Deep Dive into Support Vector Machine
By: Anupam Sharma
Tags: Support Vector Machine, Svm, Machine Learning, Data Science

Consider the above classifiers. There are two classes:

Class ‘ +1 ’: Dark data points.

’: Dark data points. Class ‘-1’: Light data points.

So, what’s wrong with the above classifiers??? If we look over the classifiers in figure 1, we cannot determine the proper regions for the two classes.The hyper plane (hyper plane is the plane that classifies the two classes. Here, the dashed line) in classifier A looks very closer to class ‘-1’. Hyper plane in classifier B looks closer to class ‘+1’. And hyper plane in classifier C looks closer to class ‘-1’. It seems that if the hyper plane is closer to a class then it is favouring that particular class more than that of the another class. And if it is so then the chance of making mistake by the classifier would be more as shown in figure 2.

Figure 2: The red coloured data point is mistakenly classified by the classifier

It can be observed that the best classifier would the one in which the hyper plane is in equal distance from the two classes (No favouritism). And it would be much better if the distance between the class and the hyper plane is maximum. And this is what happens in case of SVM as shown in figure 3. NOTE: The idea of favouritism is just for understanding the intuition. No such idea is used for formulating SVM.

Figure 3: The red coloured dashed line is the optimal hyper plane. The green coloured dashed lines define the boundary for each class. And the data points with green coloured thick outline that are on the boundary of the class are called support vectors. Hence, the name Support Vector Machine

The caption for figure 3 well describes the figure. The interesting thing about SVM is that only support vectors are considered to determine the optimal hyper plane. We will soon prove this statement.

Problem 1 : What if the data points can’t be separated by a straight hyper plane as in figure 3???

Figure 4

If we consider figure 4, no hyper plane exists in two dimensional space that can classify the data points shown in the figure. But, SVM has a solution for it that is why SVM is interesting. In such situations the given space is transformed into higher dimensional space so that the data points are separable in the new space. So, the data points are classified in the higher dimensional space and mapped back to the original space. This concept seems to be tricky. So, look over the visualisation shown below carefully.

Figure 5: Visualisation of transformation from given space to higher dimensional, classify data points in the new space and then map everything back to the original space. Source: https://www.youtube.com/watch?v=3liCbRZPrZA

The visualisation in figure 5 gives a clear idea about the transformation from given space to higher dimensional space. In the above visualisation the pink coloured hyper plane can linearly classify the data points easily in the higher dimensional space. So, data points in figure 4 can be classified as shown in figure 6.

Figure 6

Role of Kernel Functions and Problem 2

Another problem in SVM is that the process of transforming into higher dimensional space, then determining the optimal hyper plane in the new space and then transforming back to the original space is very complex and costs high overhead. Say, for example, if there are 1000 features in a 100 dimensional space and if the 100 dimensional space is transformed into 1000 dimensional space then each feature vectors will have 1000 components and 1000 * 1000 computations (so many computations because hyper plane is described as W.X + b = 0, where X is feature vector) would be required to determine the optimal hyper plane and again the hyper plane will have to be mapped back to the original space. And this whole process costs high overhead.

The solution to above problem is the kernel function. The interesting fact about kernel functions is that kernel functions does the above mapping without actually going to the the higher dimensional space. In other words the kernel functions does the above mapping without actually performing all the above computations in the higher dimensional space.

The transformation in visualisation shown in figure 5 is done using a polynomial kernel function ϕ((a, b)) = (a, b, a² + b²) as shown in figure 7.

Figure 7: Training example of SVM with kernel given by ϕ((a, b)) = (a, b, a² + b²). Source: https://en.wikipedia.org/wiki/Support_vector_machine

Please note that kernel functions is only applicable if the problem consists of dot product or inner product. And fortunately, the formulation of SVM depends on dot product (Will be proved in coming sections).