Demystifying Support Vector Machines | by Dhairya Kumar | Towards Data Science

This is my second article of the Demystifying series. You can check out the first article here —
towardsdatascience.com
In this article, I will focus on Support Vector Machines or SVM. It is one of the most popular machine learning algorithms and it enjoyed its Numero Uno status for almost a decade (the early 1990s to 2000s). However, it is still a very basic and important algorithm which you should definitely have in your arsenal. So let’s start with SVM.
The topics that I will cover in this article are as follows
SVM Introduction
Geometric Intuition
Why we use +1 and -1 for support vector planes
Loss Function
Dual form of SVM
Kernel and its types
nu-SVM
Support Vector Machine (SVM) is a machine learning algorithm that can be used for both classification and regression problems. However, it is mostly used in classification problems. In this algorithm, we plot each data item as a point in n-dimensional space (where n is the number of features you have). Then, we perform classification by finding the hyperplane that best differentiates the two classes.
If you have a n-dimensional space, then the dimension of the hyperplane will be n-1.
Ex — If you have a 2d space then it will be a line, If you have a 3d space then it will be a plane and so on.
The main idea behind SVM is to find a plane that best separates the positive and negative points and the distance between the positive plane and the negative plane is maximum.
The rationale behind choosing the decision boundaries with larger margins is that it reduces the generalisation error and the decision boundaries with smaller margins usually lead to overfitting.
Another important thing that you may have noted from the above image is that support vectors are basically the points that lie on the positive and negative hyperplane.
Now let’s take a closer look at how do we actually maximise the margin.
Let’s consider the following equations —
These are just the equations of the positive and negative hyperplanes. I have only added ‘b’ to this equation which is the y-intercept. If you are wondering why have I taken the values of positive and negative hyperplanes as +1 and -1 respectively then just hold that thought for a while as I will explain in detail the reason for doing so, later in the article.
After subtracting equations (1) and (2) , we get —
We can normalise this equation by the length of the vector w, which is defined as follows
So our final equation is —
The left side of this equation is basically the distance between positive and negative hyperplane, which is actually the margin that we want to maximize.
So our optimisation function in case of SVM is —
argmax( 2 / ||W||) for all isuch that Yi(W^T * Xi+b) >= 1
Here Yi(W^T * Xi+b) >= 1 implies that the points are perfectly linearly separable. So all the positive points lie on the positive side of the plane and all the negative points lie on the negative side
Now the problem with this approach is that you will almost never find a dataset in real life which is perfectly linearly separable. The approach that we followed is called the Hard-Margin SVM and it is rarely used in real life. So in order to use the SVM in real-world applications, a modified version was created, called the Soft- Margin SVM.
Let’s first look at the equation for Soft Margin SVM
If you are not able to comprehend this equation, don’t worry about it too much , I will explain each and every term. One thing that might look familiar to you is the term ||W||/2 . Earlier we were looking to maximize the term 2/||W|| but now since we inverted it, therefore we have changed argmax to argmin.
You might have already guessed that ‘n’ denotes the number of data points. So the 2 newly added terms in this equation are —
C         = It is a hyperparameterζ (Zeta)  = It denotes the distance of misclassified points 
In order to better understand the term zeta, let’s look at the following example.
Here, ★ represents positive points and ⬤ represents negative points.
As I said in the case of Hard-margin SVM, it is very rare that we find a dataset which is perfectly linearly separable and here we have a point x1 which is a positive point but it does not lie in the positive plane.
So in this particular case the distance between the point x1 and the plane Π is 0.5 (towards the negative plane).
For point x1 - Y(W^T * X + b)  = -0.5Since class label(Y) is +1 and the distance is -0.5, since it is towards the negative plane
We can re-write the above equation as follows —
Y(W^T * X + b)  = 1 - 1.5  So in general form we can write it as Y(W^T * X + b)  = 1 - ζ
So basically what ζ represent is the distance of the misclassified point from its actual plane.You can observe that the distance of x1 from positive plane Π + is 1.5 and that is exactly the value of ζ in this case.
As I mentioned above C is a hyperparameter and it can be tuned effectively to avoid overfitting and underfitting.
As C increases the tendency of the model to overfit increasesAs C decreases the tendency of the model to underfit increases
It is not necessary that we always choose +1 and -1. So let’s choose any arbitrary value of k here. The only restriction is that it should be greater than 0.
We can’t choose distinct values for our planes i.e we can’t take +k1 and -k2 as we want our positive and negative planes to be equally distant from our plane Π
Now our updated margin is -
2*k / ||W||for k = 5 we get10/||W||
So now we will use 10/||W|| instead of 2/||W|| that is the only difference and since k is a constant here therefore it doesn’t really matter what value we are choosing, as it will not affect our optimisation problem.
So we use +1 and -1 for simplifying the mathematical calculations.
The loss function used in SVM is hinge loss. In simple terms, we can understand hinge loss as a function whose value is non-zero till a certain point let’s say ‘z’ and after that point ‘z’ it is equal to zero.
We looked at the equation for Soft-Margin SVM.
Here the second term which contains ζ and C is the loss term. Now we will look at how we got this term.
Let Y(W^T * X + b) = Z   -- (i)// Here we are just substituting the value of Y(W^T * X + b) so that it is more readableSo from (i) we can say that If Z > = 1  then the point is correctly classified andIf Z < 1    then the point is misclassified 
If you didn’t understand the above substitution, then let me clarify it further.
Suppose you have 2 points x1 and x2 where x1 is positive and x2 is negative. Now for a point x2 which lies in the negative plane the value for (W^T * X + b) will be negative and its Y value will be -1.
So, Y*(W^T * X + b) = -1 * (-ve value) = +ve value
Similarly for a positive point x1, (W^T * X + b) will be positive and its Y value will also be positive. So, Y*(W^T * X + b) = +1 * (+ve value) = +ve value.
Now if you have another point x3 which is positive but lies in the negative plane then (W^T * X + b) will be negative but class label Y is still positive.
So, Y*(W^T * X + b) = +1 * (-ve value) = -ve value
So the crux here is that Y*(W^T * X + b) will only be positive if the point is correctly classified and we have just substituted Y*(W^T * X + b) as Z.
Now that you are comfortable with the concept of Z (hopefully), let’s look at our loss function.
So our loss function is fairly simple and if you are not able to understand how it works then I will break it down for you.
As explained earlier -
If Z >= 1   then the point is correctly classified andIf Z < 1    then the point is misclassified
So we will consider 2 cases here.
Case 1 — ( Z ≥1)
If Z ≥1 then 1-Z will be less than 0 so Max(0, 1-Z ) = 0
It makes sense intuitively as if Z≥1 then it mean we have correctly classified the point and therefore our loss is 0.
Case 2— ( Z < 1)
If Z <1 then 1-Z will be greater than 0 so Max(0, 1-Z ) = 1-Z
Final Step
As we already know that -
Y(W^T * X + b) = 1 — ζ (Refer to Understanding Zeta sub-section)
So we can rewrite it as -1 - Y(W^T * X + b) = ζAnd Y(W^T * X + b) = ZSo 1-Z = ζ
And from the above cases we can see that the term that we want to minimise is 1-Z.
This is exactly what we have written here. We just substituted 1-Z with ζ
The above Equation 1 that we derived is the primal form of SVM.However in order to leverage the power of kernels we use the dual form of SVMs.So let’s look at the dual form of SVM.
The reason for using the dual form of SVM is that it allows us to leverage the power of kernels which is a key feature of SVM and if you are not familiar with kernels then don’t worry about it too much, I will explain kernels in the next section. But for now just understand that we use the dual form of SVM in order to leverage the power of kernels.
It is proven mathematically that Equation 2 is equivalent to Equation 1.
The mathematical proof of how we actually got to this dual form is beyond the scope of this article as it is a bit mathematically intense. If you want to understand the maths behind it then you can checkout the following video —
In this video Prof. Patrick Henry Winston provided a brilliant mathematical explanation and I would highly suggest that you checkout this video to better understand the concept of SVMs.
The most important thing to note here is that the value of αi will be non-zero only for support vectors. So we basically only care about the support vectors.
We can update our Equation 2 as follows —
Earlier we were using Xi^T . Xj i.e. we were taking the dot product of Xi and Xj which is equivalent to Cosine similarity function. So we can just replace this cosine similarity function with any other function of Xi and Xj.This is called the kernel trick. Now let’s understand what the hell is a kernel.
In the above Equation 3, we can replace K with any kernel function. Now you must be wondering how does that change anything. Why does it even matter which function we use? So let’s try to answer those questions.
Suppose you have a dataset that is not linearly separable.
Now how would you use SVM to separate this data? There is no way that we can fit a plane that can separate these 2 classes.
Enter Kernels....
The main use of a kernel function is that it allows us to project our dataset onto higher dimension where we can fit a plane to separate our dataset.
So we can project our above dataset onto a higher dimension and then we can find a plane that can separate the 2 classes.This is exactly the reason why SVM was super popular in the early 90s.
The 2 most popular types of kernels are —
Polynomial KernelRadial Basis Function (RBF) Kernel
Polynomial Kernel
Radial Basis Function (RBF) Kernel
Polynomial Kernel —
So for a quadratic kernel we will have something like this —
RBF Kernel —
Here d is the distance between x1 and x2 i.e. d = ||x1-x2|| and σ is a hyperparameter.
nu is a hyperparameter that we can use for defining the percentage of error that is acceptable.
0 <= nu <= 1  // The value of nu is between 0 and 1Let's understand it with an example.Suppose nu = 0.01 and N (Number of data points)= 100,000* Percentage of errors <= 1%* Number of Support Vectors >= 1% of N i.e. 1000
So with the help of nu hyperparameter we can do 2 things —
We can control the error percentage for our model.
We can’t control but we can determine the number of support vectors.
And with that we have come to the end of this article. Thanks a ton for reading it.
You can clap if you want . IT’S FREE.
My LinkedIn, Twitter and GithubYou can check out my website to know more about me and my work.