Demystifying Support Vector Machine Part I
By: Ishan Dixit
Tags: Support Vector Machine, Machine Learning, Svm, Optimization, Artificial Intelligence

Image by Author

Introduction

SVM is the initials of Support Vector Machine which is a commonly used machine learning algorithm for binary as well as multi class classification tasks and for regression tasks also.

On a surface level SVM is basically an optimization problem involving maximization of the distance between the decision boundary separating the classes. Now the question arises why should we use this instead of a neural network ,this is because SVM searches for the optimal hyperplane as contrary to neural network which returns a hyper plane which could be or couldn’t be optimal depending on the number of iterations and other hyper parameters

So the basic premise of SVM is that it uses a line in case of two dimensional space and hyperplane in case of higher dimension to separate or classify the given data.

There are two types of SVM hard margin and soft margin SVM ,

Lets first look at the hard margin SVM and then in Part II we extend it to soft margin SVM.

Hard Margin SVM

General equation of hyperplane is

Here we are assuming x is the feature vector of training data and w and b are weights and biases respectively.

Now first we find out the margin of the hyperplane using the given data. Margin of a hyperplane is the closest distance of a point from the hyperplane in the dataset

Once we have our margin the next task is the find out our w(weights) and b(bias) so that the calculated margin can be maximized. A point to note is that the value of max margin will be same on both side of the hyperplane.

Distance from a point to hyperplane is given as

Now we calculate distance for each point in the dataset and choose the minimum one.

This expression can be reformulated as

In the above minimization the variable is x and since w is independent of x so we can take it out of the minimization expression. We will use a trick to reduce our minimization problem.

We assume that

Why this assumption holds can be understood by the following example, Let us assume M turns out to be δ,therefore

Division with δ is valid since, both w and b are scaled (up or down) by dividing a non zero constant, we get the same hyperplane. Also δ doesn’t affect our optimization problem as it can be taken out of the expression because it is a constant.

So what this shows is that any margin value can be scaled to 1,hence the assumption can be made that

The next step is to maximize the calculated margin so that the classes are as far apart as possible.

This can be restated as

Now we have our optimization statement we can focus on the constraints bounding our problem. The basic thing needed to understand the constraints or the limiting conditions is that when we input a point in the equation of hyperplane the sign of the result tells us on which side does the point lie wrt to the line, given that the point is not on the hyperplane.

Alexandre Kowalczyk (2017).Support Vector Machines Succinctly

This can be formulated as -

Since our class labels(y) are either 1 or -1,we can reformulate above three equations as

We have one more constraint as

The above two equations can be combined and written as

The final expression is

The drawback of hard margin SVM is that it has no room for allowing a small amount of error resulting, even a single outlier can determine the boundary which makes the classifier overly sensitive to noise in the data.

So thats it for the hard margin SVM ,in the next part we will look at soft margin SVM as well as how the optimization algorithms are formulated and coded.

References :

[1] Machine Learning Lecture 14 “(Linear) Support Vector Machines” -Cornell CS4780 SP17

[2] Alexandre Kowalczyk (2017).Support Vector Machines Succinctly