SVM Linear Classifier in Simplest words
By: Soumo Chatterjee
Tags: Machine Learning, Classification, Svm, Gradient Descent, Statistics

Through this blog I just want to make my readers understand how a SVM classifier works in simplest way possible.

Thereby, Support Vector Machine(SVM) is being used for separating data points of different classes by drawing a decision boundary between the classes consisting of various data points represented by each of the class. The decision boundary which is created between the sets of data points is known as hyperplane and SVM helps to create it.

The problem statements where we can use SVM : —

Clustering(grouping data points to various cluster in n-dimensional space)

Time Series Detection

Regression (e.g. Stock price prediction)

Outlier Detection (detecting presence of abnormal data points present in group of similar data points)

Classification

SVM is great where we have small dataset with about 1000 data points for discriminative classifier.

The way we build the hyperplane or, the decision boundary between the two classes in the above figure is by maximizing the margin i.e. the space between that line and both sets of classes means that points in each of the classes that are closest to the decision boundary. These points are called Support Vectors as indicated. They are so called because they support during the creation of the hyperplane.

Hyperplane that is to be created will always be of n-1 dimension where n is the set of features(columns) in the dataset, e.g. while in a 2D-Graph for separating the data points our hyperplane will be 2–1 =1 dimension i.e. a line. Similarly, for 3D-graph hyperplane will be 3–1 = 2 dimension hyperplane.

In SVM problems we use Hinge Loss function for maximum margin classification.

Hinge loss function

Where , x is the set of iterating data points

y is set of actual output corresponding to our x data points\

f(x) is the predictive output

+sign indicates as we keep on adding loss ,if we get negative value after summing up (1 — y*f(x)) then we can keep it as it is else we will make it 0.

Now we will define our objective function (like y = mx+c in linear regression) for our SVM class which is defined as :

SVM objective function

where , min λ ||w||^2 is known as Regularizer. Regularizer is a tunning knob. It tells us how best to fit our data. So if the regularizer term is too high the our model may overfit to our data points and it can not be generalized to our new data points, or if it’s value is too low then it will be going to underfit and it will never converge . So, we need a perfect regularizer term for our model to be as generalized as possible and fit to our training data. The next term in the objective function is the loss function that will generate the total loss.

Thereby, we are going to optimize this objective function consisting of both regularizer as well as loss function terms. And the way we are going to achieve this is by using Gradient Descent optimizer. So we will be doing partial derivatives for our both of the terms for optimizing our objective function during the training process.

partial derivatives for regularizer and loss function terms in objective function

Now during the iteration of each training process we will be updating the value of w (weight) according to two cases:

1 — miss-classification(not correctly classified) case when the value of total loss is less than 1 then w = w +η(yi*xi -2*λ*w) .

2 — correctly classified(value of total loss is zero) then the value our weights will be updated as w = w + η( -2*λ*w) because we don’t need to update our total loss here as we have classified it correctly.

Here , η is another tuning knob for how fast model can learn in the process.So , if η(learning rate) is too high then it can overshoot and skip the minimum loss entirely and if its too low it can take way too long to reach the minimum loss and will never converge. So we need to set for an optimal value for our η(learning rate)

One thing we should also keep in mind that the regularizer term ||λ||² = (1/(no of times we train our model(Epoch)). So, regularizer term keeps on decreasing as the no. of epochs increases.

Thus our error or total loss(hinge loss) will decrease overtime by optimizing our objective function. Thereby, maximizing the space between the data points belonging to different classes. So that we can draw a optimal hyperplane and we can use the regularizer term to find its optimal value by including it in the objective function so that our model will be best fit to our training and testing data. This is how SVM model works.

The loss we obtain at each step of our training process(epoch) is called Gradient. Gradient is the vector whose component consists of derivatives. So in all differential calculas we have derivatives. The reason for why we take derivatives to minimize the loss at each step of our training process because we want to know the rate of change in which direction our the gradient is moving. So we can also say that Gradient is a vector whose components consists of partial derivatives of whatever co-efficient or whatever function that we are approximating.

So, I hope my readers have got some intuition about how SVM works. If you are facing issues, please let me know in comments. Until Then Enjoy learning.

Credits — SIRAJ RAVAL (you tube video on SVM classifier)