Learn SVM 101 in 5 minutes
By: Haitian Wei
Tags: Supervised Learning, Machine Learning, Svm, Data Science

Motivation

For a long time I find myself stuck in a strange loop: I studied algorithms and spent long time on model details. I was confident that I understood the model. And after a month or two I pretty much forgot everything about the model. And I reviewed the model again and forgot it again.

This situation is most frustrating when you attend an interview. So I decided to stop and review my learning behavior. What’s wrong? How to break this loop? Then something came up. It’s the way how to draw a sketch that give me a hint.

So the way you finish a good sketch is that you first work out the outline and then gradually fill in the details like shadows, textures and so on. It should be the same way about learning a model.

The outline of a model is the key intuition behind the model and its pros and cons. How the math work out and its step by step deduction is the detail. If the outline is shaky the details won’t stand long.

SVM Key Intuition

Error Function

SVM try to find a hyperplane that best separate different classes of training data. Among the many hyperplane that have the classification error SVM find the hyperplane that have maximum margin, meaning the data points are furthest from the hyperplane.

Source: Udacity

So the error function of SVM is the classification error plus the margin error. And the parameter C controls how much we care about classification error.

Classification error of SVM is calculated as below, notice that data points fall in the margin area are also treated as error. The sum distance of error points to the separation plane is the classification error.

Source: Udacity

And this kind of loss function is called hinge loss.

We can see how it differs from other loss function from the below picture. The green line represents hinge loss.

Margin error help us to find the best plane among the many that have same classification error. It can also be treated as a regularization penalty.

The margin is defined as the distance between the separating hyperplane (decision boundary) and the training samples that are closest to this hyperplane, which are the so-called support vectors. This is illustrated in the following figure:

A large margin means the decision boundary has good generalize ability and will not affect by small changes of data.

Another way to interpret margin error is treat it as a regularization penalty. The norm two of weights meaning we prefer smaller weights than large weights.

So the classification error(hinge loss) and margin error combined give the final error function of SVM.

Kernel Trick

Another import aspect of SVM is the kernel trick. What if the data points can not separate linearly? We can lift the points to higher dimensions with a kernel and then separate the data with a hyperplane!

Below is a good visualization of how polynomial kernel works.

And there are a lot of kernel functions to choose from, some common kernel functions are shown in the picture below.

For more kernel method concept, like reproducing kernel Hilbert space (RKHS) and Mercer’s theorem, there is a good introduction article.

Pros and Cons of SVM

Model Strengths

It’s a robust model because it’s a regularized method. Parameters C control the penalty for mistakes. For large values of C, the optimization will choose a smaller-margin hyperplane if that hyperplane does a better job of getting all the training points classified correctly.

The kernel trick is real strength of SVM. With the kernel trick SVM can solve any non-linear decision boundaries.

SVM is defined by a convex optimization problem (no local minima) for which there are efficient methods (e.g. SMO)

Model Weaknesses

SVM is memory intensive because you need to store an NxN kernel matrix, which may be very large if N is large O(t2).

SVM is time consuming when the dataset is large O(t3).

SVM have difficulty implementing on multiple classification. If you insist on using SVM on multiple classification problems ,there are generally two ways: ‘one verse one’ and ‘one verse many’ method.

A common disadvantage of non-parametric techniques such as SVMs is the lack of transparency of results. SVMs cannot represent the result as a simple parametric function of the inputs, since its dimension may be very high. It is neither a linear combination of inputs nor has it another simple functional form.

Industry Usage

There are a lot of applications of SVM, like SVM practical industrial application for mechanical faults diagnostic, Text categorization with Support Vector Machines: learning with many relevant features.

Useful Link

https://www.quora.com/Why-are-SVMs-memory-intensive

https://www.quora.com/What-is-the-computational-complexity-of-an-SVM