Support Vector Machine
By: Dark Coder
Tags: Regression, Support Vector Machine, Machine Learning, Classification, Svm

Detailed explanation with theory and examples with code

Support Vector Machine (SVM) is a supervised machine learning algorithm which is mostly used for classification tasks. It is suitable for regression tasks as well.

Supervised learning algorithms try to predict a target (dependent variable) using features (independent variables). Depending on the characteristics of the target variable, it can be a classification (discrete target variable) or a regression (continuous target variable) task. Prediction is done with a mapping function that maps independent variables to the dependent variable. The mapping function for SVM is a decision boundary that makes the distinction between two or more classes. How to draw or determine the decision boundary is the most critical part of SVM algorithms.

Before creating the decision boundary, each observation (or data point) is plotted in n-dimensional space. “n” is the number of features used. For instance, if we use “length” and “width” to classify different “cells”, observations are plotted in a 2-dimensional space and the decision boundary is a line.

SVM in a 2-dimensional space

This is a very simple case just to illustrate the idea of SVM. It is highly unlikely that you will encounter a task like this one in real life. A decision boundary is a line in this case. If we use 3 features, the decision boundary is a plane in 3-dimensional space. If we use more than 3 features, the decision boundary becomes a hyperplane which is really hard to visualize.

Support Vectors

The main motivation is to draw a decision boundary that maximizes the distance to support vectors. The decision boundary is drawn in a way that the distance to support vectors is maximized. If the decision boundary is too close to a support vector, it will be highly sensitive to noises and not generalize well. Even very small changes in independent variables may cause a misclassification.

Bad choices for Decision Boundary

The data points are not always linearly separable like in the figure above. In these cases, SVM uses the kernel trick which measures the similarity (or closeness) of data points in a higher-dimensional space in order to make them linearly separable. It is easier to understand with the figures below. As you can see, two different classes in 2-dimensional space cannot be separated with a line. Using a kernel trick makes them linearly separable:

Figure source

Figure source

Before I go into detail about the kernel trick, I would like to mention the c parameter and a concept called “soft margin”.

Soft margin SVM and C parameter

Real data is noisy and not linearly separable in most cases. A standard SVM tries to separate all positive and negative examples (i.e. two different classes) and does not allow any points to be misclassified. This results in an overfit model or, in some cases, a decision boundary cannot be found with a standard SVM. An overfit SVM achieves high accuracy with the training set but will not perform well on new, previously unseen examples. To overcome this issue, in 1995, Cortes and Vapnik, came up with the idea of “soft margin” SVM which allows some examples to be misclassified or be on the wrong side of the decision boundary. Soft margin SVM often results in a better-generalized model. When determining the decision boundary, a soft margin SVM tries to solve an optimization problem with the following goals:

Increase the distance of decision boundary to classes (or support vectors)

Maximize the number of points that are correctly classified in the training set

There is obviously a trade-off between these two goals. Decision boundary might have to be very close to one particular class to correctly label all data points in the training set. However, in this case, accuracy on the test dataset might be lower because the decision boundary is too sensitive to noise and to small changes in the independent variables. On the other hand, a decision boundary might be placed as far as possible for each class with the expense of some misclassified exceptions. This trade-off is controlled by the c parameter.

C parameter adds a penalty for each misclassified data point. If c is small, the penalty for misclassified points is low so a decision boundary with a large margin is chosen at the expense of a greater number of misclassifications. If c is large, SVM tries to minimize the number of misclassified examples due to high penalty which results in a decision boundary with a smaller margin. The penalty is not the same for all misclassified examples. It is directly proportional to the distance to the decision boundary.

Kernel functions

The kernel function is kind of a similarity measure. The inputs are original features and the output is a similarity measure in the new feature space. The similarity here means a degree of closeness. It is a costly operation to actually transform data points into a high-dimensional feature space. The algorithm does not actually transform the data points to a new, high dimensional feature space. Kernelized SVM computes decision boundaries in terms of similarity measures in a high-dimensional feature space without actually doing a transformation. I think this is why it is also called a kernel trick.

Figure source

Kernel functions available in scikit-learn are linear, polynomial, radial basis function (RBF), and sigmoid.

RBF (or Gaussian RBF) is a widely-used kernel function. It is defined by the gamma parameter which basically controls the distance of influence of a single training point. Low values of gamma indicate a large similarity radius which results in more points being grouped together. For high values of gamma, the points need to be very close to each other in order to be considered in the same group (or class). Therefore, models with very large gamma values tend to overfit. Following visualizations explain the concept better:

Figures source

As the gamma decreases, the regions separating different classes get more generalized. Very large gamma values result in too specific class regions (overfitting).

Gamma vs C parameter

For a linear kernel, we just need to optimize the c parameter. However, if we want to use an RBF kernel, both c and gamma parameters need to optimized simultaneously. If gamma is large, the effect of c becomes negligible. If gamma is small, c affects the model just like how it affects a linear model. Typical values for c and gamma are as follows. However, specific optimal values may exist depending on the application:

0.0001 < gamma < 10

0.1 < c < 100

It is very significant to remember for SVM that the input data need to be normalized so that features are on the same scale and compatible.

Pros and Cons of SVM

Pros:

Effective in cases where the number of dimensions is more than the number of samples

When finding the decision boundary, it uses a subset of training points rather than all points so it is memory efficient

Versatile, it provides a different type of kernel functions and also possible to create custom kernel functions

Cons:

Requires careful normalization

Difficult to understand the logic behind the predictions

When we have a large dataset, training time increases which negatively affects the performance

Scikit-learn Example

I want to go over a simple case just to show the syntax and steps to follow. Let’s start with importing the libraries we need:

Scikit-learn also provides datasets for practice. The breast cancer dataset is suitable for our task:

We need to split the dataset for training and testing using the train_test_split function of scikit-learn. Then I create a support vector machine object for classification with linear kernel and train it using train dataset:

Using our trained model, we can predict the target variable in the test dataset and measure the accuracy of the model:

95% is a pretty good accuracy but, for classification tasks, there are two other measures: precision and recall. These terms are so important that they can be a topic of whole another post. In fact, they are clearly explained in Google’s machine learning crash course.

This was a simple linearly separable task. For cases that are not linearly separable, we can use kernels using the kernel parameter of SVC (i.e. kernel = ‘rbf’ ). I did not specify the c parameter so the default value (c=1) was used. In order to find the best parameters, we can use the GridSearchCV() function of scikit-learn.