Algorithms From Scratch: Support Vector Machines | by Kurtis Pykes | Towards Data Science

A popular algorithm that is capable of performing linear or non-linear classification and regression, Support Vector Machines were the talk of the town before the rise of deep learning due to the exciting kernel trick — If the terminology makes no sense to you right now don’t worry about it. By the end of this post you’ll have an good understanding about the intuition of SVMs, what is happening under the hood of linear SVMs, and how to implement one in Python.
To see the full Algorithms from Scratch Series click on the link below.
towardsdatascience.com
In classification problems the objective of the SVM is to fit the largest possible margin between the 2 classes. On the contrary, regression task flips the objective of classification task and attempts to fit as many instances as possible within the margin — We will first focus on classification.
If we focus solely on the extremes of the data (the observations that are on the edges of the cluster) and we define a threshold to be the mid-point between the two extremes, we are left with a margin that we use to sepereate the two classes — this is often referred to as a hyperplane. When we apply a threshold that gives us the largest margin (meaning that we are strict to ensure that no instances land within the margin) to make classifications this is called Hard Margin Classification (some text refer to this as Maximal Margin Classification).
When detailing hard margin classification it always helps to see what is happening visually, hence Figure 2 is an example of a hard margin classification. To do this we will use the iris dataset from scikit-learn and utility function plot_svm() which you can find when you access the full code on github — link below.
github.com
Note: This story was written straight from jupyter notebooks using python package jupyter_to_medium — for more information on this package click here — and the committed version on github is a first draft hence you may notice some alterations to this post.
import pandas as pd import numpy as np from sklearn.svm import LinearSVCfrom sklearn.preprocessing import StandardScalerfrom sklearn.datasets import load_irisimport matplotlib.pyplot as plt %matplotlib inline# store the data iris = load_iris()# convert to DataFramedf = pd.DataFrame(data=iris.data,                  columns= iris.feature_names)# store mapping of targets and target namestarget_dict = dict(zip(set(iris.target), iris.target_names))# add the target labels and the feature namesdf["target"] = iris.targetdf["target_names"] = df.target.map(target_dict)# view the datadf.tail()
# setting X and y  X = df.query("target_names == 'setosa' or target_names == 'versicolor'").loc[:, "petal length (cm)":"petal width (cm)"] y = df.query("target_names == 'setosa' or target_names == 'versicolor'").loc[:, "target"] # fit the model with hard margin (Large C parameter)svc = LinearSVC(loss="hinge", C=1000)svc.fit(X, y)plot_svm()
Figure 2 displays how the Linear SVM uses hard margin classification to ensure that no instances falls within the margin. Although this looks good for our current scenario, we must be careful to take into account the pitfalls that come with performing hard margin classification:
Very sensitive to outliersIt only works when the classes are linearly separable
Very sensitive to outliers
It only works when the classes are linearly separable
A more flexible alternative to hard margin classification is soft margin classification which is a good solution to overcome the pitfalls listed above when doing hard margin classification — mainly with solving the issue of sensitivity to outliers. When we allow for there to be some misclassifications (meaning that some negative observations may be classified as positive and vice versa), the distance from the threshold to the observations is called soft margin. In soft margin classification we aim to achieve a good balance between maximizing the size of the margin and limiting the amount of violations in the margin (the number of observations that land in the margin).
Yes, Linear SVM classifiers (hard-margin and soft-margin) are quite efficient and work really well in many cases, but when the dataset is not linearly separable, as if often the case with many datasets, a better solution is to make use of the SVMs kernel trick (once you understand the kernel trick you may notice that it is not exclusive to SVMs). The kernel trick maps non-linearly separable data into a higher dimension then uses a hyperplane to separate the classes. What makes this trick so exciting is that the mapping of the data into higher dimensions does not actually add the new features, but we still get the same results as if we did. Since we do not have to add the new features to our data, our model is much more computationally effieicent and works ust as good.
You’ll see an example of this phenomena below.
Decision Boundary: The hyperplane that seperates the dataset into two classes
Support Vectors: The observations are at the edge of the cluster (located nearest to the seperating hyperplane).
Hard Margin: When we strictly impose that all observations do not fall within the margin
Soft Margin: When we allow for some misclassification. We seek to find a balacne of keeping the margin as large as possible and limiting the number of violations (bias/variance tradeoff)
from sklearn.datasets import make_moonsfrom mlxtend.plotting import plot_decision_regionsfrom sklearn.preprocessing import StandardScalerfrom sklearn.svm import SVC# loading the dataX, y = make_moons(noise=0.3, random_state=0)# scale featuresscaler = StandardScaler()X_scaled = scaler.fit_transform(X)# fit the model with polynomial kernelsvc_clf = SVC(kernel="poly", degree=3, C=5, coef0=1)svc_clf.fit(X_scaled, y)# plotting the decision regionsplt.figure(figsize=(10, 5))plot_decision_regions(X_scaled, y, clf=svc_clf)plt.show()
Note: We applied a polynomial kernel to this dataset, however RBF is also a very popular kernel that is applied in many Machine Learning problems and is often used as a default when data is not linearly separable.
Now that we have built up our conceptual understanding of what SVM is doing, lets understand what is happening under the hood of the model. The linear SVM classifier computes the decision function w.T * x + b and predicts the positive class for results that are positive or else it is the negative class. Training a Linear SVM classifier means finding the values w and b that make the margin as wide as possible whilst avoiding margin violations (hard margin classification) or limiting them (soft margin classification)
The slope of the decision function is equal to the norm of the weight vector hence for us to achieve the largest possible margin we want to minimize the norm of the weight vector. However, there are ways to go about this for us to achieve hard margin classification and soft margin classification.
The hard margin optimization problem is as follows:
And soft margin:
Note: For this Implementation I will be doing hard margin classification, however further work will consist of Python implementations of soft-margin and the kernel trick performed to different datasets including regression based task — to be notified of these post you can follow me on Github.
from sklearn.datasets.samples_generator import make_blobs # generating a datasetX, y = make_blobs(n_samples=50, n_features=2, centers=2, cluster_std=1.05, random_state=23)def initialize_param(X):     """    Initializing the weight vector and bias    """    _, n_features = X.shape    w = np.zeros(n_features)    b = 0     return w, bdef optimization(X, y, learning_rate=0.001, lambd=0.01, n_iters=1000):     """    finding value of w and b that make the margin as large as possible while    avoiding violations (Hard margin classification)    """    t = np.where(y <= 0, -1, 1)    w, b = initialize_param(X)        for _ in range(n_iters):         for idx, x_i in enumerate(X):             condition = t[idx] * (np.dot(x_i, w) + b) >= 1            if condition:                 w -= learning_rate * (2 * lambd * w)            else:                 w -= learning_rate * (2 * lambd *  w - np.dot(x_i, t[idx]))                b -= learning_rate * t[idx]    return w, bw, b = gradient_descent(X, y)def predict(X, w, b):    """    classify examples    """    decision = np.dot(X, w) + b    return np.sign(decision)# my implementation visualizationvisualize_svm()# convert X to DataFrame to easily copy codeX = pd.DataFrame(data=X,                 columns= ["x1", "x2"])# fit the model with hard margin (Large C parameter)svc = LinearSVC(loss="hinge", C=1000)svc.fit(X, y)# sklearn implementation visualizationplot_svm()
Very good Linear classifier because it finds the best decision boundary (In a Hard Margin Classification sense)
Easy to transform into a non-linear model
Not suited for large datasets
The SVM is quite a tricky algorithm to code and is a good reminder as to why we should be grateful for Machine Learning libraries that allow us to implement them with few lines of code. In this post I did not go into the full detail of SVMs and there are still quite a few gaps that you may want to read up on such as computing the support vector machine and empirical risk minimization.
Additionally, it may be worth watch Andrew Ng’s lectures on SVMs — Click Here
Thank you for taking the time to read through this story (as it’s called on Medium). You now have a good conceptual understanding of Support Vector Machines, what happens under the hood of a SVM and how to code a hard margin classifier in Python. If you’d like to get in contact with me, I am most accessible on LinkedIn.