Support Vector Machines under the hood
By: Christos Theodoropoulos
Tags: Getting Started, Support Vector Machine, Editors Pick, Artificial Intelligence, Data Science

Support Vector Machines under the hood

1.1 A simple example: two Gaussians

An artificial dataset is constructed from two classes (1, -1 labels) of Gaussians with the same covariance matrices. “For equal covariance matrices, the decision boundary becomes linear, for the distributions having a small overlap as well as a large overlap. The fact that a linear decision boundary is optimal in the case of overlapping distributions means in fact that one has to tolerate misclassifications. [1]” The tolerance of misclassification is important in order to avoid overfitting. A good classifier should be able to generalize well in unseen data. Defining classification optimality depends on the classification task. However, in general, an optimal classifier minimizes the misclassification error / the probability of misclassification. Based on Bayesian theory the maximization of the posterior class probability leads to minimization of the misclassification probability.

The Bayes rule is the following:

Where P(Ci|x) denotes the posterior class probability of class Ci, P(x|Ci) the class conditional density (likelihood), P(Ci) the prior probability of class Ci (prior belief), P(x) the unconditional density (evidence) and n is the number of different classes (2 in the toy example of this section).

The goal is the maximization of the posterior probability:

Figure 1 — Optimal boundary (line) based on misclassification criterion (Image by author)

Figure 2 — “A minimal shaded area corresponds the probability of misclassification (right figure), when considering a moving threshold line. [1]”

Figure 1 presents the line which is used to classify the two classes. There is overlapping between the classes, so it is not possible to avoid misclassifications using a linear decision boundary. Two red points and three blue points are classified wrongly. A graphical interpretation of the minimization of misclassification probability, with the optimal definition of the decision boundary for the binary classification problem, is presented in Figure 2. The same logic applies in the example of this section, where the definition of the dashed line (Figure 2) in the intersection of the two distribution probabilities results in the optimal decision boundary.

1.2 Support vector machine classifier

The classification tasks can be divided into linear and non-linear based on the possible separability of the classes. Figure 3 shows that there are different separating hyperplanes in linear separable cases.

Figure 3 — Different separating hyperplanes [1]

A formulation for defining the optimal separating hyperplane is introduced by the notion of margin (figure 4). The idea is to maximize the distance to the nearest points of each different class which is equivalent to the minimization of the term ||w||2. The resulted classifier is:

Figure 4 — The definition of margin [1]

The idea of margin maximization is well formulated from SVM classifier model. Each SVM model has two equivalent representations, the primal and the dual. The unique solution follows from a convex optimization problem that is actually one of the main advantages of SVM method in contrast to other methods that solve a non-convex optimization problem with many local minima like artificial neural networks. The primal problem of the SVM classifier is formulated as follows:

Where w is a parameter vector and b the bias term. The Lagrangian for the primal problem is:

Where ak are the Lagrange multipliers. The solution is characterized by the saddle point of the Lagrangian and the resulted classifier is the following:

By replacing the w based on the expression in the saddle point, in Lagrangian (1.5) the dual representation (Quadratic Programming problem) is formulated:

The problem formulation of the SVM method is based on reference [1]. An important property of the dual representation is that the solution is sparse. Many elements in the vector a which contains the Lagrange multipliers are zero. The non-zero values are the support values and the support vectors are the training data which correspond to these non-zero values.

The previously described formulation requires that every training sample is classified correctly. However, in most of real-life applications, this is not possible. In order to tackle this problem, some tolerance in misclassification should be introduced. This is illustrated through the addition of slack variables ξκ in the primal representation:

The dual representation is obtained by using the Lagrangian as described before. After the introduction of the slack variables the object function has two terms. The first term is related to the maximization of the margin and it is similar to regularization term (weight decay in neural networks) and the second is related to the tolerance of misclassification. Fine-tuning of parameter c is very important to achieve the best model performance.

So far, the analysis assumed that the task is linearly separable. However, in many cases, this assumption does not hold. SVM method is introducing the idea of feature mapping and kernel trick in order to solve non-linearly separable problems. A feature map function φ(x) is used to map the initial input space (training data) to a high dimensional space where the classes are linearly separable. This technique results in the classifier:

The dual representation is obtained by using the kernel trick. Based on Mercer theorem, a positive definite kernel can be chosen such that:

Hence, the optimization problem is formulated as follows:

By defining the Lagrangian and finding the solution in the saddle the dual problem is formulated:

With the obtained classifier:

Different definite positive kernels can be used. In this section, linear and RBF kernels are investigated.