Introducing Python Package CVXOPT: Implementing SVM from Scratch
By: Zijing Zhu
Tags: Quadratic Equation, Numpy, Python, Linear Algebra, Support Vector Machine

CVXOPT is a free python package that is widely used in solving the convex optimization problem. In this article, I will first introduce the use of CVXOPT in quadratic programming, and then discuss its application in implementing Support Vector Machine (SVM) by solving the dual optimization problem.

How to use CVXOPT to solve an optimization problem

To understand how to use CVXOPT, we need to know its standard form and apply it to each individual question. According to CVXOPT API, we can solve the optimization problem in this form:

standard form

It is solving a minimization problem, with two types of linear constraints. One is an inequality constraint, and another is an equality constraint. To use the package to solve for the best x, that minimizing the object function, under the linear constraints, we just need to transform the specific question to identify matrics P, q, G, h, A, b.

Let’s take a simple example from here:

an example

In this example, we have two variables we need to solve for optimization, which is x1 and x2. First, look at the objective function 2x1² +x2² +x1x2+x1+x2, we can rewrite it as its matrix form:

P is the matrix that captures the two-dimensions’ coefficients. Thus we look at the coefficients of x1², x2², and x1x2 to construct P. Taking into consideration the 1/2 in front, P is:

And q is the matrix for x1 and x2, which is:

The next step is to build the linear constraints, lets start with the inequality to find out G and h. The standard form has equality with the less-than sign, while in the question, we have the larger-than sign. Thus, we need to transform the question by multiply negative one on both sides, we will get:

-x1 ≤ 0

-x2 ≤ 0

The corresponding G and h is:

And A, b is straightforward:

A is a diagonal matrix and b is a scalar.

With this example, I illustrate how we can transform a practical question to match the standard form of the CVXOPT package and find all the matrice to solve the optimization problem.

The application on SVM

One application of using the CVXOPT package from python is to implement SVM from scratch. Support Vector Machine is a supervised machine learning algorithm that is usually used for binary classification problems, although it is also possible to use it to solve multi-classification problems and regression problems. We define the cost function of the soft margin binary class linear SVM as:

This is the primal problem. The soft margin means that we are allowing some support vectors to across the hyperplane and be assigned to the wrong classes when finding the maximal margin.

The support vectors that cross the hyperplane is called slacks. C is a constant, which is a hyper-parameter that defines the “cost” of the slacks. When C is small, it is efficient to allow more points into the margin to achieve a larger margin. Larger C will produce boundaries with fewer support vectors. By increasing the number of support vectors, SVM reduces its variance since it depends less on any individual observation. Reducing variance makes the model more generalized. Thus, decreasing C will increase the number of support vectors and reduce over-fitting.

In solving the primal problem, we are minimizing the cost function regarding both w and b. We can rewrite the constrained optimization problem as the primal Lagrangian function with Lagrange multipliers \alpha_i ≥0 and \mu_i ≥ 0 for our two constraints, and get the following:

Instead of minimizing over w and b, subject to constraints involving \alpha, we can maximize over \alpha subject to the relations obtained previously for w and b. This is called the dual Lagrangian formulation:

the dual problem for SVM

Where x are the features and y is the target value. y is defined as 1 for the positive class and -1 for the negative class. In this article, we will show the soft margin implementation of binary class linear SVM by solving the dual problem.

First, we need to rewrite the objective function from maximizing to minimizing and rewrite the linear constraints to fit the CVXOPT package. Let’s define a matrix H, which equals to:

We can rewrite the optimizing problem as:

times -1:

From here, it is clear what are P, q, G, h, A, and b. Suppose we have m features, and n observations. P is the same as H with size equals to m*m; q is a m*1 vector of -1s; G vertically combines two m*m diagonal matrics. The top diagonal value is -1 and the bottom diagonal value is 1. h is a 2m*1 vector with m zeros on the top, and m Cs on the bottom. A is the target value vector y transposed and b is a scaler that equals to 0. Here is a python implementation using Numpy and CVXOPT. We can find more details on this website.

Furthermore, we can check the accuracy of the implementation with this function here:

Hope this article helps you understand the implementation of CVXOPT and SVM. If you are interested, you can use some dataset to test the SVM implementation here and compare it with the scikit-learn package from Python.

Thank you for reading! Here is the list of all my blog posts. Check them out if you are interested!

Python In Plain English

Did you know that we have three publications and a YouTube channel? Find links to everything at plainenglish.io!