The Magic Trick of Machine Learning — The Kernel Trick
By: Sachin Kumar
Tags: Kernel Trick, Big Data, Blog Post, Data Science, Linear Algebra

Welcome to our machine learning magic show!

You must have heard that magicians never reveal their secrets. Machine learning has several tricks that seem like magic. In this blog, we hope to reveal one secret of machine learning called the Kernel Trick. If you have encountered the problem of datasets too big to use your models on, models too complex to use your data on, or nonlinear data, then this trick will be useful for you. When used wisely, this trick can speed up algorithms, reduce memory usage, and uncover hidden patterns.

Sounds exciting and want to know more? Focus, otherwise the magic trick might elude you.

Intuition: Why is it needed?

Let’s start with a summary of the notation used and the concepts we already know.

Summary of Notations

L2 Regularized Regression

Consider the least-squares regularized objective function where we want to learn weights that minimize the following expression:

Cost (objective) function of L2 regularized least squares

Equivalently, this can be written in matrix form as:

Recalling from calculus I, to minimize an expression f(x), we need to set its derivative to zero f´ (x) = 0 and solve for x. The equivalent multidimensional case x is to calculate the gradient of the function and set it to zero vector ∇f(x) = 0 and solve the linear system of equations. For the least-squares objective function (1), the solution in matrix form would be:

The gradient of cost function and normal equations

which is known as the “normal equations”. The solution to the normal equations is:

The solution to normal equations

Once the optimal weights (w) are obtained, to make predictions on the test data Xtest, we need to calculate y = Xtest · w.

If you are not convinced, check the dimensions of the matrix operations to see that it is a sound derivation.

Check the dimensions of the matrix operations

What is the cost of computing linear regression?

Let us analyze the cost of solving the normal equations and obtaining w. For this, let us first consider the cost of the dot product of two vectors u and v. The dot product of two vectors is defined as:

dot products

As we can see, the pattern is to multiply each element of the vector and then add them. So, a dot product of an n-dimensional vector requires n multiplications and n-1 additions. Assuming the cost of multiplication and addition are constant, the total number of operations in computing the dot product is n+n-1. In big-O notation, this is O(n).

Training (learning weights w)

To learn weights w, we need to perform matrix operations. Below is the breakdown of each matrix operation involved in computing w.

–– Multiplication: Xᵀ X

The result of the matrix multiplication Xᵀ X is a d×d matrix. There will be a total of d² elements in this matrix. Each element of the resulting matrix is a dot product of two n-dimensional vectors. So, the cost of forming Xᵀ X is O(d²·n).

–– Addition: Xᵀ X+λI = B

Matrix addition does not change the size of the matrix; thus, the result will be d×d matrix. One element of the first matrix will be added to one element of the second matrix, a total of d² summations. So, the cost of adding two matrices is O(d²).

–– Matrix inversion: (Xᵀ X +λI)⁻ ¹ = B⁻ ¹ = C

The basic matrix inversion algorithm is Gauss-Jordan elimination (row reduction). Solving a d×d system of equations via Gauss-Jordan elimination is known to be O(d³).

–– Multiplication : (Xᵀ X +λI)⁻ ¹ Xᵀ = C Xᵀ = D

Similar to multiplying Xᵀ X, calculating C Xᵀ is the result of (d×d)(d×n) matrices which yields a d×n matrix whose complexity is O(dn·d) or O(d²n).

–– Multiplication : (Xᵀ X +λI)⁻ ¹ Xᵀy = Dy = w

Similar to multiplying Xᵀ X, calculating Dy is the result of (d×n)(n×1) matrices which yields a d×1 vector whose complexity is O(d·n).

–– Adding it all up the cost of all the operation yields,

Which simplifies to,

The complexity of regularized linear regression

As can be observed, the complexity depends on n and d. Let’s analyze the two scenarios.

If n>>d, this means that our data (matrix) has lots of examples (rows). In this case, we have lots of data points but few features. The number of examples dominates; therefore, the cost of performing least-squares is linear in the number of examples.

Not bad if we have tens or hundreds of examples. But what if we have 100,000? What about millions? What about billions? It’s not bad either since the computation cost is O(n), which is about the same as if we were to read n data points.

If d>>n, this means that our data (matrix) has lots of features (columns). In this case, we have lots of features but few examples. The number of features dominates; therefore, the cost of performing least-squares is cubic in the number of features.

Even with a few features, the computation cost is high. What if we have 1000 features? What about 100,000? What about millions? It would be highly infeasible with a cost of O(d³)!

But that’s not all folks! In computing the weights w, we have to form XᵀX, which will result in a (d×d) matrix. Can we even store in memory such a large matrix if d is very, very large?