Kernel Functions in Non-linear Classification | by Edwin Tai | Towards Data Science

Once the data points are non-linear separable in their original feature space, the linear classifier may be failed to determine where the decision boundary is. However, mapping the original feature space (x ∈ Rd) into the higher dimensional feature space (φ(x) ∈ Re , e>d) can help to resurrect the linear classifier to do the job correctly.
Figure 1. illustrates the concepts of classifying data points through feature mapping. Originally, the data points with the feature vectors x = [x1, x2] in the 2-D space have the concentrically circular distribution (not a strictly mathematical description here). It is impossible to use a linear classifier to distinguish the decision boundary.
Nonetheless, by incorporating a certain mapping function φ(x), the feature vectors can be transformed into 3-D feature space. The new data points with 3-D feature vectors φ(x) = [x1, x2,(x12+x22)] can now be using the linear classifier to determine the decision boundary hyperplane. This is the power of feature mapping that can allow us to deal with the more complex data distribution pattern with more expressive ability.
However, the drawbacks of using φ(x) directly are that
It is sometimes hard to explicitly construct a φ(x) directly.Increase computational power quickly with the increased feature dimensions.
It is sometimes hard to explicitly construct a φ(x) directly.
Increase computational power quickly with the increased feature dimensions.
But the kernel functions can provide an efficient way to solve this.
The idea of kernel functions is to take the inner products between two feature vectors, and evaluate inner products is not computationally costly. We can then exploit only the result of the inner products in our algorithms. For example, if we want to have the φ(x) as follows,
The kernel function is take the inner products between two feature vectors as follows,
As a result, the form of the kernel functions will be easier for us to construct than directly use the mapping functions in the higher feature dimensions.
There are several kernel composition rules that can be used to construct more complex kernel functions.
The kernel functions can even empower the feature vectors to be infinite dimensional. One of the common kernel functions is the radial basis kernel. The definition is as follows.
Because the exponential can be expanded to the infinite power series, the radial basis kernel gives much more expressiveness to the feature mapping. The following is the proof of the radial basis kernel that is a kernel function.
Recalling the perceptron algorithm here, the perceptron algorithm updates θ = θ + y( j ) x( j ) once a data point is misclassified. In the other word, the θ can be expressed alternatively as follows.
where αj is the number of mistakes the perceptron made on the j-th data point. If it is in the mapping feature space, the θ can be expressed as follows.
Utilizing the kernel form of the θ, the pseudocode of the kernel perceptron algorithm can be described as follows.
# Kernel Perceptron Algorithm# initialize αj# totally m data pointsfor i = 1 .. m do    αi = 0# totally T epoches to iteratefor t = 1 .. T do                         # totally m data points        for i = 1 .. m do        # misclassify data points                         if y(i)∑j(αjy(j)K(x(j),x(i))) ≦ 0             then            αi = αi + 1θφ(x(i))= ∑j(αjy(j)K(x(j),x(i)))
Figure 2. visualizes the updating of the decision boundary by the radial basis kernel perceptron algorithm. Note that the decision boundary drawn by the radial basis kernel perceptron algorithm can converge in 2 epoches with this kind of data distribution. The γ for the radial basis kernel uses 0.3 here.
The sample code written in Jupyter notebook for the perceptron algorithms can be found here. You can play with the data and the hyperparameters yourself to see how the kernel perceptron algorithm performs.