VC: High Dimensional PCA and Kernel PCA
Tags: Visualization, Data Science, Bioinformatics, Principal Component, Dimensionality Reduction

This post assumes you already know what PCA is. If you don’t please check my previous post.

n < p case (High dimensional PCA)

If the feature space is bigger than the number of data points, our rank is determined by n-1 because it is centred by the mean. Therefore, the degree of freedom is reduced, we can calculate the last one by the previous n-1 data points because of the mean.

We will use one fact to efficiently calculate the high dimensional PCA. All eigenvectors of S are in the span of z(the centred vector of the original data).

The proof of z span u.

This proof starts with the eigenvalue decomposition of the scatter matrix. We can think of the inner product of S and u as the sum of vector inner products. We got u is some scalar product of z. The scalar coefficient consists of eigenvalue, eigenvector, and z transpose. We can reduce the computation with this fact from p x p eigenvalue problem to n x n eigenvalue problem.

This process the calculation down to just eigenvalue decomposition of the K matrix.

We start with the eigenvalue decomposition of the scatter matrix S and it becomes the eigenvalue decomposition of matrix K because its vector product is different, it is from the different index of the vectors. We assume the eigenvector of S is normalized and we easily calculate the alpha, this is what we want to get. The alpha is the coefficient of z spanning u.

This is from u*u transpose = 1.

Alpha can be calculated with this equation

The dot product of u will be 1 and we can calculate the alpha with this fact as you can see above. The eigenvalue of K must be plus because we are going to use the square root.

The final equation. this is a simple equation to calculate.

We change the p x p calculation into n x n, this means we change the subjects of eigenvalue decomposition from the scatter matrix to the K matrix.

Kernel PCA

Make a high-dimensional data and PCA on that space.

The purpose of Kernel PCA is to overcome the limitation of PCA, it only considers the variation from a linear relationship. To preserve the nonlinear structure when we apply PCA on our data needs to use the kernel trick. The core idea is it nonlinearly maps to a higher-dimensional feature space, apply PCA there.

Kernel Trick

To apply PCA in feature space, we do not explicitly need the feature map, only a nonlinear kernel function is needed. Here are widely used examples:

Example of Kernels

Unfortunately, we need centering when we apply PCA. In this case, we easily apply the previous method (just subtracting the mean) because it is a non-linear relationship.

We are trying to get the matrix K like the normal PCA.

If we get the matrix K centered, then it is easy to calculate the alpha, the coefficient of z(the centered original data points). As you can see above, we can get centered K from just normal K and there is a more easy way to calculate the centered K because we know the centered K consists of normal K.

The centered matrix can be calculated with H and normal K.

The proof to calculate the above equation is easy and we will skip this and the algorithm is the same with previous PCA because we know the centered K.

Algorithm

Input: features and kernel function

Output: Embedded value

k is the kernel and it is centered and it is normalized by D

Compute kernel matrix K. Compute centered K by HKH. Compute spectral decomposition centered K = UΛU.transpose, sort the eigenvalue and eigenvectors in decreasing order of eigenvalues We select the desired dimension and D is the squared root of Λ only including desired dimensions.

You can compare this result and the result above. Actually, it is really similar to each other.

Advantages

Better reflects nonlinear structures. Given suitable kernels, can be applied to more abstract objects or to unfold manifolds.

Disadvantages

Less interpretable because principal modes are no longer a fixed combination of input variables Unlike with linear PCA, it is not easy to find a vector that corresponds to given Kernel PCA coordinates.

Example

The sensitivity plot [Reverter et al., 2012]

The graph is from Kernel PCA of gene expression data and the kernel is RBF kernel. The arrow indicates how much each point is changed when the single gene’s expression changes. As you can see, T-cell changes a lot but another cell dose not change much.

Actual data

This can be detected in other experiments. Therefore, Kernel PCA can represent its data well.

The next post will be MDS and ISOMAP.

This post is published on 9/5/2020.

This post is edited on 9/8/2020.