Theory of Principal Component Analysis (PCA) and implementation on Python | by Jonathan Leban | Towards Data Science

When working on a complex science project with a lot of data where each example is described by many characteristics, you may want to visualize the data. In fact, visualization in 1D, 2D or 3D is easy, but if you want to visualize your data composed of 100 characteristics, you won’t see anything in 100D. So you have to reduce the dimension and place your data in a space with a dimension equal to or smaller than 3D. This can be done using Principal Component Analysis (PCA). Another very good use of PCA is to speed up the training process of your machine learning algorithm. The more features you have, the more complex your machine learning algorithm is and the more parameters it needs to learn. Thus, the longer the computation time will be. Now, it seems obvious that if you reduce the size of your data, the time needed to learn your algorithm will decrease considerably. But, you may wonder if we reduce the number of variables, then we will lose a lot of information from our data and the result will not be accurate at all. This is the main challenge of the PCA algorithm and we will see in this article how to ensure good accuracy.
To illustrate this situation, I will use the MNIST database is a large database of handwritten digits.
In this article, I will show you the theory behind PCA and how to implement it on Python. I will also give you some tips on when to use this method.
In the principal component analysis algorithm, the objective is to find the k vectors on which to project the data in order to minimize the projection error. These k vectors will be the k directions on which to project the data. Here, k corresponds to your final dimension: if you want to look at your data in a 2D dimensional space, then k will be equal to 2.
How do we find these k vectors ?
Let’s call A the matrix which describes our data. PCA involves transforming interdependent variables (called “correlated” in statistics) into new variables that are uncorrelated with each other. These variables are called principal components and will describe the information conveyed by the data. We need to look at the covariance matrix because covariance is a measure of the joint variability of two random variables. But why covariance? Let’s think about how we can learn from the data. You look at the mean and how the variables are far from or close to the mean. This is essentially covariance: the deviation from the mean. So in the PCA, we have to calculate the covariance of our data matrix and look for the directions or vectors that collect the most information so that we can keep the PI and get rid of the rest. But it’s not as simple as that and we’re going to look at it step by step to see how we do it.
The aim of this step is to standardize the range of the continuous initial variables so that each one of them contributes equally to the analysis.
More specifically, the reason why it is critical to perform standardization prior to PCA, is that the latter is quite sensitive regarding the variances of the initial variables. That is, if there are large differences between the ranges of initial variables, those variables with larger ranges will dominate over those with small ranges (For example, a variable that ranges between 0 and 100 will dominate over a variable that ranges between 0 and 1), which will lead to biased results. So, transforming the data to comparable scales can prevent this problem.
Mathematically, this can be done by subtracting the mean and dividing by the standard deviation for each value of each variable. Recall that Ais the matrix which describes our data, thus each row is an example and each column is a feature. Let’s set n equal to the number of features et m the number of examples. Thus the matrix A is a matrix mxn.
The covariance matrix is given by the following formula:
Then, we need to diagonalize the covariance matrix. We will call S the diagonal matrix. U and V will be the transformation matrices. Thus we have:
If you have heard about PCA you may have heard about SVD. SVD is for Singular Value Decomposition and it is what we are applying here on A. The SVD theory states that it exist the following decomposition for the matrix A:
where U and V are orthogonal matrices with orthonormal eigenvectors chosen from AAT and ATA respectively. S’ is a diagonal matrix with r elements equal to the root of the positive eigenvalues of XXT or XT X. The diagonal elements are composed of singular values.
And we have S’2 = S. The next step is to reorganize the eigenvectors to produce U and V. To standardize the solution, we order the eigenvectors such that vectors with higher eigenvalues come before those with smaller values.
Comparing to eigen decomposition, SVD works on non-square matrices. U and V are invertible for any matrix in SVD.
The idea of dimensionality reduction is to keep k eigen vectors which describe the best the data. Thus, we have:
Then we add null space, which is already orthogonal to first r v’s and u’s to go from reduced to full SVD.
There is a formula we need to apply to keep a certain percentage of the total variance. Let’s say we want to keep 99% of the variance then we have the following:
Now that we have seen the maths behind the PCA we will implement it on Python.
The dataset that we will use is accessible here. Thus first we import the data and the different libraries we will need.
import numpy as npimport pandas as pdimport seaborn as snsimport matplotlib.pylab as plt# import the datadata = pd.read_csv("train.csv")# some pre-processing on the datalabels = data['label'].head(1500) # we only keep the first 1500 # Drop the label feature and store the pixel data in A and we keep # only the first 1500A = data.drop("label",axis=1).head(1500)
from sklearn.preprocessing import StandardScalerstandardized_data = StandardScaler().fit_transform(A)
First, we calculate the covariance matrix of A.
covar_matrix = np.matmul(standardized_data.T , standardized_data)
Then, we calculate the eigen values and the corresponding eigen vectors. But here we only calculate the two highest ones.
# the parameter 'eigvals' is defined (low value to heigh value) # eigh function will return the eigen values in ascending ordervalues, vectors = eigh(covar_matrix, eigvals=(782,783))
Then we project the original data sample on the plane formed by the two principal eigen vectors we have just calculated.
new_coordinates = np.matmul(vectors.T, sample_data.T)
Then, we append the labels to the 2d projected data and we create a dataframe to then use seaborn and plot the labeled points:
new_coordinates = np.vstack((new_coordinates, labels)).T# creating a new data frame for plotting the labeled points.dataframe = pd.DataFrame(data=new_coordinates, columns=("1st_principal", "2nd_principal", "label"))# plotsn.FacetGrid(dataframe, hue="label", size=6).map(plt.scatter, '1st_principal', '2nd_principal').add_legend()plt.show()
There is a lot of overlapping among classes means PCA not very good for the high dimensional dataset. Very few classes can be separated but most of them are mixed. PCA is mainly used for dimensionality reduction, not for visualization. To visualize high dimension data, we mostly use T-SNE.
We can also use the PCA module in Python to do it:
# initializing the pcafrom sklearn import decompositionpca = decomposition.PCA()# configuring the parameteres# the number of components = 2pca.n_components = 2pca_data = pca.fit_transform(standardized_data)pca_data = np.vstack((pca_data.T, labels)).T# creating a new data frame which help us in plotting the result datapca_df = pd.DataFrame(data=pca_data, columns=("1st_principal", "2nd_principal", "label"))# plot the datasns.FacetGrid(pca_df, hue="label", size=6).map(plt.scatter, '1st_principal', '2nd_principal').add_legend()plt.show()
In this section we want to find the number of eigen vectors necessary to describe our data well. Thus we need to calculate the significance of each eigen value and then the cumulative variance. I define the significance of an eigen value as:
# initializing the pcafrom sklearn import decompositionpca = decomposition.PCA()# PCA for dimensionality reduction (non-visualization)pca.n_components = 784pca_data = pca.fit_transform(standardized_data)# then we calculate the significance but the absolute value is not necessary as the eigen values are already positivesignificance = pca.explained_variance_/ np.sum(pca.explained_variance_)cum_var_explained = np.cumsum(significance)# Plot the PCA spectrumplt.figure(1, figsize=(6, 4))plt.clf()plt.plot(cum_var_explained, linewidth=2)plt.axis('tight')plt.grid()plt.xlabel('n_components')plt.ylabel('Cumulative_explained_variance')plt.show()
Let’s refresh what we have seen in the first part:
Thus we plot the cumulative sum of variance with the component. Here 300 components explain almost 90% of the variance. So we can reduce the dimension according to the required variance.
PCA is a method of reducing dimensionality, but component independence can be required: Independent Component Analysis (ICA). PCA is an unsupervised linear method, which is not the case with most unsupervised techniques. Since PCA is a dimensionality reduction method, it allows the data to be projected in 1D, 2D or 3D and thus visualize the data. It is also very useful for speeding up the training process because it considerably reduces the number of features and therefore the number of parameters.
I hope you have find what you were looking for in this article and good luck for your future projects!
PS: I am currently a Master of Engineering Student at Berkeley, and if you want to discuss the topic, feel free to reach me. Here is my email.