Principal Component Analysis
By: Andrea Grianti
Tags: Machine Learning, Python, Analytics, Unsupervised Learning, Mathematics

Principal Component Analysis

Step by step intuition, mathematical principles and python code snippets behind one of the most important algorithms in unsupervised learning

This animation shows how the covariance matrix of the projected points (A…E) get diagonalized when the rotating direction reaches two special directions (being the solution to the eigen decomposition equation below). To verify it, slow down the video or try to pause it when the direction overlaps the orange dotted line (first principal component) or the second pink dotted line (second Principal componemt). You will also notice that the diagonal elements of the matrix reaches their maximum values (max variance). (Video created by the author using Geogebra6 sw)

This formula is called eigen-decomposition equation. Initial works started in the end of 1800 but only in recent times has become of practical usage in big data analysis thanks to advances in (personal) computing power.

Hi, everybody, my name is Andrea Grianti in Milan, Italy. I wrote this to share my thoughts after many books and papers read on the subject. This is not a textbook but a starting point for further understanding of the topic. As the math/algebra behind is rather tough I divided it into four parts:

PCA Intuition Math/Algebra (easy) Math/Algebra (difficult) Python snippets

1. PCA Intuition

If you have a large data sets with thousands/millions of observations (rows) and hundreds of different variables (columns) one of the first objectives is verifying if it’s possible simplifying and reducing the dataset in order to facilitate the analysis work on a much smaller subset of the original data.

Direct elimination of variables is the obvious way but it has clearly impact on the information content of your data set. Too much or a wrong eliminations and your data set becomes useless, too less and the data set remains large and difficult to analyse.

The term ‘information’ is quite a generic subject and it’s difficult to define it. It depends on the data set. A data set could contain information for me and nothing to others and vice versa.

We could try to define the quantity of information in a data set using concepts like “Information Content” which is a concept bound to the probability that a specific value, among all those possible in the context of a variable of a data set, happens.

Following this concept, as much variety of possible outcomes a variable x has, lower is the probability to predict its value and therefore higher the information content is.

This is our first assumption to keep in mind: higher variance => higher information content even if only the context and data can tell if this assumption is good or not.

As in our data set there are many variables, correlations will likely exist and if they are linear they can be considered a form of “background noise” and redundancy. As for the first assumption only the specific context can tell if linearity exists in the correlations. If the assumption holds we can then say that we should find a way to eliminate the background noise from our data.

Why these two assumptions are important ? Because to reduce the dimensionality of a data set we should evaluate the contribution of each variable to the overall variance(=information) of the dataset in order to choose those with the greatest contribution and discard those with lowest contribution.

The battlefield for this operation is the Covariance Matrix (or the Correlation Matrix which is closely linked by the relation below, but I don’t consider that here as it is not relevant in relation to the understanding the math principles behind)

Relationship between simple covariance and correlation for two variables: sigmas being their standard deviations and rho being the correlation coefficient. More the two variables x and y are uncorrelated (low rho) less covariance we have. At an extreme where correlation is zero, covariance is zero. At the other when correlation is maximum rho=1 and covariance is the product of the standard deviations of each variable

Covariance Matrix of the X original data

When we calculate the Variance/Covariance Matrix of our original data (Cx) we see the variances along the diagonal but we also see the joint covariances in the off diagonal elements which are a (difficult to interpret) measure of the magnitude of the ‘synchronicity’ between one variable and the others.

Because of course we cannot modify our original data X in order to eliminate the correlations between variables (unless we remove completely a variable which is always possible but risky because we could unwillingly remove important information), we can try to find a way to ‘transform’ X into a different data set Y having special (eigen-…) link with X but built in a way that the new variables of Y would have a different Covariance matrix Cy where variances of these variables would be “isolated” from the interefences (correlations) of the other variables (=> covariances = 0).

This is where we would like to land. With a new covariance matrix Cy which is diagonalized. Means that we are looking to solve a problem to find Y, bound to X (we’ll see how), where all the Ys are uncorrelated between them. In this way the covariances in Cy would be zero and the variances isolated. This would clearly allow us to define the total variance of the data set as the sum of diagonal elements (Trace).

This operation is the magic of the eigen decomposition equation. By solving that equation we will find a way to transform the original X data set and by consequence the covariance matrix Cx into a new dataset Y and by consequence a diagonalized covariance matrix Cy using a special transformation matrix B.

The following figure slightly change the letters of the eigen decomposition equation (the one in red) to let it be coherent with letters used in this paper. When solved we will verify our solution (in Python) using matrix form of the same equation (eq.3,4,5) .

Assuming we are able to solve that equation… so what ? If using original data X, I was forced to work with Cx with the problem of being unable to isolate single variables contributions to overall variance based on the presence of correlations, now, solving that equation, I could work with a new Y data set whose covariance matrix is Cy and it’s diagonal so the variance contribution of every variable of Y is clearly defined and not impacted from joint intereferences/correlations. With that we can later decide, based on the values of their variances, where to cut the Y data set to keep a subset of the most important Y variables based on the weight of their contribution with respect to the sum of the values of the variances in Cy.

What’s the price to pay when we work with Y instead of X? Because variables of the new Y data set (we will see this) are a linear combination of the original X data , their meaining is undefined. So to give a contextual meaning to the y variables it’s required a certain acumen to make them ‘talking’ from a practical stand point. Of course the matter is a lot complex and there are many details we skip here, but to frame the problem and solution this is it.

In any case to go from X to Y (and from Cx to Cy) we need to understand the logic of the transformation (through a still to be defined B matrix) we can do on our original data. So we need to talk (study) about projections and linear transformations as they are strictly connected.

2. Math/Algebra behind PCA (easier)

Projection concept: in short what we draw in charts depends on the system of coordinates we use to represent the data. Think if you like to perspective in art: it’s a projection of real life measures transformed according to specific rules.

In 3 dimensions for example we use the cartesian system which is represented by a matrix (E) of orthonormal vectors (orthogonal vectors of length=1).

When we have for example a data set X (measures are in 3 dimensions: column vectors x1,x2,x3), to calculate the variance of variable x1 we apply the usual formula for variance. But what we actually do is to project the coordinates of each data point of X on the 3 directions in E through a dot product.

So in general we have X･E=X. This projection operation is “transparent” with cartesian axis and we don’t even realize that as it is “automatic”.

But if we decide to move away from Cartesian system we can build a similar set of orthonormal vectors that define a new basis B made of b1, b2, b3 (instead of E made of e1, e2, e3) like this:

And if we dot multiply (for example) a generic row vector of our data set X containing the coordinates of a single data point:

generic row of data, (1,3) means 1 row, 3 columns

with B (=3 rows,3 columns) we obtain a new row vector (=1 row, 3 columns):

A generic row of Y corresponding to a generic row of X

where each element of this vector has new “measures”/”coordinates” given by:

the three equations here are the explosion of the yi equation in the previous figure. So dimensionally the shape is (1,3)

In algebra terms and in general in our case we change basis simply with:

the first is the matrix form. In our case Y is made of 3 columns vectors (y1,y2,y3 with m rows) and B is made of 3 column vectors (b1,b2,b3 with 3 rows). So Y is (m,n) = X (m,n) . B (n,n). The 3 equations, when the solution to B will be found, will represent the Principal Components.

where X is the original data matrix (m rows by n columns) in basis E , B is a new orthonormal basis (n by n),Y (m by n) is the resulting new data matrix with the measures in the new basis B.

Note that y1,y2,y3 are new variables, whose meaning is to be semantically defined as they are s linear combination of all the X rows weighted by the corresponding components of the b1,b2,b3 vectors respectively. That is:

y1 is made of every X row weighted by b1=[b11,b12,b13]

y2 is made of every X row weighted by b2=[b21,b22,b23]

y3 is made of every X row weighted by b3=[b31,b32,b33].

So with a solution for B we could work with the y’s new variables. But what B should we use to reach of our objective to find Y with a Cy diagonal matrix?

B can be built progressively starting to find the first unit vector (direction) that maximizes the variance of X projected along that direction. Once we find a max value of variance (which is the first eigenvalue lambda1) which corresponds to a specific unit vector b1 (which is the eigenvector b1) we have found the first principal component PC1: y1=X.b1

To evaluate y1 in terms of how much variance explains, at this stage you can divide lambda1 by the trace of the covariance matrix Cx.

Then with the same principle we can find the second direction b2 (second eigenvector) as the one that maximize the variance (second eigenvalue) between all the possible projections of X along a second direction of unitary length and orthogonal to b1. When found this is the second principal component: PC2: y2=X.b2

Then the third direction b3 maximize the variance of X along a third direction defined again by a unit vector that must also be orthogonal to both b2 and b1. When found this is the third principal component: PC3: y3=X.b3

At the end of these iterations we would have built a special B=[b1*,b2*,b3*] (eigenvector matrix), a new dataset Y =[y1=X.b1, y2=X.b2, y3=X.b3] made of 3 ‘principal components’ ranked by strength of variance, a special vector of eigenvalues lambdas=(lambda1, lambda2, lambda3) where each lambda is the variance of each y1,y2,y3.

Of course the iteration can go on up to n dimension. At the end we have all the elements to reduce Y according to the corresponding weight of every varianceover the total variance sum(lambda1,lambda2,lambda3). Hopefully with few variables of Y we can “explain”” a large percentage of total variance and we have reached our objective to reduce a large dimensional dataset in X to few dimensions in Y.

3. Math/Algebra behind PCA (difficult)

The above was the verbose part to explain a manual iterative procedure to solve the eigen decomposition equation. Now we take a look at the whole mathematical procedure from the start to the solution of the eigen decomposition equation.

We call X the original data set (m rows x n columns) where each variable in columns have been already “centered” around their respective means, Cx is the covariance matrix of X, b1 is an unknown vector of n elements, y1 is a projection of X along an unknown vector b1: y1=X.b1

Variance of y1:

To maximize Var(y1) and find the corresponding first direction we use the Laplace equation with the Lagrange multiplier. The symbol is the gradient, f is the function to maximize which is in our case the variance of X.b1, g is the function where we set the constraint that the length of the b vector must be 1:

Fig.1

Partial derivatives of f over b1 (the unknown vector of n components) is:

Fig.2 If you want to find out why you can try with a simple b1 vector (b11,b12) a simmetric Cx like ([4,2][2,1]) and again b1. You will have a second degree polinomial and the two partial derivatives (for b11 and b12) make up a matrix which is 2 times the Cx multiplied by b1

Partial derivatives of g over b1 is:

Fig.3

The eigen decomposition equation becomes:

Fig.4

To find some b1 <>0 we must find the determinant of term in parenthesis and set it equal to zero. Only in this case the system of n equations in n unknown components of b1 has non trivial solutions:

Fig.5

The solution to this system of n equations in n unknowns will likely give (I keep it simple because there could be exceptions) n different lambdas. These lambdas represent the n different variances of the projected points of X along still to be determined directions.

Now we should assign the max of lambdas (let’s call it lambda1) found into the equation (in Fig.1) and find b1 by solving the system given by:

Fig.6

After we have found b1, we take lambda2 and we solve again the system in the equation above to find b2 and so on until for all the lambdas we have all the b vectors.

The calculations as you can imagine are very long for just 3 variables, that’s why we need Python and numerical algorithms to make the dirty work.

In the end we will (exceptions apart) have: the eigenvector B matrix as a sequence of b1, b2, b3 … bn column vectors; the eigenvalue vector L(ambda) representing the variances of every y of Y (being the Principal Components) or the magnitude of the eigenvectors.

4. Python snippets:

There are many examples and libraries around but here I want to use Python just to show with just numpy code snippets that you can quickly try your sample small dataset and understand what’s going on by looking at the results. I skipped the print statements as you can work on console and check yourself the content of the variables. Even charting is omitted but you can do that as you like with matplotlib or seaborn. Here the point is not building an app, but showing the understanding of the logic.

import numpy as np

import pandas as pd pd.options.display.max_columns = 200

pd.options.display.width=200

pd.options.display.max_rows=200 def fcenter(X):

data_mean=np.mean(X,axis=0) #calc mean by column

X=X-data_mean #centered data

return X def fcov(X):

covx=(X.T.dot(X))/X.shape[0] #calc cov matrix of X

#alternative to: covx=np.cov(X,rowvar=False,ddof=0)

return covx def feigen_decomp(Cx):

eigval,eigvec=np.linalg.eig(Cx) #solve eigen equation

return eigval,eigvec X=np.array([[1,1,6],[4,2,9],[2,-2,3],[-3,3,1],[-5,1,7]]) #some data

Xc=fcenter(X) #centered data

#----------------------

#this shows that variance does not change when centering data

vX=np.var(X,axis=0) #variance of columns of X

vXc=np.var(Xc,axis=0) #variance of columns of Xc

#----------------------

Cx=fcov(Xc) #Cx=covariance of Xc

L,B=feigen_decomp(Cx) #L=eigenvalues vector, B=eigenvectors matrix

Lw=L/L.sum() #weight in % of every PC (not cumulative)

Y=Xc.dot(B) #Y=Principal Components matrix (in columns = y scores)

Cy=fcov(Y) #shows that diagonal of covariance of Y coincides con L

#----------------------

#Loadings analysis

Loadings=np.sqrt(L)*B #see comments

Loadings_sq=Loadings**2 #see comments

Loadings_sq.sum(axis=0) #see comments

Loadings_sq.sum(axis=1) #see comments

One note on Loadings: Loadings tells how much of the square root of each eigenvalues in L go into the components of each eigenvector in B. In other words: high values in the Loadings matrix in positive or negative imply high relevance of corresponding original x variable (which are in rows) in relation to Principal Components (in columns). The sign tells about a positive or negative correlation between that x with that PC (grow or decrease together, or when one grows the other decrease and vice versa). Near zero values imply low releavance of that x with respect to that PC.

Second note: Loadings_sq matrix is the square of Loadings matrix and represent vertically how much of the variance of each PC is explained by each variable x of X: in fact the sum of each column is equal to L, and horizontally how much of the variance of each x is explained by each PC: infact the sum by row is equal to variance of x.

Loadings are important for trying to define a name of the PCs in relation to the names of the most relevant x that contribute to the score of the PC.

5. Conclusions

I hope you appreciate this long paper it and I hope to prepare a practical use case in one of the next papers using just python and focusing on the displaying results of PCA application and meaning using seaborn library.