Understanding the link between PCA and Eigenvectors
By: Valentina Alto
Tags: Statistics, Principal Component, Machine Learning, Eigenvectors, Data Science

In my former article, I’ve been introducing the concept of Principal Components Analysis, a powerful technique used in Machine Learning to reduce the dimensionality of your input data. An important mathematical feature of PCA is its link with the Eigenvectors (and Eigenvalues) of the original features matrix X, and in this article, I’m going to show the ratio behind this link.

So let’s start with a brief recap of PCA. Differently from features selection techniques, PCA does not use a subset of the initial features: it rather computes a new set of features obtained as linear combinations of the original ones, living in a lower-dimensional space.

So if we have a feature matrix X of n observations, each with p dimensions:

We can compute a new features matrix Z, obtained from linear combinations of the former:

Where M is at most as big as p, but of course, because of the purpose of the task (dimensionality reduction), it is chosen to be less than p.

How do we obtain Z? As said above, it is a linear combination of the original features, so that the i-th observation of the m-th component is given by:

Where:

The coefficients of the first 2 lines are known as loading factors and are computed in such a way that these new variables are uncorrelated and most of the information within the initial variables is stored into the first components. So, the idea is that p-dimensional data give you p principal components, but PCA tries to put maximum possible information in the first ones, so that, if you want to reduce your dataset’s dimensionality, you can focus your analysis on the first few components without suffering a great penalty in terms of information loss.

In this analysis, what measures the amount of information is variance, and principal components can be geometrically seen as the directions of high-dimensional data which capture the maximum amount of variance and project it onto a smaller dimensional subspace while keeping most of the information. Hence, the first principal component accounts for the largest possible variance; the second component will, intuitively, account for the second-largest variance (under one condition: it has to be uncorrelated with the first principal component) and so forth.

With this goal, we set a constrained maximization problem, where we seek to maximize the variance of each principal component, subject to the orthogonality condition between loading factors and a normalization condition (indeed, if we leave loadings free, we can increase them infinitely in order to maximize the variance).

Hence:

From the definition of Eigenvalues and Eigenvectors, we can conclude that loading factors are indeed eigenvectors. More precisely, we select as a loading factor that eigenvector associated with the highest eigenvalue.

So here it is the link between PCA and Eigenvectors! It can be shown that the above relation holds also for the remaining loading factors, with the additional constraint of being orthogonal among each other.

We can also visualize the geometric representation of this procedure. For the sake of simplicity (and the possibility of visualization), I will work with an original features space of 2 dimensions, and I will compute 2 principal components, so I’m not going to lower the dimensionality of the features space.

Imagine we are provided with the following 2-dimensional input data:

Which I’ve already centered at 0. The principal components are represented as:

Now, in order to plot our data into the new feature space (in this case, we created 2 principal components, hence it will still be a 2D plot), we simply rotate the two principal components so that PC1 is horizontal and PC2 is vertical — that means, they are the new x-axis and y-axis.

I hope this article was clear and enjoyable! If you are curious about the topic and want to read more about that, I recommend the following readings: