Statistical and Mathematical Concepts behind PCA
By: Rukshan Pramoditha
Tags: Unsupervised Learning, Machine Learning, Numpy, Data Science, Dimensionality Reduction

As I promised in the previous article, Principal Component Analysis (PCA) with Scikit-learn, today, I’ll discuss the mathematics behind the principal component analysis by manually executing the algorithm using the powerful numpy and pandas libraries. This will help you to understand how PCA really works behind the scenes.

Before proceeding to read this one, I highly recommend you to read the following article:

This is because this article is continued from the above article.

In this article, I first review some statistical and mathematical concepts which are required to execute the PCA calculations.

Statistical concepts behind PCA

Mean

The mean (also called the average) is calculated by simply adding all the values and dividing by the number of values.

Standard Deviation

The standard deviation is a measure of how much of the data lies within proximity to the mean. It is the square root of the variance.

Covariance

The standard deviation is calculated on a single variable. The covariance is the variance of one variable against another. When the covariance of a variable is computed against itself, the result is the same as simply calculating the variance for that variable.

Covariance Matrix

A covariance matrix is a matrix representation of the possible covariance values that can be computed for all features of a dataset. It is required to execute the PCA of a dataset. The following image shows such a covariance matrix of a dataset which has 3 variables called X, Y and Z.

cov(Y, X) is the covariance of Y with respect to X. It is same as the cov(X, Y). The diagonal elements of the covariance matrix give you the values of variance for each variable. For example, cov(X, X) is the variance of X.

A large value of the covariance of one variable against another would suggest that one feature changes significantly with respect to the other while a value close to zero would signify a very little change.

To calculate the covariance matrix for a given dataset, we can use numpy cov() function or pandas DataFrame cov() method.

Mathematical concepts behind PCA

Eigenvalues and Eigenvectors

Let A be an n x n matrix. A scalar λ is called an eigenvalue of A if there is a non-zero vector x satisfying the following equation:

The vector x is called the eigenvector of A corresponding to λ.

The above equation implicitly represents PCA. The following equation which is same as the above equation (but with different terms) directly represents PCA.

Where,

A is an n x n square matrix. In terms of PCA, A is the covariance matrix.

is an n x n square matrix. In terms of PCA, is the covariance matrix. Σ represents all the eigenvalues in the form of a diagonal matrix which has the diagonal elements representing eigenvalues. The amount of variability within the dataset is indicated by the corresponding eigenvalue. This is done by describing how much contribution each eigenvector provides to the dataset. The larger the eigenvalue, the greater its contribution.

represents all the eigenvalues in the form of a diagonal matrix which has the diagonal elements representing eigenvalues. The amount of variability within the dataset is indicated by the corresponding eigenvalue. This is done by describing how much contribution each eigenvector provides to the dataset. The larger the eigenvalue, the greater its contribution. U represents all the eigenvectors in the form of an n x n square matrix.

We have discussed some statistical and mathematical concepts behind PCA. In the next steps, we calculate the eigenvalues and eigenvectors using the covariance matrix of the breast_cancer dataset. Then we perform the PCA. For the entire PCA process, we will only use numpy and pandas libraries and will not use the scikit-learn library except for the feature scaling.

Execute PCA manually using numpy and pandas

Step 1: Import libraries and set plot styles

As the first step, we import various Python libraries which are useful for our data analysis, data visualization, calculation and model building tasks. When importing those libraries, we use the following conventions.

Step 2: Get and prepare data

The dataset that we use here is available in Scikit-learn. But it is not in the correct format that we want. So, we have to do some manipulations to get the dataset ready for our task. First, we load the dataset using Scikit-learn load_breast_cancer() function. Then, we convert the data into a pandas DataFrame which is the format we are familiar with.

Now, the variable df contains a pandas DataFrame of the breast_cancer dataset. We can see its first 5 rows by calling the head() method. The following image shows a part of the dataset.

Part of the breast_cancer dataset

The full dataset contains 30 columns and 569 observations.

Step 3: Obtain the feature matrix

The feature matrix contains the values of all 30 features in the dataset. It is a 569x30 two-dimensional Numpy array. It is stored in the X variable.

Step 4: Standardize the features if necessary

You can see that the values of the dataset are not equally scaled. So, we need to apply z-score standardization to get all features into the same scale. For this, we use Scikit-learn StandardScaler() class which is in the preprocessing submodule in Scikit-learn.

First, we import the StandardScaler() class. Then, we create an object of that class and store it in the variable sc. Then we use the sc object’s fit() method with the input X (feature matrix). This will calculate the mean and standard deviation for each variable in the dataset. Finally, we do the transformation with the transform() method of the sc object. The transformed (scaled) values of X are stored in the variable X_scaled which is also a 569x30 two-dimensional Numpy array.

Step 5: Compute the covariance matrix

Now, we compute the covariance matrix for all features of our dataset. Note that we use X_scaled matrix, not the X. To calculate the covariance matrix for our dataset, we can use numpy cov() function. We need to take the transpose of X_scaled because the covariance matrix is based on the number of features (30), not observations (569).

The covariance matrix of our dataset is a 30x30 2d numpy array.

Step 6: Compute the eigenvalues and eigenvectors of the covariance matrix

We can use the eig() function to calculate the eigenvalues and eigenvectors of the covariance matrix. The eig() function is in the linalg module which is a subpackage of the numpy library.

The variable eigenvalues returns all the eigenvalues.

The variable eigenvectors returns all the eigenvectors in the form of 30x30 2d numpy array.

Then we get the transpose of eigenvectors.

The eigenvector for the first eigenvalue (13.304) is the first row of the eigenvectors array. It has 30 elements.

Step 7: Sort the eigenvalues and eigenvectors from the highest to the lowest

About the first 20 eigenvalues were automatically sorted from the highest to the lowest. So, we do not need to sort the eigenvalues and eigenvectors. This is because we need just first 10 eigenvalues for our PCA process.

Step 8: Compute the eigenvalues as a percentage of the variance within the dataset

From the above eigenvalues, we need only the first 10 eigenvalues to preserve 95.15% of the variability in the data. The corresponding (selected) eigenvectors for the first 10 eigenvalues are:

Step 9: Multiply the scaled dataset by the selected eigenvectors

The dimensionality reduction process is a matrix multiplication of the selected eigenvectors and the (scaled) data to be transformed. Note that the transpose of the X_scaled is required to match the dimension when executing the matrix multiplication.

Then we get the transpose of data_new matrix (2d array).

Now the data_new array is in the right dimension. It contains the transformed data with 10 principal components.

Step 10: Convert transformed data into a pandas DataFrame

Let’s create a pandas DataFrame using the values of all 10 principal components.

Part of the transformed dataset

The transformed dataset now has 10 features (principal components) and 569 observations. The original dataset contains 30 features and 569 observations. Therefore, we have reduced the dimensionality of the data by 20 features preserving 95.15% of the variability in the data.

Step 11: Plot the values of the principal components

The output is:

To verify, you can compare the results obtained here with the results obtained using the Scikit-learn PCA() by setting n_components=0.95. The results are exactly the same!

Thank you for reading! See you in the next article.

This tutorial was designed and created by Rukshan Pramoditha, the Author of Data Science 365 Blog.

Technologies used in this tutorial

Python (High-level programming language)

(High-level programming language) numPy (Numerical Python library)

(Numerical Python library) pandas (Python data analysis and manipulation library)

(Python data analysis and manipulation library) matplotlib (Python data visualization library)

(Python data visualization library) Jupyter Notebook (Integrated Development Environment)

Statistical concepts used in this tutorial

Mean

Standard Deviation

Covariance

Covariance Matrix

Mathematical concepts used in this tutorial

Eigenvalues and Eigenvectors

Matrix Multiplication

Matrix Transpose

Machine learning used in this tutorial

Principal Componnet Analysis (PCA)

2020–08–10