Principal Component Analysis (PCA) in Machine Learning— You will never find it tough again
By: Iqbal Hussain
Tags: Machine Learning, Principal Component, Python, Unsupervised Learning, Data Science

Principal Component Analysis (PCA) in Machine Learning— You will never find it tough again Iqbal Hussain Follow Jun 4 · 9 min read

Image created by the Author

Principal Component Analysis(PCA) is a popular unsupervised machine learning technique which is used for reducing the number of input variables in the training dataset. This technique comes under dimensionality reduction technique. I assume, if you have already started reading this article that means you know PCA to some extend. Hence I am not going to discuss what it is, its advantage or disadvantage. But I will focus on how it works.

Although Python has provided all the classes and methods to create your model and perform your operations whether it be a regression, classification or clustering problem or PCA, but until and unless you know what is happening behind the scene you will not have all the fun as a Data scientist or machine learning enthusiast.

While I was learning PCA, somehow the resources I came across were not sufficient enough to make me understand the mathematics behind PCA. So, I did some research, brushed up my old mathematical and statistical skills and after doing all those, finally I could understand the magic behind the excellent dimensionality reduction technique, which is PCA(Principal Component Analysis). In this article, I will try to explain all the steps to perform PCA using the mathematical and statistical concept on a small two dimensional toy dataset and reduce its dimension into one. After completing all these by hand (I mean mathematically step by step), I will show you the same thing done programmatically in python and validate our result.

Whatever I am covering here is using a two-dimension dataset, but that should not stop you to understand it for large datasets with higher dimensions as once you can understand the concept in two dimensions, you can apply the learnings in higher dimensions.

Before I start, let me first go you through some of the basic mathematical and statistical concepts so that you can relate them later in the article. If you are already familiar with the concepts then you can directly jump into section 3 and continue. But, if you have time, I still recommend you that you go through all the sections.

Section 1: Some Mathematics

1.1 Scalar: Scalar is nothing but a number with a value. e.g. 5 is a scalar, 2.45 is a scalar.

1.2 Vector: Vector can be thought of an object with a magnitude and a direction. A coordinate can in two dimension is an example of a vector. e.g. (4,3).

We can draw a line from (0,0) to (4,3) which will represent the direction of the vector. What will be the value or magnitude then? The value of the vector will be the distance between the point (0,0) and (4,3). Note that the value will be a scalar.

Now using Pythagoras Theorem, we can find distance as:

Hence, the magnitude of the vector is 5.

Similarly, for data with 3 dimension (2,5,7) is an example of vector. Hence , for data with n dimensions (1, 4,…….,n) is a vector.

1.3 Matrix: Matrix is an array of numbers. Matrix may be square or rectangular shape. Note that all vectors can be represented as matrix. In the example above our vector can be represented as matrix as shown in the figure below:

2x1 Matrix

1.4 Determinant: It represents a number which can be calculated from a square matrix. e.g. if A is a square matrix then |A| represents determinant of A. Determinant of a 2x2 matric can be calculated as below:

1.5 Identity Matrix: Identity matrix (denoted as I) is a square matrix where the diagonal elements of the matrix are 1 and rest of the eements are 0.

1.6 Invertible matrix: Z is said to be invertible matrix (it should be square matrix) if it satisfy the following equation:

Note: Determinant of Z, that is |Z| =0 if and only if Z is not invertible.

1.7 Matrix Multiplication:

A, B are two matrices and AB is the multiplication. Two matrices are compatible for multiplication if number of columns of left side matrix is equal to number of rows of the right matrix. Here A is 2x2 and B is 2x1 matrices resulting in 2x1 matrix.

1.8 Eigenvector: Eigenvector is a non-zero vector (say, v) which when multiplied with another square matrix (say, A)produces another matrix (also, a vector) which is an scalar multiple of the original vector v.

1.9 Eigenvalue: In the above equation lambda is called the eigenvalue of the eigenvector v.

Let us derive one important equation here from the above equation:

We know,

We will use the equation later. Let’s move to the next section.

Section 2: A bit of Statistics

Now some statistics, in a very short form.

To better understand, you can consider X as a column in a dataset.

2.1 Standard deviation (S) is the measure of dispersion of data from the mean in a list of numbers. The formula to calculate the standard deviation is:

2.2 Variance is the square of the standard deviation.

2.3 Covariance: As variance depicts spread of data in one column, covariance is the joint form (related to two columns in a dataset) of the same thing.

Let us write the formula of variance (Fig: 3) in a factor form

i.e. var(X) =cov(X,X)

From the same formula we can derive covariance of two variables X and Y (two columns in a dataset) as

2.4 Z-Score: We can standardize or scale a group of observed values using z-score for which mean is deducted from each observed value and then divided by the standard deviation. The formula is:

The mean of the new observed data set becomes 0 after applying Z-score.

As of now, you just note the formulas, we will really calculate them in our example below.

Section 3: Mathematics behind PCA

Now, that you know some of the basic mathematical and statistical terminology and formulas, let’s get our hands dirty with the actual topic of discussion. We will see how to perform PCA step by step using Mathematics. I will elaborate it with a 2d dataset.

Let us consider that given dataset is a dataset with 2 columns (features) and 6 rows. Columns names are X and Y.

Note here that (2,3), (4,5), (6,5), (6,7), (7,8), (5,8) are called feature vectors.

Steps to PCA:

Step1: Standardize the data

We can standardize or scale the data using z-score. Let’s calculate corresponding mean and standard deviation for each column.

As mentioned in the section 2.4, the formula to calculate Z score for each observed value is

Using this let us transform our original data to Z-scaled data:

Step2: Find the covariance matrix

Now, let’s find the covariance matrix of the scaled data.

Step3: Find the Eigen values

As, we will consider one PCA component only we will disregard 0.261 as that is very small and will go with Eigen value 2.139.

Now, we will find the Eigen vector of the Covariance matrix corresponding to the Eigen value we have found above.

Step4: Find the Eigen Vector

Step5: Project existing data into one dimension using Eigen vector

As our main aim to perform PCA is to project the data from higher dimension to lower dimension, let’s continue with our example and project the scaled data to one dimension using the Eigen vector found above.

Let’s check the scaled data once again:

The first feature vector (-1.837, -1.643) will be transformed using the following formula:

(Transpose of Eigen vector) X (Feature vector)

Let’s put the corresponding values:

Similarly, if we project all the feature vectors in the scaled data we will end up with the following dataset with one dimension.

Hence, we have completed performing our calculations for PCA and come up with the above projected data in one-dimension.

Section 4: PCA using Python

In section 3, I showed you how we can perform PCA in a toy dataset. I will perform the same thing now using Python and validate our final projected data.

Here is the snapshot of my Jupyter notebook where I have created the dataset and performed the steps to PCA:

Hence, we can see that we have obtained the same data (approx.) by doing mathematically by hand and by programmatically. Only difference is the sign. Both are valid as python considered the negative values of the Eigen vectors while we considered the positive values. But both are correct as long as the Eigen vector is normalized to have length 1. Now, you can easily understand what is the mathematics behind the PCA algorithm.

Conclusion:

I have tried to show you PCA in a very small 2 dimensional dataset. In real life scenario, you will have a large dataset with huge number of dimensions (column or feature, whatever you say) where PCA is really useful when you do not know which of the features are not adding value to your final model. After you perform PCA in the large dataset and reduce its dimension (that’s why PCA is called, Dimensionality Reduction technique) it will be easier for visualization and will be ready for your further modelling.

I hope you have enjoyed reading and learning from this article. Please provide your valuable feedback.

All the images shown for the calculations are created by the author.