How Where and When we should use PCA
By: Bartosz Szabłowski
Tags: Principal Component, Pca, Visualization, Unsupervised Learning, Dimensionality Reduction

Let’s calculate the covariance matrix for our data set:

array([[ 1.01010101, 0.97727597, -0.01747925, -0.04152546],

[ 0.97727597, 1.01010101, -0.03990115, -0.06169213],

[-0.01747925, -0.03990115, 1.01010101, 0.87546278],

[-0.04152546, -0.06169213, 0.87546278, 1.01010101]])

3. Eigen Decomposition.

· Eigenvectors ➞ PCA directions

· Eigenvalues ➞ importance of directions

The covariance matrix is symmetric, and the eigenvectors of symmetric matrices are orthogonal. Therefore, the first principal component explains most of the variance. Orthogonal to it is the second principal component, which explains most of the remaining variance, and so on. Now it is time to perform the decomposition of the covariance matrix.

eigenvalues:

[2.0243665 1.87865542 0.03358615 0.10379597]



eigenvectors:

[[-0.62202904 -0.33364184 -0.69970849 -0.11032236]

[-0.6031108 -0.36974369 0.70534377 0.04513474]

[-0.32423873 0.63027923 0.09785443 -0.69859992]

[-0.37974529 0.59558444 -0.05764294 0.70551399]]

We obtained a vector (eigenvalues) composed of 4 eigenvalues and a 4×4 matrix storing the eigenvectors.

4. Explained total variance.

The goal of PCA is to reduce the number of dimensions. We compress the current features into new features, which are the eigenvectors (principal components) containing the most information. Information is equivalent to variance. Eigenvalues are the size of the eigenvectors. Therefore, they should be ranked in descending order. The percentage of variance explained is the eigenvalue divided by the total sum of all eigenvalues. From this, we can calculate the percentage of explained variance for each principal component of our set:

[0.501, 0.465, 0.026, 0.008]

As we can see, the sum of the first two principal components accounts for over 95% of the variance.

5. Feature transformation.

We will transform our 4-dimensional dataset into a compressed 2-dimensional dataset. Using the first two eigenvectors, we will create a projection matrix and use it to move our dataset into a subspace consisting of 2 dimensions.

We create a pair of tuples of eigenvalues and eigenvectors and then sort with decreasing eigenvalues. We choose the first two eigenvectors that account for about 95% of the variance. We created a projection matrix w onto which we transferred the observations from the dataset. Finally, we visualize the transformed dataset.

Visualization of PCA operation, Image by Author

The above example was to demonstrate the PCA algorithm [3]. We now turn to a practical application.

Principal Component Analysis — application

We will no longer perform all the steps from the PCA algorithm, we will use its implementation in scikit-learn. We will face the popular dataset iris and see if PCA will help us in better classification.

Let’s download the data and see the first 5 and last 5 rows:

+-----+-----------+-----------+-----------+-----------+-----------+

| | sep_len | sep_wid | pet_len | pet_wid | species |

|-----+-----------+-----------+-----------+-----------+-----------|

| 0 | 5.1 | 3.5 | 1.4 | 0.2 | setosa |

| 1 | 4.9 | 3 | 1.4 | 0.2 | setosa |

| 2 | 4.7 | 3.2 | 1.3 | 0.2 | setosa |

| 3 | 4.6 | 3.1 | 1.5 | 0.2 | setosa |

| 4 | 5 | 3.6 | 1.4 | 0.2 | setosa |

| 145 | 6.7 | 3 | 5.2 | 2.3 | virginica |

| 146 | 6.3 | 2.5 | 5 | 1.9 | virginica |

| 147 | 6.5 | 3 | 5.2 | 2 | virginica |

| 148 | 6.2 | 3.4 | 5.4 | 2.3 | virginica |

| 149 | 5.9 | 3 | 5.1 | 1.8 | virginica |

+-----+-----------+-----------+-----------+-----------+-----------+

As you can see, we have four dimensions and each flower is assigned a species name (Column species). All the flowers in the database are 150. Let’s calculate the basic statistics for our dataset:

+-------+------------+------------+-----------+------------+

| | sep_len | sep_wid | pet_len | pet_wid |

|-------+------------+------------+-----------+------------|

| count | 150 | 150 | 150 | 150 |

| mean | 5.84333 | 3.05733 | 3.758 | 1.19933 |

| std | 0.828066 | 0.435866 | 1.7653 | 0.762238 |

| min | 4.3 | 2 | 1 | 0.1 |

| 25% | 5.1 | 2.8 | 1.6 | 0.3 |

| 50% | 5.8 | 3 | 4.35 | 1.3 |

| 75% | 6.4 | 3.3 | 5.1 | 1.8 |

| max | 7.9 | 4.4 | 6.9 | 2.5 |

+-------+------------+------------+-----------+------------+

virginica 50

setosa 50

versicolor 50

Name: species, dtype: int64

Of course, in a real Data Science project at this stage our data mining should not end, but in this article the protagonist is PCA. Let’s try to visualize our data:

Iris dataset visualization, Image by Author

· Getting rid of “redundant” data (correlated variables)

Some of the variables carry similar information. Let’s see if, in our data, this is also the case.

Pairplot, Image by Author

As you can see, some variables are correlated with each other, most notably petal length and petal width. Both of them are also correlated with sepal length. The least helpful seems to be sepal width. We can conclude that using only one variable it is not difficult to separate classes, it is even easier to separate classes for pairs of variables. Now let us check correlations:

Correlogram of iris dataset, Image by Author

Our earlier conjecture was confirmed, the variables sepal length, petal length, and petal width are highly correlated with each other. In some machine learning models, correlated variables degrade the result because they convey more weight to one piece of information. It is now time to apply PCA, but we cannot do this on the entire dataset if we later build a model for species classification. If we were to apply PCA on the entire dataset then there would be information leakage. Let us create a dataframe X containing features and a vector of labels y, then split the data into training and test data. We should standardize our training set as I wrote earlier. The mean will be 0 for each variable, and the data will be on the same scale — standard deviation. After this operation, we can perform PCA on our set.

The training set has 100 rows.

The test set has 50 rows.

+-----+--------------+-------------+--------------+--------------+

| | PC1 | PC2 | PC3 | PC4 |

|-----+--------------+-------------+--------------+--------------|

| PC1 | 1 | 6.45038e-17 | 2.13745e-17 | -4.80269e-17 |

| PC2 | 6.45038e-17 | 1 | 6.07219e-17 | 1.28115e-17 |

| PC3 | 2.13745e-17 | 6.07219e-17 | 1 | -5.57337e-17 |

| PC4 | -4.80269e-17 | 1.28115e-17 | -5.57337e-17 | 1 |

+-----+--------------+-------------+--------------+--------------+

As we can see the variables are no more correlated.

· Feature grouping (which features represent one piece of information)

In a way, this is a continuation of the previous section, since the correlated variables carry a single piece of information. PCA is not an algorithm that clusters observations, but it is instead able to determine which features contribute the least to explaining the component structure. Let us now create a distribution of features by components. The greater the contribution of the original feature to the component, then the darker its color on the graph.

Distribution of Features by Principal Components, Image by Author

Great! We confirm what you read about earlier. The first Principal Component was created from the correlated variables. The second Principal Component is interesting because we did not see a significant correlation between sepal length and sepal width. Interesting.

· Dimension reduction without significant loss of information

The explained variance plot shows how much information is contained in each Principal Component. We can use the elbow method known for selecting hyperparameters in other machine learning models to select the appropriate number of components. Another form of visualization is the cumulative graph.

Explained variance and Cumulative explained variance, Image by Author

+-----------+----------------------+-------------------------------+

| Component | Explained Variance | Cumulative explained variance |

+-----------+----------------------+-------------------------------+

| 1 | 0.712014337051845000 | 0.7120143370518450 |

| 2 | 0.239653642891057780 | 0.9516679799429029 |

| 3 | 0.043160458797882550 | 0.9948284387407854 |

| 4 | 0.005171561259214621 | 1.0000000000000000 |

+-----------+----------------------+-------------------------------+

The graph shows that the variance of the data is 95% explained by the two components.

· Visualization of multidimensional data

We live in a three-dimensional reality. Our brains find it very well. However, everything that goes beyond three dimensions is usually very difficult for us to imagine. Using methods of dimension reduction we can “flatten” the number of dimensions of our iris dataset and visualize it in two-dimensional space. The following example provides this visualization. The colors of the points in the graph indicate the type of species.

As we can see setosa is well separated from the other classes. Virginica and Versicolor have an unclear boundary which for separating these classes may decrease the accuracy, but I am of good mind looking at this two-dimensional data representation.

· Part of the supervised learning process

We can certainly say that the 4th industrial revolution is a time of data surrounding us. We can use PCA to compress data by making our machine learning algorithms “faster” and the data set smaller. Fewer input variables can result in a simpler predictive model that can have better performance when forecasting on new data. Now is the time to teach our model on the training set and test its abilities on the test set. We will apply logistic regression! We will train the model on the original set, successively on the standardized set and all Principal Components.

+------------------------+----------+------------------------------+

| Model | Accuracy | Training time (microseconds) |

+------------------------+----------+------------------------------+

| not scaled data | 1.0 | 48403 |

| scaled data | 0.98 | 8973 |

| 4 Principal Components | 0.98 | 8977 |

| 3 Principal Components | 0.98 | 8008 |

| 2 Principal Components | 0.94 | 5985 |

| 1 Principal Components | 0.92 | 5983 |

+------------------------+----------+------------------------------+

The Iris dataset is quite small by machine learning standards. Thus, the difference in accuracy between 1.00 and 0.98 on the test set is not significant. We can see that the use of standardization has a large impact on the learning time of the model. If I were to implement the model in production it would be 3 Principal Components, since it already has a reduced dimensionality that explains 0.99 of the variance.

Principal Component Analysis Summary

In this article, I discussed the application of PCA for dimensionality reduction, visualization of multidimensional data, and its use in supervised learning. The PCA method can only be applied to numerical data, both for initial data analysis and for feature (dimension) reduction. It can help in obtaining the first approximate outlines of regularities underlying a given phenomenon. It is based on correlation, which can be very misleading (Remember: Correlation does not imply causation!). I hope dimensionality reduction using Principal Components Analysis is now clearer to you. Other dimensionality reduction techniques used:

· ICA

· LLE

· ISOMAP

· t-SNE

If you would like anything more in this article then please let me know, I will edit it or if you would like me to write an article about a specific algorithm then please let me know too.

References

[1] G. James, D. Witten, T. Hastie, R. Tibshirani, An Introduction to Statistical Learning: With Applications in R (2013)

[2] A. Geron, Hands‑On Machine Learning with Scikit‑Learn, Keras, and TensorFlow (2017), O’Reilly Media

[3] S. Raschka, V. Mirjalili, Python Machine Learning: Machine Learning and Deep Learning with Python, scikit-learn, and TensorFlow 2, 2nd Edition (2017), Packt Publishing