An Intuitive Approach to PCA
By: Brayton Hall
Tags: Beginner, Principal Component, Data Science, Data Visualization, Linear Algebra

An Intuitive Approach to PCA

With the Iris Dataset

The Iris versicolor

The Iris dataset is one of those datasets that one frequently encounters in the pursuit of acquiring or honing data science techniques. It’s small, only 150 rows, with four features, petal length and width, and sepal length and width. The featured imaged (above) is the one with average features, in the orange cluster below, named the Iris versicolor:

Easy to work with.

The clusters are extremely easy to see, the data requires no preprocessing, and there are only 3 targets, corresponding to the clusters above:

Iris setosa (Blue), Iris Versicolor (Orange), and Iris virginica (Green)

Thus, the dataset is perfect for visualization techniques, practicing linear regression or classification, and, in the case of this post, using Principal Component Analysis (PCA) for feature reduction!

What is Principal Component Analysis?

If you’re unfamiliar with PCA, or need a refresher, no worries. It’s one of those auxiliary DS techniques which may be necessary, or entirely useless, depending on the size and complexity of your data, as well as the intent of your project.

PCA stands for ‘Principal Component Analysis’, and refers to recombinations of your original data (the principal components), which is quite similar to the process of extracting, then sorting, the color from Skittles. The metaphor in this case is useful because the process of PCA isn’t just about sorting Skittles, but extracting then sorting color from Skittles. The color in the metaphor is statistical variance in the original data.

obtained from here

As a rule of thumb, PCA is useful for reducing the size of data, particularly data with many features, while retaining the ability to model. This is a highly intelligible way, though watered down, of saying ‘PCA uses singular value decomposition to find orthogonal, recombined principle components from the original features which can explain most of the variance in the original data with remarkably less memory’. A deep dive into PCA can be found here, though the purpose of the post you’re reading is to convey the intuition of PCA without a formal walkthrough of linear algebra, covariance matrices, or eigenvalues. If you are interested in the more technical aspects of PCA, the gold standard of introduction to linear algebra can be found in 3Blue1Brown’s renowned Youtube series.

Looking at the Iris Features Before PCA

Let’s look at the Iris dataset again visually, this time with both the petal and sepal features, for a total of four features. The following displays the first five rows of the 150-item Iris dataset, followed by scatterplots corresponding to the petal and sepal features.

First five Irises of the Iris dataset

The Two Petal Features, Length and Width

Clusters are easy to see.

The Two Sepal Features, Length and Width

Clusters still easy to see, but not as good. Still, these features contain more information about the Iris type.

In both cases we can easily distinguish the three types of Iris. This is important, because it tells us that both the sepal features and the petal features independently contain explanatory information relating to the type of Iris, i.e. the target class. If we could only choose petals or sepals to attempt classifying, then of course the petal features would yield much better results, but not perfect:

This overlap may seem trivial for small data, but imagine if this overlap represented millions of data points!

The Iris virginica and Iris versicolor clusters blend together a bit, and are not linearly separable. Depending on the size of the data, it may be appropriate to bring in a Support Vector Machine (SVM) here, or a neural net. In some cases, the data may even be linearly separable after applying a transformation to the data, but no such pattern seems to exist here. Thus, we apply PCA! Even though the sepal features seem even worse for classifying the target (i.e. they are even less linearly separable by Iris type), they nevertheless contain important information, or statistical variance, which may reveal more linear separability in different dimensions while also reducing the amount of features! This is why PCA is useful. We are going to break down the features we currently have (total of four), using Singular Value Decomposition, which is automatically implemented in sklearn, then re-visualize based on the new components.

Visualizing the Iris Dataset After Implementing PCA

As always, there’s a slight amount of preparation required before applying PCA. Because the features in the Iris dataset are on totally different scales (e.g. the sepal lengths are much longer than the petal widths), we need to scale them so that the new principal components treat all features equally via singular value decomposition. We will use the standard scaler in sklearn:

from sklearn.preprocessing import StandardScaler features = df[['p_length', 'p_width', 's_length', 's_width']]

scaled_features = StandardScaler().fit_transform(features)

The standard scaler basically scales each data point based on a Gaussian, or normal distribution (i.e. it subtracts the mean of each feature from each row’s corresponding column value, then divides it by the standard deviation). We can see the first five Irises again, below, after applying the standard scaler. The fact that most of the values are negative makes sense, since the blue (Iris setosa) values in the scatter plot are mostly less than the mean of the rest of the data, regardless of feature (except for sepal width, hence some positive values).

The first five Irises after standard scaling

Finally we are ready to apply PCA.

from sklearn.decomposition import PCA pca = PCA(n_components=2)

principalComponents = pca.fit_transform(scaled_features)

And that’s it! The parameter n_components allows us to select how many principle components we want to reduce our features to, in this case 2. Let’s look at the Iris dataset again, now with only two features. (In other words, we’ve reduced the size of our data by 50%).

From four features to two

Even though we’ve reduced our data by half, we still have two principle components which contain the majority of the statistical variance in the original data, and clustering or classification would still be an easy task. Even if we reduce all four components to just one, throwing out a massive 75% of our original data:

pcasolo = PCA(n_components = 1)

solo_PrincipalComponent = pcasolo.fit_transform(scaled_features)

Then we still end up with a remarkably effective single principle component (essentially just a number line) for the purposes of clustering and classification:

Wow! Through PCA, we can reduce the number of features we have, while retaining the explanatory statistical variance responsible for modeling. It may seem like a neat extension of the same old preprocessing and cleaning steps, but PCA is so effective that it almost blurs the lines between preprocessing and modeling. In the above case, PCA allowed us to create a mere number line which captures the relationships we were initially looking for in the data for determining Iris type. The advantages of this are enormous, not just for feature reduction, but for speeding up models with limited computational resources. Just to demonstrate the modeling effectiveness of PCA, take a look at how well a simple classification model performs (by accuracy) on the original data, data reduced to two principal components, and data reduced to one principal component:

('all features: ',

0.9736842105263158,

'reduced to two ',

0.9473684210526315,

'reduced to one',

0.8947368421052632)

Sure, we lost about 8% accuracy going from four features to one, but that means that we lost only 8% of our original accuracy despite throwing out 75% of our data! That’s impressive. You can imagine how important, even necessary, PCA might be for large data sets or complex data sets with many features.

If you’re interested in taking a closer look at the code and trying out PCA for yourself, the Jupyter notebook for this post can be found on my Github repository for the post. Thanks for reading!