Hands-On PCA Data Preprocessing Series. Part I: Scaling Transformers | by Kefei Mo | Towards Data Science

In this series, we will explore the combination of scaling data and the PCA. We would like to see how we can better prepare data for machine learning tasks whenever we come across a new dataset. The journey is composed of three parts.
Part I: Scalers and PCA
Part II: Meet outliers
Part III: Categorical data encoding
Review briefly the background of scalers and PCAIntroduce the dataset to work on and the taskPerform scaling-transformation on the datasetConduct PCA on the scaling-transformed dataset and evaluate the performance
Review briefly the background of scalers and PCA
Introduce the dataset to work on and the task
Perform scaling-transformation on the dataset
Conduct PCA on the scaling-transformed dataset and evaluate the performance
Understand the importance of scalers and their close relationship with PCA
Choose scalers wisely
Make relevant and pretty visualization :)
Before you start the reading, we definitely recommend you to play with the notebook (available on Colab and Github. Please find the links at the end of this post.)
As you’ve already known, PCA is sensible to the scale of a dataset, hence getting each feature to a reasonable scale is crucial to PCA. In this part, we are going to see how different scalers can affect the PCA results.
This post is inspired by this tutorial from Scikit-Learn’s document. You will see us testing different scalers using the same wine dataset, and passing the differently scaled data to the PCA step. This little toy set demonstrate the power of the combination of scaler + PCA, perhaps you would like to have it in your own tool box.
Before diving into more details, we would like to make a brief introduction about the PCA, for the sake of completeness of our journey. Readers who have already understood the maths behind will feel free to jump directly to the next section.
PCA is the abbreviation of “principal component analysis”, one of its main functionalities is to reduce the dimension (columns) of a dataset. And it is done by singular value decomposition (SVD) of linear algebra. Intuitively, data are in the form of a matrix, whose columns are the axes of high dimensional space. Data points represented with such axis are sometimes hard for machine learning models to explore. What SVD does is to find a smarter representation of the dataset (by the linear combination of the original axes). The data expressed by the new abstract axis are more model-friendly.
A few more words: the new set of axes are mutually orthogonal (even better, orthonormal). This property can be particularly lovely for tree-based models. The new set of axes is in the order of their ability to represent the dataset’s information or variability
Why do we need scaler? Imagine a dataset of house prices, containing features like the number of rooms, the area, and the price of individual apartments. The variability of these three columns is of vast difference: the maximum difference in the number of rooms can hardly exceed ten rooms, that is of order 1; the maximum difference in the area of an apartment can be of order 100 squared-feet; and the variance in the prices can easily reach the order 10K.
What does scaler have anything to do with PCA? If we are to apply PCA on such a dataset whose variability of columns differs in order, the effect of those relatively “smooth” features will be totally overwhelmed by the most furious one (the price in this example). To remedy the problem, we scale the data.
There are a bunch of different scalers available with one-line code in SciKit-Learn, like the most commonly used standard scaler and min-max scaler, and other non-linear scalers. In this blog post, we are interested in testing all these available scalers before applying PCA, and see how they act with PCA. If you are interested, here is a detailed demonstration of all scalers tested in this post.
Let’s dive in. First, we will need some preparation codes. We will use the wine data as follows.
The wine dataset is a legend dataset. The data is the results of a chemical analysis of wines grown in the same region in Italy by three different cultivators. There are thirteen different measurements taken for different constituents found in the three types(origins) of wine. The dataset contains only numerical features. Our goal is to figure out the target labels (the origins), using the thirteen different measurements. First, let’s take a quick look at the data.
# note: the code in this post is mainly for illustration purpose. More details, please refer to the original notebook. link at the end of the blog.df_wine.head()
df_wine.describe()
We kindly invite you to notice the difference in the order of the variability of features: “nonflavanoid_phenols” has a standard deviation of 0.4 while “proline” has 746. We will see later that, without any scaling of the dataset, the result from PCA will be highly biased by those features having huge variance.
The dataset’s target column contains three labels so that it can be a multi-class classification task. Here we mimic semi-supervised and unsupervised learning situations by masking out a different amount of the true labels. The reason why we do this additional step will be clear very soon. Let’s first visualize the data, just to get some insight.
The data is very clean. And all the features are numerical variables. So we don’t have any tedious data cleaning work to do. Using seaborn’s pairplot and box plot would do reveal the dataset pretty well.
fig, axes = plt.subplots(nrows=1, ncols=len(df_features.columns), figsize=(len(df_features.columns)*2,4))for index, col in enumerate(df_features.columns):    ax = axes[index]    sns.boxplot(y=df_wine.loc[:,col], x=df_wine.loc[:,'target_original'], palette=customer_palette, ax=ax)    ax.set_ylabel(ax.get_ylabel(), fontsize='x-large')    ax.set_xlabel("")plt.tight_layout(pad=0.5)plt.show()
g = sns.pairplot(data=df_wine,                  vars=df_features.columns,                  hue='target_original',                             corner=True, palette=customer_palette,                    hue_order=target_order)
# target_large_missinngg = sns.pairplot(data=df_wine,                  vars=df_features.columns,                  hue='target_large_missinng',                  corner=True,                  palette=customer_palette,                  hue_order=target_order)
# target_all_missinngg = sns.pairplot(data=df_wine,                  vars=df_features.columns,                  hue='target_all_missinng',                  corner=True,                  palette=customer_palette,                  hue_order=target_order)
You might be wondering why visualizing the same data three times with just the different colors.
We challenge you to distinguish three different target classes in three scenarios.
Scenario no.1: with color aid (the first pair plot)
Scenario no.2: with weak color aid (the second pair plot)
Scenario no.3: no color aid (the last pair plot).
The point here is that it becomes very hard for human eyes to distinguish different target classes without target labels or only a limited number of labels (Note: the scenario without target labels is similar to an unsupervised learning situation; the scenario with a limited number of labels is similar to a semi-supervised learning situation).
While humans may fail to see higher dimensional interaction between features, machine learning models may find some patterns. Yet why not try to represent the data in a more intelligent way by, for example, a proper scaler followed by the PCA?
Here we will transform the dataset using all eight different scalers available in SciKit-Learn’s preprocessing subclass. We are not paying much attention to tuning the scalers’ hyper-parameters. We plot the scatter plots of the PCA’s first two components of the transformed dataset, always keep the spirit of reducing the color aid progressively.
# customer visualization function# show scatter of 1st and 2nd PCA componentpca_scatter_plot(X_pca_dict, y_train.iloc[:,1:])
Don’t get lost in the abundance of the scatter plots. Let’s take a closer look at the ones with color aid. We’d like to conclude that, with this dataset, it’s better to scale the data, no matter which scaler we choose. But there is one exception: the normalizer. The normalizer scales data in a weird way: row-wise instead of column-wise. We are unsure of when such a scaler might be useful, but we would love to discuss with you about when to use it.
If you wonder why scaled datasets make each class more distinguishable than unscaled data. We are glad you asked. The reason is that the scaled variables can be compared equally. On the contrary, the importance of the unscaled features seen by the machine learning models is highly affected by their scales.
Here is a little exercise for readers who have the math knowledge about the PCA. The clusters of different classes don’t always appear in the same order, say from left to right being red, blue, and green; it can be green, blue, and red. Why does such a permutation phenomenon occur?
Let’s move on to investigate more about PCA acting on a differently scaled dataset. Below are box plots of each feature after PCA, different scalers are distinguished row-wise.
# customer visualization function show PCA componentspca_box_plot(X_pca_dict, df_target.loc[:,'target_original'])
Again, without any transformation, almost all the interesting information concentrates on the first component, leaving little variance to the rest of the space. We would claim that the most important ingredient of the first component is “proline” from the original dataset. Indeed the behavior of the boxes is almost identical to that of the boxes plotted for “proline” using the original feature space above.
If we look at other scalers (except the normalizer), we can see the trend of boxes bouncing less and less when going from the first component (i.e., the most informative) to the last (i.e., the least informative). More interestingly, almost all scalers result in a cut-off effect on the third component. At this stage, we may be confident enough that the first two components are representative.
We can explore even a little further to check the covariance matrices. Since PCA directly works on a covariance matrix and output the principal components. That’s the very beginning of the story.
Let’s visualize the covariance matrices by Seaborn’s heatmap function.
An important note: we usually use heatmap to show Pearson correlation coefficient matrix (CC matrix), but there is a key difference between these two matrices: the values in a covariance matrix is not necessarily between (-1,1), and the values on diagonal are not always 1. That’s why we plot the diagonal values even using lower triangle heatmap.
trans_heat_plot_abs(X_trans_dict, y_axis_labels=df_features.columns)
At this point, the heatmap might not be very intuitive comparing to the scatter plots and/or the box plots because the label information does not get into the heatmap. Also, the fine gradient of some colorbars makes some heatmaps very similar to each other.
Yet, we do have a hint from these heatmaps of how well a matrix may adapt to the PCA. A rule of thumb is, the more colorful the heatmap, the better the PCA result. Normally PCA doesn’t like a plain-looking heatmap and would output less interesting principal components. In our example, since “proline” is the one that stands out, PCA will have all eyes on it. And this is consistent with our analysis earlier. Another plain-looking one is normalizer transformed data. From the heatmap, we might guess that the ONE component from PCA would be largely from “magnesium”; hence the dull result is given by normalizer + PCA.
# plot the explained_variance cumsum using previously calculated PCA instances.pca_plot_cumsum(instance_pca_dict)
We are more or less happy with the preprocessing results, by seeing the first two dimensions of the new aspect of the feature space, found by PCA. However, are the clustering algorithms that can see higher dimensional interactions favor the same transformed matrices as we do? Let’s cluster the data using machine learning clustering models.
One advantage we have when doing clustering tasks on the labeled dataset is, we know the correct number of classes. Next, we will see how well the k-means agrees with the true labels.
Here we adopted the first metric for the blog: V-measure score. We would not go into the details to talk about this index; all you need to know is: The V-measure score is bounded in the range [0,1]: values close to 0 are bad, while 1.0 stands for a perfect match.
df_scores
We are comparing three scenarios: clustering with two original features, the two most informative PCA components, and the total feature set. We chose two features or components to keep the consistency with the previous scatter plots. While more columns might reveal more thoroughly the dataset’s information, we think that two dimensions, at least to begin with, are good enough.
Let’s take a closer look at the above table.
Firstly, let’s focus on the last column, which contains V-measure scores using the entire dataset. It’s always better to scale the data, no matter which scaler we choose. (Except the “Normalizer”)
Secondly, for the original aspect of the feature space, it’s always better to use the entire dataset. And it seems not a good idea to choose a subset of features that “seems” to illustrate better the clusters. (Just look at the comparison between the first and the third columns).
On the other hand, it is a different story for PCA transformed data. Sometimes, it results in more reasonable clusters with merely a few of the components than using all of them. (Take a look at the comparison between the second and the third columns). Even though in some of the tests, the first-2-component choice does not outperform the totality, there is a close match in terms of the V-measure score between the two choices. (since here we are comparing the calculation sources of 2-column matrices vs. 13-column matrices.)
And lastly, looking at the second column, we find that the most suitable scalers for this single task are the two non-linear ones plus the standard scaler. For this reason, you might want to try out more scalers by doing simple tests before applying directly to the most commonly used standard scaler.
We’d like to have a few more words on the less-is-more phenomenon shown by PCA. It’s often argued that throwing away information from the original dataset is not wise; it is sometimes very dangerous to do so. Yet PCA reveals the most important patterns inside the data matrix while filtering out some details. Such details could be nothing but noise. So taking the first several PCA components not only helps reduce the conditionality but also smoothens the information and makes things clearer.
kmeans = KMeans(n_clusters=3, random_state=RANDOM_STATE)pca_cluster_contour_plot(X_pca_dict,y_train, kmeans)
In this post, we have compared the results of PCA acting on differently scaled data by showing the scatter plots. Visually we can see that the combination does carve the pattern of the dataset clearer than no preprocessing. Before applying PCA, the takeaway would always check the variance of each feature in the dataset, and if there is a large gap between the variances, scale the data with a proper scaler. Depending on your task, the definition of “proper” can vary.
Next time, we will be stretching the limits of this scaler + PCA combination, by bringing in synthetic outliers and categorical features.
I hope you like this post. Please feel free to leave comments.
Announcement: this series is a colaboration of Kefei Mo and Siyun Wang Siyun Wang. 1st release: 06/08/2020.