Think twice before you use Principal Component Analysis in supervised learning tasks | by Wanshun Wong | Towards Data Science

Principal Component Analysis (PCA) is one of the most popular machine learning technique. It reduces the dimension of a given data set, making the data set more approachable and computationally cheaper to handle, while preserving most patterns and trends. This makes PCA an excellent tool for exploratory data analysis. However, we claim that using PCA in supervised learning tasks, in particular in a pipeline such as
data → PCA → supervised learning algorithm
usually does more harm than good. We will explain the major issues in this article.
We briefly recall how PCA works. Roughly speaking, PCA geometrically project a data set onto fewer dimensions, where the new variables are called principal components. This is done in such a way that the principal components are orthogonal and have the largest possible variances. The figure below illustrates a toy example where 2-dimensional data is projected onto an 1-dimensional space,
where the horizontal dotted line corresponds to the direction with maximal variance. Note that PCA is an unsupervised method, meaning that it does not make use of any labels in the computation.
Let us further assume the above toy example belongs to a binary classification task, and the two classes are coloured red and black.
We can easily see that the vertical coordinate would be a powerful feature in classifying the red class. Unfortunately, if we apply PCA then such feature would be gone. This phenomenon happens because the labels might not be correlated with the variance of the features. Even though real world data set would not be this extreme, PCA is still quite susceptible to throwing strong classification signals away.
Most supervised learning algorithms, from logistic regression, tree-based algorithm to neural network, are able to evaluate the importance of input features. These feature importances are very valuable as they
help users to discover bugs and problems such as data leakageindicate areas where further exploration and feature engineering can be beneficialindicate what kind of data is worth getting more
help users to discover bugs and problems such as data leakage
indicate areas where further exploration and feature engineering can be beneficial
indicate what kind of data is worth getting more
and so on. However, this is based on the assumption that the features are interpretable, allowing users to take relevant actions.
For example, suppose we know that the number of bedrooms is the most important feature of our current model in predicting house prices. Then we can try to get more data about bedroom sizes, the number of windows in bedrooms, etc.
On the other hand, consider the case where the input features of our model are principal components, which are linear combinations of the original features. Even if we know the most important feature is the principal component given by
0.18 * number_of_bedroom + 0.15 * house_size + ......
it is much harder for us to decide the follow-up actions.
Tree-based algorithms such as Random Forest and XGBoost are arguably the best off-the-shelf supervised learning algorithms partially because they are invariant to scaling of features. In fact they are even invariant to monotonic transformation of features. This is particularly useful when our data set contains features with different units. For example, we might have age, height, weight and salary in a census data set. We do not need to worry about whether we should use kilogram or pound for weights, how to normalize the salaries, because our tree-based algorithms will simply work the same way.
However, if we have a pipeline of PCA followed by a supervised learning algorithm, then we need to worry about feature scaling and normalization again. This is because PCA is scale sensitive. Even if we use a supervised learning algorithm that requires feature scaling and normalization, extra care is required. Blindly standardizing all features might distort the data and make variation due to noise looks significant, twisting the calculation of principal components.
Q: What should be used instead to reduce the dimension of data sets in supervised learning tasks?
A: We can first compute e.g. the Pearson correlation coefficient or Spearman correlation coefficient between each feature and the label. Then we discard those features that are not correlated with the label.
We can also train a simple model like Lasso and only keep the features with high importance. The scikit-learn documentation has a comprehensive list of feature selection methods.
Q: Are there any scenarios in supervised learning where we may use PCA?
A: PCA is great for exploring and understanding a data set. For pipelines where PCA is followed by a supervised learning algorithm, they are not suitable for model iterations for reasons listed above. However, they are handy for tasks such as quickly construct model performance benchmarks.
[1] provides a short summary of PCA. The figures on the limitation of PCA is quite enlightening.[2] is a comprehensive tutorial of PCA. It contains mathematic details as well as intuitive understanding of various aspects of the algorithm, with the help of many great examples.
[1] provides a short summary of PCA. The figures on the limitation of PCA is quite enlightening.
[2] is a comprehensive tutorial of PCA. It contains mathematic details as well as intuitive understanding of various aspects of the algorithm, with the help of many great examples.
J. Lever, M. Krzywinski, and N. Altman. Principal component analysis (2017), Nature Methods 14, 641–642J. Shlens. A Tutorial on Principal Component Analysis (2014), arXiv
J. Lever, M. Krzywinski, and N. Altman. Principal component analysis (2017), Nature Methods 14, 641–642
J. Shlens. A Tutorial on Principal Component Analysis (2014), arXiv