How do you apply PCA to Logistic Regression to remove Multicollinearity? | by Rukshan Pramoditha | Towards Data Science

Multicollinearity occurs when features (input variables) are highly correlated with one or more of the other features in the dataset. It affects the performance of regression and classification models. PCA (Principal Component Analysis) takes advantage of multicollinearity and combines the highly correlated variables into a set of uncorrelated variables. Therefore, PCA can effectively eliminate multicollinearity between features.
In this post, we’ll build a logistic regression model on a classification dataset called breast_cancer data. The initial model can be considered as the base model. Then, we’ll apply PCA on breast_cancer data and build the logistic regression model again. After that, we’ll compare the performance between the base model and this model. Finally, we’ll build a machine learning Pipeline to combine the multiple steps of applying logistic regression with PCA. After reading this post, you’ll be able to apply PCA to logistic regression models.
Let’s get started!
This dataset is one of the built-in datasets in Scikit-learn. According to the Scikit-learn documentation, it has 30 features of 569 samples. The target variable has 2 classes called 0 (not having breast cancer) and 1 (having breast cancer). Since the target variable has only 2 classes, the dataset is used for binary classification. There are no missing values in the dataset. All the values are numerical. Therefore, no cleaning is required!
The heatmap can be used to visualize the correlation coefficients of continuous variables with a beautiful colourmap.
As you can see in the heatmap, some of the features in the dataset are highly correlated with each other. So, there exists multicollinearity.
The following code block builds a logistic regression model on breast_cancer data.
The base model is very good. It is not overfitting. It performs well on new unseen data. Let’s see whether we can improve the performance of the model by applying PCA.
PCA is a linear dimensionality reduction technique (algorithm) that transforms a set of correlated variables (p) into a smaller k (k<p) number of uncorrelated variables called principal components while retaining as much of the variation in the original dataset as possible. In the context of Machine Learning (ML), PCA is an unsupervised machine learning algorithm which is used for dimensionality reduction. If the variables are not measured on a similar scale, we need to do feature scaling before applying PCA for our data. This is because PCA directions are highly sensitive to the scale of the data. The most important part in PCA is selecting the best number of components for the given dataset.
First, we apply PCA keeping all components equal to the original number of dimensions (i.e., 30) and see how well PCA captures the variance of our data.
The first component alone captures about 44% variability in the data and the second one captures about 19% variability in the data and so on. The first 6 components together capture about 88.76% variability in the data. We’re interested to keep the first 6 components.
Now, we get the transformed dataset with 6 components. For this, we need to run PCA again by setting n_components=6.
Now, we can use this transformed dataset instead of the original breast_cancer dataset to build a logistic regression model. Here are the differences between the two datasets.
The original dataset has 30 features while the transformed dataset has 6 components.
The transformed dataset only captures about 88.76% variability in the original dataset.
The corresponding values of the two datasets are completely different.
Some variables in the original dataset are highly correlated with one or more of the other variables (multicollinearity). No variable in the transformed dataset is correlated with one or more of the other variables.
fig = plt.figure(figsize=(10, 8))sns.heatmap(X_pca.corr(),             annot=True)
We cannot see any correlation between components. This is because PCA has transformed the set of correlated variables in the original dataset into a set of uncorrelated variables.
The following code block builds a logistic regression model on the transformed dataset (the dataset obtained by applying PCA).
You can compare this output with the previous output of our base model. The test accuracy has increased by 3%. Both false positives and false negatives have also been reduced. However, the transformed dataset (the dataset obtained by applying PCA), only captures about 88.76% variability in the original dataset. Therefore, what’s the reason behind the performance increase in this model? The obvious answer is PCA has effectively eliminated the multicollinearity!
A machine learning Pipeline simplifies the entire process by combining the multiple steps of applying logistic regression with PCA. Here are the steps:
The log_reg_model Pipeline simplifies the training process by sequentially applying a list of transformers and a final predictor. In our Pipeline:
StandardScaler() is a transformer.PCA() is a transformer.LogisticRegression() is a predictor.
StandardScaler() is a transformer.
PCA() is a transformer.
LogisticRegression() is a predictor.
Now, we can train all the estimators by a single .fit() call.
log_reg_model.fit(X,y)
Now, we’ve done the job promised in the introduction.
PCA is useful to remove multicollinearity. It acts as a data preprocessing step. PCA has many other use cases. It is just one of the dimensionality reduction techniques. You can find more about those things by reading the following article written by me:
towardsdatascience.com
My readers can sign up for a membership through the following link to get full access to every story I write and I will receive a portion of your membership fee.
Sign-up link: https://rukshanpramoditha.medium.com/membership
Thank you so much for your continuous support! See you in the next story. Happy learning to everyone!
Special credit goes to Gabriella Clare Marino on Unsplash who provides me with the cover image for this post. The written content, code samples, other images and content links provided in this post are copyrighted by the author.