How to Use Pairwise Correlation For Robust Feature Selection | by Bex T. | Towards Data Science

In my last article on the topic of Feature Selection, we focused on a technique to remove features based on their individual properties. In this post, we will look at a more reliable, more robust method that lets us see the connection between features and decide if they are worth keeping. This method, as you have read from the title, uses Pairwise Correlation.
First of all, let’s briefly touch on Pearson’s correlation coefficient — commonly denoted as r. This coefficient can be used to quantify the linear relationship between two distributions (or features) in a single metric. It ranges from -1 to 1, -1 being a perfect negative correlation and +1 being a perfect positive correlation.
For instance, body measurements often have a strong positive correlation. You can expect someone tall to have longer arms and legs, or someone with broad shoulders to weigh more. Also, the sales of ice cream are negatively correlated with temperature, or the further you run the slower you get, etc.
For an in-depth guide on how to use, interpret and understand correlation coefficient, refer to my separate article.
But what does correlation coefficient have to do with Machine Learning or feature selection? Well, let’s say the correlation coefficient between two features is 0.85. This means that 85% of the time, you can predict feature 2 just using the values of feature 1. In other words, if you have feature 1 in your dataset, feature 2 won’t bring much new information. That’s why there is no point in keeping feature 2 since it only adds to complexity when training a model.
Using pairwise correlation for feature selection is all about that — identifying groups of highly correlated features and only keeping one of them so that your model can have as much predictive power using as few features as possible.
The quickest and often the best method of identifying highly correlated features is to use a correlation matrix. This matrix shows the correlation between every single pair of numeric features in the dataset.
Let’s see one using the Melbourne Housing dataset which contains 13 numeric features:
Calculating the matrix is easily done by calling the .corr() method on the DataFrame. Next, we pass it to Seaborn's heatmap function which is commonly done to generate a nice visual:
Yes, the default heatmap is not so great. Let’s give it a make-up:
First, we create a custom diverging palette (blue -> white -> red). Then, we center the color bar around 0, enable the annotations to see each correlation and use 2 decimal points.
As you see, the matrix shows each correlation between pairs twice since the correlation between A and B is the same as the correlation between B and A. Also, the diagonal consists of 1s representing the correlation of features with themselves. All this extra information can be distracting. That’s why let’s get rid of the upper part of the matrix:
In line 4 of the above code snippet, we create a 2D boolean mask. First, np.ones_like creates a 2D numpy array with the same shape as our correlation matrix filled with True values. Then, passing this array to np.triu converts it to a boolean mask suited to our needs. We just pass this mask to heatmap's mask argument which gives us the beautiful plot above.
Here is a small function to plot these perfect correlation matrices:
Even though we could just find out all the highly correlated features programmatically, we should still visually explore them. Because a heatmap like above helps us see if the correlations make sense. For example, the number of newborn babies can be highly correlated with the number of storks in the vicinity even though that does not make any sense. That’s why do visual exploration to see if groups of features are actually related to each other and whether it makes sense to look for a connection between them.
Now, let’s see how we can actually remove highly correlated features. We will use a different dataset this time — Ansur Dataset. It contains every imaginable body measurement of 6000 US Army personnel and it contains 98 numeric features. It is an excellent dataset to practice your feature selection skills since there are many correlated features in this dataset:
We will only be working with numeric features, so, we will isolate them:
BTW, trying to plot the correlation matrix of a dataset with this many features is useless.
Our aim is to remove strong correlations, both negative and positive. So, we will build the matrix this time with absolute values of correlations by chaining .abs() method on .corr():
Next, we again create a boolean mask which will be used to subset the matrix:
Using the mask method of Pandas DataFrames (correlation matrix is a DataFrame) puts NaN values to the upper half and diagonal of the matrix:
>>> reduced_matrix.iloc[:5, :5]
Next, we need to set a threshold to decide whether to drop a feature or not. This threshold should be chosen carefully and only with a thorough understanding of the dataset. For our case, we will choose a threshold of 0.9:
The above list comprehension finds all other columns that meet the threshold of 0.9. Using any allows us not to use a nested for loop which would have been like this:
The list comprehension is surely a much more elegant solution. Note that if we hadn’t put NaNs to the upper half of the matrix and the diagonal, we would lose all correlated features instead of keeping one of them.
Here is a function that does the above operations on an arbitrary threshold and returns a list of columns to drop:
Now, let’s actually drop those columns:
Now, let’s see if we made the right choice by dropping so many features. We will check this by training two RandomForestRegressors: one on the full dataset and one on the reduced, feature-selected dataset. The target will be weight in pounds - Weightlbs:
We got a very good result even with so many features. Now, let’s repeat the above on the feature-selected DataFrame:
We only got a 2% reduction in test score while taking two times less runtime.
In summary, using pairwise correlation allows us to detect highly correlated features which bring no new information to the dataset. Since these features only add to model complexity, increase the chance of overfitting, and require more computations, they should be dropped.
However, get a thorough understanding of your dataset by doing proper EDA before you use this technique. Always look out for correlations that do not make sense, i. e. randomly correlated features with no connection.
If you don’t know what to read next, here I have picked some for you: