Understanding Multicollinearity and How to Detect it in Python | by Terence Shin | Towards Data Science

Be sure to subscribe here or to my personal newsletter to never miss another article on data science guides, tricks and tips, life lessons, and more!
Over the next few articles, I want to write about some really powerful topics related to regression analysis. For the longest time, I didn’t think there was much to linear regression — I simply thought it was the simplest machine learning model that was nothing more than a line of best fit.
However, as I continue to learn more about regression analysis and what it has to offer, I’m realizing that there’s a lot of powerful tools and tricks that most people don’t know about.
And so, to start it off, I wanted to talk about multicollinearity. Specifically, I’m going to cover:
What multicollinearity isWhat causes multicollinearityWhy it’s bad for linear regression modelsHow you can detect and eliminate multicollinearityHow to detect multicollinearity in Python
What multicollinearity is
What causes multicollinearity
Why it’s bad for linear regression models
How you can detect and eliminate multicollinearity
How to detect multicollinearity in Python
With that said, let’s dive into it!
Multicollinearity (or collinearity) occurs when one independent variable in a regression model is linearly correlated with another independent variable.
An example of this is if we used “Age” and “Number of Rings” in a regression model for predicting the weight of a tree.
Because there is a high correlation between the age of a tree and the number of rings that a tree has (generally, one ring per year), multicollinearity would be present in this model.
Be sure to subscribe here or to my personal newsletter to never miss another article on data science guides, tricks and tips, life lessons, and more!
In order to understand why multicollinearity is bad, we’re going to have to look at how the regression coefficients (or the model’s parameters) are estimated. This part involves a little bit of linear algebra, but feel free to skip this section if you’re not interested.
NOTE: Don’t worry if you skip this part, as long as you focus on why multicollinearity is bad and how you can eliminate it. :)
Note that the regression coefficients refer to the “slope” of each variable — in the equation y = B0 + B1x+ B2x2 , B1 and B2 are the regression coefficients. Remember that the point of a linear regression model is finding the best regression coefficients that represents the data.
In order to find the optimal regression coefficients we want to find values for them that minimize the squared error. After doing a little bit of math, you can find the optimal parameters (B1, B2, ..., Bp) with the following equation:
where B_hat is the vector that includes all individual regression coefficients and X is the design matrix which consists of the predicting variables.
Notice that we assume in the equation about that (XTX) is invertible in order to estimate B_hat.
If the columns of X are linearly dependent of each other (i.e. if multicollinearity is present), XTX is not invertible, and this results in several consequences, which you’ll see in the next section.
To recap, XTX has to be invertible in order to properly estimate regression coefficients for a multiple regression model. If XTX is not invertible, it means that the columns of X are linearly dependent of each other and multicollinearity is present.
Ultimately, the presence of multicollinearity results in several problems:
The fitted regression coefficients (beta hat) will change substantially if one of the values of one of the x variables is changed only a bit.The variance of the estimated coefficients will be inflated, which means that it will be hard to detect statistical significance. Furthermore, it’s possible that the F statistic is significant but the individual t statistics are not.Ultimately, multicollinearity makes prediction less accurate. For a given model, the underlying assumption is that the relationships among the predicting variables, as well as their relationship with the target variable, will be the same. However, when multicollinearity is present, this is less likely to be the case.
The fitted regression coefficients (beta hat) will change substantially if one of the values of one of the x variables is changed only a bit.
The variance of the estimated coefficients will be inflated, which means that it will be hard to detect statistical significance. Furthermore, it’s possible that the F statistic is significant but the individual t statistics are not.
Ultimately, multicollinearity makes prediction less accurate. For a given model, the underlying assumption is that the relationships among the predicting variables, as well as their relationship with the target variable, will be the same. However, when multicollinearity is present, this is less likely to be the case.
A simple method to detect multicollinearity in a model is by using something called the variance inflation factor or the VIF for each predicting variable.
VIF measures the ratio between the variance for a given regression coefficient with only that variable in the model versus the variance for a given regression coefficient with all variables in the model.
A VIF of 1 (the minimum possible VIF) means the tested predictor is not correlated with the other predictors.A VIF of 1 (the minimum possible VIF) means the tested predictor is not correlated with the other predictors.
The higher the VIF,
The more correlated a predictor is with the other predictors
The more the standard error is inflated
The larger the confidence interval
The less likely it is that a coefficient will be evaluated as statistically significant
An acceptable VIF is if it’s less than the max of 10 and 1/1-R2model (see below):
To give an example, I’m going to use Kaggle’s California Housing Prices dataset.
First, I imported all relevant libraries and data:
import pandas as pdimport numpy as npfrom statsmodels.stats.outliers_influence import variance_inflation_factor
Next, for simplicity, I selected only 3 columns to be my features (X variables) and removed all nulls and infinite values:
df = pd.read_csv('housing.csv')df = df[~df.isin([np.nan, np.inf, -np.inf]).any(1)]X_variables = df[['total_rooms','total_bedrooms','median_income']]
Finally, I calculated the VIF for my X variables:
vif_data = pd.DataFrame()vif_data["feature"] = X_variables.columnsvif_data["VIF"] = [variance_inflation_factor(X_variables.values, i) for i in range(len(X_variables.columns))]
Just like that, we get the final result:
Intuitively this makes complete sense. Total rooms and total bedrooms are far above the VIF threshold, indicating that there is high collinearity between these variables. We can intuitively understand this because there is a strong correlation between the number of rooms and number of bedrooms (the more bedrooms, the more rooms, and vice versa).
Be sure to subscribe here or to my personal newsletter to never miss another article on data science guides, tricks and tips, life lessons, and more!
I hope you found this useful and that you learned something new! Multicollinearity is an extremely important concept for regression analysis, so it’s definitely an important concept to understand.
Not sure what to read next? I’ve picked another article for you:
towardsdatascience.com
and another one!
towardsdatascience.com
If you enjoyed this, follow me on Medium for more
Interested in collaborating? Let’s connect on LinkedIn