How to detect and deal with Multicollinearity | by Sushmitha Pulagam | Towards Data Science

Multicollinearity is one of the main assumptions that need to be ruled out to get a better estimation of any regression model ✌️
In this article, I’ll go through the impact of multicollinearity, how to identify, and when to fix this issue with a sample dataset.
👉 What is correlation?
The correlation between two variables can be measured with a correlation coefficient which can range between -1 to 1. If the value is 0, the two variables are independent and there is no correlation. If the measure is extremely close to one of these values, it indicates a linear relationship and highly correlated with each other. This means a change in one variable is associated with a significant change in other variables.The correlation can be calculated with the Pearson correlation coefficient and Spearman rank-order coefficient.
👉 What is Multicollinearity?
Multicollinearity occurs when there is a high correlation between the independent variables in the regression analysis which impacts the overall interpretation of the results. It reduces the power of coefficients and weakens the statistical measure to trust the p-values to identify the significant independent variables. Hence, we would not be able to examine the individual explanation of the independent variables on the dependent variable.
👉 When to fix Multicollinearity?
The good news is that it is not always mandatory to fix the multicollinearity. It all depends on the primary goal of the regression model.
The degree of multicollinearity greatly impacts the p-values and coefficients but not predictions and goodness-of-fit test. If your goal is to perform the predictions and not necessary to understand the significance of the independent variable, it is not a mandate to fix the multicollinearity issue.
👉 How to test Multicollinearity?
Correlation matrix / Correlation plotVariation Inflation Factor (VIF)
Correlation matrix / Correlation plot
Variation Inflation Factor (VIF)
A correlation plot can be used to identify the correlation or bivariate relationship between two independent variables whereas VIF is used to identify the correlation of one independent variable with a group of other variables. Hence, it is preferred to use VIF for better understanding.
VIF = 1 → No correlationVIF = 1 to 5 → Moderate correlationVIF >10 → High correlation
Let’s get started with code.....
Here is the link for the dataset [House Sales]
The features in the dataset are as below. Sale Price is the target variable and the remaining are independent features.
Identifying Multicollinearity with numerical variables
Now we will calculate the VIF scores for each independent variable.
def vif_scores(df):    VIF_Scores = pd.DataFrame()    VIF_Scores["Independent Features"] = df.columns    VIF_Scores["VIF Scores"] = [variance_inflation_factor(df.values,i) for i in range(df.shape[1])]    return VIF_Scoresdf1 = df.iloc[:,:-1]vif_scores(df1)
The VIF scores are higher than 10 for most of the variables. The individual coefficients and the p-values will be greatly impacted if we build a regression model with this dataset. We will proceed on how to fix this issue.
Fixing Multicollinearity — Dropping variables
We will consider dropping the features Interior(Sq Ft) and # of Rooms which are having high VIF values because the same information is being captured by other variables. Also, it helps to reduce the redundancy in the dataset.
Let us compare the VIF values before and after dropping the VIF values.
From the above, we can notice that the VIF scores have reduced for other variables also after dropping the high-value (Interior(Sq Ft) and # of Rooms) VIF features.
Fixing Multicollinearity — Combining variables
Next, we can observe that #of Bed and #of Bath can be combined to a single variable and it helps us to capture more information from both variables.
df5 = df4.copy()df5['Total Rooms'] = df4.apply(lambda x: x['# of Bed'] + x['# of Bath'],axis=1)X = df5.drop(['# of Bed','# of Bath'],axis=1)vif_scores(X)
From the above, we can notice that all three variables (Condo Fee, Tax, and Total Rooms) came up with satisfying VIF values and we can proceed further to build a regression model.
Summary
In this article, we learned how to identify and the ways to fix the multicollinearity issue with numerical values in regression analysis. This is an iterative process and we also need domain knowledge to decide on which variables to take an appropriate action. There are other techniques such as PCA and Regularization methods also to address this issue.
Please visit my GitHub link for complete [code]
Thank You for reading and Happy Learning! 🙂