Multicollinearity in Regression. Why it is a problem? How to track and... | by Songhao Wu | Towards Data Science

Multicollinearity happens when independent variables in the regression model are highly correlated to each other. It makes it hard to interpret of model and also creates an overfitting problem. It is a common assumption that people test before selecting the variables into the regression model.
I encountered a serious multicollinearity issue before when I built the regression model for time series data. I created multiple features based on different time periods like 1-month total return, 6-month total return, and 1-year total return to have more input variables. However, these features are highly correlated with each other. For example, if one stock has performed well for the past year, then it is very likely to have done well for the recent month. I would need to either drop some of these variables or find a way to make them less correlated. I will explain later in the article different ways to solve the problem.
When independent variables are highly correlated, change in one variable would cause change to another and so the model results fluctuate significantly. The model results will be unstable and vary a lot given a small change in the data or model. This will create the following problems:
It would be hard for you to choose the list of significant variables for the model if the model gives you different results every time.Coefficient Estimates would not be stable and it would be hard for you to interpret the model. In other words, you cannot tell the scale of changes to the output if one of your predicting factors changes by 1 unit.The unstable nature of the model may cause overfitting. If you apply the model to another sample of data, the accuracy will drop significantly compared to the accuracy of your training dataset.
It would be hard for you to choose the list of significant variables for the model if the model gives you different results every time.
Coefficient Estimates would not be stable and it would be hard for you to interpret the model. In other words, you cannot tell the scale of changes to the output if one of your predicting factors changes by 1 unit.
The unstable nature of the model may cause overfitting. If you apply the model to another sample of data, the accuracy will drop significantly compared to the accuracy of your training dataset.
Depending on the situation, it may not be a problem for your model if only slight or moderate collinearity issue occurs. However, it is strongly advised to solve the issue if severe collinearity issue exists(e.g. correlation >0.8 between 2 variables or Variance inflation factor(VIF) >20 )
The first simple method is to plot the correlation matrix of all the independent variables.
I used the housing data from the Kaggle competition. The goal of the competition is to use the housing data input to correctly predict the sales price. I have selected a few numerical variables to be included in my model here.
#plot color scaled correlation matrixcorr=house_selected.corr()corr.style.background_gradient(cmap='coolwarm')
After plotting the correlation matrix and color scaling the background, we can see the pairwise correlation between all the variables. I have included the dependent variable ‘SalePrice’ here as well. This is because it is a secret trick for me when I try to select the independent variables to be included in the model.
When you are clueless about which variables to include in the model, just do a correlation matrix and select those independent variables with high correlation with dependent variable.
Back to the Multi-Collinearity issue, we can see that from the correlation matrix, quite a few variables are correlated to each other. There is one pair of independent variables with more than 0.8 correlation which are total basement surface area and first-floor surface area. Houses with larger basement areas tend to have bigger first-floor areas as well and so a high correlation should be expected.
The second method to check multi-collinearity is to use the Variance Inflation Factor(VIF) for each independent variable. It is a measure of multicollinearity in the set of multiple regression variables. The higher the value of VIF the higher correlation between this variable and the rest.
#Compute VIF data for each independent variablefrom statsmodels.stats.outliers_influence import variance_inflation_factorvif = pd.DataFrame()vif["features"] = house_selected.columnsvif["vif_Factor"] = [variance_inflation_factor(house_selected.values, i) for i in range(house_selected.shape[1])]vif
If the VIF value is higher than 10, it is usually considered to have a high correlation with other independent variables. However, the acceptance range is subject to requirements and constraints. From the results, we can see that most features are highly correlated with other independent variables and only two features can pass the below 10 threshold.
Variable Selection
Variable Selection
The most straightforward method is to remove some variables that are highly correlated to others and leave the more significant ones in the set. For example, when we plot the correlation matrix with ‘SalePrice’ included, we can see that Overall Quality and Ground living area have the two highest correlations with the dependent variable ‘SalePrice’ and thus I will try to include them in the model.
We can see that using simple elimination, we are able to reduce the VIF value significantly while keeping the important variables. However, some of the variables like Overall Quality and Years of Built still have high VIF value and they are important in predicting housing price. How? Sometimes we can use small tricks as described in the second method later to transform the variable.
2. Variable Transformation
The second method is to transform some of the variables to make them less correlated but still maintain their feature. What do I mean by this? In the housing model example, I can transfer ‘years of built’ to ‘age of the house’ by subtracting current year by years of built. For example, if the year of building is1994, then the age of the house is 2020–1994=26 years.
After I convert the years of building to house age, the VIF for the new ‘House_age’ factor drops to an acceptable range and the VIF value for overall quality also drops.
In the example of time series analysis which I mentioned at the beginning, I also converted variables to make them less correlated. For example, the total return for the past 1 month is highly correlated with the past 6 months' total return. I subtracted the past 1-month return from the past 6-month return to get the new variable on the previous 5-month return which does not include the past month. The correlation results are much more acceptable and I was able to include both variables as my model features.
3. Principal Component Analysis
Principal Component Analysis(PCA) is commonly used to reduce the dimension of data by decomposing data into a number of independent factors. It has many applications like simplifying model calculation by reducing the number of predicting factors. However, in our case here, we will just use the character of variable independence for PCA to remove the multi-collinearity issue in the model.
#Create the new data frame by transforming data using PCAimport numpy as npfrom sklearn.decomposition import PCApca = PCA(n_components=6)components=pca.fit_transform(house_adjusted)componentsDf=pd.DataFrame(data=components,columns=['component 1','component 2','component 3','component 4','component 5','component 6'])#Calculate VIF for each variable in the new data framevif = pd.DataFrame()vif["features"] = componentsDf.columnsvif["vif_value"] = [variance_inflation_factor(componentsDf.values, i) for i in range(componentsDf.shape[1])]vif
I still keep the same number of variables compared to the original data and we can see that now the 6 variables are not correlated to each other at all. We can use the new 6 variables as the independent variables to predict housing prices.
The drawback of this method is also very obvious. After PCA transformation, we do not have the identity for each variable and it will be hard to interpret the results.
We should check the issue of Multi-Collinearity every time before we build the regression model. VIF would be an easy way to look at each independent variable to see whether they have a high correlation with the rest. A correlation matrix would be useful to select important factors when you are not sure which variables to select for the model. The correlation matrix also helps to understand why certain variables have high VIF values.
In terms of methods to fix the multi-collinearity issue, I personally do not prefer PCA here because model interpretation will be lost and when you want to apply the model to another set of data you need to PCA transform again. Thus, we should try our best to reduce the correlation by selecting the right variables and transform them if needed. It is your call to decide whether to keep the variable or not when it has a relatively high VIF value but also important in predicting the result. Trial and error is always the case to include different sets of variables, build the model and test it against testing data to see whether there is any overfitting.
Do you think you are finally done with all the checks with statistical assumptions before constructing a model? No, you are not! If you want to know more about other statistical assumptions in a regression model, refer to my other article below on Normality Assumption in the regression model.