Multicollinearity in Data Science
By: Iftekher Mamun
Tags: Statistics, Linear Regression, Data, Data Science, Multicollinearity

Multicollinearity in Data Science

What is it and why is it a problem?

Recently at a meetup regarding AI, the topic of statistics came up during discussion. The statistical method is a great tool to quantify your test and check for significant impact between your independent variables (variables that you control and can change- think of the X-axis terms in a graph) and how it affects the dependent variable (the variable that changes due to the change of your independent variable- the y-axis terms in a graph).

For example, y=2x would mean that for every change in X (our independent variable), y would change by 2 (change of the dependent variable). This is quite simple when we have one independent variable for one dependent variable. However, the issue becomes a bit more complex when we have multiple variables in our dataset- referred to as features that can affect our target variable (the variable we are trying to predict using linear regression).

It becomes even more complicated when each of the features become “dependent” on each other. Another way of saying this is that when we change an independent variable and expect a change in a dependent variable, we see that another independent variable have also changed. These two independent variables are now codependent, or collinear of each other. Add in more features that are collinear of each others and we get multicollinearity.

Multicollinearity occurs when independent variables in a regression model are correlated.

There are two main types of multicollinearity. The first one is structural (independent variable x is squared), which is simply a byproduct and since more often than not that you will create it using an existing independent variable, you will be able to track it. Think of when we have a data set and we decide to use log to either scale all features or normalize them. That would be an example of structural multicollinearity. The second, and more dangerous in my opinion, is the data multicollinearity. This is already embedded within the feature set of your dataframe (pandas dataframe) and are much harder to observe.

What problems can this cause? It can reduce our overall coefficient as well as our p-value (known as the significance value) and cause unpredictable variance. This will lead to overfitting where the model may do great on known training set but will fail at unknown testing set. As this leads to higher standard error with lower statistical significance value, multicollinearity makes it difficult to ascertain how important a feature is to the target variable. And with a lower significance value, we will fail to reject the null, which leads to type II error for our hypothesis testing.

The next question I was asked during discussion was the same question you have. How do we notice them? Well, the best way I found was to test each independent feature against each other. This is great and not much work if you have limited features to work with, which is what I did for one of my projects that involved linear base model. But when you have features in the numbers of hundreds or more, it is much more difficult. At that point, it also becomes a dimensionality issue (known as the curse of dimensionality). The best at that point is to use PCA (Principal Component Analysis) to reduce your features.

Another method of detection is to look at the coefficient score.

If you can identify which variables are affected by multicollinearity and the strength of the correlation, you’re well on your way to determining whether you need to fix it. Fortunately, there is a very simple test to assess multicollinearity in your regression model. The variance inflation factor (VIF) identifies correlation between independent variables and the strength of that correlation.

Using Variance Inflation Factor- VIF- we can determine if two independent variables are collinear with each other. When measuring, if the two features have a VIF of 1, then they are not collinear of each other (ie there are no correlation between these two features). However, as the numbers increases, the higher they are correlated with each other. If VIF returns a number greater than 5, then those two features should be reduced to one using PCA.

There is also another final statistical tool that is great for analyzing the variance between features. It is ANOVA which stands for Analysis of Variances. Generally, the higher the variance between the variables, the less likely that they are related (or correlated). But perhaps this will be a topic for a future discussion. I will link my source for this discussion below, which also contains a great example you can try out.