Multicollinearity — Why is it bad?
By: Chetna Khanna
Tags: Statistical Analysis, Data Analysis, Regression, Data Science, Machine Learning

Multicollinearity — Why is it bad?

Photo by Andre Benz from Unsplash

While building our machine learning models or performing our exploratory data analysis, we use a lot of statistical terms to speak up for our data. These statistical terms have a deep hidden meaning behind them but we cannot deny the fact that they sound confusing. For instance, the terms collinearity, multicollinearity, correlation, all sounds the same, and we often end up googling about their exact meaning.

In this post, I will try to explain multicollinearity in simple terms. ✌️

What is Multicollinearity?

Multicollinearity is the presence of high correlations between two or more independent variables (predictors). It is basically a phenomenon where independent variables are correlated.

🤔 😳 Daunting?? No need to worry! Just keep reading.

Let us first understand what the term correlation means.

Correlation is the association between variables and it tells us the measure of the extent to which two variables are related to each other. Two variables can have positive (change in one variable causes change in another variable in the same direction), negative (change in one variable causes change in another variable in the opposite direction), or no correlation.

It is easy to remember these terms if we keep some examples in our minds. 😊

A simple example of positive correlation can be weight and height. The taller you are, the heavier you weigh (this is considered a general trend if we leave the exception case aside). A simple example of a negative correlation can be the altitude and oxygen level. The higher you go, the lower the oxygen level is. A simple example of no correlation can be the depth of the sea and the number of apples bought from the store. None of them is related to the other.

Hope this helped in understanding the concept of correlation. 😀

Now, coming back to our multicollinearity term. Simply put, we can say that multicollinearity occurs when two or more predictors in regression analysis are highly related to one another. For example, the level of education and annual income. It is generally considered that the more educated you are, the more you earn. Thus, one variable can be easily predicted using another variable. If we keep both these variables in our analysis, it can cause problems for our model. 🙆

Types of Multicollinearity

There are two basic kinds of multicollinearity:

Structural multicollinearity: This type of multicollinearity is caused by the researchers (people like us) who create new predictors using the given predictors in order to solve the problem. For example, the creation of variable x² from the predictor variable x. Thus, this type of multicollinearity is a byproduct of the model we specify and not present in the data itself. Data multicollinearity: This type of multicollinearity is the result of poorly designed experiments that are purely observational. Thus, it is present in the data itself and has not been specified/designed by us.

Note: In an observational study, we survey members of a sample without trying to affect them. For example, asking people whether they drink milk or coffee before going to bed and saying that the data shows that people who drink milk are likely to go to sleep earlier is an observational study. Read more about it here!

In a few places, you might come across terms like perfect multicollinearity and high multicollinearity as the two different types of multicollinearity.

Perfect multicollinearity occurs when two or more independent predictors in a regression model exhibit a perfectly predictable (exact or no randomness) linear relationship. The correlation, in this case, is equal to +1 or -1. For example, weight in pounds and weight in kilograms. However, we rarely face issues of perfect multicollinearity in a dataset.

High/Imperfect/Near multicollinearity occurs when two or more independent predictors are approximately linearly related. This is a common type and is problematic to us. All our analyses are based on detecting and dealing with this type of multicollinearity.

What kind of problems does multicollinearity cause? 🤔

The presence of correlated predictors does not provide any unique or new information to our regression analysis. In fact, it may cause our analysis or statistical inference about the data to be less reliable by making our coefficients unstable. 🙄

Consider a problem that has four predictors — Age, Gender, Number of hours of study, and level of parental education. The dependent variable (variable to be predicted) is the Student’s performance in exams. All this information is collected through a survey.

Let us suppose that the two predictors — Number of hours of study and level of parental education have a high positive correlation of 0.9.

To know which variable is more predictive of the Student's performance in exams all these variables are entered into a regression model with a basic expectation that both the number of hours of study and level of parental education would be positively correlated to performance (maybe one more than the other or maybe not). But, the model surprised by saying that none of these two variables are significant.

Does this mean that a student’s performance is not dependent on the number of hours he/she study or the level of parental education? Well, this sounds a bit incorrect.

Let us understand what caused the model to say so.

When highly correlated predictors are entered into a regression model, each of their effects will be estimated while controlling the effect for a predictor that explains all of the variability in performance that it would have itself explained. This causes neither of the predictor to be a significant predictor. The same happened in our case too.

We might even witness the R-squared value returned by the model to be high indicating that one or more predictors in the model must and should be a good predictor.

Thus, the presence of multicollinearity caused the model to give results that are both misleading and confusing and we do not want this. 🙅🏻

Do we always have to fix multicollinearity? 🙋🏻

Well, it depends on the goal of the problem. If our goal is to understand the significance of the independent variables, then go ahead and fix this issue as it affects the coefficients and p-value. However, if we do not have any such goal and we are more interested in making our predictions, then we might not worry about multicollinearity.

In short, multicollinearity is a problem for causal inference or creates difficulties in casual inference but it is not a problem for prediction or forecasting.

Fixing this issue can also be dependent on the severity of multicollinearity. We can ignore small multicollinearity in most of the cases. So, ask yourself what problem are you solving and what your end goal is, and then proceed further. It is indeed always good to know what your data is and how it behaves and then work on it.

There are several ways to detect and deal with multicollinearity out of which Variation Inflation Factor (VIF) is the most common one. Keeping the article short for now. I will cover these methods in another blog post. ✌️

Thus, we understood that the knowledge about multicollinearity can help us a lot in performing analysis with our data, so it is advised to deal with multicollinearity seriously.