Assumptions of Regression Analysis for Every Researcher
By: Karan Sharma
Tags: Regression Data Analysis, Data Science, Linear Regression, Regression Modeling, Statistics

Assumptions of Regression Analysis for Every Researcher

Photo by Rob Necciai on Unsplash

Simple Linear Regression is one of the foundations of machine learning modelling. It is the very first algorithm that you learn in your journey towards becoming a data scientist. What is the one thing that makes simple linear regression special? Well, there are five!

There are five regression assumptions. You must know all of them and consider them before performing any type of regression analysis. To begin with, let us look at the old school textbook definition of linear regression.

‘A linear regression is a statistical method of determining a linear approximation of a causal relationship between two or more variables.’

In short, you can predict a dependent variable based on single or more independent variables. The regression equation helps you do this. If we take Y as the dependent variable and X as the independent variable, the dependent variable Y is a function of the independent variable X. This is mathematically expressed as:

Y = β0 + β1X1 + ε

Where,

Y is a variable subject to prediction

β0 is the estimate of the regression constant; the intercept

β1 is the estimate of beta one multiplied by X1

β1 quantifies the effect of independent X on the dependent Y

X1 is the sample data; predictor variable

Ε is the difference between the observed and the predicted values

With the following equation above, you can predict the value of Y due to a change in the value of X. So far; the above information is sufficient to understand the importance of the five assumptions.

Now, let’s get straight to the point.

Linearity

As the name suggests, the first assumption is nothing but a linearity check; the regression equation should be linear in function. Each independent variable X is multiplied by its respective coefficient β1 and summed up to predict the value of the Y variable.

So how do you determine linearity? The easiest method is to take a single independent variable such as X1 and create a scatter plot for the same against the Y variable. If the data points form a trend line that resembles a pattern, the regression model is applicable.

If the data points do not form a trend line or resemble a non-linear, curvy pattern, linear regression is not suitable. If the relation is non-linear, you need to treat your raw data accordingly before building a regression model.

No Endogeneity

In statistics, endogeneity translates to a situation where the independent X variable exhibits some correlation to the error variable. This problem is commonly referred to as the Omitted Variable Bias. It is introduced to your regression model when you fail to include a relevant variable. Let us look at this step by step:

In a regression equation, each independent X variable explains Y; expressing the correlation Y is also explained by the omitted variable; expressing the correlation This leads to a situation in which the omitted variable becomes correlated to independent X But you as a researcher forgets to include this relevant variable in your model Therefore, the error ends up being correlated to everything else

Still, confused? Let us understand this better with the help of an example.

Mumbai is one of the densest cities on the planet. There is an absolute scarcity of open spaces. Due to this, real estate prices have sky-rocketed over the past few years. Imagine if you had to predict the price of a newly built building apartment in Mumbai based on size; square feet. The common assumption is that the bigger the apartment, the higher the price. So, you decide to build your regression model only to realize that the smaller the houses’ size, the higher the price.

How is this possible? How is big real estate cheaper than smaller apartments of a newly constructed building? On further investigation, you observe that all of Mumbai was in the sample. At the same time, the building apartment was located in South Bombay, one of the most expensive and highly demanded real estate destinations. Here, we have omitted South Bombay as the exact location for the change in the price of the apartment. Now that you take South Bombay as a different variable in your sample, your regression model will fall into the right trajectory; larger apartment prices are more expensive outside South Bombay.

From the above example, we can clearly understand that an incorrect conclusion of a variable can often lead to biased and counter-intuitive estimates.

Normality and Homoscedasticity

Let’s divide the above into two separate parts for better understanding. Normality refers to a normal distribution. It is required to make inferences on the population based on a sample. We usually assume a normal distribution for most of the cases, courtesy of the Central Limit Theorem.

Homoscedasticity means to have equal variance. What does this imply? It means that the error terms must have equal variance, one with the other. Can there be a dataset that has unequal variance? Of course! It is termed heteroscedastic data. Long story short, you need to consider normality and homoscedasticity for your regression model.

No Autocorrelation

Also known as no serial correlation, you need to have data that is assumed to be uncorrelated. Here, it means a product-moment correlation coefficient. You can relax this assumption only if you are working on time series data! How to detect autocorrelation? The simplest method is to plot the residuals of your regression model and visually look for patterns. If there are no patterns, then there is no autocorrelation.

No Multicollinearity

As the name suggests, we need data that exhibits minimum or nil multicollinearity. When two or more independent variables have a high correlation, it is known as multicollinearity. Multicollinear variables can add to your regression model’s error and will lead to incorrect estimates and p-value. You should drop them in the first instance!

How do you fix multicollinearity? This first method is to drop one of the two variables in the first instance. The second method is to take an average of the two; since they are highly correlated, they will have similar weightage.

In Conclusion

Remember, a regression model is as good as the assumptions considered by you, the researcher. Do not perform regression analysis if your model fails to adhere to any of the five assumptions mentioned in this article.