Math behind the assumptions of linear regression
By: Pawan Reddy Ulindala
Tags: Data Science, Mathematics, Linear Regression

Photo by Jeswin Thomas on Unsplash

There are certain assumptions that one has to keep in mind while applying linear regression on their data. When I started reading about linear regression, I couldn’t understand the need behind these assumptions. All we are doing in linear regression is minimizing the sum of squares of the difference between the predicted target variable(ŷ) and the original target variable(y) to find the best fit line or plane. While solving this, why do the following assumptions even come into picture? Why multi-collinearity between features is a problem. Why do we have to check the distribution of residuals? Why do we have to check for autocorrelation between observations? All these questions kept on creeping in while understanding linear regression. So, I decided to spend some more time on this and yes, I got answers to the above questions.

In this blog, I will try to explain the rationale behind these assumptions of linear regression by examining the mathematical derivations and concepts used to solve this optimization problem.

Trivial assumptions:

Linearity between features and target variable:

Consider the expression

Here, if both m1 and m2 are real numbers then we can say that x1 and x2 are linearly related to target variable(y). If m1 and m2 are not real numbers then we can say that x1 and x2 aren’t linearly related to target variable y.

For example, if m1 = √x1 and m2 =√x2 then y = x1**3/2 + x2**3/2 and here both x1 and x2 aren’t linearly dependent on y.

2. Homoscedasticity: Homoscedasticity means that variance of samples taken from the data should remain constant i.e., we need our data points(observations) to be closer to the plane that we have predicted if drawn spatially.

In the below figure, you can see that variance remains constant in the first case (Homoscedastic behaviour) and not in the second case(Heteroscedastic behaviour).

Homoscedastic behaviour

Heteroscedastic behaviour

3. No outliers: Since we are dealing with Euclidean distances, any outlier will significantly affect our predicted regression plane. It is always better to remove outliers and then perform any machine learning algorithm (not only linear regression).

Non-trivial assumptions:

To understand the reasons behind other assumptions it is necessary to go through some derivations which lead to them.

Choosing the cost function:

Choosing ordinary least squares (OLS) as our cost function comes at a cost. There are certain assumptions one has to deal with in order to finalize OLS as our cost function. The below derivation will make things more clear.

Any linear equation can be written in the form,

The term ε(i) is introduced to capture the unmodelled effects and errors caused by a given theta(θ). In other words, it can be called as an error term or residual.

We assume that ε follows Gaussian distribution (with mean zero and some variance) and ε(i)’s are independent and identically related((i.i.d).

P.D.F of ε(i) can be written as

Probability density function of ε(i)

It can also be written as,

Our goal is to get the highest probability of y(i)’s given our data i.e., we need to maximize the likelihood of pdf at each point.

We know that logarithm is a monotonic function. So, L(θ)is maximum when log(L(θ)) is maximum.

Expanding log(L(θ)) using logarithmic properties

l(θ) is maximum when the below equation is minimum.

This expression is nothing but our OLS(Ordinary Least Squares) cost function.

Note: This is one approach that justifies choosing OLS as a cost function. There can be other approaches that might infer the same end result.

Assumptions we made to choose OLS as our cost function:

In the above derivation, we assumed that residuals(ε) have a normal distribution and hence normality of residuals.

Also, as we assumed that ε(i)’s are independent and identically related((i.i.d), Residuals should also be independent and identically distributed. It basically means that there shouldn’t be any relation between residuals w.r.t time.=> No autocorrelation.

Minimizing cost function: Now that we know our cost function, our next goal is to minimize it. There are multiple ways to minimize it. We can use iterative algorithms like gradient descent, Newton’s method etc. or we can use linear algebra techniques to find optimal theta. The later one takes significantly less computational time and converges much faster when compared to others.

Minimizing cost function using linear algebra:

In matrix form, the cost function can be written as

After applying appropriate techniques, the expression finally reduces to

Inference:

Clearly, from the above expression theta(θ) exists if xtx is invertible. For X T X to be invertible, X should have linearly independent columns. In order to have independent columns, X shouldn’t have repeated columns and thus no multi-collinearity.

Sklearn’s linear regression uses LAPACK (written in FORTRAN and C language) to perform the inverse operation of the matrix (XTX). It finds the inverse using LQ or QR or Singular value decompositions depending on the drivers used in LAPACK. Matrix decomposition techniques are used to find the inverse of XTX even when XTX is not a full rank matrix i.e., X doesn’t have independent columns.

Let’s see an example explaining the same

Now, lets load a toy dataset from sklearn’s datasets module

Defining x,y and then adding a dummy column(repeated column)to check its performance

Running sklearn’s linear regression

As you can see, it’s working well even after adding a repeated column.

Now, let's implement a custom trained linear regression model

Here, it fails to perform the inverse operation as xTx is not a singular matrix.

Conclusion

In this blog, we have seen the reasons behind assumptions of linear regression by going through some derivations. We have also seen the drawbacks of implementing custom code for linear regression whereas if taken from sklearn’s library, it works well as it uses advanced matrix techniques.

Hoping that this blog has helped you in understanding linear regression better.

References:

Lecture-2 notes of cs-229 course,http://cs229.stanford.edu/syllabus-spring2019.html

Please check my other blogs here: