Is Normal Distribution Necessary in Regression? How to track and fix it? | by Songhao Wu | Towards Data Science

I believe for every person who studied statistics before, normal distribution(Gaussian distribution) is one of the most important concepts that they learned. Every time when I run a model or do data analysis, I tend to check the distribution of dependent variables and independent variables and see whether they are normally distributed. If some variables are skewed and not normally distributed, I would be a bit panic.
What should I do? Should I transform the variable? Should I remove it from the model or should I just leave it?
I always wonder whether normality is a necessary assumption and how should we tackle it. This article is about all these.
So is the normality assumption necessary to be held for independent and dependent variables? The answer is no!
The variable that is supposed to be normally distributed is just the prediction error. What is a prediction error? It is the deviation of the model prediction results from the real results.
Y = Coefficient * X + Intercept + Prediction Error
Prediction error should follow a normal distribution with a mean of 0. The calculation of confidence interval and variable significance is based on this assumption. What does it mean? For example, you are trying to analyze which variables are useful to predict housing price and you have selected the factors based on a 5% significance level. If the distribution of error significantly deviates from the mean 0 normal distribution, the factors you choose to be significant may not actually be significant enough to contribute to housing price changes. However, it would not affect your prediction if you just want to get the prediction based on the lowest mean squared error.
So what should we do? If you just want to get the prediction, then just leave it. If you want to select the significant predicting factors, after you have constructed your model and prediction, you should plot the chart to see the distribution of prediction error.
I created 1 random normal distribution sample and 1 non-normally distributed for better illustration purposes and each with 1000 data points.
#create normal and nonnormal data sampleimport numpy as npfrom scipy import statssample_normal=np.random.normal(0,5,1000)sample_nonnormal=x = stats.loggamma.rvs(5, size=1000) + 20
There are many ways to test the normality of data, below are just some examples:
Simply plot the distribution curve and see whether the plot follows the bell curve shape. The non-normal sample is clearly left-tailed.
Simply plot the distribution curve and see whether the plot follows the bell curve shape. The non-normal sample is clearly left-tailed.
import seaborn as snsimport matplotlib.pyplot as pltsns.distplot(sample_normal)plt.show()sns.distplot(sample_nonnormal)plt.show()
2. Use the Shapiro-Wilk test, built-in python library available and you can decide based on p-value you decide, usually we reject H0 at 5% significance level meaning if the p-value is greater than 0.05 then we accept it as a normal distribution. Take note that if the sample size is greater than 5000, you should use test statistics instead of the p-value as the indicator to decide.
print (stats.shapiro(sample_normal))print (stats.shapiro(sample_nonnormal))#Test Statistics, p-value#(0.9970424175262451, 0.06130971387028694) - H0 accepted#(0.9806660413742065, 2.945287624900317e-10) - H0 rejected
3. QQ-plot, a very popular plot to see whether the distribution of data follows a normal distribution.
fig = sm.qqplot(sample_normal,line='s')plt.show()fig = sm.qqplot(sample_nonnormal,line='s')plt.show()
Usually, there are 2 reasons why this issue(error does not follow a normal distribution) would occur:
Dependent or independent variables are too non-normal(can see from skewness or kurtosis of the variable)Existence of a few outliers/extreme values which disrupt the model prediction
Dependent or independent variables are too non-normal(can see from skewness or kurtosis of the variable)
Existence of a few outliers/extreme values which disrupt the model prediction
What we should do is first check the existence of outliers in both dependent and independent variables. If removal of outlier does not solve the problem, then we should transform some non-normal variables to be normally distributed using box-cox transformation.
Below is the mathematical formula for Box-Cox transformation. The Lambda value will be decided based on the data points to provide the best normal distribution shape after the transformation. We can directly use the Python package to help us transform the data.
#transform the data using box-coxsample_transformed, lambd = stats.boxcox(sample_nonnormal)#plot the distribution curve and QQ-plot for transformed datasns.distplot(sample_transformed)plt.show()fig = sm.qqplot(sample_transformed,line='s')plt.show()
We can see that after the box-cox transformation, the non-normal distributed sample is normally distributed.
In conclusion, if you try to find a significant predicting factor or define the confidence interval, please remember to check the distribution of the error term after the model has being built. If dependent variables or independent variables are very non-normal then can use box-cox transformation to transform it to make the error term more normally distributed.
Do you think you are finally done with all the checks with statistical assumptions before constructing the model? No, you are not! If you want to know more about other statistical assumption in the regression model, refer to my another article below on multi-collinearity.