All the Statistical Tests You Must Do for a Good Linear Regression | by Gustavo Santos | Towards Data Science

The idea of this post is to show the many statistical tests that are around a Linear Regression. I know that it may sound repetitive (“Yet another post about Linear Regression”), but the information I am about to write about is not widely spread as we may think.
We will cover the following subjects:
What are the necessary assumptions for a Linear Regression model
Testing the residuals’ normality
Testing the residuals’ variability
Checking for outliers in the residuals
Checking the independence of the residuals
Looking at multicolinearity
Don’t worry, I will leave the entire code at the end, where you will be able to see what I have imported for each test.
As dataset, I will be using a “toy dataset” from sklearn about wines.
For modeling and testing, I will use statsmodels , as it has all of the tests needed in the library. You can perform the tests with statsmodels and later build it with sklearn. Your call.
# Load the datasetimport sklearn.datasetsdata_ = sklearn.datasets.load_wine()# Create dataframedf = pd.DataFrame(data_.data, columns=data_.feature_names)df['target'] = data_.targetdf.rename(columns={'od280/od315_of_diluted_wines': 'test_diluted_wines'}, inplace=True)
Ok, developing a little more on the “Yet Another Post about Linear Regression” point, I’d like to say some things.
First of all, I know that the dataset above is meant to work well for Linear Regressions and I know it is for study purposes only. The idea here is to show the way each test is performed and what it looks like when the model actually pass the test, so I can give you the key to understand and assess your data.
Secondly, real life is tough, people. To be honest with you, most of the datasets from real life won’t pass those tests and you will have to work on them, transform the data, do feature engineering, tweak the model or even choose a different algorithm to model your data. That’s the job you signed up for — Data Scientist.
Last, but not least, there are many assumptions we should follow when we’re performing a Linear Regression model. Like Charles Wheelan says in his great book “Naked Statistics”:
The LR is operationally easy to perform. Even a child can do that with the great software available these days, but to perform it correctly and well done, that’s another story.
The Ordinary Least Squares (OLS) method is the most common for LR and, in short, it tries to draw a line somewhere between the points that will give us the best fit, that will give us the lowest number when we sum the difference between the real values to the regression line.
That difference is the error component of the famous equation y = a + bx + e. The OLS method will square those errors and sum the amounts, trying to reach the minimum number as the result, which will result in the coefficients for the equation with the best fit.
Being the error part of the equation, it has a lot of importance. In an excellent article, the authors Dalson FIGUEIREDO FILHO, Felipe NUNES, Enivaldo CARVALHO DA ROCHA, Manoel LEORNARDO SANTOS, Mariana BATISTA e José Alexandre SILVA JUNIOR in Revista Política Hoje, Vol.20, n. 1, 2001, describe many of the assumptions where we should test the residuals of a LR and that we will describe and code in this post:
The residuals must follow a normal distribution.
The residuals are homogeneous, there’s homoscedasticity.
There’s no outliers in the errors.
There’s no autocorrelation in the errors.
There’s no multicolinearity between the independent variables.
When we’re creating a LR model, sometimes we hear that the variables must follow a normal distribution. That is not quite true. What must follow such distribution is the residuals.
The residuals of a LR model must follow the Normal Distribution so the estimators found by the OLS method are not biased and are more efficient (FIGUEIREDO FILHO et all, 2011).
Here’s how it is done, after the dataset is loaded.
# Regression Modelmodel = smf.ols('target ~ alcohol + ash + alcalinity_of_ash + total_phenols + flavanoids + nonflavanoid_phenols + color_intensity + hue + test_diluted_wines + proline', data = df).fit()# Extract the residualsmodel_resid = model.resid
Test normality.
# Kolmogorov-Smirnov test_, p = lilliefors(model_resid, dist='norm')print('Not normal | p-value:' if p < 0.05 else 'Normal | p-value:', p)print('-------------------------------')# Andersonstat, p3, _ = scs.anderson(model_resid, dist='norm')print('Anderson:')print('Not normal | stat:' if stat > p3[2] else 'Normal | stat:', stat, ':: p-value:', p3[2])[OUT]:Normal | p-value: 0.2 ------------------------------- Anderson: Normal | stat: 0.35916418199656164 :: p-value: 0.77
I can also plot the QQ plot to see where the data deviates from the normal distribution, when that’s the case.
# QQPlotscs.probplot(model_resid, dist='norm', plot=plt)plt.title('QQ Plot')plt.show()
The residuals must be homogeneous. And if we think it logically, that makes a lot of sense. I mean, if we’re analyzing data for a linear relationship and I want to show that Y increases as X and Z increases, then it becomes clear that the errors will be within a certain range. The errors should not be too far from the line, otherwise we’re not talking about a good linear relationship.
We can check it numerically and graphically to confirm.
# Ho = Homocedasticity = P > 0.05# Ha = There's no homocedasticity = p <=0.05# Homocedasticity teststat, p, f, fp = sms.het_breuschpagan(model_resid, model.model.exog)print(f'Test stat: {stat}')print(f'p-Value: {p}')print(f'F-Value: {f}')print(f'f_p_value: {fp}')[OUT]:Test stat: 14.254047656267216p-Value: 0.16171589669144282F-Value: 1.45373117596194f_p_value: 0.16108656286704198
Graphically.
plt.scatter(y= model_resid, x=model.predict(), color='red')plt.hlines(y=0, xmin=0, xmax=4, color='orange')plt.xlabel('Predicted values')plt.ylabel('Residuals')plt.show()
Along with the last test, another good test is the presence of outliers in the residuals. That test will help us to determine if there is any spots where our model is not being able to give a sensate result. It confirms the quality of the model. It is recommended that the numbers are between -3 and 3 or even -5 and 5.
# Outliers testoutliers = model.outlier_test()print(outliers.max())print(outliers.min())[OUT]: student_resid    3.254315unadj_p          0.999499bonf(p)          1.000000dtype: float64student_resid   -2.460243unadj_p          0.001377bonf(p)          0.245188dtype: float64
The independence of the residuals means that the observations are independent as well. So the measurement of something on a moment t1 does not influence the measurement on a moment t2.
Here, the closer the statistic of the test is to 2, the better. It means that the correlation (r) is near zero and, when plugged in this formula 2 x (1-r) it will give us 2 as result.
print( f'Test stat: {durbin_watson(model_resid)}')[OUT]: Test stat: 1.5212847896408088
Multicollinearity can mess up a linear regression model.
When it is present, there is loss of information for the model calculating the OLS. Thus, the more multicollinearity, the less information for the model.
The problem with multicollinearity is the increase of the estimated parameters’ magnitude of their variance. That is because the presence of high levels of correlations between the independent variables makes impossible to estimate with precision the effect of each variable over the dependent variable. (FIGUEIREDO FILHO et all, 2011).
# check correlationsvariables = df.drop('target', axis=1)variables.corr()
You can also use the Variation Inflation Factor, as this gives you a single number to look at. It calculates the variance of a given variable influenced by the others. Rules of thumb say that the number should be below 10. Some say below 7.
from statsmodels.stats.outliers_influence import variance_inflation_factorfrom statsmodels.tools.tools import add_constant# VIF Testvif = add_constant(variables)pd.Series([variance_inflation_factor(vif.values, i) for i in range(vif.shape[1])], index=vif.columns)const                   680.485185alcohol                   2.460372malic_acid                1.656647ash                       2.185448alcalinity_of_ash         2.238732magnesium                 1.417855total_phenols             4.334519flavanoids                7.029350nonflavanoid_phenols      1.796380proanthocyanins           1.975683color_intensity           3.026304hue                       2.551447test_diluted_wines        3.785473proline                   2.823849dtype: float64
After the model passed all of those tests, we’re now in a good shape to evaluate it’s summary. statsmodels brings a great one.
print( model.summary() )
Our model is good. We should look at the R-Squared to see how much of the relationship the linear equation was able to explain. It was 89%. If we want to know how well those variables chosen actually affects the target and explains variance, then we should look at R2-Adjusted, usually also used to compare models together with AIC and BIC (those last two metrics mentioned , the lower the better).
Both R2 and the adjusted R2 give you an idea of how many data points fall within the line of the regression equation. However, there is one main difference between R2 and the adjusted R2: R2 assumes that every single variable explains the variation in the dependent variable. The adjusted R2 tells you the percentage of variation explained by only the independent variables that actually affect the dependent variable.
Look at the P > |t| column which is the p-Value of each variable. It explains the statistical significance of that for the model. The ideal is to have everything under the significance level of 0.05. There are a couple there over that threshold.
We’re loosing 10% of the variance due to something. In order to improve this model, there should be a complete data exploration, feature engineering, perform transformations if needed. Well, many possibilities.
LR is a powerful tool. Specially when you know how to create it and test it’s assumptions.
Remember to always perform those tests. They are not stiff. They should be flexible and should give you a good direction whether your model is good or not.
Here is the complete code:
What is an Acceptable Value for VIF?
What to do and not to do in LR.
The difference between R2 and R2-adjusted.
If you like this content, explore and follow my blog for more.