Testing the Assumptions of Linear Regression | by Shuangyuan (Sharon) Wei | Towards Data Science

It seems that nowadays when everyone is so much into all kinds of fancy machine learning algorithms, few people still care to ask: what are the key assumptions required for the Ordinary Least Squares (OLS) regression? How can I test if my model satisfies these assumptions? However, as simple linear regression is arguably the most popular modeling approach across every field in social science, I think it is worthwhile to do a quick recap of the fundamental assumptions for OLS and run some tests through building a linear regression model using the classic Boston Housing data.
The Gauss-Markov assumptions assure that the OLS regression coefficients are the Best Linear Unbiased Estimates or BLUE.
Linearity in parametersRandom sampling: the observed data represent a random sample from the populationNo perfect collinearity among covariatesZero conditional mean of error (i.e. E(μ|X) = 0) (also often referred as Exogeneity)Homoskedasticity (constant variance) of the errors
Linearity in parameters
Random sampling: the observed data represent a random sample from the population
No perfect collinearity among covariates
Zero conditional mean of error (i.e. E(μ|X) = 0) (also often referred as Exogeneity)
Homoskedasticity (constant variance) of the errors
It is important to note that OLS is unbiased (i.e. E(β*) = β) when assumptions 1–4 are satisfied. Heteroscedasticity has no effect on bias or consistency of OLS estimators, but it means OLS estimators are no longer BLUE and the OLS estimates of standard errors are incorrect.
The Boston house prices dataset consists of 506 observations of 14 attributes:
crim: per capita crime rate by townzn: proportion of residential land zoned for lots over 25,000 sq.ftindus: proportion of non-retail business acres per town chas: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)nox: nitric oxides concentration (parts per 10 million)rm: average number of rooms per dwellingage: proportion of owner-occupied units built prior to 1940dis: weighted distances to five Boston employment centresrad: index of accessibility to radial highwaystax: full-value property-tax rate per USD 10,000ptratio: pupil-teacher ratio by townblack: 1000(B — 0.63)2 where B is the proportion of blacks by townlstat: percentage of the lower status of the populationmedv: median value of owner-occupied homes in USD 1000's
It is natural to ask the question what factors may affect and predict the housing prices by looking at the data. I start with some simple scatter plots to visually check the relationship between variables (see below). The set of charts below plot medv against crim, rm, age, lstat, and dis. It appears that the relationships do not seem linear.
I also plot histograms to check the univariate distribution. And I decide to use log(medv) instead of medv as the dependent variable because the log transformation mitigates the skew. Similarly, I will use log(crim).
Even though my intuition and the exploratory analysis plots tell me that the percentage of the lower status of the population, number of rooms, crime rate, and distances to five Boston employment centers are probably the most important predictors. In this article, I’d like to start with running a lass model with all the variables as the baseline also as a way to select features. Here is the R code:
The lasso reg_base does not return any zero coefficients but I found the log(cim) and a few other variables are not significant. I excluded those insignificant variables in reg_1 model and here are the coefficients:
Quick interpretation: 3.3% decrease in the median housing price for 1 unit change (1 percent) increase in the percentage of the lower status of the population (lstat), and 10% increase in the housing price for 1 unit change in the number of rooms. It makes sense to me.
Residuals vs Fitted: the equally spread residuals around a horizontal line without distinct patterns are a good indication of having the linear relationships. If there are clear trends in the residual plot, or the plot looks like a funnel, these are clear indicators that the given linear model is inappropriate.
Normal Q-Q shows if residuals are normally distributed. It’s good if residuals are lined well on the straight dashed line.
Scale-Location can be used to check the assumption of equal variance (homoscedasticity). It’s good if we see a horizontal line with equally (randomly) spread points.
The residuals vs fitted plot show that the linearity assumption is more or less satisfied. The log transformation takes care of the non-linearity. However, the scale-location plot indicates heteroscedasticity.
print(bptest(reg_1, data = Boston, studentize = TRUE))studentized Breusch-Pagan testdata: reg_1BP = 64.991, df = 7, p-value = 1.51e-11
Rejecting the null hypothesis of homoscedasticity of Breusch-Pagan test indicates heteroscedasticity (HSK). We can use weighted least squares method (WLS) to correct for HSK. Since HSK has no effect on bias or consistency of OLS estimators and the WLS estimates are not very different from OLS. I will skip the correction for HSK inthis article.
VIF stands for variance inflation factors. The general rule of thumb is that VIFs exceeding 4 warrant further investigation, while VIFs exceeding 10 are signs of serious multicollinearity requiring correction.
vif(reg_1)rm dis rad tax ptratio black lstat1.780895 1.559949 6.231127 6.610060 1.408964 1.314205 2.414288
The VIF test shows collinearity for rad and tax. To mitigate the issue. I ran a new regression removing rad. The coefficients do not change much. I will not show them again to save some space.
There is no simple way to check this assumption. First, checking whether the mean of residuals is zero is not the way to do it. As long as we include an intercept in the relationship, we can always assume that E (μ) = 0, since a nonzero mean for μ could be absorbed by the intercept term. You can read the math proof here. One way to check it is to plot the residuals against row numbers that are not assigned associated with the dependent variable. The residuals should be randomly and symmetrically distributed around zero across row numbers no matter how we sort the rows, which indicates no correlation between consecutive errors. I also plotted the residuals again the independent variables to inspect if there are obvious correlations.
Lastly, I think it’s a good practice to always think and check for the omitted variable bias. I want to test the idea of including the criminal rate in the regression because I imagine that the criminal rate should affect the housing price. I compare including crim in reg_2 mode, with including log(crim) in the reg_3 model using Davidson-MacKinnon test. The interpretation of the Davidson-MacKinnon test is to reject the reg_3 specification.
Reading the regression statistics, including crim improves the R-squared as expected, and the crim coefficient is significant. However, given the skewed distribution of crim and seemingly non-linear relationship between crim and log(medv), I am not fully convinced using crim as one of the predictors is necessary.
To further check it, I decide to split the data into train and test data and run a simple out-of-sample test comparing reg_2 and reg_1. It turns out the original model reg_1 without crim returns smaller MSE on the test data. So I think it’s safe to conclude that reg_1 is the correct model.
Here is the full code for running reg_base, reg_1, comparing reg_2 vs. reg_3, and subsequently comparing reg_2 vs. reg_1.