Data science you need to know — diagnosing linear regressions | by Michael Barber | Towards Data Science

This is an ad-hoc part of my “data science you need to know series”, exploring often overlooked aspects of data science. Part 1 and part 2 are recommended reading. Part 3 is coming soon!
In this post we will cover one of the first machine learning algorithms most students learn about, linear regression.
You will see a lot of linear regressions across data analysis and data science, unfortunately, a lot of them are very weak. This generally occurs for one of two reasons:
the regression model severely violates one or more key statistical assumptions
the regression doesn’t present detailed diagnostics, making it impossible to evaluate its robustness (and therefore its conclusions)
In this post, we will learn how to avoid these issues, as well as how to effectively evaluate and critique published regression models. We will approach this from the point of view of an A/B test evaluation, but the same principles apply to productionised regression models (i.e. models in the backend).
Before we build our regression model, it is important to have thoroughly explored the data (e.g. through pair plots and density plotting) and understood the statistical power of our A/B test (see Part 1 and part 2).
Once we have explored the data, we can begin building a linear model. The simplicity of the R/Python code for linear regression can be deceiving. Whilst you can produce results with just a simple lm(x,y), you need to understand whether these results are robust.
Linear regressions have a set of assumptions and caveats associated with them which need to be checked before we interpret the regression results. If we violate these assumptions then the code will still run, and you will still get coefficients, p-values and predictions (etc), but they will likely be meaningless!
A little data science can be a dangerous thing!
Before we proceed to model building, it is good to have a definition for a good model in mind. We will here define a good model according to the following criteria:
model is capable of explaining/predicting the full range of Y values
model can accurately provide confidence intervals to inform our understanding of uncertainty (and significance)
model is not distorted by outliers
We can satisfy these requirements by fulfilling the following criteria for a linear regression:
the relationship between X and Y is linear, or can be made linear.
Errors are independent of X (a.k.a. homoskedascity)
variables are mostly un-correlated with each other
Residuals are normally distributed
Therefore, to build a good model, we will need to understand and evaluate the underlying assumptions outlined above. We will do this through a series of plots that examine a number of things, but perhaps most importantly, the residuals of a model:
A residual is the difference between a model prediction and an actually observed value.
We will explore these plots first with a well-behaved regression, to understand how plots look in a best-case scenario, and then how these plots react to heteroskedascity and non-linearity.
First, a well behaved regression model. We will (hopefully) remember the definition of a straight line:
Y=β.X+c
Where β is a matrix of coefficients, X a matrix of constants, and c the intercept.
We will use this equation to understand regression, with the addition of an e term to represent the independent error around our observations. When running a regression, our aim is to determine:
Y=β.X+c+e
so that we can accurately determine the size of coefficient β (and so our treatment effect) and e (so we can determine its significance).
Let’s use a worked example to demonstrate how regression works. First, let us define our relationship (with the aim of recovering these values from the regression results):
Intercept = 10
β = 2
Error term = 3
We’ll simulate 200 independent values based on this equation now:
set.seed(111)ex1 <- data.frame("Xvals"=runif(1:200, min=1,max=10) )ex1$Yvals = 10 + 2*ex1$Xvals + rnorm(200,0,3) # y = \beta . X +c +ehead(ex1[c("Xvals","Yvals")])
Yielding:
##      Xvals    Yvals## 1 6.336832 24.47252## 2 7.538330 21.59567## 3 4.333798 19.98488## 4 5.634314 21.88319## 5 4.398969 16.70039## 6 4.765036 16.75019
Let’s now regress Xvals against Yvals, to ask: “what correlation is there between Xvals and Yvals?”.
exreg1 <- lm(Yvals~Xvals, data=ex1)
As our regression is relatively simple, with only one independent variable, we can easily visualize the results with a simple plot:
We can see the black line does a good job of describing the relationship between Xvals and Yvals. Also note that the blue points are relatively equi-distant from the regression line; this is something we care about a lot about in regression analysis. The distance from each point (blue) to the model (black line) is known as the residual, we will collect these together (pl. residuals) to understand our model performance (below).
Before analyzing model residuals, let’s quickly examine the regression summary:
Let’s refresh what these terms mean before we proceed:
P-value: The p-value derives from testing the null hypothesis that the coefficeint = 0. If the p-value is below a certain threshold (usually 0.05) then we reject the null hypothesis and call the result “significant”. Note that significance in statistics means something like “reliable”. So a coefficient can be very small but still statistically significant (reliable).
Coefficient: Each coefficient estimates the change in the mean response per unit increase in X when all other predictors are held constant.
Constant: The intercept term in our equation.
Observations: The number of rows in our dataframe.
R^2 R-squared is a statistical measure of how close the data are to the fitted regression line. It is also known as the coefficient of determination, or the coefficient of multiple determination for multiple regression. R-squared = Explained variation / Total variation. R-squared cannot determine whether the coefficient estimates and predictions are biased, which is why you must assess the residual plots.R-squared does not indicate whether a regression model is adequate. You can have a low R-squared value for a good model, or a high R-squared value for a model that does not fit the data!
Adjusted R^2 The adjusted R-squared compares the explanatory power of regression models that contain different numbers of predictors. The adjusted R-squared is a modified version of R-squared that has been adjusted for the number of predictors in the model. The adjusted R-squared increases only if the new term improves the model more than would be expected by chance. It decreases when a predictor improves the model by less than expected by chance. The adjusted R-squared can be negative, but it’s usually not. It is always lower than the R-squared.
F statistic Does the addition of variables improve the regression model overall? The F-statistic reports on the null hypothesis that Our current model is indistinguishable from an intercept-only model. Rejection of this hypothesis (p<0.05) suggests that the inclusion of variables is helpful.
We can see that we nicely recover the intercept and coefficient terms (plus some confidence intervals proportional to our error term):
Intercept = 9.79
β = 2.07
Next, let us examine the diagnostic plots to evaluate how the model fulfills the assumptions we stated above:
Let’s take these plots one by one and diagnose them:
This plot shows the model residuals vs fitted values. Ideally, we want the residuals to be evenly distributed around the dotted black line at 0. The red line indicates the mean residual value, and ideally would lie on the 0.0 black line
A non-flat red line indicates that our data may be non-linear (think through why this is, using the “Xvals vs Yvals” graph above).
Points with un-even or non-symmetrical scatter around the 0.0 line represent either heteroskedascity or non-linearity (again — think through why this is so).
This is a QQplot of residuals. The presence of trend deviations from the diagonal line (which represents perfect normality) suggest non-normality in our residuals (see AMP Lesson 1 for more information). We don’t see any trends in our residuals here (the scatter at high and low X is minor and unalarming as it does not constitute a trend).
The scale-location plot shows the square root of the standardized residuals as a function of the fitted values. This is similar to our residuals vs fitted plot above, and should be interpreted similarly.
Finally, the leverage plot measures the impact of each point on our final output vs the residual of that point. Essentially, this graph will aid us is spotting data points that are having an outsize influence on our model. If some points have high leverage and a high (absolute) residual value then they will be skewing our model more than other data points. Cook’s distance (the dotted lines) is a measurement combining residual size and leverage.
Note that we don’t see any Cook’s distance lines on the above as our Cook’s values are too small (this is a good thing!)
Overall, these plots look reasonable, and I am happy that we have satisfied the linear model requirements.
In summary, this regression looks very valid:
The residuals vs fitted plot shows a flat, straight line and no obvious pattern in the residuals
The QQplot of residuals likewise looks reasonable. Whilst we can see a few deviations from the dashed-line, these are not what I would call “trend deviations”. We can use a Shapiro test on the residuals to confirm this (p= 0.2613), though in general I recommend simply examining the QQplot rather than running repeated hypothesis tests.
Our scale-location plot likewise is flat and without pattern
Our residuals vs leverage plot does not reveal any clear outliers.
Let’s now turn to a poorly behaved regression, specifically, we will examine the diagnostic plots produced from a non-linear relationship and compare them to the plots above:
Hmm, clearly here we have tried to fit a linear model to a non-linear relationship! This model is therefore useless (remember our definition of a good model from the start of this section). Note, that whilst we can clearly see our model is terrible (i.e. it would do a bad job of predicting Ys given a set of Xs) we are still able to achieve a great R-squared of 0.856. Do not blindly follow R-squareds into oblivion!
Let’s look at the diagnostic plots:
Compare these plots to the well-behaved regression plots to understand how they vary under non-ideal conditions. Likewise, spend some time trying to understand how these plots related to the simple X-Y plot above.
For example:
we can note that our residuals cross zero twice in our residuals vs fitted plot, and that our regression line crossed our non-linear data point trend twice also.
Likewise our scale-location plot
Note also that our residuals vs leverage plot, we clearly see a trend towards higher leverage and higher residuals (as our data points get further and further from our trend line)
If we examine our QQplot more closely:
We’ll note that our residuals plot summarises what we saw above with the Xvar, Yvar plot. That at extreme value of X, we tend to deviate from the fitted line.
Spend some time examining the relationships within these plots to improve your ability to quickly diagnose which linear-model assumption you are violating.
Finally, let’s look at what extreme heteroskedasticity looks like (remember: heteroskedascity is when our error term is dependent on X):
Note the cone-shaped data indicating heteroskedascity.
Again, compare this to our well-behaved regression to understand how heteroskedastic data appears in regression plots (hint: examine the residuals vs fitted plots, and the scale-location plots closely).
Remember — in a real regression analysis with multiple factors you will not have a simple Xvar-Yvar regression plot to easily diagnose your model, so you will need to be comfortable using these diagnostics alone!
Finally, I want to introduce a plot that isn’t automatically produced by R, but that is incredibly useful in regression diagnostics. In this graph, we will plot our fitted values vs our real (observed) values to see how the model performs, let’s plot these graphs for the above regression examples:
Spend some time examining and understanding these plots, and how these models compare against the criteria we outlined above:
model is capable of explaining both low and high Y values
model can accurately provide confidence intervals to inform our understanding of variability (and significance)
model is not distorted by outliers
given a set of Xs, our model can accurately predict Ys
One of the approaches to improving diagnostic plots for a regression is transforming data. Generally, either X or Y variables will be transformed. We will focus on Y-variable transformations here, as they are much more commonly practiced.
Common y transformations include:
ln or log10 transformations
Simple power transformations (square root or squared)
It is commonly assumed that when y-variables are transformed we can merely back-transform (that is, apply the inverse function to) the model coefficients to make them interpretable.
This is incorrect! Transforming a y-variable with alter the coefficients in more complex ways!
Generally, back-transforming coefficients for y transformed data will require solving complex differential equations (there is a notable exception here, which is covered later in the lesson), and is not as simple as a back transformation. Let’s demonstrate this by returning to our well-behaved regression:
Remember that we calculated Y so that Y=10+2βX+eY=10+2βX+e (where e is our error term). We can see this regression nicely returns this to us.
Let’s instead regress against the square root of Y, we can see our diagnostic plots still look reasonable:
And our coefficients:
This time we get an intercept of 3.079 and β of 0.253. If we square these (square being the inverse of square root) then we get:
Intercept = 9.48
Xvals = 0.064
Clearly, these do not relate to either the original formula or the original regression!
If transforming variables is necessary, then it is wise to pick a transformation that retains some physical meaning. For example taking the square root of an area (which would yield length). An important danger with transformations is introducing new censures in the data, for example, taking “change in customer expenditure (USD)” squared would make all data positive, and so make model interpretation impossible.
One notable exception to these transformations is the natural-log (in r: log()). When Y data is log transformed then coefficients can be interpreted as the % change expected in Y with a one unit increase in X.
Do not blindly transform and back-transform data!
Transformed data often yields un-interpretable coefficients (without solving the appropriate equations)
Natural-log transformations are an important exception to this rule. ln transformations yield coefficients where a one unit increase in X yields a β∗100β∗100 increase in Y.
Regressions have assumptions that must be met to produce meaningful results and good predictions.
Diagnostic plots will allow you to assess the extent to which your model is meeting these assumptions.
Transformations of the data need to be carefully thought through.
Always present diagnostic plots for any model you create! You can keep these in an appendix at the end of the document/presentation.
Some rights reserved