Avoid R-squared to judge regression model performance | by Kevin Dunn | Towards Data Science

Summary
R2 can be calculated before even fitting a regression model, which doesn’t make sense then to use it for judging prediction ability. Also you get the same R2 value if you flip the input and output around. Again, this is nonsensical for a prediction metric.
The intention of your regression model is the important factor for choosing an appropriate metric, and a suitable metric is probably not R2.
This article explains which better alternatives exist: the standard error, confidence intervals and prediction intervals.
Historical data is collected and a relationship between the input x and the output y is calculated. This relationship is often a linear regression model, written as y=a+bx, where the intercept is a and the slope is b.
The purpose is most often to use that calculated relationship and make a prediction of some future output, called ŷ, given a new input: ŷ=a+bx.
Here are the two most common reasons for building a linear regression model:
to learn more about the relationship between input and output;by far the most common usage is to get predictions of the output, based on the input.
to learn more about the relationship between input and output;
by far the most common usage is to get predictions of the output, based on the input.
Let’s look at each of these in turn.
The coefficient, b of the linear regression y=a+bx, shows what the average effect is on the output, y, for a one unit increase in the input x. This is called “learning about our system”.
For example, if you built a regression model between x=temperature measured in Celsius of your system (input) and the y=pH (the output) you might get a regression model of: y=4.5+0.12x, from which you learn two things:
that every 1 degree increase in temperature, leads, on average, to an increase of pH by 0.12
that the expected or predicted pH when using a temperature of x=0 degrees Celsius, is an output pH of 4.5 units.
But consider two cases: what if I told you the R2 of this model was 0.2, or it was 0.94. How does this change your learnings? We will come to this in the next section, where we understand a bit more what R2 is measuring.
This scenario is the one most people are familiar with. Continuing the above, it is asking what the predicted pH would be for a given new input value of temperature, x. For example, at a new temperature that we have never used before of 13°C, we expect an output pH of 4.5+0.12×13=6.06 pH units.
And again, what value do we attach to such a prediction when the model has R2 around 0.2, or if the model has R2 of 0.94?
Be warned: searching for a phrase like “R2 interpretation”, or something similar, returns many sites with faulty reasoning.
In its simplest form, R2 is nothing more than a measure of how strongly two variables are correlated. It is the square of the correlation coefficient between x and y.
where the fancy “E{...}” is the “expected value of” operation, the fancy “V{...}” is the “variance of”, and Cov{x, y} is the “covariance of” operation. The horizontal lines above x and y tell you to use the average value of those data columns.
A second interpretation is that R2 is the ratio of the Regression Sum of Squares (RegSS) to the Total Sum of Squares (TSS). Any “sum of squares” value is just a rescaled variance, so the formula here shows R2 is just the ratio of two variances:
An R2 value of 1.0 means the numerator and denominator are exactly the same, or in other words, the predictions, ŷi, are identically equally to the original values, yi. Conversely to make R2 have a value of 0.0 the numerator must equal zero, which says the predictions are simply equal to a flat line, the average value of y, no matter what the input value of x. This is the worst model you can have: it says that in the case of R2=0, the best prediction you can make for y is just to return the average value of the training y values.
It is from this equation where the common interpretation of R2 comes from: that is the percentage variation explained. The denominator is proportional to the total variation, and the numerator is the explained variation, leading to a fraction (percentage) between 0 and 1.
Related to the prior interpretation is that R2 = 1.0−RSS/TSS, coming from that fact that for a simple regression model we have TSS = RegSS + RSS.
The RSS is the Residual Sum of Squares (RSS), or mathematically
This shows nicely that to get an R2=1 you must have no residuals (RSS=0); and that an R2=0 means that your TSS = RSS, in other words, your residuals have the same variance as the raw data data.
More details and illustrations to explain this are in my free book: https://learnche.org/pid/least-squares-modelling/least-squares-model-analysis
It is very fruitful to understand the above formulas and to try interpret them yourself, in plain language. It is not easy at first, but it pays off understanding how you can make each part of the equations bigger and smaller, and how you can obtain the extreme values of 0.0 and 1.0.
From the above, here are two very simple reasons why R2 is almost never the correct metric to judge how well you can predict a new output, y, from a new input x:
If you switch the historical data around, and make y the x and let x become y, then you get exactly the same R2 value. How does that make sense even? A metric of a model’s prediction ability must depend on what is the input and what is the output.What if I told you that I can tell you what the R2 value will be before calculating the model’s slope and intercept? Again, that does not make sense. How can a good metric of prediction performance be calculated before even fitting the prediction model?
If you switch the historical data around, and make y the x and let x become y, then you get exactly the same R2 value. How does that make sense even? A metric of a model’s prediction ability must depend on what is the input and what is the output.
What if I told you that I can tell you what the R2 value will be before calculating the model’s slope and intercept? Again, that does not make sense. How can a good metric of prediction performance be calculated before even fitting the prediction model?
The above would be the equivalent of calculating the prediction ability of a neural network before even fitting it; or flipping the input and outputs around for that neural network and getting the same performance metric.
So how do we make these two very strong statements about R2? Look back at the first formula:
The numerator is symmetrical. If you switch the roles of x and y you get the same numerator value. This also holds for the denominator. Please confirm this for yourself in Python, Excel, R, MATLAB, or whatever tool you use for linear regression. Here is some R code for fitting a linear model (lm):
x = c(10, 8, 13, 9, 11, 14, 6, 4, 12, 7, 5)y = c( 8, 7,  8, 9,  8, 10, 7, 4, 11, 5, 6)summary(lm(y ~ x))  # R^2 = 0.657 summary(lm(x ~ y))  # R^2 = 0.657
Again, take a second look at the above formula. Does it depend on:
the model’s residuals? no
any of the model’s coefficients, such as the slope or intercept? no
So it is possible to calculate R2 without even fitting a regression model. The formula depends only on the raw data and not on any model! This is not some mathematical trick because things cancel out or simplify in some special way. It is just a fact of what R2 is designed to measure: the degree of correlation between two sequences, x and y.
So quite simply, that last sentence indicates when R2 should be used: only as a measure of correlation between two sequences.
Let’s look back at the two use cases:
learning from the modelmaking predictions from the model
learning from the model
making predictions from the model
If your goal is to interpret the slope or intercept, then use a metric related to that, and not R2. In this case the confidence intervals for the slope and intercept are informative. The confidence interval is two numbers, a range. Within this range you can expect, with a specified degree of confidence, that it contains the true value of the parameter (e.g. the slope). Let’s return to our temperature and pH example, and only consider the slope. The same idea holds for the intercept.
Remember that y=4.5+0.12x, meaning that every 1 degree increase in temperature, leads, on average, to an increase of pH by 0.12 units. A 95% confidence interval for the true (but unknown) slope might have been [0.09; 0.15], meaning that we have 95% probability that this range contains the true slope. Note, it is not the probability that the true slope is within this range; a subtle, but important distinction. There is no probability associated with the true value; the probability is related to the range containing the true slope, which is a fixed, but unknown, value.
Why use confidence intervals? The main reason is that the values are in the units that you care about, and not an abstract ratio of two variances, such as R2. Secondly, the wider the range, the poorer the model. Imagine the confidence interval for the slope is either [0.09; 0.15] or [0.03; 0.21]; which would you rather have?
There is a direct relationship between the width of the intervals and the value of R2, but this connection is non-linear, and has a different shape for different models.
Here is some R code you might use. Similar code is possible with Python:
x = c(10, 8, 13, 9, 11, 14, 6, 4, 12, 7, 5)y = c( 8, 7,  8, 9,  8, 10, 7, 4, 11, 5, 6)linmod = lm(y ~ x) # predict y from xsummary(linmod)confint(linmod) # confidence interval
If you want to judge the model’s predictions use either the standard error (SE) or the prediction interval, not R2. Why SE?
The standard error is the standard deviation of the residuals. This sounds more complex than R2, but it isn’t. Firstly, the standard error has units of the output variable. In other words, it has units of the quantity you actually care about, the prediction’s units! If your residuals are normally distributed (a quick and easy check with a q-q plot) then you know approximately 70% of of the residuals lie within the range [−SE; +SE], and 95% of residuals within [−2SE; +2SE]. This is what I personally use the most: calculate SE and multiply it by 4 to get an idea of the “bandwidth” of my typical residuals. It is not exact, but it is good enough to judge “is my prediction model good enough?” If you are comfortable that predictions have a spread of between [−2SE; +2SE], then you can be satisfied with your model.
You will read on countless blogs and textbooks that a low R2 is not necessarily bad, and a high R2 isn’t necessarily good. That’s correct. That is because I’ve seen cases with an R2 exceeding 0.99, but the standard error was still too large to be useful for what was required by the end user.
Even better than using the standard error calculated when you built (trained) the model is to rather use the standard deviation of the residuals from predictions on an entirely new testing dataset. This is the same metric, but just calculated from testing data, or via cross-validation if you cannot keep testing data aside.
Another option, related to the standard error, is to use the prediction interval. It is like a confidence interval, but for an entirely new prediction. In the above example, we might have had:
a new temperature measurement, x=10 degrees Celsius, leading to a predicted pH of 5.7±0.4
in other words, the predicted pH lies in a bound from [5.3 to 6.1] with 95% confidence.
Such an interval is extremely informative; right away we get a range, in the units we care about, namely the units of the output variable, y. The prediction interval is the shaded region in the illustration, and is nonlinear. It is wider the further away you go from the model center point. While this interval is not always readily available from some software packages (e.g. Excel), you can use, as an approximate rule, that the shaded area covers ±2SE.
x = c(10, 8, 13, 9, 11, 14, 6, 4, 12, 7, 5)y = c( 8, 7,  8, 9,  8, 10, 7, 4, 11, 5, 6)linmod = lm(y ~ x) # predict y from x# Get the prediction interval for 3 new values of x:newdata = data.frame(x=c(1, 8, 15))predict(linmod, newdata, interval="predict")
As before, the higher the value of R2, the smaller the prediction interval width, proving intuitively that a higher R2 is better, but you cannot derive any cut-off limit, ahead of time, of what is “a good R2”, since it depends on the data measured, and of course on what is acceptable for your purpose.
Ask yourself why you are building the linear regression model:
“I want to know more about my slope or intercept”. Use the confidence interval for them, not R2. A higher R2 is related to a smaller confidence interval.
“I want to be able to judge predictions from my model”. Use the prediction interval, or the model’s standard error. A higher R2 is related to smaller prediction intervals and lower standard error.
All alternatives described are more interpretable than R2, and are in units you care about. If you’ve made it this far, and want to read more, please see my free book.
Some rights reserved