Which Evaluation Metric Should You Use in Machine Learning Regression Problems? | by Jeff Hale | Towards Data Science

If you’re like me, you might have used R-Squared (R2), Root Mean Squared Error (RMSE), and Mean Absolute Error (MAE )evaluation metrics in your regression problems without giving them a lot of thought. 🤔
Although all of them are common metrics, it’s not obvious which one to use when. After writing this article I have a new favorite and a new plan for reporting them going forward. 😀
I’ll share those conclusions with you in a bit. First, we’ll dig into each metric. You’ll learn the pros and cons of each for model selection and reporting. Let’s get to it! 🚀
R2 represents the proportion of variance explained by your model.
R2 is a relative metric, so you can use it to compare with other models trained on the same data. And you can use it to get a rough a feel for how well a model performs, in general.
Disclaimer: This article isn’t a review of machine learning methods, but make sure you use different data for training, validation, and testing. You always want to hold out some data that your model has not seen to evaluate its performance. Also, it’s a good idea to look at plot of your model’s predictions vs. the actual values to see how well your model fit the data.
Let’s see how R2 is computed. Onward! ➡️
Here is one way to formulate R2.
1 - (SSE/SST)
SSE is the sum of squared errors; the sum of the squared differences between the actual values and predicted values.
SST is the total sum of squares (shown sometimes as TSS); the sum of the squared differences between the actual values and the mean of the actual values.
With more mathy notation:
1 - (∑(y - ŷ)2 / ∑(y - y̅)2)
Here’s what the code looks like —adapted from scikit-learn, the primary Python machine learning library.
numerator = ((y_true - y_pred) ** 2).sum()denominator = ((y_true - np.average(y_true)) ** 2).sum()r2_score = 1 - (numerator / denominator)
subtract the predicted values from the actual y valuessquare the resultssum them
subtract the predicted values from the actual y values
square the results
sum them
That’s the numerator.
subtract the mean of the actual y values from each actual y valuesquare the resultssum them
subtract the mean of the actual y values from each actual y value
square the results
sum them
That’s the denominator.
1 - the numerator/denominator is the R2. 🎉
R2 is the default metric for scikit-learn regression problems. If you want to use it explicitly you can import it and then use it like this:
from sklearn.metrics import r2_scorer2_score(y_true, y_pred)
A model that explains no variance would have an R2 of 0. A model with an R2 of 1 would explain all of the variance. Higher scores are better.
However, if your R2 is 1 on your test set you are probably leaking information or the problem is fairly simple for your model to learn. 👍
In some fields, such as the social sciences, there are lots of factors that influence human behavior. Say you have a model with just a few independent variables that results in an R2 close to .5. Your model is able to account for half of the variance in your data, and that’s quite good. 😀
It is possible to have an R2 that is negative. Negative scores occur when the predictions the model makes fit that data worse than the mean of the output values. Predicting the mean each time is a null model. See more here.
Say you have the following small toy test dataset:
All code is available on GitHub in this Jupyter notebook.
Here’s a plot of the actual and predicted y values.
The R2 of the model is 0.71. The model is accounting for 71% of the variance in the data. That’s not too shabby, although we’d like more test data. 😀
As another example, let’s say the true values for y are [55, 2, 3]. The mean is 20. Predicting 20 for each y value results in an R2 of 0.
A model that predicts [1 , 2, 2] for the true values above results in an R2 of -0.59. Bottom line, you can do far worse than the null model! In fact, you can predict infinitely worse, resulting in an infinitely low R2. 😲
As a brief aside, let’s look at the Adjusted R2 and machine learning vs. statistics.
The Adjusted R2 accounts for the addition of more predictor variables (features).
Adjusted R2 will only increase with a new predictor variable when that variable improves the model performance more than would be expect by chance. Adjusted R2 helps you focus on using the most parsimonious model possible. 😉
The Adjusted R2 is more common in statistical inference than in machine learning. Scikit-learn, the primary Python library for machine learning, doesn’t even have an Adjusted R2 metric. Statsmodels, the primary statistical library for Python does. If you want to learn more about when to use which Python library for data science, I wrote a guide here.
You can compute the Adjusted R2 if you know the number of feature columns (p) and the number of observations (n). Here’s the code:
adjusted_r2 = 1 — ((1 — r2) * (n — 1)) / ( n — p — 1)
n-1 is the degrees of freedom. Whenever you hear that term, you know you are in statistics land. In machine learning we generally care most about predictive ability, so R2 is favored over Adjusted R2.
Another note on statistics vs. machine learning: our focus is on machine learning, so prediction rather than causality. R2 — and the other metrics that we’ll see, don’t say anything about causality by themselves.
R2 tells you how much variance your model accounts for. It’s handy because the R2 for any regression problem will immediately provide some (limited) understanding of how well the model is performing. 😀
R2 is a relative metric. Let’s see a few absolute metrics now.
RMSE is a very common evaluation metric. It can range between 0 and infinity. Lower values are better. To keep this straight, remember that it has error in the name and you want errors to be low. ☝️
The RMSE can be formulated like this:
square root of mean SSE
We saw SSE in the R2 score metric. It’s the sum of squared errors; the sum of the squared differences between the actual values and predicted values.
More mathy formula:
square root of (1/n * (∑(y -ŷ)2))
In Python code:
np.sqrt(np.mean((y_true - y_pred) ** 2))
subtract the predicted values from the actual y valuessquare the resultssum themtake the averagetake the square root
subtract the predicted values from the actual y values
square the results
sum them
take the average
take the square root
Here’s how to get the RMSE with a function in scikit-learn:
from sklearn.model_selection import mean_squared_errormean_squared_error(y_true, y_pred, squared=False)
You can use the squared=False argument as of scikit-learn version 0.22.0. Prior to that you had to take the square root yourself like this: np.sqrt(mean_squared_error(y_actual, y_predicted). ☝️
Use RMSE if you want to:
penalize large errors
have the result be in the same units as the outcome variable
use a loss function for validation that can be quickly computed
You could use the Mean Squared Error (MSE) with no Root, but then the units are not as easily comprehensible. Just take the square root of the MSE and you’ve got the RMSE. 👍
In this excellent article JJ points out some issues with RMSE. Namely, that “RMSE does not necessarily increase with the variance of the errors. RMSE increases with the variance of the frequency distribution of error magnitudes.”
Also, the RMSE is not so easily interpreted. The units might look familiar, but you are squaring differences. You can’t just say that an RMSE of 10 means you are off by 10 units on average, although that’s kind of how most folks think of the result. At least, it’s how I used to. 😉
Turning to our example dataset again:
The RMSE is 0.48. The mean of the actual y values is 2.2. Together, that information tells us that the model is probably somewhere between great and terrible. It’s hard to do too much with this RMSE statistic without more context. 😐
RMSE is an imperfect statistic for evaluation, but it’s very common. If you care a lot about penalizing large errors, it’s not a bad choice. It’s a great choice for a loss metric when hyperparameter tuning or batch training a deep neural network.
Mean Absolute Error (MAE) is the average of the absolute value of the errors.
Let’s get right to math equation:
(1 / n) * (∑ |y - ŷ|)
In code:
np.average(np.abs(y_true - y_pred))
subtract the predicted values from the actual y valuestake the absolute value of each errorsum themtake the average
subtract the predicted values from the actual y values
take the absolute value of each error
sum them
take the average
Here’s how to get the MAE with a scikit-learn function:
from sklearn.model_selection import mean_absolute_errormean_absolute_error(y_actual, y_predicted, squared=False)
The MAE is conceptually the easiest evaluation metric for regression problems. It answers the question, “How far were you off in your predictions, on average?”
The units make intuitive sense. Yes! 🎉
For example, say you are predicting house sale prices and the mean actual sale price in the test set is $500,000. An MAE of $10,000 means the model was off by an average of $10k in its predictions. That’s not bad! 😀
Unlike RMSE scores, bad predictions don’t result in disproportionately high MAE scores.
The MAE will always be closer to 0 than the RMSE (or the same).
Note that computing the RMSE as an optimization metric for a model with a training loop is faster than computing MAE.
Turning to our example dataset for a final time:
The MAE is 0.37. The predictions were off from the mean of 2.2 by an average of 0.37. I can quickly understand that statement. 😀
The RMSE was 0.48 and the R2 was 0.71.
MAE is the simplest evaluation metric and most easily interpreted. It’s a great metric to use if you don’t want a few far off predictions to overwhelm a lot of close ones. It’s a less good choice if you want to penalize predictions that were really far off the mark.
So which metric should you use? In general, I suggest you report all three! 🚀
R2 gives people evaluating the performance an at-a-glance understanding of how well your model performs. It’s definitely worth report it.
RMSE is less intuitive to understand, but extremely common. It penalizes really bad predictions. It also make a great loss metric for a model to optimize because it can be computed quickly. It merits reporting.
I came out of this article with new respect for MAE. It’s straightforward to understand and treats all prediction errors proportionately. I would emphasize it in most regression problem evaluations.
Disagree? Let me know on Twitter. 👍
If you want to learn about metrics for classification problems, I wrote about them here.
I hope you enjoyed this guide to popular Python data science packages. If you did, please share it on your favorite social media so other folks can find it, too. 😀
I write about Python, SQL, Docker, and other tech topics. If any of that’s of interest to you, sign up for my mailing list of awesome data science resources and read more to help you grow your skills here. 👍
Happy reporting! ⚖️