Which Evaluation Metric Should You Use in Machine Learning Regression Problems?
By: Jeff Hale
Tags: Machine Learning, Technology, Artificial Intelligence, Python, Data Science

R-Squared (R²)

R² represents the proportion of variance explained by your model.

R² is a relative metric, so you can use it to compare with other models trained on the same data. And you can use it to get a rough a feel for how well a model performs, in general.

Disclaimer: This article isn’t a review of machine learning methods, but make sure you use different data for training, validation, and testing. You always want to hold out some data that your model has not seen to evaluate its performance. Also, it’s a good idea to look at plot of your model’s predictions vs. the actual values to see how well your model fit the data.

Let’s see how R² is computed. Onward! ➡️

Formula and code

Here is one way to formulate R².

1 - (SSE/SST)

SSE is the sum of squared errors; the sum of the squared differences between the actual values and predicted values.

SST is the total sum of squares (shown sometimes as TSS); the sum of the squared differences between the actual values and the mean of the actual values.

With more mathy notation:

1 - (∑(y - ŷ)² / ∑(y - y̅)²)

Here’s what the code looks like —adapted from scikit-learn, the primary Python machine learning library.

numerator = ((y_true - y_pred) ** 2).sum()

denominator = ((y_true - np.average(y_true)) ** 2).sum() r2_score = 1 - (numerator / denominator)

In words

subtract the predicted values from the actual y values square the results sum them

That’s the numerator.

subtract the mean of the actual y values from each actual y value square the results sum them

That’s the denominator.

1 - the numerator/denominator is the R². 🎉

R² is the default metric for scikit-learn regression problems. If you want to use it explicitly you can import it and then use it like this:

from sklearn.metrics import r2_score r2_score(y_true, y_pred)

Interpretation

A model that explains no variance would have an R² of 0. A model with an R² of 1 would explain all of the variance. Higher scores are better.

However, if your R² is 1 on your test set you are probably leaking information or the problem is fairly simple for your model to learn. 👍

In some fields, such as the social sciences, there are lots of factors that influence human behavior. Say you have a model with just a few independent variables that results in an R² close to .5. Your model is able to account for half of the variance in your data, and that’s quite good. 😀

It is possible to have an R² that is negative. Negative scores occur when the predictions the model makes fit that data worse than the mean of the output values. Predicting the mean each time is a null model. See more here.

Example

Say you have the following small toy test dataset:

All code is available on GitHub in this Jupyter notebook.

Here’s a plot of the actual and predicted y values.

The R² of the model is 0.71. The model is accounting for 71% of the variance in the data. That’s not too shabby, although we’d like more test data. 😀

As another example, let’s say the true values for y are [55, 2, 3]. The mean is 20. Predicting 20 for each y value results in an R² of 0.

A model that predicts [1 , 2, 2] for the true values above results in an R² of -0.59. Bottom line, you can do far worse than the null model! In fact, you can predict infinitely worse, resulting in an infinitely low R². 😲

As a brief aside, let’s look at the Adjusted R² and machine learning vs. statistics.

Adjusted R²

The Adjusted R² accounts for the addition of more predictor variables (features).

Adjusted R² will only increase with a new predictor variable when that variable improves the model performance more than would be expect by chance. Adjusted R² helps you focus on using the most parsimonious model possible. 😉

The Adjusted R² is more common in statistical inference than in machine learning. Scikit-learn, the primary Python library for machine learning, doesn’t even have an Adjusted R² metric. Statsmodels, the primary statistical library for Python does. If you want to learn more about when to use which Python library for data science, I wrote a guide here.

You can compute the Adjusted R2 if you know the number of feature columns (p) and the number of observations (n). Here’s the code:

adjusted_r2 = 1 — ((1 — r2) * (n — 1)) / ( n — p — 1)

n-1 is the degrees of freedom. Whenever you hear that term, you know you are in statistics land. In machine learning we generally care most about predictive ability, so R² is favored over Adjusted R².

Another note on statistics vs. machine learning: our focus is on machine learning, so prediction rather than causality. R² — and the other metrics that we’ll see, don’t say anything about causality by themselves.

Bottom Line

R² tells you how much variance your model accounts for. It’s handy because the R² for any regression problem will immediately provide some (limited) understanding of how well the model is performing. 😀

R² is a relative metric. Let’s see a few absolute metrics now.