Evaluation Metrics for Regression Analysis
By: Sam Thurman
Tags: Machine Learning, Deep Learning, AI, Data Science, Statistics

Evaluation Metrics for Regression Analysis

Methods for quantifying error and assessing predictive performance in regression modeling

Terms to know

These terms will come up, and it’s good to get familiar with them if you aren’t already:

Regression analysis — a set of statistical processes for estimating a continuous dependent variable given a number of independents

Variance — measurement of the spread between numbers in a data set

ŷ — the estimated value of y

ȳ — mean value of y

“Goodness of fit”

Goodness of fit is typically a term used to describe how well a dataset aligns with a certain statistical distribution. Here, we’re going to think of it as a way of describing how well our model is fitted to our data.

If we can think about our regression model in terms of the imaginary “best-fit” line it produces, then it makes sense that we would want to know how well this line matches our data. This goodness of fit can be quantified in a variety of ways, but the R² and the adjusted R² score are two of the most common methods for describing how well our model is capturing the variance in our target data.

R²

R² — also called the coefficient of determination — is a statistical measure representing the amount of variance for a dependent variable that is captured by your model’s predictions. Essentially, it is a measure of how well your model is fitted to the data. This score will always fall between -1 and 1, with values closest to 1 being best (value of 1 means our model is completely explaining our dependent variable).

R² uses a sort of “baseline model” as a marker to compare our regression results against. This baseline model simply predicts the mean every time, regardless of the data. After fitting the regression model, the predictions of our baseline (mean-guessing) model are compared to the predictions of our newly fitted model in terms of errors squared.

fig. 1

SSR — residual sum of squared errors for our regression model (squared difference between y and ŷ)

— residual sum of squared errors for our regression model (squared difference between y and ŷ) SST — total sum of squared errors (squared difference between y and ȳ)

We can see now that if SSR is very low, meaning our model has low error, our R² score will be closer to 1. Likewise, if we have higher amounts of error in our model, R² will swing towards 0 and in the worst cases -1.

Adjusted R²

R² is a great indicator of the goodness-of-fit of our model, but it has one major drawback — R squared will always increase as we add more independent variables. This happens regardless of whether or not our new features are actually predictive of our target, giving us a sometimes false sense of confidence in our newly included variables. This is where the adjusted R² score comes into play.

It’s important to note that adjusted R² is always going to be lower than R², and we’ll see why in just a second. Unlike with R², the adjusted R² score takes into account how effective each feature is at explaining our target, and will actually decrease as we add less-predictive features to our model.

The formula for adjusted R² looks a little messy but it’s easy to solve once we’ve calculated our R² score

fig. 2

R² — R² score for our model, calculated using the formula from fig. 1

— R² score for our model, calculated using the formula from fig. 1 N — number of items in our dataset

— number of items in our dataset p — number of independent variables

The major feature to pay attention to here is that as p (no. items in dataset) increases, R²adj will always decrease unless there is a significant boost to R². This ensures that our R²adj score reflects the usefulness of each feature.

Quantifying error

Another intuitive way of measuring success in regression modeling is to talk about the total amount of error in our model’s predictions. While R² and adjusted R² both factor in model error at some level, what if we want a separate metric to express this error? That’s where MAE, MSE, and RMSE come in handy.

MAE (Mean Absolute Error)

MAE is the most intuitive out of the three, and the calculation is performed just like the name implies. We simply calculate the absolute difference between the predicted and actual target values for each sample, and divide by the number of samples in the dataset. We take the absolute difference because we are considering error in either direction.

fig. 3

n — number of samples

This value can range from 0 to infinity, with values closest to 0 being best. It’s important to note that this value will be in terms of the continuous dependent variable you are measuring, making it easily interpretable. For example, if we’re predicting pizza prices, MAE would tell us the avg. dollars off our predictions were from the actual prices. MAE is preferred when we aren’t too worried about outliers — since we are only taking the mean of our raw errors, very poor predictions aren’t necessarily weighted any differently than very good predictions in the MAE calculation.

MSE (Mean Squared Error)

MSE works a lot like MAE, however here, our method for dealing with positive and negative errors will be to square our error term as opposed to taking the absolute value of it. Again, we divide by the number of samples to get a mean score for the dataset.

fig. 4

n — number of samples

With MSE, it’s important to note that we no longer get an error score in terms of our target variable. Instead, we get a potentially very large number that is very responsive to errors in our predictions — essentially penalizing large error values in the squaring step. Given its lack of interpretability, MSE is typically used more as a stepping stone to RMSE; however, it can be used by itself as an extremely good indicator of outliers in predictions.

RMSE (Root Mean Squared Error)

RMSE is almost identical to MSE and very easy to calculate. To find it, we simply take the square root of our MSE score.

fig. 5

Clearly, RMSE will be smaller than MSE, with the main motivation being that now we have a score in terms of our target variable (dollars, for our pizza prices example). This creates an error metric that is both sensitive to outliers — large errors are penalized — and easily interpretable — score is a continuous value with the same unit as our target.