Evaluation Metrics: Regression
By: Priyanka Dave
Tags: R Squared, Mean Absolute Error, Scikit Learn, Mean Squared Error, Evaluation Metric

What is Model Evaluation in machine learning?

Model Evaluation is the process to check how accurate or inaccurate our model is.

Scikit-learn library provides different types of evaluation metrics which we can use to check model performance.

We can implement our own evaluation metrics also based on our data set and domain requirement.

Common Regression Evaluation Metrics:

Mean Absolute Error (MAE) Mean Squared Error (MSE) Root Mean Squared Error (RMSE) R-squared

1. Mean Absolute Error (MAE):

Used to check how inaccurate our model is.

our model is. MAE = Sum of absolute error/number of samples

Absolute Error = Absolute difference between actual and predicted data.

Below it shows predicted values from 2 different models and respective absolute errors.

Calculate MAE for both the models:

MAE_Model_2 = 32996.03 / 5 = 6599.20

MAE_Model_4= 16670.90 / 5 = 3334.18

Same thing can be achieved using scikit-learn:

from sklearn.metrics import mean_absolute_error

print(mean_absolute_error(actual, prediction))

Note: MAE_Model_4 is lower than MAE_Model_2 so we can say that model used to predict prediction_4 is better. This metric fails to punish large errors in prediction as it uses simple difference between actual and predicted data.

2. Mean Squared Error (MSE):

Used to check how inaccurate our model is.

our model is. MSE is harsher than MAE. MAE measures simply the distance between predictions and actual values, where MSE squares the distance.

MSE = Sum of squared error/number of samples

Squared Error = Squared difference between actual and predicted data.

Below it shows predicted values from 2 different models and respective squared errors.

Calculate MSE for both the models:

MSE_Model_2 = 412552977.70 / 5 = 82510595.53

MSE_Model_4= 177061670.31 / 5 = 35412334.06

Same thing can be achieved using scikit-learn:

from sklearn.metrics import mean_squared_error

print(mean_squared_error(actual, prediction))

Note: MSE punishes larger errors and reduces the influence of small ones. Example: If error is less than 1, it will shrink while squaring. On the other hand, if this error is greater than 1, by squaring it we are making its contribution to the overall error bigger.

3. Root Mean Squared Error (RMSE):

Used to check how inaccurate our model is.

our model is. This is actually the square root of MSE.

RMSE = sqrt(Sum of squared error/number of samples)

Calculate MSE for both the models:

RMSE_Model_2 = sqrt(82510595.53) = 9083.53

RMSE_Model_4= sqrt(35412334.06) = 5950.82

Same thing can be achieved using scikit-learn:

from sklearn.metrics import mean_squared_error

print(np.sqrt(mean_squared_error(actual, prediction)))

or

print(mean_squared_error(actual, prediction, squared=False))

Note: RMSE of RMSE_Model_4 is lesser than in RMSE_Model_2. RMSE is the standard deviation of the error. In other words it’s a measure to check spread out of error. Lower values of RMSE indicate better fit.

4. R-squared:

Used to check how accurate our model is.

our model is. R-squared explains to what extent the variance of input variable explains the variance of the output variable.

So, if the R-squared of a model is 0.50, then approximately half of the observed variation can be explained by the model’s inputs.

R-squared values range from 0 to 1 and are commonly stated as percentages from 0% to 100%.

Formula to calculate R-squared:

Here,

y = actual values, y_hat = predicted values, y_bar = mean of actual values

R-squared formula

Note: Limitations of R-squared is, it doesn’t tell you whether your chosen model is good or bad, nor will it tell you whether the data and predictions are biased.

Create your own evaluation metrics:

MAE, MSE and RMSE uses absolute and squared errors.

Here we cannot differentiate that out of n samples, how many of them predicted below fitted line and how many of them predicted above fitted line.

Also if range of values in target variable will be high then also above metrics will not be able to show real picture. In such situation we should use percentage difference instead of simple difference.

For example, if actual value is 10 and predicted is 15 then we can say it as poor prediction but if actual value is 1000 and predicted is 1015 then we can say it as good prediction even though difference is same. In 1st case difference is 50% while in 2nd case difference is 0.5%

Summary:

To choose proper evaluation metrics for a machine learning model is very important.

Most commonly used evaluation metrics are MAE, MSE, RMSE and R-squared.

One can implement her own evaluation metrics also.