What are RMSE and MAE?. A Simple Guide to Evaluation Metrics | by Shwetha Acharya | Towards Data Science

Root Mean Squared Error (RMSE)and Mean Absolute Error (MAE) are metrics used to evaluate a Regression Model. These metrics tell us how accurate our predictions are and, what is the amount of deviation from the actual values.
Technically, RMSE is the Root of the Mean of the Square of Errors and MAE is the Mean of Absolute value of Errors. Here, errors are the differences between the predicted values (values predicted by our regression model) and the actual values of a variable. They are calculated as follows :
On close inspection, you will see that both are average of errors.
Let’s understand this with an example. Say, I want to predict the salary of a data scientist based on the number of years of experience. So, salary is my target variable (Y) and experience is the independent variable(X). I have some random data on X and Y and we will use Linear Regression to predict salary. Let’s use pandas and scikit-learn for data loading and creating linear model.
import pandas as pdfrom sklearn.linear_model import LinearRegressionsal_data={"Exp":[2,2.2, 2.8, 4, 7, 8, 11, 12, 21, 25],           "Salary": [7, 8, 11, 15, 22, 29, 37 ,45.7, 49, 52]}#Load data into a pandas Dataframedf=pd.DataFrame(sal_data)df.head(3)
#Selecting X and y variablesX=df[['Experience']]y=df.Salary#Creating a Simple Linear Regression Model to predict salarieslm=LinearRegression()lm.fit(X,y)#Prediction of salaries by the modelyp=lm.predict(X)print(yp)[12.23965934 12.64846842 13.87489568 16.32775018 22.45988645 24.50393187 30.63606813 32.68011355 51.07652234 59.25270403]
Now, we have ‘yp’ — our array of salary prediction and we will evaluate our model by plotting predicted(yp) and actual salary(y). I am using bohek for my visualizations.
from bokeh.plotting import figure, show, output_filep=figure(title="Actual vs Predicted Salary", width=450, height=300)p.title.align = 'center'p.circle(df.Exp, df.Salary)p.line(df.Exp, df.Salary, legend_label='Actual Salary', line_width=3, line_alpha=0.4)p.circle(df.Exp, yp, color="red")p.line(df.Exp,yp, color="red",legend_label='Predicted Salary', line_width=3, line_alpha=0.4)p.xaxis.axis_label = 'Experience'p.yaxis.axis_label = 'Salary'show(p)
From the graph above, we see that there is a gap between predicted and actual data points. Statistically, this gap/difference is called residuals and commonly called error, and is used in RMSE and MAE. Scikit-learn provides metrics library to calculate these values. However, we will compute RMSE and MAE by using the above mathematical expressions. Both methods will give you the same result.
import numpy as npprint(f'Residuals: {y-yp}')np.sqrt(np.mean(np.square(y-yp)))  #RMSEnp.mean(abs(y-yp))                 #MAE#RMSE/MAE computation using sklearn libraryfrom sklearn.metrics import mean_squared_error, mean_absolute_errornp.sqrt(mean_squared_error(y, yp))mean_absolute_error(y, yp)6.485.68
This is our baseline model. MAE is around 5.7 — which seems to be higher. Now our goal is to improve this model by reducing this error.
Let’s run a polynomial transformation on “experience” (X) with the same model and see if our errors reduce.
from sklearn.preprocessing import PolynomialFeaturespf=PolynomialFeatures()     #Linear Equation of degree 2X_poly=pf.fit_transform(X) lm.fit(X_poly, y)yp=lm.predict(X_poly)
I have used Scikit-learn PolynomialFeatures to create a matrix of 1, X, and X2 and passed that as input to my model.
Calculating our error metrics and ....
#RMSE and MAEnp.sqrt(np.mean(np.square(y-yp)))np.mean(abs(y-yp))2.39741.6386
Voilaaa... they are much lower this time. It fits better than our baseline model! Let’s plot y and yp (like how we did before) to check the overlap.
The gap between the 2 lines has reduced. Let’s observe the distribution of residuals or errors(y-yp) using seaborn’s residual plot function
print(y-yp)   #residuals[ 0.333921 0.447306  0.84028668 -0.136044 -4.190238 -0.434767 -0.847751  5.488121 -2.584481  1.083648]import seaborn as snssns.residplot(y, yp)  plt.show()
We see that residuals tend to concentrate around the x-axis, which makes sense because they are negligible.
There is a third metric — R-Squared score, usually used for regression models. This measures the amount of variation that can be explained by our model i.e. percentage of correct predictions returned by our model. It is also called the coefficient of determination and calculated by the formula:
Let’s compute R2 mathematically using the formula and using sklearn library and compare the values. Both methods should give you the same result.
#Calculating R-Squared manuallya=sum(np.square(y-yp))           # a -> sum of square of residualsb=sum(np.square(y-np.mean(y)))   # b -> total sum of sqauresr2_value = 1-(a/b)0.979#calculating r2 using sklearnfrom sklearn.metrics import r2_scoreprint(r2_score(y, yp))0.979
Thus, overall we can interpret that 98% of the model predictions are correct and the variation in the errors is around 2 units. For an ideal model, RMSE/MAE=0 and R2 score = 1, and all the residual points lie on the X-axis. Achieving such a value for any business solution is almost impossible!
Some of the techniques we can use to improve our model accuracy include:
Transform/scale features
Treat outliers (if many)
Add new features/feature engineering
Use different algorithms
Model hyperparameter tuning
In my next article, I will be explaining some of the above concepts in detail. You can refer to my notebook here for the above code.