Metrics and Python. In a series of three articles, we will... | by Gabriel Naya | Towards Data Science

As a programmer I lacked on studies about mathematical and statistical concepts. Now with more experience I’ve decided to compile and share some of the metrics and concepts that I’ve faced for different problems I encountered.
In this series of articles, we will briefly review each of them, their use, their formulas, examples, and implementation in Python, so we can see them all together and have this chart on hands, if necessary.Here is part one, referring to the most used metrics for regression cases.
For this article, we have used the Kaggle dataset on housing prices, because it is a typical example of regression with which many of us are familiar.Elementary engineering features have been done to be able to present a minimum reasonable model, which serves as an example for the development of metrics.
from scipy.stats import boxcox_normmaxfrom scipy.special import boxcox1pfrom sklearn.linear_model import LinearRegressiontrain = pd.read_csv('../input/home-data-for-ml-course/train.csv')y = train.SalePrice.reset_index(drop=True)features = trainend_features = ['OverallQual','GrLivArea','GarageCars','GarageArea','TotalBsmtSF','1stFlrSF','FullBath','TotRmsAbvGrd','MSSubClass','MSZoning']features = features[end_features]features['MSSubClass'] = features['MSSubClass'].apply(str)features['MSZoning'] = features.groupby('MSSubClass')['MSZoning'].transform(lambda x: x.fillna(x.mode()[0]))objects = [col for col in features.columns if features[col].dtype == "object"]features.update(features[objects].fillna('None'))numeric_dtypes = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']numerics = [col for col in features.columns if features[col].dtype in numeric_dtypes]features.update(features[numerics].fillna(0))for i in numerics:    features[i] = boxcox1p(features[i], boxcox_normmax(features[i] + 1))X = pd.get_dummies(features).reset_index(drop=True)#----------------- The modelreg = LinearRegression().fit(X, y)y_pred = reg.predict(X)
Here is a simple table with the review of the metrics we are going to describe, the last column of the table is the result of our model.
MAE measures the average magnitude of the errors in a set of predictions, without considering their direction. It’s the average over the test sample of the absolute differences between prediction and actual observation where all individual differences have equal weight.
The mean absolute error uses the same scale as the data. This is known as a scale-dependent accuracy measure and, therefore, cannot be used to make comparisons between series using different scales.
from sklearn.metrics import mean_absolute_errorprint ('Sk MAE: ' + str(mean_absolute_error(y,y_pred)) )def MAE(predict,target):    return (abs(predict-target)).mean()print ('My MAE: ' + str(MAE(y_pred,y)))
MSE is a risk function, corresponding to the expected value of the squared error loss. The fact that MSE is almost always strictly positive (and not zero) is because of randomness or because the estimator does not account for information that could produce a more accurate estimate. The MSE is a measure of the quality of an estimator — it is always non-negative, and values closer to zero are better.
“Minimizing MSE is a key criterion in selecting estimators: see minimum mean-square error. Among unbiased estimators, minimizing the MSE is equivalent to minimizing the variance, and the estimator that does this is the minimum variance unbiased estimator. However, a biased estimator may have lower MSE; see estimator bias.
In statistical modelling the MSE can represent the difference between the actual observations and the observation values predicted by the model. In this context, it is used to determine the extent to which the model fits the data as well as whether removing some explanatory variables is possible without significantly harming the model’s predictive ability.”
from sklearn.metrics import mean_squared_errorprint ('Sk MSE: ' + str(mean_squared_error(y,y_pred)) )def MSE(predict,target):    return ((predict-target)**2).mean()print ('My MSE: ' + str(MSE(y_pred,y)) )
RMSE is a quadratic scoring rule that also measures the average magnitude of the error. It’s the square root of the average of squared differences between prediction and actual observation.
Comparision RMSE vs MAE
Similarities:
Express average model prediction error in same units of the variable of interest.
Can range from 0 to ∞ and are indifferent to the direction of errors.
Lower values are better.
Difference:
Taking the square root before they are averaged, RMSE gives a relatively high weight to large errors, so RMSE should be useful when large errors are undesirable.
def RMSE(predict, target):    return np.sqrt(((predict - target) ** 2).mean())print ('My RMSE: ' + str(RMSE(y_pred,y)) )
Measure of prediction accuracy of a forecasting method in statistics, for example in trend estimation, also used as a loss function for regression problems in machine learning. It usually expresses accuracy as a percentage.
Although the concept of MAPE sounds very simple and convincing, it has major drawbacks in practical application [*], and there are many studies on shortcomings and misleading results from MAPE
def MAPE(predict,target):    return ( abs((target - predict) / target).mean()) * 100print ('My MAPE: ' + str(MAPE(y_pred,y)) )
In case of RMSLE, you take the log of the predictions and actual values. So basically, what changes is the variance that you are measuring. I believe RMSLE is usually used when you don’t want to penalize huge differences in the predicted and the actual values when both predicted and true values are huge numbers.
RMSLE measures the ratio between actual and predicted.
log(pi+1)−log(ai+1)log(pi+1)−log(ai+1)
can be written as log((pi+1)/(ai+1))log((pi+1)/(ai+1))
It can be used when you don’t want to penalize huge differences when the values are huge numbers.
Also, this can be used when you want to penalize under estimates more than over estimates.
Lets have a look at the below example
Case a) : Pi = 600, Ai = 1000 — RMSE = 400, RMSLE = 0.5108
Case b) : Pi = 1400, Ai = 1000 — RMSE = 400, RMSLE = 0.3365
import mathdef RMSLE(predict, target):    total = 0     for k in range(len(predict)):        LPred= np.log1p(predict[k]+1)        LTarg = np.log1p(target[k] + 1)        if not (math.isnan(LPred)) and  not (math.isnan(LTarg)):             total = total + ((LPred-LTarg) **2)            total = total / len(predict)            return np.sqrt(total)print ('My RMSLE: ' + str(RMSLE(y_pred,y)) )
R2 and R-Squared help us to know how good our regression model as compared to a very simple model that just predicts the mean value of target from the train set as predictions.
def R2(predict, target):    return 1 - (MAE(predict,target) / MAE(target.mean(),target))def R_SQR(predict, target):    r2 = R2(predict,target)    return np.sqrt(r2)print ('My R2         : ' + str(R2(y_pred,y)) )print ('My R          : ' + str(R_SQR(y_pred,y)) )
A model performing equal to baseline would give R-Squared as 0. Better the model, higher the r2 value. The best model with all correct predictions would give R-Squared as 1. However, on adding new features to the model, the R-Squared value either increases or remains the same. R-Squared does not penalize for adding features that add no value to the model. So an improved version over the R-Squared is the adjusted R-Squared
def R2_ADJ(predict, target, k):    r2 = R2(predict,target)    n = len(target)    return (1 -  ( (1-r2) *  ( (n-1) / (n-(k+1)) ) ) )k= len(features.columns)print ('My R2 adjusted: ' + str(R2_ADJ(y_pred,y,k)) )
Putting the main regression metrics in a single table, and programming the functions for each of them has been an excellent personal experience.Applying them on the same exercise, since it allowed me to compare their uses, understand their formulas, and while developing the article, to have a complete picture. Hopefully, the article has been useful, and any comments or corrections will be welcome.
Next week we will deliver the second part of this series, an article on classification metrics ...