Evaluation Metrics in Machine Learning Models using Python
By: Manoj Singh
Tags: Machine Learning, Statistics, Confusion Matrix, Python, Data Science

We will try to evaluate our machine learning models on different error metrics. We need to remember while evaluating models that it should be immune to class imbalance if our data set is a classical example to imbalance data set. We will deal with typical imbalance dataset examples in upcoming blogs.

A list of few popular Evaluation Metrics are followings.

For Classification Problem:

1. Confusion Matrix

2. Precision / Recall

3. F1 Score

4. Area Under the ROC curve (AUC — ROC)

5. Cohen’s Kappa

For Regression Problem:

1. Root Mean Squared Error(RMSE)

2. R-Squared/Adjusted R-Squared

Lets try an understand them sequentially.

Confusion Matrix:

This matrix represents the accuracy of the model. A confusion matrix is an N X N matrix, where N is the number of classes being predicted. Let’s take an example of two class problem for the credit card company which wants to detect fraud using the algorithm you have to build. The possible two class would be either its Fraud or Not Fraud transaction.

Fig: Confusion Matrix

Here,

True Positive (TP)= 1 i.e. actual Fraud and also predicted Fraud

False Positive(FP) = 1 i.e. actual Not Fraud but predicted Fraud

False Negative(FN) = 2 i.e. actual Fraud but predicted Not Fraud

True Negative(TN) = 996 i.e. actual Not Fraud and predicted Not Fraud

Accuracy of the model is given by

accuracy = (TP+TN) / (TP+TN+FP+FN)

In the above matrix, we have calculated the proportion of true positive and true negative in all evaluated cases.

Precision / Recall:

Precision: It is the ratio of the correct predictions to the total number of predictions for that class. It answers the question Of all the values predicted as belonging to the class “Fraud”, what percentage is correct?

Precision(P)=TP/ (TP+FP​)

The precision in our above matrix example would be =(1)/(1 + 1)=1/2=0.5

Recall: It is the ratio of the number of correct predictions to the total number of actual instances of the class. It answers the question Of all the instances of the class “X”, what percentage did we predict correctly?

Recall(R)=TP/ (TP+FN)​

The recall in our above matrix example is =(1)/(1+2) = 1/3 =0.34

Suppose two models: Model 1 and Model 2 are implemented on the credit fraud dataset where 0 indicates Not Fraud and 1 is for Fraud .Following are the observations:

Model 1 is able to correctly predict 0 [Not Fraud] better than Model 2. Thereby Model 1 has lesser False Positives, thereby higher Precision

Model 2 is able to correctly predict 1 [Fraud] better than Model 1. Thereby Model 2 has lesser False Negatives, thereby higher Recall.

This is called as Precision-Recall Tradeoff. It completely depends on a business requirement to choose precision over recall or vice-versa. In this case, we should be more concerned about the recall. Because the Fraud if not detected then it would cost a loss to a business, predicting not fraud to fraud impact can be minimized by doing extra monitoring or manual verification.

An example where the precision metric is more important are recommendations systems(youtube playlist recommendation, TED X recommendation), stock prediction, Price of house, etc.

Example where recall metric is more important are disease prediction, terrorist prediction, fraud cases, loan defaulters.

F1 Score:

What if we wish to have both good precisions and recall in a model?

In such cases, we used F1 score. It is nothing but a harmonic mean of precision and recall.

F1 score= 2PR / (P+R)

The F1 score in our above matrix example

F1 score =(2*0.5*0.34) / (0.5+0.34)=0.34/ 0.84 = 0.40

It is more useful when you have uneven class distribution.

Area Under the ROC curve(AUC — ROC):

The ROC curve is the plot between sensitivity also called as True Positive Rate and (1- specificity) also called False Positive Rate.

True Positive Rate(TPR) = TP / (TP+FN)

False Positive Rate(FPR) = FP / (FP+FN)

Fig: Confusion Matrix

Fig: Roc curve

More the area under the curve better is the model. The random line represents a random prediction of a model which is 0.5 which is considered as the worst case. So our curve should be above that random model line for the model to be good.

Python Implementation for Classification Problem for the above Matrix can explore here. Github_link_for_classification_matrix

Cohen’s Kappa:

It is a metric that compares an Observed Accuracy with an Expected Accuracy (random chance). let try to understand with help of matrix.

Fig: Cohen’s kappa Matrix

kappa is defined as K = (Po - Pe) /(1- Pe) Where,

Po = Observed Accuracy

Pe = Expected Accuracy Po simply is number of instances that were classified correctly. From above matrix we can ‘a’ and ‘d’ were classified correctly. Po = (a+d) / (a+b+c+d) Pe = Probability by random chance

= P (yes) or P (no)

= (Classifier classifies it as yes) * (Actual yes) +

(Classifier classifies it as no) * (Actual no)



Pe = [(a+b)/(a+b+c+d)] * [(a+c)/(a+b+c+d)] + [(c+d)/(a+b+c+d)] * [(b+d)/(a+b+c+d)]

If the value of observed accuracy is better than expected accuracy then the model is said to be good.

Root Mean Squared Error (RMSE):

It is used in regression problems. It follows an assumption that error is unbiased and follows a normal distribution. It is given by the following formula.

Where N is a Total Number of Observations.

fig: RMSE

Note 1) For more samples, reconstructing the error distribution using RMSE is considered to be more reliable.

2) It is highly affected by outlier values.

3) As compare to mean absolute error, RMSE gives higher weightage and punishes large errors.

R-Squared / Adjusted R-Squared:

In AUC — ROC we learned that our model should be better than accuracy by random chance, so here we have a benchmark to compare our model in classification problem but in case of regression model when RMSE decreases the model performance improves but we still do not have a benchmark to compare it. This is the reason we require R-Squared / Adjusted R-Squared. It measures the goodness of fit of a straight line in the regression model.

R-squared is always between 0 and 100%.

let’s understand a few more terminology to better understand the R-Squared.

Error Sum of Squares (SSE): It is nothing but the sum of squared residuals.

Fig: SSE

yi​ : Actual Values

y^i: Predicted value

Total Sum of Squares (SST): It is nothing but the Squared difference between the Actual Values (yi​) and the Mean of our dataset (yˉ​i​).

Fig: SST

Regression Sum of Squares ( SS(Regression) ): It is the squared difference between the Predicted values (y^​i​) and the Mean (yˉ​i​).

Fig: SS(regression)

We can observe the following,

SS(Total)=SS(Regression) + SSE

R-squared is now defined as,

Fig: R-Squared Formula

Higher the value of R-squared better is the model. The best model with all correct predictions would give R-Squared as 1. The problem with the R-squared is, it can be made artificially high by adding more number of independent variables (features) although they might be irrelevant. However, on adding new features to the model, the R-Squared value either increases or remains the same. R-Squared does not penalize for adding features that add no value to the model. So an improved version over the R-Squared is the Adjusted R-Squared.

Fig: Adjusted R-Squared Formula

where,

n = Number of data points

p = Number of features/independent variables

Note: R-squared tells you how well your model fits the data points whereas Adjusted R-squared tells you how important is a particular feature to your model.

Python Implementation for Linear regression for the above Matrix can explore here. github_link_for_regression

So, whenever you build a model, this article should help you to figure out what these parameters evaluate error metrics and how good your model has performed.

I hope this blog was useful to you. Please leave comments or send me an email if you think I missed any important details or if you have any other questions or feedback about this topic.