Choosing the Right Metric for Evaluating Machine Learning Models — Part 2
By: Alvira Swalin
Tags: Machine Learning, Classification, Towards Data Science, Metrics

In the first blog, we discussed some important metrics used in regression, their pros and cons, and use cases. This part will focus on commonly used metrics in classification, why should we prefer some over others with context.

Definitions

Let’s first understand the basic terminology used in classification problems before going through the pros and cons of each method. You can skip this section if you are already familiar with the terminology.

Source of Image: Wikipedia

Recall or Sensitivity or TPR (True Positive Rate) : Number of items correctly identified as positive out of total true positives- TP/(TP+FN)

: Number of items correctly identified as positive out of total true positives- TP/(TP+FN) Specificity or TNR (True Negative Rate): Number of items correctly identified as negative out of total negatives- TN/(TN+FP)

Number of items correctly identified as negative out of total negatives- TN/(TN+FP) Precision: Number of items correctly identified as positive out of total items identified as positive- TP/(TP+FP)

Number of items correctly identified as positive out of total items identified as positive- TP/(TP+FP) False Positive Rate or Type I Error: Number of items wrongly identified as positive out of total true negatives- FP/(FP+TN)

Number of items wrongly identified as positive out of total true negatives- FP/(FP+TN) False Negative Rate or Type II Error: Number of items wrongly identified as negative out of total true positives- FN/(FN+TP)

Source of Image: Effect Size FAQs by Paul Ellis

Confusion Matrix

F1 Score : It is a harmonic mean of precision and recall given by-

F1 = 2*Precision*Recall/(Precision + Recall)

: It is a harmonic mean of precision and recall given by- F1 = 2*Precision*Recall/(Precision + Recall) Accuracy: Percentage of total items classified correctly- (TP+TN)/(N+P)

ROC-AUC Score

The probabilistic interpretation of ROC-AUC score is that if you randomly choose a positive case and a negative case, the probability that the positive case outranks the negative case according to the classifier is given by the AUC. Here, rank is determined according to order by predicted values.

Source of Image: UNC Lecture

Mathematically, it is calculated by area under curve of sensitivity (TPR) vs.

FPR(1-specificity). Ideally, we would like to have high sensitivity & high specificity, but in real-world scenarios, there is always a tradeoff between sensitivity & specificity.

Some important characteristics of ROC-AUC are-

The value can range from 0 to 1. However auc score of a random classifier for balanced data is 0.5

ROC-AUC score is independent of the threshold set for classification because it only considers the rank of each prediction and not its absolute value. The same is not true for F1 score which needs a threshold value in case of probabilities output

Log-Loss

Log-loss is a measurement of accuracy that incorporates the idea of probabilistic confidence given by following expression for binary class:

It takes into account the uncertainty of your prediction based on how much it varies from the actual label. In the worst case, let’s say you predicted 0.5 for all the observations. So log-loss will become -log(0.5) = 0.69. Hence, we can say that anything above 0.6 is a very poor model considering the actual probabilities.

Case 1

Comparison of Log-loss with ROC & F1