Evaluation Metrics for Classification Explained | by Eunjoo Byeon | Towards Data Science

Evaluation metric refers to a measure that we use to evaluate different models. Choosing an appropriate evaluation metric is a decision problem that requires a thorough understanding of the goal of a project and is a fundamental step before all modeling process that follows. So why is it so important, and how should we choose?
Let’s say you live in a city where it is sunny about 300 days a year. During the other 65 days, it gets very uncomfortable with heavy rain or snow. In this imaginary world, there are only two weather apps. One app predicts the weather correctly 80% of the time, while the other one is right about 70% of the time. Which app would you choose?
If an app always predicts sunny in this city, this app will be 80% correct, even though it never predicts bad weather accurately. In this case, this app is useless! On the other hand, the other app might fail to predict the sunny weather about 30% of the time, but it will tell you about bad weather accurately over 80% of the time. In this case, this app that correctly predicts the weather 70% of the time is much more useful than the 80% app.
As we can see here, how we choose to evaluate our model can tell vastly different information about our data. It must be determined by the goal of the project and the properties of the dataset. No data will be without its variance, and no model can explain the world perfectly. So we must know where we would rather have such prediction errors to happen.
When a model makes a binary prediction, there are 4 possible outcomes. A model can predict yes (positive) or no (negative), and it can either be correct (true) or wrong (false). The same logic applies when a model is making a multi-class prediction. The model predicts yes or no for an individual class, and it can be right or it can be wrong, and we decide how to combine these individual outcomes.
But how we may interpret a model goes beyond these four outcomes. For one, true positive (TP) involves two events: model prediction and the actual state. Our interpretation can change based on how we want to prioritize these two events. We can see whether the model predicted positive when the actual value was positive, or we can see whether the actual value was positive when the model predicted positive.
Let’s say when the weather is sunny, there is an 80% chance that the app would predict it to be sunny. But this does not tell us anything useful in this context. Instead, we should see how often the weather is sunny when the app says so. This will tell us how precise the app’s prediction is in predicting sunny days. This is the precision metric, representing the probability of the model being correct out of all the times model said yes.
from sklearn.metrics import precision_scoreprecision_score(actual_y, predicted_y)# scoring parameter: ‘precision’
But sometimes knowing the probability of positive prediction out of all positive cases might be more important. Imagine we are predicting whether a food is poisonous or not. The precision of a model (a chance of food being poisonous when it is predicted to be) is not as important as predicting food to be poisonous when it is poisonous. Because if the model says it’s poisonous we can simply not eat it, but if it fails to tell us that, we risk eating poisonous food. So we want a model that has a high probability of predicting positive when food is poisonous. This is a case for the recall or sensitivity metric. It’s also called the true positive rate. It represents how sensitive our model is in detecting positive cases.
from sklearn.metrics import recall_scorerecall_score(actual_y, predicted_y)# scoring parameter: ‘recall’
But what if both of these are important? In other words, what should we do when not missing positive cases and not missing negative cases are equally important? Imagine you are predicting a disease that can be cured by a certain treatment. But this treatment can be detrimental to those without the disease. In this case, we need a model that is sensitive to detecting positive cases and equally precise in its detection. That’s when the F1 Score comes into play. F1 Score is the harmonic mean of precision and recall, an average between precision and recall ratios.
from sklearn.metrics import f1_scoref1_score(actual_y, predicted_y)# scoring parameter: ‘f1’
Precision and recall metrics focus on predicting positive cases: saying yes to the question of ‘Is this ___?’. But as we saw earlier there are two different ways a model can be correct: it can say yes when it should say yes, and it can say no when it should say no. Accuracy is a probability of being correct in both of these cases. We can use accuracy when classes are equally balanced. For example, let’s say we want to predict whether a picture is a dog or not. We just want the model to say yes when it’s a dog, and no when it’s not. Saying that it’s a dog when it is a cat does not have any different consequences than saying that it’s not a dog when it is a dog. In this case, we can use accuracy.
from sklearn.metrics import accuracy_scoreaccuracy_score(actual_y, predicted_y)# scoring parameter: ‘accuracy’
Lastly, specificity is a sensitivity of predicting the negative cases (probability of predicting negatives out of all negative cases). In other words, we can use specificity when it’s more important to not miss negative cases than to be wrong. Let’s say you want to know if the water from a well is drinkable or not. You’d rather mark drinkable water not drinkable than to wrongfully mark undrinkable ‘drinkable’. If we swap our positive and negative and ask a question of ‘is this water contagious?’, we would be using sensitivity instead of specificity.
Specificity does not have a built-in function within the Sci-kit Learn package. Instead, you can either use sensitivity and swap positive and negative cases or calculate it through custom function using the confusion matrix.
# custom function to calculate specificity for binary classificationfrom sklearn.metrics import confusion_matrixdef specificity_score(y_true, y_predicted):    cm = confusion_matrix(y_true, y_predicted)    return cm[0, 0] / (cm[0, 0] + cm[0, 1])print(specificity_score(y_true, y_predicted))
We reviewed how to choose evaluation metrics for our binary classification data. But what do we do when our target is not yes or no but comprised of multiple categories? One way is to count each outcome globally regardless of within-class distribution and calculate the metric. We can accomplish this by using the micro average.
Let’s think about what it means to use global counts. If we just look at the global outcome, we don’t have the four outcomes as before (TP, TN, FP, FN). Instead we have cases that are true (prediction = actual class) or false (prediction != actual class). Therefore, the micro precision, micro recall, and accuracy all represent the probability of accurate prediction and are equal. Also since the F1 Score is the harmonic mean of precision and recall, the micro F1 Score is the same as other metrics.
# these are all the samerecall_score(actual_y, predicted_y, average = 'micro')precision_score(actual_y, predicted_y, average = 'micro')accuracy_score(actual_y, predicted_y)f1_score(actual_y, predicted_y, average = 'micro')# scoring parameter: 'f1_micro', 'recall_micro', 'precision_micro'
Counting global outcomes disregards the distribution of predictions within each class (it just counts how many got it right). This can be helpful when the dataset has classes that are highly imbalanced and when we do not care about controlling prediction errors in any specific class. But as we discussed above, many of our problems do involve having specific control over where prediction errors come from.
Another method to deal with multi-class is to simply calculate the binary measures for each class. For example, if our target variable can be either cat, dog, or bird, we get a binary yes or no answer for each prediction. Is this a cat? Is this a dog? Is this a bird? This will lead to as many scores as a number of our target classes. Then we can aggregate these scores and turn them into a single metric using a macro average or weighted average.
Macro average calculates metrics for individual class then computes the average of them. This means that it gives equal weight to the results returned from each class regardless of their overall size. So it is not sensitive to the size of each class, but it takes the performance of individual class more seriously even when it’s a minority. Therefore, the macro average is a good measure if predicting minority class well is as important as the overall accuracy and we also believe that there is a reliable amount of information in the minority class to represent the ground truth pattern accurately.
The macro average recall score is the same as the balanced accuracy in multi-class problems.
print(f1_score(actual_y, predicted_y, average = 'macro'))print(precision_score(actual_y, predicted_y, average = 'macro'))# below two are the same measuresfrom sklearn.metrics import balanced_accuracy_score, recall_scoreprint(recall_score(actual_y, predicted_y, average = 'macro'))print(balanced_accuracy_score(actual_y, predicted_y))# scoring parameter: 'f1_macro', 'recall_macro', ...etc.
To recap, the micro average gives equal weight to individual observations, and the macro average gives equal weight to individual classes. The macro average by definition does not care about how much data each class has. So even when the minority class does not have enough data to show a reliable pattern, it will still weigh the minority class equally. If this is the case, we might want to consider the overall size of each class by using a weighted average.
# we can also compute precision, recall, and f1 at the same timefrom sklearn.metrics import precision_recall_fscore_supportprecision_recall_fscore_support(actual_y, predicted_y, average = 'weighted')# scoring parameter: 'f1_weighted', 'recall_weighted', ...etc.
Each of the metrics we discussed so far individually tells a part of a story, and that was why we wanted to make sure our goal is clear and aligns with the metric we choose. But what if we just want a measure that can holistically tell us how our model classification is doing relative to a random chance? Cohen’s Kappa compares the classifier’s performance to that of a random classifier. Naturally, it takes the class imbalance into account as the random classifier will rely on the distribution of each class. Unlike other measures that are probabilities, Cohen’s Kappa ranges from -1 to 1.
from sklearn.metrics import cohen_kappa_scoreprint(cohen_kappa_score(actual_y, predicted_y))
We looked at how each evaluation metrics for binary and multi-class classification are uniquely utilized for individual problems, and why it’s important to care about metrics we choose. I tried to go over the most foundational measures in plain English. But I still left off another very important metric, ROC/AUC score, which I will discuss in another post sometime.