Performance metrics for binary classifier (in simple words) | by Irene P | Towards Data Science

Say we have a simple binary classifier which accepts boxes with Schrodinger’s cats 😺 as the input and we expect the classifier to return label 1 (positive) if the cat is alive and 0 (negative) if not, but errors occur from time to time.
The reasons may be different, for example — poor quality of input data, or wrong features selected. E.g. if it’s a deep sleeping cat so it looks quite dead, the classifier can easily return 0 for it.
The figure below shows 4 possible cases, in two of them the classifier gives correct predictions, in the other two — it’s wrong:
Thus, if our null hypothesis is that ALL cats are alive (belongs to class 1), then the possible errors are:
null hypothesis was wrongly rejected
dead cat was classified as alive
null hypothesis was wrongly accepted
alive cat was classified as dead
Let’s complete the pic above with numbers: how many samples were labeled correctly, and how many times the classifier was wrong. That’s how we get the confusion matrix for binary classification.
Here is an example of a matrix constructed using the Python scikit-learn:
from sklearn.metrics import confusion_matriximport pandas as pdn = confusion_matrix(test_labels, predictions)plot_confusion_matrix(n, classes = ['Dead cat', 'Alive cat'], 											title = 'Confusion Matrix');
In this case we have:
true negatives = 616 (dead cats labeled as dead)
false negatives = 59 (alive cats labeled as dead)
true positives = 71 (alive cats labeled as alive)
false positives = 89 (dead cats labeled as alive)
All that values we can get using ravel() from numpy:
tn, fp, fn, tp = confusion_matrix(test_labels, predictions).ravel()
So, now we know how many times the model did right and wrong predictions, and the next step is to evaluate the classifier — to check if the predictions made by it were good enough. Let’s see what metrics do we have for that.
First we’ll try to use Accuracy Score for evaluation: to divide number of correct predictions by total number of predictions and get the percentage of samples were predicted correctly:
When the data is imbalanced and there are significantly more alive cats than dead (or vice versa). Let’s figure out why.
We want to find all the alive cats asap before they join the other 95, but our classifier had lost all hope and became pessimistic: now it always says the cat in the box is dead (even if it’s not).
Then it will be right 95 times (TN = 95), and wrong 5 times (FN = 5). The other values are: TP = FP = 0. Let’s calculate the classifier’s accuracy:
Looks like the classifier has great accuracy — 95%, but it do absolutely nothing.
The reason is the large number of “guessed” dead cats makes a big contribution to the accuracy score and suppress errors.
So it turns out that in order to get a high accuracy, classifier can always give a label of the predominant class. But it doesn’t fit with the problem’s solution logic: high accuracy didn’t help us to find a single box with alive cat.
Looks like we need to move from calculating the common metric for all classes to separate performance metrics for each class.
To calculate True Positive Rate (aka Recall, Sensitivity, and Hit Rate) we compare the number of correct predictions that the cat is alive with the total number of boxes with alive cats:
True Negative Rate (aka Specificity or Selectivity): here we compare the number of boxes with dead cats and the number of correct predictions that the cat is dead:
Let’s calculate both measures for our pessimistic classifier and imbalanced data (95 alive and 5 dead cats):
Well, this time the model failed to trick us: although TNR is perfect (1), but TPR is 0.
This clearly illustrates the essence of this model: it’s pretty good at finding dead cats, but terrible at finding alive ones.
All this is taken into account in balanced accuracy metric, which is an analog of accuracy, but can be used for imbalanced classes.
It is calculated as the average of sensitivity and specificity, i.e. the average of the proportion corrects of each class individually:
When all classes are balanced, so there are the same number of samples in each class, TP + FN ≈ TN + FP and binary classifier’s “regular” Accuracy is approximately equal to Balanced Accuracy.
In our case with imbalanced cats (95 dead and 5 alive):
Here we got 50% in contrast to the Accuracy score which was 95%. And, until our model guesses at least one alive cat, it won’t get a BA score higher than 50% — that’s life (and the balanced accuracy formula).
Looks like that’s all done — we can just always use balanced accuracy as it’s immune to class imbalance. But here’s another example.
Let’s say now our classifier said that there are 10 alive cats among the boxes, but... none of them were actually alive.
That is what we have then:
none of 5 alive cats were found (TP = 0), all of them were labeled as dead (FN = 5)
of the 95 dead cats, 85 were correctly classified (TN = 85) and 10 were mistaken for alive (FP = 10)
And balanced accuracy decreased from 0.5 to 0.45:
On the one hand, this makes sense, since now there are more errors (the model identified dead cats unmistakably before). On the other hand, as we’re trying to find alive cats, both models are equally useless: neither of them found a single alive cat, but metrics says that the first one is better. So what should we trust?
The point is, balanced accuracy metric gives only half its weight to how many alive cats we classified correctly. The other half is given to how many dead cats we labeled correctly, while for our task this information is useless.
An important role here is played by the task we use the classifier for. In example described above, it’s in our best interest to detect a greatest number of cats alive, but whether the model can identify dead cats doesn’t matter much (anyway, it’s too late to try to rescue them). But if one day it becomes as important to find dead cats as alive ones, we can safely use the Balanced Accuracy.
But for now, we are still focused on detecting alive cats and continue to look for a metric that can help us evaluate our model.
Let’s assume that we retrained our model and it finally predicted 5 alive cats out of 5! But we still have a few mistakes — 5 dead cats were labeled as alive too.
In other words: FP = 5, TP = 5, TN = 90, FN = 0.
Then what is model’s Recall (True Positive Rate)?
1 or 100% — awesome!
But what about those cats called alive but were dead? We all know that the classifier was wrong as many times as it was right. And, obviously, there should be a metric that reflects it.
There is such a metric — Precision:
It shows the ability of our model to differ the “alive cat” class from all the others, but, unfortunately, it does not give an idea of whether we have found all the alive cats or not.
In contrast, Recall metric shows whether we found all the alive cats, but it can’t say anything about how many times we had to open boxes with dead cats throughout these searches.
Well, let’s calculate Precision for our last experiment and then interpret the results:
So:
Recall = 1 means that no matter how many alive cats are there in total, we’ll find them all.
Precision = 0.5 means that opening a box with label “alive”, in half of the cases we’ll find a dead cat there.
(I’ll remind we still have 95 dead ☠️ and 5 alive 😸 cats)
Separately, Precision and Recall make little sense:
a model that always classifies all cats as alive (optimistic) has a perfect Recall (100%), but very low Precision (5%).
a model that deems alive only those cats about which it’s absolutely sure (pessimistic), has 100% Precision. But if there are a lot of cats sleeping like a dog, Recall will be very low.
We need to balance both metrics to get a full picture and answers to both questions:1. how many “alive” predictions were correct (Precision)2. what part of alive cats was classified correctly (Recall)
And that’s where F1 metric comes in handy.
F1 is the harmonic mean of Precision and Recall, so you can’t get a high F1 if either one is very low.
Let’s recall how our model was training and calculate F1 score for all of 3 cases:
At first, the model labeled all cats as dead:TP = FP = 0, TN = 95, FN = 5Then it labeled 10 cats as alive, but there was not a single alive cat among them:TP = 0, FP = 10, FN = 5, TN = 85Finally, the classifier predicted correctly 5 out of 5 alive cats, but also mistakenly labeled 5 dead cats as alive:TP = 5, FP = 5, FN = 0, TN = 90
At first, the model labeled all cats as dead:TP = FP = 0, TN = 95, FN = 5
Then it labeled 10 cats as alive, but there was not a single alive cat among them:TP = 0, FP = 10, FN = 5, TN = 85
Finally, the classifier predicted correctly 5 out of 5 alive cats, but also mistakenly labeled 5 dead cats as alive:TP = 5, FP = 5, FN = 0, TN = 90
You can see that with each retraining the “quality” of negative predictions decreases and positive — rises, so we expect F1 to reflect this changes:
Well, that makes sense. Cases 1 and 2 are essentially the same for us — despite the different behaviour of the model, we didn’t find any alive cats. But the third time the classifier labeled all the alive cats correctly — and F1 increased, along with our hope of saving the cats found 😸.
The thing is, unlike Balanced Accuracy, the F1 doesn’t take into account True Negatives — it’s easy to see if we expand the formula:
This means it doesn’t matter how many dead cats there were in dataset in total nor how many of them we successfully found. We focus on detecting the alive ones only as they are more important.
Let’s compare that with the Balanced Accuracy behaviour:
So if we care more about saving alive cats (detecting positives), metric F1 fits better.
F1 only cares about the samples the model said are positive, and about the samples that actually are positive and doesn’t care at all about how many negative samples we have in dataset or how many were classified correctly.
That’s why this metric is quite popular when evaluating models aimed at finding anomalies.
And on the contrary, we should prefer Balanced Accuracy more if negative samples are almost as important as positives.
For multiclass classification, if one class contains much more samples than the others, its metrics will suppress all the others. Then you can form its own contingency matrix and calculate its own F1 value for each class, and the resulting F1 shall be simple (i.e. not weighted) arithmetic mean for all classes.
the fraction of correct predictions
depends on the balance between classes, is not applicable to imbalanced datasets
for balanced datasets, is equal to Balanced Accuracy
the average of Sensitivity and Specificity
immune to class imbalance, can be applied to imbalanced datasets
for balanced datasets, is equal to Accuracy
the fraction of true positive predictions from all positive predictions
shows if the classifier is able to differ one class from all others
immune to class imbalance, can be applied to imbalanced datasets
should be balanced with Recall
the fraction of true positive predictions from all positive samples in dataset
shows if the classifier is able to detect a giving class at all
immune to class imbalance, can be applied to imbalanced datasets
should be balanced with Precision
harmonic mean of Precision and Recall
doesn’t take into account True Negatives
immune to class imbalance, can be applied to imbalanced datasets
All the metrics mentioned above can be found in scikit-learn library:
from sklearn.metrics import accuracy_scorefrom sklearn.metrics import balanced_accuracy_scorefrom sklearn.metrics import recall_scorefrom sklearn.metrics import precision_scorefrom sklearn.metrics import f1_score
I skipped a number of metrics here, which are also ok to be used if you have imbalanced dataset, like Precision–Recall curve, or Kappa score, or AUC ROС, which is immune to class imbalance too... Or it is not?
Let’s figure it out in the next part :)
* All images by Author