Hard ROC: Really Understanding & Using ROC and AUC | Towards Data Science

The ROC graphs generating code used in this post is available as part of the dython library, which can be found on my GitHub page. Examples seen in this post are also available as a notebook.
Assessing the predictions of any machine-learning model is probably the most important task of a Data Scientist — perhaps even more than actually developing the model. After all, while building super complex algorithms is the coolest thing, not knowing how to estimate their output properly is not the coolest thing.
There are several algorithms and tools dedicated to allowing a clearer view of how a model uses the inputs we supply it (like LIME and SHAP), but rarely do we discuss how the output of the model is analyzed — which is exactly what we’ll do today.
We are all familiar with the fact that we introduce biases in our input data, but when it comes to classification problems — we unknowingly introduce biases to the output too. This is due to the fact that models never really choose a class — they only output their probability assessment for each class. We then choose a class based on these probabilities, by selecting the one with the highest probability.
While this makes sense, thinking a bit more about this brings up a question — what if the prediction probabilities are noisy? Well, it’s actually quite reasonable that they are — which means we might want to adjust the probability-threshold from which we assign a class to a sample.
Let’s see an example: consider a binary classification problem, where the predicted probability of some sample to belong to the target-class is 0.53. Naively, we will decide to interpret this as the sample belongs to the target-class and output y = 1. But if we suspect the predictions are noisy, and 0.53 might be just as good as 0.48, we might decide to increase the threshold, and only output y = 1 to samples with predicted probability of 0.6 or higher. In this case, our sample will be assigned y = 0, which changes completely our overall predictions.
This means that the exact same model’s results can be interpreted in various different ways — it’s completely up to us to decide how. This means that in order to really understand how well a model performs, we must strip away this extra layer of our interpretation of the output. A very handy algorithm to do so is known as the Receiver Operating Characteristic Curve, or the ROC Curve.
ROC curves have been used in various fields for years, and in 1989 Kent Spackman has shown that Machine Learning can also benefit from it.
A ROC graph plots out the trade-off between true-positive results and false-positive results of a given class for any possible threshold. Let’s see an example so we can understand it better:
The y-axis displays the True Positive Rate (TPR) which is the number of true-positives (positive examples predicted correctly by the model) divided by the number of overall positives (TP/P). The x-axis displays the False Positive Rate (FPR) — false positives divided by the number of overall negatives (FP/N). Both are calculated for the case of y = 1, as the y = 0 class behaves the same.
Each point on the graph represents a possible threshold, and the associated TPR and FPR which follows it. When considering prediction probabilities, the threshold can vary between [0,1]. When the threshold is maximal, meaning 1, we label all samples as negatives. This means there are no true positives and no false positives — and this is represented by the starting point of the curve at (0,0). When the threshold is minimal, meaning 0, all samples are labeled as positives. This means that all positive samples were indeed labeled as positives, but also all negative samples were labeled as incorrectly as positives too. This matches the point at the end of the graph, (1,1). All ROC curves begin at (0,0) and end at (1,1). The ideal model will also go through (0,1) — meaning that some threshold exists where all positives are labeled as positives, but none of the negatives were labeled inaccurately too. In a case like this, using the threshold that matches (0,1) will yield flawless predictions.
But unfortunately, models tend not to be ideal — so we would like to measure how close is a model to the ideal one. As mentioned previously, both axes are bound in the range [0,1], and all curves begin at (0,0) and end at (1,1). This means we can simply measure the Area under the curve — the AUC. The optimal model will have AUC = 1.
We can learn even more insights about our model from the ROC graph. Note the dashed line going from (0,0) to (1,1), which represents TPR = FPR. Any point on this line means that a sample has a 50% chance of being classified correctly as a positive, and 50% chance of being incorrectly classified so. This line represents prediction by random, and matches AUC = 0.5. Models with a curve near that line are significantly underperforming, and no threshold selection will improve that.
And what if your model’s curve goes below the dashed line with AUC < 0.5? Well, this means that for some reason, your model substitutes your classes — and predicts positive instead of negative and vice versa. Double-check yourself if this ever happens.
So far we discussed what we can learn about our model possible performance from a ROC graph, but what we really care about is how can we improve our model using a ROC graph?
To improve our accuracy, we can use the ROC curve to try and find an estimated-optimal threshold (eOpT). My favorite approach is to find which point on the curve is the closest one to the ideal point (0,1), and use its corresponding threshold.
In the example displayed above, the accuracy of the model improved by 1% by simply adjusting the prediction-threshold to 0.71 instead of the default 0.5. See for yourself. Quite simple, isn’t it?
So far we’ve discussed only binary classification, where only one class exists (or two, but that’s the same thing). What happens when we face a multi-class classification problem?
When facing many classes, we compute a ROC curve for each class by itself, in a one-vs-all manner: all other classes are considered as negative examples, and only the examined class is considered positive. This yields a different ROC curve per each class. In addition, it’s useful to compute the micro and macro ROC curves of the entire classifier, just to see its overall performance (not sure what micro and macro averages are? Here’s a great explanation on StackExchange’s Data Science forum).
Let’s see how that looks like:
Adjusting the thresholds in this case gets a little bit more tricky. Here’s how I handled it:
if prediction_probability[2] >= 0.53: predicted_class = 2elif prediction_probability[0] >= 0.46: predicted_class = 0elif prediction_probability[1] >= 0.27: predicted_class = 1else: predicted_class = prediction_probability.argmax() # fallback
It gets tricky because some examples might not pass any of these thresholds (for example: [0.34, 0.16, 0.50], which why a fallback to selecting the highest one is necessary), and some might pass more than one threshold (for example: [0.49, 0.50, 0.01], which means that the if/else order matters). But once you get it right, accuracy is improved — my SVM classifier’s accuracy increased by 2.2% after readjusting the thresholds in this way.
In this post we’ve discussed a bias we introduce to our classification models from the output end, by naively interpreting predictions as labels without considering their possible noise. We’ve seen how the ROC curve can allow us to better estimate our models, stripped from the extra bias we add, and how we can fix that bias to better fit the model.
If you’d like to broaden your knowledge and understanding of ROC curves, I can highly recommend reading the paper An Introduction to ROC Analysis by Tom Fawcett. I can only hope this new tool will make your models perform better — and that you enjoyed this post. Keep safe and ROC hard!