On ROC and Precision-Recall curves | by Carlos Azevedo | Towards Data Science

In machine learning, when facing binary classification problems, there are two main metric tools that every data scientist uses: Receiver Operating Characteristic (ROC) curve and Precision-Recall (PR) curve.
The main goal of this article is to cover how to interpret these curves along with their inherent confusion matrices and thresholds.
We start by covering the raw data for these curves: the confusion matrix. Then we go through the actual curves and to finish, we show how these curves could look like when the problem is not properly set up.
The Confusion Matrix is the raw data for everything we will be showing here: it’s a table that contains the number of True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN). The advantage of using ROC and PR curves is that they summarise performance information that is relevant to binary classification problems. It makes it easier to read and to interpret compared to having several confusion matrices — one for every threshold — and then compute some ratios to have glimpse of what those numbers (TP, TN, FP, FN) mean. And voilá, we just defined what ROC and PR curves represent — ratios of numbers we get from several confusion matrices. The Confusion Matrix supports several classes but in this article, we will focus on the binary classification problem.
                  | predicted negative  | predicted positive  ||-----------------|---------------------|---------------------|| actual negative | True Negative  (TN) | False Positive (FP) || actual positive | False Negative (FN) | True Positive  (TP) |True Positives (TP): Positive samples predicted as positiveTrue Negatives (TN): Negative samples predicted as negativeFalse Positives (FP): Negative samples predicted as positiveFalse Negatives (FN): Positive samples predicted as negative
The Confusion Matrix as it is is perfectly fine when dealing with balanced data sets. It’s when we are working with imbalanced data sets that we need a more sophisticated evaluation metric that allows us to reason clearly about the trade-offs.
Wikipedia has a great page on all the metrics regarding binary classification. I find it confusing that there are several names for the same metric. Here I’ll list mainly the ones that matter for this article with the variations that are most commonly used:
Negatives (N): Total number of negative samples.N = FP + TNTrue Positive Rate (TPR): ratio of correct positive predictions to the overral number of positive examples in the dataset.TPR = Recall = Sensitivity = TP / PFalse Positive Rate (FPR): ratio of correct negative predictions to the overral number of negative samples in the dataset.FPR = 1 - Specificity = FP / NPrecision: From the positive predictions what proportion of it is correct.Precision = TP / (TP + FP)Important: Note that the y-axis of the ROC curve (Sensitivity) is the same as the x-axis of the PR curve (Recall).
Left plot: Along the y-axis, we have the score while the x-axis just helps with the visualization by making it more spread so we can easily see the score data points. There are 5 different horizontal lines that represent different thresholds: below this line, all the data points are classified as negative (class 0) and if the data point is above the line or at the same score it is classified as positive (class 1). Each of these lines has a different color in order to map them into the other two plots: ROC and PR curves.
Top right plot: This is the ROC curve. The x-axis is the FPR and the y-axis is the Sensitivity (or Recall or TPR). As the threshold on the left plot goes from the bottom to top the ROC curve develops from right to left. Each data point in this plot represents a different threshold that at each step turns a positive prediction into a negative prediction.The goal in ROC space is to be in the upper-left-hand corner — that is the point (0, 1) meaning that we have zero False Positives (FPR=0) and that we classified all positives as positive (Recall=1). The diagonal dashed line (the “curve” for a no-skill classifier2) is what we have, on average, when drawing randomly scores on the unit interval3 or if our predictions are all 0 or all 1.
Bottom right plot: This is the PR curve. Both the x-axis of the PR curve and the y-axis of the ROC curve stand for Recall so we see that they develop similarly. This curve focuses mainly on the performance of the positive class which is crucial when dealing with imbalanced classes. In the PR space, the goal is to be in the upper-right-hand corner — the top right corner (1, 1) means that we classified all positives as positive (Recall=1) and that everything we are classifying as positive is true positive (Precision=1) — the latter translates to zero False Positives. In this case, the “curve” for a no-skill classifier2 is a horizontal line obtained by P/(N+P) which is represented by the dashed line — it the case where the classifier always predicts the minority class, i.e. positive.
Each point in the ROC space or PR space represents a specific threshold that changes the confusion matrix (TP, TN, FP, FN) and thus the Recall, Precision, and FPR.
There is generally a trade-off between precision and recall, which is determined by the threshold that separates the positive cases from the negative cases — this trade-off is generally discussed with the business decision-makers to make a final decision on where it should sit.
Another dummy example follows where we are not able to get a perfect separation through thresholding:
Lets put into practice what we just learned and look at the following animation that will serve as the template for what’s coming next:
Notice how the threshold always jumps to the next sample: this is because in between samples the metrics are kept the same — so the lines connecting the points in the ROC and PR curves are meaningless.
The distance in between points in the ROC curve is constant: it only depends on the total number of samples — one dot per sample as long as no score is exactly the same, then moving the threshold changes the decision for two data points.
The distance in between points in the PR curve changes with the direction: if the step is moving horizontally (Recall) then it is constant as it is with the ROC curve. With vertical steps (Precision) it depends on how many negative samples there are — high number of False Positives (FP) leads to smaller steps — Precision = TP / (TP + FP).
By design the ROC curve always develops in a U-shape unless there’s something wrong — so regarding the animation above, as the threshold goes up, there are only two options: step to the left or downwards.
Step left: Turn a False Positive (blue cross) into a True Negative (blue dot). This behavior represents a correct decision and thus it reduces the False Positive Rate (x-axis). It is easily observed when the threshold is at the bottom moving from -1.5 up to -0.5 where every transition reduces the number False Positives successively.Step downwards: Turn a True Positive (orange dot) into a False Negative (blue cross). This behavior represents a wrong decision and thus it reduces the Recall (y-axis). It is easily observed when the threshold is at 0.5 up to 1.5 where every transition is wrongly classifying positive samples as negative.
Step left: Turn a False Positive (blue cross) into a True Negative (blue dot). This behavior represents a correct decision and thus it reduces the False Positive Rate (x-axis). It is easily observed when the threshold is at the bottom moving from -1.5 up to -0.5 where every transition reduces the number False Positives successively.
Step downwards: Turn a True Positive (orange dot) into a False Negative (blue cross). This behavior represents a wrong decision and thus it reduces the Recall (y-axis). It is easily observed when the threshold is at 0.5 up to 1.5 where every transition is wrongly classifying positive samples as negative.
So far we looked into two examples that have one thing in common: both data sets were perfectly balanced — the same number of positive and negative samples. Where the PR curve shines compared with the ROC curve is when we are dealing with imbalanced datasets and we are interested in the minority class. So let us look at an example.
Judging by the ROC curve we could be overly optimistic about our classifier — there are several thresholds we can pick from that have performance. Nevertheless, the PR curve tells us a different story — as does the plot with the data points. With large skew in the class distribution, PR curves are more insightful regarding an algorithm’s performance1: when the number of negative samples greatly exceeds the number of positive samples, a large change in the number of false positives can lead to a small change in the false positive rate used in ROC analysis. Precision, on the other hand, by comparing false positives to true positives rather than true negatives, captures the effect of a large number of negative samples on the algorithm’s performance.
The way we decide on which curve we want to optimize is context-dependant. If we are dealing with fraud detection, then optimizing for the PR curve gives the most benefit by flagging fraud without overloading the checks for false positives, whereas if we are dealing with let’s say cancer detection, there’s a high cost on false negatives and thus optimizing over the ROC curve is preferable — Nevertheless, it requires business expertise in order to decide on where should the algorithm stand in the trade-off.
Here we will show two other scenarios that are not seen that often, the reason being that it means the problem was wrongly formulated — so we will still see it when we do something wrong and it is a good reminder of what it should not look like.
Classes are set the other way around: what is positive should be negative and vice-versa. This is easily identified by the ROC curve — whenever the curve is on the wrong side of the diagonal it means we got the labels wrong.
The minority class is wrongly set as the negative class. By definition the class of interest, what we call the positive class, is the minority class. This is easily identified by the PR curve where we see that at Recall=1 the precision is 0.9 which tells us the data class imbalance is towards class 1.
And here is what it looks like when we get both wrong: the labels and the minority class. Here both curves tell us that there’s something odd — both scenarios above show up.
I hope this post made you more familiar with both the ROC curve and the PR curve in a way that it is possible to infer knowledge about the data and the classifier itself. Let me know if you feel that something is missing and that it would make sense to add to this post. I’m always open to explore the foundations in detail.
Here’s the github repo used to create the content for this post.
References
[1] Davis, Jesse, and Mark Goadrich. “The relationship between Precision-Recall and ROC curves.” Proceedings of the 23rd international conference on Machine learning. 2006.