Mechanics of the ROC Curve
By: Sahil Gupta
Tags: Machine Learning, Data Science, Towards Data Science, Statistics, Classification

The accuracy of a classifier is defined as the number of correct classifications out of all the classification performed by it. In technical terms, whenever a classified makes a classification, it can land in either of the four boxes in Figure 2. For example, this is how we would read the table for the cell “FP”: for a particular classification, we saw a label of 0, but the classifier predicts it as 1, i.e. a person who doesn't have diabetes is predicted to have diabetes by the predictive model, which explains why it is called a False Positive (FP). Accuracy is defined as,

It’s very easy to build a classifier that would be fairly accurate but completely useless. For example, we can build a random classifier using the class frequencies from our dataset. The proportion of True Negative (Not Diabetes) is 500 out of 768. At prediction time, this random classifier predicts a person as not having diabetes with a probability of 500/768, or 0.651. So, the accuracy of this classifier is 65.1%! But, we know this is completely useless in real-world applications. Clearly, there is something missing in our evaluation of a predictive model in terms of accuracy. Intuitively, accuracy is focusing on the actual truth values the model is able to capture (i.e. TP and TN). However, in Figure 1, we saw that counts are skewed. So, it would make sense to also account for the false predictions made by the model (i.e. FP and FN), because the real-life cost of missing a rarer event can be much more (which probably goes further up as the skewness in counts increase).

Until this point, we have convinced ourselves that the usual way of evaluating predictive classification models (i.e. based on accuracy) doesn’t seem to work with imbalanced datasets. The ROC curve is one tool to help us with this.

The ROC Curve

ROC curve, as the name suggests, is a graph between True Positive Rate (TPR) and False Positive Rate (FPR). These are defined as,

Let’s try to understand these terms and then see why the ROC curve might be a good choice. Let’s start with the denominators. Notice that the denominators in TPR and FPR are actual positives and actual negatives, respectively (positive refers to label 1 and negative to label 0, similar to Figure 2). Based on Figure 2 (above), for instance, we know that a diabetic person can be predicted as diabetic (TP) or not diabetic (FN) by the model. So, TPR indicates the proportion of true positives (predicted by the model) among all the positive labels. Similarly, FPR indicates the proportion of false positives (predicted by the model) among all the negative labels. The crucial thing to note here is that these two quantities are looking at the predictions of a classifier at a more granular level. We can also interpret the complement of both these quantities in a similar fashion. The complement of TPR would be FNR (false-negative rate), and for FPR it would be TNR (true negative rate). TPR is also called Sensitivity, and 1-FPR is called Selectivity. So, we can also say that the ROC curve is a graph of Sensitivity v/s 1-Selectivity.

As a side note, Sensitivity (TPR) and Selectivity (1-FPR) can also be interpreted as conditional probabilities. For instance, Sensitivity is the probability of classifying a data point (with a positive label in the PIMA dataset) as positive (notice that denominator, TP+FN, are the data points with true labels as 1, and TP are the actual positives in the data set). In mathematical terms, this is how sensitivity and selectivity are defined,

This is exactly a conditional probability description. Here, sₒ is a threshold, which is used as follows: if the score of our classifier is above the threshold (sₒ), then it is classified as a positive label (1), else negative label (0). The terms sensitivity and specificity are widely used in medicine and thinking about the definitions in terms of diseases make them a bit more intuitive: with the PIMA diabetes dataset, our aim is to build classification models that can identify diabetic vs non-diabetic patients. So, sensitivity defines the ability of our model (/test) to correctly identify diabetic patients. Similarly, specificity defines how specific our model (/test) is in terms of detecting non-diabetic patients (usually, in case of diseases, the number of patients who don’t have the disease is much more vs the patients who do).

Now, let’s observe the plane where the ROC curve lives and gain some intuition. The dotted line, where TPR = FPR, represents a random classifier.

Figure 3. The plane where the ROC curve lies. Notice that by definition FPR, TPR ∈ [0,1].

The reason is that a random classifier based on prior (from train set) class frequencies would on average get similar conditional frequencies, and hence equal TPR and FPR. Let’s suppose the prior class proportions are α and 1- α for positive and negative labels, respectively. A random classifier based on this would on average predict α fraction of samples as a diabetic. This also means that in a sample of all diabetic patients, it would on average predict α fraction of patients as diabetic, which says that the sensitivity is α. Using a similar argument, we can say that the selectivity is 1-α, which implies that FPR is α. Hence, the line TPR=FPR indeed represents a random classifier.

We can gain some further intuition by looking at the extreme points in the curve. The lower left green point (FPR=0 and TPR=0) represents a classifier that has 0 sensitivity and 1 selectivity, i.e. it is extremely good at identifying the non-diabetic patients and extremely bad at identifying diabetic patients. On the other hand, the top right corner (FPR=1 and TPR=1) represents a classifier really good at identifying diabetic patients, but bad at identifying non-diabetic patients. In other words, the classifier is super-sensitive at the cost of being non-selective. Finally, the top left corner (FPR=0 and TPR=1) represents a classifier which is ideal: it has perfect sensitivity (=1) and perfect selectivity (=1).

In a similar way, we can compare the two classifiers marked as red crosses. The left one has better selectivity and worse sensitivity compared to the right one. In general, our aim would be to build a classifier that is as close to the top left corner as possible.

The ROC Curve dashboard

Now, we are in a position to look at an actual ROC curve. Just to clear up any confusion about the score thing I mentioned in the definition of sensitivity and selectivity, these can be thought of values before we assign a label to the prediction. For example, in a logistic regression model, it’s the value of the log-odds for the data point. Commonly, an implicit threshold of 0.5 (for probability) is used (in our heads, or on paper). In the context of log-odds, this basically means that we are selecting the more probable event (remember, log-odds is the log ratio of odds of an event). Like we saw earlier, sometimes this threshold of 0.5 is not a good idea (accuracy), and we need a better threshold. The ROC Curve helps us with exactly that. Without further ado, here’s the dashboard you can play with and make these things a bit more intuitive.

You can find the Jupyter notebook for this dashboard here. The dashboard pops up once you run all the cells in the notebook. The dashboard uses Plotly and Python widgets, which makes it a bit tricky to embed it directly on medium.

Before leaving you with the dashboard, let me say a few words about how to interpret the dashboard. At the top, you can find two threshold sliders that control the thresholds (as defined in the sensitivity and selectivity definition) for Logistic Regression and Random Forest. The top-left plot is the logistic distribution of scores generated using the logistic regression model. I have included this in the dashboard to make things explicit. However, in the case of Random Forest, the functional form of distribution is not known, which is why the distribution in the case of random forest is omitted. Next, to the right of that, we have a plot of ROC curves for both models and markers which indicate the current values for the thresholds. As the sliders are moved, the markers travel on there respective ROC curves.

Further below, we have the conditional probability distribution for both models, i.e. conditional on knowing the predicted labels by the classifier. For instance, probability conditional on Pred=1 defines the probability that a person is diabetic (TP) or not (FP), given that the classifier predicted a person as a diabetic. In other words, the two plots at the bottom give us a complementary view of what happens when the threshold is varied. Notice that sensitivity (and selectivity) are conditional probabilities conditioned on knowing the true label of the data point is C=1 (or C=0).

An interesting thing that you can notice here is that for a particular model (and hyperparameter values), the ROC curve is fixed, and changing the threshold is helping us tune the model further to align better with the business objective at hand (think sensitivity and selectivity). Usually, the threshold values are selected using cross-validation.