Intuitively understand ROC and implement it in R and Python
By: Ahmar Shah, Phd, Oxford
Tags: Getting Started, Editors Pick, Machine Learning, Model Evaluation, Data Science

Intuitively understand ROC and implement it in R and Python

The field of machine learning can broadly be categorised into supervised learning, unsupervised learning, and reinforcement learning. Supervised learning uses previous examples with known outputs to determine an appropriate mathematical function to solve a classification or a regression problem. This post focusses on ROC (Receiver Operating Characteristics) curve that is widely used in the machine learning community to assess the performance of a classification algorithm. This post will help you intuitively understand what an ROC curve is and help you implement it in both R and Python. Specifically, the objectives of this post are:

(i) To provide you with an intuitive sense of what an ROC curve means and how to interpret it

(ii) To help you understand how to choose the threshold for ROC

(iii) To help you understand related advanced topics (multiclass classification, ROC-associated terms such as precision, recall, accuracy etc.)

(iv) To help you implement ROC curve analysis in R and Python

This article is divided into four parts, each dealing with an objective stated above. I recommend reading in sequence (so if you are interested in understanding how to set the threshold, I would recommend reading Part 1 first so that you understand the terminology used).

Part 1: Intuitively understand what an ROC curve is

The scenario (a Covid-19 case study): To set the stage, let us think of a case study related to the current Covid-19 pandemic. A major challenge faced by countries around the world is testing people to check if they have Covid-19. The current testing method (PCR) is slow, expensive and it has been a challenge to scale it up for many countries due to capacity constraints. Imagine that you are the CEO of a new company who has invented a fast, portable, easy to use, and cheap device that can determine if a person has Covid-19. Before the device can be rolled out, you need to show how well the device will work in practice and how it compares with other potential competitors. ROC is a great tool that can help you assess how well your device is doing and how it would compare with other devices. Let us imagine that the device outputs a probability of someone getting seriously sick after getting Covid-19. Let’s also suppose that you have 100 patients on whom you want to assess this device, 50 of them are Covid-19 positive (called the positive group henceforth) and 50 of them do not have the disease (called the negative group henceforth). If the device you built was perfect, then it would output a probability of 1 for anyone who was positive(1 means that you are certain of an event happening), and it would output a probability of 0 for anyone who was negative (in probability, 0 means that you are certain that the event did not happen). Graphically, it would look like shown in Figure 1.

Image by author (Figure 1)

However, like most things in life, most cases would have a certain level of ambiguity and the device is likely to output a score which looks like a distribution below (Figure 2).

Image by author (Figure 2)

Now where would you set the threshold to decide whether someone is Covid-19 positive or not? This is where ROC comes into play. A naïve assumption might be to set the probability to 0.5 which will almost never be the case in real-life and you would instead carry out actual studies to determine the appropriate threshold. So how does thresholding work? You set a threshold, and anything below threshold is considered to not have disease (negative) and anything above threshold is considered to have the disease (positive). Let us start by thinking about two extreme examples:

Put threshold to left-most extreme (that means you say that everyone is Covid-19 positive). If you do that, you will correctly identify all the 50 subjects in the positive group (correctly consider them to have the disease), but you will also incorrectly classify all those in the negative group (incorrectly consider them to have the disease). This situation is depicted graphically below (Figure 3).

Image by author (Figure 3)

Let us think of the other extreme case, where you set the threshold at 1. Now, you will say that nobody is Covid-19 positive and hence you will correctly identify the 50 subjects in the negative group to not have the disease but simultaneously, incorrectly classify all the 50 with in the positive group as not having Covid-19. This situation is depicted graphically below (Figure 4).

Image by author (Figure 4)

Let us define two terms before going further: Sensitivity and Specificity. Sensitivity refers to the proportion of people with disease (positive cases) that your model can correctly classify. Specificity refers to the proportion of people without the disease (negative cases) that your model can correctly classify as not having the disease. Knowing these terms, you can immediately visualize that as you increase your threshold (from 0 to 1), the sensitivity will decrease from 100% to 0%. Similarly, specificity will go from 100% to 0% as the threshold is increased.

Image by author (Figure 5)

As you can see, there is a trade-off. If you increase sensitivity, it will decrease specificity and vice versa. ROC is a tool that combines the two graphs together where you typically plot sensitivity vs (1-specificity), and each point on the ROC refers to a different combination of sensitivity/specificity associated with a specific threshold. (You cannot see the threshold set in an ROC curve, but it is implicit that each point on the ROC refers to choosing a different threshold).

How would you calculate sensitivity and specificity for a given threshold? It should be clear from the definition of the two terms but a good way to visualize this is through a “confusion matrix”. A confusion matrix is a 2x2 grid with actual cases on one grid, and predicted cases on the other grid (see Figure 6 below).

Image by author (Figure 6)

Of all the people in the healthy group (the negative class), the ones that are classified correctly are called the “True Negative”. And those that are incorrectly classified as “positive” are called False Positive. Specificity is therefore (you do not need to ever memorize this, if you understand the concept, you will simply work it out):

Specificity = TN / (TN + FP)

Similarly, of all the people in the disease group (the positive group), the ones that are classified correctly are called “True Positive” and those that are incorrectly classified as “negative” are called “False Negative”. Sensitivity is therefore:

Sensitivity = TP/(TP+FN)

An ideal test will give you the perfect separation between the scores associated with the negative cases and positive cases and the ROC will look like below (Figure 7).

Image by author (Figure 7)

A test that is not good at all will have both distributions completely overlapping and the ROC will look like the following (Figure 8).

Image by author (Figure 8)

For anything that is between the two extremes (and the most realistic scenario), the ROC will be somewhere between the two. It should be intuitively clear that the closer an ROC is to the ideal, the better the test (and similarly, the closer an ROC graph is to the random classifier, the worse the model). It should therefore also be clear that when you are comparing two classifiers, the one that is closer to the ideal scenario would be better. The way that we quantify this performance of any ROC is (very intuitively) the area under the curve. For a random classifier, it is just half of the total possible area (so it would be 0.5). For an ideal classifier, it is 1. And for any realistic classifier, it will be between 0.5 and 1.

Image by author (Figure 9)

Part 2: How do you choose the threshold for a classifier model?

A valid question that is often asked and the answer is that it is context dependent. The naïve approach is to select the threshold that corresponds to the point closest to the top left point on an ROC. Conceptually, that is assuming that the cost of making either kind of mistake (FN or FP) is equal. However, in real-life problems, the cost of making one mistake might be more profound than the other and it is context dependent. For example, imagine you have a classifier that assesses if a jet engine malfunctions (you really do not want to have a jet with a malfunctioning engine to fly, ever!). In this case, you would absolutely always want to have a 100% sensitivity so that you do not let a plane take off at the airport if there is a chance that the engine is malfunctioning. On the other hand, the cost of incorrectly classifying as engine malfunctioning means a needless inspection of airplane engine which is, again a cost but much less than letting a plane fly with malfunctioning engine.

Image by author (Figure 10)

Imagine a different scenario where it is less crucial to have 100% sensitivity. Suppose you own a factory where your product is automatically packaged in a box and you have a computer vision system that inspects every box packaged on a conveyor belt to ensure that boxes are correctly packaged. Sometimes the automatic packaging system is not able to package the box correctly. In those cases, you want to move the boxes aside and repackage them manually. However, you can only manually re-package a limited number of boxes and most customers might still buy a box despite imperfections in the packaging. In this case, you might choose a different threshold on the ROC that balances both sensitivity and specificity.

Image by author (Figure 11)

The bottom line is that choosing a threshold for an ROC is context dependent, and it is a decision that will be influenced by several factors, all dependent on the situation in which it is to be used.

Part 3: Advanced Topics in ROC

Before we go to code and show how to implement an ROC for a binary classifier, there are a few other topics that are worth clarifying.

Multi-class ROC: ROC analysis is widely used for a two-class classification and historically it was used to detect signal from noise in radar applications (that is how it got its name). However, there are situations where we could have more than two classes. In such cases, one approach that could be used is a one-vs-all approach. In this case, you develop several classifiers (equal to the number of classes) where in each case, you label every other class except one as “other” and then assess how well your classifier is able to separate a given class compared to all the other classes. You can then subsequently repeat this procedure for the remaining classes until you have carried out this procedure for all the classes. In this case, you will get multiple ROC curves and as before, choosing a threshold would be context dependent. For AUC, a simple heuristic approach is to average the AUC across all the classifiers (but this judgement is, again, context dependent).

Precision and Recall: In addition to sensitivity and specificity, there are additional terms when it comes to assessing the performance of an ROC curve that you may come across. Let us quickly go through them and relate it to sensitivity and specificity that you should now know very clearly.

Both sensitivity and specificity look at the actual cases, those who are positive and those who are negative and then correct proportions of positive are summarized by sensitivity and correct proportions of negative cases are summarized by specificity.

However, two other terms that are commonly used are precision and recall. Recall is the same as sensitivity but precision is focused on all those cases that the test has identified as positive. Of those that the test identifies as positive, how many are actually positive. Graphically, it will look like the following (Figure 12):

Image by author (Figure 12)

Accuracy: As the name implies, this is the proportion of all cases that are correctly identified (positive or negative). Mathematically, this will be:

Accuracy = (TP + TN) / (TP+TN+FP+FN)

Other terms: Sensitivity can also be called True positive rate, specificity as true negative rate, and precision as positive predictive value. (I think the terms should make their meaning clear)

Part 4: ROC implementation in R and Python

Implement ROC in R: Let us start by simulating test score points for both the Covid-19 positive and negative classes using the R built-in random number generator (rnorm)

# Generate hypothetical Covid positive scores # set parameters (mean and standard deviation for both classes)

CovidPositive_mean = 2

CovidPositive_std = 1

CovidNegative_mean = 2

CovidNegative_std = 1 # set the number points you want for each

num_postives= 1000

num_negatives= 1000 # generate positive class data points

CovidPositive_output <- rnorm(num_postives,CovidPositive_mean,CovidPositive_std) # generate negative class data points

CovidNegative_output <- rnorm(num_negatives,CovidNegative_mean,CovidNegative_std) #Combine the two vectors together in a dataframe

test_score=c(CovidPositive_output,CovidNegative_output)

test_class=as.factor(c(rep(1,length(CovidPositive_output)),rep(0,length(CovidNegative_output))))

score_dataframe<-data.frame(test_score,test_class)

You can use ggplot2 package to plot the data points generated. For the data points generated (as shown above), I can use the following to visualise the distribution of the data points generated.

library(ggplot2)

g1<-ggplot(data=score_dataframe,aes(x=test_score,group=test_class,fill=test_class))+geom_density(trim = TRUE, alpha = .5)+scale_fill_discrete(labels=c("Covid19 Negative","Covid19 Positive"),name='class')

Running the code above leads to the following figure, showing the distribution of both the positive and negative class.

Image by author

For getting the ROC, there are several R packages that you can use (check this if you are interested in exploring other packages that will help you plot ROC). In this post, we will use one of the most widely used package (ROCR) to plot the ROC.

#install.packages("ROCR") # only required if not already installed

library(ROCR) # load the package

pred_object<-prediction(score_dataframe$test_score,score_dataframe$test_class)

ROC_curve<-performance(pred_object,"tpr","fpr")

plot(ROC_curve,colorize=TRUE)

Image by author

Implement ROC in Python: Like before, we will first generate data points corresponding to the positive and negative class using a well-known Python library (numpy). To start off, import the numpy library and set the parameters for the positive and negative classes, as shown below.

from numpy.random import rand # set parameters for positive class

CovidPositive_mean = 1.5

CovidPositive_std = 1

# set parameters for negative class

CovidNegative_mean = 1

CovidNegative_std = 1 num_positives= 1000

num_negatives= 1000

Generate data points using the rand function from numpy now.

CovidPositive_output = CovidPositive_mean+rand(num_positives)*CovidPositive_std

CovidNegative_output = CovidNegative_mean+rand(num_negatives)*CovidNegative_std

Before we proceed to plot the ROC, let us visualise the distribution of the data points that we just generated. We will use the seaborn library to help us plot the distribution of the data in both the classes.

import seaborn as sn

sn.distplot(CovidPositive_output,hist=False,kde=True,kde_kws = {'shade': True},label='Covid19 Positive')

sn.distplot(CovidNegative_output,hist=False,kde=True,kde_kws = {'shade': True},label='Covid19 Negative')

If I execute the above, I end up getting the following figure for distribution.

Image by author

Before we proceed onto getting and plotting ROC, let us rearrange the data so that it can be used for getting the ROC. We will generate two arrays, one for labels and one for scores (see below), and use the concatenate function from numpy to help us join arrays.

import numpy as np labels_positive=[1]*num_positives

labels_negative=[0]*num_negatives

test_labels=labels_positive+labels_negative test_scores=np.concatenate((CovidPositive_output,CovidNegative_output))

For getting the ROC, we will use the scikit-learn functions (roc_curve and auc) as shown below.

from sklearn.metrics import roc_curve, auc

fpr, tpr,_ = roc_curve(test_labels, test_scores)

roc_auc = auc(fpr, tpr)

Finally, for plotting our results, we will use matplotlib.

import matplotlib.pyplot as plt

plt.figure()

lw = 2

plt.plot(fpr, tpr, color='navy',

label='ROC curve (area = %0.2f)' % roc_auc)

plt.xlim([0.0, 1.0])

plt.ylim([0.0, 1.0])

plt.xlabel('1-Specificity')

plt.ylabel('Sensitivity')

plt.title('ROC Curve')

plt.legend(loc="lower right")

plt.show()

Running the above results in the following ROC curve.

Image by author

Hopefully, you will now have an intuitive understanding of what an ROC curve is, how a threshold is set, the related jargon associated with ROC and how to implement it in R and Python.