What Is ROC Curve?
By: Saurav Jadhav
Tags: Supervised Learning, Evaluation Metric, Machine Learning, Classification, Data Science

In machine learning, ROC curve is an evaluation metric which measures the performance of a machine learning model by visualizing, especially when data is skewed. Let’s see what exactly that means,

Consider heart data which consist of 13 features such as age, sex, chol (cholesterol measurement). Our goal is to predict whether an individual has heart disease based on the above features, this is a binary classification problem. That is, with only two classes. Let us suppose we have 100 samples (a sample corresponds to single patient’s information) amongst which 90 samples are positive (have a heart disease) so if you say that all patients (100 of them)have heart disease, you have correctly classified 90/100 samples. You didn’t even build the model and got an accuracy of 90%. If we look carefully, we will see that the dataset is skewed that is, number of positive samples is far more than the negative samples (patients without heart disease). So it is not advisable to decide best model just on the basis of accuracy, because it does not represent the data completely. So you might get high accuracy, but your model will probably not perform that well when it comes to real-world samples. Therefore, we need a more reliable evaluation metric and hence, ROC comes into the picture.

Here we will use SVM (support vector machine) as model with different values of gamma (parameter) for comparison and with the help of ROC curve figure out which value of gamma gives us the optimal result (best model). Let’s see this in detail.

Rather than predicting samples are positive or not, we predict the probability of having heart disease for each sample, and if this probability is greater than the threshold, we say the given patient has heart disease. So let’s say we select a threshold value of 0.1 therefore, if the probability is greater than 0.1 we say that particular patient has heart disease. Now as we vary the threshold it is obvious that prediction will also vary. For example, if the threshold is 0, we will predict every sample in the data is positive (since, probability is greater than equals to 0). Similarly, if we have a high threshold of say 0.9, even though the patient has heart disease it is possible that prediction will be ‘no heart disease’, which is very risky! So we have to choose a threshold in such a way that not only we must have good predictions but also balance the tradeoff. Now we give a summary. Table.1 shows the possible results when applying a classiﬁer to the data, this table is also known as confusion matrix. Now, let us define terms given in the table require to build ROC curve.

Source: ‘Understanding Confusion matrix and applying it on KNN-Classifier on Iris Data set’ by Vishwanath Beena on medium.

True positive (TP): Given a patient’s information (age, sex… values), if your model predicts heart disease, and the patient actually has heart disease then, it is considered a true positive.

True negative (TN): Given a patient’s information, if your model predicts no heart disease, and the patient actually has no heart disease then, it is considered a true negative.

In simple words, if your model correctly predicts positive class, it is true positive, and if your model correctly predicts negative class, it is a true negative.

False positive (FP): Given a patient’s information, if your model predicts heart disease, and the patient actually has no heart disease then, it is considered a false positive.

False negative(FN): Given a patient’s information, if your model predicts no heart disease, and the patient actually has heart disease then, it is considered a false negative.

In simple words, if your model incorrectly (or falsely) predicts positive class, it is a false positive. If your model incorrectly (or falsely) predicts negative class, it is a false negative. Now,

TPR/Sensitivity is defined as:

TP / P

Where P = TP+FN

TPR is ‘The fraction of patients with heart disease which are correctly identified’.

FPR/False Positive Rate is defined as:

FP / N

FPR is ‘The fraction of patients without heart disease which are incorrectly identified as with heart disease’.

Specificity is defined as:

TN / N

Where N = TN+FP

Specificity is ‘The fraction of patients without heart disease which are correctly identified’.

ROC provides a simple way to summarize the information related to different thresholds and resulting TPR and FPR values.Table.2 illustrates TPR and FPR values for different thresholds.

Table.2. FPR and TPR values corresponding to different thresholds, for SVM with gamma is equals to 0.001

You might wonder why some threshold values are negative? I will explain this later. Now, Rather than building different tables (like Table.1.) for different values of threshold, you can just look at ROC curve to decide what threshold to select. Most of the time, the top-left value on ROC curve should give you a quite good threshold, as illustrated in Fig.1,

Fig.1. AUC (Area Under Curve) for SVM with gamma is equals to 0.001.

and the corresponding threshold value is highlighted in Table.2. Depend on how many false positive you are willing to accept, you decide optimal threshold. For example, if you don’t want to have too many false positives, you should have a high threshold value. This will however, also give you a lot more false negatives. Observe the trade-off and select the best threshold, by decreasing the threshold, value of TPR, FPR increases and specificity decreases. Similarly, when we increase the threshold TPR, FPR decreases but, specificity increases.

AUC makes it easy to compare one ROC curve to another, larger the area under the curve the better the model.

Fig.2. ROC curves for different gamma values of SVM. The red line which indicates ‘no information classifier’ classifies every sample as ‘having heart disease’.

In Fig.2.The AUC for SVM with gamma is equals to 0.001is 0.88, the AUC for SVM with gamma is equals to 0.0001 is 0.76, and the AUC for SVM with gamma is equals to 0.00001 is 0.75. We say SVM with gamma is equals to 0.001 is better model than others, since, 0.88 is close to maximum value of AUC that is one, AUC corresponds to SVM with gamma is equals to 0.001 is illustrated in Fig.1, we expect a classifier that performs no better than chance to have an AUC of 0.5, the no information classifier in Fig.2 (red line) predicts every patient as’ with heart disease’ independent of actual target (class).

You can also compare other classification models like: Logistic Regression, KNN classifier with SVM.

AUC is a widely used metric for binary classification tasks in the industry, and a metric everyone should know about.

Note: In general we use probabilities for comparison with the threshold. In case of SVM, decision function values are computed and are compared with the threshold, and can take positive or negative values, which can be seen in Table.2. But the prediction is done in the same way: if function value greater than threshold we assign the sample to the positive class (patient predicted as, ‘with disease’), otherwise we assign the sample to negative class (patient predicted as, ‘without disease’).

References: