Deep dive into ROC-AUC. deep dive into ROC-AUC
By: Songhao Wu
Tags: Classification Problem, Machine Learning, Roc, Data Science, Classification Models

I believe most people have heard of ROC curve or Area under the Curve before if you are interested in data science. However, what exactly is ROC curve and why area under the ROC curve is a good metric to evaluate classification model? I have briefly walked through in my previous article on top metrics for classification but I will go through in more details in this article.

Some Background Information

Full name for ROC is Receiver Operating Characteristic. It was first created for the usage of radar signal detection during World War II. The US has used it to improve the accuracy of detecting Japanese Aircraft from Radar. That is why it is called receiver operating characteristic.

AUC or the area under the curve is just the area under the ROC curve. Before we go into what is ROC curve, we need to understand quickly what is Confusion Matrix.

Source: Author — Confusion Matrix(Binary)

As you can see from the picture above, confusion matrix is the combination of your prediction(1 or 0) vs actual value(1 or 0). Based on your prediction value and whether you have correctly classified it is divided into 4 sections. For example, true positive is the number of cases that you correctly classify the case to be positive while false positive means the number of cases that you wrongly classify the case to be positive.

Confusion matrix contains only absolute numbers. However, based on these numbers, we can develop many new percentage based metrics. True Positive Rate and False Positive Rate are two of them.

True Positive Rate(TPR): Among the truly positive cases, how much percentage of them are captured in the model correctly

TPR=TP/(TP+FN)

False Positive Rate(NPR): Among the truly negative cases, how much percentage of them are actually false positive.

NPR=FP/(FP+TN)

Ok let us go into ROC curve now!

What is ROC curve?

Source: Author — ROC curve example

As you can see from the graph, ROC curve is simply TPR plotted against NPR. Ok end of story…

Just Joking! There are a lot more information you can read from the chart. The first thing I want to ask here is : There is only one set of TPR,FPR for the model result and how do you form so many points to plot the graph?

This goes back to how classification model works. When you build a classification model like decision tree and you want to classify maybe whether the stock will increase or decrease tomorrow based on the input. The classification model will first calculate the probability of increase or decrease based on historical value you provided. After that, based on the threshold, it will decide whether the result is increase or decrease.

Yes, the keyword here is threshold. Different threshold creates different TPR and FPR. They represent different points and form up the ROC curve. You can choose the output to be ‘Increase’ if more than 50% chance of increase based on historical data but you can also increase the threshold and only show ‘Increase’ if more than 90% chance. If you choose 90% confidence instead of 50%, you will be more confident that the stocks you choose to show ‘Increase’ is very likely to increase but you may miss out some potential stocks.

Why I plot the blue dotted line on the graph?

As we know the bigger the area under the curve(AUC), the better the classification. The ideal or perfect curve is vertical line from (0,0) to (0,1) and then to (1,1) which means the model can 100% separate positive and negative cases. However, if you pick randomly for each case, TPR and FPR should increase at the same rate. The blue dotted line shows the curve TPR and FPR when you blindly guess positive or negative for each case and for that diagonal line, area under the curve(AUC) is 0.5.

What will happen to the TPR, FPR and ROC curve if I change my threshold?

Look at the two dots on the ROC curve. Green dot has a very high threshold it means only if you are 99% sure then you can classify the case as positive. Red dot has a relatively lower threshold and it means you can classify the case to be positive if you are 90% confident.

Whats the movement of TPR and FPR from green dot to red dot?

Both TPR and FPR increase. When you reduce the threshold, more positive cases can be identified, thus TP increases and TP/(TP+FN) also increases. On the other side, inevitably you will wrongly classify some negative cases to be positive due to the lowering of threshold and so FP & FP/(FP+TN) also increases.

We can see that TPR and FPR are positively correlated and you have to balance between maximising capture of positive cases and minimising wrong classification of negative cases.

How to choose the optimal point on the ROC curve?

It is hard to define the optimal point because you should choose the most suitable threshold for the business case. However, the general rule is to maximise (TPR-FPR) which in the graph is represented by the vertical distance between orange and blue dotted line.

We can see from the chart above that among the three points, red point has the biggest TPR-FPR and should be considered as the optimal point. Moving from red point to green point, the threshold gets more strict. However, big number of positive cases are not captured from the model with less than proportionate decrease in wrongly classified cases. Moving from red point to blue point, the threshold is getting more lenient and adding more wrongly classified positive points than correctly identified positive points.

Why Area under the ROC curve is a good metric for evaluation of classification model?

A good metric of machine learning model should display true and consistent prediction ability of the model. It means if I change the testing dataset it should not have a very different score.

ROC curve does not only consider the classification results but also the prediction probability of all classes. For example, if the output is classified correctly based on 51% probability, then it is very likely to be wrongly-classified if you use another testing dataset. In addition, ROC curve also considers the performance across different threshold and so it is a comprehensive metric to assess how good the separation of the cases in different groups.

What is acceptable AUC score for a classification model?

As I have shown before, for binary classification problem, if you pick randomly you can achieve 0.5 AUC. Thus, if you are solving binary classification problem, a reasonable ROC score should>0.5. A good classification model usually has AUC score >0.9 but it is totally application dependent.

How do I calculate AUC and plot the ROC curve in Python?

If you just want to find AUC score, you can refer to sklearn metric package here. If you want to plot the ROC curve for your model results you can refer to here

# Calculate FPR, TPR and AUC

from sklearn.metrics import roc_curve, auc

fpr, tpr, treshold = roc_curve(y_test, y_score_new)

roc_auc = auc(fpr, tpr) #Plot ROC curve

plt.plot(fpr, tpr, color='darkorange',

label='ROC curve (area = %0.2f)' % roc_auc)

plt.plot([0, 1], [0, 1], color='navy', linestyle='--')

plt.xlim([0.0, 1.0])

plt.ylim([0.0, 1.05])

plt.xlabel('False Positive Rate')

plt.ylabel('True Positive Rate')

plt.title('Receiver operating characteristic example')

plt.legend(loc="lower right")

plt.show()

Here is the code for me to plot the ROC example plot which I used in this article. You need two inputs which is true value of y and also prediction probability. Note that roc_curve function only needs probability for positive case but not the probability for both classes. If you have multi-class classification problem, you can also use the package and there is example on how to plot it in the link above.

If you are interested to know what are the other top metrics for evaluation of classification model, you can refer to link below: