Deep dive into ROC-AUC. deep dive into ROC-AUC | by Songhao Wu | Towards Data Science

I believe most people have heard of the ROC curve or Area under the Curve before if you are interested in data science. However, what exactly is the ROC curve, and why area under the ROC curve is a good metric to evaluate the classification model? I have briefly walked through in my previous article on top metrics for classification but I will go through in more detail in this article.
The full name for ROC is Receiver Operating Characteristic. It was first created for the usage of radar signal detection during World War II. The US has used it to improve the accuracy of detecting Japanese Aircraft from Radar. That is why it is called receiver operating characteristic.
AUC or the area under the curve is just the area under the ROC curve. Before we go into what is ROC curve, we need to understand quickly is Confusion Matrix.
As you can see from the picture above, the confusion matrix is the combination of your prediction(1 or 0) vs actual value(1 or 0). Based on your prediction value and whether you have correctly classified it is divided into 4 sections. For example, true positive is the number of cases that you correctly classify the case to be positive while false positive means the number of cases that you wrongly classify the case to be positive.
The confusion matrix contains only absolute numbers. However, based on these numbers, we can develop many new percentage-based metrics. True Positive Rate and False Positive Rate are two of them.
True Positive Rate(TPR): Among the truly positive cases, how much percentage of them are captured in the model correctly
True Positive Rate(TPR): Among the truly positive cases, how much percentage of them are captured in the model correctly
TPR=TP/(TP+FN)
False Positive Rate(NPR): Among the truly negative cases, how much percentage of them are actually false positive.
False Positive Rate(NPR): Among the truly negative cases, how much percentage of them are actually false positive.
NPR=FP/(FP+TN)
Ok let us go into ROC curve now!
As you can see from the graph, the ROC curve is simply TPR plotted against NPR. Ok end of the story...
Just Joking!
There are a lot more information you can read from the chart. The first thing I want to ask here is : There is only one set of TPR,FPR for the model result and how do you form so many points to plot the graph?
This goes back to how the classification model works. When you build a classification model like a decision tree and you want to classify maybe whether the stock will increase or decrease tomorrow based on the input. The classification model will first calculate the probability of increase or decrease based on the historical value you provided. After that, based on the threshold, it will decide whether the result is increasing or decreasing.
Yes, the keyword here is the threshold. Different threshold creates different TPR and FPR. They represent different points and form up the ROC curve. You can choose the output to be ‘Increase’ if more than a 50% chance of increase based on historical data but you can also increase the threshold and only show ‘Increase’ if more than a 90% chance. If you choose 90% confidence instead of 50%, you will be more confident that the stocks you choose to show ‘Increase’ are very likely to increase but you may miss out on some potential stocks.
As we know the bigger the area under the curve(AUC), the better the classification. The ideal or perfect curve is a vertical line from (0,0) to (0,1) and then to (1,1) which means the model can 100% separate positive and negative cases. However, if you pick randomly for each case, TPR and FPR should increase at the same rate. The blue dotted line shows the curve TPR and FPR when you blindly guess positive or negative for each case and for that diagonal line, the area under the curve(AUC) is 0.5.
Look at the two dots on the ROC curve. Green dot has a very high threshold it means only if you are 99% sure then you can classify the case as positive. The red dot has a relatively lower threshold and it means you can classify the case to be positive if you are 90% confident.
Whats the movement of TPR and FPR from green dot to red dot?
Both TPR and FPR increase. When you reduce the threshold, more positive cases can be identified, thus TP increases, and TP/(TP+FN) also increases. On the other side, inevitably you will wrongly classify some negative cases to be positive due to the lowering of threshold and so FP & FP/(FP+TN) also increases.
We can see that TPR and FPR are positively correlated and you have to balance between maximizing capture of positive cases and minimizing the wrong classification of negative cases.
It is hard to define the optimal point because you should choose the most suitable threshold for the business case. However, the general rule is to maximize (TPR-FPR) which in the graph is represented by the vertical distance between the orange and blue dotted line.
We can see from the chart above that among the three points, redpoint has the biggest TPR-FPR and should be considered as the optimal point. Moving from red point to green point, the threshold gets more strict. However, a big number of positive cases are not captured from the model with a less than proportionate decrease in wrongly classified cases. Moving from the red point to the blue point, the threshold is getting more lenient and adding more wrongly classified positive points than correctly identified positive points.
A good metric of a machine learning model should display the true and consistent prediction ability of the model. It means if I change the testing dataset it should not have a very different score.
ROC curve does not only consider the classification results but also the prediction probability of all classes. For example, if the output is classified correctly based on 51% probability, then it is very likely to be wrongly classified if you use another testing dataset. In addition, the ROC curve also considers the performance across the different thresholds and so it is a comprehensive metric to assess how good the separation of the cases in different groups.
As I have shown before, for the binary classification problems, if you pick randomly you can achieve 0.5 AUC. Thus, if you are solving a binary classification problem, a reasonable ROC score should>0.5. A good classification model usually has AUC score >0.9 but it is totally application dependent.
If you just want to find AUC score, you can refer to sklearn metric package here.
If you want to plot the ROC curve for your model results you can refer to here
# Calculate FPR, TPR and AUCfrom sklearn.metrics import roc_curve, aucfpr, tpr, treshold = roc_curve(y_test, y_score_new)roc_auc = auc(fpr, tpr)#Plot ROC curveplt.plot(fpr, tpr, color='darkorange',         label='ROC curve (area = %0.2f)' % roc_auc)plt.plot([0, 1], [0, 1], color='navy', linestyle='--')plt.xlim([0.0, 1.0])plt.ylim([0.0, 1.05])plt.xlabel('False Positive Rate')plt.ylabel('True Positive Rate')plt.title('Receiver operating characteristic example')plt.legend(loc="lower right")plt.show()
Here is the code for me to plot the ROC example plot which I used in this article. You need two inputs which are the true value of y and also prediction probability. Note that the roc_curve function only needs probability for the positive cases but not the probability for both classes. If you have a multi-class classification problem, you can also use the package and there is an example of how to plot it in the link above.
If you are interested to know what are the other top metrics for the evaluation of the classification model, you can refer to the link below: