ROC Curve Transforms the Way We Look at a Classification Problem | by Huy Bui | Towards Data Science

The Receiver Operating Characteristic (ROC) curve is a probability curve that illustrates how good our binary classification is in classifying classes based on true-positive and false-positive rates.
The Area Under Curve (AUC) is a metric that ranges from 0 to 1. It is the area under the (ROC) curve.
Why is understanding the ROC curve and AUC important for a data scientist? To answer this question, let’s take a look at the breast cancer data set below:
from sklearn.datasets import load_breast_cancerimport pandas as pddata=load_breast_cancer()columns=data.feature_namesX=pd.DataFrame(data.data, columns=columns)y=pd.DataFrame(data.target, columns=['Target'])df=pd.concat([y,X],axis=1)
When dealing with a classification problem, one should consider the accuracy trade-off. If the data is used by the hospital to detect breast cancer, the model should emphasize the ability to detect cancer from the patients. The model also needs to carefully minimize the number of healthy patients that might be incorrectly marked as cancer since cancer treatment’s side effects can be serious.
The targets of this data set are malignant (1) and benign (0). Let’s take a look at the histogram below:
Usually, for this type of problem, the data set is much more imbalanced. According to the U.S. Breast Cancer Statistics, 12% of US women will develop invasive breast cancer over the course of their lifetime. However, this is only for educational purpose, we can ignore the realistic aspect of the dataset.
As mention above, we need to find a good balance between recall and false-positive rates. You might ask a question that why false-positive rates instead of precision. As a reminder:
Recall: also called the True Positive Rate (TPR). TPR is the ratio between the correct cancer predictions over the total number of cancer cases. Mathematically speaking, TPR=TP/(TP+FN) where FN are those that are falsely classified as benign.
Precision: is the ratio between all correct cancer and the total number of predictive cancers. Precision = TP/(TP+FP)
False Positive Rate: FPR=FP / (TN+FP)
We can see the difference between precision and FPR based on the math formula. Precision focuses on how well the model can detect cancer. Conversely, FPR focuses on the failure rate to detect cancer. There is a long discussion about Precision-Recall (PR) versus FPR-Recall (ROC) here if you want to know more.
In short, we should use precision-recall only when the positive class is much more interesting than the negative and the ratio positive/negative is small (Severe imbalanced). However, in this case, benign is about 4/7 malignant which indicates the data is only mildly imbalanced. As we discussed above, wrongly classified cancer might lead to bad consequences. Hence, using the ROC curve here is justified.
Note: The approach of precision-recall is very similar to the ROC curve, you can find it document here.
The model that we are going to use is the Logistic Regression. Usually, when we do classification, the outputs are already rounded off by some threshold. ROC curve uses different thresholds to plot the relationship between TPR-FPR. Because we want to investigate the relationship between TPR, FPR, and threshold, let’s take a look at logistic regression outputs before they are rounded off.
from sklearn.linear_model import LogisticRegressionlogreg = LogisticRegression(fit_intercept=False, C=1e12, solver='liblinear')logreg.fit(X, y.values.flatten())prediction=logreg.decision_function(X) #Values before the ouput#Define positive and negativecancer_index=y[y.Target==1].indexno_cancer_index=y[y.Target==0].indexcancer_prob=prediction[cancer_index]no_cancer_prob=prediction[no_cancer_index]#Histogramn_bins = 20fig, axs = plt.subplots(1, 2, sharey=True, tight_layout=True)axs[0].hist(cancer_prob, bins=n_bins)axs[1].hist(no_cancer_prob, bins=n_bins)plt.show()
If we put these two histograms together using kernel density estimation and truncating some outliners, we obtain a beautiful graph:
The blue line is the decision boundary. It divides the graph into 4 sections:
Pure green: no cancer (True negative)
Pure red: cancer (True positive)
Mix left: has cancer but was detected as no cancer (false positive)
Mix right: have no cancer but was detected as having cancer (false negative)
Note that a good binary classification model would minimize the mixed area. So, after obtaining a good model, investigating the ROC curve would tell us if our model does a good job of balancing between TPR and FPR. Let’s take a look at the ROC curve below:
from sklearn.metrics import roc_curve, auc# Normalize the DataX = X.apply(lambda x : (x - x.min()) /(x.max() - x.min()),axis=0) X_train, X_test, y_train, y_test = train_test_split(X, y.values.flatten(), random_state=0)y_score = logreg.fit(X_train, y_train).decision_function(X_test)fpr, tpr, thresholds = roc_curve(y_test, y_score)def ROC_curve(fpr,tpr):    # Seaborn's beautiful styling    sns.set_style('darkgrid', {'axes.facecolor': '0.9'})print('AUC: {}'.format(auc(fpr, tpr)))    plt.figure(figsize=(10, 8))    lw = 2    plt.plot(fpr, tpr, color='darkorange',             lw=lw, label='ROC curve')    plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')    plt.xlim([0.0, 1.0])    plt.ylim([0.0, 1.05])    plt.yticks([i/20.0 for i in range(21)])    plt.xticks([i/20.0 for i in range(21)])    plt.xlabel('False Positive Rate')    plt.ylabel('True Positive Rate')    plt.title('Receiver operating characteristic (ROC) Curve')    plt.legend(loc='lower right')    plt.show()    ROC_curve(fpr,tpr)
The broken line in the model is random choices (probability 50%). Since there are 2 classes, the slope of this line is 1 everywhere, and hence we get the diagonal line of the unit square. Note that the area under this line (AUC) is 0.5.
The orange line is the ROC curve. It locates above the broken line to indicate that our accuracy is better than random choices. By increasing the numeral value of the threshold, more and more observation belongs to the cancer class. Therefore, TP approaches to 1 and FN approaches to 0. This makes TPR approaches 1. Similarly, increasing the threshold values would lead to FP approaches to 1 and TN approaches to 0. This would make the FPR approaches 1. Therefore, we conclude that FPR, TPR, and the threshold are directly proportional to one another. The ROC curve only shows 2 axis FPR and TPR but now we know that threshold is the confounding variable that generates the graph.
Look at the graph again. The AUC score is 0.96 which is nearly perfect. We also want to look at the ROC curve to see at which point in the graph we have our perfect combination of FPR and TPR. For example, if we tune the current model to have a 97% TPR, the FPR would be approximately about 6%. If we lower the FPR to 5% then the TPR is only 82%. All of the different ways of tuning the model are subject-matter which varies in different fields.
We can use ROC analysis to select possibly optimal models based on the cost context or the class distribution. ROC analysis is also very similar to cost/benefit analysis of diagnostic decision making. It is a great tool every data scientist should have to evaluate their classifier performance.
You can find the full code for this blog from my repository.
Please feel free to leave any comment, question or suggestion below. Thank you for reading!