Evaluating Classification Models. A Guided Walkthrough Using Sci-Kit... | by Ishaan Dey | Towards Data Science

Authors: Ishaan Dey, Evan Heitman, & Jagerynn T. Verano
A doctor wants to know if his patient has a disease. A credit card company is interested in determining if a certain transaction is a fraud. A graduate school candidate is interested if whether or not it’s likely that she gets accepted into her program. Many applied models are interested in predicting the outcome of a binary event, which is any event that can be answered by a simple yes or no question. This is called a binary classification problem, and is distinct from multi-class classification, where we would be interested in predicting which disease an individual would have, or which product a customer would purchase from a selection.
Overview
Regression models will often report familiar metrics, such as R2, to show goodness-of-fit, or how well the model can adhere to the data. With discrete outcomes, we cannot apply the same formulas, so in this post, we will examine how to evaluate a classifier model’s performance, by breaking down the types of error a model can mistake and quantitative measures that can summarize a model’s ability to perform as we expect it to.
Throughout this post, we use a credit card transaction dataset, with targets labelling a fraudulent transaction (a binary outcome, with 1 indicating fraud). Given the low occurrence of fraud, the dataset was under-sampled to create a new distribution of 90:10 non-fraud to fraud observations. That data was further split into a 70:30 train to test split. We fitted a logistic regression model, and used a threshold of 0.5 to predict values as a fraud. You can follow along using the interactive python notebook linked below.
https://github.com/ishaandey/Classification_Evaluation_Walkthrough
For any dataset containing binary outcomes, we can label all observations with a 1 (indicating the occurrence of an event) or 0 (the absence of an event). We can fit a model to the feature variables and make a prediction for each observation given the set of feature variables. Values that are correctly classified by the model are labelled true (T), while incorrect predictions are false (F). Positive (P) and negative (N) refer to the model’s prediction, as opposed to the actual value of the observation. For example, a negative value incorrectly classified as positive is called a false positive.
As an aside, many readers will be familiar with false positives and false negatives as α and β, respectively. False positives are denoted by alpha (α) while false negatives are denoted by beta (β). Thus, when we suggest a probability of a statistic is less than α of 0.05, we say that the probability of incorrectly labelling that outcome as significant is 0.05.
Our model, produced with the function below, produced the following confusion matrix.
confusion_matrix(y_test, y_pred)
Say we come across a diagnostic tool for detecting cancer that says it can correctly predict outcomes with a 99.8% accuracy. Is that really that impressive though? According to CDC estimates, the cancer mortality rate among the general population is about 0.16%. If we were to make a dummy model, and say that regardless of the feature information available to us, each and every outcome should be classified as a negative case, we’d be performing at a 99.84% accuracy, simply because 99.84% of the observations in our sample are negative. We’d have missed diagnosing every single one of the patients with cancer, but we can still truthfully report that our model operates with a 99.84% accuracy. Clearly, a good model should be able to distinguish between what is and isn’t a positive case.
In a more technical sense, the value for accuracy can be obtained from the confusion matrix quite easily, as the number of true positives and true negatives divided by the total number of observations tested on. In simpler terms, it’s the proportion of observations correctly classified.
Accuracy = (TP + TN) / (TP + TN + FP + FN)
Going back our fraud detection data, our logistic regression model predicts with an accuracy of 97.7%. But how much better does this work than a dummy classifier? Let’s look at the actual distribution of outcome.
sns.countplot(x=fraud['Class'])
Here, the class imbalance, or the skewed distribution of outcomes is obvious. A quick check using the .value_counts() function shows that we have 137 counts of fraud cases in our data, and 1339 counts of non-fraud cases, meaning that 9.3% of our test cases are fraudulent. If we were to apply a dummy model that blindly predicts all observations to be a non-fraud case, as can be done with the sklearn.dummy package, we can see that our reported accuracy of 90.7% matches that of our frequency distribution (90.7:9.3). So, even though our dummy model is quite useless, we can still report that it performs at a 90% accuracy. The confusion matrix below shows this inability to distinguish in further detail.
y_test.value_counts() / len(y_test)dum = DummyClassifier(strategy='most_frequent')
The biggest perk of using the confusion matrix is that we can easily pull out various other values that reflect how well our model runs relative what we’d expect from a dummy model.
Sensitivity (or Recall, or True Positive Rate)
As we saw with the above example, a good model should successfully detect close to all of the actual fraudulent cases, given how much higher a cost of not catching a fraud case is relative to putting a non-fraudulent transaction under scrutiny by incorrectly suggesting that it was a fraud case. Sensitivity, also known as recall, quantifies that intuition, and reflects the ratio of correctly classified positives to actual positive cases.
Sensitivity = TP / (TP + FN)
Interpretation of sensitivity is fairly straightforward. All values range between 0 and 1, where a value of 1 indicates that the model detected every single case of fraud, while a value of 0 indicates that all the actual cases of fraud were not detected. With our logistic regression model, we have a sensitivity of 108 / 137 = 0.788.
Specificity
Specificity helps us determine how many were correctly classified as non-fraud out of total true non-fraud cases. The false positive rate, or the false alarm rate, is the opposite of specificity. In our fraud detection model, we have a specificity of 1334/1339 = 0.996
Specificity = TN / (TN + FP)
False-positive rate= 1 — Specificity
In this particular case, specificity is a less relevant metric as the cost of classifying a non-fraud case as a fraud case is lower than missing a fraud case entirely. But there are cases where false alarms are equally undesirable, such as in disease detection, where misdiagnosis would lead to unnecessary follow-up procedures.
Precision
On the other hand, we may want to test the certainty of our predictions, for example, we may in interested how many of the fraud cases that our model says it picked up were truly fraudulent. Precision does just that, by providing the proportion of true positives relative to the number of predicted positives. Intuitively, a low precision would mean that we’re giving a lot of customers headaches, in that we’re classifying more fraudulent transactions than what’s actually fraudulent. With the logistic regression model, our precision is 108/113 = 0.956
Precision = TP / (TP + FP)
In our fraud dataset, it’s more important that we minimize the number of fraud cases that go undetected, even if it comes at a cost of incorrectly classifying non-fraud cases as fraudulent, simply because the cost to the firm is far higher for the former case (potentially thousands of dollars in lost revenue or a few minutes of a customer’s time to verify their transaction). In other words, we would rather commit a type I error than a type II error. Although we would prefer a model that maximizes both sensitivity and specificity, we would prefer a model with a maximum sensitivity, as it minimizes the occurrences of a type II error.
F1 Score
Last but not least, the F1 score summarizes both precision and recall and can be understood as the harmonic mean of the two measures. An F1 score of 1 indicates perfect precision and recall, therefore the higher the F1 score, the better the model. Our logistic model shows a F1 score of 0.864.
F1 = 2 * (Precision * Sensitivity) / (Precision + Sensitivity)
How exactly are we coming up with our predictions? From our logistic regression, we compute a predicted probability of a given observation being fraudulent that falls between 0 to 1. We say that all probabilities greater than 0.5 should indicate a prediction of a fraud, while all values less than 0.5 return a prediction of a legitimate transaction.
But given how much more willing we are to make a Type I error, wouldn’t it be better to classify a case as fraudulent even if there’s only a slight probability that it is? In other words, what if we lower our threshold for discriminating between a fraud and non-fraud, such that we catch more frauds, from shifting it down from the orange to green line?
From the definitions we discussed previously, the specificity of the model would increase, as we’ve now classified more positives, but at the same time, we increase the likelihood that we’re incorrectly labelling a non-fraudulent case as a fraud, thus dropping the sensitivity as the number of false negatives is increasing. It’s a constant trade-off, and the rate at which the sensitivity increases from dropping specificity is an attribute specific to each model.
Thus far, we’ve been reporting our metrics from a confusion matrix calculated at a threshold of 0.5 (the orange line), but if we were to lower the discrimination threshold to 0.1 (the green line), the sensitivity increases from .788 to .883, while the specificity drops from .996 to .823.
classification_report(y_test, y_pred)
ROC Curve
We can see the change in these values at all thresholds using a Receiver Operating Characteristic, or ROC Curve, which plots the true positive rate against the false positive rate, or the sensitivity against 1-specificity for each threshold. We can see here the ROC curve for our logistic classifier, in which the different thresholds are applied to the predicted probabilities to produce different true positive and false positive rates.
Our dummy model is a point at (0,0) on the blue curve where the discrimination threshold is such that any probability <1.0 is predicted as a non-fraud case. This indicates that though the we correctly classified all of the non-fraud cases, we incorrectly classified all non-fraud cases. A perfectly discriminating model, on the other hand, would have a point on curve at (0,1), which would indicate our model perfectly classifying all fraud cases as well as non-fraud cases. The lines for both of these model would be generated as a function of the threshold level.
fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob[:,1])
The diagonal line shows where true positive rate is the same as false positive rate, where there is an equal chance of correctly detecting a fraud case and detecting a non-fraud case as fraudulent. This means that any ROC curve above the diagonal does better than random chance of predicting outcomes, assuming a 50/50 class balance.
Thus, all ROC curves that we’ll encounter in the field will be drawn above the y=x line, and below the line going vertically up to (0,1), then horizontally across to (1,1). We can quantify the degree to which a ROC curve performs by looking at the area under the curve, or AUC ROC. This value will adopt values between 0.5 (of the diagonal line) and 1.0 (for a perfect model). Our fraud detection model performs with an AUC ROC of 0.934, not bad for an out-of-box model.
from sklearn.metrics import aucauc = roc_auc_score(y_test,logis_pred_prob[:,1])
PRC Curve
Going back to the logit model, what happens to precision and recall when we shift our discrimination threshold from the orange to green? As we move from a threshold of 0.5 to 0.1, the recall increases from 0.788 to 0.883, since the proportion of correctly detected frauds over the actual number of frauds increases, while the precision drops from 0.956 to 0.338, as the proportion of true fraud cases over the predicted number of fraud cases decreases. Just as we did with the ROC curve, we can plot the trade-off between precision and recall as a function of the thresholds, and obtain a Precision-Recall Curve, or PRC.
precision, recall, thresholds = precision_recall_curve(y_test, y_pred_prob[:,1])
Typically, PRCs are better suited for models trained on highly imbalanced datasets, as the high true negative value used in the formulation of the ROC curve’s false positive rate can ‘inflate’ the perception of how well the model performs. PRC curves avoid this value, and can thus reflect a less biased metric for the model’s performance.
We can summarize this curve succinctly using an average precision value or average F1 score (averaged across each threshold), with an ideal value close to 1.
from sklearn.metrics import f1_scorefrom sklearn.metrics import average_precision_scoref1 = f1_score(y_test, y_pred_prob)ap = average_precision_score(y_test, y_pred_prob)
Quick review of formulas:
In this post, we discussed how to move past accuracy as a measure of performance for a binary classifier. We discussed sensitivity, specificity, precision and recall. We looked at what happens when we change our decision threshold, and how we can visualize the trade-off between sensitivity and specificity, as well as between precision and recall. As a reminder, all the visualizations and model produced can be found on the interactive python notebook, linked below.
https://github.com/ishaandey/Classification_Evaluation_Walkthrough
That’s it for this post. Let us know if you have any questions!