Classification Metrics & Thresholds Explained | by Kamil Mysiak | Towards Data Science

The metric we use to evaluate our classifier depends on the nature of the problem we want to solve and the potential consequences of prediction error. Let’s examine a very common example of cancer diagnosis (ie. classified as having cancer or not having cancer). We want our model to predict as many actual/true cancer diagnoses as possible but we also know that it is statistically impossible to correctly identify all true cancer diagnoses. Our model will eventually classify/predict someone to have cancer when they actually don’t have cancer (false positive) and predict someone not to have cancer when they actually have cancer (false negative). The question we have to ask ourselves is “What is worse? Predicting someone to have cancer when they actually don’t or predicting someone not to have cancer when they do?”. The answer in this example is obvious as the consequences of telling someone they don’t have cancer when they do far outweigh the former. Let’s keep this example in mind but let’s review the commonly used classification performance metrics.
A confusion matrix summarizes are the model’s predictions. It gives us the number of correct predictions (True Positives and True Negatives) and the number of incorrect predictions (False Positives and False Negatives). In our cancer example, if our model predicted someone to have cancer and the person has cancer that’s a true positive. When our model predicted someone not to have cancer and that person does not have cancer that’s a true negative. When our model predicted someone to have cancer but that person does not have cancer that’s a false positive (ie. the model falsely predicted a positive cancer diagnosis). Finally, when our model predicted someone not to have cancer but they do that’s a false negative (ie. the model falsely predicted a negative cancer diagnosis).
Much of the remaining performance metrics are derived from the confusion matrix therefore, it is imperative you have a good understand.
In simplest terms, accuracy details how often our model is correct. In other words, is the number of correct predictions (TP, TF) divided by the total number of predictions. Accuracy is typically the first metric but it can be very misleading if not considered carefully. For example, let’s consider an imbalanced dataset that was used to train our model. We have 1000 non-cancer diagnoses and 10 cancer diagnoses. A model was able to correctly predict 900 of the non-cancer diagnoses and 1 of the cancer diagnoses would have an accuracy of 0.89% ((900+1)/1010=0.89).
(TP+TN)/(TP+FP+FN+TN)
Precision tells us what percentage of the predicted positive class was correct. In other words, what percentage of predicted cancer diagnoses actually had cancer. Precision cares only that our model accurately predicted the positive class. I like to think of Precision as a measure of how “picky” or how “sure” a model is that it’s correctly predicting positive cancer diagnoses. A different example might be related to a zombie apocalypse. Even allowing one infected zombie into your camp will cause everyone to get infected. A model that has high precision would make sure that those people you let into your camp are healthy. However, the model would also have a high false-negative count (ie. healthy people who are considered infected). High precision relates to low FP rates. Preferred when the consequences of false positives are high.
TP/(TP+FP)
Recall is the percentage of actual positives our model predicted to be positive. What percentage of people diagnosed with cancer (by a doctor) our model predicted to have cancer. Recall cares less about accurately predicting positive cases but making sure we have captured all the positive cases since the consequences of classifying someone not to have cancer when they do are much graver. We want our model to have high recall to classify as many actual cancer diagnoses as having cancer. Unfortunately, this means that the model will also classify a larger number of individuals who do not have cancer as having cancer (ie. false positives). High recall relates to low FN rates. Preferred when the consequences of false negatives are high.
TP/(TP+FN)
It is mathematically impossible to have both high precision and high recall and this is where the F1-score comes in handy. F1-score is the harmonic average of precision and recall. If you’re trying to produce a model that balances precision and recall, F1-score is a great option. F1-score is also a good option when you have an imbalanced dataset. A good F1-score means you have low FP and low FN.
2*(Recall * Precision) / (Recall + Precision)
The receiver operating characteristics curve (ROC) plots the true positive rate against the false-positive rate at any probability threshold. The threshold is the specified cut off for an observation to be classified as either 0 (no cancer) or 1 (has cancer).
That was a mouthful........
This will help us better understand what is a threshold, how we can adjust the model’s prediction by changing the threshold, and how a ROC curve is created. This example will also bring in the concepts of TP, TN, FN, and FP you learned above.
Follow along all this will make sense, I promise :)
Logistic regression is a binary classifier and in the example above, we are trying to correctly predict obesity based on only one feature/predictor, weight. We have a dataset of 9 observations, where 4 (green) observations are not obese and 5 (red) are obese. Based on the sigmoid function (the curvy line) produced by the log regression the first 3 green non-obese observations have a 0% chance of being predicted as obese based on their weight. The last 3 red obese observations also have a 100% probability of being predicted as obese. The second red obese observation has a ~70% chance of being predicted as obese.
However, the 4th green non-obese observation has a ~85% chance of being predicted as obese based on its weight (must be very muscular), but we know that’s the wrong prediction. Furthermore, the first obese observation has a ~15% chance of being obese which once again the wrong prediction. Keep in mind the threshold is 0.5, therefore, any data point which falls at 51% probability of being obese will be classified as obese and any data point which falls below 50% will be classified as non-obese.
Let’s assume the consequences of incorrectly predicting someone not to be obese when they actually are obese are significant. In other words, we want to adjust the model in a way that would capture or predict as many actual obese individuals as possible (ie. high recall). In order to accomplish this task, we need to change our threshold. With the threshold lowered to 0.2, our model will correctly predict all 5 obese observations as obese. Any data point which falls above the 0.2 threshold will be classified as obese and vice versa.
However, by lowering the threshold to 0.2 the fourth non-obese observation was now predicted as obese. This is the trade-off we make when adjusting the model’s threshold.
Once again let’s consider our cancer example. We would be ok with a model with a 0.2 threshold as it would correctly predict all actual cancer diagnoses (ie. True Positives) and have a very high recall rate. However, the model would make a trade-off as it would ultimately predict more individuals who actually didn’t have cancer as having cancer (ie. False Positives). The consequences of the false positives are less severe than incorrectly predicting someone to not have cancer when they do.
Now a model with a 0.9 threshold would do the opposite of a 0.2 threshold. It would make sure to predict all 4 non-obese individuals as non-obese, however, the first two obese individuals would ultimately be predicted as non-obese. Now that we understand the threshold and its purpose let’s return to the ROC curve.
So where does the ROC Curve come into play?
This was an easy example with only 9 data points where the threshold was easy to see and interpret. What if you have a million observations and a more complicated situation compared to our obesity or cancer example. What would be the best threshold in that situation? Do we have to make a bunch of these graphs to find a threshold that best suits our needs?”. The answer is No. A ROC curve is a great way to quickly summarize this data so you can choose your threshold.
Here is an example of a ROC curve (green line), notice the true positive rate (actually obese and predicted as obese) on the y-axis and false positive rate (non-obese but predicted as obese) on the x-axis. Finally, remember an ROC is used to summarize the TP rate and the FP rate at various thresholds. The curve wouldn’t show you the thresholds but it will show you what your false-positive rate will be at various true positive rates.
Let’s quickly compare 3 separate thresholds on the ROC curve. A threshold of 0.9 has the following confusion matrix from which we can calculate a TP rate of 0.6 and a FP rate of 0.
Let’s plot the TP rate and FP rate points on our ROC curve. Let’s also plot the TP and FP rates for thresholds of 0.6 and 0.2 for demonstration.
The green dashed line represents the ROC curve. The individual blue points are the result of 4 separate confusion matrixes where the threshold was adjusted. Now ask yourself, “If this was the cancer example and you wanted to make sure you captured/predicted all actual cancer diagnoses, which threshold would you choose?”
If you said 0.2 you are correct! At this threshold, your TP rate is 100% which means you’re capturing/predicting all actual cancer diagnoses. Your FP rate is ~0.33 which means your model is misclassifying some non-cancer diagnoses as having cancer but that’s alright. Scaring someone and having them spend the money to get tested carries fewer consequences than telling someone they don’t have cancer when they do.
Now the ROC simply connects each point to help with visualizing threshold move from very conservative to more lenient. Finally, the ROC curve helps with visualizing the AUC.
So what is the Area Under the Curve (AUC)?
You will often see a ROC graph with many ROC curves and each curve is a different classifier (ie. log regression, SVC, decision tree, etc.). AUC is a great simple metric that provides a decimal number from 0 to 1 where the higher the number the better is the classifier. AUC measures the quality of the model’s predictions across all possible thresholds. In general, AUC represents the probability that a true positive and true negative data points will be classified correctly.
# Adjusting the threshold down from 0.5 to 0.25# Any data point with a probability  of 0.25 or higher will be #  classified as 1. clf = LogisticRegression()clf.fit(X_train, y_train)THRESHOLD = 0.25y_pred = np.where(clf.predict_proba(X_test)[:,1] >= THRESHOLD, 1, 0)y_pred_proba_new_threshold = (clf.predict_proba(X_test)[:,1] >= THRESHOLD).astype(int)
Logistic regression does not have a built-in method to adjust the threshold. That said since we know by default the threshold is set at 0.50 we can use the above code to say anything above 0.25 will be classified as 1.
I hope I was able to help clear up some confusion when it comes to classification metrics. I found that keeping all the terms (ie. recall, precision, TP, TN, etc.) clear in your mind is easier when they are memorized using one particular example (ie. cancer diagnoses).
Feel free to offer your feedback and thanks for reading.