Metrics Matter
By: Jamil Mirabito
Tags: Predictive Analytics, Statistics, Evaluation, Social Science, Machine Learning

Specificity, also known as the True Negative Rate, is the opposite of Recall. Whereas recall told us how well the model predicts a positive outcome, specificity tells us how well the model can predict a negative outcome. Let’s say roughly 500 students did not actually graduate and the model predicted that 250 of them would not graduate. This would return a specificity score of 50 percent. In this case, in particular, we might actually prioritize specificity optimization since we might care more about correctly predicting those students who would not graduate from college within six years so that we could better target additional supports to ensure that they do.

F1-Score

The harmonic average of precision and recall. 2 * [ (Recall * Precision) / (Recall + Precision) ]

Generally, F1 Score is a good high-level metric to use as it encompasses both Precision and Recall. Ideally, one would optimize the F1 score if they needed a fair balance of both Precision and Recall. For example, in predicting college graduation, we would like to create a model that accurately predicts positive outcomes and captures the totality of actual positive outcomes as we described above. F1 score is also a good metric to use when there is substantial class imbalance. In optimizing the F1 score, one can expect to increase how well the model can capture positive outcomes. Like precision and recall, the F1 score is optimized when the score approaches one.

ROC (Receiver Operating Characteristic)/ AUC (Area Under the Curve)

The ROC curve shows how the true positive rate (recall/sensitivity) compares to the false positive rate (specificity). An optimal ROC curve is one where the area under the ROC curve (AUC) approaches one. This score can be interpreted as the probability that the model ranks a positive prediction more highly than a random negative prediction.

Image by author.

Like F1 score, AUC is useful as a single-number classifier of performance and is useful even when high class-imbalance is present. AUC is also scale-invariant meaning that one does not need to scale their data in order to evaluate the model’s performance. One can keep all features in their original metrics. Although, if optimizing for performance, scaling one’s data is best practice.

AUC is also classification-threshold-invariant meaning that the AUC can be used as an effective measure of performance, regardless of which decision threshold is selected. However, if understanding how this threshold affects precision and recall is necessary for improving the model, F1 score might serve as a better metric for model evaluation.

Selecting the Best Evaluation Metric

When deciding how to evaluate a model, one must think about the question they are trying to answer and whether they would like to prioritize how well their model captures positive outcomes or how well their model captures negative outcomes. In the example above, I might optimize the model based on the specificity score given my interest in predicting who is not likely to graduate from college in six years. Alternatively, if someone were trying to predict who might be at risk of credit card default, they might optimize their model on the recall score or the F1 score so as to decrease the likelihood of missing a positive case. In this vein, how one evaluates their model depends on the use case of the model.