Beyond Accuracy: other Classification Metrics you should know in Machine Learning
By: Jack Tan
Tags: Accuracy, Python, Machine Learning, Classification, Data Science

Beyond Accuracy: other Classification Metrics you should know in Machine Learning

Photo by George Pagan III on Unsplash

“How accurate is your model?”

This is probably the most commonly asked question when one wants to know how a model performs or rather how accurate a classifier can actually predict an anticipated event.

While using accuracy to measure a classifier performance is not wrong, it may give you a false sense of optimism. A classifier with high accuracy tells you the overall performance of a model, but less so on a more localized level of the individual class it predicts.

What does that even mean? Or rather why accuracy alone is inadequate to evaluate the performance of a classifier? What other classification metrics out there can you use? In this article, we will explain how accuracy measures the classification results and what other metrics you can use to better gauge the performance of a classifier. Explanations will be accompanied by some snippets in Python for practical use.

The mantra of accuracy

Accuracy is the number of correctly predicted data points out of all the data points. More formally, it is defined as the number of true positives and true negatives divided by the number of true positives, true negatives, false positives, and false negatives. [1]

In a typical classification model, there might be a class that is of more interest than the other. For example, in a binomial classification problem, the positive class (the customer who will buy a product) normally weighs more than the negative class (the customer who will not buy a product), because you wanna make sure your marketing campaign reach the former group of customers so that they will indeed buy in the end. In this scenario, the real-world data collected to train and test the model is very often imbalanced, that means the class of interest constitutes just a small portion of the entire dataset (persons who actually buy are normally less than those who do not buy).

Accuracy measures the entire dataset with no direct regard to the imbalanced distribution and can create an “over-optimistic” interpretation of the classification results. For example, in a two-class problem with a class distribution of 90:10, the accuracy on majority-class examples will count nine times as much as the performance on minority-class examples. In other words, even if the classifier correctly predicts all the majority-class but wrongly for all the minority-class, you will still get a 90% accuracy!

But first, let’s talk about Confusion Matrix

A confusion matrix is a specific table layout that allows visualization of the performance of an algorithm, with two dimensions (“actual” and “predicted”), and identical sets of “classes” in both dimensions. [2]

Self-illustrated by the author.

A true positive is an outcome where the model correctly predicts the positive class. Similarly, a true negative is an outcome where the model correctly predicts the negative class. [3]

Naturally, we would like a classifier to maximize its ability to correctly predict test data that would fall into these two quadrants (green zones) in a confusion matrix, i.e. when the model can correctly predict both the positive and negative classes.

Now let’s visualize accuracy again with a confusion matrix, and you will quickly realise why accuracy is not the best evaluation metrics in binary classification, more so when the data are imbalanced.

Self-illustrated by the author.

As illustrated, the classifier did not correctly predict any of the positive class but still has an 80% accuracy!

Classification Metrics Beyond Accuracy

Photo by Priscilla Du Preez on Unsplash

As seen in a confusion matrix, each component represents a different scenario and can be interpreted differently to derive metrics other than accuracy. The most common ones are Precision, Recall, and F1-score. You probably know the literal meaning of these terms but how are they different in evaluating classification performance?

Precision — is used to measure the positive patterns that are correctly predicted from the total predicted patterns in a positive class.[4] Recall —also known as Sensitivity, is used to measure the fraction of positive patterns that are correctly classified. [4 ] For negative class, it is known as Specificity. F1-score — this metric represents the harmonic mean between Recall and Precision values, sometimes also known as F-measure. [4]

Thus, as seen in the definition, these metrics give us more specifics and information than Accuracy. Precision checks how precise the predictions are; Recall checks how many patterns in a given class are correctly identified; F1-score computes the harmonic average between these two values.

By using the components in a confusion matrix, the formulae for these metrics are defined as in the following illustration.

Self-illustrated by the author.

Examples in Python

Luckily the well-known Machine Learning imbalanced-learn package has a Metrics module which is equipped with methods for confusion matrix, so we just need to apply this method to our classification results.

For demonstration purposes, the dataset used in this tutorial is randomly generated using make_classification. The data is split into 70:30 ratio for training and test set while the classifier used is Random Forest. The complete Python codes written in Colaboratory are hosted on the Github repository.

Below is a “one-pager” snapshot of the codes:

Line 1–11: Classification data are created. The data are imbalanced with a 90:10 ratio. The flip_y parameter is set at 0.3 to introduce some noises in the data to make the classification task harder.

parameter is set at 0.3 to introduce some noises in the data to make the classification task harder. Line 13–16: Train-test split the data in 70:30 ratio.

Line 18–25: Create a Random Forest classifier.

Line 27–41: Evaluate the classifier with test data using confusion matrix and classification report.

The printed output for confusion matrix and classification report

The overall accuracy for this classifier is 83%, which is a pretty good value. Well, that’s only half of the picture. If we look at the minority positive class 1 in the classification report, the precision rate is 72% (12% less than the majority negative class 0) and the recall rate is only 29% (while 97% for the majority negative class 0)! That means the classifier performs relatively well for class 0 but not for class 1. This is understandable as the data are imbalanced (as in many real-life applications), thus the classifier gets to train more on samples found in the majority-class but less so on the minority-class.