Is Your Classifier Any Good?. Don’t fall for the wrong evaluation... | by Frank Zickert | Quantum Machine Learning | Towards Data Science

This post is part of the book: Hands-On Quantum Machine Learning With Python
In this post, we aim to predict whether a passenger survived the Titanic shipwreck. But rather than training a sophisticated model, we don’t even look at the data. This way, we discover how easy it is to produce seemingly good results that are, in fact, worthless.
Predicting survival on board the Titanic is a classification task since there are distinct outcome values. Specifically, it is a binary classification task because there are exactly two possible predictions (survived or died).
Before we develop a sophisticated machine learning algorithm, let’s implement the simplest algorithm we can imagine. A classifier that guesses.
We import the random number generator (line 1) and initialize it (line 2).
Our classifier is a function that takes passenger data as input and returns either 0 or 1 as output. Similar to our data, 0 indicates the passenger died and 1 the passenger survived.
In order to use the classifier, we write a Python function that runs our classifier for each item in the training set.
This function takes the classifier-function as the first argument (we can replace the classifier later) and the input data (as x) as the second parameter (line 1).
It uses Python’s map function to call the classifier with each item in x and return an array of the results.
Let’s run it.
[0, 1, 0, ... 0, 1, 1]
When we run the classifier with our train_input we receive a list of predictions.
Since our goal is to predict the actual result correctly, we need to evaluate whether the prediction matches the actual result.
Let’s have a look at the accuracy of our predictions.
350 correct predictions out of 711. Accuracy 49 %
We define another function evaluate. It takes the predictions of our algorithm and the actual results as parameters (line 1).
The term list(zip(predictions,actual)) (line 4) creates a list of 2-item lists. The 2-item lists are pairs of a prediction and the corresponding actual result.
We filter these items from the list where the prediction matches the actual result (lambda item: item[0] == item[1]) (line 3). These are the correct predictions. The length of the list of correct predictions divided by the total number of passengers is our Accuracy.
Great! We are already correct in half of the cases (more or less). This is not a surprise when guessing one out of two possible labels.
But maybe we can do even better? I mean without any effort. We know that more people died than survived. What if we always predicted the death of a passenger?
437 correct predictions out of 711. Accuracy 61 %
We’re up to an accuracy of 61% of our predictions. Not too bad, is it? This value, that is the ratio between the two possible actual values is the prevalence.
Let’s consider a different task for a moment. Let’s say you’re a doctor and your task is to predict whether a patient has cancer. Only 1% of your patients actually have cancer. If you predicted no-cancer all the time, your accuracy would be astonishing 99%! But you would falsely diagnose the patients that actually have cancer. And for the resulting lack of treatment, they’re going to die.
Maybe the accuracy of the predictions alone is not a good measure to evaluate the performance of our algorithm.
The evaluation is one main part of every machine learning algorithm. It may seem trivial at first sight. Yet, deciding on the right measure is a very important step. When you optimize your algorithm towards better performance, you will inevitably optimize towards better scores in your evaluation function.
But for now, we keep it simple. A better way to evaluate the performance of a classifier is to look at the confusion matrix.
The general idea is to compare the predictions with the actual values. In binary classification, there are two possible actual values: true or false. And there are two possible predictions: true or false.
There are four possibilities:
True Negatives (TN): a passenger who died was correctly predicted
False Positives (FN): a passenger who died was wrongly predicted to survive
False Negatives (FP): a passenger who survived was wrongly predicted to die
True Positive (TP): a passenger who survived was correctly predicted
Let’s have a look at the confusion matrix of the predict death classifier.
array([[437,   0],       [274,   0]])
Scikit-Learn provides the confusion_matrix method that we import (line 1). It takes the actual values as first and the predictions as the second parameter (line 3).
It returns a two-dimensional array. In the first row, it shows the true negatives (TN) and the false positives (FP). In the second row, it shows the false negatives (FN) and the true positives (TP).
We can define the accuracy we measured thus far as:
It does not care whether there is a systematic error, such as the algorithm’s inability to correctly predict a passenger who survived (true positives) as long as it performs well at correctly predicting passengers who die (true negatives).
The confusion matrix offers us more detailed measures of our classifier performance. These are:
precision
recall
specificity
negative predictive value (NPV)
The precision is the “accuracy of the positive predictions”. It only looks at the positive predictions, these are predictions that the passenger survived.
Let’s have a look at the code:
The precision score of the predict_death classifier is 0.0
Scikit-Learn provides a function to calculate the precision_score. It takes the list of actual values and the list of predicted values as input.
Since we did not have a single positive prediction, our precision is not defined. Scikit-Learn interprets this as a score of 0.0.
The recall is the “accuracy of the actual positives”. It only looks at the actual positives.
In Python, it is:
The recall score of the predict_death classifier is 0.0
This time, even though recall is defined (the number of actual positives is greater than 0), the score is 0.0 again, because our classifier did not predict a single survival correctly. Not a surprise when it always predicts death.
The specificity is the “accuracy of the actual negatives”. It only looks at actual negatives (deaths).
And the ‘negative predictive value’ (NPV) is the “accuracy of the negative predictions”.
These two functions are not provided out of the box. But with the values we get from the confusion matrix, we can calculate them easily:
The specificity score of the predict_death classifier is 1.00The npv score of the predict_death classifier is 0.61
The function specificity takes the confusion matrix as a parameter (line 1). It divides the true negatives (matrix[0][0]) by the sum of the true negatives and the false positives (matrix[0][1]) (line 2).
The function npv takes the confusion matrix as a parameter (line 4) and divides the true negatives by the sum of the true negatives and the false negatives (matrix[1][0]).
These four scores provide a more detailed view of the performance of our classifiers.
Let’s calculate these scores for our random classifier as well:
The precision score of the random classifier is 0.40The recall score of the random classifier is 0.52The specificity score of the random classifier is 0.51The npv score of the random classifier is 0.63
While the predict death classifier exhibits a complete absence of precision and recall, it has perfect specificity and reaches an NPV score that matches the percentage of negatives in our test dataset (the prevalence).
The random classifier yields more balanced scores. You’ll get a little bit different scores every time you run the classifier. But the values seem to stay in certain ranges. While the precision of this classifier is usually below 0.4 the npv is above 0.6.
The confusion matrix and related measures give you a lot of information. But sometimes, you need a more concise metric. In fact, the evaluation function in a machine learning algorithm must return a single measure it can optimize.
And this single measure should unmask a classifier that does not really add any value.
This post is part of the book: Hands-On Quantum Machine Learning With Python.
Get the first three chapters for free here.