A Non-Confusing Guide to Confusion Matrix | by Dario Radečić | Towards Data Science

You are hyped about machine learning. You don’t know much about statistics, but you know what the mean is. You’ve decided to skip all the prerequisites and train your first machine learning model. You’ve followed a code-along video tutorial by some Indian on YouTube. After spending 10 hours realizing you’ve missed an indentation on IF statement, your model is finally trained.
You’ve obtained above 90% accuracy. Life is good.
But I’m here to tell you, your model most probably sucks. And that’s expected, heck, that’s great!
Jokes aside, model evaluation can be difficult when you are a novice in data science and machine learning. The confusion matrix is something that confuses you, and that’s expected. You won’t believe how many stuff you can pull from stupidly simple-looking 2x2 matrix.
Here’s an example of a confusion matrix obtained after model training:
Is it good? Is it bad? The answer is your favorite — it depends (results are horrible in this case).
Here’s a more generic-looking confusion matrix:
Yeah, I know what you are thinking — good job, moron, now I’m even more confused! But hold your horses, let’s break it down into smaller, more understandable chunks.
Values from the dataset (ones from the target variable)
Values your model predicted
Value in the target variable
If predicting disease, this would be ‘Has a disease’
1 in the binary target variable
Value in the target variable
If predicting disease, this would be ‘Doesn’t have a disease’
0 in the binary target variable
Okay, that was easily understandable, but what do those true positives and negatives mean?
Good question reader. I was wondering when you will ask that. Explanations of those will be very easy for you to read. But they will be even easier to forget, so you might want to re-read them every couple of days. Or even better, write them down on a piece of paper. Writing something down will make it much easier to stick. But I’m getting off point here, we were talking about false positives and true negatives, let’s dive into those.
Your model predicted positive and the actual is positive
Would be 1 in the binomial variable
Eg. you predicted that the patient has a disease and he has it
Your model predicted positive and the actual is negative
The model predicted 1, but it’s 0 in the binomial variable
Eg. you predicted that the patient has a disease but he doesn’t have it
Type I Error
Your model predicted negative and the actual is positive
The model predicted 0, but it’s 1 in the binomial variable
Eg. you predicted that the patient doesn’t have the disease, but he has it
Type II Error
Your model predicted negative and the actual is negative
The model predicted 0, and it’s 0 in the binomial variable
Eg. you predicted that the patient doesn’t have the disease, and he doesn’t have it
You see, there isn’t anything confusing about confusion matrix. Sure, you need to memorize those terms, and yeah, they sound similar, but at least they aren’t as abstract as some other machine learning concepts you’ve probably encountered.
After reading all of that stuff about positive and negatives (a couple of times preferably), you now have a basic idea and intuition about confusion matrix, and you see that it’s not that confusing after all — it just needs to “sink in” properly.
But is that all about confusion matrix?
I hope you’re kidding. We’re only about halfway through. Maybe.
Up next I want to discuss various scorings that can be obtained from the confusion matrix. And there are a lot of them. Here’s the list of all of them, according to Wikipedia[1]:
Sensitivity / Recall / Hit Rate / True Positive Rate (TPR)Specificity / Selectivity / True Negative Rate (TNR)Precision / Positive Predictive Value (PPV)Negative Predictive Value (NPV)Miss Rate / False Negative Rate (FNR)Fall-out / False Positive Rate (FPR)False Discovery Rate (FDR)False Omission Rate (FOR)Threat Score / Critical Success Index (CSI)Accuracy (ACC)F1 ScoreMatthews Correlation Coefficient (MCC)Informedness / Bookmaker Informediness (BM)Markedness (MK)
Sensitivity / Recall / Hit Rate / True Positive Rate (TPR)
Specificity / Selectivity / True Negative Rate (TNR)
Precision / Positive Predictive Value (PPV)
Negative Predictive Value (NPV)
Miss Rate / False Negative Rate (FNR)
Fall-out / False Positive Rate (FPR)
False Discovery Rate (FDR)
False Omission Rate (FOR)
Threat Score / Critical Success Index (CSI)
Accuracy (ACC)
F1 Score
Matthews Correlation Coefficient (MCC)
Informedness / Bookmaker Informediness (BM)
Markedness (MK)
Feeling confused? That’s expected. I am also. I’ve never heard for some of those terms, but I still wanted to put them here, just to prove that you can understand everything confusion matrix has to offer without you know, going insane.
But it’s amazing to think about how you can calculate so many different scorings just from this plainly stupid-looking 2x2 matrix. I mean that are only 4 numbers, for God sake!
For the second part of this article, I want to drill down to most commonly used scorings in the realm of machine learning, and those are:
Accuracy
Recall
Precision
F1 Score
That’s right, I’ll cover only 4 of 14 because I feel that those are most important and that you’ll be perfectly capable of optimizing your model with just basic understanding of those. You don’t need to memorize the formula, those are already built-in to Python’s Scikit-Learn, but you should know when to choose one over the other.
Before diving right in, let’s see the confusion matrix you’ll be working on:
And to boil it down, in the given matrix we have:
True Negatives (TN) — 1943
False Positives (FP) — 32
True Positives (TP) — 181
False Negatives (FN) — 344
As you might have noticed, this is the same confusion matrix from the beginning of the article, just more nicely presented. Without further ado, let’s dive into the scorings!
Most intuitive to understand — a ratio of correctly predicted observation to the total observations
Not well suited for most business needs
The calculation is straightforward: all true instances are divided by the number of total instances
(TP+TN) / (TP+FP+FN+TN)
Implementation in Python:
from sklearn.metrics import accuracy_scoreprint(accuracy_score(y_true, y_pred))>>> 0.8496
The ability of a model to find all the relevant cases within a dataset
For our example, it would be the ability of the model to find all cases where the patient has the disease
The calculation is straightforward: divide True Positives (TP) by the sum of True Positives (TP) and False Negatives (FN)
TP / (TP + FN)
Python Implementation:
from sklearn.metrics import recall_scoreprint(recall_score(y_true, y_pred))>>> 0.3448
The ability of a model to identify only the relevant data points
For our example, it would be the ability of the model to correctly classify patients that don’t have the disease
Calculation: divide True Positives (TP) by the sum of True Positives (TP) and False Positives (FP)
TP / (TP + FP)
Implementation in Python:
from sklearn.metrics import precision_scoreprint(precision_score(y_true, y_pred))>>> 0.8498
The weighted average of Precision and Recall.
Takes both false positives and false negatives into account
You have low False Positives (FP) and low False Negatives (FN) — correctly identifying real threats and you are not disturbed by false alarms
The calculation needs both Recall and Precision previously calculated
2*(Recall * Precision) / (Recall + Precision)
Implementation in Python:
from sklearn.metrics import f1_scoreprint(f1_score(y_true, y_pred))>>> 0.4905
Those explanations should give you a clear picture that using accuracy as a scoring metric isn’t always a good option. In our case with patients and diseases, it would be better to use recall, because you probably want to correctly identify each patient which has a disease — just think how horrible would it be to have a large number of misclassifications in cancer detection.
And that’s right about enough materials to keep you busy for some time. When this becomes easy to you, please refer to the Wikipedia article (posted at the bottom of the article) to learn a few more scorings.
My intention was to break down this stupidly simple-looking matrix in a non-confusing way because it can get very confusing when you encounter it for the first time.
I certainly didn’t say all there is about the topic, but every key idea and concept is here, waiting to be learned. It’s so easy to forget this stuff and trying to learn it in a single day will only make those terms to mix up even further. Slow and steady wins the race.
Yes, most of the terms are called similarly and you will be confused by it — and that is perfectly normal. Review the materials every now and then, from whichever article you want.
What are your thoughts? Is there anything you don’t understand after a couple of readings? Feel free to drop a comment below.
Loved the article? Become a Medium member to continue learning without limits. I’ll receive a portion of your membership fee if you use the following link, with no extra cost to you.