Getting Acquainted with Confusion Matrix
By: Harsh Kumar
Tags: Precision, Machine Learning, Data Science, Recall, Confusion Matrix

What is Confusion Matrix?

In simple words, a confusion matrix, also known as an error matrix, is a summary of prediction results on a classification problem. It is a mechanism by which we can get insights on how well our classification model works, where it is right and where and what type of errors does the model make. Confusion matrix can be described as a table with different combinations of predicted and actual values. The table compares the values predicted by our model with the actual values. Below we can see how a confusion matrix looks like:

Pretty awesome, isn’t it? Err.. Hang on though. True Positive, False Positive, False Negative, True Negative..What do these terms even mean? Well, have no worries because that’s exactly what we are going to look at now.

True Positive : The model predicted positive and it actually was positive. Yay!

The model predicted positive and it actually was positive. Yay! False Positive( Type I Error ): The model predicted positive and it actually was negative. Bummer!

The model predicted positive and it actually was negative. Bummer! False Negative( Type II Error ) : The model predicted negative and it actually was positive. Uh-oh!

: The model predicted negative and it actually was positive. Uh-oh! True Negative : The model predicted negative and it actually was negative, Yay again!

Let us look at a fun example to comprehend it better. Suppose you really like someone ❤ but you are not sure if they like you back. Hmm.. So now, you try to predict if they like you or like you not 🌻 . Now there can be four scenarios.

Scenario I — You predict that they like you back for sure! And… wait for it… they actually do like you back! So you both live happily ever after. Yay! 😍

( True Positive)

You predict that they like you back for sure! And… wait for it… they actually do like you back! So you both live happily ever after. Yay! 😍 Scenario II — You predict they like you back so you approach them but in reality they do not. Welp! You are left heartbroken because the prediction gave you a ray of hope but now you are just friend-zoned. 😞 ( False Positive )

You predict they like you back so you approach them but in reality they do not. Welp! You are left heartbroken because the prediction gave you a ray of hope but now you are just friend-zoned. 😞 Scenario III — You predicted they do not like you back. So, to avoid embarrassing yourself, you do not approach them. What you don’t know is that they actually do like you back. And thus, your love story could never materialize! ☹️

( False Negative )

You predicted they do not like you back. So, to avoid embarrassing yourself, you do not approach them. What you don’t know is that they actually do like you back. And thus, your love story could never materialize! ☹️ Scenario IV— You predicted they do not like you back and they actually do not like you back. So you do not approach and save yourself from embarrassment. Phew! 😅 ( True Negative )

Awesome! Now that we have an idea of what a confusion matrix is and its basic structure along with its key terminologies, we should dive a bit deeper! Shall we?

Understanding different metrics — Accuracy, Precision, Recall and F1-Score

When we develop a model, we make sure to see if it is any good by looking at certain characteristics. These characteristics are the metrics with the aid of which we can determine how well our model really is! Now, Accuracy may seem to be the best metric for the purpose, right? Well, it is not always the case! Surprised much? Well, we will see how in the next section with the help of a demonstration. For now, let us discover accuracy and some hidden gems of metrics in detail.

Accuracy: Accuracy is simply the ratio of the correct number of predictions to the total predictions.

where TP = True Positives, TN = True Negatives, FP = False Positives and FN = False Negatives

Precision: This basically tells us , how often the model was right when it predicted positive. That is, how many were Actually Positive out of all the predicted Positive classes.

Recall: Recall is another metric that tells us how often did the classifier get it right when the class was Actually Positive. In other words, it calculates how many of the Actual Positives our model recorded by marking it as Positive.

F1-Score: It is a function of Precision and Recall. It is an overall measure of a model’s accuracy that combines Precision and Recall. It is basically the harmonic mean of both. Mathematically,

A good question to ask here is what is the essence of F1-Score and how does it differ from Accuracy? Well, we will see it at the end of the very next section. So..

If you still did not get much hang of it, you surely will after we go through a demonstration. Okay then! Let us continue.

Understanding it better with a demonstration

Let us take another example in which a hospital houses a total of 100 patients. 9 patients are infected with a highly contagious virus (a sad 2020 reference). The rest of the 91 do not have the virus. These are the facts, the reality. We develop a model that predicts the infected patients. We see how our model performed with the following confusion matrix.

As we can see, True Positive = 1, False Positive = 1, False Negative = 8 and True Negative = 90. In the first glance, a person would say this model is great since Accuracy = (TP + TN) / (TP + FP + TN + FN). So for our model, that would be equal to 91%. Awesome! But.. (yes there’s always a ‘but’, isn’t there?) If we take other metrics that we have just learnt into consideration, the picture isn’t so bright as it seems to be! We should be measuring how many positive cases we can predict correctly to contain the spread of the virus as quickly as possible. Or maybe, out of the correctly predicted cases, how many are positive cases to check the reliability of our model.

Let us consider them one by one. First, we calculate Precision. We do so by applying the formula, Precision = TP/(TP + FP) which turns out to be just 50%. This means the model predicted two Positive cases out of which one was actually Positive and the other case was not. Not very precise, eh? Well, this is not even the worst part. Let’s see about the situation with Recall, shall we?

We calculate Recall as Recall = TP/(TP+FN) and it comes out to be 11%!! If we look at the confusion matrix, we can directly see that out of the total 9 infected patients, we could detect only 1! Thus, we could detect only 11% of the total infected people. This is just bad. Very bad.

Accuracy only shows us how good we predicted who didn’t get the virus that is who are healthy and free of virus while we could not predict the infected ones correctly! And because of that we cannot contain the virus quickly which is not very good as that would lead to rapid spreading of the virus!

Therefore, as we saw with the help of our demonstration, basing our model’s effectiveness solely on one metric that is accuracy could turn out to be unknowingly disastrous! We need to take other metrics into account as well to get the whole picture with absolute clarity.

With this, we conclude.. oh wait.. Aren’t we forgetting something? Yup. What really is the essence of F1-Score? How is it different from Accuracy? We are yet to explore the answers to these questions. Well, in unbalanced datasets, if we have large number of True Negatives (in this example. the non infected patients), Accuracy could be misleading. Therefore, F1-Score might be a better metric to use as it provides a balance between recall and precision in the presence of unbalanced datasets.

Hope this blog helped in making the concept of confusion matrix less confusing. Thank you for reading! Cheers and as always Happy Learning! :)