Accuracy on Imbalanced Datasets and Why, You Need Confusion Matrix!
By: Joydwip Mohajon
Tags: Machine Learning, Imbalanced Data, Data Science, Confusion Matrix

Confusion Matrix

If you have an imbalanced dataset to work with, it’s always better to use confusion matrix as your evaluation criteria for your machine learning model. It gives you a very simple, yet efficient performance measures for your model. It also helps to provide a clear idea if the model you have created is performing well on your primary goal. First let’s see how a confusion matrix looks like, then I will tell you what I meant by performing well on your primary goal.

Confusion Matrix

True Positive (TP): It refers to the number of predictions where the classifier correctly predicts the positive class as positive.

True Negative (TN): It refers to the number of predictions where the classifier correctly predicts the negative class as negative.

False Positive (FP): It refers to the number of predictions where the classifier incorrectly predicts the negative class as positive.

False Negative (FN): It refers to the number of predictions where the classifier incorrectly predicts the positive class as negative.

This matrix will give you more insights as to why just using accuracy as a performance measure for your model doesn’t give you the whole picture. Now let’s look at different performance measures you can use by using the confusion matrix.

Precision: It tells you what fraction of predictions as a positive class were actually positive. To calculate precision, use the following formula: TP/(TP+FP).

Recall: It tells you what fraction of all positive samples are correctly predicted as positive by the classifier. It is also known as True Positive Rate (TPR), Sensitivity, Probability of Detection. To calculate Recall, use the following formula: TP/(TP+FN).

Specificity: It tells you what fraction of all negative samples are correctly predicted as negative by the classifier. It is also known as True Negative Rate (TNR). To calculate specificity, use the following formula: TN/(TN+FP).

Now, in a perfect world we’d want a model that has a precision of 1 and a recall of 1. That means a 100% accuracy which is often not the case for a machine learning model. So what we should try, is to get a higher precision with a higher recall value. But there is often a trade-off between these two metrics. There is a general inverse relation between these two metrics. Meaning if you increase one, more often than not you will decrease the other. Let’s look at the following examples and remember when I said “performing well on your primary goal”, okay let’s talk about that now.

High Precision, Lower Recall

High Precision, Lower Recall

As you can see, this confusion matrix gives a perfect precision of 1.00, but at the cost of a lower recall of 0.47. We use this kind of scenario when it is significantly important to have a model that gives a lower number of FP. For example, if your model ranks the results of search query, then you need to give less mismatched results (meaning, lower FP) to your user. This kind of tasks are called precision-oriented problem. So, if your goal is to build a machine learning model where it is also important to get the negative cases right, you need to build a precision-oriented model.

High Recall, Lower Precision

High Recall, Lower Precision

As opposed to the previous example, this one gives you a perfect recall of 1.00 but a lower precision of 0.36. Now when it is important to have a model that gives a lower number of FN, then we use such scenario. An example of such case would be if your model tries to detect a disease in patients, then it is far more important to minimize negative prediction (meaning, lower FN) that are actually positive cases. This kind of tasks are called recall-oriented problem. So, if your goal is to build a machine learning model that is mainly focused on getting the most positive cases right, you need to build a recall-oriented model.

Finally, I hope you have a clear idea now to why use confusion matrix rather than just taking a typical accuracy percentages for machine learning models. Thanks for reading.