How to Use Confusion Matrix For Problem-oriented Model Tuning | Towards Data Science

Master the fundamentals of the confusion matrix using Sklearn and build a practical intuition for 3 of the most common metrics used in binary classification: precision, recall, and F1 score.
Classification is a huge part of machine learning. Its benefits and applications are endless — ranging from detecting new asteroids and planets to identifying cancerous cells are all done using classification algorithms.
The type of problems classification solves is divided into two: unsupervised and supervised. Unsupervised models are called neural networks, and they can be trained on unstructured data such as video, audio, and images. In contrast, supervised models work with labeled, tabular data and are part of classic machine learning. The focus of this article is the latter; particularly, we will be exploring what all supervised classification problems have in common: confusion matrices.
A good model needs good data. So, it is important to process the available information as much as possible to achieve the best model performance even before tuning it based on confusion matrices.
A typical preprocessing workflow includes dealing with missing values, scaling/normalizing numeric features, encoding categorical variables, and do all other feature engineering steps required. We will see an example of this in this section.
We will be predicting credit card approvals using the Credit Card Approval Dataset from UCI Machine Learning Repository. Before banks can issue credit cards to new customers, there are many factors to consider: income levels, loan balances, individual credit reports, etc. This is often a difficult and mundane task, so nowadays, banks use ML algorithms. Let’s take a peek at the data:
Since it is private data, the feature names are left blank. Let’s fix that first:
The dataset contains both numeric and categorical features. The missing values in this dataset are encoded with question marks (?). We will replace them with NaNs:
Features 0, 1, 3, 4, 5, 6 and 13 contain missing values. Inspecting the data, we may guess that feature 13 contains zip codes, which means we can drop it. And for others, since they make up less than 5% of the dataset, we can drop those rows as well:
We did not use any imputation techniques because of the low number of nulls. If you want to know about other, robust imputation techniques, this article will help:
towardsdatascience.com
Let’s focus on numeric values now. Specifically, we will look at their distributions:
>>> df.describe().T.round(3)
All features have a minimum of 0, but they are all on different scales. This means we have to use some normalization, and we will see what kind by exploring these features visually:
import seaborn as sns>>> df.hist();
Features have skewed distributions which means, we will perform non-linear transform such as PowerTransformer (uses logarithms under the hood):
If you want to know more about other numeric feature transformation techniques, I got that covered too:
towardsdatascience.com
To encode categorical features, we will use a OneHotEncoder. Before isolating the columns to be used in encoding, let's separate the data into feature and target arrays:
Now, isolate the categorical columns to be OH encoded:
Finally, we will build the preprocessing pipeline:
towardsdatascience.com
At the last step, I added a RandomForestClassifier to the pipeline as a base model. We want the model to predict approved applications more accurately because that would mean more customers to the bank. This would also make the approved applications a positive class in our predictions. Let's finally evaluate the pipeline:
The default scoring of all classifiers is the accuracy score, which our base pipeline impressively achieved ~87%.
But, here is the problem with accuracy — what is the model accurate at? Can it predict suitable applications better, or is it more accurate at detecting undesirable candidates? From a business perspective, your results should answer both questions, and accuracy does not do that.
As a solution, let’s finally get introduced to a confusion matrix:
Since it is a binary classification problem, the matrix is of shape 2 by 2 (two classes in the target). The diagonal of the matrix shows the number of correctly classified samples, and the off-diagonal cells show where the model made a mistake. To understand the marix, Sklearn provides a visual one, which is much better:
This confusion matrix is much more informative. Here are a few things to notice:
The rows correspond to the actual values
The columns correspond to the predicted values
Each cell is a count of each true/predicted value combination
Paying attention to the axes labels, the first row represents the actual negative class (rejected applications) while the second row is for the actual positive (approved applications) class. Similarly, the first column is for the positive predicted and the second for the negative predicted.
Before we go on to interpreting this output, let’s fix the format of this matrix. In other literature, you may see that the actual positive class is represented as the first row and the predicted positive class as the first column. I am also used to that format and find it easier to explain.
We will flip the matrix so that the first row and column are the positive class. We will also use Sklearn’s ConfusionMatrixDisplay function that plots custom matrices. Here is a wrapper function:
We are flipping the matrix using np.flip and plotting it via ConfusionDisplayFunction which only takes a matrix and accepts custom class labels through display_labels parameter.
Let’s finally interpret this matrix:
(top left) — 78 applications were actually approved, and the model correctly classified them approved as well
(bottom right) — 95 applications were actually rejected, and the model correctly classified them rejected as well
(bottom left) — 13 applications were actually rejected, but the model incorrectly classified them as approved
(top right) — 12 applications were actually approved, but the model incorrectly classified them as rejected.
Because of the popularity of confusion matrices, each true/predicted cell combination has its own name in the community:
True Positives (TP) — actual positive, predicted positive (top left, 78)
True Negatives (TN) — actual negative, predicted negative (bottom right, 95)
False Positives (FP) — actual negative, predicted positive (bottom left, 13)
False Negatives (FN) — actual positive, predicted negative (top right, 12)
Even though you may see a matrix in a different format, the above 4 terms will always be there. That’s why before creating a model, it is useful to create a mental note of what the above 4 terms refer to in your unique case.
After you fit a model, you can extract each of the above 4 using the .ravel() method on a confusion matrix:
In this section, we will learn about metrics that let us further compare one confusion matrix to another. Let’s say we have another pipeline with LogisticRegression as a classifier:
Looking at the plots above, we might say the results of both Random Forests and Logistic Regression are similar. However, there are 3 common metrics we can derive from the confusion matrix that lets us compare them. These are called precision, recall, and F1 score. Let’s understand each one in detail:
Precision: it is the ratio of the number of correctly classified positives and the total number of predicted positive classes. In our case, it is the total number of correctly classified, approved applications (TP = 77) divided by the total number of predicted approved classifications (all predicted positives, regardless of whether they are correct or not, TP + FP = 94). Using the matrix terms, it is:
Precision: it is the ratio of the number of correctly classified positives and the total number of predicted positive classes. In our case, it is the total number of correctly classified, approved applications (TP = 77) divided by the total number of predicted approved classifications (all predicted positives, regardless of whether they are correct or not, TP + FP = 94). Using the matrix terms, it is:
You can easily remember this with the Triple-P rule — precision involves all positives and uses the terms on the left side of the matrix.
Sklearn’s official definition of precision is: “the ability of the classifier not to label a negative sample as positive.” In our case, it is the ability of our model not to label rejected applications as approved. So, if we want the model to be more accurate at filtering unsuitable applications, we should optimize for precision. In other words, increase True Positives and decrease False Positives as much as possible. 0 False Positives gives a precision of 1.
Recall: also called sensitivity, hit rate, or true positive rate (TPR). It is the ratio of correctly classified positives divided by the total number of actual positives in the target. In our case, it is the number of correctly classified, approved applications (TP = 77) divided by the total number of actual approved applications (regardless of whether they were correctly predicted or not, TP + FN = 90). Using the terms of confusion matrix:
Recall: also called sensitivity, hit rate, or true positive rate (TPR). It is the ratio of correctly classified positives divided by the total number of actual positives in the target. In our case, it is the number of correctly classified, approved applications (TP = 77) divided by the total number of actual approved applications (regardless of whether they were correctly predicted or not, TP + FN = 90). Using the terms of confusion matrix:
Recall uses the terms in the first row of the confusion matrix.
Sklearn’s official definition for recall is: “the ability of the classifier to find all the positive samples.” If we optimize for recall, we will decrease the number of False Negatives (incorrectly classified, approved applications) and increase the number of True Positives. But this may be at the cost of increasing False Positives — i. e. incorrectly classifying rejected applications as approved.
Due to their nature, Precision and Recall are in a trade-off relationship. Depending on your business problem, you may have to focus on optimizing one at the cost of the other. However, what if you wanted to have a balanced model — i.e., a model that is equally good at detecting the positives and the negatives.
In our case, this would make sense — a bank would benefit the most if it could find as many customers as possible while also avoiding undesirable candidates, thus eliminating potential loss.
The third metric, called the F1 score, tries to measure exactly that: it quantifies the model’s ability to predict both classes correctly. It is calculated by taking the harmonic mean of precision and recall:
The third metric, called the F1 score, tries to measure exactly that: it quantifies the model’s ability to predict both classes correctly. It is calculated by taking the harmonic mean of precision and recall:
Why the harmonic mean, you ask? Because of how it is calculated, the harmonic mean truly gives a balanced score. If either precision or recall has a low value, the F1 score suffers significantly. This is a useful mathematical property compared to the simple mean.
All these metrics can be calculated using Sklearn, and they are available under the metrics submodule:
RandomForest has a greater precision indicating that it is better at finding approvable applications while lowering false positives, i. e. incorrectly classified undesirable candidates.
RandomForests wins in recall too. It is similarly better at filtering out false negatives, i. e., decreasing the number of positive samples classified as negative. Since RandomForests won in both scores, we can expect it to have a higher F1 too:
As expected, RF has a higher F1 making it a more robust model for our case.
You can print out all these scores at once using the classification_report function:
Before we focus on optimizing for these metrics, let’s take a look at other scenarios to deepen our understanding.
Since the differences between these metrics are subtle, you need some practice to develop a strong intuition for them. In this section, we will do just that!
Let’s say we are trying to detect whether parachutes sold in the sky-diving shop are faulty or not. The assistance examines all available parachutes and records their attributes, and classifies them. We want to automate this process and build a model that should be exceptionally good at detecting faulty parachutes.
For example purposes, we will create the dataset synthetically:
Since there are much more working parachutes, it is an imbalanced classification problem. Let’s set up the terminology:
The positive class: faulty parachutes
The negative class: working parachutes
True positives: faulty parachutes predicted correctly
True negatives: working parachutes predicted correctly
False positives: faulty parachutes predicted incorrectly
False negatives: working parachutes predicted incorrectly
Let’s evaluate a Random Forest model on this data:
In this problem, we should try to lower the top right (False Negatives) as much as possible because even one faulty parachute means the death of a skydiver. Looking at the scorings, we should be optimizing the recall score:
It is perfectly OK if False positives increase because we will be saving people’s lives even though we might lose some money.
In the second scenario, we will try to predict customer churn (stop or continue using our company’s services). Again, let’s set up the terminology for the problem:
Positive class: continue using
Negative class: churn
True positives: wants to continue, predicted correctly
True negatives: churn, predicted correctly
False positives: churn, predicted incorrectly
False negatives: wants to continue, predicted incorrectly
We will build a synthetic dataset again and evaluate RF on it:
In this case, we want to retain as many customers as possible. This means we have to lower False Positives indicating that we should optimize precision:
In this article, we only focused on 3 metrics. However, you can derive many other scores from the confusion matrix, such as specificity, NPV, FNR, etc. After reading this article, you should be able to read the Wikipedia page on the topic. If you find yourself confused about the metrics themselves, check out this awesome article too.
Finally, let’s see how to optimize for each of the metrics we discussed today.
In this section, we will see how to boost a model’s performance for a metric of our choice. In the above sections, we used models with default parameters. To increase their performance, we have to do hyperparameter tuning. Specifically, we should find the set of hyperparameters that gives the highest score for our desired metric.
Searching for this magical set is tedious and time-consuming. So we will bring out the HalvingRandomSearchCV class which searches over a grid of possible parameters for the model and find the set that gives the highest score for the scoring function passed to its scoring parameter.
You might be surprised that I am not using GridSearch. In one of my articles I showed how Halving Grid Search is 11 times faster than regular GridSearch. And Halving Grid Search is even faster, allowing us to widen our hyperparameter space to a great extent. You can read the comparison here:
towardsdatascience.com
As a first step, we will build the hyperparameter space:
Now, we will search over this grid 3 times, optimizing for each of the metrics we discussed today:
We managed to score higher in terms of precision but got a lower score for recall and F1. It is an iterative process, so you can continue the search until the scores improve. Or, if you have time, you can change to HalvingGridSearch, which is much slower than HalvingRandomSearch but gives much better results.
The hardest part of any classification problem is truly understanding the business problem you are trying to solve and optimize for a metric accordingly. Once you theoretically construct the appropriate confusion matrix and its terms, only the coding part is left.
In terms of coding, having a good preprocessing pipeline ensures that you have the best score possible for a base model of your choice. Make sure to scale/normalize the data based on the underlying distributions. After preprocessing, create copies of your pipeline for multiple classifiers. LogisticRegression, Random Forests, KNN Classifier are good choices.
For optimization, choose either Halving Grid or Halving Random Search. They have been proven to be much better than their previous counterparts. Thanks for reading!
Confusion matrix Sklearn user guide
StatQuest on Confusion Matrix
StatQuest on Sensitivity and Specificity
Optimizing a Machine Learning Model with the Confusion Matrix
Going Beyond the SimpleImputer For Missing Data Imputation
How to Perform Operator Overloading in Python OOP
Practical Sklearn Feature Selection in 3 stages