Optimising a Machine Learning Model with the Confusion Matrix | by Rebecca Vickery | Towards Data Science

The confusion matrix, in machine learning, is a grid of values that help to evaluate the performance of supervised classification models. From this grid, you can also compute a number of metrics to give a score for the model these include precision, recall and the F1-score. Although on the surface this grid is quite simple, and the measures require only high-school level mathematics to calculate the concepts behind the matrix can be difficult to grasp.
In the following post I will give a simple introduction to the following:
The Confusion MatrixPrecision, recall and the F1-scoreA walkthrough in python illustrating how to choose the right metric and optimise the performance of your model to maximise that metric
The Confusion Matrix
Precision, recall and the F1-score
A walkthrough in python illustrating how to choose the right metric and optimise the performance of your model to maximise that metric
For this explanation let’s suppose we were working on a binary classification problem to detect whether or not a transaction is fraudulent. Our model uses characteristics of the user and transaction and returns 1 if the transaction is predicted to be fraudulent and 0 if not.
Given that machine learning models are rarely 100% accurate there is going to be a level of risk in deploying this model. If we incorrectly classify a non-fraudulent transaction as fraud then we may well lose that transaction, and possibly even the future customers business. On the other hand, if we incorrectly detect a fraudulent transaction as non-fraudulent then we might stand to lose the value of that transaction.
The confusion matrix essentially places the resulting predictions into four groups. They are as follows:
True positive (TP): the model predicts fraud and the transaction is indeed fraudulent.
False positive (FP): the model predicts fraud but the transaction is not fraudulent.
True negative (TN): the model predicts not fraud and the transaction is not fraudulent.
False negative (FN): the model predicts not fraud but the transaction is in fact fraudulent.
To illustrate how a confusion matrix looks I am going to be using some manufactured data.
There are a few python libraries that make constructing confusion matrices fairly simple, below are a couple of examples;
Scikit-learn sklearn.metrics.confusion_matrix(y_true, y_pred, labels=None, sample_weight=None) creates a very simple grid.
from sklearn.metrics import confusion_matrixy_true = [0,1,1,0,1,0,0,0,1,1]y_pred = [0,1,0,0,1,0,0,1,0,1]confusion_matrix(y_true, y_pred, labels=[0,1])
The pandas_ml library produces a more readable output.
from pandas_ml import ConfusionMatrixConfusionMatrix(y_true, y_pred)
Let’s use the grid to determine the numbers for each of the 4 groups we discussed above.
There are 3 samples where the model predicted fraud and the transaction is fraudulent, so TP = 3. This is the number in the bottom right-hand corner where the row and column are both True.
There is 1 sample where the model predicted fraud but the transaction was not fraudulent, so FP = 1.
There are 2 samples where the model predicted not fraud but the transaction was fraudulent, so FN = 2.
There are 4 samples where the model correctly predicted not fraud and the transaction was fraudulent, so TN = 4.
If we were in a business and a model such as this was going to be deployed and used to drive a value outcome there are a few things that the confusion matrix can help to determine. These include:
How good is the model at determining a fraudulent transaction?
How good is the model at differentiating a fraudulent transaction and a non-fraudulent transaction?
How many times does the model incorrectly predict that a fraudulent transaction is not fraudulent?
The measures that I mentioned earlier that you can derive from the confusion matrix can help to answer these questions.
There are a number of metrics that can be derived from the confusion matrix which all measure the performance of the model. I am going to cover explanations for three of the most commonly used.
Precision: This is essentially a measure of out of all the samples predicted to be fraud how many were correct. The calculation is, therefore;
TP / (TP + FP)
Recall (sensitivity): This is a measure of for those samples that were fraudulent how many were correctly predicted as fraudulent.
TP /(TP + FN)
The difference between these two metrics is subtle but very important. If we optimise for precision then we will get fewer false positives (fewer non-fraudulent transactions classified as fraud), 0 false positives would give a perfect score of 1. If we optimise for recall then we get fewer false negatives so we are catching more fraudulent transactions but this may be at the detriment of incorrectly classifying more non-fraudulent transactions as fraud.
F1-score: is what is known as the ‘harmonic average’ of precision and recall.
The F1-score, therefore, gives a good indicator of the overall accuracy of a classifier, whilst precision and recall give information about the areas in which specifically the model is performing well or not so well. Maximising the F1 score would create a balanced model which has the optimal score for both precision and recall. Which metric you choose to optimise for will very much depend on the use case.
There are several methods that tune a machine learning model for a particular metric that suits your specific use case. One example is the scikit-learn GridSearchCV which has a scoring parameter which allows you to select a given metric. This function performs cross-validated grid-search over a parameter grid and returns the optimal parameters for the model based on the scoring metric you provide.
In the next part of this post, I will walk through an example of optimising a model using GridSearchCV . First import the relevant libraries.
import wgetimport pandas as pdimport numpy as npfrom sklearn.model_selection import train_test_splitfrom sklearn.linear_model import LogisticRegressionfrom sklearn.metrics import classification_reportfrom pandas_ml import ConfusionMatrixfrom sklearn.metrics import precision_scorefrom sklearn.metrics import recall_scorefrom sklearn.metrics import f1_scorefrom sklearn.datasets import load_breast_cancerfrom sklearn.model_selection import GridSearchCVfrom pandas_ml import ConfusionMatrix
For this example, I am going to use a simple data set which consists of attributes for a sample of patients and an indicator describing whether or not they have breast cancer. This can be obtained directly from scikit-learn using the code below.
X, y = load_breast_cancer(return_X_y=True)
In this dataset, the malignant class is actually the negative class. To make the resulting confusion matrix easier to interpret I will relabel them so that the malignant class is positive.
for i in range(len(y)):    if y[i] > 0:        y[i] = 0    elif y[i] < 1:        y[i] = 1
Next, I am splitting the data into test and train sets.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)
I will then train a simple logistic regression model and use the model to make some predictions.
rf = LogisticRegression()model = rf.fit(X_train, y_train)y_pred = rf.predict(X_test)ConfusionMatrix(y_test, y_pred)
The confusion matrix looks like this.
Let’s calculate the metrics.
print('Precision is:'+str(round(precision_score(y_test, y_pred),2)))print('Recall is:'+str(round(recall_score(y_test, y_pred, average='binary'),2)))print('F1 score is:'+str(round(f1_score(y_test, y_pred, average='binary'),2)))
The model performs quite well for all metrics. However, if we were going to use this model in place of a doctors diagnosis for breast cancer then even 8 false negatives would be too many. So for this use case we would want to minimise false negatives and therefore want to optimise the model for recall. Now we are going to implement GridSearchCV and optimise the model for this metric.
First, we create a hyperparameter search space for two selected LogisticRegression parameters.
penalty = ['l1', 'l2']C = np.logspace(0, 4, 10)hyperparameters = dict(C=C, penalty=penalty)
We then call the logistic regression model and call GridSearchCV with the scoring parameter selected as recall.
lr = LogisticRegression()clf = GridSearchCV(lr, hyperparameters, cv=5, verbose=0, scoring='recall')
Now we fit GridSearchCV and print the best parameters for recall.
best_model = clf.fit(X_train, y_train)print('Best Penalty:', best_model.best_estimator_.get_params()['penalty'])print('Best C:', round(best_model.best_estimator_.get_params()['C'],2))
Let’s now print the confusion matrix and metrics for the best model.
y_pred_gs = best_model.predict(X_test)cm_gs = ConfusionMatrix(y_test, y_pred_gs)print(cm_gs)print('Precision is:'+str(round(precision_score(y_test, y_pred_gs),2)))print('Recall is:'+str(round(recall_score(y_test, y_pred_gs, average='binary'),2)))print('F1 score is:'+str(round(f1_score(y_test, y_pred_gs, average='binary'),2)))
You can see from these results that the recall score is now higher and we have also improved both precision and recall. The model is now better at predicting the correct labels both for malignant and benign cases.
There are many other metrics that are important to fully evaluate the performance of a classification model including specificity, AUC and ROC. However, I think those require an article of their own — this is a good one.
Thanks for reading!