Multi-class Classification: Extracting Performance Metrics From The Confusion Matrix | by Serafeim Loukas | Towards Data Science

In one of my previous posts, “ROC Curve explained using a COVID-19 hypothetical example: Binary & Multi-Class Classification tutorial”, I clearly explained what a ROC curve is and how it is connected to the famous Confusion Matrix. If you are not familiar with the term Confusion Matrix and True Positives, True Negatives etc., refer to the above article and learn everything in 6 minutes or continue reading for a quick 2 minutes recap.
Let’s imagine that we have a test that is able within seconds to tell us if one individual is affected by the virus or not. So the output of the test can be either Positive (affected) or Negative (not affected). So, in this hypothetical case we have a binary classification case.
True Positives (TP, blue distribution) are the people that truly have the virus.
True Negatives (TN, red distribution) are the people that truly DO NOT have the virus.
False Positives (FP) are the people that are truly NOT sick but based on the test, they were falsely (False) denoted as sick (Positives).
False Negatives (FN) are the people that are truly sick but based on the test, they were falsely (False) denoted as NOT sick (Negative).
To store all these measures of performance, the confusion matrix is usually used.
The confusion matrix of a classifier summarizes the TP, TN, FP, FN measures of performance of our model. The confusion matrix can be further used to extract more measures of performance such as: TPR, TNR, FPR, FNR and accuracy.
Using all the above terms, we can also construct the famous confusion matrix that consists of these metrics and then we can compute the True Positive Rate and the False Positive Rate as shown in the figure below for a binary classification case.
The confusion matrix can be well defined for any N-class classification problem. However, if we have more than 2 classes (N>2), then the above equations (in the confusion matrix figure) do not hold any more.
In this article, I show how to estimate all these measures for any number of classes (N).
In a previous post, I explained what an SVC model is so here we will use this as our classifier.
In the iris dataset, we have 3 classes of flowers and 4 features. So the classification problem is not a binary case anymore since we have 3 classes.
Let’s load the data and check how many classes and samples per class we have:
import numpy as npimport matplotlib.pyplot as pltfrom sklearn import svm, datasetsfrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import plot_confusion_matrixfrom sklearn.metrics import confusion_matrix# import some data to play withiris = datasets.load_iris()X = iris.datay = iris.targetclass_names = iris.target_nameslabels, counts = np.unique(y, return_counts=True)plt.bar(labels, counts, align='center')plt.gca().set_xticks(labels)plt.show()
We can see that we have 3 classes/groups (labels: 0,1,2) and within each class/group we have 50 (i.e. the height of the bars) samples (examples/observations).
Next, we will split the data into training and test sets before model fitting:
# Split the data into a training set and a test setX_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
Next, let’s create the classification model (linear SVC) and fit the model using the training set:
# Fit the classifierclassifier = svm.SVC(kernel='linear').fit(X_train, y_train)
Now, we will predict the labels of the test set samples and we will build the confusion matrix of our 3-class classification problem:
# Predict the labels of the test set samplespredicted_labels = classifier.predict(X_test)# Build the confusion matrix of our 3-class classification problemcnf_matrix = confusion_matrix(y_test, predicted_labels)print(cnf_matrix)#array([[13,  0,  0],#       [ 0, 15,  1],#       [ 0,  0,  9]])
To better read/understand the confusion matrix, let’s plot it:
disp = plot_confusion_matrix(classifier, X_test, y_test,                                 display_labels=class_names,                                 cmap=plt.cm.Blues)plt.show()
On the x-axis we have the predicted labels and on the y-axis we have the true labels of out test set’s samples (remember that the prediction was done using only the test set).
Ideally, a perfect classifier would result in a confusion matrix where we have values only on the diagonal i.e. in a case where we classify correctly all the test samples for all the 3 classes/groups that we have.
The values in the boxes are just counts. For instance, in our case, the upper left box has a value 13 inside and the next 2 boxes have 0 and 0. This means that we are able to correctly classify all 13 setosa test samples.
On the other hand, by looking at the second row that refers to the versicolor group, we can see that we classified correctly 15 versiolor test samples (out of the total of 16 that are in the test set) and missed 1 versicolor test samples that was wrongly predicted as “virginica”
Finally, here is the magic.
We will estimate the FP, FN, TP, TN, TPR (Sensitivity, hit rate, recall, or true positive rate), TNR (Specificity or True Negative Rate), PPV (Precision or Positive Predictive Value), NPV (Negative Predictive Value), FPR (Fall out or False Positive Rate), FNR (False Negative Rate), FDR (False Discovery Rate), and ACC (accurancy).
FP = cnf_matrix.sum(axis=0) - np.diag(cnf_matrix) FN = cnf_matrix.sum(axis=1) - np.diag(cnf_matrix)TP = np.diag(cnf_matrix)TN = cnf_matrix.sum() - (FP + FN + TP)FP = FP.astype(float)FN = FN.astype(float)TP = TP.astype(float)TN = TN.astype(float)# Sensitivity, hit rate, recall, or true positive rateTPR = TP/(TP+FN)# Specificity or true negative rateTNR = TN/(TN+FP) # Precision or positive predictive valuePPV = TP/(TP+FP)# Negative predictive valueNPV = TN/(TN+FN)# Fall out or false positive rateFPR = FP/(FP+TN)# False negative rateFNR = FN/(TP+FN)# False discovery rateFDR = FP/(TP+FP)# Overall accuracy for each classACC = (TP+TN)/(TP+FP+FN+TN)
That’s all folks ! Hope you liked this article!
[1] https://en.wikipedia.org/wiki/Confusion_matrix
[2] https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html
If you liked and found this article useful, follow me to be able to see all my new posts.
Questions? Post them as a comment and I will reply as soon as possible.
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
LinkedIn: https://www.linkedin.com/in/serafeim-loukas/
ResearchGate: https://www.researchgate.net/profile/Serafeim_Loukas
EPFL profile: https://people.epfl.ch/serafeim.loukas
Stack Overflow: https://stackoverflow.com/users/5025009/seralouk