Accuracy Visualisation: Supervised Machine Learning Classification Algorithms | by Kaushik Choudhury | Towards Data Science

Measuring the prediction accuracy of any regression or classification algorithm is vital in different stages during modelling and also when the model is live in production.
We have several ways to measure the accuracy of classification algorithms. In the Scikit-learn package, we have several scores like recall score, accuracy score etc. and then we have out of box summarised reports. In my view, most of these metrics have one or more limitations related to verbosity and difficult to understand, potential chance to misinterpret the accuracy in case of imbalance classes in the dataset, need to refer few of the scores to get holistic view etc.
To better understand the limitation, let us consider the example shown in the table. We have few parameters related to a sample of people and whether they are COVID positive. As we have more people who are COVID negative than people who are infected with the virus hence, I have considered a similar distribution in this example.
A classification algorithm trained on this datasets predicted the results as shown in the last column. The accuracy score of the classification model is close to 90 per cent. It gives us the impression that the model is quite good at predicting the result.
from sklearn.metrics import recall_scorefrom sklearn.metrics import classification_reportfrom sklearn.metrics import accuracy_score# 0- Healthy , 1-  Covidy_true = [0, 1, 0,0,0, 1,0,0,0,0,0]y_pred = [0, 0, 0,0,0, 1,0,0,0,0,0]print("The recall Score is" , recall_score(y_true, y_pred))print("The accurracy score is" , accuracy_score(y_true, y_pred))
In reality, the model could predict the COVID positive cases with only 50 per cent times. Deploying such a model only based on one metric or without understanding the areas in which the classification model is making an error can be quite costly.
We can make a full sense of the model fit for production usage after considering both the recall and accuracy score and business use case.
A visual metric, like the confusion matrix, outshines other metrics in several ways. We get an instant view on the model performance in terms of classification areas the model excelled and the areas which require fine-tuning. Based on the business use case, we can judge quickly from the false positive, false negative, true positive and true negative counts whether the model is ready for deployment.
Let us learn to visualise and interpret the result of LinearSVCand LogisticRegression with confusion matrix.
Step 1: We will use the Scikit-learn inbuilt dataset WINE in this article for discussion and Matplotlib for visualisation. In the below code, we have imported the modules which we will be using in our program.
from sklearn.datasets import load_winefrom sklearn.model_selection import train_test_splitfrom sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegressionfrom sklearn.svm import LinearSVCimport matplotlib.pyplot as pltfrom sklearn.metrics import plot_confusion_matrix
Step 2: The features ( attributes) in the Wine datasets divided into independent and dependent variables. Independent (input features) variables denoted by “X” and the dependent variable is denoted by “y”. Using the method train_test_split is divided into training and testing set.
X,y = load_wine(return_X_y=True)X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.10,random_state=0)
Step 3: Most of the supervised algorithms in sklearn require standard normally distributed input data centred around zero and have variance in the same order. As the independent variable values in the WINE dataset have a different scale, hence we need to scale it before using it in modelling.
If you would like to learn more about scaling the independent variables and different scalers in Scikit-learn package, then please refer my article Feature Scaling — Effect Of Different Scikit-Learn Scalers: Deep Dive.
SC_X=StandardScaler()X_train_Scaled=SC_X.fit_transform(X_train)X_test=Scaled=SC_X.transform(X_test)
Step 4: In the below code, a list of classifiers defined along with parameters
classifiers=[ LinearSVC(dual=False),LogisticRegression(solver="liblinear",max_iter=100)]
Step 5: Each classifier is used to train the model in sequence, and the confusion matrix is plotted based on the actual and predicted dependent variable value in the training set.
for clf in classifiers:    clf.fit(X_train, y_train)    y_pred = clf.predict(X_test)    fig=plot_confusion_matrix(clf, X_test, y_test, display_labels=["Bad Wine","Fair Wine","Good Wine"])    fig.figure_.suptitle("Confusion Matrix for  " + str(clf))    plt.show()
At first glance itself, we can see that the summation of figures from top left to bottom right diagonal is higher in Linear SVC compare to Logistics regression. It immediately suggests that Linear SVC performed better it identifying the true positive records.
Different metric scores can indicate the accuracy, but with confusion matrix, we can immediately see the classes which the algorithm is misclassifying. In the current example, both the classifiers missed in classifying most of the bad wines accurately. This level of detail information helps to perform focus fine-tuning of the model.
We have seen that the confusion matrix provides finer details about the classification model prediction pattern for different classes in a nice crisp way. In a single confusion matrix, We can get the details like false alarms (false positive) and correct rejection rates and decide whether the model is acceptable based on the business case.