Classification Model Evaluation For Data Scientists
By: Farhad Malik
Tags: Machine Learning, Fintech, Python, Data Science, Classification

Classification Model Evaluation For Data Scientists

Explaining How To Evaluate Classification Model In Data Science

Let’s consider that we want to classify data into a number of possible outcomes. We could use a binary classifier to group the items into two appropriate classes.

As an instance, let’s consider that we want to classify emails into two classes: Work-related vs Non-Work related.

We could create two email sets and then train the binary model on them. Once the model is trained, we could test the model on the real emails to assess how well it classifies the emails. But then how do we know how good our model is? This article will aim to explain the appropriate evaluation mechanism to assess the accuracy of the classification machine learning model.

Emails classified into two buckets

Article Aim

This article aims to illustrate how to evaluate a classification model.

In particular, it explains what confusion matrix is along with an analysis of sensitivity and specificity.

Sensitivity and specificity terms are heavily used in statistical tests hence it’s a must-know concept for all data scientists

Moreover, the accuracy of most, if not all of the classification machine learning models, is measured by their specificity and sensitivity. Therefore it is one of the most important topics that all data scientists should be familiar with as it explains the concepts behind the two most common metrics.

I will aim to explain the concepts using a number of use cases.

Specificity and sensitivity are extensively used in data science projects where we attempt to categorise data items into different groups

1. Let’s Understand What Classification Problems Are

Let’s consider the following three use cases where we will attempt to classify input data into two groups:

Team A: It is a technical team that is attempting to build a machine learning model to classify incoming emails to determine whether an email is a spam or not. The team wants to train a model so that it can categorise the emails correctly. Team B: Team B is a medical team that is preparing medicine to cure a flue. They want to assess whether the medicine can cure the disease by using a machine model. Lastly Team C: This team consists of financial modellers that predict whether a customer will default on its loan or not. They want to train a model that can categorise the customers correctly.

All of the three usecases revolve around the concept of classifying input data into two groups.

The teams start by gathering the input data. They then correctly label the data items to train the model. Then they gather 100 items each to test and label them respectively. Then they get the model to produce the results on the test set. The results produced by the model along with the actual labels can now be used to evaluate the model.

2. Let’s analyse the results to evaluate the model

The first is to prepare a confusion matrix as shown below.

This is known as a confusion matrix. Let’s understand what True Positive, True Negative, False Positive and False Negative terms are.

Team 1: Email filtering:

When an email is a spam and the model predicts it to be spam then it is a True Positive (TP).

If a spam email is incorrectly predicted as a non-spam email then it is False Negative (FN).

Team 2: Medical Test:

The model predicts that a patient can be cured with the medicine and the medicine did cure the patient successfully then it’s considered a True Positive (TP).

If the model predicts that a patient can be cured with the medicine but the medicine did not cure the patient then it is a False Positive (FP).

Team 3: Financial modelling:

When a model accurately predicts that a customer will default on a loan for a customer that has actually defaulted on a loan then it’s a True Positive (TP). If the model predicts that a customer will not default and the customer actually did not default on a loan then it’s a True Negative (TN).

Once the team has gathered the actual and predicted results, they can create a confusion matrix.

The last step is to compute the sensitivity and specificity metrics.

3. Formulae

Once we have computed the True Positive, True Negative, False Positive and False Negative metrics, we can compute the sensitivity and specificity of our models.

Sensitivity:

Sensitivity is known as the True Positive Rate. It is also known as the Recall. Essentially, it informs us about the proportion of actual positive cases that have gotten predicted as positive by our model.

Therefore when the value of sensitivity is high then it means our model is good at predicting the true positives correctly. It is the ratio of true positives to all positives.

Specificity:

Specificity is known as the True Negative Rate. It informs us about the proportion of actual negative cases that have gotten predicted as negative by our model. It is the ratio of true negatives to all negatives.

Therefore when the value of specificity is high then it means our model is good at predicting the true negatives.

4. What is better, sensitivity or specificity?

We should always aim to get both sensitivity and specificity as high as possible but sometimes, one measure is more important than the other.

The reasons why I presented the three use cases above was to help us understand this question.

The short answer is that it is dependent on the problem we are trying to solve.

We can plot a ROC curve and we should aim for the area under the curve to be 1. Higher the area, better the model. The area under the curve is known as AUC.

Use case 1: Email Filtering

Consider the email filtering example. If the sensitivity is high then it means the model is good at predicting the true spam emails correctly. On the other hand, if the sensitivity is low then the model will mark spam emails as non-spam emails because it implies that the true positive rate is low. As a result, we will end up getting more spams emails in our inbox.

Furthermore, if the specificity is high then it means the true negative rate is high and as a result, the model is good at identifying non-spam emails correctly. On the other hand, if the specificity is low then the model can falsely classify a non-spam email as a spam email and as a result, we will end up losing a valuable email to our spam folder.

In this usecase, it is better to have higher specificity than sensitivity because we don’t want to lose a valuable non-spam email as spam.

Therefore, for the first use case, it’s important to have a model with high specificity.

Use case 2: Medicine Test

Consider the medicine example. If the sensitivity is high then it means the model is good at predicting that the medicine can cure the disease correctly when it can actually cure the disease correctly. On the other hand, if the sensitivity is low then the model will indicate that the medicine cannot cure the patients even when the medicine can cure the disease.

Furthermore, if the specificity is high then it means the true negative rate is high and as a result, the model is good at identifying when the medicine cannot cure the disease correctly. On the other hand, if the specificity is low then the model can falsely classify that the patient can be cured and as a result, we will end up incorrectly giving the wrong medicine.

Therefore, for the second use case, it’s important to have a model with high specificity.

Use case 3: Default Predictor

For our third example of financial modellers estimating a default, it will be disastrous to incorrectly classify a customer as someone who cannot default when he/she is likely going to default.

For this usecase, it’s safer for the model to have a higher sensitivity.

Python Code

We can use Support vector machine, Logistic regression and Decision trees classifiers amongst many models to classify the data.

This code snippet illustrates how we can get a confusion matrix and output the score:

from sklearn.tree import DecisionTreeClassifier

from sklearn.linear_model import LogisticRegression

from sklearn.metrics import confusion_matrix

from sklearn.svm import SVC svc = SVC(kernel='linear', C=5).fit(X_train, y_train)

predicted = svc.predict(X_test)

cm = confusion_matrix(y_test, predicted) dtc = DecisionTreeClassifier(max_depth=10).fit(X_train, y_train)

predicted = dt.predict(X_test)

cm = confusion_matrix(y_test, predicted) lrc = LogisticRegression().fit(X_train, y_train)

predicted = lr.predict(X_test)

cm = confusion_matrix(y_test, predicted)

Summary

This article aimed to illustrate what sensitivity and specificity are. Specificity and sensitivity are used in a number of statistical experiments and hence it’s vital to understand what they are and when to choose the matric.

Thank you for reading

Specificity and sensitivity are used in data science projects where we are attempting to group data items in two clusters.

In a nutshell, sensitivity is the true positive rate and the specificity is the true negative rate.