Sensitivity Vs Specificity In Data Science
By: Farhad Malik
Tags: Sensitivity, Specificity, Fintechexplained, Data Science, Machine Learning

Sensitivity Vs Specificity In Data Science

A Must Know Topic For All Data Scientists

This article aims to illustrate what sensitivity and specificity are. The terms are heavily used in statistical tests. Moreover, the accuracy of most, if not all, of the classification machine learning models, is measured by their specificity and sensitivity. Therefore it is one of the most important topics that all data scientists should be familiar with as it explains the concepts behind the two most common metrics.

This article will explain the concepts using a number of use cases.

Photo by Isaac Smith on Unsplash

Specificity and sensitivity are extensively used in data science projects where we attempt to categorise data items into different clusters.

1. Let’s Understand What Classification Problems Are

Let’s consider the following use cases.

There exist three teams that are classifying input data into groups:

Team A: It is a technical team that is attempting to build a machine learning model to classify incoming emails to determine whether an email is a spam or not. The team wants to train a model so that it can categorise the emails correctly. Team B: Team B is a medical team that is preparing medicine to cure a particular disease. They want to assess whether the medicine can cure a disease by using a statistical model. Lastly Team C: This team consists of financial modellers that determine whether a customer will default on its loan or not. They want to train a model that can categorise the customers correctly.

I have deliberately presented three different examples because they should help us understand the concepts better.

Now, if we notice carefully, we will see that all of the three use cases revolve around the concept of classifying input data into two groups.

Cutting the chase….

Let’s now consider that the three teams carried out their tests and gathered the results.

To elaborate, let’s consider that the team gathered items and labelled them as Actual Positive and Actual Negative. This data set was used to train the model.

Then the team gathered 100 items to test and labelled them as Actual Positive and Actual Negative. Then they got the model to produce the results. The results produced by the model is labelled as Predicted Positive and Predicted Negative.

2. Now, let’s analyse the results

The results are then grouped into four categories.

Our aim now is to prepare the confusion matrix as shown below.

This is known as a confusion matrix

Team 1: Email filtering:

In this use case, when an email is a spam and the model predicts it to be spam then it is a True Positive (TP).

Team 2: Medical Test:

In this instance, the model predicts that the patient can be cured with the medicine and the medicine did cure the patient successfully then it’s considered a True Positive (TP).

Team 3: Financial modelling:

When the model accurately predicts that a customer will default on a loan for a customer that has actually defaulted on a loan then it's a True Positive (TP).

Once the team has gathered the actual and predicted results, they can create a confusion matrix.

The last step is to compute the sensitivity and specificity metrics.

3. Formulae

Once we have computed the True Positive, True Negative, False Positive and False Negative metrics, we can compute the sensitivity and specificity of our models.

Sensitivity:

Sensitivity is known as the True Positive Rate. It is also known as Recall. Essentially, it informs us about the proportion of actual positive cases that have gotten predicted as positive by our model.

Therefore when the value of sensitivity is high then it means our model is good at predicting the true positives correctly. It is the ratio of true positives to all positives.

Specificity:

Specificity is known as the True Negative Rate. It informs us about the proportion of actual negative cases that have gotten predicted as negative by our model. It is the ratio of true negatives to all negatives.

Therefore when the value of specificity is high then it means our model is good at predicting the true negatives.

Photo by Jason Coudriet on Unsplash

4. What is better, sensitivity or specificity?

We should always aim to get both sensitivity and specificity as high as possible but sometimes, one measure is more important than the other.

The reasons why I presented the three use cases above was to help us understand this question.

The short answer is that it is dependent on the problem we are trying to solve.

Use case 1: Email Filtering

Consider the email filtering example. If the sensitivity is high then it means the model is good at predicting the true spam emails correctly. On the other hand, if the sensitivity is low then the model will mark spam emails as non-spam emails because it implies that the true positive rate is low. As a result, we will end up getting more spams emails in our inbox.

Furthermore, if the specificity is high then it means the true negative rate is high and as a result, the model is good at identifying non-spam emails correctly. On the other hand, if the specificity is low then the model can falsely classify a non-spam email as a spam email and as a result, we will end up losing a valuable email to our spam folder.

In this usecase, it is better to have higher specificity than sensitivity because we don’t want to lose a valuable non-spam email as spam.

Therefore, for the first use case, it’s important to have a model with high specificity.

Use case 2: Medicine Test

Consider the medicine example. If the sensitivity is high then it means the model is good at predicting that the medicine can cure the disease correctly when it can actually cure the disease correctly. On the other hand, if the sensitivity is low then the model will indicate that the medicine cannot cure the patients even when the medicine can cure the disease.

Furthermore, if the specificity is high then it means the true negative rate is high and as a result, the model is good at identifying when the medicine cannot cure the disease correctly. On the other hand, if the specificity is low then the model can falsely classify that the patient can be cured and as a result, we will end up incorrectly giving the wrong medicine.

Therefore, for the second use case, it’s important to have a model with high specificity.

A question for you

For our third example of financial modellers estimating a default, can you tell me what is more important, sensitivity or specificity?

Clue: It will be disastrous to incorrectly classify a customer as good when he/she is likely going to default.

Email or comment and let me know your feedback

Summary

This article aimed to illustrate what sensitivity and specificity are. Specificity and sensitivity are used in a number of statistical experiments and hence it’s vital to understand what they are and when to choose the matric.

Specificity and sensitivity are used in data science projects where we are attempting to group data items in two clusters.

In a nutshell, sensitivity is the true positive rate and the specificity is the true negative rate.