What is Accuracy, Precision, Recall, and F1 score? What is its Significance in Machine Learning?
By: Ashish Mehta
Tags: Accuracy And Precision, Artificial Intelligence, Machine Learning, Data Science, Classification Metrics

What is Accuracy, Precision, Recall, and F1 score? What is its Significance in Machine Learning?

Understanding Accuracy, Precision, and Recall using a real-life example.

Image by Stefan Keller from Pixabay

Explain the difference between Accuracy and Precision?

When will you use Precision and Recall to evaluate your machine learning model?

What is the importance of the F1 score?

These are some of the favorite questions of the interviewers and they pop up in the majority of Data Science and Machine Learning interviews.

When explaining what Accuracy, Precision, Recall, and F1 score means, we tend to get confused and mix these terms up with one another.

So I thought it would be a good idea to explain these terms using real-life examples and their significance when it comes to Data Science and Machine Learning.

Understanding Accuracy, Precision, and Recall using a real-life example:

Image by Steve Buissinne from Pixabay

“John is a college student who works at a supermarket to earn some pocket money. One day, the manager of the supermarket asks John to check the stock in the storeroom, weigh all the bags of rice and prepare a report. As instructed, John walks into the storeroom and puts the first bag of rice on the weighing scale.”

Accuracy:

The true weight of the bag of rice is 20 kg. But the weighing machine displays the weight to be 8 kg.

What do you think, is the weighing machine accurate?

The answer is No. Here, the true value (20 kg) and the measured value (8 kg) are too far off from each other. Hence the machine is inaccurate.

The machine would have been accurate if the measured value would have been close to 20 kg.

In other words, Accuracy is defined as the closeness of a measured value to a true value. That means the closer the measured value to the true value, the better is the accuracy.

Precision:

“Upon weighing the first bag of rice, John recognizes that there is something wrong either with the machine or with the bag of rice. In order to cross check, he weighs the same bag of rice 3 more times. And every time, the weighing machine displays 8 kg as the weight of the bag.”

Since the weighing machine is displaying the exact same weight every time, we can say that the weighing machine is highly precise.

Precision refers to the closeness of two or more measurements to each other.

Note that Accuracy and Precision are independent of each other. As we can see in the above example, the weighing machine is highly precise but not necessarily accurate.

Recall:

“After preparing the report, John comes back to his manager. Upon reviewing, the manager asks John, how many bags of rice were blue in colour? John thinks a little and recalls that 30 bags of rice were blue in colour.”

In reality, only 15 bags of rice were blue. And John recalled that 30 bags were blue. So out of 30 instances, John was correct 15 times. Therefore, his Recall was 100% but his Precision was 50%.

Importance of Accuracy, Precision, and Recall in Data Science and Machine Learning.

After looking at a real-world example, let’s examine how Accuracy, Precision, and Recall are used in Machine Learning.

Let’s assume, You get a machine learning problem statement and a dataset. You put in your time and effort to create a machine learning model.

Now you want to judge how well does your model performs. So how do you measure your model’s effectiveness?

To measure the effectiveness of a model we make use of performance metrics.

Accuracy, Precision, and Recall are performance metrics used to measure the performance of Supervised Classification Models.

Confusion Matrix:

Before we jump into calculating Accuracy, Precision, and Recall for our classification model, we first need to understand what a Confusion matrix is.

An imaginary dataset of a group of students in a classroom

Let’s assume we have a dataset where the independent features are the marks obtained by students in a classroom and the dependent feature is “whether they passed or failed the exams.”

The dependent feature is a categorical feature with two categories: Yes and No

The category Yes is taken as Positive and category No is taken as Negative.

As a fundamental practice in machine learning, once our data is ready for model building, we divide the data into a training dataset and testing dataset before giving the data to the model.

We train our model on the training dataset and then check the performance of the model using the test dataset.

To summarize how our model has performed on the test dataset, we create a Confusion matrix for the model.

Given below is a representation of a confusion matrix:

Components of a Confusion Matrix:

The rows in a confusion matrix correspond to what the machine learning model predicted. The columns in a confusion matrix correspond to the known true values.

Since we have two categories in the dependent feature, we get a 2x2 confusion matrix.

The cell in the top left corner contains True Positives (TP). These are students who passed the exams and were correctly identified by the algorithm.

The cell in the bottom right corner contains True Negatives (TN). These are students who failed the exams and were correctly identified by the algorithm.

The cell in the Bottom left corner contains False Negatives (FN). False Negatives is when a student passed the exams but the algorithm predicted that he didn’t.

The cell in the top right corner contains False Positives (FP). False Positives is when a student did not pass the exams but the algorithm predicted that he passed.

False Positives are known as Type I error (rejection of a true null hypothesis)

False Negatives are known as Type II error (non — rejection of a false null hypothesis)

Calculating Accuracy, Precision, and Recall:

Image by Daniel Friesenecker from Pixabay

Accuracy:

Accuracy is an evaluation metric that determines the number of correct predictions made by the model.

The formula for accuracy is pretty straight forward:

Accuracy answers the question, what percent of the model’s predictions were correct? Accuracy looks at True Positives and True Negatives.

Accuracy should be used as a performance metric only if we have a balanced dataset.

For example, in the above students’ dataset, there are 10 students. 5 of them have passed and 5 of them did not pass the exams. Since there is an equal number of students in both categories, we can say that we have a balanced data set.

Now the question arises, what if we have an imbalanced dataset?

In this case, we make use of precision or recall.

Precision:

Precision answers the question, out of the number of times a model predicted positive, how often was it correct?

The formula for Precision is given as:

Precision should be used as a performance metric when the importance of False Positive is very high and the importance of false negative is very low.

For example, we take a use case of email spam detection.

In email spam detection, a false positive means that a non-spam email (Actual negative) has been identified as spam (False positive).

The user might miss upon important emails if precision is not high for spam detection models.

Recall:

Recall answers the question, out of the total actual positive values, how many of them were correctly predicted as positive?

The formula for the Recall is given as:

The Recall should be used as a performance metric when the importance of false negative is high and the importance of false positive is low.

For example, we take the use case of cancer detection in patients.

If a patient having cancer (Actual Positive) is predicted as not having cancer (Predicted Negative), it could be extremely fatal for the patient.

Significance of F1 score:

In some of the problem statements, false positive and false negative both are very important. We have to create a balance between the false positive and false negative. In such cases, we make use of another performance metric known as an F1 score.

F1 score is the harmonic mean of Precision and Recall.

The formula for F1 score is given as:

F1 score should be used as a performance metric if the number of samples in the positive class is very less and if the cost of false-positive and the cost of false negative both are very high.

For example, we take the use case of fraud detection.

In fraud detection problem, the number of fraud transactions (positives) is very less as compared to the number of genuine transactions (negatives). Hence F1 score should be used as a performance metric to evaluate the model in such cases.

Conclusion:

To use any one of Accuracy, Precision, Recall, and F1 score as a performance metric, the problem we are solving must be a supervised classification problem.

If we have a balanced dataset, we can use the Accuracy score to evaluate the model.

In the case of an imbalanced dataset, if false positive has more importance then we make use of Precision and if false negative has more importance then we make use of Recall.

Finally, if the number of true values is very less and if we want to maintain a balance between precision and recall, then we make use of the F1 score.

Selecting the right performance metric for evaluating the machine learning model really depends on the dataset you receive and the type of problem you are trying to solve.

References: