14 Popular Evaluation Metrics in Machine Learning | by Satyam Kumar | Towards Data Science

The evaluation metric is used to measure the performance of a machine learning model. A correct choice of an evaluation metric is very essential for a model. This article will cover all the metrics used in classification and regression machine learning models.
Evaluation Metrics discussed in the article:
Metrics used in Classification Models:
For a classification machine learning algorithm, the output of the model can be a target class label or probability score. The different evaluation metric is used for these two approaches.
Confusion Matrix:
A confusion matrix is the easiest way to measure the performance of a classification problem. It is used to visualize and observe the performance of the prediction of ML models. For a k class classification model, a matrix of size k*k is used to observe the prediction. For a binary class classification problem, a standard 2*2 size matrix is used.
Notations,TP: True Postive: Number of Points which are actually positive and predicted to be positiveFN: False Negative: Number of Points which are actually positive but predicted to be negativeFP: False Positive: Number of Points which are actually negative but predicted to be positiveTN: True Negative: Number of Points which are actually negative and predicted to be negative
An ML model is considered good if the numbers on principal diagonal are maximum and the number on off-diagonal should be minimum. For a binary confusion matrix, TP and TN should be high and FN and FP should be low.
Different problems have different metrics to choose from:
For the problem of cancer diagnosis, TP should be high and FN should me very low close to 0. Patients having cancer should never be predicted to be not cancer which is the case of FN.
For the problem of spam detection, FP should be very low. No mails should be predicted to be spam which is not spam.
What are type I and type II errors?
A type I error is also known as a false positive (FP). A type II error is also known as a false negative (FN).
Accuracy:
Accuracy is the most common performance metric used for classification algorithms. Accuracy is mainly used for a balanced dataset. It is defined as the ratio of the number of correct predictions made to all the predictions made.
The accuracy metric may not work well with unbalanced datasets as it may get biased towards the majority class. Below mentioned metrics overcomes this disadvantage.
TPR (True Positive Rate):
TPR is the ratio of the number of correctly predicted positive class to the total number of positive classes.
TNR (True Negative Rate):
TNR is the ratio of the number of correctly predicted negative class to the total number of negative classes.
FPR (False Positive Rate):
FPR is the ratio of the number of wrongly predicted positive class to the total number of negative classes.
FNR (False Negative Rate):
TNR is the ratio of the number of correctly wrongly negative class to the total number of positive classes.
Precision:
The precision metric used in information retrieval is defined as from all the points predicted to positive how many of them are actually positive.
The precision metric is used for ML models where low False Positive (FP) is important such as spam detection.
Recall:
The recall metric is also known as sensitivity is defined as from all the points which are actually positive how many of them are predicted to be positive. The recall is the same as the true positive rate (TPR).
The recall metric is used for ML models where low False Negative (FN) is important such as cancer diagnosis.
F-beta score:
For ML models where both FN and FP have equal importance to be low, then we can use combine the advantage of Precision and Recall in a new metric called F-beta score.
Here beta is a variable,(Beta < 1) is used when FP have more impact than FN(Beta > 1) is used when FN have more impact than FP(Beta == 1) is used when FN and FP have equal importance
when Beta=1 then the metric is called F1 score giving equal importance to precision and recall.
ROC Curve and AUC:
ROC ( Receiver Operating Characteristic) curve is used mainly for binary classification based ML models. ROC curve and AUC (Area Under Curve) can be implemented for ML models that output probability scores.
ROC curve is a curve drawn joining all the points for different threshold values in a plot of TPR vs FPR.
The output of the ML model is the probability value, for different values of threshold find y_hat values and then compute TPR and FPR. Further, draw a plot to observe the ROC curve and AUC.
For the below sample dataset:
Notation,x_i: ith data pointy: target binary class labely_hat: predicted probability valuey_hat(t): predicted binary class label for threshold - t
The ROC curve (Blue Curve) for about sample dataset is below:
For a real-world dataset the ROC curve will seem like:
The above plot is the ROC curve (Blue curve), and the area under the curve is called AUC (Area Under Curve). The orange line is the x=y line, and the ROC curve will always be above this line.
Log Loss:
Log Loss is also known as cross-entropy loss uses probability estimates to compute the performance of the model. The probability estimate of the model is between 0 and 1. The log loss of the model is always greater than 0, a score of 0 being the best model. Log Loss is the amount of uncertainty of our prediction based on how much it varies from the actual label. With the help of the Log Loss value, we can have a more accurate view of the performance of our model. The equation of log loss of binary classification is:
Notations,n: number of points in datasetp_i: predicted probability of ith pointy_i: actual output class label
Below is a sample dataset:
Notation,x_i: ith data pointy: target binary class labely_hat: predicted probability value
Metrics used in Regression Models:
For regression problems, the output of ML models is real-valued. Various metrics to compute the performance of regression models are:
Mean Absolute Error (MAE):
MAE is the simplest error metric used in regression problems. MAE is defined as the sum of the average of the absolute difference between the predicted and actual values. It computes the performance of the regression model by summing the variability of prediction from the actual value.
Notation,n: number of data pointy: actual real valuey_hat: predicted real value
Root Mean Squared Error (RMSE) and Mean Squared Error (MSE):
MSE is the same as the MAE, but the only difference is that it squares the difference between actual and predicted output values before summing them all instead of using the absolute value. RMSE is taking the square root of MSE.
Notation,n: number of data pointy: actual real valuey_hat: predicted real value
R squared Error:
R Squared metric is generally used for explanatory purposes and indicates the goodness of fit of a set of predicted output values to the actual output values.
Notation,n: number of data pointy_i: ith actual real valuey_hat: predicted real valuey_bar: mean of y
Thank You for Reading!