Is F1 the appropriate criterion to use? What about F2, F3,..., F beta? | by Barak Or | Towards Data Science

According to many data scientists, the most reliable model performance measure is accuracy. It is not only the definitive model metric, there are many others, too. Periodically, the accuracy might be high, but the false-negative (to be defined in the sequel) is also high. Another key measure is the F-measure common in machine learning these days, for evaluating the model performance. It proportionally combines the precision and recall measures. In this post, we explore different approaches where the imbalance of the two is suggested.
The confusion matrix summarizes the performance of a supervised learning algorithm in ML. It is more beneficial as it provides a more detailed analysis than the common accuracy measure. In the confusion matrix, each row represents the instance in a predicted class and each column represents the instance in an actual class. The simplified confusion matrix contains two rows and two columns, as given above, where:
For further discussion, we have to define two important measures: Precision: the number of TP samples divided by all P samples (true and false), and Recall: the number of TP divided by (TP+FN).
As both measures are of high importance, a need for one measure to combine between the two arises. Hence, the harmonic mean of the precision and the recall, also known as the F1 score, was proposed.
The measure is given by:
The main advantage (and at the same time disadvantage) of the F1 score is that the recall and precision are of the same importance. In many applications, this is not the case and some weight should be applied to break this balance assumption. This balance assumption might be applied, where there is an uneven distribution of the data such as a large amount of positive-negative.
Using the weighted mean, we can easily obtain the measure F2:
In the same way, the F3 score is obtained:
Generalizing the weighted mean approach results in the F beta measure, provided by:
This measure allows us to define how much the recall is important than the precision. Using the F beta measure in the sklearn is very easy, just follow the example:
scikit-learn.org
>>> from sklearn.metrics import fbeta_score>>> y_true = [0, 1, 2, 0, 1, 2]>>> y_pred = [0, 2, 1, 0, 0, 1]>>> fbeta_score(y_true, y_pred, average='macro', beta=0.5)0.23...>>> fbeta_score(y_true, y_pred, average='micro', beta=0.5)0.33...>>> fbeta_score(y_true, y_pred, average='weighted', beta=0.5)0.23...>>> fbeta_score(y_true, y_pred, average=None, beta=0.5)array([0.71..., 0.        , 0.        ])
In this post, I reviewed the F measures. I hope that the provided data will help those dealing with classification tasks and motivate them to use the F measures along with the accuracy.
Barak Or received the B.Sc. (2016), M.Sc. (2018) degrees in aerospace engineering, and also B.A. in economics and management (2016, Cum Laude) from the Technion, Israel Institute of Technology. He was with Qualcomm (2019–2020), where he mainly dealt with Machine Learning and Signal Processing algorithms. Barak currently studies toward his Ph.D. at the University of Haifa. His research interest includes sensor fusion, navigation, machine learning, and estimation theory.
https://www.researchgate.net/profile/Barak-Or
www.barakor.com
[1] C.J Van Rijsbergen: https://en.wikipedia.org/wiki/C._J._van_Rijsbergen
[2] Matthews Correlation Coefficient is The Best Classification Metric You’ve Never Heard Of