Good Classification Measures and How to Find Them

1 Introduction . Classification is a classic machine learning task that is used in countless applications . To evaluate classification results , one has to compare the predicted labeling of a given set of elements with the actual ( true ) labeling . For this , performance measures are used , and there are many well-known ones like accuracy , F-measure , and so on [ 13 , 15 ] . The fact that different measures behave differently is known throughout the literature [ 10 , 13 , 20 , 21 ] . For instance , accuracy is known to be biased towards the majority class . Thus , different measures may lead to different evaluation results , and it is important to choose an appropriate measure . While there are attempts to compare performance measures and describe their properties [ 3 , 6 , 13 , 14 , 26 , 28 ] , the problem still lacks a systematic approach , and our paper aims at filling this gap.1 Our research is particularly motivated by a recent paper [ 12 ] providing a systematic analysis of evaluation measures for the clustering task . We transfer many proposed properties to the classification problem and extend the research by adding more properties , new measures , and novel theoretical results . To provide a systematic comparison of performance measures , we formally define a list of properties that are desirable across various classification tasks . The proposed properties can be applied both to binary and multiclass problems . Some properties are intuitive and straightforward , like symmetry , while others are more tricky . A particularly important property is called constant baseline . It requires a measure not to be biased towards particular predicted class sizes . For each measure and each 1We describe related research in detail in Appendix A . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . property , we formally prove or disprove that the property is satisfied . We believe that this analysis is essential for better understanding the differences between the performance measures . Then , we analyze relations between different properties in the binary case and prove an impossibility theorem : it is impossible for a performance measure to be linearly transformable to a metric and simultaneously have the constant baseline property . This means that at least one of these properties has to be discarded . If we relax the set of properties by discarding the distance requirement , the remaining ones can be simultaneously satisfied . In fact , we propose a family of measures called Generalized Means ( GM ) , satisfying all the properties except distance and generalizing the well-known Matthews Correlation Coefficient ( CC ) . In addition to CC , this class also contains another intuitive measure that we name Symmetric Balanced Accuracy . To the best of our knowledge , this measure has not been previously used for classification evaluation.2 If we instead discard the constant baseline ( but keep its approximation ) , then the arccosine of CC is a measure satisfying all the properties . We also demonstrate through a series of experiments that different performance measures can be inconsistent in various situations . We notice that measures having more desirable properties are usually more consistent with each other . We hope that our research will motivate further studies analyzing the properties of performance measures for classification and other problems since there are still plenty of questions to be answered . 2 Performance measures for classification . In this section , we define measures that are commonly used for evaluating classification results . Classification problems can be divided into binary , multiclass , and multilabel . In this paper , we focus on binary and multiclass and leave multilabel for future research . There are several types of performance measures : threshold measures assume that predicted labels deterministically assign each element to a class ( e.g. , accuracy ) ; probability measures assume that the predicted labels are soft and compare these probabilities with the actual outcome ( e.g. , the cross-entropy loss ) ; ranking measures take into account the relative order of the predicted soft labels , i.e. , quantify whether the elements belonging to a class have higher predicted probabilities compared to other elements ( e.g. , area under the ROC curve or average precision ) . Our research focuses on threshold measures . Now we introduce notation needed to formally define binary and multiclass threshold measures.3 Let n > 0 be the number of elements in the dataset and let m ≥ 2 denote the number of classes . We assume that there is true labeling classifying elements into m classes and also predicted labeling . Let C be the confusion matrix : each matrix element cij denotes the number of elements with true label i and predicted label j . For binary classification , c11 is true positive ( TP ) , c00 is true negative ( TN ) , c10 is false negative ( FN ) , and c01 is false positive ( FP ) . We use the notation ai = ∑m−1 j=0 cij , bi =∑m−1 j=0 cji for the sizes of i-th class in the true and predicted labelings , respectively . Finally , we denote classification measures byM ( C ) orM ( A , B ) , whereA andB are true and predicted labelings , and write M ( c11 , c10 , c01 , c00 ) for binary ones . Table 1 ( above the line ) lists several widely used classification measures . The most well-known is accuracy which is the fraction of correctly classified elements . Accuracy is known to be biased towards the majority class , so it is not appropriate for unbalanced problems . To overcome this , Balanced Accuracy re-weights the terms to treat all classes equally . Cohen ’ s Kappa uses a different approach to overcome this bias : it corrects the number of correctly classified samples by the expected value obtained by a random classifier [ 5 ] . Matthews Correlation Coefficient is the Pearson correlation coefficient between true and predicted labelings for binary classification [ 11 ] . For the multiclass case , covariance is computed for each class , and the obtained values are averaged before computing the correlation coefficient . Finally , Confusion Entropy computes the entropy of the misclassification distribution for each class and combines the obtained values , see Table 1 and [ 30 ] for the details.4 Some measures are exclusively defined for binary classification . In this case , the classes are often referred to as ‘ positive ’ and ‘ negative ’ . Jaccard measures the fraction of correctly detected positive 2For clustering evaluation , there is an analog known as Sokal & Sneath ’ s measure [ 12 ] . 3For convenience , we list the notation used in the paper in Table 7 in Appendix . 4There can be cases when a class is not present in the predicted labels . Then , some measures may contain division by zero . A proper way to fill in such singularities is discussed in Appendix B. examples among all positive ones ( both in true and predicted labelings ) . F-measure is the ( possibly weighted ) harmonic mean of Recall ( c11/a1 ) and Precision ( c11/b1 ) . For measures that do not have a natural multiclass variant , there are several universal extensions obtained via averaging the results for m one-vs-all binary classifications [ 17 ] . For each one-vs-all classification , a particular class i is considered positive while all other classes are grouped to a negative class . Micro averaging sums up all binary confusion matrices corresponding to m one-vs-all classifications . Formally , it sets true positive as ∑m−1 i=0 cii , false negative and false positive as n− ∑m−1 i=0 cii , true negative as ( m− 2 ) n+ ∑m−1 i=0 cii . Then , a given binary measure is applied to the obtained matrix . Macro averaging computes the measure values for m binary classification sub-problems and then averages the results : 1m ∑m−1 i=0 M ( cii , ai − cii , bi − cii , n− ai − bi + cii ) , where M ( · ) is a given binary measure . Note that macro averaging gives equal weights to all one-vs-all binary classifications . In contrast , weighted averaging weights one-vs-all binary classifications according to the sizes of the corresponding classes : 1n ∑m−1 i=0 ai ·M ( cii , ai − cii , bi − cii , n− ai − bi + cii ) . 3 Properties of validation measures . As clearly seen from the above discussion , there are many options for classification validation . In this section , we propose a formal approach that allows for a better understanding the differences between the measures and for making an informed decision among them for a particular application . For this , we propose properties of validation measures that can be useful across various applications and formally check which measures satisfy which properties . In this regard , we follow the approach proposed in [ 12 ] for comparing validation measures for clustering tasks . First , we observe that some theoretical results from [ 12 ] are related to binary classification measures . Indeed , a popular subclass of clustering validation measures are pair-counting ones . Such measures are defined in terms of the values N11 , N10 , N01 , N00 that essentially define a confusion matrix for binary classification on element pairs . Thus , replacingNij in pair-counting clustering measures by cij , results in binary classification measures . We refer to Table 8 in Appendix B for the correspondence of some classification and clustering measures . In particular , Accuracy is equivalent to Rand , while Cohen ’ s Kappa corresponds to Adjusted Rand . This equivalence allows us to transfer some of the results from [ 12 ] to the context of binary classification . However , an important contribution of our work is the extension of the properties and analysis to the multiclass case . We also prove an impossibility theorem stating that some of the desirable properties can not be simultaneously satisfied and develop a new family of measures having all properties except one . Similarly to [ 12 ] , we note that all the discussed properties are invariant under linear transformations and interchanging true and predicted labelings . Hence , we may restrict to measures for which higher values indicate higher similarity between classifications . Table 2 summarizes our findings : for each measure , we mathematically prove or disprove each desirable property . Further in this section , we refer only to known measures ( above the line ) , while the remaining ones will be defined and analyzed in Section 4 . In addition to individual measures , we also analyze the properties of micro , macro , and weighted multiclass averagings : for each averaging , we analyze whether it preserves a given property , assuming the binary classification measure satisfies it . All the proofs can be found in Appendix C. Let us now define and motivate each property . 3.1 Maximal and minimal agreement . These properties make the upper and lower range of a performance measure interpretable . The maximal agreement property requires the measure to have an upper bound that is only achieved when the compared labelings are identical . Definition 1 . We say that a measure M satisfies maximal agreement if there exists a constant cmax such that for all C , M ( C ) ≤ cmax with equality iff C is diagonal . Also , for a given true labeling , there are several “ worst ” predictions , i.e. , labelings that are wrong everywhere . This leads to the following property . Definition 2 . We say that a measure M satisfies minimal agreement if there exists a constant cmin such that for all C , M ( C ) ≥ cmin with equality iff the diagonal of C is zero , i.e. , cii = 0 for all i . These properties allow for an easy and intuitive interpretation of the measure ’ s values . While all of the measures in Table 2 do satisfy maximal agreement , there are popular measures such as Recall ( c11/a1 ) and Precision ( c11/b1 ) that do not satisfy this property as the maximum can also be achieved when the compared classifications are not identical . For minimal agreement , many performance measures violate it . For example , Cohen ’ s Kappa is obtained from accuracy by subtracting the expected value of accuracy and normalizing the result . As a result of the particular normalization used , it has minimal value − ( ∑m−1 i=0 aibi ) / ( n2 − ∑m−1 i=0 aibi ) , which is clearly not constant . If a binary measure satisfies maximal agreement , then its multiclass variant obtained via micro , macro , or weighted averaging also satisfies this property as each one-vs-all binary classification agrees maximally . However , this does not hold for minimal agreement : though each one-vs-all binary classification will have zero true positives , the number of true negatives may still be positive .

Summary: The core question studied in this paper is how to choose a best performance measure (termed as "index" by the authors) in all situations for binary and multiclass classifications. To answer this question, the authors define several properties that are important for considerations when one wants to choose a good index. Then many common indices (performance measures) in classification tasks are analyzed to see if they satisfy each of the properties defined, including Accuracy, F-measure, and others. A impossibility theorem is proved to show that some desirable properties cannot be satisfied at the same time for a given index. With this theorem, a family of indices is proposed that satisfies all good properties except one.