Good Classification Measures and How to Find Them

1 Introduction . Classification is a classic machine learning task that is used in countless applications . To evaluate classification results , one has to compare the predicted labeling of a given set of elements with the actual ( true ) labeling . For this , performance measures are used , and there are many well-known ones like accuracy , F-measure , and so on [ 13 , 15 ] . The fact that different measures behave differently is known throughout the literature [ 10 , 13 , 20 , 21 ] . For instance , accuracy is known to be biased towards the majority class . Thus , different measures may lead to different evaluation results , and it is important to choose an appropriate measure . While there are attempts to compare performance measures and describe their properties [ 3 , 6 , 13 , 14 , 26 , 28 ] , the problem still lacks a systematic approach , and our paper aims at filling this gap.1 Our research is particularly motivated by a recent paper [ 12 ] providing a systematic analysis of evaluation measures for the clustering task . We transfer many proposed properties to the classification problem and extend the research by adding more properties , new measures , and novel theoretical results . To provide a systematic comparison of performance measures , we formally define a list of properties that are desirable across various classification tasks . The proposed properties can be applied both to binary and multiclass problems . Some properties are intuitive and straightforward , like symmetry , while others are more tricky . A particularly important property is called constant baseline . It requires a measure not to be biased towards particular predicted class sizes . For each measure and each 1We describe related research in detail in Appendix A . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . property , we formally prove or disprove that the property is satisfied . We believe that this analysis is essential for better understanding the differences between the performance measures . Then , we analyze relations between different properties in the binary case and prove an impossibility theorem : it is impossible for a performance measure to be linearly transformable to a metric and simultaneously have the constant baseline property . This means that at least one of these properties has to be discarded . If we relax the set of properties by discarding the distance requirement , the remaining ones can be simultaneously satisfied . In fact , we propose a family of measures called Generalized Means ( GM ) , satisfying all the properties except distance and generalizing the well-known Matthews Correlation Coefficient ( CC ) . In addition to CC , this class also contains another intuitive measure that we name Symmetric Balanced Accuracy . To the best of our knowledge , this measure has not been previously used for classification evaluation.2 If we instead discard the constant baseline ( but keep its approximation ) , then the arccosine of CC is a measure satisfying all the properties . We also demonstrate through a series of experiments that different performance measures can be inconsistent in various situations . We notice that measures having more desirable properties are usually more consistent with each other . We hope that our research will motivate further studies analyzing the properties of performance measures for classification and other problems since there are still plenty of questions to be answered . 2 Performance measures for classification . In this section , we define measures that are commonly used for evaluating classification results . Classification problems can be divided into binary , multiclass , and multilabel . In this paper , we focus on binary and multiclass and leave multilabel for future research . There are several types of performance measures : threshold measures assume that predicted labels deterministically assign each element to a class ( e.g. , accuracy ) ; probability measures assume that the predicted labels are soft and compare these probabilities with the actual outcome ( e.g. , the cross-entropy loss ) ; ranking measures take into account the relative order of the predicted soft labels , i.e. , quantify whether the elements belonging to a class have higher predicted probabilities compared to other elements ( e.g. , area under the ROC curve or average precision ) . Our research focuses on threshold measures . Now we introduce notation needed to formally define binary and multiclass threshold measures.3 Let n > 0 be the number of elements in the dataset and let m ≥ 2 denote the number of classes . We assume that there is true labeling classifying elements into m classes and also predicted labeling . Let C be the confusion matrix : each matrix element cij denotes the number of elements with true label i and predicted label j . For binary classification , c11 is true positive ( TP ) , c00 is true negative ( TN ) , c10 is false negative ( FN ) , and c01 is false positive ( FP ) . We use the notation ai = ∑m−1 j=0 cij , bi =∑m−1 j=0 cji for the sizes of i-th class in the true and predicted labelings , respectively . Finally , we denote classification measures byM ( C ) orM ( A , B ) , whereA andB are true and predicted labelings , and write M ( c11 , c10 , c01 , c00 ) for binary ones . Table 1 ( above the line ) lists several widely used classification measures . The most well-known is accuracy which is the fraction of correctly classified elements . Accuracy is known to be biased towards the majority class , so it is not appropriate for unbalanced problems . To overcome this , Balanced Accuracy re-weights the terms to treat all classes equally . Cohen ’ s Kappa uses a different approach to overcome this bias : it corrects the number of correctly classified samples by the expected value obtained by a random classifier [ 5 ] . Matthews Correlation Coefficient is the Pearson correlation coefficient between true and predicted labelings for binary classification [ 11 ] . For the multiclass case , covariance is computed for each class , and the obtained values are averaged before computing the correlation coefficient . Finally , Confusion Entropy computes the entropy of the misclassification distribution for each class and combines the obtained values , see Table 1 and [ 30 ] for the details.4 Some measures are exclusively defined for binary classification . In this case , the classes are often referred to as ‘ positive ’ and ‘ negative ’ . Jaccard measures the fraction of correctly detected positive 2For clustering evaluation , there is an analog known as Sokal & Sneath ’ s measure [ 12 ] . 3For convenience , we list the notation used in the paper in Table 7 in Appendix . 4There can be cases when a class is not present in the predicted labels . Then , some measures may contain division by zero . A proper way to fill in such singularities is discussed in Appendix B. examples among all positive ones ( both in true and predicted labelings ) . F-measure is the ( possibly weighted ) harmonic mean of Recall ( c11/a1 ) and Precision ( c11/b1 ) . For measures that do not have a natural multiclass variant , there are several universal extensions obtained via averaging the results for m one-vs-all binary classifications [ 17 ] . For each one-vs-all classification , a particular class i is considered positive while all other classes are grouped to a negative class . Micro averaging sums up all binary confusion matrices corresponding to m one-vs-all classifications . Formally , it sets true positive as ∑m−1 i=0 cii , false negative and false positive as n− ∑m−1 i=0 cii , true negative as ( m− 2 ) n+ ∑m−1 i=0 cii . Then , a given binary measure is applied to the obtained matrix . Macro averaging computes the measure values for m binary classification sub-problems and then averages the results : 1m ∑m−1 i=0 M ( cii , ai − cii , bi − cii , n− ai − bi + cii ) , where M ( · ) is a given binary measure . Note that macro averaging gives equal weights to all one-vs-all binary classifications . In contrast , weighted averaging weights one-vs-all binary classifications according to the sizes of the corresponding classes : 1n ∑m−1 i=0 ai ·M ( cii , ai − cii , bi − cii , n− ai − bi + cii ) . 3 Properties of validation measures . As clearly seen from the above discussion , there are many options for classification validation . In this section , we propose a formal approach that allows for a better understanding the differences between the measures and for making an informed decision among them for a particular application . For this , we propose properties of validation measures that can be useful across various applications and formally check which measures satisfy which properties . In this regard , we follow the approach proposed in [ 12 ] for comparing validation measures for clustering tasks . First , we observe that some theoretical results from [ 12 ] are related to binary classification measures . Indeed , a popular subclass of clustering validation measures are pair-counting ones . Such measures are defined in terms of the values N11 , N10 , N01 , N00 that essentially define a confusion matrix for binary classification on element pairs . Thus , replacingNij in pair-counting clustering measures by cij , results in binary classification measures . We refer to Table 8 in Appendix B for the correspondence of some classification and clustering measures . In particular , Accuracy is equivalent to Rand , while Cohen ’ s Kappa corresponds to Adjusted Rand . This equivalence allows us to transfer some of the results from [ 12 ] to the context of binary classification . However , an important contribution of our work is the extension of the properties and analysis to the multiclass case . We also prove an impossibility theorem stating that some of the desirable properties can not be simultaneously satisfied and develop a new family of measures having all properties except one . Similarly to [ 12 ] , we note that all the discussed properties are invariant under linear transformations and interchanging true and predicted labelings . Hence , we may restrict to measures for which higher values indicate higher similarity between classifications . Table 2 summarizes our findings : for each measure , we mathematically prove or disprove each desirable property . Further in this section , we refer only to known measures ( above the line ) , while the remaining ones will be defined and analyzed in Section 4 . In addition to individual measures , we also analyze the properties of micro , macro , and weighted multiclass averagings : for each averaging , we analyze whether it preserves a given property , assuming the binary classification measure satisfies it . All the proofs can be found in Appendix C. Let us now define and motivate each property . 3.1 Maximal and minimal agreement . These properties make the upper and lower range of a performance measure interpretable . The maximal agreement property requires the measure to have an upper bound that is only achieved when the compared labelings are identical . Definition 1 . We say that a measure M satisfies maximal agreement if there exists a constant cmax such that for all C , M ( C ) ≤ cmax with equality iff C is diagonal . Also , for a given true labeling , there are several “ worst ” predictions , i.e. , labelings that are wrong everywhere . This leads to the following property . Definition 2 . We say that a measure M satisfies minimal agreement if there exists a constant cmin such that for all C , M ( C ) ≥ cmin with equality iff the diagonal of C is zero , i.e. , cii = 0 for all i . These properties allow for an easy and intuitive interpretation of the measure ’ s values . While all of the measures in Table 2 do satisfy maximal agreement , there are popular measures such as Recall ( c11/a1 ) and Precision ( c11/b1 ) that do not satisfy this property as the maximum can also be achieved when the compared classifications are not identical . For minimal agreement , many performance measures violate it . For example , Cohen ’ s Kappa is obtained from accuracy by subtracting the expected value of accuracy and normalizing the result . As a result of the particular normalization used , it has minimal value − ( ∑m−1 i=0 aibi ) / ( n2 − ∑m−1 i=0 aibi ) , which is clearly not constant . If a binary measure satisfies maximal agreement , then its multiclass variant obtained via micro , macro , or weighted averaging also satisfies this property as each one-vs-all binary classification agrees maximally . However , this does not hold for minimal agreement : though each one-vs-all binary classification will have zero true positives , the number of true negatives may still be positive .

Summary: This paper proposed a systematic approach to choosing a suitable validation index for classification tasks. The authors considered several binary and multiclass indices, including new ones. They examined the indices with five sets of properties based on the study [6,18_Supplementary]. For each index and each property, they formally proved or disproved that the property is satisfied. They also presented the impossibility theorem for classification.