Good Classification Measures and How to Find Them
Authors: Martijn Gösgens
Anton Zhiyanov
Alexey Tikhonov
Liudmila Prokhorenkova
Venue: NeurIPS 2021 Poster (2021)
Keywords: classification
classification evaluation
performance measures
symmetric balanced accuracy
correlation coefficient

Abstract: Several performance measures can be used for evaluating classification results: accuracy, F-measure, and many others. Can we say that some of them are better than others, or, ideally, choose one measure that is best in all situations? To answer this question, we conduct a systematic analysis of classification performance measures: we formally define a list of desirable properties and theoretically analyze which measures satisfy which properties. We also prove an impossibility theorem: some desirable properties cannot be simultaneously satisfied. Finally, we propose a new family of measures satisfying all desirable properties except one. This family includes the Matthews Correlation Coefficient and a so-called Symmetric Balanced Accuracy that was not previously used in classification literature. We believe that our systematic approach gives an important tool to practitioners for adequately evaluating classification results.

Good Classification Measures and How to Find Them
Martijn Güsgens
Anton Zhiyanov
Eindhoven University of Technology Eindhoven, The Netherlands
Yandex Research, HSE University Moscow, Russia zhi zhiyanovap@@mail com
Alexey Tikhonov
Liudmila Prokhorenkova
Yandex Berlin, Germany altsoph@gmail. com
Yandex Research, HSE University, MIPT
Moscow, Russia ostroomova-la@yandex.rr
Abstract
Several performance measures can used for evaluating classification results: accuracy, F-measure, and many others. Can we say that some of them are better than others, or, ideally, choose one measure that best in all situations? To answer this question, conduct systematic analysis of classification performance measures: we formally define a list of desirable properties and theoretically analyze which measures satisfy which properties. We also prove impossibility theorem: desirable properties cannot be simultaneously satisfied. Finally, we propose a new family of measures satisfying all desirable properties except one. This family includes the Matthews Correlation Coefficient and a so-called Symmetric Balanced Accuracy that our systematic approach gives an important tool to practitioners for adequately evaluating classification results.
previously used in classification literature. We believe that
Introduction
Classification is a classic machine learning task that is used in countless applications. To evaluate classification results, one has to compare the predicted labeling of a given set of elements with the actual (truc) labeling. For this, performance measures
used, and there many well-known ones 15| The fact that different measures behave differently For instance, accuracy is known to be biased towards the majority class. Thus, different measures may lead to different evaluation results, and is important choose appropriate measure. While there attempts compare performance
like accuracy, F-measure, and so on [3
is known throughout the literature [TO
13
20
21
26 28), the problem still lacks a systematic approach, and our paper aims filling this gap Our research is particularly motivated by a recent paper [12 providing a systematic analysis of evaluation measures for the clustering task. We transfer many proposed properties the classification problem and extend the research by adding more properties, new measures, and novel theoretical results.
measures and describe their properties 33, 6 13
14
To provide systematic comparison of performance measures, formally define a list of properties that are desirable across various classification tasks. The proposed properties can be applied both to binary and multiclass problems. Some properties intuitive and straightforward, like symmetry, while others are more tricky. particularly important property called constant baseline. requires measure not to be biased towards particular predicted class sizes. For each measure and each
describe related research detail Appendix
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

property, we formally prove or disprove that the property is satisfied. We believe that this analysis is essential for better understanding the differences between the performance measures.
Then, we analyze relations between different properties in the binary case and prove an impossibility theorem: it is impossible for a performance measure to be linearly transformable to metric and simultaneously have the constant baseline property. This means that at least one of these properties has be discarded. If we relax the set of properties by discarding the distance requirement, the remaining ones can be simultaneously satisfied. In fact, we propose family of measures called Generalized Means (GM), satisfying all the properties except distance and generalizing the well-known Matthews Correlation Coefficient (CC). In addition to CC, this class also contains another intuitive measure that we name Symmetric Balanced Accuracy. To the best of our knowledge, this measure has not been previously used for classification evaluation 2 If we instead discard the constant baseline (but keep its approximation), then the arccosine of CC a measure satisfying all the properties.
We also demonstrate through a series of experiments that different performance measures can be inconsistent in various situations. We notice that measures having more desirable properties are usually more consistent with each other.
We hope that our research will motivate further studies analyzing the properties of performance measures for classification and other problems since there are still plenty of questions to be answered.
2 Performance measures for classification
In this section, we define measures that are commonly used for evaluating classification results. Classification problems can be divided into binary, multiclass, and multilabel. In this paper, we focus on binary and multiclass and leave multilabel for future research. There are several types of performance measures: threshold measures assume that predicted labels deterministically assign each element to class (e.g., accuracy); probability measures assume that the predicted labels are soft and compare these probabilities with the actual outcome (e.g., the cross-entropy loss); ranking measures take into account the relative order of the predicted soft labels, i.e., quantify whether the elements belonging to a class have higher predicted probabilities compared to other elements (c.g., area under the ROC curve average precision). Our research focuses threshold measures.
Now we introduce notation needed to formally define binary and multiclass threshold measures Let > 0 be the number of elements in the dataset and let m 2 2 denote the number of classes. We assume that there is true labeling classifying elements into m classes and also predicted labeling. Let C be the confusion matrix: each matrix element Cij denotes the number of elements with true label i and predicted label j. For binary classification, C11 is true positive (TP), Coo is true negative (TN), Ci0 is false negative (FN), and coi is false positive (FP). We use the notation a, = En= Cij. b, = Σy=o Cjf for the sizes of i-th class in the true and predicted labelings. respectively. Finally, we denote classification measures by M (C) or ( B)). where and B are true and predicted labelings, and write M (CLL C10. CoL, coo) for binary ones.
Table (above the line) lists several widely used classification measures. The most well-knoww is accuracy which is the fraction of correctly classified elements. Accuracy is known to be biased towards the majority class, so not appropriate for unbalanced problems. To overcome this, Balanced Accuracy re- re-weights the terms to treat all classes equally. Cohen's 's Kappa uses a different approach to overcome this bias: corrects the number of correctly classified samples by the expected value obtained by a random classifier 5 Matthews Correlation Coefficient is the Pearson correlation coefficient between true and predicted labelings for binary classification [I]. For the multiclass case, covariance is computed for each class, and the obtained values are averaged before computing the correlation coefficient. Finally, Confusion Entropy computes the entropy of the misclassification distribution for each class and combines the obtained values, see Table | and [30] for the details !
Some measures are exclusively defined for binary classification. In this case, the classes are often referred to as "positive" and 'negative'. Jaccard measures the fraction of correctly detected positive
Forr clustering evaluation, there is an analog known as 2 For convenience, we list the notation used in the paper Table in Appendix. 'There can be cases when a class is not present in the predicted labels. Then, some measures may contain division by zero. A proper way fill in such singularities is discussed in Appendix B
's measure 12

---

Table Commonly used (above the line) and novel (below the line) validation measures
Binary
Multiclass
F-measure (Fp)
(1+22)
micro macro / weighted
Jaccard (J)
Gu
micro macro / weighted
+000
Matthews Coefficient (CC)
22.10 Oxs 2110 b.o.
- 62) (n² - +?)
Accuracy (Acc)
Σ - 9i
Balanced Accuracy (BA)
a-md dii
Cohen's Kappa (1))
Σ
m -1
Gi -4:-2 -i= ai
n² -i-0 a,6,
Confusion Entropy (CE)
Σ
Cje logloggm-2
94
ogg + Cij loggm-2 aytbs)
2n
aytby
Symmetric Balanced Accuracy (SBA)
mm-1 ===
CH micro/ macro / weighted
Cit
2m
Generalized Mcans (GM)
Correlation Distance (CD)
arccos
examples among all positive ones (both in true and predicted labelings). F-measure is the (possibly weighted) harmonic mean of Recall (c11/1) and Precision (c11 (b)./b)) For measures that do not have natural multiclass variant, there are several universal extensions obtained via averaging the results for one-vs-all binary classifications [17]. For each one-vs-all classification, particular class i is considered positive while all other classes are grouped to a negative class.
Micro averaging sums up all binary confusion matrices corresponding to m one-vs-all classifications. 1-- Cas, false negative and false positive as > true negative as (m 2)n + - Ca.. Then, a given binary measure is applied the obtained matrix.
Formally, it sets true positive as
E
Macro averaging computes the measure values for m binary classification sub-problems and then in
averages the results: 1 m- (cns a; Cits b - Cis n a; b, + czz), where M() (-) is a given binary measure. Note that maero averaging gives equal weights all one-vs-all binary classifications.
In contrast, weighted averaging weights one-vs-all binary classifications according to the sizes the corresponding classes: 1 Em 0g M(a,, a; cas by - Cas n a; - b, + ca)-
3 Properties of validation measures
As clearly seen from the above discussion, there are many options for classification validation. In this section, we propose a formal approach that allows for better understanding the differences between the measures and for making an informed decision among them for particular application. For this, we propose properties of validation measures that can be useful across various applications and formally check which measures satisfy which properties. In this regard, we follow the approach proposed in 12 for comparing validation measures for clustering tasks.
First, we observe that some theoretical results from 12 are related to binary classification measures. Indeed, popular subclass of clustering validation measures are pair-counting ones. Such measures are defined in terms of the values N11, No No1, Noo that essentially define a confusion matrix for binary classification on element pairs. Thus, replacing N., pair-counting clustering measures by Cij. results in binary classification measures. We refer to Table Appendix B for the correspondence of some classification and clustering measures. In particular, Accuracy equivalent to Rand, while Cohen's 's Kappa corresponds to Adjusted Rand. This equivalence allows us to transfer some of the results from [2] to the context of binary classification. However, an important contribution of our work is the extension of the properties and analysis to the multiclass case. We also prove an impossibility theorem stating that some of the desirable properties cannot be simultaneously satisfied and develop new family of measures having all properties except one.
3

---

Table Properties of validation measures and averagings, VIx indicates that property satisfied only in binary case
Mcasure
Max
Min
CSym
Sym
Dist
Mon
SMon
CB
ACB
Fi (binary) (binary)
X
X
X
X
X
X
1
X
X
X
X
CC
JIX
J
J
XX
JIX
VIX
V
Acc
J
J
X
BA
J
X
J
V
V
X
VIX
X
J
V
CE
X
X
SBA
GM (binary)
x
/
CD
VIX
VIX
VIX
Preserving properties by various avcraging types
Micro
X
Macro
x
x
Weighted
x
x
X
X
note that all the discussed properties invariant under linear transformations and interchanging true and predicted labelings. Hence, we may restrict to measures for which higher values indicate higher similarity between classifications.
Similarly 12.
Table summarizes our findings: for each measure, we mathematically prove or disprove each desirable property. Further in this section, refer only to known measures (above the line), while the remaining ones will be defined and analyzed in Section 4 In addition to individual measures, we also analyze the properties of micro, macro, and weighted multiclass averagings: for each averaging, we analyze whether it preserves a given property, assuming the binary classification measure satisfies Let define and motivate each property.
it. All the proofs can be found in Appendix C
3.1 Maximal and minimal agreement
These properties make the upper and lower range of performance measure interpretable. The maximal agreement property requires the measure to have upper bound that is only achieved when the compared labelings are identical.
Definition 1. We say that measure M satisfies maximal agreement if there exists constant Cmax such that for all C, MCC S Cmax with equality iff C is diagonal.
Also, for given true labeling, there are several "worst" predictions, i.e., labelings that are wrong everywhere. This leads to the following property.
Definition 2. We say that measure M satisfies minimal agreement if there exists a constant Cmin such that for all C, MC) 2 Cmin with equality iff the diagonal of C is zero, i.e., Ci. = 0 for all i.
These properties allow for an easy and intuitive interpretation of the measure's 's values. While all of the measures in T satisfy maximal agreement, there are popular measures such Recall (C11/az) and Precision (c11/b,) that do not satisfy this property as the maximum can also be achieved when the compared classifications are not identical. For minimal agreement, many performance measures violate it. For example, Cohen's 's Kappa is obtained from accuracy by subtracting the expected value of accuracy and normalizing the result. As result of the particular normalization
used, it has minimal value -EEm a.b.) / (n² - a.b.). which is clearly not constant.
binary measure satisfies maximal agreement, then multiclass variant obtained via miero, macro, or weighted averaging also satisfies this property as each one-vs-all binary classification agrees maximally. However, this does not hold for minimal agreement: though each one-vs-all binary classification will have zero true positives, the number of true negatives may still be positive.

---

3.2 Symmetry
Definition 3. We say that a measure M is symmetric if MC() M(C)) holds for all C.
In other words, we require symmetry with respect to interchanging predicted and true labels. This property is often desirable since similarity usually understood as a symmetric concept. However, in some specific applications, there may be reasons to treat the true and predicted labelings differently and thus use an asymmetric measure. An example of an asymmetric measure is Balanced Accuracy.
Let us also introduce class-symmetry, i.e., invariance to permuting the classes.
Definition& 4. We say that a measure M is class-symmetric if. for any permutation π of the classes {I...., m. and any confusion matrix C, MCC = (C) holds, where Ć is given by čij- = +()))
Note that known multiclass measures are all llas-syymmtric,, while in binary classification tasks, there can be an asymmetry between 'positive' and 'negative' classes. Examples of well-known class-asymmetric binary classification measures are Jaccard and Fi.
3.3 Distance
In some applications, it is desirable to have a distance interpretation of a measure: whenever labeling A is similar to B, while B is similar to C, it should intuitively hold that A is also somewhat similar to C. For instance, it can be the case that the actual labels are unknown, and the labeling A is only an approximation of the truth. Then, we would want the similarity between predicted labels and A to be not too different from the similarity between predicted and the actual true labels. This would be guaranteed if the measure is a distance.
Definition 5. A measure has distance property if it can be linearly transformed to a metric distance.
A function d(A, B) is metric distance if it is symmetric, nonnegative, equals zero only when A = B, and satisfies the triangle inequality d(A, < d(A, + d(B, C.). Note that the first requirement is equivalent to symmetry (Definition B). while the second and third imply maximal agreement (Definition D Furthermore, note that if d is a distance, then e is also a distance for any e > O. Therefore, we can conclude that M is distance if and only if M satisfies symmetry and maximal agreement while Cmax - (A, B) satisfies the triangle inequality.
While most measures cannot be linearly transformed to a distance, some measures do satisfy this property. For example, the Jaccard measure can be transformed to the Jaceard distance - J(A, B). Similarly, Accuracy can be transformed to distance by 1 Acc(A, B).
3.4 Monotonicity
Monotonicity is one of the most important properties of a similarity measure: intuitively, changing one labeling such that it becomes more similar to the other ought to increase the similarity score. Then, to formalize monotonicity, we need to determine what changes make the classifications A and B more similar to each other. The simplest option is to take element on which A and B disagree and resolve this disagreement.
Definition 6. A measure M is monotone if M(C) < M(C) for any confusion matrices C and Ć such that Ć is obtained from C by decrementing off- off-diagonal entry Cab and incrementing Coa or Cbb and none of the row- or column- sums of C equal n.
The condition C is equivalent neither A nor B labeling all elements the same class. We need this to prevent contradictions with the constant baseline property that will be defined in Section 3.5
Definition 6 partial ordering over confusion matrices with the same total number of elements. However, we can relax the latter restriction and obtain the following, stronger notion of monotonicity that defines partial ordering across different numbers of elements.
Definition 7. measure M is strongly monotone if MC) < M(C) for any confusion matrices C and Č such that Č is obtained from C by either increasing a diagonal entry or decreasing an off- off-diagonal entry. Here we require that none of the row- or column- summs of C equal n and that C and C are not simultaneously diagonal or sero-diagonal matrices.
5

---

The last condition is needed since otherwise the definition would contradict the maximal (or minimal) agreement properties as (C) = Cmax 2 M()) holds when C is diagonal.
All measures in Table 2 except for CE and multiclass CC and CD satisfy monotonicity from Definition
6 Strong monotonicity is violated by many measures: for instance, the widely used Fi, Jaccard and Cohen's 's Kappa do not satisfy this intuitive property.
3.5 Constant baseline
The constant baseline is perhaps the most important non-trivia property. On the one hand, it ensures that a measure is not biased towards labelings with particular class sizes bi....,bm. On the other hand, it also ensures some interpretability for 'mediocre' predictions.
Intuitively, if predicted labels are drawn at random and independently of the true labels, we would expect them to have low similarity with the true labels. Then, if another prediction has similarly low score, we can say that it is roughly as bad as a random guess. However, this is only possible when such random classifications achieve similar scores, independent of their class sizes. To formalize this, let (bb....... bm) denote the uniform distribution over labelings with class sizes bi..... bm. We say that the class sizes bi...., 6m are unary if b; for some E (L.....)) mm) That is, if all elements get classified to the same class, so that (bi...... bm) is a constant distribution.
Definition 8. We say that a measure M has a constant baseline property if there exists (base(m) that does not depend on but may depend m, such that for any true labeling A and non non-unary class sizes bt....,br bm. it holds that B--.U(b......bb bm.) M (A, B)] = Cbase (m)
Note that we need to require the class sizes to be non-unary: if these class sizes are unary, we will have contradictions with maximal and minimal agreement when the class sizes of A are also unary. Many popular measures such as Fi. Accuracy, and Jaccard do not have a constant baseline. Furthermore, some measures that do have a constant baseline were deliberately designed to have one. For example, Cohen's 's Kappa obtained from accuracy by correcting it for chance. While definition of the constant baseline does allow for a baseline Quase (m) that depends on the number of classes m, some measures such the Matthews Coefficient and Cohen's 's Kappa have a baseline that is constant w.r.t. m.
All of the measures that satisfy constant baseline turn out to be linear functions of ca for fixed class sizes (1, ..,, Um and
Um. For such measures, linearity of the expectation can be utilized to easily compute the baseline by substituting the expected values EB ... (n))cc)) [css] = a.b, Thus, m
we also propose the following relaxation of the constant baseline property.
Definition 9. measure M is said to have an approximate constant baseline if there exists a function Coase (m) that does not depend on but may depend
such that for any class sizes a1,-..y Om o.by
and any non-unary "1,...,bme M MC) = Chase(m) where Cij
The advantage of this relaxation is that it allows us to non-linearly transform measures while still maintaining an approximate constant baseline. Take for example the Matthews Correlation Coefficient: it cannot be linearly transformed to distance while the transformations CD(A) B) arccos( (C,, B)) and 2 - CC(A, B)) do yield distances. Because Correlation Coefficient has a constant baseline, these non-linear transformations have an approximate constant baseline, see Section 4 for more details.
As can be seen from Table there is measure satisfying all the properties. In particular, there is no measure having both distance and constant baseline. In the next section, we show why this is not a coincidence.
4 Impossibility theorem for classification
In this section, we focus on binary classification and more deeply analyze the relations between the properties discussed above. Unfortunately, it turns out that the properties introduced in the previous section cannot all be satisfied simultaneously.
Theorem 1. There is no binary classification measure that simultaneously satisfies the monotonicity, distance, and constant baseline properties.

---

Proof. Let A be a labeling with a single positive and n 1 negatives. Let B1 be random labeling with a single positive and let B2 be a random labeling with two positives. The constant baseline requires R A, B1 I M (A, B2)], which gives
1 +
n -
n- (0,1,22,1 -3), 2 M(100,1, - ())
Cmax
M( 1,1,1 2)
7
n
n
which we rewrite to
1,,,,,
Cmax = (0, 1,,, (n-
2) (0, 1,2, n-2nn-))
(1)
Now, we consider labeling C with single positive that does not coincide with the positive of A and labeling B that has two positives which are the positives of A and C. The triangle inequality tells us that
Cmax (0, 1, n-) -) (1,,,,,,)), 2 - 0,
where the last step follows from symmetry (implied by distance). This is rewritten to
2M 11, 2) Cmax I (0, 1, 2).
(2)
Combining and 2 we obtain
(n 10, 1, 2) (n 20, 10,,, s M
We rewrite this to (0, 1, n 2) - M (0, 1,2, n 3), which clearly contradicts monotonicity.
Thus, we have to discard one of these properties. Obviously, discarding monotonicity would be highly undesirable since higher values would then not necessarily indicate higher similarity. For this reason, we analyze what happens if we discard either distance or constant baseline. All the results stated below are proven in Appendix D
Discarding distance by all measures discussed in this paper, we prove the following result.
Assuming some additional smoothness conditions that are, however, satisfied
Theorem 2. All binary measures that satisfy all properties except distance must be of the form
agaa n2
bubi n2
nC11 aibi
n²
where the normalization factor s(a, b) needs to satisfy some additional properties listed in Theorem
This class of measures is quite wide and contains many unelegant measures. An interesting subclass can be obtained we normalize by generalized mean, i.e., take sa, b) (g + 2/r)
Definition 10. For IR, we define Generalized Means measures as
n.111 01b1
GM,
V
Statement For any r R. the measure GM, satisfies all properties except for being a distance.
We also show that the Generalized Means measures contain two interesting special cases.
Statement Ifr + (corresponding the geometric mean). GM,(C) CC(C). Ifr (corresponding the harmonic mean), GC) BA(C) BA(CT) -1
Thus, for -1 Generalized Means equivalent to the measure 1 (BA(C) BB(CT)) that we call Symmetric Balanced Accuracy (SBA). To the best of our knowledge, this measure has not been used in the classification literature. However, in the clustering literature, similar measure is known as Sokal&Sneath's 's measure 12]. Interestingly. SBA preserves its properties for the multiclass case.
Statement 3. SBA satisfies all properties except for being distance for any > 2.
7

---

Discarding (exact) constant baseline Note that Theorem I onlly proves an impossibility for the exact constant baseline, but not the approximate constant baseline.
Statement 4. The measures CD(A, B) : (CCAAs(CC(A, B)) and CD'(A, ==
CC(1-C((A,, B)) satisfy all properties except the exact constant baseline, but including the approximate constant baseline.
Following 112 we call the measure arccos(CC(A,, B)) Correlation Distance (CD). We prove the following result (see Appendix [D2 for the details).
Statement 5. CD approximates constant baseline with one order of precision better than CD'.
Essentially, this is consequence of the fact that the transformation (arcoos(CC) is a symmetric function around the constant baseline CC = 0 while /2 - CC) is not. In more detail, we show that the leading error term of CD' is of the order B|CC(A, B)² while the leading error term for CD is of the order EE[C(A, B) Currently, we are not aware of other distance measures for which the constant baseline is approximated up to the same order of precision as CD. We thus argue that for binary classification tasks where a distance interpretation is desirable, Correlation Distance is the most suitable measure.
Inconsistency of measures in practice
In this section, we conduct several experiments that demonstrate how often performance measures may disagree in practice in different scenarios. These experiments demonstrate the importance of the problem considered in this paper and show which measures are usually more consistent than others. For binary classification, we consider all measures from Table i For F-measure, we take B = 1, for Generalized Means, we consider r 1. Recall that SBA and CC are also instances of GM with r and r + 0, respectively. Furthermore, Jaceard is a monotone transformation of Fi, and CD is monotone transformation of CC. Therefore, we omit CD and Jaccard from all inconsistency tables. The code for our experiments can be found on GitHub
5.1 Binary measures
Distinguishing measures for small datasets First, we construct simple examples showing the inconsistency of all pairs of binary classification measures. We say that two measures M1 and M2 are consistent on a triplet of classifications (A, B1, B2) if M1 (A, B1) * M1(A, B2) implies {>, =, Otherwise, we say that the measures are inconsistent. We took E ,,.,, 10} and went through all the possible triplets (A, B1, B2) of binary labelings of elements (we additionally require that all labelings contain both classes). For each triplet, we check which pairs of measures are inconsistent. We say that pair of measures is indistinguishable for given n if is consistent on all triplets.
M2(A, B1) ) M2 (A, B2). where
Table 3 lists all measures that are indistinguishable for given For instance, for n 2, all measures always consistent. For n = 4, we can distinguish Acc, Fi, and CE from other measures and each other. Inter- estingly, the remaining measures are those having the constant baseline property. Importantly, the most consis- tent measures are CC, SBA, and GM, these measures have the best properties according our analysis. This supports our intuition that "good" measures agree with each other better than those having fewer desired prop- erties. Additionally, in Appendix I.I we list six triplets (A, B1, B2) with n 10 that discriminate all pairs of different measures.
Table 3: Indistinguishable measures
73
measures
2
[Ace, BA, FI, K. CE, GM1, CC, SBAJ
3
[Ace, BA, AS, GM1, CC, SBA]
4-5
[BA, K. GM1, CC, SBAJ
6-7
[GM,, CC, SBAJ
8
[CC, SBAJ
9-10
Experiment within a weather forecasting service In this experiment, we aim at understanding whether the differences between measures may affect the decisions made while designing real systems. For this purpose, we conduct an experiment within the Yandex. Weather service.

---

There is a model that predicts the presence/absence of precipitation at particular location [18]. The prediction is made for 12 prediction intervals (horizons): from ten minutes two hours. The original model returns the probability of precipitation, which can be converted to binary labels via a threshold. There are six thresholds used this experiment, which lead six different models. The measures were logged for 12 days. To sum up, for each threshold (model), each day, and each horizon, we have a confusion matrix that can be used to compute a performance measure.
For each pair of measures, we compute how often they are in-
Table 4: Inconsistency of binary measures for rain prediction, %
consistent according to the def-
inition above. For this, we ag- gregate the results over all days that
Acc
BA
Fi
CE
GM,
CC
SBA
Acc BA
96.5
41.0
37.5
3.1
38.7
44.3
55.9
and horizons. Table there are pairs of measures with substantial disagreement: e.g.,
96.5
55.6
58.9
99.7
57.7
52.0
40.4
F1
41.0
55.6
3.3
44.2
2.2
3.4
15.0
K
37.5
58.9
3.3
40.7
1.1
6.7
18.3
accuracy and Balanced Accuracy
CE
3.1
99.7
44.2
40.7
41.9
47.5
59.1
almost always disagree. This can be explained by the fact that ac- curacy has a bias towards the ma- jority class, it prefers higher threshold, while Balanced Accuracy weighs true positives more heavily, so prefers a lower threshold. In contrast, GM1, CC, and Fj agree with each other much better. In Appendix [..] we conduct more detailed analysis. In particular, we separately consider the ten-minute and two-hour prediction horizons and show that the behavior and consistency of measures significantly depend the horizon as the horizon defines the balance between true positives, true negatives, false positives, and false negatives. We also observe that CC and SBA perfectly agree for the ten-minute horizon but have noticeable disagreement for two hours.
GM1
38.7
57.7
2.2
1.1
41.9
5.5
17.1
CC
44.3
52.0
3.4
6.7
47.5
5.5
11.4
SBA
55.9
40.4
15.0
18.3
59.1
17.1
11.4
5.2 Multiclass measures
In this section, we analyze multiclass measures. For all measures that are defined for the multiclass problems, we consider their standard expressions (if not stated otherwise). For other measures ( (Fi, Jaccard, GM1), we macro averaging.
Image classification We conduct an experiment on ImageNet [24]. a classic dataset for image classification. For this, take the top-10 algorithms that are considered to be state-of-the-art the moment of submission @ We check whether the leaderboard based on accuracy consistent with the leaderboards based other measures. Thus, apply the models the test set, compute the confusion matrices, and compare all measures defined in Table
Notably, the ImageNet dataset is balanced. This makes all measures more similar to each other. For instance, accuracy and BA are equal in this scenario. Also, the constant baseline property discussed in Section 3.5 is especially important for unbalanced datasets. Thus, measures more consistent on balanced data. Nevertheless, we notice that the ranking can be inconsistent starting from the algorithm ranked fifth on the leaderboard.
The (partial) results are shown Table Here we compare EfficienntNe-B77 NoisyStudent 31 and Bwin-B Transformer (patch size 4x4, window size 12x12, image size 3842) [19 that are the fifth and sixth models in the leaderboard. One can see that the measures inconsistently rank the algorithms: Confusion Entropy, Jaccard, and SBA disagree with accuracy and other measures. Interestingly, while Jaceard and Fi always agree for binary problems, they may disagree after the macro averaging. as we see in this case. Also, for one measure, different multiclass extensions may be inconsistent, as we see with macro averaging versus the standard definition of the multiclass Correlation Coefficient. More detailed results can be found in Appendix E.2
Sentiment analysis In the previous experiment, we noticed that despite several disagreements, the measures usually rank the algorithms similarly. This can be caused by the fact that the test set of ImageNet is balanced: all classes have equal sizes. However, in practical applications, we
(May 8, 2021).

---

Table 5: Inconsistent results on ImageNet, (fifth and sixth models in the leaderboard)
Acc/BA
1-CE
GM1
CC
CC
SBA
Efficientnet
86.46
86.30 86.27
77.525
86.44
93,41
86.28
86.44
86.419 86.423
86.57
Swin
86.43
77.531
86.42
93.51
86.26
86.42
86.61
Table 6: Ranking algorithms according to different measures on SST-5: from (best) to 7 (worst)
Acc
BA
F1
K
CE
GM1
CC
CCmacra
SBA
Eooo
1
1
Flair+BERT
2
4
5
4
5
2
2
SVM
3
3
3
3
5
3
3
4
4
Logistic
4
5
5
4
5
5
FastText
2
4
VADER
6
6
TextBlob
7
7
7
7
7
7
7
rarely encounter balanced data. Thus, also consider unbalanced classification task. In this experiment, we take the 5-class Stanford Sentiment Treehank (SST-5) dataset [27]. We compare the following algorithms: TobtBlob, VADER, Logistic Regression, SVM, FastText, Flair+ELMo, and Flai++BERT (33] 6 shows that different measures rank the algorithms differently. Among the measures shown the table, the only consistent rankings are the one provided by x and BA and the second given by Fi, GM, and Jaccard. Note that the latter ranking significantly disagrees with the ranking by accuracy.
Appendix E.2 contains an additional experiment with an unbalanced multiclass dataset, where we show the inconsistency rates of the considered measures and different multiclass extensions.
Conclusion and future work
In this paper, propose systematic approach to the analysis of classification performance measures: we propose several desirable properties and theoretically check each property for list of measures. We also prove impossibility theorem: some desirable properties cannot be simultaneously satisfied, so either distance or exact constant baseline has to discarded.
Based on the properties we analyzed in this paper, we come to the following practical suggestions. If the distance requirement is needed, Correlation Distance seems be the best option: satisfies all the properties except for the exact constant baseline, which is still approximately satisfied. Otherwise, we suggest using one of Generalized Means, including Correlation Coefficient and Symmetric Balanced Accuracy they satisfy all the properties except distance. For binary classification, CC is natural choice as it can be non-linearly transformed to a distance. For multiclass problems, Symmetric Balanced Accuracy has additional advantage: among the considered measures, only this one preserves its good properties the multiclass case. Finally, we do not advise using averagings, but needed, then macro averaging preserves more properties.
There are still many open questions and promising directions for future research. First, would like to see whether one could construct set of desirable properties that can be used as axioms to uniquely define one good measure (or parametrized group of measures). Secondly, is an open problem whether Generalized Means measures in general (or SBA in particular) can be converted to distance via a continuous transformation. Finally, our work does not cover ranking and probability- measures. Thus, we leave aside such widely used measures cross-entropy and AUC. Formalizing and analyzing their properties is an important direction for future research.
Broader impact Our work may help towards reducing certain biases in research. For instance, some measures (e.g., accuracy) are biased towards the majority class. Thus, the bias towards the majority class could be even amplified with the poor metric selection. Our work could provide some clues on how to avoid such a situation.
10

---

Acknowledgments and Disclosure of Funding
Part of this work was done while Martijn Gösgens was visiting Yandex and Moscow Institute of Physics and Technology (Russia). The work of Martijn Gösgens is supported by the Nether- lands Organisation for Scientific Research (NWO) through the Gravitation NETWORKS grant no. 024.002.003.
The authors would like thank Alexander Ganshin, Pert Vytovtov, and Eugenia Elistratova for providing the weather forecasting data.
References
[1] A. N. Albatineh, M. Niewiadomska- Bugaj, and D. Mihalko. On similarity indices and correction for chance agreement. Journal of Classification, 23(2):301-313, 2006.
[2] K. H. Brodersen, S. Ong. K. E. Stephan, and M. Buhmann. The balanced accuracy and its posterior distribution. In 2010 20th international conference on pattern recognition, pages 3121-3124. IEFE, 2010.
[3] D. Chicco and G. Jurman. The advantages of the matthews correlation coefficient (mcc) over fl score and accuracy in binary classification evaluation. BMC genomics, 21(1):1-13, 2020.
[4] S.-S. Choi, S.-H. Cha, and C. C. Tappert. A survey of binary similarity and distance measures. Journal of systemics, cybernetics and informaties, 8(1):43 48, 2010.
[5] J. Cohen. coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37-45 1960.
[6] C. Cortes and M. Mohri. Auc optimization vs. error rate minimization. Advances neural information processing systems, 16(16):313-320 2004.
[7] R. Delgado and D. NúfiezGoonzílez. Enhancing confusion entropy (cen) for binary and multiclass classification. PloS one, 14(1):e0210244 2019.
[8] R. Delgado and X.-A. Tibau. Why cohen's 's kappa should be avoided performance measure in classification. PloS one, 14(9)=e0222916, 2019.
[9] D. Dua and C. Graff. UCI machine learning repository, 2017.
[10] C. Ferri, J. Hernández-Orallo, and R. Modroiu. An experimental comparison of performance measures for classification. Pattern Recognition Letters, 30(1):27-33) 2009.
[11] J. Gorodkin. Comparing two k-category assignments by k-category correlation coefficient. Computational biology and chemistry, 28(5-)):367-37,, 2004.
[12] M. Gösgens, L. Prokhorenkova, and A. Tikhonov. Systematic analysis of cluster similarity indices: How validate validation measures. International Conference on Machine Learning (ICML), 2021.
[13] M. Hossin and M. Sulaiman. A review evaluation metrics for data classification evaluations. International Journal of Data Mining & Knnwledgee Management Process, 2015.
[14] J. Huang and C. X. Ling. Using auc and accuracy in evaluating learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 17(3):299-310, 2005.
[15] N. Japkowicz and M. Shah. Evaluating learning algorithmss a classification perspective. Cambridge University Press, 2011.
[16] S. Kosub. A note on the triangle inequality for the jaccard distance. Pattern Recognition Letters, 1 120:36-38, 2019.
[17] O. Koyejo, N. Natarajan, Ravikumar, and S. Dhillon. Consistent multilabel classification. In NIPS, volume 29, pages 3321-3329 2015.

---

[18] V. Lebedev, V. Ivashkin, I. Rudenko, A. Ganshin, A. Molchanov, S. Ovcharenko, R. Grokhovet- skiy, I. Bushmarinov. and D. Solomentsev. Precipitation nowcasting with satellite imagery. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2680-2688, 2019.
[19] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 2021.
[20] A. Luque, A. Carrasco, A. Martín, and A. de las Heras. The impact of class imbalance in classification performance metrics based on the binary confusion matrix. Pattern Recognition, 91:216-231, 2019.
[21] D. Powers. Evaluation: From precision, recall and f-measure to roc, informedness, markedness & correlation. Journal of Machine Learning Technologies, 2(1):37-63, 2011.
[22] D. M. W. Powers. The problem with kappa. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 345-355, 2012.
[23] Rao. Fine grained sentiment classification.
https://github. comp:t/rob.commprrr08777
fine- grained- gr 2021.
[24] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy,
A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):11-222, 2015.
[25] Scikit-leamn Clustering algorithms. learning.hoer html 2021.
[26] F. Sebastiani. An axiomatically derived measure for the evaluation of classification algorithms. In Proceedings of the 2015 international conference on the theory of information retrieval, pages 11-20, 2015.
[27] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Y. and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language processing, pages 1631-1644 2013.
[28] M. Sokolova and G. Lapalme. A systematic analysis of performance measures for classification tasks. Information processing & management, 45(4))427-437, 2009.
[29] V. Van Asch. Macro-am and micro-averaged averaged evaluation measures. Technical report, 2013.
[30] J.-M. Wei, X.-J Yuan, Q.-h Hu, and S.-Q. Wang. A novel measure for evaluating classifiers. Expert Systems with Applications, 37(5):3799-3899, 2010.
[31] Q. Xic, M.-T. Luong, E. Hovy, and Q Le. Self-training with noisy student improves imagenet classification. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10687-10698, 2020.
12


Reviews:
[{"review_id": "kdPq_asKbpg", "reviewer": "Reviewer_ybxe", "summary": "The core question studied in this paper is how to choose a best performance measure (termed as \"index\" by the authors) in all situations for binary and multiclass classifications. To answer this question, the authors define several properties that are important for considerations when one wants to choose a good index. Then many common indices (performance measures) in classification tasks are analyzed to see if they satisfy each of the properties defined, including Accuracy, F-measure, and others. A impossibility theorem is proved to show that some desirable properties cannot be satisfied at the same time for a given index. With this theorem, a family of indices is proposed that satisfies all good properties except one.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I think this is an interesting and comprehensive study. The guidelines (properties) provided in the paper would potentially benefit practitioners who need to choose a performance measure for a given task. Out of the properties considered, the *constant baseline* is very important and useful. However, my main concern is about the novelty and significance in the machine learning community. See below for the detailed comments.\n\nOriginality.\nThere are two main issues.\n\n1) Overall I feel that this paper is an incremental work and its novelty is of concern. This paper is based on a recent paper [1] (as the authors have acknowledged) that provides a systematic analysis of different indices for the clustering task. The authors transfer most of the properties defined in [1] to the binary and multiclass classification tasks. The transfer to binary tasks is trivial (as the authors have mentioned). One of the new contribution, that the authors claim, is to extend the properties and analyses to the multiclass case. However, I fail to appreciate how multiclass case is different from the clustering case (besides the setting of course), what the hurdles are when one transfers the properties and analyses to multiclass tasks, and whether new techniques are used. It would be more clear if the authors could elaborate the relation with [1]. \n\n2) The second concern is about the position of this paper in the machine learning community. Although this paper might be the first to present a comprehensive systematic study of different indices and properties in binary and multiclass classifications, many prospects have been touched more or less in earlier works.  I have read the related work section in the supplementary material, but I cannot fully recognize the position of this work, partially because much cited work is not from the usual machine learning venues, and I do not have enough information to bridge this paper with the background. Therefore, I decide to raise this concern to see if other reviewers could help better position this paper. Also, the authors should provide more background and related work from the machine learning venues.\n\nQuality.\nThis work is theoretical and overall is technically sound. The claims are backed by theory and experiments. \n\nClarity.\nThis paper overall is clear, and is able to communicate the main message to the reader, though some improvements are still needed. Here are some suggestions.\n\n1) Some terminologies need more explanations, for example, *reference* (true labels), *candidate* (predicted labels), *index* (performance measure), *validation index* (target performance measure). I do not know whether these terms are standard in other community, but it is the first time I have seen these terms. It would be helpful if the authors could use standard terminologies in machine learning or explain the terms so that this paper would be more accessible to people in the machine learning community. As I have said earlier that I think this work may be beneficial for practitioners working on machine learning, it is important to make this paper more accessible.\n\n2) Tables need more details in the captions to help understand the content. Please consider moving some of the details from the body of the paper to the caption of a table.\n\nSignificance.\nThis paper is a comprehensive systematic study of different indices and properties in binary and multiclass classifications. The properties studied in this work can potentially serve as guidelines for practitioners who use machine learning to help them find a good performance measure for a given task. However, the use of some nonstandard terminologies and the lack of background and related work in machine learning can have negative effects on the significance and accessibility of this paper.\n\nTake the above into consideration, especially my concern about the novelty, I would vote for a weak accept.\n\n---\n\nOther questions and comments:\n\n1) Is index A with more properties ($P_A$) strictly better than index B with less properties ($P_B$ where $P_B \\subsetneqq P_A$)? Should practitioners choose the index with as many properties as possible?\n\n2) What might be the slide effect for an index with many properties? Is it harder to compute?\n\n---\n\nTypo:\n\nPage 2, line 41, \"MCC\" -> \"CC\"\n\n---\nCitation:\n\n[1] \nGösgens, M.M., Tikhonov, A. &amp; Prokhorenkova, L.. (2021). Systematic Analysis of Cluster Similarity Indices: How to Validate Validation Measures. <i>Proceedings of the 38th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 139:3799-3808 Available from http://proceedings.mlr.press/v139/gosgens21a.html.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, but the authors are encouraged to include a disclaimer/discussion on the fairness of different indices.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}, {"review_id": "Q_SfTMr-FBO", "reviewer": "Reviewer_NwUA", "summary": "This paper proposed a systematic approach to choosing a suitable validation index for classification tasks. The authors considered several binary and multiclass indices, including new ones. They examined the indices with five sets of properties based on the study [6,18_Supplementary]. For each index and each property, they formally proved or disproved that the property is satisfied. They also presented the impossibility theorem for classification. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "At this stage, I consider this paper \"marginally above the acceptance threshold\" due to its overall contributions. \n\nOriginality: \nIt does a good job at studying various validation index for classification tasks with the five sets of properties. I like the part of the derivation of the impossibility theorem for classification because it is a good knowledge for our design of index. In addition, their proposed indices present better satisfactory of the properties than the other existing indices. \n\nQuality: Good. However, my main question is about the selection of the properties. The five sets of properties were transferred from clustering in [6]. The authors are better to discuss the extra properties in classifications. For example, this paper misses the property of Robustness to Imbalance (IMB), shown in [18_Supplementary] for classification. In fact, IBM seems to be the critical property in terms of \"class ratio changes (Brzezinski, 2020)\" in class imbalance problems. If the paper includes this part and makes the the IBM for the new indices, I will give a much higher rating. For this reason, the word \"adequately\" in the Abstract is an overstatement.   \n\nBrzezinski, D., et al. \"On the dynamics of classification measures for imbalanced and streaming data.\" IEEE transactions on neural networks and learning systems 31.8 (2019): 2868-2878.\n\n- p2. a_i=sum from {j=1} to {m} is not suitable for c_00. \n\nClarity: Good. \n\nSignificance: Yes in terms of the five sets of properties, but missing the IBM. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed the limitations in the final section. ", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12 hours"}, "extra_text": {}}, {"review_id": "011SmRSZZp", "reviewer": "Reviewer_WaUL", "summary": "This paper proposes a number of desiderata for classification measures and examines common measures, such as accuracy, Matthew’s correlation coefficient, and F1 score, to show which metrics have desirable properties. Some properties include symmetry, distance interpretation, monotonicity, and a constant baseline. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the results are numerous, if not all that difficult (from my admittedly quick glance at the appendices) and the discussion is fairly thoughtful. I’m not overly familiar with research specifically on classification metrics, so it’s unclear to me how novel some of these ideas are. In the case that they are at least somewhat novel, this paper would be a nice starting point for discussions on metrics, which I feel is somewhat lacking in the community.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have two complaints and a societal impact comment.\n\nMy primary complaint is that I would have benefitted from additional structure in the main paper. Specifically, a distinct proposition here or there would have been nice, but the exposition is otherwise reasonable to follow.\n\nA secondary complaint is that I would have liked to have seen some more examples and discussion balance and imbalance in the data. There is a slight discussion in the ImageNet discussion, but I think there could be room for a bit more, given that class imbalance is one of the motivating factors behind the use of some metrics (e.g., the ever-annoying F1 measure).\n\nThis could likely be a paper on its own, but I think the paper could potentially have negative social impacts. The reason is essentially fairness: how is selecting a desirable classification measure in practice going to affect different groups? I certainly don’t view this as disqualifying--I just bring it up because we’re asked to think a bit more about these issues now. I just think a mention should be made that in practice one may also need to consider other properties of the measure.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Throughout the discussion the reviewers agreed that the paper provides an interesting and comprehensive study of performence measures, and that it is of interest to the NeurIPS community. We urge the authors to revise the paper according to the comments provided by the reviewers in the rebuttal.

Thanks for submitting your work to NeurIPS!
Decision: Accept (Poster)