No Need for Ad-hoc Substitutes: The Expected Cost is a Principled All-purpose Classification Metric
Authors: Luciana Ferrer
Venue: Accepted by TMLR ()

Abstract: The expected cost (EC) is one of the main classification metrics introduced in statistical and machine learning books. It is based on the assumption that, for a given application of interest, each decision made by the system has a corresponding cost which depends on the true class of the sample. An evaluation metric can then be defined by taking the expectation of the cost over the data. Two special cases of the EC are widely used in the machine learning literature: the error rate (one minus the accuracy) and the balanced error rate (one minus the balanced accuracy or unweighted average recall). Other instances of the EC can be useful for applications in which some types of errors are more severe than others, or when the prior probabilities of the classes differ between the evaluation data and the use-case scenario. Surprisingly, the general form for the EC is rarely used in the machine learning literature. Instead, alternative ad-hoc metrics like the F-beta score and the Matthews correlation coefficient (MCC) are used for many applications. In this work, we argue that the EC is superior to these alternative metrics, being more general, interpretable, and adaptable to any application scenario. We provide both theoretically-motivated discussions as well as examples to illustrate the behavior of the different metrics.

Published in Transactions on Machine Learning Research (02/2025)
No Need for Ad-hoc Substitutes: The Expected Cost is Principled All-purpose Classification Metric
Luciana Ferrer
Iferrer uba.ar
Instituto de Ciencias de la Computación
CONICET- Universidad de Buenos Aires, Argentina
Reviewed on OpenReview: https: // openreview. net/ forun? id= 5PPbuCEaZs
Abstract
The expected cost (EC) is one of the main classification metries introduced in statistical and machine learning books. is based on the assumption that, for a given application of interest, each decision made by the system has a corresponding cost which depends on the true class of the sample. An evaluation metric can then be defined by taking the expectation of the cost. over the data. Two special cases of the EC widely used in the machine learning literature: the error rate (one minus the accuracy) and the balanced error rate (one minus the balanced accuracy or unweighted average recall). Other instances of the EC can be useful for applications in which some types of errors are more severe than others, or when the prior probabilities of the classes differ between the evaluation data and the use-case scenario. Surprisingly, the general form for the EC is rarely used the machine learning literature. Instead, alternative ad-hoc metries like the F-bela score and the Matthews correlation coefficient (MCC) are used for many applications. In this work, we argue that the EC is superior to these alternative metrics, being more general, interpretable, and adaptable to any application scenario. We provide both theorelically-motivaeed discussions as well as examples to illustrate the behavior of the different metrics.
1 Introduction
During the development machine learning systems, the model architecture, hyperparameters, training approaches, and other system characteristics, are determined based on the performance measured on a validation set. After development, the performance of the resulting system reported using a separate evaluation set. In both of these steps it is essential that the metric used to evaluate performance be reflective of the needs of the application of interest. In this work, focus the study and comparison of metrics that are designed to evaluate classification systems that produce a single categorical label per sample based only on the information extracted from that sample. While systems trained for this task can sometimes be used for other purposes, like ranking of samples or producing an interpretable score of each class, the metrics needed to evaluate such systems are out of the scope of this work
One of the most widely used metrics for the evaluation of categorical decisions the total error rale or, equivalently, the accuracy, which is equal to one minus the total error rate. These metrics assume that all errors are equally costly regardless of the true class of the sample. In some applications, though, like those involving medical (Ashby & Smith 2000 Kornak & Lu 2011). biometric (Van Leeuwen & Brümmer 2007 Konzalez-Rodriguuee 2014), or business and economic decisions (Harsanyi) 1978 Berger 2013). the costs may be extremely different across error types. In those cases, alternative ad-hoc metrics like the F-beta score Rijsbbrgeen 1979) or the Matthews correlation coefficient (MCC) (Matthews 1975). are often used for evaluation. In this work, we argue that the expected cost (EC), a decades-old generalization of the error rate
Rankinng systems that eventually produce categorical decisions by labelling the top M samples being from the target class also outside of the scope of this paper, the resulting label for each sample is based only that sample but the other samples under evalation.

---

Published in Transactions on Machine Learning Research (02/2025)
(Savage 1972). can be successfully used in all classification scenarios where categorical decisions are made and where the severity of an error depends only on the true class of the sample and the decision made Other metrics designed for this scenario, like the F-beta score or the MCC can be seen as ad-hoe substitutes for the EC, offering no advantages and various disadvantages over this metric.
The EC is based on a simple general assumption: given a certain use-case scenario a numerical value can be assigned to each combination of true class h and decision d, reflecting the cost incurred by the user of the system when making decision d for a sample of class h (Elkan 2001 Berger 2013 Savage 1972) Once these costs are defined for the application of interest, the performance of a classification system can be obtained as the expectation of the cost over the data. This process results in the EC metric, which is also sometimes called expected prediction error (Hastie et al. 2001], or expected loss (Bishop) 2006), and is equivalent to the negative expected utility (DeGroot 1970: Bernardo & Smith 1994]. As various authors have argued. optimizing the EC the principled way to make rational decisions (Good] 1952, Peterson 2009 Dyrland
ct al. 2022: Russell & Norvig 2010). Note that, while the EC definition is very simple, the set of costs it
relies upon are sometimes difficult to define given a certain application of interest (Kahneman) 2011] Dyrland 2022). Yet, as we discuss in this work, costs are also implicitly involved in all alternative classification et al.
metrics. The benefit of the EC is that the costs are selected explicitly rather than being obscured within the metric's definition.
The EC is used in statistical learning as the loss to be minimized when making decisions based on posterior probabilities using Bayes decision theory (Hastie et al. 2001; Bishop 2006). Yet, this metric can be used to evaluate classification systems regardless of how decisions are made (Hernández- Orallo et al., 2012) and, hence, it is not constrained to the evaluation of systems that produce posterior probabilities. The error rate and the balanced error rates which correspond to one minus the accuracy and balanced accuracy (or unweighted average recall), respectively, are, in fact, special cases of the EC which are widely used in the literature to assess performance of categorical decisions. Further, the general form of the EC with unequal costs has been widcly adopted by the specch processing community as an evaluation metric for the tasks of speaker verification and language recognition (see, e.g., Brümneer] 2010 Brümmer el al. 2021 Van Leeuwen & Brümmer 2007 Greenberg et al., 2020), where is called detection cost function (DCF). The DCF is the metric of choice for the periodic evaluations organized by NIST Other than for these tasks, though, are not aware of other applications for which the general form of the EC is used as standard practice.
While some papers stand out proposing the use of EC for medical tasks (Ashby & Smith 2000 Kornak & Luj 2011 Godau et al. 2023), they are exceptions in a literature where various other metrics are used instead Hicks et al. 2022). Interestingly, the EC has been recently highlighted as a "so far uncommon" " metric with many desirable leatures, in a recent paper on metrics for medical image analysis written by an international consortium of experts (Maier-Hein et al. 2024)
In this work, we review and compare the EC with other more widely-used classification metrics in the machine learning literature. Our study is related to recent work by Dyrland et al. (2022) who similarly argued in favor of the use of decision thcoretical metric (expected utility, in their case) and against ad-hoc metries like the F-beta and the MCC, showing empirical evidence on synthetic data that these alternative metrics do not correspond to an expected utility. In this work, we provide a thorough analysis of the EC which complements the work by Dyrland et al. (2022) in various aspects. First, we explain how the EC can be used under varions different scenarios, including the case of uncqual error severity, the case where the decisions do not coineide with the classes (e.g., when the system has an "abstain from choosing a class" option), and the case where the class priors in the evaluation data differ from the one expected in practice. We propose to address the latter case, quite common in some disciplines, by allowing the class priors to be parameters of the metric instead of being defined by the class frequencies in the evaluation data, as in all standard classification metrics. This allows us to manipulate the priors when needed, setting them to the ones that are expected at deployment. With this generalization of the EC, it is also possible to see the balanced error rate as a special case of the EC obtained when the costs depend on the class priors. We also introduce the normalized EC (NEC),, more interpretable version of the EC obtained by dividing the E.C by the F.C of the
PIn some sccnarios, the cost of error depends on characteristics other than just the true class of the sample (o.g., Zadrozny & Elkan 2001), or are determined by the end user after deployment. Such scenarios are out of the scope of this paper.
hilpp: www.wistt.ggn gowisttgoo///t fitl lad/mig/ssee.ffm sre.cfm
2

---

Published in Transactions on Machine Learning Research (02/2025)
best naive system. Finally, we compare the EC with alternative classification metrics, like the F-beta score and the MCC, both empirically, like Dyrland et al., though with alternative visualizations and including results in real datasets, but also theoretically by finding the relationship between the expressions. Based on this comparison, we argue that the EC' is superior to those metrics, being more intuitive and flexible, having a theoretical framework for making optimal decisions, and having useful properties that facilitate its interpretation. The code used to compute all metrics and generate all plots and tables in this paper is available at learning community to embrace this valuable but underused metric.
cost| We hope this work will encourage the machine
2 The expected cost (EC)
The EC is defined as a generalization of the probability of error for cases in which errors cannot be considered to have equally-sovere consequences. In the context of decision theory, the EC is used as the function to be minimized in order to make optimal decisions when the system outputs posterior probabilities. See, for
example, the work by Bishop (2006, Section 1.5), Hastie et al. 2000] Section 2.4), Duda et al. (2001] Section
2.2), Elkan 22001), DeGroot 1970)) Dyrland et al. (2022). and Savage (1972). Yet, EC can be used to assess performance of any classification system, regardless of how categorical decisions are made or whether they make such decisions based on posterior probabilities or some other type of score.
The EC is the expectation of cost function which assigns a numerical penalty to each combination of the true class of the sample, h, and decision made by the system, d. Since we assume that both h and d are categorical, this cost. function can be specified by a matrix with components C(h, d), which represent the cost, we believe that the system should incur for making decision d when the true class of the samples was h.
While there is no restriction on the values assigned to the costs, for every matrix with entries C(h, d), a new matrix equivalent to the original one can be created with entries given by C"(h, d) C(h,d) ming C(h, d). The equivalence is in the sense that systems would be ranked in the same way by both costs matrices (DeGroot 1970; Russell & Norvig, 2010; [rümmerr 2010). The resulting costs are non-negalive and the best decision for each class has a cost of zero, which often facilitates the analyses. In the rest of this paper, we assume that all costs matrices have been standardized in this way.
In this work, as generally doue in the machine learning literature, the expectation is computed with respect to the empirical distribution in the test data, i.e., as an average of the cost over the test samples. Given a test set {(h1, di). ., (hv, d where hy H [ } are the true labels and de E D Di....., Dat are the system's 's decisions for sample t, the EC is given by:
N
EC =
Σ
C(e,, de)
(1)
where C(h,, de) is the cost incurred on sample & with true class hy and system's decision d. This expression can be rewritten as:
KKK M No
M cij P.Rij
1
EC
(2)
N
where Czy C(H,, D,), Ray= -N,,N is the fraction of samples from class H2 for which the system made E, N,y being the number of samples of class H; the test set, and where P; NN./N is the empirical estimate oi the prior probability of class H; in the evaluation data. The first expression in this equation is derived by using the fact that the cost is the same for each combination of class H, and decision Dj. Hence, the summation over the samples in Equation 1 can be converted to a (double) summation over every combination of i and j by multiplying the corresponding cost by the number of samples for that combination of class and decision, Noj-
docision Dj, with N₁ =
The sccond expression for the EC in Equation ( allows for a useful generalization. While P, is originally given by N it can potentially be changed arbitrarily. This allows us set the priors so that they coincide with those we expect. to when the system is deployed which, in many applications, do not necessarily
3

---

Published in Transactions on Machine Learning Research (02/2025)
coincide with those observed in the evaluation data. Rather than forcing the evaluation data to have the priors we expect to see in practice, which would imply downsampling upsampling the data, we can simply use those priors when computing the cost and use the evaluation data as-is to compute the Raj values. This advantage of the EC has been highlighted by Godau et al. (2023). where the authors show the benefits of using an EC metric with the priors determined based on the deployment. data.
Further, note that in the definition of the EC the set of possible decisions do not need to coincide with the set of classes. Rather than being the predicted class labels, the decisions may be the actions that could be taken based on the information in the input sample (Duda et al. 2001 Section 1.3.12). Hence, the EC allows for the evaluation of more general classification systems than other classification metrics like the F-beta score or the MCC, which assume that the set of decisions corresponds to the set of possible class labels. Consider, for example, a (simplified) medical scenario where a doctor needs to make a decision based on an MRI of the brain of a patient that may or may not have a tumor. The decisions made by the doctor do not necessarily need to be one of the two classes (patient docs/does not have a tumor). They could instead be the actions that the doctor could take based the MRT image which could be, for example: "perform surgery", "send home", "do more tests". A cost for each combination of true class and decision can then be selected based on the severity of cach type of error. Internally, the system can make decisions in order to optimize the resulting EC, as explained in Section 2.4 The "do more test" decision would be taken in cases in which the system was not certain enough about the class of the sample to make a decision to operate or send the patient home.
2.1 Normalized expected cost
As explained above, any cost or utility matrix can be mapped to an equivalent cost matrix with Ciy 2 0, and at least one Cjj for each class H;. The minimum value for any EC with these properties is O, attained for perfect system that always chooses a decision with cost 0 for each sample. The maximum value that EC can take, though, depends the costs and priors used to compute it. This makes it hard to assess whether a certain value of EC is good or bad. For this reason, it is convenient to compute a normalized version of the EC (NEC) given by:
EC EC
NEC-
where
ECCaive
min
cj P.
The value used for normalization is the EC for the best naive system that does not have access to the input sample and, hence, always makes the same decision j. The system is free to choose j to optimize the EC for the chosen costs and priors. The expression for ECuavve can be easily derived by considering that, for a system that always makes the same decision j, Rij is 0 for all j + f and 1 for j. Plugging those values of Raj in Equation (2) we find that the EC for that system is Σ C3 P.. The minimum over all possible decisions j is the EC of the best naive system.
For the special case where there are two possible true labels (K 2) and two possible decisions (M 2), and we assume that C11 C22 0, the NEC reduces to:
C12PiR12 + 22 min(ci2PI,,a C21 P2)
Ra R12 + Rei if aal1 otherwise
C1
NEC
where
(4)
Ri2+ Rai
C21 P2
Hence, in this scenario, there is only one free paramcter to choose in the NEC. All combinations of priors and costs that lead to the same a correspond to the same metric.
The NEC metric has an essential property: While its value can be larger than 1, this only happens for systems that are worse than naive system. In particular, system with NEC larger than 1 can be trivially improved upon by replacing it with one that always makes the same least-costly decision j. Yet, discarding the system is usually not necessary. For systems that first generate a score for each class and then make decision based on those scores, a NEC larger than 1 indicates that the decision stage was not properly optimizod. In the binary case, optimization of the decision stage can be achieved by simply tuning the decision threshold. The problem of optimizing the decision stage for a certain F.C of choice is discussed in Section 2.4
4

---

Published in Transactions on Machine Learning Research (02/2025)
2.2 Selection of costs
The selection of costs for each lype of error is not necessarily trivial as it requires underslanding of the application scenario and, hence, should be done in consultation with field experts. This issue is discussed at length, both in general and for specific familics of applications. for example, by Raiffa (1970b); Russell & Norvig (2010); Kahneman (2011); Hunink et al. (2014): Dyrland et al. (2022). Below, we give a brief overview of the subject including some general guidelines. We refer the reader to those references for an in-depth
discussion.
When determining the cost matrix, it is often useful to allow for combination of positive and negative costs (a negative cost would be equivalent to utility) or to define, instead, utility matrices, where bad decisions correspond to negative values and good decisions to positive ones. Utility matrices can be converted into cost matrices by multiplying them by minus one. Further, it is also useful to note that shifting each row by a constant or globally scaling cost matrices does not affect the optimal decisions or the ranking of systems [Brümmer 2010] Section 3.4). In particular, this mcans that, for the standard binary case with one docision corresponding each class, all cost. matrices can be parameterized by single value. Equation (4) shows that this parameter is determined by the ratio between the costs for the two types of error.
Costs or utilitics can often be defined in monetary terms. This may be the case for companics using the output of machine learning systems to make business decisions. For example, consider a simplified loan approval scenario. A bank may use an automated system to predict whether a person will be able to repay their loan and, based on its output, make one of three possible decisions: deny the loan (D), approve the loan with a standard interest rate (AS), or approve the loan with a higher rate (AH). The utility for a person that will pay off the full loan will be 0 for decision D (assuming the cost of processing the application is negligible), and a positive value for AS and AH given by the total amount of interest that the person will pay in each case, which we will express as Is L and In ox where L is the principal loan amount requested and the as depend on the corresponding interest rate and the payment schedule. We further need to reduce the utility for the AH case by some factor B given by the fraction of clients that would decide to accept the loan despite the higher interest rate offered. We assume that all clients offered the lower interest rate accept the loan. For a person that will default on their loan, the utility for decision D would again be 0, while the utility for decisions AH and AS would be the fraction of the loan that was paid minus the total amount loaned. Assuming that, average, a fraction ? of the total money owed (principal plus interest) is paid by borrowers before defaulting, the utility for those cases may be computed as >(L+ + IH) L and >(L + Is) - L. Note that these utilities may be negative. Here we assume that borrowers that end up defaulting would always accept the high-interest loans since other banks would also offer them high rates. Hence, in this simplified loan approval scenario, the utility matrix would be given by:
D
AH
AS ag
0
71+ + a1+an)= -1 711 + as) -1 ]]
3 aa
repays the full loan and the interest does not repay the full loan
U U=L U
where the principal value of the loan, L, was factorized outside of the matrix. As mentioned above, once defined in this way for convenience, we can multiply the matrix by -1 to turn it into cost matrix C and normalize it by doing C(h. mina C(h, d) so that the minimum value in each row is 0.
As illustrated by this example, when the only possible outcomes of the decisions made with the system are monetary, selecting the cost matrix is relatively straight forward, though perhaps not trivial: the costs or utilities simply reflect the losses or gains incurred by each decision for a given class, expressed some currency. In general, though, decisions may have consequences that are not only monetary. For example, consider an application for finding a given singer within a large database of songs. In this case, as mentioned above, only the ratio of costs for the two error types needs to be determined. We need to decide how costly it is to not find song that includes the target singer, usually called míss, compared to selecting a song that does not include it, usually called false alarm. If we wish to simply retrieve some examples of the singer, false alarms may be more costly than misses, with the ratio of costs depending on how bothered we believe the user would be by being offered a song that does not contain the singer of interest. If we wish to find every single audio that includes that singer for example, for copyright reasons we should assign a much higher cost to misses than to false alarms. In this case, perhaps we are back in scenario where we can
5

---

Published in Transactions on Machine Learning Research (02/2025)
specify the costs in monetary terms since misses imply loss of revenue for the artist and false alarms need to be filtered by human annotators that require compensation for every reviewed song.
In some applications, like those related to medical decisions, the selection of costs may involve setting a value for a human life. While this is, undoubtedly, a very uncomfortable decision to make, refusing to set a value for a human life may implicitly result. its being undervalued. Russell & Norvig (2010) give a compelling example involving government agency that commissioned study on removing asbestos from schools: "The decision analysts performing the study assumed particular dollar value for the life of a school-age child, and argued that the rational choice under that assumption was to remove the asbestos. The agency, morally outraged at the idea of setting the value of a life, rejected the report out of hand. It then decided against asbestos removal implicitly asserting a lower value for the life of a child than that assigned by the analysts."
Sometimes the determination of exact costs is not possible but we have some knowledge about the range of values they should take. In this case, costs may be specified as a (discrcte or continuous) distribution. Interestingly, the expectation of the cost when the individual costs are random variables reduces to the
standard EC above, with costs given by their expected values (Dyrland et al. 2022 Raiffa 1970a.. Further, in some scenarios, one might wish to select systems based on a trade off between two or more ECs, well as other systems characteristics like memory consumption or run time.
While the difficulty in selecting costs may seem a disadvantage of the F.C, it is important to note that the trade-off between different types of error is encoded in every classification metric (this issue discussed in Sections transparent. Hence, instead of letting the metric choose the trade-off for us, we can make our best effort in choosing costs that are, at least approximately, relevant for our task of interest. As illustrated by the asbestos removal example mentioned above, refusing to explicitly select the costs may result in the implicit selection of extremely incorrect ones.
3 and The advantage of the EC over other metrics is that the trade-off is explicit and
2.3 Error rate and accuracy
The standard error rate (also called total crror or probability of error) is given by:
ER
Σ
N
(5)
N
f=1|i
Comparing this equation with Equation 2 we can see that the ER is a particular case of EC obtained when M K, and Ci for i # and GÚ for i = j, which we will call the "0-1 cost matrix". Further, since the accuracy is given by one minus the ER, we can see that accuracy is also just trivial function of one specific case of EC.
In the ER computation, errors from all classes are weighted equally. Hence, when the classes are highly imbalanced, the ER and the accuracy become somewhat insensitive to the performance in the minority classes. For this reason, when errors in the detection of the minority classes are considered more severe than those in the majority classes, the balanced ER (BER), also known as weighted ER, is used as metric instead of the ER. Alternatively, the balanced accuracy, given by one minus the balanced ER, sometimes called unweighted average recall (UAR), is also commonly used in these scenarios. The BER is defined the average of the ER values per class. That is:
K
K Σ
K
K
BER = K
No =
Σ
Ryg
(6)
R
j=1
j=
This expression coincides with the EC when M = K and the costs are selected such that Gúj = N/(KN..) = 1/(KP,) for i i j and Gj = 0 for = As a consequence of setting the costs this way, all error rates (Ry for + j) have the same influence on the final metric. Note that by deriving the balancod error rate as a special case of the EC we can see exactly what is being assumed when we compute this metric: that the cost for each type of error is given by the inverse of the frequency of the true class of the sample.
6

---

Published in Transactions on Machine Learning Research (02/2025)
The general form of EC offers greater flexibility than these two special cases of the standard and balanced error rates, allowing us to accommodate scenarios in which the errors are neither all equally costly nor costly as the inverse of the prior for each class. The EC allows us to think of each type of error independently and set their cost according to their severity.
2.4 Optimizing the decisions
Most modern classifiers are composed of two stages, one that generates posterior probabilities for the classes and a second one that makes the categorical decisions based on those posteriors. The EC and NEC can be used to assess the quality of decisions made with any strategy, like the threshold tuning approach commonly used for the two-class setting. Yet, a unique advantage of these metrics is that the optimal decisions for systems that produce posterior probabilitics can also be obtained in closed form. The decisions that thcorctically optimize the EC are called Bayes decisions (Hastie et al. 2001 Bishop 2006 Pelerson 2009). Given an EC determined by a specific cost matrix Cij, the Bayes decision for a sample x is given by:
K dr(x) = arg min Σ cy P(H())
where P(H((z) is the posterior probability for class H; given sample x as produced in an internal stage of our classifier.
One important condition needs to be satisfied for Bayes decisions to be optimal: the posteriors used to make the decisions need to be well calibrated. Loosely speaking, for well-calibrated posteriors, 90% of the times the posterior for a certain class h is 0.9, the true class is indeed h. For a formal definition of calibration, please refer, for example, to the works by DeGroot & Fienberg (1983); Brümmer (2010); Filho et al. (2028) Unfortunately, good calibration is not always achieved by modern classifiers mostly due to overfitting of the parameters to the training data, which results in overconfident posteriors (Guo et al. 2017). Fortunately, though, calibration quality can usually be easily fixed, if needed, by doing post-hoe calibration on the output of the classifier (Filho et al. 2023 Ferrer & Ramos 2025). The topic of calibration is out of the scope of this paper, but it is covered at length in the publications cited above and many others. The repository provided with this work includes tools for assessing and fixing calibration.
Assuming our classifier is calibrated, either inherently so or after adding a post-hoc calibration stage, the EC provides us with an elegant approach for making optimal decisions, even for the multi-class case. In fact, as discussed in Appendix [ the argmax decisions so widely used in the machine learning literature correspond to the Bayes decisions for the EC defined with the 0-1 cost matrix, i.e., the error rate (notably, this method is generally used without verifying whether the calibration condition is satisfied). For the binary case with one decision per class, Bayes decisions can be made by comparing one of the two posteriors with a threshold which depends on the costs (see Appendix A.2h. In the general case, making Bayes decisions for an EC of choice is as simple as implementing Equation 7 In Section 4 we show examples on making Bayes decisions for various EC metrics.
To summarize, the EC offers an important advantage over other classification metrics in that, when well- calibrated posterior probabilities are available, decisions can be made in principled way using Bayes decision theory. Alternatively, if well-calibrated posteriors are not available, decisions can be made with any of the approaches used for other metrics, like threshold tuning.
2.5 Optimizing the classifier
The scction above describes how to optimize the decisions for an EC of interest. Those decisions are made using the posteriors oulput by a classifier. Notably, even when systems are developed for a specific EC of interest, the classifiers are not usually trained to optimize such EC. In fact, perhaps the most common procedure in the machine learning litcrature on classification is to train a classifier to optimize cross-cntropy while assessing its performance using accuracy (which, as mentioned above, is one minus the EC for the 0-1 cost matrix). The reason why the classifiers are not trained to optimize accuracy is that this metric, any
7

---

Published in Transactions on Machine Learning Research (02/2025)
EC and any other metric computed on categorical decisions, is not differentiable with respect to the scores, making its use as objective function for model training inconvenient.
In general, training classifiers that will be evaluated in terms of an EC using a cross-entropy objective is a very reasonable approach. To see this, we can refer to an interesting theorem that shows that cross-entropy can be written as an integral over a family of FCs computed over Bayes decisions (Brümmer 2010 Section 8.2). By optimizing cross-entropy we are optimizing trade-off between a family of ECs. A model trained with cross-cntropy is encouraged to produce good posterior probabilities over the whole simplex, including the operating point corresponding to accuracy and infinitely many others. In some cases, though, not want to settle for a model that optimizes a trade-off between ECs, most of which are of no interest for our application. In such cases, we may wish to directly train the model to optimize our EC of interest.
might
In those scenarios, a relaxation of the EC can be used to turn it into a differentiable function that approximates the original metric, enabling its use as an objective function during optimization. The use of the EC for the purpose of model optimization is outside of the scope of this paper where we focus on their use for system evaluation. Yet, for completeness, here mention an approach for using the EC training objective proposed by Mingote et al. [2019] for a binary problem with one decision per class. First, note that, for this case, the error rates in Equation 2 can be written as
N12
N Σ
R12 =
T(P(H) (2)) < <u)
(8)
Ni.
N1.
=
N
R21
N21
Σ
I(P(H)))) > u)
(9)
N2.
N2.
t=
where 7 is the indicator function, which is 1 when its argument is true and 0 otherwise. In these equations, N1 and N21 are the number of samples for which the wrong decision was made because the posterior fell on the wrong side of the threshold u.
The indicator function is what makes the EC not differentiable. Mingote et al. 2019 proposed to approximate the indicator function by a sigmoid, computing the rates as:
R12
Σ
olu P(H()z))
(10)
Ni
t=1 11
R21
Σ
O(P(H1|2)) v)
(11)
N2.
1=1
Plugging those rates into Equation 2 we obtain differentiable version of the EC. In their approach, the threshold u is optimized along with the rest of the parameters of the model.
This simple relaxation of the EC allows for its use objective function during training. Unfortumately, generalizing this approach to the multi-class case is not trivial since the decision regions cannot be obtained by simple thresholding. We are not aware of any work that proposed a differentiable relaxation of the EC for the multi-class case.
3 Other classification metrics
In this section we describe some of the most common classification metrics in the machine lcarning literature and compare them to the EC metric. We also describe the area under the ROC curve (AUC) and the equal error rate (EER). These metrics are not strictly classification metrics in the sense considered in this work, since they do not evaluate the quality of categorical docisions but of the scores that would eventually be used to make such decisions after choosing one specific threshold. Yet, we include them in this section for completeness since they widely used in the machine learning classification literature.
8

---

Published in Transactions on Machine Learning Research (02/2025)
3.1 F-beta score
The F-bela score, here Fs, is defined one minus metric called elfectiveness, which was first introduced by |Rjjsbergen (1979). The F3 is probably one of the most widely used metrics for binary classification in modern machine learning litcrature, alongside accuracy and error rate. For example, in the SuperGLUE benchmark of natural language processing (NLP) tasks, out 10 tasks, 3 use the Fş (one uses MCC and the rest use accuracy). The SQuAD benchmark for question-answering has two metrics, one of which is Fg- The IEMOCAP benchmark for emotion recognition is evaluated with (weighted, micro or macro) Fg as well accuracy. The slot-filling SLURP benchmark"] for spoken language understanding uses Fg. The Fg is also widely used in medical imaging, where it is called Dice coefficient (Taha & Hanbury 2015), in opinion mining [Wang et al. 2019). and for epileptic scizure detection (Siddiqui et al. 2020). These are just few examples from a long list of tasks where the Fş is selected for evaluation of performance, highlighting the wide spread use of this metric.
The F3 assumes that there are only two classes, that the set of decisions coincides with the set of classes (K = M and D HH, and that the problem is not symmetrie; one of the classes taken as the class of interest to be detected. Here, we will take the class of interest to be class H2. Fg is defined follows:
Precision Recall
Precision = N22/N.2
F3 (1+82) B² Precision + Recall
where
(12)
Recall = N22/N2.
Replacing those values for the precision and recall, we get:
(1 82)
Fg = (1 N2) + 82N N12
(13)
Fa takes values between 0 and Larger values indicate better performance, contrary to the EC for which larger values indicate worse performance. For this reason, in order to compare F3 with EC, it is convenient to work with 1 1- F3, which is given by:
N21 + N12
3 R2 2 + R12P1
F- = (1 N2) + 8 N21 N12
(14)
B R42
where R.2 N.2/N is the fraction of samples the evaluation set that are labelled as class H2. The numerator in this expression is the EC for the binary case when costs for mistakes are given by C12 = 1 and 021 = 82, which we will call ECye:
ECμ= = 8 P2 + RI2P,
(15)
We can now express 1- Fg as a function of ECga and its normalized version:
ECga
NEC
1 F3
min P1)
(16)
B2 + R.2
B2P2 + R.2
That is, 1- 3 is proportional to ECga with a scaling factor given by the inverse of 8 + R.2. The relationship between F3 and ECya was shown through simulations by Dyrland et al. 2022 Figures 2 and 3), who also discuss the inadequacy of F 3 as a classification metric. Below, we analyze Equation sheding light into the issues that make F3 a poor classification metric.
16 theoretically,
The first term in the denominator of Equation (16), P2, is independent of the system. When comparing different systems on the same dataset (or different datasets with the same class priors) Pg is fixed The
htttps:/ssuper. httpe://super.gllbbbactnerrr www.bbbncccnnrrk.cccc/easerrrrrr
https://ra.jpukaat wtt..b..///Sppu/-espp..r...
httpp::/poperrewtthccoe..e www.c/o/o//.ae.oco htt.://www....oaa..o.coooooooooe
https://papprrswtthcodeee http://wwwwttcoee.ccooo////////s wttp:/d/.cco////mstttoooo wiwp://llooootlll-oooooooooo on-
Note that values of F'g across datascts with different priors are comparable each other. In particular, the F3 computed
cartain evaluation datasct not good prodictor of the F3 that will be obtained during deployment the priors between
the two differ. This scenario can addressed principled way with the EC by setting the priors in Equation 2
those that are expected during deplayment Godan al. 2023

---

Published in Transactions on Machine Learning Research (02/2025)
second term in the denominator in Equation [6 R=2, is the percentage of samples that are labeled by the system as being of class H2. Given two systems with the same value for RC Fs will favor the one that detects more samples as being from class H2, regardless of whether they are correctly or incorrectly classified (examples of this behavior will be given in Section 4 This is an odd thing to reward: having more samples from class H2 delected not beneficial in itself. If, for the application of interest, detecting the samples from class H2 is more important than detecting the samples of class H1, the principled approach for defining a metric is use an EC which penalizes errors in samples from class H2 more than errors in samples of class H1 (i.e., setting C21 > c12).
As for the EC, we can compute the best F3 that would correspond to a naive system that always outputs the same decision. If the system always chooses class H1, F3 = 0 since N22 = O. On the other hand, if the system always chooses class H2, the recall is and the precision is equal to the prior of class H2. P2, which results in F3 = (1 + P2
+ 1). This the value of F for the best naive system. Any system with an F3 worse than this value should be considered ineffective. Since the F3 for the best naive system depends on P2, given value of F3 should be interpreted differently depending on the dataset. A value of F1 of 0.4 is much better than chance for dataset with P2 0.01 which would have naive F1 of 0.02, but only just above chance for a dataset with P2 = 0.2 for which the naive F1 is 0.333. While it would be possible to normalize the F3g to make it more readily interpretable, as far as we know, this is never done in the literature.
Finally, note that, unlike for the EC where, as long as the scores are calibrated, the optimal decisions can be made with Bayes decision theory, as far as we know there is no equivalent decision theory for the Fg. For the specific scenario for which the F3 is defined, which corresponds to binary classification with one decision per class, decisions can be made by thresholding and the optimal threshold can be obtained empirically using development data. Note, though, that, even for perfectly calibrated scores, the optimal threshold for F will depend on the system under evaluation through R.2 rather than being fixed, as for the EC. Since every threshold corresponds to the best threshold for a certain EC, we can interpret an F3 that is minimized at a specific threshold as being cquivalent to the EC that is optimized at that same threshold: they are equivalent in the sense that both metrics reach their optimum value for the same threshold. Hence, the optimal threshold for F changes for each system, the implicit costs assigned each type of error also change. This mcans that F3 optimizes for different trade off between error types depending on the system under evaluation. This is, in view, undesirable characteristic for metric. We will see examples of this behavior in Section 4.1.1.
In summary, the Fs has several issues that make it a suboptimal metric: 1) it is restricted to scenarios with two classes and two decisions, 2) the interpretation of its value depends on the priors of the dataset, 3) the effective costs being optimized depend on the system under evaluation rather than being determined by the needs of the application, and 4) it cannot be optimized theoretically. All of these problems are solved by the NEC metric.
3.2 Matthews correlation coefficient
The Matthews correlation coefficient (MCC) was first introduced by Matthews 1975 for comparison of chemical structures, later proposed as a metric for binary classification by Baldi et al. (2000), and finally generalized to the multi-class case by GGrodkin (2004). In this section we will consider the binary version since this is enough to show its flaws as classification metric. Chicco & Jurman (2020) argue that MCC is the most informative single score to establish the quality of binary classifier that outputs hard decisions. EC is not considered as an option in that paper. As we discuss below, we believe EC is superior to MCC as a metric for classification performance.
As the F3, the MCC assumes that the set of decisions is the same as the set of classes. For the binary case MCC is defined as follows:
Nu - N12N211
MCC
(17)
V(NU + N21) + N12) + N12) + N21)
10

---

Published in Transactions on Machine Learning Research (02/2025)
After some manipulations and using Rij Nig/Nis we get
N2. (1- ( N.1
P2(1 - P2) = R12(1 - 122)
MCC =
+ R21)
(1 NEC,)
(18)
where NEC, R12 + R21 is the normalized BER (this can be seen by dividing Equation ( by the BER of the naive systems which is 0.5 for the binary case). As for the Fg, the relationship between MCC and EC was shown through simulations by Dyrland et al. 2022 Figures 2 and 3). Here, we analyze the relationship thcoretically. We show empirical results in Section
From Equation T8] we see that when the NEC, is larger than 1, which, as explained in Section 2.1 happens when the system worse than the best naive system, the MCC is negative. Further, we can see that MCC and 1 NEC, differ by the square root factor. The numerator this factor fixed, does not depend on the system, only on the priors (see footnote which also applies to the MCC). The denominator, other hand, does depend the system and is largest when R.2 0.5 and smaller as the decisions become more imbalanced (i.e., R.2 approaches 0 or 1). Hence, the factor will grow as the decisions become more imbalanced, making MCC larger in absolute value. For two systems with the same NEC, < 1, MCC will favor the one with larger imbalance in the decisions. As discussed with respect to F3, this behavior does not seem reasonable since the frequency with which system makes a certain decision not a good or bad quality in itself. As the FR, the MCC effectively optimizes for different trade off wet error types depending on the system under evaluation. Finally, also as for F3, as far as we know, the MCC does not have a corresponding decision theory and needs to be optimized empirically.
the
3.3
Net benefit
Net benefit, metric used in binary classification for some medical applications et al.
Vickers et al.
2016 Riley
2021
Cowley et al. 2019 is defined as:
N22
p
N12 N
NB
(19)
N
where p is a parameter of the metric called threshold probability for reasons that are explained below, We rewrite this expression as follows:
NB = P2R P P. R12 = P3 - (P2 + P P P1 R12)
(20)
1-p
1-p
where we used the following equalities: P.R.y = No/N and R2g + R21 = 1. The term in parenthesis in this equation is the EC when the number of labels and decisions is 2 (K = M = 2) and c21 = 1, and C12 = pp/1 We can then express NB as a function of the corresponding normalized EC, which we will call NEC,:
P P. NEC, NB = P2 min P2, P3
(21)
Hence, while NB is an affine function of NEC, making them equivalent for ranking systems, the NB looses the interpretability of the NEC given by the fact that NEC of 1.0 indicates that the system has the same performance a naive system. For this reason, we believe the normalized EC is preferable to the NB.
NB's single parameter is defined in such way that it coincides with the optimal threshold for this metric. We can see this by first noting that the optimal threshold for NB is the same as the optimal threshold for NEC, (NB is maximized when NEC, minimized). The threshold for NEC, is given by Equation (28) in Appendix A.2 with 621 1, and 612 = p/(1 p) resulting in p, which explains the fact that this parameter is called threshold probability.
3.4 LR+
The likelihood ratio for positive results, usually called LR++, is commonly used in diagnostic testing (Pauker & Kassirer 1975 Simundić 2009). As for Fg and NB, it is used for non-symmetric binary classification
11

---

Published in Transactions on Machine Learning Research (02/2025)
problems where one of the classes is the class of interest. It is given by:
sensitivity
LR+ 11-11-1- -seecifficty
(22)
where, if we assume class H2 the class of interest, sensitivity R22 and specificity = R11. LR+ is always positive and the larger its value, the better the system.
We can write LR+ a function of a NEC value using the fact that R22 + R21 = 1, and R1 + R1 1:
1- R21
1 - NEC.
LR+
+1
(23)
R12
R12
where NEC, = R12 + R21 as defined in Section [3.2 So, given two systems with the same NEC, the LR+ will favor the one with lower R12- Just like for F and MCC, the optimal threshold for LR+ and, hence, the effective costs for which the metric optimizes, will depend on the system's characteristics (this time, through R12). We see no reason to prefer this metric over the EC where, if favoring lower values of R12 is desired, one can directly and explicitly set C1 > 02 to the actual costs needed for the application.
3.5 Specificity at target sensitivity
Another approach for measuring performance of binary classification systems, most commonly used in medical diagnosis tasks, is to fix the sensitivity (or recall, or, in our terminology, R22) to some pre-defined value and report the specificity (R1) correspouding to the threshold that results in that value of sensitivity (Bickehhaup)) iet al. 2018 Chan et al. 2002 Goldbaum et al. 2002 Qin et al. 2019). Sometimes the opposite is done, fixing the specificity and reporting scnsitivity (Souza et al. 2010 These motrics are usod when the application of interest imposes a certain larget sensitivily or specificity below which the system would be unacceptable. It compares systems by foreing them to operate at that exact level of the target metric and comparing the other metric at that threshold. For concreteness, here we will discuss the first approach where the sensitivity is fixed at a target value and the specificity is reported, calling it SPÔSE, though the discussion directly applies to the alternative metric.
SPÔSE, while intuitive and easy to explain, has a number of problems. First, consider the scenario where a development dataset is used to compute SP@SE for a fixed sensitivity of 0.95, which corresponds to a certain threshold fdev- Now, when this threshold is used on new dataset, the sensitivity will most likely longer be exactly 0.95. What is the value of SP@SE for this new datasct? It is not the specificity at the faev threshold because the sensilivity at that threshold is not the larget value. We could, instead, set it to be the specificity for a new threshold &aev for which the sensitivity is 0.95 in the new dataset. The problem with this latter approach is that it does not address the practical sccnario in which the threshold is determined during development and used without change to make decisions on new data.
A second problem with this metric is that, just like Fg and MCC, it implicitly considers different set of costs for every system under evaluation since the decision threshold depends on the system. Finally, third issue with this metric is that it forces all systems to operate at the minimum acceptable sensitivity. When comparing various systems with each other, better systems could potentially operate at better sensitivity values and still achieve good specificity. Yet, this metric does not allow us to adapt the threshold to obtain a better trade off between the two types of error when possible.
In some applications, though, it is important to impose a minimum value of sensitivity. The approach we recommend for these cases to use a NEC as defined in Equation 4 where the a is determined using some baseline system to achieve (approximately) the desired target sensitivity at the optimal threshold for the corresponding EC, In this way, translate the original requirement on sensitivity to metric that can be used to compare systems with each other, select thresholds, and evaluate performance on different datascts without the problems described above related to SP&SE. As an added benefit, this approach makes explicit the implicit assumption on the costs that come with the selected target sensitivity. To find what the ratio of costs corresponding to the selected a is, we simply observe that 0/2/221 aP2/Pi, PI, where the priors are those corresponding to the development set. We can then analyze whether the ratio that resulted from setting the sensitivity to the selected target value is, indeed, reasonable for the task. Say, for example, that we are
12

---

Published in Transactions on Machine Learning Research (02/2025)
setting the sensitivity to 0.95 because we believe it is very important to never miss a sample of class H2. Now, after finding the corresponding a we observe that the ratio of costs for which that sensitivity is achieved for our data and system is C12/021 = 2. That is, for the desired operating point with sensitivity equal to 0.95 we are, implicitly, assigning more weight to the errors on class H1 than to those of class H2. Upon sceing this, we might decide the selected larget sensitivity was; in fact, not good choice.
3.6 Area under the curve and equal error rate
Two very common metries used to evaluate binary classification systems are the area under the ROC curve (AUC) (Brddley 1997) and the equal error rate (EER) (Brümmer et al. 2021). These metrics, unlike all other metrics covered this paper which evaluate categorical decisions, evaluate the quality of the scores. The assumption made by these metrics is that the decisions will be made by thresholding, but no commitment to specific threshold is made. Under this assumption, two types of curves be created by sweeping threshold, making categorical decisions for each case, and computing the resulting Ri values:
Receiver Operating Characteristic (ROC) curves, which correspond to Rez versus R12
Recall vs Precision (PR) curve, which correspond to Re vs N22/N.2.
For each of those two curves, the area under the curve (AUC) can be obtained as summary metric. Further, from the ROC we can find the threshold for which R12 equals Ru 1 - Re2- The value of R12 at this threshold is called Equal Error Rate (EER). Interestingly, there is a very close tie between the EER and the total error rate (EC for the 0-1 cost matrix) computed over Bayes decisions: this EC is upper bounded by min(EER, P1, P2) (Brümmer et al.) 2021), as long as calibration is perfect. If calibration is not perfect, this F.C can grow without bound, while EER will not. Note that the EER has the same problem as SPOSE: the threshold used to compute it needs to be obtained on the evaluation data itsclf, resulting a metric that does not reflect the final performance of system for which the threshold is determined during the development process.
The ROC and PR curves and the resulting AUCs are invariant to monotonic transformations of the scores since every threshold in the original space corresponds to a new threshold in the transformed space with the same R, values. As a consequence, these metrics do not assess the effect of threshold selection in the final performance of the system, which could be seen as implicitly assuming that the optimal threshold would always be selected. Further, the AUCs do not reflect the performance that a classification system used to make categorical decisions will have in practice since such systems need to commit to one specific threshold instead of integrating over all possible thresholds.
Hence, while AUC may be useful metric during the early stages of development, at some point a developer working on a system that will be used to make categorical decisions for a certain application should commit to a specific set of costs that reflect the needs of that application. If the application for which the system will be used is unknown during development or the costs camnot be determined ahead of deployment, then the system needs to be optimized for any possible threshold. In this case, using AUC as metric would be be effectively determined empirically using
appropriate, assuming that the threshold for each application some development data. Alternatively, in that scenario, performance can be assessed using strictly proper scoring rules (SPSRs) which evaluate the quality of the scores as posterior probabilities (Gneiting & Raftery 2007 Ferrer & RRmos 2025). J. A system developed to have low expected SPSR can be thresholded with the Bayes threshold without the need to tune the threshold for cach new set of costs.
Finally, note that the AUC and EER metrics cannot be naturally generalized to the multi-class case, or to binary cases with more than two decisions. Hence, when the task has more than two classes or decisions, these metrics are not applicable.
3.7 Multi-class metrics
All the metrics described in this section until this point are originally designed for binary classification. For the multi-class case, the most common metrics in the machine learning literature are accuracy and balanced
13

---

Published in Transactions on Machine Learning Research (02/2025)
accuracy which, as saw in Section 2.3 are one minus special cases of the EC. Other metrics often used for the multi-class case rely on mapping the original multi-class classification problem into a collection of binary classification problems- a task called multi-label classification (Tsoumakas & Katakis 2007 where each binary problem consists of detecting one of the classes. By turning the multi-class problem into a set of binary classification problems, an essential restriction of the multi-class classification lask is ignored: the fact that exactly one label should be assigned per sample instead of potentially zero or multiple labels as for the multi-label classification task. This mapping, though, enables the use of metrics given by summaries of binary classification metrics. The macro F is computed the binary classifiers. The F3 of macro averages of precision and recall (sometimes also called macro Fg. producing a good deal of confusion as discussed by Opitz & Burst (2019h) is computed using Equation (12 with the precision and recall calculated as averages over the per-class precision and recall for each of the binary detectors (Grandini et al. 2020). These two metrics inherit all the problems of the binary F& discussed in Scction 3.1 Finally, the F3 of micro averages of precision and recall is also given by Equation (12), but with the precision and recall computed using the total number of true positives (N22), false positives (N12) and false negatives (N21) across all classes (Zhang & Zhou 2013). This version of the F3,, when computed on the output of a multi-class classification system where only one of the binary classifiers can be active for each sample, turns out to be identical to the accuracy (Grandini et al. 2020).
the average of the Fg value over each of
One of the advantages of the EC is that it directly and naturally addresses the multi-class classification task. While the simplest way to define an EC for the multi-class case is to set all the costs to 1, as in the total error rate, or to over the priors, as in the balanced error rate, many problems benefit from the more general definition of the cost matrix. For example, in some scenarios one class may be more important than the others, requiring higher costs. Further, an abstain decision may be included with its corresponding set of costs for each true class. Importantly, as explained in Section 2.4 the EC comes with principled decision theory for making optimal decisions based on classifiers that output posterior probabilities. This is particularly useful for the multi-class case (or when the number of decisions is larger than two) where threshold tuning is not an option. We illustrate the use of the EC for the multi-class case in Section 4.1.2
4 Empirical comparison
In this section we illustrate and further discuss the concepts explained in the previous sections using empirical results, focusing on comparing the EC with the most widely used metrics in the literature. The first section shows results on various synthetic datasets which allow us to create an arbitrarily large number of samples so that conclusions are not affected by noisy results, and to manipulate the system performance to highlight the different points we wish to discuss. The second section shows results on various real datasets for speech, image, and natural language processing tasks, illustrating how the wrong choice of metric can negatively affect development decisions and highlighting the limitations of the standard metrics.
4.1 Results on synthetic datasets
The procedure for generating the synthetic scores described in detail in Appendix B samples are generated with an imbalanced prior distribution, with P1 = 0.8 and equal prior for the rest of the classes. Samples for each class are drawn from a Gaussian distribution where the mean are separated by 1.0 and the variance is 0.2, unless otherwise indicated. Then, perfectly calibrated posteriors are obtained using the known distribution and priors for each class.
Briefly, 100,000
4.1.1 Comparison of metrics for a two-class dataset
Figure [ shows results for the 2-class dataset for F1, MCC, and three different NEC with 012 1 and 02 set to 1, 4, and 16. Since P1 = 0.8, the NEC with c 4 corresponds to the normalized balancod error rate, NEC, = R12 + R2i (this can be verified by plugging these costs and priors in Equation (4)) The left. plot in the figure shows the distribution of log posterior odds, ((H2)/)//(HHI))pp classes. The rest of the plots show the behaviour of different metrics as a function of the decision threshold. The decisions are made by comparing the log odds score for each sample with the corresponding threshold, labelling the sample as class H₂ if the score is larger than the threshold and class Hi heewisee.. For each
for each of the two
14

---

Published in Transactions on Machine Learning Research (02/2025)
0.15
1.25
1.25
1.25
H1 H2
1-F1
1-MCC
NEC
1.00
NEC
1.00
NECa
1.00
NEC,
0.10
R/2
R-2
NEC,6
0.75
0.75
0.75
0.50
0.50
0.50
0.05
0.25
0.25
0.25
0.00
0.00
0.00
55
5
0
5
5
log odds
thr
thr
Figure 1: Comparison of metrics for a synthetic dataset with Gaussian per-class distributions and P1 = 0.8. The subindex in the NEC values indicates the value of 021, while C1 is 1.0 in all cases. The first plot shows the distribution of the log odds for cach class scaled by the prior probability of the class. The other three plots show various metries as function of the log odds decision threshold. The dashed lines are included only to aid visualization of the various points highlighted in the discussion.
threshold we can then obtain the Niy counts needed to compute all metrics. The dashed lines in these plots correspond to specific decision thresholds discussed below and are included to ease visualization of the metric values for those thresholds.
The second plot Figure [ shows a comparison between one minus F1 and NEC,, which are related by Equation (16). The fraction of samples labelled as class H2 for each threshold, R.2 also shown. The figure highlights two thresholds: the one that minimizes 1- F1 and another one that results in the same value of NEC1, As discussed in Section 3.1 between two systems with the same NEC1, the F1 prefers the one with larger Rs2, effectively giving more cost to the errors in class H2 by favoring systems that produce more detections of this class. Concretely, given the (hreshold to for which F1 is minimized in a certain dataset, an equivalent NEC for which the Bayes threshold for the log odds is to can be defined by using Equation (29) in Appendix A.2, setting C12 = and 021 = e to, For the example in Figure [ the NEC that would lead to the same optimal threshold as the F1 (to -0660 is the one with C21 = 1.8. That is, in this case, optimizing F1 is equivalent to optimizing NEC assuming that errors in samples of class H2 are 1.8 times more costly than those in samples of class H1. The problem is that this cost is not explicitly selected by the user of the metric and, instead, depends on the distribution of the scores under evaluation.
Figure 2 further illustrates the dependency on the scores of the effective costs corresponding to Fj. The figure shows the F2 and two NEC for two different sets of scores: (1) the same scores used for Figure [ and (2) a new set of scores oblained using the same procedure as for the first but changing the variance for the features from 0.20 to 0.06, simulating a better system. We can see that the F1 is optimized at a different threshold for both datascts, corresponding to an equivalent C21 of 1.8 for the original dataset, as discussed above, and an equivalent cost of 1.0 for the easier dataset (with C12 always fixed at 1.0). Hence, the F1 effectively optimizes for different. trade-off between error types depending on the system under evaluation. On the other hand, since both datasets are well-calibrated, the Bayes threshold for each of the NECs (Appendix A.2) optimizes the corresponding NEC. This, we believe, is one of the strongest arguments to prefer the EC over the F3 (the analysis above was done for F1 but holds for any Fg). For EC the costs can be explicitly selected and the optimal threshold for calibrated posteriors can be determined thcoretically as a function of the costs. On the other hand, for F3, the costs are implicit and depend on factors that are outside of our control making the optimal threshold and the effective costs vary across systems.
The third plot in Figure [ shows one minus MCC and the NEC value with = 4, along with R.2. The MCC and this NEC are related by Equation (1).. The figure highlights two thresholds: the one that optimizes MCC and a second one that results in the same value of NEC, as the first one. Comparing the values of MCC across these two threshold, we can observe the behavior explained in Soction 3.2 among two thresholds that are equivalent in terms of NEC,, the MCC prefers the one that results in a more imbalanced R.2- As for Fg, the optimal threshold for MCC is determined, in part, by the value of R.2- As consequence, also
15

---

Published in Transactions on Machine Learning Research (02/2025)
1.0
0.150
1-Fi
0.8
0.125
NEC1o
0.100
NEC18
0.6
0.075
0.4
0.050
0.2
0.025
0.0
0.000
-2
-1 0
2
-2
-1
1
2
thr
thr
Figure 2: Fi and two NEC metrics with C12 = 1 and 021 indicated subindex, for two different datasets. In the left plot, the metrics are computed for the same dataset in Figure 1 In the right plot they are computed on an casier dataset (note the change in the limits in the axis). The two NEC are selected to use the effective cost corresponding to F1 for each of the two datasets. For the easier dataset (right), the best threshold for F1 coincides with the Bayes threshold for NEC10- For the harder dataset (left), the best threshold for F, coincides with the Bayes threshold for NEC1..
for Fg, the equivalent cost implicit in the MCC depends on the system under evaluation giving us no explicit control over the trade-off between lypes of errors.
Finally, the fourth plot in Figure ! compares the two NECs from the middle plots with each other and with a third NEC with a larger C2l- The Bayes threshold for NEC, is given by log(1/c) (sce Equation (29) in the Appendix): larger values of C21 move the threshold
the left decreasing the error rate for class Hg and increasing it for class H1. Note that, as mentioned in Section 2.4 the Bayes threshold is only optimal if the scores are well-calibrated. If this is not the case, the threshold can be tuned empirically, as usually done for any other metric. Alternatively, a calibration stage can be added to the system enabling the use of Bayes thresholds Filhoo et al., 2023 Ferrer & Ramos 2025). In any case, the EC allows us to explicitly select the relative cost between the different error types which directly and transparently affects the resulting metric values and, when the scores are calibrated, the corresponding optimal threshold.
4.1.2 Comparison of ECs for a 10-class dataset
In this scction we illustrate the use of the EC for a 10-class problem. We compare EC and NEC obtained with the following cost matrices: 1) the 0-1 matrix (C01) which corresponds to the error rate, 2) matrix with Gy 1/(KP,) for i + j (CinvP) which corresponds to the balanced error rate, 3) the C01 matrix with the last row multiplied by 100, representing a use case where crrors in the last class are much more costly than errors in other classes (Cimb), 4) the C01 matrix with an additional column corresponding to an abstention decision, with cost of 0.05 for all classes (Cabsl), and 5) as Cabs1 but with an abstention cost of 0.3 for all classes (Cabs2). Table shoows the EC and NÉC results for these cost matrices for three methods for making categorical decisions: choosing always the best naive decision for each cost matrix, 2) making "argmax" decisions, i.e., choosing the class with the largest score for each sample, regardless of the cost matrix (for this row, decisions are the same for all cost matrices), and 3) making the Bayes decisions corresponding to each cost matrix using Equation 77.
Comparing the last two lines this table we see that, as theory shows, Bayes decisions result in better performance than argmax decisions, except for the C01 matrix for which Bayes decisions are the argmax decisions and, hence, those two results are the same. Further, the table highlights the benefit of normalization. While the EC' might be misleadingly low for some cost matrices due to the imbalanced priors and the selected cost values, the NEC solves this problem by using the EC of the best naive system as bascline with which to normalize, resulting in a more interpretable metric.
The best naive decision depends on the cost matrix as determined by Equation 3 For C01, the class with the largest prior, H,, is the least-costly naive docision. For CinvP, all decisions are equally good due to the way the costs are defined. For Cimb, the least costly decision is H10 due the large cost assigned to errors in that class. Finally, the least-costly decision for Cabsl is to abstain, while for Cabs2 it is H1 (see Appendix A.3
16

---

Published in Transactions on Machine Learning Research (02/2025)
Table 1: EC and NEC for various costs matrices defined in Section 1.1.2 for systems that make the best naive decision (the decision made by this system is indicated in parenthesis under the cost matrix name), argiax decisions, and Bayes decisions. The accuracy and balanced accuracy can be obtained as one minus the EC for C01 and CinvP, respectively. For the two cost matrices with an abstain decision, the percentage of abstain decisions made by the system is also listed (abs).
C01
CiuvP (any class)
Cimb (class H1o)
Cabs1
Cabs2
Naive Decision +
(class H1)
(abstain)
(class Hi)
Decision algorithm
EC
NEC
EC
NEC
RC
NEC
EC
NEC
abs
EC
NEC
abs
Naive
0.20
1.00
0.90
1.00
0.98
1.00
0.05
1.00
100
0.20
1.00
Argmax
0.06
0.32
0.28
0.31
0.36
0.37
0.06
1.29
0
0.06
0.32
0
Bayes
0.06
0.32
0.23
0.26
0.08
0.08
0.02
0.35
0.06
0.28
for the expression for Bayes decisions and for the best naive decisions for the Cabs cost matrices). As shown in the column "abs" for these last two cost matrices, larger percentage of abstain decisions are made with the Bayes decision algorithm for Cabsl than Cabs2, due to the fact that Cabsl assigns a lower abstention cost than Cabs2. Finally, we can see that while the lower abstention cost in Cabsl results a lower EC value than for Cabs2, the NEC shows the opposite trend. This happens because the best naive system is different for both ECs, resulting in different baselines used for normalization.
Note that the scores used for these synthelic experiments are, by design, perfectly calibrated, so Bayes decisions are optimal in this dataset. In practice, this might not be the case for a certain datasct of interest. Unlike in the scenario of the previous section where we could simply tune a threshold instead of using Bayes decision theory, for the multi-class case or, more generally, when the number of decisions is larger than 2, tuning the decision regions is not trivial (though attempts have been made in this direction, e.g., by O' et al. (2008)) In those scenarios, we recommend making sure that the system outputs are well-calibrated (Filho et al. 2023; Ferrer & Ramos 2025) and then applying Bayes decision theory as done in this section.
4.2 Results on real datasets
In this section, show results on real datasets for mumber of different tasks corrosponding to speech, image and natural language processing lasks. The datasets and systems used to generate the scores studied in this section are described in Appendix C For each system, we show two sets of results obtained on the raw scores as they come out of the system, and on calibrated scores. The calibrated scores are obtained with linear logistic regression using 5-fold cross-validation on the test data. Finally, for some datasets, we show results for resampled versions where some of the classes are downsampled to obtain a dataset with priors different from the original ones to show the effect this change has on the metrics. The details on the calibration process and the priors for each dataset are listed in Appendix C
Table 2 shows the results on all these datasets for three NECs, accuracy (ACC), two multi-class extensions of the Fi, and, for the binary cases, the standard binary F1, AUC, and EER. The multi-class extensions of the F1 are the average of the F1 over all classes (MC1) and the F1 of the average recall and precision (MC2) (see Section 4.1.2.. To facilitate comparisons, for those metrics that increase as performance improves (AUC, ACC, and the three F1), we report one minus the metric value. Decisions for the three NEC metrics are made using Bayes decision theory. For all three Fis and ACC, decisions are made with the argmax approach, ie., by selecting the class with the highest score, as is usually done in the literature. For AUC and EER. decisions are not made since those metrics evaluate scores rather than categorical decisions.
Comparing the results on raw and calibrated scores we can see that AUC and EER are insensitive to the linear logistic calibration applied in these experiments since, as explained in Section 3.6 these metrics are invariant to monotonic transformations of the scores. The small variation in AUC and EER results observed in some cases after calibration is due to the prossvalidation approach, which results in different transformations being applied to different. subsets of the data. We can also see that, as explained in Section 3.6 when the scores are
17

---

Published in Transactions on Machine Learning Research (02/2025)
Table 2: Various metrics on real scores for different speech, image and natural language processing datasets. We report three NEC values, EER, 1-ACC, 1-F; for three version of F1, and 1-AUC (lower values better for all cases). Note that 1-ACC corresponds to the EC for C01 (EC values are not shown in the table to reduce clutter). The term 'ReNN' in some of the dataset. names identifies resampled version of dataset. The colored entries highlight comparisons discussed in the text.
(see Table 31.
NEC CinyP
1-ACC
1-F
1-AUC
EER
Dataset
System
proc C01
Cimb
MC1
MC2
Bin
raw 0.996
0.992
1.000
0.497
0.667
0.667
0.332
0.048
0.116
GP12-4shh
cal 0.226
0.225
0.921
0.113
0.486
0.329
0.115
0.048
0.116
SST2
raw 0.828
0.826
1.000
0.414
0.667
0.667
0.294
0.072
0.152
GPT2-0sh
cal 0.310
0.308
0.907
0.155
0.571
0.363
0.156
0.072
0.152
raw 1.931
0.991
1.000
0.579
0.769
0.769
0.493
0.074
0.159
SST2-Res1
GPT2-0sh
cal 0.451
0.306
1.150
0.135
0.595
0.379
0.238
0.074
0.158
raw 3.298
1.000
1.000
0.660
0.833
0.833
0.625
0.077
0.150
SST2-Res2
GPT2-0sh cal 0.526
0.305
1.071
0.105
0.597
0.380
0.294
0.079
0.155
raw 0.324
0.071
0.095
0.002
0.415
0.295
0.167
0.002
0.021
SITW
XvPLDA cal 0.306
0.043
0.062
0.002
0.302
0.236
0.166
0.002
0.022
raw 0.195
0.070
0.181
0.019
0.098
0.090
0.107
0.002
0.022
SITW-Resl
XvPLDA cal 0,117
0.044
0.126
0.012
0.154
0.134
0.059
0.002
0.022
XvPLDA raw 0.192
0.103
0.352
0.038
0.056
0.054
0.106
0.002
0.022
SITW-Res2
call 0.083
0.045
0.166
0.017
0.121
0.108
0.042
0.002
0.022
raw 3.915
0.315
0.362
0.067
0.566
0.362
0.662
0.000
0.001
FVCAUS
XvPLDA cal 0.012
0.003
0.005
0.000
0.017
0.016
0.006
0.000
0.001
raw 0.420
0.055
0.055
0.004
0.223
0.187
0.223
0.003
0.022
CIFAR-IvsO
Resnet20 cal 0.430
0.041
0.041
0.004
0.312
0.239
0.230
0.003
0.023
Resnet20 raw 0.700
0.150
0.151
0.007
0.327
0.262
0.368
0.013
0.056
CIFAR-2vsO
cal 0.560
0.108
0.109
0.006
0.393
0.289
0.346
0.013
0.056
raw 0.504
0.434
0.839
0.349
0.567
0.445
TEMOCAP
W2V2 cal 0.494
0.428
0.804
0.342
0.661
0.489
W2V2 raw 0.498
0.457
0.665
0.340
0.632
0.460
IEMOCAP-Res
cal 0.477
0.435
0.620
0.322
0.598
0.442
GPT2-0sh raw 0.780
0.780
1.015
0.585
0.897
0.694
AGNEWS
cal 0.378
0.378
1.179
0.283
0.769
0.567
GPT2-0sh raw 0.742
0.946
0.998
0.497
0.992
0.714
AGNEWS-Resh
cal 0.277
0.365
0.584
0.186
0.423
0.336
Resnet20 raw 0.082
0.082
0.273
0.074
0.087
0.085
cal 0.083
0.083
0.184
0.075
0.126
0.116
CIFARIO
raw 0.068
0.068
0.357
0.061
0.064
0.064
Vgg19
cal 0.069
0.069
0.186
0.062
0.093
0.087
raw 0.053
0.053
0.169
0.047
0.053
0.052
Rep V2g-a2
0.052
0.052
0.129
0.047
0.073
0.069
raw 0.315
0.315
0.357
0.312
0.319
0.314
Resnet20
cal 0.317
0.317
0.351
0.314
0.329
0.321
CIFAR100
raw 0.264
0.264
0.465
0.261
0.260
0.258
Veg19
cal 0.266
0.266
0.358
0.263
0.271
0.260
raw 0.227
0.227
0.351
0.225
0.230
0.223
Rep
cal 0.228
0.228
0.341
0.226
0.229
0.223
18

---

Published in Transactions on Machine Learning Research (02/2025)
calibrated, the EER value is (approximately, since are dealing with limited-size datasets) upper bound for the F.C value corresponding to the C01 cost (1-ACC) (see brown entries in the table).
Results across versions of the same dataset with different priors (SST2, SITW, IEMOCAP, and AGNEWS) illustrate how the variation in priors can greatly affect the NEC values for C01 and Cimb, as well as F1 and ACC (cyan entries highlight those comparisons for SST2 and AGNews for calibrated scores). On the other hand, AUC and EER are (again, approximately, since we are dealing with limited-size datasets) unaffected by the priors. Further, for the calibrated systems, the NEC for CinvP is also (approximately) unaffected. This is, in fact, exactly was this cost matrix is designed to do. Its costs are determined for each dataset the inverse of the priors so that the final metric is immune to the priors in the dataset as long as the scores are calibrated and Bayes decisions are made or the threshold is determined by optimizing the metric on a development set. A more general version of this type of cost matrix designed to be independent of the priors of the evaluation set was used by Godau et al. (2023) to address the problem of tasks where the priors in deployment are different from the priors in the development data.
Comparing the three flavors of Fi with each other can that they are often quite different (see, for example, the blue entries in Table 2). The difference stems in part from the fact that, while the binary F1 prioritizes one of the classes, the multi-class versions are symmetric across classes. Yet, despite both multi-class versions being symmetric in the classes, they differ from each other and they can even lead to contradicting conclusions, as discussed by Opitz & Burst (2019). Importantly, none of the F1 variants (or, more generally, Fa variants) offers direct control on the trade off between error types, like the EC does. Hence, there is no principled way to select between the different variants based on the needs of the task. Further, we can see that the multi-class F1 metrics sometimes degrade after calibration (see SITW-Res1, SITW-Res2, IEMOCAP and some of the CIFAR results two of these examples are marked in orange in the table). These results illustrate the fact that argmax decisions are not necessarily optimal for the multi-class F1 metrics, even if the scores are perfectly calibrated since, if making argmax decisions with calibrated scores was optimal, no other way of making decisions would lead to better rosults 3 The fact that, in some cascs, using the raw scores to make argmax decisions leads to better results than using the calibrated scores implies that better decision strategy exists, and, hence, argmax decisions from calibrated scores are not guaranteed to optimize these metrics. As far as know, there is no principled way to optimize decisions for these metrics other than empirical optimization. This, though, is very rarely done for the multi-class case given the difficulty involved in optimizing decision regions in more than one dimension. In contrast, if scores are calibrated, for every NEC metric there is a corresponding decision strategy that is guaranteed to be optimal Ferrer& & Ramos 2025).
Finally, a general observation from Table 2 is that systems can be ranked differently depending on the metric used to rank them. For example, while calibration improves the Cimb NEC value for CIFAR10 Resnet20 system, it does not affect the other NECs the ACC, and it actually degrades the multi-class F, metrics (see red entries in the table). This also holds for the other two systems CIFAR10. Also, comparing results of the Resnet20 and Vgg19 calibrated scores for the CIFAR10 and CIFAR100 datascts we can see that Vgg19 is better than Resnet20 for both datasets in terms of all metrics except Cimb NEC, for which the Resnet20 system is slightly better (see green entries and corresponding rows for CIFAR10). If Cimb NEC was the metric of interest for this task (i.e., if errors in class 010 costed 100 times more than errors on other classes), then assossing the system performance based on ACC or one of the F₁ could lead to suboptimal development decisions. For example, in this case, using ACC would lead us to conclude that calibration is not necessary and that the Vgg19 system is better than the Resnet20 system. In contrast, for Cimb NEC, the metric that reflects the needs of the task, calibration gives gain and Resnet20 is slightly better than Vgg19, having the additional advantage of being 100 times smaller (see Appendix C These are just some examples that highlight, the importance of selecting an appropriate metric for the scenario of interest, a task that is greatly facilitated by the EC metric.
9This is also for the binary F1 metric, as can be seen in Figure 1 where the optimal threshold for F1 is not the one that corresponds to argmax decisions (which is zero in the log-odds domain).
19

---

Published in Transactions on Machine Learning Research (02/2025)
5 Conclusions
In this work we focus the problem of evaluating the quality of categorical decisions generated by a machine learning classifier. We argue that the expected cost (EC), classic evaluation function proposed decades ago, should be the primary metric to assess the performance of categorical decisions for a certain application of interest. The EC is based on the simple assumption that each decision made by the system can be assigned a cost which depends on the true class of the sample. The expectation of the cost over the data can then be used as a metric to assess the performance of system. The EC can be customized to any application for which categorical decisions are required by adapting the costs, and optionally the class priors, according to the needs of the application.
While two special cases of the FCC, the error rate and the balanced error rate are widely used in the machine learning literature, its general form is only rarely leveraged. Instead, metrics like the F3 score or the MCC are used for applications in which the error rate is deemed inadequate for performance assessment. This is often the case in applications with large class inbalance and where errors the minority class are considered more important than those in the majority class. In this work, we argue that the EC better metric than those ad-hoc alternatives. The advantages of the EC can be summarized as follows:
It is defined based on the simple and general assumption that each combination of true class and decision can be assigned corresponding fixed cost. In contrast, for other classification metrics, the trade-off between different types of error cannot be directly controlled.
The class prior distribution can be considered a parameter of the metric. This is useful in cases in which the priors in the evaluation data do not coincide with those expected at deployment.
The same general EC expression can be used to evaluate both binary and multi-class classification systems in principled way.
Unlike other classification metrics, the EC does not assume that the set of possible decisions is the set of classes. Instead, the decisions can be the actions that may be taken by the user of the system.
When normalized by the EC of the best naive systems for the selected costs and priors, the resulting metric is easily interpretable as relative value with respect to the performance of the best naive system.
Optimization of catogorical decisions can be done using Bayes decision theory as long as the system scores are well-calibrated, eliminating the need for ad-hoc solutions. Alternatively, though, when calibrated scores are not available, the decision strategy can be based on threshold or region tuning, as for any other classification metric.
The need to choose costs may seem like drawback of the EC compared to other metries like the F1 score or the MCC which have no parameters to select. Yet, is it important to note that, while these alternative metrics do not explicitly require the selection of costs, they effectively impose trade off between different types of error. This is easy to see for the binary case, where decisions are made by thresholding, by noting that the threshold that optimizes a metric for a certain set of scores is the Bayes threshold for some EC. Hence, optimizing the original metric is equivalent to optimizing the related NEC. Effective costs are implicitly considered in the optimization. Yet, those costs are selected without our control and depend on the system under cvaluation. The EC, on the other hand, allows for an explicit choice of costs. Once the costs are chosen, the trade-off between different lypes of errors is fixed and transparent and does not depend on the system under evaluation. While the costs may be hard to specify in some cases, we argue that having control over them is better than letting the metric implicitly choose them.
In conclusion, we argue that the EC, or its normalized version, should be the preferred metric for evaluating categorical decisions for a given application of interest. This work is accompanied by a python library of methods to compute the FC, along with utility methods for making Bayes decisions and calibrating scores. The code includes notebooks that produce the results in this paper. We hope this work will facilitate the wider adoption by the machine learning community of this classic, simple, and well-motivated metric.
20

---

Published in Transactions on Machine Learning Research (02/2025)
References
D. Ashby and A. F. M. Smith. Svidence-based medicine as bayesian decision decision-making making, Statistics in Medicine, 19(23)33291 3305, 2000.
P. Baldi, S. Brunak, Y. Chauvin, C. A. F. Andorsen, and H. Nielsen. Assessing the accuracy of prodiction algorithms for classification; an overview. Bioinformalies, 05 2000.
J. O. Berger. Statistical decision theory and Bayesian analysis. Springer Science & Business Media, 2013.
J. M. Bernardo and A. F. M. Smith. Bayesian Theory. John Wiley & Sons, 1994.
S. Bickelhaupt,
F. Jaeger, B. Laun, W. Lederer, H. Daniel, A. Kuder, L. Wuesthof, D. Paech, D. Bonekamp, A. Radbruch, S. Delorme, H.-P. Schlemmer, F. H. Steudle, and K. Н. Maier-Hein, Radiomics based on adapted diffuusio kurtosis imaging helps clarify most mammographic findings suspicious for cancer. Radiology, 287(3):761-770, 2018.
C. M. Bishop. Pallern Recognilion and Machine Learning. Springer, 2006.
A. P. Bradley. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pottern Recognition, 30(7):1145 1159, 1997.
N. Brümmer. Measuring, Refining and Calibraling Speaker and Language. Information Extracted from Speech. PhD thesis, Stellenbosch University, 2010.
Brümmer, L. Ferrer, and A. Swart. Out of a Hundred Trials, How Many Errors Does Your Speaker Verifier Make? In Proc. Interspeech, Brno, Czech Republic, Sept. 2021.
K. Chan, T.W. Lee, P. A. Sample, M. H. Goldbaum, R. N. Weinreb, and T. J. Sejnowski. Comparison of machine learning and traditional classifiers in glaucoma diagnosis. IEEE Transactions on Biomedical Engineering, 49(9):963 974, 2002.
D. Chicco and G. Jurman. The advantages of the matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, 21(1), Jan 2020.
L. E. Cowley, D. M. Farewell, S. Maguire, and A. M. Kemp. Methodological standards for the development and evaluation of clinical prediction rules: review of the literature. Diagnostic and prognostic research, 3: 1-23, 2019.
M. H. DeGroot. Optimal Statistical Decisions. McGraw-Hill, 1970.
M. H. DeGroot and S. E. Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society. Series D (The Statistician), (2(1/2):12-2) 1983.
R. Duda, P. Hart, and D. Stork. Pattern Classification. Wiley, 2001.
K. Dyrland, A. S. Lundervold, and P. Porta. Mana. Does the evaluation stand up to evaluation? a first-principle approach to the evaluation of classifiers. arXiv:2302.12000 May 2022.
C. Elkan. The foundations of cost-sensitive learning. Proceedings of the Sevenleenth Internalional Conference on Artificial Intelligence: 4-10 August 2001; Seattle, 1, 05 2001.
L. Ferrer and D. Ramos. Evaluating posterior probabilities: Decision theory, proper scoring rules, and calibration. Transactions Machine Learning Research, January 2025.
T. S. Filho, H. Song, M. Perello-Nieto, R. Santos-Rodrigueer M. Kull, and P. Flach. Classifier calibration: How to assess and improve predicted class probabilities: a survey. Machine Learning, 112, 2023.
T. Gneiting and A. E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 2007.
21

---

Published in Transactions on Machine Learning Research (02/2025)
P. Godau, P. Kalinowski, E. Christodoulon, A. Reinke, M. Tizabi, L. Ferrer, P. Jäger, and L. Maier-Hein. Deployment of image analysis algorithms under prevalence shifts. In Proc. of MICCAT, 2023.
M. H. Goldbaum, P. A. Sample, K. Chan, Williams, T.-W. Lee, E. Blumenthal, C. A. Girkin, L. M. Zangwill, C. Bowd, T. Sejnowski, et al. Comparing machine learning classifiers for diagnosing glaucoma from standard automated perimetry, Investigative ophthalmology & visual science, 43(1):162 169, 2002.
J. Monzalez-Rodrigneee Evaluating automatic speaker recognition systems: An overview of the NIST speaker recognition evaluations (1996-2014). Loquens, 1(1), 2014.
I. J. Good. Rational decisions. Journal of the Royal Statistical Society: Series (Methodological), 14(1): 107 114, 1952.
J. Gorodkin. Comparing two k-category assignments by k-category correlation cocfficient. Computational Biology and Chemistry, 2004.
M. Grandini, E. Bagli, and G. Visani. Metrics for multi-class classification: an overview. arXiv 2008.05756, 2020.
C. S. Greenberg, L. P. Mason, S. O. Sadjadi, and D. A. Reynolds. Two decades of speaker recognition evaluation at the National Institute of Standards and Technology. Computer Speech and Language, 60, 2020.
A. Gulli. The anatomy of a news search engine. Special Interest Tracks and Posters of the 14th International Conference on World Wide Web, pp. 880 881, New York, 2005.
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In Proc. of the 34th International Conference on Machine Leurning, Sydney, Australia, 2017.
J. C. Harsanyi. Bayesian decision theory and utilitarian ethics. The American Economic Review, 68(2): 223 228, 1978.
T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer- 2001.
J. Mananddz-Oralllo, P. Flach, and C. Ferri. A unified view of performance metrics: Translating threshold choice into expected classification loss. Journal of Machine Learning Research, 13(91), 2012.
S. A. Hicks, Strümke, V. Thambawita, M. Hammou, M. A. Riegler, P. Halvorsen, and S. Parasa. On evaluation metrics for medical applications of artificial intelligence. Scientific Reports, 12(1), apr 2022.
M. G. M. Hunink, M. C. Weinstein, E. Wittenberg, M. F. Drummond, J. S. Pliskin, J. B. Wong, and P. P. Glasziou. Decision Making in Health and Medicine: Integrating Evidence and Values. Cambridge University Press, 2 edition, 2014.
D. Kahneman. Thinking, fast and slow. 1st ed. New York Farrar, Straus and Giroux, 2011.
J. Kornak and Y. Lu. Bayesian decision analysis for choosing between diagnostic procedures. Statistics and Its Interface, 4(1):27-3) 36, 2011.
Slagnostic/proonostic prognostic prediction
L. Maier-Hein Hein, A. Reinke, P. Godau, M. D. Tizabi, F. Buettner, E. Christodoulou, B. Glocker, F, Isensee, J. Kleesiek, M. Kozubek, M. Reyes, M. A. Riegler, M. Wiesenfarth, A. F. Kavur, C. H. Sudre, M. Baumgart- ner, M. Eisenmann, D. Heckmann-Nötzel, T. Rädsch, L. Acion, M. Antonelli, T. Arbel, S. Bakas, A. Benis, M. B. Blaschko, M. J. Cardoso, V. Cheplygina, B. A. Cimini, G. S. Collins, K. Farahani, L. Ferrer, A. Gal- dran, B. van Ginneken, R. Haase, D. A. Hashimoto, M. M. Hoffman,, M. Huisman, P. Jannin, C. E. Kahn, D. Kainmueller, B. Kainz, A. Karargyris, A. Karthikosalingan, F. Kofler, A. Kopp-Schncider, A. Kreshuk, T. Kure, B. A. Landman, G. Litjens, A. Madani, K. Maier-Hein, A. L. Martel, P. Mattson, E. Meijering, B. Menze, K. G. M. Moons, H. Müller, B. Nichyporuk, F. Nickel, J. Petersen, N. Rajpoot, N. Rieke, J. Sacz-Rod Rodrigg,,, C. I. Sánchez, S. Shetty, M. van Smeden, R. M. Summers, A. A. Taha, A. Tiulpin, S. A. Tsaftaris, B. Van Calster, G. Varoquaux, and Jäger. Metrics reloaded: recommendations for image analysis validation. Nature Methods, 21 21(2):195 212, February 2024.
22

---

Published in Transactions on Machine Learning Research (02/2025)
B. W. Matthews. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. In Biochim Biophys Acla Protein Struct., 1975.
M. McLaren, L. Ferrer, D. Castan, and A. Lawson. The speakers in the wild (SITW) speaker recognition database. In Proc. Interspeech, San Francisco, Sept. 2016.
V. Mingote, A. Miguel, D. Ribas, A. Ortega, and E. Lleida. Optimization of False Acceptance/ Rejecctoon Rates and Decision Threshold for End-to-End Tex--Dependent Speaker Verification Systems. In Proc. Interspeech 2019, pp. 2903 2907, 2019.
G. S. Morrison, Rosc, and C. Zhang. Protocol for the collection of databases recordings for forensic- voice-comparison research and practice. Australian Journal of Forensic Sciences, 44(2):155 167, June 2012.
G. Morrison, C. Zhang, and E. Enzinger et. al. Forensic database of voice recordings of 500+ australian english speakers. http://databases
http://aaaaaaeee..eeeeeeeeeeeeea foremsii---oooeeccommaaiiio net 2015.
D. B. O'Brien, M. R. Gupla, and R. M. Gray. Cost-sensitive multi-class classification from probability estimates. In Proceedings of the 25th international conference on Machine learning ICML '08. ACM Press, 2008.
J. Opitz and S. Burst. Macro fl and macro fl. arXiv preprint arXit:1911.033// 2019.
S. G. Pauker and J. Kassirer. Therapeutic decision making: A cost-benefit analysis. New England Journal of Medicine, 293(5):229 234, Jul 1975.
M. Peterson. An Introduction to Decision Theory. Cambridge Introductions to Philosophy. Cambridge University Press, 2009.
Z. Z. Qin, M. S. Sander, B. Rai, C. N. Titahong, S. Sudrungrot, S. N. Laah, L. M. Adhikari, E. J. Carter, L. Puri, A. J. Codlin, et al. Using artificial intelligence to read chest radiographs for tuberculosis detection: A multi-site evaluation of the diagnostic accuracy three deep learning systems. Scientific reports, 9(1): 15000, 2019.
H. Raiffa. Decision analysis. introductory lectures on choices under uncertainty. Reading, 36(5):527-528 December 1970a.
M. M. H. Raiffa. Decision analysis. introductory lectures on choices under uncertainty. Recherches économiques de Louvain, 36(5):527 528, 1970b.
C. J. V. Rijsbergen. Information Retrieval. Butterworths, 1979.
R. D. Riley, T. P. A. Debray, G. S. Collins, L. Archer, J. Ensor, M. van Smeden, and K. 1. E. Snell. Minimum sample size for external validation a clinical prediction model with a binary outcome. Statistics in Medicine, 40(19):4230 12000-221,2200 2021.
S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach. Prentice Hall, 2010.
L. J. Savage. The foundations of statistics. Courier Corporation, 1972.
M. K. Siddiqui, R. Mennles- X. Huang, and N. Hussain. review of epileptic scizure detection using machine learning classifiers. Brain Informatics, 7(1), may 2020.
A.-M. Šimundić. Mcasures of diagnostic accuracy: basic definitions. ejifec, 19(4):203, 2009.
R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference Empirical Methods in Natural Longuage Processing, pp. 1631 1631-1642, Seattle, Washington, USA, October 2013.
M. B. Souza, F. W. Medeiros, D. Souza, R. Garcia, and M. R. Alves. Evaluation of machine learning classifiers in keratoconus detection from orbscan ii examinations. Clinics, 65(12):1223 1228, 2010.
23

---

Published in Transactions on Machine Learning Research (02/2025)
A. A. Taha and A. Hanbury. Metrics for evaluating 3d medical image segmentation: analysis, selection, and tool. BMC Medical Imaging, 15(1), aug 2015.
G. Tsoumakas and I. Katakis. Multi-label classification: An overview. International Journal of Data Warehousing and Mining (IJDWM), 3(3):1 13, 2007.
D. A. Van Leeuwen and N. Brümmer. An introduction to application-iilleendenu evaluation of speaker recognition systems. In Speaker classification I: Fundamentals, Features, and Methods. Springer-Veriee 2007.
A. J. Vickers, Van Calster, and E. W. Steyerberg. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ, 352, 2016.
R. Wang, D. Zhou, M. Jiang, J. Si, and Y. Yang. A survey on opinion mining: From stance to product aspect. IEEE Access, 7:41101-41122 2019.
B. Zadrozny and C. Elkan. Learning and making decisions when costs and probabilities are both unknown. In Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Dala Mining, KDD '01, pp. 204-213, New York, NY, USA, 2001. Association for Computing Machinery.
M.-L. Zhang and H. Zhou. A review on multi-label learning algorithms. IEEE transactions on knowledge and data engineering, 26(8):1819-188) 2013.
X. Zhang, J. Zhao, and Y. LeCun. Character-leve convolutional networks for text classification. In Proc. of NewrIPS, 2015.
A Special cases of Bayes decisions theory
In this section we discuss three special cases of Bayes decision theory, the argmax rule which is obtained when the cost function is given by the 0-1 cost matrix, the thresholding rule obtained for the binary case with only two decisions, and the case in which the cost function includes an abstention decision.
A.1 The 0-1 cost matrix
A special scenario Bayes decision theory applied to the EC arises when the cost matrix is the 0-1 square matrix given by one minus the identity matrix. In this case, the EC reduces to the error rate or one minus the accuracy (see Section 2.3). The Bayes decisions for this EC can be obtained by replacing Cy for and Ctj 1 for + in Equation 7 in Section 2.4 which gives:
K Σ
dn(x) =
arg min
P((|))
(24)
arg min 1 P(j
(25)
= arg max P(j (z).
(26)
This is the standard argmaxx decision rule used in most of the literature in machine learning for making decisions in a multi-class setting. We can now see that this rule is optimal for the 0-1 cost matrix. For any other cost matrix, the general expression in Equation should be used to make decisions.
A.2 Binary classification with two decisions
Another special case of Bayes decision theory corresponds to binary classification when the cost matrix is square (i.e., when there is only two possible decisions) with zeros in the diagonal. This is the simplest and
24

---

Published in Transactions on Machine Learning Research (02/2025)
most common case in which there is one correct zero-cost decision for each class. In that case, Bayes decisions reduce to:
ds(x)= H2, if (()
(27)
Hi; otherwise
Using the fact that P(H2lx) = 1- -(HH)))) this rule becomes:
dn(x)= H2, if P(H2|=) >t
where t= c2 +012++ 021) 1/ c21/c12)
(28)
Hi, otherwise
Sometimes we prefer to work with posterior log odds because the logarithm mitigates numerical precision problems and because their distributions are nicer for plotting (see an example in Figure 1 of the main text).
When using log odds scores, the decision rule becomes:
H2, {H2. if log(P(HH)))//((H()))) db(r)= = 1 H1, otherwise
Mogodds
where
bloodds log(e12/c21)
(29)
In summary, for binary classification with one zero-cost decision per class, Bayes decisions simply consist. of comparing the posterior for one of the classes, or some function of the posteriors, like the logodds, with a specific threshold which is function of the ratio between the two costs. While to optimize metries like the Fa or MCC we need lo sweep threshold and choose the best one for that metric, for the EC we simply need to plug the costs into Equation
28 or 29 to get the optimal threshold.
Note that, as discussed in the main text, Bayes decisions are optimal as long as the classifier is calibrated for the selected cost. If this is not the case, for the binary case above, the threshold can be tuned as for any other metric. For the general case, including multi-class cases or binary cases with more than two docisions (non-square cost matrix), the posteriors can be calibrated to enable the use of Bayes decision theory (Filho et al. 2023 For in-depth discussion of Bayes decision theory and calibration, please refer to (Ferrer & Ramos 2025
A.3 Cost matrices with an abstention option
Finally, we derive the expression for the Bayes decisions when the set of decisions is given by [H,,,.., H abstain] (the list of classes plus an abstain decision) and the cost matrix has the follow- ing form:
C= a1]
(30)
where is the identity matrix, so that 1-1 is the 0-1 cost matrix with a cost of in the diagonal and a cost of elsewhere, and where al is a vector of ones multiplied by scalar a which determines the cost of abstention, set to be equal for all classes. Replacing these costs in Equation
we get the following expression:
K dz(z)= = arg min Σ
P(H x)) = 1 P(H))). if } abstain Eillie
45 P(H.(z) arg min
(31)
aP(H) aa
if j abstain
Or, equivalently:
P(H)a=), dz(x) arg max
if j + abstain
(32)
1-a, if j == abstain
Hence, the system will abstain if all the posteriors output by the system are smaller than 1 a.
We can now easily derive the expression for the best naive decision which is the Bayes decision when the system outputs always the prior probability of cach class in place of the posteriors 10 Doing that, we obtain:
Pr 1-a,
if # abstain
dasive arg max j
(33)
if j = abstain
10This same procedure can be used to arrive at the general expression for the EC of the best naive system, Equation g in Section 2.1 which was derived in a different way in that section.
25

---

Published in Transactions on Machine Learning Research (02/2025)
Hence, the best naive decision is to abstain when all priors smaller than 1 1-a For our multiclass experiments in Section 4 where P1 = 0.8 and P. 0.022 for of 1, the best naive decision is to abstain for Cabsl where 1 1-cc a = 0.95, but not for Cabs2, where 1 - a = 0.7. In that case, the best decision is H1, which has a prior 0.8.
B Synthetic datasets for experiments
Here describe the procedure used to create syuthetic datasets for the experiments in this paper. Given number of classes K, prior for class H1 of P1, and per-class variance o, we proceeded as follows:
Set the class priors to Pi for class H1, and P. = (1 - Pi)/(K 1) for classes through K. In our experiments, P1 is set 0.8.
2. Set the total number of samples, N = 100, 000 and determined the number of samples for each class, N₁ the closest integer to NP.
3. Generated N; samples using unidimensional Gaussian distribution Wi,, with mean μ4 =i- and variance 0, taking these samples as the input features, x.
Computed the likelihoods for each class for each generated sample according to the class distributions used to draw these samples, i.e., (H,) ~ ()),
Finally, we computed the posteriors
P(H, p(x(HH)) P. p(x) ESp(o H2) P3
Hi) P.
(34)
where the p(rlh) are the likelihoods computed in step 4 and P. is the prior for class H; defined in step 1. With this procedure, the posteriors are perfectly calibrated for the test data.
The value of a was set 0.20 for all results in this work, except for those Figure where two datasets are used, with 0.20 and easier with 0.05.
C Real datasets for experiments
For the experiments with real datasets in Scction 4.2 we use a varicty of datascts from speech, image, and natural language processing lasks.
SST2 (Socher et al., 2013) a natural language processing dataset where the task is decide whether certain text has positive negative sentiment. AGNEWS (Gulli 2005 Zhang al. 2015) is another natural language processing dataset where the task is to classify news into 4 different classes. The scores for these datasets were produced with the GPT-2 model using the code provided in using zero-shot prompts.
SITW (McLaren et al. 2016 and FVCAUS (Morison et al. 2015
2012) are two speaker verification
datasets where the task decide whether two audio samples belong the same speaker or not. To obtain scores for these datasets, we ran an X-vector PLDA system using the code provided in https;
//github. www. wwtp///feeeer/DC---PPP
IEMOCAP is a speech processing dataset where the task is to classify each speech sample into set of emotions: angry, happy, sad, and neutral. The scores were downloaded from habla-liaa/ser-wtt www.tthhw22/ttreeemmsteeeeee httpp////e.atteeeeeeeeeee///////
com/
Finally, CIFARIO and CIFAR100 are two image processing datascts where the task classify the object in an image into one of 10 or 100 classes, respectively. The scores for these datasets were obtained using
the code available in http::///tttaacccc//a/ttt//////r
We run three
26

---

Published in Transactions on Machine Learning Research (02/2025)
Table Number of classes, number of samples, and class priors for each dataset included in our experiments.
Dataset
# CClasses
#Samples
Priors
SST2
1821
0.50 0.50
SST2-Res1
1301
0.70 0.30
2Ss2-Res2
1140
0.80 0.20
SITW
721788
0.99 0.01
SITW-Res
36580
0.90 0.10
SITW-Res2
18290
0.80 0.20
FVCAUS
114072
0.98 0.02
CIFAR-IvsO
10000
0.99 0.01
CIFAR-2vsO
10000
0.99 0.01
IEMOCAP
5473
0.20 0.29 0.31 0.20
IEMOCAP-Ran
3483
0.32 0.32 0.32 0.05
AGNEWS
7600
0.25 0.25 0.25 0.25
AGNEWS-Ras
5757
0.33 0.33 0.33 0.01
CLEAR10
10
10000
0.10 for all classes
CIFAR100
100
10000
0.01 for all classes
models for each of the two datasets: resnet20, vgg19, and repvgg a2. These models have approximately 0.27 million, 20 million, and 27 million parameters, respectively.
From the CIFAR100 Resnet20 scores, we also created binary classification tasks for the detection of one specific class versus all others, using the score provided by the system for that class and minus that score for the "other" class. We call these scores CIFAR-XvsO, where X identifies the target class.
For all sets of scores, we show the results for the raw scores, as they come out of the system, and for scores calibrated with logistic regression model. This model maps the logarithm of the raw posteriors generated by the systems using the following expression:
$ = softmax(a log(s) +
(35)
where is the vector of raw posteriors, is the calibrated vector of posteriors, is scalar and 3 is vector. The parameters and 3 were trained by minimizing the cross-entropy. In this work, the models were trained and the scores generated using a 5-fold cross-validation approach on the test data.
Resampled versions some of the datasets, identified by a suffix ResN are also used to achieve different priors from the ones in the original dataset. Table 3 shows the priors and total number of samples for all the datasets used in Section 4.2
27


Reviews:
[{"review_id": "5XSGjikHL8", "reviewer": "Reviewer_kbvx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength \n- This paper illustrates the advantages of using Expected Cost (EC) as an evaluation metric for machine learning models in classification problems by contrasting it with other key metrics such as F-beta score and MCC. In particular, I think the part explaining that the other metrics are a special case of EC and how optimizing the other metrics can be interpreted in the context of evaluating the model is well organized.\n\n### Weakness \nI understood the authors' argument to be that “EC is superior to other evaluation metrics for classification problems, such as F-beta scores, and that model evaluation in classification problems should essentially be done with EC”. However, EC is not a metric that can be used unconditionally; at least the following two points need to be cleared. \n\n- The outputs of the model should be sufficiently well calibrated and correctly interpreted as posterior probabilities\n- Costs for TP, TN, FP and FN should be set appropriately.\n\nThe author argues that, for example, in comparison to threshold optimization in other metrics and the setting of the cost matrix in EC, the latter is better (hence EC is superior). \nHowever, this does not seem a very convincing argument. Other evaluation metrics are used in various classification tasks, and there is a wealth of knowledge regarding their characteristics and appropriate use. On the other hand, the specification of the cost matrix is problem-dependent, and it can be said that there is no universally appropriate method for its specification. Additionally, it is unclear how to detect and correct cases where the cost matrix setting significantly deviates from the requirements of the problem. \nFrom this perspective, the authors' claim in this paper that \"EC should be used in principle over other metrics\" seems to be an overstatement.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper begins by highlighting the limitations of traditional metrics like accuracy, which assumes equal costs for all types of errors. It introduces the EC metric as a generalized form of error rate that considers varying costs for different types of errors. The authors emphasize that EC can be applied to any classification scenario where decisions incur specific costs based on the true class of the sample. The paper argues that, despite its theoretical robustness and flexibility, the EC metric is underutilized in the machine learning community, often being overlooked in favor of simpler, ad-hoc metrics.\n\nThe EC metric's primary strength lies in its ability to explicitly incorporate application-specific costs into the evaluation process, providing a more accurate reflection of the true performance of a classification system. This is particularly valuable in domains where some errors are more costly than others, such as in medical diagnostics or financial decision-making. The authors also introduce a normalized version of EC (NEC), which allows for easier interpretation by comparing the performance of a model against a naive baseline that does not consider the input data.\n\nThe paper evaluates the effectiveness of EC metrics in various scenarios by means of both theoretical arguments and empirical examples.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "EC itself is one of the well-known evaluation metrics, and naturally, there have already been various discussions about how to differentiate its use from other metrics. To make the authors' claim that \"EC is superior to other evaluation metrics and classification models should fundamentally be evaluated using EC\" more convincing, it is necessary to demonstrate that EC is clearly superior to other metrics from various perspectives. For example, \n- regarding the cost matrix described in the Weakness section, one approach might be to demonstrate that \"setting a cost matrix is easier than using other metrics\" or that \"EC has a certain degree of robustness against mis-specification of the cost matrix.\"\n\nIn order to unify the numerous evaluation metrics, shouldn't such things be at least experimentally verified?", "broader_impact_concerns": "No research ethics concerns were identified.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aav0WjIZ93", "reviewer": "Reviewer_V7Xp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n1. This work studies a significant problem and challenges the conventional wisdom regarding the utility of commonly used metrics in decision-making of model development.\n2. The preliminaries of the different classification metrics are well-detailed in the main paper.\n\n**Weakness**\n1. The empirical evaluation in Section 4 is difficult to follow. It didn't clearly present the evaluation method and clarify what constitutes positive or negative results. The writing in Sections 4.1 and 4.2 reads more like a rough description of the results without providing logical reasoning to explain \"highlight the different points we wish to discuss\" or \"illustrate how the wrong choice of metric can negatively affect development decisions\".\n2. The presentation of the results is unclear, making it challenging for readers to follow and be convinced by the findings. For instance, interpreting Table 2 is particularly difficult. It might be useful to use bold font type to emphasize which is a row/column that is worth attention.\n3. Although the introduction section claims a theoretical contribution, the paper does not present any theorems or lemmas to support its main claim.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper evaluates the conventional wisdom of classification metrics. It makes the claim that the expected cost (EC) is superior to other classification metrics, like the F-beta score and MCC. It supports the claim by providing the theoretical and empirical results, including demonstrating how it can make optimal decisions, and be more interpretable.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My main suggestions are around addressing the points listed in the weakness section. In addition to that, it could be beneficial for the paper to answer:\n\n1. Could the paper give some insights into the impact of miscalibration on the EC metric's performance? As presented in the paper, the effectiveness of the EC metric relies heavily on the assumption that the classifier’s output probabilities are perfectly calibrated. While the paper does acknowledge this and suggests post-hoc calibration, it could consider scenarios where calibration is not perfect.\n2. The paper emphasizes the flexibility of the EC metric in allowing different cost assignments for different types of errors. However, it does not provide concrete guidelines for practitioners to determine appropriate cost values for their specific applications.", "broader_impact_concerns": "There is no impact statement section in the paper and should be added.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "J2Q3eDtOZP", "reviewer": "Reviewer_BNMf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n* Putting a bunch of machine-learning metrics in a common decision-theoretic foundation is helpful for many reasons, including\n\n  1. it reveals the hidden cost function in classic metrics\n  2. It suggests other metrics that could be constructed using different choices of cost function and prior.\n  3. it reveals the class-imbalance \"problem\" to be self-inflicted\n\n* Actually doing the decision-theoretic analysis of these metrics reveals some surprising features of existing metrics, e.g. $F_{\\beta}$ and MCC have no corresponding decision theory (!)\n\n* this is IMO a pretty well-written introduction to decision theory, and in particular the important special case of discrete decisions.\n\nWeaknesses:\n\nThe framing seems odd to someone with my background, coming from statistics into ML, where we would assume *all* the metrics have a hidden decision-theoretic basis. Nonetheless, I acknowledge that this might not be part of the ML curriculum, and in fact my own prideful statisticians assumptions seem to have been faulty, since putting some classifiers into a utility-maximizing framework seems to be non-trivial, so putting this in the language of DeGroot or Savage is not so easy.\n\nAll that said, much of the paper repackages well-known results. In the ML literature — they mention Bishop 2006 and Hastie et al 2001 — which are part of the ML canon. I am aware also that Dyrland and co-authors have published some work which looks rather similar.\n\n* https://arxiv.org/abs/2302.10578 \n* https://arxiv.org/abs/2302.12006", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper seeks to unify some commonly used metrics for classification (precision, recall, Matthews Correlation Coefficient etc) into a single decision-theoretic framework. Along the way it provides a breif introduction to decision theory for the ML practitioner who uses classification metrics in their table of results without awareness of the foundations, including some worked examples.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would recommend that the authors check their work against the Dyrland references above (although since this is a resubmission, those articles may have been released while this article was under submission, so I do not wish to insist to firmly upon this, or the article might get stuck in \"revision hell.\")\n\nIt is possible that I am missing some minor errors or imperfections in this article, but nothing leaps out at me as a major blocker. I will revisit this if I notice anything in my next re-read of the article.", "broader_impact_concerns": "n.a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is some debate about whether the work is novel or just a survey of existing ideas. Although the paper builds on well-known theory, its main contribution is the clear framework it provides. The authors show that standard metrics hide important cost trade-ofs that can affect decision-making. This work is not just a review. The paper offers a practical guide on how to select and tune evaluation metrics. Overall, the paper gives useful insights that can help improve future classifier evaluations.
Decision: Accept (poster)