Evaluating Posterior Probabilities:  Decision Theory, Proper Scoring Rules, and Calibration
Authors: Luciana Ferrer
Daniel Ramos
Venue: Accepted by TMLR ()

Abstract: Most machine learning classifiers are designed to output posterior probabilities for the classes given the input sample. These probabilities may be used to make the categorical decision on the class of the sample; provided as input to a downstream system; or provided to a human for interpretation. Evaluating the quality of the posteriors generated by these system is an essential problem which was addressed decades ago with the invention of proper scoring rules (PSRs). Unfortunately, much of the recent machine learning literature uses calibration metrics---most commonly, the expected calibration error (ECE)---as a proxy to assess posterior performance. The problem with this approach is that calibration metrics reflect only one aspect of the quality of the posteriors, ignoring the discrimination performance. For this reason, we argue that calibration metrics should play no role in the assessment of posterior quality. Expected PSRs should instead be used for this job, preferably normalized for ease of interpretation. In this work, we first give a brief review of PSRs from a practical perspective, motivating their definition using Bayes decision theory. We discuss why expected PSRs provide a principled measure of the quality of a system's posteriors and why calibration metrics are not the right tool for this job. We argue that calibration metrics, while not useful for performance assessment, may be used as diagnostic tools during system development. With this purpose in mind, we discuss a simple and practical calibration metric, called calibration loss, derived from a decomposition of expected PSRs. We compare this metric with the ECE and with the expected score divergence calibration metric from the PSR literature and argue, using theoretical and empirical evidence, that calibration loss is superior to these two metrics.

Published in Transactions on Machine Learning Research (01/2025)
Evaluating Posterior Probabilities: Decision Theory, Proper Scoring Rules, and Calibration
Luciana Ferrer
fferrer@dc..uaa.o
Instituto Ciencias Computación Universidad de Buenas Aires CONICET, Argentina
Daniel Ramos
Dansel..amosSSaam.
AUDIAS Lab. Audio, Dala Intelligence and Speech
Escuela Politéenioa Superior, Universidad Autónoma Madrid, Spein
Reviewed on OpenReview: https: // openreview. net/ permmo id= qbrBOLR7fF
Abstract
Most machine learning classifiers are designed to output posterior probabilities for the classes given the input sample. These probabilities may be used to make the categorical decision on the class of the sample; provided as input to a downstream system; or provided to a human for interpretation. Evaluating the quality of the posteriors generated by these system is an essential problem which was addressed decades ago with the invention of proper scoring rules (PSRs). Unfortunately, much of the recent machine learning literature uses calibra- tion metrics most commonly, the expected calibration error (ECE) as proxy to assess posterior performance. The problem with this approach is that calibration metrics reflect only one aspect of the quality of the posteriors, ignoring the discrimination performance. For this reason, we argue that calibration metrics should play no role the assessment of posterior quality. Expected PSRs should instead be used for this job, preferably normalized for ease of interpretation. In this work, we first give a brief review of PSRs from a prac- tical perspective, motivating their definition using Bayes decision theory. We discuss why expected PSRs provide principled measure of the quality of system's posteriors and why calibration metrics are not the right tool for this job. We argue that calibration metrics, while not useful for performance assessment, may be used as diagnostic tools during system development. With this purpose in mind, we discuss a simple and practical calibration met- ric, called calibration loss, derived from a decomposition of expected PSRs. We compare this metric with the ECE and with the expected score divergence calibration metric from the PSR literature and argue, using thcoretical and empirical evidence, that calibration loss is superior to these two metrics.
1 Introduction
High-stakes machine learning applications, like those used to make health, military or legal decisions, often require systems that can provide a measure of uncertainty of the prediction given the input sample [Tomseu et al. 2020 Shinonero-Canderr
et al. 2005). classification tasks, the uncertainty is a property of the posterior probability for the classes given the input sample. We use the term probabilistic classifier to refer a classification system that outputs posterior probabílities. The posterior probabilities produced by a good probabilistic classifier be used to make decisions for a given cost function using Bayes decision

---

Published in Transactions on Machine Learning Research (01/2025)
theory (DeGroot 1970 Bernardo & Smith 1994 Jaynes 2003). They can also be readily interpreted by an end user or passed on to a downstream system.
Evaluating the quality of the posteriors produced by a classification system is not a trivial task since, unlike for the evaluation of categorical decisions for which class labels are used as ground truth, there are no ground-truth posteriors against which to compare the system-generated posteriors. Except in simulations, the true posterior distribution is not available to us. All we ever have are models trained on data. Every model we develop for a given problem provides us with new probabilistic classifier that does inference of the class of an input sample in the form of posterior distribution. The focus of this work is how assess the goodness of such models.
The conclusions in this paper are based on one fundamental premise: that classification decisions should be made rationally and that, in the face of uncertainty, rational decisions are supported by Bayes decision thoory (sce, for example, Good 1952; Poterson, 2009: Parmigiani & Inouc 2009 Savage 1972). Bayes
decision theory provides a procedure for making decisions based on given posterior distribution such that
the expectation of cost function of interest is minimized (Duda et al.) 2001 Bishop 2006 Hastie et al,
2001). Given this premise, principled way to assess the quality of posterior probabilities is through proper scoring rules (PSR), which are constructed as the cost that results from making Bayes decisions using the posteriors under evaluation. A PSR measures the quality of the class posterior distribution for one specific input sample. To obtain a metric to assess the quality of a probabilistic classifier's 's output we take the expectation of the PSR (EPSR) over the data (Brümner 2010 Filho et al. 2023). For example, the cross- entropy, widely used as objective function to train classification systems, is the expectation of the PSR given by the negative logarithmic loss.
A defining property of PSRs is that their expectation with respect to given reference probability distribution over the classes is minimized when the distribution under evaluation coincides with this reference distribution. Hence, a low PSR expectation indicates that the distribution under evaluation is close to the reference distribution with respect to which the expectation is taken. Note that we need to refer to a reference dist distribution ion since, as mentioned above, the true distribution is never available in practice.
The term PSR was first introduced by Winkler & Murphy 19688 who motivated their work by the need to assess the quality of weather forecasts. Brier score and the negative logarithmic loss (NLL), proposed in the 1950s, are special cases of PSRs (Brier) 1950 Good 1952). PSRs were later further studied in a large
number of works (see, for example, Dawid et al. 2016 Gneiting & Raftery 2007 Bröcker 2009
Ovcharov
2015 Gneiting 2011 among many others).
In contrast to this large body of literature on PSRs, many of the recent machine learning works concerned with the evaluation of posteriors do not use PSRs for the task, resorting instead to calibration metrics
(sce, for example, Guo et al. 2017: Widmann et al.
2019: Gruber & Buettner
2022 Nixon et al. 2019:
Popordanoska et al. 2022 V Vaicenavicius et al.) 2019
Van Hoorde et al. 2015; Huang et al. 2020 Müller
ct al.
2019
Mukhoti ct al.
2020 Minderer et al. 2021 Jiang et al., 2021 Desai & Durrett 2020 Dehghani
et al.
2023
A classification system is said to be well-calibrated if its output, q. coincides with a reference
posterior distribution for the classes given q, for every possible input sample, x (Bröcker) 2009 Guo et al.,
2017
Widmann et al.) 2019). As before, we need to refer to a reference posterior since the true posterior is not available in practice. The reference posterior needed to check calibration is yet another model of the posterior, one that we trust to be good.
Note that the calibration definition does not refer to posteriors for the classes given the input sample. Instead, it refers to posteriors for the classes given the system's 's output. Hence, good calibration does not imply that the system's posteriors are doing a good job of inferring the classes from the inputs, which is what matters in practice. In fact, a calibrated system can be useless. For example, a naive system that always outputs the prior probabilities of the classes is perfectly calibrated but does not provide any information about the input samples. As we will see, the overall quality of the system's posteriors, as measured by EPSRs, can be decomposed in two terms: calibration and discrimination refinement component (Filho et al., 2023; Bröcker ) The discrimination component quantifies the amount of information present in the system's 's output about the class of the samples while the calibration component quantifies the similarity between the
2

---

Published in Transactions on Machine Learning Research (01/2025)
system's output and reference distribution for the class given that output. Neither component by its own fully describes the goodness of the posteriors produced by the system.
As a consequence of the fact that calibration metrics only reflect one aspect of the posterions' performance, they do not inform how useful the classifier will be in practice. To address this problem, papers that use calibration metrics resort to a separate metric, usually accuracy, error rate, or area under the ROC curve, to complement the calibration metric (see, for example, Guo et al. 2017; Van Hoorde et al. 2015 Minderer et al. 2021; Mukhoti et al., 2020). This is problematic practice since it does not allow for direct comparison between two systems. If system's calibration error is lower but the error rate is higher than for another system, which of them is better? Which one should we choose for deployment if our main goal is to produce good posteriors for interpretation and decision-making? Also, this practice does not solve the problem of assessing whether the performance of the posteriors is good enough for a certain task. These questions are answered by EPSRs which provide comprehensive measure of the value provided by the system's 's posteriors. Using calibration metrics, on the other hand, leads to unneccssary conflict.
Papers that report calibration metrics motivate its use by arguing that good calibration is an essential characteristic for a probabilistic system to be interpretable, safe, or reliable (Guo et al.) 2017 Widmann
ct al. 2019 Mukhoti et al.) 2020; Kumar et al.) 2019 Minderer et al.) 2021; Nixon et al., 2019p. using statements like: "a network should provide a calibrated confidence" " (Guo et al. 2017), "miscalibration [--] makes [..] predictions hard to rely on" (Mukhoti et al. 2020), and "model calibration is essential for the safe application of neural networks" (Minderer et al. 2021). The implicit or explicit implication in those statements is that very discriminative system with relatively high calibration loss should not be used in high-stakes scenarios. We challenge this view and propose that calibration is neither necessary, nor sufficient for posterior probabilities to be useful to the end user. A system may be somewhat miscalibrated but still be useful, as long as the posteriors that it outputs result in sufficiently good Bayes decisions. In particular, a miscalibrated system may be much more useful than the perfectly-calibrateed calibrated naive system mentioned above. If a system's EPSR is lower than the EPSR of an alternative system, we can conclude that its postcriors are better, regardless of the calibration error of either system. Hence, when evaluating the utility of posteriors we should not be concerned with whether they are calibrated or not. Assessing the quality of probabilistic classifiers is exactly the purpose of EPSRs. When the goal is to assess the value of an individual classifier compare classifiers with each other to select the best one, there is no need to resort to the concept of calibration.
In this paper, we argue that the only purpose of calibration metrics should be to diagnose whether a system is well-calibrated in order to fix if that is not the case, much like learning curves over validation and training data are used to diagnose overfitting. While regularization, early stopping, or smaller models may be explored when overfitting is detected, various approaches can be explored when miscalibration is detected. In particular, a miscalibrated system can be very casily improved by adding a post-hoe calibration stage (Filho et al., 2023): : transformation of the system output designed to reduce the calibration error. Such a stage, if successful, would result in a new system with lower EPSR, which is our final goal. Beyond the use as a diagnostic tool during development, calibration metrics should not play any role in system evaluation since they do not provide any particular insight about the value of a system for the end user. If changing the system is not an option, assessing and reporting calibration performance has no practical role.
For the calibration analysis needed during system development, we propose use the calibration loss metric. Calibration loss is obtained from a decomposition of an EPSR into calibration and discrimination terms, and directly reflects the improvement A particular version of this metric was introduced in the literature years ago for the binary task of speaker verification (Brümmer & du Preez 2006) and later adopted in forensic science (Ramos& & Gonzalez-Rodriguess Rodriguez 2013 Ramos et al. 2017 2020). We compare it, theoretically and empirically, with the widely-used expected calibration error (ECE) metric (Naeini et al. 2015a Guo et al. 2017), ,,, and argue that ECE has no theoretical or practical advantage over calibration loss having, on the other hand, various disadvantages.
would obtain if post-hoc calibration stage was added to the system.
The rest of this paper gives an introduction to Bayes decision theory and PSRs, highlighting their tight relationship. Then, it describes the problem of calibration, introduces the calibration loss metric, and compares it with the ECE and the expected score divergence, a classic calibration metric from the statistics
3

---

Published in Transactions on Machine Learning Research (01/2025)
literature, providing novel insights and discussing the advantages of the calibration loss over those classic alternatives. Finally, it. presents empirical results on synthetic and real datasets to demonstrate and further discuss the theoretical ideas in the previous sections. A substantial part of the content in this paper revisits well-ostablished concepts, some dating back decades. Yet, we believe a discussion of these ideas is still needed in the machine learning community, given the current wide-spread use of calibration metrics as a proxy to assess the quality of a system's posteriors.
In summary, the goals of this work are to argue: 1) that calibration metrics should only be used during system development, for example, for the purpose of deciding whether post-hoe calibration stage is needed in the system, 2) that for those cases, calibration loss, simple and principled calibration metric, is preferable to the ECE and to the expected score divergence, 3) that the goodness of a probabilistic system, as will be perceived by an end user, a downstream system, or a decision stage, should be assessed using EPSRs, not calibration metrics.
2 From Bayes decision theory to calibration
The goal of this section is to review known concepts from the statistical learning literature and discuss them in the light of current. trends in the machine learning literature, where the quality of posterior probabilities is most often assessed using the ECE or one of its variants. First, we discuss the reference distribution, a necessary construct when dealing with real data where the underlying data-generating distribution is not known. Then, describe how to make optimal (Bayes) decisions for a given cost function selected for the application of interest. We then define PSRs as the cost of Bayes decisions motivating their use as metrics to evaluate the quality of posteriors. We also explain how they can be normalized to obtain interpretable metrics. Further, we explain how EPSRs can be understood as integrals over a family of Bayes risks, providing further intuition on why and how EPSRs reflect the quality of posteriors. Finally, we show how to dccompose an EPSR into calibration and discrimination components in two ways, using the traditional divergence-entropy decomposition, and using the calibration loss decomposition, and compare the resulting metrics with the widely used ECE metric.
While most of the content in this section is based on decades-old concepts from the statistical literature, our treatment is different from that in most works in that the need for a reference distribution is made explicit in every step of the way. As we will see, this uncovers some practical issues that are usually not considered in the literature.
2.1 The reference distribution
In statistics and machine learning literature, it common practice to explicilly or implicitly rely on 'the true distribution of the data'. We find this an ill-defined concept, which is not useful in practice and which, in particular, poses formidable obstacles to understanding calibration. Even if one could theoretically consider that a given dataset was created by sampling from true underlying probability distribution, such distribution will never be known to us except in simulations.
In machine learning, though, both at training time and test time, it is very useful to work with probabil- ity distributions for the data. By 'the data' we refer to some future, or otherwise unseen data, and by ddittiibutions for the data' we refer to probabilistic predictions of the unseen data. For example, if you are training a classifier by minimizing the cross-entropy over a supervised training dataset, you are effectively minimizing prediction of the cross-entropy on future, unseen data. If you are testing your classifier with error-rate or cross- cross-entrowd entropy, you are also effectively predicting how it will perform future data, We can obtain the probabilistic predictions of the data, which we will call reference distributions, by selecting mod- eling assumptions and then fitting the model to a given dataset. Different assumptions will lead to different reference distributions.
We follow the advice by Jaynes 20033 of using the preposition for, rather than to refer to the relationships between distributions and data. Distributions that we choose use to model some unseen data are not an intrinsic property of the unseen data, rather they are induced by assumptions and given data.
4

---

Published in Transactions on Machine Learning Research (01/2025)
The reference distributions can be very simple. Perhaps the simplest reference distribution is the empirical one, obtained by uniformly sampling with replacement from dataset. Expectations with respect to the empirical distribution are given by averages over the samples in the dataset. The empirical distribution is the most common reference distribution used in machine learning both for training and testing models, where cross-entropy or error rales are computed as averages over the training or test samples. As we will see, though, it turns out that empirical distributions do not provide useful references against which to judge calibration if we make use of the classical definition of calibration. It is therefore necessary for our exploration and understanding of calibration to consider more general reference distributions than the empirical ones. In the theoretical sections below we will simply leave the reference undefined, assuming enough regularity conditions are imposed in the model to result in a good predictor of future data. We will discuss a simple way to build such reference distributions in Section 3.4
2.2 Bayes decision theory
Assume that we have selected a cost function for our classification problem of interest, C(h, d), where h E H = (H!,.... Hk} is the true class the sample, d E D the decision made by the system, and H:HxD~ + R. Decisions can be categorical, in which case we take D {D.....; DD.) )- The set of decisions and the set of classes do not need to be the same. The decisions are, in general, the actions that will be taken based on the system's 's output (Duda et al. 2001p. For example, D could include "reject" "abstain" option (see, for example, [ishop 2006, section 1.5). Decisions can also be soft, in the form of distribution over classes, in which case D = S* the simplex where K-class categorical distributions live.
Given the cost function, our goal will be to make optimal decisions in the sense that they minimize the expec- tation of this cost (Peterson 2009), (M)2) (Chh, d(z))), where we have explicitly added the dependency of on the input sample, x. The expectation is taken with respect P. [x).. a reference distribution for
the class label given the input. This expectation is minimized by taking Bishop 2006 Hastie et al. 2007
d(x) dz(x)= = arg min
Σ ((HH.)))
(1)
d
The decision dn is called the Bayes decision. If decisions are made in this way for every x, then the expected cost with respect to the joint distribution (r æ) is also minimized.
2.3 Proper scoring rules
Proper scoring rules (PSRs) are family of functions specifically designed to assess the quality of posterior
probabilities (Gneiting & Raftery] 2007 Brümmer [2010]. The principle behind PSRs is that the quality of posteriors is given by the quality of the Bayes decisions made with them: better decisions imply better posteriors. Formally, given a posterior for a sample provided by a classifier, which we denote q(x) Po(1 and cost function C(h, d), we can construct PSR, (hh(,, q), as follows (Dawid Musio 2014 Brümmer 2010
C(h,)da)(
That is, C* is the cost of the Bayes decision made with q. Note that here we have expressed dy as function Bayes decisions only depend on the posterior which,
of q rather than x since, as shown in Equation ① here, is given by q-
The C* constructed in this way satisfics the defining property of PSRs which is that their expected value with respect any distribution p over the classes minimized if q coincides with p (Dawid & Musio 2014 Brümmer 2 That is,
p arg min Enup [C* 9h)
"There may sometimes be more than one minimizing decision. In such cases we assume arg min applies a tie- breaker to return single minimizing decision. The details of such tie breakers is unimportant here. WW use hold q for vectors in gK representing discrete distributions, and italic with a subindex qi refer to the ith component of the q vector.
5

---

Published in Transactions on Machine Learning Research (01/2025)
When the minimum is unique, the PSR is called strict.
A PSR measures the quality of the posterior vector q for single sample. To obtain a metric that can be used for evaluation, we can compute the expectation of the PSR with respect to the joint reference distribution:
EPSR = Ep.(b.s) (c))))
(4)
Now, since C* is function of x only through 9, we can invoke the law of the unconscious statistician (DeGroot & Schervish 2014 page 213) and compute the expectation with respect to q instead:
EPSR (E. [C (h,
This equation shows that we do not need a reference distribution for (h, to compute the EPSR. Instead, we only need a reference for q).
In practice, to compute this metric, we take the reference distribution to be the empirical distribution in the test dataset so that the EPSR is given by
N Σ C (h),
1
EPSR
qt).
(6)
N
where qu and h, are the classifier's output and the label for sample in test dataset containing N samples. Different EPSRs can be obtained by using different cost functions C. Below we will discuss three of the most common EPSRs.
2.3.1 Bayes risk
When the decision d is categorical (d (Di...., Dart), the cost function can be expressed as matrix of costs Cjj for each combination of true class H; and decision Dj- The expected cost with respect to the empirical distribution in test dataset is given by the average cost over the samples:
K M EC Σ
K M Cgp P.R.
where P. = N./N is the empirical prior probability of class H;, N; is the number of samples of class Hi, and Riz- = Ny,NN, is the fraction of samples from class H, for which the system made decision Dy. The EC can be normalized to case interpretation by dividing it by the EC of a system that outputs always the least-costly decision, which given by mins Σx= CaPi A special case of the EC is obtained for the 0-1 cost matrix where Gjj when i A (the decision incorrect), and Gj = 0 when i == j (the decision is correct). The average cost in this case reduces to the standard error rate, which equal to one minus the accuracy.
The EC can be computed regardless of how decisions are made. If decisions made using Bayes decision theory, though, the resulting F.C is an EPSR commonly called Bayes risk (Duda et al., (001).. Given a cost matrix, Equation 4 corresponds to a specific partition of the simplex S into decision regions. For binary classification with a square cost matrix (M = 2), the regions can be determined by a threshold on either of the two posteriors. For the 0-1 cost, the Bayes decision the class with the maximum posterior (Hastic et al., 2001), or argmax docision. Hence, the crror rate of argmax docisions, widcly-used metric in the machine learning classification literature, is one instance of the Bayes risk. The left plot in Figure ] illustrates cost function for binary classification when the Bayes threshold is used.
When evaluating Bayes risk, we are, as for any EPSR, mcasuring the quality of the posteriors used to make the Bayes decisions. Yet, we are restricting our assessment of the posteriors to one specific operating point defined by the cost matrix. As a result of this, the Bayes risk is not a strict PSR: Two classifiers that produce posterior distributions for which the Bayes decisions for the selected cost matrix are the same, will lead the same Bayes risk. The Bayes risk for other cost matrices, though, will not necessarily be the same for both classifiers, since their posteriors are different.
6

---

Published in Transactions on Machine Learning Research (01/2025)
2.0
0.5 -
4.0
Cost C12 G3
0.4
1.5
Log Loss
3.0
Brier Loss
0.3
h = H1
1.0
2.0
0.2
Hz
0.5
1.0
0.1
0.0
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
42
Q2
Q2
Figure Three PSRs on a binary classification task as a function the posterior for class H2, 92: (left) the cost function with 61 1 and C21 2 for the Bayes threshold (given by (12((112 C21)), (center) the logarithmic loss, and (right) the Brier loss. The blue curve corresponds the loss for samples of class Hi, which is maximum when 92 (conversely for the orange curve). The logarithmic loss goes to infinity in the extremes in which the posterior is for the wrong class, while the other two losses stay bounded. The expectation of these PSRs corresponds to the Bayes risk, the Cross-entropy, and the Brier score, respectively.
2.3.2 Cross-entropy
When we want to evaluate the quality of posteriors general rather than for a single operating point, we need to resort to strict. PSRs. One such PSR can be obtained by setting loss (NLL), C(h, d)
be the negative logarithric -og((d)) where d SK and, in slight abuse of notation, dp is the element of d corresponding to class h (if h = H;, then d, is the ith element of vector d, also denoted d,). Tt can be shown that, for this cost function, the Bayes decision (Equation is d = q, since, by Gibbs inequality, E. C,H,, d) = E, log (d)) 2 - E, log (g:) q: That is, the (soft) decision that minimizes the expected NLL is q itself. Hence, the PSR resulting from this cost function is
log gb. (Equation 2), which is again the NLL. It is easy to show that the NLL is a strict PSR (Dawid & Musio 2014). The middle plot in Figure illustrates the logarithmic loss for binary classification.
The expectation of the NLL over the data the cross-entrowd entsopy, which widely used as objective function for training. Taking the expectation with respect to the empirical distribution, get
N
CF
log(gn, (x:)),
where qh, (2x) is the system's output for sample D; for class h4, the true class of sample t. This expression can be rewritten lo make ils dependence on the class priors explicit (see Appendix AA, which allows us manipulate these priors independently of those in the test data. This is useful when the priors in our test data do not reflect those we expect see when the system is deployed, in which case we can use the target priors instead of the ones in data when computing the CE.
The absolute CE values are not casily interpretable. This issuc, though, can be solved by normalizing its value with the CE of the best naive system. A naive system is one that does not have access to the input data. The best naive system for any EPSR is the one that outputs the prior distribution in the test data [Brümmer] 2010, Section 2.4.1). The CE of a system that always outputs the prior distribution is the entropy of such distribution, - >K,, P. log(P.). Dividing the CE by this value we obtain the normalized CE, or NCE for short. The NCE values can be readily interpreted: values above 1.0 mean that the system is worse than the best naive system and, hence, one should cither 1) throw away and replace with a system that outputs the prior distribution, 2) fix by doing calibration (as we will see Section 2.4.2 A well-calibrated system will never have normalized CE larger than one.
7

---

Published in Transactions on Machine Learning Research (01/2025)
2.3.3 Brier score
If we take C(h, d) Ei I(h H,)), with d E SK we get another PSR called Brier loss. Here, T(h= Hi) the indicator function which can be shown that the Bayes decision for this loss the posterior used make the decision. Hence, for the NLL, the PSR coincides with the cost function. The right plot in Figure Iillutrtiss the behavior of the Brier loss for binary classification.
the argument true and 0 otherwise. As for the NLL,
Averaging the Brier loss over the data we get the Brier score
BS
(g.(2.)) 1(he= 11)))
(9)
N
As for the CE, this expression can be manipulated to show its dependency with the priors explicitly which allows us to turn the priors into parameters of the metric (see Appendix Ap. Further, as for CE, it can be shown that BS is a strict PSR (Dawid & Musio 2014). Unlike the CF, though, which can be infinite if the posterior for the right class 0.0 for any sample in the dataset, BS is bounded, being more forgiving of extremely incorrect posteriors (sce Figure D. Finally, we can compute normalized version of BS, NBS, by dividing its value by the BS of the best naive system, which given by Σ Pt(1 - P.)/K.
2.3.4 EPSRs as integrals over Bayes risks
As explained, for example, by Gueiting & Raftery (0077 and by Brümmer 2010 can construct PSR, Cip, by taking the following integral:
/
Cw (h1
((a) Ca q) da SK
(10)
where h is the class of the sample, q S is a posterior distribution, and ((a) is function of a E S that satisfies ((a) 2 va. The C function inside that integral is the PSR obtained explained in Section
2.3.1 for the following cost function:
1- 1(i=)) Ca(i.)) (N 1) a
(11)
where ji, is the cost for deciding Dj when the true class of the sample is H, For this construction, we take the set of decisions to be the same as the set of classes. This matrix has zeroes in the diagonal and a cost proportional to 1/a; everywhere else. Taking the expectation with respect to the data on both sides of Equation 10 we get that
EPSR
/ W(a) Risk(a) da
where
Risk(a) E [C] (h, q))
(12)
The function (a)) determines how much each Bayes risk inside the integral influences the final value. Taking ((a) to be uniform in the simplex, obtain the CE. For the binary case, taking W(a) to be Beta distribution with both parameters equal to 2, that W(a) a1 (1 -aaa)aaa az)/B((,2)) where B is the beta function, we obtain the BS. The proofs for these statements can be found in (Brümmer 2010 section 7.4). Section 3.1 shows an illustration of this way of constructing the CE and BS metrics.
2.4 Calibration
Following the literature (for example, Bröcker 2009 Guo al. 2017 Vaicenavicius et al. 2019 Nixon et al. 2019 Gruber & Buettner 2022), classifier, Pe, that for input outpuls the class posterior, q 0 [K
where = Po(H; I), has perfect calibration with respect to a reference distribution P,, if:
qi P.( q), Vi,
(13)
8

---

Published in Transactions on Machine Learning Research (01/2025)
This notion of calibration is sometimes called canonical, distribution strong calibration (Widmann et al, 2019 Gopalan et al. 2024 Popordanoska et al. 2022 Weaker definitions of calibration have been proposed, like class-wise and confidence calibration şKull et al. 2019), which refer to a single value from the posterior distribution: the value for each of the classes, or the largest value (usually called confidence), respectively. These weaker notions were proposed because they are easier to verify empirically, since the conditioning on a single value instead of the full posterior vector simplifies the problem. An example of this will be discussed Section 2.5.2 on the expected calibration loss, a metric which, for the multi-class case, evaluates confidence calibration. Unfortunately, these weaker notions of calibration, while easier to check, often very poor proxies for the strong notion, as discussed in prior works (Nixon et al. 2019 Popordanoska et al. 2022 In this work, we focus on the strong notion of calibration which, as explained in Section 2.4.1 guarantees the optimality of Bayes decisions for the given classifier. We believe weaker notions are unnecessary since, as we argue extensively throughout this work, the goal of assessing calibration should only be to decide whether adding a calibration stage would improve the quality of the posteriors and this can be done using the calibration loss metric without the need to weaken the definition. The complexity in measuring calibration is not in the definition but rather in determining the reference distribution.
In most of the literature cited above, the rolc of P. is implicitly deferred to the ill-defined concept of the 'true distribution'. With the reference made explicit, we note that classifier may be calibrated with respect to some reference distribution, but miscalibrated with respect to another. In this section we carefully analyze this definition of perfect calibration and also compare it to the optimal classifier deferring the discussion of practical ways to define that reference to Section 2.4.2
Given a reference distribution for h and x, Pr(,,r), the optimal classifier is the that outputs the class posterior, p (p....., pP] = P.(. x) with components:
= (H)))) ΣK (H),
(14)
Note that this distribution not the reference for perfect calibration of the classifier: the right hand side Equation 13 is P. q), not P.(. x). In other words, perfect calibration does not ensure that q is the
optimal classifie- it is a weaker condition that can hold for q # p, that is, for suboptimal classifier.
Before moving on to analyzing the calibration of q, we need to understand the implicit perfect calibration of the optimal classifier, Since is a function of it is also a random variable for which the joint, marginal and conditional distributions can be derived from (r z). The perfect calibration of p is defined as p; (H,, p). This equality always holds p is the optimal classifier given by Equation
This can
be seen as follows +
1
P.HH,,))
Pr(Hi, σ) dē 12:
-J
P. | (t)
J2.P-(1B=-
(15)
= Pi
Pr(T) di
PP.PP
where we used that P(H, @) is conslant with value Pt = (HH | x)), for the values of & over which we are integrating. Rearranging, we find:
(H p) P, p)
Px(p) = P. = P.
(16)
This result can be summarized succinctly as:
P. p) = = p. P(=
(17)
'A measure-theoretic (terse, perhaps less accessible) derivation given in the appendix Bröcker 2009
This derivation,
proposed by Niko Brümmer during personal communications with the authors, more similar derivation of likellhood-ratio calibration, in the appendix of (Sooten & Meester (2012).
9

---

Published in Transactions on Machine Learning Research (01/2025)
Pal:1x)
Po(. q) =
Our classifier
Calibrator
x
P+ x) =
Optinal classifier
Figure 2: Diagram of the posteriors in Section 2.4 The posterior q is the output of our classifier. After calibration according to the reference distribution (.(h, x), we obtain the posterior s. If our classifier is perfectly calibrated with respect to the reference, then s = q. On the other hand, the optimal classifier under the reference distribution P. (h,, æ) is the one that produces p. These the posteriors idcally want to have. In general, though, q is different from p. Further, perfect calibration does not guarantee that is equal to p.
This is intuitive: We have already inferred h from x in the form of p, so that if we want to infer h directly from p, without any extra information, the result remains the same.
We have shown that P. the optimal classifier for a reference x)) is perfectly calibrated with respect to the posterior distribution consistent with that reference. Now, recall that our classifier is the model Pe: that outputs the posterior q Po(- I)), with components qu Po(H; x). In general, our classifier will be different from the optimal classifier:
q Pc P.(.
(18)
Here and elsewhere, by ", we mean not equal in general. In contrast to P- x)) which provides the full joint dist distribution, all we need to practically implement a classifier is the conditional, Pa(h æ). We can however allow the thought experiment to extend this model to Po(h, = P.(x) x). Then Equation that the classifier is also perfectly calibrated- we use Pa itsolf as reference:
15
shows
Pc 9) q Pc(. 1x)
(19)
What we want to do however, is to judge the calibration of q with respect to P. instead. Again, since q is a function of x the model P. provides also the posterior P²(. q), which we will call s, against which q may be judged [ We have now defined total of three different posteriors, q s + p: our classifier, the class posterior given our classifier's 's output, and the optimal classifier, the latter two derived from the reference distribution. Their definitions and relationships may be summarized as (see also Figure
q q= Po( 1x) = Po( q) fs P.C. ()) + P.( = 1,(-1x)
(20)
The first. 't' is simply because P. * P,,. The second 'f' follows from the data processing inequality 6 we assume the function from I to q is non-invertible. As noted above, the litcrature defines perfect calibration as the special case when q = (. | q), that is, when the first 'f' in Equation (20) is replaced with equality. Even then, we still have the second " which highlights the fact that a perfectly calibrated classifier is not necessarily optimal.
The classifier Pa is optimal with respect to the reference, if P.(: [x)= q - P.(. x). But this is much stronger requirement than merely having perfect calibration, as defined by Equation [3). While = p for all æ implics perfect calibration by Equation 17 the converse is not true. An extreme counterexample is the naive classifier, say Pe = Po, that completely ignores its input, while having perfect calibration:
a Po(H; x)) (H.))) P. (H !))
(21)
where the first two equalities define Po: it outputs the class prior as given by the reference Pr, irrespective of the value of z. The last equality, which shows that Po has perfect calibration, follows because q is constant.
3 (P q) Pz(h.q) Sa.P, (h) dē.
With P, reference, the data processing incquality states that I(h;z)= I(h; > g)): where denotes mutual information (Cverr Thomas 2006). While p contains all of the information about that is present in 2, q generally contains less. The non-nvertible function + p also subject to the data processing inequality, but this special case, equality is given by Equation [5)
10

---

Published in Transactions on Machine Learning Research (01/2025)
This is just an example of perfectly-calibrated but useless classifier that highlights the problem with using calibration metrics to assess performance of probabilistic classifiers.
To understand why calibration metrics should not be used to assess posterior performance, we may also consider an analogy between a system's calibration quality and the size of its training set. These two aspects are analogous the sense that both affect but do not reflect the quality of the posteriors produced by the system. Specifically, a system may be trained with a small amount of data or be miscalibrated and still produce good posteriors. Conversely, system may be trained with large amount of data and be perfectly calibrated and still produce poor posteriors. Further, if we increase the size of the training set for given system, the quality of the system will usually improve or, at least, not degrade. Similarly, if we calibrate a given system (as explained in Section 2.4.2). it will usually improve or, at least, not degrade. Finally, given two different systems, the size of their training sets or its calibration quality are not reliable indicators of which system has better posteriors. One system may be trained with more data or be better calibrated than another but produce poorer posteriors than the other. So, just as we would not use the size of the training set as measure of the quality of a system's posteriors, we should not use calibration metrics for this purpose either. Instead, EPSRs are the right tool for that job.
2.4.1 Calibration and Bayes decisions
Most machine learning classifiers make their final decisions based only on the classifier output, q. This means that the x in Equation (1 is given by q, since those are the input features available for decision making. Replacing x with q in Equation [ we get that the Bayes decisions based on q are given by
K
(g(4) = arg min
d(H,,))
(22)
Those decisions minimize the expected cost with respect to that same reference distribution, Pr.
According to the definition of calibration given by Equation (13 system is perfectly calibrated with respect to P, if its ouput q satisfies g (H, q) This means that, for perfectly calibrated system, we can simply plug in qi in place of the postcrior in Equation (22) to get optimal decisions. In other words, Bayes decisions made with posteriors provided by perfectly calibrated system are the best possible decisions for the given system, as long as both calibration and optimality of the decisions are judged with respect to the same distribution, P,. No other strategy for making decisions based on the system's 's output would result in a better expected cost. On the other hand, we had access to the system's 's input features, there could very well exist a better decision strategy. A calibrated system does not guarantee that the Bayes decisions are the best that can be made with our system's input, x. Rather, guarantees that they are the best that can be made with our system's output, q. If the system is poor, the decisions will be poor, regardless of how well calibrated it may be.
2.4.2 Calibration transformations
13 which we have called 8; (Equation 20 can be interpreted as a The posterior PrH | q) in Equation
transformation of the classifier's posterior q = [P(([] | x), into a posterior that is perfectly calibrated with respect to P,. To see this, we can use the proof in Equation (15,, substituting p - s and x - q, to find that (H, s) = 8g. That is, s is perfectly calibrated. For this reason, s = P. q) is called calibration transformation. This transformation needs be learned from data.
For binary classification tasks, the calibration transformation can be constructed by collecting pairs q. h for several samples from the task. Noting that q [/1 1 - 91.. we can quantize yi into, say, 10 bins and compute the frequency of each class h on each bin to obtain (h q). As we will see, this is the basis for the computation of the ECE metric. Quantizing and counting, though, is often problematic, as we will discuss later this work, and is not appropriate for multi-class problems. Fortunately, a large variety of calibration transformations that do not rely on quantization have been proposed in the literature over the last decades. A review of these approaches can be found, for example, in the work by Filho et al. 2023
11

---

Published in Transactions on Machine Learning Research (01/2025)
One of the most standard calibration approaches consists of applying an affine transformation to the loga- rithm of the posterior vector, training the parameters of this transformation to minimize the cross-entropy. Since the cross-entropy is a striet EPSR, minimizing this loss guides the parameters of the affine transforma- tion toward values that produce good posteriors. Instances of this approach are linear logistic regression, also known as Platt scaling for binary classification (Plati 2000), an extension of Platt scaling for the multi-class case called
(DP) transformation (Brümmer & van Leeuwen, 2006), and temperature scaling (Guo et a 2017). In our experiments, we will use the DP transformation where the transformed posterior s = R(q) is given by
softmax(a log(g) + (3)
(23)
where q, S, and are vectors of dimension K, the mumber of classes, and a > 0 is a scalar. When taking B = 0, this transformation reduces to temperature scaling M
For the binary classification case, it is possible to obtain a monotonic transformation of one of the two posterior probabilities output by the system that leads to the lowest EPSR value on the test data itself. This transformation can be obtained with the pool-adjacen--violtoor (PAV) algorithm (Ayer et al.) 1955). The EPSR resulting after this transformation can be seen as the minimum EPSR possible on that test dataset that does not change the ranking of the posteriors. Interestingly, the transformation simultancously optimal for all EPSRs regardless of which one is used to obtain it dAhuja & Orlin 1998 Brümmer 2010). This minimum is likely not achievable any other dataset, though, since the transform is non-parametric and overfits easily.
To train any calibration transform, a dataset of q values obtained by rumning the classifier on a set of samples needs to be created. For supervised approaches like those mentioned above, the class, h, for each sample is also required. It is important to note that these samples should not be extracted from the dataset used to train the classifier we are aiming to calibrate. The posteriors that the classifiers produce on their training samples are, for most modern models, already well-calibrated. Yet., unless no degree of overfitting occurred during training, the distribution of those posteriors is not representative of the distribution which is where we wish the calibration transform to perform well. Hence, the data used to train the classifier should not be used to train the calibration model. Ideally, a fraction of the available training data should be held-out when training the classifier to be used as training data for the calibration model. It is particularly important that this data is representative of the one we expect to see when the system is deployed since calibration performance appears to be more sensitive to domain mismatch than discrimination performance Ferrrrr et al. 2021.
unseen data,
2.5 Calibration metrics
Equation 13 gives the definition of perfect calibration with respect to reference distribution. It does not provide a calibration metric, that is, a way to measure a degree of (imperfect) calibration. In this section we describe three calibration metrics: the expected divergence, commonly studied in theoretical papers; the expected calibration error, commonly used in empirical machine learning papers; and the calibration loss, our proposed metric, which we believe has various advantages and no disadvantages over both of those well-established metrics.
2.5.1 Expected score divergence
Much of the literature on calibration solves the problem of mcasuring the degree of imperfect calibration by defining a divergence between the left-hand side and the right-hand side in Equation (13),, where the left-hand side is given by the output of the system under evaluation, q, and the hand side is given by the reference posterior, s. A permissive definition of a divergence is a non-negative function d(s, q) that is zero if s = q. strict divergence is zero only at s = q and positive, or infinite othrwwise. Then, a class of metrics that evaluate the goodness of the calibration of a probabilistic classifier Pa: relative to reference
Temperature scaling, defined by Guo al. 2017), takes the pre-softmax activations of neural network input, instcad of the log posterior as in Equation Yet, both cases the final calibrated output transformed by the softmax function, making both expressions equivalent.
12

---

Published in Transactions on Machine Learning Research (01/2025)
model Pr., can be defined as the expectation of the divergence with respect to the reference distribution BBrcker 2009
D(P,, = (g) [d(R(q): q)].
(24)
where we defined R(q)= s, making explicit the fact that s is function of q.
Interestingly, it can be shown that EPSRs (Equation 5 can be decomposed into an expected divergence and
generalized entropy term (DeGroot & Fienberg 1983 Bröcker
2009
Brümmer 2010 Gneiting & Raftery
2007
Dawid 2007):
(p (C*(h, = (p.(()) [d(R(q): + Bp.(h.s) (h,,)
(25)
where
d(s. q) = Enow [C* -r (h,
(26)
which satisfies the conditions listed above. That is, it is zero if s q. and non-negative otherwise. This last property derives directly from the defining property of PSRs, Equation (3) When the PSR is strict, the divergence is zero if and only if = q. A divergence that is induced by a PSR as in Equation (26) is called a score divergence (Ovcharov) 2015). The score divergence corresponding to the Brier loss is the squared Euclidean distance, [, 4), and the one corresponding to the logarithmic loss is the Kullback-Leibler divergence, E; 8t log():/4:).
25 is the EPSR of which, for the logarithmic loss, is the entropy of this distribution. This term mcasures the discrimination refinement of q: it is the EPSR that is obtained after solving (as best as we could) any miscalibration problem by adding the calibration transform s to the system. Hence, Equation 225 expresses the overall performance of the system, given by the EPSR on the left-hand side, as a sum of a calibration and a discrimination torm.
The right-most term in Equation
Note that all terms in the decomposition depend on Pr, a model that needs to be learned from data. Different models will lead to different values for all three terms. While this may be unsettling, it is an issue inherent to the fact that the true distribution is not known in practice and, hence, it applics to all metrics designed to assess the performance of posterior probabilities.
Further, note that, in order for Equation (25) to hold, the (h q) used to obtain the expectation in the left- hand side should be consistent with s. That is, it should satisfy P. = (g)ss Hence, the left-hand side, which corresponds to the overall performance of the classifier, depends on the output of a calibration model, s. If this model poor, the assessment of the system performance may be suboptimal (examples of this problem are given in Section 3.4). This makes this classic decomposition from the theoretical literature calibration particularly problematic in practice.
In contrast, the common practice in the machine learning literature is to compute the EPSR in the left-hand side of Equation 25 by setting P, q) to be the empirical distribution on the test data, so that the EPSR is given by the average PSR over the data. Unfortunately, doing this in Equation (55 leads to a meaningless decomposition. The posterior s derived from the empirical distribution consists of a one-hot vector with a one at the index corresponding to the sample's true class for the q values obtained on the datasct and undefined values for any other value of q. This will, for most systems, have an entropy term of (unless two samples from different classes have exactly the same q value) suggesting that the system has perfect discrimination. Yet, such s would certainly not result in zero entropy on any other dataset.
In order for the decomposition in Equation 25 to be meaningful, needs to be good predictive model. That is, a model that generalizes well to unseen data. The empirical distribution does not satisfy this condition and, hence, cannot be used for the decomposition. Perhaps for this reason, the divergence/entte decomposition of EPSRs not commonly used in empirical papers. As we will see in Section 2.5.3 our proposed calibration loss metric solves this problem.
2.5.2 ECE: Expected calibration error
The most common calibration metric in current machine learning literature is the expected calibration error (FCF) (for example, Guo et. al. 2017 Liang et. al. 2023 Müller et al., 2019 Ovadia et al. 2019). which,
13

---

Published in Transactions on Machine Learning Research (01/2025)
will see, is variant of the expected divergence described above. This metric was proposed by Naeini et al. (2015a) as a metric to assess the calibration quality of binary classification systems. To compute the ECE, the posteriors for class H2 for all the samples in the test dataset are binned into M bins. The ECE is
then computed as (Guo et al. 2017h:
M Σ
Bml N
ECE
[class2(Bm) - avep2((m))
(27)
where Bm is the set of samples for which the posterior for class H2 is in the mth bin, |Bmll is the number of samples in that bin, class2(Bn, (m)) is the fraction of samples of class H2 within that bin, and avep2(Bm)) is the average posterior for class H2 for the samples in that bin.
The expression above can be rewritten to look like the divergence in Equation 24). with the expectation taken with respect to the empirical distribution of q, by replacing |Bml with a sum over all samples in that bin:
1 N ECE = N Σ ((()))),(())))
(28)
where is such that (2(x)) avm))(Bm.), and s is such that s2 = class2(Bm.). ) where Bm, is the bin corresponding to (2(2z), the output of the system for class Hz. The defined this way corresponds to a very common calibration method called histogram binning (Zadrozny & Elkan 2001 Nacini et al. 2015b). To obtain the ECE defined by Equaaionn (27), the distance d should be defined as d(s, = 82 O2|- Unfortunately, the absolute distance is not a score divergence, that is, it is not induced by any PSR. To see this, we can use the fact that all score divergences are Bregman divergences and Bregman divergences satisfy the following property [Ovcharov] 2015):
E [s] = arg min E [d(s; qo)]
(29)
where qo is a fixed vector. That is, the divergence between s and a fixed posterior vector is minimized when that vector is equal to the expectation of s. It is easy to find counterexamples (see Appendix B where this property is not satisfied for the absolute distance and, hence, we can conclude that the absolute distance is not induced by any PSR. As a consequence, the standard form of the ECE. cannot be decomposed as in Equation 25
The histogram binning calibration transformation used to compute the ECE is, in all works we found that used this metric, always trained on the test data itself. If this transformation overfits the data, the ECE will be overestimated. Alternative versions for the ECE have boen proposed where the bins are adapted to contain equal number of samples, showing that the resulting ECE less prone to overfitting (Nixon et al., 2019). Yet, this only mitigates the problem, without fully solving it. A direct solution to the problem is treat the calibration transform like any other stage of the system, cstimating its parameters held-out data through cross- volos-valiationn where RCF is computed. Further, histogram binning is not necessarily a good calibration transform for every problem (we will show examples of this in the experimental section). If the calibration transform used to compute the calibrated posteriors is not a good match for the problem, then the calibration error will be underestimated since the calibrated posteriors will be poorer than they could be. Again, the ECE could be computed using a different calibration transformation, but this not what is done in the litcrature where the ECE definition always tied to using histogram binning as transform.
the test data. This, far we know, has never been done in the literature
Finally, perhaps the biggest weakness of the ECE appears when it is used for multi-class problems. In that case, in order to be able to contimue using histogram binning as calibration transform, the multi-class problem is mapped to a new binary problem where only the quality of the confidences is evaluated, ignoring all other values in the posterior vector (Nixon et al. 2019). The confidence is the posterior corresponding to the class selected by the system which, in the literature that uses ECE taken to be the one with the maximum posterior. This effectively maps the multi-class problem into a binary classification problem: deciding whether the system was correct in its decision by using its confidence as the posterior for correctness.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Given this new problem, one now use the definition for the ECE, where class2 is the fraction of samples correctly classified by the system and avep2 is the average confidence. We call the multi-class version of ECE, ECEme, As a consequence of the fact that ECEmc only evaluates confidences rather than the full posterior, it may fail to diagnose calibration problems on any of the other components of the posterior distribution (see Section 3.3) Whenever good posteriors are required, the full vector of posteriors should be evaluated and not just its maximum value (Popordanoska et al. (022)). Notably, in some works, including (Guo et al. 2017), the multi-class definition of ECE is unnecesarily used for binary problems.
While the expression for the ECE in Equation (27 is the one used in most machine learning empirical papers, a more general form for the ECE is sometimes used in theoretical papers where the quantization of the system's 's output is not done, and the divergence is a general function that satisfies d(s, q) - 0 when s s= q and positive otherwise (Widmann et al.) 2019). Setting to be the square euclidean distance, the ECE directly corresponds to the expected divergence for the Brier loss, when s obtained by histogram binning. In fact, the transform could also be changed to a general transform in which case the ECE simply coincides with the expected divergence in Equation (24) As we discussed above, though, the expected divergence corresponds to decomposition of the raw EPSR where the expectation is taken with respect to distribution based on a calibration model, making it rather impractical decomposition. The calibration loss proposed in the next section aims to solve this problem.
2.5.3 Calibration loss
As discussed earlier in this work, in our view, the main goal of calibration metrics should be to diagnose calibration problems during system development. We then propose the following procedure for computing a calibration metric for that purpose: 1) add a calibration transform at the output of the system, and 2) assess the level of improvement obtained from this additional stage. The level of improvement, measured as the difference in EPSR before and after adding the calibration stage, indicates the degree of miscalibration of the system. We call this metric calibralion loss. If the calibration loss is large relative to the original poorly calibrated and would benefit from the addition of the
EPSR value, then it means the system proposed calibration stage. If the improvement is relatively small, then it mcans that the system is already well calibrated. Alternatively, a small relative improvement may be due to the calibration transform being poor. This is the same problem discussed several times above. In order to measure calibration, a reference distribution needs to be created first. The quality of the calibration metric will strongly depend on the quality of this reference distribution. No calibration metric is immune to this issue.
Note that the process to obtain the calibration loss is, essentially, what is done at every step during system development: a new approach is tried (in this case, the addition of calibration stage) and its benefit is assessed by measuring the relative improvement obtained compared to the baseline (in this case, the system without the calibration stage). As for every other development, process, the assessment of the gains will be more reliable if the data where the performance is computed is not used for training or tuning purposes. When computing calibration loss, this means that the test data should not be used to train the calibration transform. This approach which, as far as we know, was never proposed in the calibration literature, ensures that the estimation of the calibration error will not be biased by overfitting effects.
Formally, we define the calibration loss (CalLoss for short) as:
CalLoss = EPSR - EPSResl
(30)
where EP and EPSRcal the two EPSRs in Equation (5 with the expectation taken with respect to the empirical distribution, that is, compuled as averages over the data, as in Equation 6
N
EPSReaw =
Ch
(31)
N
E
N
EPSRcal
Σ
C*
(32)
These two EPSRs measure the overall performance of the posteriors without and with the addition of the calibration stage, respectively (see Figure 3 As we discussed with regards to Equation (25). E.SR can
15

---

Published in Transactions on Machine Learning Research (01/2025)
(21,...,,IN) [N)
(q1.....9m .N 9N}
( SN}
Our classificr
Calibrator
Equation (31)
EPSRRraw
Equation (32)
EPSR
[hi....,hNI
Figure 3: Process for computing the two EPSRs needed obtain the relative calibration loss metric (Equa- tion 33 using dataset with N samples.
be seen to measure the inherent discrimination performance of the system since it is the EPSR that remains after the miscalibration has been fixed. Again, this holds as long as the calibration transform is doing a good job roducing the miscalibration.
For of interpretation, in our experiments we report relative CalLoss, RCL for short:
RCL 100 (EPSRraw EPSRea))/ EPSR
(33)
The RCL reflects the relative improvement in EPSR that we would be able to achieve by adding a post-hoc calibration stage given by to our system.
CalLoss differs from the divergence defined by Equation 24 in the way the expectations of the PSRs are computed on both sides. Instead of computing them with respect to a distribution consistent with s they are computed with respect to the empirical distribution. As a consequence, the calibration term is no longer an expected score divergence. Instead,
simply the difference between the two empirical EPSRs. As we will show in the experimental section, as long as is good calibration model, the EPSRs computed in Equation (24) or in Equation (30) very similar, resulting in the same value for the calibration metrie, which is the difference between the two EPSRs. On the other hand, when is not good calibration model, the lent-hand side in Equation 25 can be very poor estimator of the classifier's performance, resulting, a consequence, in meaningless decomposition. In contrast, EPSR computed as in Equation [31 does not depend on a calibration model, resulting in a more reliable measure the overall performance of the classifier. Even if the calibration model is poor, the calibration loss value is still meaningful: it indicates the improvement in EPSR that can be achieved by using that same calibration model as part of the system.
A specific instance of the calibration loss metric was proposed almost two decades ago for the binary classifi- cation problem of speaker verification by Brümmer & du Preez (2006). This metric which, far as we know, has since only been used for the speaker verification and forensic applications (Ramos et al. 2017 2020) is a special case of our proposed approach above for binary classification where the calibration transform is given by the best isotonic transformation obtained on the test data itself. In this work, we consider a general form of the calibration loss where the transform can adapted to the problem of interest and, preferably, trained on held-out data rather than on the test data itself. This generalization allows use the metric for multi-class problems where the isotonic transformation is not applicable and to avoid the bias produced by the train-on-test approach.
3 Experiments
In this section we show an analysis of the metrics discussed in the previous section using both synthetic and real datasets. Synthetic datasets allow us to have the ground-truth distribution, which in turn allows us obtain perfoctly-callbratod posteriors where we would expect any good calibration metric to give a value of zero. As we will see, this does not always happen, allowing us revisit some of the theoretical issues discussed in previous sections.
16

---

Published in Transactions on Machine Learning Research (01/2025)
For synthetic data, the procedure for generating calibrated and miscalibrated posteriors is described in Appendix
C Briefly, N samples are generated by sampling the input features from multivariate Gaussian distribution for each class. The number of samples generated for each class is determined by an imbalanced prior distribution with a prior for class H1 of 0.8 and equal prior for other classes, unless otherwise noted. The per-class multivariate Gaussian distributions are determined such that the means are equidistant at an 1,2 distance of 1. The covariance is diagonal with same variance for all dimensions and shared across all classes and it controls the performance of the resulting posteriors. We set the variance to 0.15 unless otherwise indicated. Perfectly calibrated posteriors are obtained using the known feature distribution for each class. This is the optimal classifier for this data since it is derived from the generating distribution. Then, three miscalibrated versions of these posteriors are created, one by artificially manipulating the prior distribution to be mismatched to that in the data (mep), another one by scaling the logarithm of the posteriors and renormalizing the result to produce overconfident or underconfident posteriors depending on the scale (mcs); and a final onc combining both causes of miscalibration (meps).
In Section 3.1 through 3.4 we show and discuss results for datasets with 2 and 10 classes, including first an illustration of the property described in Section 2.3.4 and then focusing on the comparison of various EPSRs and calibration metrics. Finally, in Section 3.5 results on real datasots are presented and discussed, showing the effect of post-hoc calibration on the different metrics on posteriors produced by actual systems.
In practice, it important to assess the robustness of the performance estimate obtained on our test dataset, specially when the number of samples is small. Appendix L includes an example on how to obtain confidence intervals using bootstrapping. We do not include confidence intervals in the results in this section to keep both the code that produces them and the plots simple.
3.1 Illustration of EPSR construction as integrals over Bayes risks
In this section we illustrate the construction of CE and BS as integrals over Bayes risks (see Section 2.3.4) using synthetic posteriors for a 2-class dataset. We generate posteriors for four different classifiers as described above and further detailed in Appendix C
cal: Perfectly calibrated system obtained with a per-class variance of 0.15.
mcs-u: Miscalibrated system obtained by scaling the log posterions from the cal system by 0.48 to simulate an underconfident system, and then converting the result back into posteriors by doing softmax.
mcs-o: Same as mes-u but scaling the log posteriors by 2.0, to simulate an overconfident system.
cal-h: Same the cal system but with a variance of 0.19 to obtain a harder dataset.
In all cases, the prior for class H1 is set to 0.6 and the total number of samples is set to 100,000.
The left plot in Figure 4 shows the Bayes risk for those four systems for cost matrices given by
C. = | 0 1/01]
(34)
1/(1-a) a1)
while varying the value of 01 between 0 and 1. The CE for each system is simply the area under the correspouding curve. Systems cal, mes-u, and mcs-o have the same Bayes risk for aj 0.5, which corresponds to the total error rate multiplied by 2. The corresponding Bayes decisions Since mcs-u and mas-o are constructed by scaling the log posteriors of the cal system, which does not affect the argmax decisions, the total error rate is the same for all three systems. Yet, since the Bayes risk at other values of a diflers across systems, the CE also differs, being higher for the two miscalibrated systems. Finally, we can see that the calibrated posteriors for the harder dataset, cal-h, have a worse total error rate than the other three systems, but better Bayes risk than the mes postcriors at extreme values of 01 Overall, integrating the curves for mes-u, mes-o and cal-h leads to the same NCE values (recall that NCE = CE/( Σ Pt log P.), with P 0.6 and P2- = 0.4 in these synthetic datasets).
given by the argmax rule.
17

---

Published in Transactions on Machine Learning Research (01/2025)
Risk,, * an (1 /n(//((2) 2)
0.7
cal, NCE-0.34
cal, NBS=0.29
0.6
0.6
MCS-U, NCE=0.42
mcs-u, NBS=0.34
mcs-o, NCE=0.42
mcsia, NBS=0.31
cal-h, NCE=0.42
cal-h, NBS=0.35 3.
Riske
0.4
0.3
0.3
0.2
0.2
0.0
0.0
0.6
1.0
a1
Figure 4: Weighted Bayes risk curves for four different systems. Left: the weight is 1.0 so that the integral under these curves is the CF. Right: the weight is an (1 a1)/ 2) that the integral is the BS. The normalized CE, NCE, and normalized BS, NBS, are shown in the legends.
The right plot in Figure 4 shows the Bayes risk multiplied by W(s) an (1- (1-21)/3(222) so that the area under these curves corresponds to the BS. We can see that the BS de-emphasizes the performance of the system at extreme values of a1- Note that the Bayes decisions corresponding to the cost matrix defined by Equation (34) are given by thresholding the posterior for class Hi at a threshold of aı (this can be derived by replacing C(H1, D2) = 1/a1 and C(22, D2) = 1/(1 a1) in Equation (I)). Hence, the operating points that are deemphasized by BS are those corresponding to extreme threshold values. Poor posteriors in those regions will be penalized much less by BS than by CE. A similar conclusions can be derived from comparing the log-loss expression corresponding to the CE and the squared loss expression corresponding to BS (see Figure D. While the squared loss is bounded by one, which happens when the posterior for the true class is zero, the log-loss gives infinite penalization to those errors. This difference between CE and the BS results in difference in ranking of the four systems according to each metric. While systems mis-u, mes-o, and cal-h are identical in terms of CE, they are not so in terms of BS due to the different weighting given to each operating point.
As is clear from this example, different EPSRs may result in different development decisions. Hence, is important choose an EPSR that correctly reflects the needs of the application. Expression (1)) provides an intuitive way of creating new PSRs, allowing By default, though, if no specific needs are identified, the uniform weighting function, which results in the CE metric, is good general choice. The BS may not be desirable choice for high-stakes application due to the effect obscrved in Figure 4 where the behavior of the risk for extremely imbalanced cost matrices is deemphasized. This means that if a system produces poor posteriors in the extremes (very high or very low values), the BS will not correctly diagnose the problem. The CE, on the other hand, will severely penalize such systems, making
select a weight function W appropriate for the task.
more appropriate choice for high-stakes applications where having good extreme posteriors is important.
The plots in Figure
g also help illustrate our argument about the role that calibration should (or, rather, should not) play in the evaluation of posteriors. While system mes-o is miscalibrated, it is better than system cal-h which is perfectly calibrated in terms of BS (and the same in terms of CE). If we had decided that BS is the EPSR of choice for our problem, there would be no reason to select the cal-h system over the mes-o system, despite the fact that the latter system miscalibrated. If we had the chance to add a post-hoc calibration stage to our system, we could compute the calibration loss to see that the mes-o posteriors are miscalibrated which would indicate that a calibration stage is needed. Yet, after adding that stage, the performance of the system should again be assessed in terms of its new EPSR.
3.2 Results on a Binary classification problem
The left plot Figure 5 shows five different metrics for the 2-class synthetic dataset, using the procodure described in Appendix C to produce 400 samples. We chose to use a relatively small dataset to show the effect that overfitting of the calibration transform may have in some of the metrics. The metries include
18

---

Published in Transactions on Machine Learning Research (01/2025)
2.0
100
cal
maps
75
50
NRisk-D1
NRi NRisk-im
NCE
RCL-BS-DXV RCL-8S-HYV RCL-BS-Dtt RCL-8S-Htt
ECEmc
Figure 5: Various metrics for a binary classification task for four different systems, one perfectly calibrated (cal) and three miscalibrated ones (mes, mep, meps). Left: normalized overall performance metrics. The dashed line indicates the performance of a naive system. Right: calibration metrics including the binary ECE, the multiclass ECE (ECEmc), and the RCL based on BS for calibration approaches, DP (D), and histogram binning (H), trained either through cross- gross-validation the test data (xv) training on the full test dataset (tt).
normalized cross-entropy (NCE), normalized Brier score (NBS), and three normalized Bayes risks, with cost matrices given by 1) the 0-1 cost matrix (NRisk-01). 2) the 0-1 cost matrix with an additional column corresponding to reject decision with cost of 0.1 for both classes (NRisk-OIr), and 3) square imbalanced cost matrix with C12 = 1 and c21 10 (NRisk-imb). The accuracy for each system can be computed as 1-003 0.2 NRisk-01, since NRisk-01 = /0.2, where Risk-01 is total error rate.
The results show that NRisk-01 unaffected by the miscalibration in mes, a scaling in the logarithm of the posteriors which results overconfident postcriors. This happens bocause the argmax docisions, which are the Bayes decisions for this cost. function, are not affected by the scaling. NRisk-Olr and NRisk-im, on the other hand, are affected by this type of miscalibration, with NRisk-im being less affected than NRisk-01 by the mismatch in priors in mep. As expected, see that the risks have different behavior depending the cost matrix, highlighting the importance of correctly selecting this matrix for the problem of interest when categorical decisions are required.
The three risks in the figure are EPSRs since they are computed using Bayes decisions. Yet, they are not strict PSRs. They only assess the performance of the posteriors for one specific operating point given by the cost matrix, which corresponds to set of decision regions determined by Equation ① The decisions regions are, in turn, used to make categorical decisions which then used to compute the risk. Hence, two systems that results on the same categorical decisions for given cost matrix would have the same risk value, while one may have better posteriors than the other for other cost matrices.
If we want to assess the goodness of the posteriors in general, across the full simplex, we need to use the expectation of strict PSRs, like the CE or BS. These two metrics assess the goodness of the posteriors without going through the step of making categorical decisions. As discussed in Section 3.1 despite both metrics being strict PSRs, they may result in different conclusions about the quality of the posteriors. This can be observed in the left plot in Figure where the normalized CE and BS metrics show a rather different assessment of the quality of the mos and meps systems. This happens because the overconfident posteriors from the mes and meps systems are pushed to the extremes, where the BS pays little attention.
The right plot in Figure 5 shows various calibration metrics: binary ECE, multiclass ECE, and various RCL metrics. The RCL. metrics are computed using histogram binning or DP calibration trained with cross- validation on the test data obtain EPSReat (Equation 32 We can see that the histogram binning approach is problematic for the cal and mes datasets. When using cross-validee voos--valiation to train it (as for RCL- BS-Hxv), the transform overfits the training data for each fold resulting in bad calibration when applying that transform on the test data for that fold. Hence, the posteriors are actually worse than the original ones, resulting in a negative calibration loss, something that should never happen for a well-designed well- trained calibration transform. When training on the test data (as for (CL--BS-Htt) the transform overfits the data and the metric diagnoses a non-oxisting calibration problem on the cal posteriors. These results show that, for this particular problem, histogram binning is poor calibration approach. In contrast, the RCL. values obtained with the DP transform quite robust for both training approaches, since the
19

---

Published in Transactions on Machine Learning Research (01/2025)
3
100
cal
mcp
mcs
mcps
75
1
J
2
25
0
NRisk-01
NRisk-01r
NRisk-im
NCE
NBS
RCL-CE-DXv RCL-CE-Txv RCL-BS-DXv RCL-BS-TXv ECÉmc
Figure 6: Overall and calibration metrics as in Figure but for 10-class classfication task.
number of parameters in this transform is small and it is much less prone to overfitting. Both of these RCL metrics correctly diagnose the calibration problem in the me posteriors, and the correct calibration in the cal posteriors. The value these metrics tell us the relative improvement in BS that we would get from adding an DP calibration stage the system and, hence, directly quantify how badly calibrated the original posteriors are without this stage.
Note that, given the way the posteriors are generated in synthetic datasets, the DP transformation is, by design, able to perfectly reverse the miscalibration present in those posteriors. In other datasets, the DP transformation may not be necessarily optimal since misscalibration can potentially have non-linear effects in the postcriors. Other calibration transformations available in the litcrature could be explored in those cases. In our experience, though, DP calibration has given excellent results across a variely of tasks. We will see further evidence of this in Section 3.5 As discussed in the next section, this is the case for temperature scaling, which fails in cases where the prior distribution in the training and the test data are different, common scenario in many applications.
Turning to the ECE metrics, we can see that ECEmc gives a different estimate of the calibration error than the original binary ECE, which is expected since ECEme is evaluating the performance problem, as explained in Section 2.5.2 We can also see that the ECE shows a similar trend as the RCL-BS- Htt since these two metries use the same calibrated posteriors obtained with histogram binning trained on the test data, differing only in the way the distance between the raw posteriors and the calibrated posteriors is computed (see Section 2.5.2 Hence, the ECE suffers from the same problem RCL-BS-HU, discussed above, mistakenly diagnosing small calibration problem in the cal posteriors. Finally, note that the value of the ECE docs not have clear interpretation, while, as discussed above, the RCL values directly indicate the percentage of the EPSR of the original posteriors which can be reduced by doing calibration, providing a more actionable result.
different binary
3.3 Results Multi-class classification problem
In this section we show results for a multi-class classification task instead of binary task as above, including additional versions of the calibration loss, using CE as the EPSR well as BS, and using temperature scaling as well as DP calibration to obtain the calibrated posteriors needed to compute that metric. The dataset was created following the process described in Appendix C with K = 10 classes, feature variance of 0.08, and a total number of samples N = 2000.
The left. plot Figure f shows the results for various overall metries. The three risks are a direct gener- alization of the ones for the 2-class case: 1) the 0-1 cost matrix (NRisk-01), 2) the 0-1 cost matrix with an additional column correspouding and 3) a square cost matrix with imbalanced costs, cy for all i $j, except when i = 10 in which case Cij = 10 (NRisk-imb).. The conclusion from these results is similar to that for the binary case: each risk gives a different ranking of systems, and the NCE is more severely degraded by the scaling miscalibration in mos and meps than the NBS.
reject decision with costs equal to 0.1 for all classes (NRisk-O1r),
The right plot in this figure shows the relative CalLoss (RCL) based on BS and CE for two calibration methods, DP calibration and temperature scaling trained with vrossvalidatiin, and the multiclass ECE, These results show large difference wetn some of the calibration metries. Those that temperature
20

---

Published in Transactions on Machine Learning Research (01/2025)
System #Samples
cal
mep
maps
200
2000
20000
200
2000
20000
200
2000
20000
200
2000
20000
true
-3
-5
-2
-7
-10
-5
4
-2
4
-2
0
Dtt
0
0
0
0
0
0
0
0
0
0
0
posteriors
Dxv
1
0
1
1
0
0
0
0
0
Ttt
1
-1
-1
29
28
28
29
28
28
Txv
1
-1
2
-1
29
28
28
29
28
28
Htt
-6
-8
-9
-153
-143
-132
-19
-21
-17
-62
-84
-82
Hxv
-171
-144
-132
-15
-62
-82
Table 1: Relative difference between the semi-empirical cross-entropy (CE,e) and the empirical cross-entropy (CE,) computed as (CE. - /E.e)/ CE2- #100 and then rounded to the nearest integer. Results are shown for synthetic posteriors as Figure 5 for different number of samples. The semi-empirical cross-entropy is computed with respect to posteriors given by the true posterior (true), and by calibrated versions of the posteriors using DP calibration (D), temperature scaling (T), and histogram binning (H) trained with cross-validation (xv) or trained on the test data (tt).
scaling for calibration fail to diagnose the severity of the calibration problem in mep. This is because this method cannot compensale for calibration problems due to mismatched priors and, hence, it also cannot be used to diagnose them. As any other calibration metric, CalLoss fails if the calibration transform is not a good match for the problem. Yet, the CalLoss framework allows us to explore a variety of transforms. If one such transform leads to a relatively large CalLoss, and that transform was not trained on the test data itself, then we can conclude that the system miscalibrated. Importantly, the same transform that was used to diagnose the problem can be used as post-hoe calibration stage and reduce the EPSR by a percentage indicated by the RCL value.
Turning to ECEmmc, we can see that it gives different assessement of the relative severity of the miscalibration of the different datasets compared to the RCL metrics. This is partly because it uses a different way to measure the distance between the raw and calibrated scores one that is not induced by any PSR-and partly because only evaluates the quality of the maximum posterior for each sample rather than the full vector. When the miscalibration occurs on classes other than the one with maximum postcrior, the ECEme metric cannot. properly diagnose the problem. This issue with the multi-class ECE has also been discussed by Nixon et al. (2019). The CalLoss metric does not suffer from this problem it assesses the quality of the full posterior vector. In addition, note that the scale of the ECEme is not interpretable. While, for example, the RCL-CE-DXV values for miscalibrated systems indicate that over 70% of the CE is due to miscalibration, no equivalent interpretation is possible with the ECFmcc.
3.4 Effect of the reference distribution in the EPSR value
As explained in Section 2.5 EPSRs values may differ depending on the reference distribution used take the expectation, In this section, compare cross-entropy values (the expectation of the NLL) given by:
(p, (C q)) = (.),())))) log(g))
(35)
for different reference distributions: the empirical distribution in the test data used in all prior results in this section (Equation 8 and various semi-empiln empirical distributions where (,) is the empirical distribution for q. but (h q) is given by a classifier. Since we are working with synthetic data, we can derive P, | q) from the truc generating distribution. This posterior, of course, would not be available on real data. Hence, we compare with different (r q) given by calibrating the posteriors using histogram binning, DP calibration, and temperature scaling calibration trained with validatio directly the test data.
Table I shows the relative difference between the semi-empirical cross-cntropy and the empirical cross- entropy for the same four datasets used in Figure 5 with varying number of samples. We can see that, in most cases, the relative difference between the empirical and the semi-empirical cross-entropy very small, specially for the larger datasets where the empirical cross-cntropy is less noisy. Larger differences occur when the calibration transformation does not fully fix the calibration problem, which happens with temperature scaling for the mes and meps posteriors and for histogram binning in most cases, for the larger datasets.
21

---

Published in Transactions on Machine Learning Research (01/2025)
These examples illustrate the practical problem involved in using the expected score divergence a cali- bration metric. If the calibration model used to obtain s(q) P. q) in Equation (25) does not do a good job at calibrating the posteriors, the EPSR in the left-hand side may be a poor estimate of the system performance. This will, in turn, result in a meaningless decomposition with the two terms adding up to a value that does not reflect the actual performance the system. On the other hand, CalLoss meaningful regardless of the quality of the calibration model. The EPS and EPSR used to compute it reflect the performance of the system before and after calibration with that perhaps suboptimal calibration model. Hence, CalLoss always predicts the level of improvement in EPSR that can be obtained from using the selected calibration transform. For this reason, we believe that, in practice, CalLoss is better calibration metric than the expected score divergence.
3.5 Results on real datasets
In this section, we show results on real datasets for a number of different tasks corresponding to speech, image, and natural language processing (NLP) tasks.
For NL. we include SST2 and AGNEWS. SST2 (Socher et al. 2013) is a natural language processing dataset where the task is decide whether a certain text has positive or negative sentiment. In AGNEWS Gulli 2005 Zhang et al. 2015) the task is to classify news into 4 different classes. The posteriors for these datasets were produced with the GPT-2 model using the code provided in https://gitthl com/ using zero-shot prompts.
For speech processing we include three datasets, SITW, FVCAUS and TEMOCAP. STTW (MeLaren et al., 2016) and FVCAUS (Morrison et al. 2015 2012) are two speaker verification datasets where the task is to decide whether two audio samples belong to the same speaker or not. To obtain postcriors for these two datasets, we ran an X-vector PLDA system using the code provided in http::::///ttuuub.oo///uuee DCA-PLDA IEMOCAP is speech processing dataset where the task is to classify each speech sample into set of emotions: angry, happy, sad, and neutral. The posteriors were downloaded from https: Sattaaalllllaasseee http:///wwt.tttttt/////////////p
Finally, for image processing we include results on CIFAR10, which is an image processing datasets where the task is to classify the object in an image into one of 10 classes. The posteriors for this dataset were obtained using the code available in https://github. com/
cwwssification. sataafiiiaaiooocodbaeeee codebase, We evaluate three models: resmet20, vgg19, and repvgg a2. These models have approximately 0.27 million, 20 million, and 27 million parameters, respectively. We also include three medical imaging tasks from the MedMNIST dataset (Yang el al., 2023): PATH, composed of histological images where the task is to classily into 9 types of tissues; PNEUM, composed of pediatric chest x-rays classified as pneumonia or normal; and ADRENAL, composed of tomography scans of adrenal glands classified as "normal adrenal' or "adrenal wwwp:/ddd.rg//eecor//////1111111 for the Resnet-50
mass'. The posteriors were downloaded from models for the maximum image resolution.
We also created binary classification tasks for the detection of one specific class versus all others for CI- FAR100, dataset similar to CIFAR10 but with 100 classes, using the posterior obtained with resnet20 system (using the same codebase as above for CIFAR10) for that class and 1 minus that posterior for the "other" class. We call these posteriors CLFAR-XvSO, where X identifies the target class.
Table 3 in Appendix D shows the number of classes, the priors, and the total number of samples for all the datasets. SST2, SITW and FVCAUS as well as the CIsO sets are binary classification problems, while all others are multi-class problems.
For each dataset and system, we show results obtained the raw posteriors they come out of the system, and on calibrated posteriors using DP calibration trained with cross-validation on the test data. For the binary classification tasks, we also show results for the best calibrated posteriors obtained with the PAV algorithm run on the full test set. Table 2 shows three normalized risk metrics, and NCE, RCL, ECE, and ECEme for each of those systems.
Focusing on the calibration metrics for the raw posteriors (see entries highlighted in red), we can see that both ECE metrics often fail to diagnose calibration problems. For example, according to ECF, the SITW raw
22

---

Published in Transactions on Machine Learning Research (01/2025)
NRisk
Calibration ECEme
Dataset
System
proc
C01
Cab
Cimb
NCE
RCL
ECE
raw 0.996
0.812
1.000
1.073
62.4
34.4
34.8
GPT224sh
cal 0.226
0.585
0.711
0.404
-0.6
1.6
2.2
calp 0.223
0.555
0.660
0.385
-0.4
0.0
0.0
SST2
raw 0.828
0.818
1.000
0.917
46.0
20.0
27.3
GPT2-0sh
cal 0.310
0.655
0.685
0.495
-0.6
1.4
1.5
calp 0.298
0.638
0.661
0.478
-0.4
0.0
0.0
raw 0.324
0.225
0.191
0.189
16.7
0.2
0.2
SITW
XvPLDA
cal 0.306
0.190
0.153
0.158
-0.1
0.0
0.0
calp 0.304
0.188
0.152
0.155
-0.0
0.0
0.0
raw 3.915
1.992
1.049
1.966
99.6
1.8
8.6
FVCAUS
XvPLDA
cal 0.012
0.008
0.005
0.008
-1.0
0.0
0.0
calp 0.012
0.007
0.004
0.006
-0.9
0.0
0.0
Resnet-20 20 raw 0.420
0.236
0.158
0.214
7.1
0.2
0.2
CIFAR-LvsO
cal 0.430
0.199
0.130
0.199
-2.3
0.1
0.2
calp 0.400
0.175
0.132
0.169
-1.1
0.0
0.0
Resnet-20 raw 0.700
0.442
0.371
0.393
8.2
0.3
0.4
CIFAR-2vsO
cal 0.560
0.440
0.367
0.361
-0.4
0.3
0.3
calp 0.560
0.415
0.363
0.329
-0.5
0.0
0.0
Resnet-50 raw 0.278
0.736
0.479
0.802
58.1
7.7
8.6
PNEUM
cal 0.239
0.433
0.453
0.336
-2.0
2.0
2.4
calp 0.218
0.404
0.449
0.302
-1.8
0.0
0.0
Resnet-50 raw 0.928
1.013
1.122
0.931
15.9
11.1
12.0
ADRENAL
cal 0.913
0.815
0.747
0.783
-1.3
3.7
4.3
calp 0.826
0.705
0.668
0.714
-0.9
0.0
0.0
Resnet-50 raw 0.114
0.684
0.142
0.329
69.0
7.2
PATH
cal 0.076
0.281
0.106
0.102
-0.6
1.4
0.504
1.056
0.607
0.635
3.1
6.3
ЛEMOCAP
W2V2
cal 0.494
0.984
0.606
0.615
-0.1
2.7
GPT2-0sh raw 0.780
1.009
0.936
0.814
34.2
18.4
AGNEWS
cal 0.378
1.014
0.762
0.536
-0.1
3.7
Resnet-20 raw 0.082
0.406
0.121
0.122
17.2
3.9
cal 0.083
0.311
0.112
0.101
-0.8
0.8
CIFAR1O
Vgg19 raw 0.068
0.486
0.100
0.153
32.4
5.0
cal 0.069
0.322
0.097
0.103
-0.6
1.7
raw 0.053
0.309
0.076
0.092
19.9
3.2
RepVgg-a2
cal 0.052
0.239
0.067
0.074
-1.2
0.9
Table 2: Various metrics and calibrated posteriors for different speech, image and natural language processing datascts. Calibrated posteriors (cal) are obtained with DP calibration using a cross-validation procedure, For the binary tasks, calibrated posteriors obtained with the PAV algorithm (calp) are also con- sidered. We report three normalized risk values (NRisk) for the same cost. matrices as in Figure 5 normalized empirical cross-entropy (NCE), relative calibration loss for CE using DP calibration with cross-validation (RCL-CE-Dry) in Figures and G here RCL for short), binary ECE, and multi-class ECE (ECEmc). Colored entries highlight comparisons made in the text.
23

---

Published in Transactions on Machine Learning Research (01/2025)
posteriors are calibrated, while we know they are not since their NCE improves significantly after calibration. Similarly, while for FVCAUS the miscalibration is to blame for almost 100% of the NCE value, ECF., is below 10 and ECEme is close to O: both ECE metrics strikingly fail to diagnose the severity of the miscalibration of this system. A similar thing happens for PNEUM and PATH, where the ECE values are relatively low, but the RCL is quite large. On the other hand, for ADRENAL, the ECFs are higher than for PNEUM and PATH, but the RCL is much lower. Overall, we can see a very weak relationship between the ECEs and the gain that can be achieved from calibration, given by the difference between the NCE of the raw and the calibrated systems. This indicates that the ECE is not a useful tool for diagnosing misscalibration. On the other hand, the RCT, is defined as the relative gain that can be achieved from post-hoc calibration, directly indicating the impact that adding such a stage would have on the system.
In some cases, ECEme suggests that a miscalibration problem exists where RCL does not, For example, for the CIFARI00 RepVgg-a2 raw posteriors, the ECEmc of 5.6 indicates a small calibration error while the RCL of 0.8 suggest the system is very well-calibrated. First, it is important to note that the ECEme is not a relative measure. It is not possible to assess the impact that an ECE value of 5.6 would have on the performance of system. In fact, the impact will be different depending on the overall performance of the system. Unfortunately, this metric cannot be turned into a relative value like the RCL. because there is no corresponding NCE to use as reference. In contrast, an RCL value of 0.8 means that only 0.8% of the NCE value is due to miscalibration. Further, note that ECFme only assesses the performance of the maximum posterior for each sample while RCL assesses the performance of the full vector of posteriors. The ECEme would tend to overestimate the impact of a small miscalibration in the maximum posterior since it disregards the other posteriors (99 of them in this 100-class task) which may be very well calibrated. Finally, it is possible that the low RCL is due to the DP calibration transformation not being cnough to fix the misscalibration problem, resulting in an underestimation of the RCL. In practice, a developer could explore several calibration alternatives before deciding on one approach. The best calibration approach is the one that results in the lowest EPSR value (and, hence, the largest. RCL), as long as the calibration model was not trained on the test data. Exploration of various calibration techniques is out of the scope of this paper.
Another important observation from this table is the fact that, sometimes, a miscalibrated system is better in terms of the quality of its posteriors than one that is well calibrated. See, for example, the results for the CIFAR10 posteriors (entries highlighted in green). The Resnet-20 Dxv-calibrated posteriors have perfect calibration and an NCE of 0.101. The RepVgg-a2 raw posteriors, on the other hand, are miscalibrated with an RCL of 19.9%, but have an NCE of 0.092. This means that the posterior probabilities generated by the latter system are better. Further evidence of this can be found in the NRisk values, which are consistently better for the RepVag-a2 raw posteriors than for the Resnet-20 calibrates ones. This example illust why assossing calibration is not helpful to determine the quality of posterior probabilities and should never be used to compare systems with each other.
Results for CIFAR10 also illustrate the fact that two systems with the same NCE are not necessarily equally good for every operating point, as determined by a specific risk function. For example, the Resnet-20 and Vgg19 Dxv-calibrated posteriors have almost identical NCE (0.101 vs 0.103), but show different trends for the three NRisk metrics (see entries highlighted in blue and the green ones above). When the application of interest has a specific well-defined risk function, then that metric should be used to make development decisions rather than NCE, which integrates over all operating points instcad of focusing on the one relevant for the task.
Comparing the results for Dxv-calibrated (cal) and PAV-calibrated posteriors (calp) we can see that they are very similar in most cases (see entries highlighted in orange). This implies that the Dxv-calibration procedure is doing a near optimal job at calibrating the posteriors for those datasets. The small difference between cal and calp may be due to a number of reasons: 1) that the DP transformation is not sufficiently expressive for these datasets, 2) that the transforms trained with cross-validation are not generalizing well to the test sets due to the small number of samples available, or 3) that the isotonic PAV transform overfitted the test data resulting in an unrealistically low NCE value. The only way to determine whether the NCE for calp posteriors is too low or the one for cal posteriors is too large is to keep working the calibration transformation in search of one that, when trained on held-out data or through cross-validatio reduces the NCE further than Dxv calibration.
24

---

Published in Transactions on Machine Learning Research (01/2025)
4 Discussion and conclusions
In this work, we focus on the problem of evaluating the quality of probabilistic classifiers. We review the classical concept of proper scoring rules (PSRs) which are scoring functions designed to assess the performance of probability distributions. The expectation of a PSR (EPSR) reffects the quality of the posterior probabilities provided by a classifier. When comparing two systems, a lower EPSR indicates that the system's posteriors are better for making decisions and, hence, also better for interpretation by an end user. A normalized EPSR larger than 1.0 indicates that making decisions with those posteriors would be worse than making decisions simply based on the class priors.
Despite the existence of this elegant and principled tool to assess the performance of probabilistic classifiers, the current trend in much of the machine learning literature is to use calibration metrics for this purpose, under the claim that good calibration is an essential requirement for posteriors to be interpretable, safe, and reliable. In this work, we contend this view and argue that good calibration is neither necessary nor sufficient for a probabilistic classifier's posterior to be of value to the end user. A system can have a relatively high calibration error and still be useful to the user, as long as its EPSR is low. On the other hand, a system might be perfectly calibrated but provide very little value to the user if its EPSR is too high. Hence, we see reason to report calibration metrics as way to assess the value of a probabilistic classifier's outputs.
It is important to note, though, that there is also usually no reason to tolerate a miscalibrated system. Any miscalibrated system can be fixed by adding a post-hoc calibration stage at the end of the pipeline, transforming the posteriors output by the system into better posteriors. We believe the only role of calibration metrics should be to aid the developer in deciding whether such post-hoc calibration stage is needed for a given system. After that docision has been made, though, we argue, calibration metrics should play no further role in evaluation. In particular, calibration metries should not be used to compare systems with each other or to decide whether a system is safe for use in high-stakes applications. EPSRs should instead be used for these purposes.
If the role of calibration metrics is simply to reflect the performance gain that can be achieved by fixing the potential miscalibration of the system, then, we argue, they should be designed to do exactly that. A direct way to measure the impact of classifier's miscalibration is to add a post-hoc calibration stage to the system and assess the gain in EPSR provided by this addition. The difference between the EPSR of the original classifier and the EPSR after adding post-hoc calibration-callled calibration loss-is a direct measure of the miscalibration of the original system. If the calibration loss relative to the EPSR of the original system is large, then a calibration stage should be added to the system. Importantly, the calibration stage should be designed and trained as any other system stage, without using the test data to avoid overestimating the impact that a calibrator would have in the system's performance.
We compared calibration loss with the ECE, the most widely used calibration metric in the machine learning literature, and showed both theoretically and empirically that the ECE has number of important disad- vantages. First, it lacks interpretability since it is not related to a PSR and cannot be turned into relative measure of the gains that would be achieved with post-hoc calibration. In addition, the multi-class version of ECE measures only the performance of the largest. posterior probability generated by the system for each sample rather than of the full posterior, failing to directly address the question of interest. In contrast, cal- ibration loss is directly interpretable, evaluating the calibration of the full posterior distribution as the gain that can be obtained from adding a post-hoc calibrator to the system. Other problems with the ECE include the fact that relies on histogram binning, an often poor choice as calibration transformation, and that this transformation is trained on the lest data itself, leading to patential overestimation of the miscalibration due to overfitting of the transform. While calibration loss could also be computed with this choice of calibration transform, we recommend against it, favoring instead the use of direction-peserving calibration trained with cross-validation on the test data or on held-out data.
Finally, we also compared calibration loss with a calibration metric obtained from a classic decomposition of the expected PSR of the classifier. We show that, in practice, this decomposition is highly affected by the quality of the calibration transform used to compute it, loosing its meaning when the transform does not adequately fix the classifier's miscalibration. Calibration loss, on the other hand, retains its interpretation
25

---

Published in Transactions on Machine Learning Research (01/2025)
the gain that can be obtained after post-hoc calibration with the selected transform, even if the calibration transform is suboptimal.
The paper is accompanied by an open-source repository which provides code for the computation of the various metrics in this paper: https:///iithu: comm//uferrerr/xpecteeee cost All plots and tables this paper can be replicated using the notebooks under the baabboookk/valaatingg posteriors _apes_paprr___ folder that repository. We hope that this work along with the provided code will facilitate the wider adoption of these metrics that offer a principled, clegant, and general solution to the evaluation and diagnosis of probabilistic classifiers.
Acknowledgments
This material is based upon work supported by the European Union's Horizon 2020 research and innovation program under grant No
STA.:////SSEEEAAA//////E--O--EE 20SS-2200.
We thank Niko Brümmer for many enlightening discussions about calibration and PSRs, for his valuable leedback on this paper, and for contributing the proof in Section 2.4.
References
R. K. Ahuja and J. B. Orlin. Solving the convex ordered problem with applications to isotone regression. Technical Report SWP no.3988, Massachusetts Institute of Technology, Sloan School of Management, February 1998.
M. Ayer, H. D. Brunk, G. M. Ewing, W. T. Roid, and E. Silverman. An empirical distribution function for sampling with incomplete information. Annals of Mathemalical Statistics, 1955.
J. M. Bernardo and A. F. M. Smith. Bayesion Theory. John Wiley & Sons, 1994.
C. M. Bishop. Pallern Recognilion and Machine Learning. Springer, 2006.
G. Brier. Verification of forecasts expressed terms of probability. Monthly Weather review, 1950.
J. Bröcker. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society, July 2009.
N. Brümmer. Measuring, Resining and Calibraling Speaker and Language Information Extracted from Speech. PhD thesis, Stellenbosch University, 2010.
N. Brümmer and J. du Preez. Application independent evaluation of speaker detection. Computer Speech and Language, April 2006.
N. Brümmer and D. A. van Leeuwen. On calibration of language recognition scores. In Proc. Odyssey-06, Puerto Rico, USA, June 2006.
T. Cover and J. Thomas. Elements of Information Theory. Wiley, 2006.
A. Dawid. The geometry of proper scoring rules. Annals of the Institute of Statistical Mathematics, 2007.
A. P. Dawid, M. Musio, and L. Ventura. Minimum scoring rule inference. Scandinavian Journal of Statistics, 2016.
Dawid and M. Musio. Theory and applications of proper scoring rules. METRON, Apr 2014.
M. DeGroot and M. Schervish. Probability and Statistics. Pearson Education Limited, 4th edition, 2014.
M. H. DeGroot. Oplimal Statistical Decisions. McGraw-Hill, 1970.
M. H. DeGroot and S. E. Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Sociely. Series D. (The Statistician), 1983.
26

---

Published in Transactions on Machine Learning Research (01/2025)
M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P. Steiner, M. Caron, R. Geirhos,
T. Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning. PMLR, 2023.
S. Desai and G. Durrett. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
R. Duda, P. Hart, and D. Stork. Pattern Classification. Wiley, 2001.
L. Ferrer, M. McLaren, and N. Brümmer. A speaker verification backend with robust performance across conditions. Computer Speech and Language, 2021.
T. S. Filho, H. Song, M. Perello-Nieto, R. Santos-Rodrigee Rodriguez, M. Kull, and P. Flach. Classifier calibration: How to assess and improve predicted class probabilities: survey. Machine Learning, 2023.
T. Gneiting. Making and evaluating point forecasts. Journal of the American Statistical Association, 2011.
T. Gneiting and A. E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 2007.
Good. Rational decisions. Journal of the Royal Statistical Society, 1952.
P. Gopalan, Hu, and G. N. Rothblum. On computationally efficient multi-class calibration. In Proceedings of Thirly Seventh Conference on Learning Theory, June 2024.
S. G. Gruber and F. Bucttner. Better uncertainty calibration via proper scores for classification and beyond. In Proc. of NewrIPS, New Orleans, December 2022.
A. Gulli, The anatomy of news search engine. In Special Interest Tracks and Posters of the 14th Interna- tional Conference World Wide Web, New York, 2005.
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In Proc. of the 34th International Conference on Machine Learning, Sydney, Australia, 2017.
T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer- Verlag, 2001.
Y. Huang, W. Li, F. Macheret, R. A. Gabriel, and L. Ohno-Machado. A tutorial on calibration measure- ments and calibration models for clinical prediction models. Journal of the American Medical Informatics Association, 02 2020.
E. Jaynes. Probability theory: The logic of science. Cambridge university press, 2003.
Z. Jiang, J. Araki, H. Ding, and G. Neubig. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 2021.
M. Keller, S. Bengio, and S. Wong. Benchmarking non non-paramctric statistical tests. In Proc. of NeurIPS, Vancouver, December 2005.
M. Kull, M. Perello Nieto, M. Kängsepp, T. Silva Filho, H. Song, and P. Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In Proc. of NeurIPS, 2019.
A. Kumar, P. S. Liang, and T. Ma. Verified uncertainty calibration. In Proc. of NewrIPS, 2019.
P. Liang, R. Bommasani, T. Tee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu.
A. Kumar, et al. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023.
M. McLaren, L. Ferrer, D. Castan, and A. Lawson. The speakers in the wild (SITW) speaker recognition database. In Proc. Interspeech, San Francisco, Sept. 2016.
27

---

Published in Transactions on Machine Learning Research (01/2025)
M. Minderer, J. Djolonga, R. Romijnders, F. Hubis, X. Zhai, N. Houlsby, D. Tran, and M. Lucic. Revisiting the calibration of modern neural networks. In Proc. of NewrIPS, 2021.
G. S. Morrison, P. Rose, and C. Zhang. Protocol for the collection of databases of recordings for forensic- voice-comparison research and practice. Australian Journal of Forensic Sciences, June 2012.
G. Morrison, C. Zhang, and E. Enzinger et. al. Forensic database of voice recordings of 500+ australian english speakers. http: http://aattaaass..oorraiiivveeee
net 2015.
J. Mukhoti, V. Kulharia, A. Sanyal, S. Golodetz, P. Torr, and P. Dokania. Calibrating deep neural networks using focal loss. In Proc. of NewrIPS, 2020.
R. Müller, S. Kornblith, and G. E. Hinton. When does label smoothing help? In Proc. of NewrIPS, 2019.
M. P. Naeini, G. Cooper, and M. Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. In Twenty-Ninde Ninth AAAT Conference on Artificial Intelligence, 2015a.
M. P. Nacini, G. F. Cooper, and M. Hauskrecht. Binary classifier calibration: Non-parametric approach. In Proc. of SIAM Int. Conf Data Min, 2015b.
J. Nixon, M. W. Dusenberry, L. Zhang, Jerfel, and D. Tran. Measuring calibration in deep learning. Tn CVPR Workshops, 2019.
Y. Ovadia, F. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. V. Dillon, B. Takshminarayanan, and J. Snoek. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In Proc. of NewrIPS, Vancouver, June 2019.
E. Oveharov. Proper scoring rules and Bregman divergences. Bernoulli, March 2015.
G. Parmigiani and L. Inoue. Decision theory: Principles and approuches. John Wiley & Sons, 2009.
M. Peterson. An Introduction to Decision Theory. Cambridge Introductions to Philosophy. Cambridge University Press, 2009.
J. C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adu. Large Margin Classifiers, 2000.
N. Poh and S. Bengio. Estimating the confidence interval of expected performance curve in biometric authentication using joint bootstrap. In Proc. ICASSP, Honolulu, April 2007.
T. Popordanoska, R. Sayer, and M. B. Blaschko. A consistent and differentiable Lp canonical calibration error estimator. In Proc. of NewrIPS, New Orleans, 12 2022.
J. Quinonero-Candela, C. E. Rasmussen, F. Sinz, O. Bousquet, and B. Schölkopf. Evaluating predictive uncertainly challenge. In Machine Leurning Challenges Workshop. Springer, 2005.
D. Ramos and J. Maanzlle-Rodrigueer Reliable support: Measuring calibration of likelihood ratios. Forensic Science International, July 2013.
Ramos, R. Krish, J. Fierrez, and D. Meuwly. From biometric scores to forensic likelihood ratios. Handbook of biometrics for forensic science, 2017.
D. Ramos, D. Meuwly, R. Haraksim, and C. E. Berger. Validation forensic automatic likelihood ratio methods. In Handbook of forensic statistics. Chapman and Hall/CRC, 2020.
S. Raschka, Model evaluation, model selection, and algorithm selection in machine learning, arXiv preprint 1rXiv:1811.12800 2018.
L. J. Savage. The foundations of slatisties. Courier Corporation, 1972.
K. Slooten and R. Moester. Forensic identification Database likelihood ratios and familial DNA searching, 2012.
28

---

Published in Transactions on Machine Learning Research (01/2025)
R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Washington, USA, October 2013.
R. J. Tibshirani and B. Efron. An introduction to the bootstrap. Monographs on statisties and applied probability, 1993.
Tomsott, A. Preece, D. Braines, F. Cerutti, S. Chakraborty, M. Srivastava, Pearson, and L. Kaplan. Rapid trust calibration through interpretable and uncertainty-aware AI. Patterns, 2020.
J. Vaicenavicius, D. Widmann, C. Andersson, F. Lindsten, J. Roll, and T. Schön. Evaluating model cal- ibration in classification. In The 22nd International Conference on Artificial Intelligence and Stalistics. PMLR, 2019.
K. Van Hoorde, S. Van Huffel, D. Timmerman, T. Bourne, and B. Van Calster. A spline-based tool to assess and visualize the calibration of multiclass risk predictions. Journal of biomedical informatics, 2015.
D. Widmann, F. Lindsten, and D. Zachariah. Calibration tests in multi-class classification: A unifying framework. In Proc. of NeurIPS, Vancouver, December 2019.
R. L. Winkler and A. H. Murphy. Good probability assessors. Journal of Applied Meteorology, 1968.
J. Yang, R. Shi, D. Wei, Z. Liu, L. Zhao, B. Ke, H. Pfister, and B. Ni. MedMNIST v2 a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, January 2023.
B. Zadrozny and C. Elkan. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In Proc. of ICML, May 2001.
X. Zhang, J. Zhao, and Y. LeCun. Character-level convolutional networks for text classification. In Proc. of NewrIPS, 2015.
A Cross-entropy and Brier score parameterized by the priors
The cross-entropy (CE) and the Brier scores (BS) can be expressed as a function of the priors, allowing us to turn them into parameters of the metric. For the CE, the expression is given by
N Ph. log(gn, (I1)) CE = Σ No.
(36)
where No. is the number of samples of class hr, Setting the prior for each class h, Ph, to N₂/N we recover the standard CE expression (Equation Rp. The advantage of the expression above is that allows us manipulate PA, independently of the test dataset. This is useful when the class frequencies present in the test data do not reflect the prior distribution that is expected during deployment. In that case, the Pa 's can be set to those we expect to see when the system is used. The resulting CE will then better reflect the values we would measure on that target data if it was available for evaluation.
For the BS, the expression parameterized by the priors given by:
BS
Σ (a(xx)) - 1(hs H.)))
(37)
B The absolute distance loss is not a score divergence
Here we provide a countercxample show that the absolute distance loss does not satisfy the mean-as-
minimizer property of Bregman divergences (see Equation 29 ) To show this, we generate posteriors
described in Appendix C for 2-class problem with P 0.6. Then, take the expectation with respect
29

---

Published in Transactions on Machine Learning Research (01/2025)
L1post2
0.5-
1.2
2.5
0.5-
1.0
0.8
13-
0.4
1.0
0.3
0.3
010
0.2
0.4
Q01 1,0
0.6
0.8
0.0
022
0.4
0.5
002
1.0
0.1
0.2
0.4 0.6
0.5
0.0
0.2
0.4
0.5
0.8
1.0
901
901
961
Figure 7: Expected divergence between synthetic posteriors for a binary task and qo, a fixed vector of posteriors. The figure shows the expected divergence as a function of the first component of qo: which we call (01, for different divergences (blue curves) and the mean value of the first component the posterior (star). The star has to coincide with the minimum of the curve for valid score divergences proving by contradiction that the L1 loss is not score divergence.
to the empirical distribution of their divergence to a fixed posterior qu- For valid score divergences, the minimum expected divergence should occur when qo is equal the mean of s.
Figure 7 shows the expected divergence as function of the first component of qo, for four different di- vergences: 1) the L1 loss only over the posterior for H2, which is, approximately (given that the ECE actually quantizes the posteriors before computing the distance), what the ECE computes, 2) the L1 loss between the full posterior vectors, 3) the L, loss, which corresponds to the Brier score PSR, and 4) the Kullback-Leibler loss, which corresponds to the negative logarithmic loss PSR. The red stars in this plot correspond to the mean value of the first component of s. We can see that the first two losses do not satisfy the mean-as-minn miniiizer property and, hence, are not score divergences.
C Synthetic dataset for experiments
Here describe the procedure used to create synthetic datasets for the experiments in this paper. Given number of classes K, total number of samples N, prior for class H₁ of Pı, and variance a, which are taken as parameters of the simulation, proceed as follows:
Set the class priors to P1 for class H1, and P. = PP)((K--- - for classes through K. Unless otherwise indicated, P1 is to 0.8.
Determine the number of samples for each class, N; as the closest integer to P. N.
Generate N; samples using a multivariate Gaussian distribution N(js,, aT), with mean μ4 given by a one-hot vector with the one at the ith dimension and diagonal covariance matrix with equal variance in all dimensions given by which, unless is otherwise indicated, is set to 0.15. We take these samples to be the Fz, the input features for each sample.
4. Compute the likelihoods for each class for each generated sample according to the class distributions used to draw these samples, that is, P(x H,) N ol,
5. Finally, we assume two possible prior distributions: 1) the same prior distribution given by the P.s according to which the data was generated in step 3, and 2) mismatched distribution P, where = 0.1(KK 1) for + and Px = 0.9.
6. Using those two prior distributions, compute two sets of posteriors which we call cal and mep (for mis-calibree calibrated due to a mismatch in posteriors) which correspond to using the data priors and the mismatched priors, respectively, to obtain the posteriors according to:
P(H])2) P(x[H)) P(H.) P(x)
P(x) P(H.)
(38)
E, P(H,))
30

---

Published in Transactions on Machine Learning Research (01/2025)
Dataset
# Classes
#Samples
Priors
SST2
2
1821
0.50 0.50
SITW
721788
0.99 0.01
FVCAUS
2
114072
0.98 0.02
LFAR-IvsO
10000
0.99 0.01
CIFAR- 2vsO
2
10000
0.99 0.01
PNEUM
624
0.38 0.62
ADRENAL
2
298
0.77 0.23
PATH
7180
0.05 0.06 0.08 0.09 0.10 0.12 0.14 0.17 0.19
IRMOCAP
4
5473
0.20 0.29 0.31 0.20
AGNEWS
7600
0.25 0.25 0.25 0.25
CIFARIO
10
10000
0.10 for all classes
Table 3: Number of classes, number of samples, and class priors for each dataset included in our experiments.
where the P(x Hi) are the likelihoods computed in step and P(H:) are the corresponding matched or mismatched priors, P. and P., respectively.
Note that with the procedure above, the cal posteriors are perfectly calibrated for the test data. The mep posteriors, though, are not calibrated because, even though the likelihoods used to compute it are obtained from the generating distribution, the priors are mismatched to the ones used for testing. Further, we create misscalibrated versions the posteriors which we call mes and maps by scaling the cal and mep posteriors, respectively, in the log domain and then converting them back posteriors by computing the softmax. The scale is set to 5.0 unless otherwise indicated, simulating an overfitted system that produces overconfident posteriors.
D Real datasets for experiments
Table 3 shows the priors and total number of samples for all the datasets used in Section 3.5
E
Confidence Intervals
The metrics in the main text and in prior sections of this appendix are computed as averages over the whole test set. For small datasets, this estimate might be quite unreliable, not necessarily reflecting the performance we would observe in practice. Hence, having an estimate of the range of values that our metric of interest could take on a new dataset is essential, specially for small datascts. A standard method for obtaining such ranges is the bootstrapping approach (Tibshirani Efron 1993 Keller et al.) 2005 Poh Bengio 2007). Given test set of size N, the procedure for obtaining confidence intervals as follows:
Obtain B bootstrap sets size N by sampling the original test set with replacement.
Compute the metric of interest in each of the B bootstrap sets.
Compute the confidence interval with confidence level 7 by calculating the 7/2 and 100 4/2 percentiles from the list of B metric values computed above.
The confidence intervals obtained in this way assume that the system is fixed and exactly what will be deployed. Only the variability due to the test data is reflected in these intervals (Raschka) 2018 Importantly, when computing calibration metrics, the calibration transform, if computed using cross-validation train- on-test rather than using a separate calibration dataset (in which case the calibration transform can be considered part of the system and also frozen), should retrained for each bootstrap set. This allows the confidence interval reflect the variability in metric values due to changes in the transform training data. Importantly, when doing cross-vall vross-vali,ttin, the folds should be defined using the original sample identifier avoid having the same sample across more than fold.
31

---

Published in Transactions on Machine Learning Research (01/2025)
.0
100
mcp
mcs
mcps
55
50
.0
NBS
RCL-BS-AXV RCL-BS-Hxv RCL-BS-Att RCL-BS-Htt
Figure 8: Metrics with confidence intervals obtained with bootstrapping. Left: the normalized BS for the same posteriors as in Figure Right: the RCL for BS using the same calibration methods as in Figure
To illustrate, Figure shows the confidence intervals obtained from 100 bootstrap samples for the same posteriors as in Figure for NBS and BS-based RCL. The code to create this plot can be found in the accompanying repository.
32


Reviews:
[{"review_id": "TUYPhIcSub", "reviewer": "Reviewer_Jshr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I like this paper. In particular, I believe the following aspects of the paper are a strength:\n1. *Discussion of PSRs.* The paper provides a well-rounded review of proper scoring rules (PSRs), explaining their connection to Bayes decision theory. It explains why PSRs are more appropriate for evaluating the quality of posterior probabilities than calibration metrics, which only address one aspect of posterior quality. I like the pedagogy here.\n2. *Novel proposition of calibration loss.* I like the grounding and simplicity of this idea. It seems reasonable that one should simply \"add a calibration transform at the output of the system, and assess the level of improvement obtained from this additional stage\" to determine the miscalibration. \n3. *Critique of current practices.* The paper critiques common practices in machine learning where calibration metrics (like ECE) are used to assess posterior performance without considering discrimination or refinement aspects. This critique seems timely and valuable.\n\nThe only weakness in my mind is the relative lack of practical experiments. An exploration of real-world, high-stakes applications (e.g., medical diagnostics, autonomous vehicles) would strengthen the practical relevance of the proposed calibration loss. However, this kind of critique can almost always be used against authors, so I do not weight it highly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The contributions of the paper are as follows:\n* The authors advocate for the use of expected proper scoring rules (PSRs) over calibration metrics like the expected calibration error (ECE) for assessing the quality of posterior probabilities in classifiers.\n* The authors provide a practical review of PSRs, grounded in Bayes decision theory, to demonstrate why expected PSRs serve as a principled measure of a system's posterior quality, unlike calibration metrics.\n* The authors introduce a new calibration metric called calibration loss, derived from a decomposition of expected PSRs, and show that it is superior to existing metrics", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The only minor requested changes I have are:\n1. The inclusion of a greater number of diagrams in the theory section.\n2. An increase in the size of the diagrams in the experimental section.\n3. Bringing brief explanations of datasets into the main text.", "broader_impact_concerns": "I don't think there are any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sg6etlrV83", "reviewer": "Reviewer_Qdxh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I'm keeping my comments short, because I think there are some fundamental issues with the submission. If the authors respond convincingly to these issues, I will take another look at the paper and provide a more thorough review.\n\nThe paper is essentially a position paper advocating against equating the calibration of a probabilistic classifier with its goodness, which the paper represents as a widespread viewpoint/practice in machine learning. The paper is certainly correct to point out that \"calibration is neither necessary, nor sufficient for posterior probabilities to be useful to the end user\". However, I believe (and am not convinced otherwise by the paper) that this is already the dominant viewpoint in machine learning, and that there are relatively few papers actually arguing otherwise. \n\nThe paper cites a small handful of papers repeatedly in making its central claim that there is an over-reliance on calibration metrics in the ML literature. For instance, the paper says \"many of the recent machine learning works concerned with the evaluation of posteriors do not use PSRs for the task, resorting instead to calibration metrics (for example, Guo et al., 2017; [...])\". This statement surprised me, so I took a look at the first citation, Guo et al. (2017), to get a sense for how they were using calibration metrics. Guo et al. (2017) seem to use such metrics as a post-processing step to calibrate the outputted probabilities of a trained neural network. Importantly, the neural networks in their paper are first trained to minimize cross-entropy loss (a proper scoring rule). In what sense then can it be said that Guo et al. do not use a PSR for the evaluation of posteriors when their models are explicitly trained to perform well on a PSR? \n\nThe paper's actual stance on calibration metrics is confusing and difficult to pin down, and the paper sometimes seems to contradict itself. The paper, for instance, makes the following strong claim:\n\n1) \"Hence, when evaluating the utility of posteriors we should not be concerned with whether they are calibrated or not.\"\n\nwhich is followed shortly by this one:\n\n2) \"In this paper, we argue that the only purpose of calibration metrics should be to diagnose whether a system is well-calibrated in order to fix it if that is not the case [...]\"\n\nIf we should not be concerned with whether posteriors are calibrated (claim 1), then why should we care about fixing systems that are not well-calibrated (claim 2)? Moreover, how does the practice that the paper advocates in claim 2 differ meaningfully from that of Guo et al. (2017) (who use calibration metrics to diagnose whether fitted neural networks are not well-calibrated and to fix them if so)?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper advocates against an over-reliance on calibration metrics to evaluate probabilistic classifiers. To do so, the paper first provides a lengthy tutorial and discussion on Bayesian decision theory, proper scoring rules, and calibration. The paper then presents an argument that \"the goodness of a probabilistic system...should be assessed using [expected proper scoring rules], not calibration metrics\" and further advocates for a very restricted use of calibration metrics and the use of one calibration metric in particular.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper needs to be much clearer about what exactly its stance on calibration is and how it is actually different than the dominant viewpoint in machine learning. I am not aware of a large community in ML that actually believes the calibration of a probabilistic classifier equates to its goodness, regardless of its discriminative capability. I am not sure if the paper is making that claim, but if it is, it needs to substantiate it much more convincingly. My current understanding of this paper is that it is a position paper which is pointing out and correcting a currently widespread bad practice in the ML community. For this to be a useful paper, the bad practice needs to be more clearly stated, and its widespread-ness needs to be more convincingly shown.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TAnxL50O5c", "reviewer": "Reviewer_wX9M", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors of this paper clearly communicate their goals and I found it easy to understand their logic. They do a good job building up their technical tools in a cohesive fashion. One aspect I appreciated was the explicit use of a \"reference distribution\" in place of the questionable \"true\" distribution.\n\nMy main issue with the paper is that I am unpersuaded by it. The authors' position is that, because PSRs exactly capture a probabilistic classifier's quality under a cost function, it doesn't make sense to care about calibration. However, I take it as self-evident that calibration is a desirable property. So one might want a system with both a low score and high calibration. It seems reasonable that, in some settings, one might be willing to sacrifice a little Brier score for much better calibration. Thus, I feel the paper contains many statements which are not supported by the arguments. For example: \"...assessing calibration is not helpful to determine the quality of posterior probabilities and should never be used to compare systems with each other.\" (pages 20 & 22).\n\nMost readers will understand such statements as expressing opinions, so I don't view this as an issue of technical soundness.\n\nOverall, while this piece doesn't supply much in the way of new findings or new concepts, I think that it contributes to the discussion in a way that some will find interesting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper discusses the evaluation of probabilistic predictions. It focuses on proper scoring rules (PSRs) and various notions of calibration. The paper aims to persuade readers that calibration should not be relied on to evaluate systems. The bulk of the theoretical presentation is devoted to prior work. There are experiments on synthetic and real examples. The authors introduce a new notion of \"calibration loss,\" which quantifies how much a scoring rule improves when a classifier is post-processed to be calibrated. This notion generalizes an older definition (for binary classification) that appears to not be widely used.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There are several different notions of multiclass calibration that would be useful to point to, even if they don't change the discussion. There is probably a better reference, but [1] is a recent paper with relevant citations.\n\nI think the paper would be much better if the authors were able to supply a direct argument for why calibration is not a sensible desideratum. I view the current argument, that PSRs lead to a single number which is more meaningful and useful, as indirect.\n\n[1] Gopalan, P., Hu, L. & Rothblum, G.N.. (2024). On Computationally Efficient Multi-Class Calibration. Proceedings of Thirty Seventh Conference on Learning Theory, in Proceedings of Machine Learning Research 247:1983-2026 Available from https://proceedings.mlr.press/v247/gopalan24a.html.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the rebuttal and discussions, the 3 reviewers agreed with accepting the paper, by recommending accept or leaning accept.

Reviewer wX9M requested the following in the review: "There are several different notions of multiclass calibration that would be useful to point to, even if they don't change the discussion. There is probably a better reference, but [1] is a recent paper with relevant citations."
[1] https://proceedings.mlr.press/v247/gopalan24a.html
I encourage the authors to look into the reference (and relevant citations) to further enhance the discussions about multi-class calibration.
Decision: Accept (poster)