Reassessing How to Compare and Improve the Calibration of Machine Learning Models
Authors: Muthu Chidambaram
Rong Ge
Venue: ICLR 2025 Poster (2025)
Keywords: calibration
proper scoring rules
ECE
kernel regression
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: Reporting just test error and calibration error can be misleading, we provide a way to jointly plot calibration and generalization error and provide an analysis of the method.

Abstract: A machine learning model is calibrated if its predicted probability for an outcome matches the observed frequency for that outcome conditional on the model prediction. This property has become increasingly important as the impact of machine learning models has continued to spread to various domains. As a result, there are now a dizzying number of recent papers on measuring and improving the calibration of (specifically deep learning) models. In this work, we reassess the reporting of calibration metrics in the recent literature. We show that there exist trivial recalibration approaches that can appear seemingly state-of-the-art unless calibration and prediction metrics (i.e. test accuracy) are accompanied by additional generalization metrics such as negative log-likelihood. We then use a calibration-based decomposition of Bregman divergences to develop a new extension to reliability diagrams that jointly visualizes calibration and generalization error, and show how our visualization can be used to detect trade-offs between calibration and generalization. Along the way, we prove novel results regarding the relationship between full calibration error and confidence calibration error for Bregman divergences. We also establish the consistency of the kernel regression estimator for calibration error used in our visualization approach, which generalizes existing consistency results in the literature.

Published as a conference paper at ICLR 2025
REASSESSING HOW TO COMPARE AND IMPROVE THE CALIBRATION OF MACHINE LEARNING MODELS
Muthu Chidambaram & Rong Ge Department of Computer Science, Duke University
ABSTRACT
A machine learning model is calibrated its predicted probability for an out- come matches the observed frequency for that outcome conditional the model prediction. This property has become increasingly important as the impact of machine learning models has continued to spread various domains. As result, there are now a dizzying number of recent papers on measuring and improving the calibration of (specifically deep learning) models. In this work, reassess the reporting of calibration metrics in the recent literature. We show that there exist trivial recalibration approaches that can appear seemingly state-of-the-arr unless calibration and prediction metrics (i.e. test accuracy) are accompanied by additional generalization metrics such as negative log-likelhoood. We then use a calibration-based decomposition of Bregman divergences to develop a new exten- sion to reliability diagrams that jointly visualizes calibration and generalization error, and show how visualization calibration and generalization. Along the way, we prove novel results regarding the relationship between full calibration error and confidence calibration error for Bregman divergences. We also establish the consistency of the kernel regression estimator for calibration error used in visualization approach, which generalizes existing consistency results in the literature.
be used to detect trade-off's between
INTRODUCTION
Standard machine learning models trained to predict probability distributions over set of possible actions or outcomes. Model-based decision-making is then typically done by using the action outcome associated with the highest probability, and ideally one would like to interpret the model-predicted probability as notion of confidence in the predicted action/outcome.
In order for this confidence interpretation to be valid, it is crucial that the predicted probabilities are calibrated (Lichtenstein et al. 1982 Dawid 1982 DeGroot & Fienberg 1983), accurately reflect the true frequencies of the outcome conditional on the prediction. As an informal (classic) example, a calibrated weather prediction model would satisfy the property that we observe rain 80% of the time on days for which our model predicted a 0.8 probability of rain.
As the applications of machine learning models particularly deep learning models continue to expand to include high-stakes areas such as medical image diagnoses (Mehrtash et al., 2019 Elmarakeby et al., 2021; Nogales et al., (2021) and self-driving cars (Hu et (0023). so too does the importance of having calibrated model probabilities. Unfortunately, the seminal empirical investigation of Guo et al. (2017) demonstrated that deep learning models can be poorly calibrated, largely due to overconfidence.
This observation has led to a number of follow-up works intended to improve model calibration using both training-time (Thulasidasan et al., 2019, Müller et al. 2020 Wang et al. 2021) and post-training methods (Joy et al., 2022] Gupta & Ramdas (022). Comparing these proposed improvements, however, non-trivial due to the fact that the measurement of calibration in practice is itself an active area of research (Nixon et al.) 22019) Kumar et al. 2019 Blasiok et al.) 2023), and improvements with respect one calibration measure do not necessarily indicate improvements with respect another.
Even if we fix choice of calibration measure, the matter is further complicated by the existence of trivially calibrated models, such as models whose confidence always their test accuracy (see Section Most works improving calibration have approached these issues by choosing to report

---

Published as a conference paper at ICLR 2025
number of different calibration metrics along with generalization metrics (e.g. negative log- ligeliied or mean-squared error), but these choices vary greatly across works and are always non- non-exhaustive.
Motivated by this variance in calibration reporting, our work aims answer the following questions:
Do there exist any systematic issues in the reporting of calibration in the recent literature?
Is there a theoretically motivated choice of which calibration and generalization measures report, as well as an efficient means of estimating and visualizing them jointly?
1.1 SUMMARY OF MAIN CONTRIBUTIONS AND TAKEAWAYS
In answering these questions, we make the following contributions.
1. We identify multiple issues (identified in Sections 3 and 4.2 in how calibration performance is reported in the context of multi-class classification, with the core issue being that many (even very recent) papers report only some notion(s) of calibration error along with test accuracy. With respect to these metrics, the trivial recalibration strategy of always predicting with the mean confidence obtained from a calibration set outperforms the most popular post-training recalibration methods.
2. Based on this observation, we revisit the use of proper scoring rules for measuring calibration. We apply decomposition results for these scoring rules show how we can motivate calibration metric based on our choice of generalization metric, and in special cases how can do the converse well. Furthermore, Bregman divergences and leverage the structure of these divergences to also prove new results regarding the relationship between full calibration and confidence calibration, with our main result being Lemma 4.6 (confidence calibration error is a lower bound for full calibration error).
focus proper scoring rules that are
3. Finally, we propose an extension to reliability diagrams for jointly visualizing calibration and generalization. We also prove the consistency of the estimation approach we in our new visualization in Theorem 4.9 thereby extending existing estimation results in the literature. Section [5] we show how our ceabraatio-sharppeee diagrams can provide more granular comparison of recalibration methods when compared to traditional reliability diagrams (Figure? 2). Code for our visualizations is available as the Python package shharpaal
The main takeaway of our results is that the choice of calibration metric should be motivated by the choice of generalization metric or loss function and that both should be reported together in order to ensure that improvements in model calibration are not at the cost of model sharpness.
1.2 RELATED WORK
Measuring calibration. Expected calibration error (ECE) is almost certainly the most used cali- bration metric in the literature, and it is typically estimated using histogram binning (Nacini et al.. 2014). Although originally estimated using uniformly spaced bins (Guo et al., 2017). different binning schemes (such as equal mass/quantile-
binning) and debiasing procedures have been proposed to improve estimation of ECE (Nixon al.) [019 Kumar et al. 2019 Roelofs et al. 2022). As ECE is known to have a number of undesirable properties (e.g. discontinuous in the space of predictors), recent work has focused modifying ECE to fix these issues (Blasiok & Nakkiran
2023 Chidambaram et al. 2024).
Alternatives ECE include proper scoring rules (Gneiting & Raftery 2007 , smooth calibration error (Kakade & Foster 2008). maximum mean calibration error (Kumar et al. 2018), cumulative plots comparing labels and predicted probabilities [Arrieta-lbarne Ibarra et al.) (022). and hypothesis tests for miscalibration (Lee et al., (2022). Several of these approaches are analyzed under the recent theoretical framework of Blasiok et al. (2023). which puts forth a general desiderata for calibration measures.
In practice, most true calibration measures are only efficient for binary classification, and measuring calibration in the multi-class classification setting requires making concessions on the definition of calibration considered. Towards this end, a number of different calibration notions intended to efficiently work in the multi-class setting have been proposed in the literature (Vaicenavicius et al.)
2019 Kull et al. 2019 Widmann et al. 2020, Zhao et al.) 2222 Gupta & Ramdas 2022; Gopalan al. 2024 We elaborate more binary multi-class calibration in Section
2

---

Published as a conference paper at ICLR 2025
Improving calibration. Approaches for improving calibration can largely be categorized into post-training and training- saiming-time modifications. In the former category, standard approaches include histogram binning (Mincer & Zarnowitz 1969: Zadrozny & Elkan 2001), isotonic regression
(Zadrozny & Elkan 2002). Platt scaling (Platt 1999), temperature scaling (Guo et al. (017). and adaptive variants of temperature scaling (Joy et al. 2022). For the latter, data augmentation
(Thulasidasan et al. 2019 Müller et al., (020) and modified versions of the cross-entropy loss (Wang
al. (02T) have been shown lead to better calibrated models.
Calibration and proper scoring rules. Divergences obtained from strictly proper scores such as negative log-likelihood (NLL) and mean-squared error (MSE) often serve as metrics for quantifying generalization. A predictor that is optimal with respect to a strictly proper scoring rule has recovered the ground-trute conditional distribution, and is therefore necessarily calibrated. However, requiring optimality with respect to proper scoring rule is stronger condition than requiring calibration, and several works have thus explored how to decompose proper scoring rules into a calibration term and a sharpness/resolution term (Murphy, 1973 Gneiting et al. 2007 Bröcker 2008 Kull & Flach] 2015) Pohle 2020). The most related such work to ours is that of Perez-Lebel et (2023). which also puts forth the perspective that reporting calibration and accuracy is insufficient; however, they focus on a decomposition involving a grouping loss term whereas we focus on sharpness term which leads to different visualizations and different takeaways. We provide a more detailed comparison this work in Appendix
PRELIMINARIES
Notation. We use R to denote the extended real line. We use e; to denote the standard basis vector in R with non-zero i coordinate. Given n E N. we use [n:] to denote the set {1, .... ]]. For function g: K" + R we use (x(x) to denote the 7th coordinate function of g. We use A to denote the (k- - 1)--imension probability simplex in RR For a probability distribution π we use supp( to denote its support. Additionally, if π corresponds to the joint distribution of two random variables X and Y (e.g. data and label), we use x and TT to denote the respective marginals and Jy|x to denote the regular conditional distribution of Y given X. When takes values over a finite set, we will identify Ry and Tyxx=z with vectors in A and similarly identify Tylx with the corresponding function from Rd to A For a random variable X that has density with respect the Lebesgue measure, we use pxx (æ) to denote its density. We use (X) to denote the sigma-algebra generated by the random variable X.
Calibration. We restrict our attention to the classification setting, in which there exists a ground-truth distribution ι on Rdd [k] and our goal to recover the conditional distribution x We say a predictor g: : Rd 8-1 is fully calibrated if it satisfies the condition
IP(Y x(x = v) = =
(2.1)
where P(Y (XX) = v) is the regular conditional distribution of Y given gXX) at g(X) = v. Informally, the conditional distribution of the label given our prediction matches our prediction.
In the case where k = 2 (binary classification), we can identify g(X) with a single scalar and the condition (2.1 easy (approximately) verify it amounts to estimating a 1-D conditional expectation E (X) On the other hand, when k is large and g(X) has a density with respect to the Lebesgue measure on IR. the estimation problem becomes significantly more difficult due the curse of dimensionality.
For such cases, the typical approach taken in practice is to weaken 2.I to a lower dimensional condition that is casier work with. To describe such approaches, we adopt the general framework of Gupta & Ramdas (2022). Namely, given a predictor g as before, we consider a classification [c] and confidence function h A k-1 + [0, 1] that used to post- process g. For such e and h, we can define the weaker notion of confidence calibration (Kull et al. 2019
function c *-11==
(also sometimes referred to top-class or top-label calibration):
P.YY c(g(X)) M(((X)) =
(2.2)
We refer to the specific case of c(z) argmax; 2 (assuming arbitrary tie- tie-breaking) and h(2) = max; 2; as standard confidence calibration, since this is usually how confidence calibration appears in the literature. Additionally, to keep notation uncluttered, we will simply use c(X) and h(X)
3

---

Published as a conference paper at ICLR 2025
to denote (X)) and h(g())) when g is understood from context in the remainder of the paper. Confidence calibration is by far the most used notion of calibration for multi-class classification in
the literature, and will thus mostly focus our attention on the condition (2.2) moving forward.
Alternative notions are discussed in detail in Kull et al. 2019); Gupta & Ramdas (2022): Ciopalan
al. (2024).
To evaluate how close a tuple (c, h, g) is to satisfying 2.2).. we need a notion of calibration error. As mentioned earlier, the most common choice of calibration error is the Expected Calibration Error (ECE), which is simply the absolute deviation between the left and right-hand sides of (2.2):
EC h. = E [IP(Y c(X) ((X) p) - pl]-
(2.3)
This version of ECE is also referred to as the L ECE; raising the term inside the expectation to q > 1 yields the L ECE raised to the power q. Intuitively, ECE measures how much our classification accuracy conditioned on our confidence differs from our confidence. In order to estimate the ECE, it necessary to estimate the conditional probability P(Y = c(X) h(X) p). This is done using binning estimators (Naeini et al.
2014; Nixon et al.] 2019) and more recently using kernel regression estimators (Popordanoska et al.
2022 Blasiok Nakkiran 2023 Chidambaram et al. 2024).
Proper scoring rules. Alternatives to ECE for measuring calibration error include divergences obtained from proper scoring rules, which are described in full detail Gneiting & Raftery (2007). For our purposes, suffices to consider only the case of scoring rules S A x A R. in
which case S is called proper if S(u, u) 2 S(u, u) for u & and strictly proper if equality is only achieved when v u. The divergence ds associated with the scoring rule then defined to be ds(u, v) = S(u, u) S(u, v) and is guaranteed to be nonnegative so long as S is proper.
Perhaps the most popular divergences obtained from (strictly) proper scoring rules in machine learning are the mean squared error (MSE, also referred to as Brier score) and the KI. divergence (or just NLL); minimizing either one empirically ensures that recovers the ground-truth conditional distribution Tyix given sufficiently many samples and appropriate regularity conditions. As mentioned, this is stronger than necessary for achieving (2.2). which is partly why calibration errors such as ECE are appealing as they directly target the condition 2.2).
PITFALLS IN CALIBRATION REPORTING
We begin by first pointing out the well-known fact that even the full calibration condition of (..2 can be achieved by trivial choices of g. Indeed, the choice g(X) = Ty satisfies (2.2) despite the fact that g is constant predictor.
similar construction is possible for confidence calibration. We can define the confidence function h be such that h=z) P(Y argmax, g gx()), i.e. h is constant equal to the test accuracy of Taking c(z) = argmax; 2; as in the standard case, we once again have that ECF,,, (c, h, g) = 0.
since:
P(Y = c(X)
- P(Y = c(x)) = h(X)
(3.1)
by construction. Of course in practice, do actually know PP(Y argmax, (X)), but can estimate it on a held-out calibration set and set / to be this estimate. We refer to this approach as mean replacement recalibarion (MRR), and it can be viewed as a special case of the modified histogram binning procedure of Gupta & Ramdas (002)). ). Since MRR is defined purely through a modified h, there is no clear way for getting a vector A for computing metries such as NLL or MSE; for such cases, we simply set the non-predicted class probabilities to be (1 h(x))/(K 1).
The ramifications of the MRR strategy seem be apper-taee in the literature we have minimized calibration error without affecting the prediction of by using constant confidence function h. By separating the confidence function from the classification function, we have made it so that the test accuracy remains unchanged, so that this kind of trivial confidence behavior is undetectable by just reporting calibration error and accuracy. Even though MRR may seem extreme, can imagine modifications where effectively perform the procedure of replacing the predicted confidence with the calibration accuracy while doing something slightly more clever with the other predicted probabilities.
To what extent this a problem in the literature? Although many papers do report multiple calibration and generalization metrics, ECE and accuracy are often the main (or even only) reported
4

---

Published as a conference paper at ICLR 2025
metrics in the main body of these papers. For example, even the influential empirical analysis of Guo jet al.] (2017) focuses entirely on ECE results in the main paper (NLL results are, however, available in the appendix). This is also largely true of the follow-up re-assessment of deep learning model calibration of Minderer et al. (202T), which focuses on the relationships between test accuracy and ECE for different model families.
For works regarding the improvement of calibration, the work of Thulasidaaan et al. 2019) (which motivated the study of Mixup for calibration) reports only test accuracy, ECE, and related overconfi- dence measure throughout the whole paper. Similarly, the works of Zhang et al. (2020) and Wen al. (202T) analyzing how ensembling strategies can affect calibration also only report test accuracy. ECF. and some ECE variants throughout their whole papers. Calibration has also been an important topic in the context of large language models (LL.Ms), and initial investigations into the calibration of LLMs for question-answeriin -nnwerrin (Jiang et al. 2020 Desai & Durrett 2020) also only reported test accuracy and ECE.
The above listing of papers is of course non exhaustive; we have focused handful of influential papers (all have at least 100 citations) to merely point out that reporting (or at least focusing on) only ECE/similar and test accuracy is common practice. That being said, into question the advances made by the above papers many of their findings have been repeatedly corroborated by follow-up work. On the other hand, for new work calibration we strongly advocate
not all calling
for always reporting errors derived from proper scoring rules (i.e. NLL or MSE) alongside pure calibration metries such as ECE (we put forth a methodology for doing so in Sections and
Experiments. To drive home the point, we revisit the experiments of Guo et al. (2017). but focus on one of the more modern architectural choices of Minderer et al. (2021). Namely. we evaluate the most downloaded pretrained vision transformer (ViT) mode
( (osoviiskiy et al. 2021 Steiner et al. 202T) available through the t. mm library (Wightman (019) with respect to binned ECE, binned ACE (Nixon et al., 2019). SmoothECE (Blasiok & Nakkiran (002),, negative log- likeeioo,,,,d, and MSE on ImageNet- K- Val (Russakovsky et al. 2015). We split the data into 10,000 calibration samples (20% split) and 40,000 test samples, and compare the unmodified test performance to temperature scaling (TS), histogram binning (HB), isotonic regression (IR), and our proposed MRR. We follow the same TS implementation as Guo et al.) (2017) and also use 15 bins for the binning estimators (ECE, ACE, HB) to be comparable to the results in their work. The results shown in Table [
are not sensitive to
the choice of model; we show similar results for several other popular timm models in Appendix F
Model
Test Accuracy
ECE
ACE
SmoothECE
NLL
MSE/Bricr Score
VIT (Baseline)
85.14
9.42
9.44
9.44
65.35
22.73
ViT + TS
85.14
2.59
5.46
2.60
56.83
21.95
ViT + HB
82.03
5.06
11.97
4.03
00
27.17
ViT IR
83.82
4.53
8.13
4.48
00
24.13
ViT + MRR
85.14
0.18
0.18
0.18
144.66
27.51
Table 1: Comparison of different recalibration methods applied to pretrained ViT model on ImageNet- 1Kl All metries scaled by factor of 100 for readability. Histogram binning and isotonic regression have unbounded NLL as they can predict zero probability values for some classes
Remark 3.1. It is possible detect the specific instances of trivial calibration discussed above using alternatives to proper scoring rules, such as ROC-AUC applied to the confidence calibration problem. However, the confidence calibration problem can behave very differently than the overall multi-class classification problem (illustrated in Section D and typically it is very efficient to compute the proper scoring rules applied the multi-class problem directly.
4 MAIN THEORETICAL RESULTS
As can be seen in Table the MRR strategy dominates in terms of all of the pure calibration metrics, but leads to large jumps in NLL and MSE. Clearly either NLL MSE alone would have been sufficient to detect the trivial behavior of MRR. We now address the more general question: which
'The model card is available at: https::/hugginggeee
co/t mm/vi ba_ea_aaa______eeeeeeee baase_patchl 1eitchh166610 6__224
augreg2_ing innp2____________ in2ik__ffffc ft_kk
5

---

Published as a conference paper at ICLR 2025
generalization metrics should we report to prevent misleading calibration results? Our answer relies on the fact that such metrics are usually Bregman divergences. Our focus on Bregman divergences is due to their convenient theoretical properties; in particular, they defined in terms of convex functions and we can thus bring to bear the machinery of convex analysis.
We first review the definitions. Let us use Φ : A + to denote a continuously differentiable (C*), strictly convex function. We recall that the Bregman divergence dø associated with is defined to be:
[) (vy) Voly) (x y).
(4.1)
In many machine learning tasks the goal is to minimize the expectation of a conditional Bregman divergence; for example, multi-class classification we take ((x) Σ; xy log x; to obtain the KL divergence and then attempt to minimize [Td ( 9(X))] from samples. Bregman divergences are closely related with strictly proper scoring rules, and we can leverage the fact that such rules can be decomposed into calibration error and a "sharpness" term to obtain a similar decomposition for Bregman divergences.
Lemma 4.1. [Bregman Divergence Decomposition] For do defined as in (4.1 and two integrable random variables and X defined on the same probability space, it follows that:
Elda(Z, X)] B (6) Z Bi
( XT)]
T Fld ( X],
(4.2)
Sharpness Gap
Calibration Error
More general versions Lemma 4.1 have appeared in Bröcker 2009); y Kull & Flach (0155): Pohle (2020); Gruber & Buettner (2024), but we present version for Bregman divergences as we will leverage the structure of (4.1) to prove our new results in the subsequent sections. We provide a self-contained proof of Lemma in Appendix A
Some intuition for the sharpness gap term is provided in the next two propositions, the first of which appeared in Pohle (020) in different form. The second follows easily from the first and Lemma 4.1 but has not appeared elsewhere to our knowledge. Intuitively, the propositions show that a smaller sharpness gap corresponds to a predictor that is more "fine-grained"" and in that sense the constant calibrated predictor that we introduced in the previous subsection is the worst possible fully calibrated predictor. The proofs of both propositions can once again be found in Appendix A
Proposition 4.2. For 91,92 Rd + A-1 and π the joint distribution of data and label (X on Rd x [k),, (X)) S (g)( then:
[[o(E|Tyix 9x(X) S Elo(E[Ty[x 192(X)])
(4.3)
Proposition 4.3. Let π. correspond to the joint distribution of data and label (X,Y) on Rd [R]] Then every predictor g that is calibrated in the sense of (2.1) satisfies:
Efdx(Ty|x:: 9(x)) S Eldo(Tyix. my)]
(4.4)
The main takeaway from Lemma 4.1 is that when we choose to optimize a Bregman divergence, we implicitly also have a notion of calibration error that we are minimizing via (4.2). This provides a practical workflow to answer our motivating question of which metrics to report together: we first fix a Bregman divergence for our notion of generalization, and then report the same divergence but applied to the expected label conditioned on the predicted probabilities. We can also ask about the other direction - if we choose a calibration error, is there a natural Bregman divergence associated with it?
The answer is: not always. In fact, in considering this question, we can motivate a choice of L ECE. The next result shows that in the framework of Bregman divergences the "right" choice of ECE is the L ECE, since it is the conditional expectation of the Brier score.
Proposition 4.4. Let n correspond the joint distribution of data and label (X,Y) on Rd
[k), and define the L (for integers q 2 1) ECE of a function
Rdd + k-1 to be
E 9[X]] - (1 Then only the L ECE is induced by a Bregman divergence in the sense of Lemma 4.1
The proof is a simple consequence of characterizations provided in Boissonnat et al. (010) (which recall in Appendix A

---

Published as a conference paper at ICLR 2025
4.1 CONFIDENCE CALIBRATION WITH BREGMAN DIVERGENCES
So far, we have leveraged existing ideas in the literature proper scoring rules to motivate a joint choice of generalization and calibration error. However, as we alluded earlier, when considering high-dimensional classification problems it is not feasible to efficiently estimate the conditional expectation required for calibration error. The main result of this section to show that we can replace the full calibration error in with confidence calibration error and still obtain valid lower bound, which provides justification for the reporting of confidence calibration at least in the context of Bregman divergences.
In order discuss confidence calibration, need 1-D counterpart xp [0.1] E to G. For our cases of interest, the choice of yp is obvious; for example, when ((x) | (i.e. de is
Brier score), (p(xr) . Recalling that in standard confidence calibration have c(g(X))
argmax, 9¹ (X) and h(g())) max; 9 (X). our goal is then to investigate the relationship between (ld, h(X)), h())) and (Ed
g(x))
Prior to doing so, let us pause to stress the fact that the confidence calibration framework should be applied only to the calibration error and not to the entire loss. In other words, we should report the full Bregman divergence Elda(Ty) (Ty|x x 1 (X)) in addition to the calibration error, not the 1-D Bregman divergence ([d ( =c)) h( (X))]. Reporting of the latter is another pitfall and has appeared in the
recent literature on surveying confidence elicitation strategies for LLMS (Tian et al.) (023). We show in the next proposition that the I-D loss have very different behavior when compared to the full loss for the specific case of MSE/Brier score.
Proposition 4.5. Suppose satisfies the property that there exists y² : A + [k] such that
Trix= (3"(x) = 1 for xx-a.e. x. Now consider any function ÿ A A + [kc] satisfying Ú(x) + y² (x) for ax.e. Then the "always wrong" " predictor defined as grlle) 1/k and (x) = 1/k c/(k for all y + ((x) with c = =O(1/k) satisfies
E [11==ax)- - M(X)) = 0(1/k) but E [| (xXTxx--(X)11- x(x)|
22(1)
(4.5)
for the standard confidence calibration choices c(z) argmax, 24 and h(2) max; 24.
The proof follows from directly evaluating (4.3 and can be found in Appendix B Proposition 4.5 shows that the loss de incurred on the confidence calibration problem can be arbitrarily small for predictor that is always wrong despite such a predictor having true loss the multi-class classification problem bounded helow by a constant. We now return to our original goal, which was I(X)), h(X)) is lower bound for R[d (B(Tyy) x ((X)), g (X)) under the condition that Φ dominates yp applied to any individual coordinate pair (for example, this is easily seen be the case for Brier score).
showing that
Lemma 4.6. For any C², convex functions : A R and y [0, + K such that their for Bregman divergences satisfy do v) 2 do vi) for all i E [k]], it follows (with probability 1) that:
(E(TTI (g(X)) M(X)) 2 (E ((X)),
(4.6)
Corollary 4.7. The inequality 4.6p holds for the choices 6(x) = Σ, De log x, (KL divergence) and <p(x)) = x log x + (1 x:) log(1 .)) (logistic loss).
The proof of Lemma 6..6reeliss on a careful analysis of the conditional expectation terms, and can also be found in Appendix B Corollary follows from the data processing inequality.
4.2 CALIBRATION-SHARPPESSS SHARPNESS DIAGRAMS
To summarize our observations thus far, we have recommended that calibration should be compared using the calibration metric associated with the Bregman divergence for which the models were optimized, and that both should be reported together. It remains to address how to similarly modify the visualization of (confidence) calibration, which is typically done via reliability diagrams that plot E[ly=e(x)) h [X) for different values of h(X). Our goals are three-fold: we want a visualization that is efficient to compute, makes it obvious whether one model is better calibrated and/or has better overall divergence than another model, and allows one to pinpoint the regions of the predicted
probability space where model performance is weak. While there are few recent works that try to
7

---

Published as a conference paper at ICLR 2025
make progress on these goals, we not aware of solution presented as single diagram satisfying our desiderata (Dimitriadis et al. 2023 Gneiting & Resin 2023).
The approach we take is to augment the plot of E E(dv=((X) h(X)] with surrounding band whose length p(h(X)) corresponds to the sharpness gap from Lemma 4.1 at h(X), which we refer to as calibration-sharpees diagrams. For convenience, we will use p(X) in place of p(h(X)) when is understood. The function [0, 1] > the pointwise sharpness gap, and formally defined as:
p(p) E [ds (TX ((X) ((X) p] de (E[Dy=c(X) l(X) pl.p),
(4.7)
where as in the previous subsection da is the full Bregman divergence and de is its 1-D counterpart.
Note that in (4.7) we are conditioning dæ [ (xx()(()) on l(X), since want value for the divergence at different confidence levels ((X). The key property of p which allows it to be used as
visualization aid is the following.
Proposition 4.8. Suppose that & and up satisfy the conditions of Lemma 4.6 Then the function p as defined in (4.7) is nonnegative.
The proof of Proposition 4.8 (which can be found in Appendix 4.2 follows from applying Lemmas 4.1 and Lemma [46) The function p shows how we trade off calibration with generalization at any confidence level p. and we can compute p efficiently as it only requires estimating 1-D conditional expectations.
Our estimation of p relies on Nadaraya- -atsonn kernel regression (Nadaraya) 1964 Watson 1964)
with kernel K and bandwidth hyperparameter a following the estimation approaches of Blasiok & Nakkiran (2023) and Chidambaram al. (2024). We recall that given n data points (Iz y/) E R x R is defined as:
the kernel regression estimator of E
xr
Σ n
Ko -G) Tz)
Ma (a)
==
(4.8)
Ka(x Fi.
Previous work (Zhang et al. that we can obtain consistent estimators of L ECE using 4.8). We extend these results to calibration errors derived from Bregman divergences. Due to the technical nature of some of the assumptions necessary on K₂ for this result, we defer a proper presentation to Section [.2 and present only an informal version of the result below. The proof relies uniform convergence results for kernel regression applied to the individual terms in (4.1).
2020 Popordanoska et al.
2022
Chidambaram et al. 2024 has shown
Theorem 4.9. Under suitable conditions K and the sequence of bandwidths on, the plugin estimate using fi (h(x)) of the confidence calibration error with respect to d is consistent so long as h(X) has density bounded away from 0 and [[1y=m(X) h(X)) is continuous.
Constructing and reading calibration-sharpees diagrams. Using the above, we construct a diagram with three components: the calibration curve, the pointwise sharpness gap p(h(X)), and the density plot Pr(x)- Example diagrams are shown in Figures and 2 and interpreted in detail in Section The calibration curve corresponds to E[ly=<(a) h[x)) plotted at uniformly spaced values of h()) E [0, 1), and shows the conditional model accuracy vs. its confidence. A perfectly confidence
confidence-calibvatee calibrated model satisfying [E[1y yc(x)) h(x)] h(x)) is visualized as dashed line, and
any deviations from this dashed line can interpreted as weaknesses in model calibration.
The sharpness gap is plotted as band around the calibration curve, and the size of the band corresponds to (4.7) evaluated each h(()). A larger band is worse, as it means a bigger sharpness gap at predicted confidence h(x),, which implies a worse generalization error at that particular confidence level. We also plot a kernel density estimate PK(X) of the predicted confidences this intended to show where the model confidences concentrate. A density peak at h(x:)) 0.8, for example, indicates that the model typically predicts around 80% confidence for its predicted class.
We point out that another common pitfaall in several prior works is to omit this kind of density plot when discussing calibration, since it does not matter if a model is perfectly calibrated in region of its confidence space that it rarely predicts. Lastly, we also overlay the estimated values of the confidence calibration error (d,p, CAL) and the full divergence (d,,ror) all plots, as give sense the full generalization eтror of the model as well as the part attributable to calibration. More precise estimation details for each component of the diagram are provided in Appendix
8

---

Published as a conference paper at ICLR 2025
5 EXPERIMENTS
To illustrate the benefits of calibration- -sarrpptsnnnssspssssssss diagrams, we first use them to revisit the experiments of and then provide direct comparison to the standard approach of reliability diagrams. For
all of the experiments in this section, we focus on the ViT model of Section 3 for consistency and use
Brier score/MSE since we will be comparing to ECE-based visualization methods in Section 5.2 We provide visualizations for other models and datasets in Appendix [ The kernel regressions estimates for the visualized components are computed using a Gaussian kernel with bandwidth 0.05; we discuss different choices of kernel and bandwidth in Appendix All our experiments were done
in PyTTorch (Paszke et al. 2019) single A5000 GPU.
5.1
REVISITING MRR EXPERIMENTS
Condrons Expictatio eTV -.0
(eass 1 a
u 0.00
0.030 0.0
22.22.
27.27
Condioonal Expectationn of
Condiranal Expectattion
Flnket)
Confidence (6O)
Coniidecc (Nxd
Confidance MM)))
(a) ViT (Baseline)
(b) ViT TS
(c) VIT + HB
Conditona (D)
S 8:30
Cu 0.00
de w
Ae 101 21.31
Confidence
Confidance
(d) VIT + IR
(c) VIT + MRR
Figure 1: Proposed calibration-sharpne diagrams using MSE/Brier score for the experiments of Section Y-axes are not aligned here due to the large sharpness gap of MRR.
Figure shows the calibration-ssarpness diagrams (using (z y) = || yll for the ViT recal- ibration approaches Table I The first thing to notice is that the calibration-shappnes diagram immediately reveals the weaknesses of both MRR and histogram binning these methods using small fixed set of predicted confidences (a single confidence in the case of MRR), and as result incur large sharpness gaps shown by the red shaded regions.
Additionally, when comparing the better recalibration approaches of temperature scaling and isotonic regression to the baseline model, we see that these algorithms have actually (perhaps counterintu- itively) skewed the predicted confidence distributions towards even higher probabilities (as shown by the overlayed density plots of ( (h(z))). Moreover, gap visualizations that the calibration improvements from isotonic regression (and to a lesser extent, temperature scaling) have come the cost of some increased sharpness gaps. This can be expected from the fact that both methods led to larger predicted confidences, so mistakes incur larger errors.
from the pointwise sharpness
5.2 COMPARISON TO RELIABILITY DIAGRAMS
The standard approach taken by many empirical papers analyzing confidence calibration is to report reliability diagrams. These diagrams correspond to histogram binning estimators of the calibration error namely, sort the predicted confidences l1xx)) into uniformly spaced buckets in the interval [0, and plot bar corresponding to the average accuracy in each bucket. Similar to the dashed line in our calibration-sharpeess diagrams, perfectly confidence calibrated models would have bars that line up with the average confidence of each bucket (i.e. the height of the bar for the [0, 0.1] bucket should be somewhere in [0, 0.1)).

---

Published as a conference paper at ICLR 2025
de 00330
0.001 Ug 27.37
0.20 0.00
12
24.13
1.3
3:2
0--
Conditinal
((Nx))
p(nix))
Confidence (hirl)
Confedence (hict)
(a) HB, Calibration ssaibrationShharnesss
(b) IR, Calibration Calibration-Sharpeess
Deliabilty
Raliabiltty Diagram
Girgut
Outout
(c) HB, Reliability
(d) IR. Reliability
Figure 2: Comparing aaliraatonnssharpees diagrams to reliability diagrams using histogram binning and isotonic regression. Y-axes of HB and IR are aligned for more direct comparison.
Figure 2 shows reliability diagrams (using 15 bins, as was done in Guo et al. (2017)) for the histogram binning (HB) and isotonic regression (IR) recalibration approaches juxtaposed with axis-aligned cnibrationnsharppess diagrams from Figure [ The reliability diagrams suggest that IR leads to better calibration than HB the confidence range around 0.2 and in [0.5, 0.9). However, these diagrams cannot be used to compare generalization and also do not give sense of where predicted confidences concentrate.
On the other hand, the calibration calibration-harpneess diagrams allow to compare generalization while providing a more granular view calibration performance as well. They show that (1) while the calibration of HB is worse than IR, the confidence region around 0.2 is not major problem for HB since there is no confidence density there and (2) that IR has better generalization performance (with respect Brier score) due smaller sharpness gaps.
Our work is not the first to point out these drawbacks of reliability diagrams for comparing calibration; the work of Blasiok & Nakkiran (2023) on SmoothECE also discusses such issues and puts forth visualization that includes density plots combined with kernel regression estimate of the calibration eтror as opposed
binning estimate. However, their approach does not include sharpness component, which in this instance is crucial for comparing HB and IR since this is where HB really loses out. We compare to the SmoothECE visualization approach Appendix G
CONCLUSION
In this work, have reviewed the reporting of calibration metrics in the recent literature and identified several pitfalls that appear even in well-known works. To address the pointed out issues, we have proposed a reporting methodology based on proper scoring rules that are Bregman divergences, and shown in both theory and practice how our approach fits alongside the popular notion of confidence calibration in the literature. In developing our methodology, we have also introduced a new extension of reliability diagrams for visualizing calibration.
The core tenets we hope to emphasize through both our theory and experiments are: report test accuracy and calibration emror alongside the appropriate generalization metric, use visualization of calibration that shows the density of predicted confidences, and be aware of whether improvements in calibration in some regions of predicted confidence come at the cost of We believe there is still much work to be done on addressing how to compare and visualize calibra- tion/generalizatioon metrics, and hope future work continues to push in this direction.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
While calibration has become an increasingly important topic given the use of machine learning models for decision-make makinn. our work concerns only the theory and practice of reporting calibration. As result, we do not anticipate any negative societal impacts or significant ethical considerations as consequences of this work.
REPRODUCIBILITY STATEMENT
Full proofs of all results in the main paper can be found in the appendices following the references. The code necessary to recreate the experiments in the main paper as well the experiments in the appendices can be found in the supplementary material, as well as in the Python package sharpcal
ACKNOWLEDGMENTS
Rong Ge and Muthu Chidambaram supported by NSF Award DMS- MMM-2031849 and CCF-1845171 (CAREER) during the completion this work.
REFERENCES
Imanol Arrieta-lbarra, Paman Gujral, Jonathan Tannen, Mark Tygert, and Cherie Xu. Metrics of calibration for probabilistic predictions. arXiv preprint arXiv:2205.0965 2022.
Jean-Daniel Boissonnat, Frank Nielsen, and Richard Nock. Bregman voronoi diagrams. Discrete and
Computational Geomerry, 44(2), 2010. ISSN 1432-0444. doi: 10.1007/500454-010-2222 URL
http://dx doi wttp/o/g.o........ 1017:1000000000000000 010- 9256-1
Jochen Bröcker. Some remarks on the reliability of categorical probability forecasts. Monthly weather review, 136(11):44884452, 2008.
Jochen Bröcker. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Sociery, 135(6433)1512--5113 July 2009, ISSN 1477-870X. doi: 10.1002/gj.456. URL http: www. org/ 15. wtt.oo...//..1111...
Jaroslaw Blasiok and Preetum Nakkiran. Smooth ece: Principled reliability diagrams via kernel smoothing, 2023.
Jaroslaw Blasiok, Parikshit Gopalan, Lunjia Hu, and Preetum Nakkiran. A unifying theory of distance from calibration, 2023.
Muthu Chidambaram, Holden Lee, Colin McSwiggen, and Semon Rezchikov. How flawed is ece? an analysis via logit smoothing, 2024.
A Philip Dawid. The well-calibrated bayesian. Journal of the American Statistical Association, 77 (379):605-610, 1982.
Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation forecasters. Journal of the Royal Statistical Society: Series 1) (The Statistician), 32(1-2):12-22, 1983.
Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 295-302, 2020.
Lue P. Devroye. The uniform convergence of the nadaraya- regression function estimate. The Canadian Journal of Statistics / La Revue Canadienne de Statistique, 6(2):179-191, 1978. ISSN 03195724. URL httt::tttt///tt jstor. wwtp://tll/gg/ooooooo.oooooc
504
Timo Dimitriadis, Tilmann Gneiting, Alexander Jordan, and Peter Vogel. Evaluating probabilistic classifiers: The triptych. arXiv preprint arXiv:2301.10803 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICI.R, 2021.
11

---

Published as a conference paper at ICLR 2025
Haitham A Elmarakeby. Justin Hwang, Rand Arafeh, Jett Crowdis, Sydney Gang, David Liu, Saud H AlDubayan, Keyan Salari, Steven Kregel, Camden Richter, et al. Biologically informed deep neural network for prostate cancer discovery. Nature, 598(7880):348 352, 2021.
Tilmann Gneiting and Adrian Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359-37,, 2007.
Tilmann Gneiting and Johannes Resin. Regression diagnostics meets forecast evaluation: conditional calibration, reliability diagrams, and coefficient of determination. Electronic Journal of Statistics, 17(2):2226 3286, 2023. doi: 10.1214/23-EJ52180, URL 23- 23-EUS2180
Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society; Series B (Statistical Methodology), 69(2): 243-268, 2007.
Parikshit Gopalan, Lunjia Hu, and Guy N. Rothblum. On computationally efficient multi-class calibration, 2024.
Sebastian G.
Gruber and Florian Buetmer. Better uncertainty calibration via proper scores for classification and beyond, 2024.
org/ /a////b//////
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Chirag Gupta and Aaditya Ramdas. Top-label calibration and multiclass-to-binay reductions, 2022.
László Györfi, Michael Kohler, Adam Krzyžak, and Harro Walk. A distribution theory of nonparametric regression. In Springer Series in Statistics, 2002. URL https://api /semanticschola= htt.://a...ro/oooorro11::3311110
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015. URtp::////axxi org wttgabs/////..././//////// 03385
Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du. Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning-oriented autonomous driving, 2023.
Zhenghao Jiang, J. Araki, Haibo Ding, and Graham Neubig. How can we know when lan- guage models know? on the calibration of language models for question answering. Trans- actions of the Association for Computational Linguistics, 9:962-977, 2020. URL https
Tom Joy, Francesco Pinto, Ser-Nam Lim, Philip H. S. Torr, and Puneet K. Dokania. Sample-dependent adaptive temperature scaling for improved calibration, 2022.
Sham Kakade and Dean Foster. Deterministic calibration and nash equilibrium. Journal of Computer and System Sciences, 74(1):115-134) 2008
Meelis Kull and Peter Flach. Novel decompositions of proper scoring rules for classification: Score adjustment as precursor to calibration. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 68-85. Springer, 2015.
Meelis Kull, Miquel Perello-Nieto, Markus Kängsepp, Hao Song, Peter Flach, et al. Beyond temperature scaling: Obtaining well-calibrated multiclass prohabilities with dirichlet calibration. arXiv preprint arXiv: /910. 12656, 2019.
Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. In Advances in Neural Information Processing Systems, pp. 3792-3803, 2019.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In International Conference on Machine Learning, pp. 2805-2814, PMLR, 2018.
12

---

Published as a conference paper at ICLR 2025
Donghwan Lee, Xinmeng Huang. Hamed Hassani, and Edgar Dobriban. T-cal: An optimal test for the calibration of predictive models. arXiv preprint arXiv:2203.01890 2022.
Sarah Lichtenstein, Baruch Fischhoff, and Lawrence D. Phillips. Calibration of probabilities: The state of the 1980, pp.
Cambridge University Press, 1982.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Alireza Mehrtash, William M. Wells, Clare M. Tempany, Purang Abolmaesumi, and Tina Ka- pur. Confidence calibration and predictive uncertainty estimation for deep medical image segmentation. IEEE Transactions on Medical Imaging, 39:3868-3878, 2019. URL https 7/api. semantsemanteo wescchccarrrcc wttp///oooooDSS00011110000
Jacob A. Mincer and Victor Zarnowitz. The Evaluation of Economic Forecasts. In Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance, NBER Chapters. pp. 3-46. National Bureau of Economic Research, Inc, May 1969. URL https://ideas
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 14:15582-15694, 2021.
Allan Murphy, new vector partition of the probability score. Journal of applied Meteorology, 12 (4):595 600, 1973.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help?, 2020.
E. A. Nadaraya. On estimating regression. Theory of Probability & Its Applications, 911:1111-42, 1964. doi: 10. 1117/1109020 URL https::/p/// http:::://bi...r//0..11111111111
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Binary classifier calibration: Non-parametric approach. arXiv preprint arXiv: 1401.3390, 2014.
Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. In CVPR Workshops, volume 2, 2019.
Alberto Nogales, Álvaro García- García-Tejedor, Diana Monge, Juan Serrano Vara, and Cristina Antón. A survey of deep learning models in medical therapeutic areas. Artificial Intelligence in Medicine, 1 12: 102020, 2021. ISSN 0933-3657.
doi: http://ww.....ooooooooooo.oooooa
URL BRLhttps
www. www.sciience..rretttt wwcteeccccdee.cccccooo wwtp:/ecttccccc.ieeeee..oo
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Z. Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, perf-performance deep learning library. CoRR, abs/1912.01703, 2019. 01703
Alexandre Perez-Lebel, Marine Morvan, and Gaël Varoquaux. Beyond calibration: estimating the grouping loss of modern neural networks, 2023. URL
16315
John C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In ADVANCES IN LARGE MARGIN CLASSIFIERS, pp. 61-74. MIT Press, 1999.
Marc-Oliver Pohle.
The murphy decomposition and the calibration-rcsolution principle: A new perspective forecast evaluation, 2020.
Teodora Popordanoska, Raphael Sayer, and Matthew Blaschko. A consistent and differentiable Ip canonical calibration error estimator, Advances in Neural Information Processing Systems, 35: 7933-7946, 2022.
13

---

Published as a conference paper at ICLR 2025
Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C Mozer. Mitigating bias in calibration error estimation. In International Conference on Artificial Intelligence and Statistics, pp. 4036-4054. PMLR, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-252, 2015. doi: 10.1007/s1 1263-015-0816-y
Andreas Steiner, Alexander Kolesmikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint ar 106. 10270, 2021.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105-61 14, PMLR, 2019.
Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 32: 11888-1388833 2019.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback, 2023.
A. Tsybakow. Introduction nonparametric estimation. In Springer Series in Statistics, 2008. URL semant.etttt..e https://api. stttp:::/api.ssmmnttii www.emnnniiscchhooooo wcscholar.....o.......co.cc....e http::/pw....ottt...o..ooorrrrrp ID :2933599
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 3459-3461 PMLR, 2019.
Deng-Bao Wang. Lei Feng, and Min-Ling Zhang. Rethinking calibration of deep neu- ral networks: Do not be afraid of overconfidence. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural In- Curran Associates, Inc., 2ttppp/r__.eeespppee/2221111 2021. URL httt:s:://procceding..... neur ottppp/eeeeeeeeeerrrereo/oe/epee
formation Processing Systems, volume 34, pp- 11809- 11820.
File/ 61 Fawaaaacccccoeeeeeeeeee3eee
Geoffrey S. Watson. Smooth regression analysis. Sankhyā: The Indian Journal of Statistics, Series (1961-2002), 26(4):359-372, 1964. ISSN 0581572X. URL http: wwwp://wwwww//.m// http://wwwwwttoooorrorr stable/25049340
Yeming Wen, Ghassen Jerfel, Rafael Muller, Michael W. Dusenberry, Jasper Snoek, Balaji Lak- shminarayanan, and Dustin Tran. Combining ensembles and data augmentation can harm your calibration, 2021.
David Widmann, Fredrik Lindsten, and Dave Zachariah. Calibration tests beyond classification. In International Conference Learning Representations, 2020.
Ross Wightman. Pytorch image models.
fttps:///gtthu..com//w
pytorch-imase iaapi-immaeeemae model 2019.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml, volume 1, pp. 609-616. Citeseer, 2001.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probabil- ity estimates. In Proceedings of the eighth ACM SIGKDD international conference Knowledge discovery and data mining, pp. 694-699. ACM, 2002.
Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In International conference on machine learning, pp. 11117-11128 128. PMLR, 2020.
14

---

Published as a conference paper at ICLR 2025
Shengjia Zhao, Michael P. Kim, Roshni Sahoo, Tengyu Ma, and Stefano Ermon. Calibrating
predictions to decisions: A novel approach to multi-class calibration. In Advances in Neural Information Processing Systems, 2021. URL httts::/openree/// net ffrummiddiii
15

---

Published as a conference paper at ICLR 2025
CONTENTS
1
Introduction
1.1
Summary of Main Contributions and Takeaways
1.2 Related Work
2
Preliminaries
3
Pitfalls in Calibration Reporting
4
Main Theoretical Results
4.1
Confidence Calibration With Bregman Divergences
4.2
Calibration-Sharpeess Diagrams
5
Experiments
5.1
Revisiting MRR Experiments
5.2
Comparison Reliability Diagrams
6
Conclusion
10
Background Results
17
Proofs for Section 4.1
18
Proofs for 4.2
18
Calbbration-Sharpness Diagram Details
20
Detailed Comparison to Prior Work on Grouping Loss
20
Additional Experiments With Alternative Models
F.1 Further ImageNet Experiments
21
F2 CIFAR Experiments
Comparison SmoothECE
23
H
Effect of Kernel Hy Hyperparameters
25
H.1
The Impact Kernel Bandwidth
25
H.2 The Impact of Kernel Choice
26
16

---

Published as a conference paper at ICLR 2025
A BACKGROUND RESULTS
Lemma 4.1. [Bregman Divergence Decomposition) For dæ defined as in (4.1 and two integrable random variables Z and X defined on the same probability space, it follows that:
Eids (2, - [[o(Z)] - (E X
de (E X).
(4.2)
Sharpness Gap
Calibration Error
Proof. We observe that by 4.1
have:
(BIZ x), F. ( [ X])] B(((X)) B[Vo(X) [B[2 X] X)]
[ X))] E[)(X)) E(V T (Z -X)-
(A.1)
Where the last line follows from the fact that | X]] Y| Z] for any X-measurable random variable Y. Equation (42) then immediately follows from applying definition 4.1
Proposition 4.2, For 91, 92 Rd + A and a the joint distribution of data and label (X,Y) on Rd [Be,, if (X)(() C (X)). then:
Elo(B|Ty) 9x(x) < Elo(E[TYix (x(())]
(4.3)
Proof. By Jensen's inequality and the tower property of conditional expectation, we have:
Elø(E|xyIx (x() = [E (x())) g (X)]]
N (E 9x(()] 91 (X)]I
9 (X)))-
Proposition 4.3, Let π correspond to the joint distribution of data and label (X, Y) on R2 x [k]. Then every predictor g that calibrated the sense of 2.1 satisfies:
Elda(Ty(x.g9XD))
(4.4)
Proof. Firstly, observe that for any calibrated predictor we have E(d (xX)))
Next, we note that for every g Rd + k g(X) is measurable with respect to the o-algebra generated by the constant predictor xy (since this is the trivial o-algebrr). The inequality (4.4) then follows by applying Proposition 4.2 to the decomposition from Lemma 4.1
Proposition 4.4. Let π correspond to the joint distribution of data and label (X, on Rd x
[k). and define the L (for integers q 2 1) ECE of a function Rd - A--1 be
E | 1(x) ((x) Then only the T,2 ECE is induced by a Bregman divergence in the sense of Lemma 4.1
Proof. We first note that the case of q = 1 is not Bregman divergence since Bregman divergences are differentiable in their first argument. For q 1, Lemma in Boissonnat al. (010)) shows that a twice differentiable function ø induces a symmetric Bregman divergence de the Hessian of constant. This implies that for any L norm (raised to the q power) to be Bregman divergence, differentiating it twice in the first argument must yield constant. It is easy to see that this is only the case for a 2, and the result follows.
17

---

Published as a conference paper at ICLR 2025
B PROOFS FOR SECTION 4.1
Proposition 4.5. Suppose satisfies the property that there exists y' A R--1 - [kc] such that
Tyix=r (x)) = 1 for xx-a.e. I. Now consider any function y A k-1 7 [k] satisfying
ū(z) # (z) a.e. Then the "always wrong" predictor defined as g (x) 1/k + and
(z)x) 1/k e/(k: for all y + ((x) with = =((//) satisfies
(1y=a(x) - ((x)) -01//B) but || 9(x) 22(1)
(4.5)
for the standard confidence calibration choices c(z) argmax, ž; and h(=) = maxx; A/.
Proof. Observing that Ly=((X) and h(X) 1/k gives the first part of 4.5). On the other
hand, the leading term of E|| TT|x [()]|1 (1 1/k: + c/(k 1))2, which is (11)..
Lemma 4.6. For any C2, convex functions k R and yp [0, 10 R such that their Bregman divergences satisfy dz(x, v) 2 dzp Ui) for all E [lt], it follows (with probability 1) that:
(EdT(E ((())).((X)) h(X)) 2 d [E[ [E(ly=((x)) (x))),(()))
(4.6)
Proof. Firstly, recalling that (e(X) denotes the basis vector in I2t with non-zero index c(X),
=| I g(X)] [[[[[L==(x))
X g(X)]
= [E Ty x g(X)]
B TT [x(]],
(B.1)
where the last line follows due to the measurability of Ex() with respect to g(X). By the assumptions on dæ and d and the fact that h(X) (x(x))((X) we have:
dg (B[Tyi 9(X)),,((X) 2 dB( 1 ((X))) de B ((X)),
= dE ((x)),
(B.2)
The desired (46)) then follows by noting that
(E (((X)),n(X)) h(X) = (E|p(E[11))))) (()))) |(X)] - ((((x))
Bip (M()) [[(X] -(()) ((x)]
2 [( h(X))) (h(())
- (M h(X)) h(x))
(B.3)
due to Jensen's inequality and h((X)mmeauurability of (X()))..
Corollary 4.7. The inequality 4.6p holds for the choices 6(x) Σ 2y log v, (KL divergence) and (p(x) = r log r + (1 x)) log(1 x) (logistic loss).
Proof. We need only show that do(2,,) v) do( yi) for the choices of KL divergence and logistic loss. Let usual identify and with vectors in R corresponding distributions in A k-1 Then, fixing an E [k]. the map f(x) = (ernoulli(z;) is a Markov kernel, and by the data-processing inequality we have do(x,y) v) Z do(f(x), f()). Since ds(f(x) f()) (2, yi), the result follows.
C PROOFS FOR SECTION 4.2
Proposition 4.8. Suppose that & and satisfy the conditions of Lemma 4.6 Then the function p defined in (477 is nonnegative.
18

---

Published as a conference paper at ICLR 2025
Proof. From Lemma 4.1 we have:
E [dd(TTyxxx(X)) ((X) p] 2 E [Bx(x 19(X)), [(X) =p]
(C.1)
Applying Lemma then yields the result.
To state and prove a formal version of Theorem 4.9 we need a result of Devroye (1978). We first state the necessary kernel and distributional assumptions for this result. We state everything in the generality of IR as that is how they are presented in Devroye 1978). but for our particular result we only need the I-D analogues.
Assumption C.1 (Kernel Assumptions), We assume the kernel K Rd -> IR satisfies the following:
1. ((x) E [0, K*] for some K < oc.
2. ((x [[ [] for some nonincreasing function L.
3. lima-+00 uu( O.
4. L(u*)= > 0 for some u > 0.
Assumption C.2 (Distributional Assumptions). We assume the joint distribution of (X, Y)
R x R satisfies:
1. has compact support and a density xx that bounded away from O.
2. Y E|Y X] is almost surely bounded.
3. A version of E X] is bounded and continuous on supp(*x)-
In some cases above, have stated slightly stronger forms of the assumptions in Devroye (9788 for brevity, and also because our application relatively simple and works under these stronger assumptions.
Theorem C.3 (Theorem 2 in Devroye 1978), Paraphrased). Let K be kernel satisfying Assumption
C.7 with a sequence of bandwidths on, and suppose a satisfies Assumption C.2 Then if o + 0 and non
2d
log oo, we have uniform convergence of the kernel regression estimator Ma..
sup |na, (x) - -E X = 0 a.s.
(C.2)
=Esupp(xx)
We now state and prove the formal version Theorem 4.9
Theorem C.4. Let mn be in (4.8 Given sequence of bandwidths ono define the plugin
estimator d.p. as follows:
dp.n 1
Σ YI
n
p( (h(x:))) ((a))))))) (1:=(())))) M(x))).
(C.3)
Suppose K. satisfies Assumption C.1 and an satisfies on + O and no /lgn - 0o. If we addi- tionally have that E(Ly=e(X) bounded away from zero, it follows that:
h(X)= = h(x)] continuous in h(x) and that l(() has density
dp. + E[d (E(ly=e(X) ((()),)(X))
(C.4)
in probability.
Proof. By definition and properties of conditional expectation (covariance matching, the same trick we used in the proof of Lemma 4.J), we have that:
E dd (E(1y=e(X) (xX), (x))),CC k(x)] = El (E [(X))) (M())(
Bl (Mx)( - M(X)))
(C.5)
is immediate from the law of large numbers that the second and third terms in their counterparts in (.5)). It remains show the convergence of the first term of
converge to
19

---

Published as a conference paper at ICLR 2025
We observe that since ly o( 1, along with the fact that have assumed o h(()
[x)] is continuous in h(x:) and that ((X) (X) has a density bounded away from zero, we satisfy all of
the conditions of Assumption C.2 Thus we have uniform convergence of mi. from Theorem
It follows from uniform convergence and the boundedness of =e]x=)(X) [(()] that m (h()))
is constrained to a compact set for n sufficiently large. Since yp is uniformly continuous on a compact set, then have that:
R
(mm (hb(z)))
[(x)))) + 6n
(C.6)
for some 8n as + The result C.4 then immediately follows.
D AALIBRATION-SHARPEEN DIAGRAM DETAILS
Here we go over the fine-grained details of constructing calibration-sharpness: diagrams, beginning with estimation of the conditional expectations involved.
Estimation details. We estimate the term E [ (()))( [(x] in Equation 4.7) by replacing (ri,31 y:) data point pairs with (rz, de(en. ((x)))) do(e)..((a))) encoded label yΙ) and then using the kernel regression estimator (4.8). The same is done with =((x= I)()), except the label is replaced with pointwise accuracy, i.e. 11 ===(=)))- We also estimate the density Ph(x) using kernel density estimate (this is just the denominator of the kernel regression (4.8) with the same kernel, and refer to this estimate as Pk(X).
(i.e. the divergence of g(aze) from the one-hot
With these estimates, we can estimate the pointwise sharpness gap p(h(z)). We visualize this gap as band whose length is pn(x)(((x))) (h(z))p(h(zz) centered around the calibration curve E[4y=<(x) (=)]] Note that scale the band by Pa(x) P((x)(((a)) (h(x)). as this is important for making it visually obvious where most of the sharpness error is concentrated and also allows the aggregated total sharpness gap area to better correspond to the actual total sharpness gap.
Visualization details. Our kernel regression estimates for the diagrams in Figure ! were computed using a Gaussian kernel with o 0.05 (different choices of kernel and bandwidth are discussed in Appendix HD. Due to the size of ImageNet- 1 K Val, we also replace the full kernel regression estimate of (4.8) with a mean over 10 estimates in which we randomly subsample the data to 5000 data points, so to fit the entire computation into memory. The standard deviations of these estimates is shown alongside the mean calibration error scaled by 100 (dg,CAL all of the plots); in all cases they are negligible, and we checked that subsampling loses virtually nothing. Lastly, for stylistic reasons, opt to threshold the curve defining the bottom of the sharpness band at O, but do not threshold the top curve 1. In Figure and the plots of Appendix H we do not align the Y-axes of the plots due to the large differences in sharpness gaps between some methods. However, for direct comparisons between approaches (or handful of similar approaches) recommend aligning the Y-axes as we have done in Figure
note on estimating conditional expectations. We have used the Nadaraya- Watson estimator for the conditional expectation terms in all of the above as it is the standard choice for nonparametric estimation and known to be minimax optimal under certain technical conditions on the data distribution and the choice of kernel (Tsybakov) 2008). However, if there is some prior knowledge of the parametric form of EJY g(X)). then one can of course achieve better estimation results using such form, but we expect such forms would not be known in most applications. For further details regarding the asymptotic properties (bias, consistency, convergence rates) of Nadaraya- Watson style estimators recommend Tsybakov 2008 and Györf et al. (2002).
E DETAILED COMPARISON TO PRIOR WORK ON GROUPING Loss
We have focused on using the sharpness-calibration decomposition of Bregman divergences to better understand model performance, but there exist other decompositions in the proper scoring rule literature that can provide different perspectives on performance. In particular, Kull & Flach (015) proposed decomposition of divergences derived from proper scoring rules that splits the total divergence into calibration error (like we have in Lemma 4.1) as well as grouping loss and irreducible loss.
20

---

Published as a conference paper at ICLR 2025
To make these notions precise, let for simplicity consider binary classification problem with Y E {0, 1 E R. and some classifier 9 Rd + [0, 1]. In this context, we recall that the calibration condition is simply E g( = g Now let us further define C' = E[Y g(X)] and Q = B X]- Kull & Flach (2015) showed that, for any divergence dø derived from proper scoring rule (to keep background discussion brief, one can just think of this one of the Bregman divergences we consider in the main paper), we have:
Eids (X, = (Y) + [ldo(Q. + (C,
(E.
Trreducible Loss
Grouping Loss
Calibration Error
Here the irreducible loss captures the uncertainty inherent to the problem as does not depend on the classifier g. while the grouping loss captures the deviation of the conditional expectation using the predicted confidences from the true conditional expectation derived from the underlying data. Intuitively, the grouping loss measures how much information from the data is "grouped" together by our classifier - for example, if g is a bijection, then this term is zero since (XX generates the same o-algebra as X. Our interpretation of sharpness "resolution" or "fine- grannengrinddeesss of the model can be attributed to this grouping loss term, as also discussed in Pohle (2020).
The key idea in the work Perez-Lebbl al. (2023) is that this grouping loss term should be reported alongside calibration error and other metrics, and they provide similar examples to what we provide in the main paper for how a model can be calibrated and have perfect accuracy but not recover the ground-truth distribution. The main results of Peerez-Lebel et al. (2023) consequently regard the estimation of grouping loss as well as joint visualizations of the grouping loss and calibration error.
In the multi-class setting, the results Perez-Lebel et al. (2023) are applied directly to the confidence calibration problem. However, our results in Section 4 tie together the full multi-class problem with the confidence calibration problem; this allows
not reduce the entire problem to confidence calibration (where the labels are binary and the probabilities are 1-D). This is important because ultimately we care about generalization metrics in the context of the original multi-class problem, and the purpose of Lemma 4.6 was to show that we can simply replace the full, high-dimensional calibration error with its 1-D) confidence calibration counterpart while still maintaining reasonable interpretation.
Additionally, by working with the sharpness gap as opposed to grouping loss and irreducible loss, we need only concern ourselves with the estimating the conditional expectation E[y g(X) as opposed to also considering [E[Y X]. The inclusion of the irreducible loss in the sharpness gap is not an issue; as this is shared by all models, it does not have an impact on how we rank them relative to one another. To summarize, our approach differs in that we tie the calibration cuabration-shappesss decomposition in the full calibration setting confidence calibration as opposed to working with grouping loss in the confidence calibration setting, which leads to casier estimation and maintains important properties of the original multi-class problem.
F ADDITIONAL EXPERIMENTS WITH ALTERNATIVE MODELS
F.1 FURTHER IMAGENET EXPERIMENTS
We focused recalibrating pretrained VIT model on ImageNet throughout the main paper for consistency across sections; here we consider different model backbones applied to ImageNet and recreate the results of Table and Figure for each of the different models. As with the ViT model, we use pretrained models from the t timm library that have strong performance on ImageNet. In particular, we consider ResNet-50 (He al.) 2005). EfficientNet TTan & Le (201)), and ConvNext (Liu et al. 2022) backbones. Final tabular results are shown in Tables 2 to and corresponding plots are shown in Figures to
F.2 CIFAR EXPERIMENTS
We used ImageNet in the main paper due to the plethora of high quality, pretrained models available via open source libraries. To show that our observations extend to other datasets, however, we also include experiments on CIFAR-10 and CIFAR- 100. In particular, we use the pretrained ResNet-32 models of Yaofo Chen (https: /qithub. com/ www.://afoo/yooooc-ifaaaae
and calibrate them using 20% of the CIFAR- 10 and CIFAR- 100 test sets to be consistent with our ImageNet experiments. Numerical results are shown in Tables 5 and and plots are shown in Figures
21

---

Published as a conference paper at ICLR 2025
Condasrna Ereeerii Eroenitin OY 000
Sucx 20.00
Meci +U3
denen 28
Reiorn 32.54
Contiroaal
61hom
Pinikl
(a) ResNet-50 (Baseline)
(b) ResNet-50 + TS
(c) ResNet-50 + HB
Cordtoanll Expetadda
Confidincce DKAH
Confidance WAll
(d) ResNet-50 IR
ResNet-50 MRR
Figure 3: Calibration-sharpness diagrams using MSE/Brier score for the ResNet-50 experiments of Table 2
Marta Maro
deru =.10
Relta +0.40 20 0002 Rairor
F4 mm
Conteoona Experar of
Contitonal Expertation
Alniili
(a) EfficientNet (Baseline)
(b) EfficientNet + TS
EfficientNet + HB
Carolbaadd
Confidinnce 0KGn
Confidence Dkalt
(d) EfficientNet IR
(e) EfficientNet + MRR
Figure 4: Calibration-sharppess diagrams using MSE/Brier score for the EfficientNet experiments of
Table 3
22

---

Published as a conference paper at ICLR 2025
Model
Test Accuracy
ECE
ACE
SmoothECE
NLL
MSE/Brier Score
ResNet-50 (Baseline)
80.33
8.68
10.48
8.26
93.98
29.69
ResNet-50 + TS
80.33
4.97
6.00
4.85
88.63
28.69
ResNet-50 + HB
78.70
6.44
15.11
4.77
00
32,94
ResNet-50 IR
79.12
7.25
8.81
7.20
30.55
ResNet-50 MRR
80.33
0.28
0.28
0.28
185,48
35,48
Table 2: Table ! in the main paper recreated using pretrained ResNet-50 backbone
(timm/resnet50.al _nnlkk.
Model EfficientNet (Baseline)
Test Accuracy
ECE
ACE
SmoothECE
NLL
MSE/Brier Score
81.47
10.66
9.97
10.67
85.50
28.24
EfficientNet + +SS
81.47
3.44
5.85
3.41
75.17
26.94
EllicientNet HB
77.63
5.44
12.92
4.58
00
33.01
EffiiiienNett IR
80.25
4.92
7.80
4.88
00
28.75
EfficientNet MRR
81.47
0.07
0.07
0.07
175.92
33.62
Table 3: Table I in the main paper recreated using a pretrained EfficientNet backbone
(timm/efficientnet b3.ra2 inlk).
Model
Test Accuracy
ECE
ACE
SmoothECE
NLL
MSE/Brier Score
ConvNeXt (Baseline)
84.13
6.93
6.80
6.94
68.34
23.78
ConvNeXt TS
84.13
3.00
4.70
2.99
62.01
23.38
ConvNeXt + HB
80.96
3.64
12.60
4.09
28.68
ConvNeXt IR
82.72
4.93
8,29
4.84
00
25.57
ConvNeXt MRR
84.13
0.13
0.13
0.13
153.34
29.22
Table 4: Table D in the main paper recreated using pretrained (timm/convnext 1k ft__innkk inlk).
ConvNeXt backbone
and 7
The comparison between the CIFAR-10 and CIFAR-100 results highlights the advantage of the sharpness gap bands in the calibration-sharppess diagrams: we see that while calibration does not suffer too much between CIFAR-10 and CIFAR-100, sharpness is much worse on CIFAR- 100 and consequently generalization metrics (MSE/Brier Score) are also much worse
Model
Test Accuracy
ECE
ACE
SmoothECE
NLL
MSE/Brier Score
ResNet-32 (Baseline)
93.46
4.15
16.01
3.94
29.21
10.87
ResNet-32
93.46
0.72
5.79
0.88
21.86
10.00
ResNet-32 + HB
93.12
1.45
15.17
1.46
11.75
ResNet-32 IR
93.29
1.33
5.11
1.34
00
10.18
ResNet-32 MRR
93.46
0.34
0.34
0.34
38.52
12.60
Table 5: Pretrained ResNet-32 model recalibration approaches evaluated on CIFAR-10.
Model
Test Accuracy
ECE
ACE
SmoothECE
NLL
MSE/Brier Score
ResNet-32 (Baseline)
70.20
13.25
15.71
13.12
132.31
44.08
ResNet-32 + TS
70.20
1.71
2.42
1.53
111.37
40.98
ResNet-32 + HB
65.76
9.56
18.10
6.97
oC
49.84
ResNet-32 IR
68.70
6.81
7.87
6.80
OC
43.02
ResNet-32 MRR
70.20
0.20
0.20
0.20
197.85
50.63
Table 6: Pretrained ResNet-32 model + recalibration approaches evaluated on CIFAR-100.
G COMPARISON TO SMOOTHECE
As we mentioned in Section the SmoothECE of Blasiok & Nakkiran (2023) is an alternative to ECE that is calculated and visualized using similar kernel regression approach to ours. The differences
23

---

Published as a conference paper at ICLR 2025
Condatrnal Condeorn Eroemscinn OF 020
diuta 10000 2-01L
SLcx 2.12 0,00
Mecia +0 +0:55
Merroni 4e 23.76
Reitor 26:08
Contioona
6thom
pinilli
(a) ConvNeXt (Baseline)
(b) ConvNeXt + TS
(c) ConvNeXt + HB
Rece 101
bxpectada
Confidincee OKAI
Confidance WAll
(d) ConvNeXt IR
ConvNNXt MRR
Figure 5: Calibration-sharpness diagrams using MSE/Brier score for the ConvNeXt experiments of Table 4
Conteconal Experarn on of an
Valta 0.000 Mare
Seru 0.0000.00
Merta 0.00 2:00 Awior
F mn
Contitona Expertation
Stebon
C4
(a) ResNet-32 (Baseline)
(b) ResNet-32 + TS
(c) ResNet-32 + HB
12.60
Caralfaand
Athutt
300000
Confidincc 0KG1
Confidence 9Wxl
(d) ResNet-32 IR
ResNet-32 MRR
Figure 6: Calibration-sharpnes diagrams using MSE/Brier score for the ResNet-32 experiments of Jon CIFAR-10.
24

---

Published as a conference paper at ICLR 2025
1.00
Fecs 0.00
0.02
44.00
4autor 40.54
Condronn Expectatio
Condioon Exprctetion
Plnkkl
R000
Confidence 1600
Confidence (NO0
Confidance MIx))
(a) ResNet-32 (Baseline)
(b) ResNet-32 + TS
(c) ResNet-32 + HI
Ao 0.00
de wo
Corditenl
BUMAH
Confidence
Confidance
(d) ResNet-32 + IR
(c) ResNet-32 + MRR
Figure 7: Calibration-ssarpnes diagrams using MSE/Brier score for the ResNet-32 experiments of Table on CIFAR-100.
between our two approaches are that Blasiok & Nakkiran (2023) use particular choice of kernel and kernel bandwidth motivated by the framework of Blasiok al. 2023), and they do not include any kind of notion of sharpness in their visualization. Additionally. they show the density estimate of the confidences using the thickness of the calibration curve opposed to overlain density plot.
We show a SmoothECE analogue to Figure I in Figure 8 Due to memory constraints, we plot the visualization after subsampling the test data 5000 data points. While the SmoothECE plots still pick up issues with the MRR recalibration strategy, they make it much trickier to compare histogram binning (HB) and isotonic regression (IR) did in Section As we mentioned, for these cases the visualization of sharpness is crucial for understanding the tradeoffs between the methods.
H EFFECT OF KERNEL HYPERPARAMETERS
Here we examine the importance of the hyperparameters (choice of kernel and kernel bandwidth) in the calibration-sharpness diagrams of Section 5 The main takeaway from this section is that, for range of different (reasonable) kernel choices and bandwidths results remain consistent.
note on choosing kernel hy hyperparameters in practice. Our choice of using a bandwidth of 0.05 in the main paper was motivated by the recent Logit-Smoothed ECE of Chidambaram (2024), where taking the bandwidth to be the inverse of the number of bins used in binning estimators worked good analogue (and 15 20 bins standard practice for datasets of the size we consider). However, in "real life" one can of course tune both the choice of kernel and the kernel bandwidth using held-out data. In particular, properties of these models (e.g. model A has poor performance whenever its confidences are in the range 00.7, 0.8)). and we can tune our kermel/bandwidth to best distinguish these properties across models. This is essentially also how we qualitatively evaluate the different figures in the following two subsections, since our goal is to produce visualizations that display the results of Table as precise manner possible.
al.
may have set of models and various estimated (or known)
H.1 THE IMPACT OF KERNEL BANDWIDTH
We first consider the same Gaussian kernel used in Section but vary the bandwidth between 0.01 and 0.25. We find that values that are less than 0.01 lead to significant under-smoothing, and values that are larger than 0.1 lead to over- wer-smoothing. Values in the range [0.01, 0.05) appear to work best. We visualize the choices of 0.01, 0.03, 0.1, and 0.25 in Figures 9 to respectively.
25

---

Published as a conference paper at ICLR 2025
1.0
1.0
1.0
SMECE 0.085 * 0.000
SMÉCE: :0.037 = 0.008
SMECE :0.047 + 0.011
0.8
0.8
0.8
0.4
0.4
W 0.4
0.00
0.2
'0.4" 0.6
0.3
0.8
1.0
0.0.0
0.2
0.4 14
0.6
0.8
To
0..0
0.2
0.4
0.6
1.0
(a) VIT (Bascline)
(b) ViT TS
(c) ViT HB
SMECE: 0.009
SMECE: 0:011
0.6
0.2
0.2
8.00
0.4
To
8.0
(d) VIT +
IR
(e) VIT + MRR
Figure 8: SmoothECE visualization analogue to Figure
We note that when we over smooth, the ranking of calibration error can change across methods shown in Figures and [2]. This is similar to known issues with reporting binned ECE in which only small number of bins are used.
Coneioona Doletete atV 220
3.34 2.02
Rass 0.000 Sinon 95
2.34
22.75
Ma. 27.17
Conditinnl
6mkki
Conffdence ROOD
Canfiinnce (NX0
Confidance Rid)
(a) VIT (Baseline)
(b) VIT + TS
(c) VIT + HB
Mooe Oe Name
Canditional www.iiiiionnoooooo of
Condiband 5
Confidence 910m
Coffidence Pid
(d) VIT IR
VIT + MRR
Figure 9: The experiments of Figure T but using bandwidth parameter of 0.01.
H.2 THE IMPACT OF KERNEL CHOICE
We also consider the impact of changing the kernel used for the visualizations of Figure In particular, consider the popular choice of the Epanechnikov kernel, which is K (u) (3(1 u²) /4)1 wist in the ID setting. Results using this kernel with bandwidth of 0.03 and 0.05 are shown in Figures
13 and 14
26

---

Published as a conference paper at ICLR 2025
Conditon DiRebrtcnn of C0
22:73
Contitrn Expectsiion of
6lhzon
6thixil
800om
Confieece Hox)
Canfdence DOO0
Confidence Mel
(a) VIT (Baseline)
(b) VIT TS
(C) VIT HB
Contioonal om
decw 0.25 20.00
Mcow e 0.00 0.00
4
Ahhxti
2600000
(d) VIT + IR
(c) ViT + MRR
Figure 10: The experiments of Figure but using a bandwidth parameter of 0.03.
Condbcaal Expect of CX1
22.31
Contiocaal
Blnzen
Ainkil
Rdom
Costilence (oz)
Confitence (OX0
Confideace PNS5)
(a) ViT (Baselinc)
(b) VIT TS
(c) VIT + HB
S. 0.38 0.00
0.00 9C0)
4i an
AmAl
jMM
32
(d) VIT
(c) VIT MRR
Figure 11: The experiments of Figure but using a bandwidth parameter of 0.1.
27

---

Published as a conference paper at ICLR 2025
Concrion Expicaiion of .o.
0.00
1.776 0,23
22.73
21.90
Rautor 27:27
Contiooo Expecriiio
Plnkel
R0000
Confidence 1OO0
Confidence (Nx0
Confidance Rix))
(a) VIT (Baseline)
(b) VIT + TS
(c) VIT + HI
Pu D.00
Expecatinn
Conditenn Eopestattton
BUMAHI
Confidence
Confidince
(d) VIT
(c) VIT MRR
Figure 12: The experiments of Figure but using bandwidth parameter of 0.25.
As can be seen from the results, changing the kernel does not change the ranking the recalibration methods, although it does impact the actual reported calibration error. We find that the range of kernel bandwidths suitable in the Gaussian case also applies here.
Conditonal Droecattn of c00
decs
Conditonnl Exocrstion of"
Colfdeence RozD
Canfidenc (hx0
Confidence Rixl
(a) VIT (Baseline)
VIT TS
(c) VIT +
decu 0.26 20.00
dc 0,00 1.00
di 26.1
22.53
Condioonal Enperaccnn
Confidence 01011
Confidence D1A11
( VIT IR
(e) VIT MRR
Figure 13: Epanechnikov kernel using a bandwidth parameter of 0.03.
28

---

Published as a conference paper at ICLR 2025
Conde ona Expecaaiin of Cd
2.996:200 0.03
1:15=00000
Expectario
Confidece M600
Confidence (1x0
Confidance 1ND5)
(a) VIT (Baseline)
(b) ViT TS
TS
(c) VIT HB
Conddoral d10
deow = E225 20.00
Bloo = 0.00 D.00
de an
de nur 27.53
Amtxi
(d) VIT + IR
(c) VIT + MRR
Figure 14: Epanechnikov kernel using a bandwidth parameter of 0.05.
29


Reviews:
[{"review_id": "uLyY4nWCXY", "reviewer": "Reviewer_6vrr", "summary": "The paper critically examines current practices in evaluating model calibration, particularly for deep learning. It highlights flaws in reporting calibration metrics. The authors propose a new methodology based on the calibration-based decomposition of Bregman divergences and introduce new diagrams. These diagrams visualize calibration and generalization error jointly, allowing a more nuanced assessment. Additionally, the paper provides theoretical insights.", "questions": "n/a", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper studies a fundamental question in the ML calibration literature: how to compare and assess the model calibration performance. This is a very important question.\n2. This paper is easy to follow, and the main take-away is very clear.\n3. This paper points out \"pitfalls\" regarding calibration reporting in the current literature clearly.\n4. The proposed calibration-sharpness digram is interesting and novel.\n5. This paper is theoretically sound.", "weaknesses": "1. The experiment section is relatively weak. Only one dataset was examined. \n2. It would be great if the authors could open-source the code and make the new diagrams easier to use for other ML practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OWTNKYiARC", "reviewer": "Reviewer_sELH", "summary": "- The authors identify key issues in current calibration performance reporting in machine learning literature, emphasizing that focusing solely on calibration error (e.g., ECE) and accuracy can lead to misleading conclusions.\n- They show that simple recalibration strategies may outperform commonly used methods if calibration metrics are not presented alongside generalization metrics, such as negative log-likelihood (NLL).\n- The authors develop a theoretical framework that aligns calibration metrics with generalization metrics using Bregman divergences, proposing that calibration measures should be selected based on the chosen generalization metric. They establish novel connections between full calibration error and confidence calibration error.\n- They introduce “calibration-sharpness diagrams” as an extension to traditional reliability diagrams, allowing for a joint visualization of calibration and generalization errors and providing a more detailed view of the trade-offs between calibration and model sharpness.", "questions": "Based on the above Weakness points, I would like to ask the following questions. If appropriate and sufficient discussion is provided on these points, I would be pleased to consider raising my score.\n\n- Could you provide specific examples or a more detailed workflow for selecting calibration metrics based on common generalization metrics? A decision flowchart or table mapping generalization metrics to recommended calibration metrics would also be helpful for practitioners applying the proposed framework.\n-  How could the arbitrariness in selecting kernel types and hyperparameters for calibration-sharpness diagrams be minimized? Could you suggest specific strategies, such as cross-validation approaches or sensitivity analyses, to guide these choices? Additionally, are there recommended default choices for kernels and hyperparameters that perform robustly across various scenarios?\n- Given the variability in calibration performance across different kernel types and hyperparameter settings (as shown in the Appendix), could you provide more guidance on appropriate criteria for kernel selection and hyperparameter tuning? This would strengthen the reliability of calibration assessments based on the proposed diagrams.\n- Could you clarify the relationship between some theorems and the main discussion, e.g.,  regarding the “always wrong predictor” in Proposition 4.5 and its implications? An example or illustration of how the “always wrong predictor” relates to real-world calibration scenarios would be helpful. Additionally, could you explain how this proposition supports or challenges the main arguments of the paper?\n- Could you expand on the motivation behind choosing Bregman divergences for calibration metrics? Specifically, what factors led to the choice of Bregman divergences over other potential metrics? Additional context on this decision-making process would be helpful.\n  - Related to the above question, would you consider adding a brief explanation of Bregman divergences and their relationship to calibration metrics in Section 2? I believe this might improve the flow from Section 3 to Section 4 and provide readers with a clearer understanding of the theoretical foundation.\n- I may have overlooked or misunderstood certain aspects, but could you provide a more detailed guide on how to interpret the components of experimental figures like Figure 1? Specifically, explanations on how to read the sharpness band, density plot, and calibration curve, and what each represents in terms of model performance, would be helpful. Adding some sentences in the caption part of the figures or including a “How to read this figure” section in the paper could also improve accessibility.\n- Although it might slightly deviate from the core contributions of this paper, would it be valuable to provide empirical discussions on the bias of the Nadaraya-Watson estimator? For instance, as demonstrated in [Jiang et al., ICML 2020], would it be possible to use a synthetic data setup to empirically evaluate the bias of the Nadaraya-Watson estimator by estimating the true pointwise sharpness gap with Monte Carlo estimation and comparing it to the Nadaraya-Watson estimator? If this is not feasible within the setup of [Jiang et al., ICML 2020], could a simpler experimental setting be devised where the true values of the pointwise sharpness gap are estimable?\n\n### Citation:\n[Jiang et al., ICML 2020]: Jize Zhang, Bhavya Kailkhura, and T. Yong-Jin Han. Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning. ICML2020. (https://arxiv.org/abs/2003.07329).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper systematically identifies critical issues in current reporting practices for calibration and prediction accuracy, highlighting the limitations of relying solely on metrics like ECE and accuracy.\n- By utilizing Bregman divergences, the authors develop a unified framework that aligns calibration metrics with generalization metrics. This theoretical approach offers valuable insights for the development and evaluation of new calibration methods.", "weaknesses": "### Concerns for Unclear Guidelines for Selecting Calibration Metrics:\n- While the paper proposes selecting calibration metrics based on generalization metrics, it does not provide clear guidelines or step-by-step procedures for applying this principle in practice. From a practical perspective, providing concrete examples and a well-defined workflow for implementing the proposed framework would enhance the paper’s contribution and make its practical applications more accessible.\n\n### Sensitivity of Calibration-Sharpness Diagrams to Kernel and Hyperparameter Selection:\n- In calibration-sharpness diagrams based on kernel regression estimators, the choice of kernel type and hyperparameter values appears to significantly influence the results. This concern is relevant not only to the proposed method but also to other calibration error evaluations based on Nadaraya-Watson estimators, such as smooth ECE. However, these parameters pose a risk of being adjusted arbitrarily to produce favorable calibration performance. A more in-depth discussion on minimizing this arbitrariness would help ensure fair calibration evaluation.\n- Additionally, the Appendix reveals considerable variability in calibration performance across different kernel types and hyperparameter settings, raising concerns about the reliability of calibration-sharpness diagrams produced by the proposed framework. If calibration performance assessments based on these diagrams are highly sensitive to such choices, further discussion on criteria for kernel selection and strategies for tuning hyperparameters would strengthen the paper’s contributions.\n\t\n### Concerns for Readability and Presentation Clarity:\n- The presentation of certain theoretical elements in the paper could benefit from added clarity, as the connection between some propositions and the main text is not always apparent. For example, in Proposition 4.5, a more detailed explanation of the “always wrong predictor” and its implications could improve understanding. This may stem from a lack of detailed exposition rather than the inherent complexity of the material, but additional clarification would enhance accessibility.\n### Concerns for Relatively Weak Explanation for the Motivation Behind Using Bregman Divergences:\n- While the paper establishes useful calibration metrics based on Bregman divergences, it does not clearly explain why the authors hypothesized that Bregman divergence-based metrics would be effective for calibration. Specifically, the rationale behind selecting Bregman divergences over other possible metrics remains unclear, making it challenging to fully understand the motivation for the proposed framework. Although the benefits of this choice are evident in the theoretical results, a more explicit discussion on why Bregman divergences were chosen would provide valuable context.\n- I understand that the authors’ main motivation for adopting Bregman divergences could be that they provide a general metric encompassing several proper scoring rules, allowing for a unified approach to selecting calibration evaluation metrics. Emphasizing this reasoning could facilitate a smoother transition from Section 3 to Section 4. While this is a personal suggestion and not strictly necessary, including a brief explanation in Section 2 on the basic properties of Bregman divergences and their relationship to calibration metrics might improve the clarity and flow of the presentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tB6BtSID4L", "reviewer": "Reviewer_Dhyc", "summary": "This paper studies the task of calibration: measuring how well a model predicts the class probability and improving the quality.\nIt begins with studying the Expected Calibration Error (ECE), a popular metric of how well a model predicts the class probability.\nThe authors point out that the ECE can be improved by a trivial modification to the confidence function. The authors present experiments (Table 1) showing how this metric differs from proper scoring rules such as the Mean Squared Error (MSE) and the Negative Log Loss (NLL).\n\nThe paper then moves on to the study about proper scoring rules induced by Bregman divergences. The authors explain the gap between a proper scoring rule and the calibration error using a decomposition of Bregman divergences (Lemma 4.1) with some properties of this gap (Proposition 4.2 and 4.3). The pointwise variant of the gap is also defined (Eq. (4.7)).\n\nThe authors also show that the gap between the full calibration error and its one-dimensional counterpart.\n\nFinally, the paper presents visualization results based on the proposed metric (4.7).", "questions": "### Major comments\n- Are some of the findings of this paper related to those of Prez-Lebel et al. (2023)?\n\nAlexandre Perez-Lebel, Marine Le Morvan, and Gaël Varoquaux. BEYOND CALIBRATION: ESTIMATING THE GROUPING LOSS OF MODERN NEURAL NETWORKS. ICLR, 2023.\n\n- Corollary 4.7 should be more formally stated.\n\n- I could not understand the condition $d_{\\phi}(x, y) \\ge d_{\\psi}(x_i, y_i)$.\n\n- Is there any justification for thinking about the band proposed in lines 419-425? Also, the description here is not very clear to me despite its importance.\n\n- There should be a part explaining how to read the plots. What are the read curves, red areas, and blue areas? Then, there should be explanations about how to evaluate them such as what kind of curves and areas are supposed to be good. Some explanations of them are mixed with the interpretations of the results, which is making the section difficult to read.\n\n### Minor comments\n- The notation in Eq. (2.1) feels strange. If $\\mathbb{P}( \\cdot | \\cdot )$ denotes the regular conditional distribution, it should take an event in the first argument. It is also strange that it returns a vector, assuming $v \\in \\mathbb{R}^m$. Could the authors check the definition again?\n\n- I think the sign is not correct in (A.1).\n\n- The part of the proof obtaining (B.2) should be explained more clearly.\n\n- The proof of Corollary C.1 could be more precise.\n\n- Could the authors add a clear description about the standard reliability diagram?\n\n- I suggest the authors explicitly explain all the acronyms in the paper.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Theorems and propositions are interesting and seem correct. They are supported by rigorous proofs.\n- The paper studies an important problem of calibration.\n- The paper provides a good literature review, which motivates the direction of this work well.", "weaknesses": "- The paper explains ECE can be improved by a trivial modification to the confidence function but does not explain why it is a bad thing. If estimating the whole class probabilities is not the objective and we only care about the confidence of the predicted class, this solution could be just fine.\n\n- The descriptions of the method and the results are not clear to me.\n\n- The benefits of the visualization are not clear from the results. In particular, I do not agree with the claim that \"looking at the calibration-sharpness diagrams the differences between the two approaches becomes more clear.\"\n\n- The notation in some parts is a little confusing.\n\n- Some of the proofs lack details, and there are also some writing mistakes. These make the proofs a little difficult to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)