Beyond calibration: estimating the grouping loss of modern neural networks
Authors: Alexandre Perez-Lebel
Marine Le Morvan
Gael Varoquaux
Venue: ICLR 2023 poster (2023)
Keywords: calibration
grouping loss
decision making
model evaluation

Abstract: The ability to ensure that a classifier gives reliable confidence scores is essential to ensure informed decision-making. To this end, recent work has focused on miscalibration, i.e., the over or under confidence of model scores. Yet calibration is not enough: even a perfectly calibrated classifier with the best possible accuracy can have confidence scores that are far from the true posterior probabilities. This is due to the grouping loss, created by samples with the same confidence scores but different true posterior probabilities. Proper scoring rule theory shows that given the calibration loss, the missing piece to characterize individual errors is the grouping loss. While there are many estimators of the calibration loss, none exists for the grouping loss in standard settings. Here, we propose an estimator to approximate the grouping loss. We show that modern neural network architectures in vision and NLP exhibit grouping loss, notably in distribution shifts settings, which highlights the importance of pre-production validation.

Published as a conference paper at ICLR 2023
BEYOND CALIBRATION: ESTIMATING THE GROUPING LOSS OF MODERN NEURAL NETWORKS
Alexandre Perez-Lebel, Marine Le Morvan, Gaël Varoquaux Soda project team, Inria Saclay, Palaiseau, France (a exandre. perez, ma rine. 1e le-morvan, gael varooquuux)
ABSTRACT
The ability to ensure that a classifier gives reliable confidence scores essential to ensure informed decision- decision-making. making. To this end, recent work has focused on miscalibration, i.e,, the over or under confidence of model scores. Yet calibration is not enough; even perfectly calibrated classifier with the best possible accuracy can have confidence scores that are far from the true posterior probabilities. This is due to the grouping loss, created by samples with the same confidence scores but different true posterior probabilities. Proper scoring rule theory shows that given the calibration loss, the missing piece characterize individual errors is the grouping loss. While there are many estimators of the calibration loss, none exists for the grouping loss in standard settings. Here, we propose an estimator approximate the grouping loss. We show that modern neural network architectures in vision and NLP exhibit grouping loss, notably in distribution shifts settings, which highlights the importance of pre- pre-production validation.
INTRODUCTION
Validating the compliance of a model to a predefined set of specifications is important to control operational risks related to performance but also trustworthiness, fairness robustness varying operating conditions. It often requires that probability estimates capture the actual uncertainty of the prediction, i.e. are close the true posterior probabilities. Indeed, many situations call for probability estimates rather than just discriminant classifier. Probability estimates are needed when the decision left to a human decision maker, when the model needs avoid making decisions if they are too uncertain, when the context of model deployment is unknown at training time, etc.
To evaluate probabilistic predictions, statistics and decision theory have put forward proper scoring rules (Dawid, 1986; Gneiting al., 2007), such the Brier or the log-loss. Strictly proper scoring rules are minimized when a model produces the true posterior probabilities, which make them a valu- able tool for comparing models and selecting those with the best estimated probabilities (e.g. Dawid & Musio, 2014). What they do not provide though is a means of validating whether the best estimated probabilities are good enough to be put into production, whether further effort is needed to improve the model. Indeed, proper scores compound the irreducible loss -uue to the inherent randomness of problem, i.e. the aleatoric uncertainty- and the epistemic loss -which measures how far a model is from the best possible one. For example, a classifier with a Brier score of 0.15 could have optimal estimated probabilities (irreducible loss close to 0.15) or poor ones (irreducible loss close 0).
Calibration errors are another tool to evaluate probabilistic predictions, and measuring them is an active research topic the machine learning community. (Kumar et al., 2019; Minderer et al., 2021; Roclofs et al., 2022). The calibration error is in fact a component of proper scoring rules (Bröcker, 2009; Kull Flach, 2015): it measures whether among all samples to which a calibrated classifier gave the same confidence score, on average, a fraction equal to the confidence score is positive. Impor- tantly, the calibration error can be evaluated efficiently as it does not require access to the ground truth probabilities, but solely to their calibrated version. Calibration is however incomplete characteriza- tion of predictive uncertainty. It measures an aggregated error that is blind to potential individual errors compensating each other. For example, among group of individuals to which a calibrated cancer-risk classifier assigns a probability of 0.6, a fraction of 60% actually has cancer. But a subgroup of them could be composed of 100% cancer patients while another would only contain 20% of cancer patients.

---

Published as a conference paper at ICLR 2023
In general, estimating the true posterior probabilities obtaining individual guarantees is impossible (Vovk et al., 2005; Barber, 2020). Recent works have thus attempted to refine guarantees on uncertainty estimates at intermediary subgroup level. In particular, Hébert-Johnson et al. (2018) has introduced the notion of multicalibration, generalizing the notion of calibration within groups studied in fairness (Kleinberg al., 2016) every efficientlyiidenfffablle subgroup. Barber et al. (2019); Barber (2020) defines subgroups baberoups-based coverage guarantees which lie in between the coarse marginal coverage and the impossible conditional coverage guarantees. In similar vein, we study the remaining term measuring the discrepancy between the calibrated probabilities and the unknown true posterior probabilities (Kull & Flach, 2015), i.e. the grouping loss, for which no estimation procedure exists to date. In particular:
We provide a new decomposition of the grouping loss into explained and residual components, together with a debiased estimator of the explained component as a lower bound (Section 4). We demonstrate on simulations that the proposed estimator can provide tight lower-bounds on the grouping loss (Section 5.1).
We evidence for the first time the presence grouping loss on pre-trained vision and language architectures, notably in distribution shifts settings (Section 5.2).
2
CALIBRATION IS NOT ENOUGH
Calibration can be understood with a broad conceptual meaning of alignment of measures and statistical estimates (Osborne, 1991). However, in the context of decision theory or classifiers, the following definitions are used (Foster & Vohra, 1998; Gneiting al., 2007; Kull & Flach, 2015):
True posterior probabilities: Q == P(Y = 1[x),
Confidence scores: S f(X) score output by classifier,
Calibrated scores: C = P(Y = 1 S E[Q $), average true posterior probabilities for score S.
Confusion about calibration A common confusion is to mistake confidence scores a calibrated classifier with true posterior probabilities and think that calibrated classifier outputs true posterior probabilities, which is false. We identified three main sources of confusion the literature -see Appendix A for specific quotes. First, the vocabulary used sometimes leaves room for ambiguity, e.g. posterior probabilities may refer to confidence scores or to the true posterior probabilities without further specifications. Second, plain-English definitions of calibration are sometimes incorrect, defining calibrated scores as the true posterior probabilities. Lastly, even when everything is correctly defined, it is sometimes implicitly supposed that true posterior probabilities are close the calibrated scores. While it may be true in some cases, equating the two induces misconceptions.
Calibration with good accuracy does not imply good individual confidences is tempt- ing to think that a calibrated classifier with optimal accuracy should provide confidence scores close to the true posterior probabilities. However, caution is necessary: Figure shows a simple counterexample. The classi- fier presented gives an optimal accuracy as its confidence scores always on the same side of the decision threshold as the true posterior probabilities. It is moreover calibrated, as for
1.0
S(X)
@(x
0.8
0.7
Output
0.6
given score (either 0.2 or 0.7 here), the expectation of Q over the region where the confidence score is is actually equal to Yet, the confidence scores are not equal to Q as Q displays variance over regions of constant scores. This variance can be made as large as desired as long as both Q and S stay on the same side of the decision threshold to preserve accuracy. The flaws of a perfectly calibrated classifier that always predicts the same score are typically due to variations of the true posterior probabilities over constant confidence scores. As we formalize below, such variations are captured by the grouping loss (Kull & Flach, 2015). Appendix B provides a more realistic variant of this example based on the output of neural network.
0.5
decision threshold
0.3
0.2
0.1
0.0
X~ ((-11-11)
Figure 1: A calibrated binary classi- fier with optimal accuracy and con- fidence scores S(X) everywhere different from the true posterior probabilities Q(X).
2

---

Published as a conference paper at ICLR 2023
3 THEORETICAL BACKGROUND
Notations Let (X, Y) E X y be jointly distributed random variables describing the features and labels of a K-class classification task. Let be the one-hot vector of size K with its 4 entry equal to one. The label space y/ = {El....,ey, ,., is the set all one-hot vectors of size K. We assume that labels are drawn according to the true posterior distribution Q (Q1,-. ................................ Qx) AK where Qk
P(Y = c (X) and AK is the probability simplex ΔK (PI,.... PK) E 00 Ex =
) E Ax- Note that and Q are random vectors since they depend X. This section introduces the formal definition of the grouping loss, which uses the concepts of calibrated scores as well as scoring rules.
We consider probabilistic classifier f giving scores 5 = f(X) with S =
3.1 CALIBRATION IN A MULTI-CLASS SETTING
In multi-class settings various definitions of calibration give different trade offs between control stringency and practical utility (Vaicenavicius et al., 2019; Kull et al., 2019). The strongest definition controls the proportion of positives for groups of samples with the same vector of scores S. Definition 3.1. probabilistic classifier giving scores s = (s1....,33) 9y) is jointly calibrated if among all instances getting score s, the class probabilities are actually equal Calibration
P(Y = E 8) = 8k for I K.
The score S being a vector of size K the number of classes, estimating the probability of Y conditioned S difficult task that requires many samples. A weaker notion of multi-class calibration, introduced in Zadrozny & Elkan (2002), requires calibration for each class marginally: Definition 3.2. A probabilistic classifier giving scores ($1,.... ,......,)))))))))) 3k) is classwise-caiirrate if among all instances getting score Sk, the probability of class k is actually equal to 8k Classwise calibration
P(Y = Sp 8k) 8% for k I,.., K.
(2)
As classwise calibration can still be challenging estimate when the number of samples per class is too small, an even weaker definition is used in the machine learning community (Guo et al., 2017). Definition 3.3. A probabilistic classifier giving scores (. ) is top-label-calibratee if among all instances for which the confidence score of the predicted class 8, the probability that the predicted class is the correct one is Top-label calibration
P(V Cargmax(s) mmx =s.
Top-label calibration simplifies the problem by reducing it to a binary problem. However, it has an important limitation (Vaicenavicius al., 2019): it only accounts for the confidence of the predicted class, it does not tell whether smaller probabilities are also calibrated.
3.2 PROPER SCORING RULES AND THEIR DECOMPOSITION
Scoring rules measure how well an estimated probability vector S explains the observed labels Y. The two most widely used scoring rules are the log-loss and Brier score:
Scoring rules
K
Log-loss: : (L,Y) = Σ Yu log St Brier score:
(S,
Σ (S-Y Y2)²
(4)
k=1
Scoring rules are defined per sample, and the score over dataset is obtained by averaging over samples. More generally, the expected score for rule ø of the estimated probability vector S with regards to the class label Y drawn according to Q is given by (S, := Ey~q [o(S, Y)]. Proper scoring rules decompositions have been introduced in terms their divergences rather than their scores. The divergence between probability vectors S and Q is then defined as ds(S, Q) (, Q)
So( (Q, Q). The divergences for the Brier score and the log-loss read:
K
Qk Sk
K
Log-loss: : gLL(S Q)
Σ
Qu log
Brier score : BS (S, = Σ (St Qx)² (5)
k=1
Minimizing the Brier score in expectation thus amounts minimizing the mean squared error between S and the unknown Q. A scoring rule said strictly proper if its divergence is negative
and ds, Q) = 0 implies S = Q. Both the log-loss and Brier score are strictly proper.
3

---

Published as a conference paper at ICLR 2023
Scoring rules decomposition Let C' be the calibrated scores in the sense of Definition 3.1, the strongest one í.e., Ck = P(Y = e = 8) for k I....,K. K. The divergence strictly proper scoring rules can be decomposed as (Kull & Flach, 2015):
E (dd = E (d, + [dd(C, + +E.. (Q)(QQ
Calibration: CL
Grouping: GL
Inreducible:
where the expectation is taken over Y ~ Q and X. CL the calibration loss. IL is the irreducible loss which stems from the fact that one point may not have deterministic label, making perfect predictions impossible. GL. is the grouping loss. Intuitively, while the calibration loss captures the deviation of the expected score in bin vs the expected posterior probabilities, the grouping loss captures variations of the true posterior probabilities around their expectation. Together calibration and grouping form the epistemic loss, capturing intrinsic the randomness of the best possible predictor. The scoring rule decomposition (6) holds for top-label calibration (Definition 3.3) as it can be reduced to binary problem. In the case of classwise calibration, the extension is not straightforward in the general case but we prove in Proposition C.3 that it holds for the Brier score and the log-loss.
4 CHARACTERIZATION OF THE GROUPING LOSS
In this section, we focus for simplicity on all settings where the calibrated scores can be expressed as C E [Y& SS. which includes binary classification well the multi-class setting with joint or top-label calibration. For classwise calibration, Appendix C.9 shows that all the results presented in this section also hold for the Brier score and log-loss.
4.1 REWRITING THE GROUPING LOSS A FORM VARIANCE
To shed light on the grouping loss, we rewrite it using f-variances:
Definition 4.1 (f-variance). Let U. V + Rd be two random variables defined on the same probability space, and function f Rdd + R. Assuming the required expectations exist, the f-variance of U given V is:
U V = (ES(E)
S(E(U VT).
The f-variance corresponds to the Jensen gap. It is positive by Jensen's 's inequality when f is convex. Beyond positivity, it can be seen as an extension of the variance as using the square function for recovers the traditional notion of variance.
Lemma 4.1 (The grouping loss as an h-variance). Let h be the negative entropy of the scoring rule á. i.c. h + -s p). The grouping loss GL of the classifier S with calibrated scores C E[Q S]] and scoring rule writes:
[C)o(C E[V[Q []] GL(S)
(7)
The proof is given in Appendix C.I. In other words, the grouping loss associated to scoring rule ø is an h-variance of the true posterior probability Q around the average scores C' on groups of same level confidence S (Equation 7). In particular for the Brier score, the h-variance is classical variance. It measures discrepancy between Q and C' with squared norm: 1/ S] E Q CII2 S]. For the log-loss, it is Kullback-Leibler divergence: V/ QQ S] EE DKL (Q S]. These expressions highlight two challenges in estimating the grouping loss. First, it relies on the true posterior probabili- ties Q, which we do not have access Second, involves a conditioning on the confidence scores S, which are difficult to estimate for continuous scores.
4.2 GROUPING LOSS DECOMPOSITION AND LOWER- BOUND
As an ht-ariance of Q given S, evaluating the grouping loss requires access to Q(X) ) for any point X. Unfortunately ((X) is difficult to estimate, except in special settings e.g. multiple labels per sample as in Mimori et al. (2021). In fact, the scores S of a classifier are generally one's best estimate of Q, and the whole point of the grouping loss is to quantify how far this best estimate is from the unknown oracle Q. We show that it nevertheless possible to estimate a lower bound on the grouping loss. On the level where classifier score is S, it is indeed possible estimate the average of Q
4

---

Published as a conference paper at ICLR 2023
0.9
X
S,
Figure 2: Intuition. In the feature space X, the level set of confidence 5 0.7 displays F[Q S] 0.7, which we expect from a calibrated classifier. However, par- tition of the level into 2 regions 3, and R2 reveals that E[Q [.]] = 0.6 while E[Q SR 0.8, sug- gesting a high grouping loss. Intra-region variances %. QQ S, Ri and V/2 QQ) S, remain uncaptured.
E[Q[S] = 0.7
R2
R
[QQ R1 = 0.6
Level set S 0.7
0.5
on regions of the feature space. Since by definition Q is non-constant on the level set classifier with non-zero grouping loss, it allows to capture part of the grouping loss (Figure 2). Intra-region variance remains uncaptured but can reduced by choosing smarter and more numerous regions in the partition of the feature space. Theorem 4.1 formalizes this intuition:
Theorem 4.1 (Grouping loss decomposition). Let R X > N be a partition of the feature space. holds that:
GL(S)- = [|V[[EQQ S, || + [| S,
(8)
(Laystsa())
Moreover if the scoring rule is proper, then:
GL(S) 2 Leeplained (5) 2 0.
Appendix C.2 gives the proof by showing that the law of total variance is also valid for the h-variance, which allows to decompose the grouping loss into explained and residual terms. GLesplained quantifies the h-variance captured through the partition R, i.e. coarse coarse-grained h-variance reflecting between- region variations of Q. while GTrresidual captures the remaining intra-region h-variance. Due the positivity of Geeeuidua Gespppiiined is nower-boundg of the grouping loss that ranges between 0 and GL depending on how much h-variance the partition captures. Importantly, while V, / S R] cannot be estimated because the oracle Q is unknown, it is possible to estimate E[Q | S, R] and thus GLespiained
4.3 CONTROLLING THE GROUPING LOSS INDUCED BY BINNING CLASSIFIER SCORES S
The grouping loss as well GLeeplained involve conditioning on the confidence scores S, which cannot be estimated by mere counting when the scores are continuous. To overcome this difficulty, standard practice in calibration approximates the conditional expectation using a binning strategy: the classifier scores are binned into a finite number of values (Definition 4.2).
Definition 4.2 (Binned classifier), Let S X + AK be classifier. Let
Bin edge
B == (B, heisJ be a partition of Ak. The binned version of S outputs the average of S on each bin:
GLvidueed
X - S
Cp
Sp:
x E[S|SE B,] where B, is the bin S(x) into.
(10)
Sp
The binned calibrated scores are defined by:
C=:=Pt P(Y 1|SB) = E[Q [B]] E(C| Sg].
0
This is the approach taken by the popular Expected Calibration Error (ECE) (Naeini et al., 2015). However, the loss estimated for binned classifier deviates from that of the original one. In particular, binning biases the calibration loss downwards (Kumar al., 2019). Here we show that on the contrary creates an upwards bias for the grouping loss. Binning a classifier S into SB boils down to merging the level sets S into a finite number of larger level sets of confidence score Sg. For example in Figure 3, all level sets with S [0.5, 1] are merged into one level set of confidence SB = 0.75, which artificially inflates the variance of Q each bin. This intuition is formalized in Proposition 4.1
Figure 3: Binning in-
flates the grouping loss.
Proposition 4.1 (Binning- -inning-induuced induced grouping loss). The grouping loss of the binned classifier GL(SB) deviates from that of the original classifier GL(S) by an induced grouping loss GLinduaed (S, Sp):
E E Søl] = E[[Q
+ [
(11)
CL(SB)
GL(S)
Gl (S,5g)
Moreover, the scoring rule is proper:
GT GIinddaed (S, (S,Sn) 0.
5

---

Published as a conference paper at ICLR 2023
Appendix C.3 gives the proof. Proposition 4.1 shows that the difference between the grouping loss of the binned and original classifier is given by the h-variance of the original calibrated scores in bin. This result provides expression for GLinduced which can then be estimated shown in Section 4.4. Remark 1. Interestingly, the binning-induced grouping and calibration losses partly compensate each other (Corollary C.I in Appendix C.8).
Applying the decomposition of Theorem 4.1 I to the binned classifier Sn and accounting for binning using Proposition 4.1, obtain new decomposition of the grouping loss:
Proposition 4.2 (Explained grouping loss accounting for binning).
GL(S) Cppplnine SB) Ginnddcedd (S, SB) + GLe (S
(12)
If the scoring rule is proper, then:
GL( 2 GLespllinedd SB) - GLindduedd (5, SB).
(13)
GLan(S.95n
The proof is given in Appendix C.4. Importantly, contrary to the grouping loss, both terms in the lower-bound (Equation 13) can be estimated. the remainder of this paper, we will be interested in the estimation and optimization of the lower bound GL
4.4 GROUPING LOSS ESTIMATION
We now derive a grouping-loss estimation procedure by focusing on each of its components turn: GT GT resplined SB) and Giindueed (S, SB).
A debiased estimator for the explained grouping loss GLexppaaied (S)) The most natural estimator for the explained grouping loss is plugin estimator, replacing [[ S, R] by the empirical means of Y over each region. It is nonetheless generally biased. We show below that in the case of the Brier scoring rule, direct empirical estimation of GLexptained on the partition is biased upwards (cf Appendix C.6), and propose a debiased estimator.
Proposition 4.3 (Debiased estimator for the Brier score). For all class k E {I,..., K} and bin ) be the number of samples belonging to level set R(d) (resp. region R We define the empirical average Yf over these regions as:
....,,K
E S, let (a.k) (resp. n; (()
(a.h)
1
Σ
y(()) and gle,k)
y(
ka) EX"ERR!
(a,
n
K
GL (a.k) explaieed SB))
The debiased estimator of GTeespaained is: GI exsppaiine (SB)
k= SES
with:
(a,k) GT GT Geleetooeee
(S
Σ
n (e.k) n(a,k) (n/oam)) eloat))?
(8.N) (1 - ((akk) k)
n (s, p
(1-
nl (o.k)
(t,kk)
plugin estimator GLyphagin
bias estimation GLoias
Appendix C.5 gives the proof, with a debiasing logic similar to Bröcker (2012). The leftmost term corresponds to the plugin estimate: the estimator of the explained grouping loss (Theorem 4.1) with sample estimators for the quantities of interest. The two rightmost terms represent the inite-sample variance in estimating expectations over regions. They correct the upwards bias of the plugin estimate.
Estimation of the grouping loss induced by binning classifier scores. GLinduced (S, (S,Sp) involves the h-variance of the calibrated scores C inside each bin, thus its estimation requires C. A solution is estimate a continous calibration curve Ć, which amounts a one-dimensional problem for which various methods are available. In our experiments, we use a kernel-based method (e.g. LOWESS). It is then easy to compute the h-variance of Ò' inside each bin by evaluating Ô for all available samples. The resulting expression of the estimator GT snnduced given in Appendix C.7.

---

Published as a conference paper at ICLR 2023
A partition to minimize GLyesidual In order to achieve the best possible lower-bound, choose partitions in Theorem 4.1 to minimize Geseiiinaa- We use a decision tree with a loss corresponding to the scoring rule -squared loss for Brier score- the labels Y to define regions that minimize the loss on a given level set of S. As this approach relies on Y, train-test split is used to control for overfitting: partitioning of the feature space is defined using the leaves of the tree fitted one part, then the empirical means used GLeppained are estimated on the other part given this partitioning, In the experiments of Section 5.2, we work in the output space of the penultimate layer of the networks.
5
EXPERIMENTAL STUDY
5.1 SIMULATIONS: FINER PARTITIONS GIVE A TIGHT GROUPING-LOSS LOWER BOUND
Here we investigate the behavior of our estimation procedure with respect to the number of bins and number of regions on simulated data with known grouping loss. The importance of both corrections the binning-induced grouping loss (Proposition C.1) and the debiasing (Proposition 4.3) is also evaluated. For this, data Y E {0. } is drawn according to known true posterior probability Q and we consider a calibrated logistic regression classifier for the scores (details in Appendix B.2 and Figure 11). The estimation procedures First we vary the number of samples per region (e.g. region ratio) while the number of bins is fixed (Figure 4a.). Then vary the number of bins while the region ratio is fixed (Figure 4b.).
then applied according to two different scenarios.
b.
0.010
True GL
Simulation: estimat- ing the grouping loss lower bound GLLB on a simulated problem (Ap- pendix B.2, Figure 11). Right has fixed ratio samples Bins are cqual-width. Averaged curves are plotted with +1 stan- dard deviation envelop.
Figure 4:
Giptogin
0.005
GLuias
Ghinducedd
#regions
30 per bin.
GLLB
0.000
25
50
75
5
15
25
# regions per bin
samples
# bins
For a fine-enough partition (a large number of regions, and hence small region ratio), GT provides tight lower bound to the true grouping loss GL. If the average number of samples per region becomes too small, some regions have less than two samples, which breaks the estimate (grayed out area in Figure 4a.). Conversely, the naive plugin estimate GLptugin substantially overestimates the true grouping loss it does include the corrections GLinduced and GL Figure 4b. shows that to control the GLindweed due to binning, a reasonably large number of bins is needed, e.g. 15 as typical to compute ECE. Given these bins, we suggest to use tree to divide them in as many regions as possible while controlling the probability of regions ending up with less than two samples, typically targeting region ratio of dozen, to obtain the best possible lower bound GLLB-
5.2 MODERN NEURAL NETWORKS DISPLAY GROUPING LOSS
The grouping diagram: visualization of the grouping loss In a binary setting, calibration curves display the calibrated scores C' versus the confidence scores S of the positive class. To visualize the heterogeneity among region scores in level set, we add to this representation the estimated region scores p.. i.e. the fraction positives each region obtained from the partitioning of level sets (Figure 5). The further apart the region scores are, the greater the grouping loss.
Vision We evaluate 15 vision models (listed on Figure 7) from PyTorch pre-trained on ImageNet- 1K (Deng et al., 2009). Here we report evaluation on ImageNet-R (Hendrycks et al., 2021), an ImageNet variant with 15 different renditions: paintings, toys, tattoos, origami... The dataset contains 30 000 images and 200 ImageNet classes. Appendix D reports evaluation on the validation set of ImageNet-Ie IK and ImageNet-C (Hendrycks Dietterich, 2019), an ImageNet variant with corrupted versions of ImageNet images. As often with many classes, the small number of samples per class (50) does not allow to study the classwise calibration and grouping loss. Hence, following common practice, we consider top-label versions (Definition 3.3). Appendix D gives experimental details.
7

---

Published as a conference paper at ICLR 2023
100
Figure 5: Grouping diagram. Calibration curve of binned binary classifier augmented with the estimated region scores Py for partitioning each level set into regions. Region sample sizes are plotted as a gradient color. The classifier is binned into 10 equal-width bins whose sample sizes (*)) are given as an histogram. A Clopper-Pearse the region scores. Regions for which the calibrated score e(*) lie within this interval are grayed out.
Sizes 50
Fraction positives
320
500
95% confidence interval plotted
230
Confidence score
We find substantial grouping loss inside level sets for most networks on ImageNet-R (ConvNeXt Tiny and VIT L-16 in Figure 6, others in Appendix D.I), after post-hoc recalibration (Figure right). For instance, while ConvNext + Isotonic is calibrated (third graph), it is strongly over-confident in part of the feature space and under-confident in the other, creating high grouping loss.
Figure shows that the grouping loss varies across architectures, even with comparable accuracy. For example, ViT has a slightly better accuracy than ConvNeXt, but lower estimated grouping loss. Post-hoc recalibration does not affect the grouping loss (Figure 6 right and Figure 7 right), leading to the same conclusions (see Appendix C.10 for the analytical impact of recalibration on the grouping loss). We observe the same effects on ImageNet-CC (Appendix D.2). but little or none on ImageNet-Ie IK (Appendix D.3). This suggests that stronger grouping loss arises in out-of-distribution settings. Visual inspection of the images suggests that the partitions capture heterogeneity coming from how realistic an image is, or the different rendition types (Appendix D.1).
a. ConvNeXt
ViT
c. ConvNeXt
d. ViT
100
Sizes
Sizes 20
Sizes
Sizes
Correct predictions
333
94
75
600
750
1000
750
979
1577
with
25
isotonic
isotonic
recalibration
rocaa iocalibration
0.5
0
0.5
0.5
0.5
Confidence score
Confidence score
Confidence score
Confidence score
Figure Vision: Fraction of correct predictions versus confidence score of predicted class (max): St)) for ConvNeXt Tiny and ViT L-16 on ImageNet-R, without post-hoc recalibration (a. and b.) and with isotonic recalibration (c. and d.). In each bin on confidence scores, the level set partitioned into regions with decision stump constrained to one balanced split, with 50-50 train-test split strategy.
No recalibration
Isotonic
Figure 7: Evaluating vision models: de- biased estimate of the grouping loss lower bound GLLB (Equation 13) and an estimate of the calibration loss ČL, both accounting for binning, evaluated on ImageNet-R and sorted by model accuracy. Partitions R are obtained from a decision tree partitioning constrained create most # samples in bin/30 regions in each bin. Isotonic regression is used for host-hoc recalibration of the models (right). Table 1 gives the raw values.
VIT B-16
ConvNeX: Tiny
Inception
Wide ResNet-50
A Accurac
Re
Fefticient
DenseNet-121
GoogLeNet
RegNet y 4-400m
ML.3
ResNet-15
20.4
MobileNet V2
18.9
MNASNet 0.5
16.9
VGG-11
GL
ShuffleNet 0.5
CL
11.00
AlexNet
0.00
0.05
0.10
0.16
0.200.00
0006
GL + CL
8

---

Published as a conference paper at ICLR 2023
In-distribution
b. Out-of-distribution
In-distribution
d. Out-of-distribttion
Fraction of positives (%)
100
Sizes
Sizes 1352 3951
Sizes 198 22048
Sizes 151 14294
7255
15411
50
with isotonic
with isptonic recalibration
recalibration
Confidence score
Confidence score
Confidence score
Confidence score
Figure 8: NLP: Fraction of positives versus confidence score of the positive class of fine-tuned BART for zero-shot classification on the test set Yahoo Answers Topics without post-hoc recalibration (a. and b.) and with isotonic recalibration (c. and d.). The test set is either restricted to the 5 topics on which the network was trained fin-distrbuution) or to unseen topics cout-of-distriuutin In each level set clusters are built with a balanced decision stump and 50-50 train-test split strategy.
NLP We evaluate the grouping loss on BART Large (Lewis al., 2019) from HuggingFace pre-traiin te-irraned on the Multi Munti Natural Language Inference dataset (Williams et al., 2018). We consider zero-shot topic classification on the Yahoo Answers Topics dataset, composed of questions and topic labels. There are 000 test samples and 10 topics. The model is fine-tuned 10 topics of the training set, totaling 000 samples. Given question title and a hypothesis (e.g- "This text is about Science & Mathematics"), the model outputs its confidence in the hypothesis to be true. The classification being zero-shot, the hypothesis can be about an unseen topic. We evaluate the model separately the unseen topics and the 5 seen topics of the test set. Both results in binary classification task on whether the hypothesis correct or not. Appendix gives experimental details.
5 out of the
The partitioning reveals grouping loss in the out-of- distriibution setting both before and after recali- bration (Figure 8b. and d.). However, we found no evidence of grouping loss in the in in-distribution setting. As in vision, this suggests that out-of-distribution settings lead stronger grouping loss.
DISCUSSION AND CONCLUSION
A working estimator of grouping loss While calibrated scores can be estimated by solving one- dimensional problem, the grouping loss is much harder to estimate: it measures the discrepancy to the true posterior probabilities, which are unknown. We show that combining debiased partition-batt based estimators with an optimized partition captures the grouping loss well. This procedure allows us to characterize the grouping loss of popular neural networks for the first time. We find that in vision and NLP, models can be calibrated -if needed via post-hoe recalibration- but significant heterogeneity of errors remains, e.g. ConvNeXt has larger grouping loss than calibration loss.
Several avenues could be explored to better capture the grouping loss. Complex level sets may not be approximated well with the partitioning defined by tree, leaving large residual in th. 4.1. In this case, the estimated grouping loss may only be rather loose lower bound. Such a lower bound is neverthe- less useful reject models with high grouping loss. In addition, we apply the tree the penultimate layer of neural networks, where class boundaries are simplified. Finally, complementing the proposed lower bound with an upper bound would also allow identify models without grouping loss.
We need to talk about grouping loss Model should be evaluated not only on aggregate measures, but also on their individual predictions, using grouping loss. The presence of grouping loss means that the model is systematically under-confident for certain groups of individuals and over-confident for others, questioning the use such models for individual decision making. The presence of grouping loss also means that downstream tasks relying on confidence scores can be hindered, such as causal inference with propensity scores or iauullaiion-asee inference. Finally, this heterogeneity raises fairness concerns. In fact, the grouping loss and our lower bound are fundamentally related to faimess ssee sufficiency and group calibration (Barocas et al., 2017, chap 3), and multicalibration (Kleinberg al., 2016). We hope that measure of grouping loss will spur research in this area.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
All datasets are publicly available (ImageNet-R, ImageNet-C, ImageNet-IK, Yahoo Answers Top- ies) and all models involved are pre-trained and publicly available on PyTorch and HuggingFace. Simulated examples are described in Appendix B. Detailed experimental methods are given in Appendix and E. Proofs all theoretical results are listed in Appendix The source code for the implementation of the algorithm, experiments, simulations and figures is available on GitHub: ht tttp::// com/ wpw.eeeeeeeebeeeeeeeeeeeeeeeeeee /beyond/bdddd.oo cal ibniiiiiion.
ACKNOWLEDGMENTS
We acknowledge support in part by the French Agence Nationale de la Recherche under Grant ANR-20-CHIA-0002 (Learnl).
REFERENCES
Rina Foygel Barber. Is fretrbbution-fre inference possible for binary regression? Electronic Journal of Statistics, 14(2):3887-3524, apr 2020.
Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. The limits of distribution- free conditional predictive inference. Information and Inference, 10(2):455-44) 4822, mar 2019. ISSN 20498772. doi: 10.48550/arxiv. 19t.i...99000448..
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning. Nips tutorial, 2017.
Jochen Bröcker. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Sociery: A journal of the atmospheric sciences, applied meteorology and physical oceanography, 135(643):1512-1119, 2009.
Jochen Bröcker. Estimating reliability and resolution of probability forecasts through decomposition of the empirical score. Climate dynamics, 39(3):655-667, 2012.
A. P. Dawid. Probability forecasting. 1986.
A. Philip Dawid and Monica Musio. Bayesian model selection based on proper scoring rules. Bayesian Analysis, 10:479-499, 9 2014. doi: 10. 10.1214/15-BA94
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and puttern recognition, pp. 248-255. leee, 2009.
Peter A. Flach. Classifier Calibration. Encyclopedia of Machine Learning and Data Mining, pp. 1-8, 2016.
Dean P Foster and Rakesh V Vohra. Asymptotic calibration. Biometrika, 85(2):379-390, 1998.
Matthieu Garcin and Samuel Stéphan. Credit scoring using neural networks and SURE posterior probability calibration. 2021.
Tilmann Gneiting and Adrian Raftery. Strietly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102, 2007. ISSN 01621459. doi: 10.1198/ 016214506000001433
Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(2): 243-268, 2007.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration modern neural networks. In International Conference Machine Learning, pp. 1321-1330. PMLR, 2017.
10

---

Published as a conference paper at ICLR 2023
Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. Calibration of Neural Networks using Splines. 2020. ISSN 2331-8422.
Ursula Hébert-Johnson, Michael P Kim, Omer Reingold, and Guy N Rothblum. Multicalibration: Calibration for the cimpuutaionalll-Idennfffi Masses, jul 2018. ISSN 2640-3498.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corr tions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of 2021.
generalization. ICCV,
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent Trade-Offs in the Fair Determination of Risk Scores. Leibniz International Proceedings in Informatics, LIPles, 67, sep 2016. ISSN 18688969. doi: 10.48550/arxiv. 1609.05807.
Max Kuhn and Kjell Johnson. Applied predictive modeling. 2013.
Meelis Kull and Peter Flach. Novel decompositions of proper scoring rules for classification: Score adjustment as precursor to calibration. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 68-85. Springer, 2015.
Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in neural information processing systems, 32, 2019.
Ananya Kumar, Percy Liang, and Tengyu Ma. Verified uncertainty calibration. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buce E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to--sequeee pre-training for natural language generation, translation, and comprehension. pp. 7871-7880, 10 2019.
Takahiro Mimori, Keiko Sasada, Hirotaka Matsui, and Issei Sato. Diagnostic uncertainty calibration: Towards reliable machine predictions in medical domain. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 3664-3672. PMLR, 13-15 Apr 2021.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai Neil Houlsby, Dustin Tran, and Mario Lucie. Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht, Obtaining well calibrated prob- abilities using bayesian binning. Proceedings of the ... AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, 2015:2901-2907 2015.
Christine Osbome. Statistical calibration: a review. International Statistical Review/Revue Interna- tionale de Statistique, pp. 309-336, 1991.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, gigg-performance deep learning library. Advances in Newral Information Processing Systems, 32, 12 2019. ISSN 10495258.
Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C Mozer. Mitigating bias in calibration error estimation. In International Conference on Artificial Intelligence and Statistics, pp. 4036-4054. PMLR, 2022.
11

---

Published as a conference paper at ICLR 2023
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In The 22nd International Conference on Artificial Intelligence and Statistics. pp. 1459-3467. PMLR, 2019.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in random world. Springer US, 2005, ISBN 0387001522. doi: (0.1007/b106715 15.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Papers), pp. 1112-1122. Association for Computational Linguisties, 2018.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probabil- ity estimates, In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 694-699, 2002.
12

---

Published as a conference paper at ICLR 2023
SUPPLEMENTARY MATERIALS
Table of Contents
Examples of confusing statements calibration
B Examples of accurate and calibrated classifiers with high grouping loss
14
B.I Arbitrary link between true posterior probabilities Q and confidence scores $
14
B.2 Realistic example based on neural network's output
16
Proofs
19
Lemma 4.1: The grouping loss as an h-variance.
C.2 Theorem 4.1: Grouping loss decomposition
19
C.3 Proposition 4.1: Binning-indueed grouping loss
20
C.4 Proposition 4.2: Explained grouping loss accounting for binning
20
C.5 Proposition 4.3: Debiased estimator for the Brier score
20
C.6 The plugin estimator for the grouping loss lower hound is biased upwards.
23
C.7 Estimator for the induced grouping loss
24
C.8 Analysis of binning-inducce errors for the Brier score
24
Extension classwise calibration
26
C.10 Impact of recalibration the grouping loss
29
D ImageNet
30
D.1 ImageNet-R
30
D.2
ImageNet-C
31
D.3 ImageNet-18 validation set
31
E NLP
32
Grouping diagrams of all vision networks
35
ImageNet-R
35
ImageNet-C:
40
ImageNet-IK
46
13

---

Published as a conference paper at ICLR 2023
A EXAMPLES OF CONFUSING STATEMENTS ON CALIBRATION
Here we detail specific examples of confusing statements on calibration in the literature. We choose most of these examples well-cited and well regarded works.
Kuhn & Johnson (2013): "We desire that the estimated class probabilities are reflective of the true underlying probability of the sample. That is, the predicted class probability (or probabiliy-like value) needs to well-calibrated. To be well-calibrated, the probabilities must effectively reflect the true likelihood of the event of interest."
The authors write that it is desirable have confidence scores S reflective of the true posterior probabilities Q, which is indeed desirable as discussed in Section 1. However, they write this obtained through calibration. Although post-hoc recalibration makes the confidence scores closer to Q in some sense, there is an implicit shorteut. As pointed out in Section 2 and Appendix B, calibration, even with optimal accuracy, does not guarantee confidence scores S to be close to the true posterior probabilities Q.
(Gupta et al., 2020): "A classifier is said to be calibrated if the probability values it associates with the class labely match the true probabilities of correct class assignments."
The authors write that calibration is matching the confidence scores S of classifier the true posterior probabilities Q. In fact, calibration is matching the confidence scores S to the calibrated scores C, which can be far from the true posterior probabilities Q as pointed out in Section 2 and Appendix B.
Garcin & Stéphan (2021): "Ideally, we would like machine learning models to output accurate probabilities in the sense that they reflect the real unobserved probabilities. This exactly the purpose of calibration techniques, which aim to map the predicted probabilities to the true ones in order to reduce the probability distribution error of the model."
The authors write that calibration is outputting confidence scores S that are true posterior probabilities Q. As in the previous citations, calibration is outputting calibrated scores C, which can be far from Q (Section 2 and Appendix B).
Flach (2016): "A probabilistic classifier is well calibrated if. among the instances receiving predicted probability vector p. the class distribution is approximately distributed as p. Hence, the classifter approximates, in some sense, the class posterior." "The main point is that knowing the true class posterior allows the classifier to make optimal decisions. It therefore makes sense for classifier to (approximately) learn the true class posterior."
Here, calibration is rightly defined as outputting confidence scores S that are equal to the calibrated scores C. However, by writing that confidence scores S of a calibrated classifier approximate the true class posterior Q, the author makes implicit assumption that the calibrated scores C' are close to the true posterior probabilities Q, which is not guarantced in theory pointed out in Section 2 and Appendix B.
B EXAMPLES OF ACCURATE AND CALIBRATED CLASSIFIERS WITH HIGH GROUPING LOSS
Here we build simple binary classification examples of calibrated classifiers with optimal accuracy having their confidence scores far from the true posterior probabilities. In Appendix B.1 we build examples with an arbitrary link between true posterior probabilities Q and confidence scores S (up to a limit to keep the classifier's accuracy optimal). In Appendix B.2 we build a more realistic example based on the output of neural network.
B. ARBITRARY LINK BETWEEN TRUE POSTERIOR PROBABILITIES Q AND CONFIDENCE SCORES S
To show that calibration, even combined with optimal accuracy, does not impose strong constraints on how close the true posterior probabilities Q should be from the classifiers' confidence scores S, we build examples in which and S have arbitrary link. For simplicity consider binary examples with a one-dimensional feature space X. These can be extended to multiple dimensions by projecting onto vector w (via + wF))
14

---

Published as a conference paper at ICLR 2023
h: 8 + -2 2s
h: m2n(2s.)
Negutive Positive
Negulive
Aigusppopadd m1
S(X)
S(X) Q
Px
-2
X
0
Confidence score 5
X
Confidence score $
Figure 9: Calibrated but not accurate. Example of calibrated classifiers S constructed from links following the procedure described in Appendix B.1. The accuracy of these two classifiers is not optimal as Q and S are not on the same side of the decision threshold (5) wherever Q # 2. Refer to Figure 10 for example with optimal accuracy. Calibration curves (in black on 2nd and 4th plot) are obtained from million samples.
The idea is to build classifier that outputs confidence scores having most two antecedents each. One antecedent should have its true posterior probability Q at an arbitrary distance +A from the associated confidence score S, while the other has distance AA Scores with only antecedent should have Q = S. This combined with equal density weight of X' onto the two antecedents guarantees calibration: E|Q S] S. To maintain the classifier's 's accuracy optimal, the offset A is constrained to keep Q and S on the same side of the decision threshold.
To achieve this, we cut the one-dimensional feature space X into three parts: R4, R and {0}.
As classifier, we take an even function SXX) with S ({0}) reduced singleton so that each confidence score has either two antecedents (one in R and one in IRt. or one antecedent in {0). To assign an equal weight each antecedent, we choose symmetric distribution for X' e.g. standard normal distribution centered on O. We build the true posterior probabilities Q from deviations h: [0, 1] + [0, 1] and g : [0, + [0, 1] of the confidence scores S in R4 and R":
: I(>x)(((z)) + Lecog(((z))
L===0((0)
(14)
For S to be calibrated, deviations must average identity, i.e. Vs S(R), S(R), g(()) A proof this statement is given below:
Proof.
E(Q(X) S(X) = E(1xso ((X)) o|g (()) S
= (0))
S(0)
since P(X > (|S()) P(X <()(())) = ( {(00()+00)))
Hence, S(X) calibrated + [[Q(X) (( = S(X) + ()(((()) + x(S())()
= S(X).
From here, we choose h :0. + [0, 1] and define g: 2s-h(s) hs)) Note that keep g(s) [0, 1]. h is constrained by: Vs E S())) 2s - < h(s) < 2s. At this point of the procedure, classifiers S may not have an optimal accuracy. Figure shows two examples of links h. one of which saturates the constraint h(s) <= min(2s, 1).
To make the classifiers accurate, the deviations h(s) should be small enough keep S and Q on the same side of the decision threshold. This adds two constraints on h: Vs E [()) 1/2 h(s)
and V's E S(R) n [/2, 1], h(s)) > 1/ (with the convention that score of exactly predicts the positive class). Figure 10 (left) shows a classifier built following the above procedure. Figure 10 (right) shows that release the constraint 1/2(h(s) g(s)) if we tweak the distribution of X' to adapt the weights between the two antecedents accordingly (and take e.g. g(s) = La())<a)
15

---

Published as a conference paper at ICLR 2023
h: 8 + max(min (2,, 2)), 2s - 1)
h : 3 + 2 Hlcce<l + 11=1
Negalive Positive
Negalive Possitive
mi Anrporrora
SY (U0)
S(X)
(XX
Px
-2
X
2
0
Confidence score
-2
Confidence
Figure 10: Calibrated and optimal accuracy. Example of calibrated classifiers S constructed from links following the procedure described in Appendix B.1. The accuracy of these two classifiers is optimal Q and S the same side of the decision threshold (2) wherever & 2' However, confidence scores are almost everywhere different from the true posterior probabilities Q. Calibration curves (in black on 2nd and 4th plot) are obtained from million samples.
B.2 REALISTIC EXAMPLE BASED NEURAL NETWORK'S OUTPUT
The examples of Appendix B.1, while proving point, quite unusual in practice especially in the choice of classifier S. In this section we build a more realistic example based on the output of neural network. We focus on binary classification setting with a feature space X' being least woodimeenional. The classifier is taken as sigmoid of w X (akin to the last layer of neural network predicting the confidence score of the positive class). Based on this choice of model, we build class of calibrated and accurate classifiers with confidence scores S far from the true posterior probabilities Q.
The idea is to create heterogeneity in the blind spot of calibration, i.e. orthogonally to w. The perturbations creating heterogeneity must balance each other out to keep the classifier calibrated.
To achieve this, we define:
the dimension
feature space X'
w e Rd the last layer's weights.
y R + [o, 1] the link function mapping wfx to confidence scores, e.g. sigmoid.
S Rd + ((w"z)) [0. 1] the classifier's confidence scores of the positive class, w1 = O, the direction which heterogeneity will be introduced. + : IR + 1] an odd perturbation introducing balanced heterogeneity along w...
wi e Rd such that wF
Amaz
min(1 S(x) modulating the range of the perturbation to keep 1].
Q Rd + S(x) x) maz (x) [0, 1] the constructed true posterior probabilities.
X ~ (0, 2 the data distribution, with E E Rdxd having w and w. among its eigenvectors.
With the above construction, the classifier S is calibrated. Indeed,
[[Q(X) = S(x) + [[> X man S(X)] = =((X) + [ S(X) Amaz (X)
(15)
(16)
since Aman (X) is a function of S(X). We have L [[][w] X)
S(X) by construction: e is odd and Hence
and the distribution of X has symmetric weight along w since Σ is aligned E[Q(X) S(X) = S(X). Figure 11 shows two examples generated with this procedure. However, is not yet accurate. As in Appendix B. I the perturbation should be constrained keep Q and S on the same side of the decision threshold to keep the accuracy optimal. This simply achieved by
defining Amar min(1 (xx)). S(x), 1 S(x))).
16

---

Published as a conference paper at ICLR 2023
Classifier S(X) (1 exp(-w" x))" -1
S(X) p0oo X)
S(X)
X
-2
X1
2
b. Perturbation (x(2 = 2(1 + exp(-2))
A(X) vi A) (X) A(X)
=0))=
Q(X)
X
Negative Positive
True pad Coisini Q
@
8 1
x====
S(X)
X1
-2
X1
Confidence score
Perturbation (x(2) 1,50 1
A(X) v(O X) (X) A(X)
QXX)= A(X)
Q(X)
2
X
X
Negative
Positive
True Metoeaaii Selballyy Q
i
"x x=0
5(X) =
-2
0
-2
Xi
2
-2
X1
2
Confidence score
Figure 1I: Calibrated but not accurate. Example of calibrated classifier S constructed following the procedure described in Appendix B.2. Its accuracy not optimal as Q and S are not on the same side of the decision threshold (5) wherever Q / 3.4 Refer to Figure 12 for an example with optimal accuracy. Calibration curves (in black on last column) are obtained from million samples.
17

---

Published as a conference paper at ICLR 2023
a. Classifier S(X) (1
exp(-w" x)) -1
S(X) p
S(X)
X
S=
-2
X1
b. Perturbation (x2) = 2(1 + exp(-2))
A(X)= v(oo X) (X) AXX)
0x0)
Q(X)
2
X
Negative Positive
True prosi Crmbubiiiyy Q
o X 0
S(X) ]
-2
0
X1
X1
Confidence score
Perturbation (02)
2>0
11 ==000
4(x)= X0 A(X)
=00)
Q()
X'
Negative Positive
True probability Q
" X
S(X)
-2
2
X1
2
2
X1
2
Confidence score
Figure 12: Calibrated and optimal accuracy. Example of calibrated classifier S constructed following the procedure described in Appendix B.2. Its accuracy is optimal as Q and S are on the same side of the decision threshold wherever Q However, confidence scores are almost everywhere different from the true posterior probabilities Q. Calibration curves (in black on last column) is obtained from million samples.
18

---

Published as a conference paper at ICLR 2023
C PROOFS
C.I LEMMA 4.1: THE GROUPING LOSS AS AN H- H-VARIANCE
Lemma 4.1 (The grouping loss h-variance). Let be the negative entropy of the scoring rule
i.c h: :p (s p). The grouping loss GL of the classifier S with calibrated scores C = E[Q S] and scoring rule writes:
[[d((C, Q)1 [|
(7)
GL(S)
Proof of Lemma 4.1. Let be scoring rule, h:pt (p, and C [[
(C, Q)) = [Cs Q) (Q(
Definition of divergence (17)
= =[(s((C,)) M(Q))
Definition of h
(18)
K
Qx +QQ)
Definition of expected score
(19)
E
E
ex + (QQ)
Law of total expectation
(20)
Elo(C, en Qx E[A(Q)
Linearity of expectation
(21)
=
E
e(C,ex) |) E((())
(C,, ek) is a function of S
(22)
E
Σ (CC, +
+ (((())
C F[Qx
(23)
F[-M(C) ([((Q) (S]
Definition of h
(24)
E[E(A(Q) 39-h(E[() ST)
C E[
(25)
E[ViQQ [S]
Definition of [|
(26)
C.2 THEOREM 4.1: GROUPING LOSS DECOMPOSITION
Lemma C.1 (Law of total h-variance). Let X, Y, Z S + Rd be random variables defined on the same probability space and function f Rd R. The law of total variance holds for the f f-variance:
1 2]
| YY X, 2) 2 VE |x 2]
(27)
Proof.
E(((Y) = E[E[((Y) [X]
Law of total expectation Definition of Vy [Y XX]
= E| X]] + EESEEJ [X]]
-(E)))) = E| X]] + EEEEY - ((EEY(E (X))
Law total expectation
E| X + Vi[E[Y [X]
Definition of V/ |E[Y X]]
The same proof holds when the expectations and ht-ariances are conditioned on Z.
Theorem 4.1 (Grouping loss decomposition). Let R X' - be a partition of the feature space. holds that:
GL(S)= = [IV S
(8)
Gleeptine
Glaoster())
Moreover if the scoring rule is proper, then:
GL(() > GTepplaind (S) 0.
19

---

Published as a conference paper at ICLR 2023
Proof of Theorem 4.1. Applying Lemma C.I with (R, S) as (X,Y, Z) gives the decomposition. Proper scoring rules have a convex negative entropy h (see Gneiting Raftery, 2007, th. 1). Note that depending on the convention (maximization minimization of scoring rules), may find in the litterature that the entropy is either convex or concave. In the convention taken by this article (minimization of scoring rules), the entropy is concave and the negative entropy is convex. Using Jensen's s inequality, we thus have V% [Q] S, 3] 2 0. Hence both GLeppliind and Geesiiiul are positive, which gives GL 2 Lexplained.
C.3 PROPOSITION 4.1: BINNING-INDUCE) GROUPING LOSS
Proposition 4.1 Sehning-inducced grouping loss). The grouping loss of the binned classifier GL(SB) deviates from that of the original classifier GL(S)
(S) by an induced grouping loss GLinndued SB):
EQ SBl]
[QIQ + | |BB]
(11)
GL(Sn)
GL(()
Moreover, the scoring rule is proper:
GLinduced (S,Sp SB) 2
Proof of Proposition 4. I.
Vi. Sa] [[V,QQ S, [5]] [ [, [Sn]
Law of total h-variance (Lemma C.I)
[[v. 5 [B] + + [S] [BB]
SB a function
= E[V.[Q 5] [n] [C | Sp)
C = F(Q S}
EQ Sp]] = E|V [| SBll
Law of total expectation
GI.(SB) = GL.(S) + GT induced (S, Sn)
Lemma and definition of GT Giindceed
Remark: this proposition does require Sp to be the average scores the bins E[S| B,].
O
C.4 PROPOSITION 4.2: EXPLAINED GROUPING LOSS ACCOUNTING FOR BINNING
Proposition 4.2 (Explained grouping loss accounting for binning).
GL(S) GLapplained Sn) Liinduced (S, SR) + Geneiidud (Sn)
(12)
If the scoring rule is proper, then:
GL(()) 2 GTeepplined (SailaiinddSS) SB) - GLindduced S,S
(13)
GLan(S.9n)
Proof of Proposition 4.2.
GL(S) = GL(Sn) GLinduced Sp))
Propostion
GLexpaaned + GLresiduat (S GLiinducdd (S, SB) Theorem 4.1 on GL(SB)
For proper scoring rules, Theorem 4.1 gives GLeesiduat (S 2 which completes the proof.
C.5 PROPOSITION 4.3: DEBIASED ESTIMATOR FOR THE BRIER SCORE
Proposition 4.3 (Debiased estimator for the Brier score). For all class k E {l,..., K) and bin E S, let (s.k) (resp. ni k( be the number of samples belonging to level set R(d) (resp. region We define the empirical average of Y over these regions as:
R,))).
(s.h)
1
Σ
and c(a,K)
1
YO
(3
n
O
(
K
(a.b)
(a.k) Gappaanneed (SB)
The debiased estimator of GLeespiained is: GT TTeeptained (Sa
GL
GL
Am #ES
20

---

Published as a conference paper at ICLR 2023
with:
(e,k) n(skk
mj mhi mn (1 [1g nlackl)
(s)
claik) - 2(a.k)) 1((,k) 1
GL (a.k)
(nfoak)- - clam))
eagpliiied -
nl,,))
(s,k)
n
plugin estimator Klpinnio
bias estimation CLasiss
Proof. Let 8 E ( and k E K1 and define 2 plugin estimator for GL (o.k) explainedd To ease calculations, we start by rewriting the plugin estimate:
nto,yy We now compute the bias of the
(a.k)
GT (aik)
plugin
Σ p3
((ak))
(nf - cl0.k)
(28)
1=1
slaak) (nfaat)) 2 20(1.))
Epjoro (()kk) ( plaus) + (alamt))
Σ Pj
(29)
j=1
Σ Pj
(a,k)
(ofaah))= (ala.t))2
(30)
==1
From now we omit the exponent (s, k) to lighten notations. We now take the expectation of both terms the lower-bound.
E [0]] = EE [c]² + Var(c) = =2 c(1 - c)
(31)
(32)
where made of Lemma C.2 for equation 32. Similarly,
[
[,] = E [Ay] + Var(ft))
(33)
= n3 (1 - M3)
(34)
n
When ny 0 (or equivalently 9Ú 0), which happens with probability vÚ (1- A well as the right term in equation 34 are undefined. The problem disappears when multiplying by Pj. and agreeing that ü. whenever n, 0.
E
E E
$,20
[py]]
(35)
113 + 111(1 14)
Σ
E
13,00,20
(36)
n2
Σ
11 - 11j)
Pipi + E 1 18,20
(37)
Σ
[2jie + (1 vj) ">{ 11 M4) (;())
(38)
1=1
21

---

Published as a conference paper at ICLR 2023
Putting together equations 32 and 38, we get:
E
GL plugin
=
Σ
Psll - c² " Uj) 14))
c(1 - c)
(39)
n
F=1
j=1
Σ p5luz p5( c) Σ - vj)
U1 1z)
c(1 c)
(40)
n
n
==1
Gl Gappliine
GTaa
In practice Vj, which gives the probability that no sample falls in component j. is very close to unless Pj and n are very small. Hence, we will approximate yj More importantly, the expression of the bias given in 40 depends on oracle quantities μ4 and c, which are unavailable. Therefore, we resort debiasing the plugin estimate of the lower-bound using sample estimates of the bias, which gives:
Gl (s,k) expiaiied
(a.k) 12
(2/aah)) glock)
n (0.8) p. (1 f1. ((..k)) +
c(a.))( ela,k) n(,,)) -
(8.))
((an)) n (1ah) -1
plugin estimator G
(41)
where we used Bessel correction for the estimation of population variances. Finally, debiased estimator of GLexplaine is obtained by summing over the debiased estimators for all & {1, K} and all E Sk.
Lemma C.2. Define o (2.)) and ala,k) as in Proposition 4.3. Then:
Hg Hij (s.K)
(a.8)
E [1g [((c))]
(a.k) and Var (nfeth)
11j
(42)
nj (s,k)
Similarly,
F [clo.k]] (o.)) and Var (clekt) clack) (1 clock) nl(,k)
(43)
The labels y are by definition drawn from Bernoulli distribution with probability P(Y (x() = of i.e., for each sample i, the probability of the Bernoulli changes.
This lemma shows that despite these varying Bernoulli probabilities, the empirical average of labels Y, over a given subspace has the same expectation and variance as binomial variable that would be drawn with probability equal to the expectation of Q₁ over this subspace.
p.) Proof of Lemma C.2. Below write the proof for the case of (((.k) (equation 42) as the one for a(s.k) (equation 43) follows exactly the same lines. Let I(01 ( X E RR(0) }. be the subset of samples such that (0) belongs to bin R.
22

---

Published as a conference paper at ICLR 2023
[
1
E ( S - R((((() -
E
(44)
(o:k)
nj iEI
E E A
(
xx()] SSs - (x((( =j
(45)
n (2.k)
sEz("))
1
E 0 (
SSx = = R( =j]
(46)
(s.k)
n 1EIJ")
1
Σ
(47)
(e.k)
lEZ!")
(48)
where we used the law of total expectation in eq 45, the definition of Qx in eq 46, and the definition of (1 in eq 47.
Var (nfak))
F (infak) - 113 1//20112/ E (nfaas)) 2
Sh = =,,(((())) =s]
(49)
Sx = =8,((((()) ]
(nfaak))² 2
(50)
nnj ( 2
E
Σ Y())
Σ
(1)
x
Sx s, 2(x(1)) =j = - ((ufuit)
2
(51)
REZ!
(Ez(1)
F
F
Σ (+)
+ Σ
y(
t k
S2 = s, (((((())) = -
2
iET
ifFt
(52)
1
Σ (ork)
Σ (ugaee)) 2
- -(ugast)) 2
(53)
(n(fa8))
(EIJO
ixit
1
n pl (1.)) + n (c.h) (2 (0.h) 1) (1)340))² (n(omt)) 2
(a.b))
(54)
(nn
(n(oa))) 2
(0.8)
(
(55)
(8)
where we used the fact that (i) and Y! (1)
are independent when i ef in eq 52.
C.6 THE PLUGIN ESTIMATOR FOR THE GROUPING LOSS LOWER BOUND IS BIASED UPWARDS.
Analytical evaluation of the sign of the bias Let & E {1,.... K} and s E S. The bias of the
plugin estimate GI (s.8)
GT (SB) is given by (40):
Σ (1 vfa.8)) M)) (oock) (1 11, (
bias GT (a.k) (Tpeethhnnndd (SB)
clack) cleck))
(56)
mka,k)
n
1=1
23

---

Published as a conference paper at ICLR 2023
By convexity of the function x. (x - E [x:]) we have:
2
(s.k) (0.8)
n (s,h)
(s,h)
Σ nlak)
-E
n(a.k) fig (6)
(afock)
-E [E(faut)))))))
(57)
n(a,k)
Using the fact that g(o.k) = Ej=1
mi
(s,k)
and taking the expectation of both sides, we get:
((.kk)
Var(c(s.))
[(,,k) Var(/z/ok))
(58)
Finally, using Lemma C.2, we get:
(s.k) (1
(s.k)
cla.k) (1 - cls,k))
Mj
(59)
()),b)
ks,,k)
j=l
Hence, we have:
J Mj (1 14 (((rk)
(2.k)
c(a,)(1 c(a.))
ylakk Mi (1 M4 (aak)
(0/1)
bias (Faz)
(60)
(((,6)
m((,k)
20
v (s,k), this incquality does not prove that the bias is always positive.
Because the term involving
However in practice v/(a.h) (1- (=b)) n (4.))
which represents the probability that no point
belongs region j, is very close to 0 unless (s.k) is very small or the total number of points n (*.A) small. Hence, equality 60 shows that the bias can only be 'slightly' negative. In the simulations below, the upwards bias of the plugin estimate appears clearly.
C.7 ESTIMATOR FOR THE INDUCED GROUPING LOSS
Proposition C.1 (Estimator for the induced grouping loss). Let Ò be an estimator of C. An estimator of CB is CB((s)= = nल. Ei:s (()((())) with (5) the number of sample in the level set s. An estimator of the grouping loss induced by the binning of S into SB is:
GT G-induced (S, (S,SB)
Σ
(*)
1
Σ
(((1(((1)))(Caal)
(61)
n
(()
i:S ix
(x
C.8 ANALYSIS BINNING-INDUCEN ERRORS FOR THE BRIER SCORE
It is well known that binning can induce error in estimating calibration loss, leading to underestimating (Bröcker, 2012; Kumar al., 2019; Roelofs al., 2022). Proposition 4.1 shows that it also leads to errors on the grouping loss, overestimating it. Here we characterize the errors on the calibration and grouping loss for the Brier score and show that they partly compensate each other and the error on the sum of both can be bounded.
Proposition C.2 gives the deviation term induced by the binning for the calibration loss with the Brier scoring rule.
Proposition C.2 (Calibration loss decomposition). Let h be the negative entropy of the Brier scoring
rule and C E[Q S). The binned calibration loss CL(SB) deviates from the calibration loss CL(S) by a negative induced calibration loss CT Cniuuee (, SB):
E [
Cpll2]
[|| - C
E S C
(62)
CL(SB)
CL(S)
CLinduuced ( S.SB)
24

---

Published as a conference paper at ICLR 2023
The calibration loss induced by the binning, CLinduced S, SD), is always negative. CL(SB) is thus biased downward, which is already known from Kumar et al. (2019); Roelofs et al. (2022). Conversely, the grouping loss induced by the binning, (S,ndweed Sp). ). is always positive. GL(SB) thus biased upward. The mere effect of binning artificially creates grouping loss and artificially reduces calibration error. For calibrated continuous classifiers, Ciiinduced 0 and induced grouping loss is small: with N equal-width bins, GLinduced S INT. If in addition the scores are uniform on
the bins: GLimduend I2N (Lemma C.3). Both induced calibration and grouping losses can be large since [p] can be large. High Giiieceed expresses strong miscalibrations within the bin. However interestingly, both induced losses compensate. In a binary setting, the sum of induced calibration and grouping losses is contained as showed by Theorem C.I, and can be bounded by estimable quantities (Corollary C.I). While measuring CL(SB) and GL(SB) separately can lead to high binning-induced bias, measuring CL(SB) + GL through (L(5 + @T.eppaained (S)) enables reducing binning-induced errors and minorizing MSE(S, Q) (Corollary C.2).
Theorem C.1 (Bounds on induced calibration and grouping losses). In a binary setting, the calibra- tion and grouping losses induced by the binning of classifier S into Sg sums
CLindaced + GLinduced E[2Cov[S C| C|SB]
|
which is bounded by:
-E -E( [[VVSISS (2/MCISEE / (SB)BB)
CLinduced + GLinduccdd
S
MSS Spl
/2/C seai-VV5:. /VSSSB)
Suppose that [0, 1] is divided in N equal-width bins. Then:
- C - ANo S CLiinduced + GLinduced S VCB(L CB)
Corollary C.1.
-E V S CVCCB(1-C) - VSS / S CLindured + GLiinduced
E VVSISSl (2/Co(1- vis SB)
With N equal-width bins:
+ [ Cn)
CLindueed + GLindured S E CEE[VCn([1 Cn)
INT
Corollary C.2. The mean square error (MSE) between continuous S and Q is lower bounded by:
MSE(S, Q) = CL GL
2 EEECEB + LGLE
LE MS SBl
22/CIC Spl
V v
2 E-CCEB + LELF
E MMS Syl
(2/CB(1-CB)) /VSl
With N equal bins: 2 ECE, + Lar.
E (E[(VC(11-C))
where (2 E-EC is the J Expected Calibration Error of the binned classifier Sg and LGLa the grouping loss lower bound of Sp-
PROOFS
Proof of Proposition C.2. Let the negative entropy the Brier scoring rule.
SA
2+ll² = ||E[S 5m [E||| Sn] 1|²
Sn = [[S Sn) Cn = E(C) Sn]
||E[S C S 1|?
Linearity of expectation
E [[]|s - 2 Sn] V C1Sn]
Definition of V, C Sn]
E( 3n
E[|| - c - E C Sa]]
Law of total expectation
25

---

Published as a conference paper at ICLR 2023
Lemma C.3. In binary setting, suppose that [0, is divided in N equal-width bins. Then:
VIS Sp] S 1
(63)
AN²
If in addition, scores S are uniform:
vIS SB] 12N
(64)
Proof of Lemma C.3, Without loss generality, consider the first bin [0, # with binned score 81.
V[S| SB s1] = E SB = s1] E[S SB = ==1]²
Definition of the variance
S E[S SB t1] |[S| 5B =11
05 N = s² S
Nウ (1 - NE[S | Sn = NI]) SB 81]
$ 4N2
Max when NE(S SB = $1 =
For uniform scores: S = 81 U( 1). Hence S = =1] - -03 T2N
Other bins have same variance as MS SB $1] (variance is translation-iivvaiiant)
Remark: this proves that GLinduucdd I 4N2 for S calibrated (S = C - |C||sz) = ViS SB])
Proof of Theorem C.I. In a binary setting for the Brier scoring rule, have Hence:
CLinduced + GLinduced = -E Ei,, S S +
E S -E[V(S Cb E[V(C Se]]
Propositons 4.1 and C.2
1/2 V
= | viS [R]]
Expansion of V/S - C S(S
2Cov|S, Sp] V(S Sp) < 2 / /CC Snl VSS SR)]
Cauchy-Schwarz
2Cov|S, C' Sp] V[S| S3] 2 2 Cor SB] | VSS Sp]
2/
|
Cauchy-Schwarz
With N equal-width bins:
@Cov|S,Cr | MS|Sp < / SB MC Ss]
Positivity of the variance
$ N /MCSSE
VSS Spl S AN2
$ N v Cn(1 Cp)
MC! Sp] S Cn(1 CB)
2Cov|S, C [ Sa] 2 + VCC SM] INZ
MSS |S|Sp] S .Nन
1
2 CB(1 -CB) 4N2
VC Sp] S Cb(1 CB)
C.9 EXTENSION TO CLASSWISE CALIBRATION
C.9.1 PROPER SCORING RULES DECOMPOSITION
We show below that the proper scoring rules decomposition of Kull & Flach (2015) holds for cuassise-calibration (Definition 3.2) for the Brier and the log-loss.
Proposition C.3 (Brier and log-loss classwise decomposition). For the Brier score as well as the log-loss, the decomposition into calibration, grouping, and irreducible losses (Equation 6) holds when replacing the calibrated scores by the
scores (Definitio 3.2).
Proof of Proposition C.3. For all & E {1,..., scores (Definition 3.2).
14...,KK, K), let Ct = E [Yr[Sp] be the classwise-calibratedd
26

---

Published as a conference paper at ICLR 2023
Brier Score score reads:
Given any two probability vectors P and Q. the divergence associated to the Brier
K
d(P,Q) = Σ (Pk Qk)²
(65)
k=1
For all k K.....,K,IK let dx Pk. Qn (Px- Qx)².
(x Yi) (Sy- Yx)²
(66)
=(Sk- Ck + C - -Qk + + - Y/)²
(67)
(Sk- 10--(p -c,) + (C²- - -Qx)² + (Qx - -Y,)² + 2(x - -C)(C - Qk)
(68)
+ 2 - Cx)(Qx Yx) + 2(Cx Qx) - -Yt)
Taking the expectation on both sides conditional on X:
E [d4 Yx) x] = (Sx Cx) + (Cx Qx) + [[Qu - Yx)) |
(69)
+2(Sk - Cx)(Cs - Qx)
since S₁ and Qs are function of X. C& is a function of S₁ and thus of X, and E [Yx | |x] Qk-
Then taking the expectation conditional on Sg:
E (Sk, YR) Sk] = (Sx - )² + E [C Qx) Sp) + +E - Yu)² Ss}
(70)
where we use the fact that Cj is function of St. that E [Qx S.] Cv. and the property according to which for two random variables U and V and a function h, [V UJ ((U) E M(U)). Finally, taking the expectation over S₁ we get:
E [dx (S&, Yx)] = E [(S Ct)
- Q+) + E [[Qx Y+
(71)
The desired decomposition is then obtained by summing over the K classes on both sides.
log-loss
Given any two probability vectors P and Q, the divergence associated to the log loss reads:
K
Qk
d(P,Q) Σ
Qx log
(72)
Ps
For all (1...., K}, let de Pk. Qk + Qs log
Ph
Yk
de(Sk, Yn) = Yu log
(73)
St
Yv
Qx C₁
Cx
= Vy log
+ Y log
+ Y log
(74)
Q2
Sx
Yv
[xk
Ck Sk
E (S Yo) XX] = E
Yu log
X
+ Qu log
+ log
(75)
Qx
Ck
Yı
Qk
Ck
E (d Yx) [kp] = E Yx log
S
+
Qx log
Sx
+ Cx log
(76)
Q₂
Sp
= E [dl (Qu, Y2)
| Sk] + E (C Qx)
Sk] + (Se. Ck)
(77)
where we have used the same properties as those described for the proof of the Brier score classwise decomposition above. The desired decomposition is then obtained by taking the expectation over Sy and summing over the K classes.
The proper scoring rule decomposition holds for top-label calibration. Unlike classwise cal- ibration, tapelabel calibration does not define vector C' E RK of calibrated probabilities. Instead, defines a notion of calibration for simpler binary problem in which labels indicate whether the classifier predicts the correct class for given X. More precisely, the labels for this binary
27

---

Published as a conference paper at ICLR 2023
problem are given by Y' := 1y= =e max())) Since S is function of X, the random variable Y' is function of Y and X. Define now the scores associated to this binary problem as S ;= max(S) E R. Reformulated in terms of these notations, top-label calibration states that S' is well calibrated if for all s, P(Y" = S = s) 8. Thus, for classical binary problem, we can define C' E and Q' = [[ C" (resp Q") gives the probability that the classifier predicts the correct class for given score S" (resp. given input X). As the quantities S', C", Q' and Y' define classical binary problem, the decomposition (6) into calibration, grouping, and irreducible loss holds for this problem. Compared to the classwise definition of calibration and grouping. here the calibration loss measures whether on average over all points scored S across all classes, the proportion of correctly predicted points in actually S. In this setting, the grouping loss also measures to what extent there exist over-confident scores for certain classes that compensate under-confident scores for other classes.
C.9.2 RESULTS HOLD FOR BRIER AND LOG-LOSS IN CLASSWISE SETTING
Appendix C.9.1 proves the scoring rule decomposition (6) in a classwise setting for Brier and log-loss scoring rules, which is necessary for the other results to hold. However, the proof of Lemma 4.1 does not readily apply to classwise calibration. Equation 22 uses a conditioning on the full vector of joint confiences S move (CC, ex) outside of the conditional expectation on S and turn Qs into Cg in expectation. In classwise calibration the conditioning is on each marginal Sy instead of the joint S. As a result, in the general case, oCC, es) cannot be moved outside of the conditional expectation given S₁. since C depends on all marginals of C, not just Cg. However for some scoring rules, <(p ex) depends only on Pk and the proof can be adapted. This the case of the log-loss for log(Pk).
which &i (p, ex)
Lemma C.4 (Adaptation of Lemma 4.1 for classwise calibration). Suppose there exists g: RRK + R such that for all k in {1....,K K} and x in RK,, 6(z, ex.) = =((xx) Define h2 p - ex)Pk. the th component of the negative entropy of the scoring rule 6. The grouping loss GT. of the classifier with calibrated scores Cs= = E[Qx S&k and scoring rule & writes:
(l)o(C,Q)
ElVo | [
(78)
GL(S)
Proof of Lemma C.4. Define the vector with C2 = E[Qu Sy] for all k in {l...., KJ. Let ø be scoring rule, h p + (). Suppose for all ls in {1,..., K), Φl2, ex) g( with
9: RKKK + R. Then:
(C, ([sa(C, - s (Q,
Definition of ds (79)
= L ER K= ((C,ex)Q + Ek- ((Q,ew)Qu))
Definition of ss
(80)
ΣX Eo((C, Qx ex)x
Linearity of expectation
(81)
Σk=1 [l((Ca)Q [Qw)
Hypothesis on ø
(82)
ΣK E[Elo(CA) S&] [l |S]]
Law of total expectation
(83)
= E [L Sk] - Elo(Qu) Sk]l
C' is a function of S₃
(84)
-K Cl Elg(Qx)4+ [
Definition of C
(85)
=k=1 EIV,, 1Qx Sk]l
Definition of Vh2
(86)
Theorem C.2 (Results in classwise setting). Suppose Equation 78 is satisfied for the scoring rule 6. For all & E (1...., ...,, K}. ler RR X' + NN be a partition of the feature space. It holds that:
GL(S) = k== El [E[Qx Sx, RR] [Sa]]]
+ E 1I Sy, RR]l (87)
K
GLeppiane (9)
GLossterS)
K k= Elhh Q2 [B.]] || ΣK
= Σ I= El 10 Sel] + EK Ak ElVha |C2 SS...
(88)
GL(Se)
GL(S)
Glintecns (S,Sp)
GL( GLeeptained (SB) GLinduucdd (S, (S,Sz) + Gloresidual (SB)
(89)
28

---

Published as a conference paper at ICLR 2023
Moreover, if hy is comvex, then:
GL(S) 2 eeeppiaine (S)
(90)
GLinduced (S, SB) 2 0
(91)
GL(S)= 2 Geexpiiined (SB)
GLiindued SB)
(92)
GLLB(S,Ss)
Proof of Theorem C.2. Applying the law of total variance (Lemma C.1) on each of the Vn, [Q S with R₃ as conditioning variable proves Equation 87. Similarly, applying the law of total variance on each of the Vn, 1Qx S. with S₂ conditioning variable proves Equation 88. The proof for Equation 89 is the same as Proposition 4.2.
Using Jensen's s inequality, if hg is convex, then V/, 0, which proves Equation 90, and 92. L
For the log-loss scoring rule, have " exe) log(ps) and hx(p) = log(pk) )pkk wich is convex. Thus, Theorem C.2 holds for the log-loss. Unfortunately the Brier score does not satisfy the
assumptions of Lemma C.4 since &B (p. 6*) is not function of pt- But forumlation similar to Equation 78 holds for the Brier score:
[dgns (C, []]
F [[sans (Cans(C,q)) (Q Q)]
Definition of dgns
(93)
E[(C Q) (C Q)]
Definition of sges
(94)
E[(C C -C2C Q Q Q))
(95)
E[(Q Q CC
E(C Q) C
(96)
ΣK, [[ [
Linearity of expectation
(97)
= ΣR [[ Sp] - CR)]
Law of total expectation
(98)
K Σ 1= E(VQu& [Slll
Definition of the variance
(99)
with:
(100)
E[C Q) = ΣK_ E[CAQN]
(101)
= ΣR E[E[C&Q Skll
Law of total expectation
(102)
ER E[CRE[Q& [kk]
C₁ is function of Sy
(103)
-EK E(C])
Definition of C*
(104)
= =E(C-CO C1
(105)
Since V = V/ with f - 2 Equation 78 is satisfied for the Brier score. Since is convex, Theorem C.2 holds for the Brier score.
To conclude, Theorem C.2 holds for the Brier score and the log-loss in classwise setting. It is likely that some other proper scoring rules satisfy Equation 78 and Theorem C.2.
C.10 IMPACT OF RECALIBRATION ON THE GROUPING LOSSS
Lemma C.5. Let è be recalibration mapping and S' E(S) the classifier recalibrated with that mapping. The grouping loss of the recalibrated classifier GL(S') deviates from that of the original classifier GL(S) as follows:
GL(S") E [C]]
the mapping is perfect (i.e. S' C)) or invertible, then GL(S') GT.(S).
29

---

Published as a conference paper at ICLR 2023
Proof of Lemma C.5.
GL((S') = [ [S]] [I (S' [[[W/S [[]))) Law of total h-variance on S (Lemma C.1)
Definition of GL
= E[V.QQ S]] + [[[
S' is a function of S Definition of GL(() and C
= =GL(S) + [[V[C]'
If 8 C. then k/ O, hence GL(S") GL( If the mapping is invertible, then knowing S" is knowing S. Hence V4 [C [,[C| = since C' function of S. Hence GL(S") GL(S).
D IMAGENE
ImageNet-IK IK (ILSVRC2012) (Deng al., 2009) a classification dataset for computer vision with I 1000 classes. Networks studied in this article are pre-trained on the training set of ImageNet-16 IK, com- prising 1.2 million samples. Models' architectures and weights are available on Py TyTorch v0.12 (Paszke et al., 2019). We evaluated the networks on ImageNet variants' ImageNet-RR R and ImageNet-C (Ap- pendix D.1, D.2) as well as the validation set of ImageNet- 1K (Appendix D.3). We work in the high- level feature space of the networks, iee. the output space of the penultimate layer (embedding space).
For each of ImageNet-R, ImageNet-C and the validation set of ImageNet, we plot the grouping diagrams of each network with and without post-hoc recalibration, obtained with balanced decision stump. For each network, if several versions are available, we study both the smallest and the best performing one the validation set of ImageNet-Il 1K (usually the largest version). For ImageNet-R we also provide the grouping diagrams obtained with a 2-cluster k-means. Each experiment is detailed in Appendix D.I. D.2 and D.3.
Detailed experimental method First, we forward each sample of the evaluation dataset (ImageNet- R. ImageNet-C the validation set of ImageNet-I I)) through the studied network. We build confidence scores by applying softmax to the output logits. We extract representation of the input images in the high-level feature space of the network (i.e. the input space of the last linear layer). Since there is not enough samples per class (50), we restrict our study to the top-label problem (Definition 3.3). For each sample, the class with the highest confidence is predicted. The label is the network predicted a correct class (0 otherwise) and the associated confidence score is the one of the predicted class. divide the samples the evaluation set in half making sure that the confidence score distribution is the same in both resulting subsets. On set, train the isotonic regression for calibration and calibrate the confidence scores of both sets. If no post-hoe recalibration is used, we skip this step. Then, we create groups of same-level confidences by binning the confidence scores with 15 equal-width bins in [0, 1). We partition each of the 15 level sets independently. For each of them, create the partition by training the partitioning method on the training samples of the isotonic regression. We then evaluate region scores on the remaining samples to avoid overfitting. For the grouping diagrams, we mainly use balanced decision stump with 2 clusters (e.g. using scikit-learn's 's Decis s ion wowTTreeeeeeeeeeee in the bin), resulting in one split along one of the axis of the high-level feature space. For comparison, we also used k-means with 2 clusters. Constraining the partitioning methods to 2 regions is choice to provide visually informative grouping diagrams rather than to maximize the lower bound expiiiine When optimizing the lower bound, (Figure 7 and Figure 14), we increase the number of allowed regions in the partition by setting a region ratio: the number of training samples in the bin over the number of allowed regions in the bin. Fixing region ratio prevents from having regions with too few samples. In our experiments, we fix the region ratio to 30.
with min_ sann___aa___ samples leaf taken as half the samples
D.1 IMAGENET-R
ImageNet-R (Hendrycks et al., 2021) is variant of ImageNet containing renditions of the ImageNet classes. Example of renditions are: paintings, toys, tattoos and origami. There are 15 rendition types in total listed in Figure 13. The dataset contains 30 images and is limited to 200 of the 1 000 ImageNet classes. Figure 14a. compares estimated grouping loss lower bound and calibration errors of all networks (small and best versions) ImageNet-R. Overall, we observe strong grouping loss
30

---

Published as a conference paper at ICLR 2023
Deviantart
Sketch
Art-
Network
Figure 13: Comparison of renditions on ImageNet-R, Differences between the calibrated scores of samples of one rendition Crendition and the calibrated scores of all samples Call. weighted by the number of samples of this rendition in the level set and summed over the 15 bins confidence scores. Interpreta- tion: should renditions define regions of the feature space, they would ex- hibit a high grouping loss lower bound.
VGG-11
Videogame
ResNet-18
Misc
AlexNet
Renditions
Toy
DenseNet-121
Inception
Graphic
GoogLeNet
Sculpture
ShuffleNet
Origami
MobileNet V/2
Graffiti
ResNeXI-50
Wide ResNet-50
Sticker
MNASNet 0.5
Tattoo
EfficientNet-BO
Cartoon
RegNet y 4.0mmf
ViT B- 16
Embroidery
ConvNeX: Tiny
-0.1
Cenutiion
Call
in most of the networks, especially those with highest accuracy. The estimated debiased lower bound remains high after post-hoc recalibration. Grouping diagrams of all networks available at:
No post-hoc recalibration, small versions. Section D.1.2: ImageNet-R -
Section D.1.1: ImageNet-R -
No post-hoc recalibration, small versions, k-means.
Section D.1.3: ImageNet-R No post-hoc recalibration, best versions.
Section D.1.4: ImageNet-R Isotonic recalibration, small versions.
Section D.1.5: ImageNet-R Isotonic recalibration, best versions.
We also investigate whether there is heterogeneity among renditions. In Figure we observe that some renditions are better predicted than average (e.g. deviant art, sketch or art) while some others predicted worse than average (e.g. embroidery, cartoon, tattoo). These considerations would useful in fairesss setting. Also, Figure 13 highlights that if we could build regions out of renditions (i.e. ren- ditions are well separated in the feature space), this would result high grouping loss lower bound.
D.2 IMAGENET-C
ImageNet-C is variant of ImageNet containing corrupted versions of ImageNet images. Examples of corruptions are: blur, noise, saturate, contrast, brightness and compression. There are 19 corruption types in total. Each corruption has severity ranging from 5. The dataset contains the 50000 images of the validation set of ImageNet, each of them being applied 19 corruptions with 5 severity levels each. We built merged version of ImageNet-C by randomly sampling one corruption for each image. We also study one corruption only (snow). For both the merged version and the snow version, we study the maximum severity the corruption (5). Figure 14b. and c. compare estimated grouping loss lower bound and calibration errors of all networks (small and best versions) ImageNet-C merged and snow. Overall, we observe similar effect than on ImageNet-R. However, when all samples have the same corruption (snow), exhibit more grouping loss among the networks than when the 19 corruptions are randomly applied on the dataset (merged) (Figure 14c.). An intuition that heterogeneity created by corruption is canceled out by another one having heterogeneity in the opposite direction, leading to region scores closer to the average. Grouping diagrams of all networks available at:
Section D.2.1: ImageNet-C - No post-hoc recalibration, small versions.
Section D.2.2: ImageNet-C No post-hoc recalibration, best versions.
Section D.2.3: ImageNet-C Isotonic recalibration, small versions.
Section D.2.4: ImageNet-C Isotonic recalibration, best versions.
D.3 IMAGENET-1 K VALIDATION SET
The validation set of ImageNet- IK comprises 50 000 samples for 000 classes. Figure 14d. compares estimated grouping loss lower bound and calibration enrors of all networks (small and best versions)
31

---

Published as a conference paper at ICLR 2023
on the validation set of ImageNet-1K, Conversely to ImageNet-R and ImageNet-C, cannot exhibit substantial grouping loss on any of the networks. The grouping diagrams (Figure 24) show however that ConvNeXt Tiny displays more heterogeneity than the other networks this dataset. Grouping diagrams are available in:
Section D.3.1: ImageNet- No post-hoe recalibration, small versions.
Section D.3.2: ImageNet-IK: - No post-hoc recalibration, best versions.
Section D.3.3: ImageNet-1E K Isotonic recalibration, small versions.
Section D.3.4: ImageNet- K Isotonic recalibration, best versions.
E NLP
We use BART Large (Lewis et al., 2019) pre-traine rre-trained on the Multi-Genre Natural Language Inference dataset (Williams al., 2018) and fine-tuned the Yahoo Answers Topics dataset for zero-shot topic classification. The fine-tuned model is available on HuggingFace at ht /huggi i nng face. co co/ coojooddav//b /bart- Large-rrraaaa poneemmnni--. mamli-yahoor answers. Yahoo Answers Topics is composed of question titles and bodies and topic labels. There are 400 000 training samples, 60 000 test samples and topics. The dataset is available at ht : // /uwgggiiing iiggiinfacce co a tatt yahoo_ answers s_toppicss... The model is fine-tuned on 5 out of the 10 topics of the training set, totalizing 000 samples. Given a question title and a hypothesis (e.g. "This text about Science & Mathematics"), the model outputs its confidence in the hypothesis to be true for the given question. The classification being zero-shot, the hypothesis can be about an unseen topic. We evaluate model separately the 5 unseen topies and the 5 seen topics of the test set (i.e. unseen samples). This results binary classification task in which each sample composed of a question title and hypothesis and each label is I or 0 whether the hypothesis correct not. As for the clustering and calibration procedure, we used a balanced decision stump in the same way as described in Section D: "Detailed experimental method"'. We work in the high-level feature space of the network, i.e. the output space of the penultimate layer (embedding space).
topics but
a. Vision
Network
CL
CL
GLLB
GL
Accuracyt (%)
ViT B-16
0.044
0.000
0.019
0.021
30.1
ConvNeXt Tiny
0.017
0.000
0.036
0.039
29.0
Inception
0.200
0.000
0.006
0.009
27.3
Wide ResNet-50
0.082
0.000
0.020
0.019
25,9
ResNeXt-50
0.091
0.000
0.013
0.013
25.1
EfficientNet-BOO
0.028
0.000
0.015
0.018
24.1
DenseNet-122 121
0.059
0.000
0.013
0.010
23.9
GoogLeNet
0.009
0.001
0.011
0.011
23.3
RegNet y_ 400mf
0.084
0.000
0.011
0.010
21.0
ResNet-18
0.057
0.000
0.010
0.009
20.4
MobileNet V2
0.070
0.000
0.007
0.009
18.9
MNASNet 0.5
0.017
0.000
0.005
0.007
15.9
VGG-11
0.061
0.000
0.007
0.007
15.8
ShuffleNet 0.5
0.072
0.000
0.004
0.003
14.6
AlexNet
0.063
0.000
0.003
0.003
12.6
b. NLP
Setting in in-distribution
CL
CL'
GLLB
GL
Accuracy (%)
0.026
0.000
0.000
0.000 0.015
88.2
out-of-distribution
0.091
0.000
0.015
71.1
Table 1: Raw values of the estimators in the vision (Figure and NLP experiments of Section 5.2,
before (CL and GLLB) and after (CL and GL isotonic recalibration.
32

---

Published as a conference paper at ICLR 2023
a. ImageNet-R
b. ImageNet-C (merged)
No recalibration
Isotonic
No recalibration
Isotonic
ConvNeXt Large
VIT 1.116
EfficientNee-BB
32.0
VTT B-16
60.0
VIT B-16-
ConvNeXt Large
ConvNeXt Thny
ConvNeXt Tiav
ResNeXt-10 101
FfficientNet-B B7
RegNet y. 32df
RegNet y.32gf
ResNet-152
NenseNet-161
VIT 1--66-
ResNeett1101
Inception
Wide ResNet-101
DenseNet-161
20.
1esNNet-152
Wide ResNet-101
Eeficient B0
Wide RosNet-50
Inception
Ne5550
Wide ResNet-50
FfficientNet--A
Res5oXt-50
DenseNet-121
DenseNet-12
GoogLeNer NeNe
MobileNet V3L
MobileNet V3L
RegNet 400m
VGG-19 BN
Goog Nel
Reg y. 400mf
VGG-19 BN
Net-18
MNASNet 1.0
MNASNet 1.0
00.
R
4.5
Not
V2
Nob V2
ShutfleNet
1.0
ShuffleNet 1.0
MNASNel 0.5
MNASNel 0.5
VGG-11
GL
VGG-11
GL
102
ShuffleNet 0.5
ShuffleNet
AlexNet
AlexNet
0.00
0d6
0.30
0.15
0:200.00
0.06
0.00
0.05
0.30
0.15
0,200.00
0.05
GL CL
GL CL.
ImageNet-C (snow)
d. ImageNet-1K (validation set)
No recalibration
Isotonic
No recalibration
Isotonic
CouvNeXt Large
MBS
ConvNeXt
L-16-
ConxNeXt Tiny
VYT B-16-
VIT B-16
CouvNeXt Tiny
75
RegNet 32gf
DepseNet-161
26
ViT L-16
EfficientNet-Bf
ResNeXt-101
Effiiien
Wide RcsNet-101
RegNet y_
21.9
Wide ResNet-50 50
FN
Wide ResNet-101
ResNet-152
ResNeXt-101
Net
ResNet-152
ResNeXt-50
Inception
M0.6
DenseNet-161
DenseNet-121
DenseNet-12
5e
VGG-19
Get
RegNet v 400mf
Wide ResNet- Net 50
MobileNet V3L
MobileNet V3L
Efficient
VGG-19 BN
MNASNet 1.0
RegNet Net 400mf
MobileNet. V2
ResNet-1s 18
GoogLeNet
00
ShuffleNet 1.
Re*Net-15 18
MNASNet
Inception
MobileNet V2
ShuffeNNe 1.0
MNASNet 015
VGG-11 Il
VGG-11
CLED
MNASNet 0.5
GLID
ShuffleNet 0.5
CL.
ShuffleNet 0.5
AlexNet.
AlexNet
0.00
0.05
0.10
0.15
0300.00
0.06
0.00
0.05
0.10
0.15
0.200.00
0.05
GL
CL.
GL + CI.
Figure 14: Comparing vision models: debiased estimate the grouping loss lower bound GLLB (Equation 13) and an estimate of the calibration loss CT., both accounting for binning, evaluated on ImageNet-R, ImageNet-C and ImageNet- K, sorted by model accuracy. Partitions R are obtained from decision tree partitioning constrained to create at most samples in bin/so /so regions each bin. Isotonic regression used for post-hoc recalibration of the models (right).
33

---

Published as a conference paper at ICLR 2023
a. ImageNet-R
b. ImageNet-C (merged)
Network
CL
CL
GLlB
GLLB
Accuracyt (%)
Network
CL
CL
GLLB
GLLB
Accuracy! (%)
ConvNeXt Large
0.029
0.000
0.055
0.060
34,8
VIT L-16
0.004
0.000
0.016
0.016
41.4
Efficien[NetBB7
0.055
0.000
0.022
0.021
32.0
VIT B-16
0.001
0.000
0.007
0.009
40.6
ViT B-16
0.044
0.000
0.019
0.021
30.1
ConvNeXt Large
0.002
0.000
0.016
0.016
40.1
ConvNeXt Tiny
0.017
0.000
0.036
0.039
29.0
ConvNeXt Tiny
0.005
0.000
0.008
0.008
31.1
ResNeXt-101
0.114
0,000
0.019
0.018
28.9
EtficientNet-B7
0.025
0.000
0.009
0.007
28.1
RegNet y y_32gf
0.108
0.000
0.020
0.020
28.7
RegNet y y_32gf
0.053
0.000
0.002
0.002
28.0
ResNet-152
0,082
0.000
0.020
0.018
27.8
DenseNet-161
0.045
0.000
0.004
0.004
27.0
ViT I--16
0.096
0.000
0.044
0.043
27.4
ResNeXt-101
0.071
0.000
0.005
0.006
25.9
Inception
0.200
0.000
0.006
0.009
27.3
Wide ResNet-101
0.036
0.000
0.003
0.004
25.1
DenseNct-161
0.093
0.001
0.023
0.021
26.9
ResNet- 152
0.031
0.000
0.002
0.002
24.5
Wide ResNet-
0.079
0.000
0.022
0.020
26.9
EfficientNet-BO
0.003
0.000
0.002
0.002
24.4
Wide Re
0.082
0.000
0.020
0.019
25.9
Inception
0.171
0.000
-0.006
0.001
22.7
ResNeXt-50
0.091
0.000
0.013
0.013
25.1
Wide ResNet-50
0.040
0.000
0.003
0.003
22.1
EfficientNet-BO BO
0.028
0.000
0.015
0.018
24.1
ResNeXt-50
0.058
0.000
0.006
0.006
22.1
DenseNet-133 121
0.059
0.000
0.013
0.010
23,9
DenseNet-122
0.021
0.000
0.002
0.002
21.1
GoogLeNet
0.009
0.001
0.011
0.011
23.3
MobileNet V3L
0.024
0.000
0.001
0.001
20.2
MobileNet V3L
0.104
0.000
0.007
0.006
22.2
RegNet y_400mf
0.034
0.000
0.003
0.002
17.7
VGG-19 BN
0.073
0.000
0.012
0.011
21.2
GoogLeNet
0.004
0.000
0.001
0.001
17.1
RegNet y y_400mf
0.084
0.000
0.011
0.010
21.0
VGG-19 BN
0.019
0.000
0.003
0.003
16.0
ResNet-18
0.057
0.000
0.010
0.009
20.4
MNASNet 1.0
0.032
0.000
0.002
0.002
15.4
MNASNet 1.0
0.095
0.000
0.008
0.008
19.9
ResNet-18
0.027
0.000
0.004
0.004
14.8
MobileNet V2
0.070
0.000
0.007
0.009
18.9
MobileNet V2
0.028
0.000
0.002
0.002
13.9
ShuffleNet 1.0
0.091
0.000
0.007
0.006
18.8
ShuffleNet 1.0
0.073
0.000
0.002
0.003
12.7
MNASNet 0.5
0.017
0.000
0.005
0.007
15.9
MNASNet 0.5
0.009
0.000
0.003
0.003
11.7
VGG-11
0.061
0.000
0.007
0.007
15.8
VGG-11
0.022
0.000
0.005
0.004
10.2
ShuffleNet 0.5
0.072
0.000
0.004
0.003
14.6
ShuffleNet 0.5
0.084
0.000
0.006
0.005
9.3
AlexNet
0.063
0.000
0.003
0.003
12.6
AlexNet
0.029
0.000
0.001
0.001
7.8
ImageNet-CC C (snow)
ImageNet- (validation set)
Network
CL
CL'
GLiB
GLLB
Accuracyt (%)
Network
CL
CL'
GLiB
GL
Accuracy! (%)
ConvNeXt. Large
0.002
0.000
0.030
0.031
43.8
ConyNeXt Large
0.015
0.000
0.002
0.002
84.1
L-16
0.007
0.000
0.013
0.013
40.9
ConyNeXt Tiny
0.028
0.000
0.002
0.002
82.1
ViT B-16
0.004
0.000
0.005
0.005
39.5
VIT B-16
0.004
0.000
-0.000
0.000
81.1
ConvNeXt Tiny
0.011
0.000
0.021
0.021
35.6
RegNet y_32gf
0.009
0.000
-0.001
-0.000
80.9
1enseNet-161
0.069
0.000
0.019
0.019
26.2
ViT L-16
0.003
0.000
0.001
0.001
79.7
EfficientNet-B7
0.034
0.000
0.006
0.005
23.5
ResNeXt-101
0.010
0.000
-0.002
0.000
79.3
EfficientNet-BO
0.015
0.000
0.008
0.009
22.5
Wide ResNet-101
0.004
0.000
-0.001
-0.000
78.8
R
0.102
0.000
0.021
0.020
21.9
Wide ResNet-50
0.004
0.000
-0.001
-0.000
78.5
Wide Rel
0.057
0.000
0.012
0.011
21.4
ResNet-152
0.004
0.000
-0.001
0.000
78.3
RcsNeXt-101
0.133
0.000
0.025
0.025
21.1
EfficientNet-BB
0.006
0.000
0.000
0.000
77.7
ResNet-1152
0.070
0.000
0.020
0.020
20.9
ResNeXt150
0.006
0.000
-0.001
0.000
77.6
Inception
0.194
0.000
0.000
0.004
20.6
DenseNet-16 161
0.004
0.000
-0.001
-0.000
77.1
DenscNet-121
0.042
0.000
0.009
0.007
18.9
DenseNet-122
0.001
0.000
-0.000
-0.000
74.4
ResNeXt-50
0.100
0.000
0.018
0.019
17.5
VGG-19 BN
0.002
0.000
-0.001
-0.000
74.2
GoogLeNet
0.008
0.000
0.005
0.005
17.1
RegNet 400mf
0.001
0.000
-0.001
-0.000
74.0
Wide 5es
0,083
0.000
0.015
0.014
17.0
MobileNet V3L
0.006
0.000
-0,001
-0.000
74.0
MobileNet V3L
0.055
0.000
0.004
0.003
13.7
EfficientNet-B7
0.001
0.000
0.003
0.002
73,9
VGG-19 BN
0.053
0.000
0.011
0.010
12.7
MNASNet 1.0
0.003
0.000
-0.001
-0.000
73.5
RogNet 4_00mf
0.070
0.000
0.009
0.007
12.
MobileNet V2
0.001
0.000
-0.001
0.000
71.9
ResNet-18
0.088
0.000
0.009
0.010
10.5
GoogLeNet
0.006
0.000
-0.000
0.000
69.8
ShuffleNet 1.0
0.116
0.000
0.006
0.005
10.5
ResNet-18
0.001
0.000
0.000
-0.000
69.8
MNASNet 1.0
0.081
0.000
0.007
0.006
10.1
Inception
0.050
0.000
-0.007
0.001
69.5
MobileNet V2
0.100
0.000
0.007
0.006
9.0
ShuffleNet 1.0
0.008
0.000
-0.001
0.000
69.4
MNASNet 0.5
0.026
0.000
0.002
0.003
7.5
VGG-11
0.000
0.000
-0.000
0.000
69.0
VGG-11
0,047
0.000
0.005
0.004
7.1
MNASNet 0.5
0,012
0.000
-0.001
-0,000
67.7
ShuffieNet 0.5
0.094
0.000
0.002
0.002
6.2
ShuffleNet 0.5
0.004
0.000
-0.001
0.000
60.6
AlexNet
0.051
0.000
0.002
0.002
4.9
AlexNet
0.001
0.000
-0.001
-0.001
56.5
Table 2: Raw values of the estimators of Figure 14, before (CI. and GLaB) and after (CT.' and GT isotonic recalibration.
34

---

Published as a conference paper at ICLR 2023
D.1.1 IMAGENET-R- - No POST-HOC RECALIBRATION, SMALL VERSIONS
AlexNet
VGG 11
ResNet 18
DenseNet 121
100
(3)
Sizes
Sizes
Sizes
Sizes
Correct predictions
148
143
119
750
800
600 1064
600 912
1390
1217
Confidence score
1
0
Confidence score
0
Confidence score
0
Confidence score
Inception
GoogleNet
ShuffleNet V2 x0.5
MobileNet V2
100
(%)
Sizes
Sizes
Sizes
Sizes
Correct predictions
68
173
77 800 1209
57
1500
1000
600
2472
1652
1028
Confidence
Confidence
Confidence
Confidence
ResNext 50
WideResNet 50
MNASNet 0.5
EfficientNet BO
(35
Sizes
Sizes 70
Sizes
Sizes
Correct predictions
39
94
183
600
600 1183
600
1197
1937
1141
Confidence
Confidence
Confidence
Confidence
RegNet 4f00mm
VIT B-16
ConvNeXt Tiny
100
(%)
Sizes
Sizes
Sizes
Correct predictions
55
72
33
450
450
600
884
814
979
0 Confidence score
1
0
Confidence score
0
Confidence score
Figure 15: Vision: Fraction of correct predictions versus confidence score of predicted class
(max: S&) on ImageNet-R for small versions of pre-trained networks, without post-hoc recali- bration. In each bin on confidence scores, the level set is partitioned into regions with a decision stump constrained to balanced split, with 50-50 train-test split strategy.
35

---

Published as a conference paper at ICLR 2023
D.1.2 IMAGENET-R- - No POST-HOC RECALIBRATION, SMALL VERSIONS, K-MEANS
AlexNet
VGG 11
ResNet 18
DenseNet 121
100
(3)
Sizes 59
Sizes
Sizes
Sizes 58
Correct predictions
86
117
1200
900
750
800
1985
Confidence score
Confidence score
0
Confidence score
0
Confidence score
Inception
GoogleNet
ShuffleNet V2 x0.5
MobileNet V2
100
Correct predictions (8)
Sizes
Sizes
Sizes
Sizes
26
65 900 1617
59
1500
1200 2077
750
2592
1359
Confidence
Confidence
Confidence
Confidence
ResNext 50
WideResNet 50
MNASNet 0.5
EfficientNet BO
(35
Sizes
Sizes
Sizes 27 1500 2852
Sizes 123
Correct predictions
34
35
900
900
900 1717
1605
1666
o
Confidence
Confidence
Confidence
Confidence
RegNet 4_400mf
VIT B-16
ConvNeXt Tiny
100
(%)
Sizes
Sizes
Sizes 15
Correct predictions
26
86
600
750
750
1166
1486
1333
0
Confidence score
0
Confidence score
Confidence score
Figure 16: Vision:
Fraction of correct predictions versus confidence score of predicted class (max: S&) on ImageNet-R for small versions of pre-trained networks, without post-hoc recali- bration. In each bin on confidence scores, the level set is partitioned into regions with k-means clustering, with 50-50 train-test split strategy (for a fair comparison with decision stump clustering).
36

---

Published as a conference paper at ICLR 2023
D.1.3
IMAGENET-R- - No POST-HOC RECALIBRATION, BEST VERSIONS
VGG 19 BN
ResNet 152
DenseNet 161
ShuffleNet V2 x1.0
100 (%)
Sizes
Sizes
Sizes
Sizes
Currect predictions
600
750
600 1176
600 933
918
1238
Confidence score
Confidence score
Confidence score
Confidence score
MobileNet V3 Large
ResNext 101
WideResNet 101
MNASNet 1.0
100
Currect predictions 3
Sizes 47
Sizes
Sizes
36
110
66
450
958
1769
1208
870
Confidence score
Confidence score
0
Confidence score
Confidence score
EfficientNet B7
RegNet y_ __22gf
ViT L-16
ConvNeXt Large
100
Correct predictions (%)
Sizes
Sizes
Sizes
Sizes
600
900
750
450
742
0 Confidence scute
Confidence score
0
Confidence scute
0
Confidence score
Fraction of correct predictions versus confidence score of predicted class (max; Sş) on ImageNet-R for best versions of pre-trained networks, without post-hoc recalibration. In each bin on confidence scores, the level set is partitioned into 2 regions with a decision stump constrained to one balanced split, with 50-50 train-test split strategy.
Figure 17: Vision:
37

---

Published as a conference paper at ICLR 2023
D.1.4
IMAGENET-R - ISOTONIC RECALIBRATION, SMALL VERSIONS
AlexNet
VGG 11
ResNet 18
DenseNet 121
100
(2)
Sizes 19
Sizes
Sizes
Sizes
Correct predictions
1800 3505
1800 3250
1200 2129
1200 2017
Confidence score
1
0
Confidence score
1
0
Confidence score
1
0
Confidence score
Inception
GoogleNet
ShuffleNet V2 x0.5
MobileNet V2
100
(%)
Sizes
Sizes
Sizes
Sizes
Correct predictions
31
14
27
1200
1200 1842
2400
1500
1894
3793
2522
Confidence score
Confidencc
Confidence score
Confidence score
ResNext 50
WideResNNtt 50
MNASNet 0.5
EfficientNet BO
(%)
Sizes 49
Sizes 14 1200 2235
Sizes 14 1600 2410
Sizes 17
Correct predictions
900
1500 2777
1680
Confidence score
Confidence score
Confidence score
Confidence score
RegNet 4f00mf
VIT B-16
ConvNeXt Tiny
100
(%)
Sizes
Sizes 92
Sizes
Correct predictions
17
333
1200
1200
1000
2233
1928
1577
0
Confidence score
1
0
Confidence score
0
Confidence score
Figure 18: Vision: Fraction of correct predictions versus confidence score of predicted class
(max: S&) on ImageNet-R for small versions of pre-trained networks, with isotonic recalibration. In each bin on confidence scores, the level set is partitioned into 2 regions with a decision stump constrained to one balanced split, with 50-50 train-test split strategy.
38

---

Published as a conference paper at ICLR 2023
D.1.5
IMAGENET-R - ISOTONIC RECALIBRATION, BEST VERSIONS
VGG 19 BN
ResNet 152
DenseNet 161
ShuffleNet V2 x1.0
100
(%5
Sizes
Sizes
Sizes 1200 2115
Sizes 1500 2666
Currect predictions
1200
1000 1541
2115
Confidence score
Confidence score
Confidence score
Confidence score
MobileNet V3 Large
ResNext 101
WideResNet 101
MNASNet 1.0
100
Currect predictions (8)
Sizes
Sizes
Sizes
Sizes 18
411
32
34
1885
2094
2069
26588
Confidence score
Confidence score
0
Confidence score
Confidence score
EfficientNet B7
RegNet y 32gf
ViT L-16
ConvNeXt Large
100
Correct predictions (8)
Sizes
Sizes 1000
Sizes
Sizes
94
205
1200
750
1200
1901
0
Confidence scute
0
Confidence score
0
Confidence scute
0
Confidence score
Figure 19: Vision:
Fraction of correct predictions versus confidence score of predicted class (max; SR) on ImageNet-R for best versions of pre-trained networks, with isotonic recalibration. In each bin on confidence scores, the level is partitioned into 2 regions with a decision stump constrained to one balanced split, with 50-50 train-test split strategy.
39

---

Published as a conference paper at ICLR 2023
D.2.1 IMAGENET-C-ME No POST-HOC RECALIBRATION, SMALL VERSIONS
AlexNet
VGG 11
ResNet 18
DenseNet 121
100
(3)
Sizes 94
Sizes
Sizes
Sizes
Correct predictions
107
215
303
1800
1800
1500 2763
1200
3328
Confidence score
1 0
Confidence score
0
Confidence score
0
Confidence score
Inception
GoogleNet
ShuffleNet V2 x0.5
MobileNet V2
100
(8)
Sizes
Sizes
Sizes
Sizes
Correct predictions
416
121
248 1200 2197
205
1500
1800 3220
1500
2976
2834
Confidence
Confidence
Confidence
Confidence
ResNext 50
WideResNet 50
MNASNet 0.5
EfficientNet BO
(%)
Sizes
Sizes
Sizes
Sizes
Correct predictions
421
346 1959
57
280
2400 3692
1591
2533
Confidence
Confidence
Confidence
Confidence
RegNet vf400mf
VIT B-16
ConvNeXt Tiny
100
(%)
Sizes
Sizes
Sizes
Correct predictions
301
38
16
1200
900
1500
2176
1580
2858
0
Confidence score
0
Confidence score
0
Confidence score
Figure 20: Vision: Fraction of correct predictions versus confidence score of predicted class (max: S&) on ImageNet-C for small versions of pre-trained networks, without post-hoc recali- bration. In each bin on confidence scores, the level set is partitioned into regions with a decision stump constrained to balanced split, with 50-50 train-test split strategy.
40

---

Published as a conference paper at ICLR 2023
D.2.2
IMAGENET-C- No POST-HOC RECALIBRATION BEST VERSIONS
VGG 19 BN
ResNet 152
DenseNet 161
ShuffleNet V2 x1.0
100
(%5
Sizes
Sizes
Sizes
Sizes
Currect predictions
214
386
284
1500
1250
1000
1200
2785
2081
Confidence score
Confidence score
Confidence score
Confidence score
MobileNet V3 Large
ResNext 101
WideResNet 101
MNASNet 1.0
100
Currect predictions ($)
Sizes
Sizes
Sizes
Sizes
283
488
381
237
2286
1864
1847
2455
Confidence score
Confidence score
1
0
Confidence score
Confidence score
EfficientNet B7
RegNet y __g2ff
ViT L-16
ConvNeXt Large
100
Correct predictions (8)
Sizes
Sizes
Sizes
Sizes
443
1000
1250
750
1200
1528
0
Confidence scute
0
Confidence score
0
Confidence scute
0
Confidence score
Figure 21: Vision: Fraction of correct predictions versus confidence score of predicted class
(max; Sş) on ImageNet-C for best versions of pre-trained networks, without post-hoc recalibration. In each bin on confidence scores, the level set is partitioned into 2 regions with a decision stump constrained to one balanced split, with 50-50 train-test split strategy.
41

---

Published as a conference paper at ICLR 2023
D.2.3
IMAGENET-C- -ISTTONICCC ISOTONIC RECALIBRATION, SMALL VERSIONS
AlexNet
VGG 11
ResNet 18
DenseNet 121
100
Sizes
Sizes
Sizes 29 3000 5389
Sizes 71
Correct predictions
6000
4500 7623
2400 3724
9043
Confidence score
1
0
Confidence score
Confidence score
1
0
Confidence score
Inception
GoogleNet
ShuffleNet V2 x0.5
MobileNet V2
100
(%)
Sizes
Sizes
Sizes
Sizes
Correct predictions
66
21
52
1800
2400 4565
4500 7267
3000
3342
5539
Confidence
Confidence
Confidence score
Confidence
ResNext 50
WideResNet 50
MNASNet 0.5
EfficientNet BO
(%)
Sizes
Sizes
Sizes 17
Sizes
Correct predictions
35
61
44
2400
4500 6779
1800
3819
3382
3104
Confidence
Confidence
Confidence score
Confidence
RegNet 4f00mm
VIT B-16
ConvNeXt Tiny
100
Correct predictions (%)
Sizes
Sizes
Sizes
20
20
122
3000
1200
1500
4842
1979
2998
0
Confidence score
1
0
Confidence score
1
0
Confidence score
Figure 22: Vision: Fraction of correct predictions versus confidence score of predicted class
(max: S&) on ImageNet-C for small versions of pre-trained networks, with isotonic recalibration. In each bin on confidence scores, the level set is partitioned into 2 regions with a decision stump constrained to one balanced split, with 50-50 train-test split strategy.
42

---

Published as a conference paper at ICLR 2023
D.2.4
IMAGENET-C- -ISTONNIC ISOTONIC RECALIBRATION, BEST VERSIONS
VGG 19 BN
ResNet 152
DenseNet 161
ShuffleNet V2 x1.0
100
(%5
Sizes
Sizes
Sizes
Sizes 25 4500 6196
Currect predictions
3000 5747
2400
1500 2493
Confidence score
Confidence score
Confidence score
Confidence score
MobileNet V3 Large
ResNext 101
WideResNet 101
MNASNet 1.0
100
(8)
Sizes
Sizes 58
Sizes 61
Sizes 22
Currect predictions
e 73
3895
3524
3310
5241
Confidence score
1
Confidence score
1
0
Confidence score
Confidence score
EfficientNet B7
RegNet y ___2ff
ViT L-16
ConvNeXt Large
100
Correct predictions (8)
Sizes
Sizes
Sizes
Sizes
103
100
1500
2000
1200
1200
0
Confidence scute
0
Confidence score
0
Confidence scute
0
Confidence score
Figure 23: Vision: Fraction of correct predictions versus confidence score of predicted class
(max; S&) on ImageNet-C for best versions of pre-trained networks, with isotonic recalibration. In each bin on confidence scores, the level is partitioned into 2 regions with a decision stump constrained to one balanced split, with 50-50 train-test split strategy.
43

---

Published as a conference paper at ICLR 2023
D.3.1 IMAGENET-IK- - No POST-HOC RECALIBRATION, SMALL VERSIONS
AlexNet
VGG 11
ResNet 18
DenseNet 121
100
(3)
Sizes
Sizes
Sizes 28
Sizes 117 3000
Correct predictions
82
1500
2400
2400 4644
2766
Confidence score
Confidence score
Confidence score
Confidence score
Inception
GoogleNet
ShuffleNet V2 x0.5
MobileNet V2
100
(%)
Sizes
Sizes
Sizes
Sizes
Correct predictions
15
145
36 2400 3940
25
4500
1500 2954
3000
$266
5086
Confidence
Confidence
Confidence
Confidence
ResNext 50
WideResNet 50
MNASNet 0.5
EfficientNet BO
Correct predictions (3)
Sizes
Sizes 15
Sizes
Sizes 20
50
157
-4500
1200 2040
1500
7309
7233
2701
Confidence
Confidence
Confidence
Confidence
RegNet v_400mf
VIT B-16
ConvNeXt Tiny
100
Correct predictions ($)
Sizes
Sizes 54
Sizes 17
78
3000
3000
1200
5488
5047
2234
0 Confidence score
Confidence score
0
Confidence score
Figure 24: Vision: Fraction of correct predictions versus confidence score of predicted class (max: S&) on ImageNet-1K (validation set) for small versions of pre-trained networks, without post-hoe recalibration. each bin on confidence scores, the level partitioned into 2 regions with decision stump constrained to balanced split, with 50-50 train-test split strategy.
44

---

Published as a conference paper at ICLR 2023
D.3.2
IMAGENET-IK - No POST-HOC RECALIBRATION BEST VERSIONS
VGG 19 BN
ResNet 152
DenseNet 161
ShuffleNet V2 x1.0
100
(55
Sizes
Sizes
Sizes 43 4500 6852
Sizes
Currect predictions
3000
4500
3000 5868
5817
Confidence score
Confidence score
Confidence score
Confidence score
MobileNet V3 Large
ResNext 101
WideResNet 101
MNASNet 1.0
100
Currect predictions 3
Sizes
Sizes 19 8167
Sizes 36 7390
Sizes 102
31
4500
6452
5817
Confidence score
Confidence score
Confidence score
Confidence score
EfficientNet B7
RegNet =_32gf
ViT L-16
ConvNeXt Large
100
Correct predictions (3)
Sizes
Sizes
Sizes
Sizes
31
1800
4500
4500
2400
3500
3740
0 Confidence scute
Confidence score
0
Confidence scute
0
Confidence score
Figure 25; Vision: Fraction of correct predictions versus confidence score of predicted class (max; SR) on ImageNet-IK (validation set) for best versions of pre-trained networks, without post-hoc recalibration. each bin on confidence scores, the level partitioned into 2 regions with decision stump constrained to one balanced split, with 50-50 train-test split strategy.
45

---

Published as a conference paper at ICLR 2023
D.3.3
IMAGENET-IK ISOTONIC RECALIBRATION, SMALL VERSIONS
AlexNet
VGG 11
ResNet 18
DenseNet 121
100
(3)
Sizes
Sizes
Sizes 30
Sizes
Correct predictions
38
100
1200
2400 4059
2400 3997
2400 4569
2252
Confidence score
0
Confidence score
Confidence score
0
Confidence score
Inception
GoogleNet
ShuffleNet V2 x0.5
MobileNet V2
100
Correct predictions (8)
Sizes
Sizes
Sizes
Sizes
28
92 1200
20
2400
1800
2400
4075
2392
4244
Confidence
Confidence
Confidence
Confidence
ResNext 50
WideResNet 50
MNASNet 0.5
EfficientNet BO
Correct predictions (3)
Sizes
Sizes
Sizes 23 1800 3387
Sizes
16
21
15
3000
5139
4665
5155
Confidence
Confidence
Confidence
Confidence
RegNet 4f00mf
VIT B-16
ConvNeXt Tiny
100
Correct predictions (
Sizes
Sizes
Sizes
30
22
112 3000
2400
4500
4728
60.38
5314
0 Confidence score
0
Confidence score
Confidence score
Figure 26: Vision:
Fraction of correct predictions versus confidence score of predicted class (maxı S&) ImageNet-1K (validation set) for small versions of pre-trained networks, with isotonic recalibration. In each bin on confidence scores, the level set partitioned into regions with decision stump constrained one balanced split, with 50-50 train-test split strategy.
46

---

Published as a conference paper at ICLR 2023
D.3.4
IMAGENET-IK ISOTONIC RECALIBRATION, BEST VERSIONS
VGG 19 BN
ResNet 152
DenseNet 161
ShuffleNet V2 x1.0
100
(3)
Sizes
Sizes
Sizes
Sizes
Currect predictions
2400
3000 $709
3000 5294
2400 4358
Confidence score
Confidence score
Confidence score
Confidence score
MobileNet V3 Large
ResNext 101
WideResNet 101
MNASNet 1.0
100
Correct predictions B
Sizes
Sizes 28
Sizes 67
Sizes
17
45
+ 4165
$288
5867
4697
Confidence score
Confidence score
Confidence score
Confidence score
EfficientNet B7
RegNet -_22gf
ViT L-16
ConvNeXt Large
100
Correct predictions (8)
Sizes
Sizes 111 3000
Sizes
Sizes
1800
4500
4500
3473
0 Confidence scute
Confidence score
0
Confidence scute
Confidence score
Figure 27: Vision:
Fraction of correct predictions versus confidence score of predicted class (max: Sạ) on ImageNet-IK (validation set) for best versions of pre-trained networks, with iso- tonic recalibration. In each bin on confidence scores, the level set partitioned into 2 regions with decision stump constrained to one balanced split, with 50-50 train-test split strategy.
47

---

Published as a conference paper at ICLR 2023
48


Reviews:
[{"review_id": "kskbO28k9o", "reviewer": "Reviewer_sCnp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors offer a method of estimating the grouping loss, which fills in some shortcomings of common measures of network calibration. They then use their new metric to measure the grouping loss of a variety of different architectures and settings. ", "summary_of_the_review": "Overall, I think this work highlights and analyzes a known problem with ECE and recalibration methods in general. This is an important topic, and I think the knowledge here can be applied in a number of future works. ", "strength_and_weaknesses": "## Strengths\n\n- The method is rigorously defined\n- The results are interesting and highlight an exact problem and provide a possible solution to the shortcomings of ECE\n\n\n## Weaknesses\n\n- I am confused about the actual application of Proposition 4.3. $\\text{GL}_{\\text{explained}}$ will have an entry for every bin and every class, right? Then how are the figures in Figure 6 calculated on a multiclass problem like ImageNet-R? If only the top level prediction is considered, as is stated, then doesn't this add more uncertainty into the whole grouping since each class has a differing amount of irreducible uncertainty will all other classes? I understand the argument about small numbers of test instances in that particular dataset, but would it be possible to compare top-level grouping loss with per class grouping loss on a smaller dataset which has more samples, like CIFAR-10? I am curious how GL will behave when inter-class relationships are considered.\n\n- To clarify the application of proposition 4.3 to a multiclass problem, would it be correct to characterize the sum of $\\hat{\\mu}_j$ in a multiclass problem as \"count the number of times the actual class of region $j$ is the predicted class from the classifier?\"\n\n- Figure 7 does not quantify what happens to the grouping loss before and after isotonic recalibration. I would be very interested to see the information in figure 7 in the form of a table to see the actual differences in the numbers.\n\n- Isotonic calibration performance is reported, but the most popular form of recalibration (temperature scaling) is not reported. Does temperature scaling perform any different than isotonic calibration?\n\n- Do the authors have any intuition as to why the NLP tasks have a drastically lower grouping loss as compared to the vision tasks? \n\n## Minor\n\n- The end of the counterexample paragraph on page 2 refers to the grouping loss without defining what it is, which leads to confusion. \n\n- NLP Section: \"However, we found no evidence in the in-distribution one.\" This sentence is not clear. No evidence of what? What is the in-distribution 'one?'\n\n- NLP Section: \"This suggests again that our of distribution settings lead to stronger heterogeneity, and thus grouping loss.\" I find this sentence confusing. Why 'again?' was this brought up before in the text? Why would stronger heterogeneity cause larger grouping loss? It might seem intuitive that this is the case, but what is the reason that heterogeneity would cause higher grouping loss? \n\n- Conclusion: \"captures well the grouping loss.\" --> captures the grouping loss well.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "kcakrcZ6KH", "reviewer": "Reviewer_rpCw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors of the paper derive an estimator of a term that lower bounds the grouping loss term of a proper scoring rule. They show that this estimator is tighter than baselines and use it to analyze existing models. The derivations seem novel and correct; however, it is unclear to me why having this estimator is interesting or useful in practice.", "summary_of_the_review": "I am not convinced the problem studied by this paper is sufficiently significant.", "strength_and_weaknesses": "STRENGTHS:\n* The paper provides a novel estimator for the lower bound of the group loss. This term was previously hard to measure.\n* The methods used are novel, technically sound, and non-trivial. The level of technical depth required to derive the estimator is significant.\n* The experiments evaluate a lot of realistic state-of-the-art models.\n\nWEAKNESSES:\n* I am not convinced from reading this paper that measuring the grouping loss is something I would like to do in practice and why this is interesting.\n* It is not clear to me how the experiments support the main claims of the paper.\n* Some of the claims seem off off to me (see below).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VYMPO8lwCQ", "reviewer": "Reviewer_LCym", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on a decomposition of strictly proper scoring rules that reveals three terms: calibration loss, group loss, and an irreducible term. Intuitively, the group loss term (GL) deviates from the aggregate measures used in calibration; e.g. if a model is systematically over-confident on a subgroup and under-confident on another, the overall calibration can be small but GL is large. The authors argue that evaluating GL is important and they present an estimator that provides a lower bound to GL. They conduct experiments on vision and language tasks, which suggest that models continue to have a large group loss even after post-hoc calibration.\n", "summary_of_the_review": "The paper is well-written and offers an interesting insight into calibration. All arguments are supported by experiments and proofs. I find the experimental results quite interesting. My only concerns are: (1) the practical implications are unclear yet since one can simply use proper scoring rules, and (2) more details should be provided about the experiments as I mention above. \n\n\n======  Post rebuttal ======\n\nThank you for confirming my understanding and for the clarification.\n\nI see now your argument of why quantifying the group loss can be useful. Since a proper scoring rule includes the irreducible term, it might be worthwhile instead to look into the sum of calibration and group loss. As you mention in the revised draft,  a classifier with a Brier score of 0.15 could have optimal probabilities (irreducible loss close to 0.15) or poor ones (irreducible loss close to 0).\n\nI think this is a useful message and the proposed method using decision trees is simple and easy to apply. I have updated my scores.", "strength_and_weaknesses": "The paper brings insight into calibration and supports its arguments theoretically. The authors conduct experiments on several architecture families in both language and vision in real-world datasets and also present experiments on synthetic data. \n\nBut, the primary weakness in my opinion lies in the motivation behind this work and I would appreciate it if the authors clarify the following point in their rebuttal. First, using a proper scoring rule, such as the log-loss and the Brier score, would automatically capture both calibration and group loss at the same time and they can be estimated easily. Is there a reason for practitioners to report a calibration error and a group loss, when they can report a proper score. My main takeaway from the paper is that one should not rely on calibration alone, such as using ECE, but rather use proper scoring rules because they capture group loss as well, which seems to be a simple and accurate message to take away. But, I don’t see the value of trying to calculate a lower bound on the group-loss term, especially given that the results on the simulated data in Figure 4 are not encouraging. \n\nSecond, I haven’t yet figured out how the regions or partitions are identified using decision trees in Section 4. What is the input to the decision tree? For example, in vision, are those raw images? I think it would be really useful to provide examples of images for each group. \n\n\nSome minor comments:\nIn Lemma 4.1, “negative entropy” may not be the right term since the score rule can be something else.\nIn Figure 4, why is $GL_{LB}$ not a lower bound when the number of bins is small?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster