Soft Calibration Objectives for Neural Networks
Authors: Archit Karandikar
Nicholas Cain
Dustin Tran
Balaji Lakshminarayanan
Jonathon Shlens
Michael Curtis Mozer
Rebecca Roelofs
Venue: NeurIPS 2021 Poster (2021)
Keywords: calibration
uncertainty
reliable machine learning

Abstract: Optimal decision making requires that classifiers produce uncertainty estimates consistent with their empirical accuracy. However, deep neural networks are often under- or over-confident in their predictions. Consequently, methods have been developed to improve the calibration of their predictive uncertainty both during training and post-hoc. In this work, we propose differentiable losses to improve calibration based on a soft (continuous) version of the binning operation underlying popular calibration-error estimators. When incorporated into training, these soft calibration losses achieve state-of-the-art single-model ECE across multiple datasets with less than 1% decrease in accuracy. For instance, we observe an 82% reduction in ECE (70% relative to the post-hoc rescaled ECE) in exchange for a 0.7% relative decrease in accuracy relative to the cross entropy baseline on CIFAR-100.
When incorporated post-training, the soft-binning-based calibration error objective improves upon temperature scaling, a popular recalibration method.  Overall, experiments across losses and datasets demonstrate that using calibration-sensitive procedures yield better uncertainty estimates under dataset shift than the standard practice of using a cross entropy loss and post-hoc recalibration methods.

Soft Calibration Objectives for Neural Networks
Archit Karandikar'
Nicholas Cain"
Google Research archk@google. com
Google Research
nicholascainggoogle.cco
Dustin Tran Google Research
Balaji Lakshminarayanan
Jonathon Shlens Google Research shl shlens@google.c com
Google Research
trandustin@googlee.cm
balajiln@google.
Michael C. Mozer
Becca Roelofs
Google Research
Google Research
mcmozer@google.com
rolfs@google.o com
Abstract
Optimal decision making requires that classifiers produce uncertainty estimates consistent with their empirical accuracy. However, deep neural networks often under- or over-confident in their predictions. Consequently, methods have been developed to improve the calibration of their predictive uncertainty, both during training and post-hoc. In this work, we propose differentiable losses to improve calibration based soft (continuous) version of the binning operation underlying popular calibration-erro estimators. When incorporated into training, these soft calibration losses achieve state-of-the-ar single-model ECE across multiple datasets with less than 1% decrease in accuracy. For instance, we observe 82% reduction in ECE (70% relative the post-hoc rescaled ECE) in exchange for 0.7% relative decrease in accuracy relative to the cross- cross-entropy baseline on CIFAR- -00. When incorporated post-training, the setttbinning--uie based calibration еттоr objective improves upon temperature scaling, a popular recalibration method. Overall, experiments across losses and datasets demonstrate that using calibration- sensitive procedures yield better uncertainty estimates under dataset shift than the standard practice of using cross-entropy loss and post-hoe recalibration methods." ?
1 Introduction
Despite the success of deep neural networks across a variety of domains, they are still susceptible to miscalibrated predictions. Both over- and under-confidence contribute miscalibration, and empirically, deep neural networks empirically exhibit significant miscalibration [Guo et al., 2017J. Calibration error (CE) quantifies model's miscalibration by measuring how much its confidence, i.e. the predicted probability of correctness, diverges from its accuracy, i.e. the empirical probability of correctness. Models with low CE are critical domains where satisfactory outcomes depend on well-m moll-eddeeed uncertainty, such as autonomous vehicle navigation [Bojarski et al., 2016] and medical diagnostics [Jiang al., 2012, Caruana et al., 2015, Kocbek et al., 2020]. Calibration has also been shown to be useful for improving model fairness [Pleiss et al., 2017] and detecting out-of-distribution data [ulleshov and Ermon, 2017, Devries and Taylor, 2018, Shao et al., 2020]. More generally, low
"co-first author
"Code available on GitHub: http://////ttttoooooooot/t///t//
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

CE is desirable in any setting in which thresholds are applied to the predicted confidence of a neural network in order to make decision.
Methods for quantifying CE typically involve binning model predictions based on their confidence. CE is then computed empirically a weighted average of the absolute difference in average prediction confidence and average accuracy across different bins [Nacini et al., 2015]. Oftentimes these bins are selected heuristically such as equal-width (uniformly binning the score interval) and equal-mass (with equal numbers of samples per-bin) [Nixon et al., 2019].
However, these commonly used measures of CE are not trainable with gradient-based based methods because the binning operation is discrete and has zero derivatives. As a result, neural network parameters not directly trained to minimize CE, either during training during post-hoc recalibration. In this paper, we introduce new objectives based on a differentiable binning scheme that can be used to efficiently and directly optimize for calibration.
Contributions. We propose estimating CE with soft (i.e., overlapping, continuous) bins rather than the conventional hard (i.e., nonoverlapping, all-or-none) bins. With this formulation, the CE estimate differentiable, allowing us to use as (1) a secondary (i.e., auxiliary) loss to incentivize model calibration during training, and (2) a primary loss for optimizing post-hoc recalibration methods such as temperature scaling. In the same spirit, we soften the AvUC loss [Krishnan and Tickoo, 2020], allowing to it as an effective secondary loss during training for non non- Bayesian neural networks where the Av loss originally proposed for Stochastic Variational Inference (SVI) typically does work. Even when training with the cross-entropy loss results in training set memorization (perfect train accuracy and calibration), Soft Calibration Objectives are still useful as secondary training losses for reducing test ECE using procedure call interleaved training.
In an extensive empirical evaluation, we compare Soft Calibration Objectives as secondary losses to existing calibration-incentvvzzae losses. In the process, we find that soft-calibration losses outperform prior work on in-distribution test sets. Under distribution shift, we find that calibration-sensitte training objectives as a whole (not always the ones we propose) result in better uncertainty estimates compared to the standard cross-entropy loss coupled with temperature scaling.
Our contributions can be summarized as follows:
We propose simple Soft Calibration Objectives S-AvUC, SB-ECE as secondary losses which optimize for CE throughout training. We show that across datasets and choice of primary losses, the S-AvUC secondary loss results the largest improvement ECE as per the Cohen's is d effect-size metric (Figure 1). We also show that such composite losses obtain sate-of-theear single-model ECE in exchange for less than 1% reduction in accuracy (Figure 5) for CIFAR-10, CIFAR-100, and Imagenet.
We improve upon temperature scaling - a popular post-hoc recalibration method- by directly optimizing the temperature parameter for soft calibration error instead of the typical log- likelihood. Our extension (TS-SB-ECE) consistently beats original temperature scaling (TS) across different datasets, loss functions and calibration error measures, and we find that the performance is better under dataset shift (Figure 2).
Overall, our work demonstrates fundamental advantage of objectives which better incen- tivize calibration over the standard practice of training with cross-entropy loss and then applying post-hoe methods such as temperature scaling. Uncertainty estimates of neural networks trained using these methods generalize better in and out-of-distribution
2
Related Work
Many techniques have been proposed train neural networks for calibration. These can be organized into three categories. One category augments or replaces the primary training loss with a term to explicitly incentivize calibration. These include the AvUC loss [Krishnan and Tickoo, 2020], MMCE loss [Kumar et al., 2018) 8] and Focal loss [Mukhoti et al., 2020, Lin et al., 2018]. The Mean Squared Error loss also compares favourably [Hui and Belkin, 2021] to cross-entropy loss. We show that our methods outperform all these iebrrraioooinnetiviiiien
training objectives, applied across multiple primary losses. Label smoothing [Müller et al., 2020] has been shown to improve calibration and can be interpreted modified primary loss.
2

---

Figure 1: We compare the effect size of various secondary losses on ECE (equal-mass binning, lz norm) across datasets and primary losses, both with and without post-hoc tempera- ture scaling. The S-AvUC secondary loss we propose shows the strongest positive effect. followed by MMCE and SB-ECE. Note that d-value of 0.8 (resp., 2.0) is considered a large (resp., huge) positive effect and d- values obtained here are much larger. Secondary losses which incentivize calibration show strong positive effect for reducing ECE even after tempera- ture scaling.
Effect size of secondary loss for reducing ECE (higher is better)
(huge ECE 10 redn.)
MMCE
AvUC
S-AvUc
SB-ECE
Cohen's
(no
ECE
redn.)
ECE
ECE w/ TS-SB-ECE
CIFAR-100 CIFAR-100-C
CIFAR-10 CIFAR-10-C
no rescaling
30
no rescaling ng
30
TS: 1S-SB-ECE
TS. SB-EB-ECE
ECE1%)
ECE(%)
20
test
test
5
Distribution Shift Intensity
Distribution Shift Intensity
Figure 2: Post-hoc temperature scaling with the soft calibration error objective (TS-SB-ECE) reduces ECE more than standard post-hoc temperature scaling (TS), particularly under distribution shift. This result holds across datasets (left and right panels), distribution shift intensities (along abscissa) and training objectives (not shown). The training objective shown here for both datasets is the most popular one: NLL. The ECE value (equal-mass binning, & norm) shown is the mean ECE across the corruption types that constitute CIFAR-10-C and CIFAR-100- 100-C. Error bars are 1 standard error of mean (SEM), corrected for intrinsic variability due to type of corruption [Masson and Loftus, 2003].
A second category of methods are post-hoc calibration methods, which rescale model predictions after training. These methods optimize additional parameters on held-out validation set [Platt, 1999, Zadrozny and Elkan, 2002, Kull et al., 2019, Zadrozny and Elkan, 2001, Nacini and Cooper, 2016, Allikivi and Kull, 2019, Kull et al., 2017, Nacini et al., 2015, Wenger et al., 2020, Gupta et al., 2020]. The most popular technique is temperature scaling [Guo et al., 2017], which maximizes single temperature parameter on held-out NLL. We examine temperature scaling and propose an improvisation that directly optimizes temperature for soft calibration objective instead of NLL. Temperature scaling has shown to be ineffective under distribution shift in certain scenarios [Ovadia al., 2019]. We show that uncertainty estimates of methods which train for calibration generalize better than temperature scaling under distribution shift.
A third category of methods examines model changes such as ensembling multiple predictions (Lakshminarayanan et al., 2017, Wen et al., 2020] or priors [Dusenberry et al., 2020]. Similar to previous work [Kumar et al., 2018, Lin et al., 2018], focus on the choice of loss functions for improving calibration a single neural network-whert compare against ensemble models Bayesian neural networks. These techniques are complementary to ours and can be combined with our techniques to further improve performance.
during training or post-hoc-and do not
Recent works have investigated issues [Nixon et al., 2019, Kumar et al., 2019, Roelofs et al., 2020, Gupta et al., 2020] with the originally proposed ECE [Guo et al., 2017] and suggested new ones. Debiased CE [Kumar et al., 2019] and mean-sweep CE [Roelofs et al., 2020] have been shown to have lesser bias and more consistency across the number of bins parameter than ECE whereas KS-error [Gupta et al., 2020] avoids binning altogether. We report these metries in the Appendix.
3

---

3 Background
3.1 The Task and the Model
Consider a classification task over K classes with a dataset of N samples D = ((r,, Dz) ye)) drawn from the joint probability distribution ( 3) over the input space X and label space y = {1. 2....... K. The task is modelled using a deep neural network with parameters 0 whose top layer is interpreted as softmax layer. The top layer consists of K neurons which produce logits go (x) = (go(v)=)) The predictive probabilities for given input are:
fo(z) = (fo(y u)) softmax(ge (x)).
The parameters 0 of the neural network are trained to minimize (( (()o(a)))))) v) where (a, v) is sampled from ((X'3') Here L is a trainable loss function which incentivizes the predictive distribution (x) fit the label y. The model's prediction on datapoint (æz, y) is denoted by qo (x) arg max(ff( We denote by (x) = max (fe (x)) the confidence of this prediction and by boolean quantity ag ag(2.y) y) (o() the accuracy of this prediction.
3.2 Expected Calibration Error (ECE)
Given distribution () datapoints, there two standard notions of Ideal Calibration Error that we refer to as Ideal Binned Expected Calibration Error (of order p). IECE (D, 0) IECE and Ideal Expected Minnne Calibration Error (of order p), IECE (D. 0) = IECE This nomenclature is consistent with that introduced by Roelofs et al. [2020]. Both these measure, in slightly different ways, the ndh root of the ndh moment of the absolute difference between model confidence and the empirical accuracy given that confidence. They are defined follows:
IECEBin, (D,0) = [Eeo
(ato) (afo)
[[ao (i = =ce(2o)] co (2o)))) IP))
1/p
(1)
[1 (1)))(((1 = co (2o)) - co (2))) 1/p
JECE p (D,0) =
(2)
Note that the critical dependence )) is implicit in both definitions since the datapoint (æo. 30) from the outer expectation and the datapoint (æ), v1) from the inner expectation are both sampled from )
We cannot compute IECEBin. and I in practice since the number of datapoints are finite. Instead we consider a dataset D = ((z., ()) drawn from Dxx y) and partition the confidence interval [0, 1] into bins B = (B.)) each of which also corresponds to a confidence interval. We will use o; as shorthand for (ι))) and a, as shorthand for ug (aē,, Dt). We denote by b, (B, D.0) b; the bin to which o; belongs. We define the size of bin j as S,, D 0) Sj. the average confidence of bin j as CB, D, 0) = Cj and the average accuracy of bin j as A, D,0)
Ay. These expressed as follows:
S, D, 0) = [(ib, = 5}l
(3)
= 3, E
(4)
A,, = 5, E.pb.=aa.
(5)
We are now in a position define the Expected Binned Calibration Error of order p which we denote by ECEsi,,p and Expected Label-Binned Calibration Error of order p we denote by ECE,, These serve as empirical approximations the corresponding intractable ideal notions from equations and 2. They are defined as follows:
ECE (B (.) (DM N |A 115-C,17 C,11) 1/8
(6)
ECE (B, D),0) ( (15 1 csal²) 1/P
It follows from Jensen's 's inequality that ECE,,,p(B,.. (B, > ECE pp (B, [Roclofs et al., 2020].

---

4
Soft Calibration Objectives
In this section, we define quantities that can be used better incentivize calibration during training.
4.1 Soft-Binned ECE (SB-ECE)
The quantities in the definitions of ECE and p can be written in terms of a formal definition of the bin membership function. Let denote the bin-membership function for given binning
B = (B:) (1..2.... by us [0. 1] - Mas, where MM = (o E j0, 1 [Zg = 1) is the set of possible bin membership vectors over M bins. The membership function for bin j is denoted by
uB, [0, 1] + [0, 1] and is defined by (c) = ( The size, average accuracy, and average confidence bin j from equations 3, 4, and 5 can now written in terms of ug as follows:
S, E (B)()))(
(8)
C, , 2 (c) ct)
(9)
A, = -, (uB, a.).
(10)
The quantities ECEsin.p and E can further be written in terms of the quantities Sj. Cj and A, using equation and a modification of equation 7 (see equation 12 below). We know that the differentials @ECFsin. /0 and @ECEn.,p00 are non-trainable. The formulation above makes it clear that this is precisely because 0ug,;/Cc is zero within bin boundaries and undefined at bin boundaries. Morcover, this observation implies that if we could come up with a trainable soft bin- membership function uig then we could use it in place of the usual hard bin- hin-membership function us to obtain trainable version of ECE and ECE
With this motivation, we define the soft bin- membership function that has a well-defined non-zero gradient in (0. 1). It is parameterized by the number of bins M and temperature parameter T. We consider equal-width binning for simplicity and so we represent it as " rather than ug. We desire unimodality over confidence: if ε denotes the center of bin j then we want Suarr, to be positive for < Ej negative for > Ej. Similarly, we also desire unimodality over bins: < & Ej or > & > Ey. then we want that "M.T,(() > (C..T.((C....... Finally, we also want the aforementioned temperature parameter 7 control how close the binning to hard binning (i.e. how steeply membership drops off). This would give the nice property of hard-binning being a limiting condition of soft-binning. With this motivation, we define the soft bin-membership function as
(c) = softmax(gM, where (M.T,((C) = -c--,,
/T
Vi E (1,2.... M).
Figure 3 visualizes soft han--mmberrhip. We can now formulate trainable calibration error measures. We define the Expected Soft- BinneB Calibration Error (M, D, 0) and Expected Soft- Label-Binned Calibration Error SB-ECE..,p T, D):
SB SB-ECEsin, (M,T, D. M |4 c,,p) 1/P
(11)
SB-ECEB,p (M.T. D. ( أل
( MM (MM... |A, (A)- 1/P
(12)
The quantities Sy. Cj and A, in these expressions are obtained by using the soft bin membership function u uMr in place of the hard bin membership function ug in equations 8, and 10 respectively.
We use these trainable calibration eтror measures as: (1) part of the training loss function and (2) the objective that minimized to tune the temperature scaling parameter Tıa during post-hoc calibration.
4.2 Soft AvUC (S-AvUC)
The accuracy versus uncertainty calibration (AvUC) loss [Krishnan and Tickoo, 2020] categorizes each prediction that a model with parameters 0 makes for a labelled datapoint d; (x; 3/: from dataset π according to two axes: (1) accurate [A] versus inaccurate [T], based on the value of the boolcan quantity ag (x; y1) = (2) certain [C] versus uncertain [U], based whether the entropy
5

---

Visualization the soft bin membership function (M=15, T=0.0001 0.01)
10
Figure 3: Visualization of the soft bin membership function which shows that the temperature parameter determines the sharpness of the binning. Soft binning limits to hard binning as temperature tends zero.
soft membershn
Lower Temp (0.0001) Higher Temp (0.01)
022
confidence
(()e(a;))) = h4 of the predictive distribution is above or below threshold k. Denote the number of elements from D that fall in each of these categories by îLC,, ñAU âic and âuu respectively, The AvUC loss incentivizes the model to certain when accurate and uncertain when inaccurate:
[++ RAA + +MIC) A
(vUC D, 0) log
7AA + =np
(13)
where the discrete quantities are relaxed to be differentiable:
AU (. (G tanh h.)
T4Cc ". ((1 c.)(1 tanh h2)
(14)
TAC
(G-(1 tanh h;)) 7r [ (i) ((1 c.) tanh h;).
Krishnan and Tickoo [2020] have showed good calibration results using the AvUC loss SVI settings. However, in experiments found that the addition of the AvUC loss term resulted in poorly calibrated models non-SVI neural network settings (see Appendix A). One reason for this seems to be that minimizing the AvUC loss results in the model being incentivized to be even more confident in its inaccurate and certain predictions (via minimizing Ric) and even less confident in its accurate and uncertain predictions (via minimizing πuA). This conjecture is validated by experimental observations: when we stopped the gradients flowing through the c terms in equation 14, we were able obtain calibrated models (see Appendix F). Fixing this incentivization issue in a more principled manner than stopping gradients is desirable. Another desirable improvisation is replacing the hard categorization into the certain/uncertain bins with soft partitioning scheme. We meet both these objectives by defining a notion of soft uncertainty.
We want a limiting case of the soft uncertainty function to be the hard uncertainty function based on an entropy threshold k. This implics that we will continue to have a parameter λ despite getting rid of the hard threshold. As before, desire temperature parameter 7 that will determine how close the function is to the hard uncertainty function. The soft uncertainty function fx,r : [0, 1] + [o, 1] takes input the j0, 1] normalizzed entropy hī h,/ log( K) of the predicted posterior where K the number of classes. We also need Ot,, to be positive in [0. 1] and would like f,.T to and lima--11 (x,T(h") = 1. Finally, would be good to have the [0, 1] identity mapping as a special case of lu.r-family for some value of and T. We now define the soft-uncertainty function in the following way so that it meets all stated desiderata:
satisfy lima---0 (,,T (h+)
h h1(1==) logistic (=
T (h*) )
(1 (1- (1-h*)m)
Finally, we define Soft AvUC in terms of soft uncertainty by modifying equations 13 and 14. In our experiments, we use Soft AvUC as part of the loss function to obtain calibrated models.
TLAI + níc
DAAUCC,,TTÚD,C log
(15)
n/c n/y
where T'AU = E tanh/4)
n/c = Ejja,= - tn,r tanh hx))
N'sC = Ejja,=1(( (1))) tanh h;))
ziu = -ot tanh ha)h
(16)

---

Soft uncertainty plot
Soft uncertainty plat
(Fixed T=0.1, varying kappa)
(Fixed kappa 0.5, varying
1.0
T=0.01
T=0.03
T=0.10
T=0.30
0.8
0.8
T=1.00
T-3.00
kappa=0.1
uncertainty
kappam0.2
soft uncertainty
T=10.00
T=30.00
0.6
kappa=0.3
0.6
kappa-0.4
kappa-0.5
kapp=-0.6
0.4
kappa-0.7
0.4
kappamo.8
kappa=0.9
0.2
0.2
0.0
0.0
1.
normalized entropy
normalized entropy
Figure 4: Visualization of the soft uncertainty function tx,r(h") which shows that the parameter captures the soft-threshold whereas the parameter T captures the sharpness of the thresholding.
5 Results
We compare our Soft Calibration Objectives to recently proposed calibration calibration-incentivvii incentivizing training objectives MMCE, focal loss, and AvUC on the CIFAR-10, CIFAR-100, and ImageNet datasets. We evaluate the full pross-product of primary and secondary losses: the options for primary loss are cross-entropy (NLL),, focal or mean squared error (MSE) loss; and the options for secondary loss are MMCE, AvUC, SB-ECE or S-AvUC. Results for the MSE primary loss and the AvUC secondary loss are in Appendix A. Our experiments build on the Uncertainty Baselines and Robustness Metrics libraries [Nado et al., 2021, Djolonga al., 2020].
5.1 Soft Calibration Objectives for End-to- -nd Training
Our results demonstrate that training losses which include Soft Calibration Objectives obtain state-of- the-art single-model ECE on the test set in exchange for less than 1% reduction in accuracy for all three datasets that we experiment with. In fact, our methods (especially S-AvUC) without post-hoc temperature scaling are better than or as good as other methods with or without post-hoc temperature scaling on all three datasets.
The primary losses we work with for CIFAR-10/100 are the cross-entropy (NLL) loss, the focal loss and mean squared error (MSE) loss. Focal loss [Mukhoti al., 2020] and MSE loss [Hui and Belkin, 2021] have recently shown to outperform the NLL loss in certain settings. The cross-entropy loss outperforms the other
losses on Imagenet, and is thus our sole focus for this dataset.
The primary loss even by itself (especially NLL) can overfit to the train ECE [Mukhoti et al., 2020], without help from the soft calibration losses. Even in such settings, show that soft calibration losses yield reduction test ECE using a technique we call 'interleaved training' (Appendix B).
We use the Wide-Resnet-22- 22--1 10 architecture [Zagoruyko and Komodakis, 2017] trained for 200 epochs on CIFAR-100 and CIFAR-IO. For Imagenet, we use the Resnet-50 [He et al., 2015] architecture training for 90 epochs. All our experiments use the SGD with momentum optimizer with momentum fixed to 0.9 and learning rate fixed to O. The loss function we use in our experiments is PL + 3 SL + X L2 where PL and SL denote the primary and secondary losses respectively and L2 denotes the weight normalization term with E norm. We tune the 3 and À parameters along with the parameters x and relevant to the secondary losses SB-ECEn,, T, D, 0) and S-AvUC(k, T, D. 0). We tune these hyperparameters sequentially. We fix the learning rate schedule and the number of bins M to keep the search space manageable. Appendix has more details of our hyperparameter search.
7

---

CIFAR-100 (Primary Loss: Focal)
CIFAR-100 (Primary NLL)
(Primary Loss: MSE)
12
12
10
78.3%
77.3%
78.2%
78.0%
78.7%
77.2%
76.2%
78.6%
11.25
77.4% None
10.0%
77,0%
77.4%
77.9%
Nome
Addc
S-AdIC
Nome
MMCCE
Aw/c
5-tiic
58-ECE
MMCX
Avic
S-AnK
Se:ECt
Expected Callbration Error
CIFAR-10 (Primary Loss: Focal)
CIFAR-10 |Primary Loss: NLL)
CIFAR-10 (Primary Loss: MSE)
25.3%
29.0%
25.3%
93.4%
94.8%
22.0%
None
MAKE
Aviic
S-AdiC
58 ECE
None
MACE
Avdcc
S-Aviic
S8-ECE
Hone
MB
Avic
SAvvic
58 s0.
ImageNet (Primary Loss: NLLJ
None
MMCC
Avtic
S-Aviic
SBECE
Figure 5: Soft Calibration Objectives (S-AvUC, SB-ECE), when used as secondary losses with primary loss, achieve lower ECE (equal-mass binning, & norm) than the corresponding primary loss for CIFAR-10, 10, CIFAR- 100, and ImageNet. These statistically significant wins less than 1 % accuracy (reported at bottom the bar). Values reported are mean over 10 runs, and the error bars indicate + standard error of the mean (SEM). For each dataset (each row) the best (across primary losses) ECE obtained using the S-AvUC and SB-ECE secondary losses is lower than the best ECE obtained using existing techniques.
at the cost of
C-100: Accuracy vs Confidence (Soft Calibration Objectives)
C-100: Accuracy vs Confidence (Other Methods)
Accuracy vs Confidence plots for var- ious methods CIFAR- 100. NLL is significantly overconfident and NLL MMCE is somewhat over- confident. While Focal loss is underconfident, augment- ing with Soft Calibration Objectives fixes this issue, resulting curves closest to the ideal.
Figure 6:
1.0
1.0
Focal + S-AvUC
NLL
Focal SB-ECE
NLL MMCE
Ideal
0.8
Focal
Accuracy
Accuracy
Ideal
0.6
0.4
0.4
0.4 Confidence
0.2
Confidence
In Table 1, we report the with the best ECE values that also came within 1 1% of the primary loss run with the highest accuracy. Figure 6 is the
plot corresponding to Table la. In Figure 5, we visualize the ECE on the test set for all combinations of primary loss, secondary loss and dataset. The best ECE for each dataset is attained using Soft Calibration Objectives as secondary losses. More such figures are in Appendix and the complete table can be found in Appendix A.
5.2 Soft Calibration Objectives for Post-Hoc Calibration
Standard temperature scaling (TS) uses cross-entropy objective to optimize the temperature. How- ever, using our differentiable soft binning calibration objective (SB-ECE), we can optimize the temperature using loss function designed to directly minimize calibration error. Figure (and Figure 9 in Appendix C), we compare temperature scaling with soft binning calibration objective (TS-SB-ECE) to standard temperature scaling with cross-entropy objective (TS) on out- of-distribution shifts of increasing magnitude both CIFAR- 10 and CIFAR- 100. The distribution

---

Table 1:
We report average accuracy (with standard error across 10 trials). ECE, and ECE obtained after post-hoc temperature scaling (TS) for models trained with different objectives on the CIFAR-10. CIFAR-I00, and ImageNet datasets. ECE is computed with the & norm and equal-mass binning. We find Soft Calibration Objectives (SB-ECE, S-AvUC) result in better or equivalent ECEs compared to previous methods with or without TS. We also find that TS does not always improve ECE. The best ECE value is highlighted for each dataset. The best ECE with TS value is highlighted if it improves over the best value from the ECE column.
(a) LIFAR-100
CIFAR-10
Loss Fn.
Accuracy
ECE
ECE with TS
Loss
Accuracy
ECE
ECE with TS
SET.
78.740.122
9.10
5.36+0.091
NLL
95.3=00.400
5.59=0.119
1.95#0.127
MMCE
77.210.072
4.7710.121
4.06101113
95.0200031
20.00
1091 0.098
Focal
78.3 10,006
4.66 10.00
6,47 140
Focal
95.01 0L085
5.45 +0,09
1.6910.990 150
Focal SB-ECE
78.6100.62
2.30:10.105
5.16100.108 108
Focal SB-ECE
95.11 0.056
1.19:1 0.088
2.0811 0.143
Focal AvU.
78. 10.004
1.57:0.122
4.13
Focal S-AVUC
94.4.1. 0.145
1.58 0,146
1.34.1 0.172
(c) ImageNet
Loss Fn.
Accuracy
ECE
ECE with TS
NET.
75.8 0,036
4 0.043
L17F 0.45
NLL + MMCE
75.24 0.048
3.12= 0.025
2. 0.029
NLL SB-ECE
0.028
0.039
NLL + S-AAUC"
75.6 1 0.053
2.26 0.055
2.02 1 0.041
CIFAR-100 (Distribution Shift)
CIFAR-10 (Distribution Shift)
20
Errrr Rater%)
Enror Rate(k)
ECR,56
ECE(%
10
-Rocal becile 201
12
Shift Intensity
Shift Intensity
Shift Intensity
Shift intensity
Figure 7: For the 1IFAR-101100 datasets, methods which train for calibration outperform the popular methods- NLL and NIL TS under distribution shift. Focal primary loss and MMCE secondary loss result in the lowest ECEs under shift. We note that these methods start off with worse ECEs than SB-ECE and S-AvUC methods the test but end up with better ECE under increasing levels of skew. The OOD datasets that we have used here are CIFAR-10/100-C with skew levels 1-5.
shifts come from either the CIFAR-IO-C or CIFAR- datasets. Whereas Figure 2 focuses on cross-entropy loss, Figure also has the plots for other primary losses. Table 2 contains compar- isons between the two methods based on test ECE for combinations of dataset, primary loss and secondary loss. We find that TS-SB-EGE ECE outperforms TS under shift in most cases, and the performance increase similar across shifts of varying magnitude. Note that temperature scaling (TS) does not always improve ECE, especially when the training loss is different from NLL. In such cases TS-SB-EE EB-ECE still outperforms TS but may or may not result in ECE improvement.
5.3 Training for Calibration Under Distribution Shift
In previous sections we have shown that training for calibration outperforms the popular cross-entropy loss coupled with post-hoc TS on the in in-distribution test set. We find that methods which train for calibration (not always our proposed methods) also outperform the cross-entropy loss with TS under dataset shift. Prior work has shown that temperature scaling can perform poorly under distribution shift [Ovadia et al., 2019], and our experiments reproduce this issue. Moreover, we show that training for calibration makes progress towards fixing this problem. However, different methods perform best under distribution shift on different datasets. Whereas S-AVUC does well on ImageNet OOD (see figure 8), Focal loss does better than our methods CIFAR- 10-C and CIFAR- (see figure 7). We cannot prescribe one method in particular under distribution shift given these results but we have shown crucial benefit of using methods to train for calibration whole.

---

ImageNet
ImageNet-v2
ImageNet-C
ImageNet-R
7.75
10255
7.53
10.50
23.75
7.25
9.75
23.50
7.00
161
22.75
22.50
MEL-50-550
24.0
24.5
25.0
36.0
36.5
$3.0
37.5
58.5 59.0 59.2 59.4 59.5 59.8
77.0 17.2 174 17.6 728 76.0 Error Rate(%)
Error Rate(%)
Error Rate/%)
Error Rate(1))
Figure comparison methods for ImageNet (primary loss: cross-entropy) shows that S-AvUC secondary loss yields lowest ECEs under dataset shift. Error bars are +1 SEM over 10 runs.
Conclusions
We proposed Soft Calibration Objectives motivated by the goal of directly training models for calibration. These objectives SB-ECE and S-AvUC are softened versions the usual ECE measure and the recently-proposed AvUC loss, respectively. They are easy-to-implement augmentations the popular cross-entropy loss. We performed thorough comparison of existing methods of training be used
for calibration. Our experiments show that methods based on soft-calibration objectives to obtain the best ECE among such methods in exchange for less than 1% drop in accuracy. We note that model being better calibrated overall does not necessarily mean that it is better calibrated for every group and hence the fairness of our methods as well as related methods must be studied. However, our methods of training-foo for-calibration can be adapted to encourage fairness by applying the methods separately to each protected group.
Even when does not wish to incorporate secondary losses train for calibration, showed that post-hoc temperature scaling works better when tuned using the SB-ECE objective instead of the standard cross-entropy loss. Practitioners can easily replace the cross-entropy loss with our SB-ECE loss when performing post-hoc temperature scaling.
Finally, demonstrated that the uncertainty estimates of methods which train for calibration generalize better under dataset shift as compared to post-hoc calibration, which fundamental motivation for transitioning to training for calibration.
Acknowledgements
The authors thank Brennan McConnell and Mohammad Khajah who conducted initial explorations of soft binning calibration loss. The authors also thank Zack Nado, D. Sculley and Jeremiah Liu for help with implementation and suggestions for the writeup.
10

---

References
Mari-Lis Allikivi and Meelis Kull. Non-parametric Bayesian isotonic calibration: Fighting over- confidence in binary classification. In ECML/PKDD, 2019.
Mariusz Bojarski, D. Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal,
L.. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, X. Zhang, Jake Zhao, and Karol Zieha. End to end learning for self-driving cars. ArXiv, abs/1 604.07316, 2016.
R. Caruana, Yin Lou, Gehrke, Paul Koch, M. Sturm, and Noémie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. Proceedings of the 21th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, 2015.
Terrance Devries and Graham W. Taylor. Learning confidence for out-of-distribution detection in neural networks. ArXiv, abs/ 000100022444555 2018.
Josip Djolonga, Minderer Matthias, Zack Nado, Jeremy Nixon, Rob Romijnders, Dustin Tran, and Mario Lucic. Robustness Metries, 2020. URL https: //gp::///ttt thub. com/ caan/googlerreeee research/ robustness _mtrrics.e
Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 factors. In International conference on machine learning, pages 2782-2792. PMLR, 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. ArXiv, abs/1706.04599, 2017.
Kartik Gupta, Amir M. Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, C. Sminchisescu, and R. Hartley. Calibration of neural networks using splines. ArXiv, abs/2006.12800 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss vs cross- entropy in classification tasks, 2021.
Xiaoqian Jiang, M. Osl, Kim, and L. Ohno-Machado. Calibrating predictive model estimates to support personalized medicine. Journal of the American Medical Informatics Association: JAMIA, 19:263 274, 2012.
Simon Kocbek, Primoz Kocbek, Leona Cilar, and Gregor Stiglic. Local interpretability cal- ibrated prediction models: A case of type 2 diabetes mellitus screening test. arXiv preprint arXiv:2006. 13815, 2020.
R. Krishnan and O. Tickoo. Improving model calibration with accuracy versus uncertainty optimiza- tion. ArXiv, abs/2012.079233, 2020.
Volodymyr Kuleshov and S. Ermon. Estimating uncertainty online against adversary. In AAAI, 2017.
Meelis Kull, Telmo de Menezes e Silva Filho, and Peter A. Flach. Beta calibration: a well-founded and easily implemented improvement logistic calibration for binary classifiers. In AISTATS, 2017.
Meelis Kull, Miquel Perelló-Nieto, Markus Kängsepp, Telmo de Menezes e Silva Filho, Hao Song, and Peter A. Flach. Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration. In NeurIPS, 2019.
A. Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In /CML, 2018.
Ananya Kumar, Percy Liang, and Tengyu Ma. Verified uncertainty calibration. In NeurlPS, 2019.
Balaji Lakshminarayanan, Pritzel, and C. Blundell, Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurIPS, 2017.

---

Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, 2018.
M. E. J. Masson and G. R. Loftus. Using confidence intervals for graphically based data interpretation. Canadian Journal of Experimental Psychology, 57:203-220, 2003. URL http://tvee //web. uvic. ca/ psyc/masson/MLan pdf.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, S. Golodetz, P. Torr, and Dokania. Calibrating deep neural networks using focal loss. ArXiv, abs/2002.09437, 2020.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help?, 2020.
Zachary Nado, Neil Band, Mark Collier, Josip Djolonga, Michael Dusenberry, Sebastian Farquhar, Angelos Filos, Marton Havasi, Rodolphe Jenatton, Ghassen Jerfel, Jeremiah Liu, Zelda Mariet, Jeremy Nixon, Shreyas Padhy, Jie Ren, Tim Rudner, Yeming Wen, Florian Wenzel, Kevin Murphy, D. Sculley, Balaji Lakshminarayanan, Jasper Snoek, Yarin Gal, and Dustin Tran. Uncertainty Baselines: Benchmarks for uncertainty & robustness in deep learning. 2021. URL https: //github. www. htt/:/oollllunnn hoaagg////ceetttttbeeeeeeee
M. Nacini and G. Cooper. Binary classifier calibration using ensemble of near isotonic regression models. 2016 IEEE 16th International Conference on Data Mining (ICDM), pages 360-369, 2016.
M. Naeini, G. Cooper, and M. Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. Proceedings of the AAAI Conference Artificial Intelligence. AAAI Conference on Artificial Intelligence, 201 002901-22900 2015.
Jeremy Nixon, Michael W Dusenberry. Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration deep learning. In CVPR Workshops, 2019.
Yaniv Ovadia, E. Fertig, Ren, Zachary Nado, D. Sculley, S. Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snock. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. NeurlPS, 2019.
J. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. 1999.
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q. Weinberger. On fairness and calibration, 2017.
R. Roelofs, N. Cain, Jonathon Shlens, and M. Mozer. Mitigating bias in calibration error estimation. ArXiv, abs/2012.08668,, 2020.
Zhihui Shao, Jianyi Yang, and Shaolei Ren. Calibrating deep neural network classifiers on out-of- distribution datasets. ArXiv, abs/2006.08914, 2020.
Yeming Wen, Dustin Tran, and Jimmy Ba. BatchEnsemble: alternative approach efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.06715, 2020.
Jonathan Wenger, H. Kjellström, and Rudolph Triebel. Non-parametric calibration for classification. In AISTATS, 2020.
B. Zadrozny and C. Elkan. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In ICML, 2001.
B. Zadrozny and C. Elkan. Transforming classifier scores into accurate multiclass probability estimates. Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, 2002.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks, 2017.
12


Reviews:
[{"review_id": "TAk8sLKuWJm", "reviewer": "Reviewer_CL78", "summary": "This paper addresses neural networks miscalibration by introducing a soft version of two different calibration measures, namely ECE and AvUC. \n\nTo make ECE differentiable, the authors propose to soften the bin membership in ECE calculation and define differentiable bin size, bin confidence, and bin accuracy based on the soft membership function. The membership function is characterized by a categorical distribution (implemented by softmax) over the temperature-scaled Euclidean distances between the confidence score of a sample and the centers of the bins. The centers of the bins are pre-determined as the center of equal-width binning with some pre-known number of bins. The authors use the differentiable ECE as an additional term added to traditional loss functions for training classifiers. They also use this loss alone for post-hoc calibration. In addition to ECE, they soften the newly introduced AvUC loss with similar tricks.\n\nExperiments on benchmark datasets are conducted to demonstrate the effectiveness of the proposed method regarding ECE performance metrics.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the formulation that leads to soft bin membership is exciting and novel.\nWhile I appreciate the rigor that the authors aim at providing to the problem, I have several concerns/questions that need to be addressed:\n\n- It is not mentioned in the paper how the quantities in equations 8-10 are evaluated for each sample during training. Is it required to compute the bin size, accuracy, and confidence after each training iteration on all training data?\n\n- Post-hoc calibration using SB-ECE: I believe there is no need to optimize the soft-binned ECE to find a good temperature value. One can perform a search (multi-resolution) to find the optimal temperature using the hard-binned ECE. It is easy to evaluate the ECE for an input value of t, and it won't take too much time (compared to training the original model) to search for a good temperature value. It also keeps the objective exact without introducing the inaccuracies added by soft-binned ECE.\n\n- In figure 1, For CIFAR datasets, SB-ECE and S-AvUC, when added to NLL, seem not to make much improvement to calibration while making accuracy worse. Adding focal loss seems to improve them. However, this seems like a mystery in the paper. It is the addition of focal loss that makes them work. Can the authors elaborate more on that and explain why this is happening? \n\n- When training for NLL, we usually have the accuracy on the training set is 100%, and the confidence of the training samples are high, meaning that training on the NLL will result in calibrated confidence scores on training data. I am wondering how the SC-ECE and S-AvUC avoid that? Mainly, is S-AvUC a proper loss such that one can theoretically guarantee upon training one can get calibrated confidences on the training data itself? What happens if one uses these loss functions to continue training on a pre-trained model?\n\n- The numbers in Table 1 show different accuracies/ECEs compared to a similar model in [1] (Table F.1 and Table 2). \nWhy are there so many differences between these measures and the numbers in Table 1 of the paper? Some details might have been missed in training (e.g., training for 350 epochs rather than 200 epochs).\n\n- Only top-label ECE is computed. Also, only the TS method is compared. The authors have cited many strong post-hoc calibration methods in the related work but have not compared them with their method. \nFurthermore, there are many cases (especially in the applications mentioned in the introduction, such as medical diagnosis)  the calibration of other classes is necessary (That is why measures like classwise calibration or marginal calibration errors have been introduced[2,3]). These are not evaluated in the paper. Also, it would be nice if the extension of soft-calibration losses is discussed for these cases. ECE itself has many flaws as one has to select the number of bins beforehand. There also biases due to small number of bins, etc., [3]. The method introduced in the paper suffers from the same issues.\n\n- Figure S1 in the supplementary shows that the uncalibrated model is more robust under distribution shift than the proposed post-hoc method in most of the cases. These results are hidden in the main paper and have not been discussed. I hope the authors can explain the reason.\n\nMinor:\n- In abstract line 10. 82% reduction refers to which experiment/table in the main paper?\n- In eq. 6 and 11, \"i\" in the sum should be replaced with \"j.\"\n- In eq. 14, $S_{\\rm AU}$, etc., are not defined.\n\n\nReferences:\n\n[1] Mukhoti et al., \"Calibrating Deep Neural Networks using Focal Loss,\" NeurIPS 2020 (arXiv:2002.09437).\n\n[2] Kull et al., \"Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichletcalibration,\" NeurIPS 2019.\n\n[3] Kumar et al., \"Verified Uncertainty Calibration,\" NeurIPS 2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "YES", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "HdxEgv_ntY_", "reviewer": "Reviewer_rMMu", "summary": "The paper presents differentiable loss functions to improve the calibration of deep nets class probability estimates. To achieve this, the authors propose a soft approximation of binning operation, which is at the core of popular measures for model calibration error. Empirical evidence shows that the new approach achieves improved calibration performance compared to existing methods while maintaining accuracy to the extent possible.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n\nThe proposal of a loss that improves calibration is not new. For instance, R. Krishnan and O. Tickoo took this approach in their work titled “Improving model calibration with accuracy versus uncertainty optimization.” With that said, the novelty of new resides in the explicit minimization of metrics for calibration errors (based on binning), which seems to improve performance. This approach is closer to that of “Calibration of neural networks using splines” by Gupta et al. The authors also soften the AvUC loss offered by Krishnan and Tickoo, allowing the training of non-Bayesian neural networks.\n\nQuality\n\nI believe the writing of the paper can be improved. There is extensive use of acronyms and notations, which make the paper hard to read and follow. I find the proposed idea straightforward; I was hoping to see a theoretical analysis for the overall approximation error obtained by the soft operations. Also, suppose we are given a perfectly calibrated model---would the SB-ECE loss will be equal to zero?\n\nMain concern\n\nIn the experiments, the best performing combination of “accuracy loss” + “calibration loss” is not consistent across the benchmark data sets. As a result, it is not clear what should be the default approach. I also do not understand why the calibration loss reduces predictive performance; it seems to be a reasonable regularization. Would the ideal model satisfy this property? In other words, does the optimal classifier minimize the proposed losses (with infinite data)?\n\nSignificance\n\nThe problem of calibrating the probability estimates obtained by machine learning algorithms is important. The paper presents a new way to encourage calibrated predictions during training. The experiments demonstrate the effectiveness of this approach; however, it is not clear which loss works best.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are partially described. See \"main concern\" above. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "8c3ay89vfK6", "reviewer": "Reviewer_2pg6", "summary": "The paper proposes a soft-binning scheme to compute calibration metric so that the models can be trained using SGD to minimize the calibration error together with the classification error. Two metrics have been tried (ECE and AvUC) and the results show improved performance on image classification datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Post-rebuttal update\nThe authors have clarified both of my concerns about the paper. As mentioned in the discussion below, I would strongly encourage the authors to include the discussion on train ECE overfitting and the role of interleaved training. I'm increasing the score.\n\n## Strengths\n- The idea of soft-binning to make the calibration metric trainable is interesting and may have some value.\n- Overall the paper is well written.\n\n## Weaknesses\n- The main issue of miscalibration is due to overfitting in the training set (wrt to calibration) [Guo et al]. So it is not clear to me that minimize calibration error on the training set would solve the miscalibration issue on the test set. Note that it has been observed that the modern networks have perfect calibration on the training set (if a proper loss such as CE is minimized) but miscalibrated on an unseen test set [Mukhoti et al]. This is a major concern and the marginal improvements in performance may be due to some artifacts of the training scheme. Please clarify this.\n\n- Since the method is optimized for ECE it would make sense to measure calibration error using a different metric such as KS-error [Gupta et al]. This is important to show that the method is not overfitted to ECE in some way. Also, KS-error does not have any issues related to binning. \n\n- In Table 1 latest methods need to be compared such as [Kull et al, 2019, Gupta et al]. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequate", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors' response has addressed all the reviewers' concerns, and all the reviewers' final score was "weak accept", which is borderline. It seems that main issue was that they were not completely sure how the different design choices are related to the final performance, which would make this method somewhat less useful to practitioners. For example, Reviewer CL78 summarized:

It seems incorporating the soft-calibration objectives helps to improve the calibration during training. I am not getting a higher score because most of the paper is devoted to SB-ECE, while the most successful secondary loss is S-AvUC. However, there is not much theoretical support (except the previous work under the SVI setting) for different design choices made by the authors (I am not much familiar with SVI). Furthermore, there are some combinations of primary losses that have a great impact on the final results. Although the soft-calibration objectives can improve the calibration in most cases, it might not be easy for a practitioner to summarize which combinations of primary and secondary loss should be selected in advance.
Decision: Accept (Poster)