Soft Calibration Objectives for Neural Networks
Year: 2022

Abstract:
Optimal decision making requires that classifiers produce uncertainty estimates consistent with their empirical accuracy. However, deep neural networks are often underor over-confident in their predictions. Consequently, methods have been developed to improve the calibration of their predictive uncertainty, both during training and post-hoc. In this work, we propose differentiable losses to improve calibration based on a soft (continuous) version of the binning operation underlying popular calibration-error estimators. When incorporated into training, these soft calibration losses achieve state-of-the-art single-model ECE across multiple datasets with less than 1% decrease in accuracy. For instance, we observe an 82% reduction in ECE (70% relative to the post-hoc rescaled ECE) in exchange for a 0.7% relative decrease in accuracy relative to the cross-entropy baseline on CIFAR-100. When incorporated post-training, the soft-binning-based calibration error objective improves upon temperature scaling, a popular recalibration method. Overall, experiments across losses and datasets demonstrate that using calibrationsensitive procedures yield better uncertainty estimates under dataset shift than the standard practice of using a cross-entropy loss and post-hoc recalibration methods.2

heading: 1 Introduction
text: Despite the success of deep neural networks across a variety of domains, they are still susceptible to miscalibrated predictions. Both over- and under-confidence contribute to miscalibration, and empirically, deep neural networks empirically exhibit significant miscalibration [Guo et al., 2017]. Calibration error (CE) quantifies a model’s miscalibration by measuring how much its confidence, i.e. the predicted probability of correctness, diverges from its accuracy, i.e. the empirical probability of correctness. Models with low CE are critical in domains where satisfactory outcomes depend on well-modeled uncertainty, such as autonomous vehicle navigation [Bojarski et al., 2016] and medical diagnostics [Jiang et al., 2012, Caruana et al., 2015, Kocbek et al., 2020]. Calibration has also been shown to be useful for improving model fairness [Pleiss et al., 2017] and detecting out-of-distribution data [Kuleshov and Ermon, 2017, Devries and Taylor, 2018, Shao et al., 2020]. More generally, low
∗co-first author 2Code available on GitHub: https://github.com/google/uncertainty-baselines/tree/main/experimental/caltrain
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
CE is desirable in any setting in which thresholds are applied to the predicted confidence of a neural network in order to make a decision.
Methods for quantifying CE typically involve binning model predictions based on their confidence. CE is then computed empirically as a weighted average of the absolute difference in average prediction confidence and average accuracy across different bins [Naeini et al., 2015]. Oftentimes these bins are selected heuristically such as equal-width (uniformly binning the score interval) and equal-mass (with equal numbers of samples per-bin) [Nixon et al., 2019].
However, these commonly used measures of CE are not trainable with gradient-based methods because the binning operation is discrete and has zero derivatives. As a result, neural network parameters are not directly trained to minimize CE, either during training or during post-hoc recalibration. In this paper, we introduce new objectives based on a differentiable binning scheme that can be used to efficiently and directly optimize for calibration.
Contributions. We propose estimating CE with soft (i.e., overlapping, continuous) bins rather than the conventional hard (i.e., nonoverlapping, all-or-none) bins. With this formulation, the CE estimate is differentiable, allowing us to use it as (1) a secondary (i.e., auxiliary) loss to incentivize model calibration during training, and (2) a primary loss for optimizing post-hoc recalibration methods such as temperature scaling. In the same spirit, we soften the AvUC loss [Krishnan and Tickoo, 2020], allowing us to use it as an effective secondary loss during training for non-Bayesian neural networks where the AvUC loss originally proposed for Stochastic Variational Inference (SVI) typically does not work. Even when training with the cross-entropy loss results in training set memorization (perfect train accuracy and calibration), Soft Calibration Objectives are still useful as secondary training losses for reducing test ECE using a procedure we call interleaved training.
In an extensive empirical evaluation, we compare Soft Calibration Objectives as secondary losses to existing calibration-incentivizing losses. In the process, we find that soft-calibration losses outperform prior work on in-distribution test sets. Under distribution shift, we find that calibration-sensitive training objectives as a whole (not always the ones we propose) result in better uncertainty estimates compared to the standard cross-entropy loss coupled with temperature scaling.
Our contributions can be summarized as follows:
• We propose simple Soft Calibration Objectives S-AvUC, SB-ECE as secondary losses which optimize for CE throughout training. We show that across datasets and choice of primary losses, the S-AvUC secondary loss results in the largest improvement in ECE as per the Cohen’s d effect-size metric (Figure 1). We also show that such composite losses obtain state-of-the-art single-model ECE in exchange for less than 1% reduction in accuracy (Figure 5) for CIFAR-10, CIFAR-100, and Imagenet.
• We improve upon temperature scaling - a popular post-hoc recalibration method - by directly optimizing the temperature parameter for soft calibration error instead of the typical loglikelihood. Our extension (TS-SB-ECE) consistently beats original temperature scaling (TS) across different datasets, loss functions and calibration error measures, and we find that the performance is better under dataset shift (Figure 2).
• Overall, our work demonstrates a fundamental advantage of objectives which better incentivize calibration over the standard practice of training with cross-entropy loss and then applying post-hoc methods such as temperature scaling. Uncertainty estimates of neural networks trained using these methods generalize better in and out-of-distribution.

heading: 2 Related Work
text: Many techniques have been proposed to train neural networks for calibration. These can be organized into three categories. One category augments or replaces the primary training loss with a term to explicitly incentivize calibration. These include the AvUC loss [Krishnan and Tickoo, 2020], MMCE loss [Kumar et al., 2018] and Focal loss [Mukhoti et al., 2020, Lin et al., 2018]. The Mean Squared Error loss also compares favourably [Hui and Belkin, 2021] to cross-entropy loss. We show that our methods outperform all these calibration-incentivizing training objectives, applied across multiple primary losses. Label smoothing [Müller et al., 2020] has been shown to improve calibration and can be interpreted as a modified primary loss.
ECE ECE w/ TS-SB-ECE
(no ECE 0
redn.)
2
4
6
8
(huge ECE 10 redn.)
Co he
n' s d
Effect size of secondary loss for reducing ECE (higher is better)
MMCE AvUC S-AvUC SB-ECE
Figure 1: We compare the effect size of various secondary losses on ECE (equal-mass binning, `2 norm) across datasets and primary losses, both with and without post-hoc temperature scaling. The S-AvUC secondary loss we propose shows the strongest positive effect, followed by MMCE and SB-ECE. Note that a d-value of 0.8 (resp., 2.0) is considered a large (resp., huge) positive effect and dvalues obtained here are much larger. Secondary losses which incentivize calibration show strong positive effect for reducing ECE even after temperature scaling.
test 1 2 3 4 5 Distribution-Shift Intensity
10
20
30
40
EC E(
% )
CIFAR-100 / CIFAR-100-C no rescaling TS TS-SB-ECE
test 1 2 3 4 5 Distribution-Shift Intensity
0
10
20
30
EC E(
% )
CIFAR-10 / CIFAR-10-C no rescaling TS TS-SB-ECE
Figure 2: Post-hoc temperature scaling with the soft calibration error objective (TS-SB-ECE) reduces ECE more than standard post-hoc temperature scaling (TS), particularly under distribution shift. This result holds across datasets (left and right panels), distribution shift intensities (along abscissa) and training objectives (not shown). The training objective shown here for both datasets is the most popular one: NLL. The ECE value (equal-mass binning, `2 norm) shown is the mean ECE across the corruption types that constitute CIFAR-10-C and CIFAR-100-C. Error bars are ±1 standard error of mean (SEM), corrected for intrinsic variability due to type of corruption [Masson and Loftus, 2003].
A second category of methods are post-hoc calibration methods, which rescale model predictions after training. These methods optimize additional parameters on a held-out validation set [Platt, 1999, Zadrozny and Elkan, 2002, Kull et al., 2019, Zadrozny and Elkan, 2001, Naeini and Cooper, 2016, Allikivi and Kull, 2019, Kull et al., 2017, Naeini et al., 2015, Wenger et al., 2020, Gupta et al., 2020]. The most popular technique is temperature scaling [Guo et al., 2017], which maximizes a single temperature parameter on held-out NLL. We examine temperature scaling and propose an improvisation that directly optimizes temperature for a soft calibration objective instead of NLL. Temperature scaling has shown to be ineffective under distribution shift in certain scenarios [Ovadia et al., 2019]. We show that uncertainty estimates of methods which train for calibration generalize better than temperature scaling under distribution shift.
A third category of methods examines model changes such as ensembling multiple predictions [Lakshminarayanan et al., 2017, Wen et al., 2020] or priors [Dusenberry et al., 2020]. Similar to previous work [Kumar et al., 2018, Lin et al., 2018], we focus on the choice of loss functions for improving calibration of a single neural network—whether during training or post-hoc—and do not compare against ensemble models or Bayesian neural networks. These techniques are complementary to ours and can be combined with our techniques to further improve performance.
Recent works have investigated issues [Nixon et al., 2019, Kumar et al., 2019, Roelofs et al., 2020, Gupta et al., 2020] with the originally proposed ECE [Guo et al., 2017] and suggested new ones. Debiased CE [Kumar et al., 2019] and mean-sweep CE [Roelofs et al., 2020] have been shown to have lesser bias and more consistency across the number of bins parameter than ECE whereas KS-error [Gupta et al., 2020] avoids binning altogether. We report these metrics in the Appendix.

heading: 3 Background

heading: 3.1 The Task and the Model
text: Consider a classification task over K classes with a dataset of N samples D = 〈(xi, yi)〉Ni=1 drawn from the joint probability distribution D(X ,Y) over the input space X and label space Y = {1, 2, . . .K}. The task is modelled using a deep neural network with parameters θ whose top layer is interpreted as a softmax layer. The top layer consists of K neurons which produce logits gθ(x) = 〈gθ(y|x)〉y∈Y . The predictive probabilities for a given input are:
fθ(x) = 〈fθ(y|x)〉y∈Y = softmax(gθ(x)).
The parameters θ of the neural network are trained to minimize E(x,y)L(fθ(x), y) where (x, y) is sampled from D(X ,Y). Here L is a trainable loss function which incentivizes the predictive distribution fθ(x) to fit the label y. The model’s prediction on datapoint (x, y) is denoted by qθ(x) = arg max(fθ(x)). We denote by cθ(x) = max(fθ(x)) the confidence of this prediction and by boolean quantity aθ(x, y) = 1qθ(x)=y the accuracy of this prediction.

heading: 3.2 Expected Calibration Error (ECE)
text: Given a distribution D̂(X ,Y) on datapoints, there are two standard notions of Ideal Calibration Error that we refer to as Ideal Binned Expected Calibration Error (of order p), IECEbin,p(D̂,θ) = IECEbin,p, and Ideal Expected Label-Binned Calibration Error (of order p), IECElb,p(D̂,θ) = IECElb,p. This nomenclature is consistent with that introduced by Roelofs et al. [2020]. Both these measure, in slightly different ways, the pth root of the pth moment of the absolute difference between model confidence and the empirical accuracy given that confidence. They are defined as follows:
IECEbin,p(D̂,θ) = ( Ecθ(x̂0) [ |E[aθ(x̂1, ŷ1)|cθ(x̂1) = cθ(x̂0)]− cθ(x̂0)|p ])1/p (1)
IECElb,p(D̂,θ) = ( E(x̂0,ŷ0) [ |E[aθ(x̂1, ŷ1)|cθ(x̂1) = cθ(x̂0)]− cθ(x̂0)|p ])1/p . (2)
Note that the critical dependence on D̂(X ,Y) is implicit in both definitions since the datapoint (x̂0, ŷ0) from the outer expectation and the datapoint (x̂1, ŷ1) from the inner expectation are both sampled from D̂(X ,Y). We cannot compute IECEbin,p and IECElb,p in practice since the number of datapoints are finite. Instead we consider a dataset D̂ = 〈(x̂i, ŷi)〉N̂i=1 drawn from D̂(X ,Y) and partition the confidence interval [0, 1] into bins B = 〈Bi〉i∈{1,2,...M}, each of which also corresponds to a confidence interval. We will use ci as a shorthand for cθ(x̂i) and ai as a shorthand for aθ(x̂i, ŷi). We denote by bi(B, D̂,θ) = bi the bin to which ci belongs. We define the size of bin j as Sj(B, D̂,θ) = Sj , the average confidence of bin j as Cj(B, D̂,θ) = Cj and the average accuracy of bin j asAj(B, D̂,θ) = Aj . These are expressed as follows:
Sj(B, D̂,θ) = |{i|bi = j}| (3) Cj(B, D̂,θ) = 1Sj Σi|bi=jci (4)
Aj(B, D̂,θ) = 1Sj Σi|bi=jai. (5)
We are now in a position to define the Expected Binned Calibration Error of order p which we denote by ECEbin,p and Expected Label-Binned Calibration Error of order p we denote by ECElb,p. These serve as empirical approximations to the corresponding intractable ideal notions from equations 1 and 2. They are defined as follows:
ECEbin,p(B, D̂,θ) = ( ΣMi=1 Sj N̂ · |Aj − Cj |p )1/p (6)
ECElb,p(B, D̂,θ) = (
1 N̂ ΣN̂i=1|Abi − ci|p )1/p . (7)
It follows from Jensen’s inequality that ECElb,p(B, D,θ) ≥ ECEbin,p(B, D,θ) [Roelofs et al., 2020].

heading: 4 Soft Calibration Objectives
text: In this section, we define quantities that can be used to better incentivize calibration during training.

heading: 4.1 Soft-Binned ECE (SB-ECE)
text: The quantities in the definitions of ECEbin,p and ECElb,p can be written in terms of a formal definition of the bin membership function. Let us denote the bin-membership function for a given binning B = 〈Bi〉i∈{1,2,...M} by uB : [0, 1] → UM , where UM = {v ∈ [0, 1]M |Σjvj = 1} is the set of possible bin membership vectors over M bins. The membership function for bin j is denoted by uB,j : [0, 1]→ [0, 1] and is defined by uB,j(c) = uB(c)j . The size, average accuracy, and average confidence of bin j from equations 3, 4, and 5 can now be written in terms of uB as follows:
Sj(B, D̂,θ) = ΣN̂i=1uB,j(ci) (8)
Cj(B, D̂,θ) = 1Sj Σ N̂ i=1(uB,j(ci) · ci) (9)
Aj(B, D̂,θ) = 1Sj Σ N̂ i=1(uB,j(ci) · ai). (10)
The quantities ECEbin,p and ECElb,p can further be written in terms of the quantities Sj , Cj and Aj using equation 6 and a modification of equation 7 (see equation 12 below). We know that the differentials ∂ECEbin,p/∂θ and ∂ECElb,p/∂θ are non-trainable. The formulation above makes it clear that this is precisely because ∂uB,j/∂c is zero within bin boundaries and undefined at bin boundaries. Moreover, this observation implies that if we could come up with a trainable soft binmembership function u∗B then we could use it in place of the usual hard bin-membership function uB to obtain a trainable version of ECEbin,p and ECElb,p.
With this motivation, we define the soft bin-membership function that has a well-defined non-zero gradient in (0, 1). It is parameterized by the number of bins M and a temperature parameter T . We consider equal-width binning for simplicity and so we represent it as u∗M,T rather than u ∗ B. We desire unimodality over confidence: if ξj denotes the center of bin j then we want ∂u∗M,T,j/∂c to be positive for c < ξj negative for c > ξj . Similarly, we also desire unimodality over bins: if c < ξi < ξj or c > ξi > ξj , then we want that u∗M,T,i(c) > u ∗ M,T,j(c). Finally, we also want the aforementioned temperature parameter T to control how close the binning is to hard binning (i.e. how steeply membership drops off). This would give us the nice property of hard-binning being a limiting condition of soft-binning. With this motivation, we define the soft bin-membership function as
u∗M,T (c) = softmax(gM,T (c)),
where gM,T,i(c) = −(c− ξi)2/T ∀i ∈ {1, 2, . . .M}.
Figure 3 visualizes soft bin-membership. We can now formulate trainable calibration error measures. We define the Expected Soft-Binned Calibration Error SB-ECEbin,p(M,T, D̂,θ) and Expected SoftLabel-Binned Calibration Error SB-ECElb,p(M,T, D̂,θ):
SB-ECEbin,p(M,T, D̂,θ) = ( ΣMi=1( Sj N̂ |Aj − Cj |p) )1/p , (11)
SB-ECElb,p(M,T, D̂,θ) = (
1 N̂ ΣN̂i=1Σ M j=1(u ∗ M,T,j(ci) · |Aj − ci|p)
)1/p . (12)
The quantities Sj , Cj and Aj in these expressions are obtained by using the soft bin membership function u∗M,T in place of the hard bin membership function uB in equations 8, 9 and 10 respectively.
We use these trainable calibration error measures as: (1) part of the training loss function and (2) the objective that is minimized to tune the temperature scaling parameter Tts during post-hoc calibration.

heading: 4.2 Soft AvUC (S-AvUC)
text: The accuracy versus uncertainty calibration (AvUC) loss [Krishnan and Tickoo, 2020] categorizes each prediction that a model with parameters θ makes for a labelled datapoint di = (xi, yi) from dataset D̂ according to two axes: (1) accurate [A] versus inaccurate [I], based on the value of the boolean quantity aθ(xi, yi) = ai (2) certain [C] versus uncertain [U], based on whether the entropy
h(fθ(xi)) = hi of the predictive distribution is above or below a threshold κ. Denote the number of elements from D̂ that fall in each of these 4 categories by n̂AC, n̂AU, n̂IC and n̂IU respectively. The AvUC loss incentivizes the model to be certain when accurate and uncertain when inaccurate:
AvUC(κ, D̂,θ) = log (
1 + nAU + nIC nAC + nIU
) , (13)
where the discrete quantities are relaxed to be differentiable: nAU = Σi|(xi,yi)∈SAU(ci tanhhi)
nAC = Σi|(xi,yi)∈SAC(ci(1− tanhhi)) nIC = Σi|(xi,yi)∈SIC((1− ci)(1− tanhhi)) nIU = Σi|(xi,yi)∈SIU((1− ci) tanhhi). (14)
Krishnan and Tickoo [2020] have showed good calibration results using the AvUC loss in SVI settings. However, in our experiments we found that the addition of the AvUC loss term resulted in poorly calibrated models in non-SVI neural network settings (see Appendix A). One reason for this seems to be that minimizing the AvUC loss results in the model being incentivized to be even more confident in its inaccurate and certain predictions (via minimizing nIC) and even less confident in its accurate and uncertain predictions (via minimizing nAU). This conjecture is validated by experimental observations: when we stopped the gradients flowing through the ci terms in equation 14, we were able to obtain calibrated models (see Appendix F). Fixing this incentivization issue in a more principled manner than stopping gradients is desirable. Another desirable improvisation is replacing the hard categorization into the certain/uncertain bins with a soft partitioning scheme. We meet both these objectives by defining a notion of soft uncertainty.
We want a limiting case of the soft uncertainty function to be the hard uncertainty function based on an entropy threshold κ. This implies that we will continue to have a parameter κ despite getting rid of the hard threshold. As before, we desire a temperature parameter T that will determine how close the function is to the hard uncertainty function. The soft uncertainty function tκ,T : [0, 1]→ [0, 1] takes as input the [0, 1]-normalized entropy h∗i = hi/ log(K) of the predicted posterior where K is the number of classes. We also need ∂tκ,T /∂h∗ to be positive in [0, 1] and would like tκ,T to satisfy limh∗→0 tκ,T (h∗) = 0 and limh∗→1 tκ,T (h∗) = 1. Finally, it would be good to have the [0, 1] identity mapping as a special case of tκ,T -family for some value of κ and T . We now define the soft-uncertainty function in the following way so that it meets all stated desiderata:
tκ,T (h ∗) = logistic
( 1
T log h∗(1− κ) (1− h∗)κ
) .
Finally, we define Soft AvUC in terms of soft uncertainty by modifying equations 13 and 14. In our experiments, we use Soft AvUC as part of the loss function to obtain calibrated models.
S-AvUC(κ, T, D̂,θ) = log ( 1 + n′AU + n ′ IC
n′AC + n ′ IU
) , (15)
where n′AU = Σi|ai=1(tκ,T (h ∗ i ) tanhhi)
n′AC = Σi|ai=1((1− tκ,T (h ∗ i ))(1− tanhhi))
n′IC = Σi|ai=0((1− tκ,T (h ∗ i ))(1− tanhhi)) n′IU = Σi|ai=0(tκ,T (h ∗ i ) tanhhi).
(16)

heading: 5 Results
text: We compare our Soft Calibration Objectives to recently proposed calibration-incentivizing training objectives MMCE, focal loss, and AvUC on the CIFAR-10, CIFAR-100, and ImageNet datasets. We evaluate the full cross-product of primary and secondary losses: the options for primary loss are cross-entropy (NLL), focal or mean squared error (MSE) loss; and the options for secondary loss are MMCE, AvUC, SB-ECE or S-AvUC. Results for the MSE primary loss and the AvUC secondary loss are in Appendix A. Our experiments build on the Uncertainty Baselines and Robustness Metrics libraries [Nado et al., 2021, Djolonga et al., 2020].

heading: 5.1 Soft Calibration Objectives for End-to-End Training
text: Our results demonstrate that training losses which include Soft Calibration Objectives obtain state-ofthe-art single-model ECE on the test set in exchange for less than 1% reduction in accuracy for all three datasets that we experiment with. In fact, our methods (especially S-AvUC) without post-hoc temperature scaling are better than or as good as other methods with or without post-hoc temperature scaling on all three datasets.
The primary losses we work with for CIFAR-10/100 are the cross-entropy (NLL) loss, the focal loss and mean squared error (MSE) loss. Focal loss [Mukhoti et al., 2020] and MSE loss [Hui and Belkin, 2021] have recently shown to outperform the NLL loss in certain settings. The cross-entropy loss outperforms the other two losses on Imagenet, and is thus our sole focus for this dataset.
The primary loss even by itself (especially NLL) can overfit to the train ECE [Mukhoti et al., 2020], without help from the soft calibration losses. Even in such settings, we show that soft calibration losses yield reduction of test ECE using a technique we call ‘interleaved training’ (Appendix B).
We use the Wide-Resnet-28-10 architecture [Zagoruyko and Komodakis, 2017] trained for 200 epochs on CIFAR-100 and CIFAR-10. For Imagenet, we use the Resnet-50 [He et al., 2015] architecture training for 90 epochs. All our experiments use the SGD with momentum optimizer with momentum fixed to 0.9 and learning rate fixed to 0.1. The loss function we use in our experiments is PL + β · SL + λ · L2 where PL and SL denote the primary and secondary losses respectively and L2 denotes the weight normalization term with `2 norm. We tune the β and λ parameters along with the parameters κ and T relevant to the secondary losses SB-ECElb,p(M,T, D̂,θ) and S-AvUC(κ, T, D̂,θ). We tune these hyperparameters sequentially. We fix the learning rate schedule and the number of bins M to keep the search space manageable. Appendix G has more details of our hyperparameter search.
In Table 1, we report the runs with the best ECE values that also came within 1% of the primary loss run with the highest accuracy. Figure 6 is the accuracy-confidence plot corresponding to Table 1a. In Figure 5, we visualize the ECE on the test set for all combinations of primary loss, secondary loss and dataset. The best ECE for each dataset is attained using Soft Calibration Objectives as secondary losses. More such figures are in Appendix D and the complete table can be found in Appendix A.

heading: 5.2 Soft Calibration Objectives for Post-Hoc Calibration
text: Standard temperature scaling (TS) uses a cross-entropy objective to optimize the temperature. However, using our differentiable soft binning calibration objective (SB-ECE), we can optimize the temperature using a loss function designed to directly minimize calibration error. In Figure 2 (and Figure 9 in Appendix C), we compare temperature scaling with a soft binning calibration objective (TS-SB-ECE) to standard temperature scaling with a cross-entropy objective (TS) on outof-distribution shifts of increasing magnitude on both CIFAR-10 and CIFAR-100. The distribution
shifts come from either the CIFAR-10-C or CIFAR-100-C datasets. Whereas Figure 2 focuses on cross-entropy loss, Figure 9 also has the plots for other primary losses. Table 2 contains comparisons between the two methods based on test ECE for all combinations of dataset, primary loss and secondary loss. We find that TS-SB-ECE outperforms TS under shift in most cases, and the performance increase is similar across shifts of varying magnitude. Note that temperature scaling (TS) does not always improve ECE, especially when the training loss is different from NLL. In such cases TS-SB-ECE still outperforms TS but may or may not result in ECE improvement.

heading: 5.3 Training for Calibration Under Distribution Shift
text: In previous sections we have shown that training for calibration outperforms the popular cross-entropy loss coupled with post-hoc TS on the in-distribution test set. We find that methods which train for calibration (not always our proposed methods) also outperform the cross-entropy loss with TS under dataset shift. Prior work has shown that temperature scaling can perform poorly under distribution shift [Ovadia et al., 2019], and our experiments reproduce this issue. Moreover, we show that training for calibration makes progress towards fixing this problem. However, different methods perform best under distribution shift on different datasets. Whereas S-AvUC does well on ImageNet OOD (see figure 8), Focal loss does better than our methods on CIFAR-10-C and CIFAR-100-C (see figure 7). We cannot prescribe one method in particular under distribution shift given these results but we have shown a crucial benefit of using methods to train for calibration as a whole.

heading: 6 Conclusions
text: We proposed Soft Calibration Objectives motivated by the goal of directly training models for calibration. These objectives - SB-ECE and S-AvUC - are softened versions of the usual ECE measure and the recently-proposed AvUC loss, respectively. They are easy-to-implement augmentations to the popular cross-entropy loss. We performed a thorough comparison of existing methods of training for calibration. Our experiments show that methods based on soft-calibration objectives can be used to obtain the best ECE among such methods in exchange for less than 1% drop in accuracy. We note that a model being better calibrated overall does not necessarily mean that it is better calibrated for every group and hence the fairness of our methods as well as related methods must be studied. However, our methods of training-for-calibration can be adapted to encourage fairness by applying the methods separately to each protected group.
Even when one does not wish to incorporate secondary losses to train for calibration, we showed that post-hoc temperature scaling works better when tuned using the SB-ECE objective instead of the standard cross-entropy loss. Practitioners can easily replace the cross-entropy loss with our SB-ECE loss when performing post-hoc temperature scaling.
Finally, we demonstrated that the uncertainty estimates of methods which train for calibration generalize better under dataset shift as compared to post-hoc calibration, which is a fundamental motivation for transitioning to training for calibration.

heading: Acknowledgements
text: The authors thank Brennan McConnell and Mohammad Khajah who conducted initial explorations of soft binning calibration loss. The authors also thank Zack Nado, D. Sculley and Jeremiah Liu for help with implementation and suggestions for the writeup.

Review summaries:
This paper addresses neural networks miscalibration by introducing a soft version of two different calibration measures, namely ECE and AvUC.   To make ECE differentiable, the authors propose to soften the bin membership in ECE calculation and define differentiable bin size, bin confidence, and bin accuracy based on the soft membership function. The membership function is characterized by a categorical distribution (implemented by softmax) over the temperature-scaled Euclidean distances between the confidence score of a sample and the centers of the bins. The centers of the bins are pre-determined as the center of equal-width binning with some pre-known number of bins. The authors use the differentiable ECE as an additional term added to traditional loss functions for training classifiers. They also use this loss alone for post-hoc calibration. In addition to ECE, they soften the newly introduced AvUC loss with similar tricks.  Experiments on benchmark datasets are conducted to demonstrate the effectiveness of the proposed method regarding ECE performance metrics.

The paper presents differentiable loss functions to improve the calibration of deep nets class probability estimates. To achieve this, the authors propose a soft approximation of binning operation, which is at the core of popular measures for model calibration error. Empirical evidence shows that the new approach achieves improved calibration performance compared to existing methods while maintaining accuracy to the extent possible.