Soft Calibration Objectives for Neural Networks

1 Introduction . Despite the success of deep neural networks across a variety of domains , they are still susceptible to miscalibrated predictions . Both over- and under-confidence contribute to miscalibration , and empirically , deep neural networks empirically exhibit significant miscalibration [ Guo et al. , 2017 ] . Calibration error ( CE ) quantifies a model ’ s miscalibration by measuring how much its confidence , i.e . the predicted probability of correctness , diverges from its accuracy , i.e . the empirical probability of correctness . Models with low CE are critical in domains where satisfactory outcomes depend on well-modeled uncertainty , such as autonomous vehicle navigation [ Bojarski et al. , 2016 ] and medical diagnostics [ Jiang et al. , 2012 , Caruana et al. , 2015 , Kocbek et al. , 2020 ] . Calibration has also been shown to be useful for improving model fairness [ Pleiss et al. , 2017 ] and detecting out-of-distribution data [ Kuleshov and Ermon , 2017 , Devries and Taylor , 2018 , Shao et al. , 2020 ] . More generally , low ∗co-first author 2Code available on GitHub : https : //github.com/google/uncertainty-baselines/tree/main/experimental/caltrain 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . CE is desirable in any setting in which thresholds are applied to the predicted confidence of a neural network in order to make a decision . Methods for quantifying CE typically involve binning model predictions based on their confidence . CE is then computed empirically as a weighted average of the absolute difference in average prediction confidence and average accuracy across different bins [ Naeini et al. , 2015 ] . Oftentimes these bins are selected heuristically such as equal-width ( uniformly binning the score interval ) and equal-mass ( with equal numbers of samples per-bin ) [ Nixon et al. , 2019 ] . However , these commonly used measures of CE are not trainable with gradient-based methods because the binning operation is discrete and has zero derivatives . As a result , neural network parameters are not directly trained to minimize CE , either during training or during post-hoc recalibration . In this paper , we introduce new objectives based on a differentiable binning scheme that can be used to efficiently and directly optimize for calibration . Contributions . We propose estimating CE with soft ( i.e. , overlapping , continuous ) bins rather than the conventional hard ( i.e. , nonoverlapping , all-or-none ) bins . With this formulation , the CE estimate is differentiable , allowing us to use it as ( 1 ) a secondary ( i.e. , auxiliary ) loss to incentivize model calibration during training , and ( 2 ) a primary loss for optimizing post-hoc recalibration methods such as temperature scaling . In the same spirit , we soften the AvUC loss [ Krishnan and Tickoo , 2020 ] , allowing us to use it as an effective secondary loss during training for non-Bayesian neural networks where the AvUC loss originally proposed for Stochastic Variational Inference ( SVI ) typically does not work . Even when training with the cross-entropy loss results in training set memorization ( perfect train accuracy and calibration ) , Soft Calibration Objectives are still useful as secondary training losses for reducing test ECE using a procedure we call interleaved training . In an extensive empirical evaluation , we compare Soft Calibration Objectives as secondary losses to existing calibration-incentivizing losses . In the process , we find that soft-calibration losses outperform prior work on in-distribution test sets . Under distribution shift , we find that calibration-sensitive training objectives as a whole ( not always the ones we propose ) result in better uncertainty estimates compared to the standard cross-entropy loss coupled with temperature scaling . Our contributions can be summarized as follows : • We propose simple Soft Calibration Objectives S-AvUC , SB-ECE as secondary losses which optimize for CE throughout training . We show that across datasets and choice of primary losses , the S-AvUC secondary loss results in the largest improvement in ECE as per the Cohen ’ s d effect-size metric ( Figure 1 ) . We also show that such composite losses obtain state-of-the-art single-model ECE in exchange for less than 1 % reduction in accuracy ( Figure 5 ) for CIFAR-10 , CIFAR-100 , and Imagenet . • We improve upon temperature scaling - a popular post-hoc recalibration method - by directly optimizing the temperature parameter for soft calibration error instead of the typical loglikelihood . Our extension ( TS-SB-ECE ) consistently beats original temperature scaling ( TS ) across different datasets , loss functions and calibration error measures , and we find that the performance is better under dataset shift ( Figure 2 ) . • Overall , our work demonstrates a fundamental advantage of objectives which better incentivize calibration over the standard practice of training with cross-entropy loss and then applying post-hoc methods such as temperature scaling . Uncertainty estimates of neural networks trained using these methods generalize better in and out-of-distribution . 2 Related Work . Many techniques have been proposed to train neural networks for calibration . These can be organized into three categories . One category augments or replaces the primary training loss with a term to explicitly incentivize calibration . These include the AvUC loss [ Krishnan and Tickoo , 2020 ] , MMCE loss [ Kumar et al. , 2018 ] and Focal loss [ Mukhoti et al. , 2020 , Lin et al. , 2018 ] . The Mean Squared Error loss also compares favourably [ Hui and Belkin , 2021 ] to cross-entropy loss . We show that our methods outperform all these calibration-incentivizing training objectives , applied across multiple primary losses . Label smoothing [ Müller et al. , 2020 ] has been shown to improve calibration and can be interpreted as a modified primary loss . ECE ECE w/ TS-SB-ECE ( no ECE 0 redn . ) 2 4 6 8 ( huge ECE 10 redn . ) Co he n ' s d Effect size of secondary loss for reducing ECE ( higher is better ) MMCE AvUC S-AvUC SB-ECE Figure 1 : We compare the effect size of various secondary losses on ECE ( equal-mass binning , ` 2 norm ) across datasets and primary losses , both with and without post-hoc temperature scaling . The S-AvUC secondary loss we propose shows the strongest positive effect , followed by MMCE and SB-ECE . Note that a d-value of 0.8 ( resp. , 2.0 ) is considered a large ( resp. , huge ) positive effect and dvalues obtained here are much larger . Secondary losses which incentivize calibration show strong positive effect for reducing ECE even after temperature scaling . test 1 2 3 4 5 Distribution-Shift Intensity 10 20 30 40 EC E ( % ) CIFAR-100 / CIFAR-100-C no rescaling TS TS-SB-ECE test 1 2 3 4 5 Distribution-Shift Intensity 0 10 20 30 EC E ( % ) CIFAR-10 / CIFAR-10-C no rescaling TS TS-SB-ECE Figure 2 : Post-hoc temperature scaling with the soft calibration error objective ( TS-SB-ECE ) reduces ECE more than standard post-hoc temperature scaling ( TS ) , particularly under distribution shift . This result holds across datasets ( left and right panels ) , distribution shift intensities ( along abscissa ) and training objectives ( not shown ) . The training objective shown here for both datasets is the most popular one : NLL . The ECE value ( equal-mass binning , ` 2 norm ) shown is the mean ECE across the corruption types that constitute CIFAR-10-C and CIFAR-100-C. Error bars are ±1 standard error of mean ( SEM ) , corrected for intrinsic variability due to type of corruption [ Masson and Loftus , 2003 ] . A second category of methods are post-hoc calibration methods , which rescale model predictions after training . These methods optimize additional parameters on a held-out validation set [ Platt , 1999 , Zadrozny and Elkan , 2002 , Kull et al. , 2019 , Zadrozny and Elkan , 2001 , Naeini and Cooper , 2016 , Allikivi and Kull , 2019 , Kull et al. , 2017 , Naeini et al. , 2015 , Wenger et al. , 2020 , Gupta et al. , 2020 ] . The most popular technique is temperature scaling [ Guo et al. , 2017 ] , which maximizes a single temperature parameter on held-out NLL . We examine temperature scaling and propose an improvisation that directly optimizes temperature for a soft calibration objective instead of NLL . Temperature scaling has shown to be ineffective under distribution shift in certain scenarios [ Ovadia et al. , 2019 ] . We show that uncertainty estimates of methods which train for calibration generalize better than temperature scaling under distribution shift . A third category of methods examines model changes such as ensembling multiple predictions [ Lakshminarayanan et al. , 2017 , Wen et al. , 2020 ] or priors [ Dusenberry et al. , 2020 ] . Similar to previous work [ Kumar et al. , 2018 , Lin et al. , 2018 ] , we focus on the choice of loss functions for improving calibration of a single neural network—whether during training or post-hoc—and do not compare against ensemble models or Bayesian neural networks . These techniques are complementary to ours and can be combined with our techniques to further improve performance . Recent works have investigated issues [ Nixon et al. , 2019 , Kumar et al. , 2019 , Roelofs et al. , 2020 , Gupta et al. , 2020 ] with the originally proposed ECE [ Guo et al. , 2017 ] and suggested new ones . Debiased CE [ Kumar et al. , 2019 ] and mean-sweep CE [ Roelofs et al. , 2020 ] have been shown to have lesser bias and more consistency across the number of bins parameter than ECE whereas KS-error [ Gupta et al. , 2020 ] avoids binning altogether . We report these metrics in the Appendix . 3 Background . 3.1 The Task and the Model . Consider a classification task over K classes with a dataset of N samples D = 〈 ( xi , yi ) 〉Ni=1 drawn from the joint probability distribution D ( X , Y ) over the input space X and label space Y = { 1 , 2 , . . .K } . The task is modelled using a deep neural network with parameters θ whose top layer is interpreted as a softmax layer . The top layer consists of K neurons which produce logits gθ ( x ) = 〈gθ ( y|x ) 〉y∈Y . The predictive probabilities for a given input are : fθ ( x ) = 〈fθ ( y|x ) 〉y∈Y = softmax ( gθ ( x ) ) . The parameters θ of the neural network are trained to minimize E ( x , y ) L ( fθ ( x ) , y ) where ( x , y ) is sampled from D ( X , Y ) . Here L is a trainable loss function which incentivizes the predictive distribution fθ ( x ) to fit the label y . The model ’ s prediction on datapoint ( x , y ) is denoted by qθ ( x ) = arg max ( fθ ( x ) ) . We denote by cθ ( x ) = max ( fθ ( x ) ) the confidence of this prediction and by boolean quantity aθ ( x , y ) = 1qθ ( x ) =y the accuracy of this prediction . 3.2 Expected Calibration Error ( ECE ) . Given a distribution D̂ ( X , Y ) on datapoints , there are two standard notions of Ideal Calibration Error that we refer to as Ideal Binned Expected Calibration Error ( of order p ) , IECEbin , p ( D̂ , θ ) = IECEbin , p , and Ideal Expected Label-Binned Calibration Error ( of order p ) , IECElb , p ( D̂ , θ ) = IECElb , p . This nomenclature is consistent with that introduced by Roelofs et al . [ 2020 ] . Both these measure , in slightly different ways , the pth root of the pth moment of the absolute difference between model confidence and the empirical accuracy given that confidence . They are defined as follows : IECEbin , p ( D̂ , θ ) = ( Ecθ ( x̂0 ) [ |E [ aθ ( x̂1 , ŷ1 ) |cθ ( x̂1 ) = cθ ( x̂0 ) ] − cθ ( x̂0 ) |p ] ) 1/p ( 1 ) IECElb , p ( D̂ , θ ) = ( E ( x̂0 , ŷ0 ) [ |E [ aθ ( x̂1 , ŷ1 ) |cθ ( x̂1 ) = cθ ( x̂0 ) ] − cθ ( x̂0 ) |p ] ) 1/p . ( 2 ) Note that the critical dependence on D̂ ( X , Y ) is implicit in both definitions since the datapoint ( x̂0 , ŷ0 ) from the outer expectation and the datapoint ( x̂1 , ŷ1 ) from the inner expectation are both sampled from D̂ ( X , Y ) . We can not compute IECEbin , p and IECElb , p in practice since the number of datapoints are finite . Instead we consider a dataset D̂ = 〈 ( x̂i , ŷi ) 〉N̂i=1 drawn from D̂ ( X , Y ) and partition the confidence interval [ 0 , 1 ] into bins B = 〈Bi〉i∈ { 1,2 , ... M } , each of which also corresponds to a confidence interval . We will use ci as a shorthand for cθ ( x̂i ) and ai as a shorthand for aθ ( x̂i , ŷi ) . We denote by bi ( B , D̂ , θ ) = bi the bin to which ci belongs . We define the size of bin j as Sj ( B , D̂ , θ ) = Sj , the average confidence of bin j as Cj ( B , D̂ , θ ) = Cj and the average accuracy of bin j asAj ( B , D̂ , θ ) = Aj . These are expressed as follows : Sj ( B , D̂ , θ ) = | { i|bi = j } | ( 3 ) Cj ( B , D̂ , θ ) = 1Sj Σi|bi=jci ( 4 ) Aj ( B , D̂ , θ ) = 1Sj Σi|bi=jai . ( 5 ) We are now in a position to define the Expected Binned Calibration Error of order p which we denote by ECEbin , p and Expected Label-Binned Calibration Error of order p we denote by ECElb , p . These serve as empirical approximations to the corresponding intractable ideal notions from equations 1 and 2 . They are defined as follows : ECEbin , p ( B , D̂ , θ ) = ( ΣMi=1 Sj N̂ · |Aj − Cj |p ) 1/p ( 6 ) ECElb , p ( B , D̂ , θ ) = ( 1 N̂ ΣN̂i=1|Abi − ci|p ) 1/p . ( 7 ) It follows from Jensen ’ s inequality that ECElb , p ( B , D , θ ) ≥ ECEbin , p ( B , D , θ ) [ Roelofs et al. , 2020 ] .

Summary: This paper addresses neural networks miscalibration by introducing a soft version of two different calibration measures, namely ECE and AvUC.   To make ECE differentiable, the authors propose to soften the bin membership in ECE calculation and define differentiable bin size, bin confidence, and bin accuracy based on the soft membership function. The membership function is characterized by a categorical distribution (implemented by softmax) over the temperature-scaled Euclidean distances between the confidence score of a sample and the centers of the bins. The centers of the bins are pre-determined as the center of equal-width binning with some pre-known number of bins. The authors use the differentiable ECE as an additional term added to traditional loss functions for training classifiers. They also use this loss alone for post-hoc calibration. In addition to ECE, they soften the newly introduced AvUC loss with similar tricks.  Experiments on benchmark datasets are conducted to demonstrate the effectiveness of the proposed method regarding ECE performance metrics.