Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning

1 INTRODUCTION . Despite great ability in learning discriminative features , deep neural network ( DNN ) classifiers often make over-confident predictions . This can lead to potentially catastrophic consequences in safety critical applications , e.g. , medical diagnosis and autonomous driving perception tasks . A multi-class classifier is perfectly calibrated if among the cases receiving the prediction distribution q , the ground truth class distribution is also q . The mismatch between the prediction and ground truth distribution can be measured using the Expected Calibration Error ( ECE ) ( Guo et al. , 2017 ; Kull et al. , 2019 ) . Since the pioneering work of ( Guo et al. , 2017 ) , scaling methods have been widely acknowledged as an efficient post-hoc multi-class calibration solution for modern DNNs . The common practice of evaluating their ECE resorts to histogram density estimation ( HDE ) for modeling the distribution of the predictions . However , Vaicenavicius et al . ( 2019 ) proved that with a fixed number of evaluation bins the ECE of scaling methods is underestimated even with an infinite number of samples . Widmann et al . ( 2019 ) ; Kumar et al . ( 2019 ) ; Wenger et al . ( 2020 ) also empirically showed this underestimation phenomena . This deems scaling methods as unreliable calibration solutions , as their true ECEs can be larger than evaluated , putting many applications at risk . Additionally , setting HDE also faces the bias/variance trade-off . Increasing its number of evaluation bins reduces the bias , as the evaluation quantization error is smaller , however , the estimation of the ground truth correctness begins to suffer from high variance . Fig . 1-a ) shows that the empirical ECE estimates of both the raw network outputs and the temperature scaling method ( TS ) ( Guo et al. , 2017 ) are sensitive to the number of evaluation ∗firstname.lastname @ de.bosch.com Code available at https : //github.com/boschresearch/imax-calibration 1 bins . It remains unclear how to optimally choose the number of evaluation bins so as to minimize the estimation error . Recent work ( Zhang et al. , 2020 ; Widmann et al. , 2019 ) suggested kernel density estimation ( KDE ) instead of HDE . However , the choice of the kernel and bandwidth also remains unclear , and the smoothness of the ground truth distribution is hard to verify in practice . An alternative technique for post-hoc calibration is Histogram Binning ( HB ) ( Zadrozny & Elkan , 2001 ; Guo et al. , 2017 ; Kumar et al. , 2019 ) . Note , here HB is a calibration method and is different to the HDE used for evaluating ECEs of scaling methods . HB produces discrete predictions , whose probability mass functions can be empirically estimated without using HDE/KDE . Therefore , its ECE estimate is constant and unaffected by the number of evaluation bins in Fig . 1-a ) and it can converge to the true value with increasing evaluation samples ( Vaicenavicius et al. , 2019 ) , see Fig . 1-b ) . The most common variants of HB are Equal ( Eq . ) size ( uniformly partitioning the probability interval [ 0 , 1 ] ) , and Eq . mass ( uniformly distributing samples over bins ) binning . These simple methods for multi-class calibration are known to degrade accuracy , since quantization through binning may remove a considerable amount of label information contained by the classifier ’ s outputs . In this work we show that the key for HB to retain the accuracy of trained classifiers is choosing bin edges that minimize the amount of label information loss . Both Eq . size and mass binning are suboptimal . We present I-Max , a novel iterative method for optimizing bin edges with proved convergence . As the location of its bin edges inherently ensures sufficient calibration samples per bin , the bin representatives of I-Max can then be effectively optimized for calibration . Two design objectives , calibration and accuracy , are thus nicely disentangled under I-Max . For multi-class calibration , I-Max adopts the one-vs-rest ( OvR ) strategy to individually calibrate the prediction probability of each class . To cope with a limited number of calibration samples , we propose to share one binning scheme for calibrating the prediction probabilities of similar classes , e.g. , with similar class priors or belonging to the same class category . At small data regime , we can even choose to fit one binning scheme on the merged training sets of all per-class calibrations . Such a shared class-wise ( sCW ) calibration strategy greatly improves the sample efficiency of I-Max binning . I-Max is evaluated according to multiple performance metrics , including accuracy , ECE , Brier and NLL , and compared against benchmark calibration methods across multiple datasets and trained classifiers . For ImageNet , I-Max obtains up to 66.11 % reduction in ECE compared to the baseline and up to 38.14 % reduction compared to the state-of-the-art GP-scaling method ( Wenger et al. , 2020 ) . 2 RELATED WORK . For confidence calibration , Bayesian DNNs and their approximations , e.g . ( Blundell et al. , 2015 ) ( Gal & Ghahramani , 2016 ) are resource-demanding methods to consider predictive model uncertainty . However , applications with limited complexity overhead and latency require sampling-free and singlemodel based calibration methods . Examples include modifying the training loss ( Kumar et al. , 2018 ) , scalable Gaussian processes ( Milios et al. , 2018 ) , sampling-free uncertainty estimation ( Postels et al. , 2019 ) , data augmentation ( Patel et al. , 2019 ; Thulasidasan et al. , 2019 ; Yun et al. , 2019 ; Hendrycks et al. , 2020 ) and ensemble distribution distillation ( Malinin et al. , 2020 ) . In comparison , a simple approach that requires no retraining of the models is post-hoc calibration ( Guo et al. , 2017 ) . 2 Prediction probabilities ( logits ) scaling and binning are the two main solutions for post-hoc calibration . Scaling methods use parametric or non-parametric models to adjust the raw logits . Guo et al . ( 2017 ) investigated linear models , ranging from the single-parameter based TS to more complicated vector/matrix scaling . To avoid overfitting , Kull et al . ( 2019 ) suggested to regularize matrix scaling with a L2 loss on the model weights . Recently , Wenger et al . ( 2020 ) adopted a latent Gaussian process for multi-class calibration . Ji et al . ( 2019 ) extended TS to a bin-wise setting , by learning separate temperatures for various confidence subsets . To improve the expressive capacity of TS , an ensemble of temperatures were adopted by Zhang et al . ( 2020 ) . Owing to continuous outputs of scaling methods , one critical issue discovered in the recent work is : Their empirical ECE estimate is not only non-verifiable ( Kumar et al. , 2019 ) , but also asymptotically smaller than the ground truth ( Vaicenavicius et al. , 2019 ) . Recent work ( Zhang et al. , 2020 ; Widmann et al. , 2019 ) exploited KDEs for an improved ECE evaluation , however , the parameter setting requires further investigation . Nixon et al . ( 2019 ) and ( Ashukha et al. , 2020 ) discussed potential issues of the ECE metric , and the former suggested to 1 ) use equal mass binning for ECE evaluation ; 2 ) measure both top-1 and class-wise ECE to evaluate multi-class calibrators , 3 ) only include predictions with a confidence above some epsilon in the class-wise ECE score . As an alternative to scaling , HB quantizes the raw confidences with either Eq . size or Eq . mass bins ( Zadrozny & Elkan , 2001 ) . It offers asymptotically convergent ECE estimation ( Vaicenavicius et al. , 2019 ) , but is less sample efficient than scaling methods and also suffers from accuracy loss ( Guo et al. , 2017 ) . Kumar et al . ( 2019 ) proposed to perform scaling before binning for an improved sample efficiency . Isotonic regression ( Zadrozny & Elkan , 2002 ) and Bayesian binning into quantiles ( BBQ ) ( Naeini et al. , 2015 ) are often viewed as binning methods . However , their ECE estimates face the same issue as scaling methods : though isotonic regression fits a piecewise linear function , its predictions are continuous as they are interpolated for unseen data . BBQ considers multiple binning schemes with different numbers of bins , and combines them using a continuous Bayesian score , resulting in continuous predictions . In this work , we improve the current HB design by casting bin optimization into a MI maximization problem . Furthermore , our findings can also be used to improve scaling methods . 3 METHOD . Here we introduce the I-Max binning scheme , which addresses the issues of HB in terms of preserving label-information in multi-class calibration . After the problem setup in Sec . 3.1 , Sec . 3.2 ) presents a sample-efficient technique for one-vs-rest calibration . In Sec . 3.3 we formulate the training objective of binning as MI maximization and derive a simple algorithm for I-Max binning . 3.1 PROBLEM SETUP . We address supervised multi-class classification tasks , where each input x ∈ X belongs to one of K classes , and the ground truth labels are one-hot encoded , i.e. , y = [ y1 , y2 , . . . , yK ] ∈ { 0 , 1 } K . Let f : X 7→ [ 0 , 1 ] K be a DNN trained using cross-entropy loss . It maps each x onto a probability vector q = [ q1 , . . . , qK ] ∈ [ 0 , 1 ] K , which is used to rank the K possible classes of the current instance , e.g. , argmaxk qk being the top-1 ranked class . As the trained classifier tends to overfit to the cross-entropy loss rather than the accuracy ( i.e. , 0/1 loss ) , q as the prediction distribution is typically poorly calibrated . A post-hoc calibrator h to revise q can deliver an improved performance . To evaluate the calibration performance of h ◦ f , class-wise ECE averaged over the K classes is a common metric , measuring the expected deviation of the predicted per-class confidence after calibration , i.e. , hk ( q ) , from the ground truth probability p ( yk = 1|h ( q ) ) : cwECE ( h ◦ f ) = 1 K K∑ k=1 Eq=f ( x ) { ∣∣∣p ( yk = 1|h ( q ) ) − hk ( q ) ∣∣∣ } . ( 1 ) When h is a binning scheme , hk ( q ) is discrete and thus repetitive . We can then empirically set p ( yk = 1|h ( q ) ) as the frequency of label-1 samples among those receiving the same hk ( q ) . On the contrary , scaling methods are continuous . It is unlikely that two samples attain the same hk ( q ) , thus requiring additional quantization , i.e. , applying HDE for modeling the distribution of hk ( q ) , or alternatively using KDE . It is noted that ideally we should compare the whole distribution h ( q ) with 3 the ground truth p ( y|h ( q ) ) . However , neither HDE nor KDE scales well with the number of classes . Therefore , the multi-class ECE evaluation often boils down to the one-dimensional class-wise ECE as in ( 1 ) or the top-1 ECE , i.e. , E [ |p ( yk=arg maxk hk ( q ) = 1|h ( q ) ) −maxk hk ( q ) | ] .

Summary: This paper highlights the issues with the scaling method and histogram binning i.e., underestimate calibration error in scaling methods and failing to preserve classification accuracy, and sample-inefficiency in HB. They use the I-Max concept for binning, which maximizes the mutual information between labels and quantized logits. They claim that their approach mitigates potential loss in ranking performance and allows simultaneous improvement of ranking and calibration performance by disentangling the optimization of bin edges and representatives. They also propose a shared class-wise (sCW) strategy that fits a single calibrator on the merged training sets of all K class-wise problems to improve the  sample efficiency.