Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning
Authors: Kanil Patel
William H. Beluch
Bin Yang
Michael Pfeiffer
Dan Zhang
Venue: ICLR 2021 Poster (2021)
Keywords: uncertainty calibration
post-hoc calibration
histogram binning
mutual information
deep neural networks

Abstract: Post-hoc multi-class calibration is a common approach for providing high-quality confidence estimates of deep neural network predictions. Recent work has shown that widely used scaling methods underestimate their calibration error, while alternative Histogram Binning (HB) methods often fail to preserve classification accuracy. When classes have small prior probabilities, HB also faces the issue of severe sample-inefficiency after the conversion into K one-vs-rest class-wise calibration problems. The goal of this paper is to resolve the identified issues of HB in order to provide calibrated confidence estimates using only a small holdout calibration dataset for bin optimization while preserving multi-class ranking accuracy. From an information-theoretic perspective, we derive the I-Max concept for binning, which maximizes the mutual information between labels and quantized logits. This concept mitigates potential loss in ranking performance due to lossy quantization, and by disentangling the optimization of bin edges and representatives allows simultaneous improvement of ranking and calibration performance. To improve the sample efficiency and estimates from a small calibration set, we propose a shared class-wise (sCW) calibration strategy, sharing one calibrator among similar classes (e.g., with similar class priors) so that the training sets of their class-wise calibration problems can be merged to train the single calibrator. The combination of sCW and I-Max binning outperforms the state of the art calibration methods on various evaluation metrics across different benchmark datasets and models, using a small calibration set (e.g., 1k samples for ImageNet).

Published as a conference paper at ICLR 2021
MULTI-CLASS UNCERTAINTY CALIBRATION VIA MU- TUAL INFORMATION MAXIMIZATION-BASS BINNING
Kanil Patel William Beluch', Bin Yang", Michael Pfeiffer', Dan Zhang'
Bosch Center for Artificial Intelligence, Renningen, Germany 2 Institute of Signal Processing and System Theory, University of Stuttgart, Stuttgart, Germany
ABSTRACT
Post-hoc multi-class calibration is a common approach for providing high-quality confidence estimates of deep neural network predictions. Recent work has shown that widely used scaling methods underestimate their calibration error, while al- ternative Histogram Binning (HB) methods often fail to preserve classification accuracy. When classes have small prior probabilities, HB also faces the issue of severe calibration problems. The goal of this paper is to resolve the identified issues of HB in order to provide calibrated confidence estimates using only a small holdout calibration dataset for bin optimization while preserving multi-class ranking accu- racy. From an information- thheoretctt perspective, we derive the I-Max concept for binning, which maximizes the mutual information between labels and quantized logits. This concept mitigates potential loss in ranking performance due to lossy quantization, and by disentangling the optimization of bin edges and representatives allows
after the conversion into one-vs-rest class-wiss
improvement of ranking and calibration performance. To im- prove the sample efficiency and estimates from a small calibration set, we propose a shared class-wise (sCW) calibration strategy, sharing one calibrator among similar classes (c.g., with similar class priors) so that the training sets of their class-wise calibration problems can be merged to train the single calibrator. The combination of sCW and I-Max binning outperforms the state the art calibration methods on various evaluation metrics across different benchmark datasets and models, using small calibration set (e.g., Ik samples for ImageNet).
INTRODUCTION
Despite great ability in learning discriminative features, deep neural network (DNN) classifiers often make over-confident predictions. This can lead potentially catastrophic consequences in safety critical applications, e.g.. medical diagnosis and autonomous driving perception tasks. A multi-class classifier is perfectly calibrated if among the cases receiving the prediction distribution q. the ground truth class distribution is also q. The mismatch between the prediction and ground truth distribution can be measured using the Expected Calibration Error (ECE) (Guo al. 2017 Kull et al.) 2019).
Since the pioneering work of (Guo et al. 2017). scaling methods have been widely acknowledged as an efficient post-hoc multi-class calibration solution for modern DNNs. The common practice of evaluating their ECE resorts to histogram density estimation (HDE) for modeling the distribution of the predictions. However, Vaicenavicius et al. (0199) proved that with a fixed number of evaluation bins the ECE of scaling methods is underestimated even with an infinite number of samples. [iddmann jet al. 2019): Kumar et al. (2019) Wenger et al. (020)) also empirically showed this underestimation phenomena. This deems scaling methods as unreliable calibration solutions, as their true ECEs can be larger than evaluated, putting many applications at risk. Additionally, setting HDEE also faces the bias/variance trade-off. Increasing its number of evaluation bins reduces the bias, as the evaluation quantization error is smaller, however, the estimation of the ground truth correctness begins to suffer from high variance. Fig. a) shows that the empirical ECE estimates of both the raw network outputs and the temperature scaling method (TS) (Guo et al. 2017) are sensitive to the number of evaluation
" firstname. Jastname @ de. bosch.com Code available /ttttp::///// chub. cctm///cchhreeearr///mxxaaaa

---

Published as a conference paper at ICLR 2021
0.12
0.150
HB
0.10
0.125
Ours
0.08
0.100
0.06
0.075
Baseline facc 81.13 %)
HB (ace 78.60 %)
0.04
(acc 81. 13 %)
81.00 %)
0.050
0.02
0.025
100
200
400
500
1000
2000
2000
4000
5000
Number Evaluation Bins
Number of Evaluation Samples
(a) Top-1 prediction ECE (5k evaluation samples)
(b) ECE converging curve (based on 102 bootstraps)
Figure I: (a) Temperature scaling (TS), equally sized-histogram binning (HB), and our proposal, ie., sCW I-Max binning are compared for post-hoc calibrating a CIFAR (WRN) classifier. (b) Binning offers reliable ECE measure the number of evaluation samples increases.
bins. It remains unclear how to optimally choose the number of evaluation bins so to minimize the estimation error. Recent work (Zhang et al. [200]] Widmann et al. (019) suggested kernel density estimation (KDE) instead of HDE. However, the choice of the kernel and bandwidth also remains unclear, and the smoothness of the ground truth distribution is hard to verify in practice.
An alternative technique for post-hoc calibration is Histogram Binning (HB) (Zadrozny & Elkan 2001] Guo et al. 2017 Kumar et al. 2019). Note, here HB a calibration method and is different to the HDE used for evaluating ECEs of scaling methods. HB produces discrete predictions, whose probability mass functions can be empirically estimated without using HDE/KDE. Therefore, its ECE estimate is constant and unaffected by the number of evaluation bins in Fig. aa) and it can converge to the true value with increasing evaluation samples (Vaicenavicius et al. 2019), , see Fig. [b).
The most common variants of HB are Equal (Eq.) size (uniformly partitioning the probability interval [0, ID,, and Eq. mass (uniformly distributing samples over bins) binning. These simple methods for multi-class calibration are known degrade accuracy, since quantization through binning may remove considerable amount of label information contained by the classifier's outputs.
In this work we show that the key for HB to retain the accuracy of trained classifiers choosing bin edges that minimize the amount of label information loss. Both Eq. size and mass binning are suboptimal. We present I-Max, a novel iterative method for optimizing bin edges with proved convergence. As the location of its bin edges inherently ensures sufficient calibration samples per bin, the bin representatives of I-Max can then be effectively optimized for calibration. Two design objectives, calibration and accuracy, are thus nicely disentangled under I-Max. For multi-class calibration, I-Max adopts the one-vs-rest (OvR) strategy to individually calibrate the prediction probability of each class. To cope with limited number of calibration samples, we propose to share one binning scheme for calibrating the prediction probabilitics of similar classes, c.g., with similar class priors or belonging to the same class category. At small data regime, we can even choose to fit one binning scheme on the merged training sets of all per-class calibrations. Such a shared class-wise (sCW) calibration strategy greatly improves the sample efficiency of I-Max binning.
I-Max is evaluated according to multiple performance metrics, including accuracy, ECE, Brier and NLL,, and compared against benchmark calibration methods across multiple datasets and trained classifiers. For ImageNet, I-Max obtains up to 66. 16.11% % reduction in ECE compared to the baseline and up to 38. 88.14% reduction compared the state-of-the-arr GP-scaling method (Wenger et al. 2020).
RELATED WORK
For confidence calibration, Bayesian DNNs and their approximations, e.g. (Blundell et al. 2015) (Gal & Ghahramani] (016) 2016) are resource-demanding methods to consider predictive model uncertainty. However, applications with limited complexity overhead and latency require sampling- free and single- model based calibration methods. Examples include modifying the training loss (Kumar et al. 2018
scalable Gaussian processes (Milios et a (0088). sampling- free uncertainty estimation (Postels et al.
2019), data augmentation (Patel et al. 20199] Thulasidasan al., [201 Yun al.; | Hendryckss jet al. 2020) and ensemble distribution distillation (Malinin et al. 2020). In comparison, a simple approach that requires retraining of the models is post-hoc calibration (Guo al. 2017).
2

---

Published as a conference paper at ICLR 2021
Prediction probabilities (logits) scaling and binning the two main solutions for post-hoc calibration. Scaling methods use parametric or non- parametrrccc models to adjust the raw logits. [uo et al. (2017) investigated linear models, ranging from the ainlle-araamttr based TS to more complicated vector/matrix scaling. To avoid overfitting, Kull et al. (219) suggested to regularize matrix scaling with a L2 loss on the model weights. Recently, Wenger et al. (2020) adopted latent Gaussian process for multi-class calibration. Ji et al. (2019) extended TS to bin-wise setting, by learning separate temperatures for various confidence subsets. To improve the expressive capacity of TS, an ensemble of temperatures were adopted by Zhang et al. (2020). Owing to continuous outputs of scaling methods, one critical issue discovered in the recent work is: Their empirical ECE estimate is not only non-verifiable (Kumar et al., 2019), but also asymptotically smaller than the ground truth (Vaicenavicius et al. 2019). Recent work (Zhang et al. 2020; Widmann et al. 2019) exploited KDEs for an improved ECE evaluation, however, the parameter setting requires further investigation. Nixon et al. (2019) and (Ashukha et al. 2020) discussed potential issues of the ECE metric, and the former suggested to I) use equal mass binning for ECE evaluation; 2) measure both top-1 and class-wise ECE to evaluate multi-class calibrators, 3) only include predictions with a confidence above some epsilon in the class wise ECE score.
As an alternative to scaling, HB quantizes the raw confidences with either Eq. size or Eq. mass bins (Zadrozny & Elkan 2 It offers asymptotically convergent ECE estimation (Vaicenavicius et al. 2019), but is less sample efficient than scaling methods and also suffers from accuracy loss (Guo al.
2017). Kumar et al. (2019) proposed perform scaling before binning for an improved sample
efficiency. Isotonic regression (Zadrozny & Elkan 2002) and Bayesian binning into quantiles (BBQ) (Naeini et al. 2015) are often viewed as binning methods. However, their ECE estimates face the same issue as scaling methods: though isotonic regression fits a piecewise linear function, its predictions are continuous as they are interpolated for unseen data. BBQ considers multiple binning schemes with different numbers of bins, and combines them using a continuous Bayesian score, resulting in continuous predictions.
In this work, we improve the current HB design by casting bin optimization into MI maximization
problem. Furthermore, our findings can also be used to improve scaling methods.
METHOD
Here we introduce the I-Max binning scheme, which addresses the issues of HB in terms of preserving label-infornation in multi-class calibration. After the problem setup in Sec. Sec. 3.2) presents sample-efficient technique for one-vs-rest calibration. In Sec. 3.3 we formulate the training objective of binning as MI maximization and derive simple algorithm for I-Max binning.
3.1 PROBLEM SETUP
We address supervised multi-class classification tasks, where each input x E X' belongs to one of K classes, and the ground truth labels are one-hot encoded, i.e., y = [y 32..... ... yK] E {0. 1 be DNN trained using cross-entropy loss. It maps each x onto probability qk] E [0, 1jK, which is used to rank the possible classes of the current instance, e.g., arg max Is being the top-1 ranked class. As the trained classifier tends to overfit to the cross-entropy loss rather than the accuracy (i.e., 0/1 loss), q as the prediction distribution typically poorly calibrated. A post-hoc calibrator h to revise q can deliver an improved performance. To evaluate the calibration performance of o f. class-wise ECE averaged over the K classes is common metric, measuring the expected deviation of the predicted per-class confidence after calibration, i.e., hy (q), from the ground truth probability p(y = 1[k(q)):
Let f : X' + [0, 1] ke vector q = (4),
K
cw RCE f)
E k=1
Fxx { (pLu = 1(h())) ha (
(1)
K
When h is a binning scheme, hq) is discrete and thus repetitive. We can then empirically set p(y, = 11h(q)) as the frequency of label-1 samples among those receiving the same hy (q). On the contrary, scaling methods are continuous. It is unlikely that two samples attain the same hg(q). thus requiring additional quantization, i.c., applying HDE for modeling the distribution of h2 (q). or alternatively using KDE. It is noted that ideally should compare the whole distribution h(q) with
3

---

Published as a conference paper at ICLR 2021
the ground truth p(y[h(q)) However, neither HDE nor KDE scales well with the number of classes. Therefore, the multi-class ECE evaluation often boils down to the one-dimensional class- ECE as in or the top-1 BCE, i... E [P( (p(yk=arg max, (q) 1[h(q)) max, h4 (q)l]-
3.2 ONE-VS-REST (OVR) STRATEGY FOR MULTI-CLASS CALIBRATION
HB was initially developed for two-class calibration. When dealing with multi-class calibration, it separately calibrates the prediction probability qs of each class in one-vs-rest (OvR) fashion: For any class-k. HB takes yI: as the binary label for a two-class calibration task in which the class-1 means yh 1 and class-O collects all other K 1 classes. It then revises the prediction probability 9u of
y4: = 1 by mapping its logit A - log qı: log(1 qs) onto a given number of bins, and reproducing with the calibrated prediction probability. Here, we choose to bin the logit 2 instead of 9k. as the former is unbounded, i.e., Às E I. which eases the bin edge optimization process. Nevertheless, as Os and A have a monotonic bijective relation, binning qa and ) are equivalent. We note that after K class-wis calibrations we avoid the extra normalization step in (Guo et al. (017). After OvR marginalizes the multi-clas predictive distribution, each class is treated independently (sce Sec. (A]].
The calibration performance of HB depends on the setting of its bin edges and representatives. From a calibration set C = {(y, J. we can construct K training sets, i.c., S₁ = [We, )&) Vk, under the one-vs-rest strategy, and then optimize the class (CW) HB over each training set. As two common solutions in the literature, Eq. size and Eq. mass binning focus on bin representative optimization. Their bin edge locations, on the other hand, are either fixed (independent of the calibration set) or only ensures a balanced training sample distribution over the bins. After binning the logits in the calibration set Sy {(yk Ax)}, the bin representatives are set as the empirical frequencies of samples with = 1 in each bin. To improve the sample efficiency of bin representative optimization, Kumar et al. (2019) proposed to perform scaling-based calibration before HB. Namely, after properly scaling the logits {A*}. J. the bin representative per bin is then set as the averaged sigmoid-response of the scaled logits in Sy belonging to each bin.
However, pre-scaling does not resolve the sample inefficiency issue arising from small class prior Pa. The two-class ratio in S& is pk : - Pk- When pk is small we will need a large calibration set C = {(y,)) x)} to collect enough class-1 samples in Sy for setting the bin representatives. To address this, we propose to merge } across similar classes and then use the merged set for HB training. yielding one binning scheme shareable to multiple per-class calibration tasks, i.c., shared class-wise (sCW) binning instead of CW binning respectively trained on S₁. In Sec. 4 we respectively experiment using single binning schemes for all classes in the balanced multi-class setting, and sharing one binning among the classes with similar class priors in the imbalanced multi-class setting. Note, both S₂ and S serve empirical approximations
the inaccessible ground truth distribution P(Me, Ax)) for bin optimization. The former suffers from high variances, arising from insufficient samples (Fig. a)] while the latter is biased due to having samples drawn from the other classes (Fig. AJ b). As the calibration set size is usually small, the variance is expected to outweigh the approximation error over the bias (see an empirical analysis Sec. A2
3.3 BIN OPTIMIZATION VIA MUTUAL INFORMATION (MI) MAXIMIZATION
Binning can be viewed as a quantizer Q that maps the real-valued logit À E RR to the bin interval m (1,..., M] if X E Imm [9m-1. 9m), where M the total number of bin intervals, and the bin edges Im are sorted (9m-1 Im. and gu 30, IM oo). Any logit binned to Im will be reproduced to the same bin representative Tm. In the context of calibration, the bin representative I'm assigned to the logit Å is used as the calibrated prediction probability of the class-k. As multiple classes can be assigned with the same bin representative, we will then encounter ties when making top-k predictions based calibrated probabilities. Therefore, binning as lossy quantization generally ranking performance, being subject to potential accuracy loss.
does not preserve the raw
Unfortunately, increasing M to reduce the quantization error is not good solution here. For given calibration set, the number of samples per bin generally reduces as M increases, and a reliable frequency estimation for setting the bin representatives {r..} demands sufficient samples per bin.
Considering that the top-k accuracy reflects how well the ground truth label can be recovered from the logits, we propose bin optimization via maximizing the MI between the quantized logits Q(A)
4

---

Published as a conference paper at ICLR 2021
HlylA)
0.1
Density
puyy-1
p(Aly -
Highest Uncortainty
ply-0)-0.99
ply 11-0.01
0.0
binningg
Equel size brining
Equail braning
-30
-20 -10
Logit (A)
Figure 2: Histogram and KDE of CIFARIO0 (WRN) logits in S constructed from 1K calibration samples. The bin edges of Eq. mass binning are located at the high mass region, mainly covering class-O due to the imbalanced two class ratio 1 : 99. Both Eq. size and I-Max binning cover the high uncertainty region, but here only I-Max yields reasonable bin widths ensuring enough mass per bin. Note, Eq. size binning uniformly partitions the interval [0, in the probability domain. The observed dense and symmetric bin location around zero is the outcome of probability-to--oge translation.
and the label
{g arg max T(y; m Q())) (0) Q: (gm)
arg max ((m) - (mmlv) Q: 1m
(2)
where the index m is viewed discrete random variable with P(mly) 1gm_, p(Aly)d and P(m) f p(A)dA, and the equality (a) is based the relation of MI to the entropy H (m) and conditional entropy H (mlyy) of m. Such formulation offers a quantizer Q" optimal at preserving the label information for a given budget on the number of bins. Unlike designing distortion- -based quantizers, the reproducer values of raw logits, i.c., the bin representatives [rm]. are not a part of the optimization space, as it is sufficient to know the mapped bin index m of each logit. Once the bin edges {gim } are obtained, the bin representative Im to achieve zero calibration error shall equal Ply 1/m,, which can be empirically estimated from the samples within the bin interval I,m-
is interesting to analyze the objective function after the equality (a) in (2. The first term H (m) maximized if P(m) is uniform, which is attained by Eq. mass binning. A uniform sample distribution over the bins is a sample-efficient strategy to optimize the bin representatives for the sake of calibration. However, does not consider any label information, and thus can suffer from severe accuracy loss. Through MI maximization, we can view I-Max as revising Eq. mass by incorporating the label information into the optimization objective, i.e., having the second term H (mly). As result, I-Max not only enjoys a well balanced sample distribution for calibration, but also maximally preserved label information for accuracy.
In the example of Fig. the bin edges of I-Max binning are densely located in an area where the uncertainty of y given the logit is high. This uncertainty results from small gaps between the top class predictions. With small bin widths, such nearby prediction logits are more likely located to different bins, and thus distinguishable after binning. On the other hand, Eq. mass binning has a single bin stretching across this high-uncertainty area due to an imbalanced ratio between the p(Xly = 1) and p(Aly = 0) samples. Eq. size binning follows pattern closer to I-Max binning. However, its very narrow bin widths around zero may introduce large empirical frequency estimation errors when setting the bin representatives.
For solving the problem (2). formulate an equivalent problem. Theorem 1. The MI maximization problem given in (2) is equivalent to min L((gm, 2m (9m
max 1 m Q()))
(3)
Q:(mm
where the loss Li(gm, thm 1) is defined as
M
Ply = y²)
Cllgm, 2m
P(A)
Σ
P(y= log
dd
(4)
[2ny 1)
m==
9m
y'E(0.13
and {0m} ) as a of real-valued auxiliary variables are introduced here to ease the optimization.
5

---

Published as a conference paper at ICLR 2021
Eq. Size
Eq. Mass
I-Max
fly; (V)) [bits]
0.064
fly; (V)Q))))
0.062
-Eq.sizze
0.050
0.06
0.060
MaxiEE. mass init.)
0.025
4.4444444
MaxiEo. init.)
Upper Bound
+e-Eq. Eq. mass MI
0.000
0.05
0.056
0.5
1.0
0
20
40
60
80
100
Iterations
I(A; Q())) [bits]
(a) Convergence behavior
Lahel-information vs. compression rate
The KDEs of p(A(g) y) for {0, 1 shown in Fig. are used as the ground truth distribution to synthesize dataset Skde and evaluate the MI of Eq. mass, Eq. size, and I-Max binning trained over Sade. (a) The developed iterative solution for I-Max bin optimization over Skde successfully increases the MI over iterations, approaching the theoretical upper bound / (y; A). For comparison, I-Max is initialized with both Eq. size and Eq. mass bin edges, both of which are suboptimal at label information preservation. (b) We compare the three binning schemes with 2 to 16 quantization levels against the IB limit (Tishby et al. 1999) the aabel-ifformation [ Q(2)) vs. the compression rate [A; Q())) The information-rate pairs achieved by I-Max binning are very close to the limit. The information loss of Eq. mass binning is considerably larger, whereas Eq. size binning gets stuck in the low rate regime, failing to reach the upper bound even with more bins.
Figure 3: MI evaluation:
Proof. See A3 for the proof.
Next, we compute the derivatives of the loss L with respect to {gm, Φm When the conditional distribution A))lA) takes the sigmoid model, i.e., P(ylA) o((2y 1)A,, the stationary points of C, zeroing the gradients over {gm. J. have a closed- form expression
log
1te
[sm+lo (A)p)o(A
En (An)) ) (Ex.es.o(-An) (-An))
9m
=log
log
Le
mm-1
mmn-1
Φm -lo-og--
(mm+ 0 ((-))((A)d)
zolog
(5)
Itc
6m
where the approximation for Φm arises from using the logits in the training set S as an empirical
). So, we can solve the problem by iteratively and alternately updating {9m} and { based on (5) (see Algo. i the appendix for pseudocode). The
approximation p()) and Sm e Sn [9
Im 9
convergence and initialization of such an iterative method well as the sigmoid -ogell assumption
are discussed along with the proof of Theorem in Sec. A3
As the iterative method operates under an approximation the inaccessible ground truth distribution p(y, A), we synthesize an example, see Fig. to assess its effectiveness. As quantization can only reduce the MI, we evaluate I (y: A), serving as the upper bound in Fig. 3 a) for (3: Q())). Among the three realizations of Q. I-Max achieves higher MI than Eq. size and Eq. mass, and more importantly, it approaches the upper bound over the iterations. Next, we assess the performance within the framework of information bottleneck (IB) (Tishby al. 1999). see Fig. In the context of our problem, IB tackles min 1/8 x I(A; (()) - T[: Q())) with the weight factor 3 > 0 to balance between 1) maximizing the information
[(r; Q()). and 2) minimizing the compression rate T(x; Q()).. By varying B, IB gives the maximal achievable information rate for the given compression rate. Fig. b) shows that I-Max approaches the theoretical limits and provides an information-theoree perspective on the sub-optimal performance of the alternative binning schemes. Sec. A3.2 has a more detailed discussion on the connection of IB and our problem formulation.
EXPERIMENTS
We evaluate post-hoc calibration methods four benchmark datasets, i.c.. ImageNet (Degg et al. 2009). CIFAR 10/100 (Krizhessyy 2009) and SVHN (Netzer et al. 201 1). and across various modern DNNs architectures. More details reported in Sec. A8.1
Datasets and Models
Training and Evaluation Details We perform class-balanced random splits of the data test set, unless stated otherwise: the calibration and evaluation set sizes are both 25k for ImageNet, and 5k for

---

Published as a conference paper at ICLR 2021
CIFAR10/100 Different to ImageNet and CIFARIO/100, the test set of SVHN is class imbalanced. We evenly split it into the calibration and evaluation set of size 13k. All reported numbers are the means across 5 random splits; stds can be found in the appendix. Note that some calibration methods only use a subset of the available calibration samples for training, showing their sample efficiency. Further calibrator training details are provided in Sec. A8.1
We empirically evaluate MI, Accuracy (top-1 and 5 ACCS), ECE (class- wise and top-1), Brier and NLL; the latter are shown in the appendix. Analogous to (Nixon et al. 2019). we use thresholding when evaluating the class-wise ECE (cw Without thresholding, the empirical class-wise ECE score may be misleading. When a class-k has a small class prior (e.g. 0.01 or 0.001). the empirical class-wise ECE score will be dominated by prediction samples where the class-k is not the ground truth. For these cases, a properly trained classifier will often not rank this class-k among the top classes and instead yield only small calibration errors. While it is good to have many cases with small calibration errors, they should not wash out the calibration errors of the rest of the cases (prone to poor calibration) through performance averaging. These include (I) class-k is the ground truth class and not correctly ranked and (2) the classifier mis-classifies some class-j as class-k. The thresholding remedies the washing out by focusing more crucial cases (i.e. only averaging across cases where the prediction of the class-k is above a threshold). In all experiments, our primary choice of threshold is to set it according the class prior for the reason that the class-k is unlikely to be the ground truth if its a-posteriori probability becomes lower than its prior after observing the sample.
While empirical ECE estimation of binning schemes is simple, resort HDE with 100 equal size evaluation bins (Wenger et al. (020) for scaling methods. Sec. [A6 reports the results attained by HDE with additional binning schemes and KDE. For HDE-based ones, we notice that with 100 evaluation bins, the ECE estimate is insensitive to the choice of binning scheme.
4.1 EQ. SIZE, EQ. MASS VS. I-MAX BINNING
In Tab. | we compare three binning schemes: Eq size, Eq. mass and I-Max binning. The accuracy performances of the binning schemes are proportional to their MI; Eq. mass binning is highly sub- optimal at label information preservation, and thus shows severe accuracy drop. Eq. size binning accuracy is more similar to that of I-Max binning. but still lower, in particular at Acc Also note that I-Max approaches the MI theoretical limit of [w; x)=0.0068. Advantages of I-Max become even more prominent when comparing the NLLs of the binning schemes. For all ECE evalution metries, I-Max binning improves on the baseline calibration performance, and outperforms Eq. size binning. Eq. mass binning is out of this comparison scope due to its poor accuracy deeming the method impractical. Overall, I-Max successfully mitigates the negative impact of quantization on ACCs while still providing an improved and verifiable ECE performance. Additionally, one-for-all sCW I-Max achieves an even better calibration with only 1 calibration samples, instead of the standard CW binning with 25k calibration samples, highlighting the effectiveness of the sCW strategy.
Furthermore, it is interesting to note that EwEC of the Baseline classifier is very small, i.e., 0.000442, thus it may appear as the Baseline classifier is well calibrated. However, top1 ECE is much larger, i.e., 0.0357. Such inconsistent observations disappear after thresholding the class-wise ECE with the class prior. This example confirms the necessity of thresholding the dass ECE.
In Sec. A5 we perform additional ablations on the number of bins and calibration samples. Accord- ingly, a post-hoc analysis investigates how the quantization error of the binning schemes change the ranking order. Observations are consistent with the intuition behind the problem formulation (see Sec. 3.3 and empirical results from Tab. | that MI maximization is proper criterion for multi-class calibration and maximally mitigates the potential accuracy loss.
4.2 SCALING VS. I-MAX BINNING
In Tab. 2 we compare I-Max binning to benchmark scaling methods. Namely, matrix scaling with
La regularization (Kull et al. 2019) has large model capacity compared to other parametric scaling methods, while TS (Guo et al., 2017) only uses a single parameter and MnM (Zhang et al.) 2020) uses three temperatures as ensemble of TS (ETS). As non- parammeeei method, GP (Wenger et al. 2020) yields state of the art calibration performance. Additional 8 scaling methods can be found in Sec. AT Benefiting from its model capacity, matrix scaling achieves the best accuracy. I-Max
7

---

Published as a conference paper at ICLR 2021
Table 1: ACCs and ECES of Eq. mass, Eq. size and I-Max binning for the case of ImageNet (InceptionResNetV2), Due to the poor accuracy of Eq. mass binning, its ECEs are not considered for comparison. The MI empirically evaluated based on KDE analogous Fig. where the MI upper bound is (y; x)=0.0068. For the other datasets and models, we refer to A9
Binn.
sCW(?)
size
MI f
AcCrpp t AcCoops
cwECE + cw ECE prior
BCE
NLL +
Baseline
x
80,33
95.10
0.000442
0,0486
0,0357
0.8406
X
25k
0.0026
7.78
27.92
0.000173
0.0016
0.0606
3,5960
Eq. Mass
/
1k
0.0026
5.02
26.75
0.000165
0.0022
0.0353
3.5272
X
25k
0.0053
78.52
89.06
0,000310
0.1344
0.0547
1.5159
Eq. Size
/
1k
0.0062
80.14
88.99
0.000298
0.1525
0.0279
1.2671
X 25k
0.0066
80.27
95.01
0.000346
0,0342
0,0329
0.8499
I-Max
1k 0.0066
80.20
94.86
0.000296
0.0302
0.0200
0.7860
Table 2: ACCs and ECES of I-Max binning (15 bins) and scaling methods. All methods use 1k calibration samples, except for Mtx. Scal. and ETS-EnMM which requires the complete calibration set, i..., 25k/5k for ImageNe//CIFARI00 Additional 6 scaling methods can be found in A10
CIFARIOO (WRN)
ImageNet (InceptionResNetv2))
Calibrator
Acc
ew ECEts petor & ECB
AcCrpp1 t AcCopp t ew eW SSCCEels-paee ppior & top1 ECE
4
Baseline
81.35
0.1113
0.0748
80.33
95.10
0.0486
0,0357
Mix Scal. w. La
81.44
0.1085
0.0692
80.78
95.38
0.0508
0.0282
TS
81.35
0.0911
0.0511
80.33
95.10
0.0559
0.0439
GP
81.34
0.1074
0.0358
80.33
95.11
0.0485
0,0186
ETS-MrM
81.35
0.0976
0.0451
80.33
95.10
0.0479
0.0358
I-Max
81.30
0,0518
0,0231
80.20
94.86
0,0302
0,0200
I-Max w.
81.34
0.0510
0.0365
80.20
94.87
0,0354
0.0402
I-Max GP
81.34
0.0559
0.0179
80.20
94.87
0.0300
0.0121
binning achieves the best calibration on CIFAR- 100; on ImageNet, it has the best ECE, and is similar to GP on top1 ECE. For broader scope of comparison, we refer to Sec. A9
To showcase the complementary nature of scaling and binning, we investigate combining binning with GP (a top performing nonpparametric scaling method, though with the drawback of high complexity) and TS (a commonly used scaling method). Here, we propose to bin the raw logits and use the GP/TS scaled logits of the samples per bin for setting the bin representatives, replacing the empirical frequency estimates. As GP is then only needed at the calibration learning phase, complexity is no longer issue. Being mutually beneficial, GP helps improving ACCs and ECEs of binning, i.e., marginal ACC drop 0.16% (0.01% %) on Acc for ImageNet (CIFAR100) and 0. 0.24% on Acc for ImageNet; and large ECE reduction 38. 28% (49.785 78%) in cw ECEcEs-prri and 66. 66.11% (76.07%) in topi ECE of the baseline for ImageNet (CIFAR10) 100).
4.3 SHARED CLASS WISE HELPS SCALING METHODS
Though without quantization loss, some scaling methods, i.e., Beta (Kull et al. 2017). Isotonic regression (Zadrozny & Elkan 2002), and Platt scaling (Plati) 1999). even suffer from more severe accuracy degradation than I-Max binning. As they also use the one-vs-rest strategy for multi-class calibration, we find that the proposed shared CW binning strategy beneficial for reducing their accuracy loss and improving their ECE performance, with only 1 calibration samples, see Tab.
4.4 IMBALANCED MULTI-CLASS SETTING
Lastly, turn experiments an imbalanced multi-class setting. The adopted SVHN dataset has non non-uniform class priors, ranging from 6% (e.g. digit 8) to 19% (e.g. digit 0). We reproduce Tab. 2 for SVHN, yielding Tab. 4 In order to better control the bias caused by the calibration set merging in the imbalanced multi-class setting, the former ane-for-all sCW strategy in the balanced multi-class setting changes to sharing I-Max among classes with similar class priors. Despite the class imbalance,
8

---

Published as a conference paper at ICLR 2021
Table 3: ACCs and ECEs of scaling methods using the one-vs-rest conversion for multi-class
calibration. Here we compare using 1k samples for both CW and one-for-all sCW scaling.
CIFARIO0 (WRN)
ImageNet (IncpptionResNar22)
Calibrator
sCW(?) AcCupl i EC
peior spl ECE
AcCupp
AcCrpp cw top ECE
Baseline
81.35
0.1113
0.0748
80.33
95.10
0.0489
0.0357
Beta
X
81.02
0.1066
0.0638
77.80
86.83
0.1662
0,1586
81.35
0.0942
0.0357
80.33
95.10
0.0625
0.0603
I-Max w. Beta V
81.34
0.0508
0.0161
80.20
94.87
0.0381
0.0574
Isot. Reg. (IR)
X
80.62
0.0989
0.0785
77.82
88.36
0.1640
0,1255
Isot. Reg. (IR)
81.30
0.0602
0.0257
80.22
95.05
0.0345
0.0209
I-Max w. IR
81.34
0.0515
0.0212
80,20
94.87
0.0299
0,0170
Plat Scal.
X
81.31
0.0923
0.1035
80.36
94.91
0.0451
0.0961
Platt Scal.
81.35
0,0816
0,0462
80.33
95.10
0,0565
0.0415
I-Max W. Platt
81.34
0.0511
0.0323
80.20
94.87
0.0293
0.0392
Table 4: ACCs and ECES of I-Max binning (15 bins) and scaling methods. All methods use lk calibration samples, except for Mtx. Scal. and ETS-MnM, which requires the complete calibration set, i.e., 13k for SVHN. Here, we also report the class-wise ECEs using four different thresholds.
Calibrator
AcCwpt
tep ECE |
CwECEo L
cwECE x
+ Cc
CwECE
Bascline
97.08
0.0201
0.0052
0.0353
0.0356
0.0260
Mix. Scal, w. L2
97.09
0.0188
0.0050
0.0346
0,0349
0.0250
ETS-MnM
97.08
0.0152
0.0054
0.0379
0.0382
0.0256
TS
97.08
0.0106
0.0041
0,0323
0.0327
0.0206
GP
97.08
0.0104
0.0043
0,0340
0.0341
0.0212
I-Max
96.88
0.0164
0.0043
0.0244
0.0245
0.0176
I-Max w. TS
97.06
0.0088
0.0025
0.0156
0.0155
0,0112
I-Max w. GP
97.06
0.0074
0.0024
0.0148
0.0147
0.0110
I-Max and its variants perform best compared to the other calibrators, being similar to Tab. This shows that I-Max and the sCW strategy both can generalize to imbalanced multi-class setting.
In Tab. additionally evaluate the class-wise ECE multiple threholds. We ablate various thresholds settings, namely, 1) O (no thresholding): 2) the class prior; 3) 1/K (any class with prediction probability below 1/K will not be the top-1); and 4) relatively large number 0.5 (the case when the confidence on class-k outweighs NOT class-k). We observe that I-Max and its variants are consistently top performing across the different thresholds.
CONCLUSION
We proposed I-Max binning for multi-class calibration, which maximally preserves the label- information under quantization, reducing potential accuracy losses. Using the shared class-wise (sCW) strategy, we also addressed the cample-inefficien issue of binning and scaling methods that rely one-vs-rest (OvR) for multi-class calibration. Our experiments showed that I-Max yields consistent class-wise and top-1 calibration improvements over multiple datasets and model architectures, outperforming HB and state-of-the-art scaling methods. Combining I-Max with scaling methods offers further calibration performance gains, and more importantly, ECE estimates that can converge to the ground truth in the large sample limit.
Future work will investigate extensions of I-Max that jointly calibrate multiple classes, and thereby directly model class correlations. Interestingly, several closely related classes, there is no clear evidence that methods that do model class correlations, e.g. Mtrx. Scal. capture uncertainties better. In fact, I-Max empirically outperforms such methods, although all classes are calibrated independently under the OvR assumption. Non-OvR based methods may fail due to various reasons such as under-parameterizes models (e.g. TS), limited data (c.g. Mirx. Scal.) or complexity constraints (e.g. GP). Joint class calibration therefore strongly relies on new sample efficient evaluation measures that estimate how accurately class correlations are modeled, and which can be included as additional optimization criteria.
datasets such ImageNet which contain

---

Published as a conference paper at ICLR 2021
REFERENCES
David Arthur and Sergei Vassilvitskii. k-means++: the advantages of careful seeding. In Proceedings of the ACM-SIAM symposium Discrete algorithms, pp. 1027-1035, Philadelphia, PA, USA, 2007.
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. 2020.
Charles Blundell, Julien Corneiise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In International Conference on Machine Learning (ICML), pp. 1613-1622, Lille, France, July 2015.
J. Deng, W. Dong, R. Socher, L.-J. Li. K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In Proc. the IEEE/CVF Int. Conf. on Computer Vision and Pattern Recognition (CVPR), 2009.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning (ICML), 2016.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning (ICML), pp. 1321-1330, Sydney, Australia, August 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple method to improve robustness and uncertainty under data shift. In International Conference on Learning Representations (ICLR), 2020.
G. Huang, Liu, L. v. d. Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proc. of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
B. Ji, H. Jung, J. Yoon, K. Kim, and y. Shin. Bin-wise temperature scaling (BTS): Improvement in confidence calibration performance through simple scaling techniques. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 4190-4196, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009,
Meelis Kull, Telmo Silva Filho, and Peter Flach. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 623-631, Fort Lauderdale, FL, USA, April 2017.
Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In Advances in Neural Information Processing Systems (Neur[Ps), pp. 12295-12305. 2019.
Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. In Advances in Neural Information Processing Systems (NeurIPs), pp. 3787-3798, 2019.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In International Conference on Machine Learning (ICML), pp. 2805-2814, 14, July 2018
Andrey Malinin, Bruno Mlodozeniec, and Mark Gales. Ensemble distribution distillation. In International Conference on Learning Representations (ICIR), 2020.
Dimitrios Milios, Raffaello Camoriano, Pietro Michiardi, Lorenzo Rosasco, and Maurizio Filippone. Dirichlet-based gaussian processes for large-scale calibrated classification. In S. Bengio, H. Wal- lach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NeurlPs), pp. 6005-6015, 2018
10

---

Published as a conference paper at ICLR 2021
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. In Proc. of Conference on Artificial Intelligence (AAAI), pp. 2901-2907, 2015.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning, In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Jeremy Nixon, Michael W. Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
Kanil Patel, William Beluch, Dan Zhang, Michael Pfeiffer, and Bin Yang. On-manifold adversarial data augmentation improves uncertainty calibration. arXiv:1912.07455 2019.
Gabriel Pereyra, George Tucker, Jan Chorowski, Eukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. ICLR, 2017.
John Platt. Probabilities for SV machines. In Advances in Large Margin Classifiers, pp. 61-74, 01 1999.
Janis Postels, Francesco Ferroni, Huseyin Coskun, Nassir Navab, and Federico Tombari. Sampling- free epistemic uncertainty estimation using approximated variance propagation. In Proc. of the IEEE International Conference on Computer Vision (ICCV), October 2019.
Serim Ryou, Seong-Gyun Guyng Jeong, and Pietro Perona. Anchor loss: Modulating loss scale based on prediction difficulty. In Proc. of the IEEE International Conference on Computer Vision (ICCV), October 2019.
Christian Szegedy, Sergey loffe. Vincent Vanhoucke, and Alexander A. Alemi. Inception-v4. Enception-ResNe ResNet and the impact of residual connections on learning. In Proc. of the Conference Artificial Intelligence, pp. 4278 4284,, San Francisco, U.S., February 2017.
Sunil Thulasidasan, Gopinath Chennupati, Jeff Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems (NeurlPs), 2019.
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In Proc. of the 37-th Annual Allerton Conference on Communication, Control and Computing, pp- 368-377, 1999. URL WRLLhttss:///atttt http://wwwwwrrraa///////////////
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 3459-3461 Okinawa, Japan, April 2019.
Jonathan Wenger, Hedvig Kjellström, and Rudolph Triebel. Non-parametric calibration for classifica- tion. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
David Widmann, Fredrik Lindsten, and Dave Zachariah. Calibration tests in multi-class classification: A unifying framework. In Advances in Neural Information Processing Systems (NeurIPs), pp. 12257-12267. 2019.
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. Proc. of the IEEE International Conference on Computer Vision (ICCV), 2019.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In International Conference on Machine Learning (ICML), pp. 609-616, 2001.
11

---

Published as a conference paper at ICLR 2021
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probabil- ity estimates. In SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 699, January 2002.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference (BMVC), 2016.
Hongyi Zhang, Moustapha Cissé, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations (ICLR). 2018.
Jize Zhang, Bhavya Kailkhura, and T Han. Mix-n-Match: Ensemble and compositional methods for uncertainty calibration in deep learning. In International Conference Machine Learning (ICML), Vienna, Austria, July 2020.
12

---

Published as a conference paper at ICLR 2021
This document supplements the presentation of Multi-Class Uncertainty Calibration via Mutual Information Maximization--an based Binning in the main paper with the following:
A1: No extra normalization after K class-wise calibrations;
A2: S vs. Sy as empirical approximations to p(>,, yk) for bin optimization in Sec.
3.2.
A3: Mathematical proof for Theorem | and algorithm details of I-Max in Sec. 3.3
A4: Post-hoc analysis on the experiment results in Sec. 4.1
A5: Ablation on the number of bins and calibration set size;
A6: Empirical ECE estimation of scaling methods under multiple evaluation schemes;
A7: Post-hoc Calibration and During-Training Calibration;
A8: Training details;
A9: Extend Tab. 1 in Sec. 4.1 for more datasets and models;
A10: Extend Tab. Sec.
4.2 for more scaling methods, datasets and models.
A1 No EXTRA NORMALIZATION AFTER K CLASS-WISE CALIBRATIONS
There is group of calibration schemes that rely on one-vs-rest conversion to turn multi-class calibration into K class-wise calibrations, e.g., histogram binning (HB), Platt scaling and Isotonic regression. After per-class calibration, the calibrated prediction probabilities of all classes no longer fulfill the constraint, i.e., E Qse + 1. An extra normalization step was taken in Guo et al. (017)) to regain the normalization constraint. Here, we note that this extra normalization unnecessary and partially undoes the per-class calibration effect. For HB, normalization will make its outputs continuous like any other scaling methods, thereby suffering from the same issue ECE evaluation.
One-vs vesrest strategy essentially marginalizes the multi-class predictive distribution over each class. After such marginalization, each class and its prediction probability shall be treated independently. thus no longer being constrained by the multi-class normalization constraint. This is analogous to train CIFAR or ImageNet classifier with sigmoid rather than softmax cross entropy loss, e.g., Ryou et al. (2019). At training and test time, each class prediction probability is individually taken from response without normalization. The class with the largest response is then top-ranked, and normalization itself has no influence on the ranking performance.
the respective sigmoid-response
A2 S vs. S₃ AS EMPIRICAL APPROXIMATIONS TO p(>k, yk) FOR BIN OPTIMIZATION
In Sec. 3.2 of the main paper, we discussed the sample inefficiency issue when there classes with small class priors. Fig. al) shows an example for ImageNet with 1k classes. The class prior for the class-394 about 0.001. Among the 10k calibration samples, we can only collect 10 samples with ground truth is the class-394. Estimating the bin representatives from these 10 samples is highly unreliable, resulting into poor calibration performance.
To tackle this, we proposed to merge the training sets {Sk} across selected set of classes (e.g., with similar class priors, belonging to the same class category or all classes) and use the merged to train single binning scheme for calibrating these classes, i.e., shared class-wise (sCW) instead of CW binning. Fig. bllbb shows that after merging over the 1k ImageNet classes, the set S has sufficient numbers from both the positive Tab. [T showed the benefits sCW over CW binnings. Tab. 3 showed that our proposal sCW is also beneficial to scaling methods which one-vs-rest for multi-class calibration.
1 and negative y = 0 class under the one-vs one-vs-rest conversion.
As pointed out in Sec. 3.2 both S and S5. are empirical approximations to the inaccessible ground truth p(Ag, 9k) for bin optimization. In Fig. A2 empirically analyze their approximation errors. From the CIFARIO 10 test set, we take 5k samples to approximate per-class logit distribution p(Au 1y = 1) by means of histogram density estimatior]] and then use the baseline for comparison, i.e., BS, in
Here, we focus p( p(Aelyye 1) its empirical estimation suffers from small class priors, heing much challenging than p( Wy 0) as illustrated in Fig. A1
A1

---

Published as a conference paper at ICLR 2021
0.20
0.20
p 1)110 samples)
plAly 110k samples)
0.15
plAly 0)9.39k samples)
0.15
PlAly 0X9990 samples)
Density
Density
0.10
plym01=0.999
ply=1=00000
Ply-0)=0.999
p1y-1]=0.001
0.05
0.00
0.00
40
-30
20
-10
0
10
20
30
40
-30
-20
-10
0
20
20
30
Logit (A)
Logit (A)
(a) train set Sk-394 for CW binning.
(b) train. set S for shared-CW binning.
Figure AI: Histogram of ImageNet (InceptioneesNetv2) logits for (a) CW and (b) sCW training. By means of the set merging strategy to handle the two-class imbalance 1 999, S has K=1000 times more class-1 1 samples than Sy with the same 10k calibration samples from C.
[SD
VJSD(B5,
100 samples
500 samples
1000 samples
0.8
0.8
0.8
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
2
3
5 Class
4
6
7
8
5 Class
9
0
Class
2000 samples
3500 samples
5000 samples
0.8
0.8
0.6
0.4
0.4
0.4
0.2
0.2
0.2
2
3
6 Class
4
5
7
8
9
5 Class
9
0 1 2
Class
Figure A2: Empirical approximation error S vs. Sg, where used to measure the difference between the empirical distributions underlying the training sets for class-wise bin optimization. Overall, the merged set is a more sample efficient choice over St-
divergence (JSD) is
Fig. A2 The rest of the 5k samples in the CIFARIO 10 test set are reserved for constructing S₃ and S. For each class, we respectively evaluate the square root of the Jensen-Shannon divergence (JSD) from the baseline BS₁ to the empirical distribution of S or S₃ attained at different numbers of samples]
In general, Fig, A2 confirms that variance (due to not enough samples) outweighs bias (due training set merging). Nevertheless, sCW does not always have smaller JSDs than CW, for instance, the class 7 with the samples larger than 2k (the blue bar "sCW" larger than the orange bar "CW"). So, for the class-7, the bias of merging logits starts outweighing the variance when the number of samples more than 2k. Unfortunately, we don't 't have more samples to further evaluate JSDs, i.e.. making the variance sufficiently small to reveal the bias impact. Another reason that we don't 't observe large ISDs of sCW for CIFARIO is that the logit distributions of the 10 classes are similar. Therefore, the bias of sCW is small, making CIFARIO good use case of sCW. From CIFAR1O to CIFAR100 and ImageNet, there are more classes with even smaller class priors. Therefore, we expect that the sample inefficiency issue of S₁ becomes more critical. It will be beneficial to exploit sCW for bin optimization as well as for other methods based on the one ve-sst conversion for multi-class calibration.
Note, for ISD evaluation, the histogram estimator sets the bin number the maximum of 'sturges' and 'fd' estimators, both of them optimize their hin setting towards the number of samples.
A2

---

Published as a conference paper at ICLR 2021
A3 PROOF OF THEOREM AND ALGORITHM DETAILS OF I-MAX
In this section, we proves Theorem in Sec. 3.3 discuss the connection to the information bottleneck (IB) Tishby al. 1999). analyze the convergence behavior of the iterative method derived in Sec. 3.3 and modify the k-means++ algorithm (Arthur & Vassilvitskii] 2007) for initialization. To assist the implementation of the iterative method, we further provide the pseudo code and perform complexity/memory cost analysis.
A3.1 PROOF OF THEOREM 1
Theorem 1. The mutual information (MI) maximization problem given as follows:
{o } arg max I(y; Q()))
(A1)
Q:
is equivalent to
max I(y: Q())) min
C(fgm pm3)
(A2)
Q: Q:{gm1
(m
where the loss L( (m is defined
Ply w/I Ply =y)) Po(y - y'; 4m)
C(1gms 2m
p())
Σ
dl
(A3)
m== 9m
N'E(0.13
with
Po a [[2y- 1)mml
(A4)
As a set of real-valued auxiliary variables, {dm} are introduced here to ease the optimization.
Proof. Before staring our proof, we note that the upper-case P indicates probability mass functions of discrete random variables, e.g., the label y E {0. and the bin interval index m {1,.... M); whereas the lower-case p reserved for probability density functions of continuous random variables. e.g., the raw logit À E R.
The key to prove the equivalence to show the inequality
( Q(A)) -L(1mm (mlmmmmmm))
(AS)
and the equality is attainable by minimizing C over {0m}. By the definition of MI, we firstly expand [y:; Q(A))
M-1
P(y y'lm)
Iy Q(A)) ((A))
Σ
p(A)
Σ
P(y A'
P(y v/) dd,
(A6)
i=
9m
where the conditional distribution P(y(m) is given as
P(y) 5m
gm
p(ll)dd
gm plylA)P(y)dx
P(ylm) = P(y/> E [gm, Im+ 1))
(A7)
1mm+1 Om
p(A)dX
ggm+) p(A)d)
9m
From the above expression, note that MI maximization effectively only accounts to the bin edges {gm}. The bin representatives can be arbitrary as long as they can indicate the condition 9m+1). So, the bin interval index is sufficient serve the role in conditioning the probability mass function of y, ie., Pmy(m). After optimizing the bin edges, we have the freedom to set the bin representatives for the sake of post-hoc calibration.
A3

---

Published as a conference paper at ICLR 2021
Next, based on the MI expression, compute its sum with C
ply y 3/)m) Po = ys: pm))
(y: QAA)) Clggn
0m
Σ
p())
Σ
Ply = 3/ A)dd log
9m
V'E(0.1)
M
P = y'lm) Po 1t' 1ml)
P(m)
Σ
P(y v/ (m) log
D'E(U.1)
P(m)KKLD (Ply y' Pa 0m))
0.
(A8)
The equality (a) based on
9m++
D//A)a= Ply = 3', x E Im:: gm++1) P(A E [9m 9m+1)) P(y 3/ 3/Im).
9m
P(m)
(A9)
From the equality (a) to (b). it is simply because of identifying the term in [:] of the equality (a) as the Kullback-Leibler divergence (KLD) between two probability mass functions of y. As the probability mass function P(m) and the KLD both are non-negative, we reach to the inequality at (c), where the equality holds if Pa (y:: Φm) = P(ylm). By further noting that L is convex over {Øm) and Pu(y: @m) = P(ylm) nulls out its gradient over {dm) J. we then reach to
(y: Q(A)) + min L((gms 6m = O. {dm
(A10)
The obtained equality then concludes our proof
max Ty;: Q())) = max (gm)
min L( a0 (mm)) min C(fgme 2ml)) (gm:0m)
{sm)) {@m)
min (0m:0m))
C(lgma Fmn)).
(A11)
Lastly, we note that C((gm, )m)) ) can reduce to a NLL loss (as P(y) in the log probability ratio is omittable), which common loss for calibrators. However, only through this equivalence proof and the MI maximization formulation, can we clearly identify the great importance of bin edges in preserving label information. So even though {9m. 2m are jointly optimized in the equivalent problem, only { play the determinant role in maximizing the MI.
A3.2 CONNECTION TO INFORMATION BOTTLENECK (IB)
IB (Tishby et al., 1999) is a generic information theoretic framework for stochastic quantization design. Viewing binning as quantization, IB aims find a balance between two conflicting goals: 1) maximizing the information rate, i.e., the mutual information between the label and the quantized logits [y: (())): and 2) minimizing the compression rate, i.c.. mutual information between the logits and the quantized logits [(); Q(A).. It unifies them by minimizing
1 [( 1A; - = (Q===Q(A) ((s m = Q(A))
min p(m[A) 3
(A 12)
where m is the bin index assigned to and ,9 is the weighting factor (with larger value focusing more on the information rate and smaller value on the compression rate). The compression rate is the bottleneck for maximizing the information rate. Note that IB optimizes the distribution p(m[A). which describes the probability of À being assigned to the bin with the index Since it is not deterministic assignment, IB offers a stochastic rather than deterministic quantizer. Our information maximization formulation is special case of IB, i.e., /3 being infinitely large, as predominantly about how
A4

---

Published as a conference paper at ICLR 2021
well the label can be predicted from compressed representation (quantized logits), in other words, making the compression rate as small as possible is not a request from the problem. For us, the only bottleneck is the number of bins usable for quantization. Furthermore, with 3 30, stochastic quantization degenerating to a deterministic one. If using stochastic binning for calibration, outputs weighted sum of all bin representatives, thereby being continuous and not ECE verifiable. Given that, we do not use it for calibration.
As the IB defines the best trade-off between the information rate and compression rate, use it the upper limit for assessing the optimality of I-Max in Fig. b). By varying 3, IB depicts the maximal achievable information rate for the given compression rate. For binning schemes (Eq. size, Eq. mass and I-Max), we vary the number of bins, and evaluate their achieved information and compression rates. As we can clearly observe from Fig. b)). I-Max can approach the upper limit defined by IB. Note that, the compression rate, though being measured in bits, is different to the number of bins used for the quantizer. As quantization is lossy, the compression rate defines the common information between the logits and quantized logits. The number of bins used for quantization imposes upper limit on the information that can preserved after quantization.
A3.3 CONVERGENCE OF THE ITERATIVE METHOD
For convenience, we recall the update equations for {gm dm in Sec. 3.3 of the main paper here
log
Im:
log
11
log
Vm.
(A13)
1+
Φn log
pm
Um++1
(A)p(A)dA
log Exnesm or a--2n)
Exneoo a(hn)
5mm
or (()))((A)dA)
In the following, we show that the updates on {gm} and { according to A13 decrease the loss L, i.e.
continuously
Cl )'m)) 2 C(1 ('m)) 2 L((ght) 11)
(A14)
The second inequality is based on the explained property of C. Namely, it is convex over { } and the minimum for any given { is attained by Po Po dm) = Pmylm). As om is the bog-robbability ratio of Pa (y: ), we shall have
P(y = 1 m))
0+1 + log
(A15)
P(y 0 m)
where P(y mm) in this case is induced by 1 and P(yl a (2y 1)A). Plugging (
and PlylA) = ((2 - 1) into (A7), the resulting P(y = 1/ (/mm at the iteration + yields the update equation of Φn as given in A13).
To prove the first inequality, we start from showing that [ } is a local minimum of L((gms by H. The update equation {9m) } is outcome of solving the stationary point equation of L (mmr (mm))
over } under the condition p(A = =mm)= > 0 for any m
aL((gm, Hrm 0gm
Pa = --':i'' v'm
p() 9m)
Σ V'E(0,1)
Ply y'lA= gm) log
1 0 Vm
Pa :'=mm-))
(A16)
Being a stationary point is the necessary condition of local extremum when the function's first-order derivative exists at that point, i.e., first-derivaiive test. To further show that the local extremum is actually a local minimum, we resort to the second- of C((gm, 3) is positive definite at the stationary point (g/t1). Due om Φm-1 with the monotonically increasing function sigmoid in its update equation, we have
test, i.c., if the Hessian matrix
0 C(Igm, o'm.))
8 L(lgm. )'m.))
and
0, (A17)
Ogm00mm
82mm
9m -m mm
implying that all eigenvalues of the Hessian matrix are positive (equivalently, is positive definite). Therefore, ( } as the stationary point of L( of }) is local minimum.
A5

---

Published as a conference paper at ICLR 2021
It is important to note that from the stationary point equation A16 { as a local minimum is unique among {gm] with p() = 9m) > O for any m. In other words, the first inequality holds under the condition p(A A gim)) > 0 for any m. Binning is a lossy data processing. In order maximally preserve the label information, it is natural to exploit all bins in the optimization, not wasting any single bin in the area without mass, i.e., p(> gm) O. Having said that, it is reasonable to constrain } f with p(> = 9m.) > O'm over iterations, thereby concluding that the iterative method will converge local minimum based on the two inequalities (A14).
A3.4 INITIALIZATION OF THE ITERATIVE METHOD
We propose to initialize the iterative method by modifying the k-means++ algorithm (Arthur & Vas silvitskii, 2007) that was developed initialize the cluster centers for k-means clustering algorithms. It is based on the following identification
M-
C(1gm. 1m + (3):
P(A)KDL
(A18)
9m
P(A) PPy= y/ P = =': mm)]
2
PPly= min KLD (L - v'l Pa (y = vm)
(A19)
As I (y; X) is constant with respect to (19m 2m }). minimizing is equivalent minimizing the term on the RHS of (AT8). The last approximation is reached by turning the binning problem into a clustering problem, i.e., grouping the logit samples in the training set S according to the KLD measure, where { {óm } are effectively the centers of each cluster. k-means++ algorithm Arthur & Vassilvitskij (007) initializes the cluster centers based on the Euclidean distance. In our case, we alternatively use the JSD) as the distance measure to initialize [6m). Comparing with KLD,, JSD is symmetric and bounded.
A3.5 A REMARK ON THE ITERATIVE METHOD DERIVATION
The closed-form update on {9m} in (A13) is based on the sigmoid-model approximation, which has been validated through our empirical experiments. It is expected to work with properly trained classifiers that not overly overfitting the cross-entropy loss, c.g., using data augmentation and other regularization techniques at training. Nevertheless, even in corner cases that classifiers are poorly trained, the iterative method can still be operated without the sigmoid-model approximation. Namely, as shown in Fig. 2 of the main paper, we can resort to KDE for an empirical estimation of the ground truth distribution p(Aly). Using the KDBs, can compute the gradient of L over {gm} } and perform iterative gradient based update on {9m replacing the closed-form based update. Essentially, the sigmoid-moded approximation is only necessary to find the stationary points of the gradient equations, speeding up the convergence of the method. If attempting to keep the closed-form update on (9m): ). an alternative solution could to use the KDEs for adjusting the sigmoid modeel, e.g., p(yl A) R a [[2y (aa + ab)], where and b are chosen to match the KDE based approximation to p(y]A) After setting and b, they will be used as a scaling and bias term in the original closed-form update equations
log
Im
1 log
I-e 2
b
log
9m-1
Vm.
(A20)
#m
(1a aaa2+tah) (()dd ~ log Esncsm tab)
6m = log
o7 1a-aa--oo abb)p())d)
Exnesm E8m a(-adn-
A3.6 COMPLEXITY AND MEMORY ANALYSIS
To ease the reproducibility of I-Max, we provide the pseudocode in Algorithm. [ Based on it, we further analyze the complexity and memory cost of I-Max at training and test time.
We simplify this complexity analysis as our algorithm runs completely offline and is purely numpy- based. We note that despite the underlying (numpy) operations performed each step of the
A6

---

Published as a conference paper at ICLR 2021
Algorithm 1: I-Max Binning Calibration
Input: Number of bins M, logits {² and binary labels {v
Result: bin edges {gm M (go -c and Par = o0) and bin representations {Φm 10 10
M-1
Initialization: {6m} f Kmeans++ ((An [An]. J. M) (see A3.4):
for iteration 1,
200 do
for m = 1,2,.... M- do
log
1+
9m + log
+t 3m
log
Itc 9:m
end
M 1 - for m = 0, ..,,...,M
do
Sm - {Aa] n [9m: 9m+1): Φm -log Ennesm (An)) An
end end
algorithm differs, we treat multiplication, division, logarithm and exponential functions each counting as the same unit cost and ignore the costs of the logic operations and add/subtract operators. The initialization has complexity of M), for the one-dimensional logits. We exploit the sklearn implementation of Kmeans++ initialization initially used for Kmeans clustering, but replace the MSE with JSD in the distance measure. Following Algorithm [ arrive at the following complexity of O(N M + / * (10 * M + 2 * M)). Our python codes runs Algorithm. [ within seconds for classifiers large ImageNet and performed purely in Numpy. The largest storage and memory consumption is for keeping the N logits used during the I-Max learning phase.
At test time, there is negligible memory and storage constraints, only (2M - 1) floats need to be saved for the M bin representatives { J and M 1 bin edges {gm. ) MM The complexity test time is merely logic operations to compute the bin assignments of each logit and can be done using numpy's 's efficient 'quantize' function. I-Max offers a real-time post-hoc calibrator which adds an almost-zero complexity and memory cost relative to the computations of the original classifier.
We will release our code soon.
A4 POST-HOC ANALYSIS ON THE EXPERIMENT RESULTS IN SEC. 4.1
In Tab. of Sec. we compared three different binning schemes by measuring their ACCs and ECEs. The observation their accuracy performance is aligned with our mutual information maximization viewpoint introduced in Sec. 3.3 and Fig. 2 Here, we re-present Fig. 2 and provide an alternative explanation to strengthen the accuracy, e.g., why Eq. Size binning performed acceptable at the top-1 ACC, but failed at the top-5 ACC. Specifically, Fig. A3 shows the histograms of raw logits that are grouped based on their ranks instead of their labels as in Fig. 2 As expected, the logits with low ranks (i.e., rest below top-5 in Fig. A3 are small and thus take the left hand side of the plot, whereas the top-1 logits are mostly located on the right hand side. Besides sorting logits according to their ranks, we additionally estimate the density of the ground truth (GT) classes associated logits, i.e., GT in Fig, A3}. With properly trained classifier, the histogram of top-1 logits shall largerly overlap with the density curve GT, i.e., 1 prediction being correct in most cases.
understanding
how the location of bin edges affects
From the bin edge location of Eq. Mass binning, it attempts to attain small quantization errors for logits of low ranks rather than top-5. This will certainly degrade the accuracy performance after binning. On contrary, Eq. Size binning aims at small quantization error for the top-1 logits, but ignores top-5 ones. As a result, we observed its poor top-5 ACCs. I-Max binning nicely distributes its bin edges in the area where the GT logits are likely to locate, and the bin width becomes smaller in the area where the top-5 logits are close by (i.c., the overlap region between the red and blue histograms). Note that, any logit larger than must be top-1 ranked, there can exist at most one
A7

---

Published as a conference paper at ICLR 2021
GT
Top 1
Top 5 (except Top 1)
Rest
0.1
Density
0.0
I-Max binning
Equal size Binning
binning
30
-20 -10
0 Logit (A)
10
20
Figure A3: Histogram of CIFARIO0 (WRN) logits in S constructed from 1k calibration samples, using the same setting as Fig. inn the main paper. Instead of categorizing the logits according to their two-class label yi E {0. as in Fig. 2 here we them according their ranks given by the CIFARI00 WRN classifier. As baseline, we also plot the KDE of logits associated to the ground truth classes, i.e., GT.
Table Al: Comparison of sCW binning methods in the case of ImageNet Inception As sCW binning creates ties top predictions, the ACCs initially reported in Tab. I of Sec. 4.1 use the class index as the secondary sorting criterion. Here, we add Acc* upl and Acc* lups which are attained by using the logits the secondary sorting criterion. As the CW ECEs are not affected by this change, here we only report the new top1 ECE*.
Binn.
AcCopt f
Acc upl f
Accrps f
Acc* ups f
top) ECE +
upp ECEE 1
NLL +
Baseline
80.33
95.10
0.0357
0.8406
Eq. Mass
5.02
80.33
26.75
95.10
0.0353
0.7884
3.5272
Eq. Size
80.14
80.21
88.99
95.10
0.0279
0.0277
1.2671
I-Max
80.20
80.33
94.86
95.10
0.0200
0.0202
0.7860
class with prediction probability larger than 0.5. Given that, the bins located above zero are no longer to maintain the ranking order, rather to reduce the precision loss top-1 prediction probability after binning.
The second part of our post-hoc analysis is on the sCW binning strategy. When using the same binning scheme for all per-class calibration, the chance of creating ties in top-k predictions is much higher than CW binning, e.g., more than one class are top-1 ranked according to the calibrated prediction probabilities. Our reported ACCs in the main paper attained simply returning the first found class, i.e., using the class index as the secondary sorting criterion. This is certainly a suboptimal solution. Here, we investigate
how the ties affect ACCs of sCW binning. To this end, we use raw logits (before binning) as the secondary sorting criterion. The resulting ACCC,pl
and ACCr tops are shown in Tab Al Interestingly, such a simple change reduces the accuracy loss of Eq. Mass and I-Max binning zero, indicating that they can preserve the top-5 ranking order of the raw logits but not in strict monotonic sense, i.e., some are replaced by =. As opposed to I-Max binning, Eq. Mass binning has poor performance at calibration, i.e., the really high NLL and ECE. This is because it trivially ranks many classes as top-1, but each of them has very and same small confidence score. Given that, even though the accuracy loss no longer an issue, it is still not good solution for multi-class calibration. For Eq. Size binning, resolving ties only helps restore the baseline top-5 but not top-1 ACC. Its poor bin representative setting due to unreliable empirical too narrow bins can result in permutation among the top-5 predictions.
frequency estimation
Concluding from the above, our post-hoc analysis confirms that I-Max binning outperforms the other two binning schemes mitigating the accuracy loss and multi-class calibration. In particular, there exists a simple solution to close the accuracy gap to the baseline, at the same time still retaining the desirable calibration gains.
A8

---

Published as a conference paper at ICLR 2021
Table A2: Ablation on the number of bins and calibration samples for sCW I-Max binning, where the basic setting is identical to the Tab. 1 in Sec. 4.1 of the main paper.
Binn.
Bins AcCupt 1 Accops cwECE upl ECE i
AcCwpp t Accrps cw EWECEE tpl ECE
Baseline
80.33
95.10
0.0486
0.0357
80.33
95.10
0.0486
0.0357
Ik Calibration Samples
Sk Calibration Samples
GP
80.33
95.11
0.0485
0.0186
80.33
95.11
0.0445
0.0177
80.09
94.59
0.0316
0.0156
80.14
94.59
0.0330
0.0107
15
80.20
94.86
0.0302
0.0200
80.21
94.90
0.0257
0.0107
20
80.10
94.94
0.0266
0.0234
80.25
94.98
0.0220
0.0133
I-Max
30
80.15
94.99
0.0343
0.0266
80.25
95.02
0.0310
0.0150
40
80. 11
95.05
0.0365
0.0289
80.24
95.08
0.0374
0.0171
50
80.21
94.95
0.0411
0.0320
80.23
95.06
0.0378
0.0219
10
80.09
94.59
0.0396
0.0122
80.14
94.59
0.0330
0.0072
80.20
94.87
0,0300
0.0121
80.21
94.88
0,0256
0.0080
20
80.23
94.95
0,0370
0.0133
80.25
95.00
0,0270
0.0091
I-Max W. GP
30
80.26
95.04
0.0383
0.0141
80.27
95.02
0.0389
0.0097
40
80.27
95.11
0.0424
0.0145
80.26
95.08
0.0402
0.0108
50
80.30
95.08
0.0427
0.0153
80.28
95.08
0.0405
0.0114
A5 ABLATION ON THE NUMBER OF BINS AND CALIBRATION SET SIZE
In Tab. of Sec. 4.1, sCW I-Max binning is the top performing measures, In this part, we further investigate on how the number of bins and calibration set size influences its performance. Tab, A2 shows that in order benefit from more bins shall accordingly increase the number of calibration samples. More bins help reduce the quantization loss, but increase the empirical frequency estimation error for setting the bin representatives. Given that, we observe reduced ACCs and increased ECEs for having 50 bins with only 1k calibration samples. By increasing the calibration set size to 5k, then we start seeing the benefits of having more bins to reduce guantization error for better ACCs. Next, we further exploit scaling method, i.e., Wenger iet al. (2020), for improving the sample efficiency of binning at setting the bin representatives, As result, the combination is particularly beneficial improve the ACCs and top-1 ECE. Overall, more bins are beneficial to ACCs, while ECEs favor less number bins.
the ACCs, ECEs and NLL
A6 EMPIRICAL ECE ESTIMATION OF SCALING METHODS UNDER MULTIPLE EVALUATION SCHEMES
As mentioned in the main paper, scaling methods suffer from not being able to provide verifiable ECEs, see Fig. [ Here, we discuss alternatives estimate their ECEs. The current literature can be split into two types of ECE evaluation: histogram density estimation (HDE) and kernel density estimation (KDE).
A6.1 HDE-BASED ECE EVALUATION
HDE bins the prediction probabilities (logits) for density modeling. The binning scheme has different variants, where changing the bin edges can give varying measures of the ECE. Two bin edges schemes have been discussed in the literature (Eq. size and Eq. mass) as well as new scheme was introduced (I-Max). Alternatively, we also evaluate binning scheme which is based on KMeans clustering to determine the bin edges.
A6.2 KDE-BASED ECE EVALUATION
Recent work (Zhang al. 2020) presented an alternative ECE evaluation scheme which exploits KDEs to estimate the distribution of prediction probabilities {9x } from the test set samples. Using the code provided by Zhang et al. (2020), observe that the KDE with the setting in their paper can
A9

---

Published as a conference paper at ICLR 2021
Pawlf)) Piu(q)
PawlAl Pua(A)
25
0.08
Demelty
0.06
0.04
10
Probabilities
(a) CIFARIOOWRRN Top-1 (Prob. Space)
(b) CIFAR1OD/WRK Top-1 (Log Space)
0.6
Psa(g)
PualA)
0.5
0.4
Dendin
0.2
0.1
0.0
10.0
Probabilties
Logit
(c) ImageNee/Incqppionrseeez2 Top-1 (Prob. Space)
(d) ImageNet/hnceptioneeseee2 Top-1 (Log Space)
Figure A4: Distribution of the top-1 predictions and its log-space counterparts, i.e., À log q log(1 10-)) 9).
have a sub-optimal fitting in the probability space. This can be observed from Fig. A4a and Fig. A4c where the fitting is good for ImageNe/lnceptiooneeeeew though when the distribution is significantly skewed to the right (as in the case of CIFAR 100/WRN) the mismatch becomes large. We expect that case of CIFAR 100/WRN is much more common modern DNNs, due to their high capacities and prone to overfitting.
Equivalently, we can learn the distribution in its log space by the bijective transformation, i.e., and (A). As we can observe from Fig. and Fig. A4d the KDE
A = gog log(1 fitting for both models is consistently good.
Zhang et al. (2020) empirically validated their KDE in toy example, where the ground truth ECE can be analytically computed. By analogy, we reproduce the experiment and further compare it with the log-space KDE evaluation. Using the same settings in (Zhang et al., 2020), evaluation error by KDE, i.c., |ECE3
assess the ECE ECExde], in both the log and probability space, achieving prob 0.0020 vs. log 0.0017 for the toy example setting 30 0.5; Bg - 1155. For calibrated setting, 30 = 0.2; 31 = 1.9, we obtain prob 0.0029 vs. log 0.0020. So the log-space KDE-based ECE evaluation (kdeeECE1ogg) has lower estimation error than in the probability space.
less
A6.3 ALTERNATIVE ECE EVALUATION SCHEMES
Concluding from the above, Tab. A3 shows the ECE estimates attained by HDEs (from four different bin setting schemes) and KDE (from (Zhang et al. 2020). but the log space). As we can see, the obtained results evaluation scheme dependent. On contrary, I-Max binning with and without GP are not affected, and more importantly, their ECEs are better than that of scaling methods, regardless of the evaluation scheme.
A10

---

Published as a conference paper at ICLR 2021
Table A3: ECEs of scaling methods under various evaluation schemes for ImageNet InceptionRes-
NetV2. Overall, we consider five evaluation schemes, namely (1) dECE: equal size binning; (2) mÉCE: equal mass binning, (3) kECE: MSE-based KMeans clustering; (4) iECE: I-Max binning; 5) kdeECE: KDE. The HDEs based schemes, i.e., (1)-(4), 10 bins. Note that, the ECEs of
I-Max binning (as a calibrator rather than evaluation scheme) are agnostic to the evaluation scheme. Furthermore, BBQ suffers from severe accuracy degradation.
Calibrator
ACC.pp
cwdECE R
TwmBCE
KECE
4 cwiECE
kddkeeCCE
Mcan
Baseline
80.33
0.0486 1:
0.0014 0.0484
0.0521 0.0004 0.0749
0.0014
0.0540
Calibration
Samples
BBQ
53.89
0.0287 + 0.0009 0.0376 0.0014 0.0372 0.0014 0.0316 + 0.0008 0.442 0.0010
0.0353
Beta
80.47
0.0706 = 0.0003 0.0723 = 0.0005 0.0742 0.0005 0.0755 = 0.0004 0.0828 0.0003
0.0751
Isatonic Reg.
80.08
0.0644 1: 0,0015 0.0646 11: 0.0015 0.0652 1. 0.0016 0.0655 1: 0.0015 0.0704 1: 0.0014
0.0660
Platt
80.48
0.0597 +: 0.0007 0.0593 + 0.0008 0.0613 + 0.0008 0.0634 + 0.0008 0.1372 0.0028
0.0762
Voc reg.
80.53
0.0494 + 0.0002 0.0472 0.0004 0.0498
+ 0.0003 0.0805
0.0560
reg.
1,
0.0004 0,0512
0,0898
0.0590
1k Calibration Samples
TS
80.33
0.0559 + 0.0015 0.0548 0.0018 0.0573 = 0.0017 0.0598 + 0.0015 0.1003 0.005.3
0.0656
GP
80,33
0.0485 + 0.0037 0.0450 + 0.0040 0.0475 0.0039 0.0520 + 0.0038 0.0580 0.0052
0.0502
M.Mux
80.20
0.0302 1 0.0041
I-Max w. GP
80.20
0.0300 = 0.0041
Calibrator
ACC
JECE
upp EE
RECE
IEC
tpp spikdeECE
Mean
Baseline
80,33
0.0035 + 0.0010 0.0345 + 0.0010 0.0348 = 0.0012 0.0352 = 0.0016 0.0480 + 0.0016
0.0376
25k Calibration Samples
BBQ
53.89
0.2689 +: 0.0033 0.2690 + 0.0034 0,2690 = 0.0034 0,2689 + 0.0022 0.2756 1: 0.0145
0.2703
Beta
80.47
0.0346 1: 0.0022 0.0360
0.0017 0.0360 1: 0.0022 0.0357 +: 0.0019 0.0292 0.0023
0.0343
Isotonic Rog.
80.08
0.0468 + 0.0020 0.0434
0.0019 0,0436
0.0020 0.0468 0.0015 0.0437
0.0057
0.0449
+
0.0772
Vec Scal. reg.
80.53
0,0300 1: 0.0010 0.0298 : 0.0012 0.0300 1: 0.0016 0.0303 + 0.0011 0.0365 0.0023
0.0313
Mix Scal. w. L2 reg.
80.78
0.0282 + 0.0014 0.0287 + 0.0011 0.0286 0.0014 0.0289 = 0.0014 0.0324 0.0019
0.0293
1k Calibration Samples
TS
80.33
0.0439 +: 0.0022 0.0452 + 0.0022 0.0454 : 0.0020 0.0443 + 0.0020 0.0679 0.0024
0.0493
GPP
80.33
0.0186 1: 0.0034 0.0182 0.0019 0.0186 0.0026 0.0190 1: 0.0822 0.0164
0.0029
0.0182
I-Max
80.20
0.0200
A11

---

Published as a conference paper at ICLR 2021
Table A4: ECEs post-hoc and (uring-training calibration. WRN CIFARIO0 classifier trained in three modes: 1) no during-training calibration; 2) using entropy regularization (Pereyra et al. 2017); and 3) using Mixup data augmentation (Zhang al,, 2018 Thulasidasan et al., 2019). Taking each of the trained models as one baseline, we further perform post-hoc calibration. Note the best numbers per training mode is marked bold and the underlined scores are the best across the three models.
No Train Calibration
Entr. Reg.
Mixup
Post-Hoc Cal.
ECCEet prior
lop ECE
ECE Peii- + tmp ECE
cw TwECE
pprior
top ECE
+
Baseline
0. 10434
0.06880
0.08860
0.04806
0.10518
0.04972
Mix Scal. L2
0. 10308
0.06560
0.08980
0.04650
0.10374
0.04852
ETS-MaM
0.09488
0.04820
0.09050
0.03900
0.09740
0.03676
TS
0.09436
0.05914
0.09376
0.05438
0.10282
0.04536
GP
0. 1018836
0,03360
0,10520
0.03728
0. 0.10600
0.03514
I-Max
0.04574
0.01834
0.04712
0.02202
0.05534
0,02060
I-Max w. TS
0.05706
0.04342
0.04766
0.04478
0.06156
0.03264
I-Max W. GP
0.06130
0.01428
0,05114
0.01562
0.05992
0.01364
A7 POST-HOC VS. TRRINNGTRAINING CALIBRATION
To calibrate a DNN-based classifier, there exists groups of methods. One is improve the calibration during training, whereas the other is post-hoc calibration. In this paper, we focus on post-hoc calibration because it is simple and does not require re-training of deployed models. In the following, we briefly discuss the advantages and disadvantages of post-hoc and during- trring-trainig calibration.
In general, post-hoc and improve the calibration, as they can be easily combined. Exemplarily, we compare/combine post- hoc calibration methods against/with buring-training regularization which directly modifies the training objective to encourage less confident predictions through an entropy regularization term (Entr. Reg.) (Pereyra et al. 2017). Additionally, adopt Mixup Zhhngg al. 2018) which is data augmentation shown to improve calibration (Thulasidasan et al. 2019). We re-train the CIFAR 100 WRN classifier respectively using Entr. Reg. and Mixup. It can be seen in Tab, A4 that compared to the Baseline model (without training calibration Entr. Reg. or Mixup), EntrReg improves the top1 ECE from 0.06880 0. 0.04806. Further applying post-hoc calibration, I-Max and I-Max GP can reduce the 0.04806 to 0.02202 and 0.01562, respectively. This indicates that their combination beneficial. In this particular case, also observed that without Entr. Reg., directly post-hoc calibrating the Baseline model appears to be more effective, e.g., top ECE of 0.01428 and class-wise ECE 0.04574. Switching to Mixup. the best top ECE 0.01364 is attained by combining Mixup with post-hoc 1-Max w. GP, while I-Max alone without during trainina calibration is still the best at class-wise ECE.
calibration can be viewed two orthogonal ways to
While post-hoc calibrator is simple and effective at calibration, during-training techniques may deliver more than improving calibration, e.g., improving the generalization performance and pro- viding robustness against adversarials. Therefore, instead of choosing either post-hoc during training technique, we recommend the combination. While during-training techniques improve the generalization and robustness of the Baseline classifier, post-hoc calibration can further boost its calibration at a low computational cost.
A8
TRAINING DETAILS
A8.1
PRE-TRAINED CLASSIFICATION NETWORKS
We evaluate post-hoc calibration methods on four benchmark datasets, i.e., ImageNet Deng et al. (2099), CIFAR-100 Krizhevsky (2009). CIFAR- 10 Krizhevsky (2009) and SVHN (Netzer et al.
2011), and across three modern DNNs for each dataset, i.e R Szegeedy et al. (2017).
DenseNet161 Huang et al. (0177) and ResNet152 He et al. (2016) for ImageNet, and Wide ResNet (WRN) Zagoruyko & Komodakis 2016) for the two CIFAR datasets and SVHN. Additionally,
A12

---

Published as a conference paper at ICLR 2021
train DenseNet-BC (L 190, k- 40) Huang al. (2017) and ResNext8x64 Xie et al. (2017) for the two CIFAR datasets.
The ImageNet and CIFAR models are publicly available pre-trained networks and details are reported at the respective websites, i... ImageNet classifiers: https://gittub.c com/ ctcdene/pretrraanedmmoeeee...... pytorch and CIFAR classifiers: https://github. com/ feerppaw/pttorch--laaasfffc
A8.2 TRAINING SCALING METHODS
The hyper-parameters were decided based on the original respective scaling methods publications with some exceptions. We found that the following parameters were the best for all the scaling methods. All scaling methods use the Adam optimizer with batch size 256 for CIFAR and 4096 for ImageNet. The learning rate was set 10-3 for temperature scaling Guo al. (0117) and Platt scaling Platt (1999). 0.0001 for vector scaling Guo et al. (2017) and 10-5 for matrix scaling Guo al. (017). Matrix scaling was further regularized as suggested by Kull al. (2019) with L2 loss on the bias vector and the off- off-diagonal elements of the weighting matrix. BBQ Naeini et al. (2013).
isotonic regression Zadrozny & Elkan (2002) and Beta Kull et al. (0177) hyper-parameters were taken directly from Wenger et al. (2020).
A8.3 TRAINING I-MAX BINNING
The I-Max bin optimization started from k-means++ initialization, which uses JSD instead of Euclidean metric as the distance measure, see Sec. AA344] Then, we iteratively and alternatively updated } and {dm} according to (5) until 200 iterations. With the attained bin edges {gm we set the bin representatives {rm} based on the empirical frequency of class-1. If a scaling method is combined with binning, alternative setting for {rm} } is to take the averaged prediction probabilities based on the scaled logits of the samples per bin, e.g., in Tab. 2 in Sec. 4.2 Note that, for CW binning in 1 the number of samples from the minority class is too few, i.e., 25k/1k 25. We only have about 25/15 = 2 samples per bin, which are too few to use empirical frequency estimates. Alternatively, we set } based on the raw prediction probabilities. For ImageNet and CIFAR 10/100, which have test sets with uniform class priors, the used sCW setting is to share one binning scheme among all classes. Alternatively, for the imbalanced multi-class SVHN setting, we share binning among classes with similar class priors, and thus use the following class (i.e. digit) groupings: {0- 1). (2 4), (5- 9}.
A9 EXTEND TAB. FOR MORE DATASETS AND MODELS.
Tab. in Sec. 4.1 of the main paper is replicated across datasets and models, where the basic setting remains the same. Specifically, three different ImageNet models can be found in Tab, A5 Tab, A6 and Tab. A7 Three models for CIFAR 100 can be found in Tab. A8 Tab. A9 and Tab. A10 Similarly, and
CIFARI0 models can be found in Tab. ATT Tab. A12 Tab.[A12 and Tab. A13 The accuracy degradation of Eq. Mass reduces as the dataset has less number of classes, e.g., CIFARIO. This is result of higher class prior, where the one-vs-rest conversion becomes less critical for CIFARIO 10 than ImageNet. Nevertheless, its accuracy losses are still much larger than the other binning schemes, i.e., Eq. Size and I-Max binning. Therefore, its calibration performance is not considered for comparison. Overall, the observations of Tab. A5 A13 are similar to Tab. [ showing the stable performance gains of I-Max binning across datasets and models.
A10 EXTEND TAB. 2 FOR MORE SCALING METHODS, DATASETS AND MODELS
Tab. in Sec. of the main paper is replicated across datasets and models, and include more scaling methods for comparison. The three binning methods all use the shared CW strategy, therefore 1k calibration samples are sufficient. The basic setting remains the same as Tab, 2 Three different ImageNet models can be found in Tab. A14 Tab. A15 and Tab. A16 Three models for CIFAR 100
A13

---

Published as a conference paper at ICLR 2021
Table A5: Tab.
Extension: ImageNet
InceptionResNNer22
Binn.
sCW(?)
size
AcCmpt
ACCrps
t
cw CwECE_ *
rep ECE L
NLL
Baseline
x
80.33 + 0.15 95.10 + 0.15 0.0486 + 0.0003 0.0357 = 0.0009 0.8406 = 0.0095
Eq. Mass
x
25k
7.78 = 0.15 # 0.0016 + 0.0001 0.0606 = 0.0013 3.5960 = 0.0137
Eq. Mass
1k 5.02 0.13 26.75 + 0.37 0.0022 + 0.0001 0.0353 + 0.0012 3.5272 = 0.0142
Eq. Size
x
25k 78.52 + 0.15 89.06 0.13 0.1344 0.0005 0.0547 + 0.0017 1.5159 + 0.0136
Eq. Size
1k 80.14 + 0.23 88.99 + 0.12 0.1525 + 0.0023 0.0279 + 0.0043 1.2671 + 0.0130
I-Max
25k 80.27 + 0.17 95.01 + 0.19 0.0342 + 0.0006 0.0329 + 0.0010 0.8499 + 0.0105
I-Max
1k 80.20 + 0.18 94.86 + 0.17 0.0302 +: 0.0041 0.0200 + 0.0033 0.7860 + 0.0208
Table A6: Tab.
Extension: ImageNet-
DenseNet
Binn.
sCW(?)
size
AcCupl
AcCups f
cwECE *
tel ECE L
NLL
Baseline
77.21 + 0.12 93.51 + 0.14 0.0502 + 0.0006 0.0571 + 0.0014 0.9418 + 0.0120
Eq. Mass
x
25k 18.48 + 0.19 45.12 + 0.26 0.0017 + 0.0000 0.1657 = 0.0020 2.9437 = 0.0162
Eq. Mass
1k 17.21 + 0.47 45.69 + 1.22 0.0054 + 0.0004 0.1572 + 0.0047 2.9683 + 0.0561
Eq. Size
25k 74.34 0.28 88.27 + 0.11 0.1272 + 0.0011 0.0660 = 0.0018 1.6699 0.0165
Eq. Size
/
Ik 77.06 1 0.28 88.22 1 0.10 0.1519 +: 0.0016 0.0230 + 0,0050 1.3948 1 0.0105
I-Max
x
25k 77.07 = 0.13 93.40 0.17 0.0334 + 0.0004 0.0577 = 0.0008 0.9492 = 0.0130
I-Max
/
Ik 77.13 + 0.14 93.34 1 0.17 0.0263 1 0.0119 0.0201 + 0.0088
0.9229 + 0.0103
Table A7: Tab.
Extension: ImageNet- ResNet152
Binn. sCW(?)
size
AcCup1
AcCrpp f
cw cwECE_
rep ECE 1
NLL
Baseline
x
78.33 + 0.17 94.00 + 0.14 0.0500 + 0.0004 0.0512 = 0.0018 0.8760 + 0,0133
Eq. Mass
x
25k 17.45 + 0.10 44.87 0.37 0.0017 + 0.0000 0.1555 = 0.0010 2.9526 = 0.0168
Eq. Mass
/
Ik 16.25 + 0.54 45.53 0.81 0.0064 + 0.0004 0.1476 = 0.0054 2.9471 = 0.0556
Eq. Size
x
25k 75.50 0.28 88.85
0.19 0.1223 + 0.0008 0.0604 = 0.0017 1.6012 = 0.0252
Eq. Size
1k 78.24 0.16 88.81 + 0.19 0.1480 + 0.0015 0.0286 + 0.0053 1.3308 = 0.0178
T-Max
x
25k 78.24 + 0.16 93.91 + 0.17 0.0334 + 0.0005 0.0521 + 0.0015 0.8842 + 0.0135
I-Max
/
1k 78.19 + 0.21 93.82 + 0.17 0.0295 + 0.0030 0.0196 + 0.0049 0.8638 + 0.0135
Table A8: Tab.
Extension: CIFARIO0 WRN
Binn.
sCW(?)
size
AcCupt
cwF.CF 4 +
tpl FCE 4
NL.I
Baseline
81.35 + 0.13
0.1113 = 0.0010
0.0748 = 0.0018
0.7816 = 0.0076
Eq. Mass
x
5k
60.78 1 0.62
0.0129 1 0,0010
0.4538 = 0.0074
1.1084 1 0.0117
Eq. Mass
/
1k
62.04 + 0.53
0.0252 + 0.0032
0.4744 = 0.0049
1.1789 + 0.0308
Eq. Size
5k
80.39 0.36
0.1143 = 0.0013
0.0783 = 0.0032
1.0772 = 0.0184
Eq. Size
/
1k
81.12 + 0.15
0.1229 + 0.0030
0.0273 = 0.0055
1.0165 + 0.0105
I-Max
5k
81.22 + 0.12
0.0692 = 0.0020
0.0751 = 0.0024
0.7878 = 0.0090
I-Max
/
lk
81.30 1 0.22
0.0518 1 0.0036
0.0231 - 0.0067
0.7593 + 0.0085
Table A9: Tab. 1 Extension: CIFAR100 ResNeXt8x64
Binn.
sCW(?)
size
AcC
CwECE + *
tmp ECE 1
NLL
Baseline
x
81.93 1 0.08
0.0979 1 0.0015 0.0590 = 0.0028
0.7271 + 0.0026
Eq. Mass
5k
63.02 1 0.54
0.0131 = 0.0012
0.4764 = 0.0057
1.0535 = 0.0191
Eq. Mass
/
lk
64.48
+ 0.64
0.0265 = 0,0011
0.4980 = 0,0070
1.1232 = 0.0277
Eq. Size
x
5k
00.8140.2
0.1070 = 0.0008
0.0700 = 0.0030
1.0178 = 0.0066
Eq. Size
/
lk
81.99
0.21
0.1195 = 0.0013
0.0230 = 0.0033
0.9556 = 0.0071
I-Max
x
5k
81.99
0.08
0.0601 = 0.0027
0.0627 = 0.0034
0.7318 = 0.0026
I-Max
1k
81.96
+ 0.14
0.0549 + 0.0081
0.0205 = 0.0074
0.7127 + 0.0040
A14

---

Published as a conference paper at ICLR 2021
Table A10: Tab.
Extension: CIFARIO0 DenseNet
Binn.
sCW(?)
size
AcCrpp
cwECE, +
sop1 ECE +
NLL
Baseline
82.36 1 0.26
0.1223 1 0.0008
0.0762 1 0.0015
0.7542 1 0.0143
Eq. Mass
x
5k
57.23 + 0.50
0.0117 = 0.0011
0.4173 = 0.0051
1.1819 = 0.0228
Eq. Mass
/
lk
58.11 0.21
0.0233 = 0.0005
0.4339 = 0.0024
1.2049 = 0.0405
Eq. Size
5k
81.35
+ 0.23
0.1108 + 0.0017
0.0763 0.0029
1.0207 + 0.0183
Eq. Size
/
lk
0.1192 = 0.0024
0.0219 = 0.0021
0.9482 = 0.0137
T-Max
5k
82.35
0.26
0.0740 1 0.0007
0.0772 = 0.0010
0.7618 = 0.0145
I-Max
/
lk
0.0546 = 0.0122
0.0189 = 0.0071
0.7022 = 0.0124
Table A11: Tab. 1 Extension: CIFARIO WRN
Binn.
sCW(?)
size
ACCrpt f
cwECE *
ECE L
NLL
Baseline
96.12 + 0.14
0.0457 = 0.0011
0.0288 = 0.0007
0.1682 = 0.0062
Eq. Mass
X
5k
91.06 = 0.54
0.0180 = 0.0045
0.0794 = 0.0066
0.2066 = 0.0091
Eq. Mass
/
lk
91.24 + 0.27
0.0212 = 0.0009
0.0836 = 0.0091
0.2252 = 0.0220
Eq. Size
5k
96.04 + 0.14
0.0344 = 0.0008
0.0290 = 0.0013
0.2231
0.0074
Eq. Size
/
1k
96.04 + 0.15
0.0278 + 0.0021
0.0105 = 0.0028
0.2744 + 0.0812
I-Max
5k
96.10 0.14
0.0329 + 0.0011
0.0276 = 0.0007
0.1704 + 0.0067
I-Max
/
Ik
96.06 + 0.13
0.0304 + 0.0012
0.0113 = 0.0039
0.1595 + 0.0604
Table A12: Tab. 1 Extension: CIFARIO- ResNext8x64
Binn.
sCW(?)
size
AcCmp1 f
cw CC
top: ECE +
NLL
Baseline
96.30 + 0.18
0.0485 + 0.0014
0.0201 = 0.0021
0.1247 = 0.0058
Eq. Mass
x
5k
89.40 + 0.55
0.0168 = 0.0037
0.0589 = 0.0052
0.2011 + 0.0085
Eq. Mass
/
lk
89.85 + 0.61
0.0269 = 0.0051
0.0676 = 0.0127
0.2208 = 0.0172
Eq. Size
5k
96.30 1 0.20
0.0274 1 0.0013
0.0174 = 0.0013
0.1613 - 0.0101
Eq. Size
/
1k
96.17 + 0.24
0.0288 = 0.0039
0.0114 = 0.0025
0.2495 = 0.0571
I-Max
5k
96.26
0.20
0.0240 1 0.0020
0.0167 = 0.0014
0.1264 1 0.0066
I-Max
/
Ik
96.22 + 0.21
0.0254 + 0.0030
0.0104 = 0.0025
0.1397 + 0.0276
Table A13: Tab.
Extension Dataset: CIFAR10
DenseNet
Binn.
sCW(?)
size
AcCmpt f
EwE
tapa ECE L
NLL
Baseline
96.65 + 0.09
0.0404 0.001
0.0253 = 0.0009
0.1564 = 0.0075
Eq. Mass
1k
88.80 + 0.47
0.0233 + 0.0024
0.0637 = 0.0023
0.2694 + 0.0274
Eq. Mass
5k
89.51 + 0.36
0.0137 + 0.0039
0.0657 = 0.0041
0.2283 + 0.0101
Eq. Size
1k
96.64 + 0.22
0.0262 + 0.0035
0.0101
0.0035
0.2465 0.0543
Eq. Size
X
5k
96.74 + 0.07
0.0301 + 0.0012
0.0242 = 0.0013
0.1912 + 0.0075
I-Max
1k
96.59 1 0.32
0.0261 1: 0.0025
0.0098 1 0.0027
0.1208 1 0.0044
I-Max
X
5k
96.71
1 0.09
0.0284 1 0.0013
0.0233 1 0.0009
0.1608 1 0.0086
A15

---

Published as a conference paper at ICLR 2021
Table A14: Tab. 2 Extension: ImageNet- Inception
Calibrator
Acc 1
AOCugs 1
cw BCE
ECE L
NLL
Brier
Baseline
80.33 + 0.15
95.10
+ 0.15
0,0486 = 0.0003
0.0357 + 0.0009
0.8406 + 0.0095
0.1115 + 0.0007
25k Calibration Samples
2015
53.89 + 0.30
88.63 + 0.22
0,0287 = 0.0009
0.2689 + 0.0033
1,7104 0.0370
0.3273 0.001
Bota Kunl wr 2017
80.47 + 0.14
94,84 + 0.15
0.0706 = 0.0003
0.0346 = 0.0022
0.9038 + 0.0270
0.1174 + 0.0010
Isotonse Key Zedrozay
Eikkn
80.08 +:
93.46
0.0644
0.0020
1.8375 0.0587
0.1203 0.0012
Plat Platt 1999
80.48 + 0.14
95.18
+ 0.12
0,0897 = 0.0007
0.0775 + 0.0015
0,8083 + 0.0106
0.1205 + 0.0010
Vec Sca. Nille
2019
80.53 - 0.19
95.18
+ 0.16
0.0494 = 0.0002
0,0300 + 0.0010
0.8269
0.009
0.1106 + 0.0007
Mix Scal Kall el
80.78 1 0.18
95.38
1 0.15
0,0508
0.0003
0,0282 1 0,0014
0,8042
0.0100
0.1090 1 0.0006
BWS al
2019)
80.33 4: 0.16
95.10
+ 0.16
0.0561 = 0.0008
0,044 + 0.0019
0.8273
0.0105
0.1129 1: 0.0009
ETS-MiMM Zhang
2020
80.33 - 0.16
95.10
0,0479
0.0004
0,0358
0,0009
0.8426 + 0.0097
0.1115 0,0008
1k Calibration Samples
TS Guo et 2017
80.33 +: 0.16
95.10
+ 0.16
0.0559 = 0.0015
0.0439 + 0.0022
0.8293
0.0107
0.1134 +: 0.0010
GP Wenger
2020
80.33 1: 0.15
95.11
1 0.15
0,0455 1 0.0035
0.0186 1 0,0034
0.7556
1 0.0118
0.1069 0.0007
Eq. Mass
5.02 +: 0.13
26.75
+ 0.37
0.0022 = 0.0001
0.0353 - 0.0012
3,5272
0.012
0.0489 + 0.0012
Size
80.14 0.23
88.99
0,1525
0.0043
1,2671
0.0130
0,1115 0.0011
1-Max
80.20 +: 0.18
94,86 + 0.17
0.0302 = 0.0041
0.02001
+ 0.0033
0.7860 + 0.0208
0.1116 + 0.0008
Eq. Mass w. TS
5.02 0002:000 0.13
26.87 = 0.43
0,003 = 0.0001
0,0357 = 0.0012
3,5454 - 0.022
0.0490 0.0012
Fq.. Mass GP
2.123
26.87 + 0.43
0,002
0.0001
0,0353
+ 0.0012
3,4778 + 0.0217
0.0489 1 0.0012
Eq. Size w. TS
80:26 + 0.18
88.99 + 0.12
0.1470 = 0.0007
0.0391 = 0.0038
1.2721 + 0.0166
0.1136 :: 0.0012
Eq. Size GP
80.26 +
88.99
0,1508
0.0056
1.2661 0.0121
0.1105 0.0008
I-Max w. TS
80.20 1 0.18
94,87
1 0.19
0.0354
+ 0.0124
0.0402
+ 0.0019
0.8339
+ 0.0108
0.1142 1 0.0009
I-Max w. GP
80.20 4 0.18
94.87 1:
0.0300
0.0041
0.0048
0.7787 0.0102
0.1111 0.0006
can be found in Tab
A17
Tab. A18 and Tab. A19 Similarly, CIFARIO models can be found in A22
Tab. A20
Tab. A2I
and Tab.
Being analogous to Tab. 2 we observe that in most cases matrix scaling performs the best at the accuracy, but fail to provide satisfactory calibration performance measured by ECEs, Brier scores and NLLs. Among the scaling methods, GP (Wenger et al.] 2020) is the top performing one. Among the binning schemes, our proposal of I-Max binning outperforms Eq. Mass and Eq. Size at accuracies, ECEs, NLLs and Brier scores. The combination of I-Max binning with GP excels at the ECE performance. Note that, among all methods, Eq. Mass binning suffers from severe accuracy degradation after multi-class calibration. The reason behind Eq. Mass binning was discussed in Sec. 3.3 of the main paper. Given the poor accuracy, it not the scope of calibration performance comparison.
We also observe that GP performs better at NLL/Brier than the I-Max variants. GP is trained by directly optimizing the NLL as its loss. As non non-paramttric Bayesian method, GP has larger model expressive capacity than binning. While achieving better NLL/Brieer, it costs significantly more computational complexity and memory. In contrast, I-Max only relies on logic comparisons at test time. Among the binning schemes, I-Max w. GP achieves the best NLL/Brier across the datasets and models. It is noted that I-Max w. GP remains to be binning scheme. So, the combination does not change the model capacity of I-Max. GP is only exploited during training to improve the optimization on I-Max's 's bin representatives. Besides the low complexity benefit, I-Max w. GP as binning scheme does not suffer from the ECE underestimation issue of scaling methods such as GP.
We further note that cross entropy measure between two distributions, the NLL would be an ideal metric for calibration evaluation. However, empirical NI.L. and Brier favor high accuracy and high confident classifiers, each sample only having one hard label essentially implies the maximum confidence on a single class. For the this reason, during training. the empirical NLL loss will keep pushing the prediction probability to one even after reaching 100% training set accuracy. As result, the trained classifier showed poor calibration performance at test time (Guo et al. 2017). In contrast to NLL/Brier, empirical ECEs use hard labels differently. The ground truth correctness associated to the prediction confidence p is estimated by averaging over the hard labels of the samples receiving the prediction probability p or close to p. Due to averaging, the empirical ground truth correctness is usually not hard label. Lastly, we use a small example to show the difference between the NLL/Brier and ECE: for N predictions, all assigned a confidence of 1.0 and containing M mistakes, the calibrated confidence is M/N < 1. Unlike ECE, the NLL/Brier loss is only non-zero only for the M wrong predictions, despite all N predictions being miscalibrated. This example shows that NLL/Brier penalize miscalibration far less than ECE.
A16

---

Published as a conference paper at ICLR 2021
Table A15: Tab.
Extension: ImageNet- DenseNet
Calibrator
Acciept
AOCups
cw BCE
BCE
NLL
Brier
Baseline
77.21 +: 0.12
93.51
+: 0.14
0.0502 :: 0.0006
0.0571 + 0.0014
0.9418
1: 0.0120
0.1228 J. 0.0009
25k Calibration Samples
BBQ NNeini et
2015
54.69
1 0.42
86.55
1 0.19
0,0274 1 0.0007
0.2819 1. 0.0050
1,9805
0.0500
0.3355
0.0026
Beta Kull el
2017
71.35 + 0.22
93.34
+ 0.17
0.0494 = 0.0008
0.0253 + 0.0022
0.9768 + 0.0254
0.120 1: 0.0010
Key Zadrovy
76.81 1:
0.1281
Plart
Platt
1999
77443 + 0.21
93.64
+ 0.15
0,0448 = 0.0010
0.0906 + 0.0022
0.9168
0.0199
0.1297 + 0.0012
Seal Kull al.
2019
71.44 = 0.20
93.62
+ 0.17
0.492 = 0.0006
0,0516 - 0.0018
0
0.1208 0.0011
Mix Scal Kall
2019
77.56 + 0.11
93.81
+ 0.15
0.04%
0.0006
0.0491
= 0,0015
0.9159
+ 0.0158
0,1202
0.0016
BWS el
2019)
77.21 1: 0.12
93.51
1 0.14
0.0395
+ 0.0007
0,0301
+: 0.0012
0,9106
0.0166
0.1197 0.0008
K1...KMM772 Zhamg
2020
77.21 + 0.12
93.51
1 0.14
0.0357
0.0008
0,0234
0.0011
09188 0.0103
0.1194 0,0006
TX Calibration Samples
TS Guo et 7
71.21 0.12
93,51
+ 0.15
0.0375 = 0.0007
0,0300 = 0.0019
0.9166
0.1197 0.0008
GP Wenger
2020
77.22 0.12
93.51
+ 0.13
0.039 = 0.0057
0.0268 + 0,0035
0.8914 0.0120
0,1188
0.0005
Eq. Mass
17.11 1: 0.47
45.69 1 1.22
0.0054 1 0.0004
0.1572 1 0.0047
2.9683
1. 0.0561
0.1671 1 0.0046
Fq. Size
77.06 0.28
88.22 +
0.1519
0.0016
0.0230
0.0050
1.3948 0.0105
0.1206
0.0013
1-Max
77.13 + 0.14
93.34 + 0.17
0.0263
= 0.0119
0.0201
- 0.0088
0,9229 + 0.0103
0.1201 + 0.0010
Eq. Mass w. IS
17.21 - 0.47
45.73 = 1.07
0,004 = 0.0004
0.1571 = 0.0047
2.9104
0.0-82
0.1671
0.00H5
Eq. Mass GP
17.21 1: 0.47
45.71
+ 1.08
0,0054 - 0.0004
0.1571 + 0,0047
2.9090
0.0485
0,1671 0.0046
Rq. Size w. TS
77.19 1: 0.12
88.22
1: 0.10
0.1464 + 0.0005
0,0241 + 0.0032
1.3928 0.0106
0.1201 1: 0.0008
Size GP
77.19 0.12
88.22
0,1527
0.0007
0.0215
0.0037
1.3944
0.0094
0.1200 0.0005
I-Max w. TS
77.13 + 0.14
93.34
+ 0.17
0.0320 = 0.0026
0.0245 - 0.0024
0.9242 + 0.0117
0.1201 + 0.0007
w.
77.13
93.34
0,0021
0.0124
0.1201
Table A16: Tab. 2 Extension: ImageNet
ResNet152
Calibrator
Acciopt
Accups
cwBCE
ue: BCE
Brier
Bascline
78.33 + 0.17
94.00 +
0.05 + 0.0004
0.0512 = 0.0018
0,8760
0.0133
0
25k Calibration Samples
BBQ (Nccni al
2015
55.04 + 0.26
87.15 + 0.21
0.0278 = 0.0004
0,2840 = 0.0028
1,8490 + 0.0474
0.3361 + 0.0014
Betal
0.0010
0.9365 0.0249
Isotome Reg Zadrozay & Eikan 2002
77.97 1 0.07
92.33 1 0.32
0.05%0
1 0.0016
0.048 1 0.0027
1.9437
0.102
0.1248
1: 0.0015
Plati
78.56 1:
94.06
4
0,0458
0.0021
0,8557
0.0159
Vec Nili
2019
78.61
+ 0.21
94.12 1 0.18
0.099
0.0003
0.0499 + 0.0017
0.8625
0.0143
0,1159 1: 0.0012
Mix S Kall al
2019
78.54
1: 0.23
94.14
1 0.22
0.099 1 0.0004
0.0443 1: 0.0026
0.8583 1: 0.0180
0.1160
0.0016
BWS of
78.33
2 0.18
94.00
0,402
0.0005
0.0019
0.8488
+ 0.0127
0.1
0.0001
ETS-MiN] Bang cl
2020
78.33 4 0.18
94.00
+ 0.15
0.0366 = 0.0007
0.0198 + 0.0006
0.8609 + 0.0117
0.1145 + 0.0011
Calibratioo Samples
TS Guo et (0177)
78.33 0.18
94,00
: 0.15
0.0378 = 0.0007
0.0285 + 0.0023
0.8505 + 0.0126
0.1147 0.0012
GP Wenger al 2020
78.33 - 0.17
94.00
+
0.0405 = 0.0021
0.0202 0.0030
0.8366 0.0118
0.1138 0.0012
Bq, Mass
16.25 + 0.54
45.53
+ 0.81
00064
0.0004
0.1476 = 0.0054
2,9471
0,0556
Lig. Size
78.24 1: 0.16
88.81
1 0.19
0.1480 1 0.0015
0.0286
1. 0.0053
1.3308
1 0.0178
0.1167 1: 0.0011
M-Max
78.19 +: 0.21
93.82
0,0295
0.0030
0.0196
0.0049
0,8638
0.0135
0.1157 1
Eqg, Mass w. TS
16.25 4 0.54
45.54
1 0.71
0,0064
1 0.0004
0.1476 1 0.0054
2.9024
1.024000.00000 0.0401
0.1579
0.0052
Fi. Mass GP
1:
0.0054
2,9021 1: 0.040
0.1579
Eq. Size w. TS
78.27 + 0.17
88.81
+ 0.
0,1428 = 0.0007
0.0225 + 0.0022
1.3286
0.0017
0,1153
0.0013
Eg. Size w. GP
78.27
1 0.17
88.81
1 0.19
0.1475 1 0.0016
0.0138 1 0.0049
1.330 0.0171
0.1150
0.0012
1-Max TS
78.19
1: 0.21
93.82
1 0.17
0,0281
0.0029
0.0219
0.0016
0.8637
0.0125
I-Max w. GP
78.19 + 0.21
93.82
+ 0.17
0.0296 - 0.0029
0.0144 - 0.0050
0.8602 0.0122
0.1150 0.0014
A17

---

Published as a conference paper at ICLR 2021
Table A17: Tab. 2 Extension: CIFARI00 WRN
Calibrator
AcCopt
CcECE
ECE
NLL
Brier
Baseline
81.35 0.13
0.1113
0.0010 0.0748 0.0018 0.7816 0.0076 0.1082 0.0021
5k Calibration Samples
BBQ (Nacini et al.
2015)
80.44 0.19
0.0576 + 0.0018
0.0672 + 0.0044
1.7976 + 0.0443
0.1297 + 0.0019
Beta Kull al.
(2017)
$1.44+0.11
0.0952 + 0.0006
0.0379 + 0.0027
0.7624 + 0.0148
0.1018 + 0.0016
Isotonic Reg. Zadrozny & Elkan 2002
81.25
0.0597 + 0.0029
0.0487 + 0.0040
1.4015 + 0.0748
0.1059 + 0.0013
Platt Platt 1999)
81.35 0.12
0.0827 = 0.0014
0.0585 = 0.0038
0.7491 = 0.0073
0.1026 = 0.0017
Vee Scal. Kull et al.
2019
81.35 0.21
0.1063 + 0.0013
0.0687 + 0.0029
0.7619 0.0064
0.1055 - 0.0017
Mtx Scal. Kull
2019
81.44 :::4::1:000 0.20
0.1085 1 0.0008
0.0692 0.0033
0.7531
0.0078
0.1059 0.0019
BWS et al. (2019)
81.35 0.14
0.1069
1 0.0009
0.0451 + 0.0028
0.737 + 0.0057
0.1037 1 0.0017
ETS-MmMM Zhang al.
2020
81.35 0.14
0.0976 + 0.0019
0.0451 0.0027
0.7695
0.0052
0.1027 0.0020
Ik Calibration Samples
TS Guo al. 2017
81.35 0.14
0.0911 + 0.0036
0.0511 + 0.0059
0.7527 + 0.0074
0.1036 + 0.0025
GP Wenger al.
2020
81.34 + 0.12
0.1074 0.0043
0.0358 + 0.0039
0.6943 + 0.0025
0.0996 + 0.0019
Eq. Mass
6..04
0,4744 + 0.0049
1.1789 + 0.0308
0.4606 = 0.0034
Eq. Size
81.12 0.15
0.1229 + 0.0030
0.0273 = 0.0055
1.0165 = 0.0105
0.1039 = 0.0017
I-Max
81.30 - 0.22
0.0518 1 0.0036
0.0231 +:: 0.0067
0.7593 1 0.0085
0.1016 1 0.0018
Eq. Mass TS
62.04 1.04
0.0253 + 0.0034
0.4764
0.0052
1.0990 = 0.0184
0.4624 0.0037
Eq. Mass w. GP
62.04 +: 0.53
0.0252 1 0.0032
0,4749 + 0.0051
1.1110 + 0.0226
0,4610 + 0.0036
Eq. Size w. TS
81.31 0.15
0.1197 1 0.0029
0.0362 0.0065
1.0106 = 0.0113
0.1038 1 0.0026
Eq. Size w. GP
81.31 = 0.15
0.1205 + 0.0025
0,0189 + 0.0054
1.0161 + 0.0115
0.1032 + 0.0020
I-Max w. TS
81.34 0.20
0.051 + 0.0035
0.0365 + 0.0067
0.7716 + 0.0066
0.1025 + 0.0021
I-Max w. GP
81.34 + 0.20
0.0559 + 0.0089
0.0179 + 0.0046
0.7609 + 0.0080
0.1014 + 0,0014
Table A18: Tab. 2 Extension: CIFARI00
ResNeXt
Calibrator
Accopp
CcECE
tel ECE
NLL
Brier
Baseline
81.93 0.08
0.0979
0.0015
0.0590 0.0028 0.7271 0.0026 0.0984 + 0.0022
5k Calibration Samples
BBQ (Naeini et al. 2015)
81.06 0.30
0.0564 + 0.0013
0.0608 + 0.0058 1.6878 + 0.0546
0.1176 + 0.0022
Beta Kull al.
(017)
82.19 +: 0.31
0.0918 + 0.0020
0.0368 + 0.0047
0.7095 + 0.0074
0.0947 + 0.0024
Isotonic Reg. Zadrozny & Elkan 2002
$1.89
4.002
0.0503 = 0.0036
1.3015 = 0.0656
0.0995 = 0.0018
Platt Platt 1999)
82.28 0.21
0.0790 0.0025
0.0534 + 0.0047
0.7050 = 0.0045
0.0961 = 0.0026
Vee Scal. Kull et al.
2019
82.24 0.27
0.0963 1 0.0013
0.0572 + 0.0037
0.7129 + 0.0053
0.0973 1 0.0021
Mtx Scal. Kull
2019
82.38 11:8::000 0.17
0.0970 0.0014
0.0578 + 0.0040
0.7042 + 0.0046
0.0973 + 0.0023
BWS et al. (2019)
81.93 -: 0.08
0.1045
1 0.0015
0.0448 + 0.0044
0.6897 + 0.0031
0.0969 1 0.0017
ETS-MMM Zhang al. 2020
81.93 0.08
0.0932 0.0020
0.0460 0.001
0.7284
0.0029
0.0963 0.0022
Ik Calibration Samples
TS Guo al. 2017
81.93 0.08
0.0864 0.0036
0.0525 + 0.0057
0.7163 + 0.0037
0.0975 +: 0.0020
GP Wenger al. 2020
81.93 ::: 0.09
0.1025 0.003
0.0345 + 0.0038
0.6456 +: 0.0071
0.0927 + 0.0019
Eq. Mass
64.48 - 0.64
0.0265
0.4980 = 0.0070
1.1232 = 0.0277
0.4770 + 0.0051
Eq. Size
81.99 + 0.21
0.1195 0.0013
0.0230 + 0.0033
0.9556 + 0.0071
0.0974 + 0.0014
1-Max
81.96 = 0.14
0.0549 1 0.0081
0.0205 1 0.0074
0.7127 1 0.0040
0.0959 1 0.0018
Eq. Mass TS
64.48 0.64
0.0262 0.0013
0.5003 0.0066
1.0468
0.0228
0.4793 0.0048
Eq. Mass w. GP
64.48 = 0.64
0,0264 + 0.0012
0.4986 + 0.0066
1.0555 = 0.0227
0,4776 = 0.0048
Eq. Size TS
81.94 0.09
0.1179 + 0.0015
0.0343
0.0029
0.9498 = 0.0058
0.0968 = 0.0022
Eq. Size w. GP
81.94 0.09
0.1177
0.0009
0.0151 + 0.0029
0,9561 + 0.0056
0.0959 1 0.0018
I-Max w. TS
81.96 0.14
0.053
1 0.0073
0.0333
0.0023
0.7286
0.0029
0.0964 1 0.0019
I-Max w. GP
81.96 1 0.14
0,0532
1 0.0077
0.0121 1 0.0026
0.7111 + 0.0024
0.0950 1 0.0017
A18

---

Published as a conference paper at ICLR 2021
Table A19: Tab. Extension Dataset: CIFARI00 DenseNet
Calibrator
AcCopt
CWECE
ECE
NLL
Brier
Baseline
82.36 0.26
0.1223
0.0008 0.0762 - 0.0015 0.7542 0.0143 0.1041
0.0008
5k Calibration Samples
BBQ (Nacini ct al.
2015)
81.56 +: 0.22
0.0567 + 0.0020
0.0635 + 0.0052
1.5876 + 0.0914
0.1216 + 0.0026
Beta Kull al.
(2017)
82.39 + 0.28
0.0953 + 0.0013
0.0364 + 0.0034
0.6935 + 0.0185
0.0966 + 0.0008
Isotonic Reg. Zadrozny & Elkan 2002
82.05 0.26
0.0506 + 0.0025
1.3030 + 0.1107
0.1019 + 0.0014
Platt Platt 1999)
82.34 0.28
0.0866 + 0.0012
0.0491 = 0.0012
0.6835 = 0.0138
0.0969 = 0.0015
Vee Scal. Kull et al.
2019)
8
0.0711 + 0.0015
0.7362 = 0.0173
0,1028 + 0.0015
Mtx Scal. Kull
2019)
82.53 1: 0.19
0.1214 0.0006
0.0733 + 0.0013
0.7360 + 0.0153
0.1025 1 0.0015
BWS et al. (2019)
82.36 1 0.27
0.1028
1 0.0013
0.0445 T 0.0021
0.682 + 0.0125
0.0975 1 0,0008
ETS-MmMM Zhang al.
2020
82.36 0.27
0.1007
0.0016
0.0387 0.0012
0.6986
0.0111
0.0969 0.0008
Ik Calibration Samples
TS Guo al. 2017
82.36 0.27
0.0938 + 0.0017
0.0447 + 0.0023
0.6851 + 0.0115
0.0976 + 0.0008
GP Wenger al.
2020
82.35 0.27
0.1021 + 0.0032
0.0338 + 0.0011
0.6536 + 0.0120
0.0943 + 0.0007
Eq. Mass
58.11
0,4339 + 0.0024
1.2049 = 0.0405
0.4317 + 0.0017
Eq. Size
82.22 = 0.30
0.1192 0.0024
0.0219 = 0.0021
0.9482 = 0.0137
0.0997 = 0.0014
I-Max
82.32 1: 0.22
0.0546 1 0.0122
0.0189 1 0.0071
0,7022 1 0.0124
0.0967 1 0,0019
Eq. Mass TS
58.11 0.21
0.0233
0.0006
0.4347 = 0.0024
1.1483 = 0.0102
0.4324 + 0.0017
Eq. Mass w. GP
58.11 - 0.21
0.0233 + 0.0005
0,4342 + 0.0024
1.1508 + 0.0099
0,4319 + 0.0018
Eq. Size w. TS
82.40 0.24
0.1134 1 0.0014
0.0245 0.0025
0.9427 1 0.0137
0.0986 1 0.0013
Eq. Size w. GP
82.40 -: 0.24
0.1166 + 0.0021
0,0126 + 0.0012
0.9455 + 0.0142
0.0985 + 0.0013
I-Max w. TS
82.36 0.21
0.048 + 0.0090
0.0237 + 0.0009
0.7040 + 0.0104
0.0967 + 0.0010
I-Max w. GP
82.36 0.21
0.0535 + 0.0121
0.0114 + 0.0025
0.6988 + 0.0104
0.0964 + 0.0010
Table A20: Tab. 2 Extension: CIFARIO WRN
Calibrator
Accoot
CcECE
tep ECE
NLL
Brier
Baseline
96.12 0.14
0.0457
0.0011
0.0288
0.0007 0.1682 0.0062 0.0307 + 0.0008
5k Calibration Samples
BBQ (Nacini et al. 2015)
95.98 0.15
0.0290 + 0.0047
0.0198 + 0.0044 0.2054 + 0.0156
0.0314 + 0.0005
Beta Kull et al.
(017)
96.31 + 0.06
0.0504
1:044:00 0.0015
0.0208 + 0.0023
0.1335 + 0.0039
0.0271 + 0.0007
Isotonic Reg. Zadrozny & Elkan 2002
9.120
002411#00002
0.0138 = 0.0017
0.1764 = 0.0241
0.0273 = 0.0005
Platt Platt 1999)
96.24 0.09
0.0489 + 0.0011
0.0177 = 0.0015
0.1359 = 0.0039
0.0270 = 0.0006
Vee Scal. Kull et al.
2019
96.27 1 0.11
0.0449 1 0.0008
0.0229 + 0.0008
0.1437 + 0.0050
0.0286 1 0.0007
Mtx Scal. Kull
2019
96.20 :1 0.10
0.0444 0.0005
0.0277 + 0.0007
0.1625 1 0.0062
0.0302 1 0.0008
BWS et al. (2019)
96.12 - 0.14
0.14
0.0467
1 0.0012
0.0195 1 0.0014
0.1395 + 0.0077
0.0279 1 0.0007
ETS-MMM Zhang al. 2020
96.12 + 0.14
0.0647
0.0014
0.0329 0.0012
0.1478
0.0038
0.0270 0.0006
Ik Calibration Samples
TS Guo al. 2017
96.12 1: 0.14
0.0486 +: 0.0024
0.0205 + 0.0009
0.1385 + 0.0048
0.0278 1: 0.0007
GP Wenger el al. 2020
96.10 - 0.13
0.0549 +: 0.0021
0,0146 +:: 0.0022
0.1281 +: 0.0055
0.0269 + 0.0009
Eq. Mass
91.24 0.27
0.0212
0.0836 = 0.0091
0.2252 = 0.0220
0.0858 = 0.0055
Eq. Size
96.04 1.04=00 0.15
0.0278 + 0.0021
0.0105 + 0.0028
0.2744 + 0.0812
0.0305 + 0.0015
1-Max
96.06 = 0.13
0.0304 1 0.0012
0.0113 1 0.0039
0.1595 + 0.0604
0.0274 1 0.0013
Eq. Mass TS
91.24 0.27
0.0219 + 0.0005
0.0837 0.0092
0.1944 0.0093
0.0853 0.0054
Eq. Mass w. GP
91.24 - 0.27
0,0212 0.0008
0.0821 + 0.0088
0.1918 = 0.0091
0,0851 + 0.0054
Eq. Size w. TS
96.13 0.12
0.0286 + 0.0018
0.0125 0.0024
0.1940 = 0.0063
0.0296 = 0.0009
Eq. Size w. GP
96.13 1 0.11
0.0266 + 0.0016
0.0066 + 0.0028
0.1917 + 0.0058
0.0292 + 0.0009
I-Max w. TS
96.14 1 0.13
0.0293
1 0.0010
0.0163
0.0012
0.1417 T 0.0047
0.0280 1 0.0008
I-Max w. GP
96.14 1 0.13
0.0276
1 0.0011
0.0074 1 0.0035
0.1331 T 0.0042
0.0268 1 0.0008
A19

---

Published as a conference paper at ICLR 2021
Table A21: Tab. 2 Extension: CIFARIO ResNeXt
Calibrator
AcCopt
CwECE
lep ECE
NLL
Brier
Baseline
96.30 0.18
0.0485
0.0014
0.0201 0.0021
0.1247 0.0058
0.0266 0.0013
5k Calibration Samples
BBQ (Nacini ct al.
2015)
96.18 + 0.12
0.0256 + 0.0027
0.0166 + 0.0020
0.1951 + 0.0134
0.0286 + 0.0004
Beta Kull al.
(2017)
96.31 + 0.22
0.0517 + 0.0011
0.0148 + 0.0016
0.1163 + 0.0040
0.0256 + 0.0011
Isotonic Reg. Zadrozny & Elkan 2002
96.35 =.20
0.0241 + 0.0016
0.0129 + 0.0008
0.1686 + 0.0099
0.0264 + 0.0011
Platt Platt 1999)
96.34
0.0511 + 0.0008
0.0143 = 0.0017
0.1159 = 0.0042
0.0256 = 0.0011
Vee Scal. Kull et al.
20199
96.37
= 0.0495 0.0017
0.0161 + 0.0017
0.1189 = 0.0053
0.0258 + 0.0013
Mtx Scal. Kull
2019
96.34
0.0492 + 0.0020
0.0187 + 0.0020
0.1225 0.0060
0.0263 0.0014
BWS et al. (2019)
96.3
0.0514 0.0013
0.015 1 0.0008
0.1199 1 0.0048
0.0257 1 0,0012
ETS-MmMM Zhang al.
2020
96.3 + 0.19
0.0547 0.0013
0.0159 0.0027
0.1193 0.0043
0.0257 0.0011
Ik Calibration Samples
TS Guo al. 2017
96.30 0.19
0.0524 0.0028
0.0150 + 0.0009
0.1182 + 0.0051
0.0257 + 0.0012
GP Wenger al.
2020
96.31 + 0.17
0.0529 + 0.0017
0.0125 + 0.0021
0.1176 + 0.0051
0.0258 + 0.0011
Eq. Mass
89.85 = 0.61
0.0269 0.0051
0,0676 + 0.0127
0.2208 = 0.0172
0.0841 - 0.0042
Eq. Size
96.17 = 0.24
0.0288 + 0.0039
0.0114 = 0.0025
0.2495 = 0.0571
0.0277 = 0.0008
I-Max
96.22 0.21
0.0254 1 0.0030
0.0104 1 0.0025
0.1397 1 0.0276
0.0265 1 0.0012
Eq. Mass TS
89.85 0.61
0.0269 + 0.0054
0.0676 0.0128
0.1966 = 0.0104
0.0844 = 0.0043
Eq. Mass w. GP
89.85 - 0.61
0.0266 1 0.0049
0.0669 + 0.0126
0.1962 + 0.0106
0,0841 + 0.0043
Eq. Size w. TS
96.29 0.18
0.0270 0.0022
0.0062 0.0024
0.1574 1 0.0091
0.0264 1 0.0013
Eq. Size w. GP
96.29 - 0.18
0,0271
+ 0.0020
0.0063 + 0.0030
0.1576 + 0.0093
0.0264 - 0.0012
I-Max w. TS
96.28 0.19
0.0224 + 0.0016
0.0053 + 0.0024
0.1208 + 0.0058
0.0259 + 0.0012
I-Max w. GP
96.28 0.19
0.0223 + 0.0018
0.0052 + 0.0029
0.1 + 0.0061
0.0259 + 0.0012
Table A22: Tab. Extension: CIFARIO
DenseNet
Calibrator
Accopt
CcECB 1 1
tep lpt ECE
NLL
Brier
Baseline
96.65 0.09
0.0404
0.0010 0.0253 0.0009 0.1564 0.0075 0.0259 + 0.0007
5k Calibration Samples
BBQ (Nacini et al. 2015)
96.75 0.19
0.0245 + 0.0030
0.0170 + 0.0022
0.1806 + 0.0105
0.0279 + 0.0010
Beta Kull et al.
(017))
96.81 + 0.10
0.0468 + 0.0003
0.0154 + 0.0013
0.1151 + 0.0042
0.0234 + 0.0007
Isotonic Reg. Zadrozny & Elkan 2002
96.84 = 0.08
0.0236+
0.0140 = 0.0024
0.1501 = 0.0137
0.0241 = 0.0007
Platt Platt 1999)
96.82 = 0.11
0.0459 0.0007
0.0141 = 0.0010
0.1154 + 0.0040
0.0233 - 0.0007
Vee Scal. Kull et al.
2019
96.84 0.14
0.0413 1
0.0223 + 0.0010
0,1373 + 0.0077
0.0249 1 0,0007
Mtx Scal. Kull
2019
96.73 - 0.09
0.0402
0.0017
0.0245 1 0.0008
0.1531 + 0.0081
0.0257 1 0.0007
BWS et al. (2019)
96.65 0.10
0.0423
1 0.0010
0.0188 1 0.0016
0.1239 + 0.0065
0.0239 1 0.0006
ETS-MMM Zhang al. 2020
96.65 0.10
0.0527
0.0012
0.0212 0.0012
0.1196 0.0038
0.0230 0.0007
Ik Calibration Samples
TS Guo al. 2017
96.65 0.10
0.0425 1: 0.0005
0.0169 + 0.0010
0.1186 + 0.0051
0.0237 1: 0.0006
GP Wenger el al. 2020
96.66 - 0.09
0.0135 + 0.0025
0.1143 +: 0.0048
0.0228 1 0.0007
Eq. Mass
88.80 - 0.47
0.0024
0.0637 = 0.0023
0.2694 = 0.0274
0.0881 = 0.0033
Eq. Size
96.64 + 0.22
0.0262 0.0035
0.0101 + 0.0035
0.2465 + 0.0543
0.0256 + 0.0003
1-Max
96.59 + 0.32
0.0261 + 0.0025
0.0098 1 0.0027
0.1208 1 0.0044
0.0239 1 0.0005
Eq. Mass TS
88.80 0.47
0.0234 + 0.0026
0.0626 0.0023
0.2102 0.0051
0.0877 0.0030
Eq. Mass w. GP
88,80 = 0.47
0.0233 + 0.0026
0,0634 + 0.0025
0.2098 = 0.0053
0,0880 + 0.0030
Eq. Size w. TS
96.75 0.10
0.0133
0.0014
0.1657 = 0.0056
0.0249 + 0.0007
Eq. Size w. GP
96.77 1: 0.10
0.0242
0.0022
0.0050 + 0.0012
0.1612 + 0.0048
0.0245 1 0.0005
I-Max w. TS
96.81 0.15
0.0229
1 0.0016
0.0125
0.0017
0.1224
0.0056
0.0239 1 0.0007
I-Max w. GP
96.81 1 0.15
0,0218
0.0012
0.0048 1 0.0009
0.1173 1 0.0054
0,0231 + 0.0005
A20


Reviews:
[{"review_id": "RFkv0Svqg_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: the authors went out of their way to address my concerns about the absence of the unbalanced class setting: they added a new datasets (SVHN), new results (table 4) and updated some of their explanations. All these additions seem satisfactory. I was also pleased with the feedback about computational cost (R3). I improved my rating. \nWhile I agree with the concerns of reviewer 4 (those I could understand), they would apply to every publication I have read about calibration, and I think the authors addressed these concerns to the best of our current knowledge.\n\nThis paper proposes an information maximization binning scheme for calibration. Starting with a good introduction, a clear progression leads to the core algorithm described by theorem 1. Limits of previous histogram-based approaches, both in terms of performance or reliability of metric, are clearly demonstrated with clear figures and proper references.\nWhile using information measures to drive histogram binning has been done, I assume that the current classification setting where one maximizes the MI between the logit and the class is novel (the authors do not give pointer to previous work here, only mentioning the Info Bottleneck without references).\n\nTheorem 1 leads to an alternative minimization algorithm with analytical steps. I did not check the convergence behavior proof but Figure 3 is convincing enough. I did not fully understand the information bottleneck limit.\n\nExperiments show first that the information binning strategy is far superior than equal-mass or size binning. Table 2 and 3 then shows how it improves on most scaling algorithms used for calibration. One detail I am not comfortable with: the ECE_{1/K} hack, as it looks like a last-minute addition to give even stronger gains to the I-MAX method. A more principled introduction would be better (see below).\n\nThis would be an excellent paper except for the following, which casts doubts whether all the steps of the method generalize to an unbalanced multiclass setting. It is probably possible to fix or explain before publication.\nThis paper relies on a very unnatural and unfortunate state-of-affair in ML: classes are equally distributed on the test data. The phrasing does even consider any other possibility, and some of the algorithms seem to be quite specific to this setup, requiring significant changes in the “real world” case where test classes are not equally distributed.\n\nAt the end of section 3.2, the authors propose an algorithm to merge {S_k} across K classes based on the observation that they have similar distribution. Rather than a proof, they run a simulation on ImageNet (Sec A.2) that shows it is better than binning each S_k separately. While the experiment is elegant, it probably strongly relies on the fact that each S_k has the same 1:K-1 split. What would happen if the classes follow a more realistic Zipf law, as observed in real NLP classification tasks? I would assume that the merging process could still be applicable, but applied to groups of {S_k} with similar class-0/class-1 distributions.\n\nIn section 4.1, the trick to remove from the measure of the ECE classes where the predicted probability is less than 1/K also depends on a uniform 1/K prior. It should also be adapted to a non-uniform prior.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NGrOCRPb-r9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update after the rebuttal: The authors have answered my concerns. I believe the paper should be accepted and would be a nice contribution to the current research.\n\n===============================================================\n\n\n\nThe paper proposes a novel approach for post-hoc calibration of outputs of the neural networks to estimate uncertainty of its prediction. The paper considers the histogram binning approach (in contrast to scaling approaches existing in the literature) and utilises the information theory in building bins.\n\nStrong points:\n* The work is very well placed in the context of the existing literature identifying the current gaps\n* Theoretically sound motivation of the approach\n* Extensive empirical evaluation\n\nWeak points:\n* Some discussion of the cost of the proposed method is lacking - i.e. how much in terms of computational time and memory this new calibration method is? \n* The methods are compared with respect to accuracy and Expected Calibration Error (ECE) only. It has been shown that ECE is not a good metric for comparing different methods (see, e.g. Ashukha, A., Lyzhov, A., Molchanov, D. and Vetrov, D., 2020. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. ICLR 2020). \n\nI am recommending acceptance of the paper, though addressing the weak points above would largely improve the paper. The reasons for this decision is that strong points outweigh weak points: the proposed idea is interesting, it is shown that it is promising in practice (subject to not very good metrics) and the paper is mostly well written and easy to follow.\n\nQuestions to authors: \nCould you please address raised weak points?\n\nAdditional feedback (not necessarily important for evaluation, but could help to improve the paper):\n1. The part on shared class-wise binning is rather rushed in the main paper and it is not very clear. It is also rather independent contribution from the main I-Max calibration contribution. It would be better to somehow put them under one umbrella\n2. Section 2. “Bayesian DNNs, e.g. (Blundell et al., 2015) and their approximations (Gal\n& Ghahramani, 2016)” – a very arguable statement, I would suggest rephrasing it. Since Blundell et al. proposed variational inference which is also an approximation, and Gal & Ghahramani work is not an approximation of Blundell et al.’ model\n3. Section 4.2. “Namely, matrix scaling w. L_2” dot after w is read as a full stop which is confusing\n4. After eq.(5). \"So, we can solve the problem by iteratively and alternately updating ${g_m}$ and ${\\phi_m}$ based on (A12).\" - it seems eq. 5 and A12 are the same and it would be more convenient to refer to eq. 5 in the text right after it.\n5. I am a little bit missing the overall procedure of the proposed calibration. I.e. all details are there (especially if refer to appendix), but after reading the main paper, there is no feeling that I can now go and implement it for my problem. Maybe a pseudocode can help, or just step-by-step guidance\n\nMinor:\nSection 3, first paragraph: “Sec. 3.2)” – redundant bracket\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "N8mXXF7dF0J", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper highlights the issues with the scaling method and histogram binning i.e., underestimate calibration error in scaling methods and failing to preserve classification accuracy, and sample-inefficiency in HB. They use the I-Max concept for binning, which maximizes the mutual information between labels and quantized logits. They claim that their approach mitigates potential loss in ranking performance and allows simultaneous improvement of ranking and calibration performance by disentangling the optimization of bin edges and representatives. They also propose a shared class-wise (sCW) strategy that fits a single calibrator on the merged training sets of all K class-wise problems to improve the  sample efficiency.\n\nThe paper is well written and the authors provide enough motivation and intuition of why maximizing the mutual information between labels and quantized logits would help multi-class calibration. There are some concerns and issues that I think needs to be addressed. \n\n1- One approach in estimating uncertainty in classification is to choose a model and a regularized loss function to inherently learn a good representation. For example using confidence as a term for regularization in neural networks is proposed in Regularizing neural networks by penalizing confident output distributions (ICLR 2017) that penalizes low-entropy output distributions. I think it is worth comparing the results with such existing work and discussing the advantages and disadvantages since a similar concept has been used one while training the model and this paper as a post-hoc calibration. \n\n2- It is interesting that a single calibrator on the merged training sets of all K class-wise problems (sCW) performs well. As it is mentioned in the paper, it introduces bias due to having samples drawn from the other classes. In HB, increasing its number of evaluation bins reduces the bias, but in (sCW) such bias can not be controlled. Moreover, Figure A2 shows it achieves smaller JSDs which is not expected. Is there any reason for that? What would happen if the number of bins is increased?\n\n\n3- Based on the experimental results, it seems I-Max performs better than other binning approaches. However, compared to the scaling methods it seems GP (Wenger et al. 2020) performs better at NLL/Brier than the I-Max variants. \n\n4- Even though the paper shows combining I-Max with GP improves the ECE, it is not clear how the issues of each approach will be handled. For example, the ECE might be underestimated.  \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to maximizing the mutual information to optimize the bin for multiclass calibration. The idea, technique, and presentation are good. The paper solves some multiclass calibration  issues. The author should revise the paper according the reviewer's comments before publish.
Decision: Accept (Poster)