Mitigating bias in calibration error estimation
Authors: Rebecca Roelofs
Nicholas Cain
Jonathon Shlens
Michael Curtis Mozer
Venue:  (2021)
Keywords: calibration error
uncertainty estimation
statistical bias

Abstract: Building reliable machine learning systems requires that we correctly understand their level of confidence. Calibration focuses on measuring the degree of accuracy in a model's confidence and most research in calibration focuses on techniques to improve an empirical estimate of calibration error, $\mathrm{ECE}_\mathrm{BIN}$. Using simulation, we show that $\mathrm{ECE}_\mathrm{BIN}$ can systematically underestimate or overestimate the true calibration error depending on the nature of model miscalibration, the size of the evaluation data set, and the number of bins. Critically, $\mathrm{ECE}_\mathrm{BIN}$ is more strongly biased for perfectly calibrated models. We propose a simple alternative calibration error metric, $\mathrm{ECE}_\mathrm{SWEEP}$, in which the number of bins is chosen to be as large as possible while preserving monotonicity in the calibration function. Evaluating our measure on distributions fit to neural network confidence scores on CIFAR-10, CIFAR-100, and ImageNet, we show that $\mathrm{ECE}_\mathrm{SWEEP}$ produces a less biased estimator of calibration error and therefore should be used by any researcher wishing to evaluate the calibration of models trained on similar datasets.

Under review as a conference paper at ICLR 2021
MITIGATING BIAS IN CALIBRATION ERROR ESTIMATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Building reliable machine learning systems requires that we correctly understand their level of confidence. Calibration focuses measuring the degree of accuracy in a model's confidence and most research in calibration focuses on techniques improve empirical estimate of calibration error, ECEaIN Using simulation, we show that ECEaIN can systematically underestimate or overestimate the true calibration error depending the nature of model miscalibration, the size of the evaluation data set, and the number of bins. Critically, ECEsI is more strongly biased for perfectly calibrated models. We propose simple alternative calibration eπror metric. ECEswLER. in which the number of bins is chosen to be as large as possible while preserving monotonicity in the calibration function. Evaluating our measure on distributions fit to neural network confidence scores on CIFAR- 10, CIFAR-100, and ImageNet, we show that ECEsweee produces a less biased estimator of calibration eтror and therefore should be used by any rescarcher wishing to evaluate the calibration of models trained on similar datasets.
Sample B
ECE,N Distribution
INTRODUCTION
Raw data
Machine learning models are increasingly de-
Empinical accuraje
0.88
True callbratio on
ECE Bins
0.15
ployed in high-staken settings like self-driving cars
Mean Sample
(Caesar et al. 2020) Geiger et al. [0033] [Snn al.) 2020) and medical diagnoses (Esteva et al. 22117 2019 Gulshan al. 2016) where a model's ability to recognize when it is likely to be incorrect is crit- ical. Unfortunately, such models often fail in unex- pected and poorly understood ways, hindering our ability to interpret and trust such systems (Azulay & Weiss 2018 Biggio & Roli 2018 Hendrycks & Dietterich 2019 Recht et al. 2019 Szegedy et al., 2013). To address these issues, calibration is used to ensure that a machine learning model produces confidence scores that reflect the model's ground truth likelihood of being correct (Platt et al. 1999) Zadrozny & Elkan 2001 2002).
0.4
8.10
0.0
0.0 0.2 0.4 0.6 088 1.0 Predicted confidence
0.00
10
15
20
ECEaN (9)
Figure 1: Bias in ECEEIN for perfectly cali- brated models. Simulated data from perfectly calibrated model with confidence scores mod- eled to ResNet- 10 CIFAR-10 10 output (He et al., 2016 Kängsepp 2019). We show a reliability
diagram for sample of size 200 and the distribution of ECE scores computed across 10 independent simulations. Even though the model is perfectly calibrated, ECEsI systemati- cally predicts large calibration errors.
To obtain an estimate of the calibration error, or ECE the standard procedure (Guo et al. 2017 Naeini et al. 2015) partitions the model confidence scores into bins and compares the model's predicted accuracy to its empirical accuracy within each bin. We refer to this specific metric as ECE Although recent work has pointed out that ECE.N is sensitive to implementation hyperparameters (Kumar et al.) 2019 Nixon et al. 2019), measuring the statistical bias in ECEgIN, or the difference between the expected ECEpIn and the true calibration error (TCE), has remained largely unaddressed.
In this paper, we address this problem by developing techniques to measure bias in existing calibration metrics. We use simulation to create a setting where the TCE can be computed analytically and thus the bias can be estimated directly. As Tigure[Thightee highlights we find empirically that ECF, has non-negligible statistical bias and systematically predicts large errors for perfectly calibrated models.
Nacini et al. (2015) introduce ECE as an acronym for Expected Calibration Error. However, ECE is not proper expectation whereas the true calibration ervor is computed under expectation. To resolve this confusion, we prefer to read ECE Estimated Calibration Error.

---

Under review as a conference paper at ICLR 2021
ECEsweer ECEgiv
ECEsweer
resnetl 10 10
resnetl 10 cc100
ECEpıv
resmetl10 SD c10
EGEweer ECEarv
resnetl10 SD c100
ECEswee
ECEaw
resnet wide32 c10
ECEawee
resnet wide32 c100
ECEaweer
ECEgrv
ECEsiN
ЕCЕμиит
densenet40 c10
densemet40 _1000
ECEai
ECEaI
ECE:WEEP
ECESHEEP
1es imgnet
densenet16] imgnet
ECEgiv
ECEaM
Figure 2: Bias affects which recalibration algorithm preferred. For ten models, we report which recalibration method is determined to be superior based either ECEBIN ECE5wEER. The wide bar indicates the superior method using entire validation set (mean of X instances); narrow bars each random sample of 10% of the original validation set. Recalibration methods tested histogram binning, temperature scaling, and isotonic regression.
Motivated by monotonicity in true calibration curves arising from trained models, we develop simple alternate for measuring calibration errоr, the monotonic sweep calibration error (ECEswene) which chooses the largest number possible while maintaining monotonicity the approximation to the calibration curve. Our results suggest that ECEyweeP is less biased than the standard ECEsIN and can thus more reliably estimate calibration error.
Does the use of improved ECE measure affect which recalibration method is preferred? In Figure we examine this question using 10 pre-trained models, and compare the standard ECE measure, ECEpIN with 15 squual-witt-sppacee and evaluation we find that ECE,I produces a different selection of the preferred recalibration method on 30% of the models. (We use histogram binning (Zadrozny & Elkan 2001). temperature scaling (Guo et al. 2017). and isotonic regression (Zadrozny & Elkan 2002) as the recalibration techniques.) When reduce the size of the validation and evaluation by 10% and recalibrate with these smaller sets, ECERN produces a different selection on 22% of the cases ( with 10 bins, we see disagreement 27% of the cases). Thus, the of improved ECE measure has significant implications not only for estimation of calibration error but for improving calibration with methods like temperature scaling.
bins, to ECEswrоP- With large dataset sizes for recalibration
BACKGROUND
fix) Beta(e.8, 0.05)
=
Perfect calibration
Consider binary classification setup with imput X X. target output Y = {0, 1}, and we have model f: (0, 1] whose output represents confidence score that the true label Y is 1.
True calibration error (TCE). We define true cal- ibration егror the difference between model's predicted confidence and the true likelihood being correct under the Ep norm:
r00
Figure 3: Curves controlling true calibra- tion error. Our ability measure calibration is contingent on both the confidence score distribution (c.g., f(X) Beta(2.8, 0.05)) and the true calibration curve (e.g., Ey [ f(X) c
TCE(f) = (Ex lIf(x) - Ey (XI((I))))" (1)
The TCE is dictated by two independent features of a model: (1) the distribution of confidence scores
f(x) F over which the outer expectation is computed, and (2) the calibration curve Ey f(X)], which governs the relationship between the confidence score f(x) and the empirical accuracy (see Tigure3for illustration).
In our experiments, we measure calibration error using the & norm because it increases the sensitivity of the eπrоr metric to extremely poorly calibrated predictions, which tend be more harmful in applications. In addition, the mean squared prediction error of the classifier, or Brier score (Brier 1950), can be decomposed into terms corresponding to the squared & calibration error and the variance of the model's correctness likelihood (Kuleshov & Liang 2015 Kumar et al. 2019).
We standard validation sets of size 5, 000 examples for CIPAR-10/100 and 25, 000 examples for ImageNet and evaluation sets of size 10, 000 for CIFAR-10/100 and 25, 000 for ImageNet.
2

---

Under review as a conference paper at ICLR 2021
2.1 ESTIMATES OF CALIBRATION ERROR
To estimate the TCE of a model f. assume we are given dataset containing n samples, {r;, yi J= We can approximate TCE by replacing the outer expectation in Equation by the sample average and replacing the inner expectation with an average over a set of instances with similar f(z) values:
ECE (f) En f(z:) म EjeN, 8o 1)(")²
(2)
where N; is instance i's set of neighbors in model confidence output space.
Label-binned calibration error (ECE,,)) The label-binnee calibration error uses binning define
N and estimate the model's is empirical accuracy EE [X]] Specifically, the instances are partitioned into b bins, where B& denotes the set of all instances in bin ks allowing us to express Equation 2 terms of the binned neighborhood:
in
Bj 3/. Ef = (+ Ei EieB, If(x;) - Us P): where Øk- = Be T
1
Binning is commonly implemented using either equal width binning (Guo al. 2017 7 Nacini et al.)
2015), which creates bins by dividing the model confidence domain [0, 1] into equal sized intervals, or
equal mass hinning (Nixon et al.] 2019), which creates bins by dividing the n samples into partitions with equal number of instances.
Binned calibration error (ECEnrx). contrast to ECE, which operates on the original instances but uses binning to estimate empirical accuracy, ECE.I collapses all instances in bin into single instance and compares the per-bin empirical accuracy to the per-bin confidence score, weighted by the per-bin number of instances. Given b bins, where Bş is the set of instances in bin k and letting S and 5k be the per-bin average confidence score and label, ECEAN is defined under the lp norm as
ECE,IN = (E k=
[&
f The 3²
(4)
Importantly, ECERIN always underestimates ECE,R, ECE (s) 2 ECE ( which follows by applying Jensen's inequality on each inner term (1,2,...,0
.b) b}
Eqs.
3 and
B& EieBy (X,) Yul 2 EieB, fx Y
(5)
3 MEASURING BIAS THROUGH SIMULATION
We focus on bias rather than variance because the variance can be estimated from finite set of samples through resampling techniques whereas the bias is unknown quantity that reflects systematic error. We also found empirically that the variance seems relatively insensitive to the estimation technique and number of bins (see Appendix B). The bias of calibration error estimator, ECE for some estimation algorithm A, is the difference between the estimator's 's expected value with respect to the data distribution and the TCE:
B = E|ECEA| TCE.
(6)
If we assume a particular confidence score distribution F and true calibration curve T(X) [Y f(X) (see Figure for examples), we can compute the TCE by numerically evaluating the integral implicit in the outer expected value of Equation
We then compute sample estimate of the bias as follows. First, we generate n samples (f(x;), yı ) such that f(xx)) ~ F and [Y] f(x)= cf == T(c), and compute the ECE on the sample. We repeat this process for m simulated datasets and compute the sample estimate of bias as the difference between the average ECE and the TCE
(n) m Σ ECEA TCE.
Bias in ECEBIN for varying hyperparameters. Using simulation, we next investigate the bias in ECE,N as function of the number of samples n and the number of bins. We compute ECE&N with equal width binning and we assume parametric curves for f(x) and Ey [Y f(X) c| that are fit to the ResNet-110 CIFAR-10 model output (see Section 5.1 for de- tails on how we compute fits). Kumar et al. (2019) Proposition 3.3 shows that any binned version of calibration error systematically underestimates TCE in the limit of infinite data.
In the remainder of the text, will use the term "bias" to refer to this sample estimate of bias.
3

---

Under review as a conference paper at ICLR 2021
Bias in Equal Width ECERIN
4.34
4.52
4.65
4.72 -4.78
..52
However, for finite number of samples Figure showws that ECE can either overestimate or un- derestimate TCE and that increasing the number of bins does not always lead to better estimates TCF. Intuitively, each example lies in its own bin, and the prediction error computed with respect Moreover, regardless of binning scheme, there ex- ists a bin number for each sample size that results in the lowest estimation bias and this optimal bin count grows with the sample size. Appendix B we show the variance associated with this ex- periment, and also include results for alternative calibration metries.
-3.71
4.00
-4.4
Bins
6
-1.443
2214
2.69
3.04
-3.26
-3.40
the number of bins explodes,
-0.37
-1.12
-2.01
-2.24
target of 0 or 1.
32
2.66
1.50
0.52
-0.26
-0.83
-1.2
3.32
0.30
-0.30
200
400
800
1600
3200
6400
# Samples
Figure 4: ECE with equal width binning can overestimate TCE and the optimal number of bins depends on number of samples.
4 MONOTONIC CALIBRATION METRICS
Though Section 3 shows that there exists an optimal number of bins for which ECE,N has the lowest bias, unfortunately, this number depends on the binning technique, the number of samples, the confidence score distribution, and the true calibration curve. This observation motivates us to seek method for adaptively choosing the number of bins.
Monotonicity in the true calibration curve implies that model's expected accuracy should always increase as the model's confidence increases. Although such a requirement seems reasonable for most any statistical model, it is not obvious how prove why or when "reasonable" model would attain such property. We offer a rationale for why should be expected of machine learning models trained with a maximum likelihood objective, e-g., cross-entropy or logistic loss (Murphy 2012). Namely, from ROC (receiver operating characteristic) analysis of maximum likelihood models, an apprecaeeee observation of ROC curves is that a model trained to maximize the likelihood ratio must have a convex ROC curve in the limit of infinite data (Green et al. (1966), Section 2.3). The slope of the ROC curve is related to the calibration curve, and convex ROC curve implies monotonically increasing calibration curve (the converse is also true) (Chen et al. 2018 Gneiting & Vogel 2018).
In practice, several potential confounds may lead to measuring a non-monontonic calibration curve. First, finite data size effects may lead to fluctuations the true positive or false positive rates, but do not reflect the behavior of the underlying model. Second, deviations in the domain statistics between cross-validated splits in the data may lead to unbounded behavior; however, we assume that such domain shifts are negligible as cross-validane validated splits are presumed to be selected i.i.d. Given that deviations from non-monotonic calibration curves are considered artificial, we posit that any method that is trying to assess the TCE of underlying model may freely assume monotonicity in the true calibration curve. Note that this proposition already guides the entire field of re-calibration to require that re-calibration methods only consider monotonic functions (Platt al. 1999 Wu al. 2012
Zadrozny & Elkan 2002).
Monotonic sweep calibration error (ECF Accordingly, we leverage the underlying mono- tonicity in the true calibration and propose the monotonic sweep calibration error, a metric that choose the largest number of bins possible such that it and all smaller bin sizes preserve monotonicity in the bin heights Vk.
BCE = max;
Bal
The 1²)² s.t. 01
82
S
Do.
Ag' =b)
We can compute the monotonic sweep calibration error by starting with b bins (since b is guaranteed to be monotonic binning) and gradually increasing the number of bins until we either
'Note that a third potential reason for a non mon-monotonic calibration curve is that classitier could be trained with non likelihood- based statistical criteria, moment matching. However, the lack of monotonic behavior in the calibration curve of such model may actually be sign that the model is not reasonable or admissible model for consideration given task (Chen al. 2018 Pesce al. 2010
4

---

Under review as a conference paper at ICLR 2021
(a)
1
(b)
(ECEam) (%, simulated)
25 (c)
resnet110_cm
resnet110 _0__c10
ELY]f(x) =
20
resnet wwdc32_cccccc
densenet40_c10
15
PDF
resnet110 c100
10
10_500_
resnet wit___icc2__cc0 c1000
5
densenatHO c100
resnet152_ jiignet
densenett61_mmg Jmgnet
(X)
1
(X) =
5
15 25
ECEain (%)
Figure 5: Maximum likelihood fits to empirical datasets illustrate large skew in their density distribution and calibration function. For each dataset, (a) confidence distributions were with two-parameter beta distribution and (b) calibration curves were fit via generalized linear models across multiple model families, with the best model selected via the Akaike information criterion (details in Appendix A Across models, the dataset source systematically affects both curves. (c) We plot the overall quality of the fits by computing the ECEsIN on the original data vs. the ECEBıN averaged over 1000 simulated trials. Curves well-fit fit the data should lie close the identity line.
reach a non-monotonic binning, in which case we return the last b that corresponded to a monotonic binning, until every sample belongs to its own bin (b n)).
Algorithm 1: Monotonic Sweep Calibration Error
for b- to do Compute ECERIN bin heights (jts) with & bins if Binning is not monotonic then b= b=b-1;
break;
return ECEDIN computed with b bins
5 RESULTS
5.1
PARAMETRIC FITS CAPTURE CALIBRATION CURVE AND SCORE DISTRIBUTION
TCE is analytically computable when the true calibration curve. To what extent can parametric forms capture the diversity and complexity of real world data? In many applications, only sample-based approximations to these functions available. In order to estimate ECE,I bias in real-world data, we develop parametric models of empirical logit datasets that enable direct measurement of TCE.
parametric forms for the confidence distribution and
We consider 10 logit datasets (including those studied in Guo et al. (017 ), arising from training four different neural model families (ResNet, ResNet-SD, Wide-ResNer and DenseNet) three different image datasets (CIFAR-10/1)) and ImageNet) (Deng et al., |2009 He et al. 2016 Huang
al. 2016 2017 Krizhevsky 2009 LeCun al. 1998 Zagoruyko & Komodakis 2016). For each
dataset (Figure 5). we compute confidence scores by applying soltmax and top-1 selection to logits
from Kängsepp (2019).
Computing TCE directly for real-world data via Equation 1 is infeasible because of the expectation across X. Instead, we model the distribution of the scores f(X) directly with two-parameter beta distribution, which we fit using maximum likelihood estimation (note that in many cases, the confidence scores are heavily skewed). Calibration functions are computed by fitting multiple (binary) generalized linear models (GLM) to the calibration data. From these candidate models, select the model of best fit using the Akaike Information Criteria (AIC). The models considered include logit, log, and "logflip" (log(1 .r)) link and transformation functions, up first order in the transformed domain, resulting in monotonic calibration functions. See Appendix A for additional details.
We find that the parametric forms for the calibration curve and distribution of scores well captured by these simple GLM and Beta models. Figure 5]abb) shows the resulting fits, with parameters summarized in Appendix A We observe significant skew in the score distribution which, as discussed
5

---

Under review as a conference paper at ICLR 2021
CIFAR-10
CIFAR- 100
ImageNet
-
-6-
Blas
Blas
Blas
-12
14
8
-6
20
-8
200
800
3,200
12,800
200
800
3,200
12,800
Num samples (log scale)
Num samples (log scale)
Num samples (log scale)
EM
EWsweep
EMdebias
ResNet SD
DenseNet
KDE
ResNet
Wide ResNet
Bias=0
EMsweep
Figure 6: EMsweep is less biased than alternative calibration metries. We plot bias versus number of samples n. for calibration metrics on simulated data drawn from the CIFAR-10, CIFAR- 100, and ImageNet fits (Section 5.1). The dataset the model was trained has greater influence bias than the model architecture. Metrics based on equal mass binning consistently outperform equal width binning. Exploiting monotonicity the EMsweep metric helps the most small sample sizes.
in Section 5.2 poses challenge measuring calibration error with equal-width bins. We find that the dataset has more influence on the fits than the neural model, with ImageNet models the least skewed and CIFAR-I0 the most (correlating with model accuracy). Figure 4 45ccc indicates that ECE scores computes simulated data from the fits closely match ECE,IN scores computed on the real data, as witnessed by the fact that the points lie near the line of equality.
5.2 SIMULATION RESULTS ON DISTRIBUTIONS FIT TO 10FAR-10/100, AND IMAGENET
Using the resulting parametric fits from Section 5.1 we evaluate ECEBI and ECESW using both equal mass binning and equal width binning and compare these values to the analytically computed TCE. In addition, we include comparison to the recently proposed debiased estimator, ECEDEBIAS, using equal mass binning (Kumar et al., 2019) and smoothed Kernel Density Estimation (KDE) method for estimating calibration error (Zhang al. 2020). We abbreviate each method as follows:
BCE using equal width binning and 15 bins (Guo al.
2017 Naeini et al. 2015
EM: ECE,N using equal mass binning and 15 bins (Nixon et al.
2019
EMdebias: Debiased calibration metric using equal mass binning (ummrr et al. 2019
XDE: KDE calibration metric (Zhang et al. 2020)
Naweepp ECEsweeR using equal width binning, and
EMsweep: ECESwFER using equal mass binning.
We choose 15 bins for ECEsIN and ECEOErIAS. following the standard set by Guo et al. (2017). For comparison different choices of fixed number of bins, Appendix B includes an analysis the bias and variance of ECE,IN. ECESwEEP- and ECEpERIAS across different bin numbers and sample sizes for the curves corresponding to CIFAR- 10 ResNet- 10, CIFAR- 100 Wide ResNet-32 and ImageNet ResNet-152 models. We that the optimal number of bins varies with the number of samples, and using a different fixed number bins introduces bias at varying sample sizes.
Figure 6 plots the bias (estimated using best-fit curves for each neural network model trained on the CIFAR- 10, CIFAR-100, 100, and ImageNet datasets. An unbiased estimator would have Bias 0, which highlight visually in green. We
1,000 simulations) versus the sample size for the

---

Under review as a conference paper at ICLR 2021
find that the dataset the model was trained on has more influence the calibration metric behavior than the model architecture, which may be unsurprising given that (. that the dataset heavily influences both the distribution of confidence scores and the true calibration curve.
Equal width versus equal mass binning: Overall, metrics that employ equal mass binning show less bias than those with equal width binning. Surprisingly, for EW and EWsweep CIFAR-10, well as EW on CIFAR-100 100, we see increasing absolute bias as the number samples increases across all model architectures tested. We propose possible explanation for this phenomenon. As Figure (a) shows, models trained on CIFAR-10 10 and CIFAR- 1-0 have highly skewed confidence distributions and, as result, equal width binning places the majority of the instances in the top bin. As we increase the number of samples, we increase the likelihood that we generate sample that populates one of the lower bins, which, due their low sample density, may have poorer average estimate of the TCF.
ECEsweer versus alternative metrics: Our experiments show that ECEswEeP with equal mass binning has either similar or less bias than alternative calibration error metrics, and at low sample sizes, the WCEP method is consistently less biased than all other metrics. However, the ECEswEEP does not show improvements over other metrics when combined with equal width binning, and we do not recommend using the combination in practice.
KDE estimator Compared to all calibration metrics we evaluate, the KDE estimator has much higher bias across the CIFAR-10, 10, CIFAR-100 100, and ImageNet simulations. Our results suggest that the heuristic used to choose the kernel bandwidth and the specific 'triweight" kernel worked well for the synthetic example evaluated in (Zhang et al. 2020), but fails generalize to the more realistic synthetic examples we study. Specifically, Zhang et al. (2020) assumes a Gaussian distribution for P(X and logistic confidence score distribution, which result in notably different qualitative shapes than the logit distributions we obtain from models trained on CIFAR-10/100 or ImageNet (see
Figure 5 a, b) or the reliability diagrams and confidence score distributions from Kängsepp (2019).
Debiased estimator. The debiased estimator (Kumar et al. 2019) uses a jackknife technique to estimate the per-bin bias in the standard ECEBIN, and subtracts off this bias to achieve a better binned estimate of the calibration error. However, unlike ECESWEEP. the debiased estimator still has hyperparameter b that controls the number of bins. On the CIFAR-10/10N and ImageNet simulations in Figure [6] the debiased estimator with 15 bins is more competitive to equal mass ECEswEEr than any other estimation method we test, but the equal mass ECE&wEEP method still outperforms the debiased estimator for low sample sizes. In Appendix B we also show that the equal mass debiased estimator has higher variance than the equal mass ECESWEEP (except when b I 4, when all estimators have high bias).
5.3
BIAS VERSUS TRUE CALIBRATION ERROR
EW
EMsweep
20
20
18
18
Estimated Callbration Error (6)
Estimated Callbration Emrr (86)
16
16
12
10
20
n-200. Betal_tt 0.1)
n-200, Bk)-8eta11.1. 0.1)
n=5.000, (ixi-Belall.1. 0.11 (ix)--nafoomss
5,000 Ex1-Bwall1. 0.11 n=200. Unf-omfo
n=50000 foo- Unform0000 y-x 12% emrer
n=5.000, Eni- y-x
2
12% unor
0
18
20
16 18
True Calibration (%)
True Calibration Error (%)
Figure 7: Bias in calibration estimation increases as TCE decreases. We plot average ECE (%) for FW (left) and EMsweep (right) versus the TCE (%), with varying sample size and score distributions. The estimator bias is systematically worse for better calibrated models, and the effect is more egregious with fewer samples. At n 200 samples, depending on the score distribution, an EW estimate of 12% could either correspond to 5% or 8% TCE. The EMsweep metric is able to mitigate the bias and ambiguity in calibration error estimation to a certain extent.
7

---

Under review as a conference paper at ICLR 2021
We next evaluate the estimation bias of the baseline EW and our best estimator from the previous section, EMaweep, as we systematically vary the TCE. We are interested in the low calibration error regime because goal of many recalibration algorithms is reduce the calibration error of the model to 0%. Figure 7 shows the average estimated calibration error for EW and EMsweep versus the TCE. The average calibration error is computed across
1,000 simulated datasets, and we include results for two sample sizes, n 200 and π = 5,000, and two score distributions, f(z)- ~ (niform(O, 1) and f() Beta(1.1) 0.1). ) the beta distribution fit to the CIFAR-I00 Wide ResNet_32 32. To control the TCE, we assume Ey LY f(X) = c] cd and vary d E [1, 10). When 1 the true calibration curve is Ey [Y f(X) c| = c, which means the model's predicted confidence score is exactly equal to its empirical accuracy and thus the TCE is 0%. As we increase d, we move the true calibration curve farther away from the perfect calibration curve, which increases
the TCE of the model.
The bias the calibration error estimation can seen visually as the difference between the ECE and the TCE. Perfect estimation (0 bias) corresponds to the y = line. Bias is highest when the model is perfectly calibrated (TCE is 0%) and generally decreases as TCE increases. Using larger sample size of = 5, 000 reduces the bias, but when the model is perfectly calibrated, the ECEBIN can still be off by 2%. The EMsweep metric significantly reduces this bias.
In practice, we do not know the distribution of scores F, the true calibration curve Ey [Y or the TCE. So, given a measurement of calibration error, how much bias can we expect? we measure an ECEBI of 20%, expect it to be fairly close the TCE. However, if see ECEgI of 2%, it is possible that the model may actually be perfectly calibrated! We further explore this dilemma in the next section.
f(X) c).
5.4 How WELL CAN WE DETECT MISCALIBRATION?
n=400
Consider the situation where we have a model whose TCE unknown and wish to test the hypothesis that the model is miscalibrated, i.e., TCE > O. Our ability to de- tect miscalibration depends the TCE, the sample size (n), and the method for estimating calibration error. We conduct a simulation with fz) Beta 1, 1) and true cal- ibration curve from the family Ey Y f(X) c cd.
n=800
n=1600
Type il error (miss) rate
n-3200
n-6400
n=12800
0.6
EMsweap
where d is varied to obtain range of TCE. Allowing for
type I eгror rate of .05 (also known as the false-alarm rate, or the rate of mistakenly claiming miscalibration when model is perfectly calibrated), we obtain type II error rates (also known as the miss rate, or the rate of failing to detect miscalibration). Figure 8 shows the type II error rate as a function of TCE and n for EMsweep and EW. Our results indicate that EMsweep obtains a significantly lower failure rate than EW, particularly for under 10,000 samples. More generally, we note limitations with both methods: to detect a miscalibration of 2%, over 10,000 samples needed; and if one has under 500 samples, the miscalibration must be greater than 10% to be detected reliably.
0.2
10
TCE (%)
Figure 8: Probability of failing to de- teet miscalibration (type II error, or miss rate), plotted as a function of TCE for various sample sizes (n), with type I error rate fixed at 0.05. EMsweep (dashed lines) obtains significantly lower failure rate than EW (solid lines).
DISCUSSION
Much research in model calibration has focused on recalibrating models, i.e., transforming (1x to f'(x) (Platt et al. 1999 Zadrozny & Elkan 2001 2002). We focus on estimating calibration error, because without a good estimate of TCE, there is little point in trying to recalibrate models. What implications do our results have choosing and evaluating recalibration algorithms?
One possibility that bias affects all recalibration methods in the same way, which would imply that we should still be able to select recalibration method A over method B with biased estimator. However, our results show that the distribution of confidence scores significantly impacts the bias in calibration error estimation, even when the sample size is fixed. Since recalibration methods are inherently designed to modify the confidence score distribution, we cannot assume that bias will affect all methods in the same way.
8

---

Under review as a conference paper at ICLR 2021
Another possibility is that the bias is small compared to the calibration error differences would measure. However, our results suggest that this is not true. Even when the number of evaluation samples is high, = 5,000, Figure 7 shows that it is entirely possible that we might measure an ECHBIN of 2% when the model is in fact perfectly calibrated. Moreover, Figure 2 shows that the preference of recalibration algorithm can change depending on whether ECEgweer or ECFaI is used to measure calibration error, implying that bias might meaningfully affect the conclusions of previous studies of calibration error such as those in Guo et al. (2017).
Several authors attempt a different approach to recalibration: improving model calibration during training. For instance, Mukhoti et al. (2020) trains a model with batch size of 128 across multiple types of losses including maximum mean calibration error (Kumar et al., 2018) and Brier loss (Brier) 1950) which explicitly tries to minimize a calibration loss using 128 examples at a time. However, our results suggest that training a model with naive estimates of calibration error as an objective using a batch size < (1000) a potentially flawed endeavor, particularly because the distribution of scores from the model is changing throughout training, and any potential measure of calibration may be more affected by the distribution of scores (as opposed to the calibration curve).
RELATED WORK
Sensitivity of ECE,I to hyperparameters. Several works have pointed out that ECEBIN is sensitive to implementation details. Kumar et al. (2019) show that ECE increases with number of bins while Nixon et al. (2019) find that ECEaIN scores are sensitive to several hyperparameters, including lp norm, number of bins, and binning technique. In addition, Nixon et al. (2019) find that ECEBI with equal mass binning produces more stable rankings of recalibration algorithms, which is consistent with our conclusion that equal mass ECEAN is less biased estimator of TCE. However, in contrast to prior work, we study the sensitivity of the bias in ECE to implementation hyperparameters.
Metries for calibration error estimation. Gupta et al. (2020) propose a calibration error metric inspired by the Satinoooroo-SSmimoo (KS) statistical test that estimates the maximum difference between the cumulative probability distributions P (((()) and P(Y f(X)). The KS is similar to the maximum calibration error (MCE) (Naeini et al. 2015) in that it computes a worst-case deviation between confidence and accuracy, but the KS is computed on the CDF, while the MCE uses binning and is computed on the PDF. In contrast, our work focuses between confidence and accuracy. As mentioned in Guo et (2017). both the worst case and average difference are useful measures but may be applicable under different cireumstances.
measuring the average difference
CONCLUSIONS AND FUTURE WORK
If we are to rely on the predictions from machine learning models in high stakes situations like autonomous vehicles, content moderation, and medicine, we must be able to detect when these predictions are likely to be incorrect. Given that the default confidence scores produced by machine learning models do not necessarily correspond to the model's 's empirical accuracy, recalibration is necessary in order to produce reliable and consistent output. However, it is impossible for a recalibration algorithm to achieve perfect calibration if we cannot measure calibration accurately. Our results show that the statistical bias in current calibration error estimators grows as we approach perfect calibration, but this bias can be mitigated by using monotonic estimation techniques. We conclude with some directions for future work:
Simulation as an evaluation tool. We have shown that simulation a powerful technique for evalu- ating calibration error estimators. However, especially for small sample sizes, the ECEswEEP method does not completely eliminate estimation bias, and we only evaluated finite set of distributions arising from image classification datasets and models. We hope future work can use simulation as an effective tool for developing both new calibration metrics and new recalibration techniques, and that evaluations can be extended to more diverse set of datasets and models.
Distribution shifts. Since models deployed in real world application will necessarily make pre- dictions on out-of distribution examples and since we would like these predictions to be calibrated (Ovadia et al. 2019), it is important that we are able to accurately measure and improve calibration error on non-ird data. However, in such situations, we do not necessarily expect the calibration curve to be monotonic. Thus, exploring how distribution shifts can modify the shape of the true calibration curve is an important future direction.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? arXiv preprint 1722:::800..1177,.0
Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition, 2018. hhttps://arke wwtp:///ggg///////////a// 03141
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
Holger Caesar, Varun Bankiti, Alex Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 11621-11631, 2020.
Weijie Chen, Berkman Sahiner, Frank Samuelson, Aria Pezeshk, and Nicholas Petrick. Calibration of medical diagnostic classifier scores to the probability of disease. Statistical methods in medical research, (115):139411444 2018.
J. Deng, W. Dong, R. Socher, I.-J. Li, K. Li, and I.. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPRO9, 2009.
Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks. nature. 542(7639):115-118 2017.
Andre Esteva, Alexandre Robicquet, Bharath Ramsundar, Volodymyr Kuleshov, Mark DePristo, Katherine Chou, Claire Cui, Greg Corrado, Sebastian Thrun, and Jeff Dean. A guide to deep learning in healthcare. Nature medicine, 25(1):24-2) 29, 2019.
dataset. The International Journal of Robotics Research, 32(1 1):11111223 2013. Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti
Andrew Gelman, John B. Carlin, Hal S. Stern, and Donald B. Rubin. Bayesian Data Analysis. Chapman and Hall/CRC, 2nd ed. edition, 2004.
Tilmann Gneiting and Peter Vogel. Receiver operating characteristic (roc) curves. arXiv preprint arXiv:1809.04808, 2018
David Marvin Green, John A Swets, et al. Signal detection theory and psychophysics, volume 1. Wiley New York, 1966.
Varun Gulshan, Lily Peng, Mare Coram, Martin C Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus 2016.
photographs. Jama, 3 316(22):2402-244
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. International Conference on Machine Learning (ICML), 2017. URL httpt://aaaa
w.org/abs//170 07019
Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. Calibration of neural networks using splines. arXiv preprint arXiv:2006. /2800, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Computer Vision and Pattern Recognition (CVPR), 2016. URL https://aaax
v. wrorrgabs//512..
htte//aabb/112.333335
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR),
2019. https: //axxi org/ hbtp://b/180000 01697]
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference computer vision, pp. 646-661. Springer, 2016.
10

---

Under review as a conference paper at ICLR 2021
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Volodymyr Kuleshov and Percy S Liang. Calibrated structured prediction. In Neural Information Processing Systems (NeurIPS), 2015. URL //ap://ppppaa...p nips. cc/ htc/paper/5658 ca
iontiio.... ion. .ddf
Ananya Kumar, Percy S Liang. and Tengyu Ma. Verified uncertainty calibration. In Neural Informa- tion Processing Systems (NeurIPS), 2019. URL httth:////aaaat
org/ /abb//199... 10155
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In International Conference Machine Learning (ICML), pp. 2805-28144 14, 2018.
Markus Kängsepp. Nn_calibration. http:///ghhhhhoooa/aaaaasaaaaaaa 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11))2278-2322 1998.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip HS Torr, and Puneet K Dokania. Calibrating deep neural networks using focal loss. arXiv preprint arXiv: 2042.793777 2020.
Kevin P Murphy, Machine learning: a probabilistic perspective. MIT press, 2012.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In AAAI Conference Artificial Intelligence. NIH Public Access, 2015. URL https: /tt////wwww /www ncbi nnlm nih. http::///mmmmmaataace/////////00
90/
Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Mcasuring calibration in deep learning. In CVPR Workshops, pp. 38 38-41,222 2019.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanann and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing 2019.
Systems, pp.
Lorenzo L Pesce, Charles E Metz, and Kevin S Berbaum. On the convexity of roc curves estimated from radiological test results. Academic radiology, 17(8):960-9)) 06000-998.-10 2010.
John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5400, 2019.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2446-2454, 2020.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2013. http::/ wwtp://////a//////eo// 6199
Yuan Wu, Xiaoqian Jiang, Jihoon Kim, and Lucila Ohno-Machado. I-spline smoothing for calibrating predictive models. AMIA Summits on Translational Science Proceedings, 2012:39, 2012.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In loml, volume 1, pp. 609-616. Citeseer, 2001.
11

---

Under review as a conference paper at ICLR 2021
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass proba- bility estimates. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2002.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint urXiv:1605.07146, 2016.
Jize Zhang, Bhavya Kailkhura, and Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. arXiv preprint 20XIv:2003.07:29,
12

---

Under review as a conference paper at ICLR 2021
A MAXIMUM-LIKELHHOO FITS
Table ! provides parameters fit to the top-scores obtained for each of 10 empirical datasets by maximum likelihood estimation.
Table 1: Parameters of best fit for distribution functions investigated Section 5.1
resnet 10_c10 _c10
2.7752
0.0478
resnetl 10 S0__S__100
2.1714
0.0394
resnet_ www.wwddwl2llllc
2.3806
0.0379
densenet40 cc00
1,9824
0.0397
resnet 10_clo_eeo __c100
1.1823
0.1081
resnetl 10 SD c100
1.1233
0.1147
resnet resnet_wide32] wide32 c100
1.0611
0.0650
densenet40_cll =100
1,0805
0.0808
resnet152 __2gnne
1.1359
0,2069
densenet161 Liiegnee 1.1928
0.2206
Global optimia â E [0, 200). 3 E [0, 50) are approximately computed using a recursively-refining brute-force search until both parameters are established to within an absolute tolerance of 1e-5.
Each step in the recursion contracts a linear sampling grid (N 11) by factor of 7 = centered on the previously established optimal parameter, subject to the constraints a, > 0. Experiments confirmed that the computed optima were robust to the hyperparameters N,
26: 3-1
arg min 0,8
Σ In
[(a)T(9) T(a+P)
Table provides parameters fit to calibration functions. For each sample image 2; in the image dataset, define 8; fx;z be the score (the output of the top-scoring logit after softmax) and {0,1} to be the classification (μ 1 when the top-scoring logit correctly classified image a;)) for the sample image. The loss for the binary generalized linear model (GLM) across different combinations of link functions g(y) and transform functions ((s) was optimized via the standard loss
(Gelman al. (2004)) min Σ by bi
- p. -p)
P=-9 + byt(s;))
For each dataset, the GLM of best fit was selected via the Akaike Information Criteria using the likelihood the optimized parameter values.
13

---

Under review as a conference paper at ICLR 2021
Table 2: Parameters of best fit for calibrations functions investigated in Section 5.1
bp
datase_namme
gim_name
datzsel_name
gim_same
resne:110_c00
logtip_ hawppl____pp______ M0_b1
2779.22
-0.24
0.30
#esset110 10000_c100
loget Jagit, b0_bt
7873.61
-0.88
0.49
logat logflip ho bl
2790 40
-0.55
-0. 33
logffip logflip N0. bi
7878. 19
-0.09
0.35
Jogft.ogfffp... hl
2827 51
-0.31
logffip_lagffip bi
7952.25 28
0.38
logit logit 50.
2840.22 70
-0.38
0.36
logil logflip N0. bl
7944.61
-1.04
-0.52
Joght_
2900002
loge__logi__
8315.51
0.32
logftip. logflip.
2932.09
0.34
8457.82
2.18
log_
3221
-0.86
loget_logffip.
8110366
logit logit 50
3799
1,99
log. log.
9983.07
3.30
logflip Jo
3811 58
-2.13
logst_logit
10803.27
0.80
log bi
3829 05
-0.13
log, log b0
10810.90
-0.37
Jngflpoo
3868 40
1,95
logtlip logftip__0
10825:15
-1.16
log log bl
4281 78
4.75
logit logflip ND
1083448
0.76
1smect10_SD_sitt
logit logflip__eeee. 60_bi
2498.96
-0,27
0.35
resteL MameL_wdde2__cor
logtlip logtip 60_
7183.93
0.21
logit logil
2502 52
logit logit
logit logttip_b
2508
logtlpp logttip_
logit logit ho bl
2538.41
-0.26
lught lugfiip No
7297.00
-1.06
-0.34
logflip Loglip_oogllpp.. Jogflip__00000... b0_bl
2550.29
-0.35
0.27
Jogin_ bi
7625.21
0.19
logflip logflip bl
2572.85
0.35
log. log. b0. bl
7650.97
-0.24
2.51
log log blbb1
2594.91
-0.08
1.98
loget logflip b1
7795 28
-0.17
log hi
3137
-0.19
ligflip losflip 10
8977 59
-0.98
logftip_ Jogflip_
3150.42
-1,80
Jogflpp
8967.38
logit loget_oot log. log
1.58
log,
3179667
1,56
logst_logi
log loz
5:48
resee_wide3__coo
logit blgi
24K3,3
0.26
densenett0_cile
logst Jogir b0_b1
8158.28
-0.97
0.34
logfiip logflip 50 bl
2487 69
-0.47
0.22
logflip. logllip 60 bl
8229.43
-0.12
0.22
logat Jogat__ b0_b1
2511 1.29
-0.15
0.28
logst Jogllp bo_bl
8267.77
-1.08
-0.35
lagit logflip 10 bl
2558 45
-0.26
-0.28
ligftip losflip h1
8368.55
0.25
logit Jagfip_ log. ligfli_logfipp..
25
logit log.
8783.50
2713.17
logftppgfipp_bl
log. log
2981
-0.21
logit logit
10138.24
logftip logflip_00
2983.05
-1.70
logit lagflip bo
10182.61
0..55
logit logit ho
2989 90
1,49
logflip Togflip 60
10242.15
-0.94
Jogel_o
3055.55
1.45
log Jog. bü
10261001
-0.50
log
4582.09
4.61
logg log
13322 10
5.25
dessenct40_clU
logit
2910.62
resuet152_
logttip logtlip
18729.85
0.58
logh NO
logit
22
logit_ogflip_
-0.38
-0.31
log log b0_bb
1.32
logftip logftp. 10_b1
3001
-0.31
logttip logflip
18872.14
0.65
Jogut
3021
0.25
logit_logi, bi
19074.37
0.57
logttip logflip bl
3027 78
0.31
loght logftip No bl
19095.40
-0.82
-0.79
log b0.b1
3153.28
-0.12
2.04
log. log_
19840125
1.53
log bi
3531 22
-0.22
logil Jogflip bl
20062 10
-0.50
logtii
3589-11
-1.60
logtlipJoggtip Jogtlppooppppppee b0
26935.09
N0
1601
1.37
Logfliog
logit logflpp
27012.77
log. log
4735
4.27
logit logi,
27084 11
resne:110_c000
Jogglpp Jogflpp_oOO_O... b0_bl
8181.97
-0.11
0.28
densenet161 @ensenet6olimmnnet
log log
18202.41
-0.05
1.27
logat logit to bl
8206 19
-0.88
0.39
loght loge: N0. bl
18463 70
-0.25
0.68
logftip Jogfip bi
830] 28
0.31
logflip Jogflip_b b1
18521.48
0.67
logit logflip 50. bl
8371.53
-1,01
-0. 40440
logflip logflip 60 bl
18534.07
-0.10
0.61
Jogst__
8732.11
logrt_logi_b b1
18822:25
0.60
bit
8918.1
losflip. No,
18113.25
-0.77
logftlo
log. Jog_lo._.1
logit. logfip
10903
0.74
logit lugfip. b1
19562.58
-0.54
Jogst_o
10943.95
0.72
logst Jagflip b0
26425.38
1.19
logftip. lagflip
10964.91
-1.12
logftp. logftpp
26445.01
-1.45
log log bü
11002 20
-0.40
logit, Jogi, bo
26519.76
1.18
log log bl
11850.89
4.26
log. log h0
26662.65
-0.27
14

---

Under review as a conference paper at ICLR 2021
Table ECE reported in Figure c)..
FCE2 (%)
<ECE2> (%, simulated)
resnetl 10_c10
6.67
8.42
resnetl 10 S0_$SDecl0
6.54
8.79
resnet_ resnet_wide322 wide32_ch
6.09
8.44
densenet40_cll cc0
6.70
8.09
resnetl _0_0000
20.26
18.87
resnetl 10 SD c100
17.44
15.78
resnet resnet_wide32
20.40
17.53
densenet40 c100
23.12
19.69
Desnet1.2_imgen densenet1ó1 jimgnet
6.85
9.26
6.15
6.87
B BIAS AND VARIANCE IN CALIBRATION METRICS B. BIAS
We evaluate bias for various calibration metrics using both equal-width and equal-mass binning we vary both the sample size and the number of bins b. These plots should be seen as an alternative visualization to 6 where we additionally compare different choices for the fixed number bins b. Since the ECEsWEEP metrics adaptively choose different number of bins for each sample size, we display the bin number for this metric as 1.
We find that ECEBIN can overestimate the true calibration error and there exists optimal number of bins that produces the least biased estimator that changes with the number of samples n. Additionally, equal mass binning generally results in a less biased metric than equal width binning.
CIFAR-10 ResNet-110. Figure assume parametric curves for (x)) and Ey [Y f(X) that we obtain from likeiihomiiee fits to CIFAR-10 ResNet- 110 model output.
CIFAR-100 Wide ResNet-32. Figure assume parametric curves for p (z)) and Ey [Y f(X) that we obtain from maximum-likel fits to CIFAR-100 Wide ResNet-32 model output.
ImageNet ResNet-152. Figure 11 assume parametric curves for p( (x)) and Er [Y f(X) c) that we obtain from maximum-lise ihood fits to ImageNet ResNet- 152 model output.
15

---

Under review as a conference paper at ICLR 2021
Equal Width ECEgiN
Bias in Equal Mass ECEBIN
4.34
-4.52
-4.65
-4.72 -4.78
-4.82
2
-2.58
-2.59
-2.56
-2.50
-2.50
-2.53
-3.28
-3.71
-4.02
-4.21
-4.34
-4.42
4
-0.70
-0.75
-0.73
-0.66
-0.66
-0.69
-1.43
-2.14
-2.99
-3.04
-3.26
-3.40
8
0.16
-0.13
-0.19
-0.18
-0.20
-0.24
Bins
Bins
16
0.62
-0.37
-1.12
-1.67
-2.01
-2.24
16
1.17
0.42
0.14
0.01
-0.07
-0.14
32
2.66
1.50
0.52
-0.26
-0.83
-1.22
2.82
1.36
0.66
0.32
0.13
-0.01
64
4.54
3.32
2.14
1.13
0.30
-0.30
64
5.50
3.01
1.56
0.83
0.41
0.17
200
400
800
1600
3200
6400
200
400
800
1600
3200
6400
# Samples
Samples
Blas in Equal Width ECEDFeAAS
Blas in Equal Mass ECE
-4,90
-4.93
2
-3.00
-2.79
-4.62
-4.60
..53
4.50
4.50
4
-1.34
-1.05
Bins
-3.30
-3.48
-3.54
3354
33.4
-3.5
Bins
8-
-0.97
-0.66
-0.45
-0.31
-0.27
16
-1.58
-2.15
-2.36
-2.45
-2.47
2..00
16-
-0.89
-0.59
-0.37
-0.24
-0.19
-0.20
32
0.35
-0.56
-1.11
-1.11
-1.56
-1.55
32
-0.96
-0.55
-0.31
-0.17
-0.11
-0.13
64
2.28
1.14
0.19
-0.41
-0.79
-1.00
64
-1.40
-0.55
-0.29
-0.12
-0.07
-0.08
200
400
800
1600
3200
6400
200
400
800
1600
3200
6400
# Samples
Samples
Bias Equal Width ECEsWEEP
Bias Equal Mass ECEsWEE?
# Bins
Bins
-3.52
-3.96
...4
-4.39
-4.50
-4.55
-1
0.05
-016
-0.17
-0.14
-0.15
-0.19
200
400
800
1600
3200
6400
400
800
1600
3200
6400
# Samples
# Samples
Figure 9: Bias for various calibration metrics assuming curves fit to CIFAR-10 ResNet-110 output. We plot bias for various calibration metrics using both equal-width binning (left column) and equal-mass binning (right column) as we vary both the sample size n and the number of bins b.
B.2 VARIANCE
We also compute the variance for various calibration metrics using both equal-width and equal-mass binning as we vary both the sample size n and the number bins As expected, the variance decreases with number of samples, but, unlike the bias, there is clear dependence the number of bins.
CIFAR-10 ResNet-1100 100. Figure 12 assume parametric curves for p(f(x)) and Ey(Y f(X) c) that we obtain from maximum-like fits to CIFAR-10 10 ResNet- 10 model output.
CIFAR-100 Wide ResNet-32. Figure 13 assume parametric curves for p( (x)) and Ey(Y f(X)
c] that we obtain from maximum-likellfiod fits to CIFAR-100 Wide ResNet-32 model output.
ImageNet Res Figure 14 assume parametric curves for p( (x)))) and Er S(X) c)
that we obtain from maximum-liked ihood fits to ImageNet ResNet- 152 model output.
16

---

Under review as a conference paper at ICLR 2021
Bias in Equal Width ECEaIN
Bias in Equal Mass ECEBIN
-6.29
-6.43
-6.49
-6.54
-6.55
-6.54
2
-2.07
-2.06
-2.04
-2.05
-2.04
-2.02
-5.12
-5.48
-5.66
-5.79
-5.83
5.84
4
-1.02
-1.16
-1.20
-1.25
-1.25
-1.23
Bins
-3.15
-3.86
.4.28
.4.50
.4.61
.4.65
Bins
8
-0.06
-0.49
-0.68
-0.78
-0.81
-0822
16
-0.83
-1.91
-2.69
-3.12
-3.34
-3.44
16
1.30
0.51
0.08
-0.14
-0.23
-0.26
32
1.59
0.19
-0.98
-1.74
-2.15
-2.38
32
3.29
1.65
0.78
0.33
0.12
0.03
64
3.83
2.31
0.86
-0.26
-1.01
-1.43
64
6.53
3.45
1.76
0.87
0.43
0.22
200
400
800
1600
3200
6400
200
400
3200
# Samples
# Samples
Bias in Equal Width ECEprews
Bias in Equal Mass ECEDEBIAS
-6.81
-6.69
-6.61
-6.60
-6.58 -6.55
-2.42
-2.23
-2.12
-2.09
-2.06
-2.03
-6.14
-6.00
-5.93
-5.92
-5.89
-5.87
-1.59
-1.44
-1.34
-1.32
-1.28
-1.25
-4,80
-4.86
-4.79
-4.76
-4.73
-4.71
-1.12
-1.02
-0.94
-0.91
-0,88
-0.85
* Bins
Bins
-3.03
-3.49
-3.64
-3.62
33.99
.3.6
-0,69
-0.49
-0.42
-0.39
-0.35
-0.32
32
-0.81
-1.89
-2.49
-2.66
-2.63
-2.62
32
-0.45
-0.27
-0.20
-0.16 -0.12
64
1.41
-0.05
-1.20
-1.74
-1.89
-1.90
64
-0.47
-0.23
200
400
800
1600
3200
6400
200
400
800
1600
3200
6400
Samples
Samples
Bias Equal Width ECEsWEEP
Bias Equal Mass ECEsWEEP
Bins
Bins
-1
-5.80
-6.10
66.09
66.00
-5.83
-5.58
-1
-0.49
-0.58
-0.58
-0.56
-0.46
-0.38
200
800
1600
3200
6400
800
1600
3200
6400
# Samples
# Samples
Figure Bias for various calibration metrics assuming curves fit CIFAR-100 Wide ResNet- 32 output. We plot bias for various calibration metrics using both equal-width binning (left column) and eaual-mass binning (right column) as we vary both the sample size n and the number of bins b.
17

---

Under review as a conference paper at ICLR 2021
Bias in Equal Width ECEaIN
Bias in Equal Mass ECEBIN
-1.35
-1.72
-1.89
-1.96
-2.00
-2.03
2
-1.10
-1.25
-1.29
-1.29 -1.26
-1.26
0.23
-0.65
-1.09
-1.31
-1.40
-1.47
4
-0.02
-0.53
-0.75
-0.85
-0.88
-0.90
Bins
2.69
1.05
0.11
-0.39
-0.63
-0.77
Bins
8
1.60
0.64
0.15
-0.10
-0.19
-0.26
16
5.98
3.23
1.53
0.57
0.07
-0.20
16
3.82
1.95
0.96
0.43
0.18
0.04
32
9.78
6.22
3.50
1.77
0.83
0.31
32
7.06
3.96
2.12
1.08
0.55
0.26
64
13.74
9.91
6.36
3.61
1.92
0.95
64
11.93
7.12
4.05
2.19
1.15
0.58
200
400
800
1600
3200
6400
200
400
3200
# Samples
# Samples
Bias in Equal Width ECEprews
Bias in Equal Mass ECEDEBIAS
-2.82
-2.40
-2.21
-2.12
-2.08
-2.07
-2.29
-1.79
-1.54
-1.41
-1.32 -1.29
-2.53
-2.00
-1.74
-1.63
-1.56
-1.55
-1.83
-1.37
-1.15
-1.05
-0.98
-0.95
-1.91
-1.44
-1.11
-1.00
-0.94
-0.92
-1.33
-0.79
-0.55
-0.44
-0.36
-0.35
* Bins
Bins
-0.65
-1.09
-0.71
-0.59
-0.52
00.99
-1.10
-0.65
-0.33
-0.23
-0.14
-0.12
32
2.20
-0.22
-0.49
-0.38
-0.29
-0.26
32
-1.15
-0.68
-0.29
64
6.96
2.56
0.33
-0.24
-0.19
-0.15
64
-1.33
-0.34
200
400
800
1600
3200
6400
200
400
800
1600
3200
6400
Samples
Samples
Blas Equal Width ECEsWEEP
Bias Equal Mass ECEsWEEP
Bins
Bins
-1
-0.44
-0.80
-0.88
-0.86
-0.80
-0.72
-1
0.73
0.36
0.17
0.01
0.01
-0.03
200
800
1600
3200
6400
800
1600
3200
6400
# Samples
# Samples
Figure 11: Bias for various calibration metrics assuming curves fit ImageNet ResNet-152 output. We plot bias for various calibration metrics using both equtl-width binning (left column) and equal-mass binning (right column) as we vary both the sample size n and the number of bins b.
18

---

Under review as a conference paper at ICLR 2021
Varrance Equal Width ECEBIN
Variance
Equal Mass ECEgIn
2
1.71
1.21
0.91
0.62
0.45
0.32
2
2.38
1.71
1.26
0.87
0.63
0.45
4
2.00
1.49
1.11
0.78
0.56
0.40
4
2.97
2.17
1.59
1.09
0.78
0.56
Bins
8
2.43
1.79
1.39
0.99
0.71
0.52
Bins
8
3.08
2.25
1.66
1.15
0.81
0.59
16
2.67
1.96
1.47
1.07
0.78
0.58
16
3.06
2.22
1.66
1.15
0.82
0.59
32
2.73
1.98
1.48
1.09
0.80
0.59
3.03
2.20
1.65
1.15
0.82
0.59
64
2.79
1.93
1.43
1.06
0.78
0.59
64
3.08
2.21
1.65
1.14
0.82
0.60
200
400
800
1600
3200
6400
200
400
800
1600
3200
6400
# Samples
Samples
Variance in Equal Width ECEDEBIAS
Variance in Equal Mass ECEpEaIAS
2
2.06
1.46
0.99
0.65
0.46
0.32
2
2.44
1.73
1.26
4-
2.82
1.96
1.25
0.83
0.57
0.41
3.11
2.20
1.61
1.10
0.78
Bins
8-
3.44
2.39
1.65
1.09
0.74
0.54
Bins
8
3.36
2.33
1.69
1.16
0.82
0.59
16
3.70
2.64
1.82
1.21
0.84
0.60
16
3.53
2.37
1.71
1.16
0.82
0.60
32
3.55
2.59
1.84
1.26
0.88
0.62
32
3.91
2.47
1.75
1.19
0.83
0.60
64
3.39
2.38
1.76
1.27
0.90
0.64
64
4.83
2.75
1.84
1.20
0.84
0.60
200
400
800
1600
3200
6400
200
400
800
1600
3200
6400
# Samples
Samples
Variance Equal Width ECEsWEEP
Variance Equal Mass ECEsWEEP
Bins
Bins
-1
2.61
1.76
1.20
0.85
0.63
0.50
-1
3.16
2.30
1.69
1.16
0.83
0.60
200
1600
3200
6400
400
800
1600
3200
6400
# Samples
# Samples
Figure 12: / Variance for various calibration metrics assuming curves fit to CIFAR-10 ResNet- 110 output. We plot / Variance for various calibration metrics using both equal-width binning (left column) and equal-mass binning (right column) as we vary both the sample size n and the number of bins b.
19

---

Under review as a conference paper at ICLR 2021
Variance Equal Width ECEBIN
Variance Equal Mass ECEgIN
2
2.41
1.70
1.26
0.88
0.60
0.44
2
3.27
2.30
1.68
1.17
0.81
0.58
4
2.47
1.80
1.35
0.94
0.65
0.47
4
3.36
2.33
1.73
1.19
0.83
0.59
8
2.64
1.96
1.47
1.04
0.72
0.53
8
3.42
2.36
1.76
1.23
0.85
0.61
Bins
Bins
16
2.79
2.06
1.58
1.12
0.78
0.57
16
3.38
2.38
1.76
1.23
0.86
0.62
32
2.72
2.01
1.58
1.14
0.82
0.58
3.26
2.34
1.74
1.23
0.86
0.62
64
2.73
1.96
1.55
1.13
0.82
0.59
64
3.13
2.31
1.71
1.21
0.86
0.61
200
400
800
1600
3200
6400
200
400
800
1600
3200
6400
# Samples
Samples
Variance in Equal Width ECEDEBIAS
v Variance in Equal Mass ECEpEaIAS
2
2.49
1.73
1.27
0.88
0.60
2
3.32
2.32
1.69
1.17
0.81
4-
2.74
1.89
1.39
0.95
0.65
0.47
3.45
2.36
1.74
1.20
0.83
0.60
Bins
8-
3.12
2.16
1.54
1.06
0.73
0.53
Bins
8
3.58
2.41
1.77
1.23
0.85
0.61
16
3.47
2.39
1.72
1.17
0.80
0.57
16
3.67
2.48
1.80
1.24
0.87
0.62
32
3.30
2.42
1.79
1.23
0.85
0.59
32
3.79
2.53
1.80
1.25
0.87
0.62
64
3.16
2.35
1.83
1.26
0.87
0.61
64
4.29
2.69
1.84
1.26
0.87
0.62
200
400
800
1600
3200
6400
200
400
800
1600
3200
6400
# Samples
Samples
Variance Equal Width ECEsWEEP
Variance Equal Mass ECEsWEEP
Bins
Bins
-1
2.61
1.77
1.31
0.98
0.72
0.61
-1
3.44
2.44
1.81
1.24
0.88
0.64
200
1600
3200
6400
400
800
1600
3200
6400
# Samples
# Samples
Figure 13: Variance for various calibration metrics assuming curves fit to CIFAR-100 Wide ResNet-32 output. We plot VVariance for various calibration metrics using both equal- binning (left column) and equal-mass binning (right column) as we vary both the sample size n and the number of bins b.
20

---

Under review as a conference paper at ICLR 2021
VVrrinnce Equal Width ECEBIN
Variance Equal Mass ECEgIN
2
2.29
1.59
1.12
0.78
0.56
0.41
2
2.89
2.10
1.49
1.05
0.76
0.55
4
2.50
1.81
1.27
0.91
0.67
0.49
4
2.78
2.02
1.43
1.01
0.73
0.53
8
2.64
1.93
1.40
1.01
0.75
0.54
8
2.84
2.07
1.47
1.03
0.75
0.55
Bins
Bins
16
2.56
1.90
1.39
1.01
0.75
0.54
16
2.73
2.03
1.46
1.03
0.75
0.55
32
2.41
1.83
1.33
0.98
0.75
0.54
2.68
1.96
1.42
1.01
0.75
0.55
64
2.29
1.68
1.27
0.97
0.72
0.53
64
2.59
1.92
1.37
1.00
0.73
0.54
200
400
800
1600
3200
6400
200
400
800
1600
3200
6400
# Samples
Samples
Variance in Equal Width ECEDEBIAS.
v Variance in Equal Mass ECEpEaIAS
2
2.86
1.77
1.17
0.80
0.57
0.42
2
3.41
2.29
1.54
1.07
0.77
4-
3.56
2.22
1.38
0.95
0.68
0.50
3.48
2.26
1.50
1.03
0.74
0.53
Bins
8-
4.26
2.74
1.63
1.09
0.78
0.55
Bins
8
3.80
2.45
1.58
1.07
0.76
0.55
16
4.85
3.26
1.79
1.15
0.81
0.56
16
4.16
2.71
1.66
1.10
0.77
0.56
32
4.91
3.75
2.12
1.24
0.85
0.58
32
4.87
3.13
1.78
1.14
0.80
0.57
64
4.20
3.57
2.41
1.48
0.91
0.59
64
5.67
3.84
2.05
1.24
0.82
0.57
200
400
800
1600
3200
6400
200
400
800
1600
3200
6400
# Samples
Samples
Variance Equal Width ECEsWEEP
Variance Equal Mass ECEsWEEP
Bins
Bins
-1
2.21
1.64
1.29
0.96
0.73
0.55
-1
2.82
2.14
1.53
1.08
0.76
0.55
200
1600
3200
6400
400
800
1600
3200
6400
# Samples
# Samples
Figure 14: Variiace for various calibration metrics assuming curves fit to ImageNet ResNet- 152 output. We plot / Varriance for various calibration metrics using both equal-width binning (left column) and equal-mass binning (right column) as we vary both the sample size and the number of bins b.
21

---

Under review as a conference paper at ICLR 2021
C
WHAT NUMBER OF BINS DOES EQUAL MASS ECE SWEEP CHOOSE?
500 samples
500 samples
2000 samples
2000 samples
19
5000 samples
2000 samples
32000 samples
32000 samples
18
128000 samples
128000 samples
$12000 samples
15.0
512000 samples TCE
TCE
ECE
15
7.5
14
2.5
11
175
00
number
(a) Uncalibrated model.
(b) Perfectly calibrated model.
Figure 15: Bins chosen by equal mass ECESWEEP method. We plot equal mass ECEMN % versus number of bins for various sample sizes n. We highlight the TCE with horizontal dashed line and show the average number of bins chosen by the ECESwEEP method for different sample sizes with vertical dashed lines. When the model uncalibrated (left) ECEswEEP chooses bin number that is close to optimal. However, for perfectly calibrated models (right). the optimal number of bins is small (<=4), and ECES does not good job selecting good bin number. The incorrect bin selection may partially explain why ECHswwur still has some bias for perfectly calibrated models. However, we note that any binning-base technique that always outputs a positive number will never be completely unbiased for perfectly calibrated models.
For Figure 15| the uncalibrated plot assumes EY S(X) = c] = logistic(10 e 5) while calibrated plot assumes Ey [Y ((X) = c. Both experiments assume f(x) Uniform(O, 1).
22

---

Under review as a conference paper at ICLR 2021
D DIFFERENCES IN EW LEC__BIN VS. EM ST_ECE_SWEEE
In Table we compare the values for ECEBIN reported in Guo et al. (017)) Table 1 against our computation of the same quantities using 15 equal-width bins, but using the logits reported in Kängsepp (2019). We report both absolute and relative differences between these two quantities. The table has rows sorted according to the ECEai obtained Guo et al. (2017).
Table Comparison ECE values (and associated rank orderings) computed using ew_ece_bin _in vs em em_ece_sweep from uncalibrated logits.
Uncalibrated ECE(%)
x=ew bin
#Fem_sweeq)
x-y
100(x-y)/x
resnetl 10netl 110_SSD_100 4D_c10
4.11(0))
4.10(0)
0.01
0.24
resnet resnet_wide32 wide32_chl c10
4.51(1)
4.48(1)
0.03
0.66
resnetl c10
4.75(2)
4.75(2)
-0,00
-0.00
densenet40_cco
5.50(3)
5.49(3)
0.01
0.13
densenet161 Limmggee
5.72(4)
5.72(4)
-0.00
-0.00
resnet152 0m2gneettttt
6.54(5)
6,54(5)
0.00
0.00
resnetl LO__SD_______________________ c100
15.86(6)
15.83(6)
0.03
0.18
resnet 10 10_cc00
18.48(7)
18.48(7)
0.00
0.00
resnet_ resnet_wide32 c100
18.78(8)
18.78(8)
-0.00
-0.00
Sensenet40_c100
21.16(9)
21.16(9)
0.00
0.00
Table 5: Comparison ECE values (and associated rank orderings) computed using ew _c_e bin vs em, _me ece_sweep from logits calibrated using temperature scaling Guo et al. (2017). Red indicates differences in the sorted order of each entry the column.
Temp. scaling(%)
x=ew_bin
y=em_sweep
X-y
100(x-y)/x
resnetl 10_$0_0___l_
0.56(0)
0.36(1)
0.19
34.99
resnet_ Seeet__idee__ee___
0.78(1)
0.21(0)
0.57
72.65
densenet40 c100
0,90(2)
0.72()
0.18
20.45
densenet40_cdd c10
0.95(3)
0.90(3)
0.05
4.98
resnetl 10 Q_c10
1.13(4)
0.91(4)
0.23
19.90
resnet 10. SD S00
1.21(5)
0.98(5)
0.23
18.96
resnet resnet_wide32 2ide32_c100
1.47(6)
1.31()
0.16
10.82
densenet161_img Limmggt
1.94())
1.88(8)
0.06
3,01
resnet152_imgen
2.08(8)
2.13(9)
-0.05
-2.59
resnetl 10 c100
2.38(9)
1.87(7)
0.51
21.38
23

---

Under review as a conference paper at ICLR 2021
Table 6: Comparison of ECE values (and associated rank orderings) computed using ew w__ee___in vs. _e_e___eeeeppppeeeee from logits calibrated using isotonic regression Zadrozny Elkan 2002
Isotonic regression(%e)
x=ew bin
y=em_ prem_sseepp
x-y
100(x-y)/x
resnetl 10 SD__c0
1.03(0)
0.70(0)
0.33
32.15
resnet_ wide32_clll
1.19(1)
0.77(1)
0,42
35.41
resnetl 10_
1.47(2)
0.93(2)
0.55
37.14
densenet40_com cc0
1.68(3)
1.61(3)
0.08
4.66
densenet161 j_iinnet
4.64(4)
4.64(4)
-0.00
-0.00
resnet 10 SD 100cc100 c100
4.89(5)
4.86(5)
0.04
0.81
densenet40 _100 resnet152 imgnet resnet resmet_wide32 c100
5.01(6)
4.95(6)
0.06
1.14
5.15(7)
5.11(7)
0.04
0.74
5.76(8)
5.64(8)
0.12
2.15
resnetl 10_c100
6.19(9)
6.05(9)
0.15
2.40
Table 7: Comparison of ECE values (and associated rank orderings) computed using ew ______________________________e_ bin vs em ee_eee________eeeeeemeeeeeeeeeee _ce_sseep from logits calibrated using histogram binning Zadrozny Elkan (2001). Red indicates differences in the sorted order of each entry in the column.
Histogram binning(%%
x=ew_bin
y=em_sweep
x-y
100(x-y)/x
resnet ceeeet_iieeeeeeeeeee c10
0.56(0)
0.56(0)
0.00
0.00
r0snetl SD_c10
0.62(1)
0.57(1)
0.05
7.61
resnet152_ imgnet
0.78(2)
0.76(4)
0.02
2.14
densenet161 imggnet
0.80(3)
0.64(2)
0.16
19.86
densenet40_cll 1000
0.81(4)
0.80(5)
0.01
1.02
resnetl 10_c10
0.84(5)
0.67(3)
0.17
20.67
resnet 10 c100
0.91(6)
0.91(6)
0.00
0.00
densenet40 c10
1.27(7)
1,27(7)
-0.00
-0,00
resnet_wide32
1.45(8)
1.45(8)
0.00
0.00
1.58(9)
1.55(9)
0.03
1.82
24


Reviews:
[{"review_id": "sITqER2ztS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThis paper highlights a major flaw with the commonly-used $ECE_\\text{BIN}$ calibration metric, namely that it is biased for perfectly-calibrated models. Through a large number of empirical experiments (including through simulation), the authors show that a newly proposed metric, $ECE_\\text{SWEEP}$ is able to produce less biased estimates of calibration error.\n\n### Paper Strengths\n\n1. The paper has many empirical experimental and simulation data to support the claim that $ECE_\\text{SWEEP}$ is a useful calibration metric.\n\n2. The paper points drawbacks of commonly-used calibration metrics such as $ECE_\\text{BIN}$, namely that it is biased for perfectly-calibrated models, and suggests alternatives.\n\n### Major Concerns\n\n1. In the theoretical discussion, the paper assumes binary classification. However, the empirical experiments are conducted on multi-class classification datasets. I did not find any mention of how the binary classification discussion is generalized to the multi-class setting.\n\n2. The discussion about how the simulations were conducted was difficult to follow. Are the $m$ simulated datasets essentially subsets of the full CIFAR-10/100 or ImageNet datasets? (This seems to be implied, but is never made explicit. Otherwise, it can read as if these datasets are synthetic images.) And how do you draw samples such that \"the confidence score distribution matches the neural model's best fit Beta distribution and the true calibration curve matches the neural model's best fit GLM\"?\n\n3. The equation for $ECE_\\text{SWEEP}$ seems misleading. Are you really trying to maximize the quantity in parentheses over all possible values of $b$? And if so, why does that mean that this maximization will result in the largest number of bins, under the monotonicity constraint?\n\n4. Is it always possible to satisfy the monotonicity constraint for $ECE_\\text{SWEEP}$? Some more theoretically-sound discussion of $ECE_\\text{SWEEP}$ would be much appreciated.\n\n5. The discussion on Page 3 about Figure 2 can be expanded upon. Why does the \"optimal bin count grow with the sample size\"? This is stated as fact without explanation.\n\n6. In general, while the experiments are mostly convincing, it would be useful to have some theoretical notions for why/when $ECE_\\text{SWEEP}$ is less biased than $ECE_\\text{BIN}$.\n\n\n### Minor Concerns\n\n1. There are many typos, grammatical errors, and mis-referenced figures throughout the manuscript. Please correct them.\n\n2. Running experiments on new datasets is understandably time-consuming. However, given that this paper is heavily dependent on empirical evidence, it would be helpful to see experiments on datasets beyond image classification.\n\n### Original Rating\n\n**Rating** - 5: Marginally below acceptance threshold\n\n**Confidence** - 3: The reviewer is fairly confident that the evaluation is correct\n\n### Post-Rebuttal Update\n\nI applaud the authors for providing detailed responses to my (and other reviewers') questions and for updating the manuscript appropriately. Several follow-up thoughts to my original questions:\n\n1. **Binary classification.** I think I understand what you mean now. Basically you are still performing multi-class classification, but then you treat the calibration problem as a binary: is the top-1 model output correct or not? I think this still can be made clearer in the manuscript.\n\n2. **Simulation procedure**: Thank you for the clarifications.\n\n3. **$\\text{ECE}_\\text{SWEEP}$ equation**: I am still concerned that the equation given for $\\text{ECE}_\\text{SWEEP}$ differs from Algorithm 1. (Thanks for moving the algorithm into the main manuscript!) The issue is that I don't think Algorithm 1 is taking the maximum over the quantity in the parentheses of the $\\text{ECE}_\\text{SWEEP}$ equation. Algorithm 1 is not actually computing $\\max_b g(b)$ for some function $g(b)$. Instead, it is first finding the maximum $b$ that satisfies some criterion, then calculating $g(b)$ at that selected $b$.\n\n  If the equation and the algorithm are the same, it would imply that increasing the number of bins will increase the estimated calibration error. However, this does not seem to be true. Consider a binary dataset that is perfectly balanced: $\\bar{y_1} = \\frac{1}{n} \\sum_{i=1}^n y_i = 0.5$, and consider a constant model, i.e. $\\forall x: f(x) = 0.6$. Then for the case $b=1$, $ECE = 0.1$. But for the case $b=2$, $ECE = 0$ (assuming equal-width binning).\n\n4. **Monotonicity constraint**: Thanks for pointing out the trivial constraint satisfaction.\n\n5. **\"optimal bin count grows with the sample size\"**: I understand the empirical and intuitive reasoning for why this should be true. However, I still wish that this notion could be made more formal.\n\n6. **Theoretical notions for why/when $\\text{ECE}_\\text{SWEEP}$ is less biased than other estimators of calibration error**: I am now more convinced that this is difficult to show, and I understand that a lot of the literature is based on empirical evidence. However, given that the results are empirical, I still would like to see experiments are non-image datasets.\n\n**Updated Rating** - 6: Marginally above acceptance threshold", "extra_scores": {}, "extra_text": {}}, {"review_id": "etmZ5i_8aql", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the manuscript, \"Mitigating bias in calibration error estimation\", the authors examine the problem of estimating the 'calibration error' of a binary classifier; the problem context being that addressed by techniques such as Platt scaling.  Here the class of estimators considered are what I would call 'bin smoothers', namely function estimators constructed from piecewise constant sub-estimators.  That these estimators suffer biases in the finite sample regime---and indeed asymptotically if the number of bins is not allowed to increase with sample size---does not seem remarkable to me (even without reference to the extensive literature on this class of problem), and likewise is noted (though not numerically explored) by previous authors considering the calibration error topic (e.g. Nixon et al. 2019).  The question then is whether this bias can be shown to have meaningfully affected the conclusions of previous studies of calibration error corrections (in particular, overturning or questioning the results of studies that were themselves important in the field).  From the experimental section presented here it is not clear that this is the case.  On the other hand, if the value of the 'sweep' method is to be the focus then a comparison should be made against a selection of competitive methods for bin smoothing in contemporary use (e.g. moving windows, adaptive bandwidths) rather than simply the equal width and equal mass bins of fixed number.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "HL2xM1I9eM", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post Rebuttal:\n\nClearly I was too cavalier in suggesting alternative bin selection methods, as the authors have clarified. I think the paper is much improved by the comparison to other techniques such as ECE_debiased. The rebuttal also made me re-read with the view of seeing the demonstration of how bias varies with sample size as a major part of the contribution. In addition, the concrete demonstration of the relevance of the discussion via fig. 2 is nice.\n\nOn the whole, my opinion has shifted to be much more positive. I have been rating the paper somewhere between 6 and 7, and in such a case I would rather err on the side of acceptance. \n\nTwo asides:\n1. I only realised this now, but I had misentered my confidence - it should have been 3 and not 5! I have corrected this now.\n\n2. Of course, actual constants matter in practice, but one way to select number of bins by considering errors in $\\overline{y}_k$ is that the squared error in each of these scales as $B/n,$ which, when added over $B$ bins induces error at scale $B^2/n$. With the same logic as the maximum number of bins in ECE_sweep, this suggests one way to set $B$ to be as $B = \\lfloor \\sqrt{n} \\rfloor,$ which controls this error term. Of course, this is only a heuristic, and the analysis is incomplete because it's not accounting for correlations between these errors and $(f(x_i) - \\overline{y}_k).$\nIn a larger sense, I still think that just looking at $15$ bins because that's all that has been looked at in the literature is not quite the right thing - basically I think that when studying an autotuning method, such as ECE_sweep, it is important to illustrate how it performs with respect to simple strategies based on some basic heuristics. This was the reason I wanted to score the paper at $6$, but I didn't bring this up when the authors asked for references and so felt it would be unfair to penalise on this basis.\n\n-----\nSummary of claims:\n\nThe paper studies the estimation of the calibration error of models, which is a problem of increasing relevance due to the rich settings that modern ML systems are being employed in. In particular, the focus is on mitigating the _bias_ in the commonly used binned estimators. The authors make the case that the biases of the binned estimators vary with the number of bins and the number of samples in a nontrivial manner, which is detrimental to point estimation. As a form of autotuning the number of bins that should be used, the  authors use the fact that the true calibration curve should be monotone increasing to propose choosing the number such that the resulting estimates of the average values of the $y$s in the resulting bins are monotonic. The bias of this estimator is then demonstrated on a number of simulations for which the laws of the classifier outputs and the calibration curves are obtained by fitting to models trained on standard datasets.\n\n----\n\nStrengths:\n\nThe problem being studied is certainly relevant, and the work is well contexualised. I found the idea of using the monotonicity of the $\\{\\\\overline{y}_k\\}$ as a criterion to choose an appropriate binning scale to be clever. I also like the problem of testing if the true calibration error is large or not. Finally, the method of section 5.1 is a nice way of generating realistic laws and calibration curves, and adds meaningful depth to the simulations.\n\n----\n\nWeaknesses: \n\n\n1\\) The nature of the contribution with respect to ECE_sweep is not clearly described in the text. Concretely, this amounts to a way to choose the number of bins using data (i.e., autotuning a hyperparameter in the estimate). While this, of course, leads to a different estimator, this is not something fundamentally different. I would much rather that the paper was upfront about the contribution. (In fact, I was pretty confused about the point the paper was making until I realised this).\n\n2\\) I don't think the baseline comparisons made in the experiments are appropriate. The proposal is a method to choose the appropriate number of bins in the estimate, and should be compared to other methods to do so instead of to an arbitrary choice of number of bins as is done in section 5.2. Without this comparison, I have no way to judge if this is a good autotuning method or not. Reasonable comparisons could be, e.g., choosing $b$ by cross validation, or, in equal mass binning, choosing $b$ so that each bin has a reasonable number of samples for the error $\\overline{y}_k$ to not be too large.\n\n3\\) While the focus of the paper is on bias, it should be noted that by searching over many different bin sizes, the variance of  ECE_sweep may be inflated. If this is to such an extent that the gains in bias relative to other autotuning methods are washed out, then this estimator would not be good. To judge this requires at least that the variances for ECE_sweep are reported, but these are never mentioned in the main text.\n\n4\\) Choice of law in simulation in section 3, which are used to illustrate the dependence of bias on the number of bins, not aligned with the laws/curves in figure 3. Taking the latter as representative of the sort of laws and calibration curves that arise in practice, there are two issues:\n\n4a\\) The pdfs of $f$ tend to be a lot more peaked near the end than the one explored in section 3 - this is borne out by the values of $\\alpha, \\beta$ in the fits in Table 1. Beta(1.1,1) is remarkably flat compared to the curves in Fig 3.\n\n4b\\) There seem to be a few different qualitative properties of the calibration curves - monotone but with a large intercept at 0; those with an inflection point in the middle; and those with the bulk lying below the $y=x$ line. In particular, all of them tend to have at least some region above the $y=x$ line. The choice of curve $c^2$ in section 3 doesn't completely align with any of these cases, but even if we make the case that it aligns with the third type, this leaves two qualitative behaviours unexplored. \n\nIn fact, the choice of laws is such that the error of the hard classifier that thresholds $f$ at $1/2$ is $26\\\\%$. I don't think we're usually interested in the calibration of a predictor as poor as this in practice.\n\nAll of this makes me question the relevance of this simulation. Is the dependence of the bias on the number of bins as strong for the estimated laws as it is for these? Seeing the the equivalents of figs 7 and 8 for the laws from section 5 would go a long way in sorting this out.\n\n5\\) Experiments: As I previously mentioned, I don't think the correct baselines are compared to. Instead of posing the method against other autotuning schemes, just one choice of the number of bins is taken. This already makes it near impossible to judge the efficacy of this method.\n\nDespite this, even the data presented does not make a clear case for ECE_sweep. In Fig. 4 we see that the bias of EW_sweep is even worse than EW. This already means that the sweep estimate doesn't fix the issues of ECE_bin in all contexts. It _is_ the case that EM_sweep has better bias than EM, but again, for samples large enough for the variances to be in control, it seems like these numbers are both converging to the same, so I don't see any distinct advantage when it comes to estimation. (of course, this is moot because this isn't the right comparison anyway)\n\nAlso, Fig. 5 is flawed because it compares EW and EM_sweep. It should either compare EM and EM_sweep, or EW and EW_sweep, I don't see why EW and EM_sweep are directly comparable.\n\n\nMinor issues:\n\na\\) Algorithm (1) and the formula for ECE_sweep in section 4 don't compute the same thing. In algorithm (1), you find the largest $b$ such that the resulting $\\{\\overline{y}_k\\}$ is a monotone sequence, and return the ECE_bin for this number of bins. In the formula, you maximise the ECE_bin for all b that yield a monotone $\\{\\overline{y}_k\\}$. From the preceding text, I assumed that the quantity in Algorithm (1) is intended. \n\nb\\) Why is the $L_p$ norm definition of the ECEs introduced at all? In the paper only $p = 2$ is used throughout. I feel like the $p$ just complicates things without adding much - even if you only present the $L_2$ definition, the fact that a generic $p$ can be used instead should be obvious to the audience.\n\nc\\) Design considerations for ECE_sweep - it is worth noting that accuracy is not all that we want in an estimate of calibration error. For instance, one might reasonably want to add this as a regulariser when training a model in order to obtain better calibrated solutions. One issue with ECE_sweep is that it introduces a problem in that how the number of bins in the ECE_sweep estimate changes with a small change in model parameters seems very difficult to handle, which makes this a nondifferentiable loss. Broader issues of this form, and a discussion of how they may be mitigated, could lead to a more well rounded paper.\n\n----\n\nComments:\n\na\\) Exact monotonicity in the ECE_sweep proposal - I find the argument stemming from the monotonicity of the true calibration curve, and the idea to use this to nail down a maximum binning size interesting. However, why should we demand exact monotonicity in the bin heights? Each $\\\\overline{y}_k$ will have noise at the scale of  roughly $\\sqrt{b/n},$ (for equal mass binning with $b$ bins), and in my opinion, violation of monotonicity at this scale should not be penalised. Also, what if a few $\\overline{y}_k$s decrease but most are increasing (i.e., the sequence has a few falling regions, but the bulk is increasing)? Perhaps instead of dealing with this crudely, the error of a shape constrained estimator may serve as a better proxy.\n\nb\\) Isn't the procedure for parametrically fitting the pdf of $f$, and $\\\\mathbb{E}[Y|f(X)],$ and then integrating the bias a completely different estimator for TCE of a model? In fact, if the laws are a good fit, as is claimed in section 5.1, then this plug in estimator might do well simply because the integration is exact. In fact, since the fit is parametric, this can further be automatically differentiated (if, say, $f$ were a DNN), and thus used to train.\n\nc\\) It would be interesting to see what number of bins are ultimately adopted in the ECE_sweep computations that are performed. \n\n----\n\nOverall opinion: The lack of comparison to appropriate baselines makes it near impossible for me to judge the validity of the proposed estimator. I feel like this is a deep methodological flaw when it comes to evaluating the main proposal of the paper. This is a real pity because I quite like some of the ideas in the paper.\n\nDue to the inability to evaluate the main contribution of the paper, i am rating it a strong reject. I'd be completely open to re-rating it if appropriate comparisons are performed, and the case for the method is properly made.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2A4u4aWrZ0B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces an estimation approach for the true calibration error (TCE). TCE is compared to the binning-based estimator of calibration error (ECE) to measure bias.  Also, a maximum estimator of calibration error ECE_sweep is introduced and compared against standard binning ECE_bin across image datasets in classification tasks. \n\n**Strengths**\n-  With extensive sensitivity analysis, the paper illustrates the importance of  non-biased ${\\rm ECE}$  calibration evaluation and model selection\n\n**Weakness**\n\nThe reviewer appreciates the extensive sensitivity analysis. However, the contributions seem incremental in terms of methodology.\n\n*Clarity*:\n- The writing could be improved by the following:\n1) Simulation steps for estimating bias in section 3  can be written in algorithm form\n2) Computation or approximation of TCE described in section 5 is difficult to follow\n3) Sections arranged in terms of concepts, e.g., TCE is introduced in section 2 and an approximation approach for TCE described in section 3 and section 5\n4) The proposed ${\\rm ECE}_{\\rm sweep}$ estimator moved from the appendix to the main section\n5) What is the motivation for the proposed ${\\rm ECE}_{\\rm sweep}$\n\n*Strong technical assumptions*:\n- Estimation of TCE relies on parametric formulations for confidence distribution and the true calibration curve;  such parametric assumptions are violated in practice\n- The paper assumes monotonicity in the true calibration curve, which is ill-justified \n- Provided that the proposed solutions rely on TCE, which is unknown. To evaluate bias, the proposed solution requires the above assumptions to hold\n\n*Additional experiments*:\n- How does the proposed solution compare to non-binning based approaches?\n- Sensitivity analysis of top-r estimates, where r is the rank?\n ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an adjustment to the ECE metric to make it less biased in the small sample case by including the assumption that the confidence output by a classifier is monotonic with the true correctness probability.  The main idea is to successively make finer bins until a non-monotonicity is observed.  The paper is interesting, but the magnitude of the contribution would be just enogh for a short paper if such a track existing in ICLR.  

Reviewers have raised concerns about the discrepancy between their revised ECE formula and the Algorithm accompanying it, although that has been fixed through the author feedback phase.  Another concern is that for a paper whose core technical contribution is a revised metric for measuring calibration, a more thorogh empirical study over larger datasets is required.
Decision: Reject