Better Uncertainty Calibration via Proper Scores for Classification and Beyond
Authors: Sebastian Gregor Gruber
Florian Buettner
Venue: NeurIPS 2022 Accept (2022)
Keywords: Calibration
Predictive Uncertainty
Classification
Regression

Abstract: With model trustworthiness being crucial for sensitive real-world applications, practitioners are putting more and more focus on improving the uncertainty calibration of deep neural networks.
Calibration errors are designed to quantify the reliability of probabilistic predictions but their estimators are usually biased and inconsistent.
In this work, we introduce the framework of \textit{proper calibration errors}, which relates every calibration error to a proper score and provides a respective upper bound with optimal estimation properties.
This relationship can be used to reliably quantify the model calibration improvement.
We theoretically and empirically demonstrate the shortcomings of commonly used estimators compared to our approach.
Due to the wide applicability of proper scores, this gives a natural extension of recalibration beyond classification.

Better Uncertainty Calibration via Proper Scores for Classification and Beyond
Sebastian G. Gruber
German Cancer Research Center (DKFZ), Germany German Cancer Consortium (DKTK), Frankfurt, Germany Goethe University Frankfurt, Germany sebastian, gruberOdkfz.de
Florian Buettner
German Cancer Research Center (DKFZ), Germany
German Cancer Consortium (DKTK), Frankfurt, Germany Frankfurt Cancer Institute, Germany
Goethe University Frankfurt, Germany
florian. buettnerddkf..de
Abstract
With model trustworthiness being crucial for sensitive real-world applications, practitioners are putting more and more focus on improving the uncertainty cali- bration of deep neural networks. Calibration errors are designed to quantify the reliability of probabilistic predictions but their estimators are usually biased and inconsistent. In this work, introduce the framework of proper calibration errors, which relates every calibration error to a proper score and provides a respective upper bound with optimal estimation properties. This relationship to reliably quantify the model calibration improvement. We theoretically and empirically demonstrate the shortcomings of commonly used estimators compared to our approach. Due to the wide applicability proper scores, this gives natural extension of recalibration beyond classification.
be used
Introduction
Deep learning became a dominant cornerstone of machine learning research in the last decade and deep neural networks can surpass human-level predictive performance on wide range of tasks [[7] However, Guo et al. [4] have shown that for modern neural networks, better classification accuracy can come at the cost of systematic overconfidence in their predictions. Practitioners in sensitive forecasting domains, such as cancer diagnostics [77 genotype-baut based disease prediction [2 or climate prediction [59]. require for models to not only have high predictive power but also to reliably communicate uncertainty. This raises the need to quantify and improve the quality of predictive uncertainty, ideally via dedicated metric. An uncertainty-ave probabilistic predictions which represent the true likelihood of events depending the very prediction. To quantify the extend to which this condition is violated, calibration errors have been introduced.
19
47
model should give
In general, their estimators are usually biased [46] and inconsistent (53) This, in turn, is highly problematic since we cannot quantify how reliable model is if we do not know how reliable the metric is. Especially the medical field domain that requires high model trustworthiness, but with low expert availability and/or disease frequency we often encounter small data regime. Resampling strategies can be viable options for optimization on small datasets but also reduce the evaluation set
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

0.25
Improvement
MO
Calibration Eslimala
0.05
015
(ourc)
KDL TCL2
0.10
0.00
Tob d CCL
0.00
TC.F,
0.00
15b OIND;
2014
10000
2154
5966
10000
160
165
1163
2000
7120
rat30
2004
TULD CWCE
Test
Test sel size
Tast 5et sire
KDC CE.
(a) TS of DenseNet 40 CIFARIO
(b) ETS of Wide ResNet 32 CIFARI00
(c) DIAG of DenseNet 161 on ImageNet
Figure I: Estimated calibration improvement for various settings. The calibration error is estimated before and after recalibration method (TS ETS DIAG) applied and the difference (i.e. calibration improvement) is shown for increasing test set size. All common calibration estimators are sensitive with respect to the test set size and can substantially over- or underestimate the effect of performing recalibration Only RBS robustly estimates the improvement in calibration error for all test set sizes.
size even more. We will discover that little data exacerbates the estimation bias and propose reliable alternatives for improving uncertainty calibration.
Since deep neural networks often yield uncalibrated confidence scores 36 a variety of different post-hoc recalibration approaches have been proposed 31 These methods use the validation set transform predictions returned by trained neural network such that they become better calibrated. key desired property of recalibration methods is to not reduce the accuracy after the transformation. Therefore, most modern approaches are restricted to accuracy preserving transformations of the model outputs [14, 42 45 63.. When recalibrating a model, it is crucial to have reliable estimate of how much the chosen method improves the underlying model. However, when using current estimators for calibration errors, their biased nature results in estimates that are highly sensitive to the number of samples in the test that are used compute the calibration error before and after recalibration (Fig. [ c.f. Section 5) The source code is openly available at https //githut www./momm//////.. lab wwt.err.wnrrraatyy.cllaa.aaaaaaa
Our contributions for better uncertainty calibration are summarized the following. We...
give an overview of current calibration error literature, place the errors into a taxonomy, and show which are insufficient for calibration quantification. This also includes several theoretical results, which highlight the shortcomings of current approaches.
introduce the framework of proper calibration errors, which gives important guarantees and relates every element to a proper score. We can reliably estimate the improvement an injective recalibration method w.r.t. a proper calibration error via its related proper score even in non-classification settings.
show that common calibration estimators are highly sensitive w.r.t. the test set size. We demonstrate that for commonly used estimators. the estimated improvement of recalibration methods is heavily biased and becomes monotonically worse with fewer test data.
Related Work
In this section, we give an extensive overview of published work regarding quantifying model calibration and model recalibration. Important definitions will be directly given, while others are placed in Appendix C These will be the basis for our theoretical findings. Further, will motivate the definition of proper calibration errors, which are directly related to proper scores. Consequently, will also present important aspects of the framework around proper scores in later parts of this section.
For consistency with other calibration estimators, refer KDE proposed by
43
KDE CE

---

2.1 Calibration errors
For clarity, we introduce shortly the notation used throughout this work. Assume we have random variables X and Y corresponding to feature and target variable, and feature and target space L and V. We have Py, Pylx E P, where IP refers to the distribution of Y, y to the conditional distribution given X, and P set of distributions on V. Even though some approaches explore calibration for regression tasks 48 588, it is most dominantly considered for classification. To distinguish between the general case and n-class classification, we refer to P0, n. as the n-dimensional simplex of corresponding categorical distributions.
A popular task is the calibration of the predicted top-label C = arg max; f& (X) of model f: 2¹ + Sn [4 30
51 [53.. Here, the top-label confidence should represent the accuracy of this very prediction. Formally, we try to reach the condition fc (X) P (Y = C fc (X)). However, the condition is weaker as one might expect. and referring to model fulfilling this condition as (perfectly) calibrated can give false sense of security [53] [57] This holds especially in forecasting domains, where low probability estimates can still be highly relevant. For example, assigning probability mass to an aggressive type of cancer can still trigger action even if it is not predicted as the most likely outcome. Further, we might also interested in predictive uncertainty for non- classification tasks. Consequently, prediction f (X) should be equal to the complete conditional distribution РYTS(X) of the target given this very prediction introduced by Widmann et al. [58] More formally, we state:
use the stricter and more general condition that the complete
Definition 2.1. A model f: 2 + S is calibrated if and only if f (X)= = s())
Note that f can be a set of arbitrary distributions, which incorporates 9n (classification) as a special case.
One of the first metrics for assessing model calibration that is still widely used in recent literature is the Brier score (BS) 16
For model f: 2. + Pn the Brier score [ is defined as
BS (f) = E [x)
(X) - r'
(1)
where y" is one-hot-encoded Y. The estimator of the BS is equivalent to the mean squared error, illustrating that it does not purely capture model calibration. Rather, the BS can be interpreted as a comprehensive measure of model performance, simultaneously capturing model fit and calibration. This becomes more obvious via the canonical decomposition of the BS into calibration and sharpness term BS Based on this decomposition, we can derive the following error. For 1 S p E R, the LP calibration error (CEp) of model f: 2 + Pn is defined as
CEp (1)
E [x)
The BS decomposition only supports the squared case, but a general T. formulation became more
Popordanoska et al. 43 proposed to estimate CFpp via multivariate kernel density estimation. In general, calibration estimation is difficult due to the term Py)s(X) since we never have samples of every possible prediction for continuous models. This is in contrast to the original work of Murphy [3 where only models with finite prediction space are considered. To assess the calibration a continuous binary model Platt 42 used histogram estimation, transforming the infinite prediction space to a finite one. This is also referred to equal width binning. Similarly, Nguyen & O' O'Connor 40 introduced an equal mass bimning scheme for continuous binary models. Both, equal width and equal mass binning schemes, suffer from the requirement of setting a hyperparameter. This can sigmificantly influence the estimated value [3] and there is optimal default since every setting has different bias-variance tradeoff [41]. The first. calibration estimator for a continuous one-vs-all multi-class model was given by Naeini et al. (39 and is still the most commonly used measure to quantify calibration. It is referred the expected calibration error (ECE) of model f: 2.' + P, and defined as
common in recent years + 5%
57
x
m
ECE ( Σ
pi /conf; acc;
(3)
with the bin frequency Ps = P (X) E B,), the bin-wise mean confidence conf, E [fo (X) fo (X) e Bi], and the bin accuracy acc; = P (Y C (X) B.), for
3

---

bins B; =
(-= We can estimate pt, conf,. and acc; via the respective means. These are then used in equation to estimate the ECE. This estimator biased [31 53
[-1
m
Kull et al. 29 and Nixon et al. independently introduced another calibration estimator, which also captures the extend to which the condition TP (Y = k: fu (X)) = f& (X) is violated for each class k E y They respectively use equal width and equal mass binning. Similar to these estimators, Kumar et al. 33I introduced the class-wise calibration error (CWCE,) and, similar to the ECE, the top-label calibration error (TCEp). They only formulated the squared case p = 2 but suggested the extension of their definitions to general p-norms, which we provide in Appendix C
Furthermore, Kumar et al. 31 and Vaicenavicius et al. 3 proved independently that using a fixed binning scheme for estimation leads to lower bound of the expected error. Zhang et al. [63 circumvent binning schemes by using kernel density estimation to estimate the TCFp
Orthogonal ways to quantify model miscalibration have been proposed to not depend on binning or kernel density estimation schemes. Gupta et al. T6 introduced the Kolmogorov-Sminn -mmiroy calibration error (KS) (c.f. Appendix C which is based the Bolmogorov-Smimoon test between empirical cumulative distribution functions. Its estimator does not require setting a hyperparameter but the authors did not provide further theoretical aspects.
Estimators of the TCEp and CWCE, are in general not differentiable. Consequently, Kumar et al. BC proposed the Maximum mean calibration error (MMCE) (c.f. Appendix C which has a differentiable estimator. It is a kernel-based error, comparing the top-lahel confidence and the conditional accuracy, similar to the ECE.
Widmann al. 157 argued that the MMCE is insufficient for quantifying calibration of model, similar as the ECE. They further proposed the Kernel calibration error (KCE) (c.f. Appendix Cc. It based on matrix-valued kernels and unlike the MMCE, which only uses the top-label prediction, includes the whole model prediction. The squared KCE has an unbiased estimator based on a U-statistic with quadratic runtime complexity with respect to the data size. However, even though the KCE is positive, the U-statistic estimator can give negative values. To this end, the authors use the estimated value test statistic w.r.t. the null hypothesis that the model calibrated.
Furthermore, Widmann et al. 57 and Widmann et al. (58) proposed to unify different definitions of calibration errors in theoretical framework, which also includes variance regression calibration. However, the framework allows calibration errors, which are zero even if the model is not calibrated all.
2.2 Recalibration
A plethora of recalibration methods have been proposed to improve model calibration after training by transforming the model output probabilities [14] 16
18 28
60 []] 633
These methods are optimized on specific calibration set, which is usually the validation set. Key desiderata of these methods include for the algorithms to be accuracy accuracy-preserving and data-efficient
63, reflecting that typical use-cases include settings in sensitive domains where accuracy should remain unchanged and often little data is available to train and evaluate the models. Such accuracy- preserving methods only adjust the probability estimate such a way that the predicted top-label remains the same. The most commonly used accuracy- preseervinn recalibration method is temperature scaling (TS) [. where the model logits are divided by single parameter T E E> before computing the predictions via softmax. A more expressive extension of TS is ensemble temperature scaling (ETS) [63], where a weighted ensemble of TS output, model output, and label smoothing is computed. Rahimi et al. A5] proposed different classes of order- preserving transformations. A specifically interesting one is the class of diagonal intra order-preserving functions (DIAG). Here, the model logits are transformed elementwise with scalar, monotonic, and continuous function, which represented by neural networks of unconstrained monotonic functions (55)
2.3 Proper scores
Greiting& & Raftery [K3 give an extensive overview of proper scores. Unfortunately, their presented definitions assume maximization as the model training objective. To stay in line with recent machine learning literature, we flip the sign when it required in the following definitions, similar as in 4 We specifically do not constrain ourselves to classification, which is special case. Assume give

---

prediction in 9 for an event in V/ and we want to score how good the prediction was. A function S P x y + R with I = = ( 20) is called scoring rule or just score. Examples are the Brier score and the log score for classification, or the It is defined as S (P. = [ + log To compare
score (DSS), which extends
the MSE for variance regression
distributions, we use the expected score sg: P x 3 + R defined as s9 of
A scoring rule S is defined to be proper if and only if () 2 ss (Q holds for all P.Q P.
and strictly proper and only if P * Q = 8y (P.Q) > sy QQ. . In other words, a score is proper if predicting the target distribution gives the best expected value and strictly proper if no other prediction can achieve this value. Given a proper score S and P. E P, the associated divergence dg: 3 30 - Rzo is defined as dg (P,
$g (P: Q) - 8y (Q, Q) and the associated generalized entropy gs: : P + R as gs (Q) = sy (Q, Q). For strictly proper S, ds is only zero
P Q: for (strictly) proper S, gs is (strictly) concave. An example of divergence-entropyy combination is the Kullback-Lend Leibler divergence and the Shannon entropy associated to the log score.
Associated entropies and divergences used in the calibration-sharpnae decomposition introduced by Bröcker [ for proper scores of categorical distributions. In Lemma 4.1 we will prove that this result holds for proper scores of arbitrary distributions, long as additional conditions are met. Further, associated divergences will be the backbone for our definition of proper calibration errors in Section 4
3 Theoretical analysis of calibration errors
In this section, we present theoretical results regarding calibration errors used in current literature. First, we place these calibration errors into taxonomy, which we introduce in Theorem 3. Next. we give an example of how much errors lower in the hierarchy can differ from CE2 in Proposition 3.2
Last, we analyse the ECE estimate with respect to the data size in Proposition 3.3 This proposition the basis for explaining the empirical bhess-)behaviour of the ECE observed in Section S are presented in Appendix D
All proofs
To the best of the authors' knowledge, other publications provided relations between at most two distinct calibration errors none at all while introducing new one. The taxonomy in the following theorem is a novel contribution, improving overview of a whole body of work regarding quantifying calibration.
Theorem 3.1. Given model f X - Pn and 1 1 R. we have
(f)= CEp -
TCEp (f) =
BS ( =
KCB (f) calibrated
CWCE, (1) =0
MMCE (f) 0
ECE ( 0,
KS (J) =
where statements inside curly brackets {...} are equivalent. Further, we have
KS ECE(J) (MMMME())
BS BS) CEp (1) 2 CWCE, ( 2 TCE, (J) 2 TCE, (0)
0,
where only holds for p < The kernel dependent constant e E R is given in Appendix D.2 according Kumar al. 130]
From this theorem follows that it is ambiguous to refer to perfect calibration just because there exists calibration error which is for model. The proof of this theorem also contains n - CE (/) CE (f) for p S q, which is a contradiction to Theorem demonstrate how large the gap between calibration errors can be.
(6). Next, we
Proposition 3.2. Assume the model f: V. + Pn, is surjective. There exists joint distribution of X and Y such that for all E E (MMCE, KS, ECE, TCEp, CWCE, 1 1 p R}:
1
E E(f) =0 and CE2 (f) 2 0.99
n
5

---

Recall that CB2 (f) = if and only if f is calibrated. In other words, most used calibration errors can be zero, but the model is still far from calibrated. An example of model transformation with perfect ECE but uncalibrated outputs given in Proposition D.2
0.100
0.075
ECE
0.050
Among all calibration errors, the ECE is still the most com- 15
0.025
monly used ICC
23
26
55
36
37
45
50
5
even
though its estimator is knowingly biased B1
46
Let ECE
0.000
denote the ECE estimator for n data instances as originally defined in Guo et al. [14]. The following gives an analysis how this estimate behaves approximately with respect to n and how this is further impacted by the ground truth ECE value. For simplicity, we omit the fixed model from the notation.
100
1000
10000
Data size
Figure 2: Estimated ECE of sim- ulated models with perfect cali- bration (blue), mediocre calibra- tion (orange), and bad calibration (green). Better calibration exac- erbates ECE bias with respect to the data size, leading to unreliably calibration improvement quantifica- tion.
ECE defined in Proposition 3.3. For M(n) Fo E [FCF(M) 2
Appendix D.S we have
dp(n) 0
d' (dn)² 0, and
dP dn dECE
0.
In words, the ECE estimator can be approximated by a differ- entiable function, which is strictly convex and monotonically decreasing in the data size. The smaller the data size, the larger the change of the function. Further, this change is also influ- enced by the true ECE value, such that, for low ground truth ECE, the function changes even more rapidly with the data size. Analogous results can be found for CWCE,, CWCF2, and TCE, with binning estimators as used Kull et al. (29), Nixon et al. 41 and Kumar et al. 31
To confirm the goodness of the approximation, we evaluated the ECE estimator simulated models in Figure 2 The models are designed such that their true level of calibration is known. Including the results of Figure M the empirical curves are consistent with Proposition
3.3
Further details on the
simulation are given Appendix
The results in this section motivate formal framework of proper calibration errors which are zero if and only the model is calibrated and with robust estimators.
Proper calibration errors
In this section, we introduce the definition of proper calibration errors. We provide an easy-to- estimate upper bound and investigate some properties. preliminary step, we generalize proper score decomposition. Again, all proofs presented in Appendix D
Bröcker 4 introduced calibration -hhrppesnns decomposition of proper scores w.r.t. categorical distributions. We extend this decomposition to proper scores of arbitrary distributions.
Lemma 4.1. Let D be a set of arbitrary distributions for which exists proper score with some mild conditions. For random variables Q and Y with Q, Ty,, PriQ E 3, we have the decomposition
E (Q, = gs (Py)
- E ds Prio) [ds (Q,Pr
generalized entropy
sharpness
calibration
Substituting Q with f (X) and 5 with the Brier score, the calibration term equals the previously defined CE2 of model f. Lemma 4.1 motivates the following definition, which introduce:
Definition 4.2. Given model f: 2 G. we say
CEs (/) = E ds (X). (x),()))))))
a (strictly) proper calibration error if and only ds a divergence associated with a (strictly) proper score S.

---

This gives CEBS = CE2 as an example of a strictly proper calibration error for classification since the Brier score is a strictly proper score on Pn, Strictly proper calibration errors have the highly desired property: CEs (f) =
is calibrated. Since proper scores are not restricted to classification, the above definition gives a natural extension of calibration errors beyond classification.
Additionally, by generalizing the definition of proper scores, we can show that the squared KCE is a strictly proper calibration error (Appendix F.. But, in general, there does not exist an unbiased estimator of a proper calibration error, since we cannot estimate [gs (Fy))(x)) in an unbiased manner. Because we do not want lower bounds for errors used in sensitive applications, we introduce the following theorem about how to construct an upper bound.
Theorem 4.3. For all proper calibration errors with in 9s (P) E IR, there exists associated calibration upper bound
U3 (f) CE3 (f)
defined as Us ( (X) - Pe gs (P). Under classification setting and further mild conditions, we have limAcc((s-11 Ws (f) CEs (f) = 0.
In other words, we can always construct a non-trivial upper bound of a proper calibration error as long as the generalized entropy function has finite infimum. The calibration upper bound approaches the true calibration error for models with high accuracy. Our proposed calibration upper bounds are provably reliable to use since they all have a minimum- variance unbiased estimator. In the following example, we derive the calibration upper bound Vss of the Brier Score.
Example 4.4. The scoring rule induced by the Brier score is defined Sns (ʃ (X), Ilf (X) - (XI'''I where y' is the one-hot encoding of Y. Using the definition of the associated entropy gives JBs (Q) Byy~ [Sus Y)] By-q ||I Y' To find its infimum, note that 11-11 2 0 and gas (11.0....,0) .)))....))).........)) = 0. Thus, inf pe Ias (P) = O, which gives 4s (f) = E (x) - y'] BS BS()) (f). This makes the Brier score itself an upper bound of its induced calibration error.
Additionally, Theorem 3.1 motivates the usage of Vas (f). Given dataset
{(X,, Yi) y..., (Xn, Yx)] and model f. we will estimate this quantity via V (f) V (x.) x2112 In general, any unbiased estimator becomes biased after non-linear
transformation t. since t [0) # t (E
But, if is continuous, our estimator is still asymptotically unbiased and consistent 449 We will further investigate the empirical robustness w.r.t. data size in Section with t as the square root and / (f) as the root calibration upper bound (RBS).
Furthermore, Ug has the following properties, which are helpful for the application of recalibration method optimization and selection.
Proposition 4.5. Given injective functions h, T P we have
Us (h o f) -ss (f) = CFs (h f) CEs (f) Us (h - f) > Us (h' o f) + CEs (h 0 f) CEs (h' o f) and (assuming S is differentiable)
d (h o f) dh
dCEs (h f) dh
This is generalization of Proposition 4.2 presented in 63 It tells us that we can reliably estimate the improvement of any injective recalibration method via the upper bound. Furthermore, we get access to the calibration gradient and can compare different transformations. At first, injectivity seems like significant restriction. But, we argue the following that injectivity rather than being eucuracy-preservee is desired property of general recalibration methods. For example, we can
follows from Continuous Mapping Theorem and Theorem 3.2.6 of Takeshi 40
7

---

3.0
CWCE22
0.6
100b TCE₂
0.5
25
15b CWCE2
15b d TCE2
Relative bias
0.4
2.0
KDE TCE2
0.3
KS
RBS (ours)
1.5
0.2
0.1
1.0
100
167
278
464
774
1292
2154
3594
5995
10000
100
167
278
774
1292
2154
3594
5995
10000
Test set size
Test set size
Figure 3: Left: Different calibration error estimates versus the test size of ResNet Wide 32 and CIFAR100. The red line corresponds to the square root of the Brier score which is an upper bound of the CE2. The other errors are lower bounds. Right: Relative change versus data size with respect to eлror at full size. Averaging across a multitude of models shows systematic trend. An unbiased estimator would give a flat line.
construct recalibration method. which is calibrated and accuracy preserving, but only predicts finite set of distinct values (see Appendix [). Specifically, we would only predict two distinct values for any input in binary classification. To exclude such naive solutions which substantially reduce model sharpness, we restrict ourselves to injective transformations of Sn + Pn. These do provably not impact the model sharpness and preserve, at least partly, the continuity of the output space. Examples of injective transformations are TS, ETS, and DIAG. These state-of-the methods show very competitive performances even when compared to non-injective recalibration methods Further, Proposition 4.5 also holds when replacing Us with the expected score 8y without further conditions. This is useful when Us does not exist, but we still want to perform recalibration as in Section 5 in the case of the DSS.
45 [x]
5 Experiments
In the following, we investigate the behavior of calibration error estimators in three settings. First, we use varying test set sizes for the estimators and compare their values. This will show how well the inequalities in Theorem 3.J hold in practical settings and how robust the estimators are. Second, we explore what the estimated improvements of several recalibration methods are. This is done after the recalibration methods are already optimized only vary the size of the test set and compute calibration errors on these test sets before and after recalibration. In both settings, the straighter a line is. the more robust and, consequently, trustworthy the estimator for practical applications. Third, we investigate how our framework can be used to improve calibration for tasks beyond classification by performing probabilistic regression with subsequent recalibration.
given validation set;
In all experiments we evaluate the following estimators: CWCE2 with 15 equal width bins ('15b CWCE3') ) CWCE2 with 100 equal width bins ('10 CWCE2") ) ECE with 15 equal width bins ('ECE") TCE, with 100 equal width bins (010 TCE2') ) TCE, with 15 equal mass bins and debias term (bb d TCE2') TCE₂ with kernel density estimation ('DDE TCE, ) KS ('KS') and the root calibration upper bound V ('RBS (ours)'). bin amounts are chosen based on past literature We also evaluate the KDE estimator of CE, ('KDE CE, ') with automatic bandwidth conducted across several model-dataset combinations, for which logit sets are openly accessible 29 45 This includes the models Wide
B1 [6
selection based on 43 for CIFAR10. The experiments
ResNet 32 62 DenseNet 40, and DenseNet 161 (2) and the datasets CIFARIO, CIFAR1O0 27
and
https://github. com/ wttc/ooimmomaaaaeeeee /morooR/

---

and ImageNet 11 We did not conduct model training ourselves and refer to 12 and 45 for further details. We include TS, ETS, and DIAG as injective recalibration methods. Further details and results on additional models and datasets are reported in the Appendix G
Robustness of calibration errors to test set size We illustrate the estimated values of our in- troduced upper bound and the other errors, which are lower bounds of the unknown CE2 on the left of Figure On the right, we aggregate across several models to show the systematic drop-off according to Proposition 3.3 The relative bias is computed by Error(n)/ Error Error(10000) and allows an aggregation of models with different calibration levels. Included models are DenseNet 40, Wide ResNet 32, ResNet 110 SD, ResNet 110, and LeNet 5, all trained on CIFARIO 10. All values represent the calibration of the given model without recalibration transformation. Only our proposed upper bound and KS stable, and Appendix C shows this holds across wide range of different settings. The theoretically highest lower bound (CWCE, with 100 bins) is also constantly the highest estimated lower bound, but it is sensitive the test set size. Results for further settings presented in Appendix show similar results.
Quantifying recalibration improvement Next, we assessed how well all estimators were able to quantify the improvement in calibration error after applying different injective recalibration methods (Fig. D. Only our proposed upper bound estimator RBS is again robust throughout all settings. According to Proposition 4.5 and since RBS is asymptotically unbiased and consistent, it can be regarded as reliable approximation of the real improvement of the presented recalibration methods. For all other estimators, there is general trend to estimate recalibration improvement higher for large test set sizes. In other settings, especially for small test set sizes, calibration improvement is underestimated to the extent that negative improvements (poorer calibration than before) are suggested. Results on other settings presented in Appendix G show similar results. Taken together, these experiments demonstrate the unreliability of existing calibration estimators, in particular, when used to evaluate recalibration methods. In contrast, our proposed upper bound estimator is stable across different settings.
Variance regression calibration We consider variance regression to demonstrate the usefulness of proper calibration errors outside the classification setting. To this end, predict sales prices with an uncertainty estimate in the UCI dataset Residential Building, which consists of a high feature (107) to data instances (372) ratio [44] Our model of choice is fully-connected mixture density network predicting mean and variance 2 Similar to classification, we are interested in recalibration of the predicted variance to adjust possible under- overconfidence. We use our proposed framework to derive a proper calibration error induced by the proper score DSS for recalibration. Further, we compare DSS [2] to squared KCE (SKCE) (58) and analyze the average predicted variance throughout model training. We use Platt scaling (xr + wx + with parameters w. b E R (22) on the predicted variance in each training iteration to show how recalibration benefits uncertainty awareness. We expect high uncertainty awareness at the start of the model training with a drop- off later iterations. As can be seen in Figure 4 recalibration is able adjust the uncertainty estimate of a model as desired. Further, the DSS estimate, which captures predicted mean and variance correctness, directly communicates the improved variance fit. Contrary, the SKCE estimate appears more erratic between iteration steps and seemingly ignores the changes in variance and the recalibration improvement.
One might also be interested at how the predicted variance corresponds to the mean squared error throughout model training. As we can see in Table [ only after calibration using proper calibration error, the average predicted variance (Avg Var) corresponds to the mean squared error.
Next, assess how well the predicted variance corresponds to the instance-level error, compute the ratio between the squared error of the predictive mean jI and the predicted variance o² (SE
Var Ratio) for each individual sample via 1 D (#L- .An SE Var Ratio of '1' for given of
instance means that the predictive uncertainty (variance) exactly matches the squared error, an SE Var Ratio of '10' means that the model is overconfident and the squared error is ten times as large as the predicted variance. As we can see in Table 2 recalibration through our framework gives consistently conservative estimates on the squared error, whereas the uncalibrated uncertainties highly overconfident (with errors more than ten times larger than the prediction).
We perform further variance regression experiments in Appendix G

---

1e7
45
1e-5
3.5
Model
3.0
Uncalibrated
40
Mean predicted variance
Callbrated
2.5
35
2.0
DSS
SKCE
30
1.5
25
1.0
20
0.5
15
0.0
500
1000
1500
2000
500
1000
1500
2000
500
1000
1500
2000
Training iteration
Training iteration
Training iteration
Average predicted variance throughout model training before and after recalibration. Initially, due to bad fit, recalibration adjusts the variance accordingly for better communicated uncertainty. Once the model fit improves, the predicted variance requires less adjustment due to less uncertainty in each prediction. Middle: DSS communicates reasonably changes in the variance due to recalibration. Right: SKCE fails to capture the variance trend and behaves erratically.
Figure 4: Left:
Table I: Comparing the mean squared error (MSE) with the average predicted variance (Avg Var) before and after recalibration for various training iterations. Recalibration gives a better average match between prediction and real error.
Iteration
500
750
1000
1250
1500
1750
2000
MSE
10.48
5.87
4.3
3.51
3.12
2.74
2.57
Avg Var (Calibrated)
11.04
6.89
5.4
4.55
4.11
3.63
3.32
Avg Var (Uncalibrated)
0.83
0.83
0.83
0.83
0.83
0.82
0.82
6 Conclusion
In this work, we address the problem of reliably quantifying the effect of recalibration on predictive uncertainty for classification and other probabilistic tasks. This critical for adjusting under- or overconfidence via recalibration. To this end, we first provide a taxonomy of existing calibration errors. We discover that most errors are lower bounds of a proper calibration error and fail assess if a model is calibrated. This motivates our definition of proper calibration errors, which provides a general class of errors for arbitrary probabilistic predictions. Since proper calibration errors cannot be estimated in the general case, we introduce upper bounds, which directly measure the calibration change for injective transformations. This allows us reliably adjust model uncertainty via recalibration. We demonstrate theoretically and empirically that the estimated calibration improvement can be highly misleading for commonly used estimators, including the ECE. In stark contrast, our upper bound is robust changes in data size and estimates robustly the improvement via injective recalibration. We further show additional experiments that our approach can be applied successfully to variance regression.
Table 2: Instance level ratio between the squared error and the predicted variance before and after recalibration for various training iterations. Recalibration improves the instance level prediction of the squared error.
Iteration
500
1000
1500
2000
SE Var Ratio (Calibrated)
2.12
0.79
0.79 2.56
0.79 # 2.59
SE Var Ratio (Uncalibrated)
11.33
30.72
5.36
15.09
4.07 12.13
3.51 # 11.22
10

---

References
Aitchison, J. and Shen, S. M. Logistic-normal distributions:
distibrrttotaiiiisiituuttttssss Some properties and uses. Biometrika, 67(2):
261-272. 1980. ISSN 00063444. URL http://wwww...ottooooo///aaaaaa/
Bishop, C. Mixture density networks. 1994.
Brier, W. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):
3, 1950. doi: 10. 10111777555000000111100000010010 2O0p://000000000000000000000000.
URL. httppppaa////oooorraaaaaaaaa
/rt. http:////....a.o..a.oo.o..oo.... 078__000__ofeiit_22_...ooo
amstaoc.orrg/veww/
[4] Bröcker, J. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal
Meteorological Society, 135(643):1512 1511-111991,,19999,9 Jul 2009. ISSN 1477-870X. doi: 10.1002/qj.456 URL
http://dx. doi org/ 1atgg/1..0 1002/g1 456
[5] Capiński, M. and Kopp. P. E. Measure, integral and probability, volume 14. Springer, 2004.
[6] Chung, Y., Neiswanger, W., Char, I., and Schneider, J. Beyond pinball loss: Quantile methods for calibrated uncertainty quantification, 2021.
Dawid, A. P. The well-calibrated bayesian. Journal of the American Statistical Association, 77(379): 605-610, 1982. doi: 10. 10.10800016214550 1459. 198449999.19.... 19t..19....71111 10477856. URL https::ss/// htw.///mlw.ataaddmmli.i.i. com/ /oi/ 1bs/10.1080/01622 108000062//0190 1982. 10477856
[8] Dawid, A. P. and Sebastiani, P. Coherent dispersion criteria for optimal experimental design. Annals of Statistics, pp. 65-81, 1999.
[9] Deng, J., Dong. W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
[10] Fan, H., Ferianc, M., Que, Z., Niu, X.. Rodrigues, M. L., and Luk, W. Accelerating bayesian neural networks via algorithmic and hardware optimizations. IEEE Transactions Parallel and Distributed Systems, 2022.
[11] Friedman, Multivariate adaptive regression splines. The annals of statistics, 19(1):1667, 1991.
[12] Gneiting, T. and Katzfuss, M. Prohahilistic forecasting. Annual Review Statistics and Its Application, 1: 125 151, 2014.
[13] Gneiting, T. and Raftery. E. Strictly proper scoring rules, prediction, estimation. Journal of American Statistical Association, 102(477):359 378, 2007, doi: 10. 10.11980012244500000014331 URL https://dot org/ 10. 1118//01662155000000133333
[14] Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In International Conference Machine Learning, pp. 1321-1330. PMIRR 2017.
[15] Gupla, A., Kvernadze, G., and Srikumar, V. Bert & family eat word salad: Experiments with text understanding. ArXiv, abs/2101.03453 2021.
[16] Gupta, K., Rahimi, A.. Ajanthan, T., Mensink, T., Sminchisescu, C., and Hartley, R. Calibration of neural networks using splines. In International Conference Learning Representations, 2020.
[17] Haggenmüller, S., Maron, R. C., Hekler, Utikal, J. S., Barata, Bamhill,
Beltraminelli, H., Berking, C., SttzeStablein, B., Blum, A., Braun, S. A., Carr, R., Combalia, M., Feruunde--Figueras,
M.-T,, Ferrara, G., Fraitag, S., French, L. E., Gellrich, F. F., Ghoreschi, K., Goebeler, M., Guitera, P., Haenssle, H. A., Haferkamp. S., Heinzerling, L., Heppt, M. V., Hilke, F. J., Hobelsherger, S., Krahl, D.. Kutzner, H., Lallas, A., Liopyris. K., Llamas- Velasco, M.. Malvehy, J., Meier. F., Müller. C. S., Navarini, A. A.. Navarrete- Decceen, C., Perasole, A., Poch. G., Podlipnik. S., Requena, L., Rotemberg. V. M., Saggini, A., Sangueza, O. P., Santonja, C., Schadendorf, D., Schilling, B., Schlaak, M., Schlager, J. G., Sergon, M., Sondermann, W., Soyer, H. P., Starz, H., Stolz, W., Vale, F.,, Weyers, W., Zink, A., Krieghoff- Hennnigg- F., Kather, J. N., von Kalle, C., Lipka, D. B., Fröhling, S., Hauschild, A., Kittler, Skin cancer classification via convolutional neural networks: systematic review of studies involving human experts. Eurupean Journal of Cancer, 156:202- 2021. ISSN 0959-8049. doi: https::/doi.org//// wtt.:/ooooo/....1...... http:///00110////.......00 0.6.049, URL https: httpe:////wwccciecceireettt htt.44//...aatttp/////ssaaaaaaae
H., and Brinker,
articleep31/S0959800000
[18] Hastie, T. and Tibshirani, R. Classification by pairwise coupling. The annals of statistics, 26(2):451-47 1998.

---

[19] He, K.. Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance imagenet classification, In Proceedings of the IEEE international conference computer vision. pp. 1026-1034, 2015.
[20] Hc, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-770077 770-778, 2016.
[21] Hoeftding, W. A Class of Statistics with Asymptotically Normal Distribution. The Annals of Mathematical Statisticx, 19(3):293 325, 1948. doi: 10.1214/aoms/ 177730196. URL httpss/:://do http:::///oo..rg//111/ org/10. 1214/ aoma/ 1078//17773019
[22] Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q, Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4700-4708, 2017.
[23] Islam, A.. Chen, C.-F,, Panda, R., Karlinsky, L., Radke, J., Feris, S. A broad study on the transferability visual representations with contrastive learning. 202/ IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8825-8835, 2021.
[24] Joo, T., Chung, U., and Sco, M. Being bayesian about categorical probability. In ICMI,, 2020.
[25] Katsaouni, N., Tashkandi, A., Wiese, L., and Schulz, M. Machine learning based discase prediction from genotype data. Biological Chemistry, 402(8):871-888)) 1885-88, 2021. doi: 10.55 2021-0109, URL https://dos org/ 10. 1515/hsz 10511//sz-2221-11151 2021-0109
[26] Kristiadi, A., Hein, M., and Hennig, P. Being bayesian, even just bit, fixes overconfidence in relu networks. In ICMI., 2020.
[27] Krizhevsky, A. et al. Leamingg multiple layers of features from tiny images. 2009.
[28] Kull, M., Filho, T. S., and Flach, P. Beta calibration: a well-founded and easily implemented improvement logistic calibration for binary classifiers. In Singh, A. and Zhu, J. (eds.). Proceedings of the 20th International Conference Artificial Intelligence and Statistics, volume 54 of Proceedings Machine Learning Research, pp. 623-631. PMLR, 20-22 Apr 2017. htttttttp//:poceeeeet
mlr press/
v54/kull17a.hte
[29] Kull, M.. Perello Nieto, M.. Kängsepp. M.. Silva Filho, T. Song. H., and Flach, P. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in Neural Information Processing Systems, 32:12316-12326, 2019.
[30] Kumar, A., Sarawagi, S., and Jain, U. Trainable calibration measures for neural networks from kernel mean embeddings. In Dy, J. and Krause, A. (c Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2805-2814, PMLR, 10-15 Jul 2018. URL httppss//pproceeeinngg mlr. www.mmrrrrress..... www.spreesa.... ppresa/800/kuaa11850 /aa.rraaaaaaaaaaaaa..a html
[31] Kumar, A., Liang, P., and Ma, T. Verified uncertainty calibration. Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 3792-3803, 2019.
[32] LeCun, Y., Bottou, I.., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):227-2333 1998.
[33] Liu, C., Zoph, B., Neumann, M. Shlens, J., Hua, W., Li, L.-., Fei-Fei, L.,, Yuille, A., Huang, J., and Murphy, K. Progressive neural architecture search. In Proceedings of the European conference on computer vision (ECCV), pp. 19-34, 2018.
[34] Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. and Wilson, G. A simple baseline for bayesian uncertainty deep learning, Advances Neural Information Processing Systems. 32:13153 13164, 2019.
[35] Menon, A. K., Rawat, A. S., Reddi, S. J., Kim, S., and Kumar, S. A statistical perspective on distillation. In ICML, 2021.
[36] Minderer, M., Djolonga, J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N.. Tran, D., and Lucie, M. Revisiting the calibration modern neural networks, Advances Neural Information Processing Systems, 34, 2021.
[37] Morales-Alvarez. P., Hernández- D., Molina, R., and Hernández-Lobato, J. M. Activation level uncertainty in deep neural networks. In ICLR, 2021.
12

---

[38] Murphy, A. H. A new vector partition of the probability score. Journal of Applied Meteorology and Climatolgyy, 12(4):595 600, 1973. doi: 101.77//12.444501777771000511111
2. URL
0595_ let___avoo_____cco_o xml
[39] Nacini, M. P.. Cooper. G. F., and Hauskrecht, M. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI'15, pp. 2901-2907. AAAI Press, 2015. ISBN 2262511290.
[40] Nguyen, K. and O'Connor, B. Posterior calibration and exploratory analysis for natural language processing models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1587- 1598, 2015.
[41] Nixon, J., Dusenberry. M. W. Zhang, L.,, Jerfel, G., and Tran, D. Measuring calibration in deep learning. In CVPR Workshops, volume 2019.
[42] Platt. J. C. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances Marge-Margin Classifiers, pp. 61-74. MIT Press, 1999.
[43] Popordanoska, T., Sayer, R., and Blaschko, M. B. A consistent and differentiable Ip canonical calibration error estimator. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho. K. (eds.), Advances in Neural Information Processing Systems, 2022. URL http:://ppeeeeeeeiees httpps///ppprreeeee..eeee/////// aett/torumiid-iMeppp///
[44] Rafiei, M. H. and Adeli, H. A novel machine learning model for estimation of sale prices of real estate units. Journal of Construction Engineering and Management, 142(2)04405006 2016.
[45] Rahimi, A., Shaban, A., Cheng. C.-A., Hartley, R., and Boots, B. Intra order- auder-preserving functions for calibration of multi-class neural networks. Advances Neural Information Processing Systems, 33: 14466-1367, 2020.
[46] Roelofs, R., Cain, N., Shlens, J., and Mozer, C. Mitigating bias in calibration error estimation, 2021.
[47] Schritwieser, J., Antonoglou, Hubert, T., Simonyan. K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D.. Graepel, T., et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839)6044-09, 2020.
[48] Song, H., Diethe, T., Kull, M., and Flach, P. Distribution calibration for regression. International Conference on Machine Learning, pp. 5897-5906. PMLR, 2019.
[49] Takeshi, A. Advanced econometrics. Harvard university press, 1985.
[50] Tian, J., Yung, D., Hsu, Y.C.., and Kira, 7., A geometric perspective towards neural calibration via sensitivity decomposition. NeurlPS, 2021.
[51] Tomani, C., Gruber, S., Erdem, M. E., Cremers, D., and Buettner, F. Post-hoc uncertainty calibration for domain drift scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10124-10132, June 2021.
[52] Tsagris, M., Beneki, C., and Hassani, H. On the folded normal distribution, Mathematics, 2(1):12-28, feb 2014. doi: (0.3390/math2010012 URL htttps://dot org/ro///00 10. 3390%.2Fmath210001.
[53] Vaicenavicius, J., Widmann, D., Andersson, C., Lindsten, F., Roll, J., and Schön, T. Evaluating model calihration in classification. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 3467. PMLR, 2019.
[54] Wang, X., Liu, H., Shi, C., and Yang, C. Be confident! towards trustworthy graph neural networks via confidence calibration. In NeurlPS, 2021.
[55] Wehenkel, A. and Louppe, G. Unconstrained monotonic neural networks. Advances in Neural Information Processing Systems, 32: 155555555 2019.
[56] Wenger, J., Kjellström, H., and Triebel, R. Non- parametric calibration for classification. In International Conference on Artificial Intelligence and Statistics, pp. 178-190. PMLR, 2020.
[57] Widmann, D., Lindsten, F., and Zachariah, D. Calibration tests in multi-class classification: A unifying framework. Advances in Neural Information Processing Systems, 32: 12:17227 2019.
[58] Widmann, D., Lindsten, F., and Zachariah, D. Calibration tests beyond classification. In International Con- ference Learning Representations, 2021. URL Stttpp///ppomerrieeeee///oorttte 3xf89wMMx
13

---

[59] Yen, M.-H., Liu, D.-W., Hsin, Y.-.., Lin, C.-E., and Chen, C.-C. Application of the deep learning for the prediction of rainfall in southern taiwan. Scientific reports, 91):119, 2019.
[60] Zadrozny, B. Elkan, Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. ICML, 1, 05 2001.
[61] Zadrozny, B. and Elkan, C. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the Eighth ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '02, pp. 694-699, New York, NY, USA, 2002. Association for Computing Machinery. ISBN 1581135673 13567X. doi: 10.11457775047.775555
StRLhhttps://ao http////ooorr///////// 1ttg//00.114/775444 .751111
[62] Zagoruyko, S. and Komodakis, N. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
[63] Zhang, J., Kailkhura, B., and Han, T. Y.J. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In International Conference on Machine Learning, pp. 11117- 11128. PMLR, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We mention required conditions for the proofs either directly or refer to Appendix D
(c) Did you discuss any potential negative societal impacts of your work? [No] We see positive societal impacts from improved uncertainty awareness via recalibration.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes] Major condi- tions are directly stated; minor conditions in Appendix D as referred.
(b) Did you include complete proofs of all theoretical results? [Yes] In Appendix D
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] Full code is located in the supplementary material and further experimental details are in Appendix
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Relevant training details are located in Appendix G A lot of models are not trained by ourselves. Instead, we loaded their results from publicly accessible URLs. We refer to each URL and the original work, where the training was conducted.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We repeated experiments until the error bars are barely visible or not visible anymore. Only exception is Figure 4i where aggregations hinder visibility due high variances in training different seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix G
4. If you using existing assets (c.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We used assets from 29 and Rahimi et al. [4]] located
and https://github.co
(b) Did you mention the license of the assets? [No] Each lincense can be viewed in the
respective github repository.
14

---

(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We include the code for reproducing the experiments and figures in the supplementary material.
(d) Did you discuss whether and how consent obtained from people whose data you're 're using/curating? [No] The data publicly accessible and we cited each respective source.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] The data is often abstract in nature (we used pretrained logits provided by publicly accessible source) or downloaded the dataset from the UCI dataset database.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] No crowdsourcing human subjects
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A] No crowdsourcing or human subjects
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] No crowdsoureing human subjects
15


Reviews:
[{"review_id": "b_b5KOYk2A", "reviewer": "Reviewer_bU5e", "summary": ". The paper makes a theoretical treatment of how calibration error is estimated, how biased these estimations are, proposes a taxonomy relating most calibration errors, argue that most calibration errors are lower bounds of the true calibration error, and propose upper bounds that can be estimated efficiently. The paper additionally proposed the concept of proper calibration errors, which are related to proper scoring rules, and are proper in terms that a zero proper calibration error results in a calibrated model.\n\nThe contributions are:\n- A taxonomy and categorization of calibration errors and how they relate to each other.\n- A definition of proper calibration errors, considering that most well known calibration errors used to measure miscalibration, are not proper.\n- A proposal for an estimable upper bound for proper calibration error, since most proper calibration errors are intractable to compute.\n- Experimental results that shows how practice matches theory, particularly about properly estimating calibration error without bias.", "questions": "- What is RBS in detail? I believe that it is not explicitly described in the paper (other than Line 292).", "limitations": "The paper does not describe negative societal impacts and/or limitations, and I do not believe this is required. There are positive societal impact that comes with better calibrated models and uncertainty quantification in general.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- This paper touches an important topic, we usually use ECE and other calibration errors as metrics, but these are biased (they have parameters that can be tricked like number of bins), and this paper provides a good theoretical analysis and formulation on how calibration errors are related, and the fact that zero calibration error does not guarantee a calibrated model due to calibration error estimation biases and other issues. This work makes a good theoretical analysis of how these calibration errors are related, and some possible reasons on why these calibration errors are biased in estimating the true calibration error.\n- This paper proposes the concept of proper calibration errors, which I find a good generalization from proper scoring rules, to have calibration errors with guarantees that the model will be calibrated. I believe this is a good contribution to the state of the art and is novel.\n- Proper calibration errors are untractable to compute, and this paper proposes the calibration upper bound which is asymptotically unbiased and tractable to compute given some assumptions.\n- The evaluation is made with a good selection of calibration errors from the literature, on a good selection of models (ResNet, Wide ResNet), and datasets/tasks, mainly on CIFAR10/100 and ImageNet for classification, and UCI Residential Building for regression. I like that the paper has results on both classification and regression.\n- This work evaluates how the calibration error and its bias changes with the size of the test set, this is often not evaluated, and only a single test set size used, which I believe is a good contribution to the literature, and clearly shows the estimation bias of calibration errors.\n- The supplementary material is pretty long/detailed and has all the proofs for the lemmas/theorems in the main paper, as well as additional results and reproducibility details. I have not checked the proofs in the supplementary.\n- The paper is well written and is good but not so easy to read. It is heavy on statistical details, but overall it is readable and understandable, I only have some minor details on notation in Minor Issues.\n\nWeaknesses\n- I believe that the only Weaknesses of the paper is some missing notation and that the mathematical details might be a bit too much for machine learning practitioners, see Minor Issues for more details. Having a section like takeaways for practitioners would be the best (for example, how to compute upper bound calibration error in an algorithmic way for some commonly used proper calibration errors).\n\nMinor issues\n\n~~- What exactly is the RBS? The paper defines it as root of U_BS, but I believe U_BS is not completely defined, my guess is that it is the upper bound of the brier score? Clarification is needed.~~\n\n- Some notation required to read the paper is available in the supplementary,  it would be best to have some of this notation in the main paper for easier readability, for example P_{Y | f(X)} and others.\n- One possible suggestion about the literature is the paper [1] which shows expected calibration error with variations of the training set size, which can be compared with this papers' variations of the test set, it is not the same but can be related. I do not suggest this as a reference but only as further research and/or general interest.\n- The figures in the paper look blurry in HiDPI screens and with zooming, I think this is because the figures have been produced as raster images (PNG or JPG), if you produce the figures directly as PDF, they will be vector based and easily zoomable and higher quality.\n\n[1]: https://arxiv.org/abs/2111.0980", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RunfpzHBC7P", "reviewer": "Reviewer_u81U", "summary": "The paper studies the reliability of common calibration estimators used to assess quality of the uncertainty estimates of machine learning models. Using the proposed framework of “proper calibration errors”, the authors create a taxonomy of calibration estimators, providing bounds on their performance.  Specifically, the framework defines proper calibration error as the expected divergence between the model predictions and true target distribution. Here, the divergence comes from a popper scoring rule. The suggested improvements based on this framework are shown to be effective on several practical applications.", "questions": "See Strengths And Weaknesses section.", "limitations": "See Strengths And Weaknesses section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The paper addresses a very important issue of reliability of calibration metrics especially in low data regime. It is shown that in extreme cases, a model with perfect calibration metric value can be severely miscalibrated. Overly optimistic estimation of calibration error can lead to false trust in the model’s uncertainty estimation quality and can lead to errors that can be quite risky in critical applications.\n- The paper is easy to read. The taxonomy of calibration estimators is quite useful.\n- Interesting theoretical analysis that connects various calibration estimators to proper scores. This connection helps provide some guarantees on the calibration estimation and helps generalize the estimator to non-classification settings.\n- Tackles the more stricter notion of calibration than the typical top-class calibration in the classification setting.\n\nWeaknesses \n\nIn critical applications, apart from calibration over a test set, we are interested in the reliability of the confidence score at individual instance level. It is not clear if the upper-bound to the proper calibration error is particularly useful for this. How well would this metric correlate with selective prediction performance or be suitable to assess out-of-distribution detection performance?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KCS3W1DWse", "reviewer": "Reviewer_7EUs", "summary": "The paper introduces a framework of proper calibration errors proposing an upper bound as a reliable estimator for recalibration. The manuscript provides a comprehensive overview of calibration error and post-hoc calibration literature. This work also shows the popularly used calibration estimators are highly sensitive to the test-set size, while the proposed upper bound is robust to changes in data size.", "questions": "* In Figure 1 and Figure 3, is the validation set size that is used to recalibrate the model is constant irrespective of test-test size? Is the test-set i.i.d and uniformly drawn from test data distribution for evaluation in sensitivity experiments (Fig 3)?\n* Can the authors explain the basis for how models with perfect calibration, mediocre calibration and bad calibration defined in ECE estimator simulation in Figure 2?\n*  What is the ResNet-Wide-32 model’s accuracy on CIFAR100 used for evaluation of calibration estimator sensitivity to test-set size in Figure 3? I suggest accuracy with different test-set size is plotted in Fig 3 to compare how well the calibration estimators correlate with the model accuracy.  ", "limitations": "not applicable.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* A strong related work section with comprehensive literature review on calibration error and model recalibration.\n* The experiments show the proposed upper bound is stable and not sensitive to test dataset size.\n\nWeaknesses:\n* There is no theoretical justification why the existing calibration estimators are sensitive to test-set data size if they are i.i.d. and drawm uniformly in random from test data distribution.\n* Given the Brier score is in range (-\\infty, \\infty), it is not clear how can square root of negative number (not real number) can be used as proper calibration error.  Or is there any approximation or rescaling performed to the upper bound?\n* The term “root calibration upper bound (RBS)” for the proposed method is introduced in the Experiments section but could not find it in Section 4 where proper calibration errors are discussed. I suggest the paper introduce this term and its properties in Section 4.\n* Reliability diagram and miscalibration area can be a better evaluation method to compare different calibration estimators to align with the actual definition of calibration. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept