A Consistent and Differentiable Lp Canonical Calibration Error Estimator
Authors: Teodora Popordanoska
Raphael Sayer
Matthew B. Blaschko
Venue: NeurIPS 2022 Accept (2022)
Keywords: uncertainty calibration
calibration error estimator
dirichlet kernel density estimation

Abstract: Calibrated probabilistic classifiers are models whose predicted probabilities can directly be interpreted as uncertainty estimates. It has been shown recently that deep neural networks are poorly calibrated and tend to output overconfident predictions. As a remedy, we propose a low-bias, trainable calibration error estimator based on Dirichlet kernel density estimates, which asymptotically converges to the true $L_p$ calibration error. This novel estimator enables us to tackle the strongest notion of multiclass calibration, called canonical (or distribution) calibration, while other common calibration methods are tractable only for top-label and marginal calibration. The computational complexity of our estimator is $\mathcal{O}(n^2)$, the convergence rate is $\mathcal{O}(n^{-1/2})$, and it is unbiased up to $\mathcal{O}(n^{-2})$, achieved by a geometric series debiasing scheme. In practice, this means that the estimator can be applied to small subsets of data, enabling efficient estimation and mini-batch updates. The proposed method has a natural choice of kernel, and can be used to generate consistent estimates of other quantities based on conditional expectation, such as the sharpness of a probabilistic classifier. Empirical results validate the correctness of our estimator, and demonstrate its utility in canonical calibration error estimation and calibration error regularized risk minimization.

A Consistent and Differentiable Lp Canonical Calibration Error Estimator
Teodora Popordanoska" ESAT-PSI, KU Leuven
Raphael Sayer*
University of Tübingen
teodora. cocdora_pppordaastt@etuloooo be
Matthew B. Blaschko
ESAT-PSI, KU Leuven
matthew.C blassckkoesatt. kuleuven.be wtw.://aeeaaa..kkueeeeee..e
Abstract
Calibrated probabilistic classifiers are models whose predicted probabilities can directly be interpreted uncertainty estimates. It has been shown recently that deep neural networks are poorly calibrated and tend to output overconfident pre- dictions. As a remedy, we propose low-bias, trainable calibration error estimator based on Dirichlet kernel density estimates, which asymptotically converges to the true Lp calibration error. This novel estimator enables to tackle the strongest notion of multiclass calibration, called canonical (or distribution) calibration, while other common calibration methods are tractable only for top-label and marginal calibration. The computational complexity of our estimator is O(n²). the conver- gence rate is Onn-1/2) and it is unbiased up to On 2), achieved by geometric series debiasing scheme. In practice, this means that the estimator can be applied to small subsets of data, enabling efficient estimation and mini-batch updates. The proposed method has a natural choice of kernel, and can be used to generate con- sistent estimates of other quantities based on conditional expectation, such the sharpness of probabilistic classifier. Empirical results validate the correctness estimator, and demonstrate its utility in canonical calibration error estimation and calibration error regularized risk minimization.
1 Introduction
Deep neural networks have shown tremendous success in classification tasks, being regularly the best performing models in terms of accuracy. However, they are also known to make overconfident predictions [Guo et al. 2017), which particularly problematic in safety-critical applications, such as medical diagnosis [Esteva al. 2017 2019] or autonomous driving [Caesar et al. 2020 [Sun et al.) 2020). In many real world applications is not only the predictive performance that is important, also the trustworthiness of the prediction, i.c., uncertainty estimates. To this end, it is necessary that the models are uncertainty calibrated, which means that, for instance, among all cells that have been predicted with a probability of 0.8 to be cancerous, 80% should indeed belong to a malignant tumor.
interested in accurate predictions with robust
The field of uncertainty calibration has been mostly focused on binary problems, often considering only the confidence score of the predicted class. However, this so called top-label (or confidence) calibration |Guo et al. 2017) is often not sufficient in multiclass settings. A stronger notion of calibration is marginal (or class-wise)
Kull et al. 2019), that splits up the multiclass problem
'Equal contribution
Most this work was done while Leuven
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Table I: Properties of ECEKDE and other commonly used calibration error estimators.
Properties
Consistency
Scalability
De-biased
Differentiable
ECEEDDE (Our)
ECE
X Vaicenavicius al. 2019
Roclofs al. 2022
Mix-n-Match
Zhang et al. 2020
MMCR
X Kumar ct al. 2018
into K one-vs-all binary ones, and requires each to be calibrated according to the definition of binary calibration. The most strict notion of calibration, called canonical (or distribution) calibration Bröcker 2009 Kull and Flach 2015] Vaicenavicius et 2019). requires the whole probability vector to be calibrated. The curse of dimensionality makes estimation of this form of calibration difficult, and current estimators, such as the binned estimator Ebén [Nacini et al. 2015). MMCE [Kumar et al. 2018] and Mix-n-Mateh (Zhang al.) 2020]. have computational statistical limitations that prevent them from being successfully applied in this important setting. Specifically, the binned estimator is sensitive to the binning scheme and is asymptotically inconsistent in many situations Vaaicnavvcius et al. 2019). MMCE is not a consistent estimator of T,p calibration error, and Mix-n-Match, although consistent, is intractable in high dimensions and the authors did not implement it in more than one dimension.
We propose tractable, differentiable, and consistent estimator of the expected Lp canonical cal- ibration error. In particular, we use kernel density estimates (KDEs) with a Beta kernel in binary classification tasks and a Dirichlet kernel in the multiclass setting, these kernels are the natural choices to model densities over a probability simplex. In Table we summarize and compare the properties of our ECE DE estimator and other commonly used estimators. ECE K RDE scales well to higher dimensions and it is able to capture canonical calibration with Onn²) complexity.
Our contributions can be summarized as follows: 1. We develop a tractable estimator of canonical T.p calibration error that is consistent and differentiable. 2. We demonstrate a natural choice of kernel. Due to the scaling properties of Dirichlet kernel density estimation, evaluating canonical calibration becomes feasible in cases that cannot be estimated using other methods. 3. We provide a second order debiasing scheme to further improve the convergence of the estimator. 4. We empirically evaluate the correctness of our estimator and demonstrate its utility the task of calibration regularized risk minimization on various network architectures and several datasets.
2 Related Work
Calibration of probabilistic predictors has long been studied in many fields. This topic gained attention in the deep learning community since Guo et al. [2017) observed that modern neural networks are poorly calibrated and tend to give overeonfident predictions due overfitting on the NLL loss. The surge of interest resulted in many calibration strategies that can be split in two general categories, which we discuss subsequently.
Post-hoc calibration strategies learn a calibration map of the predictions from a trained predictor in post-hoc manner, using held-out calibration set. For instance, Platt scaling [Platt [999] fits a logistic regression model on top of the logit outputs of the model. A special case of Platt scaling that fits single scalar, called temperature, has been popularized by Guo al. 2017 an accuracy- preserving, easy to implement and effective method to improve calibration. However, it has the undesired consequence that it clamps the high confidence scores of accurate predictions [Kumar et 2018]. Similar approaches for post-hoc calibration include histogram binning [Zadrozny and Elkan 200T], isotonic regression [Zadrozny and Elkan 2002). Bayesian binning into quantiles [Nacini and Cooper 2016). Beta (Kull et al. 2017] and Dirichlet calibration [ull et al. 2019). Recently, Gupta al. [202]] proposed a binning- frree calibration measure based on the Kolmogorov-Smiing In this approach, the recalibration function is obtained via spline fittine--ittnn, rather than minimizing loss function on a calibration set. Ma al. 2 integrate ensamble- baaee and post-hoe calibration methods in an accuracy- perrrcciip truth discovery framework. /Zhao et al. (202] introduce new notion of calibration, called decision calibration, however, they do not propose an estimator of calibration error with statistical guarnatces.
test.
2

---

Trainable calibration strategies integrate a differentiable calibration measure into the training objective. One of the earliest approaches is regularization by penalizing low entropy predictions [Pereyra et al., 2017]. Similarly to temperature scaling, it has been shown that entropy regularization needlessly suppresses high confidence scores of correct predictions (Kumar et al. 2018). Another popular strategy is MMCE (Maximum Mean Calibration Error) [Kumar et al. [2088. where the entropy regularizer is replaced by a kernel-based surrogate for the calibration error that can be optimized alongside NLL. It has been shown that label smoothing (Szegedy et al. 2016 Müller et al. 2019], i.e. training models with a weighted mixture of the labels instead of one-hot vectors, also improves model calibration. Liang et al. 20200 propose to add the difference between predicted confidence and accuracy as auxiliary term to the cross-entropy loss. Focal loss [Mukhoti et al. 2020
Lin et al. 2017] has recently been empirically shown produce better calibrated models than many of the alternatives, but does not estimate a clear quantity related to calibration error. [Bohdal et al. [021] derive differentiable approximation to the commonly-use
binned estimator of calibration error by computing differentiable approximations to the 0/1 loss and the binning operator. However, this approach does not eliminate the dependence the binning scheme and it is not clear how it can be extended to calibration of the whole probability vector.
Kernel density estimation PPazzen 1962 Rosenblati 1956 Silverman 1986 is a non-parametric
method to estimate a probability density function from a finite sample. Zhang et al. 2020 propose a KDE-based estimator of the calibration error (Mix-n Mix-n-Match Match) for measuring calibration performance. Although they demonstrate consistency of the method, it requires a numerical integration step that is infcasible in high dimensions. In practice, they only implemented binary calibration, and not canonical calibration.
Although many calibration strategies have been empirically shown to decrease the calibration error, very few of them are based on an estimator of miscalibration. Our estimator is the first consistent, differentiable estimator with favourable scaling properties that has been successfully applied to the estimation of Lp canonical calibration error in the multi-class setting.
3 Methods
We study a classical supervised classification problem. Let (S2, A, be a probability space, where Q is the set of possible outcomes, A = A(?) the sigma field of events and P. A - [0, is probability measure, let X' Rd and y {1. ...., K). Let Ω - X' and y S2 + > be random variables, while realizations are denoted with subscripts. Suppose we have a model f X + A where A denotes the K 1 dimensional simplex as obtained, e.g., from the output of final softmax layer in a neural network. We measure the (mis-)calibration of the model in terms of Lp calibration error, defined below.
Definition 3.1 (Calibration error, Naaiin et al., 2015 Kumar et al. 2019 Wenger et al. 2020 ) The hp calibration error of f is:
CE = E
(B Ely [(x)] s(x))
(1)
We note that we consider multiclass calibration, and that f(x) and the conditional expectation in Equation ( therefore map points a probability simplex. We say that classifier f is perfectly calibrated if (f,(f) = O.
In order to empirically compute the conditional expectation in Equation we need to perform density estimation over the probability simplex. In a binary setting, this has traditionally been done with binned estimates [Nacini et al. 2015 Guo al. [017 Kumar et al. (2019). However, this is not differentiable w.r.t. the function f. and cannot be incorporated into a gradient-basee training procedure. Furthermore, binned estimates suffer from the curse of dimensionality and do not have a practical extension to multiclass settings. We consider an estimator for the CE, based on Beta and Dirichlet kernel density estimates in the binary and multiclass setting, respectively. We require that this estimator is consistent and differentiable, such that we can train it in a calibration error regularized risk minimization framework. This estimator is given by:
n
CE,(f)
n Σ
Ely f(x)] ((a2)
-((,,)
(2)
3

---

where Ely I [x)
denotes Ely | f(x)] evaluated at f(x) f(zz). Let Pa,y Vi)
f(xz)
Pyl==a, (1)) (1,) be the joint density, Then we define the estimator of the conditional expec-
tation as follows:
EBEE Us Pf(s)),) Uk)
Byy [())]
Σ ye Pv. ((x) JVEY
Pr(w) (F())
((f());; =: Bly [(x))
1-1
E k(f()); f(a))
where & is the kernel of a kernel density estimate evaluated point and Pf( is uniquely determined by Pa and/ f. Proposition 3.2. Assuming that (I())((())) is Lipschitz continuous over the interior of the simplex, there exists a kernel k such that Elyy [(x) is a pointwise consistent estimator of Bly [(()), that is:
plim k(f(x);; (a)))) Emey 3k (f))
(4)
k(((<);;(;)))
(f())
Proof. Let & be a Dirichlet kernel OOimeet and Ahoosna--eelgadee 2022 By the consistency of the Dirichlet kernel density estimators [Ouimet and Volosana-Delgade 2022, Theorem 4] Lip- schitz continuity of the density over the simplex a sufficient condition for uniform conver- gence of the kernel density estimate. This in turn implies that for a given f. for all f(x) E (0, 1 El {(f(=): 7 Emey yk (f(=))(((()) yk) and 1 E.= f(x());(()))) & n
Let g(x) = 1/2, then the set of discontinuities g applied the denominator of the Lh.s. of has measure zero since 1 (i==k(((x)) f(x4) = 0 with probability zero. From the con- tinuous mapping theorem [Mann and Wald 1943]. it follows that n/(Ei=
(f Let
f(x))) & (fPr)) Since products of convergent (in probability) sequences of random vari- ables converge in probability to the product of their limits [Resnick] 2019]. we have that Σ.= (zf(;); ((:))) f(x))) which is equal to the r.h.s. of
& EgeEy Uh Ps(o). (f(), (f()))))))(()))
The most commonly used loss functions are designed to achieve consistency in the sense of Bayes optimality under risk minimization, however, they do not guarantee calibration neither for finite samples nor in the asymptotic limit. Since we are interested in models f that are both accurate and calibrated, we consider the following optimization problem bounding the calibration error CE(f): arg min Risk(f), s.t. CE(f) S B for some B O, and its associated Lagrangian:
arg min (Risk(f) + -CE())) JEF
(5)
Mean squared error in binary classification As a first instantiation of this framework we consider a binary classification setting, with mean squared error MSE(f) [[(f(x)-y)) the risk function. jointly optimized with the Le calibration error CE2:
f= arg argminn MSE(F) CE2(1)² arg min FEF
MSE(f)) + > Bly f(=))²)
(6)
feF
where y 1 E [0, 1). The full derivation using the MSE decomposition MMrphhy
1973
Degroot
and Fienberg 1983 Kuleshov and Liang 2015) Nguyen and O' O'Connor 2013 Is given in A Appendix
For optimization wish to find estimator for E Ely by f()] Building upon Equation 3 partially debiased estimator can be written as:
Eies (((())::
Eiss (k:(((z))) { ((4))))))
1
Ely f())
(7)
k(f<x)
Eezy (k(f(Iz); f(x))))
Thus, the conditional expectation is estimated using ratio of unbiased estimators of the square of mean.

---

Proposition 3.3. Equation 7 is ratio of two U-statistics and has a bias converging as O (1)
The proof is given in Appendix
Proposition 3.4. There exist de-biasing schemes for the ratios in Equation and Equation achieve an improved O (+:3 convergence of the bias.
3 that
Proofs are given Appendix and
In a binary setting, the kernels k(; are Beta distributions defined as:
[(a; + B.) T(o:) ((3))
kn([(z)), = [(x) ( ((a))
(8)
Bouezmarni and Rolin 2003 Zhang and Karunamuni 2010], where h is bandwidth parameter in the kernel density estimate that goes to Osas n + 00. We note that the computational complexity of this estimator (nn)). If we would use this within gradient descent training procedure, the density can be estimated using a mini-bater and therefore the (n²)) complexity is w.r.t. the size of a mini-batch, not the entire dataset.
with oz; f(ax)) 1 and B; = ===-(mm))
Chen 1999
The estimator in Equation (7) is a ratio of two second order U-statistics that converge as 1/2 [Ferguson 2005). Therefore, the overall convergence will be n n-1/2 Empirical convergence rates calculated in Appendix G and shown to be close to the theoretically expected value.
Multiclass calibration with Dirichlet kernel density estimates There several definitions of multiclass calibration that vary in terms of how strictly they define the calibration the probability vector f=x). The strongest notion of multiclass calibration, and the one that we focus on in this paper, is canonical (also called multiclass or distribution) calibration [Bröcker] 22099 Kull and Flach
2015 Vaicenavicius et al. 2019), which requires that the whole probability vector f (x) is calibrated (Definition) 3.1). Its estimator is:
Eie, KDi ( f(2z))
CE,(f)
Σ
(9)
Ens KDn (f(z)) f(z.))
where KDi is a Dirichlet kernel defined as:
r k= ask) k= I(ank)
K II f(x;) pm-
(fir()(z;),
(10)
with a; = fxx)) Ouimet and Tolosana- -Delgado On?)) irrespective of
2022 As before, the computational complexity
This estimator is differentiable and furthermore, the following proposition holds: Proposition 3.5. The Dirichlet kernel based CE estimator is consistent when (f()) is Lipschitz
continuous:
Σe وe (T))))()))) f(x;)) E 1 f(=)))
plim TL
Σ j=1
[f((f())]
Eit kDu (f);) f(x.))
Proof. Dirichlet kernel estimators are consistent when the density is Lipschitz continuous over the simplex [Ouimet and Delosana-Delgadoo/ [022] Theorem 4], consequently. by Proposition 3.2 the term inside the norm is consistent for any fixed f (æ;) (note that summing over i / j ensures that the ratio the KDE's does not depend on the outer summation). Moreover, for any convergent sequence also the norm of that sequence converges to the norm of its limit. Ultimately, the outer sum is merely the sample mean of consistent summands, which again is consistent.
With this development, we have for the first time consistent, differentiable, and tractable estimator of Lp canonical calibration enror with (nn?) computational cost and O(n -1/2) convergence rate, with debiasing scheme that achieves Onn (2) bias for p {1,2].
5

---

Empirical validation of ECE
Accurately evaluating the calibration error is crucial step towards designing trustworthy models that can be used in societally important settings. The most widely used metric for evaluating miscalibration, and the only other estimator that can be straightforwardly extended to measure canonical calibration, the histogram- based estimator ECE Ebin. However, as discussed in Vaicenavicius et al. (2019), Widmann al. [2019]. Ding al. [2020] Ashukha et al. [2020), it has numerous flaws, such as: (i) is sensitive to the binning scheme (ii) it is severely affected by the curse of dimensionality, as the number of bins grows exponentially with the number of classes (iii) it asymptotically inconsistent
in many cases.
To investigate its relationship with estimator ECEKDE we first introduce extension of the top-label binned estimator to the probability simplex in the three class setting. We start by partitioning ttangle-shaped bins and assign the probability scores to the
the probability simplex into equally-sized corresponding bin, as shown in Figure Then, we define the binned estimate of canonical calibration
error as follows: (f) E [f(x)) f(
E (f - f( where
(()())) is the histogram estimate, shown in Figure Tb The surface of the corresponding Dirichlet KDE is presented in Figure Tcl See Appendix (F for (i) an experiment investigating their relationship for the three types of calibration (top-label, marginal, canonical), and with varying number of points
used for the estimation, and (ii) another example of the binned estimator and Dirichlet KDE on
CIFAR-10.
(a) Splitting the simplex in 16 bins
(b) Histogram
(c) Dirichlet KDE
Figure Extension the binned estimator gbin to the probability simplex, compared with ECEKDE The ECEKDE achieves better approximation to the finite sample, and accurately models the fact that samples tend to be concentrated near low dimensional faces of the simplex.
Synthetic experiments We consider an extension of EC arbitrary number of classes and investigate its performance compared to ECEKDE, Since on real data the ground truth calibration error is unknown, we generate synthetic data with known transformations, with the following procedure. First, we sample uniformly from the simplex using the Kraemer algorithm (Smith and Tromble 2004]. Then, we apply temperature scaling with t1 0.6 to simulate realistic scenarios where the probability scores are concentrated along lower dimensional faces of the simplex. We generate ground truth labels according to the sampled probabilities, and therefore, obtain perfectly calibrated classifier. Subsequently, the classifier is miscalibrated by additional temperature scaling with 12 = 0.6. Figure Papdeiicss the performance of the two estimators as function of the sample size generated data for 4 and 8 classes. DC converges to the ground truth value obtained by integration both cases, whereas E provides poor estimates even with 20000 points.
In another experiment with synthetic data we look at the bias of the sharpnes term in binary setting. In Figure 2b we plot the estimated value the sharpness term for varying number samples, both using the partially debiased ratio from Equation (7). and the ratio debiased with the scheme introduced in Appendix [D] A sigmoidal function is applied to the calibrated data to obtain an uncalibrated sample that is used to compute the partially debiased and the fully debiased ratio of the sharpness term. The ground truth value is obtained by using 100 million samples to compute the ratio with the partially debiased version, it converges asymptotically to the true value due its consistency. We use
The sharpness is defined as (ar (x))) Kuleshov and Liang 2015). Here we neglect the term that does
not depend f(x), and thereby refer B Bly {(x))
the sharpness.

---

bandwidth of 0.5 and average over 10000 repetitions for each number of samples that range from 32 16384. We fix the location of the KDE fa,) 0.17.
Num dissess
Num
2.5x 10
Periiityy eesase Iicersttut
04
0.0
8 0.5
0.4
0.4
9. 10
0.3
A3x to
0.2
5000
10000
15000
9232
19802
Oround
ECg
62ght
Ecg
ECg
123 Mumbur of samp
(a) EC DCEKDE
(b) Debiasing
Figure 2: 2a Performaacc of EC and ECEKDE symthetic data for varying number of classes, as a function of the sample size. Ground truth represents the true value of the integral. ECC Ebbn is calculated using several common choices for the number of bins (n binb represents number of bins per-class.) n bins" and b" are found as optimal values according to Doane's formula (Doane 1976) and LOO MLE, respectively. ECERDE converges the true value in all settings, in contrast to ECE 2b Sharpness term evaluated for different numbers of samples with the partially debiased ratio from Equation (7), and with the debiasing scheme derived in Appendix D on synthetic data.
5 Calibration regularized training
Empirical setup To showcase our estimator in applications where canonical calibration is crucial, consider two medical datasets, namely Kather and DermaMNIST. The Kather dataset (Kather et al., 2016] consists of 5000 histological images of human colorectal cancer and it has eight different classes of tissue. DermaMNIST (Yang al., [021] is a pre-processed version of the HAM1 dataset Tschandl er al. 2018). containing 10015 dermatoscopic images of skin lesions, categorized in seven classes. Both datasets have been collected in accordance with the Declaration of Helsinki. According to standard practice in related works, we trained ResNet [ee et al.) 2016), ResNet with stochastic depth (SD) (Huang al.) 20166). DenseNet (Huang et al. 2017] and WideResNet [Zagoruyko and Komodakis 2016 networks also on CIFAR- 10/100 (Krizhessky 2009]. J. We use 45000 images for training on the CIFAR datasets, 4000 for Kather and 7007 for DermaMNIST. The code is available at https: gi thub. com/ wtw/ http//poooodnnoskaaeeeeeeo ecea
Baselines Cross-entropy. The first baseline model is trained using cross-entropy (XE), with the data preprocessing, training procedure and hyperparameters described in the corresponding paper for the architecture.
Trainable calibration strutegies. KDE-XE denotes the joint training of XE with our proposed estimator EC as defined in Equation ( MMCE Kumar et al. 2018 a differentiable measure of calibration with property that it is minimized at perfect calibration, i.e., MMCE is 0 if and only if CE2, = 0. It is used as a regulariser alongside NLL, with the strength of regularization parameterized by A. Focal loss (FL) Mukkhot et al.) 2020 is an alternative to the cross-entropy loss, defined as C, = - (1 1f (y)x))) log(f x )), where 7 is a hyperparameter and f(y(z) is the probability score that neural network if outputs for class y approach is the cemple-dependent FL-53, where ? = 5 for fylxx) E [0, 0.2), and 7 = otherwise.
input Their best-performing
Post-hoc calibration strategies. Guo et al. 2017] investigated the performance of several post-hoc calibration methods and found temperature scaling to be a strong baseline, which we use as a representative of this group. It works by scaling the logits with scalar T > 0, typically learned on validation set by minimizing NL... Following Kumar et al. [2018) and Mukhoti et al. [2020]. we also use temperature scaling as post-processing step for our method.
7

---

Metries We report L1 canonical calibration using our ECERDE estimator, calculated according Equation 9 Additional experiments with L, and L2 top-label calibration on CIFAR-10/100 can be found in Appendix
Hyperparameters A crucial parameter for KDE is the bandwidth b, a positive number that defines the smoothness of the density plot. Poorly chosen bandwidth may lead to undersmoothing (small bandwidth) or oversmoothing (large bandwidth), as shown in Figure 3 commonly used non-parametric bandwidth selector is maximum like-
3
RDC 2
HDC 2-33
lihood cross validation [Duin 1976). For our experiments we choose the bandwidth from a list of possible values by maximizing the leave-one-out likelihood (LOO MLE). The À parameter for weighting the calibration error w.r.t the loss typically chosen via cross- validation or using holdout validation set. We found that for KDE-XE, val- ues of E [0.001,0.] 0.2] provide a good trade-off in terms of accuracy and calibration error. The p parameter is se- lected depending on the desired T, calibration error and the corresponding theoretical guarantees. The rest of the hyperparameters for training are set as proposed in the corresponding papers for the architectures we benchmark. In particular, for the CIFAR- 100-00000000//0 datasets we used batch size of 64 for DenseNet and 128 for the other architectures. For the medical datasets, we used a batch size of 64, due to their smaller size.
Figure 3: Effect of the bandwidth on the shape of the estimate.
5.1 Experiments
An important property of our ECEKDE estimator is differentiability. allowing be used in a calibration regularized training framework. In this section, we benchmark <DE-XE with several baselines on medical diagnosis applications, where the calibration of the whole probability vector is of particular interest. For completeness, also include an experiment on CIFAR-10.
Table 2 summarizes the canonical L ECEKDE and Table the accuracy, measured across multiple architectures. The bandwidth is chosen by LOO MLE. For MMCE and KDE-XE, we train the models with several values for the regularization weight, and report the best performing one. In Table 2 we notice that KDE-XE consistently achieves very competitive ECE values, while also boosting the accuracy, as shown in Table Interestingly, we observe that temperature scaling does not improve canonical calibration eπror, contrary to its reported improvements observation that temperature scaling is less effective for stronger notions of calibration consistent with similar finding in Kull et al. [2019), where the authors show that although the temperature- scaled model has well calibrated tapelabel confidence scores, the calibration error is much larger for class-wise calibration.
top-label calibration. This
Table 2: Canonical L, DE (L) for different loss functions and architectures, both trained from scratch (Pre T) and after temperature scaling validation set (Post T). Best results across Pre T methods are marked in bold.
Dataset
Model
XB
MMCE
FL.53
KDE-XE (Our)
Pre
Post T
Pre T
Post T
Pre T
Post T
Pre
Post T
ResNet-110
0.335
0.304
0.343
0.300
0.325
0.248
0,311
0.289
ResNet-1 10 (SD)
0.329
0.334
0.235
0.159
0.209
0.122
0.198
0.147
Kather
Mide-ResNNet-2--1
0.177
0.259
0.201
0.241
0.270
0.328
0.162
0.212
DenseNet-40
0.244
0.251
0.159
0.218
0.165
0.207
0.114
0.154
ResNet-110
0.579
0.602
0.575
0.603
0.684
0.618
0.467
0.516
ResNet-1 10 (SD)
0.534
0.571
0.470
0.526
0.567
0,594
0.461
0.538
DermaMNIST
Wide ReeleRReNNee-28--
0.546
0.599
0.470
0.512
0.623
0.608
0.455
0.599
Den
0.573
0.578
0.514
0.558
0.577
0.557
0.366
0.418
ResNet-110
0.133
0.170
0.171
0.196
0.138
0.171
0.126
0.163
Res (SD)
0.132
0.172
0.164
0.203
0.156
0.201
0.178
0.223
CIFAR-10
Wide-ResNeet28-110
0.083
0,098
0,143
0.155
0.147
0.177
0.077
0.091
DenscNet-40
0.104
0.131
0.133
0.155
0.081
0.081
0.098
0.124
8

---

Table Accuracy (1) computed for different architectures. Best results are marked in bold.
Dataset
Model
XE
MMCE
FL-53
KDE-XE (Our)
ResNet-1
0.840
0.860
0.844
0.860
ResNet-110 (SD)
0.870
0.900
0.885
0.914
Kather
Wide-ResNet-28-110
0.933
0.899
0.873
0.921
DenseNet-40
0.913
0.93
0.916
0.941
ResNet- 10
0.720
0.721
0.674
0.744
ResNet-110 (SD)
0.743
0.753
0.689
0.764
DermaMNIST
Wide-ResNet-2--10
0.736
0.741
0.715
0.754
DenseNet-40
0.741
0.758
0.705
0.748
ResNet-110
0.925
0.929
0.922
0.929
ResNet-110 (SD)
0.926
0.925
0.92
0.907
CIFAR-10
Wide-ResNet-28-111
0.954
0.947
0.936
0.954
DenseNet-40
0.947
0,944
0.948
0.947
the performance of several architectures and datasets in terms of accuracy and L ECERDE for various choices of the regularization parameter for MMCE and KDE-XE. The 95% confidence intervals for ECEKDE DE are calculated using 100 and 10 bootstrap samples the medical datasets and CIFAR-10, respectively. In all settings, KDE-XE Pareto dominates the competitors, for several choices of X. For example, on DermaMNIST trained with DenseNet, KDE-XE with À 0.2 reduces ECEKDE from 66% to 45%.
535
0.45
0.70
0.275
0.40
KDEXXE
H1
0.00
WR001
0.205
0.2
0.58
50.20
0.25
0.01
0.176
0.20
KEEXE
0.45
0.100
0.82
0.85
0.88
0.88
0.90
0.82
0.70
0.72
0.74
0.78
0.78
0.90
0.01
0.92
0.05
ACC
ACC
ACC
(a) ResNet- (SD) on Kather
(b) DenseNet on DermaMNIST
(c) ResNet- 110 on CIFAR-
Figure 4: Canonical calibration on various datasets and architectures. The numbers next to the points denote the value of the regularization parameter. KDE-XE outperforms the competitors, both in terms of accuracy and calibration error, for several choices of
Training time measurements In Table we summarize the running time per epoch the four architectures, with regularization (KDE- XE), and without regularization (XE). We ob- serve only an insignificant impact on the train- ing speed when using KDE-XE, dispelling any concerns w.r.t. the computational overhead.
Table 4: Training time [sec] per epoch for XE and KDE-XE for different models on CIFAR-10.
Dataset
Model
X
KDE-XE
ResNet- 10
51.8
53.0
ResNet-110 (SD)
45.0
46.0
CIFAR-10
To summarize, the experiments show that our estimator is consistently producing competitive calibration errors with other state-of-the-art ap- proaches, while maintaining accuracy and keep- ing the computational complexity at O We note that within the proposed calibration calibration-regularrize training framework, this complexity is w.r.t. to match and the added cost is less than couple percent. Furthermore, the O(n²) complexity shows up in other related works (Kumar al. 2018 Zhang et al.) 2020). and is intrinsic to the problem of density estimators of calibration error. As future work, larger scale benchmarking will be beneficial for exploring the limits of canonical calibration using Dirichlet kernels.
Wide-ResNet-28-11
152.9
154.9
DenseNet-40
103.2
106.8

---

6 Conclusion
In this paper, we proposed a consistent and differentiable estimator of canonical L, calibration error using Dirichlet kernels. It has favorable computational and statistical properties, with a complexity of Onn)), convergence of O( 1/2), and a bias that converges as (n 1) which can be further reduced to ((n=3) using our debiasing strategy. The ECEK can be directly optimized alongside any loss function in the existing batch stochastic gradient descent framework. Furthermore, we propose using it measure of the highest form of calibration, which requires the entire probability vector to be calibrated. To the best of our knowledge, this is the only metric that can tractably capture this type of calibration, which is crucial in safety-critical applications where downstream decisions are made based on the predicted probabilities. We showed empirically on range of neural architectures and datasets that the performance of our estimator in terms of accuracy and calibration error is competitive against the current state-of wate-of-the-arr while having superior properties as consistent estimator of canonical calibration error.
Acknowledgments
This research received funding from the Research Foundation Flanders (FWO) through project number S00142IN, and the Flemish Government under the "Onddrzooekppogrramma Artificiële Intelligentie (AI) Vlaanderen" programme. R.S. was supported in part by the Tübingen Al centre.
Ethical statement
The paper is concerned with estimation of calibration error, a topic for which existing methods are deployed, albeit not typically for canonical calibration error in a multi-class setting. We therefore consider the ethical risks to effectively the same as for any probabilistic classifier. Experiments apply the method to medical image classification, for which misinterpretation of benchmark results with respect to their clinical applicability has been highlighted as a risk, see e.g. Varoquaux and Cheplygina 2022)
References
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. In International Conference on Learning Representations, 2020.
Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Meta-calibration: Meta-learning of model calibration using differentiable expected calibration error. arXiv preprint 1rXiv:2106.06613. 2021.
Taoufik Bouezmarni and Jean-Marie Rolin. Consistency of the beta kernel density function estimator. The Canadian Journal of Statistics / La Revue Canadienne de Statistique, 31(1):89-99 98, 2003.
Jochen Bröcker. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society, 1356643):1512-1519 Jul 2009.
Holger Caesar, Varun Kumar Reddy Bankiti, Alex Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, page 11621-11631, 06 2020.
Song Xi Chen. Beta kernel estimators for density functions. Computational Statistics Data Analysis, 31:131-145, 1999.
M. Degroot and S. Fienberg. The comparison and evaluation of forecasters. The Statistician, 32: 12-22, 1983.
Yukun Ding, Jinglan Liu, Jinjun Xiong, and Yiyu Shi. Revisiting the evaluation of uncertainty esti- mation and its application to explore model complexity-uncertiin trade-off. In 2020 IEEF/CVF Conference on Computer Vision and Puttern Recognition Workshops (CVPRW), pages 22-31, 2020.
10

---

David P. Doane. 1976.
Aesthetic frequency classifications. The American Statistician, 30(4):181-183,
Robert Duin. On the choice of smoothing parameters for parzen estimators of probability density functions. IEEE Transactions on Computers, (25(11):1175-117) 1976.
Andre Esteva, Brett Kuprel, Roberto A. Novoa, Justin Ko, Susan M. Swetter, Helen M. Blau, and Sebastian Thrun. Dermatologist-lvvl classification of skin cancer with deep neural networks. Nature, 542:115-, 2017.
Andre Esteva, Alexandre Robiequet, Bharath Ramsundar, Volodymyr Kuleshov, Mark DePristo, Katherine Chou, Claire Cui, Greg Corrado, Sebastian Thrun, and Jeff Dean. A guide to deep learning in healthcare. Nature Medicine, 25, 01 2019.
Thomas S. Ferguson. U-statistics. In Notes for Statistics 200C. UCLA, 2005.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321-1330. PMLR, 2017.
Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. Calibration of neural networks using splines. International Conference on Learning Representations, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. European conference on computer vision, pages 661. Springer, 2016.
Gao Huang. Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Jakob Kather, Cleo-Aron Weis, Francesco Bianconi, Susanne Melchers, Lothar Schad, Timo Gaiser, Alexander Marx, and Frank Zöllner. Multi-class texture analysis in colorectal cancer histology. Scientific Reports, 6:27988, 06 2016.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Volodymyr Kuleshov and Percy $ Liang. Calibrated structured prediction. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Meelis Kull and Peter Flach. Novel decompositions of proper scoring rules for classification: Score adjustment as precursor to calibration. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 68-85. Springer, 2015.
Meelis Kull, Telmo Silva Filho, and Peter Flach. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statisties, volume 54 of Proceedings of Machine Learning Research, pages 623 631. PMLR, 20-22 Apr 2017.
Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining wall-calibrated multi-class probabilities with dirichlet calibration. Advances in neural information processing systems, 32, 2019.
Ananya Kumar, Percy Liang, and Tengyu Ma. Verified uncertainty calibration. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bun Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 3792-3803. 2019.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In ICML, 2018.

---

Gongbo Liang, Yu Zhang, Xiaoqin Wang, and Nathan Jacobs. Improved trainable calibration method for neural networks on medical imaging classification. In British Machine Vision Conference. 2020.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
Chunwei Ma. Ziyun Huang, Jiayi Xian, Mingchen Gao, and Jinhui Xu. Improving uncertainty calibration of deep neural networks via truth discovery and geometric optimization. In Cassio de Campos and Marloes H. Maathuis, editors, Proceedings of the Shirtt-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 75-85. PMLR, 27-30 Jul 2021.
Henry В Mann and Abraham Wald. On stochastic limit and order relationships. The Annals of Mathematical Statistics, 14(3):17-2266 1943.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Tomr and Puneet Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 15215888-1229, 2020.
Rafael Müller, Simon Korblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
A. Murphy. A new vector partition of the probability score. Journal of Applied Meteorology, 12: 595-600, 1973.
Mahdi Pakdaman Nacini and Gregory F Cooper. Binary classifier calibration using ensemble of near isotonic regression models. In 2016 IEEE 16th International Conference on Data Mining (ICDM), pages 360-369. IEEE, 2016.
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pages 2901-2907, 2015.
Khanh Nguyen and Brendan O'Connor. Posterior calibration and exploratory analysis for natural language processing models. arXiv preprint arXiv::508s 05 154, 2015.
R.C. Ogliore, G.R. Huss, and Nagashima. Ratio estimation in SIMS analysis. Nuclear Instruments and Methods in Physics Research Section B-beam Interactions With Materials and Atoms, 269, 06 2011.
Frédéric Ouimet and Raimon folosana-Delgadee Asymptotic properties of dirichlet kernel density estimators. Journal of Multivariate Analysis, 187:104832, 2022.
Emanuel Parzen. On estimation of a probability density function and mode. The Annals of Mathe- matical Statistics, 3363):1065-1076, 1962.
Gabriel Pereyra, George Tucker, Jan Chorowski, Fukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint / 066548, 2017.
John C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, pages 61-74. MIT Press, 1999.
Sidney Resnick. A probability path. Springer, 2019.
Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C. Mozer. Mitigating bias in calibration error estimation. In Gustau Camps- Francisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 4036-4054. PMLR, 28-30 Mar 2022.
Murray Rosenblatt. Remarks on some nonparametric estimates of density function. The Annals of Mathematical Statistics, 27(3):832 837, 1956.
12

---

Jun Shao. Mathematical Statistics, page 180. Springer Texts in Statistics, second edition, 2003.
B. W. Silverman. Density Estimation for Statistics and Data Analysis. Chapman & Hall, 1986.
Noah A. Smith and Roy W. Tromble. Sampling Uniformly from the Unit Simplex. 2004.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Alcksei Timofeev, Scott M. Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. 2020 IEEE/CVF Conference on Computer Vision and 2020.
Pattern Recognition (CVPR), pages
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
Myint Tin. Comparison of some ratio estimators. Journal of the American Statistical Association, 60 (809))294-307, 1965.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The HAM10000 dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci. Data, 5:1 1116,, 2018.
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik I Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in elassification. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3459-3467. PMLR, 2019.
Gaël Varoquaux and Veronika Cheplygina. Machine learning for medical imaging: methodological failures and recommendations for the future. npj Digital Medicine, 5:48, 04 2022.
Jonathan Wenger, Hedvig Kjellström, and Rudolph Triebel. Non-parametric calibration for clas- sification. In International Conference on Artificial Intelligence and Statistics, pages 178-190, 2020.
David Widmann, Fredrik Lindsten, and Dave Zachariah. Calibration tests in multi-class classification: A unifying framework. Advances in Neural Information Processing Systems, 32, 2019.
Jiancheng Yang, Rui Shi, and Bingbing Ni. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), Apr 2021.
B. Zadrozny and C. Elkan. Transforming classifier scores into accurate multiclass probability estimates. Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, 2002.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. ICML, 1, 05 2001.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference, 2016.
Jize Zhang, Bhavya Kailkhura, and T. Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In International Conference on Machine Learning, 2020.
Shunpu Zhang and Rohana Karunamuni. Boundary performance of the beta kernel estimators. Journal of Nonparametric Statistics, 22:81-104, 01 2010.
Shengjia Zhao, Michael Kim, Roshni Sahoo, Tengyu Ma, and Stefano Ermon. Calibrating predictions to decisions: A novel approach to multi-class calibration. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 2333-22324. Curran Associates, Inc., 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Please refer our ethical statement.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as a URL)? [Yes] It is in the supplementary material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Figure
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] [No] Table includes compute times. Most of our results provided in big O complexity.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [No] We do not release the data. Data license available via the citation.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you't re using/curating? [Yes] Medical datasets used in this paper conform the Declaration of Helsinki.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] Medical datasets used in this paper conform to the Declaration of Helsinki.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] We did not use crowdsourcing or conducted research with human subjects
(b) Did you describe any potential participant risks, with links to Institutional Review Board (RRB) approvals, applicable? [N/A] We did not use crowdsourcing or conducted research with human subjects
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] We did not use crowdsourcing or conducted research with human subjects
14


Reviews:
[{"review_id": "rVtDxr-0Lf8", "reviewer": "Reviewer_GyKQ", "summary": "Classifiers with deep neural networks tend to output overconfident class-posterior probabilities.\nThis paper proposes a novel method for calibrating the confidence (i.e., the predicted class-posterior probability) during training a classifier with deep neural networks. The proposed method adds a term to the loss for penalizing miscalibrated confidence scores.\nThe added term is an estimate of the $L_p$ distance between the predicted class-posterior probability and the true one, which is differentiable so that we can use the gradient descent method for training.\nThey show that the estimator of the discrepancy is statistically consistent, and calculation of it takes $O(B^2)$ time for mini-batch size $B$.\nThe paper presents experiments with a synthetic dataset and a three benchmark datasets whose results support the usefulness of the proposed method.", "questions": "- Denoting the mini-batch size as $B$, the overhead in the training time is indeed $O(B^2)$, but we only need to evaluate the function $B$ points in each iteration (which is nice). If this is correct, I suggest the authors mention this in the paper.\n- I am not very sure if \"scalable\" is the right word for describing the overhead of $O(B^2)$. I suggest simply removing this from the title or using milder words.\n- I request that the authors clearly present the mini-batch sizes used in the experiments rather than just referring to other papers.\n- I do not think the statements about consistency in the paper are precise. The convergence has to be stated in terms of convergence in probability. I suggest making them more formal. The proofs are also very rough should be improved.\n- Also, the paper only provides point-wise convergence analyses although the proposed method performs optimization over a function class. I suggest providing convergence analysis that holds uniformly over the function class. \n- Are the color scales of Figure 1(b) and Figure 1(c) the same?\n- \"realistic scenarios where the probability scores are concentrated along lower dimensional faces of the simplex\": What makes the authors believe it is \"realistic\"?\n- How did the authors choose the bandwidth (0.5) and the location of KDE (0.17) for the synthetic data experiments?\n- \"We found that for KDE-XE, values of $\\lambda \\in [0.001, 0.2]$ provide a good trade-off in terms of accuracy and calibration error\": How do we define a \"good\" trade-off?\n- l.190: What does \"high-cost settings\" mean?\n- What is the \"partially\" debiased estimator?", "limitations": "As I mentioned in \"Questions\" section, the training time overhead $O(B^2)$ can limit the mini-batch size and thus create bias. It would be nice if the authors could provide the maximum mini-batch size they can handle.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strength\n- The paper tackles the important task of confidence calibration for classification with deep neural networks.\n- The authors provide some guarantees for their estimator of the miscalibration penalty term. Specifically, the estimator has $O(1/n)$ bias, and the rate of convergence is $O(1/n)$. The bias can be further improved to $O(1/n^2)$.\n- The overhead in computation time added by the proposed method is $O(B^2)$ for mini-batch size $B$, which may be small when $B$ is small. In fact, they empirically showed that adding the proposed term only increases the training time by a few percentage points in their experiments.\n\n# Weaknesses\n- Their convergence analysis is done for a fixed function and is not guaranteed to hold for all functions of the feasible space of their optimization problem. Analysis of uniform convergence would be preferable.\n- For a mini-batch size $B$, the estimator will have a bias of $O(1/B^2)$. For example, if $B$ is constant determined by the GPU memory size, we will have constant bias.\n- Proofs in the paper are too brief and it is difficult to confirm the statements are correct.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jL4HcTBT-gD", "reviewer": "Reviewer_x5cZ", "summary": "In this paper, the authors propose a novel strategy to estimate distributional or canonical calibration error based on Dirichlet kernel density estimates which asymptotically converges to the true Lp calibration error.  Such a calibration penalty is differentiable and can be incorporated while training a classifier.  The proposed approach improves the L1 ECE across multiple classification benchmarks and model architectures.", "questions": "How does adding the KDE calibration into the training pipeline explicitly control over-confidence in NN ? It would be beneficial to plot the accuracy vs confidence reliability plots along with comparisons with existing methods. \n\nCan such a strategy be directly adopted for regression tasks ? It would be beneficial to demonstrate empirical results for regression. \n\nCan this training strategy lead to models that can detect distribution shifts much better than other training strategies that include different calibration objectives ?  How does canonical calibration improve robustness to distribution shifts ? \n\nFig. 4 can be better expanded and scaled.\n\nThese are some of the typos I had spotted\nLine 106 - It would be beneficial for readers if you can expand the notations used in the definition of the probability space (omega, A, P). \nLine 149 - Need to specify that lambda and gamma are used interchangeably\nLine 111: Left paranthesis can be removed before 'Calibration'\nEq. 12, the index j needs to be defined\n\nSuggestions\nIt would be great if you can change some of the arxiv references to their actual conference/journal references (if published)\n", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe idea of using Dirichlet kernel density estimates as the means to estimate canonical calibration is novel. The paper clearly illustrates how the conventional metrics such as ECE^BIN do not clearly convey the notion of canonical calibration and why it is important to estimate such a calibration error.  The work is mathematically well grounded and demonstrates a number of synthetic as well as real-data experiments. Overall the paper is well written with minor typos. \n\nWeaknesses\n\nKindly refer to the questions below\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "deKK3azGyWY", "reviewer": "Reviewer_Nx3X", "summary": " The paper proposes a differentiable (and therefore trainable using gradient descent), low-bias calibration error estimator based on Dirichlet kernel density estimates, which is consistent w.r.t $L_{p}$ calibration error. \n\n- The computational complexity of the estimator in the paper reported is $\\mathcal{O}\\left(n^{2}\\right)$, while the convergence rate is $\\mathcal{O}\\left(n^{-1 / 2}\\right)$, and it is unbiased up to $\\mathcal{O}\\left(n^{-2}\\right)$, which the authors achieve by debiasing the estimate.\n\n- This estimator shows strong performance in estimating the exact $L_1$, $L_p$ calibration error in a synthetic dataset with known calibrations. Since it is also a differentiable function, it can be used in conjunction with crossentropy loss to train neural nets, and it shows improved accuracy and calibration properties over other differentiable proxies.", "questions": "I mostly have minor nits to the paper, and I think it is a well-rounded paper in general, that covers most of my doubts.\n\n1. \"Interestingly, we observe that temperature scaling does not improve canonical calibration error, contrary to its reported improvements on top-label calibration.\" I think this line warrants much more study. Temperature scaling is a very commonly used technique to acheive quick calibration, and if it is not showing any improvements on the ECE metric proposed by the paper, that poses some interesting questions. Temperature scaling clearly improves model calibration, results in better performance across benchmarks on ECE (binned), Brier score, and OOD performance. However, this improvement in performance is not being captured by the new ECE metric? Do the authors have any intuition as to why? Is the KDE independent to division by a scalar term?\n\nMinor nits:\n- Line 81: \"Maximum\"", "limitations": "I believe the authors have addressed limitations well, and have a good section talking about potential societal impact.", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Originality\n\nThe field of uncertainty calibration has seen significant research in recent years in trying to find a differentiable approximation to calibration error, in order to be able to train calibration error using gradient descent. To my knowledge, previous methods have attempted to (1) perform soft binning, obtaining a differentiable approximation to the binned ECE (Bohdal et al 2021), (2) Mix-n-Match (Zhang et al 2020), which uses numerical integration and doesn't extend to multiclass calibration as far as I am aware. This paper addresses both these concerns quite well, as their metric is independent of binning, and doesn't require an expensive integration step.  I think the approach is quite original, and the authors have done a good job performing a literature survey. To my limited knowledge in this field, I didn't notice any similarities to previous work, and it seems like the current approach mitigates many concerns other methods have had.\n\n2. Quality\n\nI believe this paper is well written, has a clear motivation, strong experiments, and well-derived proofs and theorems. Not only does this paper propose a differentiable proxy to Lp calibration error with good guarantees, this proxy seems to provide good regularisation to empirical risk minimization, resulting in better classification performance on real-world datasets.\n\n3. Clarity\n\nI found the paper mostly easy to follow. I will admit that I found it a bit difficult to follow Appendix C, though this is due to my limited knowledge in proving such quantities. I feel like there wasn't much discussion about the bandwidth parameter $b$, and maybe explaining a little bit more how the leave-one-out MLE selection was performed would make this a bit clearer.\n\n4. Significance\n\nModel calibration is a very important field, and there has been much work on trying to find a differentiable proxy to ECE. To this extent, this paper accomplishes a significant milestone, as they propose a consistent, low-bias, differentiable approximator to Lp calibration error.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept