Year: 2021

Abstract:
Calibrated probabilistic classifiers are models whose predicted probabilities can directly be interpreted as uncertainty estimates. This property is particularly important in safety-critical applications such as medical diagnosis or autonomous driving. However, it has been shown recently that deep neural networks are poorly calibrated and tend to output overconfident predictions. As a remedy, we propose a trainable calibration error estimator based on Dirichlet kernel density estimates, which asymptotically converges to the true Lp calibration error. This novel estimator enables us to achieve the strongest notion of multiclass calibration, called canonical calibration, while other common calibration methods only allow for toplabel and marginal calibration. The empirical results show that our estimator is competitive with the state-of-the-art, consistently yielding tradeoffs between calibration error and accuracy that are (near) Pareto optimal across a range of network architectures. The computational complexity of our estimator is O(n), matching that of the kernel maximum mean discrepancy, used in a previously considered trainable calibration estimator (Kumar et al., 2018). By contrast, the proposed method has a natural choice of kernel, and can be used to generate consistent estimates of other quantities based on conditional expectation, such as the sharpness of an estimator.

heading: 1 INTRODUCTION
text: Deep neural networks have shown tremendous success in classification tasks, being regularly the best performing models in terms of accuracy. However, they are also known to make overconfident predictions (Guo et al., 2017), which is particularly problematic in safety-critical applications such as medical diagnosis or autonomous driving. Therefore, in many real world applications we do not just care about the predictive performance, but also about the trustworthiness of that prediction, that is, we are interested in accurate predictions with robust uncertainty estimates. To this end, we want our models to be uncertainty calibrated which means that, for instance, among all cells that have been predicted with a probability of 0.8 to be cancerous, in fact a fraction of 80 % belong to a malignant tumor.
Being calibrated, however, does not imply that the classifier achieves good accuracy. For instance, a classifier that always predicts the marginal distribution of the target class is calibrated, but will not be very useful in practice. Likewise, a good predictive performance does not ensure calibration. In particular, for a broad class of loss functions, risk minimization leads to asymptotically Bayes optimal classifiers (Bartlett et al., 2006). However, there is no guarantee that they are calibrated, even in the aysmptotic limit. Therefore, we consider minimizing the risk plus a term that penalizes miscalibration, i.e., Risk +λ · CalibrationError. For parameter values λ > 0, this will push the classifier towards a calibrated model, while maintaining similar accuracy. The existence of such a λ > 0 is suggested by the fact that there always exists at least one Bayes optimal classifier that is calibrated, namely P(y|x). To optimize the risk and the calibration error jointly, we propose a differentiable and consistent estimator of the expected Lp calibration error based on kernel density estimates (KDEs). In particular, we use a Beta kernel in binary classification tasks and a Dirichlet kernel in the multiclass setting,
as these kernels are the natural choices to model density estimation over a probability simplex. Our Dirichlet kernel based estimator allows for the estimation of canonical calibration, which is the strongest notion of multiclass calibration as it implies the calibration of the whole probability vector (Bröcker, 2009; Appice et al., 2015; Vaicenavicius et al., 2019). By contrast, most other state-ofthe-art methods only achieve weaker versions of multiclass calibration, namely top-label (Guo et al., 2017) and marginal or class-wise calibration (Kull et al., 2019). The top-label calibration only considers the scores for the predictied class, while for marginal calibration the multiclass problem is split up into K one-vs-all binary ones, each of which is required to be calibrated according to the definition of binary calibration. In many applications marginal and canonical calibration are preferable to top-label calibration, since we often care about having reliable uncertainty estimates for more than just one class per prediction. For instance, in medical diagnosis we do not just care about the most likely disease a certain patient might have but also about the probabilities of other diseases.
Our contributions can be summarized as follows:
1. We develop a trainable calibration error objective using Dirichlet kernel density estimates, which can be minimized alongside any loss function in the existing batch stochastic gradient descent framework. 2. We propose to use our estimator to evaluate canonical calibration. Due to the scaling properties of Dirichlet kernel density estimation, and the tendency for probabilities to be concentrated in a relatively small number of classes, this becomes feasible in cases that cannot be estimated using a binned estimator. 3. We show on a variety of network architectures and two datasets that DNNs trained alongside an estimator of the calibration error achieve competitive results both on existing metrics and on the proposed measure of canonical calibration.

heading: 2 RELATED WORK
text: Calibration of probabilistic predictors has long been studied in many fields. This topic gained attention in the deep learning community following the observation in Guo et al. (2017) that modern neural networks are poorly calibrated and tend to give overconfident predictions due to overfitting on the NLL loss. The surge of interest resulted in many calibration strategies that can be split in two general categories, which we discuss subsequently. Post-hoc calibration strategies learn a calibration map of the predictions from a trained predictor in a post-hoc manner. For instance, Platt scaling (Platt, 1999) fits a logistic regression model on top of the logit outputs of the model. A special case of Platt scaling that fits a single scalar, called temperature, has been popularized by Guo et al. (2017) as an accuracy-preserving, easy to implement and effective method to improve calibration. However, it has the undesired consequence that it clamps the high confidence scores of accurate predictions (Kumar et al., 2018). Other approaches for post-hoc calibration include: histogram binning (Zadrozny & Elkan, 2001), isotonic regression (Zadrozny & Elkan, 2002), and Bayesian binning into quantiles (Naeini & Cooper, 2015). Trainable calibration strategies integrate a differentiable calibration measure into the training objective. One of the earliest approaches is regularization by penalizing low entropy predictions (Pereyra et al., 2017). Similarly to temperature scaling, it has been shown that entropy regularization needlessly suppresses high confidence scores of correct predictions (Kumar et al., 2018). Another popular strategy is MMCE (Maxmimum Mean Calibration Error) (Kumar et al., 2018), where the entropy regularizer is replaced by a kernel-based surrogate for the calibration error that can be optimized alongside NLL. It has been shown that label smoothing (Szegedy et al., 2015; Müller et al., 2020), i.e. training models with a weighted mixture of the labels instead of one-hot vectors, also improves model calibration. Liang et al. (2020) propose to add the difference between predicted confidence and accuracy as auxiliary term to the cross-entropy loss. Focal loss (Mukhoti et al., 2020; Lin et al., 2018) has recently been empirically shown to produce better calibrated models than many of the alternatives, but does not estimate a clear quantity related to calibration error.
Kernel density estimation (Parzen, 1962; Rosenblatt, 1956) is a non-parametric method to estimate a probability density function from a finite sample. Zhang et al. (2020) propose a KDE-based estimator of the calibration error for measuring calibration performance. However, they use the triweight kernel, which has a limited support interval and is therefore applicable to binary classification, but does not have a natural extension to higher dimensional simplexes, in contrast to the Dirichlet kernel
that we consider here. As a result, they consider an unnatural proxy to marginal calibration error, which does not result in a consistent estimator.

heading: 3 METHODS
text: The most commonly used loss functions are designed to achieve consistency in the sense of Bayes optimality under risk minimization, however, they do not guarantee calibration - neither for finite samples nor in the asymptotic limit. Since we are interested in models f that are both accurate and calibrated, we consider the following optimization problem bounding the calibration error CE(f):
f = arg min f∈F
Risk(f), s.t. CE(f) ≤ B (1)
for some B > 0, and its associated Lagrangian
f = arg min f∈F
( Risk(f) + λ · CE(f) ) . (2)
We measure the (mis-)calibration in terms of the Lp calibration error. To this end, let (Ω,A,P) be a probability space, let X = Rd,Y = {0, 1, ...,K}. Let x : Ω → X and y : Ω → Y be random variables while realizations are denoted with subscripts. Furthermore, let f : X → 4K be a decision function, where 4K denotes the K dimensional simplex as is achieved e.g. from the output of a final softmax layer in a neural network. Definition 3.1 (Calibration error, (Naeini et al., 2015; Kumar et al., 2019; Wenger et al., 2020)). The Lp calibration error of f is:
CEp(f) = ( E [∥∥∥E[y | f(x)]− f(x)∥∥∥p
p
]) 1 p
. (3)
We note that we consider multiclass calibration, and that f(x) and the conditional expectation in Equation 3 therefore map to points on a probability simplex. We say that a classifier f is perfectly calibrated if CEp(f) = 0. Kumar et al. (2018) have also considered a minimization problem similar to Equation 2. Instead of using the CEp they use a metric called maximum mean calibration error (MMCE) that is 0 if and only if CEp = 0. However, it is unclear how MMCE relates to the canonical multiclass setting or to the norm parameter p for non-zero CEp.
In order to optimize Definition 3.1 directly, we need to perform density estimation over the probability simplex in order to empirically compute the conditional expectation. In a binary setting, this has traditionally been done with binned estimates (Naeini et al., 2015; Guo et al., 2017; Kumar et al., 2019). However, this is not differentiable w.r.t. the function f , and cannot be incorporated into a gradient based training procedure. Furthermore, binned estimates suffer from the curse of dimensionality and do not have a practical extension to multiclass settings. A natural choice for a differentiable kernel density estimator in the binary case is a kernel based on the Beta distribution and the extension to the multiclass case is given by the Dirichlet distribution. Hence, we consider an estimator for the CEp based on Beta and Dirichlet kernel density estimates in the binary and multiclass setting, respectively. We require that this estimator is consistent and differentiable such that we can train it according to Equation 2. This estimator is given by:
̂CEp(f)p = 1
n n∑ h=1 [∥∥∥ ̂E[y | f(x)]∣∣∣ f(xh) − f(xh) ∥∥∥p p ] , (4)
where ̂E[y | f(x)] ∣∣∣ f(xh)
denotes ̂E[y | f(x)] evaluated at f(x) = f(xh). If Px,y has a probability density px,y with respect to the product of the Lebesgue and counting measure, we can define: px,y(xi, yi) = py|x=xi(yi) px(xi). Then we define the estimator of the conditional expectation as follows:
E[y | f(x)] = ∑ yk∈Y yk py|x=f(x)(yk) =
∑ yk∈Y yk px,y(f(x), yk)
px(f(x)) (5)
≈ ∑n i=1 k(f(x); f(xi))yi∑n i=1 k(f(x); f(xi)) =: ̂E[y | f(x)] (6)
where k is the kernel of a kernel density estimate evaluated at point xi.
Proposition 3.2. ̂E[y | f(x)] is a pointwise consistent estimator of E[y | f(x)], that is:
lim n→∞ ∑n i=1 k(f(x); f(xi))yi∑n i=1 k(f(x); f(xi)) = ∑ yk∈Y yk px,y(f(x), yk) px(f(x)) . (7)
Proof. By the consistency of kernel density estimators (Silverman, 1986; Chen, 1999; Ouimet & Tolosana-Delgado, 2021), for all f(x) ∈ (0, 1), 1n ∑n i=1 k(f(x); f(xi))yi
n→∞−−−−→∑ yk∈Y yk px,y(f(x), yk) and 1 n ∑n i=1 k(f(x); f(xi))
n→∞−−−−→ px(f(x)). The fact that the ratio of two convergent sequences converges against the ratio of their limits shows the result.
Mean squared error in binary classification As a first instantiation of our framework we consider a binary classification setting, with the mean squared error MSE(f) = E[(f(x) − y)2] as the risk function, jointly optimized with the L2 calibration error CE2. Following Murphy (1973); Degroot & Fienberg (1983); Kuleshov & Liang (2015); Nguyen & O’Connor (2015) we decompose (full derivation in Appendix A) the MSE as:
MSE(f)− CE2(f)2 = E [( 1− E[y | f(x)] ) E[y | f(x)] ] ≥ 0. (8)
Similar to Equation 2, we consider the optimization problem for some λ > 0:
f = arg min f∈F
( MSE(f) + λCE2(f) 2 ) . (9)
Using Equation 8 we rewrite:
MSE(f) + λCE2(f) 2 =(1 + λ) MSE(f)− λ ( MSE(f)− CE2(f)2 ) (10)
=(1 + λ) MSE(f)− λE [( 1− E[y | f(x)] ) E[y | f(x)] ] . (11)
Rescaling Equation 11 by a factor of (1 + λ)−1 and a variable substitution γ = λ1+λ ∈ [0, 1)
f = arg min f∈F
( MSE(f) + λCE2(f) 2 )
= arg min f∈F
( MSE(f)− γE [( 1− E[y | f(x)] ) E[y | f(x)] ]) (12)
= arg min f∈F
( MSE(f) + γE [ E[y | f(x)]2 ]) . (13)
For optimization we wish to find an estimator for E[E[y | f(x)]2]. Building upon Equation 6, a partially debiased estimator can be written as:1
̂ E [ E[y | f(x)]2 ] ≈ 1 n n∑ h=1
(∑ i 6=h k(f(xh); f(xi))yi )2 − ∑ i 6=h (k(f(xh); f(xi))yi)
2(∑ i 6=h k(f(xh); f(xi)) )2 − ∑ i6=h (k(f(xh); f(xi))) 2 . (14)
In a binary setting, the kernels k(·, ·) are Beta distributions, i.e. denoting zi := f(xi) for short, then:
kBeta(z, zi) := z αi−1(1− z)βi−1 Γ(αi + βi)
Γ(αi) Γ(βi) , (15)
with αi = zih +1 and βi = 1−zi h +1 (Chen, 1999; Bouezmarni & Rolin, 2003; Zhang & Karunamuni, 2010), where h is a bandwidth parameter in the kernel density estimate that goes to 0 as n → ∞. We note that the computational complexity of this estimator is O(n2). Within the gradient descent training procedure, the density is estimated using a mini-batch and therefore the O(n2) complexity is w.r.t. a mini-batch, not the entire dataset.
The estimator in Equation 14 is a ratio of two second order U-statistics that converge as n−1/2 (Ferguson, 2005). Therefore, the overall convergence will be n−1/2. Empirical covergence rates are calculated in Appendix D.3 and shown to be close to the theoretically expected value.
1We have debiased the numerator and denominator individually (Ferguson, 2005, Section 2), but for simplicity have not corrected for the fact that we are estimating a ratio (Scott & Wu, 1981).
Multiclass calibration with Dirichlet kernel density estimates There are multiple definitions regarding multiclass calibration that differ in the strictness regarding the calibration of the probability vector f(x). The weakest notion is top label calibration, which, as the name suggests, only cares about calibrating the entry with the highest predicted probability, which reduces to a binary calibration problem again (Guo et al., 2017). Marginal or class-wise calibration (Kull et al., 2019) is the most commonly used definition of multiclass calibration and a stronger version of top label calibration. Here, the problem is split into K one-vs-all binary calibration setting, such that each class has to be calibrated against the other K − 1 classes:
MCEp(f) p = K∑ k=1 E [∣∣∣E[y = k | f(x)k]− f(x)k∣∣∣p]. (16)
An estimator for this calibration error is:
̂MCEp(f)p = K∑ k=1 1 n n∑ j=1 ∣∣∣∣∣ ∑ i 6=j kBeta(f(xj)k; f(xi)k)[yi]k∑ i 6=j kBeta(f(xj)k; f(xi)k) − f(xj)k ∣∣∣∣∣ p . (17)
The strongest notion of multiclass calibration, and the one that we want to consider in this paper, is called canonical calibration (Bröcker, 2009; Appice et al., 2015; Vaicenavicius et al., 2019). Here it is required that the whole probability vector f(x) is calibrated. The definition is exactly the one from Definition 3.1. Its estimator is:
̂CEp(f)p = 1
n n∑ j=1 ∥∥∥∥∥ ∑ i6=j kDir(f(xj); f(xi))yi∑ i 6=j kDir(f(xj); f(xi)) − f(xj) ∥∥∥∥∥ p
p
(18)
where kDir is a Dirichlet kernel defined as: kDir(z, zi) := Γ( ∑K i=1 αi)∏K
i=1 Γ(αi) K∏ j=1 z αij−1 j (19)
with αi = zi/h+ 1 (Ouimet & Tolosana-Delgado, 2021). As before, the computational complexity is O(n2) irrespective of p. This estimator is differentiable and furthermore, the following proposition holds: Proposition 3.3. The Dirichlet kernel based CE estimator is consistent, that is
lim n→∞
1
n n∑ j=1 ∥∥∥∥∥ ∑n i 6=j kDir(f(xj); f(xi))yi∑n i6=j kDir(f(xj); f(xi)) − f(xj) ∥∥∥∥∥ p
p
= E [∥∥∥E[y | f(x)]− f(x)∥∥∥p
p
]p . (20)
Proof. Dirichlet kernel estimators are consistent (Ouimet & Tolosana-Delgado, 2021), consequently, by Proposition 3.2 the term inside the norm is consistent for any fixed f(xj) (note, that summing over i 6= j ensures that the ratio of the KDE’s does not depend on the outer summation). Moreover, for any convergent sequence also the norm of that sequence converges against the norm of its limit. Ultimately, the outer sum is merely the sample mean of consistent summands, which again is consistent.

heading: 4 EMPIRICAL SETUP
text: We trained ResNet (He et al., 2015), ResNet with stochastic depth (SD) (Huang et al., 2016), DenseNet (Huang et al., 2018) and WideResNet (Zagoruyko & Komodakis, 2016) networks on CIFAR-10 and CIFAR-100 (Krizhevsky, 2009). We use 45000 images for training. The code will be released upon acceptance.
Baselines Cross-entropy: The first baseline model is trained using cross-entropy with the data preprocessing, training procedure and hyperparameters described in the corresponding paper for the architecture. Trainable calibration strategies MMCE (Kumar et al., 2018) is a differentiable measure of calibration with a property that it is minimized at perfect calibration. It is used as a regulariser alongside NLL, with the strength of regularization parameterized by λ. Focal loss (Mukhoti et al., 2020) is an alternative to the popular cross-entropy loss, defined as Lf = −(1 −
f(y|x))γ log(f(y|x)), where γ is a hyperparameter and f(y|x) is the probability score that a neural network f outputs for a class y on an input x. Their best-performing approach is the sampledependent FL-53 where γ = 5 for f(y|x) ∈ [0, 0.2) and γ = 3 otherwise, followed by the method with fixed γ = 3. Post-hoc calibration strategies Guo et al. (2017) investigated the performance of several post-hoc calibration methods and found temperature scaling to be a strong baseline, which we use as a representative of this group. It works by scaling the logits with a scalar T > 0, typically learned on a validation set by minimizing NLL. Following Kumar et al. (2018); Mukhoti et al. (2020), we also use temperature scaling as a post-processing step for our method.
Metrics The most widely-used metric for expected calibration error (ECE) is a binned estimator (Naeini et al., 2015), which divides the interval [0, 1] into bins of equal width and then calculates a weighted average of the absolute difference between accuracy and confidence for each bin. A better binning scheme involves determining the bin sizes so that an equal number of samples fall into each bin (Nguyen & O’Connor, 2015; Mukhoti et al., 2020). We report the ECE (%) with 15 bins calculated according to the latter, so-called adaptive binning procedure. We compute the 95% confidence intervals using 100 bootstrap samples as in Kumar et al. (2019). We consider multiple versions of the ECE metric based on the Lp norm and the type of calibration (top-label, marginal, canonical). Top-label calibration error only considers the probability of the predicted class, marginal requires per-class calibration and the canonical is the highest form of calibration which requires the entire probability vector to be calibrated. We report L1 and L2 ECE in the marginal and canonical case. Additional experiments with top-label and marginal calibration on both CIFAR-10 and CIFAR100 can be found in Appendix B.
Hyperparameters A crucial parameter for KDE is the bandwidth, a positive number that defines the smoothness of the density plot. Poorly chosen bandwidth may lead to undersmoothing (small bandwidth) or oversmoothing (large bandwidth). A commonly used non-parametric bandwidth selector is maximum likelihood cross validation (Duin, 1976). For our experiments we choose the bandwidth from a list of possible values by maximizing the leave-one-out likelihood. The λ parameter for weighting the calibration error w.r.t the loss is typically chosen via cross-validation or using a holdout validation set. The p parameter is chosen depending on the desired Lp calibration error and the corresponding theoretical guarantees.

heading: 5 RESULTS AND DISCUSSION

heading: 5.1 BINARY CLASSIFICATION
text: We construct a binary experiment by splitting the CIFAR-10 classes into 2 classes: vehicles (plane, automobile, ship, truck) and animals (bird, cat, deer, dog, frog, horse). Figure 1a shows how the choice of the bandwidth parameter influences the shape of the estimate.
Figure 1b shows the effect of the regularization parameter γ on the performance of a ResNet-110 model. The orange point represents a model trained with MSE loss, and the blue points (KDE-MSE) correspond to models trained with regularized MSE loss by an L2 calibration error for different values of γ. As expected, the calibration regularized training decreases the L2 calibration error at the cost of slightly increased error.

heading: 5.2 EVALUATING CANONICAL CALIBRATION
text: Accurately evaluating the calibration error is another crucial step towards designing trustworthy models that can be used in high-cost settings. In spite of its numerous flaws discussed in Vaicenavicius et al. (2019); Ding et al. (2020); Ashukha et al. (2021), such as its sensitivity to the binning scheme, the histogram-based estimator remains the most widely used metric for evaluating miscalibration. Another downside of the binned estimator is its inability to capture canonical calibration due to the curse of dimensionality, as the number of bins grows exponentially with the number of classes. Therefore, because of its favourable scaling properties, we propose using our Dirichlet kernel density estimate as an alternative metric (KDE-ECE) to measure calibration. To investigate its relationship with the commonly used binned estimator, we first introduce an extension of the top-label binned estimator to the probability simplex in the three class setting. We start by partitioning the probability simplex into equally-sized, triangle-shaped bins and assign the probability scores to the corresponding bin, as shown in Figure 2a. Then, we define the binned estimate of canonical calibration error as follows:
CEp(f) p ≈ E [ ‖H(f(x))− f(x)‖pp ] ≈ 1 n n∑ i=1 ‖H(f(xj))− f(xi)‖pp (21)
where H(f(xj)) is the histogram estimate, shown in Figure 2b. The surface of the corresponding Dirichlet KDE is presented in Figure 2c. In Figure 3 we show that the KDE-ECE estimates of the three types of calibration closely correspond to the their histogram-based approximations. Each point in the plot represents a ResNet-56 model trained on a different subset of three classes from CIFAR-10. See Appendix C for another example of the binned estimator and Dirichlet KDE on CIFAR-10 and an experiment with varying number of points used for the density estimation.

heading: 5.3 MULTICLASS CLASSIFICATION
text: In this section we evaluate our proposed KDE-based ECE estimator that was jointly trained with cross entropy loss (KDE-CRE) against other baselines in a multiclass setting on CIFAR-10 and CIFAR-100. We found that for KDE-CRE, values of λ ∈ [0.01, 0.1] provide a good trade-off in terms of accuracy and calibration error. Table 1 summarizes the accuracy and marginal L1 ECE% (computed using 15 bins), measured across multiple architectures. For MMCE, we report the results with λ = 1 and for KDE-CRE we use λ = 0.01. An analogous table measuring marginal L2 ECE is given in Appendix B.
We notice that for both pre and post temperature scaling, KDE-CRE achieves very competitive ECE scores. Another encouraging observation is that the improvement of calibration error comes at almost no cost in accuracy. An important advantage of our KDE-based method is the ability to directly train and evaluate canonical calibration. In Figure 4 we show a scatter plot with confidence intervals of the L1 and L2 KDE-CRE models for canonical calibration and the other baselines on CIFAR-10. We measure the canonical calibration using our KDE-ECE metric from section 5.2. In three of the architectures, both L1 and L2 KDE-CRE either dominate or are statistically tied with cross-entropy (CRE). Similarly, Figure 5 shows a scatter plot of L1 and L2 KDE-CRE models trained to minimize marginal calibration error. In this case, we measure L2 marginal ECE with the standard binned estimator. In most cases, our methods Pareto dominate the other baselines. A general observation can be made, however, that the models trained with cross-entropy have a surprisingly low marginal calibration error, contrary to previous findings that show poor calibration when considering only the most confident prediction (top-label calibration). An additional experiment comparing the CRE baseline with KDE-CRE for canonical calibration on a benchmark dataset of histological images of human colorectal cancer is given in Appendix D.2, which clearly illustrates the superior performance of our method, both in terms of accuracy and calibration error in this context. To summarize, the experiments show that our estimator is consistently producing competitive calibration errors with other state-of-the-art approaches, while maintaining accuracy and keeping the computational complexity at O(n2). We evaluate the computational overhead of CRE and KDECRE and summarize the results in a table in Appendix D.1, which shows that the added cost is less than a couple percent. There are several limitations in the current work: A larger scale benchmarking will be beneficial for exploring the limits of canonical calibration using Dirichlet kernels. Furthermore, while we showed consistency of our estimator, we did not fully derive and implement its debiasing. Due to space constraints, this was not the focus of the paper and is left for future work.

heading: 6 CONCLUSION
text: In this paper, we proposed a consistent and differentiable estimator of an Lp calibration error using Dirichlet kernels. The KDE-based estimate can be directly optimized alongside any loss function in the existing batch stochastic gradient descent framework. Furthermore, we propose using it as a mea-
sure of the highest form of calibration which requires the entire probability vector to be calibrated. We showed empirically on a range of neural architectures that the performance of our estimator in terms of accuracy and calibration error is competitive against the current state-of-the-art, while having superior properties as a consistent estimator of canonical calibration error.

heading: A DERIVATION OF THE MSE DECOMPOSITION
text: Definition A.1 (Mean Squared Error (MSE)). The mean squared error of an estimator is MSE(f) := E[(f(x)− y)2]. (22)
Proposition A.2. MSE(f) ≥ CE2(f)2
Proof.
MSE(f) :=E[(f(x)− y))2] = E[((f(x)− E[y | f(x)]) + (E[y | f(x)]− y))2] (23) =E[(f(x)− E[y | f(x)])2]︸ ︷︷ ︸
=CE22
+E[(E[y | f(x)]− y)2] (24)
+ 2E[(f(x)− E[y | f(x)])(E[y | f(x)]− y)] which implies
MSE(f)− CE2(f)2 =E[(E[y | f(x)]− y)2] (25) + 2E[(f(x)− E[y | f(x)])(E[y | f(x)]− y)]
=E[(E[y | f(x)]− y)2] + 2E[(f(x)E[y | f(x)]] (26) − 2E[f(x)y]− 2E[E[y | f(x)]2] + 2E[E[y | f(x)]y]]
=E[E[y | f(x)]2] + E[y2]− 2E[E[y | f(x)]y] (27) + 2E[(f(x)E[y | f(x)]]− 2E[f(x)y] − 2E[E[y | f(x)]2] + 2E[E[y | f(x)]y]]
=E[y2] + 2E[(f(x)E[y | f(x)]]− 2E[f(x)y] (28) − E[E[y | f(x)]2]
=E[(2f(x)− y − E[y | f(x)])(E[y | f(x)])− y] (29) =E[(f(x)− y)(E[y | f(x)]− y)] (30)
+ E[(f(x)− E[y | f(x)])(E[y | f(x)]− y)]. By the law of total expectation, we will write the above as
MSE(f)− CE2(f)2 = E[E[(f(x)− y)(E[y | f(x)]− y) (31) + (f(x)− E[y | f(x)])(E[y | f(x)]− y) | f(x)]].
Focusing on the inner conditional expectation, we have that
E[(f(x)− y)(E[y | f(x)]− y) + (f(x)− E[y | f(x)])(E[y | f(x)]− y) | f(x)] =E[y | f(x)](f(x)− 1)(E[y | f(x)]− 1) + (1− E[y | f(x)])f(x)E[y | f(x)]
+ E[y | f(x)](f(x)− E[y | f(x)])(E[y | f(x)]− 1) + (1− E[y | f(x)])(f(x)− E[y | f(x)])E[y | f(x)] (32)
=(1− E[y | f(x)])E[y | f(x)] ≥ 0 ∀f(x) (33) and therefore
MSE(f)− CE2(f)2 = E[(1− E[y | f(x)])E[y | f(x)]] ≥ 0. (34)
The expectation in Equation 34 is over variances of Bernoulli random variables with probabilities E[y | f(x)].

heading: B RESULTS
text: Table 2 summarizes the marginal L2 ECE and accuracy for the two datasets across multiple architectures and training loss functions. The scatter plots in Figures 6 and 7 show the accuracy and both L1 and L2 ECE, for top-label and marginal calibration on CIFAR-10 and CIFAR-100, respectively. KDE-CRE is trained by directly minimizing the metric that is evaluated, e.g., in the first column we minimize marginal L1 calibration error and in the last column we optimize the L2 top label calibration error. Other methods do not have the flexibility of choosing the type of calibration and the Lp norm.

heading: C RELATIONSHIP BETWEEN THE BINNED ESTIMATOR AND THE KERNEL DENSITY ESTIMATOR
text: Figure 8 shows an example of the binned estimator in a three-class setting on CIFAR-10. The points are mostly concentrated at the edges of the histogram, as can be seen from Figure 8b. The surface of the corresponding Dirichlet KDE is given in 8c. Figure 9 shows the relationship between the binned estimator and our KDE-ECE metric. The points represent a trained Resnet-56 model on a subset of three classes from CIFAR-10. In every row, a differnt number of points was used to estimate the KDE-ECE.

heading: D EXPERIMENTS FOR REBUTTAL
text: D.1 TRAINING TIME MEASUREMENTS
In Table 3 we summarize the running time per epoch for training with (KDE-CRE) and without (CRE) regularization for the two datasets and four architectures. KDE-CRE does not create an overhead of more than a couple percent over the CRE baseline.
D.2 CANONICAL CALIBRATION IN A MEDICAL APPLICATION
An additional experiment with a medical application, where the canonical calibration is of particular interest, was performed on the publicly-available Kather dataset (Kather et al., 2016), which consists of 5000 histological images of human colorectal cancer. The data has eight different classes of tissue. Figure 10 shows a comparison in performance of the CRE baseline with our KDE-CRE method. The canonical L1 (left) and L2 (right) calibration is measured using our KDE-ECE metric. The results clearly illustrate that our method significantly outperforms the cross-entropy baseline, both in terms of accuracy and calibration error, for several choices of the regularization parameter.
D.3 BIAS AND CONVERGENCE RATES
Figure 11 shows a comparison of the groud truth, computed from 3000 test points with KDE-ECE against KDE-ECE and binned ECE estimated with a varying number of points used for the estima-
tion. The used model is a ResNet-56, trained on a subset of three classes from CIFAR-10. The figure shows that the two estimates are comparable and both are doing a reasonable job. Figure 12 shows the absolute difference between the ground truth and estimated ECE using our KDE estimator and a binned estimator with varying number of points used for estimation. The results are
averaged over 120 ResNet-56 models trained on a subset of three classes from CIFAR-10. Both estimators are biased and have some variance, and the plot shows that the combination of the two is in the same order of magnitude. The empirical convergence rates (slope of the log-log plot) is given in the legend and is shown to be close to the theoretically expected value of -0.5.
D.4 CHOICE OF THE BATCH SIZE
In Figure 13 we investigate the choice of the batch size on CIFAR-10. To this end, we use two differently shuffled dataloaders that draw random batches from the same training set. The first dataloader provides batches to the loss term (CRE) while the second dataloader provides the batches for the regularization (KDE). The batch size for the loss term is fixed in all experiments, while the
batch size for the regularization varies. The orange point is our normal experimental set-up with just one dataloader (i.e. the same points are used for loss and KDE-ECE computation) as a comparison. The plot shows that our chosen batch size of 128 is appropriate for our purposes.

Review summaries:
The paper proposes a new approach for calibrating neural network outputs. The idea is to train the neural network with a regularized loss function that is a linear combination of prediction and calibration errors. The calibration error is measured as the Lp norm of the difference between predicted class probabilities and the expected true class probabilities given the predicted class probabilities where the latter term is computed using a kernel density estimate with a Dirichlet Kernel. The approach is evaluated in terms of accuracy and expected calibration error on CIFAR-10 and CIFAR-100.

The paper proposes a regularization term to augment loss functions, where the regularization term effectively minimizes the calibration error of the model. The term itself is a kernel density estimator over the K-simplex space (hence Dirichlet kernel is the natural choice).  The authors claim the estimator is consistent (but not unbiased, though they partially debias it) and empirically verify that their method yields tradeoff between accuracy and calibration that is near Pareto optimal.