ESD: Expected Squared Difference as a Tuning-Free Trainable Calibration Measure
Authors: Hee Suk Yoon
Joshua Tian Jin Tee
Eunseop Yoon
Sunjae Yoon
Gwangsu Kim
Yingzhen Li
Chang D. Yoo
Venue: ICLR 2023 poster (2023)
Keywords: calibration

Abstract: Studies have shown that modern neural networks tend to be poorly calibrated due to over-confident predictions. Traditionally, post-processing methods have been used to calibrate the model after training. In recent years, various trainable calibration measures have been proposed to incorporate them directly into the training process. However, these methods all incorporate internal hyperparameters, and the performance of these calibration objectives relies on tuning these hyperparameters, incurring more computational costs as the size of neural networks and datasets become larger. As such, we present Expected Squared Difference (ESD), a tuning-free (i.e., hyperparameter-free) trainable calibration objective loss, where we view the calibration error from the perspective of the squared difference between the two expectations. With extensive experiments on several architectures (CNNs, Transformers) and datasets, we demonstrate that (1) incorporating ESD into the training improves model calibration in various batch size settings without the need for internal hyperparameter tuning, (2) ESD yields the best-calibrated results compared with previous approaches, and (3) ESD drastically improves the computational costs required for calibration during training due to the absence of internal hyperparameter. The code is publicly accessible at https://github.com/hee-suk-yoon/ESD.

Published as a conference paper at ICLR 2023
ESD: EXPECTED SQUARED DIFFERENCE AS A TUNING- FREE TRAINABLE CALIBRATION MEASURE
Hee Suk Yoon
Joshua Tian Jin Teel
Eunseop Yoon'
Sunjae Yoon
Gwangsu Kim!
Yingzhen Li Chang D. You
KKorea Advanced Institute of Science and Technology (KAIST) 2 Imperial College London
{hskyoon, joshuateet j, esyoon97, sun jae @kais st.stttt ac.kr
2000120gmaii com yingzhen. 1i0 imperial. 1angzheen.@@mmmmmal.... uk cd.yoo@kaist. kr
ABSTRACT
Studies have shown that modern neural networks tend to be poorly calibrated due to over-confident predictions. Traditionally, post-processing methods have been used to calibrate the model after training. In recent years, various trainable calibration measures have been proposed to incorporate them directly into the training process. However, these methods all incorporate internal hyperparameters, and the performance of these calibration objectives relies on tuning these hyper- parameters, incurring more computational costs as the size of neural networks and datasets become larger. As such, we present Expected Squared Difference (ESD), tuning-fiees frre (i.c., hyperparameter-ffre) trainable calibration objective loss, where we view the calibration error from the perspective of the squared difference between the expectations. With extensive experiments (CNNs, Transformers) and datasets, we demonstrate that (1) incorporating ESD into the training improves model calibration in various batch size settings without the need for internal hyperparameter tuning, (2) ESD yields the best-calibrated results compared with previous approaches, and (3) ESD drastically improves the computational costs required for calibration during training due to the absence of internal hyperparameter. The code is publicly accessible at htttp::/ggihuu.coom////// suk-yoon/ESD
several architectures
INTRODUCTION
The calibration of neural network measures the extent which its predictions align with the true probability distribution. Possessing this property becomes especially important in real-world applications, such as identification (Kim & Yoo! [017] Yoon al. [2022) autonomous driving (Bojarski et al., 2016 Ko et al. (017). and medical diagnosis (Kocbek et al., 20200 Pham al. 2022), where uncertainty- based decisions the neural network are crucial to guarantee the safety of the users. However, despite the success of modern neural networks in accurate classification, they are shown to be poorly calibrated due to the tendency of the network to make predictions with high confidence regardless of the input. (i.c., over-confident predictions) (Guo al. 2017).
Traditionally, postpprocsssing methods have been used, such as temperature scaling and vector scaling (Guo et al. 2017), to calibrate the model using the validation after the training by adjusting the logits before the final softmax layer. Various trainable calibration objectives have been proposed recently, such as MMCE (Kumar al. 2018) and SB-ECE (Karandikar al.) 2021), which are added to the loss function as a regularizer to jointly optimize accuracy and calibration during training. key advantage of calibration during training is that it is possible to cascade post- post-processing calibration methods after training to achieve even cetter-calibrated models. Unfortunately. these existing approaches introduce additional hyperparameters in their proposed calibration objectives, and the performance of the calibration objectives is highly sensitive to these design choices. Therefore these hyperparameters need to be tuned carefully on a per model per dataset basis, which greatly reduces their viability for training large models and datasets.
"Equal contribution Corresponning Author

---

Published as a conference paper at ICLR 2023
To this end, propose Expected Squared Difference (ESD), trainable calibration objective loss that is hyperparameter- free. ESD is inspired by the KSS-rroo (Gupta et al. (021), and it views the calibration error from the perspective of the difference between the two expectations. In detail, our contributions can be summarized as follows:
We propose ESD as trainable calibration objective loss that can be jointly optimized with the negative lig-liielihoo loss (NLL) during training. ESD is a binning-foe -ree calibration objective loss, and additional hyperparameters are required. We also provide an unbiased and consistent estimator of the Expected Squared Difference, and show that it can be utilized in small batch train settings.
With extensive experiments, we demonstrate that across various architectures (CNNs Transformers) and datasets (in vision & NLP domains), ESD provides the best calibration results when compared to previous approaches. The calibrations of these models are further improved by post-processigg methods.
We show that due to the absence of an internal hyperparameter in ESD that needs to be tuned, it offers a drastic improvement compared to previous calibration objective losses with regard the total computational cost for training. The discrepancy computational cost between ESD and tuning-requing calibration objective losses becomes larger as the model complexity and dataset size increases.
2 RELATED WORK
Calibration of neural networks has gained much attention following the observation from Guo et al. (2017) that modern neural networks are poorly calibrated. One way to achieve better calibration is to design better neural network architectures tailored for uncertainty estimation, e.g., Bayesian Neural Networks (Blundell et al. 2015 Gal & Ghahramani (016) and Deep Ensembles (Lakshminarayanan
al. 2017). Besides model design, post-processing calibration strategies have been widely used to calibrate a trained machine learning model using hold-out validation dataset. Examples include temperature scaling (Guo al. (017). which scales the logit output of a classifier with a temperature parameter, Platt scaling (Platt 1999). which fits logistic regression model on top of the logits; and Conformal prediction (Vovk et al.] |005| Lei et al. 2018) which uses validation set to estimate the quantiles of given scoring function. Other post-processing techniques include histogram binning (Zadrozny Elkan 200T), isotonic regression (Zadrozny & Elkan 2002), and Bayesian binning into
quantiles Pakddaman Nacini et al. 2015).
Our work focuses on trainable calibration methods, which train neural networks using a hybrid objective, combining primary training loss with auxiliary calibration objective loss. In this regard, one popular objective is Maximum Mean Calibration Error (MMCE) (Kumar et al. 2018), which is kernel umbedddingbbased measure of calibration that is differentiable and, therefore, suitable calibration loss. Moreover, Karandikar et al. (202T) proposes a trainable calibration objective loss, SB-ECE, and S-AvUC, which softens previously defined calibration measures.
3 PROBLEM SETUP
3.1 CALIBRATION ERROR AND METRIC
Let us first consider an arbitrary neural network as fo D D 1] with network parameters f, where D is the input domain and C' is the number of classes in the multiclass classification task. Furthermore, we assume that the training data, (x, yi Di) are sampled i.i.d. from the joint distribution IP(X, Y) (here we use one-hot vector for y). Here, Y (Yi.... .... Yc), and y (1)1 .... UC) are samples from this distribution. We can further define multivariate random variable Z = fe(X) as the distribution of the outputs of the neural network. Similarly. Z (Z,... ..., Zc), and (21 ..., &c) are samples from this distribution. We use (kK,i to denote the output confidence and the one-hot vector
element associated with the K-th class of the i-th training sample. Using this formulation, a neural network is said to be perfectly calibrated for class K, if and only if
PP(KK 1 =ạ) 2K-
(1)
2

---

Published as a conference paper at ICLR 2023
Intuitively, Eq. ① requires the model accuracy for class K to be 2 on average for the inputs where the neural network produces prediction of class K with confidence 2x. In many cases, the research of calibration mainly focuses the calibration of the max output class that the model predicts. Thus, calibration error is normally reported with respect to the predicted class (i.e., max output class) only. As such, k will denote the class with maximum output probability. Furthermore, as the indicator function which returns one if the Boolean expression is true and zero otherwise.
also write T(-)
With these notations, one common measurement of calibration error the difference between confidence and accuracy which is mathematically represented as (Guo et al. 2017)
Ez, [IP(Y& 1|Z)) Z
To estimate this, the Expected Calibration Error (ECE) (Nainn et al. 2015) uses B number of bins with disjoint intervals By
(
] j- = 0, 1,.1,,.. B 1 to compute the calibration error as follows:
B
N
1
1 < B
jt1
B .kk (00d=11 =1) zx,i)
ECE
Th,4 S
D
1=
3.2 CALIBRATION DURING TRAINING
post-processing method and calibration during training are the two primary approaches for calibrating a neural network. Our focus in this paper is on the latter, where our goal is train a calibrated yet accurate classifier directly. Note that calibration and predictive accuracy are independent properties of a classifier. In other words, being calibrated does not imply that the classifier has good accuracy and vice versa. Thus, training classifier to have both high accuracy and good calibration requires jointly optimizing a calibration objective loss alongside the negative log- log-liielioodd (NLL) with a scaling parameter À for the secondary objective:
min NLL 0) + A CalibrationOOjeccivee.... O). 8
3.2.1 EXISTING TRAINABLE CALIBRATION OBJECTIVES NEED TUNING
Kumar et al. (2018) and Karandikar et al. (202I) suggested that the disjoint bins in ECE can introduce discontinuities which are problematic when using it calibration loss in training. Therefore, additional parameters were introduced for the purpose of calibration during training. For example,
Kumar et al. (2018) proposed Maximum Mean Calibration Error (MMCE), where it utilizes Laplacian Kernel instead of the disjoint bins in ECE. Furthermore, Karandikan et al. (2021) proposed soft-binned ECE (SB-ECE) and soft AvUC, which are softened versions of the ECE metric and the AvUC loss (Krishnan & Tickoo 2020), respectively. All these approaches address the issue of discontinuity which makes the training objective differentiable. However they all introduce additional design choices MMCE requires careful selection of the kernel width ø, SB-ECE needs to choose the number of bins M and softening parameter T, and S-AvUC requires a user-specified entropy threshold k in addition the softening parameter T'. Searching for the optimal hyperparameters can be computationally expensive especially as the size models and dataset become larger.
3.2.2 CALIBRATION DURING TRAINING SUFFERS FROM OVER- HITTING PROBLEM
NLL loss is known implicitly train for calibration since it is a proper scoring rule, so models trained with NLL can overfit in terms of calibration error (Mukhoti et al. 2020 Karandikar et al.) 202T). Figure provides an example of an accuracy curve and its corresponding ECE curve for model trained with N.L.loss, We see that the model is overfitting in terms of both accuracy and ECE, which causes the gap between train and test ECE become larger during training. As such, adding a calibration objective the total loss will not be able to improve model calibration it mainly helps improve calibration of the model with respect to the training data only.
Kumar et al. (2018) tackle this issue by introducing a weighted version of their calibration objective loss. They consider using larger weights to incorrect prediction samples after observing that the fraction of incorrect to the correct samples the training data is smaller than that of the validation and test data. Instead of changing the calibration objective function itself, Karandikar et al. (202T) introduced new training scheme called interleaved training, where they split the training data
3

---

Published as a conference paper at ICLR 2023
Acc. Train set Acc Test eet
EGE of Train set
EDS of Text ser
Accuracy
20-
ECE
18-
M.
10-
Epoch
Epech
Figure I: Accuracy (%) curve (left) and its corresponding ECE (%) curve (right) during training with negative engllikelihood (NLL) loss. It could be seen that since NLL implicitly trains for calibration error, the ECE of the train set approaches zero while the ECE of the test set increases during training.
further and dedicate a small portion of it to optimize the calibration objective loss. We will follow this strategy and introduce the technical details in the later sections.
ESD: EXPECTED SQUARED DIFFERENCE
We propose Expected Squared Difference (ESD) as a tuning-free (i.e., hyperparameter- -ree) calibra- tion objective. Our approach is inspired by the viewpoint calibration error as a measure of distance between distributions to obtain binning-free calibration metric (Gupta al. [021]. By using this approach, there is no need to employ kernels or softening operations to handle the bins within calibration metries, such as ECE, in order to make them suitable for training.
In particular, we consider calibration error as the difference between the two expectations. We first start with the definition of perfect calibration in Eq. []:
P(, 1 2k) 2
VZz E [0, 1] Vzz E [0
(5)
+ PY, = 1, Z =x)) 24 P( =x)
(by Bayes rule).
Now considering the accumulation of the terms both sides for arbitrary confidence level [0, 1]. the perfect calibration for an arbitrary class ho can now be written as:
/ P(2 1, 24 = zu zz)dz = [ ZZP = )=
(6)
EZ. Y. (Z S a, Yv = 1)] = Ez, 12x S a)],
Va E [0,
This allows us to write the difference between the two expectations as
dx(a) I. P(Y 1. = 24) 24,1(( Z5 = )dz
EZ (1 S a) 1) Z+)]l
and (a) = 0, Va [0, 1] if and only if the model is perfectly calibrated for class k. Since this has to hold Va E [0, 1], we propose ESD as the expected squared difference between the two expectations:
(4x
Ez, EE
v. (Z < ())(1 =) -Z))]]-
Due to the close relationship between (a) and calibration of neural network, the difference between an uncalibrated and a calibrated neural network can be clearly observed using Ez, [de (Zz) visually shown in Appendix A Since ESD 0 iff. the model is perfectly calibrated shown in the following theorem, this metric is a good measure of calibration.
Theorem 1. 1Zzz [ V.
[TZ & (T(( = 1) - Z+)]] = 0 iff. the model is perfectly calibrated.
Yv. [[(Z < 2,)(1(Y, 1) Zx)] is non-negative random variable induced by Z Ez; (Z,) (Z,)² 0 iff. P(dx (z)) 0) Furthermore,
Proof. Since dy (Z,)2 E2 2,
P(da(Z)) 0) = da(a) Va I where I is the support set of 7
4

---

Published as a conference paper at ICLR 2023
Thus,
Ez (d = iff. dx(a) = Va E..
From lemma 1.1 below, we have (a) = 0 Va E [0 iff. (a) = 0 Va s.t. P(Z = a) +
Consequently, Ezz [B2,, (7 S (TV( = =1) 7+)]] iff. the model perfectly calibrated.
Lemma 1.1 Ler I be the support set of random variable Zk, then dx(a) = 0 vo E I iff dx(a)= = 0 Va E [0, 1].
Proof. The backward direction result is straight-forward, so we only prove for the forward direction: If (a) 0 Va E I, then (a) = Va E [0, 1].
For arbitrary af I" let arg min h la a'l and a' We then have,
P(Y 1, Z = 2x) P 2k)d2k dx(a = /
P(Yx: = 1|2x 2x) = zw) (Z dx)
P(Y; = 1/ P Z2 = 2x) P(Z = =z)
P(Y; = 1|2x = 2α )P(cs = =x) ExP(Zx = zz dzz
P(Yx 1 2x)P((xx =x) =xP(Zx 2xx zz)dzal
0.
4.1 AN ESTIMATOR FOR ESD
In this section, use (Zk, vk.,) to denote the output confidence and the one-hot vector element associated with the k-th class of the i-th training sample respectively. As the expectations in the true Expected Squared Difference (Eq. 8) intractable, we propose Monte Carlo estimator for it which is unbiased. A common approach to use naive Monte Carlo sampling with respect to both the inner and outer expectations give the following:
Ez E2,,, [1 S Z.) 1) Z+)I]
gr
(9)
N
where 9s N - Σ
gij and gij = I(zk.j I 24, [T,)) = k) 2n.
However, Eq. (9) results in a biased estimator that an upper bound of the true Expected Squared
Difference. To account for the bias, we propose the unbiased and consistent estimator (proof in Appendix B of the true Expected Squared Difference!
N g
S2
N Σ
ESD
where S
gi - g.
-1
N
(10)
341
'To avoid confusion, ESD from this point onward will refer the estimator instead of its expectation form.
5

---

Published as a conference paper at ICLR 2023
4.2 INTERLEAVED TRAINING
Negative log-likelihood (NLL) has be shown to greatly overfit to ECE of the data it is trained on. Thus, training for calibration using the same data for the NLL has limited effect on reducing the calibration error of the model. KKrandikar et al. (202T) proposed interleaved training where they split the train set into two subsets one is used to optimize the NLL and the other used optimize the calibration objective. Following this framework, let Ptrain denote the entire train set. We separate the train set into two subsets Dirain and D'al- The joint training of ESD with NLL becomes,
min NLL((Drrain 0) A ESD cal: 0)-
(11)
With this training scheme, NLL is optimized on Prraen and ESD is optimized on Deat This way, Karandikar et al. (2021) showed we can avoid minimizing ECE that is already overfit to the train set.
5
EXPERIMENTAL SETTING
5.1
DATASETS AND MODELS
Image Classification
For image classification tasks we use the following datasets:
images for train, validation, and test split was used. We resized the images to (32x32) before imputting the network.
MNIST (Deng 2012):
CIFARIO & CIFAR 100 (Krizhevsky al. a bb: 45, 45,000/5,000 (25550001100000 0,000 images for train, vali- dation, and test split was used. Used random cropping of 32 with padding of 4. Normalized each RGB channel with mean of 0.5 and standard deviation of 0.5.
ImageNet100 (Deng et al., 2009): A subset dataset from ImageNet Large Scale Visual Recognition Challenge 2012 with 100 classes. Since the labels of test sets are unavailable, we use the official validation set as the test set, and we dedicate 10% of the training data as the validation set for the experiments (i.e., 117,
split for train/val/test set).
Natural Language Inference (NLI) NLI is a task in Natural Language Processing where involves classifying the inference relation (entailment, contradiction, or neutral) between two texts (MacCartney & Manning 2008). For NLI tasks we use the following datasets:
SNLI (Bowman et al. 2015): SNLI corpus is a collection of human Human--witten English sentence pairs manually labeled for balanced classification with the labels entailment, contradiction, and neutral. The data consists of 550, 01211000001000000 sentence pairs for train/val/test set respectively. The max length of the input is set to 158.
ANLI (Nie et al. 2020): ANLI dataset a large-scale NLI dataset, collected via an, adver- sarial human-and -an--aanndddd-------ee model-in-the--e sentence pairs for train/val/tes set respectively. The max length of the input is set to 128.
procedure. The data consists of 162,865/3,200/3.22
For the Image Classification datasets, we used Convolutional Neural Networks (CNNs). Specifically, we used LeNet5 (Lecun et al. 1998), ResNet50, ResNet34, and ResNet18 (H et al. 2016), for MNIST, CIFAR10, CIFARION 100, and ImageNet100, respectively. For the NLI datasets, finetuned transformer based Pre-trained Language Models (PLMs). Specifically, we used Bert-base (Devlin al. 2019) and Roberta-base (Liu al.), for SNLI and ANLI, respectively.
5.2
EXPERIMENTAL SETUP
We compare our Expected Squared Difference (ESD) to previously proposed trainable calibration ob- jectives, MMCE and SB-ECE, on the datasets and models previously mentioned. For fair comparison, interleaved training has been used for all three calibration objectives. For MMCE, in which Kumar al. (0018) proposed an unweighted and weighted version of the objective, we use the former to set the method to account for the overfitting problem mentioned in section 3.2.2 consistent to interleaved training. For the interleaved training settings, we held out 10% of the train set to the calibration set. The regularizer hyperparameter X for weighting the calibration measure with respect to NLL is chosen via fixed grid search For measuring calibration error, we use ECE with 20 equally sized
Forr A, search for [0.2, 0.4, 0.6, 0.8, 1.0, 2.0, 3.0, ... 10.0] (Appendix

---

Published as a conference paper at ICLR 2023
Dataset (Model)
ECE
Acc.
ECE
Loss Fn.
Acc.
ECE
after TS
after VS
after VS
NLL (baseline)
98.810.033
0.91100080
0.3110.044
98.7100058
0.43110123
MNIST
+MMCE
98.4100.20
0.36:10029
0.33100068
98.41:0.250
0.32:10.055
(LeNet5)
+SB-ECE
97.9+0.177
0.41+00067
0.45++0.103
97.9+0.073
0.398
+ESD (ours)
98.6:#0.204
0.30
0.29-000330
98.6+0.167
0.281
NLL (baseline)
92.9=0.159
5.4910.105
1.89
92.3+00625
1.96=
CIFARIO
+MMCE
91.511.340
4.92.10.292
1.92.10.274
91.3110.429
2.52.10.586
(Resnet50)
+SB-ECE
91.6110.28
4.86.10.319
1.6300
91.5110.34
1..99
+ESD (ours)
92.1=0.141
3.08
1.60
92.1400314
1.61-0.287
NLL (baseline)
68.410.911
23.8:00403
5.7410.306
67.310.551
9.78100640
C0FAR100
+MMCE
66.5+00644
13.9+0.555
4.91+0.457
66.3+0.773
1.575
(Resnet34)
+SB-ECE
6..3+
14.7++1.370
4.94+00240
66.4+0.369
4.57
+ESD (ours)
67.410.356
13.6::0.500
0.85
67.1
1.289
NLL (baseline)
75.8:10.397
10.3100.686
2.311
75.91
2.86110459
ImageNet100
+MMCE
74.3+00248
3.83+00664
1.94+0.262
74.5+04444
2.29=00227
(Resnet18)
+SBECE
74.490.596
4.28+
2.06+0.260
74.7=:0472
2.13
+ESD (ours)
74.6:10.320
1.8010.262
1.7210.350
74.8:10.336
1...0
NLL (baseline)
90.2+00434
4.20+00420
90.1 1.04
0.96+
SNLI
+MMCE
89.410.491
1.11
10
09.58
100
(Bert-base)
+SB-ECE
89.120.668
1.82
0.99
89.2:00765
0.90=10.10
+ESD (ours)
09.3=00536
0.9
0.73
09,4
0.61=0.120
NLL (baselinc)
49.42:0323
35 55.9
4.16=2.445
48.420.492
5.10=0.657
ANLI
+MMCE
49.0=0.471
31.2++0.500
0.71
47.790.429
4.79=00693
(Roberta-base)
SB-ECE
48.5110.811
33.911.377
3.98
47.3100281
5.1010.119
+ESD (ours)
48.01:0.51
28.810.543
3.49
47.11:04229
1.42
Table 1: Average accuracy (%) and ECE (%) (with std. across 5 trials) for baseline, MMCE, SB-ECE. ESD after training and after post-processing with temperature scaling (TS) or vector scaling (VS).
bins. For the image classification tasks we use AdamW (Loshchilov Hutter 2019) optimizer with 10-3 learning rate and 10 weight decay for 250 epochs, except for ImageNetIOO. in which case we used 10- weight decay for 90 epochs. For the NLI tasks, we use AdamW optimizer with 10-5 learning rate and 10-2 weight decay for 15 epochs. For both tasks, we use a batch size of 512.
The internal hyperparameters within MMCE (Ø) and SB-ECE (M, T) were sequentially optimized 3 following the search for optimal A. Following the original experimental setting of SB-ECE by Karandikar et al.) (202T), we fix the hyperparameter M 15. Similar the model selection criterion utilized by Karandikar et al. (2021). we look at the accuracy and the ECE of all possible hyperparameter configurations in the grid. We choose the lowest ECE while giving up less than 1.5% accuracy relative to baseline accuracy on the validation set. All experiments were done using NVIDIA Quadro RTX 8000 and NVIDIA RTX A6000.
6 EXPERIMENTAL RESULT
In Table we report the accuracy and ECE for the models before and after post-processing (i.c., temperature scaling and vector scaling) of various datasets and models. Compared to the baseline or other trainable calibration objective loss (MMCE and SB-ECE), jointly training with ESD as the secondary loss consistently results in better-calibrated network for all the datasets and models with around 1% degradation of accuracy. Moreover, we observe that applying post-processing (i.e.. temperature scaling (TS) and vector scaling (VS)) after training with a calibration objective loss generally results in better-calibrated models over baseline post-processing, in which case ESD still outperforms other methods. Comparing the calibration outcomes in temperature scaling with vector scaling for models trained with ESD, vector scaling worked comparable if not better post-processing method, except for ANLI, with minimal impact on the accuracy for all datasets. Additionally, Ovadia et al. (209)) demonstrated that postprocessing methods, such temperature
Foor d. search [0.2, 0.4, 0.6, 0.8]. For T, we search [0.0001, 0.001, 0.01, 0.1].
7

---

Published as a conference paper at ICLR 2023
(E50)
ECE
ECE
0-
0.05
0.10 Softening parameter (1) SB-ECE
0:15
0.20
0.25
Kernel width (4)
MMCE
Figure 2: ECE performance curve of MMCE (left) and SB-ECE (right) with respect to their varying internal hyperparameters on MNIST, CIFARIO 10, SNLI datasets.
Single Computation
Total Computation Cost
Compitation Cost (GPU hours/epach)
LSB
Computation Cost
(GPL hours/spoch) 3 F
GS
Model (dataset)
Model (dataset)
Figure 3: Computational cost of single-run training (left) and total cost considering hyperparameter tuning (right). The x-axis in both cases are the order of increasing model complexity.
scaling, not effective in achieving good calibration under distribution shifts. Building this Karandikar K training enhance model's ability maintain calibration under distribution shifts. This finding has been replicated in Appendix
et al. (202T) have provided evidence that incorporating calibration during model
We show in Figure that across different models and datasets the calibration performance of MMCE and SB-ECE is sensitive to the internal hyperparameter. This shows the importance of hyperparameter tuning these methods. On the other hand, ESD does only outperforms MMCE and SB-ECE in terms of ECE but does not have an internal hyperparameter to tune for.
7 ABLATION STUDY
7.1 COMPUTATIONAL COST HYPERPARAMETER SEACH
We investigate the computational cost required to train model with ESD compared to MMCE and SB-ECE. As depicted in Figure 3 the computational cost required to train model for a single run remains nearly identical across different models and datasets. However, considering the need for tuning additional hyperparameters within MMCE and SB-ECE, the discrepancy in total computational cost between ESD and tuning-require calibration objective losses becomes more prominent as the model complexity and dataset size increases.
7.2 BATCH SIZE EXPERIMENTS
Table 2 shows that ESD is robust to varying batch sizes which could be due to the fact that it is an unbiased and consistent estimator. Regardless of the batch size, being unbiased guarantees that the average of the gradient vectors of ESD equals the gradient of the true estimate (Appendix D
8

---

Published as a conference paper at ICLR 2023
Dataset (Model)
ECE
Acc.
ECE
Loss Fn.
Acc.
ECE
after TS
after VS
after VS
NLL (b512)
92.91101559
5,49.10.105
189:10.105
92.3110.225
1.96.11.596
CIFARIO
+ESD (b512)
92.11101441
3.08:10.692
1660:0.089
92.110.3114
1.61:00287
(Resnet50)
+ESD (b256)
91.5++0137
3.15+0.587
1.6140.073
91.9++0573
1.60++0.022
+ESD (b128)
91.5++0.519
2.93+40.599
1.70+L0.600
91.7+-0.46
2.20+H0.584
+ESD (b64)
91.3+:0.753
3.91::0.847
188
91.5++0257
2.65 1.48
NLL (b512)
90.2.10.344
4,20.10.420
104410.112
90.11003977
0.96.10.099
SNLI (Bert-base)
+ESD (b512)
89.3100536
0,98110.165
0.73
89,410.583
0.61
+ESD (b256)
89.9+00314
1.11+002236
0.76+0.241
88.940.146
0.62+ 147
+ +ESD (b128)
89.8+00435
0.99+00244
0.68+0.190
89.6+0:490
0.58+0..18 187
+ESD (b64)
89.1110660
1.0611.2285
0.83+:0260
88.820.672
0.8110.153
Table 2: Average accuracy and ECE (with std. across 5 trials) for ESD after training with batch sizes 64, 128, 256, and 512.
7.3 ARE INDICATOR FUNCTIONS TRAINABLE?
Recent papers, Kumar et al. 2018) and Karandikar et al. (2021). have suggested that ECE is not suitable for training as a result of its high discontinuity due binning, which can be seen form of an indicator function. However, the results from our method suggest that our measure was still able to train well despite the existence of indicator functions. addition, previous measures also contain indicator functions in the form of argmax function that introduces discontinuities but remains to be trainable. As such, this brings to rise the question of whether calibration measures with indicator functions can be used for training. To investigate this, ran ECE as auxiliary calibration loss on different batch sizes and observed its performance on CIFAR 100 (Table 3).
We found that ECE, contrary previous belief. is trainable under large batch sizes and not trainable under small batch sizes while ours maintains good performance regardless of batch size. The poor performance of ECE under small batch size setting could potentially be attributed the high bias present in such cases. From this, it seems to suggest that indicator functions do not seem to inhibit the training for calibration.
Loss Fn.
Acc.
ECE
NLL (b512)
68.6440.334
22.4=0.10
FECE (h512)
68.2+00025
14.6=0.252
ECE (b256)
67.911.057
18.1=0.552
+ECE (b128)
68. 10.084
21.0=0.552
Table 3: Average accuracy and ECE (with std. across 5 trials) for CIFAR100 after training with ECE on batch sizes 128, 256, and 512.
8 CONCLUSIONS
Motivated by the need for a tuning-free free trainable calibration objective. we proposed Expected Squared Difference (ESD), which does not contain any internal hyperparameters. With extensive comparison with existing methods for calibration during training across various architectures (CNNs & Trans- formers) and datasets (in vision & NLP domains), we demonstrate that training with ESD provides the best-calibrated models compared to other methods with minor degradation in accuracy over baseline. Furthermore, the calibration of these models is further improved after post-processing. In addition, we demonstrate that ESD can be utilized in small batch settings while maintaining performance. More importantly. in contrast to previously proposed trainable calibration objectives, ESD does not contain any internal hyperparameters, which significantly reduces the total computational cost for training. This reduction in cost is more prominent as the complexity the model and dataset increases, making ESD more viable calibration objective option.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2022-0-00184, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy Ethics), and Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea gov- (MST) (No. 2021-0-0138) Development of Causal Al through Video Understanding and Reinforcement Learning, and Its Applications to Real Environments).
REFERENCES
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613-1622. PMLR, 2015.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End end learning for self-driving cars. CoRR, abs/1604.07316, 2016. URL
org. abs 1604.0731111 07316
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguisties, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large- hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. (00/CCVP.22009552088888
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6): 141-142, 2012.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-41111 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/ URL https wwwelannhhoooy.oorg//l...
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, 06-11 Aug 2017. URL https: pttep::/pprccedinng. .mlr mlr. www.l...res///0 quol7a.html
Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. Calibration of neural networks using splines. In International Confer- ence on Learning Representations, 2021. URL httpt:::///peeeee https:://opereeview net P @Ce8DEWNN2W)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
doi: 10. 10.10CC222.2
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In International Conference on Learning Representations, 2019. URL httt::htt::/:ppeee httpa//nweeeeeeeeeeooooooooooooo
10

---

Published as a conference paper at ICLR 2023
Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael Curtis Mozer, and Rebecca Roclofs. Soft calibration objectives for neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neu- ral Information Processing Systems, 2021. URL htttps://openreviiee wwwrewiiww..... net. forum i LVD 3hOS33
Junyeong Kim and Chang D. Yoo. Deep partial person re-identification via attention model. In 2017 JEEE International Conference on Image Processing (ICIP), pp. 2425-3429 2017. doi: 10.1109/111.2017.89999 181.899918...
ByungSoo Ko, Ho-Jin Choi, Chansol Hong, Jong-Hwan Kim, Oh Chul Kwon, and Chang D. Yoo.
Neural network- based autonomous navigation for a homecare mobile robot. In 2017 IEEE International Conference on Big Data and Smart Computing (BigComp), pp. 403 2017. doi: 10.1 10.1009BICCC0M2211.7788177
Simon Kocbek, Primoz Kocbek, Leona Cilar, and Gregor Stiglic. Local interpretability of calibrated prediction models: A case of type 2 diabetes mellitus screening test, 2020. URL https://
arxiv.org/ http://////../0......000 13815
Ranganath Krishnan and Omesh Tickoo. Improving model calibration with accuracy versus un- certainty optimization. In Proceedings of the 34th International Conference on Neural Informa- tion Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). a. URL/ :/ww:. cs toronto. www.nttt..uuu/kiiia http//kriz/// wwt./kriiiiiiaataaa html
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. C00 (canadian institute for advanced research). b. URLhttp::
cs. twronto......o........ edu/ -ttp//kkrzz/ee wii.rriiciife far
far.hhmmm html
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2805-2814. PMLR, 10-15 Jul 2018. URL wtRLhttts///pprceddiii
mlr. Seress/v8b/kume
8a. html
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (6(1)))277-2222 1998. doi: 10. 10.1109/5.72677 776691..
Jing Lei, Max G'Sel, Alessandro Rinaldo, Ryan Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523): 1114, 2018
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL pittpss:/opeeneeeee wwt.:/trtoormmmmm forum? id=
BkgbRiCqy) Cq
Bill MacCartney and Christopher D. Manning. Modeling semantic containment and exclusion in natural language inference. In Proceedings of 22nd International Conference on Computational Linguistics (Coling 2008), pp. 521-522112822 Manchester, UK, August 2008 Coling 2008 Organizing Committee. [t..htt.ss:/ / http:///o000...0/0//-1100555
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip H. S. Torr, and Puneet K. Dokania. Calibrating deep neural networks using focal loss. In Proceedings of the 34th Interna- tional Conference on Neural Information Processing Systems, NIPS 20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.

---

Published as a conference paper at ICLR 2023
Mahdi Pakdaman Nacini, Gregory F Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. AAAI, pp. 2901-2907, 2015.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial
NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885-4901, Online, July 2020.
Association for Computational Linguistics. doi: 10. 10.186633v1/20000 acl- mallmaii..44... URL https //aclanthology. org/ acl www.aain..mmaam...mm.. main.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht, Obtaining well calibrated probabil- ities using bayesian binning. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1), Feb. 2015. URL https::/ojss..... aaai. org/ index
/ 9602
Trung Xuan Pham, Jin Woong Choi, Rusty John Lloyd Mina, Thanh Xuan Nguyen, Sultan Rizky Madjid, and Chang D. Yoo. Lad: A hybrid deep learning system for benign paroxysmal positional vertigo disorders diagnostic. IEEE Access, 10: 13995-11440077 2022. doi: 10. 1091AA//SCSS 3215625.
John C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In ADVANCES IN LARGE MARGIN CLASSIFIERS, pp. 61-74, MIT Press, 1999.
A. W. van der Vaart. Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 1998. doi: 10.1017/CB097805) 1802256.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in random world.
Springer Science & Business Media, 2005.
Sunjae Yoon, Dahyun Kim, Ji Woo Hong, Junyeong Kim, and Chang D. Yoo. Dual-scale doppler attention for human identification. Sensors, 22(17), 2022. ISSN 1424-8220. doi: 10.3390/
com/ 1424 s22176363. URL https: /www 3
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Proceedings of the Fighteenth International Conference on Machine Learning, ICML. '01, pp. 609-616, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558607781.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass prob- ability estimates. In Proceedings of the Fighth ACM SIGKDD) International Conference on Knowledge Discovery and Data Mining, KDD '02, pp. 694-699 699,, New York, NY, USA, 2002. Association for Computing Machinery. ISBN 158113567X, doi: 10. 7577177770447777551111 URL
https///ooi..rr//001111///7 775151
12

---

Published as a conference paper at ICLR 2023
A
VISUAL INTUITION OF EXPECTED SQUARED DIFFERENCE (ESD)
From Eq. da(a) /' P(Y 1, 2x.) 22 2x d2z
[B a))( =) Zx)]l-
This can be viewed as the difference between two quantities:
Cumulative Accuracy
P(Yx = 1, = dxx
Ez [1Z S a, Ytx 1)]
T=Eu S a, Va,s = 1).
1=1
Cumulative Confidence
2x (75 =2x 2x
Ez Vi [7 [0]
zk, S a.).
Due to the close relationship between (a) and the calibration of a neural network, the average squared difference (Ez; [d& (Z) between the cumulative accuracy and confidence closely corre- sponds with the calibration of a network (Figure 4 That is, the average squared difference between the cumulative accuracy and confidence is larger for an uncalibrated network compared to calibrated network. Jointly training with our proposed Expected Squared Difference (ESD) an auxiliary loss tries to minimize this squared difference between the two curves during training on average, thus achieving better calibrated model.
1.0
Cumulatine Confidencca Cumulative Asourayy
Curuulltive Corfile icl Cumubative Asorray
08
ECE: 23,3
ECE: 6.12
02
0.6
Quantile (a)
Quantile (a)
Uncalibrated
Calibrated
Figure 4: Visual intuition plot showing the cumulative confidence and cumulative accuracy with varying quantile scores of prediction confidence (a) for an uncalibrated (left) and calibrated (right) network. The uncalibrated network was obtained by training Resnet34 on CIFAR100 with NLL, and the calibrated network was acquired by temperature scaling on the aforementioned trained network.
B PROOF THAT ESD IS AN UNBIASED AND CONSISTENT ESTIMATOR
In Theorem 2, prove that ESD is unbiased estimator, that is taking the expectation of ESD is equal to the true Expected Squared Difference.
Theorem 2 ESD is unbiased estimator, i.e. Ez, Y, [ESD] Ex [d)()) where dy (Z,) [E v. (7 (T(x& 1) Z+]]]-
13

---

Published as a conference paper at ICLR 2023
8
Proof. Let
Yx
k... and G; =g,²
N-I
We have that
1
R Ez. Yu
[ESD]
R. Zz Vi [Gi]
(by linearity of expectation)
N
N
1
Ez Mk. [4
(by lemma 2.1)
N
i=1
1
E2g [d] (21)
(since Ez.,, Yu. [uf] E2gldh((2 (2x))
N
1222 (Z)))
Lemma 2.1 -e,, = 22 where = Ez.r S ZI(i( = 1) - -z+]]
Proof. Since samples are i..d., for fixed i, it holds that:
E ZZ, Ve. [i]] = 14i
52
o?
Ezn N
g1
(where Var(gi] = o?)
(12)
N -
Var
E Ezu. Yv. 19. /r²
Shifting variables around, we get
4 Nn. 19.21
Varlgi]
$2 91
o?
EZu..t
J/²
(since Varíg.
N N-1
N
Gi].
In Theorem 3, we prove that ESD is consistent estimator, which means that ESD converges in probability the true Expected Squared Difference.
Theorem 3 ESD is consistent estimator; i.e. ESD F, Ez, [d] (2))-
14

---

Published as a conference paper at ICLR 2023
Proof. Since ESD is an unbiased estimator, it is sufficient to prove -- Var 0. For simplicity, use where Σny Eij== Σn Ej= and Ez,, [ Ezg [d (2() Vi,
1
Var|EDD]
Var
[
Cov(G,,
2 i=1
N
Since Ez. Yo [G:] = Ezg (x(() Vi,
1
Var
EZ,, Yo [G,G,] E3 (z(())
N
N
f,j=1
N
N
1
2
(Ez, Yo 1. "
E2, (2((1))
(by lemma 3.1 and 3.2)
N2
N
+
N2
(N 2)
1,==1
ifj
2
4N(N
-
N(N 1)
+
(E Zu Yn 10 2022 2
Ez (Z))) (by lemma 3.3)
N
N²(N
2)
N2
Thus, by using lemma 3.4 and the non-negativeness of variance,
0 < lim Var EESD < 12000
lim Ez, Yv 2 12-200
1 (2)(((()) = 0.
Consequently, lim Var[ESD] = O.
Lemma 3.1 Var[G.] $2 00.
Proof.
Var[G;] = Ez Yo [G
E2
GG
Zx, 2x,yy Yo
< EZu. Vo (G
52
S2
= Ez Vo
2
9:
+
91
N -
N -
27
82
< Ez Vv
D1 +
gt
N
1
(
82 N-11
27
By lemma 3.2, Ez,,r, (9?] < and Ezn. Vn
2
1. Thus,
N-
27
52. N-I
Ez, Yv
2 2.
52
Lemma 3.2
N-1 < N=z and Ø 1.
15

---

Published as a conference paper at ICLR 2023
N
N
Proof. By the triangle inequality, |
N-1
Σ
gim
S N-1
pim
S 1, since Vi [giml S 1.
m= m
m=
Furthermore, we have
S2
N
N
Fim +
Fim )2
N
(N
1)2
(N
1) - 2)
T2
m= migh
1
1
2 +
gim 1)²
Jim
(N 1)
(N 1)2
2)
N
N
1+
1)?
(N
1)2
(N- - 1)N 2)
v=== 172
The last inequality is by Vmn giml S and the direct calculation implies that
S2 N
2
N 2 N
Lemma 3.3 Ez,,Y, 1g,",, Ezz, Yv 2 Vi,J where + j.
Proof.
Σ Jim
Σ Jin
Σ 9ig9jn
Σ
9im
m,n=1
(m,n)> (15)
(mn]>(i..] mfn
(t
my (1,)
Σ m,n=1 mzlii)
Jim + Dij9i
(since these terms are disjoint terms)
N
N Σ
N
Σ
Σ
mime D2n
g1202a
+
mlml 021
m. an =1 n' (1.2)
m .n m' (1.2)
(m' nn of (1.)), m 'n'
N Σ
Dim F2m + 912021
m' m
m (112)
N Σ
9im D2n'
m =1
#1, #2
Proof of claim above:
As the summation be divided into five distinct cases,
16

---

Published as a conference paper at ICLR 2023
Let i and be arbitrary such that i J.
Case (m, * { and m + n,
Jimlij I[Zumm S ZI 1) mm ( S ZR.)) = 1) - Zk.nl
(Zz,i Zy.m, Gs Lu.ns Ys Yk,n)
d h1 (Zk,1 km 23 Zz,n's Kk,m Mk,n)).
where (m',n') * {1,2} and m' * n'
Case 2: m and + {i,j}.
9is9in = 1(Zx, S (1x,))((YRs) = - Znnle(ZRin S (x.))L((Knn> = =1) Zn.m]
(Z Zz.j Zki Yk.t Mk.n)
- (Zx,1 2, Zk,n' Ye,2, Yk,e)).
where m' =nand n' + {1,2}
Case 3: n = and m + {i,j},
JimOji mm S 7k. ( =) 7kkm (ZZk S ZT = Zx,ii
= Z Zu,mi Zg.5, Vk.m: Yk,s)
1 12 ZZ,, Zu.m', Zu.2, Ya Vr.1)- Case 4: m # {i.j},
where n' = and m' + {1,2]
Jim9jm = 1(Zxm S ZI, 1) Zx.m) (Z2, S Zxs)(T(Yk,n 1) Zm
h44 Lu,iz Zu.ms Zu Yu,m)
A (Z Zm.'', Zu.2, Yk.m)). Case 5: m j and = i,
where m' =n' + {1,2}
Pッgli 1(Zx. S ZZx,)[((Yss) =) Ze.) S Zn.)][((Yn 1) Zu.i
= No Lo Zr.jo Ykits Yx.s)
1 (z Zuz Yk.1. Yx.2).
where m' = 2 and n' = 1
This follows from the fact that Zk,s 's and Ya,''s 's are i.i.d. random variables. Therefore,
91ml 91me92n )²
Jim Jim
m mnbin
.n'=1
mn' mln1,nn'z
3 d 91 '22
E2, Yv 1g. gg
EZ Yv 1gi g2
Lemma 3.4 lim N- Ez, Yi 10.
20 = =E3 (2)))
Proof. For arbitrary g, E, μe by the strong law of large number. Thus, Φ1 P, μu and Øz P, u22- Therefore, since marginal convergence in probability implies joint convergence in probability (Vaart 1998).
Fg (1i, P2). (gi,92)
By continuous mapping theorem, since [(x, y) _²n²² 2 is continuous function,
2 2 F.
9/2
Additionally, since 1 2g2 VN, is uniformly bounded and thus uniformly integrable. Com- bining this with the fact that it converges in probability,
lim Ez Ez.. Y, [ 2 Ez, v. 11@pz) N
17

---

Published as a conference paper at ICLR 2023
Thus,
lim Ez (r = Ez., [[]pE]] Y2 = 133 [dH() N-000
C EXPECTED SQUARED DIFFERENCE (ESD) PSEUDOCODE
In this section, we provide pseudocode for calculating the Expected Squared Difference (ESD) for a given batch output.
Algorithm 1: Pytorch-like Pseudocode: Expected Squared Difference (ESD)
n n:number of samples in a siini-bacch.
confidence: tensor of elements containing max softmax outputs of each sample from a neural network.
correct: tensor of elements containing 1/0 corresponding to the correctness of each sample.
def ESD loss (n, confidence. correct): compute the difference between confidence and correctness.
diff correct. proecttflootii () confidence
Prepare the split between inner and outer expectation estimation.
split = torch. ones (n - torch.eye (n)
compute the inner expectation estimation. confidence_mat= confidence expand (n,
ineq = torch. htt.://eellootieeeeeee...oo..eee .floo
ffloat ()
diff mat diff ()) x.mat = torch.mul (diff inei) mean_row torch. (x_mat, dim 1 x.mat.squared = torch.mul (x_mat, x_mat)
var 1/ (n- torch. sum (x.mat.squareet dim=1) (n-1) (n-2)
corch.mul (mean_row, mean_row)
compute the outer expectation estimation. d sq vector = torch.mul (mean row, mean row) var/ (n-1)
ESD torch. sum (d.k.sq.vecttor) /n return ESD
D UNBIASED ESTIMATORS AND ITS GRADIENT
Since ESD is an unbiased estimator,
E nu. Yi [ESD] Ezg (Z))
VøEz [Z,)] = (EE2
[oEx,r(ESD]
(by law of unconscious statistician)
(since (X, Y) are independent of 0)
Thus, the gradient of the unbiased estimator is on average the gradient of the desired metric.
E ADDITIONAL INFORMATION ON THE CHOICE OF LAMBDA RANGE
To validate the choice of grid range in our experimental settings, we plot the variations in accuracy with respect to increasing for all methods (Figure 5). We that choice of À contains points
18

---

Published as a conference paper at ICLR 2023
CIFAR 10 (Resnet50)
ImageNet 100 (Resnet18)
CIFAR 100 (Resnet34)
Threshold Accuracy
Acurrat
30
30-
34
Auturacy
074
0.5
072
Arsurnty
080
0M
042
MMCE
Figure 5: Accuracy plot with respect to varying values across different datasets and models trained with MMCE, SB-ECE, and ESD. The threshold accuracy represents the value 1.5% below the baseline accuracy, which was used as the model selection criterion as stated in section 5.2
within the acceptable range of accuracy (i.e., within 1.5% degradation in accuracy compared to baseline). However, after a particular value, which is different on per method per dataset basis, the accuracy decreases consistently with increasing A. As such, the grid chosen is suitable for the experiments conducted.
TRAINABLE CALIBRATION MEASURES UNDER DISTRIBUTION SHIFT
NEL-WISE
AUSCET ALL HERD
LEEE
Accuracy
ECE
VEROIT
NLL NLLAMUGE SLLVESO
16
LIVRACE ECD
Shift Intensity
Shift Intemsity
Shift Intensity
Shift Intensity
(a) CIEAR 10-C
(b) CIFAR 100-C
Figure 6: Accuracy and ECE plot for varying intensities of distribution shifts in (a) CIFAR 10-C and (b) CIFAR 100-C across models trained using NLL as well as those jointly trained with an auxiliary calibration objective (i.e., NLL+MMCE, NLL+SB-ECE, NLL+ESD) and their performance after post-processing with temperature scaling (i.e., NLL+T, NLL+MMCE+T, NLL+SB-ECE+T, NLL+ESD+T).
In Figure 6 evaluate the performance of calibration methods under distribution shift benchmark datasets, CIFAR 10-C and CIFAR 100-C, introduced in [enddrykk & Dietterich (201)). For models jointly trained with an auxiliary calibration objective INLL+MMCE, NLL+SB ECE, NLL+ESD), we observe that they are more robust to distribution shifts when compared to those solely trained with NLL. addition, prior work (Ovadia et al.) 2019). has shown that the calibration performance of neural network after temperature scaling may be significantly reduced under distribution shifts. Our results on CIFAR 10-C and CIFAR 100-C imply a similar trend. Stacking calibration during training methods with temperature scaling (NL++MMCE-T, NLL+SB-ECE+T, NLL+ESD+T), we observe that they perform comparable to temperature scaled models trained with NLL (NLL+T) with the exception of ESD, where it performs marginally better. As such, training with ESD could potentially improve model's 's robustness after temperature scaling regarding distribution shifts and mitigate this issue.
19


Reviews:
[{"review_id": "b1motHOB5tB", "reviewer": "Reviewer_md7e", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a trainable objective less ESD as the extra calibration loss jointly optimized with the NLL loss during training. It is claimed as a binning-free objective lease without need to tune any additional parameters. The experiments were extensively conducted across three architectures (MLPs, CNNs, and Transformers) and both vision and NLP datasets. The experimental results seem to strongly validate the efficacy of the proposed method.", "summary_of_the_review": "Based on the above statements, I would like to weakly accept the paper at the initial stage. After carefully reading the other reviewers' comments and the authors' response, I would like to keep the initial rating.\n", "strength_and_weaknesses": "#### Strengths\n- Overall the paper is well written and easy to follow.\n- The idea of ESD is very interesting and should be able to bring a positive impact to the AI fields not limited to both vision and NLP domain.\n- Strong theoretic explanation with solid mathematical functions. \n\n#### Weaknesses\n- The accuracy has no obvious advantages when compared to other competing baselines, according to Table 1.\n- As the paper proposes a general calibration loss, I would like to suggest the authors to conduct the experiments on the more general large-scale vision datasets (e.g., MS-COCO and ImageNet).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QuMmJbNreY", "reviewer": "Reviewer_smEk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper present Expected Squared Difference (ESD), a tuning-free (i.e., hyperparameter-free) trainable calibration objective loss, which views the calibration error fromthe perspective of the squared difference between two expectations.", "summary_of_the_review": "This paper proposes a hyperparameter-free calibration method, ESD and produces accurate predictions and calibration results.", "strength_and_weaknesses": "Strengths:\n1. Build a well-established  trainable calibration objective loss ESD, which is binning-free and does not need hyperparameters. \n2. Extensive experiments to prove that the esd produces more accurate ECE without tuning the hyperparameters.\nWeaknesses:\n1. This calibration method itself does not need any hyper-parameter but still incorporates some other hyperparameter, such as the regularizer hyperparameter λ, monte carlo sample number N, etc. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8YONtaf8pGm", "reviewer": "Reviewer_XMyj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a hyperparameter free training based calibration method using proposed calibration metric, namely expected squared difference (ESD). The proposed calibration metric is inspired from the recently introduced KS metric that compares cumulative distribution.", "summary_of_the_review": "I am in favor of acceptance of the paper as its a novel way of calibration and proposed method looks clean. I suggest some few additions to the paper which will make it stronger. If my concerns are addressed, I will increase my rating.", "strength_and_weaknesses": "Strengths:\n- The paper is easy to follow and well-written.\n- There has been a lot of research focused on post-hoc calibration methods in the literature but this paper looks upon the solving the mis-calibration at the training time without significant additional computational costs. This problem is particularly challenging as the overfitting is a significant challenge to solve this task. \n- The paper achieves impressive results on multiple benchmarks of calibration while proposing a theoretically concrete approach to calibration.\n\nWeaknesses:\n- Since this is a training based calibration method, I would suggest the authors add calibration results on large scale datasets such as ImageNet. \n- An important addition to the paper, should be along the lines of calibration performance under distribution shifts [a] as this has been known to be difficult for post-hoc calibration methods but I would assume the proposed method should be able to perform better under such a scenario.\n\nReferences:\n- Ovadia et al. Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. NeurIPS 2019.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0ojrtdrjyN", "reviewer": "Reviewer_ywx4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new method to improve calibration of multi-class classifiers. It is similar to the existing methods of MMCE and SB-ECE in the sense that it adds a trainable auxiliary loss function (Expected Squared Difference, ESD) to the cross-entropy loss (NLL) during training of the classifier. Similarly to SB-ECE, the paper uses interleaved training where the new auxiliary loss and NLL are trained on separate parts of the training set. ESD is parameter-free and its estimator is unbiased. The experiments on 5 datasets demonstrate that compared to MMCE, SB-ECE and the baseline NLL it results in better-calibrated predictions (lower ECE) both before and after post-hoc calibration with temperature scaling or vector scaling. About 1% of accuracy is sacrificed compared to the baseline NLL.", "summary_of_the_review": "The paper proposes a parameterless method and improves the state-of-the-art. The proposed method ESD is simpler and better than the existing methods MMCE and SB-ECE. There are some weaknesses highlighted above but these can be fixed for the camera-ready v\nersion.", "strength_and_weaknesses": "Strengths:\n* The paper is well-written and clear.\n* The proposed method ESD has been justified theoretically and the ESD estimator has been proved to be unbiased and consistent.\n* The experiments on 5 datasets demonstrate that ESD results in better calibrated probabilities than earlier methods, while not sacrificing more accuracy.\n\n\nWeaknesses:\n* The intuition behind the proposed method could have been explained more, perhaps on a toy example.\n\n* More information about the experimental results could have been reported, for example as an appendix. It would be good to know whether the range of values for the hyperparameter lambda is sufficient and which values were eventually chosen by the method on different datasets. Also, it would have been good to report NLL as an evaluation measure, in addition to accuracy and ECE. E.g., Table 1 could have had extra rows for NLL, NLL after TS, and NLL after VS, or alternatively, this information could have been provided in the appendix. The paper does not mention why accuracy after vector scaling is so low for CIFAR100 and ANLI. Can there really be so much overfitting even though the number of classes is quite high?\n\n* The conclusion fails to mention that some accuracy has been sacrificed to achieve better calibration. On the positive side, this fact has been mentioned in Section 6 about experimental results.\n\n* Notation $k$ is overloaded in Section 3 - first it is the total number of classes, then just any class in Eq.(1), and then the class with maximum output probability in Eq.(2). This can create confusion, in particular because Eq.(1) is about being classwise-calibrated whereas later the paper only studies confidence calibration.\n\nMinor weaknesses:\n* Table 3 and Section 7.3 fails to mention which dataset it is about. I assume CIFAR100?\n\n* The last sentence of page 7 (first paragraph of Section 6) states the decrease of accuracy by more than 2% - is it not about 1% in this case (47.08 vs 48.0) because the comparison is all between ESD and ESD+VS?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster