Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error
Authors: Ondrej Bohdal
Yongxin Yang
Timothy Hospedales
Venue: Accepted by TMLR ()

Abstract: Calibration of neural networks is a topical problem that is becoming more and more important as neural networks increasingly underpin real-world applications. The problem is especially noticeable when using modern neural networks, for which there is a significant difference between the confidence of the model and the probability of correct prediction. Various strategies have been proposed to improve calibration, yet accurate calibration remains challenging. We propose a novel framework with two contributions: introducing a new differentiable surrogate for expected calibration error (DECE) that allows calibration quality to be directly optimised, and a meta-learning framework that uses DECE to optimise for validation set calibration with respect to model hyper-parameters. The results show that we achieve competitive performance with existing calibration approaches. Our framework opens up a new avenue and toolset for tackling calibration, which we believe will inspire further work on this important challenge.

Published in Transactions on Machine Learning Research (08/2023)
Meta-Calibration. Learning of Model Calibration Using Differentiable Expected Calibration Error
Ondrej Bohdal The University of Edinburgh
ondrej.bohdaaleedd.cco uk
Yongxin Yang
yongsin.gangg@qmuul. uk
Queen Mary University of London
Timothy Hospedales
t.hospedales@ed. wed.dallleeeed.cooo uk
The University of Edinburgh
Samsung Al Center Combridge
Reviewed OpenReview: https: //teer////pee/vveeu.u net/ forum? id= R2hUure381
Abstract
Calibration of neural networks is topical problem that is becoming more and more important neural networks increasingly underpin real-world applications. The problem is especially noticcable when using modern neural networks, for which there is a significant difference between the confidence of the model and the probability of correct prediction. Various strategies have been proposed to improve calibration, yet accurate calibration remains challenging. We propose novel framework with two contributions: introducing new differentiable surrogate for expected calibration error (DECE) that allows calibration quality to be directly optimised, and meta-learning framework that uses DECE to optimise for validation set calibration with respect to model hyper-parameters. The results show that achieve competitive performance with existing calibration approaches. Our framework opens up a new avenue and toolset for tackling calibration, which we believe will inspire further work on this important challenge.
1 Introduction
When deploying neural networks to real-world applications, il crucial that models' own confidence estimales accurately match their probability of making a correct prediction. model over-confident about its predictions, we cannot rely it; while well-calibrated models can abstain or ask for human feedback in the case of uncertain predictions. Models with accurate confidence estimates about their own predictions be described well-calibrated. This is particularly important in applications involving safety or human impact such as autonomous vehicles (Bojarski et al. 2016 Wiseman 2022) and medical diagnosis (Jiang et al. 2012: Caruana et al.] 2015 El-Sappagh et al.] 2023), and tasks that directly rely on calibration such as outlier detection (Hendrycks Gimpel 2017 Liang et al. 2018 Wang et al. 2023). However, modern neural networks are known be badly calibrated (Guo al. 2017 Gawlikowski al. 2021).
This challenge of calibrating neural networks has motivated growing area of research. Perhaps the simplest approach is to post-process predictions with techniques such temperature scaling (Guo et al. 2017). However, this has limited efficacy (Wang et al.] 2021) and fails in the common situation of distribution shift between training and lesting data (Ovadia el al. 2019 Tomani et al.) 2021). also reduces network's confidence in correct predictions. Another family of approaches modifies the model training regime to improve calibration. Muller al. (2019) show that label smoothing regularization improves calibration by incrcasing overall predictive entropy. But it is unclear how to set the label smoothing parameter so as optimise calibration. Mukhoti et al. (2020) show that Focal loss leads to better calibrated models than standard

---

Published in Transactions on Machine Learning Research (08/2023)
14
Training Validation
Training Validation
12
50-
10-
Error (%)
ECE
30
10
50
100
150
200
250
300
350
0 50
100
150
200
250
300
350
Epoch
Epoch
Figure Illustationn of the calibration generalisation gap. ECE and classification error during training of ResNet18 on CIFAR-10, using cross-entropy loss and showing mean and std. across three random seeds. Training ECE and error fall to 0. However, calibration overfitting occurs and validation ECE increases. This motivates the need for meta-learning to lune hyper-parammeers to optimise validation calibration.
cross-entropy, and Kumar et al. (2018) introduce a proxy for calibration error to be minimised along with standard cross-cntropy on the training set. However, this does not ensure calibration on the test set. Ovadia et al. (2019) Mosser & Naeini (0022) show that. Bayesian neural network approaches
comparatively well-calibrated, however these are difficult and expensive to scale to large modern architectures.
The above approaches explore the impact of various architectures and design parameters on calibration. In this paper we step back and consider how to optimise for calibration. Direct optimisation for calibration would require a differentiable metric for calibration. However, calibration is typically measured using expected calibration error (ECE), which not differentiable due to its internal binning/counting counting operation. Therefore our first contribution is a high-quality differentiable approximation to ECE, which we denote DECE. A second consideration is how to optimise for calibration given that calibration itself is quantity with generalisation gap between training and validation (Carrell et al. 2022). We illustrate this challenge in Figure 1 which shows how validation calibration worsens as training calibration improves during training, To this end, our second contribution is to introduce framework for meta-learning model calibration: We fit a model on the training set using given set of hyper-parametors, evaluate it on a disjoint validation sot, and optimise for the hyper-parameters that lead the best validation calibration as measured by DECE. Our framework for differentiable optimisation of validation calibration is generic and can potentially be used with any continuous model typer-parameters. Our third contribution is a specific choice of hyper-paramette which, when meta-learned with suitable calibration objective, are effective for tuning the base model's calibration. Specifically, we propose non-uniform label-smoothing, which can be tuned by meta-learning to penalise differently each unique combination of true and predicted label.
To summarize, we present a novel framework and toolset for improving model calibration by differentiable optimisation of model lyper-parametent with respect to validation calibration. We analyse our differentiable calibration metric in detail, and show that closely matches the original non-differentiable metric. When instantiated with label smoothing typer-parametters our empirical results show that our framework produces high-accuracy and well-calibrated models that are competitive with existing methods across a range of benchmarks and architectures.
2 Related work
Calibration Since finding modern neural networks are typically miscalibrated (Guo et al. 2017 model calibration has become a popular area for research (Gawlikowski el al. (021] with many applications, incl including medical image segmentation (Judge et al. 2022) and object detection (Munir et al. 2023) Guo et al. 2017 study a varioty of potential solutions and find simple post-training rescaling of the logits temperature scaling
works relatively well. Kumar al. (2018) propose a ternel-based measure of calibration called MMCE
that they use regularization during training of neural networks. Mukhoti al. (2020) show Focal loss
2

---

Published in Transactions on Machine Learning Research (08/2023)
relatively simple weighted alternative to cross-entropy can be used to train well-calibrated neural networks. The classic Brier score (Brier 1950), which is the squared error ween the soft vector with probabilities and the ground-truth one-hot encoding, has also been shown to work well. Similarly, label smoothing (Müller cct al. 2019) has been shown to improve model calibration. Noise during training more broadly can also be beneficial for improving calibration (Ferianc et al., 2023). These aforementioned methods do not optimise for calibration metric (e.g., ECE) directly, because the calibration metrics are usually non-differentiable In this work, we propose a new high-quality differentiable approximation to ECE, and utilize it with meta-learning.
Karandikar et al. 2021 have proposed banted ECE (SB-ECE) as auxiliary loss to be used during training to encourage better calibration. The approach makes the binning operation used in ECE differentiable, leading to an additional objective that is more compatible with gradient-based methods. SB-ECE does not make the accuracy component of ECE differentiable, but we make all components of ECE differentiable. We also try to obtain a highly accurate approximation of the binning operation, while SB-ECE binning estimate for the left-most and right-most bin can be inaccurate as a result of using bin's center value. We provide additional comparison with SB-ECE in the appendix, showing DECE provides a closer approximation to ECE than SB-E SB-ECE, and that empirically our meta-learning approach with DECE leads to better calibration.
We use the newly introduced DECE metric as part of the meta-objective for gradient-bar based mota-lcarning Gradient-basod mota-learning has become popular since the seminal work MAML Finn let al.
Meta-learning
2017) has successfully used to solve the challenging problem of few-shot learning (Wang et al. 2020
Song et al. 2023 Bohdal et al. 2023 Nevertheless, gradient- based meta-learning is not limited to few-shot lcarning problems. but it can also be used to solve various other challenges, including training with noisy
labels (Shu et al. 2019 Algan & Ulusoy 2022). dataset distillation (Wang et al. 2018 Bohdal et al. 2020
Yu et al. 2023 domain generalization and adaptation (Li et al. 2019; Balajji et al. 2018 Bohdal et al.
2022
molecular property prediction (Chen et al. 2023) and many others.
Gradient-based meta-learning is typically formulated as a bilevel optimisation problem where the main model is trained in the inner loop and the meta-knowledge the case of few-shot learning it is possible to fully train the model within the inner loop also known offline mela-learuing. In more realistic and larger scale settings such as ours, il is only feasible to do one or a few updates in the inner loop. This approach is known as online meta-learning (Hospedales et al.) 2021) and mcans that we jointly train the main model as well as the mnta-knnoledge. Online meta-learning is most commonly done using the so-called Ti - T2 method (Luketina el al. 2016) that updales the mela-knowledge by backpropagating through one step of the main model update. This is the approach that we adopt, however more advanced or efficient approaches are also available (Lorraine et al. 2020 Bohdal al. 2021
hyper-parameters are trained in the outer loop. In
Label smoothing We use label smoothing as the mota-knowledge that we use to demonstrate the benefits of using our DECE metric. Label smoothing has been proposed by [Szeedy et al. (2016) as a technique to alleviate overfitting and improve the generalization of neural networks. Tt consists of replacing the one-hot labels by their softer alternative that gives non-zero target probabilities to the incorrect classes. Label smoothing has been studied in more detail, for example Müller et al. (2019) have observed that label smoothing can improve calibration, but at the same time it can hurt knowledge distillation if used for training the teacher. Mukhoti et al. (2020) have compared Focal loss with label smoothing among other approaches, showing that simple label smoothing strategy has limited scope for state-of-the-art calibration.
However, we demonstrate that using meta-learning and our DECE objective, more expressive form of label smoothing can achieve stato-of-tho-art calibration results. Note that meta-lcarning has already been used for label smoothing (Li et al., 2020), but using it as mela-knowledge to directly optimise calibration is new
3 Methods
3.1
Preliminaries
We first discuss expected calibration error (ECE) Nacini et al. 2015), before we derive a differentiable approximation to it. ECE measures the expected difference (in absolute value) bet the accuracies and
3

---

Published in Transactions on Machine Learning Research (08/2023)
the confidences of the model examples that belong to different confidence intervals. ECE is defined
M
Bm
ECE
aace (Bm) conf (Bm)),
n
m=1
where accuracy and confidence for bin Bm are
acc (Bm)
Σ
1 1 Us)
Bm
im
conf (Bm)
Pi.
Bm
B
There are M interval bins each of size 1/M and n samples. Confidence P. is the probability of the top prediction as given by the model for example We group the confidences into their corresponding bins, with bin Bm covering interval (" M. The predicted class of example is ii while yi is the true class of example i and 1 is indicator function.
ECE metric is not differentiable because assigning examples into bins is not differentiable and also accuracy is not differentiable due to the indicator function. We propose approximations to both binning and accuracy and derive a new metric called differentiable ECE (DECE).
3.2 Differentiable ECE
ECE is composed of accuracy, confidence and binning, but only confidence is differentiable.
Differentiable accuracy In order to obtain differentiable approximation to accuracy, we consider approaches that allow us to find a differentiable way to calculate the rank of given class. Two approaches stand out: differentiable ranking (Blondel et al. (020)) and an all-pairs approach (Qin al. 2010). While both allow us approximate the rank in a differentiable way, differentiable ranking implemented on CPU only, which would introduce potential bottleneck for modern applications. All-pairs approach has asymptotic complexity of O(n²) for classes, while differentiable ranking is O(n n)). However, if the number of classes is not in thousands or millions, differentiable ranking would be slower because of not using GPUs. We use the all-pairs approach lo estimate the rank of a given class.
All-pairs QQin et al. 2010 calculates a rank of class as (-)), = 1+ ii [ < Φ, where & are the logits. We can obtain soft ranks by replacing the indicator function with sigmoid scaled with some temperature value Ta to obtain reliable estimates of the rank of the top predicted class. Once the rank [R(-)] for true class 1 is calculated, can estimate the accuracy
max(0, 2 [R])).
Soft binning Our approach is similar to YYang et al. 2018). We take confidence A for example x; and pass it through one-layer neural network softmax((wp, + b)/z)) parameterized with different values of w and b as explained in (Yang et al.] 2018), with temperature to control the binning. This leads to M different probabilities, saying how likely is that P. belongs to the specific bin BMEL. We will denote these probabilities as (x;) = p( (p.).
Putting these parts logether, define DECE using a minibatch of examples as:
M DECE Σ
iist Om (I:) Jace (Bm)) conl (Bm)) )
acc (Bm) = Σm Om (mr)
Om Om (x:)1 (5s = vi),
1=1
conf (Bm) = Σn Om (2x)
On (2:))..
i=1
4

---

Published in Transactions on Machine Learning Research (08/2023)
3.3 Meta-learning
Differentiable ECE provides an objective optimise, but we still need to decide how utilize it. One option could be to directly use it as extra objective in combination with standard cross-entropy, used by few existing attempts (Karandikar et al. 2021 Kumar et al. 2018). However, expect this to be unhelpful as calibration on the training set is usually good the issue being failure of calibration generalisation to the held out validation or test set (Carrell et al. 2022), as illustrated in Figure Meanwhile multi-task training with such non-standard losses may negatively affect the learning dynamics of existing well tuned model training procedures. To optimise for calibration of held-out data, without disturbing standard model training dynamics, we explore the novel approach of using DECE as part of the objective for syper-paramett meta-learning in outer loop that wraps inner learning process of conventional cross-cntropy-driven model training.
Meta-learning objective We formulate our approach as a bilevel optimisation problem. Our model is assumed to be composed of feature extractor and classifier Φ. These are trained minimise CE cross-entropy loss on the training set. The goal of meta-learning to find hyper-parameters so that training with them optimises the meta-objective computed on the meta-validation set. In our case the meta-objective is a combination of cross-entropy and DECE to reflect that the meta-learned syper-parameters should lead to both good generalization and calibration. More specifically the meta-objective is Lval EEE+ADECE: with hyper-parametes specilying how much weight is placed on calibration. The bilevel optimisation ion problem can then be sunmarized as:
wt arg min Coal
LCE+XDECE(" (w)),
6. 0 (w) arg min Ctrain (Φ O...
Φ.0
To solve this efficiently, we adopt online meta-learning approach (Tuketina al. 2016 Hospedales al., 2021) where alternate base model and hyper- hyper-parameee parameter updates. This is an efficient strategy as do not need to backpropagate through many inner-loop steps or retrain the model from scratch for each update of mela-knowledge.
When simulating training during the inner loop, we only update the classifier and keep the feature extractor frozen for efficiency, as suggested by (Balaji et al., 2018). Base model training is done separately using full model update and more advanced optimiser.
We give the overview of our meta-learning algorithm in Algorithm D The inner loop that trains the main model (0, Φ) (line 10) is conducted using hyper-parameteers w, while the outer loop (line 12) that trains the parameersss does not directly use it for evaluating the meta-objective (e.g. no learnable label smoothing is applicd to the meta-validation labels that are used in the outer loop). We backpropagate through one stop of update of the main model.
Hyper-parameter choice A key part of meta-learning is to select suitable knwwelno (hyper- parameters) that we will optimise to achieve the meta-learning goal (Hospedales et al.) (021). Having cast calibration optimisation as meta-learning process, we are free to use any of the wide range of typer-parameters surveyed in [Hospedales et al.) p. Note that in contrast to grid search that standard temperature scaling (Guo et al.] 2017 Mukhoti et al. 2020) and other approaches rely on, we have gradients with respect to hyper-parameters and so can therefore potentially optimisc calibration with respect to high-dimensional hyper-parametees. In this paper we explore two options to demonstrate this generality: 1) Unit-wise L2 regularization coefficients of each weight in the classifier layer, inspired by (Balaji et al. 2018 and (Lorraine
et al. 2020); and 2) various types of learnable label smoothing (I.S) (Müleer et al. 2019 However, we found
LS to be better overall, so our experiments focus on this and selectively compare against L2.
Learnable label-smoothing Learnable label smoothing learns one or more coefficients to smooth the one-hot encoded labels. More formally, if there are K classes in total, yk is 1 for the correct class k e and
5

---

Published in Transactions on Machine Learning Research (08/2023)
Algorithm
1: Input: a, f: inner and outer-loop learning rates
Output: trained feature extractor 0, classifier & and label smoothing w
3: w w~ p(w)
4: p,0~p(6),p(0) while training do
Sample minibatch of training Tt. Vr and meta-validation Zee Uo examples
// For LS: use to smooth Yt
8:
// For L2: add unit-wise weight-decay w
9:
Calculate C; = Los (fæce (ze),,(.w))
10: Update 0, Φ + 0 o aV.
11:
Calculate C. = LCE+ADECE (Fooe (xx)
12:
Update w +- - WVo.
13:
end while
yk is 0 for all classes + e, then with learnable label smoothing w the soft label for class k becomes
yk yu We,k) + we.k/K K.
In the scalar case of label smoothing, we is the same for all classes, while for the vector case it takes different values for each class c. We consider scalar and vector variations as part of ablation.
Our main variation of meta-calibration uses non-uniform label smoothing. It computed using the overall strength of smoothing wa for the correct class and also wh,k saying how w is distributed across the various incorrect classes k # Given correct class c, with this variation the soft label for class le is calculated as:
wd
LS = y/(1 11--w w() + w uk
+ >i w
where we normalize the distribution weights to sum to and use small value e to avoid division by 0. The learnable label smoothing parameters are restricted non-negative values, with the total label smoothing strength most 0.5. In practice the above is implemented by lcarning a vector of elements specifying the strengths of overall label smoothing for different correct classes c, and a matrix of K (K - 1) elements specifying how the label smoothing is distributed across the incorrect classes k + c.
Learnable L2 regularization In the case of learnable L2 regularization (ef: Balaji el al. (2018)), )), the goal is to find unit-wise 1.2 regularization coefficients for the classifier layer & so that training with them optimises the meta-objective that includes DECE (0 is the feature extractor). The inner loop loss becomes
L: = Lox (foce (ax). v.) + w|l012
4 Experiments
Our experiments show that DECE-driven meta-learning can be used to obtain excellent calibration across a variety of benchmarks and models.
4.1 Calibration experiments
Datasets and settings We experiment with CIFAR-10 and CIFAR-100 benchmarks (Krizheskyy 2009).
SVHN (Nezzer et al. 2011) and 20 Newsgroups dataset (Lang 1995), covering both computer vision and NLP. For CIFAR benchmark, we use ResNet18, ResNet50, ResNct110 He al. 2015) and WidcResNet26-1 10 2016) models. For SVHN we use ResNet18, while for 20 Newsgroups we use global pooling CNN (Lin et al. 2014). We extend the implementation provided by (Mukhoti et al. 2020) to
Zagoruyko & Komodakis

---

Published in Transactions on Machine Learning Research (08/2023)
implement and evaluate our meta-learning approach. We use the same authors for lair comparison, which we summarize next.
selected by the
CIFAR and SVHN models are trained for 350 epochs, with a multi-step scheduler that decreases the initial learning rate of 0.1 by a factor of 10 after 150 and 250 epochs. Each model is trained with SGD with momentum of 0.9, weight decay of 0.0005 and miníbatch size of 128. 90% of the original training set used for training and 10% for validation. In the case of meta-learning, we create a further separate meta-calidation set that is of size 10% of the original training data, so we directly train with 80% of the original training data. 20 Newsgroups models are trained with Adam optimiser with the default parameters, 128 minibatch size and for 50 epochs. As the final model we select the checkpoint with the best validation accuracy.
For DECE, we use M = 15 bins and scaling parameters Ta = 100, T. = 0.01. Learnable label smoothing coefficients are optimised using Adam (Kingma & Ba 2015 optimiser with learning rate 0.001. The meta-learnable paramcters are initialized at 0.0 (no label smoothing or L2 regularization initially). The total nunber of mneaa-parameters is K x K, and K for the non-uniform, scalar and vector label smoothing respectively, while it is 512 x K + K for learnable L2 regularization. We use X = 0.5 in the meta-objective, and we have selected it based on validation set calibration and accuracy after trying several values.
Results We first follow the experimental setup of Mukhoti et al. (020) and compare with the following
alternatives: 1) cross-entropy, 2) Brier score (Biie 1950), 3) Weighted MMCE (Kumar et al. 2018) with À == 2, 4) Focal loss (Lin ct al. 2017) with y 3, 5) Adaptive (sample dependent) focal loss (FLSD) Mukkooti 3 for predicted probability p E [0,0.2] and p E [0.2, 1) respectively. 6) Label smoothing (LS) with a fixed smoothing factor of 0.05. In all cases we report the mean and standard deviation across three repetitions to obtain a more reliable estimate of the performance. In contrast, Mukhoti et al. (2020) report their results on only one run, so in the tables we include our own results for the comparison with the baselines.
et al. 2020) with 7 7 and
We show the test ECE, test ECE after temperature scaling (TS) and test error rates in Tables 1 2 and respectively. Meta CaltaCaaibbationn leads to excellent intrinsic calibration without the need for post-processing
(Table D. which is practically valuable because post-processing is not always possible (Kim & Yun 2020)
or reliable (Ovadia et al.) 2019). However, even after post-processing using TS Meta-Calibration gives competitive performance (Table 2 as evidenced by the best average rank across the considered scenarios. Table 3 shows that Meta- Calibration maintains comparable accuracy to the competitors, even if it does not have the best average rank there. Overall CalabCaaibr leads significantly better intrinsic calibration, while keeping similar or only marginally worse accuracy.
Table 1: Test ECE (%, 4): Our Mcta-Calibratte (MC) leads to excellent intrinsic calibration.
Dataset
Model
CR
Brier
MMCE
F1.3
FLSD-553
MC (Ours)
ResNes18
4.23 0.15
0.03
4.36
0.16
2.11
2.22
0.04
0.06
1.17 0.26
ResNet50
4.20
0.01
1.95 + 0.15
4.49 0.18
1.48
0.19
1.68
0.14
2.58 + 0.26
1.09
0.09
CIFAR-10
ResNet110
4.81
0.12
2.58 4 0.17
4.20
0.74
1.82
2 0.20
2.16
0.22
1.96 : 0.36
1.07
0.12
WideResNe126:10
3.37 1 0.11
1.03 1 0.08
3.48
0.06
1.57
1: 0.32
1.50
1 0.15
3.68 :: 0.10
0.94
0.10
ResNer18
8.79 1 0.59
5.19 = 0.18
7.41 = 1.30
2.83 :: 0.27
2.47 + 0.12
6.87 = 0.29
2.52 = 0.35
ResNet50
12.56
1.44
4.82 1 0.36
9.02 1.72
4.78 1: 1.00
5.43
0.31
5.94 + 0.52
3.07 0.18
CIFAR- 100
ResNet110
11.96
0.83
6.52
= 0.56
12.29 = 1.25
6.61
= 1.12
7.38
0.25
10.09 = 0.39
2.80 = 0.58
WideReNNet26-00
12.39
4.26 0.30
8.35 2.79
2.36
0.13
2.30
0.36
0.96
3.86 0.34
SVHN
ResNet18
2.98
0.08
1.94 0.10
3.14 0.10
2.69
0.06
2.83 0.17
3.85
0.01
1.14 0.12
Newsgroups
Global Pooling CNN
18.58
0.80
16.49 0.70
14.68 1.03
7.51
6.13 1.84
5.14 + 0.64
2.56 0.38
Average rank
6.4
3.5
2.8
3.1
1.8
1.3
Note that while Brier score, Focal loss and FLSD modify the base model's loss function, our Mata-Calibration corresponds to the vanilla cross-entropy baseline, but where label smoothing is tuned by our DECE-driven hyper-parameter meta-learning rather than being selected by hand.
7

---

Published in Transactions on Machine Learning Research (08/2023)
Table 2: Test ECE with temperature scaling (%, 4): Our Meta- Veta-Calibration (MC) obtains excellent calibration also after temperature scaling.
Datase
Modiel
Brier
MMCE
F1-3
FLSD--3
T.5
MC (Chrss
ResNer18
1.36
0.4:2...0 0.30 (2.30)
1.25 0.03 (1,00)
1.25 1. 0.08 (2.30)
1.10 2: (0.90)
1.45 2. 0.31 (0.57)
1.31 #: 8.08 (0.90)
1.34 4: 0.47 (0.97)
CIFAR-10
ResNetI10
50
1,49
1.30 0.17 (2.53)
2: 0.19 (2.57)
0.97 + 002 (1:17)
135 0.35 (1.13)
1.20 1 0.45 (1.50)
1.35 + 0.38 (2.50)
1.21 1: 0.07 (1.10)
1.10- 0.31 (1.07)
1,33 4: 0.14 (1.30)
1.21 + 0.31 (1.00)
2.16 4: 0.21 (0.50)
1.12 + 0.15 (0.00)
1.30 0.37 (0.97)
1.00
0.09 (1.00)
WedeReeNet26- 10
0.13
103
(1.00)
0.90
1.20
(0.87)
1,00
0.02
(0.90)
1:32 900
0.10
BesNet1s
0.22 (1.30)
4.21
(0.00)
6,00
(1.13)
285 0.27 (1.00)
2.47 0.32 (1.00)
4:37 0.45
0.58 (1.05)
BeaNatt
(1.57)
2:25
(3:10)
0.11
(1.30)
0.42
ResNet110
0.51 (1.57)
3.71
(1.17)
2.74
(1.40)
3.77
12.10)
4.13 0.40 (1.30)
6.04 0.31 (1.10)
0.33 (1.03)
11
0.26 (1.50)
2.19
(1:10)
3.52
(1.40)
2:50 0.12 (1.05)
2.50
0.55 (1.00)
5.02
0.24 (1.07)
:: 0.19 (1:10)
SVHN
18
+ 0.01 (2.10)
0.85
(1.900)
1.10
0.01 (2.90)
0.90 (0.85)
1,11 + (0.87)
1:15
0.51 (0.87)
0.12 (1.00)
20 Newsgroups Global Pooling CNN
2,85 4: 0.34 (3.67)
4.32
0.79 (2.97)
4,00 a 0,22 (2.00)
3.59 ds 0.34 (1.43)
2.76 a: 0.20 (1.83)
0.19 # 0:30 (3.10)
2:50 0.12 (0.97)
Joongg tack
3.7
3.8
4.4
3.0
8.9
6.2
2.8
Table 3: Test error (%, 1): Our Meta- Calibration (MC) obtains excellent calibration with only small increases in the test error.
Dataset
Model
CE
Brier
MMCE
F1.-3
FLSDD-3
L.
MC (Ours)
ResNe118
4.99 1: 0.14
5.27
1 0.21
5.17 1 0.19
5,06 1: 0.09
5.22
0.04
4.94 1: 0.13
5.22
1: 0.06
CIFAR-10
Res/Ne550
0.02
5.15
5.13
0.12
5.27
0.22
5.26
0.11
5.46
0.05
Nos
5.40 + 0.10
5.97
0.17
5.70
+ 0.12
5.67
0.33
5.87
0.13
5.45 + 0.11
6.09
0.22
WideResNet26-10
3.99 + 0.07
4.20
+ 0.03
4.11 0.06
4.18 0.03
4.22 + 0.05
4.05 + 0.07
4.36
0.20
ResNet18
22.85 + 0.17
23.50
+ 0.17
23.80 + 0.18
22.87 = 0.16
23.23
0.32
22.35 + 0:27
23.88
= 0.20
ResNet500
22.41 + 0.24
24.81
1 0.33
22.43
1 0.05
22.27 = 0.13
22.76
1 0.27
21.85
+ 0.06
23.22
: 0.48
CIFAR-100
ResNet110
22.99 1: 0.19
28.29
1: 1.42
23.81
0.58
23.12 1: 0.26
23.71
0.24
23.08
1. 0.15
24.51
1: 0.41
WideResNet26-10
20.41
0.12
20.77
0.05
20.60
0.10
19.80
0.40
0.25 19.97
20.82
0.42
22.35
0.03
SVHN
RosNot18
0.08
3.90
4.15
0.08
0.07
4.18
0.06
0.09
4.08
Newagroupe
Global Pooling CNN
26.64 +: 0.27
26.39
26.92
0.32
27.65
0.38
27.59
0.94
26.10
0.31
27.26
0.59
Average rank
4.9
4.2
4.8
5.9
4.2
Further analysis
Alternative calibration metrics We investigate if models meta-trained using DECE also perform well when evaluated using more advanced calibration metrics than ECE. Tn particular, we evaluate performance using dass ECE (CECE) (Kunar et al. 2019 Widmann et al. 2019 Vaicenavicius et al. 2019 Kull el al. 2019) that considers the scores of all classes in the predicted distribution, instead of only the class with the highest probability. The results in Table 4 confirm that models meta-trained using DECE have excellent calibration also in terms of the CECE criterion.
Table 4: Test Classwise- ECE (%, 4): Our Meta-Calibration (MC) leads to excellent calibration also when using a more advanced calibration metric.
Dataset
Model
CE
Brier
MMCE
FL-3
FLSD-53
LS-0.05
MC (Ours)
18
0.87 + 0.04
0.46 + 0.02
0.91 + 0.03
0.52 + 0.02
0.53 0.03
0.73 +.01
0.41 + 0.01
ResNet50
0.88 + 0.01
0.46 + 0.02
0.93 + 0.04
0.44 = 0.02
0.44 + 0.03
0.63 + 0.02
0.48 = 0.04
CIFAR-10
ResNet110
0.99 + 0.02
0.58
: 0.04
0.89 0.14
0.50 = 0.02
0.55 : 0.05
0.66 + 0.03
0.15 = 0.01
WideResNet
0.71 + 0.01
0.37 : 0.00
0.74 + 0.01
0.43 = 0.02
0.43 1 0.04
0.72 + 0.02
0.34 = 0.00
ResNet18
0.23 1 0.01
0.24 1 0.00
0.22 0.01
0.20 1 0.00
0.20 0.00
0.26 1 0.00
0.19 1: 0.00
CIPAR-100
Net50
0.03
0.20
0.01
0.24 0.03
0.00
0.20
0.00
0.19
ResNet110
0.34 + 0.02
0.23
0.29 0.02
0.22 0.01
0.23 0.00
0.26 + 0.00
0.19 = 0.00
WideResNet
0.29 + 0.02
0.19 = 0.00
0.23 + 0.03
0.18 0.00
0.18 + 0.00
0.21 + 0.01
0.19 0.00
SVHN
ResNet15
0.62 + 0.02
0.52 + 0.02
0.65 + 0.02
0.67 = 0.03
0.68 + 0.04
0.81 + 0.05
0.29 + 0.03
20 Newsgroups Global Pooling CNN
2.01 + 0.07
1.80 + 0.04
1.63 + 0.09
1.29 + 0.03
1.22 + 0.14
0.97 + 0.06
1.00 + 0.04
Average rank
6.0
3.3
5.8
2.5
2.8
5.1
1.6
choice We present a general metric that can be used for optimising hyper- parameters for superior calibration. While our main experiments are conducted with non-uniform label smoothing, we demonstrate the generality of the framework by also learning alternative meta-parameters. Tn
Alternative
byper-parameters
8

---

Published in Transactions on Machine Learning Research (08/2023)
particular, we also consider scalar and vector version of label smoothing well learnable L2 regularization. We perform the additional evaluation using ResNet18 on the CIFAR benchmark.
The results in Table confirm learnable L2 regularization also leads to clear improvement in ECE over the cross-entropy baseline. However, the error rate is slightly increased compared to learnable LS, hence we focused on the latter for our other experiments. Scalar and vector T.S (MC-S and MC-V) have both improved the calibration, but non-uniform label smoothing (MC) has worked better thanks to its larger expressivity.
Table 5: Comparison of hyper-parameter choice for meta-calibration: CIFAR benchmark with ResNet18 model. Test errors (%, J) and test ECE (%, 4). Other variants of Meta- Cetta-Clibration also lead to strong improvements in calibration, with non-uniform label smoothing leading to the best calibration overall.
Dataset
Method
ECE (1)
Error (1)
CE
4.23 + 0.15
4.99 0.14
MC
1.17 + 0.26
5.22 0.06
CIFAR-10
MC-S
1.48 + 0.26
5.17 = 0.13
MC-V
1.51 +: 0.26
5.07 0.03
MC-12
1.78 + 0.22
5.49 = 0.14
CE
8.79 + 0.59
22.85 0.17
MC
2.52 + 0.35
23.88 0.20
CIFAR-100
MC-S
6.13 1.20
24.07 0.17
MC-V
3.98 + 0.23
23.96 0.12
MC-12
4.18 + 0.26
26.10 0.14
Ablation study on meta-lcarning objective design Recall our framework in Equation 1 sctup perform conventional model training in the inner optimisation, given hyper-parameters; and meta-learning of prper-parameters in the outer optimisation, by minimising combination of cross-entropy and our DECE metric as evaluated on the meta-validation set. While we view this setup as being the most intuitive, other architectures are also possible in terms of choice of objective for use in the inner and outer layer of the bilevel optimisation. As a comparison to our DECE, we also evaluate the prior metric MMCE previously proposed as proxy for model calibration (Kumar et al. 2018).
From the results in Table we can conclude that: 1) Meta- Meta-learning with combined CE and DECE meta- objective is beneficial for improving calibration (M5 MO). 2) Alternative outer-loop objectives CE (M2) and DECE (M3) improve calibration but not significantly as the combined meta-objective (M5 vs M2 and M5 vs M4). 3) MMCE completely fails as a meta-objective (M3). 4) DECE improves calibration when used as a secondary loss in multi-tas task learning, but at greater detriment to test error (M1 vs M0). 5) Our combined meta-objective (M5) is the best overall.
Table 6: Ablation study on losses for inner and outer objectives in bilevel optimisation using CIFAR-10 and CIFAR-100 with 18sNet18.
CIFAR-10
CIFAR-100
Model
Meta-Loss
Loss
ECE (%, 4)
Error (%, 4)
ECE (%, 4)
Error (%, 4)
MO: Vanilla CE
CE
4.23 + 0.15
4.99
+ 0.14
8.79 +:2 0.59
22.85 + 0.17
M1: Multi-task
CE
DECE
3.80 0.03
10.24
1 0.21
4.40
0.39
29.49
+ 0.17
M2: Meta-Calibration
CE
CE
1.31 1 0.36
5.13
1 0.24
3.00
1 1.12
23.72
1 0.40
M3: Meta-Calln Cota-Calibration
MMCE
CE
44.24
0.70
6.77
1 0.25
21.94
2.39
25.40
1 0.31
M4: Meta-Calibration
DECE
CE
1.26 - 0.44
5.21
1 0.14
3.28 1 0.31
23.83
+ 0.14
M5: Meta-Caaibration
CE DECE
CE
1.17 0.26
5.22
1 0.06
2.52
0.35
23.88
+ 0.20
A koy contribution of this work is DECE, a differentiable approximation expected calibration error. In this section we investigate the quality of our DECE approximation. We trained the same ResNet18 backbone both CIFAR-10 and CIFAR-100 benchmarks
Evaluating DECE approximation ECE
9

---

Published in Transactions on Machine Learning Research (08/2023)
for 350 epochs, recording DECE and ECE values at various points. The results in Figure 2a show both Spearman and Pearson correlation coefficient between DECE and ECF. In both cases they are close to 1, and become even closer to as training continues. This shows that DECE accurately estimates ECE, while providing differentiability for end-to-end optimisation. We further we show in Figure 2b that their mcan values are very close lo each other.
CIFAR-10D
0.5
CIFAR-10
0.5
CIFAR-100
ConrelatioiRCE .6cc
1,000-
ECE
0.1
0.4
DECE
0,595
Calleration
0.96
0.3
0.990
0.04
0.2
0.2
0.985
0.90
Spearman
0.1
0.1
0.980
Pearson
100
200
250
300
100
200
305
250
250
Epoch
Epoch
Epoch
Epach
(a) Correlation between DECE and ECE is close to 1.
(b) Mean ECE and DECE are close to each other.
Figure 2: Evaluation of how DECE approximates ECE, using ResNet18 on CIFAR.
What hyper-parameters are Icarned? We show our approach learns non-trivial hyper-paramcee scttings to achieve its excellent calibration performance. Figure 3 shows how the learned overall strength of smoothing evolves during training for both CIFAR-10 and CIFAR-100 benchmarks using ResNet18. We show the mean and standard deviation across three repetitions and all classes.
From the figure we observe label smoothing changes in response to changes in learning rate, which happens after 150 and 250 epochs. For CIFAR-100 with more classes it starts with large smoothing values and finishes with smaller values. The large standard deviations are due to the model making use of a wide range of class-wist wise smoothing paramcters. It would be infeasible to manually select a curriculum for label smoothing at different stages of training, it would be to tune a range of smoothing parameters: The ability to opt these hyper parammaareeeerrs automatically is a key benefit of our framework.
Label smoothing
CIFAR-10
5 smoothing
CIFAR-100
0.15
0.5
0.10
0.4
0.3
0.05
0.2
0.1
0.00
0.0
0
50
100
150
200
250
300
350
50
100
150
200
250
300
350
Epoch
Epoch
Figure 3: Overall label smoothing during training for CIFAR, using ResNet18. The learned smoothing strategy is non-trivial and adapts according to the learning rate schodule.
We also analyse how the smoothing is distributed across the different classes in Figure 4 and 5 The results show that the smoothing is indeed non-uniform, demonstrating the model does exploit the ability to learn a complex label-smoothin distribution. The learned non-uniform label-smoothing distribution can be observed to subject visually similar classes to more smoothing (igure[[()) which makes sense reduce the confidence of the most likely kinds of specific errors. This idea is quantified more systematically for CIFAR-100 100 in Figure 5 which compares the average degree of smoothing between classes in the same superclass, and those in different superclasses. The results show that within-superelass smoothing is generally much stronger than across-superclass smoothing, even though the model receives no amnotation or supervision about superclasses. It learns this smoothing structure given the objective of optimising (meta-)yalidation calibration.
We further analysed the hyper-parandn parameters in the case of learnable L2 regularization and show as part of the appendix. The figure shows we learn range of regularization values to achieve good calibration outcome.
10

---

Published in Transactions on Machine Learning Research (08/2023)
0.0
0.2
0.0
0.0
0.0
0.1
0.1
0.0
0.0
0.0
0.1
0.2
0.5
0.5
0.1
0.0
0.3
0.0
0.1
0.1
0.2
0.0
0.0
0.0
0.0
0.1
0.0
0.1
0.5
0.1
0.0
0.0
0.4
0.0
0.0
0.4
0.1
0.0
0.1
0.1
0.3
0.0
0.0
0.03
0.01
0.01
0.07
0.03
0.07
0.01
0.01
0.01
0.03
0.025
0.050
0.0
0.1
0.6
0.1
0.0
0.1
0.1
0.0
0.0
0.3
2
6
9
0.0 0.1
0.3
0.7
0.2
0.1
0.0
0.0
0.0
0.1
0.2
0.1 0.1
0.1
0.2
0.1
0.2
0.0
0.0
0.0
0.1
0.2
0.2
0.1
0.0
0.0
0.3
0.1
0.2
0.5
0.1
0.1
0.0
0.0
0.0
0.0
0.0
0.0
(a) Overall level of smoothing in various classes.
(b) Distribution of smoothing across varions classes.
Figure 4: Analysis of learned non-uniform label smoothing for CIFAR-10, using ResNet18 model. Visually similar classes receive more smoothing e.g. cat and dog (3 and 5), and automobile and truck (1 and 9).
Diferent Same
0.08
0.08
0.11
0.06
0.10
0.15
0.03
0.01
0.01
0.1 001
001
0.07
0.01
Figure 5: CalibCaanoootion learns to give more label smoothing within the same superclass compared to other superclasses in CIFAR-100, using ResNet18 model.
This highlights the value of our differentiable framework that enables efficient gradient-based optimisation of many paperr-arammter...
Analysis of accuracy vs calibration using simulated data Meta-Calibratton leads significantly better calibration, while kceping similar or only marginally worse accuracy. To study if there is a trado-off between accuracy vs calibration we perform an experiment using simulated data. More specifically the experiment with simulated data consists of 1) sampling parameters of binary logistic regression model (oracle) over 2D data, 2) sampling the label for each data point from a binomial distribution with class probability given by the oracle. 3) We then use the sampled data to train (i) vanilla logistic regression model, (ii) label-smoothing logistic regression model and (iii) a Mela-Calibratiin logistic regression model. This enables us to compare the best-case classifier accuracy and calibration with the results of learned models. Our experiment repeated across three random seeds that we can report the mean and standard deviation.
The results in Table 7 confirm iela-Calibration matches both the test accuracy and ECE of the oracle, obtaining close to the best possible calibration. Analogous cross-entropy training well as simple label smoothing (LS) have significantly larger ECE. The ECE of Meta-Calibration and oracle is close to 0, but not precisely 0 due to sampling effects (i.e. because we do not use an infinite amount of data). We have also visualized the estimated class probabilities different data points, together with the decision boundary in Figure The visualization shows that Meta-Calibration and the oracle are similarly calibrated (e.g., similar point shades close the decision boundary), while difference in calibration (point shading) perceptible for cross-entropy and L.S.
5 Discussion
This work pioncering step using meta-learning directly optimisc model calibration. Learnable rather than hand-tuned calibration important as different models and datasets have very different calibration properties, precluding neessie--fissalll solution (Minderer et. al. 2021 There many ways work could
11

---

-1 Published in Transactions on Machine Learning Research (08/2023)
Table 7: Analysis of accuracy and calibration on simulated data with known oracle. Test accuracy and ECE (%) are reported, with the mean and standard deviation computed across three random seeds. Our Meta- Calibration (MC) matches both the accuracy and ECE of the oracle, obtaining close to perfect calibration.
Metric
Oracle
Cross- Suuss-Entropy
LS
MC (Ours)
Accuracy (1)
87.62 + 1.87
87.55 + 1.86
87.57 + 1.87
87.55 + 1.81
ECE (U) 1.38 + 0.32
9.81 + 1.14
3.61 + 0.43
1.40 = 0.19
Oracle
Cross Entsr-y
2
2
00
Class
x
0
0
04
Class
-1
-1
02
-2
-2
1
LS
Meta- Meta-Calibration
2
21
1
Class
2 0
04
Class
-1
02
02
-2
-2
X1
Figure Visualization of the estimated probabilitics across test data points, with the decision boundary.
be extended in the future. One direction is to target different hyper- ppeerpparmmeeess beyond the label smoothing and L2 classifier regularization evaluated here, such as loss learning. While did not explore this here, the framework could also be used to unify post-hoc methods such as temperature scaling by treating temperature as the hyper-parameter Secondly, better meta-learning algorithms such as implicit meta-learning (Lorraine) ltt al. 2020) could better optimise the meta-objective. A third direction to extend the differentiable metric itself c.g. adapt to various domain-spoeific calibration mcasures for example ones relevant to the finance industry (Liu et al. 2019).
A drawback of our approach is the computational overhead added by meta-learning compared to basic model training. However, it is still manageable and may be worth it when well-calibrated models are crucial. We give an overview in Table in the appendix. Ongoing advances in efficient leta-learners (Bohdal et al. 2021 can make the overhcad smaller. In terms of social implications, our work aims to improve the reliability of neural networks, but there still are risks the neural networks will fail to accurately estimate their confidence.
6 Conclusion
We introduced a new DECE metric that accurately represents the common calibration ECE measure and makes differentiable. With DECE, we can directly optimise hyper-parameters for calibration and obtain competitive results with hand-designed architectures. We believe DECE opens up new avenue for the community to tackle the challenge of model calibration in optimisation- based ways.
Acknowledgments
This work was supported in part by the EPSRC Centre for Doctoral Training in Data Science, funded by the UK Engineering and Physical Sciences Research Council (grant EP/L016427/1) and the University of Edinburgh.
12

---

Published in Transactions on Machine Learning Research (08/2023)
References
Gorkem Algan and Πkky Ulusoy. Metalabelnet: Learning to generate soft-labels from noisy-labels. Transactions Image Processing, 2022.
IEEE
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. MetaReg: towards domain generalization using meta-regularization. In NeurIPS, 2018.
Mathieu Blondel, Olivier Teboul, Quentin Berthet, and Josip Djolonga. Fast differentiable sorting and ranking, In ICML, 2020.
Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: learn labels instead of images. In NeurIPS MetaLearn Workshop, 2020.
Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. EvoGrad: efficient gradient- gradient-based meta-learning and hyperparameter optimization. In NewrIPS, 2021.
Ondrej Bohdal, Da Li, Shell Xu Hu, and Timothy Hospedales. Feed-forward source-free latent domain adaptation via cross-attention. In ICML 2022 Workshop Pre-training: Perspectives, Pitfalls, and Paths Forward, 2022.
Ondrej Bohdal, Yinbing Tian, Yongshuo Zong, Ruchika Chavhan, Da Li, Henry Gouk, Li Guo, and Timothy Hospedales. Meta Omnium: A benchmark for general-purpose learning-to-leamn. In CVPR, 2023.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End to end learning for self-driving cars. In orXiv, 2016.
Glenn W Brier. Verification of forecasts expressed in terms of probability. In Monthly weather review, 1950.
Annabelle Carrell, Neil Mallinar, James Lucas, and Preetum Nakkiran. The calibration generalization gap- In ICML 2022 Workshop on Distribution- Freee Uncertainty Qnantification, 2022.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intelligible models for healtheare. In KDD, 2015.
Wenlin Chen, Austin Tripp, and José Miguel Hornández-Lobato. Mcta-learning adaptive deep kernel gaussian processes for molecular property prediction. In ICLR, 2023.
Shaker El-Sapp,gh, Jose Maria Alonso-Moral, Tamer Abuhmed, Farman Ali, and Alberto Bugarín-Diz Trustworthy artificial intelligence in alzheimer's discase: state of the art, opportunities, and challenges. Artificial Intelligence Review, 2023.
Martin Ferianc, Ondrej Bohdal, Timothy Hospedales, and Miguel Rodrigues. Impact of noise calibration and generalisation of neural networks. In TCML, 2023 Workshop Spurions Corvelations, Invariance, and Stability, 2023.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseo Lee, Matthias Humt, Jianxiang Feng, Anna M. Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, M. Shahzad, Wen Yang, Richard Bamler, and Xiaoxiang Zhu. A survey of uncertainty in deep neural networks. ArXiv, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. ICML, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2015.
13

---

Published in Transactions on Machine Learning Research (08/2023)
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
Timothy M Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning neural networks: survey. Transactions on Pattern Analysis and Machine Intelligence, 2021.
Xiaoqian Jiang, Melanie Osl, Jihoon Kim, and Lucila
Calibrating predictive model estimates support personalized medicine. Journal of the American Medical Informatics Association, 2012.
Thierry Judge, Olivier Bernard, Mihaela Porumb, Agisilaos Chartsias, Arian Beqiri, and Pierre-Mare Jodoin.
Crisp- - reliable uncortainty estimation for medical image segmentation. In MICCAI, 2022.
Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Takshminarayanan, Jonathon Shlens, Michael C Mozer, and Becca Roclofs. Soft calibration objectives for neural networks. In NewrIPS, 2021.
Sungnyun Kim and Se-Yo Yun. Task calibration for distributional uncertainty in few-shot classification. In OpenReview, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR, 2015.
Ranganath Krishnan and Omesh Tickoo. Improving model calibration with accuracy versus uncertainty optimization. In NewrIPS, 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In NewrIPS, 2019.
Ananya Kumar, Percy Liang, and Tengyu Ma. Verified uncertainty calibration. In NeurIPS, 2019.
Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In ICML, 2018.
Ken Lang. Newsweoder: Learning to filter netnews. ICML, 1995.
Wei-Hong Li, Chuan-Sheng Foo, and Hakan Bilen. Learning to impute: general framework for semi- supervised learning. In Xiv, 2020.
Yiying Li, Yongxin Yang, Wei Zhou, and Timothy M Hospodales. Featuro-critic networks for heterogencous domain generalization. In JCML, 2019.
Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR, 2018.
Min Lin, Qiang Chen, and Shuicheng Yan. Network network. In ICLR, 2014.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár, Focal loss for dense object detection. ICCV, 2017.
Shuaiqiang Liu, Anastasia Borovykh, Lech A Grzelak, and Cornelis W Oosterlee. A neural network-based framework for financial model calibration. Journal of Mathematics in Industry, 2019.
Jonathan Lorraine, Paul Vicol, and David Duvenand. Optimizing millions of hyperparameters by implicit differentiation. In AISTATS, 2020.
Jelena Luketina, Mathias Berglund, Aaltofi Klaus Greff, and Tapani Raiko. Scalable gradient-based tuning of continuous regularization hyperparameters. In ICML, 2016.
14

---

Published in Transactions on Machine Learning Research (08/2023)
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. In NeurIPS, 2021.
Lukas Mosser and Ehsan Zabihi Naeini. A comprehensive study of calibration and uncertainty quantification for bayesian convolutional neural networks an application to seismic data. Geophysies, 2022.
Jishnu Mukholi, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip H S Torr, and Puneet K Dokania. Calibrating deep neural networks using focal loss. In NeurIPS, 2020.
Rafael Müller, Simon Kornblith, and Geofrrey Hinton. When does label smoothing help? In NeurIPS, 2019.
Muhammad Akhtar Munir, Muhammad Haris Khan, Salman Khan, and Fahad Shahbaz Khan. Bridging precision and confidence: A train-lime loss for calibrating object detection. In CVPR, 2023.
Pakdaman Naeini, Gregory F Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In AAAI, 2013.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupercised Feature Learning, 2011.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snock. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In NewrIPS, 2019.
Tao Qin, Tie Yan Liu, and Hang Li. A general approximation framework for direct optimization of information retrieval measures. Information relrievol, 2010.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-Weight-NGr learning an explicit mapping for sample weighting. In NewrIPS, 2019.
Yisheng Song, Ting Wang, Subrota K Mondal, and Jyoti Prakash Sahoo. A comprehensive survey of few-shot learning: Evolution, applications, challenges, and upportunities. ACM Compuling Surveys, 2023.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, and Jon Shlens. Rethinking the Inception architecture for computer vision. In CVPR, 2016.
Christian Tomani, Sebastian Gruber, Muhammed Ebrar Erdem, Daniel Cremers, and Florian Buettner. Post-hoc uncertainty calibration for domain drift scenarios. In CVPR, 2021.
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In ATSTATS, 2019.
Deng-Bao Wang, Lei Feng, and Min-Ling Zhang. Rethinking calibration deep neural networks: do not be afraid of overconfidence, In NewrIPS, 2021.
Qizhou Wang, Junjie Ye, Feng Liu, Quanyu Dai, Marcus Kalander, Tongliang Liu, Jianye Hao, and Bo Han. Out-of-distribution detection with implicit outlier transformation. In ICT.R, 2023.
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation. In orxiv, 2018.
Yaqing Wang, Quanming Yao, James Kwok, and Lionel M. Ni. Generalizing from a few examples: A survey on few-shot learning. ACM Computing Surveys, 2020.
David Widmann, Fredrik Lindsten, and Dave Zachariah. Calibration tests in multi-class classification: A unifying framework. In NewrIPS, 2019.
Yair Wiseman. Autonomous vehicles. In Research Anthology on Cross-Disciplinary Designs and Applications of Automation, 2022.
Yongxin Yang, Irene Garcia Morillo, and Timothy M Hospedales. Deep neural decision trees. In ICML Workshop on Human Interprelability in Machine Learning, 2018.
15

---

Published in Transactions on Machine Learning Research (08/2023)
Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. ArXis, 2023.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVG, 2016.
Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: learning adapt domain shift. In NeurIPS, 2021.
A Learned L2 regularization values
We show the learned unit-wise L2 classifier regularization values in Figure 7 The results show there is a large variability in the coofficients and they take both positive and negative values.
CIFAR-10
CIFAR-100
1000
15000
Count
750
10000
500
5000
250
-.0
1.5
1100
-05
0.0
0.5
1.0
1.5
2.0
-2.0
-1.0
0.5
0.0
0.5
1.5 2.0
Figure Histograms the learned unit-wise 1,2 classifier regularization values.
B Reliability analysis
We perform reliability analysis and show the percentage of samples with various confidence levels in Figure S for CIFAR-10 and CIFAR-100. We use ResNet18 and take the best model from training early stopping. The figure shows learnable label smoothing leads visually better alignment between the expected and actual confidence binning. It also leads to softening the confidenccs of prodictions, which is expected for label smoothing.
CIFAR-10
CE
1.0
Expected
Accuracy
1.0
Expected
Samples 190
100
Samppee (99)
0.0
0.00
0.25
0.50
0.75
1.00
0.0
0.50
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0
0.00
0.25
0.50
0.75
1.00
Confidence
Confidence
Confidence
Confidence
CIFAR-100
CE
MC
CE
MC
Accuracy
Accuracy
Expected
Samples (S)
30
Camples (8))
40
0.5
Actual
0.5
Actual
25
0.00
0.00
Confidence
Confidence
Confidence
Confidence
Figure Reliability analysis for CIFAR-10 and CIFAR-100, using ResNet18 model.
Training times analysis
We report the training times of the different approaches Table (for simplicity we only include CE bascline as all baselines have similar training time). We see that even training with Meta-Calibration takes longer, the overall training time remains manageable. If excellent. calibration is key, longer training time is acceptable.
16

---

Published in Transactions on Machine Learning Research (08/2023)
Table Training times hours. We used one NVIDIA Titan X for each experiment. Meta-Calibration takes longer, but if excellent calibration priority, the additional time is acceptable.
Dataset
Model
CE
MC (Oux)
ResNet18
2.8h + 0.1h
7.0h - 0.2h
ResNet50
8.9h + 0.1h
21.9h + 0.1h
CIFAR-10
ResNet110
17.3h + 0.3h
44.6h
7.1h
WideResNet26-10
9.2h + 0.3h
29.7h + 0.2h
ResNet18
2.7h + 0.1h
12.4h + 0.2h
ResNet50
8.9h + 0.1h
26.4h + 0.2h
CIFAR-100
ResNet110
17.3h + 0.2h
50.5h + 1.0h
WideResNet226-00
9.3h + 0.4h
39.4h + 6.7h
SVHN
ResNet18
4.4h + 0.6h
9.6h - 0.1h
20 Newsgroups
Global Pooling CNN
0.1h + 0.0h
0.3h + 0.1h
D Cross-domain evaluation
Setup To evaluate our framework's calibration performance under distribution shift, we use the CIFAR-C benchmark (Hendrycks & Dietterich 2019). CIFAR-C contains relatively large varicty of image corruptions, such as adding log, pixelation, changes to the brightness and many others (overall I 19 corruption types). There are 5 levels of severity for each of them, which can be interpreted as providing 95 domains. We select 22 of these as test domains following (Zhang et al.) 2021 These include impulse noise, motion blur, fog, and elastic transform at all levels of severity, and spatter and JPEG compression at the maximum level of severity. All other domains from (Hendrycks & Dietterich 2019) are used during training and validation.
All approaches are trained with augmented (corrupted) data so that they can better generalize across domains. In each step we sample corruption to use from the set of training corruptions. Meta- Calibration in particular samples a separate corruption for the inner and outer loop so that it can train meta-parameters that are more likely to generalize to new domains.
Results We have evaluated our approach using CIFAR-C benchmark and ResNet18 model, and we show the results in Table 9 and I0 We include both the average across all test domains well as the result the most challenging domain (worst case). We report the mean and standard deviation across three repetitions. Rela-Calibration is still helpful for obtaining better calibration in the cross-domain case, but not to as large an extent as for clean data. While the efficacy of this algorithm likely depends on whether the domain-shifts seen during meta-tran tretantrainin are representative in strength those seen during meta-testing, we still view these as encouraging initial results that tools from multi-domain meta-learning can be adapted address model calibration under domain shift. Studying how to more successfully exploit meta-learning for calibration in cross-domain scenarios is an interesting research question that the ML community can focus in the future.
Table 9: Test ECE (%, 4) for our cross-domain experiments on CIFAR-C.
Dataset
Case
CE
LS
Brier
MMCE
FL
FLSD
MC (Ours)
Average
7.28 +: 0.03
3.13 + 0.19
4.20 + 0.81
4.12 + 0.23
4.02 - 0.26
4.13 + 0.01
3.00 + 0.28
CIPAR-10-C
Worst case
15.55 + 0.32
9.43 + 0.65
9.19 + 1.84
5.47 + 0.78
4.97 + 0.64
9.05 + 0.88
10.97 + 0.99
Average
7.77
0.09
5.13 0.05
4.89
+ 0.06
6.07 = 0.59
6.33
0.23
7.06
0.54
5.26 + 2.03
CIPAR-100-C
Worst case
14.75 1.09
0.71
7.40
0.32
8.30 1 0.75
8.53 1 0.16
8.94
0.61
6.79 1 2.37
17

---

Published in Transactions on Machine Learning Research (08/2023)
Table 10: Test error (%, 4) for our cross-domain experiments on CIFAR-C.
Dataset
Case
CE
LS
Brier
MMCE
FI
FLSD
MC (Ouns)
Average
10.60
10.55
10.67 + 0.11
11.08 + 0.18
11.05
0.15
10.13
0.22
CIFAR-10-C
Worst case
21.83
21.78
21.32
0.42
23.52 + 0.29
23.34
21.03
0.78
Average
34.20
0.09
35.26
+ 0.09
34.27
+ 0.23
34.26
1: 0.22
34.14
1 0.38
33.32
0.10
34.68 + 0.13
CIFAR-100-C
Wonst case
57.10
0.98
56.56
+ 0.91
56.95
+ 0.27
56.00 + 0.20
55.87 + 1.47
54.60
0.44
53.76 = 0.51
E
Comparison to SB-ECE
Soft-binned ECE (SB-ECE) approximates the binning operation in ECE that it differentiable and can be used auxiliary loss during training (Karandikar et al. 2021). Comparing DECE with SB-ECE, our DECE has both conceptual and empirical advantages. Conceptual advantages also make the accuracy component of ECE differentiable, 2) SB-ECE bimning estimate for the left-most and right-most bin can be inaccurate as a result of using bin's center value, while our binning approach does not. suffer from this. The empirical advantages are:
follows: 1) We
DECE provides a closer approximation to ECE than SB-ECE as empirically evaluated in Figure g and 10 The quality of binning in SB-ECE controlled by temperature parameter T, and we try both lower T 0.0001 and higher temperatures T 0.01. The results show DECE provides a significantly better approximation to ECF than SB-ECC regardless the value of the temperature. In fact, we see that the quality of SB-ECE approximation is relatively insensitive to the value of the temperature parameter T.
Our Meta-Calibratton with DECE leads to better calibration. Karandikar et al. (2021) propose SB-ECE and SAvUC (soft version of accuracy versus uncertainty calibration loss (Krishnan & Tickoo 20201) to be used as auxiliary losses to encourage better calibration. We evaluate SB-ECE and SAvUC on our benchmark setup, both in their original form (as a regularizer added to CE or Focal Loss) and in meta-learning framework (as part of meta-objective for LS learning) in Table 11 The results confirm the benefits of using Cattiaaaiiittooon with meta-objective that includes DECE. We additionally include comparison of error rates in Table 12 The error rates remain similar to Teea-Calibbattinn and the other baselines, although instabilities can occur that lead to noticeably worse error rates.
Karandikar et al. 2021 | also introduce an alternative way of training called interleaved training. In such training each epoch is split into two and a separate set is used for training with respect to calibration. We have implemented and evaluated interleaved training as described in (Karandikar al. 2021), and we report test. ECE and error (%) in Table 13 and 14 respectively. The results suggest interleaved training generally leads to worse ECE than approach, in most cases significantly worse. Interleaved training also leads to significantly worse error compared to our HletaCaaiibattinn and the other baselines. We attribute the empirical benefits of CataaCCalbbationn to using the calibration objective for training only the label smoothing meta-parameters and also the specialised meta-training that uses an inner and outer loop.
Table 11: Test ECE (%, J)
comparison Meta- Calibration (MC) that uses DECE vs SB-ECE and SAvUC.
Dataset
Model
CE+SBECE
CE+SAvUC
FL3+SBECE
FL3+SAvUC
MC-SBECE
MC-SAvUC
MC (Ours)
ResNet18
5.37 + 0.16
3.26 + 0.02
1.31 + 0.01
1.91 + 0.09
2.63 + 0.66
4.18 = 0.49
1.17 + 0.26
CIFAR-10
ResNet50
3.29 + 0.37
3.36 + 0.10
1.85 = 0.03
1.54 4 0.17
2.51 + 0.90
2.50 = 0.42
1.09 = 0.09
RosNet18
1.15
5.68
0.26
2.77 1 0.08
2.58 0.10
5.72 0.21
5.49
0.54
2.52
1 0.35
CIFAR-100
ResNet50
5.72 1 0.39
12.34 1 0.00
5.86 + 0.01
4.93 1 0.18
3.10 0.13
7.53
0.63
3.07
4 0.18
18

---

Published in Transactions on Machine Learning Research (08/2023)
CIFAR-10
CIFAR-100
CIFAR-10
CIFAR-100
Mean estimated value
0.5
Mean estimated value
0.5
Pearson Corr. coeff:
1.00
Pearson corr. coeff.
1.00
ECE
0.98
0.98
0.4
0.4
DECE
0.95
0.96
SB-ECE
0.94
0.94
0.2
0.2
DECE SB-ECE
0.1
0.1
0.88
0.69
0.0
0.0
200
300
100
300
100
200
300
200
Epoch
Epoch
Epoch
Epoch
Figure 9: T = 0.01: Pearson correlation coefficient between ECE/DECE and ECE/SB-ECE, and the mean estimated value of RCE, DECE and SB-ECE for CIFAR-10 and CIFAR-100 using ResNet18.
CIFAR-10
CIFAR-100
CIFAR-10
CIFAR-100
Pearson corr. coeff:
1.00
Pearson corr. coeff.
1.00
Mean estimated value
0.5
Mean estimated value value
0.5
ECE
0.4
DECE
0.3
0.3
SB-ECE
0.94
0.94
0.92
0.92
DECE SB-ECE
0.1
0.1
0.90
0.90
0.88
0.88
0.0
0.0
100
200
300
100
300
100
200
100
200
300
Epoch
Epoch
Epoch
Epoch
Figure 10: T 0.0001: Pearson correlation coefficient between ECE//DEC and ECE/SR-FCE, and the mean estimated value of ECE, DECE and SB-ECE for CIFAR-10 and CIFAR-100 using 18
Table 12: Test error (%, 4)
comparison Meta-Calibration (MC) that uses DECE vs SB-ECE and SAvUC.
Dataset
Model
CE+SBECE
CE+SAvUC
FL3++BECC
FL3+SAVUC
MC-SBECE
MC-SAvUC
MC (Ours)
ResNet18
8.97
0.04
5.07 1 0.06
5.16 1
5.20 1 0.11
5.10 1 0.12
5.17 1 0.25
5.22 0.06
CIFAR-10
ResNet50
12.39
0.39
5.03 + 0.10
5.04 0.10
5.26 + 0.23
5.30 + 0.05
5.16 + 0.21
5.46 + 0.05
18
27.55
0.18
22.64
0.32
22.87
0.34
22.88
0.26
23.98
0.14
23.83 + 0.25
23.88 + 0.20
CIFAR-100
ResNet50
28.33
0.39
22.09
0.13
22.40 +: 0.21
22.02 0.41
23.51
+ 0.45
23.18 +: 0.26
23.22 + 0.48
Table 13: Test RCE (%, 4)
comparison of interleaved training and Meta- Ceda-Calibration (MC).
Dataset
Model
CE-SBECH
CE+SAeUC
FL3+SBECE
FL33+SAUUC
MC [Ours)
ResNet18
4.81 + 1.25
4.67
0.12
6,90 # 0.18
1.52 + 0.08
1.17 + 0.26
CIFAR-10
ResNet50
3.44 4: 0.37
4.55
0.15
6.38 + 0.82
1.38 4 0.31
1.09 + 0.09
ResNet18
5.17 + 1.23
10.01 + 0.65
4.81
= 1.05
1.73 = 0.24
2.52
+ 0.35
CIFAR- 100
ResNet50
6.74 + 0.29
11.56
2.98
4.35
0.48
2.86
0.27
3.07
4 0.18
Table 14: Test error (%, 4) comparison of interleaved training and Meta- Peta-Calibration (MC).
Dataset
Model
CE+SBECE
CE+SAVUC
FLS+SBBCE
FL3+SAvUC
MC (Ows)
ResNet18
19.09
5.56
5.66 = 0.13
14.22 4 0.63
5.96 + 0.13
5.22 0.06
CIFAR-10
ResNet50
13.51 + 1.01
5.56 = 0.13
12.13 4 0.56
6.15 4 0.45
5.46 1: 0.05
ResNet18
37.66 = 2.85
26.94 + 166
39.97 + 5.71
26.61 4 0.17
23.88 + 0.20
C1FAR-100
ResNet50
35.18 1.24
24.73 = 0.39
35.61 + 2.22
25.45 1 0.97
23.22 + 0.48
19

---

Published in Transactions on Machine Learning Research (08/2023)
F Comparison with the setup of Guo et al.
(0017)
Our main results adopt the sctup from Mukhoti et al. 2020 as they provide a complete codebasc and full details of their experiments. In this section we compare this setup with the one in (Guo el al. 2017 that reports better ECE values in their paper. The comparison will focus on ResNet110 model trained with cross-entropy as is the only model present in both our results and the results from Guo et al. (0117
We have tried to estimate the details of the training used by Guo et al. (017), based on the relatively limited information in the paper and the parameters used in the demo code for different model. After estimating the hyper-parameeers of (Guo et al. 2017); the differences compared to are the following: Guo et al. (2017) train for 500 epochs and decay the learning rate by a factor of 10 after 250 and 375 epochs. In our main experiments we train for 350 epochs and decay the learning rate after 150 and 250 epochs. Minibatch size is 64 in (Guo et al. 2017), while we use 80 for ResNet110 (in general we use 128, but it was downscaled to 80 to fit into memory of our GPUs). We use weight decay of 5e-4 and no Nesterov momentutm, while Guo et al. (211)) use Nesterov momentum and potentially use weight decay of 0.001. The demo code provides the value of weight decay, but it is then not used, so we have tried both including and excluding it. While the demo code uses 300 epochs, the paper says that 500 epochs the additional experiments. Further, the way that temperature scaling (TS) is done in (Mukhoti et al. 2020) and (Guo et al. 2017) is different. Mukhhoti et al. (2020) use grid search over a range of values and argue that this gives stronger baselines than selecting the temperature by minimising validation set NLL. While we follow Mukhoti el al. (2020) in our paper, we also evaluate the alternative option. Based on the demo code, we try 50 iterations for optimising the temperature, but we also try 10 iterations because that value is mentioned in the paper of Guo et al. (2017).
used for ResNet110, so we have used 500 epochs for
We report the results of our additional experiments in Table 15 The results from the respective papers are at the top of each part, from which we observe that the stochastic depth model is better than the standard model. However, our comparison will focus on the standard model. Comparing (Guo et al.) 2017) and Muuhhoti et al., 2020) we see that the test error in (Guo et al. 2017 is significantly worse, as a result of the differences in the training and evaluation procedure. This likely to relate to the general observation (Guo et al. 2017) that "the network learns better classification accuracy at the expense of well-modelled probabilities" : The two papers report results from one run only, but for our experiments we report mean and standard deviation across three runs.
Our experiments that try to match the setup of Guo et al. (2017) show that grid-search based temperature scaling from (Mukhoti et al. 2020) is significantly better, confirming the observations of Mukhoti et al. (202)). We also observe that weight decay is beneficial, but while it gets us closer to the reported calibration, there is a significant difference in the error rate. Overall we see that neither of the configurations obtains similar values to the ones reported in (Guo al. 2017] for both CIFAR-10 and CIFAR-100 datasets and across all metrics. One of the configurations that we have evaluated "With weight decay, grid search TS" slightly improves over the setup from Mukhoti et al. (2020), but it comes instead of 350). To summarize, our additional comparison with the setup of Guo et al. (2017) has provided more detailed justification for following the setup of Mukhoti et al. (2020) in our paper.
increased runtime cost (500 epochs
20

---

Published in Transactions on Machine Learning Research (08/2023)
Table 15: Comparison of different setups for ResNet110 model. The results suggest neither configuration
obtains similar results as reported in /uuo et al. 2017 for both CIFAR-10 and CIFAR-100 datasets.
Dataset
Setup
Test ECE (%, 4)
Test ECE with TS (%, 4)
Test error (%, 4)
Guo et al.
2017
standard model
4.60
0.83
6.21
Guo al.
2017
stochastic depth model
4.12
0,60
5.64
(Mukhoti
2020
standard model
4.41
1.09
4.89
No weight decay, grid-search TS
5.96 + 0.04
1.50 + 0.24
6.39 + 0.07
No weight decay, optim. TS (50 iters)
5.53 + 0.12
5.16 0.11
5.93 = 0.14
CIFAR-10
No weight decay, optim. TS (10 iters)
5.53 0.12
5.28 0.12
5.93 =
With weight decay, grid-search TS
4.04 + 0.20
0.75 0.13
4.56 = 0.29
With weight decay, optim. TS (50 iters)
4.13 + 0.12
3.59 0.18
4.63 + 0.06
With weight decay, optim. TS (10 iters)
4.13 0.12
3.80 0.16
4.63 0.06
Our cross-entropy
4.81 + 0.12
1.49 + 0.19
5.40 = 0.10
Our CelaaCaalirattinn
1.07 0.12
1.33 0.37
6.09 0.22
Guo et al.
2017
standard model
16.53
1.26
27.83
Guoo 2017
stochastic depth model
12.67 62.67
0.96
24.91
MMukhoti
al. 2020
standard model
19.05
4.43
22.73
No weight decay, grid-search 'TS
25.68 + 0.50
1.85 + 0.33
28.71 + 0.55
No weight decay, optim. TS (50 iters)
25.88 0.27
22.84 : 0.22
20.24 + 0.27
CIFAR-100
No weight decay, optim. TS (10 iters)
25.88 + 0.27
23.96 + 0.23
29.24 + 0.27
With weight decay, grid-search TS
16.28 +: 0.22
3.54 + 0.28
21.82 + 0.19
With weight decay, optim. TS (50 iters)
16.73 0.07
11.02
22.17 + 0.24
With weight decay, optim. TS (10 iters)
16.73 0.07
12.86 + 0.14
22.17 + 0.24
Our cross-entropy
14.96 + 0.83
3.77 + 0.51
22.99 + 0.19
Our Meta-Calibration
2.80 0.58
2.55 +
24.51 + 0.41
21


Reviews:
[{"review_id": "EjF8vz6Drr", "reviewer": "Reviewer_5hfA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\n\n* The method section and algorithm are well explained. \n* A differentiable objective for calibration error is a novel contribution.\n* The ablation experiment in Figure 2 shows that the new objective aligns well with the established ECE objective. \n\n\n# Weaknesses:\n\n* Table 1 and 2 are confusing when comparing Guo et al results. The Guo et al. paper from 2017 shows ECE on CIFAR10 and 100 of 0.6% and 0.96%, respectively. However, the current results in this paper seem to be of higher/worse value, at 0.94% and 2.8%. Can this difference be explained?\n\n* The discussion section mentions an increase in complexity and computation for the proposed method. Could this be quantified and compared? Does the increase in computation outweigh the benefits of ‘better’ calibration?\n\n\n### Minor question:\n\n* Figure 1 shows a discrepancy between ECE on train and validation set. However, this divergence only seems to happen on epoch 150, what makes this point so characteristic? \n* On the reliability analysis in Figure 6: what information does this figure provide over the usual ECE metric? I have the impression that ‘the percentage of samples with various confidence levels‘ is merely a function of the dataset. When a calibrated model achieves 90+% accuracy on an easy dataset, then a well calibrated model would by definition have many high confidence predictions. Likewise for a ‘hard dataset’ one would expect even a well calibrated model to have many low confidence outputs. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of calibration in deep learning. Modern deep learning can have discrepancy between confidence outputs (usually the value of softmax) and its local probability of success. The popular ECE is non-differentiable and cannot be used for gradient-based optimisation. Hence the proposal of a differentiable alternative. Such an objective is then used in meta-learning. However, the results of this paper are incongruent with results in previous literature. Therefore, the conclusion of the paper is not entirely clear. For example, the results in Table 1 and Table 2 are worse than Guo et al 2017 that the paper claims to improve upon.\n\nSummary of contributions:\n\n  * Although the experimental results leave some major questions, a differentiable objective for calibration error is a valid contribution. Also the study of calibration as a meta-learning problem seems novel. \n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The most important concern is to address the ECE values in Table 1 and Table 2, see previous section.\n\n\n", "broader_impact_concerns": "The paper addresses the problem of model calibration, which would have a positive impact on sensitive applications of deep learning. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TBiCdUn6vX", "reviewer": "Reviewer_8KmU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n* Overall the paper is well written and easy to follow.\n* The authors propose DECE to soften the ECE metric in order to make a differentiable calibration objectives. Although, this method is similar to Soft-binned ECE (SB-ECE) [1], the paper provides detailed comparison to SB-ECE in Appendix C.\n* Calibration during training is particularly challenging as the overfitting is a significant challenge to solve this task (Figure 1). The paper proposes a meta-learning based method (i.e., meta-calibration) to solve this issue.\n\nWeakness\n\n* Lack of calibration results on large scale datasets such as ImageNet.\n* The meta-calibration is proposed to solve the overfitting problem of trainset ECE. However, the paper does not compare this method to previously proposed solutions: Weighting method from [2] and interleaved training from [1]. Especially, the comparison between interleaved training vs. meta-calibration seems crucial since both method requires a calibration set, which takes about 10% of the training set dedicated to minimizing the calibration objective loss. What benefit would meta-calibration have over interleaved training?\n\n[1] Karandikar et al., Soft Calibration Objectives for Neural Networks, 35th Conference on Neural Information Processing Systems (NeurIPS 2021).\n\n[2] Kumar et al., Trainable Calibration Measures For Neural Networks From Kernel Mean Embeddings, Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, PMLR 80, 2018.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors tackle two key challenges encountered during model training: 1) the non-differentiable nature of the calibration metric and 2) the overfitting of the calibration error to the training set.\n\nIn response to the first challenge, they devise a novel solution: DECE, a differentiable approximation to Expected Calibration Error (ECE). This approach facilitates direct calibration optimization during training.\n\nTo address the second challenge, the authors propose a meta-learning framework that uses DECE to optimize for calibration with a held-out calibration set. \n\nThey evaluate their methods and show that their approach yields well-calibrated, high-accuracy models that are competitive with existing methods on various benchmarks and architectures.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Comparison between interleaved training [1] and meta-calibration.\n* Showing ImageNet result or NLP classification tasks would strengthen the results.\n* Karandikar et al. [1] shows their results under data shift senarios (i.e., CIFAR-10C and CIFAR-100C). Does the proposed method still outperform other baselines under data shift senarios?\n\n[1] Karandikar et al., Soft Calibration Objectives for Neural Networks, 35th Conference on Neural Information Processing Systems (NeurIPS 2021).\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BQ4xrXlvHE", "reviewer": "Reviewer_4bkh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. Impressive calibration error on the studied data sets.\n2. The paper is clear and well-written.\n\n**Weaknesses**\n\nBeyond the issue of marginal novelty compared to previous work, here are concrete concerns that I have regarding the proposed method.\n\n1. *Comparisons with baseline methods.* In particular, the method by Karandikar et al. (2021) seems to be very relevant, as they also offered a differentiable soft-binned ECE loss to improve the calibration metric.  In the paper, the comparison between the methods is deferred to the appendix, which is okay. However, the authors did not include a comparison between the accuracy of the two methods. Thus it is not clear whether the method proposed in this work is superior both in terms of calibration and accuracy. The second method that is of interest to compare is the one by Müller et al. (2019(, which also combines meta-learning with label smoothing. This comparison is meaningful as label smoothing has been shown to improve the ECE calibration metric. \n\n2. *Accuracy of the classifier obtained by the proposed method compared to other techniques.* I find it disappointing that, for example, the accuracy of FLSD-53 with wide resnet is better than the proposed method while having a comparable ECE.\n\n3. This brings me to the third concern: it is *not clear how to handle the trade-off in accuracy vs. calibration*. I would love to see an experiment with *simulated* data where an oracle classifier is known – i.e., the classifier has access to the true conditional distribution of Y|X, from which the label is sampled. Importantly, the oracle in such an experiment is calibrated since we have access to the true conditional class probabilities. With this at hand, it will be important to see that, at the limit of large sample size, the proposed approach can achieve perfect calibration with prediction accuracy close to the oracle.\n\n4. The fourth concern is about the *generality of the results*. The experiments are conducted on 3 data sets, from which two (CIFAR10 and CIFAR100) are similar in nature. Since this paper is mostly empirical, it is most important to communicate to the reader that the proposed method is preferred on large-scale data sets. One such data set is ImageNet.\n\n5. Lastly, the training time of the proposed method is 3X to 4X longer than the training time of a standard CE model. Here, it would be essential to compare the training time with the other baseline methods to have a better perspective.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper offers a differential ECE objective to be minimized during training, on a disjoint subset of “validation” data. One interesting feature of this work is the use of meta-learning combined with label smoothing to improve the model's overall calibration. Overall, the classifiers fitted with this method enjoy impressive ECE although the accuracy is lower than that of standard classifiers trained via the cross entropy loss. \n\n*First contribution: Differentiable ECE*\n\nThe authors suggest using an all-pairs approach (Qin et al., 2010) to approximate the accuracy and soft binning (Yang et al., 2018) to approximate the overall ECE metric.\n\n*Second contribution: Meta-learning*\n\nAfter defining the meta-learning objective (both for L2 regularization on the classifier’s weights and label smoothing), the authors found that optimizing the hyper-parameters of label smoothing leads to better performance. This step's goal, inspired by previous work by Müller et al. (2019), is to learn one or more coefficients to smooth the one-hot encoded labels.\n\nNote that both contributions are only of marginal novelty, as these ideas were suggested in the previous work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness above.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers think the novelty is minor, but they commend the paper as of high clarity. The main concerns raised by them are:
1. Large-scale dataset experiments are missing;
2. Missing some comparisons with soft-binned ECE and interleaved training which are closely related to the proposed approach;
3. Worse ECE results compared with Guo et al. (2017).

In revision the authors added more experiments and baselines as suggested by point 2 above. The reviewers are satisfied for most of the additional results, however the 3rd point above still remain unaddressed.

The authors replied Concern 3 as "we followed the setting of Mukhoti et al. (2020) and used their code to build our codebase". While in different settings the ECE results for the calibration results may be different, the authors need to clarify in the paper on exactly how different the settings are. For example, if the network architecture and data augmentation procedure used in Guo et al. (2017) are exactly the same as what were used in this paper, then I personally don't think "using different learning rates" is a valid reason for not attaining Guo et al. (2017) performance, unless the authors have evidence to claim that Guo et al. results are not reproducible.

I suggest the authors to revise accordingly -- I think the acceptance should be conditioned on clarifying the above point.
Decision: Accept (poster)