Towards Unbiased Calibration using Meta-Regularization
Authors: Cheng Wang
Jacek Golebiowski
Venue: Accepted by TMLR ()

Abstract: Model miscalibration has been frequently identified in modern deep neural networks. Recent work aims to improve model calibration directly through a differentiable calibration proxy. However, the calibration produced is often biased due to the binning mechanism. In this work, we propose to learn better-calibrated models via meta-regularization, which has two components: (1) gamma network (gamma-net), a meta learner that outputs sample-wise gamma value (continuous variable) for Focal loss for regularizing the backbone network; (2) smooth expected calibration error (SECE), a Gaussian-kernel based, unbiased, and differentiable surrogate to ECE that enables the smooth optimization of gamma-net. We evaluate the effectiveness of the proposed approach in regularizing neural networks towards improved and unbiased calibration on three computer vision datasets. We empirically demonstrate that: (a) learning sample-wise $\gamma$ as continuous variables can effectively improve calibration; (b) SECE smoothly optimizes gamma-net towards unbiased and robust calibration with respect to the binning schemes; and (c) the combination of gamma-net and SECE achieves the best calibration performance across various calibration metrics while retaining very competitive predictive performance as compared to multiple recently proposed methods.

Published in Transactions on Machine Learning Research (06/2024)
Towards Unbiased Calibration using Meta-Regularization
Cheng Wang
cumgommamazon..cm
Amazon
Berlin, Germany
Jacek Golcbiowskit
jacekgo Damazon.ccom
Amazon
Berlin, Germany
Reviewed on OpenReview: https: openreviee.... net/ forum? id= Yf85HCCG4W
Abstract
Model miscalibration has been frequently identified in modern deep neural networks. Re- cent work aims to improve model calibration directly through differentiable calibration proxy. However, the calibration produced is often biased due to the binning mechanism. In this work, we propose to learn better-calibrated models via meta-reggularization, which has two components: (1) gamma network (7-Net), a meta learner that outputs sample-wise gamma values (continuous variable) for Focal loss for regularizing the backbone network; (2) smooth expected calibration error (SECE), Gaussian-kernel based, unbiased, and dif- Terentiable surrogate to ECE that enables the smooth optinization of y-Net. We evaluate the effectiveness of the proposed approach in regularizing neural networks towards improved and unbiased calibration on three computer vision datasets. We empirically demonstrate that: (a) learning sample-wise as contimuou variables can effectively improve calibration; (b) SECE smoothly optimizes 7-Net towards unbiased and robust calibration with respect to the binning schemes; and (c) the combination of 7-Net and SECE achieves the best calibration performance across various calibration metrics while retaining very competitive predictive performance as compared to multiple recently proposed methods.
1 Introduction
Deep Neural Networks (DNNs) have demonstrated promising predictive performance various domains, including computer vision (Krizhevsky al. 2012), speech recognition (Graves al.) 2013). and natural language processing (Vaswani et al.) 2017). Consequently, trained deep neural network models fre- quently deployed and utilized in real-world systems. However, recent work (Guo al. 2017) has pointed out that these highly accurate, negative log likelihood-trainedd deep neural networks are often poorly cali- brated (Niculescu-Mil Mizil & Caruana 20055)). Their predicted class probabilities do not accurately estimate the truc probability of correctness, resulting in primarily overconfident and under-confident predictions. Deploying such miscalibrated models in real-world systems poses significant risk, particularly when model outputs are directly utilized serve customers' requests in applications such as medical diagnosis (Caruana et al. 2015) and autonomous driving (Bojarski et al.) 20166). Better-calibrated model probabilities can serve essential signal toward building more reliable machine learning systems.
A recent trend aims to learn calibrated model by training it to minimize errors on calibration metrics. One representative work is from Kumar et al. (001)), where they developed a differentiable equivalent of Expected Calibration Error (ECE), the Maximum Mean Calibration Error (MMCE). Mukhoti et al. (2020) found that focal loss [in et al. 2017). as a good alternative to standard cross-entie cross-entropy, can effectively improve calibra- tion. Thoy also highlighted the crucial role of the gamma paramcter in focal loss in making this approach
EEqual contribution.

---

Published in Transactions on Machine Learning Research (06/2024)
gamma network
validation Set (outer loop)
SECE
learned sample-wise gamma
training Set (inner loop)
Focal
backbone network
representation classifier
Figure 1: Our proposed approach for regularizing the base network towards better calibration includes two new components: y-Net and SECE The inner loop optimizes the backbone network (e.g., ResNet), which uses focal loss as objective function. The >-Net in the outer loop takes the extracted sccond-to-last layer representation of backbone network input and learns to outpul sample-wise for focal loss in a continuous space. The 7-Net is optimized by using the proposed SECF, Gaussian kernel-based, unbiased, and differentiable calibration error.
effective and proposed a sample-dependent schedule based on heuristics (Lambert-W W function (Corless et al. 1996)) for the gamma in focal loss (FLSD), which showed superior calibration performance compared basc- approach termed meta-calibration.
lines. [Boddal al. (2021 2023) proposed using a meta-learning- In their formulation, the backbone network learns to optimize standard cross-entropy loss, while differ- entiable proxy for calibration error (i.c., DECE) is used to tune the parameters of the weight regularizer.
Motivated by advancements in this area, we introduce a learnable approach that involves learning sample- wise y values via a meta-network, referred to as the gamma network (i.e., 7-Net), as illustrated in Figure [ The key distinctions from Focal loss (Lin et al. 2017) and FLSD (Mukhoti al. 2020), where global 7 parameter (e-g., setting = or 7 = 5) is employed approach (1) learns more finely-grained gamma values at the sample level; (2) the
fixed parameter scheduled across training
epochs, twofold: learned yy values are continuous variables. The output of the 7-Net, namcly the Iearned values, is then utilized to regularize a backbone network towards improved calibration. The optimization of the >-Net is achieved with a proposed differentiable surrogate for expected calibration error (ECE) (Naeini et al., 2015): Smooth ECE (SECE), an unbiased Gaussian kernel based ECE estimator. SECE optimizes 7-Net towards unbiased calibration by avoiding the binning step as in (Zhang al. 2020).
Our contributions be summarized follows:
We propose propose--Nee, Net, a meta-network designed learn sample-wise y values for Focal loss as continuous variables, rather than relying on a single pre-defined y value. To the best of our knowledge, our proposed method learns the most fine-grained 7 values for Focal loss.
We propose a beseel ECE SECCE which is a variation the kernel density estimate (KDE) approximation (Zhang al.) 2020 This estimator smoothly regularizes >-Net towards improved and unbiased calibration.
Through extensive experiments and analysis, we empirically demonstrate that 7-Net effectively calibrates models, while SECE provides stable and smooth calibration. The combination of both achieves competitive predictive performance and superior scores across multiple calibration metrics compared to the baselines.
2

---

Published in Transactions on Machine Learning Research (06/2024)
2
Related Work
Calibration of machine learning (ML) models using Platt scaling (Platt et al. 1999) and Isotonic Regres- sion (Zadrozny & Elkan 2002) has shown significant improvements for SVMs and decision trees. With the advent of neural networks, Niculescu-Mizil & Caruana (2005a) showed that those methods can produce well-calibrated probabilities even without any dedicated modifications.
Recently, Guo et al. (2017) and Mukhoti et al. (2020) have shown that modern NNs are noticeably more accurate but poorly calibrated due to negative log likelihood (NLL) overfitting. MMnneerer et al. (2021) revisited this problem and found that architectures with large sizes have a larger effect on calibration, but nevertheless, more accurate models tend to produce less calibrated predictions.
One of the ways to tackle calibration is to rely on post-hoc corrections using nonpparametric approaches such as histogram binning (Zadrozny& & Elkan 2001), isotonic regression (Zadrozny & Elkan methods such as Bayesian binning into quantiles (BBQ) and Platt scaling (Platt et al., 1999). Beyond those four, temperature scaling (TS) a single-parameter extension of Platt scaling (Platt et al.
2002) and parametric
1999) and the
most recent addition to the offering of post-hoc methods. Recent work has shown that a single paramcter
TS leads to good calibration performance with minimal added computational complexity (Guo et al. 2017;
Minderer et al. 2021). There have been multiple extensions to temperature scaling in recent years. Kull et al. [2019) proposed Dirichlet calibration which assumes probability distributions are parameterized Dirichlet distributions, and an attention-based mechanism was proposed to tackle noise in validation data (Mozafari
et al. 2018). Laves et al. (2019) extended TS to dropout variational inference to calibrate model uncertainly by inserting temperature value in Monte Carlo integration.
Beyond post-hoe methods, other approaches do not explicitly include the calibration objective but rather im- plicitly guide the training towards better calibration performance. There are basically two lines of work: (1) data augmentation-basse based methods and (2) regularization based methods. For the former line, the represen- tative methods include label smoothing (LS) Muüleer et al., 2019) that mitigates miscalibration by softening hard labels with an introduced smoothing modifier in the standard loss function (e.g., cross-entropy). Mix-up training Thulasidasan et al.) 2019) that extends mixup (Zhang et al., 2018) to generate synthetic samples during model training by combining two random elements from the dataset.
Extending the portfolio of eegularization-bases methods for calibration, focal loss [Lin et al. 2017 has been used as maximum entropy regularizer. One of its extensions, FLSD (Mukhoti et al.) 2020 computes the gammma value for focal loss based on heuristics (Lambert-W function (Corless et al. 1996 ), dual Focal (Tao et al. 2023) takes into account the logit corresponding to the ground truth label and the largest logit ranked after it. AdaFocal (Ghosh et al.) 2022) utilizes the calibration properties of focal (and inverse- focal) loss and adaptively modifics gamma for different groups of samples based on previons gamma value the knowledge of model's ' under/ove coeeccooicceeee on the validation set. Most recently, Krishnan & Tickoo (2020) introduced a differentiable accuracy versus uncertainty calibration (AvUC) loss function that allows model to learn to provide well-calibrated uncortainties. Liu et al. (2022) analyzed some current calibration methods such as label smoothing (Müller et al. 2019). focal loss Lin et al. 2017), ; and explicit confidence penalty (Pereyra et al. 2017) from a wwwttraanee-opiinizztiiiee perspective and pointed out those can be seen as approximations of a liner penalty. The authors proposed to add a margin into learning objectives. Cheng & Vasconcelos (2022) proposed calibration by pairwise constraints (CPC), which aims to strengthen the calibration supervision by casting multiclass searino into pairwise binary calibration. Thus authors proposed binary discrimination constraints (BDC) loss and binary exclusion constraint (BEC) loss as addition to standard cross-entropy loss increase the supervision rate for calibration of the training process.
Within regularization--asee methods, some approaches use differentiable calibration proxies regularizers. For instance, Kumar et al. (2018) developed differentiable equivalent of ECE, the Maximum Mcan Cali- bration Error (MMCE), which can be oplimized directly to train calibrated models. Similarly Karandikar et al. (021)) proposed a softened version of ECE (SB-ECE) by employing a softmax-basee bin- nitt-mmmbeship function with temperature, cnabling the soft ECE to serve as either a primary or auxiliary loss objective. Another interesting exploration is presented in Bohdal et al. (2021), where instead of utilizing devel- oped differentiable ECE (DECE) for optimizing the main network, the authors proposed meta-learning
3

---

Published in Transactions on Machine Learning Research (06/2024)
approach (Luketina al. 2015 to train model that minimizes DECE, thereby finding the optimal param- eters of the 1.2 regularizers.
Compared to the aforementioned regularization methods, to the best of our knowledge, our proposed method learns the most fine-grained gamma values for focal loss via a meta-network. This network is optimized toward stable and unbiased calibration, mitigating the biases caused by the binning mechanism.
3 Preliminaries
3.1
Model Calibration
Calibration (Guo al. 2017 measures and verifies how well the prodicted probability estimates the true likelihood of correctness. Assune a model M trained with datasel {x, v), X, V. Let P (2..P2.....PP be the predicted softmax probability of K classes. If M makes 100 independent predictions, each with confidence arg max 0.9, calibrated M is correct 90 times. Formally, accuracy(M(D)) = confidence(M(D) if M is perfectly calibrated on dataset D.
Reliability Diagrams (DeGroot & Fienberg 1983 Niculescu-Mizil Caruana 2005a visualize whether a model overconfident or underconfident by grouping predictions into bins according to their prediction probability. The predictions are grouped into M bins bo..b....bba b..). The accuracy of samples in each bin bm is computed as: ACC(b..) [bml Eiet... ( y4), where indexes all examples that fall into bin bms 0: are Yi are the prediction and ground-trute truth, respectively. Let P be the probability for the predicted class yi for the i-th sample (i.e. the class with highest predicted probability for i-th sample), then the average confidennc is defined as: CONF(bm) |h.. Eieb.. P-- A model is perfectly calibrated if ACC(bm) coNF(bm). ), Vm and in diagram the bins would follow the identity function. Any deviation from this indicates miscalibration.
Expected Calibration Error (ECE) NNacini et al. 2015 ECE computes the difference between model accuracy and confidence. It quantifies the discrepancy between predicted probabilities and observed frequen- cies by partitioning data points into bins of similar confidence. It takes the form
M
Ibm
ECE
|CCC(b)) - conF(bm)) N
(1)
where N is the total number of samples and bn represents a single bin.
Maximum Calibration Error (MCE) (Nacini et al. 2015 is particularly important in high-risk appli- cations where reliable confidence measures are absolutely necessary. It measures the worst-case deviation
between accuracy and confidence, MCE ma.meel...... (ACc(b..) - CoNF(b,.)l. For a perfectly calibrated model, the ideal ECE and MCE are equal to O.
Besides ECE and MCE, we also report Classwise ECE NNxon et al. 2019 Adaptive ECE Nixon et al. 2019), we describe these metrics in Appendix A.1
3.2
Focal Loss
For a classification task, the focal loss (FL) Lin et al. 2017) can be defined as Ct = - pi,,, )7 log Pa, where 7 is a hyper-parameter. It was originally proposed handle imbalanced data distributions but Mukhoti et al. (2020) found that the models trained with focal loss are better calibrated with respect to cross-entropy trained counterparts. This is because focal loss be interpreted as trade-off between minimizing Kullback Leibler (KL) divergence and maximizing the entropy of predictions, depending on MMukhoti et al. 2020
C > KL(g I) - Hl(g) - (pH((p)
(2)
where p is the prediction distribution, q is the one-hot encoded target class, and H(g) is constant. Models trained with this loss learn to strike balance between narrowing p (high confidence in predictions) due to
More theoretical findings can be found in the paper.
4

---

Published in Transactions on Machine Learning Research (06/2024)
the KL term and boarding p (avoiding overconfidence) due to the entropy regularization term. The authors provided principled approach to select the for focal loss based on the Lambert-W function (Corless et al. 1996). Motivated by this, our work proposes to learn more fine-grained, sample-wise 7 with a meta-network.
3.3 Meta- Learnne
Model calibration can be formulated as minimizing a mulli-component loss objective where the base term is used to optimize predictive performance while a regularizer maintains model calibration (Bohdal et al. 2021 Lin et al. 2017). Tuning the hyper-parameters of the regularizer can be a difficult process when conventional methods are used (Li ct al., 2016 Snock et al. 2012; Falkner et al., 2018). In this work, we adopt the meta learning approach (Luketina el al., 2015). At each training iteration, the model takes a mini-batch from the training dataset Dirain and the validation dataset Deat to optimize
arg min _(0, 0, Dtroin, Duai)= = CFL, (0, Derain) + LSECE(Ó, Deat) 0,0
(3)
First, the base loss function LFL,, is used to optimize the parameter of the backbone model O on Dtrain: note that the y parameter for this step is predicted by 3-Net. Following that, the validation mini-batch Deat is used to optimize the parameters of the meta-network (--Net) & using a validation loss LSECE- The validation loss is a function of the backbone model output and does not depend on g-Net directly. The dependence between the validation loss and parameters of the meta-network is mediated via the parameters of the backbone model as discussed by (Luketina et al. 2015
4 Methods
This section introduces the two components of our approach: the y-Net learns to parameterise the focal loss and the SECE provides differentiability to 7-Net towards calibration optimization.
4.1 7 Nett Learning Sample-Wise Gamma for Focal Loss
Sampleedependent y value for focal loss showed it effectiveness of calibrating deep neural networks (Mukhoti
ct al.
In this work, instead of computing and scheduling y value based on Lambert-W W function (Corless et al. 1996). We propose learnable approach which learns more fine-grained and local y values, i.e., each sample has an individual 7: in a continuous space. Formally, 7-Net takes the representations from the second-to-last layer of a backbone network (i.e. ResNet (He et al. 2016). Let x E Rgbdd (b: batch size, d: hidden dimension) be the extracted representation, A E Raxi be a k-head self-attention matrix that followed by a liner layer with parameters W E Rdx1, The 7-Net transforms the representation to sample-wise ?
2020
x A, E &RAxk p SOFTMAX(a), E R
= p AT, E gbdd 7= |X W//T, E 1xx
(5)
Here we use |-| ensure the y positive valued and tune the temperature 0.01 as hyperparameter. This
is similar to the temperature setups as in meta-calibration (Bohdal et al. 2021 2023
Those operations
result in a set of sample-wise y to be used in focal loss C/. Note that, (x xCa
E D is learned a
continous variable rather than discrete value as in original FL formulation Lin et al. 2017 and scheduled
FL (FLSD) (Mukhoti et al. 2020).
Practical Considerations: To ensure 7 2 0, we could apply operations based on min-max scaling or activation functions such as sigmoid, ReLu(Rectified Linear Unit) Agarap (0018) and softplus. However, our experimental evidence shows that the formulation presented in Equation 5 performs better across datasets.
4.2 SECE Smooth Expected Calibration Error
Conventional calibration measurement via Expected Calibration Error (ECE) discussed in Section 3.1 is computed over discrete bins by finding the accuracy and confidence of examples in each interval. However,
5

---

Published in Transactions on Machine Learning Research (06/2024)
this approach is highly dependent different settings of bin edges and the number of bins, making it biased estimator of the true value (Minderer et al. 2021
The issue of bin-based ECE can be traced back to the discrete decision of binning of samples Minderer et al, 2021]. The larger bin numbers, the less information is retained. Small bins lead to inaccurate measurements of accuracy. For instance, in single-example bins, the confidence is well defined but the accuracy that interval of model confidences is more difficult to assess accurately from a single point. Using the binary accuracy of single example leads to the Brier score (Brier al. 1950) which has been shown to not be perfect measure of calibration.
To find a good representation of accuracy within the single-example bin (representing a small confidence interval), we leverage the accuracy of other points in the vicinity of the single chosen example weighted by their distance in the confidence space. Formally, the soft estimation accuracy within a single-example bin:
SACC (b,) Σ (()s))((zs+ 2z)
(6)
K (x),2') = exp (-1 r3 2'y 11
(7)
2h
where is the bin housing example xa and K(:-)) is a chosen distance measure, for example a Gaussian kernel and h is the bandwidth. Moreover, 24 and T(;; respectively denote the confidence and accuracy of the th example. As in meta-calibration (Bohdal et al. 2021), we use the all-pairs approach (Qin et al. 2010 to ensure differentiability. Having good measures of soft accuracy and confidence for each single-example bin, we can write the updated ECF metric soft-FECE. in the form
M M Σ
SECE
JSACC (i) - CONF(i)),
(8)
where represents single example and M the number of examples. The new formulation is (1) differen- tiable as long as the kernel we use is differentiable and (2) enables smooth control over how accuracy over confidence bin is computed via tuning the kernel parameters. Regarding (2), choosing a Dirac-delta function recovers the original Brier score while choosing broader kernel enables a smooth approximation of accuracy over all confidence values.
Connection to KDE-based ECE Estimator ZZang et al. 2020) presents a KDE-based ECE estimator that relies on kernel density estimation to approximate the desired metric. Canonically, ECE is computed integral over the confidence space:
EỘE² - -f ||2 [1a= [==)(()]l
where 2 = {21, 22, }1} denotes model confidence distribution over L classes, || |1 denotes the tth power of the la norm, and p(2) represents the marginal density function of model's confidence #(2) on given dataset. The p(z) and #(2) are approximated using kernel density estimation. We argue that SECE is a special instance with d = of EČE" d and estimate ECE with max probability 2,, = arg max 22, # for single instance. And SECE is an upper bound of EČE":
EĆE-
|z - ==)))
(10)
121 *(z))) / |1x #(xe)) ((
(11)
$ / 12c +(())) = SECF
(12)

---

Published in Transactions on Machine Learning Research (06/2024)
with 1 [|, # and density functions:
h L M
Ee=1 ((i) (2))2) Eis K (Zz, zi)
d(z))=
K (x,,)) tt((z)
(13)
M
where h the kernel width and (tt) represents the binary accuracy of point i. The p(2() is a mixture of Dirac deltas centered on confidence predicted for individual points within the dataset (used to approximate ECE). Replacing binary accuracy with accuracy computed using an all-pairs approach (Qin et al., 2010) results in SECE, which is differentiable and can be computed efficiently for smaller batches of data since the integral is replaced with sum over all examples in batch.
There exist other studies that develop differentiable calibration error metrics, for instance, DECE (Boldal et al. 2021) and SB-ECE (Karandikar et al. 2021). As pointed out in (Zhang et al.) 2020): non non-paraettric density estimators are continuous, thus avoiding the bimning step and potentially reduce binning biases. The difference between SECE and KDF-based estimators do not invalidate the analysis of the EČE as an unbiased estimator of ECE (Theorem 4.1 in (Zhang al.) (020)) as the all-pairs accuracy an unbiased estimator of accuracy and the details of the p(z) distribution are not used in the derivation beyond setting the bounds. As result, the analysis described in (Zhang et al.) 2020) to show their KDE-based approximation to ECE is unbiased can be re-used to show the same property for SECE.
Discussion:
4.3 Optimising 7-Net with SECE
The newly introduced 7-Net and the SECE metric can be applied together to optimize selected ML models for both cross-entropy and calibration error using the meta learning scheme introduced in Section 3.3 As the probability calibration via maximising entropy is performed in a continuous space, we argue that SECE is an efficient learning objective to optimize >-Net and provide calibration regularization to base network trained with focal loss.
Under the umbrella of meta-learning, two sets of parameters are learned simultaneously: the parameters of base network f", well the é, the parameters of ~-Net denoted as ft, The former, fo, is used to classify each new example into appropriate classes and the latter, f7, is applied to find the optimal value of the focal loss parameter for each training example. Algorithm 1 in Appendix describes the learning procedures.
5 Experiments
We conducted series of experiments to achieve the following objectives:
Examine both the predictive and calibration performance of the proposed method. We used error (accuracy) to measure the predictive performance and Negative Log-Likelihood (NLL), Expected Calibration Error (ECE), Maximum Calibration Error (MCE), Adaptive Calibration Error (ACE), and Classwisc ECE for calibration performance.
Observe the calibration behaviours of the proposed method during training.
Empirically evaluate the learned values for Focal loss and the robustness of different methods to the binning mechanism.
Implementation Details. We implemented our methods by adapting and extending the code from Bo- hdal et al. 2021) with Pytorch (Paszke et al. 2019). For all experiments we used their default sottings (using 18 as base model, batch size 128, data augmented with random crop and horizontal flip) unless otherwise stated. Each experiment was run times with different random seeds, and results were averagod. We conducted our experiments on CIFAR-10 and CIFAR-100 (in (Bohdal et al. 2021)) ) as well Tiny- Imageem (Ya Le. 2015).. For meta-learning, we split the training set into 8:1:1 as training/va validation, keeping the original test sets untouched. The experimental pipeline for all three datasets

---

Published in Transactions on Machine Learning Research (06/2024)
identical. The models were trained with SGD (learning rate 0.1, momentum 0.9, weight decay 0.0005) for up to 350 epochs. The learning rate was decreased at 150 and 250 epochs by a factor of 10. The model selection was based on the best validation error.
The 7-Net is implemented with a multi-head attention layer with k heads and a fully-connected layer. k is set to the number of categories. The hidden dimension is set to 512, the temperature SECE, we used the Gaussian kernel with bandwidth of 0.01 (selected via grid search) for both datasets. We initialized y 1.0. During inference, the meta-network is not present excepted except in our ablation study learned values in Section 5.2
fixed at 0.01. For
Baselines, We extensively compare our method with baselines including standard cross-entropy (CE), cross-entropy with post-hoc temperature scaling (TS) (Platt et al., 1999). Focal loss with standard garnma value (Focal), y = 1 {Lin et al.) 2017), Focal Loss with scheduled gamma (FLSD) (Mukhoti et al. 2020). MMCE (Maximum Mean Calibration Error) (Kumar et al. 2018) and Label Smoothing with smooth factor 0.05 (LS-0.05) or (LS-0.1) (Mäleer et al. 2019) and Mix-Up (a 1.0) (Thulasidasan et al.) 2019). In mela-
learning setting, we include CE-DECE qnueta-calibrationnBa (Bohdal et al. 2021 for learning unit-wise weight regularization), which serves as our meta-learning baseline, CE-SECE, F -EEE,, focal loss with learnable sample-wise and FD,-DECE.
Table 1: The predictive (test error) and calibration performance of different methods on CIFAR-10 (Top), CIFAR-100 (Middle) and Tiny-ImageNet (Bottom). The best scores are bold. The mcan and standard de- viation numbers are reported by averaging 5 runs with random seeds. As an alternative calibration method, our approach generally exhibits better calibration while retaining competitive predictive performance com- pared to conventional as well as meta-learning baselines.
Methods
Error
NLL
ECE
MCE
ACE
Classwise ECE
CIFAR 10
CE
4.812 = 0.122
0.335 + 0.01
4,056 + 0.092
33.932
5.433
4.022 + 0.136
0.848 + 0..23
CE (TS)
4.812 + 0.122
0.211 + 0.005
3.083 + 0.140
26.693 = 2.959
3.046 + 0.157
0.656 + 0.022
Focal
4.874 + 0.100
0.207 + 0.005
3.193 + 0.104
28.034 = 5.702
3.174 + 0.098
0.690 + 0.018
FLSD
4.916 1 0.074
0.211 1 0.005
6.904 1 0.462
19.246 11.071
6.805 1 0.446
1.465 1 0.088
LS (0.05)
4.744 0.126
0.232 + 0.003
2,900 = 0.085
24.860 8.599
3.985 = 0.154
0.727 = 0.009
LS(O.1)
4.918 0.085
0.266 + 0.004
7.566 + 0.41
16.033 3.783
7.611 = 0.161
1.637 + 0.056
Mixup(a=1.0)
4.126 0.068
0.273 + 0.033
12.863 3.2
20.739 4.205
12.833 3.161
2.678 + 0.615
MMCE
4.808 = 0.082
0.333 + 0.012
4.027 + 0.082
41.647 = 10.275
4.013 = 0.091
0.845 + 0.014
CE-DECE
5.194 = 0.161
0.301 + 0.038
4.106 = 0.402
41.346 13.325
4.088 = 0.395
0.868 = 0.074
CE-SECE
5.222 = 0.168
0.289 + 0.027
4.062 = 0.241
50.81 + 21.705
4.049 = 0.251
0.852 = 0.040
DECE
5.434 = 0.095
0.193 + 0.009
2.257 + 0.787
56.633 = 23.856
2.396 = 0.669
0.557 + 0.165
PL,-SECE
5.428 1 0.144
0.193 1 0.010
2.138 1 0.819
22.725 - 5.756
2.357 1 0.541
0.556 1 0.165
CIFAR- 100
CE
22.570 + 0.438
0.997 + 0.014
8.380 + 0.336
23.250 = 2.436
8.347 + 0.344
0.233 + 0.006
CE (TS)
22,570 + 0.438
0.959 + 0.008
5,388 + 0.393
13.454 = 2.377
5.360 = 0.316
0.208 - 0.003
Focal
22.498 + 0.214
0.900 = 0.007
5.044 + 0.203
12.454 0.893
5.015 = 0.207
0.203 - 0.004
FLSD
22.656 + 0.113
0.876 = 0.005
5.956 + 0.804
14.716 1.387
5.958 = 0.802
0.241 = 0.008
LS (0.05)
21.810 T 0.172
1,070 T 0.011
8.108 1 0.346
20.268 1.536
8.106 1 0.346
0.272 T 0.006
LS(0.1)
22.244 0.155
1.052 1 0.011
4.754 1 0.709
17.228
0.923
4.777 1 0.647
0.239 1 0.004
Mixup(a=1.0)
21.210 + 0.227
0.917 + 0.017
9.716 + 0.754
16.01 + 1.335
9.722 = 0.740
0.315 1 0.011
MMCE
22.490 0.143
1.021 = 0.007
8.713 = 0.245
23.565 1.141
8,670 = 0.305
0.238 - 0.004
CE-DECE
23.406 0.323
1.148 1 0.006
7.309 1 0.245
22.565 = 1.446
7.253 1 0.315
0.241 1 0.002
CE-SECE
23.448 0.302
1.153 T 0.015
7.668 1 0.330
24.261 1.614
7,609 1 0.295
0.244 1 0.002
FL,-DECE
23.712 0.204
0.888 1 0.009
1.879 + 0.440
8,271 1 2,651
1.838 0.371
0.195 1 0.005
FL,,-SECE
23.686 + 0.377
0.877 + 0.004
1.940 + 0.365
7.480 + 1.867
1.939 + 0.379
0.192 + 0.006
Tiny-ImageNett
CE
40.110 + 0.110
1.838 + 0.171
8.059 + 1.296
15.73 + 1.905
8.006 + 1.282
0.154 + 0.001
Focal
39.415 1 0.625
1.896 1 0.009
7.600 1 0.309
13.771 - 0.897
7.469 1 0.301
0.152 1 0.002
FLSD
39.705 1 0.075
1,904 1 0.025
14.501 1 1.078
21.528 2.116
14.501 1 1.078
0.202 1 0.006
LS (0.1)
39.395 1 0.305
2.185 1 0.001
16.777 1 0.476
29.088 1 1.835
16.901 1 0.460
0.199 1 0.001
Mixup(a=1.0)
39.890 + 0.271
1.932 1 0.054
12.133 + 2.069
31.440 = 0.968
12.028 + 2.079
0.193 1 0.009
MMCF
40.310 + 0.100
1.826 + 0.177
8.206 + 1.219
16.802 2.339
8.165 + 1.269
0.149 - 0.001
CE-DECE
41.350 + 0.000
2.228 + 0.033
10.694 + 0.503
20.888
0.430
10.553 + 0.553
0.160 1 0.000
CE-SECE
41.005 1 0.145
2.213 1 0.058
10.928 1 1.125
21.362
2.526
10.912 1.069
0.157 1 0.003
FL,C
40.625 0.095
1.826 = 0.007
5.944 + 1.090
11.542
1.990
6.077 = 1.095
0.155 - 0.007
-LCEE
40.850 + 0.140
1.829 + 0.005
5.794 + 0.756
11.477 1.563
5.848 = 0.751
0.156 + 0.005
8

---

Published in Transactions on Machine Learning Research (06/2024)
5.1 Predictive and Calibration Performance
Table presents the performance comparison across approaches, Temperature Scaling (TS) can effectively roduce the errors calibration metrics compared to uncalibrated CE models (baseline). Label smoothing and Mixup achieve the best test errors on datasets but exhibit. higher ECE and MCE scores. Focal loss and FLSD can improve calibration in general, but we also observed high MCE score for FLSD. On the other hand, we found that MMCE exhibits higher calibration errors as compared to baseline, this aligns with the findings from (Bohdal et al., 2021). We further discussed the predictive-calibratio trade-off in Appendix A.4 Our proposed approach FL,-SECE, achieves lower errors on most of calibration metrics with competitive predictive performance. Compared to the prior work (CE-DECE), our method (FL,-SECE) showed significant calibration improvement across multiple calibration metrics. Particularly, on MCE (which measures the worst-case mismatch between accuracy and confidence), there are 18.62% 15.09% and 9.41% improvements on CIFAR-10, CIFAR-100 and Tiny-ImageNet, respectively.
1.
1.0
CCE 8.2
5.07%
CCE
5.22%
5A1%
022 KCE 24.27%
0.3-
32.30%
0.5
32.03%
0.5
22.53%
Accurac
0.2
00
0.2
880
03
0.2
600
0.5
060
Confidence
Curfidence
Curfiidence
Conficence
(a) CE (22.41%)
(b) CE (TS) (22.41%)
(c) Focal (22.53%)
(d) F2SDD22.72%%
(e) LS-0.05 (21.56%)
1.0
1.
1.
1.0
10
ECE 3,00%
7.35%
7.84%
2.38%
0.8
0.8
6.9
0.8
2.5
Accunar
Ra
00.
880
800
6:80
060
Confidence
Condidence
Conficence
Conditence
Conficence
(f) MMCE (22.30%)
(g) CE-DECE(23.05%) (h) (L,-DECC(2110015)) (i) CE-SECE(23.58%)
(i) F S2.EEE3333377
Figure The reliability diagram plot for models on CIFAR-100 test set. The (-) represents test error. The diagonal dash line represents perfect calibration. The red bar represents the gap between the observed accuracy and the desired accuracy of the perfectly calibrated model (the diagonal) it is positive if the observed accuracy is lower and negative otherwise. The model from the 5th run is used.
Figure 2 illsstrates the reliability diagram plots of models on CIFAR-100. (The plots for CIFAR-10 are Appendix Figure 7
The diagrams show that for CIFAR-100, >-Net based methods achieve better ECE compared to SECE and meta-loss for optimizing 7-Net ensures smooth and stable calibrations, potentially reduces calibration biases in bins. Among compared methods, our method FL,-SECE achieves considerably lower errors on calibration metrics. When comparing meta-learning based approaches: CE-DECE (meta-learning baseline), CE-SECE, FECF- FE.,. We can see that our method (FT.,SFFCFF) achieves comparable test error and better scores across calibration metrics. Particularly, it improves ECE by an average of roughly 4.198% on three datasets, additionally improving MCE by an average of 14.37% MCE and ACE score by 3.917%.
To observe the learning behavior of models, we plotted the curves showing test ECE changes in Figure 3 (a-b), and test error (included in Appendix Section 1. It is noteworthy that while the test error curves exhibit similar behaviors, the calibration behaviors vary significantly across methods. The primary advantage of N-Net with SECE lies in its stable and smooth calibration behavior. After the t50th epoch, when the learning rate is decreased by factor of 10, we observe an increase in ECE scores for all approaches, particularly for FLSD (green lines). This indicates that during model training, the learning procedure aims to align the probability distribution as closely as possible with the ground-truth distribution (one-hot representation).
9

---

Published in Transactions on Machine Learning Research (06/2024)
0.14
LSI0.051
0.12
FL
MMGE
FLED FL-YSECE
FL
MMCE
F FLaySECE
0.15
CE-DFCE
0.10
Test ECE
0.08
Test
0.10
0.06
0.04
0.05
0.02
0.00
0.00
160
200
300
100
200
300
Epoch
Epoch
CIFAR-10
(b) CIFAR-100
4.0
2.05
3.5
2.00
1.95
3.0
1.90
2.5
1.85
2.0
1.80
1.5
1.75
mean
std.
1.0
meen
std.
0
50
100
150 200 Epoch
250
300
350
50
100
150 200 Epoch
250
300 350
(c) CIFAR-10
(d) CIFAR-100
Figure 3: (a-b): ECE curves on the test dataset of CIFAR-10 (a) and CIFAR-100 (b)-(c-d): The mean and standard deviation (std.) of y on test dataset at each epoch. Low std. score indicates samples share similar gamma values, and high std. score indicates more samples have different y values.
causing models to become overconfident. It also demonstrates that our approach is more robust to this phenomenon.
5.2
Ablation Study
5.2.1
Learning ? continous variables
Figure 3 (--)) presents the changes in y values on test dataset over epochs. In the earlier stages of training, the sample-wise "i for Ty E Drest have similar values (with low variance) because they initialized with N 1.0 for Duai- The y parameter is observed to have higher standard deviation in the later stage of training as 7-Net Icarns the optimal value for each example in the datasct rather than relying on global values, showcasing the flexibility of the network. It is also noted that is learned in a continuous space, which is different to the discrete values the pre-defined in Focal Loss Lin et al. 2017) and FL.SD (Mukhoti
et al. 2020
5.2.2 Calibration bias and robustness
In Figure 5 we examine the robustness of those methods with different binning schemes by varying the number of bins, which is one causes of introducing calibration bias (Minderer et al. 2021). shows that y-Net based approaches (FL,,DEEE and FL,,-EEE)) maintain much lower ECE score when throughout all bin numbers from 10 1000 showing the trained based network is robustly calibrated, Furthermore FL,,-SECE is also able to maintain lower MCE compared to other methods.
10

---

Published in Transactions on Machine Learning Research (06/2024)
1.0
1.0
1.0
ECE
7 7.09%
ECE
7.86%
ECE # 2.39%
ECE 1.711%
0.8
MCF
21.31%
0.8
MCE
22.42%
0.8
MCE -.4%%
0.8
NCE -7.6%
Accurac
0.6
Accuracy
0.6
Accuracy
0.6
0.2
0.2
0.2
0.2-
9.0
0.4 6B
0
0.4 0.6 Confidence
0.8
0000
0.2
0.4
00
0.4 CA Confidence
Confidence
(a) CE-DECE
(b) CE-SECE
(c) FLa-DECE
(d) FL,-SECE
1.0
1.0
ECE
ECE 7.96%
2.6.%
MCE 28.1156
MCE 27.25%
0.8-
0.8-
10.05%
Accuracy
0.6
Accuracy
3.6
Accurary
0.6
Accuracy
0.6-
0.4
0.4
0.4
9.0
022
0.4 0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
6.0
0.2
0.4
0.6
0.8
1.0
0.2 0.4 0.6
0.8
Confidence
Confid dence
Confidence
Confidence
(e) CE-DECE
(f) CE-SECE
(g) FL,E-ECEE
(h) FL, -SCE
1.0
1.0
1.0
1.0
ECE 8.20%
ECE 8.06%
ECE 3.20%
ECE 5.04%
0.8
MCE 34.58%
MCE 33.13%
0.8
NCE 43.45%
0.8
MCE 41.23%
Accuraty
Accuracy
Accurary
0.6
Aeguracy
0.6
0.4
0.4
0.4
0.4
0.2
02
0.2
0.2
0.00
0.8
08.0
1.
8
Confid
dence
Confidence
Confidence
(i)
CE-DECE
(j) CE-SECE
(k) FL,-DECE
(1) FL,-SECE
Figure& 4: The reliability diagram plots on CIFAR-100 with large bin numbers (top to bottom: 20, 50, 100).
100
80
ECE
Test MCE
5-
yORE
40
3.
(S0 (b)
NMCC
10'
10a Bin numbers
10 101
10a
10
Bin numbers
Figure 5: The changes in ECE (left) and MCE (right) scores on the CIFAR-10 test dataset with increasing bin numbers in the range of [10, 20, 50, 100, 200, 500, 1000) are illustrated. FL,,-SECE demonstrates superior robustness to increasing bin numbers, as evidenced by lower MCE. A similar plot for CIFAR- 100 is provided in the Figure g of the Appendix.
Here we can elcarly see the role of SECE in stabilizing the calibration with y-Net against the change in the number of bins. Table]showw shows the individual calibration gain from SECE as compared to using learnable 7 only with our method F FL,.SECE, With SECE, we can see improved predictive and calibration performance. Importantly, from Figure g and F we observe that it reduces calibration bias introduced by different binning mechanisms. This is mostly reflected in reduced ECE/MCE scores, i.c., the model exhibits high MCE on a particular bin if calibration bias exists.
11

---

Published in Transactions on Machine Learning Research (06/2024)
Methods
Error
NLL
ECE
MCE
ACE
Classwise ECE
Fl,,
5.632 = 0.118
0.197 + 0.009
2.177
0.619
46.172 28.240
2.319 + 0.407
0.553 0.120
SECE
5.428 i 0.144
0.193 0.010
2.138
0.819
22.725
5.756
2.357 + 0.541
0.556
0.165
FL,
28.148 8.127
1.051 = 0.278
3.044
1.542
10.082
3.441
3.016 = 1.511
0.226
0.063
SL,-SECE
23.686 1 0.377
0.877 1 0.004
1.940
0.365
7.480 1 1.867
1.939 1 0.379
0.192
0.006
Table 2: The calibration gain from SECE on CIFAR-10 (lop) and CIFAR-100 (bottom).
6 Discussion
Though we follow the general setup in (Bohdal et al. 2021p. our work makes several novel contributions: (1) We introduce learnable sample-wise continuous variables for focal loss. To the best of our knowledge, this is the first work to introduce fine-grained for focal loss for the purpose of model calibration, learned from a meta-network. (2) We propose SECE, which reduces model calibration bias across different binning mechanisms, an aspect rarely discussed in the literature. Additionally, SECE is also more efficient than DECE in (Bohdal et al. 2021 as it uses simple summation rather than predicting the bin assignment using networks. (3) We further showcase the importance and fcasibility of using gradient-based meta- learning Finn et al. 2017) in alleviating model miscalibration. As shown in Table 3 meta-regularizztionn can achieve very competitive predictive and calibration performance compared to conventional regularization methods.
Novelties.
Methods
ECE (6)
Test Error(%)
CIFAR-10
Patra al.
2023
0.59
6.28
Hebbalaguppe et al.
2022
0.93
7.18
Hebbalaguppe al.
20/2
0.70
7.08
LJu el al.
2022
without marign)
3.72
5.24
Liu ct
2122
1.16
4.75
Karandikar
2021
(BCCCE))-EECE)
1.19
4.90
Karandikar
(Focal-SB-AvUC)
1.58
5.60
2023
Dual Focal with ResNet50)
0.46
5.17
Tao ct al.
2023
Dual Focal with ResNet110)
0.98
5.02
Ghosh al.
2022
(AdaFocal with ResNet50)
0.66
5.30
Ghosh et al 2022
(AdaFocal with ResNet50)
5.27
Meta- RetglaKggalazzziiin
(Ours)
2.14
5.43
CIFAR-100
Patra et al.
2023
1.74
26.57
Hlcbbalaguppe ct al.
2022
1.49
31.58
Hebbalaguppe ct al.
2022
0.72
29.80
Karandıkar al.
2021
(Focal+SB-ECCH)
2.30
21.40
Karandikar
2021
SBo
1.57
21.90
Tao al. 2023
Dual Focal with ResNet50)
1.08
22.67
Tao
2023
Duaal Focal with ResNet110)
2.90
22.59
Chosh 2022
(AdaFocal with ResNet50)
1.36
22.60
Gloshh al.
2022
AddaRocal with ResNet50)
1.40
22.79
Mella-Rngaalasslllo
(Ours)
1.94
23.69
Table 3: Comparing meta-regularizztioon (ours) to recent (non-meta) regularization methods CIFAR- 10 and CIFAR-100. Meta-regularrzztion provides very competitive predictive and calibration performance compared to conventional regularization methods (the numbers are obtained from the corresponding papers).
Limitations. Our proposed method utilizes Y-Net for learning 7, which increases the number of parameters by approximately 0.46% during training. However, since 7-Net is not retained after training, there is no additional computational complexity during inference.
7 Conclusion
In this work, we have introduced meta-learning based approach for acquiring well-calibrated models and demonstrated the advantages of two newly introduced components. By learning sample-wise y for Focal
12

---

Published in Transactions on Machine Learning Research (06/2024)
loss using N-Net, we achieve both strong predictive performance and unbiased, robust calibration. The opti- mization of 7-Net with SECE proves crucial in ensuring stable calibration compared to baselines. Through extensive empirical results on three computer vision datasets, we have demonstrated that our method en- hances calibration capability without altering the original networks.
References
Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803. 08377, 2018.
Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Meta-calibration: Meta-learning of model calibra- tion using differentiable expected calibration error. arXiv preprint ArXic:2106. 09613, 2021.
Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Meta-calibration: Learning of model calibration using dilferentiable expected calibration error. Trunsuctions Machine Learning Research, 2023.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint MXXi:1604.07311 2016.
Glenn W Brier et al. Verification of forecasts expressed in terms of probability. Monthly weather review, 78 (1):1-3 1950.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pncumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and dala mining, pp. 1721 1730, 2015.
Jiacheng Cheng and Nuno Vasconcelos. Calibrating deep neural networks by pairwise constraints. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 13709-13718, 2022.
Robert M Corless, Gaston H Gonnet, David EG Hare, David J Jeffrey, and Donald E Knuth. On the lambertw function. Advances in Computational mathematics, 5(1):329-38) 1996.
Morris DeGroot and Stephen Fienberg. The comparison and evaluation of forecasters. The Statistician, 1983.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimization at scale. 35th International Conference on Machine Learning, 1CML 2018, 4:2323 2341, 7 2018. doi: 10.44550/arxiv.uu
URL https://arxix: wwwg/bs///////o/oo/oe 3tg/abs/1180.00174rr1
Chelsoa Finn, Picter Abbeel, and Sergey Levine. Model-agnostic meta-lcarning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126 1135. PMLR, 2017.
Arindam Ghosh, Thomas Schaaf, and Matthew Gormley. Adafocal: Calibration-awart adaptive focal loss. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Newral Information Processing Systems, volume 35, pp.
Curran Associates, Inc., 2022.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. Tn 2013 IEFE international conference acoustics, speech and signal processing, pp. 6645-6649. leee, 2013.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321 1330. PMLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Ramya Hebbalaguppe, Jatin Prakash, Noclabh Madan, and Chetan Arora. A stitch in time saves nine: A train-time regularizing loss for improved neural network calibration. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16081-16090, 2022.
13

---

Published in Transactions on Machine Learning Research (06/2024)
Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael C Mozer, and Becca Roelofs. Soft calibration objectives for neural networks. Advances in Neural Information Processing Systems, 34:29768 29779, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412, 6980, 2014.
ar preprint
Ranganath Krishnan and Omesh Tickoo. Improving model calibration with accuracy versus uncertainty optimization. Advances in Newral Information Processing Systems, 33:18237 18248, 2020.
Alex Krizhevsky, Πlya Sutskever, and Geoffrey Hinton. Imagenct classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
Meelis Kull, Miquel Perello Nicto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Be- yond lemperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. NewrIPS, 32, 2019.
Aviral Kumar, Sunita. Sarawagi, and Ujiwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In International Conference on Machine Learning, pp. 2805 2814. PMLR, 2018,
Max-Heinrieh Laves, Sontje Thler, Karl-Philipp Kortmann, and Tobias Ortmaier. Well-calibrated model uncertainty with temperature scaling for dropout variational inference. arXiv preprint arXiv:1909. 13550, 2019.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18:1-52, 3 2016. ISSN 15337928. doi: 10.48550/arxiv, wtr.ii..603.0000 1603.06560. URL https:: //arxi///xxivvvv wwwpexiww..orreeeeeeeeee abbp//60..0666.44 06560v4
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pp. 1980-2988 2017.
Bingyuan Liu, Ismail Ben Ayed, Adrian Galdran, and Jose Dolz. The devil is in the margin: Mlargin-bssed label smoothing for network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 80 88, 2022.
Jelena Luketina, Mathias Berglund, Klans Greff, and Tapani Raiko. Scalable gradient-based tuning of con- tinuous regularization ion hyperparameters. 33rd International Conference on Machine Learning, ICML 2016, 6:4333 144333-4441, 11 2015. doi: 10. http0//axvv.111111.
URL https://arxix: oeeeiii.rrr//e wtggabs////..oo..o.ogggoeee awtpa///sl..//////.////e//tttttt 1111. 06727v3
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in Neural Infor- mation Processing Systems, 34, 2021.
Azadeh Sadat Mozafari, Hugo Siqueira Gomes, Wilson Leão, Steeven Janny, and Christian Gagné. Attended temperature scaling: A practical approach for calibrating deep neural networks, 2018. URL https:
//arxiv.org/ wttpoo/g/////////o/////////////e 18100 11586
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puncet Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 13:15288-15299 2020.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht, Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Ninth AAAI Conference Artificial Intelligence (AAAT), 2015.
Alexandru Niculescu- Mizil and Rich Caruana. Predicting Good Probabilities with Supervised Learning. In Proceedings of the 22nd International Conference Machine Learning (TCML), 2005a.
14

---

Published in Transactions on Machine Learning Research (06/2024)
Alexandru Niculescu- Mizil and Rich Caruana, Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pp. 625 632, 2005b.
Jeremy Nixon, Mike Dusenberry, Ghassen Jerfel, Timothy Nguyen, Jeremiah Liu, Linchuan Zhang, and Dustin Tran. Measuring calibration in deep learning, 2019. http:ttt/////rww.ooo/////a
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, ZZeming Lin, Natalia Gimelshein, Luca Autiga, et al. Pytorch: An imperative style, high-performance deep learning library. Aduances neural information processing systems, 32, 2019.
Rishabh Patra, Ramya Hebbalaguppe, Tirtharaj Dash, Gautam Shroff, and Lovekesh Vig. Calibrating deep neural networks using explicil regularisation and dynamic data pruning. In Proceedings of the IEEE/CVP Winter Conference on Applications of Computer Vision, pp. 1541 1549, 2023.
Gabriel Pereyra, George Tucker, Jan Chorowski, Fukass Kaiser, and Geoffrey Hinton. Regularizing neural
networks by penalizing confident output distributions, arXiv preprint arXiv: 701.06548, 2017.
John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Tao Qin, Tie Yan Liu, and Hang Li. A general approximation framework for direct optimization of in- formation retrieval measures. Inf. Retr. 13(4):375 397, aug 2010. ISSN 1386-4564. doi: 10.1007/ s10791-009-9122 URL https:///oi. org/ 10. 100.1000/100991-009-90000 912--
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Ade. Newral Inf. Process. Syst. 25, pp. 1-9, 2012. ISSN 10495258, doi: 2012arXiv1206.2945 2944S.
URL https://arxix: wtggpdf////// 2944.
Elias Stengel-Fskin and Benjamin Van Durme. Calibrated interpretation: Confidence estimation in semantic parsing. Transactions of the Association for Computational Linguistics, 11:1213 1231, 2023.
Linwei Tao, Minjing Dong, and Chang Xu. Dual focal loss for calibration. In International Conference Machine Learning, pp. 16833-33499. PMLR, 2023.
Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Newral Information Processing Systems, 32, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Xuan Yang Ya Le. Tiny imagenct visual recognition challenge, 2015. URL http::/visiin.ee wta.nn.tanooooooo edu/ pdf
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In Proceedings of the International Conference on Machine Learning (ICML), pp. 609 616. Citeseer, 2001.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference Knowledge discovery and dula mining, pp. 694 699. ACM, 2002.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Jize Zhang, Bhavya Kailkhura, and T. Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning, 2020. URL
15

---

Published in Transactions on Machine Learning Research (06/2024)
A Appendix
A.1 Evaluation Metrics
Besides ECE and MCE, we evaluate models on Classwise ECE and Adaptive ECE.
Classwise ECE Nixon et al. 2019). Classwise ECE extends the bin-based ECE to measure calibration across all the possible classes. In practice, predictions are binned separately for each class and the calibration error is computed at the level of individual class-bins and then averaged. The metric can be formulated
M K bm.el (ACO(bm.e) cCNF((b..)) Σ m==
CECE =
(14)
NK
where N is the total number of samples, K is the number of classes and bm,e represents single bin for class c. In this formulation, ACC(bm.c) represents average binary accuracy for class over bin bm.e and CoNF(bin.e) represents average confidencee for class over bin bm.c.
Adaptive ECE further extends the Classwise variant of expected calibration error by introducing a new binning strategy which focuses the measurement on the confidence regions with multiple predictions and. Concretely Adaptive ECE (ACE) spaces the bin intervals such that each contains equal number of predictions. The metric can be formulated
Adaptive ECE Nixon et al. 2019
n K ΣΣ
ACE = RK
|ACo(br,c) - co,e(
(15)
where r is a calibration range that defined by the IN I index of the sorted and thresholded predictions. In this formulation, ACC(br,) represents the average accuracy for class e over the calibration range r and coNF(br,e) represents the average confidence for class over calibration range r.
A.2 Algorithm
Algorithm [ describes the learning procedures, it relies on the training and validation sets Dtrains Deats two networks f and f with parameters 0 and Φ The optimization proceeds in an iterative process until both sets converge. Each iteration starts by sampling mini-bated of training data (xf. ys) ~ Dirain well as mini-batch of validation data yx) Deal- The training mini-batch is used to find the optimal value of 7 = y= (x() and following that, compute the focal loss on the outputs of the backbone network ft, The gradient of the loss (x((f((x), y.) is then used to update the parameters of the backbone network using the Adam algorithm (Kingmna & Ba 2014).
Once the backbone model f updated, it is used to compute the value of the auxiliary loss SECE on the validation mini-batch SECE (f")))) The auxiliary SECE loss is differentiable proxy of the true expected calibration error and can be minimized w.r.t. ó to find the optimal parameters of >-Net. It is important to note that SECE is computed based on the outputs of fx(x)) and the only dependence on y is an indirect one based on the updates of 0, as discussed in Section 3.3
A.3
Additional Experiments
A.3.1
Test Error Curves
Figure 0 presents the test crror on CIFAR-10 and CIFAR-100 for the compared methods. In general, they exhibit. similar behavior.
A.3.2 Reliability Diagram Plots
Figure examines the robustness of these methods with different binning schemes by varying the number of bins. It shows that 7-Net based approaches (FL,-DECE and FLy-SECE) maintain much lower ECE score throughout all bin numbers from 10 to 1000 indicating that the trained based network is robustly calibrated.
16

---

Published in Transactions on Machine Learning Research (06/2024)
Algorithm 1: Meta optimization with 7-Net and SECE
Input: fo and fr with initialized and
Output: Optimized and ó
Data: Training and validation scts: Dirains Dval
while 0 not converged do
v2) ~ Derein: yx) ~ Dall: # Sample a mini-batch from both datasets
7i = (x,); # learning sample-wise "
C4 = =x)((xxxx) y(): # Compute training loss based := 0 nVo # Update parameters of
ussnn training loss gradient
SECE SECE (J(()),,)? # Compute SECE auxiliary loss using validation batch
o: Φ no V SECCE # Update parameters of fa using SECE gradient
CE
LS(0.05)
CE-SECE FL-Y-DECE
0.9
LS(0.05)
CE-SECE FL-Y-DEEE FL-YSECE
0.5
FL
MMCE
FL
MMCE
CE-DECE
FL-Y-SECE
Test Error
0.4
Test Error
0.3
0.2
0.4
0.1
0.2
100
200
300
0
100
200
300
Epoch
Epoch
(a) CIFAR-10 (Error)
(b) CIEAR-100 (Error)
Figure 6: Test error both test datascts.
-.20%
ECE +3.32%
ECE
6.90%
2.40%
0.6
34
0.8
0.8
MCE 39.13%
0.5
VICE 37.70%
0.5
18.75%
Acouratd
0.4
0.4
0.4
0.44
48.0
0.6
1.5
0.2
0.8
1.0
0.80
0.4
2.8
0.3
044
Confidence
Confidence
Confidence
(a) CE (4.94%)
(b) CE (T3) (4.94%)
(c) Focal (5.01%)
(d) FLSD(5.02%)
LS-0.05 (4.67%)
4..7%
ICE
-4.31%
ECE 1.0%%
187%
144%%
0.6
39
0.8
65.74%
0.8
MCE
0.5
VICE 28.3234
0.5
Accurasy
Accuracy
Accuracy
0.4
0.4
0.4
0.4
0.44
0.80
0.6
1.5
08.0
7.5
08
0.8
D.A Confidence
3.8
0g
0.4
Confidence
Confidence
Confidence
MMCE (4.83%)
(g) CE-DECE(5.10%)
(h) PL,-DECE(
CE-SECE (5.06%)
FL. (L.C(555(5)
Figure 7: The reliability diagram plots for models on CIFAR-10 test set. The (-) denotes test error. The diagonal dashed line represents perfect calibration. The red bar represents the gap between the observed accuracy and the desired accuracy of the perfectly calibrated model (diagonal) is positive the observed accuracy is lower and nogative otherwise. The model from the 5th run is used.
17

---

Published in Transactions on Machine Learning Research (06/2024)
12
10
CE-SECE
FL
Test ECE
Test Test MCE
60
Y DECE SECE
FLSO
LS1O 05)
40
MINCE
10a
10a Bin numbers
10a 10a
10a Bin numbers
10a
Figure 8: The changes in ECE (left) and MCE (right) scores on the C1FAR-100 test dataset with increasing bin numbers in the range of [10, 20, 50, 100, 200, 500, 1000] are illustrated. Our proposed approach shows better robustness on bin sizes.
ECE 4.35%
ECE 3.67%
1.84%
MCE 65.74%
MCE 68.93%
75.65%
28.55%
Accuracy
0.6
Accuracy
0.6
0.6
0.66
9.4
0.4
0.4
0..0
0.4 0.6
0.8
065
0.2
0.4
0.6
088
1.0
0.2
0.4 0.8
02
1o
0.86
0.2
0.4 0.8
0.8
1.0
Confidence
Confi dence
Confidence
Confidence
(a) CE-DECE
(b) CE-SECE
(c) FL,-
(d) FL, -BCE
1.0
1.0
1.0
1.0
ECE 4, 4,38%
ECE 3.96%
ECE
2.01%
ECE 1..5%
0.8
MCE 05, 74%
MCE
66 08%
KCE 75, 03%
0.6
MCE 31.27%
Accuraty
Accuracy
Accuracy
Agguracy
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0.0
066
0.8
88.0
0.000
Confii
Confi dence
Confidence
Confidence
(e)
CE-DECE
(f) CE-SECE
(g) FL,-DECE
(h) FL,-SECE
1.0
1.0
ECE 4.43%
ECE
-2.18%
22000% 41.62%
0.8
68.75%
0.8
66.09%
0.8
0.8-
Accuract
0.6
Accurazy
0.6
0.6
Accuracy
0.6
0.2
0.2
0.0
0.6
0.2
6.0
0.6
0.8
1.0
066
0.2
0.4
0.6
0.8
1.0
08
0.2 0.4
0.6
0.
1.0
0
0.2
0.0 0.6
0.8
1.0
Confid ence
Confidence
Confidence
Confidence
(1) CE-DECE
(i) CE-SECE
(k) -L,,-DECE
FL,, SECE
Figure 9: The reliability diagram plots for models on CIFAR-10 with large bin numbers (from top to bottom: 20, 50, 100). The diagonal dashed line represents perfect calibration, while red bar indicates the devialion from perfect calibration in each bin. The model from the 5th run is used.
In Figure g we provide the comparison across meta-lcarning based approaches via reliability plots with large bin numbers and corresponding ECE and MCE. FL,-SECE is able to maintain the lowest ECE and MCE; it shows robustness to increased bin numbers.
18

---

Published in Transactions on Machine Learning Research (06/2024)
Methods
Error
NLL
ECE
MCE
ACE
Classwise ECE
5.354
0.097
0.220 0.015
3.038
+ 0.159
33.419
3.035
0.157
0.677 0.031
CE-DECE
5.850
0.236 0.017
3.245 + 0.471
25.062
3.240 0.471
0.072
CE-SECE
5.895
0.271
0.246 0.032
3.555
1 0.428
24.518
3.613
3.555 0.425
FL,-DECE
6.084
0.188
0.199 = 0.012
2.151
= 1.499
24.644
11.227
2.139 = 1.457
0.605 0.201
FLy-SECE
6.263 0.266
0.208 + 0.021
2.549 + 2.233
20.220 + 5.235
2.559 2.217
0.672 0.355
Table 4: DenseNet (100 layers) on CIFAR-10 with vecta-laaminggbasee methods, 5 repetitions. gamma-nct
based methods exhibit better calibration across multiple metries (NLL, ECE, ACE, Classwise ECE) while maintaining competitive test error.
A.3.3 Experiments with DenseNet
We conducted experiments on DenseNet with meta-learning basclines. Table 4 presents the effectiveness of gamma-net based methods, which exhibit better calibration across multiple metrics with comparable predictive performance.
A.4 The PredictiveCCalibration Trade-off
Figure 10 depicts the empirical trade-off between predictive (measured by test error) and calibration (mea- sured by ECE) performance. We observe that many methods exist the Pareto front. For instance, mixup (ar = 0.1), LS(0.1) and FL,-SECE achieves the low Lest error but high ECE CIFAR datasets. This indicates there is predictive-calibration trade-off for most of methods. Similar Pareto fronts are observed in pretained large language models (LLMs) (Stengel-Eskin & Van Durme 2023). However, as shown in Table when evaluating the ECE improvement for our proposed FL,- -SECE method, the reduction in predictive capability modest.
Methods
CIFAR-100
CIFAR-100
A Arror)
ΔECE
(Aerror)
ΔECE
(Aerror)
AgCE
Focal
0.062
-0.863
-0.072
-3.336
-0.695
-0.459
FLSD
0.104
2.848
0.086
-2.424
-0.405
6.442
LS(1.0)
0.106
3.510
-0.326
-3.626
-0.715
8.718
Mixup(a = 1.0)
-0.686
8.807
-1.360
1.336
-0.220
4.074
MMCE
-0.004
-0.029
-0.080
0.333
0.200
0.147
CE-DECE
0.382
0.050
-0.164
-1.071
1.240
2.635
FECE
0.616
-1.918
1.116
-6.440
0.740
2.265
Table 5: The test error difference (Aerror) and ECE difference (A (AECE) for different methods compared to the bascline (CE).
19

---

Published in Transactions on Machine Learning Research (06/2024)
Pipppla
Miceplo 19)
11.5
050
14-
SEOE
ECE(%)
U50
12
PNCE
DECE
42
4.4
4.5
4.8
5.0
52
5.4
21.5
22.0
22.5
23.0
23.5
39.50 30.75 40.00 40 25 40.50 40.75 41.00 41.25 Error(%)
Error(%)
Ersor(%)
(a) CIFAR-10
(b) CIFAR-100
(c) Tiny- ImageNet
Figure 10: The trade-off between predictive and calibration performance of different methods the test set of CIFAR-10, CIFAR-100 and Ting-ImageNet. The Parelo front are observed for most of methods.
20


Reviews:
[{"review_id": "yeR0RxVM9s", "reviewer": "Reviewer_XKgH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe work introduces a meta-network capable of learning sample-wise gamma in the focal loss, which results in competitive calibration performance to previous methods.\n\n--- \n\nWeaknesses \n\n1. The authors said the FLSD learns a sample-dependent schedule for gamma in focal loss. In the first line of the second page, the authors also mention that ‘different from FLSD, we learn a gamma parameter for each sample’. Could the authors clarify the difference?\n2. To my understanding, $L_\\text{SECE}$ is dependent on the output of the backbone model. Does this mean the $D_\\text{val}$ is just the output of the backbone model?\n3. Looking at Table 1, although the calibration performance is better in most cases, the proposed method tends to have the highest prediction error rate (i.e., lowest accuracy). The calibration error and accuracy are a trade-off in most cases, and it is not clear if the proposed method just prefers a better-calibrated prediction by sacrificing accuracy. Could the author provide a Preto front analysis to show that the proposed method achieves a superior solution rather than simply obtaining a prediction with lower calibration error and higher error rate?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Training a classifier with focal loss has been known to improve the calibration of the network prediction. However, the gamma parameter in the focal loss plays a crucial role in making this approach effective. Motivated by a previous work [1] that used a meta-learning approach during calibration during training, this paper also proposes a meta-learning approach to learn a sample-wise gamma via a meta-network. \n\n\nReference:\n\n[1] Bohdal et al., Meta-learning of model calibration using differentiable expected calibration error, 2021", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses above", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UryNnhvfCA", "reviewer": "Reviewer_mW6C", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The key problem of this paper is lack of comparison and discussion with related works:\n\nFor Gamma-Net providng sample-wise gamma values for focal loss, there are some focal loss based calibration methods such as [1][2], the author should compare with them. In addition, for the meta learning Gamma-Net itself, more motivation or empirical examples are needed.\nFor Smooth Expected Calibration Error (SECE), it is very similar with the [3], although it is compared in the experiments, it should be discussed in the main paper and related works, in terms of methodology.\n\n---\n\n[1] Dual focal loss for calibration\n\n[2] Adafocal: Calibration-aware adaptive focal loss\n\n[3] Soft calibration objectives for neural networks", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a meta-learning approach to address model miscalibration in deep neural networks. It introduces two main components: the Gamma-Net, which outputs sample-wise gamma values for focal loss, enabling fine-grained calibration regularization, and the Smooth Expected Calibration Error (SECE), an unbiased and differentiable calibration error metric using a Gaussian kernel. This combination allows for effective, sample-wise calibration adjustment. Tested across three computer vision datasets, the approach significantly enhances calibration performance without sacrificing predictive accuracy, showcasing its potential for practical deployment in various applications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please the authors include more comparison with SOTA calibration methods and discuss the difference between SECE and [Soft calibration objectives for neural networks]", "broader_impact_concerns": "The paper proposes a meta-learning approach to address model miscalibration in deep neural networks. Gamma-Net is an adaptive way to apply focal loss. The adaptive way can also impact other methods of this field and bring better calibration.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iRntphfliJ", "reviewer": "Reviewer_fXcz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* Strengths: the proposed loss function is differentiable w.r.t. to both the model parameters and the focal loss parameter. The experimental results show that the proposed method can improve the calibration of the model by taking steps on the gradient of this loss function.\n* Experimental results compare multiple methods across three datasets. Improvements are shown in the ECE values, which is a common metric for calibration. The results are consistent across datasets, which is a good sign that the method does not overfit a specific dataset.\n\nWeaknesses:\n\n*  The motivation for 'better calibration' is not clear. In most experimental results, the classwise calibration error is an order of magnitude lower than the error rate. In the case of CIFAR100, the classwise ECE is even two orders of magnitude lower than the error rate. What does an improvement in ECE then mean?\n* Following on the previous question, most improvements are well within the noise standard deviation of DECE, which is prior work. Therefore, are the results significant?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of calibrating neural networks. The introduction assumes that neural networks are ill-calibrated and proposes a new model and new loss function to train better-calibrated networks. Experimental results show that the method of sample-wise prediction of focal loss during training can improve the ECE values. However, both Table 1 and Figure 5 show no significant difference between DECE (prior work) and the proposed method. Therefore, the second claimed contribution remains to be argued about.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* The definition of pi(x) is not clearly defined. The text states 'and pi(xi) are the confidence and accuracy of ith example.', but what does that mean? do you mean it's the **difference** between confidence and accuracy?\n* Figure 3a: The ECE increases during training for most of the methods. This effect also seems present for the proposed method, although to a lesser extent. The text makes this observation, but no explanation is given. How could the ECE significantly increase when the learning is a proper scoring rule?\n* Figure 5: there seems to be no significant difference between DECE and SECE. Could one conclude that most improvement come from the new focal loss and not the new Differentiable loss function?\n\nTypo's?\n\n * 'Learnable sample-wise at as continuous variables for focal loss.'\n * 'Additionally, SECE is more also efficient than DECE''\n * 'We further showcase the importance and fesiablity'", "broader_impact_concerns": "n.a. better calibration would improve many applications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a meta-learning approach for calibrating neural network uncertainty prediction, providing numerical evidence it approaches SOTA performance across different calibration metrics and tasks.

The initial reviews pointed towards insufficient numerical benchmarking and comparison to the literature, and a possible accuracy vs. calibration trade-off. During the rebuttal phase, the authors have clarified some points raised and update the manuscript, including further comparison to other methods in Table 3 and an additional section (Appendix A.4) following a reviewer's request.

The reviewers were overall positive with the revised version, except for an additional comment:

> *In the rebuttal, also, the authors state that "Error rate and classwise ECE measure (...) are not directly comparable." That remains an open question to me. For example, if the error rate is 10 in 100 samples, but the ECE, which is the L1 distance between prediction and accuracy is 1/100, then the calibration error is an order of magnitude lower than the error rate. In that case, what does an improvement in ECE mean? This trade-off is also discussed in Figure 7 of [1]. The rebuttal alludes to this point by saying, "The difference between SECE and DECE are indeed within one standard deviation apart in most cases, but we attribute it to the overall low values of calibration errors."*

Therefore, my recommendation is for the acceptance of the manuscript, conditioned on a minor revision to reflect the discussion above.
Decision: Accept (poster)