Optimizing Estimators of Squared Calibration Errors in Classification
Authors: Sebastian Gregor Gruber
Francis R. Bach
Venue: Accepted by TMLR ()

Abstract: In this work, we propose a mean-squared error-based risk that enables the comparison and optimization of estimators of squared calibration errors in practical settings.
Improving the calibration of classifiers is crucial for enhancing the trustworthiness and interpretability of machine learning models, especially in sensitive decision-making scenarios.
Although various calibration (error) estimators exist in the current literature, there is a lack of guidance on selecting the appropriate estimator and tuning its hyperparameters.
By leveraging the bilinear structure of squared calibration errors, we reformulate calibration estimation as a regression problem with independent and identically distributed (i.i.d.) input pairs.
This reformulation allows us to quantify the performance of different estimators even for the most challenging calibration criterion, known as canonical calibration.
Our approach advocates for a training-validation-testing pipeline when estimating a calibration error on an evaluation dataset.
We demonstrate the effectiveness of our pipeline by optimizing existing calibration estimators and comparing them with novel kernel ridge regression-based estimators on standard image classification tasks.

Published in Transactions on Machine Learning Research (02/2025)
Optimizing Estimators of Squared Calibration Errors in Classification
Sebastian G. Gruber
Aobostiin.gguube@ddfff.
German Cancer Consortium (DKTK), partner site Frankfurt/ FrankfurtMaina, Manna, partnership between DKFZ and UCT Frunkfurt-Morbuue Germany, Fronkfurt am Main, Germany German Cancer Research Center (DKFZ), Heidelberg, Germany
Goethe University Frankfurt, Germany
Francis Bach
Inria, Ecole Normale Supérieure, PSL Research University, Poris, France
Reviewed on OpenReview: //pp:re/epeerevier// net/ forum? id= BPDV20000
Abstract
In this work, we propose a mean-squared prror-based risk that enables the comparison and optimization of estimators of squared calibration errors in practical settings. Improving the calibration of classifiers is crucial for enhancing the trustworthiness and interpretability of machine learning models, especially in sensitive decision-making scenarios. Although various calibration (error) estimators exist in the current literature, there is a lack of guidance on selecting the appropriate estimator and tuning its hyperparameters. By leveraging the bilinear structure squared calibration errors, we reformulate calibration estimation as a regression problem with independent and identically distributed (i.i.d.) input pairs, This reformulation allows us to quantify the performance of different estimators for the most challenging calibration criterion, known canonical calibration. Our approach advocates for a saainingvaliidation--eee
pipeline when estimating calibration error an evaluation dataset. We demonstrate the effectiveness of our pipeline by optimizing existing calibration estimators and comparing them with novel kernel ridge regression-bant estimators on real-world image classification lasks.
1 Introduction
In the field of machine learning, classification tasks involve predicting discrete class labels for given instances [Bishop & Nasrabadi 2006). As these models are incrcasingly being employed in critical applications such healthcare (Haggenmüller et al. 2021 autonomous driving (Feng et al. 2020), weather forecasting (Gneiting & Raftery 2005), and financial decision-making (Frydman et al. 1985), the need for reliable and interpretable predictions has become of critical importance. A key aspect of reliability in classification models is the calibration of their predicted probabilities (Murphy & Winkler 1977 Hekler et al.) 2023). Calibration refers to the alignment between predicted probabilities and the true likelihoods of outcomes, ensuring that predictions are only accurate but also mcaningful in terms of their confidence scores (Murphy 1973). Despite the advancementts in model architectures and learning algorithms, many modern classifiers, such as deep neural networks, prone to producing overconfident predictions (Minderer et al. 2021). This overconfidence be attributed to several factors, including the model's complexity, training data limitations, and inherent. biases in learning processes (Guo et al. 2017). Consequently, even models that achieve high accuracy might suffer from poor calibration, leading to polential misinterpretations and suboptimal decisions.
To quantify the extent which model miscalibratod, calibration errors have been introduced Nacini et al. 2015). However, their estimators are usually biased (Roclofs et al. (2022) and inconsistent (Vaicenavicius et al. 2019 Other calibration errors with unbiased estimators exist. but they lack theoretical derivation and

---

Published in Transactions on Machine Learning Research (02/2025)
are difficult to interpret [Widmann et al. 2019 2021 Marx et al. 2024 This, in turn, is highly problematic since we camnot quantify how reliable a model is if we either do not know how reliable the metric is or how to interpret it. In this work, we tackle the former problem. We propose mean-squared error risk minimization to find an optimal calibration estimator for any squared calibration error. This risk can be applied in any practical scenario to compare and select different estimators, and works for all notions of calibration, even for the notoriously difficult canonical calibration. Our contributions are as follows:
We propose novel risk applicable for squared calibration estimators in Section 3.1 which represents an optimization objective for comparing calibration estimators proposed by the literature.
We formulate calibration-evaluuttoo pipeline based on our risk in Section 3.2.2 which allows optimize calibration estimators used by the litcrature.
We propose novel kernel ridge regression-based calibration estimators Section and compare these with optimized baselines on common image classification models in Section 5
2 Background
In the following, we offer an extensive introduction to the background of this work. First, we offer a more extensive motivation for calibration. Second, we give briefly measure theoretic preliminaries, followed by the different notions of calibration used for classification by the literature. We then introduce and discuss commonly used estimators for these notions.
2.1 An Exemplary Motivation for Calibration
Uncertainty calibration in classification is directly connected to practitioners' concerns in real-world application as the following example demonstrates. Assume we are sensitive classification task, c.g., we aim to prodict the classes Y E (No-Tumon Benign-Tumor, Malignand-Tumman
for given patient X via classifier In such setting, the practitioner may be interested in the uncertainty of a given prediction f(X) to assess how reliable the prediction is. One possible uncertainty is the predicted probability for cach outcome, c.g., f(X)= =(No- Malignant-Tumor: Tumor: 0.7, Benign-Tumor: 0.2, Maligmant- Tumor: 0.1 }, which aims to predict the unknown ground truth X), and IP(Y = MMlaigannt-Turee Tumor X). However,
probabilities P(Y No-Tumor X), IP(Y = Benign-Tunne similar to how the predicted class may be untruthful, so may the predicted probabilities. To see this, assume we also have a dataset of 10.000 prediction-target pairs in the previous example. Now, further assume there exist 100 instances of prediction-targel pairs within this dataset with the above prediction. Ideally, if the predicted probabilities were truthful, then among these 100 instances, there would be 70 instances of No-Tumor, 20 instances of Benign- Tumor, and 10 instances of Valigmant-Tumon If this holds for all predictions, we refer to the underlying classifier as being calibrated. In summary, the central question for calibration is, given a probability prediction fX), how closely do the outcome probabilities TP(Y) (X)) match f (X)? Squared calibration errors are then used to quantify the expected square difference between prediction and outcome probabilities (Murphy) 1973). In the following, we formulate these concepts in a mathematically rigorous manner similar to (Vaicenavicius al. 2019).
2.2 Measure Theoretic Preliminaries
Throughout this work we mcasure theoretic definitions formalize our contribution, its assumptions, and its limitations. Specifically, assume measure space (S2, F (1) is given. We associate with random variable X:Ω + 2' (a measureable function) probability space (22,,xx, xx,Pxx with o-field Fx = = (A) A A .) and pushforward measure Px = = x Further, we may also assume a second random variable Y:Ω Φ such that we have joint probability space (. x 3 ",,,x.. Pxy), where Fxy and IPxy are defined analogous Øx and Px. Further, if a random variable Y is discrete, we use Py to represent both its probability measure and the associated probability vector in the simplex A = [(p..... .pr) E [0o 11 E P; = 1]. where d is the number of unique outcomes. In the same sense may Pr|x = dp as measurable function 2 Ad. dFx
2

---

Published in Transactions on Machine Learning Research (02/2025)
Finally, say statement holds j-almost surely (H-a.s.) if it is true except set of p-measure zero (Capinski & Koppl 2004).
2.3
Mean-Squared Error Risk Minimization and Calibration
The mean- sqared error (MSE) is one of the most common loss functions used in regression problems, see, Schölkopf & Smola 2002 Bishop & Nasrabadi 2006; [Goodfellow et al. 2016 Murphy
c.g.,
EEfron
1994:
2022
Bach
2024). Its expected loss for a vector-valued sample Py from a target distribution Py and a
prediction E TR" is defined by
MSE (c. Py) Ey-py [[]c-Y13].
(1)
Tt holds that the expectation of the target term in the difference is the unique minimizer, i.e.,
B[Y]= = arg min LISE (c, Py).
(2)
nERd
Now, let's introduce another random variable such that (X, ~ Pxy follows a joint distribution and can be used as input for a regression model m: 2 - R to approximate the target conditional distribution m (x) = [[Y X = æ]. Then, the corresponding risk of m is defined by
MAE (m) = Ex-Px [LMSF (m (X), Py|x]] E(x,y--xx [[lm (
(3)
Given m", where the inequality holds for of positive probability mass, follows that
MSE (m) MSE (msE(m")
(4)
However, we have the measure theoretic limitation that there might be a null set A E Ix and some m such that in (x) # m (x: for all x E A while MSE (m) - (m"). This limitation risk minimization is usually irrelevant in machine learning applications. However, in our case, it will require additional theoretical assumptions on the target conditional distribution make risk minimization a usable tool for assessing calibration estimators.
The MSE can also be used for classification lasks with a one-hot encoded target in Equation D often referred to as Brier score (Brier] 1950). Then, Murphy 1973) introduces the concept of calibration for a classifier J: 2 Ad d by showing that.
MSE =x)=Ex-ry (Pix)lll Ex-rx [llpy- -aly--yyiin PYI 111x111
||PP|
The first term the right- hand side is usually referred to the calibration term, and the second sharpness term, coining Equation (5) as calibration-sharppeee decomposition of the Bricr score (Gneiting et al. 2007
Gruber & Buettner 2022 Kuleshov & Deshpande 2022 Gruber et al.) 2024 Sun et al. 2024). In the
calibration term, [(X) is compared with the target distribution PP( [(X) given the full predicted vector. In current literature, this notion of calibration referred to as canonical calibration /Vaicenavicius et al.)
2019: Popordanoska et al.] 2022b Gupta & Ramdas 2022
Gruber et al. 2024). Formally, we say the model
is canonically calibrated il and only if
P(Y i (X) = p) p for all p P 1=...==1..d
(6)
The corresponding L canonical calibration error defined as
CCE, = Ex Pyiss))l
(7)
which is equal to the calibration term in Equation 5 It holds that CCE, (/) = 0 if and only if f is canonically calibrated IP Px-almost surely.
However, canonical calibration errors are notoriously difficult to estimate and represent a calibration strictness which may not be necessary in practice (Vaicenavicius al. 2019). Consequently, other notions of calibration have been proposed, which discuss next.
3

---

Published in Transactions on Machine Learning Research (02/2025)
2.4 Alternative Notions of Calibration
Besides canonical calibration, multiple notions of calibration have been introduced in the literature Zadrozzn & Elkan
2002: Vaicenavicius et al. 2019 Kull et al.
2019
Gupta & Ramdas 2022). Respective calibration
errors assess the degree to which a classifier violates à given notion. In recent literature, the most common
notion top-label confidence calibration Naeini et al.
2015
Guo et al. 2017 Joo et al. 2020 Kristiadi et al.
2020
Rahimi et al. 2020 Tomani et al. 2021 Minderer et al.
2021 Tian et al. 2021 Islam et al. 2021 Menon
ct al.
2021
Moralcs-Alvarez al., 2021 Gupta al. 2021
Wang ct al. 2021: Fan ct al.
2022 Dehghani
jet al.
2023
Chang al. 2024). Here, we compare if the predicted top-label confidence max [i
max fi (X)) given the prediction. Formally, we say the classifier is top-label confidence calibrated if and only if
matches the conditional accuracy PP (Y arg maX fi (X)
-))= = for all p E [].
PP Y arg mag (x) max (x) =p)
(8)
The corresponding 2 top-label confidence calibration error is defined as
TCE, (f)= (f)= = Ex max [ (x) - PlY arg max fi (X) max fa (x)
99)
It holds that TCE, (f) and only if f is top-label confidence calibrated Px-almost surely. It is easier to estimate than CCE₂ since the target conditional distribution is only based on scalar random variable independent of the number of classes. However, top-label confidence calibration is weaker condition than canonical calibration since the implication CCE, (f) = = TCE, (f) = 0 does not generally hold in the
reverse direction (rrbber & Buettner 2022
Other notions of calibration exist, which also reduce the prediction to a scalar, such as class wise calibration (Zarrzzy & Likan 2002 Kull et al. 2019 Kunar et al., 2019). Gupta & Ramdas (2022) introduce further of such notions. In general, these notions are transformations of the full probability vectors to a lower dimensional space. Similar to top-label confidence calibration, this makes them easier to estimate than canonical calibration but also turns them into weaker condition due to the information loss of the
transformation (Vaicenavicius et al. 2019 Gruber & Buettner 2022).
2.5 Calibration Estimators
We assume a dataset of ii.d. samples
Y,) Pxy estimate the calibration of given
classificr f: 1 + A The most common approach to estimate calibration errors based on scalar conditionals are binning schemes [Naeini et al. 2015 Guo et al. 2017 Minderer el. al. 2021 Detlefsen el al. 2022). A prominent estimator the so-called expected calibration error (ECE), which is a binning-bass estimator of the L top-label confidence calibration error (Guo et al.
2017). In essence, the conditional target distribution PP PYY arg max; fa (X) max, fa (X)) is estimated via a histogram binning scheme, which places all top-label confidence predictions into mutually distinct bins Bn {i arg max, f3 (X,) E Lm); m I,..., M, based on partition Um Im 1]. The analogous L²² estimator given by
Baul n
TCE bin (f)= =
Σ
(ace (Bm) conf (B,,.)²²
(10)
with ace (B)= [.| ΣIEB 1 myaargmax, fxX.)) and conf con [ LEEB arg max, f, (X.) Kumar et al. 2019). This
estimator is primarily suitable for larget distributions conditioned on a scalar random variable. The choice of bin intervals 4,...,,4 IM is user-defined and the estimator only converges to TCE, for an adaptive scheme (Vaicenavicius al. 2019). Patel al. (2021) and Roclofs al. (2022) propose approaches to automatically select appropriate bins. However, in practice, remains an open challenge to definitively select the optimal choice for a specific dataset and classifier. Analogous binning-based estimators for class- wise calibration exist.
4

---

Published in Transactions on Machine Learning Research (02/2025)
in the literature and share these limitations [Kumar et al. 2019 Nixon et al. 2019 2019
Vaicenavicius et al.
Estimating canonical calibration is more difficult than other notions of calibration due to the target distribution
P(Y (X)) being conditioned on a vector-valued random variable. Pooordaaoska et al. (2022b) propose
use a kernel density ratio estimator, which is closely related the satrraaa-aatson-ssimnaaat (Bierens)
1996). The estimator for CCE2 is given by
Σ, kair (f(x)): ex, CCE2 (f) = (xx) Σ, kair (f (x
(11)
where refers to the unit vector with 1 at index and kair is chosen to be the Dirichlet. kernel, which is specifically suited for the simplex space (Ouimet & folosana-Delgadde 2022). The authors also propose analogons kernel density based estimators for top-label and class- wise calibration crrors, which we will denote as TCE2 and CWCE Even though the kernel density approach is advantageous compared to the bimning approach for higher dimensions, it still shares some of its limitations. Popordanoska et al. (2022b) show that the estimator converges in the infinite data limit, but it is still not clear what is the optimal choice of kernel and kernel hyperparameters the finite data, regime. Further, the runtime complexity CCF
0
To summarize, a lot of different approaches have been proposed by the literature to estimate calibration errors. However, it is not clear how to compare different estimators and pinpoint an optimal choice in finite data setup in practice.
3 Mean-Squared Error Risk for Calibration Estimators
In this section, we present our main contribution: A mean-squared error based risk, which can be applied to compare different calibration estimators in a practical, finite data setup. We first discuss its measure theoretic foundations in Section 3.1 and, then, propose a training-inference inference pipeline for estimating the calibration error in practice in Section 3.2 This pipeline is analogous how model training, model selection, and test error evaluation is done in practice in machine learning (Bishop & Nasrabadi] 2006). All formulations are with respect to canonical calibration, since this is the most general case. Other notions, like top-label confidence calibration, can be derived by restricting the canonical case to binary classification. All missing proofs are presented in Appendix C
3.1 Theoretical Definition and Properties
Note that for the CCE, calibration error it is sufficient to find function A":A" A² - -R such that
h" (p.p') = (p- PYISI p' PYI(X)=)
(12)
since from this follows that
Ex [1* (F(X)) CCE,
(13)
Indeed, in a later section, we will discover that current estimators already implicitly use such a form. general, we refer to a function A d x A + R as calibration estimation function. We now propose a risk, which quantifies how close such a h is to l². To achieve this, we slightly modify the mean-squared error loss function of Equation [1 in the following.
Definition 1. For a prediction E I, a target product measure Py Pv, and constants p.p E A we define the calibration estimator loss by
LCE (c, Py o Pv:PP.'' By. [((p [llp-ey,p's (ilo--eyp--ev--eee -c))
(14)
where e, refers to the wnit vector with at inder i.
5

---

Published in Transactions on Machine Learning Research (02/2025)
Similar to the mean-squared error, it holds that Len has unique minimizer given by
(p Py Pv) arg min TIEE (c, Py @ pv:p.)))
(15)
CER
We use this definition of a novel loss define the respective risk in the following.
Definition 2. For calibration estimator function h: A² x A - R, define the calibration estimation risk by
cr (h) = Ex,x [LCE (x)) PyINIIIIM (x
(16)
with X, X ri Px.
Similar to MSE in Equation 3
we may also express ICE in simpler form, since it holds
ACB (h) = Exxxxr
(17)
with
Pxv. This formulation will be used in a later section to construct the empirical risk. Next, we establish that our proposed risk can distinguish the right solution almost surely.
Theorem 1. For any h: A" x IK for which h h does not hold P f(x) @ P(xx)=almost surely we have that
ccr (h) > ECE (h").
(18)
This property would be sufficient in practice, if we were using h with arguments sampled from P((x) @ f(x) estimate the calibration error. However, as demonstrated by Equation (13) we predict CCE, via the same sample in both arguments. Mirroring the arguments results in possible P Pą() @ Py(xy-mull set since only elements in the diagonal 9 (-a) = {(p. p) I E 4 c A x A are considered. Consequently, the diagonal of an optimum h" identified via &ee may "slip through' the f(() @ Px guarantee in Theorem | and in turn may result in vE[h" [(X]] CCE2 (/). To avoid such theoretical exceptions, we require additional theoretical assumptions, which we state in the following.
Theorem 2. Assume function h: A x Ad
is continuous in all points of the diagonal (A \ with A being a IP (Xx))-oull set, and the target as a function PyΙлxς A aa continuous P pixx-almost surely. Further, assume the boundary of the support of IP Rx) does not involve a singular distribution, then it holds that
ACr (h) ReB = V [IEx) CCE,
(19)
This result states under which conditions estimation. We briefly discuss these conditions, which are of purely technical nature and should not influence practical results. First, the continuity of h is non-problematic since it is uscr-defined and infinitcly many points of discontimuity are allowed (as long as they have no probability mass). The continuity of Prıバ()) may be considered the most relevant condition in practice, since we usually do not know about the nature of the target distribution. However, again, infinitely many points of discontimuity are allowed, as long as their overall probability mass is zero. The last assumption, namely that the boundary of the support is not allowed to be part of singular distribution, disqualifies certain theoretically crafted distributions. One such example is the Cantor distribution, which consists of infinitely many disconnected points each of zero probability (Teschl 2014). In summary, the purpose of the conditions in Theorem 2 is to guarantee that samples from P P(X] @ P((x) can be arbitrarily close to the diagonal 9 (A*) and that this closeness indicates how well h matches h² Φ (Aa))
expect that optimal risk indicates truthful calibration
Remark. Alternatively to our approach, one might also use a loss for finding probabilistic model 9 (p) = РУІлIHo and then use h (p) = (p (p),p' 1obp'''9p 9 (v')) = (p P -PYIsx-=) as a solution. However, learning a predictive space A becomes increasingly more difficult for higher dimensions d than regression problem in R. For example, approach is invariant to any orthogonal matria. M since (Mx, My) (a. (x,y)-

---

Published in Transactions on Machine Learning Research (02/2025)
3.2 Estimating Calibration for Finite Data
In this section, propose novel calibration evaluation pipeline for the finite data regime according to our theory. First, we give an unbiased and consistent estimator of the risk in form of an U-statistic. Then, we mimic the saaaiing-vaidaatio--essiii pipeline of a conventional machine learning model for a debiased calibration estimate. This procedure allows to select between different calibration estimators and optimize their hyperparameters.
3.2.1 Risk Estimator
Note that the risk formulated in Equation 17 is expectation of two i.i.d tuples of random variables (X, Y) and (x',"". Consequently, given an i.i.d. dataset (X1, (X Yn) Pxy, we can construct an U-statistic estimator (Shao 2003) via
......)
R1 2 (113))--0_((3)))==----1111101111 dCE (h) n nnn - 1)
(20)
It holds that E ACE [h]] ECe (h), and ACE (4) Ice (h) in distribution if n co. The estimator has quadratic complexity in n. However, an estimator with linear complexity can be constructed as well by excluding certain index combinations.
3.2.2 Galirration-Evalaatioo Pipeline
In general, we cannot expect find h. However, the empirical risk allows to find f, H,, close h" where H, model class with hyperparameter 7) E θ and search space O. Examples for the number of bins in Equation (10) the bandwidth in Equation (II). Consequently, similar to traditional machine learning, we need training-valiinn vaaiiing-alidaaion-tssss split to achieve debiased estimation of the calibration error once we optimized Specifically, we use training set to fit h. and a validation set to find an optimal Thal The final calibration estimate is then computed by
CF2
=
Ryy. (X)),
(21)
Yhee
XED
where Dio is the test set of size Rhe, and CE2 is representative for different notions of calibration. We may also use multiple training and validation sets via cross-validation, similar to conventional hyperparameter optimization (Bischl et al. 2023). Our proposed pipeline for estimating the calibration error of a classifier via cross-validation is presented in Algorithm [] The algorithm mimics hyperparameter optimization of a conventional machine learning model (Bischl et al. 2023). Consequently, using our pipeline also implics a computational complexity of O (k(@)), where le the number of folds cross-validation.
comparison, (Guo et al.) 2017 or (Minderer et al. 2021), has O(1) complexity since
using a default hyperparameter, no optimization happens.
Remark. We recommend not compuring the holdout lest set risk of different calibration estimation functions since this would put bias the final calibration estimation. Neglecting this is similor selecting an optimal classifier based the test accuracy in classification task.
In this section, have established our optimization and evaluation pipeline based on calibration estimation functions. Next, show how already existing calibration error estimators used in the literature can be framed as calibration estimation functions.
4 Calibration Error Estimators as Calibration Estimation Functions
In this section, we first formulate the calibration estimation functions implicitly used in the literature. Then, introduce two novel calibration estimators based on kernel ridge regression, which minimize regularized
7

---

Published in Transactions on Machine Learning Research (02/2025)
Algorithm 1 Evaluating the calibration of given classifier and dataset by optimizing the calibration
estimator. The evaluation dataset is split into holdout set for estimating the calibration error, and another set, which is used for optimizing the calibration estimator via cross-validation.
Input: Evaluation dataset D ((X,,Y))..... .i),...., (Xx: Yn)), classifier f. model class H, of calibration ostimation functions with hyperparameter search space è ô n, k number of folds.
Drr + [(f (1). (x ..)) > compute classifier predictions
Dopt: Do + split (Dpr) > create optimization and holdout test set
{(Di D1 ..... (D train )) C' (Vfolds(Dopt)
create cross cross-vlidattion folds
for i= 1....... do
for 1 1 do
Ri + f Hp (H ).ram)) risk, ECE (h)n)
fit for given hyperparameter and training set
compute risk for hn with data D according to Eq. 20
end for end for
Thal + arg minneu -i-, risk; > get the hyperparameter with the smallest average risk
Define Bm (x., y) = [ Nim (x,
Compute CF22 (f) with pi and Die according Eq- 21
return CE2 (f)
versions of the empirical risk in Equation 20 All calibration estimation functions can be seen as preliminary to future rescarch, since may find function classes with lower validation risk by expanding the search space and computational resources. All missing proofs are located in Appendix C
4.1 Reframing Binning and Kernel Density based Calibration Estimators
In this section, we show that the binning estimator TCE of Equation (10) and the kernel density ratio estimator CCE of Equation 11] are the mean prediction of an implicil calibration estimator function of the form
h ff...
(22)
i=l
for some h: Ad x x R and an i..d. dataset (X,, V1) define
(Xn, Y.) Pxy. For the binning-based based estimator,
Abin (p.z))
Σ
(conf (B,..) acc (Bm)) pm
(conf (B,.) acc (B,)) p
(23)
m=1
where I, ....MM acc (Bm),, and conf (Bm) are defined above in Equation
10 It holds that
(TCE)" bin (n) = Abin (X.).
(24)
Similarly, one may formulate the debiasod (but not unbiased) estimator of Kumar al. (2019)
Further, we can also put. the estimator of Popordanoska al.
2022b) in the form of Equation 22 by defining
En ey Kair (X,))
En ey kair (f()))
Axae (p.p') = p- En kair (Xx)) .p)
p
(25)
En kair (f (Xx),))
where kat is the Dirichlet. kernel as defined above in Equation
Again, it holds that
(CCE kde (n)²
1 Σ
hude (f(X),
(26)

---

Published in Transactions on Machine Learning Research (02/2025)
We will also use analogous estimator TCE2 for estimating TCE₂ in the experiment sections. Extending the binning-based estimator to CCE₂ is in general not possible. Further, our calibration evaluation pipeline allows us to compare the risk of different calibration estimators. This motivates the introduction of novel calibration estimators, especially for canonical calibration errors, to have a larger search space to optimize over. In the following, we introduce a novel class of calibration estimators based on kernel ridge regression, which minimizes the empirical risk in Equation (20) plus regularization term under typical kernel ridge regression assumptions.
4.2 Novel Calibration Estimators based on Kernel Ridge Regression
Here, we propose two novel calibration estimators, which are derived as closod-form solutions under the typical kernel ridge regression assumptions, see, e-g-, (Schölkopf & Smola 2002 Bach 2024h. The following approach is based on the notion of ordinary Kronecker kernel ridge regression (Stock et al. 2018 Specifically, we require a reproducing kernel Hilbert space (RKHS) H with an associated feature map di: kye, inner product (.,,),,, and norm I1 Denote with H @ H the tensor product of the Hilbert space with itself, with respective feature map (o @ Φ): 12xx A HH @ H.. Next, assume that (f) - ey, (x) - ev) = (g" (o & o) (p.p')) xo + E for some g E H @ H and mean noise term e. Define the kernel ridge objective for a g H6 @ H via
H,, kernel
CEA
(0)
(100 n
1
+
(27)
where h (p.p') (g. lo () (p
xax. This objective is in essence the risk in Equation 20 plus regular- ization constant. Then, closed-form minimizer can be found, which results in the predictor
hakr pp) vec (Y FAI @ K(ix) + An Xn²1)" (k (p) @ (xx())))))
(28)
where @ becomes the Kronecker product, A YY = (xx ... s(x,) -y.)) E dxx. K {(x) = (e (f (Xx)))) E IR and kyx) (p) = (k (X,, P))eei..) E R.. Without further mod- ifications, computing Equation 288 has runtime complexity (n"") due to the matrix inverse, which is practically infeasible. To reduce the complexity, we classically for Kronecker products make use of the eigenvalue decomposition K ax (laggAA......A... A, 2/po) which is in O (nn)) Define Λsxx E =x" with [Axxl,] X and denote with the Hadamard product. It holds that
hakr (p.r')=1 = kJ f (xx o A fnm
(29)
This representation consists of multiplications of n matrices and in consequence is in (n²)) A more general approach uses the Schur decomposition to achieve such a reduction in complexity (Moravitz Martin & Van Loan 2007). Naively computing the empirical generalization error ÅCE (harr) on an evaluation set (Xi, Yi) ..... X Yir) for some n' x has complexity O (n"), which is again prohibitive in practice. However, can also reduce this complexity to O (nn²) since it holds
BeE (hake) = n' n''n'
(xx - -vv))= - -H 11/ j+i
(30)
with
kkr KJ Q ARX o 2 A y Annx)nnn)))) (xy) KRNAX E R'"w' and [xxxxil ke (f(X)), (X,))
(31)
Alternatively, one may also fit a kernel ridge regressor for the problem assumption =( with g" Ho: 19, and then use the result plug-in for the inner product. This is referred to
9

---

Published in Transactions on Machine Learning Research (02/2025)
0.025
Empirical Risk
0.020
0.015
0.010
10-1
100
10
Figure 1: Simulated experiment for estimating CCE₂ in a task with 5 classes and 500 instances. The empirical risk correctly identifies the ideal calibration estimator with = indicated by the red line. Standard deviations across multiple seeds indicate the empirical risk stability.
two-step kernel ridge regression (tock al. then becomes
2018
U-statistic regression
Park et al. 2021 The model
bukee (p.p') = k six) (p) (Kx) + Xnt)
A ((x) Ay (Kxx)* + Anl) kxx) (p').
(32)
It holds that hukler Marr if the kernel used for hyker incorporates the regularisation constant À or when À = [took et al. 2018 We define the calibration estimators based on kernel ridge regression for CCE₂ as
CCE klr (
kak (f(X),,
(33)
and
CCE2 (/)
Nukee (X(X),,
(34)
7
s=1
We also use analogous estimators TCE and TCE for estimating TCE,
We have now established a large pool of possible calibration estimation functions we can compare and optimize. In the next section, we perform top-label confidence and canonical calibration evaluations with the discussed and proposed estimators. Specifically, comparison and the proposed calibrattion-evalattion pipeline of Section 3.2.2 for calibration estimation.
the proposed calibration estimation risk in Equation (20) for
5 Experiments
In this section, we demonstrate how to use our proposed risk framework in practice. We first run a simulation with known ground truth. Then, we evaluate the risk of the different calibration estimation functions defined in Section 4 and the respective estimated calibration error across a variety of image classification datasets and models. We focus on top-label confidence since is the most prominent, and canonical calibration, which is the most general. The source code publicly available https:://ii thwgg.thhu.....mm how///ee/CCruueeer/gt///m//aa Calibration Estimators
5.1 Simulation
We construct simulation experiment with known ground truth demonstrate how our proposed risk identifies the solution. For this, draw i.i.d. samples Pi,..., Ps00 Dir (a1,..., o5)) from Dirichlet distribution with
10

---

Published in Transactions on Machine Learning Research (02/2025)
Table 1: Validation set square root risk VCE x 100 of TCE2 estimators for CIFAR10 models with optimized hyperparameters. Lower is better. The estimator TCE performs worse and TCE2 better or equal than the other estimators. The large risk of TCE translates to an outlier calibration estimation Figure 2a
Model
LeNet-5
Densenet-40
ResNetWido-32
Resnet-110
Resnet-110 SD
Estimator
TCE 15mbiss
14.96
0.31
6.14 0.12
5.03 + 0.2
5.4 i 0.24
4.73 0.25
TCE.
14.96 + 0.31
6.12 t 0.12
5.03 = 0.2
5.39 + 0.23
4.72 t 0.25
TCE dee
14.98 + 0.31
13.7 + 0.27
12.31 + 0.65
7.46 + 0.35
10.65 + 0.58
TCE kkr
14.96
0.31
6.13 0.12
5.03 + 0.2
5.39
+ 0.24
4.72 0.25
14.96 + 0.31 TCE2 T
6.11 + 0.12
5.03 + 0.2
5.38 + 0.24
4.72 + 0.25
Table Validation set risk V CR x 100 TCE2 for Cifar100 models. Lower is better. Similar as before, the estimator TCE2 performs worse than the other estimators except for LeNet-5, The best performing are TCE2 and TCE2
Model
LeNet-5
Densenet-40
ResNetWide-B!
Resnet-110
Resnet- 110 SD
Estimator
TCD2 15-bins
18.51 + 0.16
20.66 + 0.40
18.40
0.19
18.67 + 0.43
17.18 + 0.20
TCE2 2
hins
18.50 t 0.16
20.43 + 0.38
18.24
0.17
18.61
0.42
17.11 0.20
TCE2
18.50 + 0.16
23.74 + 0.41
23.28
+ 0.18
19.69 t 0.47
18.21 t 0.19
TCE kkr
18.50 + 0.16
20.52 + 0.39
18.29
0.18
18.59
0.42
17.11 0.20
TCE
18.50 + 0.16
20.51 + 0.40
18.28 + 0.18
18.61 + 0.43
17.12 + 0.21
concentration parameters 01 =... 0.04. These samples represent the (in practice unknown) ground truth probability vectors of classification task with 500 instances and 5 classes. Then, sample Y, P for 1,... 500 target labels. We set the hypothetical model predictions to f(,)) softmax 10 log P. for i = 1, 500, which represent miscalibrated predictions. The concentration parameters were set such that the model would have an accuracy of = 90%. We then define calibration estimation function hsim (p. p') (p softmax 100 (00log)) p) softmax Oogg'')) which has single learnable parameter 0 E R referred to as temperature. The cstimation function was chosen such that it matches the ground truth haim h² if and only if 0 1. In Figure 1 we plot the mean results with standard deviations of the empirical risk according to 100 repetitions of the experiment. As can be seen, the empirical risk correctly identifies the correct calibration estimation function with =
5.2 Real World Settings
In the following, evaluate and compare estimators discussed in Section according to novel calibration- evaluation pipeline introduced in Section 3.2.2 standard image classifier setups, like CIFAR and ImageNet.
Technical Setup
The experiments are conducted across several model-dataset combinations, whose logit sets are openly accessible (Kull et al. 2019 Rahimi et al. 2020 Gruber & Bucttner 2022 The image classification datasets
in use are CIFAR10 with 10 classes, CIFAR100 with 100 classes (Krzzhessyy 2009), and ImageNet with
1,000 classes (Deng 2009). Since we restrict ourselves evaluating the calibration error estimate of
11

---

Published in Transactions on Machine Learning Research (02/2025)
Estimator
0.14
TCE - bins
TCE, estimate
0.12
TC
TCE₂ estimate
0.25
TCE TC TCE
0.10
0.20
0.08
0.06
0.15
LeNet-5
Densenet-40
ResNerWide-32
Resnet-110 110
Resnet-110 SD
LeNet-5
Densenet-40
ResNetWide-32
Resnet-110
Resnet-11 110 SD
Model
Model
(a) CIFAR10
(b) CIFAR100
Figure 2: Different TCE, estimates different models. Most calibration estimates approximately agree with
each other. Only TCE: is an outlier for Densenet-40, ResNetWide-32, and Resnet-110 SD. However, also shows increased calibration estimation risk in these cases (c.f. Table 1
102
10
Runtime [s]
100
10-1
10
10-3
TCE
bins
TCE
TCE Estimator
TCE
TCE
Figure 3: Average runtime with error bars of the estimators in Figure 2a the hyperparameters has substantial computational cost.
single CPU thread. Optimizing
the models, we only use the set each dataset (CTFAR: n 10,000, ImageNet: 25,000). Modifying or selecting models based on the calibration estimate would require using the validation instead. The included models are LeNet-5 (LeCun et. al. 1998), ResNet-110, ResNet-110 SD, ResNet-152 (He al.) 2016), Wide ResNet-32 Zagoruyko & Komodakis 2016), DenseNet-40, DenseNet-161 (Huang et al.) 2017 and PNASNct-5 Large (Liu et al. 2018). We did not conduct model training ourselves and refer to Kull et al.
(019 and Rahimi et al. (2020) for further details. We evaluate top-label confidence calibration and canonical calibration estimators for these classifiers.
We run the calibration-cvaluation pipcline proposed in Section 3.2.2 with random split of the original test set, using 80% for tuning the calibration estimator function via crossvvalidation and 20% for the calibration test set Pe, which computes the mean in Equation (1 In all experiments, 5-fold cross-validation to
12

---

Published in Transactions on Machine Learning Research (02/2025)
optimize the hyperparameters of calibration estimator function. For the best performing hyperparameter, the models across all folds are used as an ensemble predictor for the final calibration estimation on the set 2ee- This approach also allows us to include error bars according to the cross validation folds.
As calibration estimator functions, we consider hpins Mxde: Make: and haker for top-label confidence, as well as hkder Nika: and hukke canonical calibration. For both kernel ridge regression models, we use the RKHS of the exp(-r
RBF kernel hybt (z.y) 3) = exp (-> -->||>-p/ - = We set y= = 2 based on preliminary evaluations. We also evaluate hain with 15 bins without hyperparameter optimization, which we refer to as TCE This corresponds to a common default choice in current practice (Guo et al. 2017 Detlefsen et al. 2022 More details on the hyperparameter search spaces are given in Appendix B
Results
We now discuss the experimental results. All reported risks are with respect to the holdout sets in the cross-validaaton folds. The reported calibration estimations are with respect to the calibration test set (20% of the original test set). The error bars for the risk and calibration estimations are the standard errors according to the cross-validation folds. In Table [ we show the performance across different models of CIFAR10, and in Table 2 across models of CIFAR100. Here, compare the calibration estimation functions for top-label confidence calibration. As can be seen, calibration estimation function dominates all others. Specifically, TCE2 performs worst for all models, even when we consider the error bars. The estimator TCE2 outperforms the other estimators, however, the difference is too marginal with respect to the error bars to come to confident conclusion. For the CIFAR100 models, TCE performs more similar to the other estimators. Further, the optimized binning estimator TCE shows the strongest performance and not TCE2 anymore. However, again, the error bars are too large to designate a definitive ranking. Further, for the LeNet-5 model in CIFARIO and CIFAR100, our risk is not sufficiently sensitive to rank the estimators.
In Figure 2 we depict the corresponding TCE2 estimations. As can be seen, only TCE2 is occasionally outlier relative to the other estimators, which is expected based on the reported risk values of Table and Table 2 Even though, we cannot spot direct connection between all risk values and the estimated calibration values in Figure 2 the large risk of TCE is indicative of a worse estimation. However, it is not surprising that differences in the risk do not always translate to differences in the estimated values, since the loss docs not measure in which direction a calibration estimator function gives wrong predictions. Figure 3 shows the wall-clock runtime of each estimator Figure [a] averaged across the classifiers for CIFAR10. Since all estimators only use the classifier outputs, the classifiers have no systematic influence on the runtime. As can be scen, using our optimization pipcline adds substantial computational costs to the calibration estimation. This is analogous to the costs of hyperparameter optimization for conventional machine learning models (Bischl et al. 2023).
In Table 3 we report the risk values of the canonical calibration estimator functions for CIFAR100 classifiers. As can be seen, CCE2 kde performs better than in previous results, outperforming the other approaches in some cases. We can also see that CCE2 performs better than CCE which is a continuous trend across all results. However, the error bars dominate the performance difference and no clear cut conclusion can be made. In Figure 4 we show the respective calibration estimates.
In Appendix B we offer additional results regarding top-label confidence calibration for ImageNet classifiers and canonical calibration for CIFAR10 classifiers. In summary, no calibration estimator outperforms the other approaches across all settings. Additionally, risk performance is often indicative of outlying calibration estimates. This underlines the requirement of a risk to assess which estimator to use for evaluating the calibration of a new model in practice. We may expect to find better estimalors by extending the search space (e.g., by considering different kernels), or by including other model classes, like boosted trees or neural networks (Bishop & Nasrabadi] (2066). However, the proposed risk may not be sufficiently scnsitive to rank the estimators according to their performance. Future research may involve exploring alternative loss functions for more sensitive results.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Table 3: Validation set square root risk CE x 100 of CCE₂ estimators for CTFAR100 models. Again,
lower is better. Contrary previous results, the estimator CCE2 kdl manages to outperform the kernel ridge regression based estimators in some scenarios.
Model
LeNet-5
Densenet-40
RedeetWidde32
Resnet-110
Resnet- Bosnet-110 SD
Estimator
CCE2 kde
8.38 + 0.06
5.61 + 0.09
4.98 i 0.05
5.14 + 0.1
4.79 + 0.03
CCE2 kkr
8.36 + 0.06
5.56 i 0.09
5.00 i 0.05
5.16 0.1
4.80 0.03
CCE2
8.35 i 0.06
5.56 + 0.09
5.01 i 0.05
5.16 + 0.1
4.80 + 0.02
Estimator
0.250
CCE
CCE₂ estimate
0.225
CCE CCE
0.200
0.175
0.150
0.125
LeNet-5
Densenet-40
ResNetWide-32
Resnet-110
Resnet-110 SD
Model
Figure 4: Different CCE₂ estimates for CIFAR100 models. The risk values of Table 3 do not relate to the calibration estimate but only indicate which estimator to trust more (here: CCE kde
6 Conclusion
In this work, we introduced mean-squared error based risk to compare different calibration estimators. This is the first approach in the literature to compare different calibration estimators on real-world datascts. We offer measure theoretic conditions for when the risk identifies an ideal estimator. We also derive novel calibration estimators as closed-form minimizers of the empirical risk based on kernel ridge regression assumptions. Further, using an empirical risk enables to perform hyperparameter optimization and estimator selection a training-valian validation-testing pipeline, similar to conventional machine learning. In the experiments, we optimize the syperparameters of common calibration estimators in the literature popular real-world benchmarks, and compare the risks of different optimized estimators. No dominating calibration estimator was found, which emphasises the requirement of using our risk to detect an appropriate estimator for new settings in practice.
Acknowledgments
The authors are very thankful to Achim Hekler, who computed the logits for the Vision Transformer experiments in this work. Further, the authors would like to thank David Holzmüller, Eugène Berta, and Florian Buetiner for fruitful discussions regarding this work.
The majority of this work was conducted during a research visit at Inria, which was partly funded by a "DKFZ Cancer Research Academy Short-term Research Visit Grant".
14

---

Published in Transactions on Machine Learning Research (02/2025)
References
Francis Bach. Learning Theory from First Principles. MIT Press, 2024.
Herman J. Bierens. Topics in Advanced Bconometrics: Estimation, Testing, and Specification of Cross-section and Time Series Models. Cambridge University Press, 1996.
Bernd Bischl, Martin Binder, Michel Lang, Tobias Piclok, Jakob Richter, Stefan Coors, Janek Thomas, Theresa Ullmann, Marc Becker, Anne-Laure Boulesteix, et al. Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges. Wiley Interdisciplinary Reviews: Data Mining ond Knowledge Discovery, 13(2),44484 2023.
Christopher M. Bishop and Nasser M. Nasrabadi. Pattern Recognition and Machine Learning, volume 4. Springer, 2006.
Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1
3, 1950.
Thomas Capelle. FastAl Timm. http:://ftthho.cccm///eeellaaaaa http://wwww..aaaaaooooaaaaaaaaar httis///iaiaimllooo/aiiiiiiieeee py: 2022.
Marck Capiñiski and Peter Ekkehard Kopp. Measure, Integral and Probability, volume 14. Springer, 2004.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cuuxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1-45 45, 2024.
Mostafa. Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Pe- ter Steiner, Mathilde Caron, Robert Geirhos, and Ibrahim Alabdulmohsin. Scaling vision transformers 22 billion parameters. In International Conference on Machine Learning, pp.
2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, pp. 248 148-225,,4- 2009.
Nicki Skafte Detlefsen, Jiri Borovec, Justus Schock, Ananya Harsh Jha, Teddy Koker, Luca Di Liello, Daniel Stancl, Changshong Quan, Maxim Grechkin, and William Falcon. Torchmetrics mcasuring reproducibility in pytorch. Journal of Open Source Software, 7(70):4101, 2022.
Alexey Dosovilskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Bradley Efron. An Introduction to the Bootstrap. CRC press, 1994.
Hongxiang Fan, Martin Ferianc, Zhiqiang Que, Xinyu Niu, Miguel L. Rodrigues, and Wayne Luk. Accelerating Bayesian neural networks via algorithmic and hardware optimizations, IEEE Transactions on Parollel and Distributed Systems, 3311))33387 3399, 2022.
Di Feng, Christian Haaso-Schützz Lars Rosenbaum, Heinz Hertlein, Claudius Glaesor, Fabian Timm, Werner Wiesbeck, and Klaus Dietmayer. Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Transactions Intelligent Transportation Systems, 22(3):1341 111:1111111111 1360, 2020.
Halina Frydman, Edward 1. Altman, and Duen-Li Kao. Introducing recursive partitioning for financial classification: the case of financial distress. The Journal of Finance, 20(1):269-291, 1985.
Tilmann Gneiting and Adrian E. Raftery. Weather forecasting with ensemble methods. Science, 310:248 249, 2005.
15

---

Published in Transactions on Machine Learning Research (02/2025)
Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Sociely Series B: Statistical Methodology, 69(2):243 2007.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Sebastian G. Gruber and Florian Buettner. Better uncertainty calibration via proper scores for classification and beyond. In Advances in Neurol Information Processing Systems, 2022.
Scbastian G. Gruber, Teodora Popordanoska, Alcksci Tiulpin, Florian Bucttner, and Matthew B. Blaschko. Consistent and asymplotically unbiased estimation of proper calibration errors. In International Conference on Artificial Intelligence and Statistics, pp. 3466 3474, 2024.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330, 2017.
Ashim Gupta, Giorgi Kvernadze, and Vivek Srikumar. Bert & family eat word salad: Experiments with text understanding. ArXiu, abss/2101.03453 2021.
Chirag Gupta. and Aaditya Ramdas. Top-label calibration and multiclass-to-binaryy reductions. In International Conference on Learning Representations, 2022.
Sarah Haggenmüllerr, Roman C. Maron, Achim Hekler, Jochen S. Utikal, Catarina Barata, Raymond L. Barnhill, Helmut Beltraminelli, Carola Berking, Brigid Betz-Stablein, Andreas Blum, Stephan A. Braun, Richard Carr, Mare Combalia, Maria-Teresa Fernandez-Figueraas, Gerardo Ferrara, Sylvie Fraitag, Lars E. French, Frank F. Gellrich, Kamran Ghoreschi, Matthias Goebeler, Pascale Guitera, Holger A. Haenssle, Sebastian Haferkamp, Lucie Heinzerling, Markus V. Heppt, Franz J. Hilke, Sarah Hobelsberger, Dieter Krahl, Heinz Kutzner, Aimilios Lallas, Konstantinos Liopyris, Mar Velasco, Josep Malvehy, Friedegund Meier, Cornelia S.T. Müller, Alexander A. Navarini, Cristián Navarrete- Antonio Perasole, Gabriela Poch, Sebastian Podlipnik, Luis Requena, Veronica M. Rotemberg, Andrea Saggini, Omar P. Sangueza, Carlos Santonja, Dirk Schadendorf, Bastian Schilling, Max Schlaak, Justin G. Schlager, Mildred Sergon, Wiebke Sondermann, H. Peter Soyer, Hans Starz, Wilhelm Stolz, Esmeralda Vale, Wolfgang Weyers, Alexander Zink, Eva. Sroghhoff-Hennnng Jakob N. Kather, Christof von Kalle, Daniel B. Lipka, Stefan Fröhling, Axel Hauschild, Harald Kittler, and Titus J. Brinker. Skin cancer classification via convolutional neural networks: systematic review of studies involving human experts. European Journal of Concer, 156: 202-216, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pottern Recognátion, pp. 770-778, 2016.
Achim Hekler, Titus J. Brinker, and Florian Buettner. Test time augmentation meets post-hoc calibration: uncertainty quantification under real-world conditions. In AAAI Conference on Artificial Intelligence, pp. 14856-14864, 2023.
Gao Huang, Zhuang Lin, Laurens Van Der Maaten, and Kilian Q. Weinborger. Densely connected convolutional networks. In Conference on Computer Vision and Pattern Recognition, pp. 4700 4708, 2017.
Ashraful Islam, Chun-Fu Chen, Rameswar Panda, Leonid Karlinsky, Richard J. Radke, and Rogério Schmidt Feris. A broad study on the transferability of visual representations with contrastive learning. International Conference on Computer Vision (ICCV), pp. 8825 8835, 2021.
Taejong Joo, Uijung Chung, and Minji Seo. Being Bayesian about categorical probability. In ICML, 2020.
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig, Being Bayesian, even just bit, fixes overconfidence in relu networks. In TCML, 2020.
Alex Krizhevsky, Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
Volodymyr Kuleshov and Shachi Deshpande. Calibrated and sharp uncertainties in deep learning via density estimation. In International Conference on, Machine Learning, pp. 11683-111111 11693, 2022.
16

---

Published in Transactions on Machine Learning Research (02/2025)
Meclis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in Neural Information Processing Systems, 32:12316 12326, 2019.
Ananya Kumar, Percy Liang, and Tengyu Ma. Verified uncertainty calibration. In Advances Newral Information Processing Systems, pp. 3792-3803, 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document rocognition. Proceedings of the IEEE, (6(11):227-2333 1998.
Chenxi Liu, Barret. Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European conference computer vision (ECCV), pp. 19 19-34, 2018
Charlie Marx, Sofian Zalouk, and Stefano Ermon. Calibration by distribution matching: Trainable kernel calibration metrics. Advonces in Neural Information Processing Systems, 36, 2024.
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In TCML, 2021.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Pablo Morales- Morales-Alvarez, Daniel Hernández-Lobato, Rafael Molina, and José Miguel Hernández-Lobato. Activation-lovel uncortainty in deep noural networks. In ICLR, 2021.
Carla D. Moravitz Martin and Charles F. Van Loan. Shifted kronecker product systems. SIAM Journal on Matric Analysis and Applications, 29(1):184-198, 2007.
Allan H. Murphy. A new vector partition of the probability score. Journal of Applicd Meteorology and Climatology, 12(4):595 600, 1973.
Allan H. Murphy and Robert L. Winkler. Reliability of subjective probability forecasts of precipitation and temperature. Journal of the Royol Statistical Society. Series C (Applied Statistics), 26(1):41-47, 1977.
Kevin P. Murphy. Probabilistic Machine Learning: An Introduction. MIT Press, 2022.
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 2901 2907, 2015.
Jeremy Nixon, Michael W. Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. In CVPR Workshops, volume 2, 2019.
Frédéric Ouimet and Raimon Tolosana-Delgado. Asymptotic properties of dirichlet kernel density estimators. Journal of Multivariate Analysis, 187:104832, 2022.
Junhyung Park, Uri Shalit, Bernhard Schölkopf, and Krikamol Muandet. Conditional distributional treatment effect with kernel conditional mean embeddings and u-statistic regression. In International Conference on Machine Learning, pp. 8401-8412, 2021.
Kanil Patel, William H. Beluch, Bin Yang, Michael Pfeiffer, and Dan Zhang. Multi-class uncertainty calibration via mutual information maxinization-based binning. In International Conference on Learning Representations, 2021.
Teodora Popordanoska, Raphael Sayer, and Matthew B. Blaschko. A consistent and differentiable L, canonical calibration error estimator. In Advances in Neural Information Processing Systems, 2022a.
Teodora Popordanoska, Raphael Sayer, and Matthew B. Blaschko. A consistent and differentiable L, canonical calibration error estimator. In Advances in Neural Information Processing Systems, 2022b.
17

---

Published in Transactions on Machine Learning Research (02/2025)
Amir Rahimi, Amireeza Shaban, Ching-An Cheng, Richard Hartley, and Byron Boots. Intra order- preserving functions for calibration of multi-class neural networks. Advances in Neural Information Processing Systems, 33:13456 13467, 2020.
Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C. Mozer. Mitigating bias in calibration error ostimation. In International Conference on Artificial Intelligence and Statistics, pp. 4036-4054, 2022.
Bernhard Schölkopf and Alexander J. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, 2002.
Jun Shao. Mathematical statistics. Springer Science & Business Media, 2003.
Michiel Stock, Tapio Pahikkala, Antti Airola, Bernard De Baets, and Willem Waegeman. A comparative study of pairwise learning methods based on kernel ridge regression. Neural Computation, 30(8):2245-222) -2883,, 2018.
Zeyu Sun, Dogyoon Song, and Alfred Hero. Minimum-risk recalibration of classifiers. Advances in Newral Information Processing Systems, 36, 2024.
Gerald Teschl. Mathemalical Methods in Quantum Mechanics, volume 157. American Mathematical Soc., 2014.
Junjiao Tian, Dylan Yung, Yen-Chang Hsu, and Zsolt Kira. A geometric perspective towards neural calibration via scnsitivity docomposition. In NewrIPS, 2021.
Christian
Tomani, Sebastian G. Gruber, Muhammed Ebrar Erdem, Daniel Cremers, and Florian Buettner. Post-hoc uncertainty calibration for domain drift scenarios. In Conference Computer Vision and Pattern Recognition (CVPR), pp. 10124-10132, June 2021.
Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification, In International Conference on Artificial Intelligence and Statistics, pp.
2019.
Xiao Wang, Hongrui Liu, Chuan Shi, and Cheng Yang. Be confident! towards trustworthy graph neural networks via confidence calibration. In NewrIPS, 2021.
David Widmann, Fredrik Lindsten, and Dave Zachariah. Calibration tests in multi-class classification: A unifying framework. Advances in Neural Information Processing Systems, 32:12257-12267, 2019.
David Widmann, Fredrik Lindsten, and Dave Zachariah. Calibration tests beyond classification. In Interna- tional Conference Learning Representations, 2021.
Jiancheng Yang, Rui Shi, and Bingbing Ni. Medmnist classification decathlou: A lightweight automl benchmark for medical image analysis. In International Symposium on Bioreddical Imaging, pp. 191-195, 2021.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. International Conference on Knowledge Discovery and Data Mining, pp. 694 699. Association for Computing Machinery, 2002.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference, 2016.
18

---

Published in Transactions on Machine Learning Research (02/2025)
A Overview
Here, we first give additional experimental content in Appendix The missing proofs are located Appendix C
B Extended Experiments
In the following, we discuss further experimental details and results. Specifically, we evaluate the calibration of VisionTransforme classifiers (Dosovitskiy et al., 2020) trained on McdMNIST datasets (Yang et al. 2021 according to approach.
Additional Details
We use the implementation of the calibration estimator function /idde given by the original authors (Popor- danoska et al. 20226). For a small fraction of imputs, this implementation returns NaN as the prediction. We remove these instances from the risk and calibration estimation calculation of lidee which has a neglectable effect.
As hyperparameter search spaces for the TCE experiments, consider {5i i i= 1,..., 20} for the number of bins in hpins a bandwidth in (1--
l,..., 15 i == ......5) for the Dirichlet kernel of hide according to PPooordanska et al. (2022a), a regularization constant E {n 0.5 102+1
1..... 9) for hARE and À E {n² 100 - .9....99 9 for haker For the CCE experiments, we consider the same set of bandwidths for the Dirichlet kernel of hude: a regularization constant E fn { 10 i= 1,....118) 18] for NRkes and À E fnz 0.5 100-50555 i 1, 1,...,18,, 18) for hukkr-
All experiments are run on an Intel(R) Xeon(R) Gold 5218R with 2.1 GHz and Macbook Pro M1.
Additional Results
In the following, we discuss the risks and calibration estimations of some cases left out the main paper and VisionTransfon Transformer classifiers.
In Table 4 we show the risk of the top-label confidence calibration estimators for ImageNet with various models. All calibration estimation functions show similar risk except TCE which is worse for DenseNet- 161 and Resnet-152. This is in agreement with Figure 26 where the estimated calibration values are also mostly similar. The risks in Table 5 for canonical calibration estimators in the case of CIFARIO show slightly different results: Here, the risk fails to distinguish the performance between the different estimators. Only CCE outperforms the other approaches for Resnet-110.
We train the Vision Transformer architocture (Dosovitskiy et al. 2020 on the MedMNIST classification datasets Blood, OCT, and Derma (Yang et al. (0221) Specifically lly, Huggingfaed and fino-tune with a modification of (Capelle 2022). The respective risk values are in Table 6 The kernel density estimator performs worse than the other approaches. No definitive winner can be declared due to the wide error bounds. In Figure 0 we show the associated calibration estimates. Similar to before, the risk values do not link to the calibration estimates.
pre-trained classifier from
In summary, the results mimic the ones in the main paper and it is not apparent which estimator to use in practice without considering our proposed risk. However, the risk may be insensitive regarding the various estimator performances.
http:://muggingffccc.c//ttaaiiiu wawppttthsseeephhh___ea 224. (iiiii_inik__f______ ftninik (Accessed on 2nd Jan 2025)
19

---

Published in Transactions on Machine Learning Research (02/2025)
Table 4: Validation set square root risk C x 100 of TCE₂ for different ImageNet models. Lower is better. The estimator TCE2 performs worse than the other estimators for DenseNet-161 and ResNet-152 152. For Pnasnet-5, all estimators perform similarly.
Model Estimator
DenseNet-161
Resnet- 152
Pnasnet-5
TCE2 15-bins
12.17 + 0.16
12.58 + 0.14
10.63 + 0.16
TCE2
bins
12.17 + 0.16
12.57 + 0.14
10.63 t 0.16
TCE bde
12.31 + 0.17
12.77 + 0.14
10.63 t 0.16
kkr
TCE bkr
12.17 # 0.16
12.37 # 0.14
10.63 t 0.16
TCE
12.17 # 0.16
12.57 + 0.14
10.63 + 0.16
Table 5: Validation set risk ICE 100 of CCE₂ for CIFARIO models. Lower is better. All estimators perform similarly.
Model
LeNet-5
Densenet-40
ResNetWidd-32
Resnet-110 Resnet-110 SD
Estimator
CCE
13.53 t 0.27
5.13 - 0.13
4.22 0.16
4.46 + 0.14
3.89 + 0.2
CCF2 kkr
13.53 + 0.27
5.13 + 0.13
4.22 + 0.16
4.47 + 0.14
3.89 + 0.2
CCEaker
13.53
0.27
5.13 0.13
4.22 0.16
4.47 # 0.14
3.89 + 0.2
C
Missing Proofs
Here, we present the missing proofs of the main part. Specifically, we prove Theorem in Section C.1 Theorem 2 in Section C.2 and various statements of Section 3.2 in Scction C.3
C.1 Proof for Theorem
We show that RCr (h) > CCF (h*).
For this, require that (p p Pvv) is the unique minimizer of LCE (,, Py Pv:p.p)), which holds since
LOB (c, @ Pvip.p)
de
2
Ey. [[c - ey ey.
de
(35)
= 2Ey, [|c- - (p- ey, -vv
= = - (p- P Pv)).
8² and 8. LCB Pc @ Pv:p.p')
Based on the assumption that 34 E & (()) with FX (A) > 0 we have
Vp.p' E A: h (p.p') p') = (p,p')
= Vp.p's E A: Lew (1 (p. p'), PYISIXEP @ PrISI LeR (hi (p,p). Prinx @ Prisixpw:PPp))
San LCE(H (n ( @ FrINxYwiPP
(36)
AnA
> > And
Loe (hi (p.r'), PYINX @ () P (p.))
20

---

Published in Transactions on Machine Learning Research (02/2025)
Estimator
Estimator
TCE birs TC TCE TC TCE
CCE
TCE₂ estimate
0.075
CCE₂ estimate
0.08
CCE
CCE
0.070
0.06
0.065
0.04
0.060
0.02
DenseNet-161
Resnet-152
Pnasnet-3 5
LeNet-5
Densenet-40
ResNetWide-32 32
Resnet-110
Resnet- Resnet-110 SD
Model
Model
(a) TCE2 ImageNet
(b) CCE₂ CIFARIO
Figure 5: Different calibration estimates of different models. Most calibration estimates approximately agree with each other. This is in agroement with the similar risk values for each estimator in Table 4 and Table 5
Table 6: Validation set risk ICE x 100 of TCE, estimators for VisionTTrasten models various McdMNIST datasets. Lower is better. The error bounds are too wide to make definitive statement about which estimator is best. Contrary, we can claim that the kernel density estimator performs worst on the Blood and Derma datasets.
Dataset
Blood
Derma
OCT
Estimator
TCE I 1ō-bins
0.54 t 0.12
6.0 0.42
5.9 + 0.94
TCE bins
0.52 t 0.12
6.0 + 0.42
5.87 + 0.93
TCE kde
0.96 t 0.23
7.78 0.58
5.96 + 0.95
TCE2 kkr
0.52 + 0.12
5.99 + 0.41
5.87 + 0.94
TCE ukke
0.52 t 0.12
5.99 + 0.42
5.87 t 0.94
where the inequality follows since h" (p.p') = (p-PPIsx)---1' -P(((X)=)) is the unique minimizer of Lee (y,Prix @ P.PP). From the unique minimizer property also follows that for all B E & {(}]s(x) holds
J LeE (h(p,r'), PYISIII Pynn (p))) B
(37)
z 5 LeR (p.r', PrIs(xX) PYINx PPP (p.x))..r))
Since AA x 1xx (AA E [x))s(x)) it holds
21

---

Published in Transactions on Machine Learning Research (02/2025)
Estimator
0.12
TCE bins
0.10
TCE₂ estimate
TCE
0.08
TCE
TCE
0.06
TCE
0.04
0.02
0.00
Derma
OCT
Blood
Dataset
Figure 6: Different top-label confidence calibration estimates of VisionTraassormee models trained MedMNIST datasets. Most calibration estimates approximately agree with each other. Like in previous instances, this demonstrates that the calibration estimate is not directly linked to the respective risk value in Table
RCr
= Ex.x (Le PYIAX @ PYISIXENPPP))
ICE (p..)) =Sarrasuun
РУIЛI)= Pylnemann))) (xx))))))))
+ + LCE (h (p. ()) PrIS() ())) (nx) (p.p))
(38)
(a*xaT[(XX))) LCr (h" (p. pp. PYIS(M @ PYInx)/ (Ix] @ (p.x))
+ " AxA
LCE (h" Do.P')Prrixxx, @ P @ p (p.p')
Ex [T (hi (p,r'), PYI(X)1 /P
ECE (
C.2 Proof for Theorem 2
A sketch of the necessity of Theorem is given in Figure 7 which also illustrates the proof.
(p,ii)) p.p' E Py for simplicity. It continuous
Proof. We use == (X) and I (p.p') at every point in which h and h" continuous. We denote with D the Pull set of p's for which h and h*
0.
clse,
22

---

Published in Transactions on Machine Learning Research (02/2025)
A
Figure 7: Blue indicates a possible support set supp (Pp @ Pp) S During training we optimize all blue dots and areas, but during testing we only evaluate their intersection with the rod line (a possible mull sct).
A A"xA", and red line indicates ((p.)) p) 1p E a").
are not continuous at point (p,p). Then,
Ep [p[((P,))
S f (p.
S p)
f (p. dPP (p)
(39)
(p. p) dP (p) + f
f p) dPp (p)
intsupp(Pp) p)
bd sppp(pp))
sutpup(e))) f (p. p) dP (p) S
bdsspp((p)) f (p. p) dP (p).
The last line holds since the support of measure is closed, and, consequently, splitting it up into in- terior and boundary does not add new 'mass' (Teschl 2014). For the following, denote with B (p. c) = for R² ||æ pll2 <)) the open (euelidean) ball with center and radius e. Further, since h and h" are by assumption continuous in the points ((p,p) p E By \ D), follows that ʃ is also continuous at these points, and, consequently, also lower semicontimuous. We first deal with the interior term by using this lower-semicontinouus property giving
In Intsupooonn f (p, p) dIP (p)
Inttmpp((p)))D I (prpp))-ppp))
liminf I (P1..P2
= lim intsupp(PpllD
inf in (p), ((p.p),.
(40)
c-
< lim c0 intsupp(Pp))
essinf (f (2). B(p. p). d) (p. (p.)))) (p)
Since E int supp (Pp). it holds Pp (B \ (rH) 0 for all > 0 following the definition of int and supp
Tesch1 2014 Let. Sq (p, = for Id ||lp u|lo < < be the open hypercube with center p and side length 2c.
23

---

Published in Transactions on Machine Learning Research (02/2025)
It holds (p, c) c (p, and Sq vdc) B B(p c). Then
< p B p. /2de) \p)
$ p (sq p. V2dc \ {p))
P p @ Pr *>((sq (p. V2de) (p)) (sq V2de) (o)))
(41)
Pr @ Pr ((5(( p. V2dc) x Sq V2de)) K(p, pl))
$
Pp @ Pp (Sq (p. m), 2dd ((p.))))
=
$ p @ PB (1p p). v) ((.,))))
Consequently, B ((p, p), <) ((p,p) p)} is not null set (w.r.t. Pp @ Pp), and, thus, ess inf (ʃ (P1, P2) (21.P2) E B ((v, p) t) ((p,p)))
f Mll. f(pr.P2)d(P,, PPp))
(42)
$ f mllm.)).))) (F e PP))pppp
s [[ (P, =0)
where we used the given assumption h (P, P") - h (P,PP) PP)
in the last line. Continuing Equation (40) it
follows
lim e0 suts
essinf (f (p1: P2) (pI, P2) E ((p.p)) ((p.))) dPp (p) 0.
(43)
Next, we deal with the boundary of the support. Since assume that it consists of at most countably infinite elements with probability mass (which we denote as (pI...... it holds
f Inatanppte m))
f (p.
I (p. dP
[ pelp.....)
(p.p)
Σ
Σ
1 (p.)))
(44)
pelp p'Elpi....
p (Pp /Pp) (p.r')
(p Mpp...)
(p. Pp vPP (v.2')
$ { gd (v.r) @ PP (0.0') (p.p'))
= 0,
where the last line holds since for all A PXP we have PPp @ /Pp (A)
Pp Pf(( =.
From Equation
43 and Equation 44 follows that f P') = 0 = f(P,)) 6.5. 0. Consoquently, we have
h (P,, P') a.c. h" (P.P') (P h" (P, P).
24

---

Published in Transactions on Machine Learning Research (02/2025)
Remark. Note that any random variable with outcomes restricted to A (PI.....Pa)) e spect to the dimensional Lebesgue measure X². This would then foll outside of the conditions stated in Theorem However, we can cireumvent this simply by transforming (bijectively) the outcome space to A ((P.....,PP..)) E [0, 10s E Pi s1] R which has non-zero mass according to the d dimensional Lebesgue mensure d-1
g has singular distribution, since A is null set with re-
C.3 Proofs for Section 3.2
We give proofs of various statements of Section 3.2
Proof for Equation (27
Instead of proving the whole kernel ridge regression approach end-to-end, bring Equation form of an ordinary kernel ridge regression objective and then show that solution in Equation the ordinary solution.
27
into the
28 matches
For this, define y, vec; A xxx) AYRx) = (Xx mod n+1) - EY, mod mo1 ff (Xpijma) - -Expuml E and F; =
(X; mod f (X(x)) A d x A with i = = 1...n, and R (Ø) (2). for A x A
well
H H @ H with ( (o Φ) (f1, P2).
Then, we can write Equation 27 as
(g)
- -xx
+ Alllll
=1 5=
(45)
I (ö. σ (F:)
+
which is ordinary kernel ridge regression in the last line is reached under certain assumptions if
Bach
2024
Bach 2024
shows its unique minimum
g= (Y
(Rx)) >n²1) (6())......
(46)
with [K (xxik ( Now, to reach our solution, note that it holds (((F)),(()))
le If (X; mod (1)) nn1), (f (Xp.m)) P2) and K K KYty, which gives
(p))
= (P Pre) (K [Kxx) + An²1)" (&
(1 (X, mod n+1), .t)) & (Xp1mm) ........ ,. (f mod n+1) .p)) (f (Xpnema) ,2)) veo (K) ((x) K + Xn² (k (Pr) @ k k(xx) (7z))
(47)
The last line the predictor we stated in Equation
28
Proof for Equation (29
By definition of havkr and by using the eigenvalue decomposition KYxi = 11x
fxy we have
25

---

Published in Transactions on Machine Learning Research (02/2025)
Mykr (p.p') p¹)
= -vec" (1rxx)ARN ( @ 1(x)+ + >n²1)" (kyix) (p) @ (() s())
voc'
(Arrii (2Pxx) @ Q(x)) sA
(A @ Axx) + An 2n²1)= (g) (x)
@ Q (kx (x) f(x (2'))
= =((e)1x) @ (111))) vec م ((x))) (Amx)
@ Axx An -1 (0) |x,]
o (10))) (p) @ k
@ (p')
(48)
(B) Note it holds that (A @ (3) vec (C) vec (BCA') for matrices A, B, C and vec (A) vec (B) = tr ().
Then, with the Hadamard product o and Åx e "x= with [A [Axxl (Anxilalssx)), +2m= we have
(10m mi) @ ((ixi) vec ( n Ar.cx))) (Msc)) @ Axx) + xn xn21) ( @ ( (k (x) @ kx(x)(())
/xx) Σ vec; (o) ((x) S(X A voc; (en (p) (x) (D') ((x) (1x))
2
(49)
ir ( (v) sim| (D') Q ( (()) O Q jix ANY
Ax Ax
= =k /k (1'm((')) ( (ΛΛι)) o A A100))))))))) (1n (v),
which shows Equation 29
Proof for Equation (30)
Given the definition of Hkke we have
H [K] 9 1 AXIIIY ERRiX
(50)
=k nix (s (x:)) (Ax /ww. (xxx)) E Ra'nn
where the last line follows since [Kxt] (xx))], = =[K])]
26


Reviews:
[{"review_id": "KX9g5tFp7M", "reviewer": "Reviewer_3Qdv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I enjoyed reading the paper and found the idea of evaluating calibration estimators interesting, and the proposed solution makes sense.\n\nWeakness:\nI do not have any major complain for the work. The only thing that I think could be improved is the flow of the paper, especially for Section 4, where the first part of section 4 is unifying some existing calibration functions as the optimal solutions of the risk, and the second part talks about new calibration functions that come from close form optimal solutions of the risk. The title of Section 4 does not provide a good summarization of the Section, at least for me.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a mean-squared error based risk to compare calibration estimators. The main results are a collection of theoretical results for the newly proposed risk, which includes properties of the optimal solution to the proposed risk, a practical framework for finding a better calibration estimators using the proposed risk function, a unified view of binning TCE and kernel density CCE as the optimal solutions of the risk function under different conditions, and some new calibration estimators inspired by the close-form solutions of the risk function under certain assumptions. The paper also provide synthetic data and real-world data to show that the proposed risk function can indeed compare the performance of calibrations estimators.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Nothing major, see weakness.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0kMztVyhkU", "reviewer": "Reviewer_mJox", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Well-written and clear motivation and explanations. Mathematical notation is clear.\n- Sec. 2 provides a summarization and framework for calibration metrics and estimators.\n- Proposed method is mathematically justified and captures a fuller notion of calibration than metrics used in the literature.\n\nWeaknesses:\n- My biggest question is about computational complexity and runtime. As a practitioner who wants to quantify calibration, what is the cost of using this method over other more simpler metrics of calibration?\n- While evaluations seem thorough, it would be nice to see experiments with more recent transformer-style architectures.\n- Overall, the results in tables and figures are not striking, but for the purposes of a venue like TMLR is ok for me.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a novel framework to optimize and compare calibration error estimators for classification tasks\nMy understanding is that the authors are proposing a richer calibration estimator which is more faithful to the canonical calibration estimator. This is in contrast to other more simpler versions of calibration like top-label (confidence) calibration error, which is the dominant notion of calibration in the literature.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Minor:\n- Page 3, under Sec. 2.3: \"degree a classifier\" should be \"degree to which a classifier...\"\n- Results in Fig. 2 should have higher dpi.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "da9Ngcz1NZ", "reviewer": "Reviewer_5D5w", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* The paper introduces a mean-squared-error based risk framework that enables evaluation and optimization of squared calibration error estimators in real-world settings.\n* It also leverages the bilinear structure of squared calibration errors and reformulates the calibration task to a regression problem. The paper also introduces kernel ridge regression estimators for the calibration task.\n* The paper benchmarks existing calibration estimators against the proposed kernel ridge regression based estimators on standard image classification tasks and illustrates the benefits of the proposed calibration framework in practice.\n* The proofs, to the best of my knowledge, seem kosher.\n\n**Weaknesses**\n\n* The presentation although technically precise could benefit from greater accessibility. Currently the paper is tailored to an expert audience well versed into the issues and approaches on calibration of classification models. Including a motivating example to illustrate the nature of the problem and the role of squared calibration errors would be helpful to the readers as in other papers (e.g., Vaicenavicius et al., 2019; Gupta et al., 2022).\n* Related to the above, currently the way the pipeline (Section 3.2.2) is presented does not provide sufficient clarity to practitioners in terms of how the proposed strategy can be readily adopted. It would be beneficial for the practitioners if the components can be outlined in an algorithm form, so that input/steps can be followed practically.\n* The paper addresses the computational complexity of various estimators and outlines strategies to mitigate these challenges in practical scenarios. However, including runtime comparisons for the real-world settings examined would significantly enhance the discussion. Concrete benchmarks would provide readers with a clearer understanding of the trade-offs between computational efficiency and estimator performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel framework to enhance the calibration of classification models, a topic that has gained increasing interest in recent literature. The paper proposes achieving improved calibration by directly optimizing estimators of squared calibration errors. This approach not only aims to refine the calibration process, but also can offer practical guidance on selecting and tuning these estimators to enhance their effectiveness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the **Weaknesses** in the section above. Overall, this is a solid contribution to the literature, once the weaknesses mentioned are addressed.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, all minor concerns were addressed by the authors during the discussion period, and all reviewers were unanimous and decisive on accepting the paper to TMLR. Congratulations to the authors on a solid contribution!
Decision: Accept (poster)