Confidence Calibration of Classifiers with Many Classes
Authors: Adrien Le Coz
Stéphane Herbin
Faouzi Adjed
Venue: NeurIPS 2024 poster (2024)
Keywords: Calibration
Classification
Uncertainty quantification
Primary area: safety_in_machine_learning
TL;DR: We transform the problem of calibrating a multiclass classifier into calibrating a single surrogate binary classifier and show that it significantly improves existing calibration methods.

Abstract: For classification models based on neural networks, the maximum predicted class probability is often used as a confidence score. This score rarely predicts well the probability of making a correct prediction and requires a post-processing calibration step. However, many confidence calibration methods fail for problems with many classes. To address this issue, we transform the problem of calibrating a multiclass classifier into calibrating a single surrogate binary classifier.  This approach allows for more efficient use of standard calibration methods. We evaluate our approach on numerous neural networks used for image or text classification and show that it significantly enhances existing calibration methods.

Confidence Calibration of Classifiers with Many Classes
Adrien Le Coz Coz
2,3
Stéphane Herbin 2.3 Faouzi Adjed'
IRT SystemX
ONERA DTIS 3 Paris-Saclay University
adrien. 2wrvbepass iinb:xxsssiibb...... com
stephane herbin@onera.fr faouzi. aejiedddddddrrr-sssteee..
Abstract
For classification models based neural networks, the maximum predicted class probability is often used confidence score. This score rarely predicts well the probability of making a correct prediction and requires post-processing calibration step. However, many confidence calibration methods fail for problems with many classes. To address this issue, we transform the problem of calibrating multiclass classifier into calibrating single surrogate binary classifier. This approach allows for more efficient use of standard calibration methods. We evaluate our approach on numerous neural networks used for image or text classification and show that it significantly enhances existing calibration methods. Our code can be accessed at the following link: atttp:///githu com/ wwwaaallgllllaaeeee caw.rratalibiiiinnn
Introduction
The considerable performance increase of modern deep neural networks (DNNs) and their potential deployment in real-world applications has made reliably estimating the probability of wrong decisions key concern. When such components medical or transportation), estimating this probability is crucial to mitigate catastrophic behavior. One way address this question treat it as an uncertainty quantification problem [2 [2), where the uncertainty value computed for each prediction is considered as confidence. This confidence can be used to reject uncertain decisions proposed by the DNN [3], for auttof-distribution detection [22], or to control active learning (34] or reinforcement learning based systems (76). When confidence values reliably reflect the true probability of correct decisions, i.e., their accuracy, a predictive system said to be calibrated. In this case, confidence values can be used as a reliable control for decision-making.
expected be embedded in safety-critical systems (e.g..
We are interested in producing uncertainty indicator for decision problems where the input is high dimensional and the decision space large, typically classifiers with tens to thousands of classes. For this kind of problem, DNNs are common predictors, and their outputs can be used to provide an uncertainty value at no cost, i.c., without necessitating heavy estimation such as Bayesian sampling 15 or ensemble methods 33 Indeed, most neural architectures for classification instantiate their decision as softmax layer, where the maximum value can be interpreted as the maximum of the posterior probability and, therefore, confidence. Unfortunately, uncertainty values computed in this way are often miscalibrated. DNNs have been shown to be over-confident [17]. meaning their confidence is higher than their accuracy: predictions with 90% confidence might be correct only 80% of the time. A later study (44) suggests that model architecture impacts calibration more than model size, pre-training, and accuracy. For ImageNet classifiers, the accuracy and the number of model parameters are not correlated to calibration, but model families are
These studies show that it difficult anticipate the calibration level of confidence values computed directly from DNNs and exhibit the benefits of a complementary post-processing calibration. This calibration process can be seen as learning step that exploits data from a calibration set, distinct from the training set, and is used to learn a function that maps classifier outputs into calter-calibrated values. This process is typically lightweight and decoupled from the issue of improving model
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

performance. A standard baseline for posteerresssing calibration is Temperature Scaling [17] where the penultimate logit layer is scaled by a coefficient optimized on the calibration set.
Many post-processing calibration methods have been developed for binary classification models Applying these methods multiclass classifiers requires some adaptation. One standard approach reformulates the multiclass setting into many One-versus-Ar Alll binary problems (one per class)
550
69 70
70 One limitation of this approach is that it does not scale well. When the number of classes
is large, the calibration data is divided into highly unbalanced subsets that do not contain enough positive examples to solve the -ell-eruus-Al binary problems. Other methods based on Platt scaling [50] involve learning set of parameters whose size grows with the number of classes. For problems with many classes, they tend to overfit, as we demonstrate in this work.
The main idea of our work is to reformulate the multiclass confidence estimation into a single binary problem. This problem can be phrased as the unique question: "Is the prediction correct?". In this formulation, the confidence score is defined as the maximum class probability of the binary problem that outputs 1 if the predicted class correct and 0 otherwise. The intent is that the confidence score accurately describes whether the prediction is correct, regardless of the class, We show that this novel approach, which we call Top-versus-All (TvA), significantly improves the performance of standard calibration methods: Temperature and Vector Scaling [17]. Dirichlet Calibration [29], Histogram Binning [69]. Isotonic Regression 70,, Beta Calibration [28] and Bayesian Binning into Quantiles [6]. We also introduce a simple regularization for Vector Scaling or Dirichlet Calibration
that mitigates overfitting when the number of classes is high relative to the calibration data size. We
conduct experiments on multiple image and text classification datasets and many pre-trained models.
Our main contributions are the following:
We discuss four issues of the standard approach to confidence calibration.
To solve these issues, we develop the Top-versus-AlI approach to confidence calibration of multiclass classifiers, transforming the problem into a single binary classifier's calibration.
This straightforward reformulation enables more efficient use of existing calibration methods, achieved with minimal modifications to the methods' original algorithms.
Applied scaling methods for calibration (such as Temperature Scaling). TvA allows the use of the binary cross-entropy loss, which is more efficient in decreasing the confidence of wrong predictions and leads to stronger gradients in the case of Temperature Scaling. Applied to binary methods for calibration (such as Histogram Binning), TvA significantly improves their performance and makes them eecuracy-preservvin
We demonstrate our approach's 's scalability and generality with extensive experiments on im- age classification with state-of-the-art models for complex datasets and on text classification with Pre-trained Language Models (PLMs) and Large Language Models (LLMs).
2
Related work
Calibration There are various notions of multiclass calibration. One consider confidence 17
class-wise 288. top-r [1] top-label [18), decision (75). projection smooth [16), or strong [60 65
calibration. For recent surveys, we refer 10 and [63]. In this work, we focus confidence calibration and not on the calibration of the full probability vector. Indeed, confidence calibration is useful for many applications that only require single confidence value: selective classification [3).
out-of-distribuit detection [22], or active learning 34 For these applications, stronger notions of calibration are both difficult and useless. Also, class-wise calibration metrics do not appropriately scale to large numbers of classes, a setting we consider in this work, as explained in Appendix
Metries Several metries have been proposed quantify calibration error. The most common is the Expected Calibration Error (ECE) 46) (see Equation 2 ECE has flaws: the estimation quality is influenced by the binning scheme, and it not a proper scoring rule 14 60 48). Despite its flaws, remains the standard comparison metric for confidence calibration. Variants of ECE have also been
developed: classwise-ECE (29) ECE with equal mass bins (8 44 or top-label-ECE, which adds a conditioning on the predicted class [18]. The Brier score 4 is also used to measure calibration. The proximity-informed expected calibration error (PIECH) evaluates the miscalibration due to proximity bias (68). We mainly use the standard ECE in this work, and the Appendix contains more metrics.

---

Training calibrated networks Several solutions have been proposed in the literature to improve
calibration by training neural networks in specific ways, generally by making use of new loss term
6 S While these methods directly optimize calibration during the training phase of the networks, they require a high development time, often compromise accuracy, and are not adapted to pre-trained foundation models. That is why we prefer to focus on calibrating already-traine models.
31
58
26
Post-processing (or post-hoc) calibration methods Another approach is calibrate already- trained models. This lowers the development time by decoupling accuracy optimization and calibra- tion. In this paper, we divide post-hoc calibration methods into two categories: scaling and binary. Scaling methods are derived from Platt scaling [50] and optimize some parameters to scale the logits. Temperature Scaling [17] is a popular simple post-processing calibration method. The logits vector scaled by a coefficient, which modifies the probability vector. Vector Scaling [17 is more expressive
and has good performance many cases [7 48 29]. Matrix Scaling can also be considered for more
expressiveness but is difficult to apply without overfitting [[7]. Dirichlet Calibration (29] proposes regularization strategy for Matrix Scaling. [72] developed Ensemble Temperature Scaling. Scaling can be combined with binning (30]. Besides logits or probabilities, features can also be used [5] Another family of methods tackles binary classification. We designate them as binary methods. Histogram Binning [69] divides the prediction into B bins according to the predicted probability. For each bin, a calibrated probability is computed from the calibration data. The probability becomes discrete: it can only take B values. With some modifications, it outperforms scaling methods [9]. Isotonic Regression 770 learns a piecewise constant function to remap probabilities. Bayesian Binning into Quantiles 46 brings Bayesian model averaging to Histogram Binning. Beta Calibration (8) uses a beta distribution to obtain a calibration mapping.
Our work reformulates the multiclass calibration problem and allows more efficient use of all these calibration methods, with little to no change in their algorithms.
Multiclass to Binary Using binary calibration methods for multiclass classifier requires adapting the multiclass setting. This is usually done with a One- versus-All approach [70 [7]. The multiclass setting is decomposed into T, One-vern versus-All independent problems: one binary problem for each class. 18] introduce the notion of top-label calibration, i.e., confidence calibration with additional conditioning on the predicted class (top-label). They describe general multiclass-to-binan frame- work to develop top-label calibrators. [6] derive L(L - 1)/2 pairwise binary problems. The approach requires training the classifier from scratch, and its performance decreases with the number of classes. Our work tackles this multiclass-to-binal research problem. Contrary to [6), the One-versus-Al approach (70 and top-label calibrators 18], our approach works well for problems with many classes. The methods I-Max [49] and IRM (72) use a shared class-wise strategy to compute single calibrator. The calibrator is applied all class probabilities separately, so the class probabilities ranking and prediction might change. In contrast, TvA applied to binary methods rescales the confidence after the class prediction is made. I-Max and IRM consider the full class probabilities vectors, while TvA only considers confidence values. Also, they build on top of Histogram Binning and Isotonic Regression, respectively, while we apply our approach to many calibration methods. A concurrent work building on an intuition similar to ours derives a calibration method based on a Correctness- Loss [38] Appendix discusses the differences between our approach and others.
3
Problem setting
3.1 Background
Confidence calibration of classifier We consider the classification problem where an input is associated with class label E y = [1, 2, .... L). The neural network classifier f provides a class prediction from a final softmax layer a that transforms intermediate logits into probabilities. The classifier prediction is the most probable class ÿ = arg maxkey fa (z) with (x) referring to the probability of class k, and the confidence score defined as s = maxkey fx(x). Note that we use the term confidence to denote the maximum class probability. With y the real label, we consider the confidence calibration definition from 17 that says that the classifier ʃ calibrated if:
P( u|s = p) = p. E
where the probability is over the data distribution. Equation ① expresses that the probability of being correct when the confidence is around p indeed p. For instance, if we consider the set of predictions
3

---

with confidence of 90%, they should be correct 90% of the time. The conditional probability of (1) not rigorously defined mathematically (the event {s = p} has zero probability), and interval-based empirical estimators are often used to define metrics capable of evaluating how well ① is satisfied. This is the case of ECE, which approximates the calibration error by partitioning the confidence distribution into B bins. The absolute difference between the accuracy and confidence is computed for each subset of data in the bins. The final value a weighted sum of the differences of each bin.
ECE Σ Th N
ac) conf(6))
Where ny is the number of samples in bin b, Nis the total number of samples, acc(b) is the accuracy in bin b, and cbnf(b) is the average confidence in bin b. ECE can be interpreted visually by looking at diagrams such as those of Figure ECE computes the sum of the red bars (difference between bin accuracy and average confidence) weighted by the proportion of samples in the bin.
Post-procced processcgssiigg calibration methods We are considering the scenario where a classifier has already been trained, and the objective is to enhance its calibration. Post-processing calibration methods aim to remap the classifier probabilities to better-calibrated values without modifying the classifier. They typically
a calibration set different from the training set optimize parameters or
learn a function. We note the calibration data Deal = {z,, y:) N } We focus on post-processing calibration because it enables better utilization of aff-the-shelff models and separates model training (optimized for accuracy) from calibration. These advantages significantly reduce the development cost of obtaining a well-performing and well-calibrated model, contrary to optimizing calibration during training. We categorize the post-processing calibration techniques considered in this paper into two groups: scaling methods and binary methods.
3.2 Issues related to current approaches
Behavior of current scaling methods Scaling methods for calibration optimize one or more coefficients that scale the logits vector to minimize on calibration data the cross-entropy loss defined as IcE -E 1x=y (z)( = - log(fy (z))) Minimizing Ic therefore increases the probability of the true class. We can distinguish two cases to understand what happens during the optimization: whether the prediction ğ is correct or not, In the first case, the confidence score is = [x)): minimizing Ie increases the confidence fy(r). In the second case, the prediction is incorrect, which implies that [y( <s. s. Minimizing lG increases the probability of the true class fy (x) but does not directly change the confidence (because s * fz(r)). Instead, the confidence (which was attributed to a wrong class) is indirectly lowered through the softmax normalization.
+ Issue I: Cross-entropy loss only indirecly lowers confidence in wrong predictions.
We identified another issue of some scaling methods. By design, the number of parameters optimized by Vector Scaling and Dirichlet Calibration grows with the number of classes. When the number of classes is high, these methods overfit the calibration set as shown in Figure
Issue 2: Vector Scaling and Dirichlei Calibration overfit calibration sets with many classes.
One-versus-All approach for binary methods The One-versus-Ale All (OvA) calibration approach 70 allows adapting calibration methods for binary classifiers to multiclass classifiers. To do so, it decomposes the calibration of multiclass classifiers into sets of L binary calibration problems: one for each class k. For each problem, the considered probability is fa(x)) and the associated label ly=k E {0, 1}. When calibrating a classifier from data, each binary problem is highly imbalanced with a ratio between positive and negative examples equal to - if the classes are equally sampled. For instance, for ImageNet, the ratio is 1/999: out of 25000 examples, only 25 have a positive label.
Issue 3: OvA approach leads to highly imbalanced binary problems.
At test time, each of the L class probabilities is calibrated by a separate calibration model. The resulting probability vector can be normalized to ensure a unit norm. Because each probability is calibrated independently, their ranking can change, thus modifying the predicted class. In Table 9 we see that accuracy is often negatively impacted in practice.
Issue 4: OvA approach can change the predicted class and negatively impact the accuracy.

---

Top-versus-All approach to confidence calibration
Algorithm -op-versus-All approach to confidence calibration
Input:
Dont: {(zi,y: y= the calibration data
ʃ: the multiclass classifier
g. a calibration function
> e.g,, Temperature Scaling
Preprocessing:
14 f arg maxtey (xi)
Compute class predictions Compute predictions correctness Create surrogate binary classifier 4 Build binary calibration set
fo + maxkey fk
D {(I) 1
Learn calibration function:
Learn g to calibrate the surrogate binary classifier fo on D
Inference:
Use g to calibrate the confidences of the original multiclass classifier f
4.1 General presentation
In the calibration definition (T) and the standard ECE metrics, only the confidence, i.c., the maximal probability, reflects the likelihood of making an accurate prediction. The probabilities of other classes are not taken into account. However, the standard approach to calibration uses the entire set of probabilities, not just confidence, which introduces unnecessary complexity. We aim to simplify the process by reformulating the problem of calibrating multiclass classifiers into a single binary problem. This problem can be phrased as: "Is the prediction correct?". In this setting, we do not calibrate the predicted probabilities vector but only a scalar: the confidence. The remaining probabilities are discarded. This is equivalent to calibrating a surrogate binary classifier that predicts whether the class prediction is correct. Since this correctness classifier only considers the maximal probability versus all others, we call our approach Top-versus-All (TvA).
Replacing the standard approach by TvA is straightforward. Given the standard calibration data Deat = {x, ) we add few data preprocessing steps. First, compute the class pre- dictions ÿ and their correctness: yb Tg=y- Second, create the surrogate binary classifier (x) maxxey (a).. Finally, build the calibration set for the surrogate binary classifier:
D = {(x 1
(3)
After this preprocessing, we choose standard calibration function g. e.g., Temperature Scaling, to calibrate the surrogate binary classifier. The learning of the calibration function follows its original underlying algorithm but uses the modified calibration data D The learned calibration function then applied to the confidences of the original multiclass classifier. Algorithm [ describes our approach. In the Appendix, Algorithm 3 provides more details and highlights differences with the standard approach of Algorithm 2
After this general presentation, we explain how TvA impacts the two categories of calibration methods, scaling and binary, in Subsections 4.2 and 4.3 respectively. We also justify its behavior.
4.2 Top-versus-Al approach for scaling methods
Because our Top-versus-Al setting reformulates the calibration of multiclass classifiers into a binary problem, the natural loss is the binary cross-entropy:
(4) Minimizing this loss results in confidence estimates that more accurately describe the probability of being correct, regardless of the L. less likely class predictions. Using the binary cross-entropy as a calibration loss makes an important difference compared to the usual multiclass cross-entropy. The cross-entropy loss takes into account the probability of the correct class, while with TvA the binary cross-entropy takes into account the probability of the predicted class (i.c., the confidence).
IBCE = -Iy² log s + (1 y²)) log(1 -s)
5

---

As for the standard approach, only two cases are possible. When the prediction is correct, (BCE = - lo)g(s) lfg ICE. We get the same result as the cross-entropy loss: minimizing directly increases the confidence. But when the prediction is incorrect, BCE - log(1 - s) * Icg.
Minimizing the loss now directly decreases the confidence. This is key difference compared to using the multiclass cross-entropy loss.
The impact of the reformulation can be seen for Temperature Scaling, which optimizes coefficient T' that scales the logits 2g.- The reformulation generates stronger gradients when the prediction is incorrect:
BCE 0T
OlCE 8T
for > 0.5
(5)
with MgCE T --= -Inax = -x =k fk) and Olce 4 (2y - Fx fk). See Appendix
ST Dioor the mathematical calculations. Because for interesting problems, the confidence verifies > 0.5 most of the time (as shown in Figure(T). our approach strengthens the gradients. The optimization of the temperature 7 is more efficient as confident incorrect predictions are more heavily penalized. This effect not mitigated by the choice of learning rate, which does not vary with 8. Applying standard Temperature Scaling usually results in overconfident probabilities, but approach limits this overconfidence. This is verified experimentally Table which displays the average confidences for TS without and with TvA.
Solutioo for Issue 1: Use the binary cross-entie moss-entropy loss resulting from TvA approach.
Regularization of scaling methods Overfitting of Vector Scaling and Dirichlet Calibration can be reduced with simple L2 regularization that penalizes the coefficients of the vector u that are far from the reference value 1.
(v)
Σ
(u 1)²
(6)
This regularization allows these methods take advantage of their additional expressiveness without being subject to overfitting. The loss for Vector Scaling becomes BBC + >rreg(r) where À is a hyperparameter. The loss for Dirichlet Calibration uses additional matrix regularization terms. is the only additional hyperparameter introduced by our method, and it applies only to Vector Scaling and Dirichlet Calibration.
Solution for Issue 2: Use L2 regularization.
4.3 Top-versus-Al approach for binary methods
Our TvA approach replaces the One-versus-AN approach apply binary methods to the multiclass setting. TvA transforms the multiclass setting into a single binary problem that uses the binary calibration dataset In this dataset, the proportion of positive labels equals the classifier's accuracy
a. The ratio between negative and positive examples (1-4)N accuracy ImageNet and calibration dataset of 25000 examples, there are 5000 negative and 20000 positive examples (ratio of 1/4). This still bit imbalanced but orders of magnitude smaller than the class-wise binary calibration datasets of the One-versus-Ar approach (ratio of 1/999).
1. For a classifier with 80%
eN
Soluttion for Issue 3: By not dividing the calibration data into class-wise datasets, the TvA approach yields much better balanced binary calibration problem.
The Top-versus-Ae approach operates confidence alone, not the full class probabilities vector. This means that the class prediction is already done, and the ranking of the class probabilities does not change, unlike with One-versus-All. The classifier's prediction and accuracy are unaffected. This scheme allows decoupling accuracy improvements (during training time) and calibration (during post-processing calibration), thus avoiding compromises and reducing development time.
Sollution for Issue 4: By operating on confidence alone, the Top-versus-Al approach does not impact the classifier's prediction accuracy for binary methods applied the multiclass scenario.

---

Samples
Global Accuracy
Global Accuracy Avg. Confidence
Global Accuracy Avg. Confidence
... Global Accuracy Avg. Confidence
Avg. Confidence
0.4
0.2
0.0
0.2
0.4
0.6
0.8
0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Acouncy
Perfoct Calibration
Perfect Calibration
Perfect Calibration
... Perfect Calibration
0.0
0.2
0.4
0.6
0.8
0
0.2
0.4
0.6
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Conlidence
Conlidence
Conlidence
Conlidence
(a) ResNet-50 original
(b) ResNet-50 TS
(c) ResNet-50 TSNa
(d) ResNet-50 IIB
Accurccy of Samples
Global Accuracy
Global Accuracy Avg. Confidence
Global Accuracy
Global Accuracy
0.6
Avg. Confidence
Avg. Confidence
Avg. Confidence
0.4
0.2
0.0
0.2
0.6
0.8
0.6
0.4
0.2
0.8
Perfoct Calibration
Perfect Calibration
Perfect Calibration
Perfoct Calibration
0.6
0.4
0.2
0.0
0.2
0.6
0.8
0.6
Contidence
Contidence
Coulidence
Contidence
ViT-B/16 original
(i) ViT-B/16 VS
ViT-B/16 VS..g
(h) ViT-B/16 HBTTA
Figure 1: Reliability diagrams for ResNet-50 and ViT-B/16 when using Temperature Scaling (TS), Vector Scaling (VS), and Histogram Binning (HB) on ImageNet. The subscript TvA signifies that the
TvA reformulation was used, and reg means our regularization (6) was applied. Red bars show the differences between bin accuracy (blue bar) and accuracy for perfect calibration (dashed red line). As the methods improve the calibration, these differences are reduced and the average confidence (vertical dotted line) will get closer the global accuracy (vertical dashed line).
5 Experiments
5.1 Setting
Datasets and models For image classification, we used the datasets CIFAR-IO (C10) and CIFAR- 100 (C100) (27) with and 100 classes respectively, ImageNet (IN) 7 with 1000 classes, and ImageNet-2IK (IN21K) (54) with 10450 classes. For text classification,
used Amazon Fine Foods (AFF) [43] and DynaSent (DF) [51] for sentiment analysis with classes, MNLI [66] for natural language inference with 3 classes, and Yahoo Answers (YA) [73] for topic classification on 10 classes. Experiment results are averaged over five random seeds that randomly split the concatenation of the original validation and test sets into calibration and test sets.
We used the following models for image classification: ResNet 21, Sode-ResNNer266. (WRN) 711.
DenseNNet-1e /2/ [24], MobileNerV3 V3 (MN3) (23) ViT (9) ConvNeXI [41) EfficientNet [56 57 Swin
40 39), and CLIP (22) which matches input images to text descriptions in a shared embedding space, assigning labels based the highest similarity score. For text classification, used the PLMs
RoBERTa [37] and T5 [53].
More details about datasets, calibration sets sizes, and model weights can be seen in Appendix
Baselines Our Top-versus-All (IvA) reformulation and regularization (reg) can be applied to different calibration methods. We have tested the following scaling methods: Temperature Scaling (TS) and Vector Scaling (VS) [17], and Dirichler Calibration (DC) [29] with the best best-performing variant Dir-ODIR, which regularizes off diagonal and bias coefficients. We also tested the following binary methods: Histogram Binning (HB) (69] using for each case the best- best-performing variant between equal-mass or equal-size bins, Isotonic Regression (Iso) [70), Beta Calibration (Beta) (28), and Bayesian Binning into Quantiles (BBQ) [46]. For comparison, we include methods with state-of- the-art results on problems with many classes: I-Max [49] and IRM [72]. See Appendix C for more
details these methods. More details code implementations can be seen in Appendix

---

Table I: ECE in % (lower is better). The subseript TvA denotes that our reformulation was applied to the calibration method. IRM and I-Max are competing methods. The best method for a given model is in bold. Methods in purple impact the model prediction, potentially degrading accuracy; methods in teal do not. Values are averaged over five random seeds. Results are averaged over models of the same family. Detailed results for all models can be seen in Tables 5 and of the Appendix.
scaling methads
binary methods
Dhttaset
Models
Uncal.
IRM
I-Max
TS
T8
VS
VSegna
DC
DCoggrs
Iso
Istr.
BBQ
BBQ
HB
C10
NonsNets
1.77
0.67
0.61
1.20
1.25
1.17
1.17
1.16
1.37
1.17
0.77
1.06
0.43
CLIP
5.03
1.03
0.94
0.78
0.73
2.56
1.85
2.56
1.84
1,05
0.86
1.39
0.86
1.82
0.73
C100
ConeNets
6,04
1.30
1.15
4.65
2.96
4.91
2.35
4.89
2.35
5.33
1.38
9.63
1.35
9.56
1,002
CLIP
10.37
2,90
2.57
2.54
2.51
7.78
2,86
7.55
1.84
2.53
7,48
1.39
RexNet
15.26
1,31
1.07
2,65
1,89
2.77
1.67
3.59
2.23
3.05
0.79
8.41
0.76
7.49
0.55
BTNNtt
15:72
0.68
0.48
3.48
2.59
3.67
1.26
3.65
1.23
2.83
0.68
6.55
0.64
4.39
01.43
ComNNeXt
16,46
0.82
0.58
3.67
2.25
4.05
1.37
4.04
1.35
2.97
0.75
7.41
0.68
5.13
0.52
VIT
4,40
0.81
0.61
4.09
2.96
4.31
2.02
4.31
1.99
3.60
0.77
6.64
0.73
0.52
Swin
0.75
3.63
2.91
4.04
1.70
L.67
3.19
7.09
0.48
CLIP
1.96
1,08
0.72
1.89
1,82
1.63
1.05
32.03
67.65
2.35
0.92
8.31
0.93
7.16
0.80
IN2IK
MN3
12,31
ctr.
et7.
8.69
4.39
2.52
2/10
58.84
81.16
2,00
0.21
GET.
0.20
5.500
0.17
VIT-B/16
6.27
8.92
6.55
2.38
1.54
8.22
3,20
2.14
0.22
0.24
7.89
0.12
AFF
T5
5.47
0.27
0.26
1.10
1.15
1.52
1.42
1.18
1.31
0.37
0.27
0.39
0.28
2.87
0.17
RoBERTa
7.37
0.30
0.28
2:40
2:33
1.41
1.85
1.38
1.68
0.52
0.27
0.75
0.35
4.02
0.20
2.19
2.17
6.13
2,00
1.38
RoBERTa
16.12
1.56
1.50
12.07
12:07
14.66
6.80
13.90
5.57
1.71
1.53
1.64
1.14
1.05
0.91
MNLI
TS
7.04
0.72
0.70
2.81
2.80
4.46
1.79
4.31
1.82
0.80
0.74
1,38
0.69
2.09
0.43
RaBERTa
9.22
0.89
0.71
5:72
5.72
6.99
1,92
6.59
1.99
1,00
0.92
1.67
0.84
1.02
0.60
0.80
0.81
1.07
3.70
1.16
3.75
1.73
2,81
0.69
RoBERTa
19.59
0.97
0.79
12.39
12.38
16.47
2.52
16.07
2.21
1.92
0.99
5,00
0.75
3:41
0.58
5.2 Top-versus-Al
We observe that initially, ResNet-50 is highly underconfident and ViT-B/16 a bit underconfident. Applying TS and VS solves the underconfidence and makes the models slightly overconfident. TvA improves these methods, and the average confidence gets closer to the accuracy. HBTvA makes the calibration almost perfect.
For visual qualitative results, Figure displays reliability diagrams 47.
Table shows the results of applying the Top-versus-Al All reformulation to several calibration methods. For clarity, results are averaged over families of models (models based on the same architecture) and the full results are available in Tables and of the Appendix. In most cases, the TvA reformulation significantly lowers the ECE by dozens of percent. Without TvA, binary methods often perturb the prediction and degrade the classifier's ' accuracy (see Table making them inapplicable in a practical setting. TvA solves the issue as it only scales the confidence (after the prediction made) and makes binary methods outperform scaling methods.
Improvements due to TvA are consistent across models. However, exceptions are observed for CLIP: it is the model family with the lowest ECE pre-calibration. but the highest ECE post-calibration for ImageNet. CLIP's multimodal training regime, zero-shot adaptation as a classifier, and very large training dataset might cause this different behavior. CLIP's low ECE was also observed in [4 (64] specifically tackles the calibration of fine-tuned CLIP, a setting not considered here.
We also found that DC is sensitive to hyperparameter tuning, and its performance is usually not much better than VS, which is consistent with (29) In some cases, the optimization diverges, leading to very poor results, e.g., for CLIP on ImageNet.
Improvements due to TvA are also consistent across datasets, although they tend to increase with the number of classes. Improvements on ImageNet are usually better than on CIFAR- 100, whose improvements are usually better than on CIFAR-10. This is notable with e.g. TS or HB. The magnitude of improvement is usually higher for binary methods, e.g., HB, than scaling methods, e.g.. TS, especially for many classes. This indicates that Issue 3 is more serious than Issue 1.
For text datasets with only three classes (AFF, DS, and MNLI), TS does not benefit from TvA, but other methods do, despite the small number of classes. According to (5) TS is among the best calibration methods for the text classification tasks considered here, even compared to ones that retrain the model. Even so, our method HBTvA TvA significantly outperforms it.
8

---

Some methods' current implementations could not handle the large scale of ImageNet-2IK,, resulting out-of momoor errors written "err." in the Table. For I-Max and IRM, this is because they consider the full probability vectors while TvA efficiently uses data by considering only confidence values. Indeed, TvA handles this scale without difficulty.
Additional results are included in Appendix H Tables and 6 contain the full results for ECE, with the standard deviations in Table 7 Table 8 reveals that ImageNet networks are mostly underconfident. This is aligned with M and goes against previous knowledge on overconfidence, which was initially believed to be linked to network size [7]. Table provides the accuracies after calibration. Table exhibits that ECE with equal-mast mass bins has similar values as standard ECE. Table IT shows that TvA mostly lowers the Brier score, except for Iso, which has the lowest score overall.
Calibration methods can also be applied to Large Languages Models (LLMs) using In-Context Learning (ICL) to tackle text classification tasks [77 20 25 78 The primary goal of these methods to improve model accuracy. TvA was not designed for this objective, but can still be applied on top of an existing method that improves the accuracy. TvA then lowers the calibration error while keeping the accuracy gain. Results for GPT-J [62 and Llama-2 [59] are in Table I2
To summarize the results for practical use, our experiments show that Histogram Binning (within the TvA or I-Max setting) is the best calibration method overall, providing ECE values mostly below 1 1%. This is the method we advise using. However, suppose the underlying application requires confidence with continuous values, c.g., to rank the predictions for selective classification. In that case, advise using method that improves the AUROC, shown in Appendix G such as TS Iso.
5.3 Solving overfitting with regularization and TvA
On ImageNet, VS and DC overfit the calibration set, degrading the calibration on the test set. The lower performance of VS relative to TS indicates this overfitting. As visualized in Figure combining the binary cross-entropy loss used in the TvA reformulation and an additional regularization term prevents overfitting. We fixed the value À 0.01 as it works well across models. Initializing the vector coefficients to T with T obtained by TSm. helps further improve performance.
5.4 Influence of the calibration set size
The size of the calibration influences the performance of the different methods, as seen in Figure TS and Th,, do not henefit from more data due to their low expressiveness. VS does not improve the ECE because of the overfitting problem. In contrast, VS benefits from calibration data. With enough data (s= 15000), it outperforms TS,.- Binary methods using the standard One-versus-All approach have poor performance and need large amount of data to be competitive. Using TvA, they get excellent performance with little data.
10
VS
ECE test [%6]
ECE test [%]]
VSTVA
6
TSTVA
4
VSng
2
NRQQvA
5
VSng
ECE test [%]
41
3
TS
50
100
150
200
epochs
2
5,000
10,000 15,000 20,000 25,000
Figure 2: Test ECE evolution during training with ResNet-50 on ImageNet. The combination of regularization and TvA prevents overfitting of Vector Scaling. Temperature Scaling with TvA is shown for reference.
calibration set size
Figure 3: Influence of the calibration set size for ResNet- 101 on ImageNet. Binary methods the top and scaling methods at the bottom.

---

Limitations
Our approach tackles confidence calibration and is unlikely improve performance for stronger notions of calibration, such as class-wise wise calibration. However, confidence calibration is useful for many practical cases, such as selective classification [13, out-of-distributtin detection [22], or active learning [34] Also, calibration improvements are less significant for problems with few classes (s 10) than for problems with many classes, but our approach still provides the best results.
Conclusion
Reducing the miscalibration of neural networks is essential to improve trust in their predictions. This can be done after the model training with an optimization using calibration data. However, many current calibration methods do not scale well to complex datasets: binary methods under the One-versus-Al setting do not have enough per-class calibration data, and scaling methods are inefficient. We demonstrate that reformulating the confidence calibration of multiclass classifiers single binary problem significantly improves the performance of baseline calibration techniques. The competitiveness of scaling methods is increased, and binary methods use per-class calibration data more efficiently without altering the model's accuracy. In short, our TvA reformulation enhances many existing calibration methods with little no change in their algorithm. Extensive experiments with state-of-the-ar image classífication models on complex datasets and with text classification demonstrate our approach's 's scalability and generality.
10

---

Acknowledgments and Disclosure of Funding
We thank Tahar Nabil, Houssem Ouertatani, and Pol Labarbarie for their constructive feedback the paper draft.
This work has been supported by the French government under the "France 2030" program, as part of the SystemX Technological Research Institute. This work was granted access the HPC/AI resources of IDRIS under the allocation 2024- AD011013372RI made by GENCI.
References
[1] Momin Abbas, Yi Zhou, Parikshit Ram, Nathalic Baracaldo, Horst Samulowitz, Theodoros Salonidis, and Tianyi Chen. Enhancing in-context learning via linear probe calibration. arXiv preprint ar arxiv:240. /2406, 2024.
[2] Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Sacid Na- havandi. A review of uncertainty quantification in deep leamiing: Techniques, applications and challenges. Information Fusion, 76:243-297, 297, December 2021.
[3] Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky. Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary Vito, Elias Ellison. Will Feng, Jiong Gong. Michael Gschwind, Brian Hirsh. Sherlock Huang. Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang. Jason Liang. Yinghai Lu. CK Luk. Bert Maher, Yunjie Pan. Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Fikan Wang. Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. PyTorch Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS '24). ACM, April 2024.
[4] Glenn Brier. Verification of forecasts expressed terms probability. Monthly weather review, 78(1):1-3, 1950.
Yangyi Chen, Lifan Yuan, Ganqu Cui, Zhiyuan Liu, and Heng Ji. Close Look into the Calibration of Pre-trained Language Models. In Proceedings of the 61 Annual Meeting of the Association for Computational I Linguistics ( (Volume 1: Long Papers), pages 1343-1367, Toronto, Canada, 2023. Association for Computational Linguistics.
[6] Jiacheng Cheng and Nuno Vasconcelos. Calibrating Deep Neural Networks by Pairwise Constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13709- 13718, 2022.
[7] Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, June 2009.
[8] Qingxiu Dong, Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint arXiv:2301.0000 000244 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer. Georg Heigold, Sylvain Gelly, et al. image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[10] Telmo Silva Filho, Hao Song, Miquel Perello-Nieto, Raul Rodiiiguz Meelis Kull, Peter Flach. Classifier Calibration: A survey on how to assess and improve predicted class probabilities. Machine Learning, 12((9):32 12-6260,, September 2023.
[11] Ido Galil, Mohammed Dabbah, and Ran EI- Yaniv, What Can we Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers? In The Eleventh International Conference on Learning Representations, February 2023.
[12] Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, Muhammad Shahzad. Wen Yang, Richard Bamler, and Xiao Xiang Zhu. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, July 2023.
11

---

[13] Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017.
[14] Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):35--378, 2007.
[15] Ethan Goan and Clinton Fookes. Bayesian neural networks: An introduction and survey. Case Studies in Applied Bayesian Data Science: CIRM Jean-Moriet Chair, Fall 2018, pages 45 87, 2020.
[16] Parikshit Gopalan, Lunjia Hu, and Guy N Rothblum. On computationally efficient multi-class calibration. arXiv preprint arXiv:2402.07821, 2024.
[17] Chuan Guo, Geoff Pleiss, Sun, and Kilian Q Weinberger. On calibration of modemn neural networks. In International conference on machine learning, pages 1321-1330. PMLR, 2017.
[18] Chirag Gupta and Aaditya Ramdas. Top-label calibration and multiclass-to-binaryy reductions. In Interna- tional Conference on Learning Reprexentations, January 2022.
[19] Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. Calibration of Neural Networks using Splines. International Conference on Learning Representations, December 2021.
[20] Zhixiong Han, Yaru Hao, Li Dong, Yutao Sun, and Furu Wei. Prototypical calibration for few-shot learning of language models. The Eleventh International Conference on Learning Representations, 2022.
[21] Kaiming He, Xiangyu 7hang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778, 2016.
[22] Dan Hendrycks and Kevin Gimpel. A Baseline for Detecting Misclassified and Out-of-Dode oft--strrriittionn Examples in Neural Networks. In International Conference on Learning Representations, 2017.
[23] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314-1324, 2019.
[24] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700-4708, 2017.
[25] Zhongtao Jiang, Yuanzhe Zhang, Cao Liu, Jun Zhao, and Kang Liu. Generative calibration for in-context learning. arXiv preprint arXiv:2310.100 10266, 2023.
[26] Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael C Mozer, and Becca Roelofs. Soft calibration objectives for neural networks. Advances in Neural Information Processing Systems, 34:29768-29779 2021.
[27] A. Krizhevsky, Learning Multiple Layers of Features from Tiny Images. 2009.
[28] Meelis Kull, Telmo M. Silva Filho, and Peter Flach. Beyond sigmoids: How to obtain well-calibrated probabilities from binary classifiers with beta calibration. Electronic Journal of Statistics, 11(2):052-50000 January 2017.
[29] Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Ohtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in neural information processing systems, 32, 2019.
[30] Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. Advances in Neural Information Processing Systems, 32, 2019.
[31] Aviral Kumar, Sunita Sarawagi. and Uijwal Jain. Trainable Calibration Measures Neural Networks from Kernel Mean Embeddings. In Proceedings of the 35th International Conference on Machine Learning, pages 1805-2814 14, PMLR, July 2018
[32] Fabian Küppers, Jan Kronenberger, Amirhossein Shantia, and Anselm Haselhoff. Multivariate confidence calibration for object detection, In The IEEECVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
[33] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
12

---

[34] Mingkum Li and L.K. Sethi. Confidence- based active learning. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 28(8):1251-12611 August 2006.
[35] Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Taking Step Back with KCal: Multi-Class Kernel- Based Calibration for Deep Neural Networks. In The Eleventh International Conference on Learning Representations, Septemher 2022.
[36] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9): 1-5,, 2023.
[37] Yinhan Liu, Myle Ott, Naman Goyal. Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: htt::/9077 /907. / /692, 2019.
[38] Yuchi Liu, Lei Wang, Yuli Zou, James Zou, and Liang Zheng. Optimizing calibration by gaining aware of prediction correctness. arXiv preprint arXiv:2404. 13016, 2024.
[39] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao. Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12009-12019, 2022.
[40] Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[41] Zhuang Liu, Hanzi Mao, Chao Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976 1 11986, 2022.
[42] Vision maintainers and contributors. Torchvision: Pytorch's 's computer vision library. https
gggthubb.com/pttorr///visssio 2016.
[43] Julian John McAuley and Jure Leskovec. From amateurs connoisseurs: modeling the evolution of user expertise through online reviews. Proceedings of the 22nd International Conference on World Wide Web, WWW '13 page 897-908, New York, NY, USA, 2013. Association for Computing Machinery.
[44] Matthias Minderer, Josip Djolonga. Rob Romijnders, Frances Hubis, Xiaohua Zhai Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the Calibration of Modern Neural Networks. In Advances in Neurul Information Processing Systems, volume 34, pages 15682-15694, Curran Associates, Inc., 2021.
[45] Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Punect Dokania. Calibrating Deep Neural Networks using Focal Loss. In Advances in Neural Information Processing Systems, volume 33, pages 15299. Curran Associates, Inc., 2020.
[46] Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining Well Calibrated Probabilities Using Bayesian Binning. Proceedings of the AAAI Conference on Artificial Intelligence, (I), February 2015.
[47] Alexandru Niculescu-Min -iziil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine Learning, ICML '05, pages 625-632, New York, NY, USA, August 2005. Association for Computing Machinery.
[48] Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran, Measuring calibration in deep learning. In CVPR workshops, volume 2, 2019.
[49] Kanil Patel, William H Beluch, Bin Yang, Michael Pfeiffer, and Dan Zhang. Multi-class uncertainty calibration via mutual information maximization-base binning, In International Conference on Learning Representations, 2020.
[50] John Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances large margin classifiers, 10(3):61-74, 1999.
[51] Christopher Potts, Zhengxuan Wu, Atticus Geiger, and Douwe Kiela. DynaSent: A dynamic benchmark for sentiment analysis. In Chengqing Zong, Fei Xia. Wenjie Li, and Roberto Navigli. editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2388 2404, Online. August 2021. Association for Computational Linguistics.
13

---

[52] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[53] Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-61 2020.
[54] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zannik-Manor. Imagenet-21k pretraining for the masses. In Thirty-fftth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[55] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potis. Recursive deep models for semantic compositionality sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.
[56] Mingxing Tan and Quoc I.c. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learming, pages 6105-61 14. PMLRR 2019.
[57] Mingxing Tan and Quoc Le. Efficientmetv2: Smaller models and faster training. In International conference on machine learning, pages 10096-10106. PMLR, 2021.
[58] Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tammoy Bhattacharya, and Sarah Michalak. On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
[59] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307,09288 2023.
[60] Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in elassification. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3459 3467. PMLR, 2019.
[61] Ellen M. Voorhees and Dawn Tice. Building a question answering test collection. Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '00. page 200-207, New York, NY, USA, 2000. Association for Computing Machinery.
[62] Ben Wang and Aran Komatsuzaki, GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https: / /i thub. conm/kinggooollz/meiii. www.ssssssstttttatareeereeeeeee. jax May 2021.
[63] Cheng Wang. Calibration in deep learning: A survey of the statc-of-the-art. arXiv preprint arXiv:2308.0122 012222 2023.
[64] Shuoyuan Wang. Jindong Wang, Guoqing Wang. Bob Zhang, Kaiyang Zhou, and Hongxin Wei. Open- vocabulary calibration for vision-language models. arXiv preprint 2402:24022.06555 2024.
[65] David Widmann, Fredrik Lindsten, and Dave Zachariah. Calibration tests in multi-class classification: A unifying framework. Advances in neural information processing systems, 32, 2019.
[66] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Marilyn Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies, Volume I (Long Papers), pages 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguisties.
[67] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Mai, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jemnite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame. Quentin Lhoest, and Alexander M. Rush. Transformers:
-otte-of-thearr natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics.
[68] Miao Xiong, Ailin Deng, Pang Wei W Koh, Jiaying Wu, Shen Li, Jianqing Xu, and Bryan Hooi. Proximity- informed calibration for deep neural networks. Advances in Neural Information Processing Systems. 36:68511 1-665111668888....11111111888881 2023.
14

---

[69] Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In Proceedings of the Eighteenth International Conference Machine Learning. ICML '01, pages 609-616, San Francisco, CA, USA, June 2001. Morgan Kaufmann Publishers Inc.
[70] Bianca Zadrozny and Charles Elkan, Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '02, pages 694-699, New York, NY, USA, July 2002. Association for Computing Machinery.
[71] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:600.007146. 2016.
[72] Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In International conference machine learning, pages 11117-11128. PMLR, 2020.
[73] Xiang Zhang, Junbo Zhao, and Yann I LeCun. Character-level convolutional networks for text classification. In C. Cortes, Lawrence, Lee, M. Sugiyama, and Garnett, editors, Advances Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[74] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character- level convolutional networks for classification. In C. Cortes, Lawrence, Lee, M. Sugiyama, and Garnett, editors, Advances Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[75] Shengjia Zhao, Michael Kim, Roshni Sahoo, Tengyu Ma, and Stefano Ermon. Calibrating predictions to decisions: A novel approach to multi-class calibration. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 22313-22324. Curran Associates, Inc., 2021.
[76] Xujiang Zhao, Shu Hu, Jin-Hee Cho, and Feng Chen. Uncertainty-baase Decision Making Using Deep Reinforcement Learning. In 2019 22th International Conference on Information Fusion (FUSION), pages 1-8, July 2019.
[77] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before Improving few-shot performance language models. In International conference on machine learning, pages 12697-12706. PMLR, 2021.
[78] Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, and Subhrajit Roy. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. arXiv preprint arXiv:2309.17249, 2023.
15

---

A
Appendix contents
discusses the broader impacts of the work.
discusses the proposed approach in more details, and compares with other methods.
contains a theoretical justification for -op-versus-All in the case of Temperature Scaling.
discusses the limits of classwise-ECE and top-label-ECE for high number of classes.
describes implementation details, including the computing time in Table
G shows the impact of different calibration methods selective classification.
provides additional results: full ECE results in Table 5 and 6 standard deviations in Table 7
confidences in Table accuracies in Table equal-mast bins ECE in Table TO Brier score in Table
experiments for in-context learning of LLMs in Table I2
B
Broader impacts
Our reformulation of the confidence calibration of multiclass classifiers as binary problem is both simple and general. It has several benefits. On the theoretical side, it might lead to new perspectives on the confidence calibration problem and the development of new calibration methods. On the practical side, existing calibration methods can be adapted to problem reformulation by adding just a few lines of code. This is an easy and quick way to improve the calibration of classification models. Better-calibrated models are more trustworthy: potential incorrect predictions easily identifiable and preventable. However, this also comes with potential risks. The knowledge that model is well-calibrated might lead to undue trust in the system and the tendency to overlook prediction errors. Even well-calibrated models are not entirely reliable, and developers and users must remember this. Post-pree fost-procsssing calibration requires data not included in the training set, which leaves less data available for a thorough evaluation of the model. Calibration does not fix biases in the data. Finally, tested the calibration improvement only on in-distribution data, but real systems might receive out-of-distribution data (e.g., an image of a new class) or adversarial examples. For such inputs, the classifier predicted probabilities (and thus the confidence) are unreliable, even for well-calibrated models, and pipeline to filter such data is necessary.
more
C Details the method
Algorithm -er-versus-al approach
Input:
Algorithm 2 Standard approach
Deat: {(,,,, ) the calibration data
Input:
f: the multiclass classifier
Deat: {(,,, (1) the calibration data
gi calibration function
Dr cgg. Temperature Scaling
f: the multiclass classifier
Preprocessing:
g: a calibration function & egg. Temperature Scaling
M: arg maxkey fe(
D Compute class predictions
Learn calibration function:
ly,n
Dr Compute predictions correctness
if is scaling method then
f +- maxvey fi N
Create surrggat binary classifier
loss ( == Cross- Cross-Entropy
D + [[xx,, )
t Build hinary calih.
Learn g calibrate ff by minimizing I Deal
Learn calibration function: g is scaling method then
else g binary method then
for le = to I. do
b Oneeversus-Al appmach
loss = Binary Cross Cooss-Entropy if g is vector Dirichlet scaling
D +- [( U:) = 2
Learn Ok to calibrate ff on D
loss 1
1+ Mreg
D regularization
end for
end if
(g), (g..0,.....9))
Learn g to calibrate f² by minimizing on D else if g is binary method then
end if
Inference:
Learn g calibrate f TV
Use g to calibrate contidences from f
end if
Inference:
Use to calibrate confidences from f
16

---

Comparison with the standard approach Algorithm 2 describes the standard approach to post- processing calibration, and Algorithm B describes our approach in more details and shows in blue the differences with the standard approach. Our approach adds a preprocessing step to keep only the confidences instead of the full probabilities vector. It can be seen as creating a surrogate "correctness" classifier and its associated calibration data. The calibrator is learned for the surrogate classifier and applied to the original classifier at inference time. Also, we add regularization for some scaling methods and we have only one binary calibrator instead of one per class.
Comparison with IRM and I-Max IRM 72 and I-Max 49 are, like TvA, multiclass-to-biigg reductions. This is why TvA cannot be applied on top of them: they already transform the multiclass problem into binary one using a different strategy.
The shared class-wise strategy of [49] and the data ensemble strategy of 72) are described very briefly in subsections 3.2 and 3.3.2 of their respective papers and not rigorously justified. Our understanding that these two strategies do exactly the same thing. To build the calibration set, they concatenate all the class probability vectors so that we get a big probability vector of size N.L (N samples and L classes) as predictions and similarly concatenate the one-hot embedding of the target class (a big vector with N ones and NN. 1) zeros) as targets. Then, they learn single calibrator. For each example, this calibrator aims to simultaneously increase the probabilities for the target class (target is 1) and decrease all the other class probabilities (target is 0). The single calibrator is applied to each class probability separately, meaning that the ranking of class probabilities can change, modifying the classifier prediction.
Our strategy derives from transforming the multiclass calibration into a single binary problem. The intuition is to learn the calibrator on surrogate binary classifier and apply this calibrator to the original classifier. This binary classifier built on top of the original classifier (by applying the max function to the class probabilities vector). They thus share their confidence. However, the binary classifier aims solve a different task: predicting the correctness of the original classifier. To build the calibration set, we concatenate all the confidences (a vector of size N) as predictions and concatenate all the correctnesses as targets (also a vector of size N). The correctness value of a given example is 1 if the class prediction is correct; otherwise, it is 0. Then, we learn single calibrator, similar to the strategy above. However, there is a key difference: this calibrator aims to increase the probabilities for correct predictions and decrease them for incorrect predictions. Note that our probabilities are all confidences (the maximum class probabilities), meaning we only consider the confidences, which the calibrator directly increases decreases. In the strategy from [49] and (72), the calibrator has to manage all class probabilities (L times more), even the ones that do not matter, including the lowest class probabilities close to 0. This is less efficient (actually, while this can surely be fixed, the original implementations of IRM and I-Max could not run on ImageNet-20 IK). This point is closely linked to the analysis of the binary cross entropy loss for scaling methods in Subsection 4.2: when the prediction is incorrect, increasing the probability of the correct class indirectly decreases the confidence (strategy from 49 and 722 while our strategy directly decreases the confidence.
I-Max more complex because it modifies the Histogram Binning algorithm, while our approach does not. Additionally, (35 found that I-Max produces unusable probability vectors. Indeed, they do not sum up to 1, and normalizing them degrades the method's performance.
We wrote our paper with practicality and generality in mind. Contrary to 49 and (72). we demon- strate the generality of our strategy by applying on top of existing calibration baselines of different natures (scaling and binary). One of our main goals is that practitioners can easily and quickly try our TvA approach, using just few lines of code, which can significantly improve the calibration performance of their existing calibration pipeline while having no impact on the predicted class by design (except for VS and DC).
Comparison with ProCal Another recent calibration method is ProCal 68). However, its primary objective differs from ours: it "focuses on the problem of proximity bias in model calibration, a phenomenon wherein deep models tend to be more overconfident on data of low proximity". Its goal is to lower the difference in the confidence score values between regions of low and high density, i... to make the confidence score independent of a local density indicator called "proximity." " There is no theoretical guarantee, however, that minimizing the proximity bias improves the confidence calibration, the focus of our work. Theorem 4.2 about the PIECE metric is direct consequence of Jensen's 's inequality and is true for any random variable D, not necessarily proximity score.
17

---

Theorem 1 is an interesting bias/variance decomposition of the Brier score. However, as this type of decomposition usually states, the error may come from bias (here, a wrong initial calibration) or high estimation variance (which can be related to low density but is not expressed such in the decomposition). We experimentally compare our approach to the ProCal algorithm using the code provided by its authors and observe in Table that our approach gives much better ECE confidence calibration and, for half of the models, also better PIECE values.
ProCal aims to achieve three goals: mitigate proximity bias, improve confidence calibration, and provide a plug-and aad--lan method. We share the last two goals. Concerning improving confidence calibration, our approach has better results, as shown in Table Both approaches are plug-and plyy. but they apply very differently. ProCal applied after existing calibration methods further improve calibration. It thus does not solve any of the four issues we identified (e.g., cross- cross-entropy loss is still inefficient, and One-versus-AN still leads to highly imbalanced problems). Our Top-versus-Ar approach is reformulation of the calibration problem that uses a surrogate binary classifier. Existing approaches are applied to this surrogate classifier, which how the four issues are solved. We do not propose new method but new way of applying existing methods. Our approach does not introduce new hyperparameters (except in the particular case regularizing scaling methods). ProCal introduces several new hyperparamcters, such the choice of the distance, the number of KNN neighbors, or a shrinkage coefficient.
Comparison with Correctness-Awan Loss A concurrent work (38] builds calibration method on top an intuition similar to ours: binarize the calibration problem. However, what the authors do with this intuition differs vastly from our approach. They derive a Correctness-Aware Loss (Eq. of their paper), which almost the standard binary cross-entropy loss we use for scaling methods but without logarithm. They use this loss learn separate model that predicts sample- wise temperature coefficient. This a new calibration method, which is not straightforward to implement due to the numerous hyperparameters (network architecture, image transformations..)). It also requires multiple inferences at test-time, which can be problematic in some production models. Our approach is, again, a calibration method but general reformulation of the calibration problem that enhances existing methods. By looking at their Table 1, they get an ECE of 2.22 on ImageNet (in-distribution), while our approach achieves values around 0.5 for most models in our paper's 's Table 1. Their method, contrary to TvA, improves the AUROC, but in our understanding, it seems mostly due to the use of image transformations not from their proposed loss. Their method seems to work best in out-of-distribution scenarios, which is not the main objective of our paper. However, these good results for AUROC and out-of-distributio scenarios make this method complementary to our approach, and combining the two in some way could be promising.
18

---

Table 2: ECE, MCE, ACE, and PIECE in %. The experimental setting is the one used for Table 4 of 68 The baselines are no calibration (conf), Temperature Scaling (TS), Multi Isotonic Regression (MIR). and Histogram Binning (HB). The ProCal calibration method [68] is applied after one of the baselines, as symbolized by the "+" symbol. Our approach, Top-versus-
changes what the baselines optimize and is symbolized by "TvA". " We apply it for TS, Isotonic Regression (Iso), and HB. The best values for each model and metric are in bold.
Overall, HBTvA is the best calibration method as it always gets the lowest ECE and ACE. Our TvA approach lowers PIECE and even achieves the lowest value for half the models.
Model
Method
ECE ())
MCE (L)
ACE (L)
PIECE (4)
BeiT
conf
3.60
1.52
3.58
4.26
TS MIR HB
2.99
0.76
3.56
0.59
0.14
0.64
1.88
4.96
1.83
6.13
7.20
conf+ProCal
1.02
0.33
0.94
1.69
TS+ProCal
1.52
0.76
1.45
2.05
MIR+ProCal
0.61
0.15
0.71
1.41
HBoPrrCCa
5.53
4.13
5.81
6.39
TSTvA
2.10
0.49
2.20
2.88
ISOTAA
0.65
0.14
0.68
1.92
HBB
0.44
1.70
Mixer
conf
10.78
5.00
10.78
10.86
TS
4.81
5,57
MIR
1.14
0.25
3.41
IIB
9.65
4,00
9.97
13.09
conf+ProCal
2.64
0.94
2.55
3.23
TS+ProCal
1.32
0.38
1.22
2.13
MIR+ProCal
0.83
0.14
0.88
2,08
HB+ProCal
6.57
4.43
7.32
7.83
TSTAA
2.51
0.74
2.50
4.32
ISoT.A
0.86
0.15
0.86
3.18
HB
0.64
0.75
3.17
ResNet50
conf
8.59
4,58
8.74
TS
2.48
MIR
0.75
0.18
0.82
1.79
HB
7.63
2.61
9.32
10.17
conf+ProCal
2.63
1.31
2.61
3.26
TS-ProCal
1.66
0.66
1.53
2.50
MIR+ProCal
0.76
0.17
0.74
1.78
HB+ProCal
6.32
4,38
7.52
7.65
TSTAA
6.89
1.17
6.95
7.13
ISOYVA
0.76
0.19
0.73
1.79
HBN
0.55
0.59
1.35
conf
1.14
0.33
1.83
TS
1.46
0.46
2.03
MIR
0.64
0.13
0.75
1.54
HB
4.59
2.32
7.07
7.20
conf+ProCal
0.81
0.22
0.81
1.71
TS+ProCal
0.83
0.25
0.84
1.74
ProCPr
0.78
0.14
0.76
1.59
HB+ProCal
6.80
4.72
7.55
TSTA
1.08
0.25
1.05
1.86
Iso
0.51
HB
0.37
0.17
0.51
1.20
19

---

D Theoretical justification for Top-versus-All in the case of Temperature Scaling
We define L the number of classes, fz) the classifier estimated probability for class and data sample z, y the correct class, and the confidence s(x) max; fx(z). The cross-entropy loss is (2, y) = Ek=1 1 = y} -Io((f(())) = log (x)) Because the last layer the classifier is a softmax function, (x) = Σ. 5 eaN with z the logits vector. Note that we omit the writing variable in the following for clarity.
Temperature scaling optimizes coefficient T > 0 that scales the logits vector. Predicted probabilities become (z)
6²0/""
Exe5t
Let us first develop the standard cross-entropy loss when temperature scaling is applied:
IcE = log(fu) = - -log(
(-mr====-((/))))) loog Σ Σx (tt/T)
e²2/T
(Iog(e5//T)
eaa/T)
Fy
+log( Σ E
T
b
h
Let us compute its gradient:
OlCE
Zy
+
log Ex EalTT)
3.. 6
by application of the chain rule on the second term.
0T
T
Ø En ezx/T
0T
deaL/T
2y
+
72
Σn ezN/T
0T
2y
1
1/7
+
7
Σx e2t/T
OT
Zy
+
1
log(e**
(e )1/T
Σx eax/T
T22
2k ex//T
2y
Σ
+
T
Σx earlT'
-T2
1
Ex th :
72
Zyy
ExeOST
2k cP//T
Zy Σ
T
Ej etiTT
Zz Σ
2x fk
T
For our TvA approach, the problem becomes binary. The classification output becomes the confidence s(x) max (z) and the ground truth label becomes binary representation of the prediction correctness: n ls=y with ū(x) arg maxxey fu (œr) and the indicator function. The loss we use is the binary cross entropy (BCE(x,)) v) = - -yy² log s(x) + (1 - (1=")) log(1 s(z)))
20

---

Let us compute the gradient:
RCE
RCE
ds
0T
0s
8T
1
3² (1-3")1-3 -
(1
-1
0s
8
0T
(1 8)
s(1 y²)
0s
6(1 8)
8(1 8)
07
8 y6 Üs
s(1 s)
0T
65m/T Ee 6'k/T
8 y
ea//T
Em/T
because & = max
Σx e with 2m = max 2
s(1 8)
OT
Σ eax/T
&
yb
De 0m/
Σn e - c a eui
BT
s(1
8)
y6
loR(e (E )2/2 Ex Ex -emm/T Σx
log(e"<) (1=
8
s(1
Ex
y6
em/l
mm) En Saeat Ex (o.logee)) (E2))/))
s(1 8)
T2
8 y
@m/
-2m Σx eAN/T + +E e
s(1 8)
T
=mm Ea/aT + Ex 2k e
8
s(1
- 8)
T
18.
Ex ExeeT
1
2m E EaTT + Ex 24 c
T
1-8 8
Exeu/
1
Σ 2b 3.6.00 C
T
1
Σx ean/T
2m
1 3
Ex Zk eFxI/T eEx/T
T
1
7m
2ke eFk/T Ej eas/T
2m
T
1
7m Σ 2n fk
(8)
72
First case, the prediction is correct: y² = and zm = Zy. Let us inject these in 08
Blace Of
77 (Zy - Ei zz fk) Bles We thus get the same gradient as the standard cross-entropy loss.
07
Second case, the prediction is incorrect: y¹ O and
Zg-
becomes:
AlacE 87
77 - (zm Ex 2k fk). By comparing to 7.. we have the term TM (zm - fk)) >
T2 (zy Σx 24 fa) = 87
1
Bleg and the remaining part (8
s-1 when s > 0.5.
So to recapitulate, @ > Olcg when > 0.5, which corresponds to the vast majority of data 07
points as the classifier gets better calibrated. This is shown in Figure
We also have lim.-1 - oo. In practice, is not close enough to generate exploding
gradients, so it just means that as confidences for wrong predictions gets higher, does the gradient to reduce the confidences.
The conclusion is that for correct predictions, our approach does not change the optimization, but for incorrect predictions, the gradient is stronger and penalizes more heavily confident predictions that are wrong. This is also proven experimentally by looking at Table where we see that average
21

---

confidences of temperature scaling with the TvA approach (TS are lower than the ones using the standard upproach (TS), for almost networks. This makes the average confidences closer the accuracy, showing reduced overconfidence.
E
Limits of classwise-ECE and top-label-ECE for a high number of classes
10'
10
10
number Samples (log
10a
10
10
10a
10
10a
10
10
10'
10
109
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
class probability for class 677
class probability for cluss 537
cluss probability for class 845
Figure 4: Histogram of class probabilities for random classes, for ViT-16/B on ImageNet.
Let us define the ECE for class j:
Σ Th acc(bb, cbnf(b.j))
ECE₁
The difference compared to 2 is that now acc(b. j) corresponds the proportion of class in the bin. Also, conf(b. now is the average probability given to class j for all samples the bin. Then, classwise-ECE 29 takes the average for all classes:
ECEp Σ ECE,
Classwise- ECE considers the full probabilities vectors: all the class probabilities for each prediction. However, this metric does not scale to large numbers of classes. Let us see why with an example.
Let test of N samples, N/L for each of the L classes (the dataset is balanced), and high-accuracy classifier fairly calibrated. The classifier predicts N probability vectors of length L. Predicted probabilities for class are all the values of the vector dimension j. Because the classifier has high accuracy and fairly calibrated, around N/L values are close to 1 (corresponding to mostly correct predictions), and the remaining ones, around N N/ are close (because the predicted class is not class j. and the predicted probability is high for another class).
To compute ECE, with equal size 15 bins, the predicted probabilities for class are partitioned into 15 bins. The first bin (with probabilities close to 0), contains n A N N/L samples while the last one (with probabilities close 1) contains ng N/ samples. The remaining bins are usually more empty. That means that the calibration error the first bin is weighted 74 h1/ng- = L times more than the last one. For the 1000 classes of ImageNet, L 1 999. Figure 4 shows the number of samples (ng) in each bin for an ImageNet classifier.
Because the impact of the calibration emoor in the bin with the high probabilities is negligible relative to the bin with the low probabilities, the classwise- mostly measures whether probabilities close to are calibrated. We argue this is not what we are interested in: what matters more is the calibration of higher values of the probabilities.
Top-label ECE 18 is another interesting metric that does not scale to large numbers of classes either. Top-label-ECE divides data into subsets according to the predicted class, computes the ECEs of these subsets, and averages them. For an ImageNet test set of 25000 samples (25 per class), data is divided into 1000 subsets Av 25 samples each (the classifier is high-accuracy, most of the time the predicted class is equal to the true class). The ECE is computed for each subset containing only 25 samples. To compute the ECE, samples are typically partitioned into 15 bins. The number of samples per bin does not allow correct estimation of the average confidence accuracy.
22

---

Implementation details
F.1 Models weights
Model weights for CIFAR are from 45
Model weights for ImageNet come from torchvision 42
Model weights for ImageNet-215 21 are from 54
CLIP weights are from OpenAl's Hugging Face
Original weights for T5 and RoBERTa come from the Transformers library 67 The models
are fine-tuned for each task using prompt-based learning 36 For more details, see [5].
We used the model GPT-J http://///huggggg.......///eaaaa 2.0 license) and Llama-2 http:://huggnnggacccco/// http://f.....cccoo.aaeaaooo
(Apache
httpa///////////ggoo///////////a 2at.aaamaaaaaaa---2-m Ili-
cense).
F.2
Datasets
CIFAR-1O and CIFAR- (C100) [27 contain 60000 32x32 images corresponding to 10 and 100 classes, respectively. Data is split into subsets of 45000/5000/1 10000000000000 images for train/validation/eest. We concatenate the original validation and test sets, and randomly split that into a calibration set of size 5000, and a test set of size 10000.
ImageNer (IN) 77] contains 1.3 million images from 1000 classes. Following 17
randomly split the original validation test of size 50000 into a calibration set and test set, both of size 25000.
ImageNet-22 (IN2/K) [54] in winter21 version, contains million images in the train set, and 522500 in the test set (50 for each of the 10450 classes). We randomly split the test set into equal-sized calibration and test set (261250 samples each, 25 per class).
Amazon Fine Foods 4% a collection of customer reviews for fine foods sold on Amazon. Reviews are categorized into bad, neutral, and good. The original validation set size is 78741 and test size 91606. We randomly split them into 78741 samples for calibration and 91606 for test.
DynaSent 51 is dynamic benchmark for sentiment analysis consisting of sentences annotated as positive, neutral, and negative. The original validation set size 11160 and test size 4320. We randomly split them into 11 116 samples for calibration and 4320 for test.
MNLJ (66) contains pairs of sentences labeled as contradiction, neutral, and entailment. The original validation set size 19635 and test size 9815. We randomly split them into 19635 samples for calibration and 9815 for test.
Yahoo Answers (YA) 7R contains question-answert pairs corresponding to 10 different topics. The original validation set size is 14000 and test size 60000. We randomly split them into 14000 samples for calibration and 60000 for test.
TREC [6 contains questions categorized into 6 classes. The training set contains 5500 labeled questions, and the test set contains another 500.
SST-5 (55) contains 11855 sentences corresponding to sentiments (from very negative to very positive).
DBpedia (74] contains text for topic classification with 14 classes. The training set contains 560000 samples, and the test set 5000.
F3
Code
We used the library netcal 32
(Apache-2.0) license) for their implementation of binary methods for calibration and adapted their reliability diagrams code. For HBB we tested equal- size and equal-mass bins, and chose the best variant for each case. All hyperparameters were kept at their default values (10 bins for HB).
We took inspiration from the official implementation of temperature scaling: hhttps// (MIT license).
github. com/ htt.g//isstemppratrreeereeee/eee
23

---

We took inspiration from the official implementation of Dirichlet calibration:
https
//github. com/ htt.a//aaalllllllaaoooooooooooo. dnn (MIT license).
We used the official implementation of I-Max: httg:///gthaa.coo/obssseeeeerrr
imax-calibratiin (AGPL-3300 license).
We used the official implementation of IRM: http:///ttaabb.c///bbgggg./
Mix-n- Calibration (MIT license).
For evaluation, we used codes from https://githut htt.:/ttuub.cmm///eeeniiiiiooooo
uncertainnty-merric--- (Appchhe license) and httpt:tttptptiiiiiee Cacchuu..ommmdooaaaa IdoGalil/
perfoorraaeeeoo (MIT liccene))
3 (BSD-style license). We used PyTorch 2.0.0
We used CIFAR models from
45 https://gtaab.ccm//rr////ooooo
calibration (MIT license).
We used ImageNet-21K models
http:://gttab..cn//AAbbb...
ImageNeet21K (MIT license).
We used CLIP models from HuggingFace's Transformers library 67 (Apache-2.0 license).
We used pretrained language models from Transformers [67 and calibration codes from S
(MIT license).
We used code from tion of LLMs using ICL, itself built upon https::/github..com http://idbbb.commmconzhhooho
htt.gtaww.../mmmoiiaaaaaaaaaaaaa for the calibra-
few-shot- learning (Apache-2.0 license).
Table 3: Computing time (in seconds) of the calibration on ImageNet, using one NVIDIA V100 GPU, The first column denotes the data preprocessing time, which includes computing the model logits for all calibration examples. Post-hoc calibration methods do not usually require much computing power compared to classifier training.
Modell
IKM
V8
III
Ise
IBBQ
ResNecr-0
2021
66
G Impact on selective classification
Selective classification aims to improve model's prediction performance by trading-off coverage: reject option allows to discard data that might result in wrong predictions, thus improving the accuracy on the remaining data. A strong standard baseline uses thresholding on the maximum softmax probability outputted by the classifier
[3]. Improving confidence calibration means uncertainty is better quantified and should result in better selective classification.
Results in Table show the superiority of Histogram Binning (applied with the right framework) in reducing the calibration error ECE. Unfortunately, does not translate into improvements in selective classification. AUROC standard metric for selective classification [1I. Table shows that Histogram Binning actually degrades the AUROC, while the best method is Isotonic Regression. Our TvA framework does not significantly impact the AUROC.
This paper addresses confidence calibration, usually measured by ECE, AUROC a global rank- based metric for selective classification: relies the relative values of the scores, not their absolute values. Even though calibration and selective classification are related, improvement calibration does not directly translate better selective classification. This has been clearly demonstrated experimentally by [II.
good example of that difference is the behavior of HBBTAA: : is the best calibration method overall but actually degrades the AUROC in most cases. Such a difference can be explained by the fact that able discriminate between certain and
selective classification benefits from continuous uncertain examples finely, but HB quantizes the confidences into, e.g., 10 different values.
24

---

Table 4: AUROC in % (higher is better). Methods in purple impact the model prediction, potentially degrading accuracy: methods in teal do not. Improvements from the uncalibrated model are colored in blue and degradations in orange.
(a) CIFAR-IN 10
Made
Unral.
IRM
1-Max
1S
18
VS
V8rcis
DC
DC
Bota
Bett..
Ive
BINQ
BRIQQ..
[BII
IIRl.
305Net-30
92.09
01.79
91.30
92.01
91.95
92.71
92.76
92.71
92.76
91.94
92.09
92.29
91.87
75.63
85.53
75.78
$4.09
1esNet-110
92.26
92.25
91.53
92.19
92.18
92.18
92.31
92.16
92.30
91.33
92.26
92.40
92.20
73.44
85.01
74,46
$4.56
WRN
91.17
91.17
90.95
91.21
91.19
91.50
92.32
91.82
92.31
90.45
91.17
92.28
91.18
75.99
66.62
73.91
55.38
DenseNer
50.46
90.15
99.65
90.58
90.85
90.84
91.30
90.84
91.39
39.98
90.95
91.46
90.02
77.07
$7.07
74.61
$3.00
CLI OCT 1222)
59.85
89:72
89.23
89.97
89.91
90.73
91.07
90.55
91.14
90.58
89.85
90.28
89
88.50
$9.41
88.00
59.76
CLI (WT--2763)
91.00
90.96
90.72
91.10
91. 10
91.29
91.99 599
91 88
06
91.75
91. 10.00
50. 09
90.85
89 36
90.57
88: 12
89.97
CLP (WT--/14)
99.22
95:15
94. 32
93.32
91.31
93.00
93.78
93.79
93.88
93.17
99.22
90.78
93.05
87.30
91.73
68.65
91.33
(b) CIFAR-100
Midel
Taral.
IRM
1-Maa
TS
18,s
VS
v8,.ms
DC
IC.....
Botta
Bellayy
Ive
BBQQ
IRQQ
10ll..
RecNea-50
55.80
85.76
85.03
65.23
85.60
65.90
66.50 50
85.91
86.51
$5.49
65.80
87. 1919
85.69
81:21
55.31
61:00
63.00
Ne-Ne 10
55.00
84.90
83.26
84.94
84.85
64.82
S5.34
$4.84
85.53
84:57
65.03
56.32
84.07
80 80-54
$4.47
61.44
62.93
WRN
$7.90
87:52
87.02
87.90
87.48
67.83
58.17
$7.85
88.14
87.45
87.90
58.65
87.46
83.59
96.46
83.77
65.69
DeneeNee
56.17
86.02
85.64
86.12
86.00
$6.50
56.79
$6.39
85.79
95.01
86.17
$7.09
36.
33.02
35.42
83:47
54.86
CLIP (N7F-122)
63.06
83.14
83.
83,70
83.95
16
$3.99
00
84.17
63.06
54.61
82.95
85.59
33.00
85.61
62,80
CLIP (NTT-1911))
$0.57
83:51
61.66
06
18
87.57
$4.45
82.51
85.76
60.24
CLIP (NTT-1/14)
$4.26
84:23
84.14
96.56
57.52
87.50
84.26
$6.04
84.11
(c) ImageNNtt
Model
Uscal.
IRM
1-Max
T3
18
VS5
V8..n>
DC
DC
Beta
Belinos
Ise
1shns
BBQ
BBQQ..
100...
ReoNer-18
63:73
RosNet-34
85.78
86.13
$6.41
ResNet-0
50.53
80.13
85.0
55.57
$5.58
56.91
83.63
ResNet-101
54.18
83.57
$5.56
57.09
ErNNee
64.92
86-61
55.51
81.91
57.34
81.57
50:38
ErNNetV2.
55-77
LrtNeeV2
62.36
56.51
LHNNeVV2-L
54.83
$1.78
ComeNeXi55
ComNNeG-R
62.35
VIT-8/52
65.57
85.60
55.00
55.33
83.56
55.55
$2.60
65.11
85.55
$4.92
85.32
65.36
85.53
85.34
85.50
$1.32
65.52
87.19
85.50
$1.56
85.39
51.21
65.09
VYT-1/32
55.42
85.45
64.74 76
85
65.73
K5.19
85.29
85.20
65.30
82.07
65.42
57.25
85.41
81.51
$5.40
$1.42
65.09
V9T-1/16
65.85
85.8%
84.63
85,
86.00
$4.31
$4.65
86.32
$4.64
80.96
65.85
56.90
85.83
79.76
35.66
10.08
64.85
VIT-H/14
57.28
87.24
85.00 60
87.5 51
87
86.30
56.71
86.77
$6.75 74
82.15
67.24
56.65
87.32
79.33
56.54
50.08
65.30
Swin-T
55.68
85.71
85.04
86,
86
65.79
55.86
$5.80
85.86
83.14
85.68
$7.10
35.66
82.15
$5.64
51.
65.16
SwinSS
$5.37
55.36
94.75
85.99
85.78
$4.99
55.30
$5.00
55.20
80.92
65.37
56.02
55.35
$0.25
$5.32
50.05
84.80
Swin-B
$4.11
84.26
83.38
83.3
84.91
$3.53
$4.38
53.92
$4.20
79.78
84.11
56.55
$4.19
75.94
$4.24
75.00
53.09
SwinV2-T
$5.90
85.79
85.10
86.74
86.56
55.82
$6.04
$5.80
56.02
$3.06
85.80
$7.29
35.78
82.58
55.76
$1.47
55.26
S
55.75
85.75
$5.03
86.61
86.39
$5.19
55.51
$5.19
$5.50
91.77
55.75
$7.18
35.74
80.54
55.75
50.41
54.73
SwinV2 B
$5.15
85.13
$4.19
86.07
83.82
$4.43
64.70
$4.40
$4.65
$0.31
85.15
56.99
85.16
79.57
$4.58
79.42
$3.86
CLIP (VT (/22)
50.56
80.55
90.24
80.59
80.57
$1.73
63.21
77.72
77.92
$1.14
50.55
$1.43
50.52
81.85
50.53
$2.14
90.18
CLIP (VT-1116)
$1.12
$1.05
90.95
81.16
61.15
$2.29
83.64
78.10
$4.57
$1.52
61.12
$2.17
81.0
82.35
$1.07
63.42
90.76
CLIP (WTFL/14)
$2.98
82.95
82.50
82.90
62.86
63.06
$5.07
30.09
79.85
82.75
62.98
83.29
82.95
82.85
$2.80
63.87
62.36
(d) ImageNet-21K
Modd
Uncal
1RM
1.Max
78
15.,
V5
DC
DC...a
Bota
Betin.
BBQQ
BBQ
HBB
MNS
68 70
em
em
67.50 80
65.89
80.00
81.00
61..44
51 14
emr
69. 79
79.62
68:77
em
69.77
90.86
68:40
VIT-R116
72.90
em
74.36
73.17
79.78
$1.42
76.20
78.92
72.99
79:46
73.10
73,20
90.27
71.95
(c) Amazon Fine Foods
Medel
Unaal
IRS1
1-Max
TS
1S...
VSS
VS
DC
Beta
Botta..
Ive
Info..
BBQQ
HBQQ...
EIB...
T3
85.04
$4.59
84.32
85.11
85.11
87.84
87, .9.99
$8.08
88. 02.02
$7.46
85.04
$7.00
$3.04
85555 35.53
$3.47
80.53
79.41
TS large
$1.53
81.34
80.52
80.75
80.74
85.5 $5.84
87.
$7.47
87 $7.80
$7.27
$1.33
$5.15
$1.38
83.35
78,22
77.53
74.25
83.52
$3.50
80.50
93:34
83.34
84.96
86,
86.4
86. 56
36.27
83.51
83.93
83.51
81.0 $1.02
25.00
75.00
73.35
RobsEiUT--arre
87.88
87.67
82.16
87.99
87.99
88.04
63.25
68.70
88
88.40
87.55 67.88
87.54
67,68
$1.52
24.90 90
25.63
25,30
(f) DynaSent
Madel
Utcal.
BRM
1-Max
T8
1S...
DCC
ISC... m.
Hita
Norta..
Ise
Inh...
BBQQ
BINQ...
11B
FIll..
TS
77.55
77.51
78.12
76
78.
78
78.01
77:91
76.
25.04
TSsargo
77.69
76.63
77.81
77
77.83
70
71.61
79:30
77.61
73,
RoBLRTa
75:15
72.77
15.30
75.47
75,00
76.05
75.31
75.16
76.10
75.06
64.47
66.97
RoUURTT large
75.83
72.80
76.34
76.34
75.25
76.52
76.12
76.18
76.13
76.06
61.33
59.53
68.27
(g) MNLI
Medel
Uncal.
IRM
TS
1S...
Vie th
DC ns.
Beta
Beta..
Ise
Iseo..
BBQ
BBQ...
E1B
EIB..
$4.37
$2.08
82.01
72.69
82:99
92.48
83.49
83.63
83.58
83
83.58
74.25
(h) Yahoo Anwsers
Modelt
1S...
VS
DC
Betta..
Inno..
BBQQ
BBQQ..
15
$3.60
$1.55
81.10
31.61
81.61
81.64
K3.65
$1.63
81.60
$1.59
$1.60
$1.70
$1.53
30.46
50.85
78,99
90.01
TS large
$1.09
$1.00
80.45
91.08
91.07
81.16
81.17
$1.22
81.33
$1.07
81.09
$1.15
$1,05
79.24
79.35 09.89
77.41
79.06
78.03
78.59
76.52
78.80
78.80
78.23
78.20
788 39.90
79.07
78.68
78.63
86
78.57
74.45
73.17
23.20
67.91
79.13
79.08
76.92
79:41
79:41
79.30
79.66
79.32
79.81
79.34
79.15
79:35
79.04
75.00
72.89
23,67
72.95
25

---

H Additional results
Table 5: ECE in % (lower is better, best in bold) full results for image classification datasets. Averages on 5 seeds. Mean relative improvemets from TvA are shown (negative values for reductions of ECE.). Methods in purple impact the model prediction, potentially degrading accuracy; methods in teal do not. Values are averaged over five random seeds.
(a) CIFAR-I0
Model
Uncal
IRM
1.Mox
18
15..
Y8
VS
DC
DC
Beto
Belin.
Iss
Istea
BBQQ
BBQQ..
HHB
HiB...
RosNis-90
1.50
0.68
0.91
2.16
0.74
1.27
RoNNs-100
2.57
0.54
1.36
1.35
1.35
2.97
1.20
0.56
1.45
142
WRN
1:21
0.78
0.64
1.10
0.22
1.19
1.19
1.75
1.16
0.82
0.88
0.79
1152
DetseNet
1.52
0.59
1.32
1.61
1.21
1.19
2.04
0.58
1.07
1.00
1.30
improvement CoryNNts
116
25%
39%
31%
57%
CLIP (WIE-/99)
4.77
1.39
1.35
1.02
1.79
1888
2.80
1.46
1:79
1.05
2.92
0.998
CLIP (VIE-816)
5.30
1.04
0.95
1.91
2.92
1.58
2.14
CLIP (VIE-L/14)
4.93
1.98
1.94
0.81
1.01
Mean improvment CLIP
81
26%
2616
36%
36%
59%5
(b) CIFAR- 100
Model
Uncol
IRM
1-Max
TS
TN..
DCC
DC
Beth
Beth..
1ue
IBQQ
IIBQQ..
IIB...
RcsNNct-00
6.56
1.37
1.35
4.93
2.97
5.23
2.24
5.22
2.24
5.59
3.39
5.70
1.40
10.07
1.47
9.62
1.37
Res/ees110 1.0
7.95
1.40
1.31
5,05
4.04
5.32
2.65
5.30
2.70
5.10
4.74
6.59
1.34
5.53
1.44
10.04
1.23
WRN
4.41
1.24
0.95
4.42
2.70
4.57
2.25
4.55
2.27
4.50
2.75
4.42
1.41
10.02
1.26
8.45
0.94
DenseNer
5.23
1,20
0.97
4.19
2.12
4.33
2.21
4.51
2.20
4.99
2.84
4.61
1.35
9.91
1.24
10.12
0.76
Mron imgroverent ConcNNets
-37%
-57%
-529
-999
-73%
-86%
-80%
(VIT-B/R2)
9.51
8.74
7.97
1.98
8.33
(VIT-B/1))
10.63
8.64
3.04
5.14
1.50
2.75
7.00
CLIP (VITL)14)
10.96
3,15
2,68
5,96
2.06
6.54
1.74
3.99
2.46
6,46
improvement CLIP
-80%
-64%
(c) ImageNet
Model
Uncal.
IBM
1-Max
18
18...
VS....
DC
DC
Bota
Bataos.
Boi
Ielys
BBQQ
IRRQ...
11B...
RosNet 18
2.71
1.01
0.57
1.87
1.85
1.75
2.12
3:45
3.44
4.39
1.44
3.87
0.54
9.58
0.91
9.54
0.57
RecNet 34
3,63
0.84
0.56
1.77
1.90
1.87
2.02
3.46
2.98
4.97
1.11
4.50
0.83
9.17
0.55
5.42
0.60
RecNet 50
41.15
2.56
2.59
3:25
1.66
3.26
0.50
3.27
0.94
11.30
2.30
1.23
0.68
8.44
0.66
5.80
0.50
ResNer-101
13.56
0.82
0.58
5.72
2.22
4.22
1.62
4.20
1.58
9.21
1.87
3.01
0.71
6.35
0.61
6.18
0.52
Miss imgrowement
Re
-225k
-26/6
-57%
-76R%
.697%
-91.%
-92/
EctNet B7
12.61
0.61
0.40
3:71
2.96
3.84
1.41
3.82
1.35
9.32
2.27
2.93
0.65
6:93
0.3%
4.89
0.40
LitNeVV2
16.92
0.68
0.43
3.60
3.34
3.91
1.43
3.90
1.45
K.03
2.59
2.90
0.67
7.66
0.68
5.33
0.47
LnNNeVV2 24.38
0.30
0.70
3.77
2.71
3.86
1.16
3.82
1.14
8.32
1.999
2.89
0.75
6.55
0.75
4.36
0.49
B-NevV2-L
3.65
0.63
0.39
3.86
1.34
3.08
1.05
3.06
0.98
9.45
0.99
2.51
0.64
5.06
0.54
2.99
0.37
Mess ingrovement EITNes
-27%
-66/6
-789%
-766%
-0099
-300%
ComNeeI:
7.34
ComNeXi
7,46
ComNeX:
4.10
0.77
7.92
5.02
Com-NXX-L
4.02
7.02
inpeemement CuncNoXi
-66/6
-00%
B/32
6.37
0.77
0.60
4.02
2.17
4.67
6.58
1.68
0.94
9.51
7.76
0.53
B/L6
5.61
0.36
0..4
3.30
3.25
4.29
1.95
7.36
2.29
0.79
5.38
6.79
0.51
WIFL/2
4,21
0.35
0.75
5.00
3.89
5.37
2.55
2.49
6.33
2.57
0.76
9.31
7.39
WIFL/1
5,17
0.99
0.77
5.77
4.63
5.29
5:27
2.58
7.44
3,05
4.10
0.85
6.83
0.78
7.38
0.54
VI-R014
0,60
0.60
0.40
1.84
0.88
1.95
1.22
2.00
1.17
7.84
0.75
2.48
0.62
1.67
0.63
3.60
0.42
Moss imprevenent
VIT
-31.%
-51%
-538
-70%
-78%%
-85%
-925%
Swin T
6,82
0.76
0.45
3:08
1.85
3.45
1.38
3.44
1.33
7.72
1.61
2.94
0.72
6.72
0.67
6.31
0.43
Swin 5
3,65
0.78
0.54
3.65
2.95
4.17
1.37
4.17
1.76
7.91
2.31
3.29
0.77
7.20
0.30
5.72
0.44
Swin-B
4.77
0.72
0.45
5.88
3.43
4.22
1.98
4.21
1.96
7.95
2:27
3:33
0.75
6.83
0.68
4.20
0.52
SwinV2-T
8.51
0.30
0.46
5.61
3.25
3.92
1.51
3.91
1.49
6.68
1.76
3.08
0.81
7.61
0.79
6.20
0.52
SwinV
6.07
0.75
0.46
5.79
3.50
4.74
1.54
4.93
1.71
R.51
3.76
3.16
0.74
7.18
0.67
5.00
0.41
SwinV
5.50
0.69
0.59
5.82
3.68
4.25
1.50
4.02
1.73
7.53
2.68
3.34
0.67
6.77
0.63
4.42
0.55
Mean impcovement Swits
-21%
-38%6
.5%%
-73%
-77%
-90%%
.11%
CLI (WT-B/32)
1.50
0.56
0.78
1.70
1.58
1.38
0.92
35.01
70.52
3,57
0.82
2.22
0.84
8.12
6.63
0.82
(WT--/16)
1:31
0.75
199
1.61
0.87
$4.02
66.04
4.53
2.35
1.01
8.48
0.91
6.96
0.74
<I 1P (VIT-) /14)
2,57
0.97
0U67
2.04
1.99
1.89
1.36
06.06
66.40
5.95
1.35
2.49
0.92
8.33
7.86
Meus Ingrovement CLIP
.4%
-36%
-7%%
.61%
-89%
.89%
(d) ImageNet-21K
Medel
Uncal.
1.Max
18..
YS
VS...
DC
Bete
Betns.:
Iso
Isho
BBQ
BBQ
MN3
12:34
8.69
4.39
2.52
2.40
$9.84
81.16
1.02
2,00
0.21
0:20
5.50
0L17
V0T-8/16
6.27
8.92
6.55
2.38
1.54
5.22
3:20
3.72
2.14
0.22
0.24
7.89
0.12
Mean improvement
38%
20%
412
90%
98%
26

---

Table 6: ECE in % (lower is better, best in bold) - full results for text classification datasets. Averages on seeds. Mean relative improvements from TvA are shown (negative values for reductions of ECE). Methods in purple impact the model prediction, potentially degrading accuracy; methods in teal do not. Values are averaged over five random seeds.
(a) Amazon Fine Foods
V8..s.
DC
Beibu.
IIBQ...
0.28
1.26
1.34
1.25
0,80
0.41
1,34
1,49
1.34
1,63
Meen
15
4%
-6%
14%
-78%
-20%
-28%
-9455
RoBERTs
7,50
0.28
0.30
2.27
2.21
1.37
199
1.51
1.76
7.48
4,30
0.31
0.26
1.11
0.37
4.07
0.34
RoBERTT-large 6,83
0.32
0.25
2.52
2.44
1.45
1.78
124
1.57
6.36
4.45
1.72
0.26
0.38
0.34
3.96
0.16
Mean imprevement RodlKTb
36
32%
2216
36%
37%
39%
9816
(b) DynaSent
Unral.
8-Max
18..
188
Vi
DCC
DC
Meta
Bettus
Ieno.
BBQ
RBQ
FIl
MIK..
1.16
2.04
1.53
large
3.19
7.38
2,03
7.15
4.72
1.62
improvement
-10%
RoBERTa
1.50
13.20
13.20
7,83
15.02
10.45
1.64
1,19
1.26
RoBERTo-lagee
14.58
10.94
1.05
Meat improvement RoBEEKta
54%
42%
31%
14%
(c) MNLI
Midel
Unral:
IRM
6-Max
18
T8..
V88
DC
DCCcsss
Beta
Beta..
BRQ
BIQQ..
HIR...
T5
6,45
0.71
0.67
1.37
1.15
3:25
1.58
3.21
2.01
7.74
2.12
0.84
0.73
0.95
0.80
1.91
0.47
T5: large
7,59
0.74
0.72
4.46
4.45
5.66
1.71
5.41
1.64
8.20
4.45
0.77
0.76
1.78
0.57
2.28
0.40
Mean improvemett TS
-1%
-36%
54%
59%
-7%
-4%%
-79%
10:36
0.81
7.83
7.38
11.06
RoBERToolagge 8.18
0.61
6.35
5.51
8.30
0.34
Mian inprevenent RoBERTs
-72%
700%
-5%
.195
(d) Yahoo Anwsers
Model
Uncal.
IRM
B.Max
YS
VS,
DC
ns.
Reta
IB
0.74
0.90
0.6
0.97
2.70
1.01
2,66
0.94
7.50
1.64
1.57
0.79
0.95
0.71
T5. T5-large
9.04
0.57
0.72
1.47
1.73
4.70
1.31
4,54
1.36
10.34
2.39
1.90
0.85
3.01
0.95
3.15
1167
Mei improsemeet T5
31%
.67%
-68%
-78%
-529
-667
-91%
RoBERTa
19.53
1.03
0.32
12.02
12.00
16.26
2.29
15.35
1.25
20.15
9.41
1.96
1.08
5.05
0.72
3.56
0.60
RobERti large
19:68
0.90
0.86
12.77
12:75
16.67
2.75
16.30
2.70
20.18
10.19
1.87
0,94
4.95
0.78
3.26
0.57
Mine inpresement RoBERTs
-0%
-85%
-86%
-51%
-88%
-85%
-83%
27

---

Table 7: Standard deviations of ECE in % for 5 seeds.
(a) CIFAR-
Model
Uncat
IRM
1-Mix
TS
18
V8rgrs
Betn
Betau.
Ise
Ishoa
BBQQ
1IB...
RoNNes50
0.36
0.15
0.15
0.14
0.26
0.25
0.15
0.24
0.36
0.29
0.29
0.19
0.34
0.39
0.12
0.37
11.09
RosNe-110
0.30
0.22
0.08
0.12
0.30
0.14
0.15
0.12
0.15
0.14
0.21
0.22
0.18
0.24
0.20
0.17
11:19
WRN
0.09
0.17
0.21
0.29
0.21
0.25
0.07
0.25
0.08
0.09
0.57
0.25
0.20
0.19
0.22
0.26
11.24
DenseNet
0.31
0.08
0.16
0.12
0.38
0.09
0.09
0.09
0.08
0.13
0.78
0.14
0.35
0.15
0.08
0.19
0.10
CLIP IVT (X3))
0.12
0.89
0.35
0.17
0.13
0.51
0.37
0.30
0.32
0.29
0.32
0.13
0.44
0.55
0.41
0.39
0.32
CLIP NIT BV16)
0.37
0.35
0.31
0.19
0.10
0.22
0.15
0.22
0.13
0.13
0.91
0.18
0.27
0.09
0.25
0.11
0.29
CLIP (WTF-L214)
0.06
0.22
0.21
0.11
0.99
0.11
0.15
0.10
0.16
0.22
0.15
0.19
0.18
0.21
0.11
0.18
0.09
(b) CIFAR-100
Model
Uncal
1RM
1/Max
18
18..
V8
V5.g.s
IBC
Betn
Beth..
1se
Isow
BBQQ
BBIQ...
111]
11B...
RoNNs.00
0.47
0.47
0.37
0.49
RoNNo-100
0.38
0.25
0.31
0.33
0.33
0.34
0.60
0.36
WRN
0.38
0.25
0.24
0.36
0.71
0.33
DesseNet
0.30
0.17
0.30
0.35
0.68
0.31
0.22
CLIP IVIT
0.13
NTT BV16)
0.21
0.42
0.54
0.37
CLIP (VIF)/14
0.07
0.53
0.19
0.23
0.19
0.34
0.99
0.27
(c) ImageNet
Model
Uncal
IRM
1/Mox
TS
Beto
BBQQ
H1B...
Ro
0.34
0.12
0.14
0.11
0.31
0.13
0.13
0.15
0.14
0.38
0.19
0.03
0.17
0.23
0.24
0.22
0.37
RoNNo-34
0.12
0.16
0.24
0.12
0.30
0.14
0.20
0.13
0.17
0.54
0.10
0.24
0.37
0.19
0.21
0.06
11.33
RosNes50
0.21
0.20
0.15
0.25
0.34
0.32
0.15
0.31
0.15
1.71
0.15
0.22
0.36
0.15
0.25
0.21
11.09
ResNe: 101
0.15
0.11
0.22
0.11
0.18
0.15
0.25
0.17
0.25
1.02
0.05
0.19
0.11
0.23
0.16
0.25
11.4
BRNee 97
0.07
0.13
0.11
0.10
0.12
0.15
0.15
0.14
0.22
1.91
0.17
0.13
0.30
0.16
0.16
0.11
1.06
BRNeiV22 5
0.15
0.19
0.08
0.17
0.19
0.22
0.21
0.22
0.17
1.10
0.24
0.25
0.15
0.32
0.15
0.20
0.24
DRNeeVV2 M
0.18
0.12
0.13
0.17
0.15
0.16
0.25
0.18
0.22
1.10
0.47
0.13
0.10
0.24
0.15
0.26
0.06
BRNeiV2-L
0.11
0.07
0.12
0.15
0.12
0.21
0.17
0.18
0.19
1.68
0.25
0.25
0.07
0.33
0.17
0.41
0.55
ConoNeXe-T
0.16
0.08
0.12
0.25
0.28
0.29
0.30
0.28
0.33
1.52
0.42
0.31
0.12
0.23
0.21
0.15
0.10
Com-NcX--S
0.14
0.26
0.18
0.21
0.17
0.34
0.27
0.2
0.78
1.87
0.21
0.15
0.12
0.14
0.14
0.79
0.08
Com-N:X--B
0:50
0.09
0.16
0.30
0.26
0.56
0.33
0.36
0.30
2.12
0.36
0.35
0.10
0.11
0.06
0.40
0.12
ConsNeX0-4.
0.16
0.10
0.09
0.26
0.17
0.21
0.27
0.20
0.28
1.33
0.12
0.28
0.34
0.26
0.19
0.33
0.13
VIT-B/22
0.20
0.17
0.19
0.29
0.17
0.34
0.34
0.37
0.30
0.99
0112
0.21
0.35
0.21
0.16
0.28
1.14
VIT-B/16
0.15
0.07
0.13
0.26
0.17
0.3
0.36
0.35
0.34
0.54
0.24
0.28
0.58
0.25
0.11
0.25
0.30
1/32
0.10
0.15
0.11
0.19
0.30
0.25
0.2
0.28
0.22
0.31
0.17
0.29
0.11
0.20
0.16
0.25
1.14
VIT L/16
0.22
0.24
0.39
0.19
0.15
0.35
0.36
0.35
0.37
0.81
0.12
0.30
0.28
0.51
0.23
0.1
0.22
VIT 1014
0.15
0.15
0.89
0.21
0.28
0.19
0.20
0.20
0.21
0.75
0.15
0.20
0.39
0.34
0.16
0.29
0.18
Swin T
0.17
0.13
0.19
0.19
0.12
0,20
0.6
0.20
0.16
0.70
0.10
0.23
0.15
0.34
0.20
0.12
0.17
Swin-S
0.10
0.18
0.18
0.19
0.19
021
0.5
0.2
0.30
0.48
0.27
0.24
0.16
0.22
0.27
0.30
0.14
Swin-B
0.07
0.14
0.16
0.21
0.25
0.24
0.90
021
0.30
0.80
0.11
0.28
0:21
0.24
0.15
0.34
0.13
SwiaV2-T
0.30
0.14
022
0.00
0.13
0.32
0.11
SwisV2 S
0.17
0.26
Swis22
0.4
0.11
(VIT
0.18
0.17
0.31
0.17
CLI (VI--21)))
0.5
(d) ImageNet-221
Midell
Unenk
I8M
1-Max
18
V8,,
DC
Beta Betayy
BBQ...
|1B
XIN3
0.04
en.
0.10
0.07
0.07
0.08
7.34
17.57
0.25
0.08
0.06
0.44
0.04
0.05
B/16
0.08
0.09
0.16
0.38
0.14
0.23
0.18
0.29
0.10
0.06
0.06
0.05
0.05
(e) Amazon Fine Foods
Midel
Uncal:
1RM
1-Max
18
1S..
vS....
DC
DC
Beta
Betb..
Iso
Bon...
BBQQ
IIBQ...
11B
0.09
0.06
0.10
0.05
0.05
0.07
0.15
0.00
0.14
0.08
0..9
0.11
0.06
0.06
0.04
0.34
0.06
T5-lage
0.000
0.06
3.07
0.06
0.25
0.02
0.18
0.07
0.23
0.09
0.26
1007
0.15
0.05
0.07
0.13
0,00
RodERTa
0.12
0.08
0.06
0.14
0.14
0.07
0.05
0.06
0.12
0.15
0.04
9.07
0.07
0.06
0.09
0.12
0.08
0.10
0.11
0.06
0.89
0.10
0.05
0.16
0.99
0.26
0.13
0.07
0.08
0.04
0.07
0.13
0.05
(1) DynaSent
Model
Uneal.
IRM
1:3aax
TS... VSS
vS....
DC
DCC...s
Bota
Beteo..
Ise
BBQ
BIBQ....
HB
HiH...
0.21
0.35
0.56
0.47
0.55
0.22
0.20
0.33
0.20
0.46
0.35
RoBERTT
0.41
0.63
0.54
0.38
0.45
RuBiERT--aage
0.42
0.35
0.69
(g) MNLI
Model
Uncal.
IRM
1-Max
TS
15..
VSS
VS.
DC
DC....
Butta
Betao.
be
Bon..
BBQ
BBQQ..
|BB
HRh..
T5
0.17
0.19
0.20
0.25
0.22
0.20
0.21
0.25
0.15
0.18
0.25
0.25
0.15
0.37
0.24
0.17
0.15
T5 large
0,20
0.16
0.15
0.35
0.54
0.30
0.40
0.34
0.58
0.21
0.21
0.21
0.14
0.12
0.25
0.23
0.26
RolEETT
0.41
0.35
0.25
0.36
0.36
0.37
0.36
0.36
0.80
0.34
0:20
0.17
0.34
0.18
0.27
0.45
0:21
0.19
0.13
0.18
0.12
0.12
0.13
0.16
0.15
0.17
0.19
0.21
0.28
0.16
0.19
0.36
0.27
0.16
(h) Yahoo Answers
Model
Uneal:
18M
1-Max
18
1S..
VX
DC
Moda
Beta..
Ise
Bor...
BBQQ
IIBQ...
HB
HIN...
0.25
T5-lung
0.08
0.09
RoBERTa
0.29
0.44
021
RoDERTA large
0.15
0.06
0.29
0.32
0.20
0.13
1.25
0.24
0.18
28

---

Table 8: Average confidence in %. Methods in purple impact the model prediction, potentially degrading accuracy; methods in teal do not. Overconfidence (average confidence > accuracy) is shown in violet and underconfidence (average confidence < accuracy) in brown.
(a) CIFAR-10
Medel
Ulral.
IRM
1-Max
TS
18
DC
Beta
Betmu.
Iehus
BBQQ
IURQ...
IBBI
Iill...
ResNer-30
94.9
96.5
94.5
94.8
95.6
95.2
95.6
95.5
95.6
95.5
96.3
95.1
95.3
94,9
94.3
94.99
994.9
95.0
RosNer-110
94.6
97.1
94.9
94.8
95.7
95.3
95.7
95.7
95.7
95.7
97.5
95.0
95.4
94.8
94.8
94.8
94.8
94.5
WRN
95.3
95.9
96.1
96.0
95.8
96.5
96.8
95.3
96.3
95.3
97.4
96.0
95.5
96.0
96.2
95.0
96.2
96.0
DenseNet
95.0
95.7
95.0
94.9
95.9
95.6
96.0
95.6
95.9
95.6
96.9
95.2
95.5
95.0
95.1
95.0
95.1
94.9
CLIP (VT (M/3))
69.2
K3.4
87.5
87.3
97.7
57.8
87.5
88.5
87.6
88.55
89.8
86.4
89.7
67.9
89.2
87.9
88.8
$7.9
CLIP (MT- 111-1
90.2
64.8
89.8
89.5
89.8
90,1
89.8
90.4
89.5
900.5
91.1
90.9
91.9
90.2
91.3
90.2
90.3
90.2
CLIP (WIT-L/14)
95.3
90:4
95.0
88.7
94.9
95.0
94.5
94.8
96.6
94.8
98.7
0
95.2
95.2
95.9
95.2
95.7
95.2
(b) CIFAR-100
Medel
Act.
Ulraa..
I1MM
1-Max
YS
18,,
V8
VS..ms
DCC
Brits
Bolby,.
Loo
Telpss
BBQQ
ISBQ
RecNes-50
76:3
K3.9
72.2
22.0
80.9
76.9
81.4
72.5
81.4
77.4
50.3
78.0
81:5
26-9
24.0
75.9
17.6
76.8
RccNes-110 10
75.0
K2.8
75.5
75.3
79.6
75.5
80.0
76.2
80.0
76.2
79.3
76.9
80.6
75.2
71.4
75.2
73.0
73.3
WRN
79.6
83.0
70.5
79.3
83.0
79.3
83.4
79.7
85.3
70.7
81.1
79.6
83.2
79.3
75.5
70.3
77.3
79.3
NenseNet
76.3
81.0
76.2
76.0
79.6
96.0
80.1
76.6
80.1
76.6
98.7
15.6
79.9
75.8
72.6
75.8
74.0
76.0
CLI (WIT-1932)
62.3
50.8
60.7
63.3
62.3
59.0
63.9
58.3
63.5
60.7
62.9
65.2
60.3
57.0
62.3
57.8
62.3
CLIP (WIT-//1))
66.7
56.0
64.5
63.9
65.9
57.1
62.7
68.4
62.0
67.9
66.5
68.5
66.9
61.0
65.0
67.0
CI (WIT-1/14)
76.0
63.0
73.9
25.4
76.
76.0
78.6
73.4
78.4
73.5
74.6
78.5
76.0
72.8
76.0
73.4
(c) ImageNet
Madel
Acc.
Uaccl.
IRM
1.Max
TS
TS...
vs
VS..ms
DC
DC
Beta
Betow.:
Ishrs
BBQQ
IMBQQ...
18BI
100...
ReeNer-18
70.8
69:4
73.1
69:8
00..5
RooNo1344
73.4
73.1
74.3
74.2
68.3
73.0
76.8
73.3
20.3
73.3
RooNer-511
77.9
52.2
801.3
6913
81.2
71.1
ResNes-101
81.7
81.4
85.5
53.4
85.1
33.1
72.5
82.2
K2.0
78.2
EatNec
87.88
55.7
85.5
75.0
ExtNeeV2
85.3
76.3
UHNNeeV2
85.1
DHNNetV2
95.6
ComNeXi-T
CsmNoXi-8
85.0
85.0
84.0
84.0
84.1
CamNeXX44.
26.4
66.2
85.8
85.8
84.5
84.5
84.5
NST-8/32
17.3
77.4
25.7
71.0
75.5
91.2
81.0
$6.8
62.6
82.8
85.3
K2.7
73.8
81.2
$6.0
61.0
78.8
81.0
30.5
AST-1/32
72.0
74.2
77.3
77.2
81.6
78.8
82.2
79.0
82.2
79.0
71.3
77.0
80.8
76.9
73.9
76.9
76.0
76.9
VST-1/15
79.6
78.8
80.1
80.0
84.5
$1.7
85.1
82.1
85.1
83.1
72.9
80.1
83.6
79.7
78.3
79.7
79.8
70.7
VIT-H/14
68.6
80.0
88.6
88.4
90.4
60.3
90.
89.5
90.5
80.5
50.8
4
90.8
88.6
88.3
85.6
89.3
M8.7
Swin-T
31.5
74.7
81.3
81
84.5
$2.6
850
82.7
85.0
82.6
73.7
81.3
$4.0
81.5
78.8
81.5
$0.9
514
Swin-S
83.2
79.9
83.3
83.1
86.8
54.6
87.3
84.7
87.3
84.7
75.3
83.7
86.1
83.1
81.7
83.1
$3.1
83.2
Swin-B
83.6
79.7
83.8
83.6
87.5
55.4
880
85.6
88.0
85.5
75.8
83.6
36.7
83.5
82.8
83.5
$4.0
83.5
SwinV2-T
$2.0
73.7
82.1
81.9
85.6
53.4
86J1
83.4
86.0
K3.4
73.4
82.2
$4.7
62.2
79.4
82.2
$1.5
52.2
SwinV2.5
$3.7
77.7
83.8
83.7
87.5
55.2
88.0
85.4
38.0
85.3
75.3
86.1
36.7
83.7
82.3
83.7
53.8
$3.7
SwinV2
$4.1
73.9
$1.3
84.1
37.9
55.7
88.4
85.8
38.4
85.8
76.7
86.5
37.1
64.2
83.1
84.1
$4.3
54.2
CLIP (MT (/32)
57.3
57.3
57.1
56.7
57.9
57.5
59.3
60.1
67.1
70.7
55.4
57.5
61.88
57.2
52.7
57.2
55.0
57.2
CLIP (VIT-B/1))
62.9
62.5
62.5
62.
6334
63.2
66.6
65.6
69.9
66.2
60.2
62.5
67.4
63.0
59.4
63.0
61.4
63.0
CLIP (WIT-/114)
70.1
72.1
70.0
69.8
70.9
70.3
73.0
72.8
76.9
66.6
65.7
70.5
74.6
70.1
67.3
70.1
69.2
69.9
(d) ImageNet-21K
Midel
Acc.
Ulcal.
I8M
1.Ma |
TS..
v8..o.
DC
DC...
Beta
Betno
Bio
1ahp
BBQ
BBQ
HBB
HB...
MN3
15.4
27.7
emn.
24.0
174
35.5
33.3
71.9
81.2
err
15.1
35.1
15.5
em
15.3
15.7
15.3
VI--/166
19.3
21.4
177.
25.8
21.0
43.4
41.3
48.7
41.5
19.3
43.8
19.2
cm
19.1
20.6
19.2
(c) Amazon Fine Foods
Middi
Aut:
Unco.
IIM
1-Max
18
TS..
VS
DC
DC
Bats
Bottays
Joa
Intoo
BBQ
BRQQ..
IBB
EIB...
T5
59.8
95.0
59.8
39.8
90.2
90.3
90.8
91.5
90.7
91.6
96.0
39.5
90.5
89.8
90.4
39.8
900.2
59.8
TS large
91.5
97.5
91.6
91.6
92.0
91.9
92.3
95.3
92.3
93.3
97.8
91.5
92.1
91.6
92.1
91.6
91.9
91.6
RobiAAiaa
90.0
97.8
90.0
90.0
92.2
92.2
90.6
91.7
90.7
91.7
98.0
50.0
90.5
90.0
90.4
90.0
90..2
90.0
91.4
98.2
91.5
91.5
93.9
93.9
92.3
93.3
92.3
93.3
98.3
91.5
92.1
91.5
91.9
91.5
91.7
91.4
(f) DynaSent
Misell
Acc:
Elwokk
IRM
1-Max
TTS
T%
V8
DCC
DCC
Betta
[IRQ
BRQQ..
IIB
78.8
85.8
78.5
78.5
78.9
83.6
50.5
80.6
89.7
78.7
78.4
75..2
TS-lage
63.2
91.8
$1.7
85.0
39.5
83.9
94.1
/5
$1.88
81.6
RoblKia
77.7
95.1
77.9
90.9
93.5
85.4
92.7
79.4
77.7
78.0
77.8
RobUURTa lange
$1.2
81.2
92.1
$1.7
$1.5
81.0
(g) MNLI
Modet
Acc.
Uncul.
IRM
1-Max
TS..
VS,, mvs
DC n4.
Beta
Bottays
BBQ
BBQQ..
EBB
HIB...
Rofsficts
96.7
96.3
97.1
95.0
(h) Yahoo Answers
Modil
Unekk
IRMM
I-Max
TS...
vS....
DC.
Battayy
BRQQ...
R1S...
15
75.3
92.0
75.6
75.1
74.6
79.0
76.1
75.0
76.0
74.8
76.5
15:4
75.4
75.4
75.0
75.4
large
75.6
$4.6
75.3
15.6
75.4
74.8
80.3
16.3
30.5
76.4
15.9
77.0
75.6
15.8
75.6
75.3
75.6
72.4
91.9
72.6
72.5
$4.4
64.4
36.6
74.66
69.6
74.7
92.8
72.0
74.4
12.3
12.7
72.3
12.4
72.3
72.8
92.5
73.1
73.0
85.6
85.6
89.5
75.6
89.3
76
72.6
74.8
72.9
73.2
72.9
73.0
73.0
29

---

Table 9: Accuracy % (higher is better). Methods in purple impact the model prediction, potentially degrading accuracy; methods in teal do not. Because classifiers can be well calibrated when not accurate (by having low accuracy and low confidence), it is important to monitor the accuracy. even better when the methods preserve the accuracy by design.
(a) CIFAR-10
Midel
Unral.
IBM
1-Max
18
Bettayy
1ntps
BBQQ
1000..
ResNea-50
$4.80
04.89
94.70
ResNes-110
94.55
94.51
94.35
95.79
95.65
95.59
DenseNee
58:17
92.31
(VTT-/14)
95.28
(b) CIFAR-100
Moddl
Uncel.
IRM
1-Max
T8
15..
v8....
DCC.....
Bota
B-db..
1ve
1sh..
BBQQ
BB0Q...
Hll...
ReoNe-50
06.71
76.72
16.71
06.71
76.63
76.50
76.50
76.67
06.71
76:34
76.31
76:71
74.80
76.71
ResNas-11
75.00
75.00
74.96
74.66
74.14
73.12
WRN
79.57
79.57
79.36
79.08
79.57
79.57
77:35
DesseNet
76.26
76.21
76.25
76.26
76.31
76.31
76.30
76.07
76.26
76.26
74.42
76.26
CLIP (N3T-B232)
62.33
61.60
62.33
67.77
63.95
66.40
66.48
62.33
62.33
64.06
CLIP B/16)
66.56
66.10
66.65
66.66
71.35
70.04
70.21
67.38
67.55
66.06
CLIP L/14)
73.96
$0,09
79.02
79.36
75.96
77.05
75.96
(c) ImageNett
Model
Uncal.
IRM
1.Max
BesNe-188
69.77
69.76
69.37
69:77
69:77
69.81
69.21
68.12
67.60
69.72
60:77
69.34
69:37
68.34
69:37
66.65
69.77
BcsNct-34
73.23
73.19
72:77
73.25
73.23
73.19
72.70
71.96
71.53
73.10
73.23
72.51
73.23
72.21
73.23
70.46
73.23
BesNet-30
50.85
80.80
50.26
50.85
50.85
50.93
50.51
$1.94
90.81
$0.60
80.85
50.47
30.85
78.13
$0.85
78.49
50.85
BesNet-101
$1.86
81.83
$1.54
81.86
81.96
$1.77
$1.65
$1.30
$1.06
81.74
81.86
$1.44
$1.86
80.82
$1.86
79. 19.50
$1.96
ErNNet B7
$4.16
84.15
$4.05
$4.15
54.16
$4.45
64.31
$4.45
$4.30
$4.27
$4.15
54.29
$6.16
85.71
$4.16
$2.72
$4.16
ExNNetV2 5
$4.27
88.19
$3.99
$4.27
14.22
$4..3
64.23
$4.34
$4.22
84.26
64.27
$3.58
$4.27
85.52
$4.27
$2.55
$4.27
DHNeeV2 M
55.06
85.05
$1.90
55.00 105
10.00
$5.28
$5.19
$5.28
$5.17
85.
85.00
$4.57
35.06
84 $6.19
55.07
$3.92
$5.06
EHNeeV2-1
55.80 65.80
85.75
$5.60
85. 900
655 800
$5.85
85.83
$5.89
85.67
$5.89
85.55 85,80
$5.58
85 30
85.25
85 50
64.25
85 $5.90
ComcNex-T
$2.50
82.00
$2.14 16
R1:50
81350
$2.44
80.70
$2.45
60.24
82:45
81.59
83 100
$2.50
81.51
$2,50
60:37
82,50
ComeheXi8
53.65
83.59
83. 36
K3665
$1.63
$3.55
63.63
83.55
83.64
R366
53128
$3.65
82.80
$3.65
61
K3,65
ComNeei-B
54.04
84.01
85.76
14
5404
$4.09
$3.08
84.10
83.06
$4.14
54:04
53.88
84.04
83.22
54.04
82
64.04
ComNeXi44.
54.38
84.37
84.23
54.41
54.33
$4.41
84.54
84.44
54:35
54.12
84 84.38
83.08
54.38
83.00
64.38
VIT-8/32
75.95
75.91
75.69
75:95
25:95
75.81
75.65
75.83
75.66
75.85
75.05
75.36
75.95
74.59
75.95
73.13
75.95
VIT-B/16
$1.04
81.01
80,00
131
81.04
$1.00
30.58
$1.01
80N7
80.96
81.04
50.63
$1.04
80.38
$1.04
79.06
$1.04
V/T-L/22
76.96
76.94
76.84
76.96
73.9%
70.79
70.73
76.79
70.72
70.46
76.9%
76.37
76.96
76.04
76.96
74.55
76.9
L/15
79.54
79.64
79.59
79:64
79.64
79,50
79651
7921
72.64
29.66
79.64
79.47
79.64
79.20
79.64
77.82
79.64
VIT H1114
$8.82
98.61
$5.08
85.02
59.02
$3.02
35,50
55.39
$5,46
88.63
55.62
58.1
38.62
98.35
38.62
$7.24
59.02
Swin I
$1.49
81.45
$1.28
81.69
$1.09
$1.85
$1,42
81.35
$1.42
$1.46
$1.09
$1.07
$1.49
90.77
$1.49
79.43
$1.49
Swin
63.21
83:20
83.04
R3,21
83,21
83.13
83.00
83.13
$3.0
83.21
83,21
$2.99
83.21
92.76
83.21
$1.40
83,21
Swin-B
83:57
83.51
$3,00
53.75
83.51
83.76
83.63
83,80
$3.30
33.60
K2.16
53.00
S.inV3.T
$3.00
82:01
81.81
83:02
K2.13
81.08
82.05
67.03
16
81:37
10:08
83.0
SwinV2-
53.74
83.64
83.74
83.81
83.73
83.73
83.74
80:31
63.74
SwinV-B8
$4.01
84
54.03
$4.16
$3.51
(M22)
57.54
55.94
56.08
(d) ImageNet-21K
Mode
Usool.
1.Mux
75
T5..
vs.
DC
DC
Beta
Botoy.
1se
BBQ
BBQ
HIB....
MNS
15.36
13.07
0.00
96
15
16
21:17
VIT-R116
19.18
(e) Amazon Fine Foods
Model
1S...
Vi
Besta...
Inn...
BBQQ...
T5
89.81
99.85
39.85
39.81
89.81
90.04
90.15
90).11
90.34
90.59
89.81
90,00
59.81
50.55 00.58
59.81
99..6
89.81
TS large
91.57
91.58
91.61
91.57
91.57
91.67
91.82
92.04
91.96
92.10
91.57
92.14
91.57
92.14
91.55
91.57
RobEERTs
89.95
39.55
89.96
$9.95
89.95
89.90
90.15
90.29
90.21
90.56
89.95
90.59
69.00
90.59
69.30 95
90.54
69.05
91.42
91.55
91.45
91
91. 42
91.42
9..64
91.74
91 91.69
91.94
91.42
91.98
91.41
91. .00
91.35 39
91.91
91.42
(f) DynaSent
Medel
Ulcol.
TS...
Beta
Bect....
Ise...
BBRQ...
TS
78.81
78,58
78.82
7888
76.83
18.84
78.81
78, 78.80
7%
78.51
76.83
78.70
78.83
78.
78.81
78.77
71.83
TS-largo
82.20
82.16
82.19
82.20
82.30
82.20
K2.21
62. 30
82
$2.36
82.20
82.31
63.30
82. 19
62.30
82.20
82.20
R.BERR
77.23
77.67
77.71
77.72
77.72
77.72
77.24
77.76
77.70
77.57
75.72
77.75
77.72
77. 17.66
77.72
77.97
77.72
RuBERTTilagee
81.19
81.29
81.33
81.19
81.19
81.19
$1.19
$1.24
81.44
31.28
81.19
$1.50
81.19
$1.51
$1.19
X1.66
81.19
(g) MNLI
Medd
Uscol.
I8M1
1.Max
15
15...
VS
DC
DC.,.
Beta
Bethoa
bso
BBQQ
BBQ=..
|IB
BBB..
TS
65.35
98.35
88.38
38.35
88.35
88.
63,
68. 19
88.35 39
$8.41
88:35
$8.5
68.35
$8.30
68.35
88.
$8.35
TS-bagg
90.07
90.10
90
90.07
90.07
90.07
90.15
90.23
90.21
90.00
9007
90.15
90.07
50 06
90.07
90.16
90.07
RoRFACD
86.41
86 40
86.43
86.41
86
86.41
43
66.45
86.48
86-44
85-41
86. 57
66-4
$6-41
96.41
86. 96
86.
83.89
88:00
88.92
$8.89
88.90
88.00
88.96
68.08
89.02
$8.00
88.89
85.95
58. 96
60.89
88..0 92
88.80
(h) Yahoo Answers
Madell
Ulccol
IRM
1.Max
TS
TS...
YS
DXC
DC
Beta
Bela..
Ise...
I1B4Q
BBQQ...
| |1B
HIB...
T5
25.85
75:35
75-26
75.35
15.85
75.34
75.34
75.
75. 35.35
75.34
75.35
75.30
75.35
75:22
73.35
75.25
75.35
T5-kage
25.57
75.60
75.56
15:57
15.57
75.59
75.55
75.49
75.66
75.53
75.57
75.70
75.57
75.5
75.57
75.67
75.57
RUBERTA
72.38
72:37
72.35
72.38
72.38
72.39
72:43
72.74
73.05
72.72
72.38
73.00
72.58
72.97
72.
72.91
72.38
RoBERTa large
72.84
72:53
72:81
72.84
72:84
72.83
72.99
73,01
73.40
72.98
72.84
73.48
72.54
73.29
72.84
73.38
72.84
30

---

Table 10: ECE with 15 equal mass bins in % (lower is better). Methods in purple impact the model prediction, potentially degrading accuracy; methods teal not.
(a) CIFAR-10
Model
Uncal.
1RM
1-Mix
T8
V8rgrs
Betn
Ise
BBQQ
IIB...
RcsNes50
1.51
0.71
0.62
1.33
1.24
1.25
1.29
1.25
1.29
2.03
1.55
0.93
0.70
2.45
0.47
2.35
11.66
ResNee-110 110
2.58
0.48
0.51
1.78
1.72
1.74
1.74
1.72
1.75
2.91
1.56
0.99
0.43
2.59
0.54
2.71
11223
WRN
1,30
0.58
0.51
1.74
1.75
1.48
1.52
1.49
1.52
1.74
2.09
0.95
0.62
2.07
0.54
2.15
0.46
DenseNet
2.04
0.55
0.46
2,00
2.07
1.48
1.70
1.47
1.70
2.03
2.45
0.78
0.56
2.71
0.09
2.75
0.29
CLIP LVIT (L32)
4.33
1.33
1.30
0.93
0.91
2.75
1.79
2.80
1.80
1.66
1.34
1.05
1,35
2.15
1.47
2.47
0.90
CLIP (VI BV16)
5.38
1.12
1.01
0.59
0.48
2.91
1.91
2.87
1.87
1221
1.90
1.05
0.87
1.63
0:91
2.41
0.63
CLIP (WIF-L214)
4,50
0.54
0.50
0.56
0.48
1.98
1.74
1.9
1.72
0.96
0.94
0.55
0.46
0.95
0.51
0.96
0.37
(b) CIFAR-100
Midel
Uncok
IRM
1-Max
18
VS
DC
Beth
Betbhy
Jus
Inh.a
[BQQQ
IIBQQ..
118
RenNee-SD
6.53
1.55
1:26
5.00
3.54
5.15
2.31
5.14
2:30
5.54
3.90
5.80
1.44
10.10
2.08
10.97
1.19
ReaNe-1100
7:83
1.25
1:18
5.31
4.34
5.11
2.86
5.07
2.90
5.97
4.95
6.49
1.50
9.72
2.11
10.89
1.83
WRN
4.33
1,07
0.95
4.36
2.79
4.05
2.38
4.45
2.37
3.40
2.85
4.26
1.22
10.29
0.94
9.99
0.83
DenseÑee
5:16
1.12
0.86
4.25
2.30
4.47
2.25
4.88
2.25
4.83
2.94
4.50
1.37
10.65
1.27
10.60
1.07
CLIP (VI EUrS)
0.51
2,10
1.74
1,86
1.28
8.72
3.49
7.97
1.99
6.51
2.25
2.32
1.28
7.95
2.13
7:99
1:35
CLIP (VIT-1//66)
10.64
3.35
1.04
2,66
2.72
8.68
3.15
8.20
1.22
7.0%
2.55
2.68
1.39
7.14
2.02
7:03
1.82
CLIP (VIT-121-)
10.96
3.14
2.94
2.47
2.53
6.01
2.05
6.57
1.74
6.5
1.93
3.90
6.65
1.66
6.48
(c) ImageNet
Medel
Cucal.
IRM
E-Max
T3
TS..
V8
VS..a
DC
DC
Beto
Betin.:
Eso
1shrss
BBQQ
BB00s.
HIB...
RooNet-18
1.86
1.71
RosNes-14
3.61
0.52
1.75
1.81
1.11
0.71
9.17
0.20
RosNes-00
2.75
2.60
0.76
5.45
RcsNor-101
0.36
4:21
7.84
6.68
BBNat-B7
6.91
BINNatV2.5
7.58
BrNNetV2
0.73
6.68
UnNNetV2
6:00
5,30
ComNeX::1
ComNoX-B8
Csm-NXX-1.
4.42
6.27
VIT-B/32
4.64
0.71
9.34
5.30
0.58
B/16
5,56
0.75
0.55
4.19
3.18
4.27
7.24
2.39
0.20
7.68
7.12
0.57
WIFL/Z
4,13
0.35
0.25
5.30
5.37
2.67
2.64
5.19
0.22
9.18
1.05
3.84
063
WIEL/S
5,17
1.02
0.59
5.92
5:20
5228
2.54
526
2
7.36
3.59
4,10
0.76
7.50
1.29
3,50
0.6
VIEH044
0.61
0.56
0.36
1775
0.83
1388
108
192
1.05
K00
0.70
2.46
0.60
3.96
0.53
4.64
041
Swin-T
6.82
0.77
0.49
5.10
1.82
3.45
1.35
3.4
1.33
7.70
1.67
2.54
0.71
7.50
0.90
6.87
0.48
SwinS
3,57
0.59
0.52
3.92
2.98
4.17
1.44
4.17
1.82
7.88
2.33
3.20
0.59
6.78
0.33
6.97
0.50
Swin-
4.65
0.60
0.36
4.36
3.71
4.22
2.04
421
2.04
7.89
2.81
3.33
0.63
6.50
0.7%
6.64
0.41
SwinV2-T
8.31
0.72
0.46
3.62
1.21
3,92
1.60
3.00
1.59
K.65
1,83
3,07
0.71
7.97
0.67
6.84
0.52
SwinV2-S
6.06
0.64
0.45
4.18
3.32
4.24
1.88
4.5
1.53
8.45
2.34
3.15
0.56
7.04
0.79
6.75
0.50
SwinV2.B
5.27
0.51
0.46
4.42
3.68
4.25
1.95
4.22
1.58
7.47
2.74
3.33
0.57
6.56
0.75
6.33
0.42
CLI (MT (/22))
1.51
1.00
0.73
1.62
1.56
1.42
0.81
36.01
70.32
3.59
0.64
2.25
0.98
7.58
1.04
6.66
0.79
CLP (VT (/16)
1.78
1.25
0.74
190
1.83
1.00
0.30
02
66.04
4.51
1,07
2.31
0.95
8.17
0.53
6.96
6.71
CUI (VIT-L/14)
2,54
1.37
0.64
2.03
2.04
1.79
1.32
25.05
66.39
5.93
1.50
2.39
1.10
9.00
0.58
7.87
0.84
(d) ImageNet-21K
Medd
Uncal
IRM
1.Max
18
15..
VS
V8
DC
DC
Beta
Betoy.
1sh.
BBQ
BBQ
HB
HB:
MNS
12.36
cm
om:
5.69
4,43
2.51
2.38
$8.64
$1.16
err.
1.14
1.90
0.28
cm
0.31
5.53
0.19
WT-R166
6.54
<m
9.03
6,86
1.5
16
3.50
3.73
2.14
0.21
<m
0:43
7,00
0.17
(c) Amazon Fine Foods
Midel
Uncal.
IRM
1-Mas
T8
1S..
VS
VS....
DC
DCC...
Bosta
Betbow
Iso
Bon...
BBQQ
IBQQ...
IBB
10h...
5.17
0.28
0.22
1.35
1.36
1.35
1.43
0.99
1.30
5.44
0.27
0.47
0.29
1.14
0.21
2.51
0.21
T5 Jurge
5,76
0.27
0.22
1.82
1.84
1.71
1.67
1.37
1.38
3.71
1.81
0.75
0.26
1.97
0.17
3.18
0.17
RoDlRTaa
7.89
0.35
0.25
2.27
2.21
1.46
2.11
1.60
1.90
7.47
4.29
0.45
0.52
2.73
0.22
3.76
0.21
RoDERTa-lagg large
6.62
0.36
0.18
2.53
2.45
1.47
2.13
1.2
1.90
6.35
4.2
0.56
0.28
2.81
0.24
3.68
0.16
(f) DynaSent
Model
Unral.
IRM
8-Max
TS
18.,
VS,
DC
ICC,,
Neta
Betbyy
Ise
Iens,.
BBQQ
IIQQ..
FIB
7.92
1.76
1.35
1:60
1.60
4.75
2.12
4.58
2.20
2.8%
1.46
1.89
2.00
1.51
1.31
75-lagge
3.35
7.25
2,00
6.98
2.07
1.79
RoBERTa
17.34
2.38
13.14
13.14
7.68
14.96
6.25
10.36
1.27
RoBERTa large
14.50
1.51
1.14
19.85
10.87
13.43
5.53
12:71
4.53
9.09
1.93
1.41
2:75
1.09
(g) MNLI
Model
Uncal.
IBM
L-Mas
VS...
VS..
DC.,...
Beta
Betaus
Ise
BBQQ
BBQQ..
HEP,..
3.21
5.31
RoMeECIa
1.84
(h) Yahoo Answers
18..
V8
DCC
Balliyy
Inno..
NIRQ...
11hh,.
0.53
0.90
0.98
2.66
0.92
2.64
0.88
7.68
1.68
1.47
0.36
0.86
0.52
Jagge
9.04
0.38
0.68
1.46
1.71
4.67
1.26
4.82
1.27
2.36
1.81
0.56
3.72
0.89
0.66
RoBERTT
19.53
0.39
12.02
12.00
16.26
2.18
15.35
1.70
20.15
9.36
1.94
0.99
5.15
0.72
6.48
0.62
RoBSERTT
19.65
0.92
12.06
12.75
16.67
2.34
16.29
2.68
20.16
10.11
1.87
0.94
6.53
0.74
6.70
0.60
31

---

Table 11: Brier score of the predicted class in 10-2 (lower is better). Methods in purple impact the model prediction, potentially degrading accuracy; methods in teal do not.
(a) CIFAR-10
Mode
Uncal
13
Betu
ReaNee-50 50
3.75
3.64
3.64
3.67
3,66
3.67
3.65
3,66
3,65
5.82
3,69
3.67
3,63
4.05
3.26
4.00
3.78
Rm-Ne-110
3.55
1.64
3.65
3.25
5.72
3.72
3.73
3.75
3.73
4.02
371
3,62
3.54
4.21
3.26
4.25
3,50
WRN
3.06
3,02
3,02
3.11
3.68
1.11
3.07
111
3.00
3.17
3.10
3,02
3.01
9.37
3.09
9.45
3.33
3.68
3.54
3.56
3.69
366
3.65
3.62
3.65
3.62
5.78
3
3.56
3.53
4.01
3.58
4.01
3,67
CLIP (VIE-R3))
7.63
7.37
7.33
7.27
7.26
6.31
6.19
6.24
6.16
6.43
7.33
6.35
7.35
6.42
7.37
6.62
7.36
CLIP (VIT-8/16)
6.48
6.08
6.07
6.02
6.01
5.15
5.05
5.12
5.03
5:21
6.15
5.20
6.39
5.24
6.08
5.39
6.33
CLIP (VIT-L/14)
3,62
3.20
3.22
3.15
3.15
2,65
2.61
2.63
2,61
2.74
3.20
2.64
3.39
2.76
3.18
2.74
3.19
(b) CIFAR-100
Medd
Uncal.
183
1-Max
15
15
VI5
VS,.m
DC
DC....
Beta
Betino
Ise
109n-
BBQ
BBQQ..
HBBr.
RecNea-50
12:75
11.96
12.07
12.44
12.14
12.30
11.91
12:39
11.92
12:50
12.16
12.08
11:59
14:26
12.12
13.65
12.01
RosNea-1110 10
13.86
12.69
12:24
13.20
13.94
13.20
12.82
13:26
12.82
13:37
13.13
13.00
12.70
14:88
12.50
14:34
12.75
WRN
11:05
10:72
10.34
11.05
10.85
11.03
10.71
11.03
10:72
11.08
10.83
10.77
10.75
12.65
10.52
12.00
10.86
DaveeNia
12.49
12.09
12.10
12.35
12.16
12.30
12.04
12:30
12.04
12:47
12.16
12.11
12.07
14.06
12.33
13.41
12.11
CLIP (MIT-B192)
17:25
16.20
16.12
15.88
15,90
15.98
14.72
16.03
15.00
15.92
16.25
14.97
16.23
15.55
16.31
15.36
16.25
CLIP (MIT-B/16)
17:21
16.01
15.94
15.42
15.43
15.37
13.99
15.44
14.06
15.49
15.87
14.39
15.85
14.69
15.92
14.53
15,88
CLIP (VIT-L/14)
14.78
13:33
13.27
12.70
12.71
11.53
10.67
11.69
10.90
12:59
13.17
11.00
13.16
11.56
13.18
11.38
13.15
(c) ImageNet
Mede
Uncal.
I8M
1.Max
15...
V55
V8..ns
DC
DC
Beta
Boli..
1se
16hp.
BBQ
BBQQ..
H00..
ResNet 18
13.93
13.86
13.85
13.94
13.93
13.92
13.92
14.48
14.53
14.50
13.87
13.89
13.87
15.56
13:90
15,00
13,93
RcoNet-14
13.15
12.99
12.99
13.70
14:49
RcoNas-00
12.33
10.98
10.98
13.14
10:71
11.90
11.08
RosNat-111
12.65
10.70
10.65
10.70
11.97
11.72
11.40
B7
11.28
9.71
9.72
11:02
9.41
10:35
EINNeV225
12.39
9.66
10.61
10:37
LrNNetV2
16.35
10.40
DrNNetV2
9.70
Com-NeNt-
11.79
ComeNeX-B
13.86
10.20
ComoNeX-4..
11.58
0.58
7.6
VIT-8/12
12.68
12:26
12.33
12.54
12.50
13.28
12.11
VIT-8/16
11.02
10.71
11.01
10.53
11.98
12.09
11.54
VIT-L/22
12.15
12.02
12:35
12.49
12.21
12
12.21
13.20
11.92
11:93
13.87
13.45
VIT-L/16
11.39
11.22
13.73
11.58
11.88
12:92
11.13
11.11
12.71
VIT-
7.47
7.51
7.49
7.58
3,46
Soin
11.09
10.59
10.64
10.64
10.71
10.64
10,03
11.68
10.64
10.51
10.11
11.89
11,06
Soin
10.14
9.58
10.04
10.23
10.06
10.32
10.32
10.13
11.40
10.06
9.95
9.98
11.48
10,00
11.14
10.06
Soin U
10,18
9.50
10.00
10.21
10.23
10,10
10.25
10,10
11.42
9,82
9.91
11446
9.93
11,06
10,00
SuinV2-T
11.06
10.33
10.39
10.45
10.29
10.54
10.38
10.54
10.38
11665
10.38
10.25
10.34
11:69
10.36
11.43
10.37
SwinV2-
10.05
9.61
9.67
9.91
9.72
10.01
9.78
10.01
9.71
11.3
9.70
9.57
9.60
11:15
9.54
10.00
9.69
SwiVV-BB 2-B
10.00
9.64
9.70
9.97
9.70
10.04
9.83
10.04
9.83
11.6
9.76
9.59
9.54
11:11
9.56
16.78
9.71
CLIP (WTT-B/92)
17.76
17:75
17.72
17.75
13.76
17.06
16.37
31.17
50.78
17.52
17.74
17.34
17.76
1800
17.79
17.38
17.81
CLIP (VTT-B/16)
16.99
16.99
16,89
16.98
16.96
16.12
15.54
30.53
44.33
16.74
16.97
16.22
16.98
17.14
16.99
16.54
17.03
CLIP (VTT-1/14)
14.96
14.1
14.93
14.97
14.99
14.17
13,67
26.10
47.35
14.55
14.91
14.32
14.92
15.39
14.93
14.57
14.99
(d) ImageNet-21K
Medd
Uncal
IRM
1.Max
T8
15,,
VS
DC
DC,
Bota
Betayo
Iue
BBQ
BBQs.
HB
HBB
MNS
14.08
em
13:35
12:71
17.17
15.26
49.00 94.94
69/66 18:22
emr.
11.95
1706
11.93
em
11.93
10.70
11.95
VIT-R166
13.51
cm
13:54
13:49
18.14
17.11
20
12.96
18:12
12.76
m
12:77
12.91
12.96
(e) Amazon Fine Foods
Model
Uneal.
IRM
1.3ax
T8
TS..
YS
vS...
DC
DC...a
Bota
Belao.
Ise
Bsteo
BBQQ
BBQQ...
HB
HBh..
T.S
2.74
7.27
7.30
7.27
7.20
6.3
6.85
6.7%
6.75
7.30
7:30
6,69
7.28
6.72
7.20
7.19
7.38
TS-harge
7,00
6.34
6.36
5.40
6.40
6.11
5.94
5.76
5.83
6.57
6.46
5.80
6.34
5.92
6.38
6.42
6.41
RoBERTs
8.66
7.30
7.35
7.42
7.42
7.22
7.02
6.87
6.99
8.12
7.55
6.66
7.30
7.11
7.48
7.70
7.57
RaB
7.43
6.15
6.23
6.30
6.29
6.18
6.01
5.89
6.00
6.36
6.85
5.86
6.14
6.21
6.49
6.74
6.22
(f) DynaSent
Medd
Uncol.
IRM
1-Max
15
15..
VS
DC
DC
Beta
Belk..
be
Ise..
BBQ
BBQ
E1B
EIB..
TS
14.08
13:06
13.96
13.89
13.89
14.15
13.64
14.09
14.02
13.84
13.98
14:00
14.05
14.21
13.95
TSstarge
12:48
12:52
12:52
13.08
17.21
12:16
12.80
12.14
17.50
17.60
12.76
12.54
RoRFFRD
14:06
17.3
17.03
15.59
17.66
1662
14.97
15.64
15.06
15.01
16.14
13:31
13.35
14:65
14:65
15.54
13.51
15.29
14:70
13.09
13.30
13.91
13.83
14.20
13.25
(g) MNLI
Moxell
Uncaat.
IRM
B.Max
T8
TS..
VSagas
DCC....
Beta
Botby.
1se
Ise...
BBRQQ
IIBQQ...
HBB
1EB...
8.21
8.21
5.28
75-larpe
8.56
7.52
7.76
7.76
7.98
7.49
7.80
7.7%
8.67
7.54
9.67
10.24
10:23
10.57
9.57
10.48
10.05
10003
10.48
9.50
(h) Yahoo Answers
Medel
Uncol.
IRM
TS
TS...
VS....
DC
Beta
Beto..
Ise
Ioe...
BBQQ
BBQQ..
|HB
HB...
14.18
14:21
14:14
14:15
14.23
14:14
14:21
14.13
14:20
14.18
14:41
14.34
14:75
14:34
14.35
14.25
14.44
14.21
14.41
14.28
14.34
14.49
14.33
14.29
RuBBERT
15.04
17.57
17.56
19.20
15.92
18.84
15.54
17.15
15.64
15:04
16.75
16.55
17.20
15.99
RoBERTI--lage
20.94
15.68
17.57
17.57
19.23
15.99
18.96
15.30
21.21
17.14
15.38
15.68
16.71
16.42
15:72
32

---

Table 12: Calibration methods applied in-context learning of LLMs. Accuracy and ECE are in %.
Dataset
TREC
SST-5
DBpedia
Acc: (1)
BCE (
Acc. (T)
ECE ())
Acc. (1)
ECE (
Model
Shots
Method
Uncalibrated
24.7
29.7
33.7
27.4
ConC.
40.0
14.0
40.7
10.3
47.7
24.6
LinC
58.9
26.4
46.3
12.8
LinC+HBB,
58.9
6.5
46.3
7.0
62.2
5.7
Uncalibrased
43.7
12.1
36.3
30.9
58.7
14.2
ConC
41.7
13.6
50.7
14.2
82.7
6.9
9.1
GPT-J 6B
LinC+HBB,
59.9
3.9
50.1
7.3
84.4
5.1
Uncalibrated
15.7
ConC
40.3
14.4
54.3
8.8
94.0
6.9
LinC
$7.9
9.7
5.7
LinC+IIB,.
$7.9
5.2
53.6
7.1
94_3
4,8
Uncalibrared
36.0
26.0
48.3
9.7
92.3
9.2
ConC
46.7
15.5
43.7
11.7
92.0
6.8
LinC
LinC+HB
60.7
6.6
51.7
7.6
93.9
2.6
Uncalibrased
48.7
21.4
34,0
17.6
54.3
19.7
ConC
71.7
18.7
33.3
17.2
75.3
17.2
LinC
73.3
16.2
LinC+IIB,
73.3
9.3
47.6
6.7
84.4
4.1
Uncalibrated
63.0
8.6
41.3
4
11.4
CooC
76.0
5,9
41.0
12.6
92.3
5.2
LinC
79.7
6.3
48.7
4,4
Llama-2 13B
LinC+HB,
79.7
6.0
48.7
9.6
93.1
3.0
Uncalibrased
60.0
12.0
50.7
94.0
9.9
ConC
71.3
6.9
51.3
18.6
95.3
3.8
75.6
LinC+HB...
75.6
4.0
52.9
7.6
95.3
2.2
Uncalibrated
70.0
5.2
55.0
7.0
94.7
5,9
CooC
73.7
12.6
44.0
22.8
94.3
3.9
LinC
73.5
9.4
3.9
LinC+IIB,
73.5
7.0
50.2
4.2
95.3
2.1
Large Language Models (LLMs) exhibit an in-context learning (ICL) capability. meaning they can learn from just few examples in the context. works by constructing a prompt that includes input-output pairs demonstrating the considered task, followed by a query for new input. See [8] for survey. Recent works develop calibration methods whose main goal is to improve the performance of ICL for LLMs, without requiring complicated model fine-tuning. [77] uses a customized variant of Platt scaling (more specifically, Vector Scaling). Their method infers good values of the vector scaling parameters in data-free procedure. The idea is that for a "content-free" input, e.g., "N/A", the calibrated probability has a 50% chance (for binary classification task) of removing bias toward the positive or negative class. In our paper, we denote this method as ConC. [[] builds on top of this work but uses a calibration set learn the scaling parameters by minimizing the cross-entropy loss. This can be considered Matrix Scaling. We denote this method as LinC. [78 proposes per-class normalization
the probabilities on a given batch. (25) estimates the in-context model label marginal p(y) from limited data and uses it to calibrate the model probabilities. Paper 20 uses a Gaussian mixture model.
In our experiments, we have tested a two-step calibration. First, we use the state-of-the-art method LinC maximize the accuracy by learning scaling parameters a calibration set. Then, we apply HBTvA to scale the confidences to lower the calibration error ECE, while preserving the accuracy gains. We the same calibration set for the two methods. LinC performance depends on hyperparameter values, but to keep the experiments simple, we fixed the following values: 100 epochs. a learning rate of 0.001, and 300 calibration samples. means that the reported performance of LinC suboptimal and could be enhanced even more. We used the same experimental setting as [1. We used the models GPT-J with 6B parameters (62) and 1lama-2 with 13B parameters [59]. The text classification datasets are TREC [61] for question classification with 6 classes, SST-5 (55) for sentiment analysis with classes, and DBpedia 774 for topic classification with classes. The 0-shot, I-shot, 4-shot, and 8-shot learning settings were tested. Five different sets of 300 test samples were randomly selected, and results are averaged over seeds. We evaluated the accuracy and ECE for each configuration. Please Table T2 for the results. In most cases, LinC+HB TvA achieves the best accuracy and ECE.
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims in the abstract and introduction are that our approach solves the failure of many calibration methods when applied to problems with many classes and that we conduct extensive experiments. This reflects our analyses in Sections and and experimental results in Section and Appendices Gnd
and H
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Section 6 is a separate "Limitations" section in the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
34

---

Answer: [NA]
Justification:
No strong theoretical result, but mathematical calculations are in Appendix D
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We use publicly available models and data and experiments details provided in Section and and Appendix Our provided code can be used to reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
35

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use publicly available models and data and provide clean code imple- mentation aiming for easy reuse and reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
Poolicc/uuides/Coeeeuuliiis wtw.eeeewuumiisioooooooooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccppplgguuuiiaeeeaec
Saleee//CoSSSSbisssS.S.. www.ssssiicgcyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details are provided Section 5 Appendix F and in the provided code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All our tables show values averaged over five random seeds that generate different calibration and test datasets, as mentioned in Section Standard deviations are reported in Table 7
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
36

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Table in the Appendix provides computing times and the GPU model used.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpps:////euriii hww.nnuu...../. http//rripss.cccpllll///eeteeeee cc/ppbblic/Ethcccccuuelllllll
Answer: [Yes]
Justification: The research conducted in the paper does conform with the NeurIPS Code of Ethics and we preserve anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix B discusses the broader impacts of the work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
37

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We believe that improved confidence calibration does not have a high risk of
misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The papers producing the used datasets are cited. The code implementations used are cited and the URLs are provided Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
38

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Our provided code is well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
39

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
40


Reviews:
[{"review_id": "wy8zM4nnYg", "reviewer": "Reviewer_S8uZ", "summary": "This paper proposes a new learning objective for model calibration in multi-class classification tasks. The authors analyze the issues with the current softmax-based scaling method and argue that it only considers the confidence (the probability of the predicted class) without accounting for other remaining classes during calibration. The proposed Top-versus-All method divides the calibration data into class-wise datasets. Experiments on various models, tasks, and calibration baseline methods have shown some effectiveness of the proposed approach.", "questions": "1. The results in Table 3 are not aligned with the paper's narrative. Why not discuss the mechanism or reason behind this? I think it is a vital issue.\n\n2. Can the authors please provide strong reasons to convince me that your method is superior to similar works [2] and [3]?", "limitations": "Yes. They have a limitation section.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. The experiment is conducted on many models and test sets.\n\n2. The motivation is clear and convincing.\n\n3. The writing is clear, and easy to understand.", "weaknesses": "1. This paper calibrates a surrogate binary classifier that predicts whether the class prediction is correct. However, according to the results in Table 3, in the majority of experimental results, the AUROC is worse when using the proposed method. The AUROC is not improved, meaning that the calibrator has not improved its ability to effectively distinguish between correct and incorrect model predictions, thus failing to learn to predict whether the class prediction is correct. This is contradictory to the paper's main objective and narrative. \n\n2. The authors answered YES for Theory Assumptions and Proofs in the checklist. However, no proofs are shown. Some mathematical calculations are presented in Appendix D, but those are not proofs. They overclaim the theoretical contribution.\n\n3. The evaluation metrics are limited. Only quantitative results on ECE and AUROC are reported (even though the AUROC is not improved). Other popular metrics such as ACE, MCE, or PIECE are not provided.\n\n4. Authors report the results on CLIP models but do not discuss or compare them to recent calibration methods for CLIP models. For example:\n    \n   [1] Wang, S., Wang, J., Wang, G., Zhang, B., Zhou, K. and Wei, H., 2024. Open-Vocabulary Calibration for Vision-Language Models. International Conference on Machine Learning (ICML), 2024.\n\n\n5. There are several existing model calibration papers in the literature that focus on directly estimating the probability of correctness for each sample/prediction.\n\n     [2] Xiong, M., Deng, A., Koh, P.W.W., Wu, J., Li, S., Xu, J. and Hooi, B., 2023. Proximity-informed calibration for deep neural networks. Advances in Neural Information Processing Systems, 36, pp.68511-68538.\n\n     [3] Liu, Y., Wang, L., Zou, Y., Zou, J. and Zheng, L., 2024. Optimizing Calibration by Gaining Aware of Prediction Correctness. arXiv preprint arXiv:2404.13016.\n\n    For example, in paper [3], the idea is almost the same with minor differences. It also learns to predict the probability of correctness of the model's predictions as confidence. However, it can distinguish between correct and incorrect predictions based on their AUROC results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your careful consideration of our paper. We are glad that you appreciated our experiments and found our motivation and writing clear. We answer your questions below, and the weaknesses in an additional Comment.\n\n**Questions**\n\n> 1. The results in Table 3 are not aligned with the paper's narrative. [...]\n\nOur paper addresses confidence calibration, usually measured by ECE. AUROC is a global rank-based metric for selective classification: it relies on the relative values of the scores, not their absolute values. Even though calibration and selective classification may be related, improvement in calibration does not directly translate to better selective classification. This has been clearly demonstrated experimentally by [A].\n\nA good example of that difference is the behavior of $HB_{TvA}$: it is the best calibration method overall but actually degrades the AUROC in most cases. Such a difference can be explained by the fact that selective classification benefits from a continuous score able to discriminate between certain and uncertain examples finely, but HB quantizes the confidences into, e.g., 10 different values.\n\nWe included experiments for selective classification by computing the AUROC to see if it could benefit from improvements in calibration. Our experiments confirmed the findings of [A].\n\n[A] *Ido Galil, Mohammed Dabbah, and Ran El-Yaniv. What Can we Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers? ICLR 2023*\n\n> 2. Can the authors please provide strong reasons to convince me that your method is superior to similar works [2] and [3]?\n\nWe are glad that you open the discussion, and we hope the following arguments can convince you.\n\nThe primary objective in [2] differs from ours: it \"focuses on the problem of proximity bias in model calibration, a phenomenon wherein deep models tend to be more overconfident on data of low proximity,\". The goal in [2] is to lower the difference in the confidence score values between regions of low and high density, i.e., to make the confidence score independent of a local density indicator called \"proximity.\" There is no theoretical guarantee, however, that minimizing the proximity bias improves the confidence calibration, the focus of our work. Theorem 4.2 about the PIECE metric is a direct consequence of Jensen's inequality and is true for any random variable $D$, not necessarily a proximity score. Theorem 5.1 is an interesting bias/variance decomposition of the Brier score. However, as this type of decomposition usually states, the error may come from bias (here, a wrong initial calibration) or high estimation variance (which can be related to low density but is not expressed as such in the decomposition). We experimentally compare our approach to the ProCal algorithm using the code provided by [2] and observe that our approach gives much better ECE confidence calibration and, for half of the models, also better PIECE values (see attached PDF file).  \nAs you mentioned, both works however share similarities. [2] proposes a calibration method to achieve three goals: mitigate proximity bias, improve confidence calibration, and provide a plug-and-play method. We share the last two goals. Concerning improving confidence calibration, our approach has better results, as shown in the attached PDF. Both approaches are plug-and-play, but they apply very differently. The method by [2] is applied *after* existing calibration methods to further improve calibration. It thus does not solve any of the four issues we identified (e.g., cross-entropy loss is still inefficient, and One-versus-All still leads to highly imbalanced problems). Our Top-versus-All approach is a reformulation of the calibration problem that uses a surrogate binary classifier. Existing approaches are applied to this surrogate classifier, which is how the four issues are solved. We do not propose a new method but a new way of applying existing methods. Our approach does not introduce new hyperparameters (except in the particular case of regularizing scaling methods). [2] introduces several new hyperparameters, such as the choice of the distance, the number of KNN neighbors, or a shrinkage coefficient.\n\nConcerning paper [3], according to the NeurIPS 2024 FAQ for Authors, \"Authors are not expected to compare to work that appeared only a month or two before the deadline.\", which is the case for this work. However, we can still discuss it. In our understanding, the main intuition is similar indeed: binarize the calibration problem. However, what they do with this intuition differs vastly from our approach. [3] derives a loss (Eq. 7), which is almost the standard binary cross-entropy loss we use for scaling methods but without a logarithm. They use this loss to learn a separate model that predicts a sample-wise temperature coefficient. This is a new calibration method, which is not straightforward to implement due to the numerous hyperparameters (network architecture, image transformations...). It also requires multiple inferences at test-time, which can be problematic in some production models. Our approach is, again, not a calibration method but a general reformulation of the calibration problem that enhances existing methods. By looking at their Table 1, they get an ECE of 2.22 on ImageNet (in-distribution), while our approach achieves values around 0.5 for most models in our paper's Table 1. In our understanding of their results, the AUROC improvements seem mostly due to the use of image transformations, not from their proposed loss. Their method seems to work best in out-of-distribution scenarios, which is not the main objective of our paper.\n\nWe will include the discussions on these relevant papers and the AUROC results in the final version of the paper.\n\nWe hope that our rebuttal has addressed your concerns to your satisfaction. We would be grateful if you could consider revising your score in light of our response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "59vk1M3lFF", "reviewer": "Reviewer_RhL1", "summary": "The paper proposes a method to improve confidence calibration in neural network-based classification models. It transforms the multiclass calibration problem into a binary classification surrogate, demonstrating enhanced performance across various neural network applications in image and text classification.", "questions": "1. how does the method apply to CLIP? On the contrastive softmax loss? What if the number of classes is varying during training CLIP? I.e., the batch size is varying.", "limitations": "Limitations are well discussed in the paper", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper's discussion on the standard approach to confidence calibration provides a comprehensive overview of existing methods and their limitations, highlighting the need for more effective strategies.\n   \n2. The Top-versus-All approach presented in the paper offers a straightforward yet powerful solution to the multiclass calibration challenge. By transforming the problem into a binary classification task, it simplifies the calibration process while maintaining effectiveness in improving prediction confidence.\n\n3. The scalability and generality of the proposed method to LLMs are particularly noteworthy. By demonstrating its applicability across diverse neural network architectures used in image and text classification tasks, the paper underscores its potential impact on enhancing calibration methods for a wide range of applications.", "weaknesses": "1. The paper lacks rigorous theoretical evidence to substantiate its claims, which weakens the strength of the proposed solutions. Without detailed proofs, the effectiveness of the Top-versus-All approach and other methods in improving confidence calibration remains somewhat speculative.\n\n2. While the paper claims scalability to LLMs, the experimental evaluation primarily focuses on smaller-scale models. This limited scope raises questions about the method's performance and applicability to state-of-the-art LLMs like LLaMA or Gemma. Extending experiments to these latest models would provide more convincing evidence of scalability and effectiveness across a broader range of neural architectures.\n\n3. Figure 3 in the paper indicates a notable variance in the ECE test results. This variability suggests inconsistent performance across different settings or datasets, potentially indicating limitations in the method's stability and reliability under varying conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper. We appreciate that you value our discussion on the limitations of existing approaches, the straightforwardness of our approach, and its scalability and generality. We have addressed each of your concerns below.\n\n**Weaknesses**\n\n> 1. The paper lacks rigorous theoretical evidence to substantiate its claims, which weakens the strength of the proposed solutions. Without detailed proofs, the effectiveness of the Top-versus-All approach and other methods in improving confidence calibration remains somewhat speculative.\n\nPlease see our global answer which addresses this weakness.\n\n> 2. While the paper claims scalability to LLMs, the experimental evaluation primarily focuses on smaller-scale models. This limited scope raises questions about the method's performance and applicability to state-of-the-art LLMs like LLaMA or Gemma. Extending experiments to these latest models would provide more convincing evidence of scalability and effectiveness across a broader range of neural architectures.\n\nThis might be a misunderstanding. Experiments in Table 1 show results with T5 and RoBERTa, which we call Pre-trained Language Models, and which we believe you call smaller-scale models. What we call LLMs are GPT-J 6B and LLaMA-2 13B (as you suggested), whose results are included in the Appendix, Table 11. This is mentioned in the Experiments section, lines 318-322. Does that resolve the issue?\n\n> 3. Figure 3 in the paper indicates a notable variance in the ECE test results. This variability suggests inconsistent performance across different settings or datasets, potentially indicating limitations in the method's stability and reliability under varying conditions.\n\nThere is indeed some variance for scaling methods (figure at the bottom). We believe this is because these methods involve a learning process, which might not be perfect each time as we did not optimize each run due to the scale of our experiments. However, this variability is not increased when our TvA approach is applied: it comes from the underlying methods.\nAlso, please note that for binary methods (figure at the top), especially when TvA is applied, the variance is very small.\nAdditionally, Table 6 of the Appendix includes the standard deviations for ECE. In most cases, compared to the underlying methods, applying our approach TvA either does not impact the variance or reduces it.\n\n**Questions**\n\n> 1. how does the method apply to CLIP? On the contrastive softmax loss? What if the number of classes is varying during training CLIP? I.e., the batch size is varying.\n\nIn this paper, we use a pre-trained CLIP model as a zero-shot classifier with the standard method, without retraining. We compute the cosine similarities between the images and the text \"a photo of a {c}\" with c taking the values of all the labels for the considered task, e.g., 1000 different class names for ImageNet. The cosine similarities are converted to \"logits\" by multiplying them by a coefficient. \nFor more details, you can check the provided code full_code/utils.py, lines 137-171, where the CLIPClassifier class is defined. We hope we answered your question and will clarify this point in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vu2dmPqRnr", "reviewer": "Reviewer_eq8Z", "summary": "The paper relates to post-hoc confidence calibration i.e. the calibration of\nthe top-prediction of a classifier such that ``when the class l is predicted\nwith confidence q, the probability of the actual class being l is also q''.\nThe calibration does not update the classifier.\n\nThe paper tackles two issues of past calibration methods:\n- The one vs all approaches that that do not scale well with the number of\n  classes (there are as many binary calibrators as number of classes)\n- Each binary calibrator is tuned on datasets which imbalance increases with\n  the number of classes: the positives are the instances of class $k$ and the\n  negatives are all the other instances\n- The overfitting of scaling methods due to their parametric nature (the number\n  of parameter increases with the number of classes)\n\nTo do so, it proposes an alternative formulation of the calibration problem:\ninstead of calibrating the top-prediction directly, the paper calibrates an\nintermediate binary classifier that estimates whether the top-prediction is\ncorrect or not.\n\nThis addresses the two previous issues as:\n- There is only one calibration optimization that calibrates the single binary\n  classifier described above, which is independent of the number of classes\n- The imbalance remains but is reduced and linked to the classifier's accuracy\n- The calibration of the binary classifier with scaling methods requires less\n  parameters and is hence less prone to overfitting\n\nThe proposed calibration can be optimized with off-the-shelf binary calibrators\nand the experiments compare the proposed calibration against the traditional\ncalibrations, i.e., calibrating the classifier's top prediction directly. \n\nThe experiments compare the proposed formulation that reduce the ECE compared\nto the traditional calibrations.\nThe proposed formulation usually reduce the ECE compared to the traditional\ncalibration and improves over IRM and I-MAX.\nThe improvement is more notable on datasets with a higher number of classes.", "questions": "Q1. One of the claimed advantage of the proposed method (TvA) is the \"stronger\n  gradient\" of TvA compared to that of the default scaling methods (understand\n  the gradient's magnitude is higher).\n  However, it is not clear how this is an advantage on its own as i) the impact of the\n  gradient's magnitude is mitigated by the choice of learning rate; ii) a\n  higher gradient magnitude does not imply a better optimization (this is\n  specific to each optimization problem).\n  Can more details be given on why `stronger gradients' are an advantage?", "limitations": "- The advantage of 'preserving' the accuracy is limited by the fact that existing\n  calibration method that affect the accuracy usually induce a variation of ~1%\n  only, and can even affect positively the accuracy i.e. increase it.\n  This leads to the question of whether 'preserving of the accuracy' is really a\n  limitation for other methods.\n\n- Issue 1 describes the limitation of using cross entropy loss during\n  calibration as it increases the probability of the true class which only\n  **indirectly** impacts the probability of the top-prediction i.e. the\n  confidence.\n  L: 173 describes this optimization as 'inefficient' compared to the proposed\n  one, however it is hard to quantify the 'efficiency' of an optimization so a\n  better word might be 'indirect'.\n\nMisc.:\n- Eq6: v is undefined, does it refer to the logit probability vector?\n- lack of legends for the blue and red bars (which is accuracy, which is\n  confidence) even though one can understand\n- Legend for Global acc. and avg. confidence not very visible", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1. The paper is well-written and pleasant to read.\n\nS2. The proposed calibration formulation is simple yet efficient: it calibrates\n   a single intermediate binary problem that estimates whether the classifier's\n   prediction is correct. Thus, it can leverage the strengths of existing\n   methods on a simpler optimization problem.\n\nS3. By design, the proposed method preserves the classification accuracy since\n   it does not edit the network.\n\nS4. The experiments are exhaustive in terms of datasets and calibration methods:\n- the experiments are run on image and text classification on multiple datasets\n  with a wide range of number of classes: 10 on cifar-10 to 10K on ImageNet-21K.\n- scaling methods (Temperature and Vector Scaling, Dirichlet Calibration) and\n  binary methods (Isotonic regression, BBQ and Histogram Binning); and other\n  methods tackling the one-vs-all limitations (IRM and I-MAX).", "weaknesses": "W1. Not really a weakness but a consequence of the design choice: the proposed method keeps the classifier untouched so that the calibration does not impact the accuracy.\nHowever, improving the confidence of incorrect predictions could help improving the classification accuracy by re-distributing the probabilities. \nIn practice though (Tab.8), the accuracy varies only by a few points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for spending the time and effort to write this thorough review, which demonstrates a deep understanding of our work. We are grateful that you praise our writing and recognize our approach's efficiency and the exhaustivity of our experiments. The points you raised are relevant, and we will update our paper accordingly. We believe your confidence score could be higher. Please see our responses below.\n\n**Weaknesses**\n\n> W1. Not really a weakness but a consequence of the design choice: the proposed method keeps the classifier untouched so that the calibration does not impact the accuracy. However, improving the confidence of incorrect predictions could help improving the classification accuracy by re-distributing the probabilities. In practice though (Tab.8), the accuracy varies only by a few points.\n\nIndeed, TvA applied to binary methods keeps the classifier accuracy untouched which means that it cannot improve accuracy. However, we consider that an advantage because in practice binary methods otherwise usually degrade accuracy (Table 8).\n\n\n**Questions**\n\n> Q1. One of the claimed advantage of the proposed method (TvA) is the \"stronger gradient\" of TvA compared to that of the default scaling methods (understand the gradient's magnitude is higher). However, it is not clear how this is an advantage on its own as i) the impact of the gradient's magnitude is mitigated by the choice of learning rate; ii) a higher gradient magnitude does not imply a better optimization (this is specific to each optimization problem). Can more details be given on why `stronger gradients' are an advantage?\n\nThe reasoning about \"stronger gradient\" was inspired by two articles: [1] and [2] (see page 5 for both).\n\nMore specifically in our case, when predictions are incorrect, the gradient with TvA is proportional to $\\frac{s}{s-1}$ with $s$ the confidence. This term means that as the confidence for wrong predictions gets higher, so does the gradient to reduce the confidence. This effect is not mitigated by the choice of learning rate (which does not vary with $s$) and we believe it allows a better optimization. We will clarify this point in the final version.\n\n[1] *Zhao, Z., Liu, Z., & Larson, M. (2021). On Success and Simplicity: A Second Look at Transferable Targeted Attacks. Advances in Neural Information Processing Systems (Vol. 34, pp. 6115-6128)*\n\n[2] *Naseer, M. M., Khan, S. H., Khan, M. H., Shahbaz Khan, F., & Porikli, F. (2019). Cross-Domain Transferability of Adversarial Perturbations. Advances in Neural Information Processing Systems (Vol. 32)*\n\n**Limitations**\n\n> The advantage of 'preserving' the accuracy is limited by the fact that existing calibration method that affect the accuracy usually induce a variation of ~1% only, and can even affect positively the accuracy i.e. increase it. This leads to the question of whether 'preserving of the accuracy' is really a limitation for other methods.\n\nGiven the effort required to gain 1% accuracy on ImageNet, losing it during calibration is not negligible in our opinion. In principle, we believe that decoupling accuracy optimization (during model training) and calibration (during post-hoc calibration) is a more efficient approach than train time calibration (optimizing both accuracy and calibration during model training) or having calibration methods impacting the accuracy. Indeed, it avoids to manage compromises between two objectives during development. Preserving accuracy is a characteristic of our approach, which we consider an advantage in most cases, but which other people might view differently.\n\n> Issue 1 describes the limitation of using cross entropy loss during calibration as it increases the probability of the true class which only indirectly impacts the probability of the top-prediction i.e. the confidence. L: 173 describes this optimization as 'inefficient' compared to the proposed one, however it is hard to quantify the 'efficiency' of an optimization so a better word might be 'indirect'.\n\nCorrected, thank you.\n\n> Misc.:\n\n$v$ is defined line 238, this is the parameters vector that scales the logits values. While Temperature Scaling uses a single coefficient to scale the logits, Vector Scaling scales all class logits independantly (1 coefficient per class).\n\nWe added the signification of the blue and red bars and improved the legend for global accuracy and average confidence.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3IK7XjURxl", "reviewer": "Reviewer_yKjq", "summary": "This paper presents a top-versus-all (TvA) approach for the confidence calibration of classifiers with many classes. The authors categorizes the post-processing calibration methods into two groups: scaling and binary methods, and lists several problems with these approaches. The authors reformulate the problem of calibrating a multiclass classifier into a single binary classifier, where the confidence score represents the maximum class probability of the binary problem. They show the effectiveness of the proposed method on numerous neural networks used for image or text classification.", "questions": "Please see my comments in Weaknesses.", "limitations": "The authors have addressed the limitations in the main text.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) This paper nicely presents the shortcomings of the existing methods and describes how the proposed method addresses these challenges, the proposed method is innovative and provides a simple and efficient solution to calibrating classifiers with many classes.\n2) This paper presents extensive experimental results on multiple datasets and network architectures.\n3) This paper provides a well-structured review of relevant literature and clearly explains the motivation and significance of the proposed approach.", "weaknesses": "1) The theoretical contribution of the TvA approach and its impact on calibration could be further strengthened.\n2) The propose method may exhibit limited novelty, as it builds upon established approaches rather than introducing entirely novel elements.\n2) The calibration of CLIP appears to be better than other models on ImageNet, can you explain why? Additionally, it seems like most calibration methods don't work on CLIP, which confuses me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback. We value that you praise our approach, its motivation, and significance and acknowledge the extensiveness of our experiments. We hope our response can address your concerns.\n\n**Weaknesses**\n\n> 1. The theoretical contribution of the TvA approach and its impact on calibration could be further strengthened.\n\nPlease see our global answer which addresses this weakness.\n\n> 2. The propose method may exhibit limited novelty, as it builds upon established approaches rather than introducing entirely novel elements.\n\nThe novelty of our approach is to calibrate a surrogate binary classifier. This is a new reformulation of the confidence calibration problem. It does not build upon established approaches but rather applies on top of existing methods such as HB.\n\n> 3. The calibration of CLIP appears to be better than other models on ImageNet, can you explain why? Additionally, it seems like most calibration methods don't work on CLIP, which confuses me.\n\nIndeed, CLIP has a different behavior from other models, which is consistent with the results of (Galil et al., 2023). There could be several reasons to explain the good \"out of the box\" calibration of CLIP and its particular behavior. The multimodal training regime and zero-shot adaptation as a classifier are quite different from standard image classifiers. CLIP's \"logits\" are based on the cosine similarity of images with textual prompts representing the classes: they are not like standard image classifier logits trained with a fixed number of classes. The pre-training dataset is also different from ImageNet (much larger and containing images with a textual legend, not the class label).\n\nThe fact that many calibration methods do not improve CLIP's calibration is probably also due to the standard methods having difficulties improving already well-calibrated models. This also happens with ViT-H/14.\n\nWe will clarify this in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ss6Xocp5YI", "reviewer": "Reviewer_oe96", "summary": "The paper addresses the issue of miscalibrated confidence scores in neural network classifiers, particularly for problems with many classes. Traditional methods often fail in these scenarios. The authors propose transforming the multiclass calibration problem into a single binary classification problem, termed Top-versus-All (TvA), allowing the use of standard calibration methods more efficiently. This approach improves the performance of calibration methods like Temperature Scaling and Vector Scaling by better utilizing the calibration data and reducing overfitting. Experiments on various datasets and models demonstrate the scalability and effectiveness of this method.", "questions": "see Weaknesses", "limitations": "see Weaknesses", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "Simple Method: The Top-versus-All (TvA) approach is straightforward and easy to implement, requiring minimal changes to existing calibration methods.\n\nGood Performance: The TvA method significantly enhances the performance of standard calibration techniques, consistently improving Expected Calibration Error (ECE) across various datasets and models.\n\nScalability: The method scales well to handle classifiers with a large number of classes, addressing a common limitation in traditional calibration methods.", "weaknesses": "1. Lack of novelty. I read a paper very similar to the idea previously, make top vs all, but I am sorry that I cannot find the link. And as I remember, the Multiclass to Binary idea has been used in this literature multiple times.\n\n2. Lack of the comparison with some SOTA post hoc methods such as [1] , the performance may not succeed.\n\n3. Lack of comparison with some SOTA train time methods, such as [2].\n\n4. Overall, the method lack the analysis and comparison with other SOTA methods.\n\n[1] Proximity-Informed Calibration for Deep Neural Networks\n[2] Dual focal loss for calibration", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We appreciate that you find that our method is simple, scalable, and has good performance. We hope our response can address your concerns.\n\n**Weaknesses**\n\n> 1. Lack of novelty. I read a paper very similar to the idea previously, make top vs all, but I am sorry that I cannot find the link. And as I remember, the Multiclass to Binary idea has been used in this literature multiple times.\n\nWe are sorry we were not able to find such a reference. It would be helpful if you could provide it to us. Also, the Related Work includes a paragraph on Multiclass to Binary, and Appendix C includes a comparison with competing approaches. We thus already discussed in the paper how our approach differs from other work.\n\n> 2. Lack of the comparison with some SOTA post hoc methods such as [1] , the performance may not succeed.\n\nIndeed, we have missed this nice reference; thank you for pointing it out. We have conducted preliminary experiments following the experimental setting of paper [1]'s Table 4. Results are available in the PDF attached to the global response. They show that our approach performs better. We plan to add more comparisons with their method in the final version of our paper.\n\n\n> 3. Lack of comparison with some SOTA train time methods, such as [2].\n\nAs we have stated in a paragraph starting line 90, we focus on calibrating already-trained models. Train time methods require a high development time, often compromise accuracy, and are not adapted to pre-trained models. Because of the computation time required by train-time methods, paper [2]'s experiments are much more limited than ours. Their biggest dataset is Tiny ImageNet, a subset of downscaled images from ImageNet. In our paper, we not only consider ImageNet but also the even more complex ImageNet-21K dataset. We have compared 26 models for ImageNet, each one calibrated with 5 different seeds, while [2] only studies a single model for ImageNet. A fair comparison with [2] would require huge computing resources. To us, fast development is one of the main strengths of post-hoc calibration methods, the main scope of the paper.\n\nFurthermore, many train time methods actually benefit from post-hoc methods, as can be seen in Table 1 of [2]. This means that our work could also be applied on top of train time methods to further improve calibration.\n\nWe will add the reference to [2] in the \"Training calibrated networks\" paragraph of the Related work for the final version.\n\n\n> 4. Overall, the method lack the analysis and comparison with other SOTA methods.\n\nWe already have included the SOTA methods IRM and I-Max, and besides the preliminary results in the attached PDF, we plan to add more comparisons with [1] in the final version, as you suggested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G8gzMWaQep", "reviewer": "Reviewer_QaKm", "summary": "The paper proposes a confidence calibration method.  It reformulates a multi-class problem as the binary task \"is the prediction correct?\".  Then a given calibration method  (e.g. Temperature Scaling) is applied to that binary task.", "questions": "ggg", "limitations": "ggg", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed method is very simple and can be efficiently applied to tasks with a large number of classes,", "weaknesses": "The main justification of a calibration model is its improved results (usually measured by ECE).   \nIn the case of TS the paper reports a minor improvement. However,  in the case of HB, the paper reports a huge calibration improvement.  It is not clear what is the reason for the improvement difference.\n\nAs far as I understand the HB method, we first split the validation samples  to bins according to the confidence.    \n\n\nAlthough the proposed idea is very very simple,  the  method presentation is overcomplicated and there is  an over-seling of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We are glad that you find our approach simple and efficient. Please find our rebuttal below.\n\n**Weaknesses**\n\n> The main justification of a calibration model is its improved results (usually measured by ECE).\nIn the case of TS the paper reports a minor improvement. However, in the case of HB, the paper reports a huge calibration improvement. It is not clear what is the reason for the improvement difference.\n\nThe reason that improvements for HB are more significant than for TS is simply that standard HB (and other binary methods) does not perform well when the number of classes is high. This is what we describe as \"Issue 3\" in the paper. For experimental results, you can see in Table 1 that improvements for HB are moderate for C10 (10 classes), higher for C100 (100 classes), even higher for IN (1k classes), and even higher for IN21K (10k classes). Concerning TS, TvA solves \"Issue 1\" (the inefficientness of the loss) which is not as serious as \"Issue 3\".\nWe do not see how this different behavior is a weakness, but we will include the explanation in the final version.\n\n\n> As far as I understand the HB method, we first split the validation samples to bins according to the confidence.\n\nThis is correct but we do not really understand how this describes a weakness. Could you please clarify this point?\n\n> Although the proposed idea is very very simple, the method presentation is overcomplicated and there is an over-seling of the method.\n\nWe agree that the idea is simple, this is what we consider to be one of its main strengths. Easy and quick to implement, it can improve existing calibration methods. We chose to explain the limitations of existing approaches before explaining how our approach applies to scaling and binary calibration methods. Algorithm 1, and Algorithms 2 and 3 of the Appendix provide, in our opinion, simple presentations of the approach. Also, other Reviewers praise the paper's clarity: \"writing is clear, and easy to understand\" from Reviewer S8uZ; \"well-written and pleasant to read\" from Reviewer eq8Z; \"nicely presents\" from Reviewer yKjq. As for the \"over-seling\", could you be more specific?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the Reviewers for the time spent reviewing our paper and for the constructive comments. In this global response, we group, summarize, and discuss the main identified strengths and weaknesses.

**Strengths**

The Reviewers mostly agree on three strengths:
- *The method is simple and efficient* (5 Reviewers: RhL1, eq8Z, yKjq, oe96, and QaKm).
- *The experiments are comprehensive* (5 Reviewers: S8uZ, RhL1, eq8Z, yKjq, and oe96).
- *The writing is clear* (4 Reviewers: S8uZ, RhL1, eq8Z, and yKjq).

These are indeed what we consider to be some of the main strengths of our paper, and we are glad that most Reviewers successfully identified them.
We also would like to emphasize the reproducibility and practical applicability of our work. Our approach is "plug-and-play" and easily integrates with most existing calibration methods with minimal code modification. We have made our complete codebase available (currently hosted on Anonymous GitHub), allowing practitioners to utilize our method and replicate our results.

**Weaknesses**

Two weaknesses are shared by several Reviewers:
- *The theoretical contribution could be further strengthened* (3 Reviewers: S8uZ, RhL1, and yKjq).
- *Some references are missing* (2 Reviewers: S8uZ, and oe96).

Concerning theoretical contribution, the primary focus of our paper is on experimental research, with an emphasis on practicality. To compensate for the lack of theoretical evidence, we conducted experiments on complex real-world problems, such as natural image and text classification. Given the extensiveness of our experiments, which five Reviewers highlighted, we consider the effectiveness of Top-versus-All to be clearly demonstrated. We also aimed to clarify our intuition by highlighting four shortcomings of standard approaches and discussing how our method addresses them. Moreover, our paper is not devoid of any theory: we developed a theoretical justification for the specific case of Temperature Scaling on lines 227-235 that derives from calculations included in Appendix D. When predictions are incorrect, the gradient with TvA is proportional to $\frac{s}{s-1}$ with $s$ being the confidence. This term means that as the confidence for wrong predictions gets higher, so does the gradient to reduce the confidence. Despite the limited scope of our theoretical justification, we believe it is a valuable outcome that partly explains our good experimental results.

Concerning the missing references, we thank the Reviewers for providing them to us. In particular, two Reviewers cited [1]. We thus conducted preliminary experiments to compare it to our approach. Results are available in the attached PDF. We used [1]'s public code to reproduce the experimental setting closest to ours, corresponding to the results of Table 4 in [1]. The comparison shows that our approach is always better for confidence calibration (the main focus of our work) and competitive for reducing the proximity bias (the main focus of their work), with our approach being the best for 2 of the 4 models studied. In addition to these better experimental results, we argue how our approach is superior to [1] in our response to Reviewer S8uZ.

Other weaknesses were mentioned by only one Reviewer each. To us, this means that, while valid, they are not major concerns. We address all weaknesses and questions in the rebuttals addressed to each review below.


[1] Xiong, M., Deng, A., Koh, P.W.W., Wu, J., Li, S., Xu, J. and Hooi, B., 2023. Proximity-informed calibration for deep neural networks. Advances in Neural Information Processing Systems, 36, pp.68511-68538.

Meta-review: The paper lacks strong theoretical contributions and there is some concern about novelty.  However, its simplicity, practicality, and  experimental validation could make it a valuable contribution to the field of confidence calibration, particularly when dealing with a large number of classes.  The paper had many reviews (with some of them conflicting) and I have to discard some of the negative reviews on novelty due to the fact that there was no concrete justification by the reviewers.  I find that the paper is borderline accept as a poster.
Decision: Accept (poster)