Title: Confidence Calibration of Classifiers with Many Classes
Venue: NeurIPS 2024
Accepted: yes
Keywords: Calibration

Classification

Uncertainty quantification
Abstract: For classification models based on neural networks, the maximum predicted class probability is often used as a confidence score. This score rarely predicts well the probability of making a correct prediction and requires a post-processing calibration step. However, many confidence calibration methods fail for problems with many classes. To address this issue, we transform the problem of calibrating a multiclass classifier into calibrating a single surrogate binary classifier.  This approach allows for more efficient use of standard calibration methods. We evaluate our approach on numerous neural networks used for image or text classification and show that it significantly enhances existing calibration methods.

Confidence Calibration of Classifiers with Many Classes

Abstract

For classification models based on neural networks, the maximum predicted class probability is often used as a confidence score. This score rarely predicts well the probability of making a correct prediction and requires a post-processing calibration step. However, many confidence calibration methods fail for problems with many classes. To address this issue, we transform the problem of calibrating a multiclass classifier into calibrating a single surrogate binary classifier. This approach allows for more efficient use of standard calibration methods. We evaluate our approach on numerous neural networks used for image or text classification and show that it significantly enhances existing calibration methods. Our code can be accessed at the following link: https://github.com/allglc/tva-calibration .

1 Introduction

The considerable performance increase of modern deep neural networks (DNNs) and their potential deployment in real-world applications has made reliably estimating the probability of wrong decisions a key concern. When such components are expected to be embedded in safety-critical systems (e.g., medical or transportation), estimating this probability is crucial to mitigate catastrophic behavior. One way to address this question is to treat it as an uncertainty quantification problem [2, 12], where the uncertainty value computed for each prediction is considered as a confidence. This confidence can be used to reject uncertain decisions proposed by the DNN [13], for out-of-distribution detection [22], or to control active learning [34] or reinforcement learning based systems [76]. When confidence values reliably reflect the true probability of correct decisions, i.e., their accuracy, a predictive system is said to be calibrated . In this case, confidence values can be used as a reliable control for decision-making.

We are interested in producing an uncertainty indicator for decision problems where the input is high dimensional and the decision space large, typically classifiers with tens to thousands of classes. For this kind of problem, DNNs are common predictors, and their outputs can be used to provide an uncertainty value at no cost, i.e., without necessitating heavy estimation such as Bayesian sampling [15] or ensemble methods [33]. Indeed, most neural architectures for classification instantiate their decision as a softmax layer, where the maximum value can be interpreted as the maximum of the posterior probability and, therefore, as a confidence. Unfortunately, uncertainty values computed in this way are often miscalibrated. DNNs have been shown to be over-confident [17], meaning their confidence is higher than their accuracy: predictions with 90% confidence might be correct only 80% of the time. A later study [44] suggests that model architecture impacts calibration more than model size, pre-training, and accuracy. For ImageNet classifiers, the accuracy and the number of model parameters are not correlated to calibration, but model families are [11].

performance. A standard baseline for post-processing calibration is Temperature Scaling [17], where the penultimate logit layer is scaled by a coefficient optimized on the calibration set.

Many post-processing calibration methods have been developed for binary classification models [50, 69, 70]. Applying these methods to multiclass classifiers requires some adaptation. One standard approach reformulates the multiclass setting into many One-versus-All binary problems (one per class) [70]. One limitation of this approach is that it does not scale well. When the number of classes is large, the calibration data is divided into highly unbalanced subsets that do not contain enough positive examples to solve the One-versus-All binary problems. Other methods based on Platt scaling [50] involve learning a set of parameters whose size grows with the number of classes. For problems with many classes, they tend to overfit, as we demonstrate in this work.

The main idea of our work is to reformulate the multiclass confidence estimation into a single binary problem. This problem can be phrased as the unique question: "Is the prediction correct?". In this formulation, the confidence score is defined as the maximum class probability of the binary problem that outputs 1 if the predicted class is correct and 0 otherwise. The intent is that the confidence score accurately describes whether the prediction is correct, regardless of the class. We show that this novel approach, which we call Top-versus-All (TvA), significantly improves the performance of standard calibration methods: Temperature and Vector Scaling [17], Dirichlet Calibration [29], Histogram Binning [69], Isotonic Regression [70], Beta Calibration [28], and Bayesian Binning into Quantiles [46]. We also introduce a simple regularization for Vector Scaling or Dirichlet Calibration that mitigates overfitting when the number of classes is high relative to the calibration data size. We conduct experiments on multiple image and text classification datasets and many pre-trained models.

Our main contributions are the following:

We discuss four issues of the standard approach to confidence calibration.

To solve these issues, we develop the Top-versus-All approach to confidence calibration of multiclass classifiers, transforming the problem into a single binary classifier's calibration. This straightforward reformulation enables more efficient use of existing calibration methods, achieved with minimal modifications to the methods' original algorithms.

Applied to scaling methods for calibration (such as Temperature Scaling), TvA allows the use of the binary cross-entropy loss, which is more efficient in decreasing the confidence of wrong predictions and leads to stronger gradients in the case of Temperature Scaling. Applied to binary methods for calibration (such as Histogram Binning), TvA significantly improves their performance and makes them accuracy-preserving.

We demonstrate our approach's scalability and generality with extensive experiments on image classification with state-of-the-art models for complex datasets and on text classification with Pre-trained Language Models (PLMs) and Large Language Models (LLMs).

2 Related work

Calibration There are various notions of multiclass calibration. One can consider confidence [17], class-wise [28], topr [19], top-label [18], decision [75], projection smooth [16], or strong [60, 65] calibration. For recent surveys, we refer to [10] and [63]. In this work, we focus on confidence calibration and not on the calibration of the full probability vector. Indeed, confidence calibration is useful for many applications that only require a single confidence value: selective classification [13], out-of-distribution detection [22], or active learning [34]. For these applications, stronger notions of calibration are both difficult and useless. Also, class-wise calibration metrics do not appropriately scale to large numbers of classes, a setting we consider in this work, as explained in Appendix E.

Metrics Several metrics have been proposed to quantify calibration error. The most common is the Expected Calibration Error (ECE) [46] (see Equation 2). ECE has flaws: the estimation quality is influenced by the binning scheme, and it is not a proper scoring rule [14, 60, 48]. Despite its flaws, it remains the standard comparison metric for confidence calibration. Variants of ECE have also been developed: classwise-ECE [29], ECE with equal mass bins [48, 44], or top-label-ECE, which adds a conditioning on the predicted class [18]. The Brier score [4] is also used to measure calibration. The proximity-informed expected calibration error (PIECE) evaluates the miscalibration due to proximity bias [68]. We mainly use the standard ECE in this work, and the Appendix contains more metrics.

Training calibrated networks Several solutions have been proposed in the literature to improve calibration by training neural networks in specific ways, generally by making use of a new loss term [31, 58, 26, 6, 5]. While these methods directly optimize calibration during the training phase of the networks, they require a high development time, often compromise accuracy, and are not adapted to pre-trained foundation models. That is why we prefer to focus on calibrating already-trained models.

Post-processing (or post-hoc) calibration methods Another approach is to calibrate alreadytrained models. This lowers the development time by decoupling accuracy optimization and calibration. In this paper, we divide post-hoc calibration methods into two categories: scaling and binary. Scaling methods are derived from Platt scaling [50] and optimize some parameters to scale the logits. Temperature Scaling [17] is a popular simple post-processing calibration method. The logits vector is scaled by a coefficient, which modifies the probability vector. Vector Scaling [17] is more expressive and has good performance in many cases [17, 48, 29]. Matrix Scaling can also be considered for more expressiveness but is difficult to apply without overfitting [17]. Dirichlet Calibration [29] proposes a regularization strategy for Matrix Scaling. [72] developed Ensemble Temperature Scaling. Scaling can be combined with binning [30]. Besides logits or probabilities, features can also be used [35]. Another family of methods tackles binary classification. We designate them as binary methods. Histogram Binning [69] divides the prediction into B bins according to the predicted probability. For each bin, a calibrated probability is computed from the calibration data. The probability becomes discrete: it can only take B values. With some modifications, it outperforms scaling methods [18, 49]. Isotonic Regression [70] learns a piecewise constant function to remap probabilities. Bayesian Binning into Quantiles [46] brings Bayesian model averaging to Histogram Binning. Beta Calibration [28] uses a beta distribution to obtain a calibration mapping.

Our work reformulates the multiclass calibration problem and allows more efficient use of all these calibration methods, with little to no change in their algorithms.

Multiclass to Binary Using binary calibration methods for a multiclass classifier requires adapting the multiclass setting. This is usually done with a One-versus-All approach [70, 17]. The multiclass setting is decomposed into L One-versus-All independent problems: one binary problem for each class. [18] introduce the notion of top-label calibration, i.e., confidence calibration with additional conditioning on the predicted class (top-label). They describe a general multiclass-to-binary framework to develop top-label calibrators. [6] derive L ( L -1) / 2 pairwise binary problems. The approach requires training the classifier from scratch, and its performance decreases with the number of classes. Our work tackles this multiclass-to-binary research problem. Contrary to [6], the One-versus-All approach [70] and top-label calibrators [18], our approach works well for problems with many classes. The methods I-Max [49] and IRM [72] use a shared class-wise strategy to compute a single calibrator. The calibrator is applied to all class probabilities separately, so the class probabilities ranking and prediction might change. In contrast, TvA applied to binary methods rescales the confidence after the class prediction is made. I-Max and IRM consider the full class probabilities vectors, while TvA only considers confidence values. Also, they build on top of Histogram Binning and Isotonic Regression, respectively, while we apply our approach to many calibration methods. A concurrent work building on an intuition similar to ours derives a calibration method based on a Correctness-Aware Loss [38]. Appendix C discusses the differences between our approach and others.

3 Problem setting

3.1 Background

Confidence calibration of a classifier We consider the classification problem where an input x is associated with a class label y ∈ Y = { 1 , 2 , ..., L } . The neural network classifier f provides a class prediction from a final softmax layer σ that transforms intermediate logits z into probabilities. The classifier prediction is the most probable class ˆ y = arg max k ∈Y f k ( x ) with f k ( x ) referring to the probability of class k , and the confidence score defined as s = max k ∈Y f k ( x ) . Note that we use the term confidence to denote the maximum class probability. With y the real label, we consider the confidence calibration definition from [17] that says that the classifier f is calibrated if:

where the probability is over the data distribution. Equation (1) expresses that the probability of being correct when the confidence is around p is indeed p . For instance, if we consider the set of predictions

with a confidence of 90% , they should be correct 90% of the time. The conditional probability of (1) is not rigorously defined mathematically (the event { s = p } has zero probability), and interval-based empirical estimators are often used to define metrics capable of evaluating how well (1) is satisfied. This is the case of ECE, which approximates the calibration error by partitioning the confidence distribution into B bins. The absolute difference between the accuracy and confidence is computed for each subset of data in the bins. The final value is a weighted sum of the differences of each bin.

Where n b is the number of samples in bin b , N is the total number of samples, acc ( b ) is the accuracy in bin b , and conf ( b ) is the average confidence in bin b . ECE can be interpreted visually by looking at diagrams such as those of Figure 1: ECE computes the sum of the red bars (difference between bin accuracy and average confidence) weighted by the proportion of samples in the bin.

Post-processing calibration methods We are considering the scenario where a classifier has already been trained, and the objective is to enhance its calibration. Post-processing calibration methods aim to remap the classifier probabilities to better-calibrated values without modifying the classifier. They typically use a calibration set different from the training set to optimize parameters or learn a function. We note the calibration data D cal = { ( x i , y i ) } N i =1 . We focus on post-processing calibration because it enables better utilization of off-the-shelf models and separates model training (optimized for accuracy) from calibration. These advantages significantly reduce the development cost of obtaining a well-performing and well-calibrated model, contrary to optimizing calibration during training. We categorize the post-processing calibration techniques considered in this paper into two groups: scaling methods and binary methods.

3.2 Issues related to current approaches

Behavior of current scaling methods Scaling methods for calibration optimize one or more coefficients that scale the logits vector to minimize on calibration data the cross-entropy loss defined as l CE = -∑ L k =1 1 k = y · log( f k ( x )) = -log( f y ( x )) . Minimizing l CE therefore increases the probability of the true class. We can distinguish two cases to understand what happens during the optimization: whether the prediction ˆ y is correct or not. In the first case, the confidence score is s = f y ( x ) : minimizing l CE increases the confidence f y ( x ) . In the second case, the prediction is incorrect, which implies that f y ( x ) < s . Minimizing l CE increases the probability of the true class f y ( x ) but does not directly change the confidence (because s = f y ( x ) ). Instead, the confidence (which was attributed to a wrong class) is indirectly lowered through the softmax normalization.

.

̸

↪ → Issue 1: Cross-entropy loss only indirecly lowers confidence in wrong predictions.

We identified another issue of some scaling methods. By design, the number of parameters optimized by Vector Scaling and Dirichlet Calibration grows with the number of classes. When the number of classes is high, these methods overfit the calibration set as shown in Figure 2.

.

↪ → Issue 2: Vector Scaling and Dirichlet Calibration overfit calibration sets with many classes.

One-versus-All approach for binary methods The One-versus-All (OvA) calibration approach [70] allows adapting calibration methods for binary classifiers to multiclass classifiers. To do so, it decomposes the calibration of multiclass classifiers into sets of L binary calibration problems: one for each class k . For each problem, the considered probability is f k ( x ) , and the associated label 1 y = k ∈ { 0 , 1 } . When calibrating a classifier from data, each binary problem is highly imbalanced with a ratio between positive and negative examples equal to 1 L -1 if the classes are equally sampled. For instance, for ImageNet, the ratio is 1/999: out of 25000 examples, only 25 have a positive label.

.

↪ → Issue 3: OvA approach leads to highly imbalanced binary problems.

At test time, each of the L class probabilities is calibrated by a separate calibration model. The resulting probability vector can be normalized to ensure a unit norm. Because each probability is calibrated independently, their ranking can change, thus modifying the predicted class. In Table 9, we see that accuracy is often negatively impacted in practice.

.

4 Top-versus-All approach to confidence calibration

Algorithm 1 Top-versus-All approach to confidence calibration

Input :

D cal : { ( x i , y i ) } N i =1 the calibration data f : the multiclass classifier g : a calibration function

Preprocessing :

ˆ y i ← arg max k ∈Y f k ( x i )

y

b

i

1

y ˆ

=y

f

b

D

TvA

cal

i

i

k

f

k

(

x

i

, y

b

N

i

)

}

i

=1

Learn calibration function :

Learn g to calibrate the surrogate binary classifier f b on D TvA cal

Inference :

Use g to calibrate the confidences of the original multiclass classifier f

4.1 General presentation

In the calibration definition (1) and the standard ECE metrics, only the confidence, i.e., the maximal probability, reflects the likelihood of making an accurate prediction. The probabilities of other classes are not taken into account. However, the standard approach to calibration uses the entire set of probabilities, not just confidence, which introduces unnecessary complexity. We aim to simplify the process by reformulating the problem of calibrating multiclass classifiers into a single binary problem. This problem can be phrased as: "Is the prediction correct?". In this setting, we do not calibrate the predicted probabilities vector but only a scalar: the confidence. The remaining probabilities are discarded. This is equivalent to calibrating a surrogate binary classifier that predicts whether the class prediction is correct. Since this correctness classifier only considers the maximal probability versus all others, we call our approach Top-versus-All (TvA).

Replacing the standard approach by TvA is straightforward. Given the standard calibration data D cal = { ( x i , y i ) } N i =1 , we add a few data preprocessing steps. First, compute the class predictions ˆ y and their correctness: y b = 1 ˆ y = y . Second, create the surrogate binary classifier f b ( x ) = max k ∈Y f k ( x ) . Finally, build the calibration set for the surrogate binary classifier:

After this preprocessing, we choose a standard calibration function g , e.g., Temperature Scaling, to calibrate the surrogate binary classifier. The learning of the calibration function follows its original underlying algorithm but uses the modified calibration data D TvA cal . The learned calibration function is then applied to the confidences of the original multiclass classifier. Algorithm 1 describes our approach. In the Appendix, Algorithm 3 provides more details and highlights differences with the standard approach of Algorithm 2.

After this general presentation, we explain how TvA impacts the two categories of calibration methods, scaling and binary, in Subsections 4.2 and 4.3, respectively. We also justify its behavior.

4.2 Top-versus-All approach for scaling methods

Because our Top-versus-All setting reformulates the calibration of multiclass classifiers into a binary problem, the natural loss is the binary cross-entropy:

Minimizing this loss results in confidence estimates that more accurately describe the probability of being correct, regardless of the L -1 less likely class predictions. Using the binary cross-entropy as a calibration loss makes an important difference compared to the usual multiclass cross-entropy. The cross-entropy loss takes into account the probability of the correct class, while with TvA the binary cross-entropy takes into account the probability of the predicted class (i.e., the confidence).

▷ e.g., Temperature Scaling

▷ Compute class predictions ▷ Compute predictions correctness ▷ Create surrogate binary classifier ▷ Build binary calibration set

̸

As for the standard approach, only two cases are possible. When the prediction is correct, l BCE = -log( s ) = -log( f y ) = l CE . We get the same result as the cross-entropy loss: minimizing it directly increases the confidence. But when the prediction is incorrect, l BCE = -log(1 -s ) = l CE . Minimizing the loss now directly decreases the confidence. This is a key difference compared to using the multiclass cross-entropy loss.

The impact of the reformulation can be seen for Temperature Scaling, which optimizes a coefficient T that scales the logits z k . The reformulation generates stronger gradients when the prediction is incorrect:

with ∂l BCE ∂T = 1 T 2 · 1 s -1 · (max k z k -∑ k z k · f k ) and ∂l CE ∂T = 1 T 2 ( z y -∑ k z k · f k ) . See Appendix Dfor the mathematical calculations. Because for interesting problems, the confidence verifies s > 0 . 5 most of the time (as shown in Figure 1), our approach strengthens the gradients. The optimization of the temperature T is more efficient as confident incorrect predictions are more heavily penalized. This effect is not mitigated by the choice of learning rate, which does not vary with s . Applying standard Temperature Scaling usually results in overconfident probabilities, but our approach limits this overconfidence. This is verified experimentally in Table 8, which displays the average confidences for TS without and with TvA.

.

↪ → Solution for Issue 1: Use the binary cross-entropy loss resulting from TvA approach.

Regularization of scaling methods Overfitting of Vector Scaling and Dirichlet Calibration can be reduced with a simple L2 regularization that penalizes the coefficients of the vector v that are far from the reference value 1 .

This regularization allows these methods to take advantage of their additional expressiveness without being subject to overfitting. The loss for Vector Scaling becomes l BCE + λl reg ( v ) where λ is a hyperparameter. The loss for Dirichlet Calibration uses additional matrix regularization terms. λ is the only additional hyperparameter introduced by our method, and it applies only to Vector Scaling and Dirichlet Calibration.

.

↪ → Solution for Issue 2: Use L2 regularization.

4.3 Top-versus-All approach for binary methods

Our TvA approach replaces the One-versus-All approach to apply binary methods to the multiclass setting. TvA transforms the multiclass setting into a single binary problem that uses the binary calibration dataset (3). In this dataset, the proportion of positive labels equals the classifier's accuracy a . The ratio between negative and positive examples is (1 -a ) N aN = 1 a -1 . For a classifier with 80% accuracy on ImageNet and a calibration dataset of 25000 examples, there are 5000 negative and 20000 positive examples (ratio of 1/4). This is still a bit imbalanced but orders of magnitude smaller than the class-wise binary calibration datasets of the One-versus-All approach (ratio of 1/999).

. ↪ → Solution for Issue 3: By not dividing the calibration data into class-wise datasets, the TvA approach yields a much better balanced binary calibration problem.

The Top-versus-All approach operates on confidence alone, not the full class probabilities vector. This means that the class prediction is already done, and the ranking of the class probabilities does not change, unlike with One-versus-All. The classifier's prediction and accuracy are unaffected. This scheme allows decoupling accuracy improvements (during training time) and calibration (during post-processing calibration), thus avoiding compromises and reducing development time.

. ↪ → Solution for Issue 4: By operating on confidence alone, the Top-versus-All approach does not impact the classifier's prediction or accuracy for binary methods applied to the multiclass scenario.

0

0.2

0.4

0.6

0.8

1

0.0

0.2

0.4

0.6

0.8

1.0

%of Samples

Avg. Confidence

0

0.2

0.4

0.6

0.8

1

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

Perfect Calibration

(a) ResNet-50 - original

0

0.2

0.4

0.6

0.8

1

Avg. Confidence

0

0.2

0.4

0.6

0.8

1

Confidence

Perfect Calibration

(b) ResNet-50 - TS

0

0.2

0.4

0.6

0.8

1

Avg. Confidence

0

0.2

0.4

0.6

0.8

1

Confidence

Perfect Calibration

(c) ResNet-50 - TSTvA

0

0.2

0.4

0.6

0.8

1

Avg. Confidence

0

0.2

0.4

0.6

0.8

1

Confidence

Perfect Calibration

(d) ResNet-50 - HBTvA

0

0.2

0.4

0.6

0.8

1

0.0

0.2

0.4

0.6

0.8

1.0

%of Samples

Avg. Confidence

0

0.2

0.4

0.6

0.8

1

0.0

0.2

0.4

0.6

0.8

1.0

Confidence

Perfect Calibration

(e) ViT-B/16 - original

0

0.2

0.4

0.6

0.8

1

Avg. Confidence

0

0.2

0.4

0.6

0.8

1

Confidence

Perfect Calibration

(f) ViT-B/16 - VS

0

0.2

0.4

0.6

0.8

1

Avg. Confidence

0

0.2

0.4

0.6

0.8

1

Confidence

Perfect Calibration

(g) ViT-B/16 - VSreg_TvA

0

0.2

0.4

0.6

0.8

1

Avg. Confidence

0

0.2

0.4

0.6

0.8

1

Confidence

Perfect Calibration

(h) ViT-B/16 - HBTvA

5 Experiments

5.1 Setting

Datasets and models For image classification, we used the datasets CIFAR-10 (C10) and CIFAR100 (C100) [27] with 10 and 100 classes respectively, ImageNet (IN) [7] with 1000 classes, and ImageNet-21K (IN21K) [54] with 10450 classes. For text classification, we used Amazon Fine Foods (AFF) [43] and DynaSent (DF) [51] for sentiment analysis with 3 classes, MNLI [66] for natural language inference with 3 classes, and Yahoo Answers (YA) [73] for topic classification on 10 classes. Experiment results are averaged over five random seeds that randomly split the concatenation of the original validation and test sets into calibration and test sets.

We used the following models for image classification: ResNet [21], Wide-ResNet-26-10 (WRN) [71], DenseNet-121 [24], MobileNetV3 (MN3) [23], ViT [9], ConvNeXt [41], EfficientNet [56, 57], Swin [40, 39], and CLIP [52] which matches input images to text descriptions in a shared embedding space, assigning labels based on the highest similarity score. For text classification, we used the PLMs RoBERTa [37] and T5 [53].

More details about datasets, calibration sets sizes, and model weights can be seen in Appendix F.

Baselines Our Top-versus-All (TvA) reformulation and regularization (reg) can be applied to different calibration methods. We have tested the following scaling methods: Temperature Scaling (TS) and Vector Scaling (VS) [17], and Dirichlet Calibration (DC) [29] with the best-performing variant Dir-ODIR, which regularizes off-diagonal and bias coefficients. We also tested the following binary methods: Histogram Binning (HB) [69] using for each case the best-performing variant between equal-mass or equal-size bins, Isotonic Regression (Iso) [70], Beta Calibration (Beta) [28], and Bayesian Binning into Quantiles (BBQ) [46]. For comparison, we include methods with state-ofthe-art results on problems with many classes: I-Max [49] and IRM [72]. See Appendix C for more details on these methods. More details on code implementations can be seen in Appendix F.

5.2 Top-versus-All

For visual qualitative results, Figure 1 displays reliability diagrams [47]. We observe that initially, ResNet-50 is highly underconfident and ViT-B/16 a bit underconfident. Applying TS and VS solves the underconfidence and makes the models slightly overconfident. TvA improves these methods, and the average confidence gets closer to the accuracy. HBTvA makes the calibration almost perfect.

Table 1 shows the results of applying the Top-versus-All reformulation to several calibration methods. For clarity, results are averaged over families of models (models based on the same architecture) and the full results are available in Tables 5 and 6 of the Appendix. In most cases, the TvA reformulation significantly lowers the ECE by dozens of percent. Without TvA, binary methods often perturb the prediction and degrade the classifier's accuracy (see Table 9), making them inapplicable in a practical setting. TvA solves the issue as it only scales the confidence (after the prediction is made) and makes binary methods outperform scaling methods.

Improvements due to TvA are consistent across models. However, exceptions are observed for CLIP: it is the model family with the lowest ECE pre-calibration, but the highest ECE post-calibration for ImageNet. CLIP's multimodal training regime, zero-shot adaptation as a classifier, and very large training dataset might cause this different behavior. CLIP's low ECE was also observed in [44, 11]. [64] specifically tackles the calibration of fine-tuned CLIP, a setting not considered here.

We also found that DC is sensitive to hyperparameter tuning, and its performance is usually not much better than VS, which is consistent with [29]. In some cases, the optimization diverges, leading to very poor results, e.g., for CLIP on ImageNet.

Improvements due to TvA are also consistent across datasets, although they tend to increase with the number of classes. Improvements on ImageNet are usually better than on CIFAR-100, whose improvements are usually better than on CIFAR-10. This is notable with e.g., TS or HB. The magnitude of improvement is usually higher for binary methods, e.g., HB, than scaling methods, e.g., TS, especially for many classes. This indicates that Issue 3 is more serious than Issue 1.

For text datasets with only three classes (AFF, DS, and MNLI), TS does not benefit from TvA, but other methods do, despite the small number of classes. According to [5], TS is among the best calibration methods for the text classification tasks considered here, even compared to ones that retrain the model. Even so, our method HBTvA significantly outperforms it.

Some methods' current implementations could not handle the large scale of ImageNet-21K, resulting in out-of-memory errors written as "err." in the Table. For I-Max and IRM, this is because they consider the full probability vectors while TvA efficiently uses data by considering only confidence values. Indeed, TvA handles this scale without difficulty.

Additional results are included in Appendix H. Tables 5 and 6 contain the full results for ECE, with the standard deviations in Table 7. Table 8 reveals that ImageNet networks are mostly underconfident. This is aligned with [11] and goes against previous knowledge on overconfidence, which was initially believed to be linked to network size [17]. Table 9 provides the accuracies after calibration. Table 10 exhibits that ECE with equal-mass bins has similar values as standard ECE. Table 11 shows that TvA mostly lowers the Brier score, except for Iso, which has the lowest score overall.

Calibration methods can also be applied to Large Languages Models (LLMs) using In-Context Learning (ICL) to tackle text classification tasks [77, 20, 25, 78, 1]. The primary goal of these methods is to improve model accuracy. TvA was not designed for this objective, but it can still be applied on top of an existing method that improves the accuracy. TvA then lowers the calibration error while keeping the accuracy gain. Results for GPT-J [62] and Llama-2 [59] are in Table 12.

To summarize the results for practical use, our experiments show that Histogram Binning (within the TvA or I-Max setting) is the best calibration method overall, providing ECE values mostly below 1%. This is the method we advise using. However, suppose the underlying application requires a confidence with continuous values, e.g., to rank the predictions for selective classification. In that case, we advise using a method that improves the AUROC, shown in Appendix G, such as TS or Iso.

5.3 Solving overfitting with regularization and TvA

On ImageNet, VS and DC overfit the calibration set, degrading the calibration on the test set. The lower performance of VS relative to TS indicates this overfitting. As visualized in Figure 2, combining the binary cross-entropy loss used in the TvA reformulation and an additional regularization term prevents overfitting. We fixed the value λ = 0 . 01 as it works well across models. Initializing the vector coefficients to 1 T with T obtained by TS TvA helps further improve performance.

5.4 Influence of the calibration set size

The size of the calibration set influences the performance of the different methods, as seen in Figure 3. TS and TSTvA do not benefit from more data due to their low expressiveness. VS does not improve the ECE because of the overfitting problem. In contrast, VS reg_TvA benefits from more calibration data. With enough data ( ≈ 15000), it outperforms TS TvA . Binary methods using the standard One-versus-All approach have poor performance and need a large amount of data to be competitive. Using TvA, they get excellent performance with little data.

50

100

150

200

5

10

epochs

ECE test [%]

VS

VSTvA

TSTvA

VSreg

VSreg_TvA

0

2

4

6

8

ECE test [%]

BBQ

HB

BBQTvA

HBTvA

5

,

000

10

,

000

15

,

000

20

,

000

25

,

000

2

3

4

ECE test [%]

VS

TS

VSreg_TvA

TSTvA

calibration set size

6 Limitations

Our approach tackles confidence calibration and is unlikely to improve performance for stronger notions of calibration, such as class-wise calibration. However, confidence calibration is useful for many practical cases, such as selective classification [13], out-of-distribution detection [22], or active learning [34]. Also, calibration improvements are less significant for problems with few classes ( ≤ 10 ) than for problems with many classes, but our approach still provides the best results.

7 Conclusion

Reducing the miscalibration of neural networks is essential to improve trust in their predictions. This can be done after the model training with an optimization using calibration data. However, many current calibration methods do not scale well to complex datasets: binary methods under the One-versus-All setting do not have enough per-class calibration data, and scaling methods are inefficient. We demonstrate that reformulating the confidence calibration of multiclass classifiers as a single binary problem significantly improves the performance of baseline calibration techniques. The competitiveness of scaling methods is increased, and binary methods use per-class calibration data more efficiently without altering the model's accuracy. In short, our TvA reformulation enhances many existing calibration methods with little to no change in their algorithm. Extensive experiments with state-of-the-art image classification models on complex datasets and with text classification demonstrate our approach's scalability and generality.

Acknowledgments and Disclosure of Funding

We thank Tahar Nabil, Houssem Ouertatani, and Pol Labarbarie for their constructive feedback on the paper draft.

This work has been supported by the French government under the "France 2030' program, as part of the SystemX Technological Research Institute.

This work was granted access to the HPC/AI resources of IDRIS under the allocation 2024AD011013372R1 made by GENCI.

References

Momin Abbas, Yi Zhou, Parikshit Ram, Nathalie Baracaldo, Horst Samulowitz, Theodoros Salonidis, and Tianyi Chen. Enhancing in-context learning via linear probe calibration. arXiv preprint arXiv:2401.12406 , 2024.

Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion , 76:243-297, December 2021.

Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS '24) . ACM, April 2024.

Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review , 78(1):1-3, 1950.

Yangyi Chen, Lifan Yuan, Ganqu Cui, Zhiyuan Liu, and Heng Ji. A Close Look into the Calibration of Pre-trained Language Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1343-1367, Toronto, Canada, 2023. Association for Computational Linguistics.

Jiacheng Cheng and Nuno Vasconcelos. Calibrating Deep Neural Networks by Pairwise Constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1370913718, 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, June 2009.

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint arXiv:2301.00234 , 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2020.

Telmo Silva Filho, Hao Song, Miquel Perello-Nieto, Raul Santos-Rodriguez, Meelis Kull, and Peter Flach. Classifier Calibration: A survey on how to assess and improve predicted class probabilities. Machine Learning , 112(9):3211-3260, September 2023.

Ido Galil, Mohammed Dabbah, and Ran El-Yaniv. What Can we Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers? In The Eleventh International Conference on Learning Representations , February 2023.

Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, Muhammad Shahzad, Wen Yang, Richard Bamler, and Xiao Xiang Zhu. A survey of uncertainty in deep neural networks. Artificial Intelligence Review , July 2023.

Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. Advances in neural information processing systems , 30, 2017.

Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association , 102(477):359-378, 2007.

Ethan Goan and Clinton Fookes. Bayesian neural networks: An introduction and survey. Case Studies in Applied Bayesian Data Science: CIRM Jean-Morlet Chair, Fall 2018 , pages 45-87, 2020.

Parikshit Gopalan, Lunjia Hu, and Guy N Rothblum. On computationally efficient multi-class calibration. arXiv preprint arXiv:2402.07821 , 2024.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

Chirag Gupta and Aaditya Ramdas. Top-label calibration and multiclass-to-binary reductions. In International Conference on Learning Representations , January 2022.

Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. Calibration of Neural Networks using Splines. In International Conference on Learning Representations , December 2021.

Zhixiong Han, Yaru Hao, Li Dong, Yutao Sun, and Furu Wei. Prototypical calibration for few-shot learning of language models. In The Eleventh International Conference on Learning Representations , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Dan Hendrycks and Kevin Gimpel. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. In International Conference on Learning Representations , 2017.

Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1314-1324, 2019.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4700-4708, 2017.

Zhongtao Jiang, Yuanzhe Zhang, Cao Liu, Jun Zhao, and Kang Liu. Generative calibration for in-context learning. arXiv preprint arXiv:2310.10266 , 2023.

Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael C Mozer, and Becca Roelofs. Soft calibration objectives for neural networks. Advances in Neural Information Processing Systems , 34:29768-29779, 2021.

A. Krizhevsky. Learning Multiple Layers of Features from Tiny Images. 2009.

Meelis Kull, Telmo M. Silva Filho, and Peter Flach. Beyond sigmoids: How to obtain well-calibrated probabilities from binary classifiers with beta calibration. Electronic Journal of Statistics , 11(2):5052-5080, January 2017.

Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in neural information processing systems , 32, 2019.

Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. Advances in Neural Information Processing Systems , 32, 2019.

Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. Trainable Calibration Measures for Neural Networks from Kernel Mean Embeddings. In Proceedings of the 35th International Conference on Machine Learning , pages 2805-2814. PMLR, July 2018.

Fabian Küppers, Jan Kronenberger, Amirhossein Shantia, and Anselm Haselhoff. Multivariate confidence calibration for object detection. In The IEEECVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , June 2020.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems , 30, 2017.

Mingkun Li and I.K. Sethi. Confidence-based active learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 28(8):1251-1261, August 2006.

Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Taking a Step Back with KCal: Multi-Class KernelBased Calibration for Deep Neural Networks. In The Eleventh International Conference on Learning Representations , September 2022.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 55(9):1-35, 2023.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Yuchi Liu, Lei Wang, Yuli Zou, James Zou, and Liang Zheng. Optimizing calibration by gaining aware of prediction correctness. arXiv preprint arXiv:2404.13016 , 2024.

Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12009-12019, 2022.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11976-11986, 2022.

TorchVision maintainers and contributors. Torchvision: Pytorch's computer vision library. https: //github.com/pytorch/vision , 2016.

Julian John McAuley and Jure Leskovec. From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. In Proceedings of the 22nd International Conference on World Wide Web , WWW'13, page 897-908, New York, NY, USA, 2013. Association for Computing Machinery.

Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the Calibration of Modern Neural Networks. In Advances in Neural Information Processing Systems , volume 34, pages 15682-15694. Curran Associates, Inc., 2021.

Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating Deep Neural Networks using Focal Loss. In Advances in Neural Information Processing Systems , volume 33, pages 15288-15299. Curran Associates, Inc., 2020.

Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining Well Calibrated Probabilities Using Bayesian Binning. Proceedings of the AAAI Conference on Artificial Intelligence , 29(1), February 2015.

Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine Learning , ICML '05, pages 625-632, New York, NY, USA, August 2005. Association for Computing Machinery.

Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. In CVPR workshops , volume 2, 2019.

Kanil Patel, William H Beluch, Bin Yang, Michael Pfeiffer, and Dan Zhang. Multi-class uncertainty calibration via mutual information maximization-based binning. In International Conference on Learning Representations , 2020.

John Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers , 10(3):61-74, 1999.

Christopher Potts, Zhengxuan Wu, Atticus Geiger, and Douwe Kiela. DynaSent: A dynamic benchmark for sentiment analysis. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 2388-2404, Online, August 2021. Association for Computational Linguistics.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1-67, 2020.

Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) , 2021.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing , pages 1631-1642, 2013.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR, 2019.

Mingxing Tan and Quoc Le. Efficientnetv2: Smaller models and faster training. In International conference on machine learning , pages 10096-10106. PMLR, 2021.

Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 3459-3467. PMLR, 2019.

Ellen M. Voorhees and Dawn M. Tice. Building a question answering test collection. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR '00, page 200-207, New York, NY, USA, 2000. Association for Computing Machinery.

Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax , May 2021.

Cheng Wang. Calibration in deep learning: A survey of the state-of-the-art. arXiv preprint arXiv:2308.01222 , 2023.

Shuoyuan Wang, Jindong Wang, Guoqing Wang, Bob Zhang, Kaiyang Zhou, and Hongxin Wei. Openvocabulary calibration for vision-language models. arXiv preprint arXiv:2402.04655 , 2024.

David Widmann, Fredrik Lindsten, and Dave Zachariah. Calibration tests in multi-class classification: A unifying framework. Advances in neural information processing systems , 32, 2019.

Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Marilyn Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online, October 2020. Association for Computational Linguistics.

Miao Xiong, Ailin Deng, Pang Wei W Koh, Jiaying Wu, Shen Li, Jianqing Xu, and Bryan Hooi. Proximityinformed calibration for deep neural networks. Advances in Neural Information Processing Systems , 36:68511-68538, 2023.

Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In Proceedings of the Eighteenth International Conference on Machine Learning , ICML '01, pages 609-616, San Francisco, CA, USA, June 2001. Morgan Kaufmann Publishers Inc.

Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '02, pages 694-699, New York, NY, USA, July 2002. Association for Computing Machinery.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146 , 2016.

Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In International conference on machine learning , pages 11117-11128. PMLR, 2020.

Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

Shengjia Zhao, Michael Kim, Roshni Sahoo, Tengyu Ma, and Stefano Ermon. Calibrating predictions to decisions: A novel approach to multi-class calibration. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 22313-22324. Curran Associates, Inc., 2021.

Xujiang Zhao, Shu Hu, Jin-Hee Cho, and Feng Chen. Uncertainty-based Decision Making Using Deep Reinforcement Learning. In 2019 22th International Conference on Information Fusion (FUSION) , pages 1-8, July 2019.

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning , pages 12697-12706. PMLR, 2021.

Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, and Subhrajit Roy. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. arXiv preprint arXiv:2309.17249 , 2023.

A Appendix contents

B discusses the broader impacts of the work.

C discusses the proposed approach in more details, and compares with other methods.

D contains a theoretical justification for Top-versus-All in the case of Temperature Scaling.

E discusses the limits of classwise-ECE and top-label-ECE for a high number of classes.

F describes implementation details, including the computing time in Table 3.

G shows the impact of different calibration methods on selective classification.

H provides additional results: full ECE results in Table 5 and 6, standard deviations in Table 7, confidences in Table 8, accuracies in Table 9, equal-mass bins ECE in Table 10, Brier score in Table

11, experiments for in-context learning of LLMs in Table 12.

B Broader impacts

Our reformulation of the confidence calibration of multiclass classifiers as a binary problem is both simple and general. It has several benefits. On the theoretical side, it might lead to new perspectives on the confidence calibration problem and the development of new calibration methods. On the practical side, existing calibration methods can be adapted to our problem reformulation by adding just a few lines of code. This is an easy and quick way to improve the calibration of classification models. Better-calibrated models are more trustworthy: potential incorrect predictions are more easily identifiable and preventable. However, this also comes with potential risks. The knowledge that a model is well-calibrated might lead to undue trust in the system and the tendency to overlook prediction errors. Even well-calibrated models are not entirely reliable, and developers and users must remember this. Post-processing calibration requires data not included in the training set, which leaves less data available for a thorough evaluation of the model. Calibration does not fix biases in the data. Finally, we tested the calibration improvement only on in-distribution data, but real systems might receive out-of-distribution data (e.g., an image of a new class) or adversarial examples. For such inputs, the classifier predicted probabilities (and thus the confidence) are unreliable, even for well-calibrated models, and a pipeline to filter such data is necessary.

C Details on the method

Algorithm 2 Standard approach

Algorithm 3 Top-versus-all approach

Comparison with the standard approach Algorithm 2 describes the standard approach to postprocessing calibration, and Algorithm 3 describes our approach in more details and shows in blue the differences with the standard approach. Our approach adds a preprocessing step to keep only the confidences instead of the full probabilities vector. It can be seen as creating a surrogate "correctness" classifier and its associated calibration data. The calibrator is learned for the surrogate classifier and applied to the original classifier at inference time. Also, we add regularization for some scaling methods and we have only one binary calibrator instead of one per class.

Comparison with IRM and I-Max IRM [72] and I-Max [49] are, like TvA, multiclass-to-binary reductions. This is why TvA cannot be applied on top of them: they already transform the multiclass problem into a binary one using a different strategy.

The shared class-wise strategy of [49] and the data ensemble strategy of [72] are described very briefly in subsections 3.2 and 3.3.2 of their respective papers and not rigorously justified. Our understanding is that these two strategies do exactly the same thing. To build the calibration set, they concatenate all the class probability vectors so that we get a big probability vector of size N.L ( N samples and L classes) as predictions and similarly concatenate the one-hot embedding of the target class (a big vector with N ones and N. ( L -1) zeros) as targets. Then, they learn a single calibrator. For each example, this calibrator aims to simultaneously increase the probabilities for the target class (target is 1 ) and decrease all the other class probabilities (target is 0 ). The single calibrator is applied to each class probability separately, meaning that the ranking of class probabilities can change, modifying the classifier prediction.

Our strategy derives from transforming the multiclass calibration into a single binary problem. The intuition is to learn the calibrator on a surrogate binary classifier and apply this calibrator to the original classifier. This binary classifier is built on top of the original classifier (by applying the max function to the class probabilities vector). They thus share their confidence. However, the binary classifier aims to solve a different task: predicting the correctness of the original classifier. To build the calibration set, we concatenate all the confidences (a vector of size N) as predictions and concatenate all the correctnesses as targets (also a vector of size N). The correctness value of a given example is 1 if the class prediction is correct; otherwise, it is 0 . Then, we learn a single calibrator, similar to the strategy above. However, there is a key difference: this calibrator aims to increase the probabilities for correct predictions and decrease them for incorrect predictions. Note that our probabilities are all confidences (the maximum class probabilities), meaning we only consider the confidences, which the calibrator directly increases or decreases. In the strategy from [49] and [72], the calibrator has to manage all class probabilities (L times more), even the ones that do not matter, including the lowest class probabilities close to 0. This is less efficient (actually, while this can surely be fixed, the original implementations of IRM and I-Max could not run on ImageNet-21K). This point is closely linked to the analysis of the binary cross entropy loss for scaling methods in Subsection 4.2: when the prediction is incorrect, increasing the probability of the correct class indirectly decreases the confidence (strategy from [49] and [72]) while our strategy directly decreases the confidence.

I-Max is more complex because it modifies the Histogram Binning algorithm, while our approach does not. Additionally, [35] found that I-Max produces unusable probability vectors. Indeed, they do not sum up to 1 , and normalizing them degrades the method's performance.

We wrote our paper with practicality and generality in mind. Contrary to [49] and [72], we demonstrate the generality of our strategy by applying it on top of existing calibration baselines of different natures (scaling and binary). One of our main goals is that practitioners can easily and quickly try our TvA approach, using just a few lines of code, which can significantly improve the calibration performance of their existing calibration pipeline while having no impact on the predicted class by design (except for VS and DC).

Comparison with ProCal Another recent calibration method is ProCal [68]. However, its primary objective differs from ours: it "focuses on the problem of proximity bias in model calibration, a phenomenon wherein deep models tend to be more overconfident on data of low proximity". Its goal is to lower the difference in the confidence score values between regions of low and high density, i.e., to make the confidence score independent of a local density indicator called "proximity." There is no theoretical guarantee, however, that minimizing the proximity bias improves the confidence calibration, the focus of our work. Theorem 4.2 about the PIECE metric is a direct consequence of Jensen's inequality and is true for any random variable D, not necessarily a proximity score.

Theorem 5.1 is an interesting bias/variance decomposition of the Brier score. However, as this type of decomposition usually states, the error may come from bias (here, a wrong initial calibration) or high estimation variance (which can be related to low density but is not expressed as such in the decomposition). We experimentally compare our approach to the ProCal algorithm using the code provided by its authors and observe in Table 2 that our approach gives much better ECE confidence calibration and, for half of the models, also better PIECE values.

ProCal aims to achieve three goals: mitigate proximity bias, improve confidence calibration, and provide a plug-and-play method. We share the last two goals. Concerning improving confidence calibration, our approach has better results, as shown in Table 2. Both approaches are plug-and-play, but they apply very differently. ProCal is applied after existing calibration methods to further improve calibration. It thus does not solve any of the four issues we identified (e.g., cross-entropy loss is still inefficient, and One-versus-All still leads to highly imbalanced problems). Our Top-versus-All approach is a reformulation of the calibration problem that uses a surrogate binary classifier. Existing approaches are applied to this surrogate classifier, which is how the four issues are solved. We do not propose a new method but a new way of applying existing methods. Our approach does not introduce new hyperparameters (except in the particular case of regularizing scaling methods). ProCal introduces several new hyperparameters, such as the choice of the distance, the number of KNN neighbors, or a shrinkage coefficient.

Comparison with Correctness-Aware Loss Aconcurrent work [38] builds a calibration method on top of an intuition similar to ours: binarize the calibration problem. However, what the authors do with this intuition differs vastly from our approach. They derive a Correctness-Aware Loss (Eq. 7 of their paper), which is almost the standard binary cross-entropy loss we use for scaling methods but without a logarithm. They use this loss to learn a separate model that predicts a sample-wise temperature coefficient. This is a new calibration method, which is not straightforward to implement due to the numerous hyperparameters (network architecture, image transformations...). It also requires multiple inferences at test-time, which can be problematic in some production models. Our approach is, again, not a calibration method but a general reformulation of the calibration problem that enhances existing methods. By looking at their Table 1, they get an ECE of 2.22 on ImageNet (in-distribution), while our approach achieves values around 0.5 for most models in our paper's Table 1. Their method, contrary to TvA, improves the AUROC, but in our understanding, it seems mostly due to the use of image transformations, not from their proposed loss. Their method seems to work best in out-of-distribution scenarios, which is not the main objective of our paper. However, these good results for AUROC and out-of-distribution scenarios make this method complementary to our approach, and combining the two in some way could be promising.

D Theoretical justification for Top-versus-All in the case of Temperature Scaling

We define L the number of classes, f k ( x ) the classifier estimated probability for class k and data sample x , y the correct class, and the confidence s ( x ) := max k f k ( x ) . The cross-entropy loss is l CE ( x, y ) = -∑ L k =1 1 { k = y } · log( f k ( x )) = -log( f y ( x )) . Because the last layer of the classifier is a softmax function, f y ( x ) = e zy ∑ k e z k with z the logits vector. Note that we omit the writing variable x in the following for clarity.

Temperature scaling optimizes a coefficient T > 0 that scales the logits vector. Predicted probabilities become f y ( x ) = e zy /T ∑ k e z k /T .

Let us first develop the standard cross-entropy loss when temperature scaling is applied:

Let us compute its gradient:

For our TvA approach, the problem becomes binary. The classification output becomes the confidence s ( x ) = max j ∈Y f j ( x ) and the ground truth label becomes a binary representation of the prediction correctness: y b = 1 ˆ y=y with ˆ y ( x ) = arg max k ∈Y f k ( x ) and 1 the indicator function. The loss we use is the binary cross entropy l BCE ( x, y ) = -( y b · log s ( x ) + (1 -y b ) · log(1 -s ( x )) )

Let us compute the gradient:

First case, the prediction is correct: y b = 1 and z m = z y . Let us inject these in (8): ∂l BCE ∂T = 1 T 2 · ( z y -∑ k z k · f k ) = ∂l CE ∂T . We thus get the same gradient as the standard cross-entropy loss.

Second case, the prediction is incorrect: y b = 0 and z m > z y . (8) becomes: ∂l BCE ∂T = 1 T 2 · s s -1 · ( z m -∑ k z k · f k ) . By comparing to (7), we have the term 1 T 2 · ( z m -∑ k z k · f k ) > 1 T 2 ( z y -∑ k z k · f k ) = ∂l CE ∂T and the remaining part of (8) | s s -1 | > 1 when s > 0 . 5 .

So to recapitulate, | ∂l BCE ∂T | > | ∂l CE ∂T | when s > 0 . 5 , which corresponds to the vast majority of data points as the classifier gets better calibrated. This is shown in Figure 1.

We also have lim s → 1 | s s -1 | = ∞ . In practice, s is not close enough to 1 to generate exploding gradients, so it just means that as confidences for wrong predictions gets higher, so does the gradient to reduce the confidences.

The conclusion is that for correct predictions, our approach does not change the optimization, but for incorrect predictions, the gradient is stronger and penalizes more heavily confident predictions that are wrong. This is also proven experimentally by looking at Table 8 where we see that average

confidences of temperature scaling with the TvA approach (TS TvA ) are lower than the ones using the standard approach (TS), for almost all networks. This makes the average confidences closer to the accuracy, showing reduced overconfidence.

E Limits of classwise-ECE and top-label-ECE for a high number of classes

0.0

0.2

0.4

0.6

0.8

1.0

10

0

10

1

10

2

10

3

10

4

class probability for class 677

number of Samples (log scale)

0.0

0.2

0.4

0.6

0.8

10

0

10

1

10

2

10

3

10

4

class probability for class 537

0.0

0.2

0.4

0.6

0.8

1.0

10

0

10

1

10

2

10

3

10

4

class probability for class 845

Let us define the ECE for class j :

The difference compared to (2) is that now acc ( b, j ) corresponds to the proportion of class j in the bin. Also, conf ( b, j ) now is the average probability given to class j for all samples in the bin. Then, classwise-ECE [29] takes the average for all classes:

Classwise-ECE considers the full probabilities vectors: all the class probabilities for each prediction. However, this metric does not scale to large numbers of classes. Let us see why with an example.

Let us use a test set of N samples, N/L for each of the L classes (the dataset is balanced), and a high-accuracy classifier fairly calibrated. The classifier predicts N probability vectors of length L . Predicted probabilities for class j are all the values of the vector at dimension j . Because the classifier has a high accuracy and is fairly calibrated, around N/L values are close to 1 (corresponding to mostly correct predictions), and the remaining ones, around N -N/L , are close to 0 (because the predicted class is not class j , and the predicted probability is high for another class).

To compute ECE j with equal size 15 bins, the predicted probabilities for class j are partitioned into 15 bins. The first bin (with probabilities close to 0), contains n 1 ≈ N -N/L samples while the last one (with probabilities close to 1) contains n B ≈ N/L samples. The remaining bins are usually even more empty. That means that the calibration error in the first bin is weighted n 1 /n B = L -1 times more than the last one. For the 1000 classes of ImageNet, L -1 = 999 . Figure 4 shows the number of samples ( n b ) in each bin for an ImageNet classifier.

Because the impact of the calibration error in the bin with the high probabilities is negligible relative to the bin with the low probabilities, the classwise-ECE mostly measures whether probabilities close to 0 are calibrated. We argue this is not what we are interested in: what matters more is the calibration of higher values of the probabilities.

Top-label ECE [18] is another interesting metric that does not scale to large numbers of classes either. Top-label-ECE divides data into subsets according to the predicted class, computes the ECEs of these subsets, and averages them. For an ImageNet test set of 25000 samples (25 per class), data is divided into 1000 subsets of ≈ 25 samples each (the classifier is high-accuracy, most of the time the predicted class is equal to the true class). The ECE is computed for each subset containing only 25 samples. To compute the ECE, samples are typically partitioned into 15 bins. The number of samples per bin does not allow a correct estimation of the average confidence or accuracy.

F Implementation details

F.1 Models weights

Model weights for CIFAR are from [45].

Model weights for ImageNet come from torchvision [42].

Model weights for ImageNet-21K are from [54].

CLIP weights are from OpenAI's Hugging Face.

Original weights for T5 and RoBERTa come from the Transformers library [67]. The models are fine-tuned for each task using prompt-based learning [36]. For more details, see [5].

We used the model GPT-J https://huggingface.co/EleutherAI/gpt-j-6b (Apache2.0 license) and Llama-2 https://huggingface.co/meta-llama/Llama-2-13b (license).

F.2 Datasets

CIFAR-10 (C10) and CIFAR-100 (C100) [27] contain 60000 32x32 images corresponding to 10 and 100 classes, respectively. Data is split into subsets of 45000/5000/10000 images for train/validation/test. We concatenate the original validation and test sets, and randomly split that into a calibration set of size 5000, and a test set of size 10000.

ImageNet (IN) [7] contains 1.3 million images from 1000 classes. Following [17], we randomly split the original validation test of size 50000 into a calibration set and a test set, both of size 25000.

ImageNet-21K (IN21K) [54], in its winter21 version, contains 11 million images in the train set, and 522500 in the test set (50 for each of the 10450 classes). We randomly split the test set into equal-sized calibration and test set (261250 samples each, 25 per class).

Amazon Fine Foods [43] is a collection of customer reviews for fine foods sold on Amazon. Reviews are categorized into bad, neutral, and good. The original validation set size is 78741 and test size 91606. We randomly split them into 78741 samples for calibration and 91606 for test.

DynaSent [51] is a dynamic benchmark for sentiment analysis consisting of sentences annotated as positive, neutral, and negative. The original validation set size is 11160 and test size 4320. We randomly split them into 11160 samples for calibration and 4320 for test.

MNLI [66] contains pairs of sentences labeled as contradiction, neutral, and entailment. The original validation set size is 19635 and test size 9815. We randomly split them into 19635 samples for calibration and 9815 for test.

Yahoo Answers (YA) [73] contains question-answers pairs corresponding to 10 different topics. The original validation set size is 14000 and test size 60000. We randomly split them into 14000 samples for calibration and 60000 for test.

TREC [61] contains questions categorized into 6 classes. The training set contains 5500 labeled questions, and the test set contains another 500.

SST-5 [55] contains 11855 sentences corresponding to 5 sentiments (from very negative to very positive).

DBpedia [74] contains text for topic classification with 14 classes. The training set contains 560000 samples, and the test set 5000.

F.3 Code

We used the library netcal [32] (Apache-2.0 license) for their implementation of binary methods for calibration and adapted their reliability diagrams code. For HB, we tested equalsize and equal-mass bins, and chose the best variant for each case. All hyperparameters were kept at their default values (10 bins for HB).

We took inspiration from the official implementation of temperature scaling: https:// github.com/gpleiss/temperature_scaling (MIT license).

We took inspiration from the official implementation of Dirichlet calibration: https: //github.com/dirichletcal/experiments_dnn (MIT license).

We used the official implementation of I-Max: https://github.com/boschresearch/ imax-calibration (AGPL-3.0 license).

We used the official implementation of IRM: https://github.com/zhang64-llnl/ Mix-n-Match-Calibration (MIT license).

For evaluation, we used codes from https://github.com/JeremyNixon/ uncertainty-metrics-1 (Apache-2.0 license) and https://github.com/IdoGalil/ benchmarking-uncertainty-estimation-performance (MIT license).

We used PyTorch 2.0.0 [3] (BSD-style license).

We used CIFAR models from [45] https://github.com/torrvision/focal_ calibration (MIT license).

We used ImageNet-21K models [54] https://github.com/Alibaba-MIIL/ ImageNet21K (MIT license).

We used CLIP models from HuggingFace's Transformers library [67] (Apache-2.0 license).

We used pretrained language models from Transformers [67] and calibration codes from [5] https://github.com/lifan-yuan/PLMCalibration (MIT license).

We used code from https://github.com/mominabbass/LinC for the calibration of LLMs using ICL, itself built upon https://github.com/tonyzhaozh/ few-shot-learning (Apache-2.0 license).

G Impact on selective classification

Selective classification aims to improve a model's prediction performance by trading-off coverage: a reject option allows to discard data that might result in wrong predictions, thus improving the accuracy on the remaining data. A strong standard baseline uses thresholding on the maximum softmax probability outputted by the classifier [13]. Improving confidence calibration means uncertainty is better quantified and should result in better selective classification.

Results in Table 1 show the superiority of Histogram Binning (applied with the right framework) in reducing the calibration error ECE. Unfortunately, it does not translate into improvements in selective classification. AUROC is a standard metric for selective classification [11]. Table 4 shows that Histogram Binning actually degrades the AUROC, while the best method is Isotonic Regression. Our TvA framework does not significantly impact the AUROC.

This paper addresses confidence calibration, usually measured by ECE. AUROC is a global rankbased metric for selective classification: it relies on the relative values of the scores, not their absolute values. Even though calibration and selective classification are related, improvement in calibration does not directly translate to better selective classification. This has been clearly demonstrated experimentally by [11].

A good example of that difference is the behavior of HBTvA: it is the best calibration method overall but actually degrades the AUROC in most cases. Such a difference can be explained by the fact that selective classification benefits from a continuous score able to discriminate between certain and uncertain examples finely, but HB quantizes the confidences into, e.g., 10 different values.

(a) CIFAR-10

Model

Uncal.

IRM

TS

TSTvA

VS

VSreg_TvA

DC

DCreg_TvA

Beta

BetaTvA

TvA

BBQ

BBQTvA

HB

HBTvA

ResNet-50

92.09

91.79

91.30

92.01

91.98

92.71

92.76

92.71

92.76

90.94

92.09

92.29

91.87

75.63

85.53

75.78

84.69

ResNet-110

92.26

92.25

91.53

92.19

92.18

92.18

92.31

92.16

92.30

91.33

92.26

92.40

92.20

73.44

85.01

74.46

84.56

WRN

91.17

91.17

90.36

91.21

91.19

91.80

92.32

91.82

92.31

90.45

91.17

92.28

91.18

75.99

86.62

73.91

85.38

DenseNet

90.46

90.15

89.68

90.48

90.45

90.84

91.40

90.84

91.39

89.98

90.46

91.46

90.12

77.07

87.07

74.61

83.07

CLIP (ViT-B/32)

89.85

89.72

89.73

89.97

89.97

90.73

91.07

90.95

91.14

90.58

89.85

90.28

89.66

88.86

89.41

88.01

88.76

CLIP (ViT-B/16)

91.00

90.96

90.72

91.10

91.10

91.79

91.99

91.88

92.06

91.75

91.00

90.69

90.83

89.36

90.57

89.12

89.97

CLIP (ViT-L/14)

93.22

93.15

94.32

93.32

93.31

93.69

93.78

93.79

93.87

93.17

93.22

92.78

93.05

87.29

91.73

88.65

91.33

(b) CIFAR-100

Model

Uncal.

IRM

TS

TSTvA

VS

VSreg_TvA

DC

DCreg_TvA

Beta

BetaTvA

TvA

BBQ

BBQTvA

HB

HBTvA

ResNet-50

85.80

85.78

85.03

85.73

85.62

85.92

86.50

85.91

86.51

85.49

85.80

87.19

85.69

81.21

85.31

81.90

83.90

ResNet-110

85.03

84.99

83.76

84.94

84.85

84.82

85.34

84.84

85.33

84.57

85.03

86.32

84.97

80.54

84.47

81.40

82.93

WRN

87.59

87.52

87.02

87.59

87.48

87.83

88.17

87.85

88.14

87.45

87.59

88.65

87.46

83.59

86.46

83.77

85.69

DenseNet

86.17

86.02

85.64

86.12

86.00

86.60

86.79

86.59

86.79

86.01

86.17

87.09

86.11

83.02

85.42

83.47

84.86

CLIP (ViT-B/32)

83.06

83.14

82.99

83.74

83.70

83.95

85.16

83.99

85.02

84.17

83.06

84.61

82.95

85.59

83.00

85.61

82.82

CLIP (ViT-B/16)

82.57

82.55

82.51

83.65

83.66

83.95

85.29

84.04

85.76

84.18

82.57

84.45

82.51

85.76

82.46

85.62

82.24

CLIP (ViT-L/14)

84.26

84.22

84.14

85.51

85.50

86.56

87.57

86.62

87.50

85.64

84.26

86.94

84.11

86.92

84.12

86.67

84.07

(c) ImageNet

Model

Uncal.

IRM

TS

TSTvA

VS

VSreg_TvA

DC

DCreg_TvA

Beta

BetaTvA

TvA

BBQ

BBQTvA

HB

HBTvA

ResNet-18

85.73

85.70

85.37

85.64

85.65

85.63

85.88

85.30

85.39

84.39

85.73

86.12

85.69

83.19

85.40

83.73

85.03

ResNet-34

86.18

86.19

85.78

86.11

86.10

86.25

86.38

85.86

85.99

84.64

86.18

86.41

86.14

82.26

85.77

83.03

85.77

ResNet-50

80.53

80.54

80.12

85.93

85.69

85.60

85.57

85.58

85.57

82.34

80.53

86.91

80.49

85.27

80.52

83.63

79.96

84.18

84.20

83.57

85.96

85.71

85.38

85.56

85.34

85.53

82.38

84.18

87.09

84.18

82.48

84.11

81.16

83.80

EffNet-B7

84.92

84.84

84.10

86.61

86.34

85.18

85.51

85.18

85.52

81.91

84.92

87.14

84.87

81.57

84.94

80.38

84.26

EffNetV2-S

85.77

85.87

85.29

87.02

86.86

85.30

85.67

85.28

85.68

82.55

85.77

87.42

85.72

82.44

85.75

80.86

84.80

EffNetV2-M

82.36

82.36

81.58

85.26

84.92

83.66

84.19

83.64

84.23

80.51

82.36

86.51

82.32

81.24

82.23

79.45

81.71

EffNetV2-L

84.63

84.58

83.96

86.33

86.05

85.77

85.94

85.75

85.87

82.27

84.63

86.70

84.55

81.78

84.56

80.56

84.08

ConvNeXt-T

82.35

82.31

81.84

85.47

85.18

85.60

85.57

85.59

85.59

81.93

82.35

86.97

82.29

82.58

82.30

81.44

81.88

ConvNeXt-S

82.29

82.36

81.87

85.27

84.88

84.81

85.01

84.81

85.01

81.22

82.29

86.98

82.26

81.29

82.20

79.78

81.86

ConvNeXt-B

82.27

82.27

81.70

85.13

84.75

84.40

84.88

84.40

84.90

80.87

82.27

87.01

82.22

81.79

82.25

79.89

81.69

ConvNeXt-L

82.35

82.35

81.42

84.81

84.38

84.04

84.58

84.04

84.54

80.24

82.35

86.79

82.32

80.49

82.23

79.15

81.75

ViT-B/32

85.57

85.60

85.10

86.31

86.13

85.95

85.98

85.93

85.98

83.56

85.57

87.16

85.54

83.39

85.55

82.69

85.11

ViT-B/16

85.52

85.55

84.92

86.32

86.12

85.36

85.53

85.34

85.56

81.82

85.52

87.19

85.50

81.56

85.39

81.21

85.09

ViT-L/32

85.42

85.45

84.78

85.93

85.73

85.19

85.29

85.20

85.30

82.07

85.42

87.25

85.41

81.51

85.40

81.42

85.09

ViT-L/16

85.85

85.83

84.63

86.16

86.00

84.33

84.65

84.32

84.64

80.96

85.85

86.97

85.83

79.76

85.66

80.08

84.85

ViT-H/14

87.28

87.24

86.60

87.53

87.34

86.74

86.71

86.77

86.78

82.15

87.28

86.65

87.22

79.33

86.54

80.08

85.32

Swin-T

85.68

85.71

85.04

86.50

86.34

85.79

85.86

85.80

85.86

83.14

85.68

87.10

85.66

82.15

85.64

81.39

85.16

Swin-S

85.37

85.36

84.75

85.99

85.78

84.99

85.20

85.00

85.20

80.92

85.37

86.92

85.35

80.25

85.32

80.05

84.80

Swin-B

84.11

84.26

83.38

85.26

84.91

83.93

84.18

83.92

84.20

79.78

84.11

86.55

84.19

78.94

84.24

79.09

83.09

SwinV2-T

85.80

85.79

85.10

86.74

86.56

85.82

86.04

85.80

86.02

83.06

85.80

87.29

85.78

82.58

85.76

81.47

85.26

SwinV2-S

85.75

85.75

85.03

86.61

86.39

85.19

85.51

85.19

85.50

81.77

85.75

87.18

85.74

80.54

85.75

80.41

84.73

SwinV2-B

85.15

85.13

84.19

86.07

85.82

84.43

84.70

84.40

84.65

80.31

85.15

86.99

85.16

79.57

84.98

79.42

83.86

CLIP (ViT-B/32)

80.56

80.56

80.24

80.59

80.57

81.73

83.21

77.72

77.92

81.14

80.56

81.43

80.52

81.85

80.53

82.14

80.18

CLIP (ViT-B/16)

81.12

81.08

80.96

81.16

81.15

82.28

83.64

78.10

84.57

81.52

81.12

82.17

81.10

82.33

81.07

82.42

80.76

CLIP (ViT-L/14)

82.98

82.95

82.50

82.90

82.86

83.66

85.07

80.09

79.86

82.75

82.98

83.29

82.93

82.85

82.80

82.87

82.36

Model

Uncal.

IRM

TS

TSTvA

VS

VSreg_TvA

DC

DCreg_TvA

Beta

BetaTvA

TvA

BBQ

BBQTvA

HB

HBTvA

MN3

68.79

err.

err.

67.80

65.89

80.00

81.00

61.24

51.94

err.

68.79

79.62

68.77

err.

68.77

90.86

68.40

ViT-B/16

72.99

err.

err.

74.36

73.17

79.78

81.42

76.29

78.92

err.

72.99

79.66

73.10

err.

73.20

90.27

71.95

(e) Amazon Fine Foods

Model

Uncal.

IRM

TS

TSTvA

VS

VSreg_TvA

DC

DCreg_TvA

Beta

BetaTvA

TvA

BBQ

BBQTvA

HB

HBTvA

T5

85.04

84.99

84.32

85.11

85.11

87.84

87.99

88.09

88.02

87.46

85.04

87.03

85.04

85.53

83.47

80.53

79.41

T5-large

81.33

81.34

80.52

80.75

80.74

85.84

87.94

87.47

87.80

87.27

81.33

85.15

81.38

83.35

78.27

77.53

74.25

RoBERTa

83.52

83.50

80.50

83.34

83.34

84.96

86.75

86.44

86.56

86.27

83.51

83.93

83.51

81.62

75.17

75.80

73.35

RoBERTa-large

87.88

87.67

82.18

87.99

87.99

88.04

88.25

88.70

88.33

88.40

87.88

87.34

87.88

81.52

74.92

75.63

75.30

(f) DynaSent

Model

Uncal.

IRM

TS

TSTvA

VS

VSreg_TvA

DC

DCreg_TvA

Beta

BetaTvA

TvA

BBQ

BBQTvA

HB

HBTvA

T5

78.01

77.80

77.51

78.12

78.12

78.11

78.34

78.31

78.31

78.20

78.01

78.57

77.91

76.88

76.81

74.14

76.04

T5-large

77.61

77.69

76.63

77.81

77.81

77.82

79.18

78.87

79.35

78.83

77.61

79.20

77.61

74.72

74.34

69.26

73.32

RoBERTa

75.16

75.15

72.77

75.30

75.30

75.23

75.47

75.60

76.05

75.31

75.16

76.10

75.06

64.47

66.97

59.47

68.49

RoBERTa-large

76.18

75.83

72.80

76.34

76.34

76.25

76.52

76.05

76.12

75.67

76.18

76.13

76.06

61.33

66.46

59.53

68.27

(g) MNLI

Model

Uncal.

IRM

TS

TSTvA

VS

VSreg_TvA

DC

DCreg_TvA

Beta

BetaTvA

TvA

BBQ

BBQTvA

HB

HBTvA

T5

84.44

84.39

83.31

84.51

84.51

84.55

85.01

85.07

85.15

84.59

84.44

84.89

84.37

79.85

80.50

70.97

78.39

T5-large

82.01

81.83

79.68

82.10

82.10

82.08

81.99

82.26

82.30

82.14

82.01

82.11

81.92

66.48

69.68

59.63

72.69

RoBERTa

82.36

82.33

79.46

82.50

82.50

82.48

82.83

82.82

82.89

82.78

82.36

82.99

82.32

67.99

72.67

60.25

73.19

RoBERTa-large

83.58

83.49

79.43

83.63

83.63

83.58

83.78

83.63

83.75

83.49

83.58

83.58

83.50

68.68

72.21

61.39

74.25

Model

Uncal.

IRM

TS

TSTvA

VS

VSreg_TvA

DC

DCreg_TvA

Beta

BetaTvA

TvA

BBQ

BBQTvA

HB

HBTvA

T5

81.60

81.53

81.10

81.61

81.61

81.64

81.65

81.63

81.65

81.59

81.60

81.70

81.53

80.46

80.81

78.99

80.03

T5-large

81.09

81.00

80.45

81.08

81.07

81.16

81.17

81.22

81.29

81.07

81.09

81.15

81.05

79.24

79.89

77.41

79.06

RoBERTa

78.63

78.59

76.52

78.80

78.80

78.73

78.70

78.90

79.07

78.68

78.63

78.86

78.57

74.45

73.17

73.20

67.91

RoBERTa-large

79.13

79.08

76.92

79.41

79.41

79.30

79.46

79.37

79.81

79.34

79.13

79.35

79.04

75.00

72.69

73.67

72.96

H Additional results

(a) CIFAR-10

(d) ImageNet-21K

(a) Amazon Fine Foods

(a) CIFAR-10

(a) CIFAR-10

(a) CIFAR-10

(b) CIFAR-100

(c) ImageNet

(d) ImageNet-21K

(e) Amazon Fine Foods

(f) DynaSent

(g) MNLI

(h) Yahoo Answers

(a) CIFAR-10

(a) CIFAR-10

Large Language Models (LLMs) exhibit an in-context learning (ICL) capability, meaning they can learn from just a few examples in the context. It works by constructing a prompt that includes input-output pairs demonstrating the considered task, followed by a query for a new input. See [8] for a survey. Recent works develop calibration methods whose main goal is to improve the performance of ICL for LLMs, without requiring a complicated model fine-tuning. [77] uses a customized variant of Platt scaling (more specifically, Vector Scaling). Their method infers good values of the vector scaling parameters in a data-free procedure. The idea is that for a "content-free" input, e.g., "N/A", the calibrated probability has a 50% chance (for a binary classification task) of removing a bias toward the positive or negative class. In our paper, we denote this method as ConC . [1] builds on top of this work but uses a calibration set to learn the scaling parameters by minimizing the cross-entropy loss. This can be considered as Matrix Scaling. We denote this method as LinC . [78] proposes a per-class normalization of the probabilities on a given batch. [25] estimates the in-context model label marginal p(y) from limited data and uses it to calibrate the model probabilities. Paper [20] uses a Gaussian mixture model.

In our experiments, we have tested a two-step calibration. First, we use the state-of-the-art method LinC to maximize the accuracy by learning scaling parameters on a calibration set. Then, we apply HBTvA to scale the confidences to lower the calibration error ECE, while preserving the accuracy gains. We use the same calibration set for the two methods. LinC performance depends on hyperparameter values, but to keep the experiments simple, we fixed the following values: 100 epochs, a learning rate of 0.001, and 300 calibration samples. It means that the reported performance of LinC is suboptimal and could be enhanced even more. We used the same experimental setting as [1]. We used the models GPT-J with 6B parameters [62] and Llama-2 with 13B parameters [59]. The text classification datasets are TREC [61] for question classification with 6 classes, SST-5 [55] for sentiment analysis with 5 classes, and DBpedia [74] for topic classification with 14 classes. The 0-shot, 1-shot, 4-shot, and 8-shot learning settings were tested. Five different sets of 300 test samples were randomly selected, and results are averaged over 5 seeds. We evaluated the accuracy and ECE for each configuration. Please see Table 12 for the results. In most cases, LinC+HBTvA achieves the best accuracy and ECE.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our main claims in the abstract and introduction are that our approach solves the failure of many calibration methods when applied to problems with many classes and that we conduct extensive experiments. This reflects our analyses in Sections 3 and 4 and experimental results in Section 5 and Appendices G and H.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 6 is a separate "Limitations" section in the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: No strong theoretical result, but mathematical calculations are in Appendix D.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We use publicly available models and data and experiments details are provided in Section 5 and Appendix F. Our provided code can be used to reproduce our results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We use publicly available models and data and provide a clean code implementation aiming for easy reuse and reproducibility.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details are provided in Section 5, Appendix F; and in the provided code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All our tables show values averaged over five random seeds that generate different calibration and test datasets, as mentioned in Section 5. Standard deviations are reported in Table 7.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Table 3 in the Appendix provides computing times and the GPU model used.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper does conform with the NeurIPS Code of Ethics and we preserve anonymity.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Appendix B discusses the broader impacts of the work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We believe that improved confidence calibration does not have a high risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The papers producing the used datasets are cited. The code implementations used are cited and the URLs are provided in Appendix F.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our provided code is well documented.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper proposes a confidence calibration method.  It reformulates a multi-class problem as the binary task "is the prediction correct?".  Then a given calibration method  (e.g. Temperature Scaling) is applied to that binary task.

STRENGTHS:
The proposed method is very simple and can be efficiently applied to tasks with a large number of classes,

WEAKNESSES:
The main justification of a calibration model is its improved results (usually measured by ECE).   
In the case of TS the paper reports a minor improvement. However,  in the case of HB, the paper reports a huge calibration improvement.  It is not clear what is the reason for the improvement difference.

As far as I understand the HB method, we first split the validation samples  to bins according to the confidence.    


Although the proposed idea is very very simple,  the  method presentation is overcomplicated and there is  an over-seling of the method.

QUESTIONS:
ggg

LIMITATIONS:
ggg
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the issue of miscalibrated confidence scores in neural network classifiers, particularly for problems with many classes. Traditional methods often fail in these scenarios. The authors propose transforming the multiclass calibration problem into a single binary classification problem, termed Top-versus-All (TvA), allowing the use of standard calibration methods more efficiently. This approach improves the performance of calibration methods like Temperature Scaling and Vector Scaling by better utilizing the calibration data and reducing overfitting. Experiments on various datasets and models demonstrate the scalability and effectiveness of this method.\n\nSTRENGTHS:\nSimple Method: The Top-versus-All (TvA) approach is straightforward and easy to implement, requiring minimal changes to existing calibration methods.\n\nGood Performance: The TvA method significantly enhances the performance of standard calibration techniques, consistently improving Expected Calibration Error (ECE) across various datasets and models.\n\nScalability: The method scales well to handle classifiers with a large number of classes, addressing a common limitation in traditional calibration methods.\n\nWEAKNESSES:\n1. Lack of novelty. I read a paper very similar to the idea previously, make top vs all, but I am sorry that I cannot find the link. And as I remember, the Multiclass to Binary idea has been used in this literature multiple times.\n\n2. Lack of the comparison with some SOTA post hoc methods such as [1] , the performance may not succeed.\n\n3. Lack of comparison with some SOTA train time methods, such as [2].\n\n4. Overall, the method lack the analysis and comparison with other SOTA methods.\n\n[1] Proximity-Informed Calibration for Deep Neural Networks\n[2] Dual focal loss for calibration\n\nQUESTIONS:\nsee Weaknesses\n\nLIMITATIONS:\nsee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-06"}

comments: SUMMARY:
This paper presents a top-versus-all (TvA) approach for the confidence calibration of classifiers with many classes. The authors categorizes the post-processing calibration methods into two groups: scaling and binary methods, and lists several problems with these approaches. The authors reformulate the problem of calibrating a multiclass classifier into a single binary classifier, where the confidence score represents the maximum class probability of the binary problem. They show the effectiveness of the proposed method on numerous neural networks used for image or text classification.

STRENGTHS:
1) This paper nicely presents the shortcomings of the existing methods and describes how the proposed method addresses these challenges, the proposed method is innovative and provides a simple and efficient solution to calibrating classifiers with many classes.
2) This paper presents extensive experimental results on multiple datasets and network architectures.
3) This paper provides a well-structured review of relevant literature and clearly explains the motivation and significance of the proposed approach.

WEAKNESSES:
1) The theoretical contribution of the TvA approach and its impact on calibration could be further strengthened.
2) The propose method may exhibit limited novelty, as it builds upon established approaches rather than introducing entirely novel elements.
2) The calibration of CLIP appears to be better than other models on ImageNet, can you explain why? Additionally, it seems like most calibration methods don't work on CLIP, which confuses me.

QUESTIONS:
Please see my comments in Weaknesses.

LIMITATIONS:
The authors have addressed the limitations in the main text.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper relates to post-hoc confidence calibration i.e. the calibration of\nthe top-prediction of a classifier such that ``when the class l is predicted\nwith confidence q, the probability of the actual class being l is also q''.\nThe calibration does not update the classifier.\n\nThe paper tackles two issues of past calibration methods:\n- The one vs all approaches that that do not scale well with the number of\n  classes (there are as many binary calibrators as number of classes)\n- Each binary calibrator is tuned on datasets which imbalance increases with\n  the number of classes: the positives are the instances of class $k$ and the\n  negatives are all the other instances\n- The overfitting of scaling methods due to their parametric nature (the number\n  of parameter increases with the number of classes)\n\nTo do so, it proposes an alternative formulation of the calibration problem:\ninstead of calibrating the top-prediction directly, the paper calibrates an\nintermediate binary classifier that estimates whether the top-prediction is\ncorrect or not.\n\nThis addresses the two previous issues as:\n- There is only one calibration optimization that calibrates the single binary\n  classifier described above, which is independent of the number of classes\n- The imbalance remains but is reduced and linked to the classifier's accuracy\n- The calibration of the binary classifier with scaling methods requires less\n  parameters and is hence less prone to overfitting\n\nThe proposed calibration can be optimized with off-the-shelf binary calibrators\nand the experiments compare the proposed calibration against the traditional\ncalibrations, i.e., calibrating the classifier's top prediction directly. \n\nThe experiments compare the proposed formulation that reduce the ECE compared\nto the traditional calibrations.\nThe proposed formulation usually reduce the ECE compared to the traditional\ncalibration and improves over IRM and I-MAX.\nThe improvement is more notable on datasets with a higher number of classes.\n\nSTRENGTHS:\nS1. The paper is well-written and pleasant to read.\n\nS2. The proposed calibration formulation is simple yet efficient: it calibrates\n   a single intermediate binary problem that estimates whether the classifier's\n   prediction is correct. Thus, it can leverage the strengths of existing\n   methods on a simpler optimization problem.\n\nS3. By design, the proposed method preserves the classification accuracy since\n   it does not edit the network.\n\nS4. The experiments are exhaustive in terms of datasets and calibration methods:\n- the experiments are run on image and text classification on multiple datasets\n  with a wide range of number of classes: 10 on cifar-10 to 10K on ImageNet-21K.\n- scaling methods (Temperature and Vector Scaling, Dirichlet Calibration) and\n  binary methods (Isotonic regression, BBQ and Histogram Binning); and other\n  methods tackling the one-vs-all limitations (IRM and I-MAX).\n\nWEAKNESSES:\nW1. Not really a weakness but a consequence of the design choice: the proposed method keeps the classifier untouched so that the calibration does not impact the accuracy.\nHowever, improving the confidence of incorrect predictions could help improving the classification accuracy by re-distributing the probabilities. \nIn practice though (Tab.8), the accuracy varies only by a few points.\n\nQUESTIONS:\nQ1. One of the claimed advantage of the proposed method (TvA) is the \"stronger\n  gradient\" of TvA compared to that of the default scaling methods (understand\n  the gradient's magnitude is higher).\n  However, it is not clear how this is an advantage on its own as i) the impact of the\n  gradient's magnitude is mitigated by the choice of learning rate; ii) a\n  higher gradient magnitude does not imply a better optimization (this is\n  specific to each optimization problem).\n  Can more details be given on why `stronger gradients' are an advantage?\n\nLIMITATIONS:\n- The advantage of 'preserving' the accuracy is limited by the fact that existing\n  calibration method that affect the accuracy usually induce a variation of ~1%\n  only, and can even affect positively the accuracy i.e. increase it.\n  This leads to the question of whether 'preserving of the accuracy' is really a\n  limitation for other methods.\n\n- Issue 1 describes the limitation of using cross entropy loss during\n  calibration as it increases the probability of the true class which only\n  **indirectly** impacts the probability of the top-prediction i.e. the\n  confidence.\n  L: 173 describes this optimization as 'inefficient' compared to the proposed\n  one, however it is hard to quantify the 'efficiency' of an optimization so a\n  better word might be 'indirect'.\n\nMisc.:\n- Eq6: v is undefined, does it refer to the logit probability vector?\n- lack of legends for the blue and red bars (which is accuracy, which is\n  confidence) even though one can understand\n- Legend for Global acc. and avg. confidence not very visible", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to improve confidence calibration in neural network-based classification models. It transforms the multiclass calibration problem into a binary classification surrogate, demonstrating enhanced performance across various neural network applications in image and text classification.\n\nSTRENGTHS:\n1. The paper's discussion on the standard approach to confidence calibration provides a comprehensive overview of existing methods and their limitations, highlighting the need for more effective strategies.\n   \n2. The Top-versus-All approach presented in the paper offers a straightforward yet powerful solution to the multiclass calibration challenge. By transforming the problem into a binary classification task, it simplifies the calibration process while maintaining effectiveness in improving prediction confidence.\n\n3. The scalability and generality of the proposed method to LLMs are particularly noteworthy. By demonstrating its applicability across diverse neural network architectures used in image and text classification tasks, the paper underscores its potential impact on enhancing calibration methods for a wide range of applications.\n\nWEAKNESSES:\n1. The paper lacks rigorous theoretical evidence to substantiate its claims, which weakens the strength of the proposed solutions. Without detailed proofs, the effectiveness of the Top-versus-All approach and other methods in improving confidence calibration remains somewhat speculative.\n\n2. While the paper claims scalability to LLMs, the experimental evaluation primarily focuses on smaller-scale models. This limited scope raises questions about the method's performance and applicability to state-of-the-art LLMs like LLaMA or Gemma. Extending experiments to these latest models would provide more convincing evidence of scalability and effectiveness across a broader range of neural architectures.\n\n3. Figure 3 in the paper indicates a notable variance in the ECE test results. This variability suggests inconsistent performance across different settings or datasets, potentially indicating limitations in the method's stability and reliability under varying conditions.\n\nQUESTIONS:\n1. how does the method apply to CLIP? On the contrastive softmax loss? What if the number of classes is varying during training CLIP? I.e., the batch size is varying.\n\nLIMITATIONS:\nLimitations are well discussed in the paper", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new learning objective for model calibration in multi-class classification tasks. The authors analyze the issues with the current softmax-based scaling method and argue that it only considers the confidence (the probability of the predicted class) without accounting for other remaining classes during calibration. The proposed Top-versus-All method divides the calibration data into class-wise datasets. Experiments on various models, tasks, and calibration baseline methods have shown some effectiveness of the proposed approach.\n\nSTRENGTHS:\n1. The experiment is conducted on many models and test sets.\n\n2. The motivation is clear and convincing.\n\n3. The writing is clear, and easy to understand.\n\nWEAKNESSES:\n1. This paper calibrates a surrogate binary classifier that predicts whether the class prediction is correct. However, according to the results in Table 3, in the majority of experimental results, the AUROC is worse when using the proposed method. The AUROC is not improved, meaning that the calibrator has not improved its ability to effectively distinguish between correct and incorrect model predictions, thus failing to learn to predict whether the class prediction is correct. This is contradictory to the paper's main objective and narrative. \n\n2. The authors answered YES for Theory Assumptions and Proofs in the checklist. However, no proofs are shown. Some mathematical calculations are presented in Appendix D, but those are not proofs. They overclaim the theoretical contribution.\n\n3. The evaluation metrics are limited. Only quantitative results on ECE and AUROC are reported (even though the AUROC is not improved). Other popular metrics such as ACE, MCE, or PIECE are not provided.\n\n4. Authors report the results on CLIP models but do not discuss or compare them to recent calibration methods for CLIP models. For example:\n    \n   [1] Wang, S., Wang, J., Wang, G., Zhang, B., Zhou, K. and Wei, H., 2024. Open-Vocabulary Calibration for Vision-Language Models. International Conference on Machine Learning (ICML), 2024.\n\n\n5. There are several existing model calibration papers in the literature that focus on directly estimating the probability of correctness for each sample/prediction.\n\n     [2] Xiong, M., Deng, A., Koh, P.W.W., Wu, J., Li, S., Xu, J. and Hooi, B., 2023. Proximity-informed calibration for deep neural networks. Advances in Neural Information Processing Systems, 36, pp.68511-68538.\n\n     [3] Liu, Y., Wang, L., Zou, Y., Zou, J. and Zheng, L., 2024. Optimizing Calibration by Gaining Aware of Prediction Correctness. arXiv preprint arXiv:2404.13016.\n\n    For example, in paper [3], the idea is almost the same with minor differences. It also learns to predict the probability of correctness of the model's predictions as confidence. However, it can distinguish between correct and incorrect predictions based on their AUROC results.\n\nQUESTIONS:\n1. The results in Table 3 are not aligned with the paper's narrative. Why not discuss the mechanism or reason behind this? I think it is a vital issue.\n\n2. Can the authors please provide strong reasons to convince me that your method is superior to similar works [2] and [3]?\n\nLIMITATIONS:\nYes. They have a limitation section.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-14"}