Confidence Calibration of Classifiers with Many Classes

Authors: adrien  le coz
stéphane  herbin
faouzi  adjed
irt  systemx
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 15:39:41

Abstract: For classification models based on neural networks, the maximum predicted class probability is often used as a confidence score. This score rarely predicts well the probability of making a correct prediction and requires a post-processing calibration step. However, many confidence calibration methods fail for problems with many classes. To address this issue, we transform the problem of calibrating a multiclass classifier into calibrating a single surrogate binary classifier.  This approach allows for more efficient use of standard calibration methods. We evaluate our approach on numerous neural networks used for image or text classification and show that it significantly enhances existing calibration methods.

Main paper: 
 introduction: 
 The considerable performance increase of modern deep neural networks (DNNs) and their potential deployment in real-world applications has made reliably estimating the probability of wrong decisions a key concern. When such components are expected to be embedded in safety-critical systems (e.g., medical or transportation), estimating this probability is crucial to mitigate catastrophic behavior. One way to address this question is to treat it as an uncertainty quantification problem [2, 12] , where the uncertainty value computed for each prediction is considered as a confidence. This confidence can be used to reject uncertain decisions proposed by the DNN [13] , for out-of-distribution detection [22] , or to control active learning [34] or reinforcement learning based systems [76] . When confidence values reliably reflect the true probability of correct decisions, i.e., their accuracy, a predictive system is said to be calibrated. In this case, confidence values can be used as a reliable control for decision-making.We are interested in producing an uncertainty indicator for decision problems where the input is high dimensional and the decision space large, typically classifiers with tens to thousands of classes. For this kind of problem, DNNs are common predictors, and their outputs can be used to provide an uncertainty value at no cost, i.e., without necessitating heavy estimation such as Bayesian sampling [15] or ensemble methods [33] . Indeed, most neural architectures for classification instantiate their decision as a softmax layer, where the maximum value can be interpreted as the maximum of the posterior probability and, therefore, as a confidence. Unfortunately, uncertainty values computed in this way are often miscalibrated. DNNs have been shown to be over-confident [17] , meaning their confidence is higher than their accuracy: predictions with 90% confidence might be correct only 80% of the time. A later study [44] suggests that model architecture impacts calibration more than model size, pre-training, and accuracy. For ImageNet classifiers, the accuracy and the number of model parameters are not correlated to calibration, but model families are [11] .These studies show that it is difficult to anticipate the calibration level of confidence values computed directly from DNNs and exhibit the benefits of a complementary post-processing calibration. This calibration process can be seen as a learning step that exploits data from a calibration set, distinct from the training set, and is used to learn a function that maps classifier outputs into better-calibrated values. This process is typically lightweight and decoupled from the issue of improving model performance. A standard baseline for post-processing calibration is Temperature Scaling [17] , where the penultimate logit layer is scaled by a coefficient optimized on the calibration set.Many post-processing calibration methods have been developed for binary classification models [50, 69, 70] . Applying these methods to multiclass classifiers requires some adaptation. One standard approach reformulates the multiclass setting into many One-versus-All binary problems (one per class) [70] . One limitation of this approach is that it does not scale well. When the number of classes is large, the calibration data is divided into highly unbalanced subsets that do not contain enough positive examples to solve the One-versus-All binary problems. Other methods based on Platt scaling [50] involve learning a set of parameters whose size grows with the number of classes. For problems with many classes, they tend to overfit, as we demonstrate in this work.The main idea of our work is to reformulate the multiclass confidence estimation into a single binary problem. This problem can be phrased as the unique question: "Is the prediction correct?". In this formulation, the confidence score is defined as the maximum class probability of the binary problem that outputs 1 if the predicted class is correct and 0 otherwise. The intent is that the confidence score accurately describes whether the prediction is correct, regardless of the class. We show that this novel approach, which we call Top-versus-All (TvA), significantly improves the performance of standard calibration methods: Temperature and Vector Scaling [17] , Dirichlet Calibration [29] , Histogram Binning [69] , Isotonic Regression [70] , Beta Calibration [28] , and Bayesian Binning into Quantiles [46] . We also introduce a simple regularization for Vector Scaling or Dirichlet Calibration that mitigates overfitting when the number of classes is high relative to the calibration data size. We conduct experiments on multiple image and text classification datasets and many pre-trained models.Our main contributions are the following:• We discuss four issues of the standard approach to confidence calibration.• To solve these issues, we develop the Top-versus-All approach to confidence calibration of multiclass classifiers, transforming the problem into a single binary classifier's calibration. This straightforward reformulation enables more efficient use of existing calibration methods, achieved with minimal modifications to the methods' original algorithms.• Applied to scaling methods for calibration (such as Temperature Scaling), TvA allows the use of the binary cross-entropy loss, which is more efficient in decreasing the confidence of wrong predictions and leads to stronger gradients in the case of Temperature Scaling. Applied to binary methods for calibration (such as Histogram Binning), TvA significantly improves their performance and makes them accuracy-preserving.• We demonstrate our approach's scalability and generality with extensive experiments on image classification with state-of-the-art models for complex datasets and on text classification with Pre-trained Language Models (PLMs) and Large Language Models (LLMs). 
related work: 
 Calibration There are various notions of multiclass calibration. One can consider confidence [17] , class-wise [28] , top-r [19] , top-label [18] , decision [75] , projection smooth [16] , or strong [60, 65] calibration. For recent surveys, we refer to [10] and [63] . In this work, we focus on confidence calibration and not on the calibration of the full probability vector. Indeed, confidence calibration is useful for many applications that only require a single confidence value: selective classification [13] , out-of-distribution detection [22] , or active learning [34] . For these applications, stronger notions of calibration are both difficult and useless. Also, class-wise calibration metrics do not appropriately scale to large numbers of classes, a setting we consider in this work, as explained in Appendix E.Metrics Several metrics have been proposed to quantify calibration error. The most common is the Expected Calibration Error (ECE) [46] (see Equation 2). ECE has flaws: the estimation quality is influenced by the binning scheme, and it is not a proper scoring rule [14, 60, 48] . Despite its flaws, it remains the standard comparison metric for confidence calibration. Variants of ECE have also been developed: classwise-ECE [29] , ECE with equal mass bins [48, 44] , or top-label-ECE, which adds a conditioning on the predicted class [18] . The Brier score [4] is also used to measure calibration. The proximity-informed expected calibration error (PIECE) evaluates the miscalibration due to proximity bias [68] . We mainly use the standard ECE in this work, and the Appendix contains more metrics.Training calibrated networks Several solutions have been proposed in the literature to improve calibration by training neural networks in specific ways, generally by making use of a new loss term [31, 58, 26, 6, 5] . While these methods directly optimize calibration during the training phase of the networks, they require a high development time, often compromise accuracy, and are not adapted to pre-trained foundation models. That is why we prefer to focus on calibrating already-trained models.Post-processing (or post-hoc) calibration methods Another approach is to calibrate alreadytrained models. This lowers the development time by decoupling accuracy optimization and calibration. In this paper, we divide post-hoc calibration methods into two categories: scaling and binary. Scaling methods are derived from Platt scaling [50] and optimize some parameters to scale the logits.Temperature Scaling [17] is a popular simple post-processing calibration method. The logits vector is scaled by a coefficient, which modifies the probability vector. Vector Scaling [17] is more expressive and has good performance in many cases [17, 48, 29] . Matrix Scaling can also be considered for more expressiveness but is difficult to apply without overfitting [17] . Dirichlet Calibration [29] proposes a regularization strategy for Matrix Scaling. [72] developed Ensemble Temperature Scaling. Scaling can be combined with binning [30] . Besides logits or probabilities, features can also be used [35] .Another family of methods tackles binary classification. We designate them as binary methods.Histogram Binning [69] divides the prediction into B bins according to the predicted probability. For each bin, a calibrated probability is computed from the calibration data. The probability becomes discrete: it can only take B values. With some modifications, it outperforms scaling methods [18, 49] . Isotonic Regression [70] learns a piecewise constant function to remap probabilities. Bayesian Binning into Quantiles [46] brings Bayesian model averaging to Histogram Binning. Beta Calibration [28] uses a beta distribution to obtain a calibration mapping.Our work reformulates the multiclass calibration problem and allows more efficient use of all these calibration methods, with little to no change in their algorithms.Multiclass to Binary Using binary calibration methods for a multiclass classifier requires adapting the multiclass setting. This is usually done with a One-versus-All approach [70, 17] . The multiclass setting is decomposed into L One-versus-All independent problems: one binary problem for each class. [18] introduce the notion of top-label calibration, i.e., confidence calibration with additional conditioning on the predicted class (top-label). They describe a general multiclass-to-binary framework to develop top-label calibrators. [6] derive L(L -1)/2 pairwise binary problems. The approach requires training the classifier from scratch, and its performance decreases with the number of classes.Our work tackles this multiclass-to-binary research problem. Contrary to [6] , the One-versus-All approach [70] and top-label calibrators [18] , our approach works well for problems with many classes. The methods I-Max [49] and IRM [72] use a shared class-wise strategy to compute a single calibrator. The calibrator is applied to all class probabilities separately, so the class probabilities ranking and prediction might change. In contrast, TvA applied to binary methods rescales the confidence after the class prediction is made. I-Max and IRM consider the full class probabilities vectors, while TvA only considers confidence values. Also, they build on top of Histogram Binning and Isotonic Regression, respectively, while we apply our approach to many calibration methods. A concurrent work building on an intuition similar to ours derives a calibration method based on a Correctness-Aware Loss [38] . Appendix C discusses the differences between our approach and others.3 Problem setting The classifier prediction is the most probable class ŷ = arg max k∈Y f k (x) with f k (x) referring to the probability of class k, and the confidence score defined as s = max k∈Y f k (x). Note that we use the term confidence to denote the maximum class probability. With y the real label, we consider the confidence calibration definition from [17] that says that the classifier f is calibrated if:EQUATIONwhere the probability is over the data distribution. Equation (1) expresses that the probability of being correct when the confidence is around p is indeed p. For instance, if we consider the set of predictions with a confidence of 90%, they should be correct 90% of the time. The conditional probability of (1) is not rigorously defined mathematically (the event {s = p} has zero probability), and interval-based empirical estimators are often used to define metrics capable of evaluating how well (1) is satisfied. This is the case of ECE, which approximates the calibration error by partitioning the confidence distribution into B bins. The absolute difference between the accuracy and confidence is computed for each subset of data in the bins. The final value is a weighted sum of the differences of each bin.ECE = B b=1 n b N |acc(b) -conf(b)| (2)Where n b is the number of samples in bin b, N is the total number of samples, acc(b) is the accuracy in bin b, and conf(b) is the average confidence in bin b. ECE can be interpreted visually by looking at diagrams such as those of Figure 1 : ECE computes the sum of the red bars (difference between bin accuracy and average confidence) weighted by the proportion of samples in the bin.Post-processing calibration methods We are considering the scenario where a classifier has already been trained, and the objective is to enhance its calibration. Post-processing calibration methods aim to remap the classifier probabilities to better-calibrated values without modifying the classifier. They typically use a calibration set different from the training set to optimize parameters or learn a function. We note the calibration data D cal = {(x i , y i )} N i=1 . We focus on post-processing calibration because it enables better utilization of off-the-shelf models and separates model training (optimized for accuracy) from calibration. These advantages significantly reduce the development cost of obtaining a well-performing and well-calibrated model, contrary to optimizing calibration during training. We categorize the post-processing calibration techniques considered in this paper into two groups: scaling methods and binary methods.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Section 6 is a separate "Limitations" section in the paper. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
issues related to current approaches: 
 Behavior of current scaling methods Scaling methods for calibration optimize one or more coefficients that scale the logits vector to minimize on calibration data the cross-entropy loss defined as l CE = -L k=1 1 k=y • log(f k (x)) = -log(f y (x)). Minimizing l CE therefore increases the probability of the true class. We can distinguish two cases to understand what happens during the optimization: whether the prediction ŷ is correct or not. In the first case, the confidence score is s = f y (x): minimizing l CE increases the confidence f y (x). In the second case, the prediction is incorrect, which implies that f y (x) < s. Minimizing l CE increases the probability of the true class f y (x) but does not directly change the confidence (because s ̸ = f y (x)). Instead, the confidence (which was attributed to a wrong class) is indirectly lowered through the softmax normalization.. → Issue 1: Cross-entropy loss only indirecly lowers confidence in wrong predictions.We identified another issue of some scaling methods. By design, the number of parameters optimized by Vector Scaling and Dirichlet Calibration grows with the number of classes. When the number of classes is high, these methods overfit the calibration set as shown in Figure 2 .. → Issue 2: Vector Scaling and Dirichlet Calibration overfit calibration sets with many classes.One-versus-All approach for binary methods The One-versus-All (OvA) calibration approach [70] allows adapting calibration methods for binary classifiers to multiclass classifiers. To do so, it decomposes the calibration of multiclass classifiers into sets of L binary calibration problems: one for each class k. For each problem, the considered probability is f k (x), and the associated label 1 y=k ∈ {0, 1}. When calibrating a classifier from data, each binary problem is highly imbalanced with a ratio between positive and negative examples equal to 1 L-1 if the classes are equally sampled. For instance, for ImageNet, the ratio is 1/999: out of 25000 examples, only 25 have a positive label.. →Issue 3: OvA approach leads to highly imbalanced binary problems.At test time, each of the L class probabilities is calibrated by a separate calibration model. The resulting probability vector can be normalized to ensure a unit norm. Because each probability is calibrated independently, their ranking can change, thus modifying the predicted class. In Table 9 , we see that accuracy is often negatively impacted in practice.. →Issue 4: OvA approach can change the predicted class and negatively impact the accuracy.4 Top-versus-All approach to confidence calibration Algorithm 1 Top-versus-All approach to confidence calibration Input:D cal : {(x i , y i )} Ni=1 the calibration data f : the multiclass classifier g: a calibration function ▷ e.g., Temperature Scaling Preprocessing: ŷi ← arg max k∈Y f k (x i ) ▷ Compute class predictions y b i ← 1 ŷi=yi ▷ Compute predictions correctness f b ← max k∈Y f k ▷ Create surrogate binary classifier D TvA cal ← {(x i , y b i )} N i=1 ▷ BuildQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We use publicly available models and data and provide a clean code implementation aiming for easy reuse and reproducibility.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
general presentation: 
 In the calibration definition (1) and the standard ECE metrics, only the confidence, i.e., the maximal probability, reflects the likelihood of making an accurate prediction. The probabilities of other classes are not taken into account. However, the standard approach to calibration uses the entire set of probabilities, not just confidence, which introduces unnecessary complexity. We aim to simplify the process by reformulating the problem of calibrating multiclass classifiers into a single binary problem. This problem can be phrased as: "Is the prediction correct?". In this setting, we do not calibrate the predicted probabilities vector but only a scalar: the confidence. The remaining probabilities are discarded. This is equivalent to calibrating a surrogate binary classifier that predicts whether the class prediction is correct. Since this correctness classifier only considers the maximal probability versus all others, we call our approach Top-versus-All (TvA).Replacing the standard approach by TvA is straightforward. Given the standard calibration dataD cal = {(x i , y i )} N i=1, we add a few data preprocessing steps. First, compute the class predictions ŷ and their correctness: y b = 1 ŷ=y . Second, create the surrogate binary classifier f b (x) = max k∈Y f k (x). Finally, build the calibration set for the surrogate binary classifier:EQUATIONAfter this preprocessing, we choose a standard calibration function g, e.g., Temperature Scaling, to calibrate the surrogate binary classifier. The learning of the calibration function follows its original underlying algorithm but uses the modified calibration data D TvA cal . The learned calibration function is then applied to the confidences of the original multiclass classifier. Algorithm 1 describes our approach. In the Appendix, Algorithm 3 provides more details and highlights differences with the standard approach of Algorithm 2.After this general presentation, we explain how TvA impacts the two categories of calibration methods, scaling and binary, in Subsections 4.2 and 4.3, respectively. We also justify its behavior.Because our Top-versus-All setting reformulates the calibration of multiclass classifiers into a binary problem, the natural loss is the binary cross-entropy:EQUATIONMinimizing this loss results in confidence estimates that more accurately describe the probability of being correct, regardless of the L -1 less likely class predictions. Using the binary cross-entropy as a calibration loss makes an important difference compared to the usual multiclass cross-entropy. The cross-entropy loss takes into account the probability of the correct class, while with TvA the binary cross-entropy takes into account the probability of the predicted class (i.e., the confidence).As for the standard approach, only two cases are possible. When the prediction is correct, l BCE = -log(s) = -log(f y ) = l CE . We get the same result as the cross-entropy loss: minimizing it directly increases the confidence. But when the prediction is incorrect,l BCE = -log(1 -s) ̸ = l CE .Minimizing the loss now directly decreases the confidence. This is a key difference compared to using the multiclass cross-entropy loss.The impact of the reformulation can be seen for Temperature Scaling, which optimizes a coefficient T that scales the logits z k . The reformulation generates stronger gradients when the prediction is incorrect:EQUATIONwith∂l BCE ∂T = 1 T 2 • 1 s-1 • (max k z k -k z k • f k ) and ∂l CE ∂T = 1 T 2 (z y -k z k • f k ).See Appendix D for the mathematical calculations. Because for interesting problems, the confidence verifies s > 0.5 most of the time (as shown in Figure 1 ), our approach strengthens the gradients. The optimization of the temperature T is more efficient as confident incorrect predictions are more heavily penalized. This effect is not mitigated by the choice of learning rate, which does not vary with s. Applying standard Temperature Scaling usually results in overconfident probabilities, but our approach limits this overconfidence. This is verified experimentally in Table 8 , which displays the average confidences for TS without and with TvA.. →Solution for Issue 1: Use the binary cross-entropy loss resulting from TvA approach.Regularization of scaling methods Overfitting of Vector Scaling and Dirichlet Calibration can be reduced with a simple L2 regularization that penalizes the coefficients of the vector v that are far from the reference value 1.EQUATIONThis regularization allows these methods to take advantage of their additional expressiveness without being subject to overfitting. The loss for Vector Scaling becomes l BCE + λl reg (v) where λ is a hyperparameter. The loss for Dirichlet Calibration uses additional matrix regularization terms. λ is the only additional hyperparameter introduced by our method, and it applies only to Vector Scaling and Dirichlet Calibration.. →Solution for Issue 2: Use L2 regularization.Our TvA approach replaces the One-versus-All approach to apply binary methods to the multiclass setting. TvA transforms the multiclass setting into a single binary problem that uses the binary calibration dataset (3) . In this dataset, the proportion of positive labels equals the classifier's accuracy a. The ratio between negative and positive examples is(1-a)N aN = 1 a -1.For a classifier with 80% accuracy on ImageNet and a calibration dataset of 25000 examples, there are 5000 negative and 20000 positive examples (ratio of 1/4). This is still a bit imbalanced but orders of magnitude smaller than the class-wise binary calibration datasets of the One-versus-All approach (ratio of 1/999).. →Solution for Issue 3: By not dividing the calibration data into class-wise datasets, the TvA approach yields a much better balanced binary calibration problem.The Top-versus-All approach operates on confidence alone, not the full class probabilities vector. This means that the class prediction is already done, and the ranking of the class probabilities does not change, unlike with One-versus-All. The classifier's prediction and accuracy are unaffected. This scheme allows decoupling accuracy improvements (during training time) and calibration (during post-processing calibration), thus avoiding compromises and reducing development time.. →Solution for Issue 4: By operating on confidence alone, the Top-versus-All approach does not impact the classifier's prediction or accuracy for binary methods applied to the multiclass scenario. The subscript TvA signifies that the TvA reformulation was used, and reg means our regularization (6) was applied. Red bars show the differences between bin accuracy (blue bar) and accuracy for perfect calibration (dashed red line).As the methods improve the calibration, these differences are reduced and the average confidence (vertical dotted line) will get closer to the global accuracy (vertical dashed line). 
setting: 
 Datasets and models For image classification, we used the datasets CIFAR-10 (C10) and CIFAR-100 (C100) [27] with 10 and 100 classes respectively, ImageNet (IN) [7] with 1000 classes, and ImageNet-21K (IN21K) [54] with 10450 classes. For text classification, we used Amazon Fine Foods (AFF) [43] and DynaSent (DF) [51] for sentiment analysis with 3 classes, MNLI [66] for natural language inference with 3 classes, and Yahoo Answers (YA) [73] for topic classification on 10 classes. Experiment results are averaged over five random seeds that randomly split the concatenation of the original validation and test sets into calibration and test sets. We used the following models for image classification: ResNet [21] , Wide-ResNet-26-10 (WRN) [71] , DenseNet-121 [24] , MobileNetV3 (MN3) [23] , ViT [9] , ConvNeXt [41] , EfficientNet [56, 57] , Swin [40, 39] , and CLIP [52] which matches input images to text descriptions in a shared embedding space, assigning labels based on the highest similarity score. For text classification, we used the PLMs RoBERTa [37] and T5 [53] . More details about datasets, calibration sets sizes, and model weights can be seen in Appendix F.Baselines Our Top-versus-All ( TvA ) reformulation and regularization ( reg ) can be applied to different calibration methods. We have tested the following scaling methods: Temperature Scaling (TS) and Vector Scaling (VS) [17] , and Dirichlet Calibration (DC) [29] with the best-performing variant Dir-ODIR, which regularizes off-diagonal and bias coefficients. We also tested the following binary methods: Histogram Binning (HB) [69] using for each case the best-performing variant between equal-mass or equal-size bins, Isotonic Regression (Iso) [70] , Beta Calibration (Beta) [28] , and Bayesian Binning into Quantiles (BBQ) [46] . For comparison, we include methods with state-ofthe-art results on problems with many classes: I-Max [49] and IRM [72] . See Appendix C for more details on these methods. More details on code implementations can be seen in Appendix F. For visual qualitative results, Figure 1 displays reliability diagrams [47] . We observe that initially, ResNet-50 is highly underconfident and ViT-B/16 a bit underconfident. Applying TS and VS solves the underconfidence and makes the models slightly overconfident. TvA improves these methods, and the average confidence gets closer to the accuracy. HB TvA makes the calibration almost perfect.Table 1 shows the results of applying the Top-versus-All reformulation to several calibration methods.For clarity, results are averaged over families of models (models based on the same architecture) and the full results are available in Tables 5 and 6 of the Appendix. In most cases, the TvA reformulation significantly lowers the ECE by dozens of percent. Without TvA, binary methods often perturb the prediction and degrade the classifier's accuracy (see Table 9 ), making them inapplicable in a practical setting. TvA solves the issue as it only scales the confidence (after the prediction is made) and makes binary methods outperform scaling methods.Improvements due to TvA are consistent across models. However, exceptions are observed for CLIP: it is the model family with the lowest ECE pre-calibration, but the highest ECE post-calibration for ImageNet. CLIP's multimodal training regime, zero-shot adaptation as a classifier, and very large training dataset might cause this different behavior. CLIP's low ECE was also observed in [44, 11] . [64] specifically tackles the calibration of fine-tuned CLIP, a setting not considered here.We also found that DC is sensitive to hyperparameter tuning, and its performance is usually not much better than VS, which is consistent with [29] . In some cases, the optimization diverges, leading to very poor results, e.g., for CLIP on ImageNet.Improvements due to TvA are also consistent across datasets, although they tend to increase with the number of classes. Improvements on ImageNet are usually better than on CIFAR-100, whose improvements are usually better than on CIFAR-10. This is notable with e.g., TS or HB. The magnitude of improvement is usually higher for binary methods, e.g., HB, than scaling methods, e.g., TS, especially for many classes. This indicates that Issue 3 is more serious than Issue 1.For text datasets with only three classes (AFF, DS, and MNLI), TS does not benefit from TvA, but other methods do, despite the small number of classes. According to [5] , TS is among the best calibration methods for the text classification tasks considered here, even compared to ones that retrain the model. Even so, our method HB TvA significantly outperforms it. Some methods' current implementations could not handle the large scale of ImageNet-21K, resulting in out-of-memory errors written as "err." in the Table . For I-Max and IRM, this is because they consider the full probability vectors while TvA efficiently uses data by considering only confidence values. Indeed, TvA handles this scale without difficulty.Additional results are included in Appendix H. Tables 5 and 6 contain the full results for ECE, with the standard deviations in Table 7 . Table 8 reveals that ImageNet networks are mostly underconfident. This is aligned with [11] and goes against previous knowledge on overconfidence, which was initially believed to be linked to network size [17] . Table 9 provides the accuracies after calibration. Table 10 exhibits that ECE with equal-mass bins has similar values as standard ECE. Table 11 shows that TvA mostly lowers the Brier score, except for Iso, which has the lowest score overall.Calibration methods can also be applied to Large Languages Models (LLMs) using In-Context Learning (ICL) to tackle text classification tasks [77, 20, 25, 78, 1] . The primary goal of these methods is to improve model accuracy. TvA was not designed for this objective, but it can still be applied on top of an existing method that improves the accuracy. TvA then lowers the calibration error while keeping the accuracy gain. Results for GPT-J [62] and Llama-2 [59] are in Table 12 .To summarize the results for practical use, our experiments show that Histogram Binning (within the TvA or I-Max setting) is the best calibration method overall, providing ECE values mostly below 1%. This is the method we advise using. However, suppose the underlying application requires a confidence with continuous values, e.g., to rank the predictions for selective classification. In that case, we advise using a method that improves the AUROC, shown in Appendix G, such as TS or Iso.On ImageNet, VS and DC overfit the calibration set, degrading the calibration on the test set. The lower performance of VS relative to TS indicates this overfitting. As visualized in Figure 2 , combining the binary cross-entropy loss used in the TvA reformulation and an additional regularization term prevents overfitting. We fixed the value λ = 0.01 as it works well across models. Initializing the vector coefficients to 1 T with T obtained by TS TvA helps further improve performance.The size of the calibration set influences the performance of the different methods, as seen in Figure 3 . TS and TS TvA do not benefit from more data due to their low expressiveness. VS does not improve the ECE because of the overfitting problem. In contrast, VS reg_TvA benefits from more calibration data.With enough data (≈ 15000), it outperforms TS TvA . Binary methods using the standard One-versus-All approach have poor performance and need a large amount of data to be competitive. Using TvA, they get excellent performance with little data.  
limitations: 
 Our approach tackles confidence calibration and is unlikely to improve performance for stronger notions of calibration, such as class-wise calibration. However, confidence calibration is useful for many practical cases, such as selective classification [13] , out-of-distribution detection [22] , or active learning [34] . Also, calibration improvements are less significant for problems with few classes (≤ 10) than for problems with many classes, but our approach still provides the best results.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Details are provided in Section 5, Appendix F; and in the provided code.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 Reducing the miscalibration of neural networks is essential to improve trust in their predictions. This can be done after the model training with an optimization using calibration data. However, many current calibration methods do not scale well to complex datasets: binary methods under the One-versus-All setting do not have enough per-class calibration data, and scaling methods are inefficient. We demonstrate that reformulating the confidence calibration of multiclass classifiers as a single binary problem significantly improves the performance of baseline calibration techniques. The competitiveness of scaling methods is increased, and binary methods use per-class calibration data more efficiently without altering the model's accuracy. In short, our TvA reformulation enhances many existing calibration methods with little to no change in their algorithm. Extensive experiments with state-of-the-art image classification models on complex datasets and with text classification demonstrate our approach's scalability and generality.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: All our tables show values averaged over five random seeds that generate different calibration and test datasets, as mentioned in Section 5. Standard deviations are reported in Table 7 .Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: Table 3 in the Appendix provides computing times and the GPU model used.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Appendix B discusses the broader impacts of the work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We believe that improved confidence calibration does not have a high risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The papers producing the used datasets are cited. The code implementations used are cited and the URLs are provided in Appendix F.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: Our provided code is well documented.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: B discusses the broader impacts of the work. C discusses the proposed approach in more details, and compares with other methods. D contains a theoretical justification for Top-versus-All in the case of Temperature Scaling. E discusses the limits of classwise-ECE and top-label-ECE for a high number of classes. F describes implementation details, including the computing time in Table 3 . G shows the impact of different calibration methods on selective classification. H provides additional results: full ECE results in Table 5 and 6 , standard deviations in Table 7 , confidences in Table 8 , accuracies in Table 9 , equal-mass bins ECE in Table 10 , Brier score in Table 11 , experiments for in-context learning of LLMs in Table 12 .Our reformulation of the confidence calibration of multiclass classifiers as a binary problem is both simple and general. It has several benefits. On the theoretical side, it might lead to new perspectives on the confidence calibration problem and the development of new calibration methods. On the practical side, existing calibration methods can be adapted to our problem reformulation by adding just a few lines of code. This is an easy and quick way to improve the calibration of classification models. Better-calibrated models are more trustworthy: potential incorrect predictions are more easily identifiable and preventable. However, this also comes with potential risks. The knowledge that a model is well-calibrated might lead to undue trust in the system and the tendency to overlook prediction errors. Even well-calibrated models are not entirely reliable, and developers and users must remember this. Post-processing calibration requires data not included in the training set, which leaves less data available for a thorough evaluation of the model. Calibration does not fix biases in the data. Finally, we tested the calibration improvement only on in-distribution data, but real systems might receive out-of-distribution data (e.g., an image of a new class) or adversarial examples. For such inputs, the classifier predicted probabilities (and thus the confidence) are unreliable, even for well-calibrated models, and a pipeline to filter such data is necessary. Comparison with the standard approach Algorithm 2 describes the standard approach to postprocessing calibration, and Algorithm 3 describes our approach in more details and shows in blue the differences with the standard approach. Our approach adds a preprocessing step to keep only the confidences instead of the full probabilities vector. It can be seen as creating a surrogate "correctness" classifier and its associated calibration data. The calibrator is learned for the surrogate classifier and applied to the original classifier at inference time. Also, we add regularization for some scaling methods and we have only one binary calibrator instead of one per class.: ŷi ← arg max k∈Y f k (xi) ▷ Compute class predictions y b i ← 1 ŷi =y i ▷ Compute predictions correctness f b ← max k∈Y f k ▷ Create surrogate binary classifier D TvA cal ← {(xi, y b i )} N i=1 ▷ BuildComparison with IRM and I-Max IRM [72] and I-Max [49] are, like TvA, multiclass-to-binary reductions. This is why TvA cannot be applied on top of them: they already transform the multiclass problem into a binary one using a different strategy.The shared class-wise strategy of [49] and the data ensemble strategy of [72] are described very briefly in subsections 3.2 and 3.3.2 of their respective papers and not rigorously justified. Our understanding is that these two strategies do exactly the same thing. To build the calibration set, they concatenate all the class probability vectors so that we get a big probability vector of size N.L (N samples and L classes) as predictions and similarly concatenate the one-hot embedding of the target class (a big vector with N ones and N.(L -1) zeros) as targets. Then, they learn a single calibrator. For each example, this calibrator aims to simultaneously increase the probabilities for the target class (target is 1) and decrease all the other class probabilities (target is 0). The single calibrator is applied to each class probability separately, meaning that the ranking of class probabilities can change, modifying the classifier prediction.Our strategy derives from transforming the multiclass calibration into a single binary problem. The intuition is to learn the calibrator on a surrogate binary classifier and apply this calibrator to the original classifier. This binary classifier is built on top of the original classifier (by applying the max function to the class probabilities vector). They thus share their confidence. However, the binary classifier aims to solve a different task: predicting the correctness of the original classifier. To build the calibration set, we concatenate all the confidences (a vector of size N) as predictions and concatenate all the correctnesses as targets (also a vector of size N). The correctness value of a given example is 1 if the class prediction is correct; otherwise, it is 0. Then, we learn a single calibrator, similar to the strategy above. However, there is a key difference: this calibrator aims to increase the probabilities for correct predictions and decrease them for incorrect predictions. Note that our probabilities are all confidences (the maximum class probabilities), meaning we only consider the confidences, which the calibrator directly increases or decreases. In the strategy from [49] and [72] , the calibrator has to manage all class probabilities (L times more), even the ones that do not matter, including the lowest class probabilities close to 0. This is less efficient (actually, while this can surely be fixed, the original implementations of IRM and I-Max could not run on ImageNet-21K). This point is closely linked to the analysis of the binary cross entropy loss for scaling methods in Subsection 4.2: when the prediction is incorrect, increasing the probability of the correct class indirectly decreases the confidence (strategy from [49] and [72] ) while our strategy directly decreases the confidence. I-Max is more complex because it modifies the Histogram Binning algorithm, while our approach does not. Additionally, [35] found that I-Max produces unusable probability vectors. Indeed, they do not sum up to 1, and normalizing them degrades the method's performance.We wrote our paper with practicality and generality in mind. Contrary to [49] and [72] , we demonstrate the generality of our strategy by applying it on top of existing calibration baselines of different natures (scaling and binary). One of our main goals is that practitioners can easily and quickly try our TvA approach, using just a few lines of code, which can significantly improve the calibration performance of their existing calibration pipeline while having no impact on the predicted class by design (except for VS and DC).Comparison with ProCal Another recent calibration method is ProCal [68] . However, its primary objective differs from ours: it "focuses on the problem of proximity bias in model calibration, a phenomenon wherein deep models tend to be more overconfident on data of low proximity". Its goal is to lower the difference in the confidence score values between regions of low and high density, i.e., to make the confidence score independent of a local density indicator called "proximity." There is no theoretical guarantee, however, that minimizing the proximity bias improves the confidence calibration, the focus of our work. Theorem 4.2 about the PIECE metric is a direct consequence of Jensen's inequality and is true for any random variable D, not necessarily a proximity score.Theorem 5.1 is an interesting bias/variance decomposition of the Brier score. However, as this type of decomposition usually states, the error may come from bias (here, a wrong initial calibration) or high estimation variance (which can be related to low density but is not expressed as such in the decomposition). We experimentally compare our approach to the ProCal algorithm using the code provided by its authors and observe in Table 2 that our approach gives much better ECE confidence calibration and, for half of the models, also better PIECE values.ProCal aims to achieve three goals: mitigate proximity bias, improve confidence calibration, and provide a plug-and-play method. We share the last two goals. Concerning improving confidence calibration, our approach has better results, as shown in Table 2 . Both approaches are plug-and-play, but they apply very differently. ProCal is applied after existing calibration methods to further improve calibration. It thus does not solve any of the four issues we identified (e.g., cross-entropy loss is still inefficient, and One-versus-All still leads to highly imbalanced problems). Our Top-versus-All approach is a reformulation of the calibration problem that uses a surrogate binary classifier. Existing approaches are applied to this surrogate classifier, which is how the four issues are solved. We do not propose a new method but a new way of applying existing methods. Our approach does not introduce new hyperparameters (except in the particular case of regularizing scaling methods). ProCal introduces several new hyperparameters, such as the choice of the distance, the number of KNN neighbors, or a shrinkage coefficient.Comparison with Correctness-Aware Loss A concurrent work [38] builds a calibration method on top of an intuition similar to ours: binarize the calibration problem. However, what the authors do with this intuition differs vastly from our approach. They derive a Correctness-Aware Loss (Eq. 7 of their paper), which is almost the standard binary cross-entropy loss we use for scaling methods but without a logarithm. They use this loss to learn a separate model that predicts a sample-wise temperature coefficient. This is a new calibration method, which is not straightforward to implement due to the numerous hyperparameters (network architecture, image transformations...). It also requires multiple inferences at test-time, which can be problematic in some production models. Our approach is, again, not a calibration method but a general reformulation of the calibration problem that enhances existing methods. By looking at their Table 1 , they get an ECE of 2.22 on ImageNet (in-distribution), while our approach achieves values around 0.5 for most models in our paper's Table 1 . Their method, contrary to TvA, improves the AUROC, but in our understanding, it seems mostly due to the use of image transformations, not from their proposed loss. Their method seems to work best in out-of-distribution scenarios, which is not the main objective of our paper. However, these good results for AUROC and out-of-distribution scenarios make this method complementary to our approach, and combining the two in some way could be promising. We define L the number of classes, f k (x) the classifier estimated probability for class k and data sample x, y the correct class, and the confidence s(x) := max k f k (x). The cross-entropy loss isl CE (x, y) = - L k=1 1{k = y} • log(f k (x)) = -log(f y (x)).Because the last layer of the classifier is a softmax function, f y (x) = e zy k e z k with z the logits vector. Note that we omit the writing variable x in the following for clarity.Temperature scaling optimizes a coefficient T > 0 that scales the logits vector. Predicted probabilities become f y (x) = e zy /T k e z k /T . Let us first develop the standard cross-entropy loss when temperature scaling is applied:l CE = -log(f y ) = -log( e zy/T k e z k /T ) = -log(e zy/T ) -log( k e z k /T ) = - z y T +log( k e z k /T )Let us compute its gradient:∂l CE ∂T = z y T 2 + ∂ log( k e z k /T ) ∂ k e z k /T • ∂ k e z k /T∂T by application of the chain rule on the second term.EQUATIONFor our TvA approach, the problem becomes binary. The classification output becomes the confidence s(x) = max j∈Y f j (x) and the ground truth label becomes a binary representation of the prediction correctness: y b = 1 ŷ=y with ŷ(x) = arg max k∈Y f k (x) and 1 the indicator function. The loss we use is the binary cross entropyl BCE (x, y) = -y b • log s(x) + (1 -y b ) • log(1 -s(x))Let us compute the gradient:EQUATION-First case, the prediction is correct: y b = 1 and z m = z y . Let us inject these in (8):∂l BCE ∂T = 1 T 2 • (z y -k z k • f k ) = ∂l CE ∂T .We thus get the same gradient as the standard cross-entropy loss. -Second case, the prediction is incorrect: y b = 0 and z m > z y . (8) becomes:∂l BCE ∂T = 1 T 2 • s s-1 • (z m -k z k • f k ).By comparing to (7) , we have the term 1T 2 • (z m -k z k • f k ) > 1 T 2 (z y -k z k • f k ) = ∂l CE∂T and the remaining part of (8)| s s-1 | > 1 when s > 0.5. So to recapitulate, | ∂l BCE ∂T | > | ∂l CE ∂T | when s > 0.5, which corresponds to the vast majority of data points as the classifier gets better calibrated. This is shown in Figure 1 . We also have lim s→1 | s s-1 | = ∞. In practice, s is not close enough to 1 to generate exploding gradients, so it just means that as confidences for wrong predictions gets higher, so does the gradient to reduce the confidences.The conclusion is that for correct predictions, our approach does not change the optimization, but for incorrect predictions, the gradient is stronger and penalizes more heavily confident predictions that are wrong. This is also proven experimentally by looking at Table 8 where we see that average confidences of temperature scaling with the TvA approach (TS TvA ) are lower than the ones using the standard approach (TS), for almost all networks. This makes the average confidences closer to the accuracy, showing reduced overconfidence. Let us define the ECE for class j:ECE j = B b=1 n b N |acc(b, j) -conf(b, j)|The difference compared to (2) is that now acc(b, j) corresponds to the proportion of class j in the bin. Also, conf(b, j) now is the average probability given to class j for all samples in the bin. Then, classwise-ECE [29] takes the average for all classes:ECE cw = L j=1 ECE jClasswise-ECE considers the full probabilities vectors: all the class probabilities for each prediction. However, this metric does not scale to large numbers of classes. Let us see why with an example.Let us use a test set of N samples, N/L for each of the L classes (the dataset is balanced), and a high-accuracy classifier fairly calibrated. The classifier predicts N probability vectors of length L.Predicted probabilities for class j are all the values of the vector at dimension j. Because the classifier has a high accuracy and is fairly calibrated, around N/L values are close to 1 (corresponding to mostly correct predictions), and the remaining ones, around N -N/L, are close to 0 (because the predicted class is not class j, and the predicted probability is high for another class).To compute ECE j with equal size 15 bins, the predicted probabilities for class j are partitioned into 15 bins. The first bin (with probabilities close to 0), contains n 1 ≈ N -N/L samples while the last one (with probabilities close to 1) contains n B ≈ N/L samples. The remaining bins are usually even more empty. That means that the calibration error in the first bin is weighted n 1 /n B = L -1 times more than the last one. For the 1000 classes of ImageNet, L -1 = 999. Figure 4 shows the number of samples (n b ) in each bin for an ImageNet classifier.Because the impact of the calibration error in the bin with the high probabilities is negligible relative to the bin with the low probabilities, the classwise-ECE mostly measures whether probabilities close to 0 are calibrated. We argue this is not what we are interested in: what matters more is the calibration of higher values of the probabilities.Top-label ECE [18] is another interesting metric that does not scale to large numbers of classes either. Top-label-ECE divides data into subsets according to the predicted class, computes the ECEs of these subsets, and averages them. For an ImageNet test set of 25000 samples (25 per class), data is divided into 1000 subsets of ≈ 25 samples each (the classifier is high-accuracy, most of the time the predicted class is equal to the true class). The ECE is computed for each subset containing only 25 samples.To compute the ECE, samples are typically partitioned into 15 bins. The number of samples per bin does not allow a correct estimation of the average confidence or accuracy.• Model weights for CIFAR are from [45] .• Model weights for ImageNet come from torchvision [42] .• Model weights for ImageNet-21K are from [54] .• CLIP weights are from OpenAI's Hugging Face.• Original weights for T5 and RoBERTa come from the Transformers library [67] . The models are fine-tuned for each task using prompt-based learning [36] . For more details, see [5] .• We used the model GPT-J https://huggingface.co/EleutherAI/gpt-j-6b (Apache-2.0 license) and Llama-2 https://huggingface.co/meta-llama/Llama-2-13b (license).• CIFAR-10 (C10) and CIFAR-100 (C100) [27] contain 60000 32x32 images corresponding to 10 and 100 classes, respectively. Data is split into subsets of 45000/5000/10000 images for train/validation/test. We concatenate the original validation and test sets, and randomly split that into a calibration set of size 5000, and a test set of size 10000.• ImageNet (IN) [7] contains 1.3 million images from 1000 classes. Following [17] , we randomly split the original validation test of size 50000 into a calibration set and a test set, both of size 25000.• ImageNet-21K (IN21K) [54] , in its winter21 version, contains 11 million images in the train set, and 522500 in the test set (50 for each of the 10450 classes). We randomly split the test set into equal-sized calibration and test set (261250 samples each, 25 per class).• Amazon Fine Foods [43] is a collection of customer reviews for fine foods sold on Amazon.Reviews are categorized into bad, neutral, and good. The original validation set size is 78741 and test size 91606. We randomly split them into 78741 samples for calibration and 91606 for test.• DynaSent [51] is a dynamic benchmark for sentiment analysis consisting of sentences annotated as positive, neutral, and negative. The original validation set size is 11160 and test size 4320. We randomly split them into 11160 samples for calibration and 4320 for test.• MNLI [66] contains pairs of sentences labeled as contradiction, neutral, and entailment. The original validation set size is 19635 and test size 9815. We randomly split them into 19635 samples for calibration and 9815 for test.• Yahoo Answers (YA) [73] contains question-answers pairs corresponding to 10 different topics. The original validation set size is 14000 and test size 60000. We randomly split them into 14000 samples for calibration and 60000 for test.• TREC [61] contains questions categorized into 6 classes. The training set contains 5500 labeled questions, and the test set contains another 500.• SST-5 [55] contains 11855 sentences corresponding to 5 sentiments (from very negative to very positive).• DBpedia [74] contains text for topic classification with 14 classes. The training set contains 560000 samples, and the test set 5000.• We used the library netcal [32] (Apache-2.0 license) for their implementation of binary methods for calibration and adapted their reliability diagrams code. For HB, we tested equalsize and equal-mass bins, and chose the best variant for each case. All hyperparameters were kept at their default values (10 bins for HB).• We took inspiration from the official implementation of temperature scaling: https:// github.com/gpleiss/temperature_scaling (MIT license).• We took inspiration from the official implementation of Dirichlet calibration: https: //github.com/dirichletcal/experiments_dnn (MIT license). • We used the official implementation of I-Max: https://github.com/boschresearch/ imax-calibration (AGPL-3.0 license). • We used the official implementation of IRM: https://github.com/zhang64-llnl/ Mix-n-Match-Calibration (MIT license). • For evaluation, we used codes from https://github.com/JeremyNixon/ uncertainty-metrics-1 (Apache-2.0 license) and https://github.com/IdoGalil/ benchmarking-uncertainty-estimation-performance (MIT license). • We used PyTorch 2.0.0 [3] (BSD-style license).• We used CIFAR models from [45] https://github.com/torrvision/focal_ calibration (MIT license). • We used ImageNet-21K models [54] https://github.com/Alibaba-MIIL/ ImageNet21K (MIT license). • We used CLIP models from HuggingFace's Transformers library [67] (Apache-2.0 license).• We used pretrained language models from Transformers [67] and calibration codes from [5] https://github.com/lifan-yuan/PLMCalibration (MIT license). • We used code from https://github.com/mominabbass/LinC for the calibration of LLMs using ICL, itself built upon https://github.com/tonyzhaozh/ few-shot-learning (Apache-2.0 license). Selective classification aims to improve a model's prediction performance by trading-off coverage: a reject option allows to discard data that might result in wrong predictions, thus improving the accuracy on the remaining data. A strong standard baseline uses thresholding on the maximum softmax probability outputted by the classifier [13] . Improving confidence calibration means uncertainty is better quantified and should result in better selective classification.Results in Table 1 show the superiority of Histogram Binning (applied with the right framework) in reducing the calibration error ECE. Unfortunately, it does not translate into improvements in selective classification. AUROC is a standard metric for selective classification [11] . Table 4 shows that Histogram Binning actually degrades the AUROC, while the best method is Isotonic Regression. Our TvA framework does not significantly impact the AUROC.This paper addresses confidence calibration, usually measured by ECE. AUROC is a global rankbased metric for selective classification: it relies on the relative values of the scores, not their absolute values. Even though calibration and selective classification are related, improvement in calibration does not directly translate to better selective classification. This has been clearly demonstrated experimentally by [11] .A good example of that difference is the behavior of HB TvA : it is the best calibration method overall but actually degrades the AUROC in most cases. Such a difference can be explained by the fact that selective classification benefits from a continuous score able to discriminate between certain and uncertain examples finely, but HB quantizes the confidences into, e.g., 10 different values. Large Language Models (LLMs) exhibit an in-context learning (ICL) capability, meaning they can learn from just a few examples in the context. It works by constructing a prompt that includes input-output pairs demonstrating the considered task, followed by a query for a new input. See [8] for a survey. Recent works develop calibration methods whose main goal is to improve the performance of ICL for LLMs, without requiring a complicated model fine-tuning. [77] uses a customized variant of Platt scaling (more specifically, Vector Scaling). Their method infers good values of the vector scaling parameters in a data-free procedure. The idea is that for a "content-free" input, e.g., "N/A", the calibrated probability has a 50% chance (for a binary classification task) of removing a bias toward the positive or negative class. In our paper, we denote this method as ConC. [1] builds on top of this work but uses a calibration set to learn the scaling parameters by minimizing the cross-entropy loss. This can be considered as Matrix Scaling. We denote this method as LinC. [78] proposes a per-class normalization of the probabilities on a given batch. [25] estimates the in-context model label marginal p(y) from limited data and uses it to calibrate the model probabilities. Paper [20] uses a Gaussian mixture model.In our experiments, we have tested a two-step calibration. First, we use the state-of-the-art method LinC to maximize the accuracy by learning scaling parameters on a calibration set. Then, we apply HB TvA to scale the confidences to lower the calibration error ECE, while preserving the accuracy gains. We use the same calibration set for the two methods. LinC performance depends on hyperparameter values, but to keep the experiments simple, we fixed the following values: 100 epochs, a learning rate of 0.001, and 300 calibration samples. It means that the reported performance of LinC is suboptimal and could be enhanced even more. We used the same experimental setting as [1] . We used the models GPT-J with 6B parameters [62] and Llama-2 with 13B parameters [59] . The text classification datasets are TREC [61] for question classification with 6 classes, SST-5 [55] for sentiment analysis with 5 classes, and DBpedia [74] for topic classification with 14 classes. The 0-shot, 1-shot, 4-shot, and 8-shot learning settings were tested. Five different sets of 300 test samples were randomly selected, and results are averaged over 5 seeds. We evaluated the accuracy and ECE for each configuration. Please see Table 12 for the results. In most cases, LinC+HB TvA achieves the best accuracy and ECE.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: Our main claims in the abstract and introduction are that our approach solves the failure of many calibration methods when applied to problems with many classes and that we conduct extensive experiments. This reflects our analyses in Sections 3 and 4 and experimental results in Section 5 and Appendices G and H. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: The research conducted in the paper does conform with the NeurIPS Code of Ethics and we preserve anonymity.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper lacks strong theoretical contributions and there is some concern about novelty.  However, its simplicity, practicality, and  experimental validation could make it a valuable contribution to the field of confidence calibration, particularly when dealing with a large number of classes.  The paper had many reviews (with some of them conflicting) and I have to discard some of the negative reviews on novelty due to the fact that there was no concrete justification by the reviewers.  I find that the paper is borderline accept as a poster.

Reviews: review: paper_summary: paper_summary: This paper proposes a new learning objective for model calibration in multi-class classification tasks. The authors analyze the issues with the current softmax-based scaling method and argue that it only considers the confidence (the probability of the predicted class) without accounting for other remaining classes during calibration. The proposed Top-versus-All method divides the calibration data into class-wise datasets. Experiments on various models, tasks, and calibration baseline methods have shown some effectiveness of the proposed approach.
strength_weakness: strength_weakness: 1. This paper calibrates a surrogate binary classifier that predicts whether the class prediction is correct. However, according to the results in Table 3, in the majority of experimental results, the AUROC is worse when using the proposed method. The AUROC is not improved, meaning that the calibrator has not improved its ability to effectively distinguish between correct and incorrect model predictions, thus failing to learn to predict whether the class prediction is correct. This is contradictory to the paper's main objective and narrative. 

2. The authors answered YES for Theory Assumptions and Proofs in the checklist. However, no proofs are shown. Some mathematical calculations are presented in Appendix D, but those are not proofs. They overclaim the theoretical contribution.

3. The evaluation metrics are limited. Only quantitative results on ECE and AUROC are reported (even though the AUROC is not improved). Other popular metrics such as ACE, MCE, or PIECE are not provided.

4. Authors report the results on CLIP models but do not discuss or compare them to recent calibration methods for CLIP models. For example:
    
   [1] Wang, S., Wang, J., Wang, G., Zhang, B., Zhou, K. and Wei, H., 2024. Open-Vocabulary Calibration for Vision-Language Models. International Conference on Machine Learning (ICML), 2024.


5. There are several existing model calibration papers in the literature that focus on directly estimating the probability of correctness for each sample/prediction.

     [2] Xiong, M., Deng, A., Koh, P.W.W., Wu, J., Li, S., Xu, J. and Hooi, B., 2023. Proximity-informed calibration for deep neural networks. Advances in Neural Information Processing Systems, 36, pp.68511-68538.

     [3] Liu, Y., Wang, L., Zou, Y., Zou, J. and Zheng, L., 2024. Optimizing Calibration by Gaining Aware of Prediction Correctness. arXiv preprint arXiv:2404.13016.

    For example, in paper [3], the idea is almost the same with minor differences. It also learns to predict the probability of correctness of the model's predictions as confidence. However, it can distinguish between correct and incorrect predictions based on their AUROC results.
questions: questions: 1. The results in Table 3 are not aligned with the paper's narrative. Why not discuss the mechanism or reason behind this? I think it is a vital issue.

2. Can the authors please provide strong reasons to convince me that your method is superior to similar works [2] and [3]?
limitations: limitations: Yes. They have a limitation section.
paper_summary: paper_summary: The paper proposes a method to improve confidence calibration in neural network-based classification models. It transforms the multiclass calibration problem into a binary classification surrogate, demonstrating enhanced performance across various neural network applications in image and text classification.
strength_weakness: strength_weakness: 1. The paper lacks rigorous theoretical evidence to substantiate its claims, which weakens the strength of the proposed solutions. Without detailed proofs, the effectiveness of the Top-versus-All approach and other methods in improving confidence calibration remains somewhat speculative.

2. While the paper claims scalability to LLMs, the experimental evaluation primarily focuses on smaller-scale models. This limited scope raises questions about the method's performance and applicability to state-of-the-art LLMs like LLaMA or Gemma. Extending experiments to these latest models would provide more convincing evidence of scalability and effectiveness across a broader range of neural architectures.

3. Figure 3 in the paper indicates a notable variance in the ECE test results. This variability suggests inconsistent performance across different settings or datasets, potentially indicating limitations in the method's stability and reliability under varying conditions.
questions: questions: 1. how does the method apply to CLIP? On the contrastive softmax loss? What if the number of classes is varying during training CLIP? I.e., the batch size is varying.
limitations: limitations: Limitations are well discussed in the paper
paper_summary: paper_summary: The paper relates to post-hoc confidence calibration i.e. the calibration of
the top-prediction of a classifier such that ``when the class l is predicted
with confidence q, the probability of the actual class being l is also q''.
The calibration does not update the classifier.

The paper tackles two issues of past calibration methods:
- The one vs all approaches that that do not scale well with the number of
  classes (there are as many binary calibrators as number of classes)
- Each binary calibrator is tuned on datasets which imbalance increases with
  the number of classes: the positives are the instances of class $k$ and the
  negatives are all the other instances
- The overfitting of scaling methods due to their parametric nature (the number
  of parameter increases with the number of classes)

To do so, it proposes an alternative formulation of the calibration problem:
instead of calibrating the top-prediction directly, the paper calibrates an
intermediate binary classifier that estimates whether the top-prediction is
correct or not.

This addresses the two previous issues as:
- There is only one calibration optimization that calibrates the single binary
  classifier described above, which is independent of the number of classes
- The imbalance remains but is reduced and linked to the classifier's accuracy
- The calibration of the binary classifier with scaling methods requires less
  parameters and is hence less prone to overfitting

The proposed calibration can be optimized with off-the-shelf binary calibrators
and the experiments compare the proposed calibration against the traditional
calibrations, i.e., calibrating the classifier's top prediction directly. 

The experiments compare the proposed formulation that reduce the ECE compared
to the traditional calibrations.
The proposed formulation usually reduce the ECE compared to the traditional
calibration and improves over IRM and I-MAX.
The improvement is more notable on datasets with a higher number of classes.
strength_weakness: strength_weakness: W1. Not really a weakness but a consequence of the design choice: the proposed method keeps the classifier untouched so that the calibration does not impact the accuracy.
However, improving the confidence of incorrect predictions could help improving the classification accuracy by re-distributing the probabilities. 
In practice though (Tab.8), the accuracy varies only by a few points.
questions: questions: Q1. One of the claimed advantage of the proposed method (TvA) is the "stronger
  gradient" of TvA compared to that of the default scaling methods (understand
  the gradient's magnitude is higher).
  However, it is not clear how this is an advantage on its own as i) the impact of the
  gradient's magnitude is mitigated by the choice of learning rate; ii) a
  higher gradient magnitude does not imply a better optimization (this is
  specific to each optimization problem).
  Can more details be given on why `stronger gradients' are an advantage?
limitations: limitations: - The advantage of 'preserving' the accuracy is limited by the fact that existing
  calibration method that affect the accuracy usually induce a variation of ~1%
  only, and can even affect positively the accuracy i.e. increase it.
  This leads to the question of whether 'preserving of the accuracy' is really a
  limitation for other methods.

- Issue 1 describes the limitation of using cross entropy loss during
  calibration as it increases the probability of the true class which only
  **indirectly** impacts the probability of the top-prediction i.e. the
  confidence.
  L: 173 describes this optimization as 'inefficient' compared to the proposed
  one, however it is hard to quantify the 'efficiency' of an optimization so a
  better word might be 'indirect'.

Misc.:
- Eq6: v is undefined, does it refer to the logit probability vector?
- lack of legends for the blue and red bars (which is accuracy, which is
  confidence) even though one can understand
- Legend for Global acc. and avg. confidence not very visible
paper_summary: paper_summary: This paper presents a top-versus-all (TvA) approach for the confidence calibration of classifiers with many classes. The authors categorizes the post-processing calibration methods into two groups: scaling and binary methods, and lists several problems with these approaches. The authors reformulate the problem of calibrating a multiclass classifier into a single binary classifier, where the confidence score represents the maximum class probability of the binary problem. They show the effectiveness of the proposed method on numerous neural networks used for image or text classification.
strength_weakness: strength_weakness: 1) The theoretical contribution of the TvA approach and its impact on calibration could be further strengthened.
2) The propose method may exhibit limited novelty, as it builds upon established approaches rather than introducing entirely novel elements.
2) The calibration of CLIP appears to be better than other models on ImageNet, can you explain why? Additionally, it seems like most calibration methods don't work on CLIP, which confuses me.
questions: questions: Please see my comments in Weaknesses.
limitations: limitations: The authors have addressed the limitations in the main text.
paper_summary: paper_summary: The paper addresses the issue of miscalibrated confidence scores in neural network classifiers, particularly for problems with many classes. Traditional methods often fail in these scenarios. The authors propose transforming the multiclass calibration problem into a single binary classification problem, termed Top-versus-All (TvA), allowing the use of standard calibration methods more efficiently. This approach improves the performance of calibration methods like Temperature Scaling and Vector Scaling by better utilizing the calibration data and reducing overfitting. Experiments on various datasets and models demonstrate the scalability and effectiveness of this method.
strength_weakness: strength_weakness: 1. Lack of novelty. I read a paper very similar to the idea previously, make top vs all, but I am sorry that I cannot find the link. And as I remember, the Multiclass to Binary idea has been used in this literature multiple times.

2. Lack of the comparison with some SOTA post hoc methods such as [1] , the performance may not succeed.

3. Lack of comparison with some SOTA train time methods, such as [2].

4. Overall, the method lack the analysis and comparison with other SOTA methods.

[1] Proximity-Informed Calibration for Deep Neural Networks
[2] Dual focal loss for calibration
questions: questions: see Weaknesses
limitations: limitations: see Weaknesses
paper_summary: paper_summary: The paper proposes a confidence calibration method.  It reformulates a multi-class problem as the binary task "is the prediction correct?".  Then a given calibration method  (e.g. Temperature Scaling) is applied to that binary task.
strength_weakness: strength_weakness: The main justification of a calibration model is its improved results (usually measured by ECE).   
In the case of TS the paper reports a minor improvement. However,  in the case of HB, the paper reports a huge calibration improvement.  It is not clear what is the reason for the improvement difference.

As far as I understand the HB method, we first split the validation samples  to bins according to the confidence.    


Although the proposed idea is very very simple,  the  method presentation is overcomplicated and there is  an over-seling of the method.
questions: questions: ggg
limitations: limitations: ggg

Comments: title: Answer to Reviewer QaKm
Answer to Reviewer oe96
Answer to the weaknesses
comment: Thanks for your response, I'm keeping my score in view of the limited novelty of the method.
The rebuttal addressed all the comments in the review, thank you.

The comment on the confidence of the review is noted but the score will remain as is as it reflects that I am not an expert in the area so it is hard to assess the contribution against previous works (except for what is described in the related work of course).
Thank you for clarifying your point. For your curiosity, here is what we believe to be the reason. TS is already data efficient as only a single parameter is learned. It only needs a few calibration data to converge to its optimal value but does not benefit from having more data (which is seen in Figure 3 of the paper). By applying our approach to TS, we improve its performance. Our approach changes how calibration methods are applied but does not change the methods themselves. The performance of TS is still limited by the fact that only a single parameter is learned.
We are glad that we successfully addressed your concern about performance. However, we regret you find the novelty trivial. The main intuition is indeed trivial, but we believe our approach's motivation and justification are clear. Also, we consider this triviality as a strength because it allows the approach to be straightforward and easy to implement, as you mentioned.

We are sincerely grateful that you engaged in the discussion and raised your score.
TS is the most standard calibration method and the fact that applying the proposed method to TS does not result in a clear improvement is a weakness. Hence, my score remains 4.
I have read the comments from other reviewers, the author has addressed my concern on performance by providing the comparison with other works. But I think the novelty is still trival. I will raise my score to Reject instead of strong reject.
**Weaknesses**

> 1. [...] The AUROC is not improved [...]

We address this point in our response to the related question; please see the rebuttal above.

> 2. The authors answered YES for Theory Assumptions and Proofs in the checklist. [...]

Please see our global answer, which addresses this weakness.
We do not believe that we are making a significant theoretical contribution to the field. If there is any way that our work can be interpreted otherwise, we would appreciate being informed. We have misinterpreted the checklist and will answer NA for Theory Assumptions and Proofs to address potential confusion.

> 3. The evaluation metrics are limited. [...]

Besides ECE and AUROC we also include ECE with equal-mass bins (also sometimes called ACE), Brier score, and display qualitative results with reliability diagrams. Also, please note that what we call ECE with equal-mass bins is actually called ACE in some works. ACE uses equal-mass bins and is computed either classwise (as in the original definition) or not (as in [2] by looking at their code). To remove ambiguity, we call it ECE with equal-mass bins. As for the classwise version of ACE, it is not estimated well when the number of classes is high. We discussed this issue in Appendix E. Concerning MCE and PIECE, we will add the results in the Appendix of the final version of the paper. In the meantime, they are included in the attached PDF.

> 4. [...] recent calibration methods for CLIP models.

We were not aware of [1], which is not yet published; we thank you for providing the reference. We used CLIP as a zero-shot classifier and considered it as any other classifier. Our method is built with generality in mind, while [1] applies specifically to Vision-Language Models. We plan to discuss this approach and compare it to ours for CLIP in the final version, given that the code is available. 

> 5. [...] existing model calibration papers [2] [3] [...]

We address this concern in our answer to the related question in the rebuttal above.