Does confidence calibration improve conformal prediction?
Authors: HuaJun Xi
Jianguo Huang
Kangdao Liu
Lei Feng
Hongxin Wei
Venue: Accepted by TMLR ()

Abstract: Conformal prediction is an emerging technique for uncertainty quantification that constructs prediction sets guaranteed to contain the true label with a predefined probability. Previous works often employ temperature scaling to calibrate classifiers, assuming that confidence calibration benefits conformal prediction. However, the specific impact of confidence calibration on conformal prediction remains underexplored. In this work, we make two key discoveries about the impact of confidence calibration methods on adaptive conformal prediction. Firstly, we empirically show that current confidence calibration methods (e.g., temperature scaling) typically lead to larger prediction sets in adaptive conformal prediction. Secondly, by investigating the role of temperature value, we observe that high-confidence predictions can enhance the efficiency of adaptive conformal prediction. Theoretically, we prove that predictions with higher confidence result in smaller prediction sets on expectation. This finding implies that the rescaling parameters in these calibration methods, when optimized with cross-entropy loss, might counteract the goal of generating efficient prediction sets. To address this issue, we propose \textbf{Conformal Temperature Scaling} (ConfTS), a variant of temperature scaling with a novel loss function designed to enhance the efficiency of prediction sets. This approach can be extended to optimize the parameters of other post-hoc methods of confidence calibration. Extensive experiments demonstrate that our method improves existing adaptive conformal prediction methods in both image and text classification tasks.

Published in Transactions on Machine Learning Research (06/2025)
Does confidence calibration improve conformal prediction?
Huajun Xi
1B111806@@maii.sssscccc........
Deportment of Statistics and Data Science Southern University of Science and Technology
Jianguo Huang
Sianguo.haanggnita
College of Computing and Data Science Nangang Technological University
Kangdao Liu
kangdaoliu@gmaailcoom
Deportment of Computer and Information Science Universily of Macou
Lei Feng
feng leidBould edu sg
Information Systems Technology and Design Pillar Singapore University of Technology and Design
Hongxin Wei 1
weilusssutecch.c edu.cn
Department of Statistics and Data Science
Southern University of Science ond Technology
Reviewed on OpenReview: https: // wpw.revee.........eoeee net/ por/forum? id- 6DDoTuTvdE
Abstract
Conformal prediction is an emerging technique for uncertainty quantification that constructs prediction sets guarantced to contain the true label with predefined probability. Previous works often employ lemperature scaling to calibrate classifiers, assuming that confidence cali- bration benefits conformal prediction. However, the specific impact of confidence calibration on conformal prediction remains underexplored. In this work, we make two koy discoveries about the impact of confidence calibration methods on adaptive conformal prediction. Firstly, we empirically show that current confidence calibration methods (e-g., temperature scaling) typically lead to larger prediction sets with lower confidence in adaptive conformal prediction. Secondly, by investigating the role of temperature value, observe that high-confidence predictions produced by a low temperature lead to small prediction sets for adaptive con- formal prediction. Thcorctically, we prove that higher-confidence predictions with lower temperatures result in smaller prediction sets on expectation. This finding implies that the rescaling parameters in these calibration methods, when optimized with cross-entropy loss, might counteract the goal of generating small prediction sets. To address this issue, we propose Conformal Temperature Scaling (ConfTS), variant of temperature scaling with novel loss function designed lo enhance the elficiency of prediction sets. This approach can be extended to optimize the parameters of other post-hoc methods of confidence calibration. Extensive experiments demonstrate that our method improves existing adaptive conformal prediction methods in both image and text classification tasks.
1 Introduction
Ensuring the reliability of model predictions is crucial for the safe deployment of machine learning such as autonomous driving (ooarski et al. 2016) and medical diagnostics CCruanaa et al. 2015). Numerous methods
"Equal Contribution
i "Correspond veihxtsustech.

---

Published in Transactions on Machine Learning Research (06/2025)
have been developed to estimate uncertainty and incorporate it into predictive models, including confidence calibration (Guo et al. 2017) and Bayesian neural networks (Smith 2013). However, these approaches do not. provide formal theoretical guarantees for the reliability of model predictions. In contrast, conformal prediction offers a systematic approach to construct prediction sets that are thcoretically guaranteed to contain the true label with desired probability (Vovk et al. 2005 Shafer & Vovk, 2008) Balasubramanian el al. 2014 Angelopoulos & Bates 2021). This framework thus provides trustworthiness in real-world scenarios where wrong predictions are dangerous.
In the literature, conformal prediction is frequently associated with confidence calibration, which expects the model to predict softmax probabilities that faithfully estimate the true correctness (Wei et al. 2022]
Yuksekgonul et al. 2023 Wang 2023 Wang et al.
2024). For example, existing conformal prediction methods
usually employ temperature scaling (Guo et al. 2017]. a post-hoc method that rescales the logits with a
scalar temperature, for a better calibration performance (Angelopoulos al.) 2021 Lu et al.] 2022 2023
Gibbs et al., 2023). The undorlying hypothesis that well-calibrated models could yield procise probability estimates, thus enhancing the reliability of generated prediction sets. However, the rigorous impacts of current confidence calibration techniques conformal prediction remain ambignous in the literature, which motivates our analysis of the connection between conformal prediction and confidence calibration.
In this paper, we empirically show that existing methods of confidence calibration increase the size of prediction sets generated by adaptive conformal prediction methods (this effect does not apply to non- adaptive conformal methods such as LAC (Sadinle et al. 2019)). Moreover, we find that predictions with high confidlecee (rescaled with small temperature value) tend to produce efficient prediction sets while maintaining the desired coverage guarantees. However, simply adopting extremely small temperature value may result in meaningless prediction sets, as some tail probabilities can be truncated to zero due to the finite- precisspeeiiinn issue. Theoretically, we prove that smaller temperature value leads to larger non-conformity scores, resulting in more efficient prediction sets on expectation. This highlights that rescaling parameters of post-hoc calibration mothods, optimized by the cross-entropy loss, might counteract the goal of generating efficient prediction sets.
To validate our theoretical findings, we propose a variant of temperature scaling, Conformal Temperature Scaling (ConfTS), which rectifies the optimization objective through the efficiency gap, i.c., the deviation between the threshold and the non-conformily score of the ground iruth. In particular, ConfTS optimizes the temperature value by minimizing the efficiency gap. This approach can be extended to optimize the parameters of other post-hoe methods of confidence calibration, e.g., vector scaling and Platt scaling. Extensive experiments show that ConfTS can effectively enhance the efficiency of existing adaptive conformal prediction techniques, APS (Romano et al., (020) and RAPS (Angelopoulos et al., (021]. Notably, we empirically show that post-hoe calibration modhods optimized by our loss function can also improve the efficiency of prediction sets in both image and text classification (including large language models), which demonstrates the generality of our method. In addition, we provide an ablation study of loss functions to show that the proposed loss function can outperform the ConfTr loss (Stutz et al. 2022p. In practice, our approach is straightforward to implement within deep learning frameworks, requiring no hyperparameter tuning and additional computational costs compared to standard temperature scaling.
We summarize our contributions as follows:
We discover that current confidence calibration methods typically lead to larger prediction sets adaptive conformal prediction, while high-confidence predictions (using small temperatures) can enhance the efficiency of prediction sets. We further identify practical limitation where extremely small temperature values cause numerical precision issues.
We provide a theoretical analysis by proving that applying smaller temperature values in temperature scaling results in more efficient prediction sets on expectation. This theoretical insight explains the relationship between confidence calibration and conformal prediction.
We validate our theoretical findings by developing Conformal Temperature Scaling (ConfTS), a variant of temperature scaling that exploits the relationship between temperature and set efficiency.
2

---

Published in Transactions on Machine Learning Research (06/2025)
Extensive experiments demonstrate that ConfTS enhances the efficiency of prediction sets in adaptive conformal prediction and can be extended to other post-hoc methods of confidence calibration.
2 Preliminary
In this work, we consider the multi-class classification task with K classes. Let X' C Rd be the input space and Y' = 21, }} } be the label space. We represent a pre-trained classification model by f x' + R* Let (X ~ Pxy denote a random data pair sampled from a joint data distribution Pxy: and fy(Ø) denote the y-th element of logits vector f(x) with an instance æ. Normally, the conditional probability of class y is approximated by the softmax probability output (2)) defined as:
evlae//
P(Y = y/ 2) my = ( Σ efc(w//t
where is the softmax function and denotes the temperature parameter (Guo et al. 2017). The temperature softens the output probability with / >>> and sharpens the probability with < 1. After training the model, the temperature can be tuned on a held-out validation set by optimization methods.
Conformal prediction. To provide theoretical guarantees for model prodictions, conformal prediction (Vovk et al., 2005) is designated for producing prediction sets that contain ground-truth labels with desired probability rather than predicting one-hot labels. In particular, the goal of conformal prediction is to construct set-valued mapping C: X + 2 that satisfies the marginal coverage:
P(Y E ((X))21-1
(2)
where a E (0,1) 1) denotes a user-specified error rate, and C(x) C V is the generated prediction set. particular, the probability is with respect to the randomness of data sample (X, In the following, we will use 'coverage" to represent 'marginal coverage' for convenience.
Before deployment, conformal prediction begins with a calibration step, using held-out calibration set Dent := [(z,, //) We calculate the conformfory score = S(x;, yi) for each example (æ,, Vi), where 8; is measure of deviation between an example and the training data, which we will specify later. Then, we determine the - a quantile of the non-conformity scores as a threshold:
= inf { [í: S(x, vi) S s71 [n + 1)(1 a)
(3)
n
n
For a test instance In+1, we first calculate the non-conformity score for each label in V, and then construct the prediction set C(#n41) by including labels whose non-conformity score falls within r:
C(#n+1) tu E y S(x $ r)
(4)
Notably, small prediction sets are often preferred. As demonstrated in previous work (Cresswell et al.), the reduction in the prediction set size has practical significance, as smaller prediction sets are more informative to enable accurate human decision making. In the following, we introduce the term 'efficiency' to compare conformal prediction methods: a method is more efficient when it produces smaller prediction sets.
In this paper, we focus on adaptive conformal prediction methods, which are designed to improve the adaptiveness of prediction set, which requires prediction sets to communicate instance-wise uncertainty [Romano et al. 2020). However, they usually sulfer from inefficiency in practice: these methods commonly produce large prediction sets (Angelopoulos et al. 2021). In particular, we take the two representative methods: APS [Romano et al. 2020) and RAPS (Angelopoulos et al. 2021).
Adaptive Prediction Set (APS). (Romano et al.) 2020) In the APS method, the non-conformity score of data pair (xr,)) y) is calculated by accumulating the sorted softmax probability, defined as:
SAPs(æ, y) (()) + u (x()))(=))
(5)
3

---

Published in Transactions on Machine Learning Research (06/2025)
where (z)) (2)())) are the sorted softmax probabilities in descending order, and o(y, xx))
denotes the order of (y))), Le., the softmax probability for the ground-truth label y. In addition, the term u. is an independent random variable that follows a uniform distribution on [0,
Regularized Adaptive Prediction Set (RAPS). (Angelopoulos et al., 2021) The non-conformity
score function of RAPS encourages a small set size by adding a penalty, as formally defined below:
SRAPS(æ, (x) + + (o(g. + À ((() kireg)*
(6)
where (z)+ = max 2), kreg controls the number of penalized classes, and À is the penalty term.
Notably, both methods incorporate uniform random variable u to achieve exact 1-a a coverage (Angelopoulos et al. 2021). Moreover, use coverage and overage size to evaluate the prediction sets. A detailed description of the metrics is provided in Appendix A
3 Motivation
3.1 Adaptive conformal prediction with calibrated prediction
Confidence calibration Guo et al., 2017) expects the model to predict softmax probabilities that faithfully estimate the true correctness: Vp E [0, 1J, PP(Y- (x) =} = p. To quantify the degree of miscalibration, the Expected Calibration Error (ECE) is defined as the difference between accuracy and confidence. With N samples grouped into K bins {b),-.. bk], the ECE calculated as:
lbul N
ECE Σ
(acc(bu) conf(bb)]
where ace(-) and conf(.) denotes the average accuracy and confidence in bin by-
In conformal prediction, previous work claims that deep learning models are often badly miscalibrated, leading to large prediction sets that do not faithfully articulate the uncertainty of the model (Angelopoulos et al., 2021). To address the issue, researchers usually employ temperature scaling (Guo et al.) 2017] to process the model outputs for better calibration performance. However, the precise impacts of current confidence calibration techniques adaptive conformal prediction remain unexplored, which motivates our investigation into this connection.
To figure out the correlation between confidence calibration and adaptive conformal prediction, we incorporate various confidence calibration mothods to adaptive conformal predictors for a ResNet50 model (He et al.) 2016) on CIFAR-100 dataset (Krizhevsky et al. 2009). Specifically, we use six calibration methods, including four post-hoc methods vector scaling (Guo al. 2017), Platt scaling (Platt et al. 1999), temperature scaling (Guo et al.] 2017).
Bayesian methods (Daxberger et al.) 2021), and two training methods label
smoothing (Szegedy et al. 2016 mirup (Zhang al. 2018 More details of calibration methods and setups are presented in Appendix
B and Appendix
Confidence calibration methods deteriorate the efficiency of adaptive conformal prediction. In Table [ we present the performance of confidence calibration and conformal prediction using APS and RAPS with various calibration methods for a ResNet50 model. The results show that the influences of those calibration methods are consistent: models calibrated by both post-hoc and training calibration techniques generate large prediction sets with lower ECE (i.e., better calibration). For example, the ImageNet dataset, temperature scaling enlarges the average size of prediction sets of APS from 9.06 to 12.1, while decreasing the ECE from 3.69% to 2.24%. This finding demonstrates an inverse relationship between calibration performance and prediction set efficiency. In addition, incorporating calibration methods into conformal prediction does not violate the 1 1 a marginal coverage as the assumption of data exchangeability is still satisfied: we use a hold-out validation dataset for conducting confidence calibration methods. In addition, we present the results of the LAC score (Sadinle et al. 2019) in Appendix D.1 where we observe clear correlation between confidence calibration methods and conformal prediction.
4

---

Published in Transactions on Machine Learning Research (06/2025)
Table 1: The performance of APS and RAPS on CIFAR-100 dataset with ResNet50 model, using various calibration methods. In particular, we apply label smoothing (LS), Mixup (Mixup), Bayesian methods (Bayesian); vector scaling (VS), Platt scaling (PS), and temperature scaling (TS). We do not employ calibration techniques in the bascline (Base). We repeat each experiment. for 20 times. "4" indicates smaller values are better. and " indicate whether the performance is superior superior/inferior to the baseline. The results show that existing confidence calibration methods deteriorate the efficiency of APS and RAPS.
Method
Base
I.S
Mixup
Bayesian
TS
PS
VS
Accuracy
0.77
0.78
0.78
0.77
0.77
0.77
0.77
ECE
8.79
4.39
2.96 A
4.30
3.62
3.81
4.06
Coverage
0.90
0.90
0.90
0.90
0.90
0.90
0.90
APS
Avg.size + 4.91
11.9
12.5
7.55
6.69
7.75
7.35
Coverage
0.90
0.90
0.90
0.90
0.90
0.90
0.90
RAPS
Avg.size +
2.56
9.50
10.2
6.16
3.58
3.72
3.85
Coverage
0.95
0.95
0.95
0.95
0.95
0.95
0.95
0.05
APS
Avg.size +
11.1
19.8
20.1
15.6
12.8
13.9
11.3
II
Coverage
0.95
0.95
0.95
0.95
0.95
0.95
0.95
RAPS
Avg.size +
6.95
14.5
15.5
9.34
10.4
11.0
8.70
Overall, we empirically show that current confidence calibration methods negatively impact the efficiency of prediction sets, challenging the conventional practice of employing temperature scaling in adaptive conformal prediction. While confidence calibration methods are primarily designed to address overconfidence, we conjecture that high confidence may enhance prediction sets in efficiency.
3.2 Adaptive conformal prediction with high-confidence prediction
In this section, we investigate how the high-confidence prediction influences the adaptive conformal prediction. In particular, we employ temperature scaling with different temperatures E [0.4, 0.5. ... 1.3] (defined Eq. (I)) to control the confidence level. The analysis conducted on the ImageNet dataset with various model architectures, using APS and RAPS at 0.1.
High confidence enhances the efficiency of adaptive conformal prediction. In Figures la and 1b we present the average size of prediction sets generated by APS and RAPS under various temperature values t. The results show that a highly-confident model, produced by a small temperature value, would decrease the average size of ion sets. For example, using VGG16, the average size reduced by four times from 20 to 5, with the decrease of the temperature value from 1.3 to 0.5. In addition, we present the effect of lemperature on conditional coverage in Appendix ??. There naturally arises a question: is il always beller for efficiency to take smaller temperature values?
In Figure Ic we report the average size of prediction sets produced by APS on ImageNet with ResNet18, using extremely small lemperatures (i.e. E (0.12, 0.14 ... 0.23). Different from the above, APS generates larger prediction sets with smaller temperatures in this range, even leading to conservative coverage. This problem stems from floating point numerical errors caused by finite precision (see Appendix for a detailed explanation). The phenomenon indicates that it non-trivial to find the optimal temperature value for the highest efficiency of adaptive conformal prediction.
5

---

Published in Transactions on Machine Learning Research (06/2025)
22.5
20.0
BesNet18
ResNet50
14
Reshet18
coverage
17.5
ResNet100
12
ResNec101
Averagn siree
012.5
VGG16
Average
VGG16
Coverage
7.5
5.0
2.5
0.4
0.6
1.0
1.2 14
0.6
1.0
1.2
14
0.12
0.14
0.15
0:18
0.20
Tem perptrruere
Temperature
Tomparatura
(a) APS with various
(b) RAPS with various
(c) APS with small é
Figure 1: (a) & (b): The performance of APS and RAPS with different temperatures on ImageNet. The results show that high-confidence predictions, with a small lemperal lead to efficient prediction sels. (Temperature softens the softmax vector with T' 1 and sharpens with T < 1.) (c): The performance APS for Res.Net18 on ImageNet with eatremely low temperatures. this setting, APS generates large prediction with conservative coverage due to finite precision.
3.3 Theoretical explanation
Intuitively, confident predictions are expected to yield smaller prediction sets than conservative ones. Here, we provide a theorctical justification for this by showing how the reduction temperature decreases the avcrage size of prediction sets the case of non-randomized APS (simply omít. the random term Eq. (5)). We start by analyzing the relationship between the temperature and the APS score. For simplicity, assuming the logits fx (xx,, then, the non-randomized
vector f(x) = [f1 (a);, f2(x)...... (z=]]² satísfics f1 f2(x) > APS score for class & ) is given by:
ef(()/tt
S(x,k,t) = Σ
j==t el(()//*
Then, we can derive the following proposition on the connection of the temperature and the score: Proposition 3.1. For instance x E X, let S(x, t, be the non-conformity score function of an arbitrary class & E V, defined as in Eq. 7 Then, for a fixed temperature tu RR and Vt E (0, to), have
S(x, k. to) S ktt.
The proof is provided in Appendix F. Proposition 3.1 we show that the APS score increases as temperature decreases, and vice versa. Then, for a fixed temperature to Rt, we further define e(k, S(x:, k, t) S(x, k, to) 0 the difference of the APS scores. As a corollary of Proposition 3.1 conclude that <k,,t) t) is negatively correlated with the temperature t. We provide the proof for this corollary in Appendix F.2 The corollary is formally stated as follows:
Corollary 3.2. For any sample E X and fized temperature to, the difference e(k,t) t) is decreasing function with respect to E (0, to).
In the following, we further explore how the change in the APS score affects the average size the prediction set. In the theorem, we make two continuity assumptions on the CDF of the non-conformity score (see A Appendix F.3), following prior works [eii 2014 Sadinle et al. 2019). Given these assumptions, we can derive an upper bound for the expected size of (xx for any (0,t)):
Theorem 3.3. Under assumptions in Appendir
F.3
there exists constants c1, (0, 1] such that
[[(,, [)]]
Σ ci ([[e(k, t)", Vt (0,ta).
x
kEY
Interpretation. The proof of Theorem 3.3 is presented in Appendix F.3 Through Theorem 3.3 we show that for any temperature f, the expected size of the prediction set C(x,,) () has upper bound with respect.

---

Published in Transactions on Machine Learning Research (06/2025)
to the non-conformity score deviation Recalling that increases with the decrease of temperature t, we conclude that lower temperature results in larger difference 6, thereby narrowing the prediction set C(x,t, t). Overall, the analysis shows that tuming temperature values can potentially enhance the efficiency of adaptive conformal prediction. In practice, we may employ grid search to find the optimal T for conformal prediction, but requires defining the search range of T and cannot be extended to post-hoc calibration methods with more parameters, like Platt scaling and Vector scaling. Thus, we propose an alternative solution for automatically optimizing the parameters to enhance the efficiency of conformal prediction.
3.4
An alternative method for improving efficiency
In the previous analysis, we empirically and theoretically demonstrate that standard temperature scaling optimized by negative log-likelihood often leads to degraded efficiency, while searching for relatively small temperature can potentially address this issue. In this work, we propose alternative method, Conformal Temperature Scaling (ConfTS), to automatically optimize the paramcters of post-hoc calibration methods. This is a variant of Lemperature scaling that directly optimizes the objective function toward generating efficient prediction sets, and can be extended to other post-hoc calibration methods.
For a test example (ar y), conformal prodiction aims to construct an efficient prediction set C(x) that contains the true label y. Thus, the oplimal prediclion set meeting this requirement is defined as:
(a(x)) = (k y S(x, k) S v)x,
Specifically, the optimal prediction set is the smallest set that allows the inclusion of the ground-truth label. Recall that the prediction set is established through the т calculated from the calibration set (Eq. (3),, the optimal set can be attained if the threshold well approximates the non-conformity score of the ground-truth label S(x, y). Therefore, we can measure the redundancy of the prediction set by the differences between thresholds 7 and the score of true labels, defined as:
Definition 3.4 (Efficiency Gap). For an example (æ,v), a threshold t and non-conformity score function S(.), the efficiency gap of the instance x is given by:
g y, 7) = T S(x,)))
In particular, a positive efficiency gap indicates that the ground-truth label y included in the prediction set y C(x), and vice versa. To optimize for the optimal prediction set, we expect to increase the efficiency gap for samples with negative gaps and decrease it for those with positive gaps. We propose to accomplish the optimization by tuning the temperature t. This allows us to optimize the efficiency gap since S(æ, y) and т are functions with respect to the temperature & (see Eq. 77).
Conformal Temperature Scaling. To this end, we propose our method - Conformal Temperature Scaling (dubbed ConfTS), which rectifies the objective function of temperature scaling through the efficiency gap. In particular, the loss function for ConfTS is formally given as follows:
LConTTs (ax, y; = (T(1) y))
(8)
where r(t) is the conformal threshold and 5(x,V. denotes the non-randomized APS score of the example (x, y) with respect to t (see Eq. (7). By minimizing the mcan squared error, the ConfTS loss encourages smaller prediction sets for samples with positive efficiency gaps, and vice versa.
The optimization of ConfTS. To preserve the exchangeability assumption, we tune the temperature minimize the ConTTS loss held-out validation set. Following previous work (Stutz et al. 2022). split the validation set into two subsets: one to compute r(t), and the her to calculate the ConlTS loss with the obtained T(t). Specifically, the optimization problem can be formulated as:
= arg min EER+
Σ
CConCTs(FF, 36;1)
Diosal
(E..B:))E
where Dioss denotes the subset for computing ConfTS loss. Trained with the ConfTS loss, we can optimize the temperature L. for adaptive prediction sets with high efficiency without violating coverage. Since the
7

---

Published in Transactions on Machine Learning Research (06/2025)
pre-defined alpha determines the threshold т, ConfTS method can yield different temperature values for each a. In addition, the Connrs can be replaced by ConITr loss (Stutz et al. losses designed for different goals (e.g., improving conditional coverage). In Subsection proposed loss is superior to ConfTr loss with improved efficiency (see Table
2022 or new training
4.2 we show that our
Extensions to other post-hoe calibration methods. Noteworthy, our ConfTS loss general method and can be easily incorporated into existing post-hoc calibration methods such as Platt scaling (Platt et al, 1999) and vector scaling (Guo et al. 2017 Formally, for any rescaling function dg with parameters 9, we can formulate the method as follows. First, we define the k-th softmax probability after rescaling as:
e [(m]]
The (x; 0) o(óg f(x))
Σ elve f(m));
The corresponding non-conformity score for each class i E y is given by S(x,k; -Et= mx( With the threshold r(0), we rewrite the ConfTS loss by
Cooutrs(z, y: t) = (T(0) y1
y, 0))
Then, the optimization objective can be formulated as:
0" arg min
Σ
Con (1i, yz; 0).
Dioss
D
(E::DE)) Ploso
We present the algorithms of our proposed methods step-by-step in Appendix G Moreover, it is worth noting that our method does not conflict with post-hoe confidence calibration, as it only changes the scaling parameters (e.g,, temperature value). During inference, one may use different scaling parameters according to the objective, whether for improved calibration or smaller prediction sets.
4 Experiments
4.1 Experimental setup
Datasets. We evaluate ConfTS on both image and text classification tasks. For image classification, we employ CIFAR-100 (Krizhevsky et al. 2009),, ImageNet (Deng et al. 2009), and ImageNet- (Recht et al. 2019). For text classification, we utilize AG news (Zhang al. 2015) and DBpedia (Auer et al.) 2007] dalasets. For ImageNet, split the lest datasel containing 50,000 images into 10,000 images for calibration and 40,000 for testing. For CIFAR-100 and ImageNet-Vàs we split their test datasets, each containing 10,000 images, into 4,000 images for calibration and 6,000 for testing. For text datascts, we split each test datasct equally between calibration and lesting. Additionally, we split the calibration ion set into two subsets of equal size: one subset is the validation set to optimize the temperature value with ConfTS, while the other half is the conformal set for conformal calibration.
Models. For ImageNet and ImageNet-We V2, we employ 6 pre-trained classifiers from TorchVision ct al. 2019 18, ResNet50, ResNet 101 (He et al. 2016). DenseNet121 (Huang et al. 2017), VGG16 [Simonyan & Zisserman 2015) and ViT-B-16 (Dosovitskiy et al. 2021). We also utilize the same model architectures for CIFAR-106 100 and train them from scratch. For text classification, we finetune a pre-trained BERT (Devlin 2018) and GPT-Neo-1.3B (Black et al. 2021) on each dataset. The model architecture consists of a frozen pro-trained encoder followed by a trainable lincar classifier layer. For each datasot, we employ the AdamW optimizer with learning rate of 2e-5. The training conducted over epochs with batch size of 32. The models are trained for 100 epochs using SGD with momentum of 0.9, weight decay of 0.0005, and a batch size of 128. We set the initial learning rate as 0.1 and reduce it by a factor of 5 at 60 epochs.
Paszke
Conformal prediction algorithms. We leverage three adaptive conformal prediction methods, APS and RAPS, to generate prediction sets at error rate a E {0.1, 0.05). In addition, we set the regularization hyperparameter for RAPS to be: kreg = and X E {0.001, 0.002, 0.004, 0.006, 0.01, 0.015, 0.02). For the evaluation metries, we employ coverage and average size assess the performance of prediction sets. All experiments are repeated 20 times with different seeds, and report average performances.
8

---

Published in Transactions on Machine Learning Research (06/2025)
Table 2: Performance of ConfI'S using APS and RAPS on ImageNet dataset. The tuned is the temperature value optimized by our loss function. We repeat each experiment 20 times. "L" indicates that smaller values are better. Bold numbers are superior results. Results show that our ConfT' can improve the performance of APS and RAPS, maintaining the dosired coverage rate.
APS
RAPS
Model
Error rate
Tuned T
Coverage
Average size +
Coverage
Average size +
Base
ConfTS
0.1
0.593
0.900 / 0.900
14.09 / 7.531
0.900 / 0.900
9.605
5.003
ResNet18
= 0.05
0.591
0.951 / 0.952
29.58 / 19.59
0.950 / 0.950
14.72 / 11.08
= 0.1
0.705
0.899 / 0.900
9.062 / 4.791
0.899 / 0.900
5.992
3.561
ResNet50
= 0.05
0.709
0.950 / 0.951
20.03 / 12.22
0.950 / 0.951
9.423
5.517
0.1
0.793
0.900 / 0.899
6.947
4.328
0.900 / 0.899
4.819
3.289
ResNet1
0.05
0.785
0.950 / 0.950
15.73 / 10.51
0.950 / 0.950
7.523
5.091
0.1
0.659
0.900 / 0.899
9.271 / 4.746
0.900 / 0.900
6.602
3.667
DenseNett22
0.05
0.675
0.950 / 0.949
20.37
11.47
0.949 / 0.949
10.39
6.203
0.1
0.604
0.901
0.901
11.73 / 6.057
0.901 / 0.900
8.118
4.314
VGG16
0.05
0.627
0.951
0.951
23.71 / 14.78
0.950 / 0.950
12.27
8.350
0.1
0.517
0.900
0.901
14.64 / 2.315
0.902 / 0.901
6,889
1.800
ViT-B-16
0.05
0.482
0.951 0.950
36.72
9.050
0.950 / 0.950
12.63
3.281
4.2
Main results
ConfTS improves current adaptive conformal prediction methods. In Table 2 we present the performance of APS and RAPS (A = 0.001) with ConfTS on the ImageNet dataset. A salient observation is that ConfTS drastically improves the efficiency of adaptive conformal prediction, while maintaining the marginal coverage. For example, on the VIT model at a = 0.05, ConfISS reduces the average size of APS by times from 36.72 to 5.759. Averaged across six models, ConfTS improves the efficiency of APS by 58.3% a 0.1. We observe similar results on CIFAR-100 and ImageNet-V2 dataset in Appendix [ and Appendix [H] Moreover, ConfIS remains effective for RAPS across various penalty terms
ImageNet shown in Appendix J Furthermore, in Appendix K we demonstrate that ConfTS can lead to small prediction sets for SAPS Huang et al., 2024). recent technique of adaptive conformal prediction. In addition, we find that the tuned temperature values are generally smaller than 1.0 and different for varions settings, which demonstrates the importance of the automatic method. Overall, empirical results show that ConfTS consistently improves the efliciency of existing adaptive conformal prediction methods.
Our method can work with other post-hoc calibration methods. We extend the application of ConfTS loss (Eq. (8) other post-hoc calibration methods. We introduce conformal Platt scaling (dubbed ConfP'S and conformal vector scaling (dubbed ConfVS), where the parameters are optimized using ConfTS loss. We employ ResNet50 and VGG16 models the ImageNet dataset for thimage task, as well as BERT and GPT-Neo-13B on the DBpedia dataset for the text task. The error rate is = 0.1. Table 3 shows that both ConfPS and ConfVS can help construct efficient prediction sets. This indicates that replacing cross-entropy loss with ConfTS loss in post-hoc calibration methods consistently enhances the efficiency of adaptive conformal prediction. In addition, we provide the calibration performance of our methods in Appendix L Overall, these results validate the effectiveness of ConfTS loss across different calibration methods.
ConfTS maintains the adaptivencss. Adaptiveness (Romano et al.] 2020 Angelopoulos et al. 2021 Seedat et al. 2023) requires prediction sets to commmicate instance-wise uncertainty: easy examples should obtain smaller sets than hard ones. In this part, examine the impact of ConfTS on the adaptiveness of
9

---

Published in Transactions on Machine Learning Research (06/2025)
Table The average size of APS and RAPS using ConfPS and ConfVS. ConfPS and ConfVS are the variants of Platt scaling and vector scaling, optimized by our ConfTS loss. "A and " " indicate the performance is superior aaperior/inferior /inferiorrrrrr to the baseline. The results show that by rescaling the logits with ConfPS and ConfVS, the
algorithm can construct efficient prediction sets, demonstrating the generality of our loss function.
APS
RAPS
Dataset
Model
Baseline
ConfTS
ConfPS
ConfVS
Baseline
ConfTS
ConfPS
ConfVS
ResNet50
9.062
4.791
2.571
4.564
5.992
3.561
2.446 A 3.303
ImageNet
DenseNet121
9.271
4.746
3.169
5.345
6.602
3.667
3.224
3.683
VGG16
11.73
6.057
3.729
A
7.020
8.118
4.314
3.558
4.745
ViT-B-32
14.64
2.315
1.743
4.797
6.899
1.800
1.575
2.549
BERT
2.105
1.886
1.808
A
1.979
2.004
1.802
1.794
1.949
AG news
GPT-Neo-1.3B
2.022
1.911
1.749
1.897
2.018
1.909
1.728
1.884
BERT
3.557
2.905
2.96
3.869
3.458
2.908
2.837
3.744
Dbpedia
GPT-Nco-1.33
3.171
2.178
1.826
1.884
3.137
2.144
1.768
2.415
Average
13.89
6.697
4.889
7.839
9.557
5.526
4.733
6.068
Table 4: The average size of APS and RAPS with various post-hoe calibration methods optimized by our
loss and ConfIr loss. Bold numbers superior results between two loss functions. The results show that ConfTS loss achieves better performance than ConfTr loss in most cases.
ConfTS
ConfPS
ConfVS
Model
Baseline
Our loss
ConfTr loss
Our loss
ConfTr loss
Our loss
ConfTr loss
APS
9.062
4.719
/
8.864
2.571
/
2.657
4.564
/
4.471
ResNet50
RAPS
5.992
3.561
/
5.980
2.446
/
2.500
3.303
/
3.333
APS
11.73
6.057
/
9.822
3.729
/
4.193
7.020
/
6.757
VGG16
RAPS
8.118
4.314
6.825
3.558
/
3.921
4.745
/
4.742
Average
8.726
4.663
/
7.873
3.076
/ 3.318
4.908
/
4.826
prediction sets and measure the instance difficulty by the order of the ground truth oly. (xx).. Specifically, partition the sample by label order: 1, 2-3, 4-6, 7-10, 11-100, 101-1000, following (Angelopoulos al. 2021). Figure 2a and Figure show that prediction sets, when applied with ConfTS, satisfy the adaptiveness property. Notably, employing ConfTS promote smaller prediction sets for all examples ranging from easy to hard. In addition, we provide a discussion on conditional coverage in Appendix ??. Overall, the results demonstrate that APS with ConfTS succeeds in producing adaptive prediction sets: examples with lower difficulty obtain smaller prediction sets on average.
Ablation study on the size of validation and calibration set. In the experiment, ConfIS splits the calibration data into two subsets: validation set for tuning the temperature and conformal set for conformal calibration. In this part, analyze the impact of this split the performance of ConfTS by varying the validation and conformal datasct sizes from 3,000 to 8,000 samples while maintaining the other part at 5,000 samples. We use ResNei18 and ResNet55 on ImageNet, with APS al a = 0.1. Figure 2 and 2d show that the performance of ConfTS remains consistent across different conformal dataset sizes and validation dataset sizes. Basod on these results, we choose a calibration set including 10000 samples and split it into two equal subsets for the validation and conformal set. In summary, the performance of ConfTS robust to variations in the validation dataset and conformal dataset size.
10

---

Published in Transactions on Machine Learning Research (06/2025)
Average size
60
80
ConfTS
Base
Average size
80
70
ConfTS
Base
7.5
8.0
Average Siz
7.0
7.5
40-
30-
50
20-
40
30
6.5
6.0
5.5
ResNet18
ResNet50
6.5
6.0
5.5
Net
Net50
10-
10
5.0
5.0
4.5
4.5
3k
5k
6k
7k
8k
4k
5K
6k
Difficulty
Difficulty
Conformal size
Validation size
(a) ResNet18
(b) ResNet50
(c)
(d)
Figure 2: (a)&:b): Average sizes of examples with different difficulties using APS on ResNet18 and ResNet50, respectively. Results show that ConfTS can maintain adapliveness. (c)&(d) Average sizes of APS employed
with ConfIS under various sizes of (c) conformal dataset (d) validation dataset. Results show that our ConfISS is robust to variations in the validation and conformal dataset size.
Table 5: The performance ConfTS using various non-conformity scores to compute the efficiency gap. We consider standard APS and RAPS score as well as their non-randomized variants. Each experiment is repeated 20 times. " "Avg.size" and "Cov." represent the results of average size and coverage, and 'Base' presents the results without ConfTS. "4" indicates smaller values are better. " and " indicate the performance is superior mrrrrior/inffriin /inferiorrrr to the baseline. Bold numbers superior results.
Base
APS no random
RAPS no random
APS random
RAPS random
Model
Score
Avg.size 4
Cov.
Avg.size
Cov.
Avg.size +
Cov.
Avg.size 4
Cov.
Avg.size +
Cov.
ResNet18
APS
14.09
0.900
7.531
0.900
7.752 A
0.900
13.67 A
0.900
13.97
0.900
RAPS
9.605
0.900
5.003 A
0.900
5.346
0.900
11.36 v
0.900
11.58
0.900
APS
9.062
0.900
4.791 A
0.900
5.201
0.900
12.92
0.900
16.43 V
0.900
RexNet50
RAPS
5.992
0.900
3.561 A
0.900
3.782 A
0.900
9.838
0.900
11.70 V
0.900
Our loss function outperforms ConfTr loss.
Previous work (Stutz et al.
2022 proposes Conformal
Training (ConfTr)), which enhances prediction set efficiency during training through novel ConfIr loss function. In particular, the ConfTR loss is defined as
Eatp CConETR max 0, Σ al Cappre 1 S(x, v)l) - k )
where = by default to prevent penalizing singletons and P a smoothing parameter. this experiment, we set == 1.2. We compare the performance of ConfTS, ConfPS, and ConfVS when trained with both ConfTr loss and proposed ConfTS loss. In particular,
the ConfTr loss to replace line 9 of the
algorithms presented in Appendix G
Using ResNet50 and VGG16 models on ImageNet, we generate prediction sets with APS and RAPS at an error rate a = 0.1. The results in Table 4 demonstrate that while both loss functions improve prediction set officiency, our loss typically achieves better performance than the ConfTr loss. For example, with the ResNet50 model, ConITS loss reduces the average size of APS 4.791, compared to 8.864 when using ConfTr Joss. Overall, the proposed loss function superior the ConfIr loss in optimizing the calibration methods.
Ablation study on the non-conformity score in ConfTS. In this ablation, compare the performance of ConfIS trained with various non-conformity scores in Eq. including standard APS and RAPS, as well as their non-randomized variants. Table 5 presents the performance of prediction sets generated by standard APS and RAPS (A 0.001) methods with different. variants of ConfTS, employing ResNet18 and ResNet50 on ImageNet. The results show that ConfTS with randomized scores fails to produce efficient prediction sets, while non-randomized scores result in small prediction sets. This because the inclusion of the random variable leads to the wrong estimation of the efficiency gap, thereby posing challenges the optimization process in ConfTS. Moreover, randomized APS consistently performs better than randomized RAPS, even
11

---

Published in Transactions on Machine Learning Research (06/2025)
when using standard RAPS to generate prediction sets. Overall, our findings show that ConfTS with the non-randomized APS outperforms the other scores in enhancing the efficiency of prediction sets.
Discussion on the conditional coverage. In this part, we discuss the impact of tem- perature on the conditional coverage and how ConlTS affects the conditional coverage. In par- ticular, the conditional coverage (Vovk) 2012) requires conformal prediction methods sat- isfy the marginal coverage at the instance level. The size-slralified coverage violalion (SSCV) (Aneloppoloss et al. 2021) is often employed to
ResNer50
ResNott1:
Densehett22
VGG18
Temperature
evaluate the conditional coverage of prediction sets:
(a)
(b)
Figure 3: (a): The performance of APS on SSCV with different temperatures on ImageNet. The round marks rep- resent the temperature value obtained from ConfTS. (b): The SSCV performance of ConfTS using APS on ImageNet. dataset. A smaller SSCV is better.
Ei Sy yi E C (x::) - - (1-a)l,
SSCV = sup 8
|S
where { N, is disjoint set-size strata, satis- fying U S; (1, D/I. A lower SSCV value indicates better conditional coverage per-
formance. Following prior work (Angelopoulos et al. 2021 we set the partitioning of the set sizes as: 0-1, 2-3, 4-10, 11-100, and 101-1000. Figure 3a presents the SSCV performance of APS with varying temperatures. The results show that the optimal lemperature value for the lowest SSCV of APS is generally smaller than 1.0, so vanilla temperature scaling causes worse performances on conditional coverage, as typically encourages a large temperature. In contrast, our ConfTS usually leads to a relatively low temperature, achieving promising performance on conditional coverage.
Figure 5a presents the SSCV performance of APS with varying temperatures. The results show that the optimal temperature value for the lowest SSCV of APS is generally smaller than 1.0, so vanilla temperature scaling causes worse performances on conditional coverage, as it typically encourages a large temperature. In contrast, our ConfTS usually leads to a relatively low temperature, achieving promising performance on conditional coverage. In Figure 5b, we present comparison of SSCV performance between LAC, APS, and +PnffS on various model architectures. In this setting, the results show that our mothod can enhance the conditional coverage of APS, while LAC achieves the worst results on SSCV. For example, ResNet50, ConfTS reduces the SSCV of APS from 0.033 to 0.025, and the SSCV of LAC is 0.125 much larger than method. Despite the empirical improvements, conditional coverage as it is not included in the training objective. We hope this work can inspire future work to design specific training losses to improve conditional coverage.
emphasize that our method cannot guarantee improved
Based on the analysis above, we provide the following guidelines for practitioners to select the appropriate conformal technique:
For minimal prediction set sizes with marginal coverage: use the LAC non-conformity score, which is proven to yield the smallest prediction sets, though it offers limited conditional coverage.
For conditional coverage with compact sets: opt for APS or RAPS as the non-conformity score and apply ConITS, Con(VS, or CoufPS to further reduce prediction set sizes (see Table 2
5 Related Work
Conformal prediction. Conformal prediction (Papadopoulos et al. 2002 Vovk et al. 2005 is a statistical
framework for uncertainty qualification. Some methods leverage post- hoc techniques to enhance prediction sets (Romano al. 2020 Angelopoulos et al.
2021
Ghosh et al. 2023 Huang et al. 2024p. For example, caleulates the score by accumulating the sorted softmax
Adaptive Prediction Sets (APS) (Romano et al.
2020
12

---

Published in Transactions on Machine Learning Research (06/2025)
values in descending order. However, the softmax probabilities typically exhibit long-tailed distribution, and thus, those tail classes are often included in the prediction sets. To alleviate this issue, Regularized Adaptive Prediction Sets (RAPS) (Angelopoulos et al., 2021) exclude tail classes by appending penalty to these classes, resulting in efficient prediction scts. These post-hoc mothods often employ temperature
scaling for better calibration performance (Angelopoulos el al. 2021 Lu et al. 2022 Gibbs et al. 2023 Lu
ct al. 2023). In our work, we show that existing confidence calibration methods could harm the efficiency of adaptive conformal prediction.
Some works propose training time regularizations to improve the efficiency of conformal prediction (Colombo
& Vovk 2020 Stutz et al. 2022 Einbinder et al.; 2022 Bai et al.) Correia et al.) 2024). For example,
uncertainty-aware conformal loss function (Einbinder et al. 2022) optimizes the efficiency of prediction sets by
encouraging the non-conformity scores to follow a uniform distribution. Moreover, conformal training (Stutz ct al. 2022) constructs efficient. prodiction sets by prompting the threshold to be less than the non-conformity scores. In addition, information- conformal training (Correia et al., 2024) incorporates side information into the construction of prediction sets. In this work, we focus on post-hoc training methods, which only require the pre-trained models for conformal prediction. ConfI'S is easy to implement and requires low computational resources.
Confidence calibration. Confidence calibration has been studied in various contexts in recent years. Numerous methods have been developed to enhance the calibration performance of machine learning models. Some works address the miscalibration problem by post-hoc methods, including histogram binning ZZadrozny & Elkan, 2001 and Platt scaling (Platt et al. 1999). Besides, regularization methods like entropy regularization Pereyra et al.] 2017) and focal loss (Mukhoti et al., 2020) are also proposed to improve the calibration performance of deep neural networks. A concurrent work (Dabah & Tirer 2024) also investigates the effects of temperature scaling on conformal prediction. However, they only focus on the temperature scaling and do not extend the conclusion to other post-hoc and training methods of confidence calibration. In this work, we provide a more comprehensive analysis with both post-hoc and training methods of confidence calibration. In addition to the analysis, we also provide a theoretical explanation and rodree a novel method to optimize the parameters of post-hoc calibrators automatically.
6 Conclusion
In this paper, we investigate the relationship between two uncertainty estimation frameworks: confidence calibration and conformal prediction. We make two discoveries about this relationship: firstly, existing confidence calibration methods would lead to larger prediction sets for adaptive conformal prediction; secondly, high-confidence prediction could enhance the efficiency of adaptive conformal prediction. We prove that applying a smaller temperature to prediction could lead to more efficient prediction sets on expectation. Inspired by this, we propose a variant of temperature scaling, Conformal Temperature Scaling (ConfTS), which rectifies the optimization objective toward generating efficient prediction sets. Our method can be extended to other post-hoe calibrators for improving conformal predictors. Extensive experiments demonstrate that our method can enhance existing adaptive conformal prediction mothods, in both image and text classification tasks. Our work challenges the conventional wisdom of utilizing confidence calibration for conformal prediction, and we hope it can inspire specially- desigged methods to improve the two frameworks of uncertainty estimation.
Limitations. In this work, the conclusions of our analysis are mostly for adaptive conformal prediction methods, without generalizing to the LAC score. In addition, the proposed method only focuses on enhancing the efficiency of prodiction sets and may not help in conditional coverage, similar to current training methods for conformal prediction. We believe it can be interesting to design loss functions specifically tailored for improving conditional coverage, in future works.
13

---

Published in Transactions on Machine Learning Research (06/2025)
7 Acknowledgements
This rescarch is supported by the Shenzhen Fundamental Rescarch Program (Grant No. JCY1)002300000)88800000)). We gratefully acknowledge the support of the Center for Computational Science and Engineering at the Southern University of Science and Technology for our research.
References
Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution- free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
Anastasios Nikolas Angelopoulos, Stephen Bates, Michael I. Jordan, and Jitendra Malik. Uncertainty sets for image classifiers using conformal prediction. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpedia: A nucleus for web of open data. In international semanlic web conference, pp. 722-735. Springer, 2007.
Yu Bai, Song Mei, Huan Wang, Yingbo Zhou, and Caiming Xiong. Efficient and differentiable conformal prediction with general function classes. In International Conference on Learning Representations.
Vineeth Balasubramanian, Shen-Shyang Ho, and Vladimir Vovk. Conformal prediction for reliable machine learning: theory, adaptations and applications. Newnes, 2014.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. If you this software, please cite it using these metadata, 58 (2), 2021.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint or
2016.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1721-1730, 2015.
Nicolo Colombo and Vladimir Vovk. Training conformal predictors. In Conformal and Probabilistic Prediction and Applications, pp. 55-64 64. PMLR, 2020.
Alvaro HC Correia, Fabio Valerio Massoli, Christos Louizos, and Arash Behboodi. An information theoretic perspective conformal prediction. arXiv preprint arXiv:2405.0214 2024.
Jesse Cresswell, Yi Sui, Bhargava Kumar, and Noël Vouitsis. Conformal prediction sets improve human decision making, In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024.
Lahav Dabah and Tom Tircr. On calibration and conformal prediction of deep classificrs. arXiv preprint [xxx:2402.05806 2024.
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Malthias Bauer, and Philipp Hennig. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems, 14:20089-20100 2021.
Jía Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255. Ieee, 2009.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. orXiv preprinl Xiv: 18:0:181..00 04805, 2018.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview net, 2021.
Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia, and Yanfei Zhou. Training uncertainty- classifiers with conformalized deep learning. Advances in Neural Information Processing Systems, 35:22380-22395, 2022.
Subhankar Ghosh, Taha Belkhouja, Yan Yan, and Janardhan Rao Doppa. Improving uncertainty quantification of deep classifiers via neighborhood conformal prediction: Novel algorithm and theoretical analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 7722 7730, 2023.
Isaac Gibbs, John J Cherian, and Emmanuel J Candès. Conformal prediction with conditional guarantees.
arviv preprint ar Xi::2305.1266,, 2023.
Chuan Guo, Gcoff Plciss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321 1330. PMLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Compuler Vision and Pallern Recognition, pp. 770 778, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional nolworks. In Proceedings of the IEFE Conference on Compuler Vision and Pollern Recognition, pp- 4700- 4708, 2017.
Jianguo Huang, HuaJun Xi, Linjun Zhang, Huaxiu Yao, Yue Qiu, and Hongxin Wei. Conformal prediction for deep classifier via label ranking. In Forty-forst International Conference on Machine Learning, 2024.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Jing Lei. Classification with confidence. Biometrika, 101(4):755-700 769, 2014.
Charles Lu, Syed Rakin Ahmed, Praveer Singh, and Jayashree Kalpathy- Estimating test performance for Al medical devices under distribution shift with conformal prediction. CoRR, hbss/220.005790 2022.
doi: 10.48550/ARM5 ARXIV 2207.05796.
Charles Lu, Yaodong Yu, Sai Praneeth Karimireddy, Michael Jordan, and Ramesh Raskar. Federated conformal predictors for distributed uncertainty quantification. In International Conference on Machine Learning, pp. 22442229644. PMLR, 2023.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puncet Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 33: 15288 15299, 2020.
Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence machines for regression. In Machine Learning: ECML 2002: 13th Europeon Conference Machine Learning Helsinki, Finland, August 19-23, 2002 Proceedings 18, pp. 345-356. Springer, 2002.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performanee deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. Regularizing neural networks by penalizing confident output distributions. In 5th International Conference on Learning Repre- sentations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.ner net, 2017.
John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 10(3):61-74 74, 1999.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. (389-5400) PMTR, 2019.
Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive coverage. Advances in Newral Information Processing Systems, 53:3581-359) 2020.
Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least ambiguous set- set-valued classifiers with bounded error levels. Journal of the American Statistical Association, 114((25):223 223-2334422232 2019.
Nabeel Scedat, Alan Jeffares, Fergus Imrie, and Mihaela van der Schaar. Improving adaptive conformal prediction using self-supervised learning. In International Conference on Artificial Intelligence and Statistics, pp. 10160 10177. PMLR, 2023.
Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Ralph C Smith. Uncertainty quantification: theory, implementation, and applications, volume 12. Siam, 2013.
David Stutz, Krishnamurthy Dvijotham, Ali Taylan Cemgil, and Arnaud Doucet. Learning optimal conformal classifiers. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.nee, 2022.
Christian Szegedy, Vincent Vanhoucke, Sorgey Loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818 2818-2826, 2016.
Sunil Thulasidasan, Gopinath Chenmupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Vladimir Vork. Conditional validity of inductive conformal predictors. In Asion Conference on Machine Learning, pp. 475-490. PMLR, 2012.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world, volume 29. Springer, 2005.
Cheng Wang. Calibration deep learning: A survey of the state-of-tho-art. arXiv preprint vrXiw:2308.0122 2023.
Shuoyuan Wang, Jindong Wang, Guoqing Wang, Bob Zhang, Kaiyang Zhou, and Hongxin Wei. Open- vocabulary calibration for finc-tuned CLIP. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.nng net, 2024.
Hongxin Wei, Reuchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In International Conference on Machine Learning, pp. 23631-23644 PMLR, 2022.
Mert Yuksekgonul, Linjun Zhang, James Zou, and Carlos Guestrin. Beyond confidence: Reliable models should also consider atypicality. In Thirty-seventl Conference on Neurol Information Processing Systems, 2023.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In International Conference on Machine Learning, volume 1, pp. 609 616, 2001.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
16

---

Published in Transactions on Machine Learning Research (06/2025)
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character- convolutional networks for text classification.
A ddances in neural informalion processing systems, 28, 2015.
17

---

Published in Transactions on Machine Learning Research (06/2025)
A Conformal prediction metrics
In practice, we often use coverage and average size to evaluate prediction sets, defined as:
Coverage
Σ
1 E c((;)))
(10)
[Dinst
(#i.ge)EDecat
Average size
Σ
((x:)),
(11)
Drast
where 1(.) is the indicator function and Drest denotes the test dataset. The coverage rate measures the percentage of samples whose prediction set contains the true label, i.., an empirical estimation for P(Y E C(X)). The average size measures the efficiency of prediction sets. For informative predictions (Vovk) 2012 Angelopoulos al. 2021), the prediction sets are preferred to be efficient (i.e., small prediction sets) while satisfying the valid coverage (defined in Eq. (2
B
Confidence calibration methods
Here, we briefly review three post-hoc calibration methods, whose parameters are optimized with respect to negative log-likelihood (NLL) on the ion set, and three training calibration methods. Let a be the softmax function and f E R an arbitrary logits vector.
Platt Scaling (Platt et al. 1999) is parametric approach for calibration. Platt Scaling learns two scalar parameters o, E I and outputs
π = oaaf + b).
(12)
Temperature Scaling (Guo et al., 2017) is inspired by Platt scaling (Platt et al, 1999 using a scalar parameter for all logits vectors. Formally, for any given logits vector f. the new prodiction is defined by
(f/()-
Intuitively, t softens the softmax probabilities when & > 1 so that it alleviates over-confidennc.
Vector Scaling (Guo et al. 2017) is a simple extension of Platt scaling, Let f be an arbitrary logit. vector, which produced before the softmax layer. Vector scaling applies linear transformation:
π oMMf+Mf +).
where M R and e R
Label Smoothing (Szegedy al., 2016) softens hard labels with introduced smoothing parameter in the standard loss function (e.g., cross-entropy):
K
C= Σ vi log De ye - -x) + ta/K,
where yk is the soft label for k-th class. is shown that label smoothing encourages the differences between the logits of the correct class and the logits of the incorrect class to be a constant depending on a.
Mixup (Zhagg et al., 2018 is another classical work in the line of training calibration. Mixup generates synthetic samples during training by convexly combining random pairs of inputs and labels as well. To mix up two random samples (z,, and (Iz, Uj), the following rules are used:
-= a. + (1 a) 7 = =-y + (1 a)8y
where (Z,, D:) the virtual featuro-targot of original pairs. Previous work Thulaaiddssan et al., 2019) observed that compared to the standard models, mixup-trained models are better calibrated and less prone to overconfidence in prediction out-of-distribution and noise data.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Bayesian Method (Daxberger et a 2021). Bayesian modeling provides principled and unified approach to mitigale poor calibration and overconfidence by equipping models with robust uncertainty estimates. Specifically, Bayesian modeling handles uncertainty in neural networks by modeling the distribution over the weights. In this approach, given observed data D = { {X, y), we aim to infer a posterior distribution over the model parameters using Bayes' theorem:
D( p(D(0)p(0) ((D)
(13)
Here, (D[)) represents the likelihood, p(0) is the prior over the model parameters, and p(D) is the evidence (marginal likelihood). However, the exact posterior p(0) D) is often intractable for deep neural networks due the high-dimensional parameter space, which makes approximate inference techniques necessary.
One common method for approximating the posterior is Laplace approzimation (LA). The Laplace approxi- mation assumes that the posterior is approximately Gaussian in the vicinity of the optimal parameters 0MAP, which simplifies inference. Mathematically, LA begins by finding the MAP estimate:
MAPP = arg max lag + log p(0).
(14)
Then, the posterior is approximated by a Gaussian distribution:
D(0 R N(OMAP, H-',, H loggg((()))
(15)
A A=0MAA
The LA provides an efficient and scalable method capture uncertainty around the MAP estimate, making it widely used bascline in Bayesian deep learning models.
C Experimental setups for motivation experiments
We conduct the experiments CIFAR-100 (Krizhevsky et al. 2009). We split the test dataset including 10,000 images into 4,000 images for the calibration set and 6,000 for the test set. Then, we split the calibration set into two subsets of equal size: one the validation set used for confidence calibration, while the other half is the conformal set used for conformal calibration. We train a ResNet50 model from scratch. For post-hoc methods, we train the model using standard cross-entropy loss, while for training methods, we use their corresponding specific loss functions. The training detail is presented in Section 4.1 We leverage APS and RAPS to generate prediction sets at error rate 0.1, and the hyperparameters set to be kreg and À- 0.001.
D Experiment results of LAC
D.1 How does confidence calibration affects LAC?
In this part, we investigate the connection between and confidence calibration methods. We employ three pre- trained classifiers: ResNet18, 1es (He et al. 2016), DenseNet121 (Huang el al. 2017) on CIFAR-100, generating LAC prediction sets with a = 0.1. In Table 6 the results show that different post-hoc methods have varying impacts on LAC prodiction sets, while all of them can maintain the dcsired coverage rate. For
example, the original average size of ResNet18 with respect to THR is 2.23, increases 2.40 with vector scaling, 2.34 with temperature scaling, and decreases to 2.20 with Platt scaling.
19

---

Published in Transactions on Machine Learning Research (06/2025)
Table The performance of LAC prediction sets employed with different calibration methods: baseline (BS),
vector scaling (VS), Platt scaling (PS), and temperature scaling (TS). indicates smaller values are better.
Model
Metrics
BS
VS
PS
TS
Avg.size 4
2.23
2.42
2.26
2.34
ResNet18
Coverage
0.90
0.90
0.90
0.90
Avg.size
1.88
1.98
1.83
1.83
ResNet101
Coverage
0.90
0.90
0.90
0.90
Avg.size 4 1.68
1.68
1.69
1.65
DenscNect211
Coverage
0.90
0.90
0.90
0.90
D.2 LAC with high-confidence prediction
In Figure 1 we compare the performance of LAC prediction sets deployed with different lemperatures. We observe that when used with small temperature, models tend to generate large prediction sets, while the coverage rate stabilizes about 0.9, maintaining the marginal coverage. Morcover, we observe that models typically construct the smallest prediction set when the temperature approximates 1. Therefore, cannot search for an appropriate temperature that benefits LAC.
038
derreeeet12
Coverage
vggll
Average
35
15-
6ondenet12
vgg 10
29
Temperature
Temperature
(a) Coverage
(b) LAC
Figure 4: The performance comparison of prediction sets with different temperalures.
E Why numerical error occurs under an exceedingly small temperature?
In Section 3.3 we show that an exccedingly low temperature could pose challenges for prediction sets. This problem can be attributed to numerical errors. Specifically, in Proposition 3.1 we show that the softmax probability tends to concentrate in top classes with small temperature, resulting in long-tail distribution. Thus, the tail probabilities of some samples could be small and truncated, eventually becoming zero. For example, in Figure 5 the softmax probability is given by π(Ø) [0.999997,2 x 10-5, x 10-6,.... and the prediction set size should be 4, following Eq. (4) However, due to numerical error, the tail probabilities, i.e., ns. Fe are truncated to be zero. This numerical error causes the conformal threshold to exceed the non-conformity scores for all classes, lcading a trivial set. Furthermore, as the temperature decreases, numerical errors occur in more data samples, resulting in increased trivial sets and consequently raising the average set size.
20

---

Published in Transactions on Machine Learning Research (06/2025)
0.99997
t
2e-5
*truncated
1e-6
1e-7
1e- 20 1e-20
711
1
75
755
TE6
Figure 5: An example of softmax probabilities produced by small temperature.
F
Proofs
F.1 Proof for Proposition 3.1
We start by showing several lemmas: the Lemma F. Lemma F.2 and Lemma F.3
Lemma F.1. For any given logits (f1.... fkk with fi fo fk, and a constant 0 < t < 1, we have:
ehit
eh
(a)
Ei=e Exeetilt
eflt
Σ ehi
elx/t
elk
(b)
Exgehhi ehit
K
5K 2i=1 el
Proof. Lot = --11 Then, we have
elift
ell+s)s
efi
eti
2i=1es Eistehi ehit
K Σ ell+s)si
Lizl efi (1)
K
Σ K
ビ eh
exxit
(l-s)xxx
ctx
eh
E elift elitt
K Σ e((+))
E
k eliettfi-MMi
5 k eh
Lemma F.2. For any given logils (fi,... fk) with Si f2 erists j > 1 such that
SK, and a conslant < 1, if there
eiit
cli
k
ci/t
Σx ch
then, for all 1,2,... .s. we have
el/lt
eSh
(16)
E
efift
K Li=1 eh
Proof. It suffices to show that
ehi-l/t
ets-1
(17)
Ekeliit elift
Lizl el
21

---

Published in Transactions on Machine Learning Research (06/2025)
since the rest cases where & == 1,2, . would hold by induction.
The assumption gives us
ellt eb
>i= ehit
Σ
Let -1, which follows that
eb/t
e((+s)))
eb
(0)
eb
E ehit
Σ i== e((+))s
Che
k
Σ
-i=1
eh
The inequality (a) indicates that
Σ e ((
< eli
Therefore, we can have
elo-t/t
((++))5-11
edo-1
eli-1
el--1
ΣK ebit
Σx K
2i=1 ell+s)si
ΣK eheelti-----)
Ek ehe
5K
el
which proves the Eq.
17 Then, by induction, the Eq.
16 holds for all S k <.
Lemma F.3. For any given logits (fi,... fk), where fi k = 1,2, ... K, we have
> f2
fK, constant 0 < < 1, and for all
k elit
k
ef
(18)
j= efilt
2jj=1 eli
t=l
The equation holds if and only if k = K.
Proof. The Eq. 18 holds trivially at K, since both sides are equal to 1:
K
ft/t
K
ef
=
(19)
k ==
elit
el
2==1
We continue by showing the Eq.
18 at k K 1.
The Lemma
F.1
gives us that
ejxlt
etx
(20)
Σ efi/
Σk
eh
Subtracting the Eq. 20 by the Eq.
20 directly follows that
K-1
eilt
K-1
ef
(21)
E elii
1==1 eb
which prove the Eqq. 18 at k = K - 1. We then show that the Eq.
18 holds at A = 2, which follows that the Eq. 118 remains true for all k = 1, K - by induction. Here, assume that
K-2
cilt
K-2
eh
Σ
(22)
Eki
cl
and we will show that the Eq. that
(22 leads to a contradiction. Subtracting Eq.
22 by the Eq.
21 gives us
elk-1/t
efk-1
(23)
Ekehhtt
elift
ΣK el
22

---

Published in Transactions on Machine Learning Research (06/2025)
implies that Considering the Lemma F.2 the Eq.
23
elit
el
(24)
k
efilt
R siml
eh
holds for all 1,2, K - 2. Accumulating the Eq.
24
from k = 1 to K gives us that
K-2
efit
K-2 eh
K 1==1
ebit
5 k j= el
==1
This contradicts our assumption (Eq. (22 ). It follows that Eq. 18 holds k K- Then, by induction,
the Eq. (18] remains true for all & 1, K 1. Combining with the Eq. (19 we can complete our proof.
Proposition F.4 (Restatement of Proposition 3.1 For any sample a E X, let S(x,k, be the non- conformity score function with respect to an arbitrary class k E y, defined as Eq. 7 Then, for a fixed temperature to and Vt E (0, to), we have
k(
Proof. We restate the definition of non-randomized APS score in Eq. 7
k
el
S(x, v. Σ
Lj=z
k
Let = t/to E 1) and f. = fe/to- We rewrite the formulation of S(x, k to) and S(x, k,t) by
ef
S(x,y,to) =
== 2ij=1 eb
chiaa
S(x,v,t)
k -i=1 elsta"
Since the scaling parameter to does not change the order of (fr, f2. fkk, i.c A> ... > fx and a E (0, 1), then by the Lemma F.3 we have S(x,y.1o) S(æ,
F.2 Proof for Corollary 3.2
3.2 For any sample a E X and a fixed temperature lo, the difference e(k,t) f) is a decreasing function with respect to E (0, to).
Corollary F.5 (Restatement of Corollary
Proof. For all 4,, la satisfying 0 < f < l2 < to, will show that e(k,t1) > e(k, 12). Continuing from Proposition 3.1 we have S(x, v2 < S(x, y, ti). It follows that
e(k, (1) = S(x, k, (1) S(x, k, to)
> S(x, k 12) S(x, k. to) c(k fz).
23

---

Published in Transactions on Machine Learning Research (06/2025)
F.3 Proof for Theorem 3.3
In the theorem, make two continuity assumptions on the CDF of the non-conformity score following Lei
2014 Sadinle et al. 2019). We define (-) as the CDF of S(æ, k t), assuming that
(1) c1 02 E (0, 1] s.t. Ak E y, a s (s + - G[]( = col=??,
(25)
(2) > 0 s.t. inf I (s) (s) 2p.
kis
To prove Theorem 3.3 start with lemma:
Lemma F.6. Give pre-trained model, data sample æ, and temperature satisfying t" < to. Then, under
assumtion 25 we have
P(k E C(2, k $ C(ar (3) 2 (2e(k))
Proof. Let (-) be the probability measure corresponding to (.), and C²(()) = fa: S(æ,v, < 88. Then, we have
pto (Ct PC (T(t") + c(k,t")) (T())
(26)
(0) 2 o (T((") + t((ktt") +
where (a) comes from the assumption (2). Let T = r(t)) - =(k,tt - 1//
Then, replacing the T r(**) in
Eq. 26 with T", we have
plo (C (7 2 Gy (7((²) - 105 (1) +
(a)
G (T((")
(27)
@ (Ct
where (a) is due to the assumption (1):
(T(())) - C. (T((") -c( (h) S c2l1c 1/1 p.
(b) and (c) is because of the definition of threshold :: Cj (T(**) (T((T)) = a. The Eq. (28 follows that
r(to) =(k) () - 1//
(28)
Continuing from Eq. 28 holds for all y E y that
P(k E (o)),tt), $ C(x,
P(S(x,y, <()) r(to)
P(r(F)) S(x, ((1)) E(k, {²)}
2 P(r((") > S(x,, - (")))r))))))) - 1// -c55p11/77
@ (T(()) G (r(t")
((e((,) + p//9 2 el (2((k,
where (a) comes from the construction of prediction set: C(x) if and only if S(æ, v) S T. (b) is because of the definition of e. (c) and (d) is due to the definition of G and assumption (1).
Theorem F.7. Under the assumption equation 25 there crists constants c1, (0, 1] such that
E [[[ t)]l K
Σ t]),( Vt E (0.
xEX
24

---

Published in Transactions on Machine Learning Research (06/2025)
Proof. For all t to, we consider the expectation size of C(x, t):
E
11,,
= Σ
E
1sk E ((),
αEA
kEY
Σ
E
[1 E C(),
ZEx
kEY
Σ
P( E C(a,1))
KEY
Σ
P(k & C(æ, (H.
kEY
Due to the fact that
P(k tox, k & C(x,t)) S P(k & C(x,t)),
we have
E [1 OI < Σ1 P(kk E C(x, 10), h & C(., ())]
=EX
kEY
Continuing from Lemma
F.6 we can get
E [[(((()))]
t)l s K(1 e1 (2c(kk))))
K --Σ ())
.EX
kEY
25

---

Published in Transactions on Machine Learning Research (06/2025)
G Pseudo-algorithms of ConfTS, ConfPS and ConfVS
In this section, we present the pseudo-algorithms of the proposed methods, including ConfTS (Algorithm ConfPS (Algorithm 2), and ConfVS (Algorithm 8 The essence of our method to train a logits rescaling function with respect to the ConfTS loss. The loss function can be replaced by ConfTr or other loss functions designed for various targets.
Algorithm Conformal Temperature Scaling (ConfTS)
Require:
Pre-trained model Validation set Dval { y:) 1e) Significance level a, learning rate 17 timits
N
Ensure: Optimal temperature t*
1: Split van into two equal subsets Dioes = {1 Bi) and D Doonf = {(1,, v:))
2: t + tinst
for i do
(x)(f)(()//t)
Compute calibrated probabilities: (2,,1';t) Σ exp(f)(x)/0)
for all y' [l....,K}
for each data sample (2i, ys) E Dval do Compute the non-randomized APS score with respect to n(0, (8(())²
end for
8: Compute the conformal threshold r()) as the [(+1)((-o) quantile of scores in Deout: (5:(1))?a
Compute empirical risk of ConfTS loss on Dosss:
R(t) (Co
(T(() S(()))
71
n
10: t- a 81 R(t)
II: end for
12: return
Algorithm 2 Conformal Platt Scaling (ConfPS)
Require: Pre-trained model f, Validation set Dval {2 9i) 1m Significance level a, learning rate 7), Cinits
binit, N
Ensure: Optimal parameters a" and b*
1: Split Dval into two equal subsets Dioes {1ri, 9:) Wi and Dronf ((x;, 1i))
2: a + ainits b + bivit
3: for i= to do
expla-f, for all v/ [I,...,KK
Compute calibrated probabilities: a',, = Σ E fxp(a-f)(x)+t)
5: for each data sample (s) E D do Compute the non-randomized APS score with respect to v:a b): (a, b)}2
end for
Compute the conformal threshold r(a, b) the [(+1)(1-0]] quantile of scores in Doonti (Si(a,)))2)
Compute empirical risk of ConfTS loss on Dioss:
A(aa b) =
1 Σ
LConTTs (a,,
(T(a, 6) (a( Σ
11
n 1=1
de 10: a A b), b b+ b b- 17 36 fi b)
11: end for
12: return b
26

---

Published in Transactions on Machine Learning Research (06/2025)
Algorithm Conformal Vector Scaling (ConfVS)
Require: Pre-trained model f, Validation set. Dval = {(0 yi)) Significance level a, learning rate f)
Winit: binits N
Ensure: Optimal matrix M and vector b* Split Vall into two equal subsets Dioss Di)l= and Doont ((111 yi))
dinits b 4- binit for i- to N do
exp(M
4: Compute calibrated probabilities: T(2 ;' Σ (xp(M-fz(o)+b))
for all s' fl..... K}
5: for each data sample (z),y:) E Daal do
6: Compute the non-randomized APS score with respect to T(., M, b): (5,
7: end for
8: Compute the conformal threshold r(M, b) as the [x++)(1-0]] quantile scores in Doont {S:(M, 6)
Compute empirical risk of ConfTS loss on Dosss
Coonfrs 96 (.. = 1 Σ
R(M,b)- = Σ
(T(M, S,(,
1=1
1=1
10:
M < M 7. aM R(M,b), b b-- R(
11: end for
12: return M,
Results of ConfTS on ImageNet-V2
In this section, we show that ConfTS can effectively improve the elficiency of adaptive conformal prediction on the ImageNet-Ve V2 dataset. In particular, we employ pre-trained ResNet50, DenseNet121. VGG16, and ViT-B-16 on ImageNet. We leverage APS and RAPS to construct prediction sets and the hyper-paramcters of RAPS are set to be kieg and x 0.001. In Table 7 results show that after being employed with ConfTS, APS, and RAPS tend to construct smaller prediction sets and maintain the desired coverage.
Table 7: The performance comparison of conformal prediction with baseline and ConfTS under distribution shifts. *** denotes significant improvement (Iwo-sample (-lest at a 0.1 confidence level). "4" indicates smaller values are better. Bold numbers are superior results. Results show that ConfIS can improve the efficiency of APS and RAPS on new distribution.
ResNet50
DenseNet121
VGG16
VIT
Metrics
Baseline
ConfTS
Baseline
ConfTS
Baseline
ConfTS
Baseline
ConfTS
Avg.size(APS)
24.6
11.9*
50.3
13.3*
27.2
17.9*
34.2
10.1*
Coverage (APS)
0.90
0.90
0.90
0.90
0.90
0.90
0.90
0.90
Avg.size
13.3
11.3*
13.7
9.67*
16.3
13.6"
14.9
4.62*
(Roerage(RAAS))
0.90
0.90
0.90
0.90
0.90
0.90
0.90
0.90
Results of ConfTS on CIFAR-100
In this section, we show that ConfIS can effectively improve the efficiency of adaptive conformal prediction the CIFAR100 datasct. In particular, train 1esNet11. ResNet50, ResNet191, ResNext50, ResNext101, DenseNet121 12 and VGG16 from scratch CIFAR-100 datasets. We leverage APS and RAPS to generate prediction sets at error rates a E (0.1,0.05), The papper-prrmmter for RAPS is set to be kyeg = and A = 0.001. In Table results show that after being employed with ConfIS, APS, and RAPS tend to construct smaller prediction sets and maintain the desired coverage.
27

---

Published in Transactions on Machine Learning Research (06/2025)
Table 8: The performance comparison the baseline and ConfTS on CIFAR-100 100 dataset. We employ five models trained on CIFAR-100. **" denotes significant improvement (two-sample t-test at a 0.1 confidence level). "4" indicates smaller values are better. Bold numbers are superior results. Results show that our ConfTS can improve the performance of APS and RAPS, maintaining the desired coverage rate.
0.1
a 0.05
Model
Score
Coverage
Average |
size
Coverage
Average size 4
Baseline ConfTS
APS
0.902 / 0.901
7.049 / 6.547*
0.949
0.949
12.58 / 11.91 *
ResNet 18
RAPS
0.900
0.901
5.745 / 4.948*
0.949
0.949
8.180 / 7.689*
APS
0.901
0.900
5.614 / 5.322*
0.951 0.951
10.27 / 10.00"
ResNet50
RAPS
0.900
0.900
4.707 / 4.409*
0.951
0.950
7.041 / 6.811*
APS
0.900
0.900
5.049 / 4.917*
0.949 0.949
9.520 / 9.405
Res.Ve1101
RAPS
0.901 / 0.900
4.324 / 4.145*
0.950 0.950
6.515 / 6.450*
APS
0.900
0.900
1.668
4.436*
0.950 0.950
8.911 / 8.626*
ResNext50
RAPS
0.901 / 0.901
4.050 / 3.811*
0.951 0.951
6.109 / 5.854*
APS
0.900
0.900
4.125
/ 3.988*
0.950 0.950
7.801 / 7.614*
ResNext101
RAPS
0.901 / 0.901
3.631 / 3.492*
0.950 0.950
5.469 / 5.253*
APS
0.899
0.899
4.401 / 3.901
0.949 0.949
8.364 / 7.592*
DenseNet121
RAPS
0.898
0.898
3.961 / 3.434*
0.950
0.949
6.336 / 5.222"
APS
0.900
0.900
7.681 / 6.658*
0.949
0.950
12.36 / 11.70*
VGG16
RAPS
0.899
0.900
6.826
5.304*
0.949
0.949
10.32"
11.70
J Results of ConfTS on RAPS with various penalty terms
Recall that the RAPS method modifies APS by including a penalty term À (see Eq. 16 In this section, we investigate the performance of ConfTS RAPS with various penalty terms. In particular, employ the same model architectures with the main experiment on ImageNet (see Section 4.1 and generate prediction sets with RAPS (Kreg 1) at an error rate a 0.1, varying the penalty À E (0.002, 0.004, 0.006, 0.01, 0.015, 0.02] and setting kreg to 1. Table and 10 show that our ConfTS can enhance the efficiency of RAPS across various penalty values.
Table 9: The performance of ConfTS on RAPS with various penalty terms E f0.002, 0.004, 0.006) ImageNet. a*" denotes significant improvement (two-sample t-test a 0.1 confidence level). "4" indicates smaller values are better. Bold numbers superior results. Results show that our ConfTS can enhance the efficiency of RAPS across various penalty values.
0.002
À 0.004
0.006
Model
Coverage
Average size
Coverage
Average size
Coverage
Average size !
Baseline / ConfTS
ResNet18
0.901 / 0.900
8.273
4.517*
0.901 / 0.901
6.861 / 4.319*
0.901
0.901
6.109
4.282*
ResNet50
0.899 / 0.900
5.097
3.231*
0.899 / 0.900
4.272 / 2.892*
0.899
0.900
3.858 / 2.703*
ResNet101
0.900 / 0.900
4.190
2.987*
0.901 / 0.899
3.599 / 2.686*
0.900
0.900
3.267
2.516*
DenseNet121
0.901 / 0.901
5.780
3.340*
0.900 / 0.900
4.888 / 3.014*
0.900
0.900
4.408
2.836*
VGG16
0.901
0.900
7.030
3.902*
0.901 / 0.900
5.864 / 3.514*
0.901
0.900
5.241
3.344
VTT-B-16
0.901 / 0.900
5.308
1.731
0.901 / 0.901
4.023 / 1.655*
0.901
0.901
3.453 / 1.611*
28

---

Published in Transactions on Machine Learning Research (06/2025)
Table 10: The performance of ConfTS on RAPS with various penalty terms E {0.01, 0.015, 0.02) at ImageNet. ax* denotes significant improvement (two-sample t-test at a 0.1 confidence level). "4" indicates smaller values are better. Bold numbers are superior results. Results show that our ConfIS can enhance the efficiency of RAPS across various penalty values.
0.01
À 0.015
0.02
Model
Coverage
Average size
Coverage
Average size +
Coverage
Average size 4
Baseline / CoufI'
ResNet18
0.901 / 0.901
5.281 / 4.449*
0.901 / 0.901
4.712 / 4.683*
0.900
0.900
4.452* / 4.917
ResNet50
0.899 / 0.900
3.380
2.505*
0.900 / 0.901
3.048 / 2.373*
0.901
0.901
2.860
2.321 "
ResNet101
0.900 / 0.900
2.902
2.317*
0.900 / 0.899
2.643 / 2.168*
0.900
0.900
2.484
2.096*
DeuseNet121
0.900 / 0.900
3.843
2.657*
0.900 / 0.900
3.452 / 2.587*
0.901
0.899
3.213
2.750
VGG16
0.900 / 0.900
4.537
3.371*
0.900 / 0.900
4.060 / 3.423*
0.899
0.899
3.744
3.530*
ViT-B-16
0.901 / 0.900
2.872
1.564*
0.901 / 0.900
2.508 / 1.543*
0.900
0.900
2.285 / 1.535"
K
Results of ConfTS on SAPS
Recall that APS calculates the non-conformity score by accumulating the sorted softmax values in descending order. However, the softmax probabilities typically exhibit long-tailed distribution, allowing for easy inclusion of those tail classes in the prediction sets. To alleviate this issue, Sorted Adaptive Prediction Sets (SAPS) (Huang et al., (024) discards all the probability values except for the maximum softmax probability when computing the non-conformity score. Formally, the non-conformity score of SAPS for a data pair (æ, y) can be calculated as
{ u Amar (z), if oly.a(x)) (x)) = 1, Amaz + f())) - u) X, else,
S S2o U;
where A is a hyperparameter representing the weight of ranking information, Amaz (x) denotes the maximum softmax probability and u is a uniform random variable.
In this section, we investigate the performance of ConfIS on SAPS with various weight terms, In particular, we employ the same model architectures with the main experiment ImageNet (see Sention 4.1 and generate prediction sets with SAPS at an error rate 0.1, varying the weight À {0.01, 0.02, 0.03, 0.05, 0.1, 0.12). Table 11 and Table 12 show that our ConfTS can enhance the efficiency of SAPS across various weights.
Table 11: The Performance of ConfTS on SAPS with various penalty terms X E 00.055, 0.01, 0.02]. denoles significant improvement (two-sample t-test at a 0.1 confidence level). "I" indicates smaller values are better. Bold numbers are superior results. Results show that our ConfIS can enhance the efficiency of SAPS across various penalty values.
A 0.005
A 0.01
A= = 0.02
Model
Coverage
Average size 1
Coverage
Average size
Coverage
Average size
Baseline / CoufTS
ResNet18
0.901 / 0.900
37.03
27.38*
0.901 / 0.902
19.91 / 14.81*
0.900
0.901
11.21
8.469*
ResNet50
0.899 / 0.899
27.13
21.37*
0.899 / 0.899
14.45 / 11.48*
0.899
0.899
8.016 / 6.510*
ResNet101
0.901 / 0.901
24.89
20.78*
0.901 / 0.901
13.21 / 11.16*
0.901
0.901
7.350 / 6.287*
DenseNet121
0.900 / 0.901
30.54
22.67*
0.900 / 0.901
16.28 / 12.30*
0.901
0.901
9.085 / 6.968*
VGG16
0.900 / 0.900
34.88
25.57*
0.900 / 0.900
18.56 / 13.71*
0.901
0.900
10.34 / 7.788*
ViT-B-16
0.901
0.900
18.90
11.51*
0.901 / 0.900
10.11 / 6.379*
0.900
0.900
5.669
3.784"
Average
0.900 / 0.900
28.89
21.54
0.900 / 0.900
15.42 / 11.63*
0.900
0.900
8.611 / 6.634*
29

---

Published in Transactions on Machine Learning Research (06/2025)
Table 12: The performance of ConfTS on SAPS with various penalty terms E {0.03, 0.05, 0.13. .** denotes significant improvement (two-sample t-test at a 0.1 confidence level). "4" indicates smaller values are better. Bold numbers are superior results. Results show that our ConfI'S can enhance the efficiency of SAPS across various penalty values.
0.03
0.05
À
Model
Coverage
Average size
Coverage
Average size +
Coverage
Average size +
Baseline
CoufI'
ResNet18
0.900 / 0.900
8.206
6.269*
0.900 / 0.900
5.747 / 4.716*
0.901
0.901
4.143* / 4.381
ResNet50
0.899
0.899
5.853
4.838*
0.899 / 0.900
4.122 / 3.464*
0.899
0.900
2.753
2.460"
ResNet101
0.901
0.901
5.364
4.640*
0.901 / 0.901
3.756 / 3.293*
0.899
0.900
2.511
2.286*
DenseNet121
0.900 / 0.900
6.600
5.151*
0.900 / 0.900
4.601 / 3.672*
0.900
0.900
3.063 / 2.811"
VGG16
0.900
0.900
7.504
5.785*
0.900 / 0.900
5.225
/ 4.173*
0.900
0.900
3.483*
3.551
ViT-B-16
0.900
0.900
4.197
2.905*
0.900 / 0.900
2.995 / 2.212*
0.901
0.900
2.114 / 1.768
Average
0.900 0.900
6.287
4.931*
0.900 / 0.900
4.407
3.588*
0.900
0.900
3.011
2.909
L Calibration performance of ConfTS, ConfPS, and ConfVS
In this part, we demonstrate the tradeoff between the ECE and the prediction set size by comparing standard scaling methods - lemperature scaling (TS), Platt scaling (PS), and vector scaling (VS) - to their conformal variants ConfTS, ComfPS, and ConfVS. We conduct the experiment on ImageNet dataset, with ResNet50 and DenseNet121. We generate prediction sets with APS score and error rate a = 0.1. The results in Table [3 show that TS, PS, and VS generate large prediction aligned with our finding in Section 3.1 In contrast, ConfTS, ConfPS, and ConIVS essentially reduce the prediction set size with higher ECE. The results are intuitive as our methods are designed for conformal prediction, instead of confidence calibration. Importantly, our method does not conflit with confidence calibration, as it only replaces the temperature value. During inference, one may use different temperature values according to the objective, whether for improved calibration performance or efficient prediction sets.
with lower ECE (i.c., better calibration) that are
Table 13: The ECE results of ConITS, ConIPS and ConIVS ImageNet dataset.
Model
ResNet50
DenseNet121
Method
Tuned T
ECE
Avg.size
Tuned T
ECE
Avg.size
Base
1.000
4.36
9.062
1.000
3.66
9.271
TS
1.140
3.02
12.29
1.081
2.93
12.06
ConfTS
0.705
47.78
4.791
0.659
45.63
4.746
PS
2.55
12.49
2.34
10.72
ConfPS
19.23
2.57
21.6
3.169
VS
2.44
11.29
2.69
11.33
ConfVS
12.05
4.56
12.9
5.345
30


Reviews:
[{"review_id": "HKDJwAV3N6", "reviewer": "Reviewer_UXE6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This manuscript explores an important area where two UQ paradigms, calibration and conformal prediction, fruitfully overlap. Specifically, it explores the effect of calibrating a multiclass predictor ,via one of the oft-used heuristic post-hoc calibration methods, on the size of downstream conformal prediction sets, when scores like APS and RAPS (which directly take in model probabilities) are sued. This question has to my knowledge somewhat flown under the radar, in comparison to the more direct question of optimizing the model outputs for the purposes of conformal prediction efficiency (as developed in Stutz et al (2022) and follow-up works). As such, I believe this paper is a timely contribution.\n\nThe method ConfTS is conceptually very straightforward, and yet is shown to provide significant efficiency increases in almost all of the performed experiments. Moreover, the authors show that it can even improve over the method of Stutz et al, which employs a generally more complex pipeline to obtain its results. This simplicity should lead to good illustrative value for the readers interested in calibration methods that can improve prediction set sizes.\n\nOn the theory side, the paper demonstrates the following simple fact: taking the sum of the softmax vector entries corresponding to top-k predicted probabilities (where k and the probabilities are fixed), driving the temperature down monotonically decreases this sum. Under some regularity assumptions, this then leads to the conclusion that the efficiency of the prediction sets under APS can be tuned by tuning the temperature. While the softmax fact is common knowledge in the intuitive sense that as t -> 0, the prediction set will converge to including only the top label due to the probability mass concentrating there, but to my best knowledge this fact has not been explicitly stated/derived before. Thus, even though this fact is quite simple (as a matter of fact, it can be obtained simply by taking the derivative of the softmax, so the longer argument in the paper is not necessary) but I do think it adds some value to the literature to have it written down in this form.\n\nFurthermore, the paper is quite well-written, and does a good job explaining the setup and executing on the experiments. \n\nTo sum up, while the paper's contributions are quite simple and straightforward at first glance, to my knowledge such a study has not been explicitly performed before, and coupled with the paper's writing it should have value for the TMLR audience.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a study of the interaction between conformal prediction set size efficiency and heuristic calibration methods in multiclass prediction settings. It takes the APS nonconformity score and temperature scaling as a starting point, and mathematically shows that as temperature decreases to 0, the average prediction set size under non-randomized APS goes down. Furthermore, it presents several experiments that further support this point, for both APS and RAPS nonconformity scores and for several other calibration methods (including Platt scaling and mixup). \n\nNext, extending the insight of the relationship between the temperature and conformal set efficiency, it proposes a loss function and method (that consists in minimizing this loss) called ConfTS (conformal temperature scaling). This method (as well as its generalization that optimizes for the parameters of other calibration methods such as Platt scaling) is then empirically tested on a suite of multiclass experiments, showcasing the improvements in conformal set efficiency achieved by ConfTS.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As one \"weakness\", I would like to point out the empirical evaluation of ConfTS. Specifically, I would like to see how the proposed method ConfTS affects various calibration metrics of the underlying models, in comparison to optimizing the temperature/the parameters of the calibration methods for the sake of calibration error decrease. In other words, what is the calibration suboptimality incurred by ConfTS in order to achieve improvements in the prediction set sizes? This could be fruitfully illustrated, for the benefit of the reader, by e.g. depicting some Pareto frontiers showing the tradeoff between ECE and prediction set sizes as a function of the parameter (e.g. temperature).\n\nFurthermore, admittedly the experiments presented (especially at the beginning of the paper before ConfTS was introduced) were executed on somewhat old tasks. That could also be improved, especially as it is empirically known that calibration of NNs out of the box has gone up since Guo et al's paper in 2017.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9Y3EBtuHpc", "reviewer": "Reviewer_gbFS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths and Weaknesses\n\nThe core idea of the paper is conveyed in a logical and coherent order:\n\n- An initial empirical study reveals that the commonly held belief — that standard calibration methods should be used in combination with conformal prediction (CP) — might be misfounded if **interval size** is of concern. (And of course it should be: completely trivial sets can achieve nominal coverage.)\n- It appears that existing calibration methods tend to produce lower-confidence predictions — although this is not explicitly stated (see my related requested change).\n- This motivates the next section, where an empirical study on **high-confidence predictions** (via low-temperature settings in temperature scaling) shows that tighter prediction sets can be obtained.\n- A theoretical explanation is then provided to account for this phenomenon — again, in the **specific context of temperature scaling**.\n- The paper concludes by proposing a new loss function for selecting the calibration temperature, introducing what is effectively a **new family of methods**.\n- Finally, an extensive empirical study is conducted on these methods. The empirical performance is very convincing.\n\nThat said, there is **insufficient emphasis** on the fact that both the empirical study in Section 3.2 and the theoretical study in Section 3.3 pertain **only to temperature scaling**.\n\nThis weakens the accuracy of broad claims such as the one made in the first bullet point of the contributions:\n\n> “We discover that current confidence calibration methods typically lead to larger prediction sets in adaptive conformal prediction, while high-confidence predictions (using small temperatures) can enhance the efficiency of prediction sets.”\n\nYou provide evidence for the second clause — that high-confidence predictions yield tighter sets — in the context of **temperature scaling, Platt scaling, and vector scaling**. This has **not** been demonstrated for the **other calibration methods** explored in Table 1. These distinctions should be made clearer to avoid overgeneralization of your findings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary of Contributions\n\nThe paper investigates the commonly held practice of applying **confidence calibration** methods (e.g., temperature scaling) prior to performing **adaptive conformal prediction (CP)**. It challenges this convention by demonstrating, both empirically and theoretically, that such calibration methods can **adversely impact the efficiency** (i.e., tightness) of prediction sets in CP, even while maintaining conditional (?) coverage.\n\nTo address this, the authors propose a novel loss function — **Conformal Temperature Scaling (ConfTS)** — which directly optimizes for the tightness of the prediction set. This loss function replaces the typical negative log-likelihood used to select temperature parameters in post-hoc calibration methods.\n\nImportantly, this approach is **modular** (though there is missing emphasis on this) and can be integrated into **temperature scaling, Platt scaling, and vector scaling**. When used as a precursor to CP, these modified calibrators result in significantly **tighter prediction sets** without sacrificing coverage.\n\nExtensive experiments across multiple datasets and model architectures in both image and text classification settings support the tightness of their method. The paper also presents theoretical insights and ablation studies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I regard responses to these as crucial, but I am open to understanding why some of my proposed changes should not be actioned.\n\n### Abstract\n- It needs to be made more clear that current calibration methods result in higher temperatures, which lead to larger prediction sets in adaptive conformal prediction (CP).  \n- Then, the second point you make — about high-confidence predictions (resulting from low temperatures) — can be more clearly situated in this context.  \n- There are several places in the abstract where a prediction set is described as “efficient” or the word “efficiency” is otherwise employed to the same effect. I think this term is far too vague and could be conflated with computational efficiency to the casual reader.\n\n### Introduction\n- Referring to temperature scaling (Guo et al., 2017) and BNNs (Smith, 2013), it is written that “these approaches lack theoretical guarantees of model performance.”  \n  - It is unclear what exactly is meant by this.  \n- The next sentence sets up a vague contrast: “conformal prediction, on the other hand, …”  \n  - This contrast would benefit from being made more precise.\n\n### Definition of Marginal Coverage (around Equation 2)\n- It is not specified what the probability is over, making it impossible for readers to understand what is “marginal” about marginal coverage.\n\n### Missing Definition of Conditional Coverage\n- Immediately below Equation (4), it is written that the paper will focus on adaptive CP, which is “designed to improve conditional coverage.”  \n- If this is your focused notion of coverage, it is odd that there is no accompanying definition, especially when one is provided for marginal coverage in Equation (2).\n\n### Consistent Usage of Marginal vs. Conditional Coverage\n- By the end of Section 2, the reader understands there are two distinct notions of coverage: marginal and conditional.  \n- When you make statements henceforth such as “Notably, both methods incorporate a uniform random variable \\( u \\) to achieve exact \\( 1 - \\alpha \\) coverage,” it seems important to clarify which type of coverage is being referred to.  \n- There are multiple other such instances. If the remainder of the paper will only consider conditional coverage, please state this explicitly and note that “coverage” will refer to that notion from then on.\n\n### Section 3.1: Definition of Expected Calibration Error (ECE)\n- The definition of the ECE at the beginning of Section 3.1 is incomplete for readers not already familiar with it.  \n- Either define it completely or omit the partial definition and instead reference a standard source.\n\n### Table 1\n- The triangles indicating superior/inferior performance are rather distracting, especially since the directionality of the arrows is reversed for ECE and Avg Size metrics.  \n- It would be helpful to reconsider the use of these symbols altogether especially you use the convention of boldface later on to express superiority.\n\n- Additionally, I’m not familiar with all the confidence calibration methods listed.  \n  - Do they all involve choosing some type of temperature parameter?  \n  - If so, then the typical value of the temperature should be reported so that we can observe the tendency to select high-temperature (low-confidence) predictions.  \n  - This would help set up Section 3.2 more clearly and help the reader appreciate why you study low-temperature (high-confidence) predictions.\n\n### Extensions to Other Post-Hoc Calibration Methods\n- It is written that ConfTS can be extended to Platt scaling and vector scaling.  \n- What about the other calibration methods in Table 1?  \n- Can you set out more general conditions on the calibration method under which ConfTS can be applied?\n\n### ConfTR\n- I was surprised to learn about an important related work, ConfTR, only on page 10 in a short paragraph within the experiments section.  \n- It was even more confusing to learn that ConfTS can be trained with either the ConfTR loss or the ConfTS loss.  \n- If the method ConfTS can be defined via alternative losses, then this suggests a level of modularity that is not reflected in the naming. This modularity should be clarified earlier in the paper.\n\n### Pseudo-Algorithms\n- I think it will be quite helpful to include pseudo-algorithms for your ConfTS/PS/VS methods.  \n- These should illustrate how the loss function employed (e.g., ConfTR loss vs. ConfTS loss) affects the method.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CQxU0ndZAX", "reviewer": "Reviewer_FtzM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is well-written and easy to follow\n* The figures and tables are presented in a way that is easy to interpret\n\nWeaknesses: \n* The paper currently ignores a key motivation for using adaptive score functions like APS and RAPS—namely, their ability to achieve approximate X-conditional coverage. There is no discussion or computation of X-conditional coverage, even in a rough way (e.g., defining groups in feature space and computing group-conditional coverage). If you are in a setting where you do not care about X-conditional coverage and only care about set size and marginal coverage, then you could just use 1-softmax (what the paper calls the LAC score) as your conformal score function, as this score tends to give smaller sets than APS or RAPS.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the affect of temperature scaling (TS) on the average set size of conformal prediction sets generated using the APS and RAPS score functions. The authors find that TS with lower temperature values generally results in smaller sets on average (the only exception is at very small temperature values at which point numerical precision becomes an issue). They then propose an alternative to post-processing softmax vectors before applying APS: rather than apply TS, one can apply “Conformal Temperature Scaling” which optimizes the TS parameter to minimize the “efficiency gap,” leading to smaller set sizes. They present empirical results on several image and NLP datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical:\n1. There should be some attempt to compute X-conditional coverage and consideration of X-conditional coverage should be included from the beginning. Currently, the paper presents reductions in set size as a unilateral gain without consideration of the tradeoff with X-conditional coverage. The problem that is studied is an interesting one, but the paper in its current state does not capture the full story.\n\n2. It is not obvious to me that Conformal Temperature Scaling (Eq 8) actually yields calibration. Do you have a table that reports the ECE after applying ConfTS? If ECE is not improved by ConfTS relative to no-calibration, then it should be emphasize that ConfTS is not being proposed as a general purpose alternative to TS but rather as a pre-processing step only applicable before applying APS/RAPS.\n\nSmall comments:\n* To aid in interpretation of Figure 1, perhaps there could be a brief remark about what is the range of temperature values that sharpens vs. flattens the softmax vector, relative to no calibration.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)