On Temperature Scaling and Conformal Prediction of Deep Classifiers
Authors: Lahav Dabah
Tom Tirer
Venue: ICML 2025 poster (2025)
Keywords: classification
temperature scaling
conformal prediction
conditional coverage
prediction sets
Primary area: deep_learning->everything_else
TL;DR: We theoretically and empirically analyze the impact of temperature scaling beyond its usual calibration role on key conformal prediction methods.

Abstract: In many classification applications, the prediction of a deep neural network (DNN) based classifier needs to be accompanied by some confidence indication. Two popular approaches for that aim are: 1) *Calibration*: modifies the classifier's softmax values such that the maximal value better estimates the correctness probability; and 2) *Conformal Prediction* (CP): produces a prediction set of candidate labels that contains the true label with a user-specified probability, guaranteeing marginal coverage but not, e.g., per class coverage.  In practice, both types of indications are desirable, yet, so far the interplay between them has not been investigated. Focusing on the ubiquitous *Temperature Scaling* (TS) calibration, we start this paper with an extensive empirical study of its effect on prominent CP methods. We show that while TS calibration improves the class-conditional coverage of adaptive CP methods, surprisingly, it negatively affects their prediction set sizes. Motivated by this behavior, we explore the effect of TS on CP *beyond its calibration application* and reveal an intriguing trend under which it allows to trade prediction set size and conditional coverage of adaptive CP methods. Then, we establish a mathematical theory that explains the entire non-monotonic trend.
Finally, based on our experiments and theory, we offer guidelines for practitioners to effectively combine adaptive CP with calibration, aligned with user-defined goals.

On Temperature Scaling and Conformal Prediction of Deep Classifiers
Lahav Dabah
Tom Tirer!
Abstract
ciated with each prediction of the DNN. This is crucial in high-stakes scenarios where human lives are at risk, such as medical diagnoses (Miotto et al., 2018) and autonomous vehicle decision-maie (Grigorescu al., 2020).
In many classification applications, the prediction of a deep neural network (DNN) based classifier needs to be accompanied by some confidence in- dication. Two popular approaches for that aim are: 1) Calibration: modifies the classifier's softmax values such that the maximal value better esti- mates the correctness probability; and 2) Confor- mal Prediction (CP): produces prediction set of candidate labels that contains the true label with a user-specified probability, guaranteeing marginal coverage but not, e.g., per class coverage. In prac- tice, both types of indications are desirable, yet. so far the interplay between them has not been investigated. Focusing on the ubiquitous Temper- ature Scaling (TS) calibration, start this paper with an extensive empirical study of its effect on prominent CP methods. We show that while TS calibration improves the class-conditional cov- crage of adaptive CP methods, surprisingly, it negatively affects their prediction set sizes. Mo- tivated by this behavior, we explore the effect of TS on CP beyond its calibration application and reveal an intriguing trend under which it al- lows to trade prediction set size and conditional coverage of adaptive CP methods. Then, we es- tablish mathematical theory that explains the entire non-monotonic trend. Finally, based on our experiments and theory, we offer guidelines for practitioners to effectively combine adaptive CP with calibration, aligned with user-defined goals.
In practice, DNN classification models typically generate post-softmax vector, akin to probability vector with non- negative entries that add up one. One might, intuitively, be interested in using the value associated with the prediction as the confidence (Cosmides & Tooby, 1996). However, this value often deviates substantially from the actual correct- ness probability. This discrepancy, known as miscalibration, is prevalent in modern DNN classifiers, which frequently demonstrate overconfidence: the maximal softmax value surpasses the true correctness probability (Guo et al., 2017), To address this issue, post- post-processing calibration methods are employed to adjust the values of the softmax vector. In particular, Guo et al. (2017) demonstrated the usefulness of simple Temperature Scaling (TS) procedure single parameter variant of Platt scaling (Platt al., 1999)). Since then, TS calibration has gained massive popularity (Liang et al., 2018; Ji et al., 2019; Wang et al., 2021; Frenkel & Goldberger, 2021; Ding et al., 2021; Wei et al., 2022). Thus studying it of high significance.
Another post-processing approach for uncertainty indica- tion is Conformal Prediction (CP), which was originated in (Vovk al., 1999; 2005) and has attracted much attention recently. CP algorithms are based on devising scores for all the classes per sample (based on the softmax values) that are used for producing set of predictions instead of a single predicted class. These methods have theoretical guarantees for marginal coverage: given a user-specified probability, the produced set will contain the true label with this prob- ability, assuming that the data samples are exchangeable (c.g., the samples are i..d.). Note that this property does not ensure conditional coverage, i.e., coverage of the true label with the specified probability when conditioning the data, e.g., to a specific class. Consequently, CP methods are usually compared by both their prediction set sizes and their conditional coverage performance.
1. Introduction
Modern classification systems are typically based on deep neural networks (DNNs) (Krizhevsky et al., 2012; He et al., 2016; Huang et al., 2017). In many applications, it is nec- essary quantify and convey the level of uncertainty asso-
'Faculty of Engineering, Bar-Ilan University, Ramat Gan, Israel, Correspondence to: Lahav Dabah <lahavdabah @ gmmi..com>. comb>
Clearly, in critical applications, both calibration and CP are desirable, as they provide complementary types of informa- tion that can lead to comprehensive decision. However, so far the interplay between them remains largely unexplored.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
Specifically, the works (Angelopoulos al., 2021; Lu al., 2022; Gibbs et al., 2023; Lu et al., 2023) apply initial TS calibration (rather than any other calibration method) before applying their CP methods. Yet, none of them investigates what is the effect of this procedure on the CP methods.
x E R produces a logits vector z =2(x)= RC that is
fed into a final softmax function a R RC defined
exp (24)
as o, (2)
Σj= exp (zz) Typically, the post-softmax
vector +(x) = (z(x)) is being treated as an estimate of the class probabilities. The predicted class is given by (xx = argmax, (x(
In this work, we study the effect of TS, arguably the most common calibration technique, on three prominent CP meth- ods: Least Ambiguous set-valued Classifier (LAC) (Lei & Wasserman, 2014; Sadinle al., 2019), Adaptive Prediction Sets (APS) (Romano et al., 2020), and Regularized Adap- tive Prediction Sets (RAPS) (A (Angelopoulos et al., 2021). Note that LAC (aka THR), APS and RAPS are probably the three most popular CP methods for classification. Indeed, important papers in this field, such as (Angelopoulos et al., 2021) and (Stutz et al., 2022), do not consider any other CP method. Our discoveries on the effect of TS calibration in this paper have motivated us to explore the TS mechanism also beyond its calibration application.
2.1. Calibration and Temperature Scaling
The interpretation of # (x) as an estimated class probabili- ties vector promotes treating Tg(x) (x) as the probability that the predicted class j(x) is correct, also referred to as the model's confidence. However, it has been shown that DNNs are frequently overconfident - - is larger than the true correctness probability (Guo et al., 2017). Formally, P (ū(X) = Y f p) < p with significant margin. Post- prosessoosssing calibration techniques aim reducing the aforementioned gap. They are based optimizing certain transformations of the logits z(-). yielding probability vector # +-) that minimizes an objective computed over a dedicated calibration set of labeled samples fx,, Vi W (Platt et al., 1999; Zadrozny & Elkan, 2002: Naeini et al., 2; Nixon et al., 2019). Two popular calibration objec- tives are the Negative Log-Likelihood (NLL) (Hastic et al., 2005) and the Expected Calibration Error (ECE) (Naeini et al., 2015), detailed in Appendix B.3.
Our contributions can be summarized as follows:
We conduct an extensive empirical study on DNN classifiers that shows that initial TS calibration af- fects CP methods differently. Specifically, we show that its effect is negligible for LAC, but intriguing for adaptive methods (APS and RAPS): while their class- conditional coverage is improved, surprisingly, their prediction set sizes typically become larger.
Temperature Scaling (TS) (Guo et al., 2017) stands out as, arguably, the most common calibration approach, sur- passing many others in achieving calibration with minimal computational complexity (Liang et al., 2018; Ji et al., 2019; Wang et al., 2021; Frenkel & Goldherger, 2021; Ding et al., 2021; Wei al., 2022). It simply uses the transformation + z/T before applying the softmax, where T > 0 (the temperature) is a single scalar parameter that is set by mini- mizing NLL or ECE. Importantly, TS preserves the accuracy rate of the network (the ranking of the elements and in particular, the index of the maximum - is unchanged).
Following these findings, we investigate the impact of TS on CP for wide range of temperatures, beyond the values used for calibration. We reveal that by modifying the temperature, TS enables to trade the prediction set sizes and the class-conditional coverage performance for RAPS and APS. Moreover, metrics of these properties display a similar non-monotonic pattern across all models and datasets examined.
We present a rigorous theoretical analysis of the impact of TS on the prediction set sizes of APS and RAPS, offering comprehensive explanation for the complex non-monotonic patterns observed empirically.
Hereafter, we use the notation (x) := (z(x)/T) to denote the output of the softmax when taking into account the temperature. Observe that T = 1 preserves the original probability vector. Let us denote by T the temperature that is optimal for TS calibration. Since DNN classifiers are commonly overconfident, TS calibration typically yields some T > 1, which "softens" the original probability vector. Formally, TS with T > 1 raises the entropy of the softmax output (see Proposition A.5 in the appendix).
Based on our theoretically-heee backeed findings, we propose practical guidelines to effectively combine adaptive CP methods with TS calibration, which allow users to con- trol the prediction set sizes and conditional coverage trade-off.
2. Background and Related Work
The reliability diagram (DeGroot & Fienberg, 1983; Niculescu-Mizil & Caruana, 2005) is a graphical depiction of a model before and after calibration. In Appendix B. 2, we provide details on this procedure and display reliability diagrams for the dataset-model pairs examined our study.
Let us present the notations that are used in the paper, followed by some preliminaries on TS and CP. We con- sider a C-classes classification task of the data (X, Y) dis- tributed on Rd x [C], where [C] := {I..... .....,C C}. The clas- sification tackled by a DNN that for each input sample
2

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
2.2. Conformal Prediction
There have been also efforts alleviate the exchangeability assumption (Tibshirani al., 2019; Barber et al., 2023). In this paper we will focus three prominent CP methods. Each of them devises a different score s(x, y) based on the output of the classifier's softmax *(-)
Conformal Prediction (CP) is a methodology that is model- designed for generating prediction set of classes (X) for given sample X, such that Y E Ca (X) with probability 1 - a for predefined E (0, 1), where Y is the true class associated with X (Vovk et al., 1999; 2005; Papadopoulos et al., 2002). The decision rule is based on a calibration set of labeled samples fx, Vi ) which we hereafter refer to as the CP set, to avoid confusion with the set used for TS calibration. The only assumption in CP is that the random variables associ- ated with the CP set and the test samples are exchangeable (e.g., the samples are i.i.d.).
agnostic and distribution-fiee
Least Ambiguous Set-valued Classifier (LAC) (Lei & Wasserman, 2014; Sadinle et al., 2019). In this method, (x). Accordingly, given ÍLAC associ- ated with a through step 2, the prediction sets are formed as: CT.AC(xx : {v (x) > QrAAc). LAC tends to have small set sizes (under the strong assumption that (xx matches the posterior probability, it provably gives the smallest possible average set size). On the other hand, its conditional coverage limited.
s(x, y) =
Let us state the general process of conformal prediction given the CP set {x,,ya yi y=) and its deployment for a new (test) sample Fx+1 (for which yn++ is unknown), as pre- sented in (Angelopoulos & Bates, 2021):
A daptive Prediction Sets (APS) (Romano et al., 2020), The objective of this method is to improve the conditional coverage. Motivated by theory derived under the strong assumption that * (x) matches the posterior probability, it uses sxx, v) Σ i=1 (t()) (x), where (t()(x) denotes the i-th element in a descendingly sorted version of fxx and Ly is the index that is permuted to after sorting. Following steps 2 and 3, yields QAPs and CAPS (x).
1. Define heuristic score function s(x, y) E R based on some output of the model. A higher score should encode lower level of agreement between and y.
2. Compute @ as the [(n + 1)(1 - all
quantile of the
scores (s(x1, U1), ...,,...,,,.. SXx, )-
Regularized Adaptive Prediction Sets (RAPS) (An- gelopoulos et al., 2021). A modification of APS that aims improving its prediction set sizes by penalizing hard exam- ples to reduce their effect on q. With the same notation as
3. At deployment, use q to create prediction sets for new samples: Ca (x {y: : s(x.+1.8) v) S q)
by
CP methods possess the following coverage guarantee.
APS, in RAPS we have s(x, v)
i= (t)) (x) + X(Ly- krre))+ where X, kreg 2 0 are regularization hyperparame- ters and we use the notation (-)+ == max [-,0], 0}. Following steps 2 and 3, yields GRAPS and CRAPS(X)) (x).
Theorem 2.1 (Theorem in (Angelopoulos & Bates, 2021)). Suppose that {(X;, Y=))=z and (Xn+1, Yn+1) are i..d., and define as in step 2 above and Ca (X)+1) as in step 3
above. Then the following holds:
Note that all these CP methods can be readily applied on TT-(.) (.) after TS calibration. This is done, in (Angelopoulos et al., 2021; Lu et al., 2022; Gibbs et al., 2023; Lu et al., 2023), where the authors stated that they applied TS calibra- tion before examining the CP techniques. However, none of these works has examined how TS calibration impacts any CP method. All the more so, no prior work has exper- imented applying TS with a range of temperatures before employing CP methods.
P (Yn+1 E (C((Xx+)) 21 a.
(1)
The proof of this result is based on (Vovk et al., 1999). A proof of an upper bound of l-a+1/(+++)) also exists. This property called marginal coverage since the probability is taken over the entire distribution of (X,Y² Y).
Note that this property does not imply the much more strin- gent property of conditional coverage:
After completing our work, we became aware of a concur- rent work (Xi et al., 2024), which also studies the effect of TS on CP methods. Yet, their results are only a small subset of our results. They do not consider conditional coverage and explore only a limited range of temperatures, which masks the non non-monotonic effect the prediction set size of APS and RAPS (which they do not compare to LAC). In contrast, our paper provides a complete picture of the effect of TS across a wide range of temperatures on both the pre- diction set size and the class-conditional coverage of APS, RAPS, and LAC. This complete picture shows practitioners
P P(a++1 E
=
(2)
In fact, coverage for any value x of the random X is im- practicable (Vovk, 2012), (Foygel Barber et al., 2021), and useful intuitive relaxation is to consider class-conditional coverage.
CP methods are usually compared by the size of their predic- tion sets and by their proximity to the conditional coverage property. Over time, various CP techniques with distinct ob- jectives have been developed (Angelopoulos & Bates, 2021).
3

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
that tuning the temperature for APS and RAPS introduces trade-off effect, and we provide a practical way control it. Lastly, note that several theoretical results prediction sets and calibration have been reported in (Gupta et al., 2020). Yet, that work is limited to binary classification and does not provide an explanation for the fact that calibration affects CP methods differently and non-monotonic manner, as shown and analyzed in our paper.
et al., 2021; Angelopoulos & Bates, 2021).
Metrics. We re metrics over the validation set, which we denote by ((x! Vj ((aa) Mer comprising of the samples that were not included in the calibration set or CP set. The
metrics are as follows.
Average set size (AvgSize) - The mean prediction set size of the CP algorithm:
Nusl
AvgSize
((x((a)))) )|-
3. The Effect of TS on CP Methods for DNN Classifiers
Neal
Marginal coverage gap (MarCovGap) The deviation of
In this section, we empirically investigate the effect of TS on the performance of CP algorithms. Specifically, we
the marginal coverage from the desired 1 az:
consider different datasets and models, and start by reporting the mean prediction set size, marginal coverage, and class- conditional coverage of CP algorithms, with and without an initial TS calibration procedure. Then, we extend the empirical study to encompass a wide range of temperatures and discuss our findings.
MarCovGap
Nual
l( (val))
(val)
E (vol)) 1 -11- -x)
E((x
Neal
Top-5% class-coverage gap (TopCovGap) - The deviation from the desired 1 a coverage, averaged over the 5% of classes with the highest deviation:
3.1. Experimental setup
Datasets. We conducted our experiment on CIFAR-I0, CIFAR-100 (Krizhevsky et al., 2009) and ImageNet (Deng al., 2009) chosen for their diverse content and varying levels of difficulty.
TopCovGap
Σ 1 11 (val) xET,
(/ E C((x(vol) -1
Top5ye[c yE[C|
[rg
Models. We utilized a diverse set of DNN classifiers, based on ResNets (He et al., 2016), DenseNets (Huang et al., 2017) and ViT (Dosovitskiy et al., 2021).
where Top5 is an operator that returns the mean of the 5% highest elements in the set and I, = {i E [Neat] y (oul) y is the indices of validation examples with label We use top-5% classes deviation due to the high variance in the maximal class deviation.
For CIFAR-10: ResNet34 and ResNet50. For CIFAR- ResNe150 and 121 For ImageNet: R5sNet152, DenseNet121 and ViT-B/16. Details on the training of the models are provided in Appendix B.1.
We also evaluate the average class-coverage gap (AvgCov- Gap) - metric that measures the deviation from the target coverage, averaged across all classes. The experimen- tal results for this metric closely mirror those observed with the TopCovGap metric. Thus, we defer its formal definition and experimental evaluation to Appendix B.5.3.
TS calibration. For each dataset-model pair, we create a calibration set by randomly selecting 10% of the validation set. We obtain the calibration temperature T by optimizing the ECE objective. The optimal temperatures when using the NLL objective are very similar, as displayed in Table 3 in the Appendix B. This justifies using ECE the default for the experiments.
Note that for these metrics: the lower the better. Similar metrics have been used in (Ding et al., 2023; Angelopoulos et al., 2021). All the reported results, per metric, are the median-or of means along 100 trials where we randomly select the calibration/CP sets, similarly to (Angelopoulos et al., 2021).
CP Algorithms. For each of the dataset-model pairs, we construct the "CP set" (used for computing the thresholds of CP methods) by randomly selecting (5%, 10%, 20%} of the validation set, while ensuring not include in the CP set samples that are used in the TS calibration. The CP methods that we examine are LAC, APS, and RAPS, detailed in Section 2.2 (we use the randomized versions of APS and RAPS, as done in (Angelopoulos et al., 2021)). For each technique, we use a = 0.1 and a = 0.05, so the desired marginal coverage probability is 90% and 95%, as common in most CP literature (Romano al., 2020; Angelopoulos
3.2. The effect of TS calibration on CP methods
For each of the dataset-model pairs we compute the afore- mentioned metrics with and without an initial TS calibra- tion procedure. In Table 1, we report the the calibration temperature T", the accuracy (not affected by TS), and the median-of-meen -feaanss of the prediction set sizes metric, AvgSize.
4

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
Table Prediction Set Size. AvgSize metric along with 7 and accuracy for dataset- pains using LAC, APS, and RAPS algorithms with 0.1, CP set size 10%, pre- and post-TSS calibration.
Accuracy(
AvgSize APS
AvgSize after TS
Dataset-Model ImageNet, 155
7
Top-1
Top-5
LAC
RAPS
LAC
APS
RAPS
1.227
78.3
94.0
1.95
6.34
2.71
1.92
11.11
4.30
ImageNet, 1enseNet121
1.024
74.4
91.9
2.73
9.60
4.70
2.76
11.32
4.88
ImageNet, ViT-B/16
1.180
83.9
97.0
2.22
1.93
19.27
2.34
CIFAR-100, ResNet50
1.524
80.9
95.4
1.62
5.31
2.88
1.57
9.14
4.96
CIFAR- 100, DenseNet121
1.469
76.1
93.5
2.13
4,26
2.98
2.06
6.51
4.27
CIFAR-10. 10, ResNet50
1.761
94.6
99.7
0.91
1.04
0.98
0.91
1.13
1.05
CIFAR-IO, ResNet34
1.802
95.3
99.8
0.91
0.94
0.93
1.05
Table Coverage Metries, MarCovGap and TopCovGap metries for dataset-model pairs using LAC, APS, and RAPS algorithms with 0.1, CP size 10%, pre- and post-TS calibration.
MarCovGap( %)
GapCovGap TS( )
TopCovGap( %)
Top TS(%)
Dataset-Model ImageNet. ResNet152
LAC
APS
RAPS
LAC
APS
RAPS
LAC
APS
RAPS
LAC
APS
RAPS
0.1
0
0
23.1
16.0
17.6
23.9
13.8
15.2
ImageNet, 22
0
0.1
0.1
24.9
15.7
18.0
25.2
14.9
17.6
ImageNet, ViT-B/16
0
0.1
0.1
24.8
14.2
14.7
24.9
12.2
12.5
CIFAR-100, ResNet50
0.1
0
0
0.1
13.9
12.6
11.7
12.9
9.0
7.9
CIFAR-100, DenseNet121
0
0
0
0
0.1
11.5
9.5
9.7
12.2
7.8
8.0
CIFAR-I0, ResNe150
0
0
0.1
11.1
5.0
4.8
11.2
2.4
2.6
CIFAR-10, 10, ResNet34
0
0.1
9.5
3.0
2.8
9.1
2.2
2.2
of the marginal and conditional coverage metries, MarCovGap and TopCov- Gap. In both tables, the specified coverage probability is 90% (a- = 0.1), and we use 10% of the samples for the CP set and 10% of the samples for the calibration set. Due to space limitation, the results for coverage probability 95% (a- 0.05) and sizes (5%, 20%} of the CP sets are deferred to Appendix B.4. The insights gained from Tables and 2 hold also for the deferred results.
In Table 2, we report the
datasets. (Note that while LAC has lower AvgSize than APS and RAPS, its conditional coverage is worse, as shown by TopCovGap in Table 2). Third, perhaps the most thought provoking observation, for APS and RAPS the TS calibra- tion procedure has led to increase in the mean prediction set size. Especially, when the value of the optimal temperature T is high. For instance, for ResNet50 CIFAR-100, TS calibration increases AvgSize of APS from 5.31 to 9.14 9.14. This behavior is quite surprising.
Examining the results, we first see that the TS calibration temperatures, T", in Table are greater than 1, indicat- ing that the models exhibit overconfidence. The reliability diagrams before and after TS calibration are presented in Appendix B.3.2.
In Appendix B.4.1, 1. we verify that the increase in the mean set size for APS and RAPS is not caused by a small number of extreme outliers. We observe that for many samples in the validation set the prediction set size increased due TS calibration. Only for a small minority of the samples the prediction set decreases. Yet, this teaches that cannot make universal (uniform) statement about the impact of TS on the set size of arbitrary sample, but rather consider a typical/average case.
By examining MarCovGap in Table 2, we see that all CP methods maintain marginal coverage both with and without the initial TS procedure (the gap is at most 0. 1., i.c., 0,001). That is, TS calibration does not affect this property, which consistent with CP theoretical guarantees (Theorem 2.1).
3.3. TS beyond calibration
As for the conditional coverage, indicated by the TopCov- Gap metric in Table 2, there is distinct trend observed for the LAC method. On the other hand, in the adaptive CP methods, APS and RAPS, there is noticeable improve- ment (TopCovGap decreases), especially when 7* is high. We turn to examine Table 1, which reports the effect of TS calibration on the prediction set size of the CP methods. First, we see that for the CIFAR-10, where the models' ac- curacy is very high, the effect AvgSize is minor. Second, see that the effect on LAC is negligible also for other
The intriguing observations regarding TS calibration - es- pecially, both positively and negatively affecting different aspects of APS and RAPS - prompt us to explore the ef- fects of TS on CP, beyond calibration. In Figure we present the average prediction set size (AvgSize), the class- conditional coverage metric (TopCovGap), and the thresh- old value of the CP methods (ÿ) for temperatures ranging from 0.5 to 5 with an increment of 0. The reported results are the median-of-mean for 100 trials. We present here three diverse Mataset-mode pairs, and defer the others to
5

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
ImageNet, VIT
CIFAR-100, DenseNet121
CIFAR-10, ResNet34
Mean prediction size 15
Moan prodictoon size
Meen prediction set size
Temperature
Temmeratuuee
Tetspersttre
ToyCorCop
ToyyCox
TopCovCosp
012
Temperature
Temperature
Temnerature
Moaan 4
Mean
Temperature
Temperature
T
Temmerature T
Figure 1. AvgSize (top), TopCovGap (middle), and mcan threshold @ (bottom) for LAC, APS and RAPS with a 0.1 versus the temperature T. The vertical line marks 7 obtained by calibration.
Appendix B.5 due space limitation. The appendix also includes different settings, such various calibration set sizes and coverage levels, detailed in Section 3.1.
trade-off remains also beyond the maximun/minimum of PygSize/ToCCoGGpp Nevertheless, the existence of non- monotonic trends with extreme points is surprising and not intuitive.
Figure displays interesting non-monotonic trends of Avg- Size and Top for APS and RAPS. Across all dataset- model combinations, these metrics exhibit similar patterns: AvgSize (top row) increases until reaching a peak, and then starts declining; TopCovGap (middle row) decreases until reaching minimum, then reverses and starts increasing. The threshold value Ụ (bottom row), the other hand, de- creases monotonically for APS and RAPS. For LAC no clear pattern is evident.
4. Theoretical analysis
In this section, we provide mathematical reasoning for em- pirical observations regarding the effect of TS the pre- diction set size of APS and RAPS presented in Section 3. Theoretically analyzing our other findings, such as the ef- feet of TS on the conditional coverage, are left for future research. All the proofs are provided in Appendix A.
Let T. denote the "critical" temperature at which AvgSize reaches its peak (not to confused with the optimal calibra- tion temperature, )*)). Restricting our view T To, we generalization of the observations of the TS calibration experiments (which compare 7 = and 7 T*). Specif- ically, for the APS and RAPS algorithms, as T' increases AveSize increases while TopCovGap decreases. This re- veals that in this range of T. there is a trade-off between the two crucial properties of APS and RAPS prediction sizes and conditional coverage which can be con- trolled by increasing/decreasing T. In fact, overall, the
4.1. APS/ RAPS threshold decreases as T'' increases
In Section 3.3, we observe that increasing the temperature monotonically reduces the threshold of APS and RAPS. Let us prove this theoretically. Later, we will use this result in our theory on the effect of TS on the prediction size.
Let z(x) be the logits vector of sample Let मं a (z/T) be the softmax vector after TS with temperature T. We denote by 8T(x, y) the score of the APS method when
6

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
applied on Rr. Namely, sx = Σ *T, where #T,()) denotes the i-th element in a descendingly sorted version of रT and Ly is the index that y is permuted to after sorting. Recall that the RAPS algorithm is based on the same score with an additional regularization term that is not affected by TS.
Ly
12
Without T5
With
10
The following theorem states that a cumulative sum of a sorted softmax vector, analogous to the APS score, de- creases as the temperature T increases.
Theorem 4.1. Let z E C be a sorted logits vector, i.e, 21 2 22 2 ... 20, and let L [C]. Let कτ (z/TT) and oं = =z/ with T > T > 0. Then, we have 1.
2000
4000
6000 8000
Samples indexes sorted by score value
Σ j=ز -.,² = Σ Tr.j. The inequality is strict, unless
Figure 2. A for each sample sorted by APS score value (from low to high), with and without TS calibration applied before sorting the scores, for CIFARIOO 100-0eeNe550...
L L- C or 21
Z0.
Note that Theorem 4.1 1 is universal: it holds for any sorted logits vector. Denote the threshold obtained by applying the CP method after TS by âr. Based on the universality of the theorem, we can establish that increasing the temperature T decreases âr for APS and RAPS.
Σ i-1 मा. 2 qr). The following holds:
If 0cT T <
and LT > 1 then :
LT-1 # -==
LT-1
मा, S q - dr = L2 Lr
i=1
Corollary 4.2. The threshold value år of APS and RAPS decreases monotonically as T increases.
If T >>>
and L > 1
then
M4 i=1
tr. 2 q qr
= L Lr
1=
(3)
4.2. The effect of TS on prediction set sizes of APS
In Section 3.3, we observe a consistent dependency on the temperature parameter I' across all models and datasets: the mean prediction set size of APS and RAPS switches from increasing decreasing T passes some value Ta. In this section, we provide theoretical analysis to elucidate this behavior. For simplification we focus on APS.
Note that our definitions of L and Lr imply that their mini-
mal value is 1. Thus, in the case of 0 < T < 1 and Lr = 1, we trivially have L 2 Lr. Similarly, in the case of T > 1 and L. 1, we trivially have S Lr-
The right-hand side of the new inequalities pertains the difference between the threshold values before and after applying TS. Analyzing this term requires understanding the properties of the quantile sample of APS.
Let (r and i denote the thresholds of APS when applied with and without TS, respectively. For a new test sample x with logits vector z that is sorted in descending order, ेr = oz (T) and मे = (z)) are the softmax outputs with and without TS, which are sorted as well. We denote Tor min(e: E
Let z denote the logits vector of the sample associated with APS threshold (without applying TS), which we dub "the quantile sample". The CP theory builds on the score of the quantile sample being larger than (1 a:% of the scores of other samples with high probability.
1= tT. 2 (r). T. min Σ #; 2 i) as the prediction set sizes for this sample according to APS with and without TS, respectively.
As illustrated in Figure 2, there is a strong correlation be- tween the score value and the difference A: == x(1) =(2)- This implies that, for typical values of a (e.g., 0.1), the quantile sample exhibits highly dominant first entry in its logits vector z².
We aim to investigate the conditions under which the events LT 2 L and Lr < L occur. Since analyzing these events directly seems to be challenging, we leverage the follow- ing proposition that establishes alternative events that are sufficient conditions.
Notably, even if the samples are sorted according to their scores after applying TS, this strong correlation still holds, Specifically, denoting by zitt the logits vector of the quantile sample when TS is applied, we observe in Figure 2 that it still exhibits a highly dominant first entry, with a A value similar to that of z Recall that the softmax operation
Proposition 4.3. Let z E CC such that 2 2 2g 2 > 20, क. =o(z/T) and से (z)). Let g. du (0,1]
such that ifT > 1 then ğ 2 ûr and i < T < 1 then
< Φr. Let L min(e: Σ i=1
म. g) LT min(é:
7

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
applied the logits vectors is invariant to constant shifts and further amplifies the dominance of larger entries (Tt; x exp(z;)). Thus, having .1,(1) (2)(2) 2 = =(2) > 1, practically implies that 2 ~ z up to an additive constant. In Appendix B.6, we demonstrate this similarity across additional dataset-model pairs and temperature settings. Therefore, it is reasonable to make the technical assumption that both i and âr correspond to the same sample in the CP set, denoted here by the sorted logits vector z².
10
1/
SL
Ir
2
10
For the rest of the analysis, we define the "gap function":
T
(T):TMM) = Σ imi 01(2)-
M
Σ M (z/T))
im1
Figure 3. b(T) of Thm. 4.4, for C ={10, 1000).
Σ Ei=1 exp(zi)
M Ei=eeppe exp(=i/T)
(4)
25=1 exp(zz)
Ej= exp(=;/T)
Let us consider the case of TS with T > 1. The theorem establishes that for a sample with a sorted logits vector z satisfying A > (T') we have that g(z) decreases monotonically as 21 increases (indeed, when 21 increases then A increases, and thus the inequality A= > br (T) remains satisfied). Since z has larger dominant entry than typical z, this implies that g(;; T, M) > g(z";T, MM. Thus, under our technical assumption that & and qr correspond to the same sample, we can apply Proposition 4.3 and get LT 2 - a larger prediction set of APS. Conversely, by the same logic, the effect of TS with T < 1 on a sample with a sorted logits vector satisfying A >> "r<i(T), is smaller prediction set of APS.
where z is a logits vector sorted in descending order.
With our assumption that q and êr are associated with the same quantile sample z², we have that q - îr in (3) can be written as g(2; T, L.), where I denotes the rank of the true label of z².
In Proposition A.6 in Appendix A. we prove that VM E [C]. 1g(2), T, LL)) g(z², T, M) decays exponentially with Az. Due to the large A of the quantile sample illustrated in Figure 2, we approximate g(z", T, L9) ~ g(z", T, M). Substituting M- LT 1 or M L 1, depending on the branch in (3), this justifies studying the following events:
[0(z; M) s T) if <<<<1 (g(2; 7, M) 2 M)
(5)
We now show that the bounds in Theorem 4.4 do not re- quire unreasonable values of A and T. In particular, the theorem explains the phenomenon for a "typical" z 2--e.g. e.g.. the sample with the median score in Figure 2 which in turn explains why we see the increase/decrease of the mean prediction set. Indeed, for this sample, according to Figure 2 we have A R 8. For C = 100 (as in this CIFAR-100 experiment), the bounds in the theorem are complied by this sample for the temperature ranges 0 < T < 0.83 and 1.25 < T < 2.33. Since the calibrated temperature in this CIFAR100 1021000-RsNeetS0 experiment is T = 1.524, which falls in this range, we rigorously proved increased prediction set for the median sample after TS calibration. Interestingly, the broad temperature range that is covered by our theory indicates that our bounds are sufficiently tight to establish additional fine- -rainnd insights.
T>1
Returning to consider z as the sorted logits vector of a test sample, typically it has lower score than the quantile sample 2², and thus as illustrated in Figure also lower Az. Consequently, it is intuitive to associate an increase in the score with increase in the first entry of the sorted logits vector, 21. We now present our key theorem that establishes connection between the difference Az and the sign of v. ((; T, M). depending on I' and on the following "bound function":
If 0 0<7 T <:
T
T
4(C - 1)2) T
max
T-1 In
T+1
B(T)=
If T> 1:
max [
T
T
To demonstrate the signifi- cance of our theory, we analyze the bounds b(T) established in Theorem 4.4. We leverage our theory to explain the entire mon-monotonic trend in the empirical results on the effect of TS on the mean prediction set size of APS (and the closely related RAPS), showed in Figure 1.
Implications of the bounds.
In(4T),
T+1 (0T((-1))))
[--1
(6)
Theorem 4.4. Let E C such that 21 2 22 2 ... 2 20
and denote Az = 22 Then, the following holds:
[If 0<T <T<> 1 A b(T) V.,g(z; T,M, M) >> 0
If T>1: A b(T) = Vz,g(Z;T,M) < 0
In Figure 3 we present the bound as a function of T for C = (10, 100, 1000]. According to the analysis, samples whose
8

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
A= is above the bound are those for which the TS operation yields larger (resp. smaller) prediction set size when T (resp. T < 1). We denote by To the temperature value at which the bound attains its minimum value for T >> 1. This value can be computed (numerically) by the intersection of the functions in the max operation at the T branch in (6). Note that affected by the number of classes C. Inspecting Figure we gain the following insights.
TS with 7'
x(x)
X
trained Model
TS with f
xTp
Adapaiee
r Optimal (etperatiev calibration
f Trade-off temperature for PS size and C.C.
For < T 1: The bound increases function of T. Thus, T decreases, greater proportion of samples satisfy the bound, which is aligned with a reduction in the mean prediction set sizes.
Figure 4, Guideline for using TS calibration and adaptive CP.
trends, the user can choose Te that best fit their requirements. Furthermore, note that the procedure required to produce approximated curves of metrics vs. T is done offline dur- ing the calibration phase and its runtime is on the order of minutes.
For 1 < 7 < T. The bound decreases as T increases, indicating that more samples comply with the bound (have Ly 2 L), which is aligned with an increase in the mean
prediction set sizes.
In Appendix D, we further show the practical significance of our findings and guidelines. Specifically, for users that prioritize class-conditional coverage, we show that applying TS with temperature of the minimum in the approximated TopCovGap curve (created using only the calibration set as explained in Appendix C) followed by RAPS outperforms Mondrian CP (Vovk, 2012), which is tailored to classwise CP, in both TopCovGap and AvgSize metrics. Code for reproducing our experiments and applying our guidelines is available at https: / https:///iia //tg//gttttt//t com/ lahavdababl TS4CP.
A = Ta: The bound attains a minimum, corresponding to the maximum number of samples satisfying the bound, which aligned with the largest mean prediction set size.
For T > To: The bound again increases, indicating that as T continues rise, fewer samples satisfy the bound, which is aligned with decrease in the prediction set sizes.
We that Te describes the temperature which the trend of the prediction set size shifts from increasing to decreasing as T increases. Our theory shows that T. shifts to lower temperatures as C' increases. A similar trend is observed in the empirical results shown in Figure 1.
6. Conclusion
In this work, we studied the effect of the widely used TS calibration on the performance of CP techniques. These popular complementary approaches are useful for assessing the reliability of classifiers, in particular those that are based DNNs. Yet, their interplay has not been examined so far. We conducted an extensive empirical study on the effect of TS, even beyond its calibration application, on prominent CP methods. Among our findings, we discovered that TS enables trading prediction set size and cansiiooniti coverage performance of adaptive CP methods (APS and RAPS) through non-monotonic pattern, which is similar across all models and datasets examined. We presented a theoretical analysis on the effect of TS on the prediction set sizes of APS and RAPS, which offers comprehensive explanation for this pattern. Finally, based on our findings, we provided practical guidelines for combining APS and RAPS with calibration while adjusting them via dedicated TS mechanism to better fit specific requirements.
5. Guidelines for Practitioners
Based on theoretically- backed findings, propose guideline, depicted in Fig. 4, for practitioners that wish to use adaptive CP methods (e.g., due to their better condi- tional coverage). Specifically, we suggest to use TS with two different temperature parameters on separate branches: T that is optimized for TS calibration, and π that allows trading the prediction set sizes and conditional coverage properties of APS/RAPS to better fit the task's 's requirements. Our experiments and theory show that 7e should be scanned up to value Te.
Note that one does not know in advance what values of the metrics are obtained per value of T. However, since we propose to separate the calibration and the CP proce- dure, the calibration set can also be used to evaluate the CP algorithms without dangering exchangeability. Indeed, in Appendix C, we demonstrate how using small amount of calibration data we can approximate the curves of AvgSize and TopCovGap vs. T that appear in Figure (which were generated using the entire validation set that not accessi- ble the user in practice). According to the approximate
Acknowledgment
The work was supported by the Israel Science Foundation
(No. 1940/23) and MOST (No. 0007091) grants.

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
Impact Statement
Frenkel, L. and Goldberger. J. Network calibration by class- based temperature scaling. In 2021 29th European Sig- nal Processing Conference (EUSIPCO), pp. 1486-1490. IEEE, 2021.
Our work focuses on investigating the interplay between approaches aiming at making modern classifiers reliable. Our finding can guide practitioners to use these approaches better. Improved reliability estimation can significantly in- fluence and guide society toward better future.
Gibbs, I,, Cherian, J. J., and Candès, E. J. Conformal prediction with conditional guarantees. arXiv preprint arXiv:2305. 2023.
References
Grigorescu, S., Trasnea, B., Cocias, T.. and Macesanu, G. A survey of deep learning techniques for autonomous driving. Journal of Field Robotics, 37(3):362-386 2020.
Angelopoulos, A. N. and Bates, S. A gentle introduction to conformal prediction and distribution- free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q- On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Angelopoulos, A. N., Bates, S., Jordan, M., and Malik, J. Uncertainty sets for image classifiers using conformal prediction. In International Conference on Learning Representations, 2021.
Gupta, C., Podkopaev, A., and Ramdas, A. Distribution free binary classification: prediction sets, confidence inter- vals and calibration. Advances in Neural Information Processing Systems, 33:3711-3723 2020.
Barber, R. F., Candes, E. J., Ramdas, A., and Tibshirani, J. Conformal prediction beyond exchangeability. The Annals of Statistics, 51 31(2)816-845 2023.
Cosmides, L. and Tooby, J. Are humans good intuitive statisticians after all? rethinking some conclusions from the literature on judgment under uncertainty. cognition, 58(1): 1873),1-73, 1996.
Hastie, T. Tibshirani, R., Friedman, J., and Franklin, The elements of statistical learning: data mining, inference and prediction. The Mathematical Intelligencer, 27(2): 83-85, 2005.
DeGroot, M. H. and Fienberg, S. E. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), (21112):12-22 1983.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700 47004 2017.
Ding, T., Angelopoulos,
A., Bates, S., Jordan, M., and Tib- shirani, R. J. Class conformal prediction with many classes. Advances in Neural Information Process- ing Systems, 36, 2023.
Ji, B., Jung, H., Yoon, J., Kim, K., et al. Bin-wise tempera- ture scaling (bts): Improvement in confidence calibration performance through simple scaling techniques. In 20/9 IEEE/CVF International Conference Computer Vision Workshop (ICCVW), pp. 4190-4196. IEEE, 2019.
Ding, 7.,, Han, X., Liu, P., and Niethammer, M. Local temperature scaling for probability calibration. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6889-6899, 2021.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
Foygel Barber, R., Candes, E. J., Ramdas, A., and Tibshirani, R. J. The limits of distribution- free conditional predictive inference. Information and Inference: A Journal of the IMA, 10(2):455-48) 2021.
Lei, J. and Wasserman, L. Distribution-free prediction bands for non- non-parametric regression. Journal of the Royal Sta- tistical Society Series B: Statistical Methodology, 76(1): 71-96, 2014.
10

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
Liang, S., Li, Y., and Srikant, R. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference Learning Representations, 2018.
Tibshirani, R.J., Foygel Barber, R., Candes, E., and Ramdas, A. Conformal prediction under covariate shift. Advances in neural information processing systems, 32, 2019.
Vovk, V. Conditional validity of inductive conformal pre- dictors. In Asian conference on machine learning, pp. 475-490. PMLR, 2012.
Lu, C., Ahmed, S. R., Singh, P., and Kalpathy-Crann Estimating test performance for medical devices under distribution shift with conformal prediction. arXiv preprint (x1::22077007796 2022.
Vovk, V., Gammerman, A., and Saunders, C. Machine- learning applications of algorithmic randomness. In Pro- ceedings of the Sixteenth International Conference on Machine Learning, pp. 444-453, 1999.
Lu, C., Yu, Y., Karimireddy, S. P., Jordan, M., and Raskar, R. Federated conformal predictors for distributed uncertainty quantification. In International Conference on Machine Learning, pp. 29942222964 PMLR, 2023.
Vovk, V., Gammerman, A., and Shafer, G. Algorithmic learning in a random world, volume 29. Springer, 2005.
Miotto, R., Wang, F., Wang, S., Jiang, X., and Dudley, J. T.
Wang, D.-B., Feng, L., and Zhang, M.-L. Rethinking cal- ibration of deep neural networks: Do not be afraid of overconfidence. Advances in Neural Information Pro- cessing Systems, 34:1 14:11809-11820, 2021.
Deep learning for healtheare: review, opportunities and challenges. Briefings in bioinformatics, 19(6):1236-1244) 1246, 2018.
Naeini, M. P., Cooper, G., and Hauskrecht, M. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelli- gence, volume 29, 2015.
Wei, H., Xie, R., Cheng, H., Feng, L., An, B., and Li, Y. Mitigating neural network overconfidence with logit normalization. International Conference on Machine Learning, pp. 23631-236666 122444 PMLR, 2022.
Niculescu-Mizil, A. and Caruana, R. Predicting good prob- abilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning. pp. 625-60 632, 2005.
Xi, H., Huang, J., Liu, K., Feng, L.,, and Wei, H. Does confidence calibration improve conformal prediction? arXiv preprint arXiv:2402.04344 2024.
Zadrozny, B. and Elkan, C. Transforming classifier scores into accurate multiclass probability estimates. In Proceed- ings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 694-699, 2002.
Nixon, J., Dusenberry, M. W. Zhang, L., Jerfel, G., and Tran, D. Measuring calibration in deep learning. In CVPR workshops, volume 2019.
Papadopoulos, H., Proedrou, K., Vovk, V., and Gammerman, A. Inductive confidence machines for regression. In Ma- chine Learning: ECML 2002: 13th European Conference on Machine Learning Helsinki, Finland, August 19-23, 2002 Proceedings 13, pp. 345-335-555 346. Springer, 2002.
Platt, J. al. Probabilistic outputs for support vector ma- chines and comparisons to regularized likelihood meth- ods. Advances in large margin classifiers, 10(3):61-74, 1999.
Romano, Y., Sesia, M., and Candes, E. Classification with valid and adaptive coverage. Advances in Neural Infor- mation Processing Systems, $3:3581-3591 2020.
Sadinle, M., Lei, J., and Wasserman, L. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association, 114(525):223- 234, 2019.
Stutz, D., Cemgil, A. T., Doucet, A., et al. Learning optimal conformal classifiers. International Conference on Learning Representations, 2022.

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
A. Proofs.
Theorem A.1. Let a E RC be a sorted logits vector, i.e.,
21 2 22 2 2C, and let L E [C]. Let क. - o(z/T) and
t (z/T) with > T > 0. Then, we have Σ j=1 TIJ 2 TT.j The inequality is strict, unless L = C or
21 2C.
Before we turn to prove the theorem, let us prove an auxiliary lemma.
Lemma. Let 24, 2g E R such that 2, 2 2j and ler T T 2 0. Then, the following holds
exp(2;/T) exp(2,/T) (exee//))exp(22/771 exp(2;/T) exp(2)/T).
The inequality is strict, unless T T or 2; = Zj.
Proof. Given that 2, - 2j 2 and T > T 2 0, it follows that:
(z - ( 2/)/T.
Since the exponential function is monotonically increasing, applying it to both sides of the inequality yields:
exp((z)- - 2,)/T) 2 -xp((=x--)) 21)/T).
Multiplying both sides by exp(z;/T) we obtain:
exp(2:/T) exp(zz/T) exp(2:/T) exp(2;/T),
which is the desired result. The strict monotonicity of the exponential function implies that equality holds only (z: - 2,)/T = (zi - 2,)/T, i.. if T or 2, Zz-
Proof. Back to the proof of the theorem.
Let I (1,2...., and J = {L + 1, + C)..,,.. Because is sorted, Vi I,j we have 2 23. Therefore, according to the auxiliary lemma in Theorem A.I: exp(z;/T") exp(z;/T) 2 exp(z/T) exp(z;/T) for any combination of E I.j J. Consequently, summing this inequality over all i E I,j J, we get
Σ
Σ ++1
exp(zi/7) exp(2,/T)
Σ
exp(2:T) exp(e,/T)
1=1 ++
$
L N
exp(zi/T)
Σ
exp(2;/T)
Σ
exp(21/1)
Σ
exp(2;/T).
j=L+1
1=
In the last line, we separated the summation of the indexes i T,jeJ. J.
12

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
Adding
exp(2: (T)
(T) to both sides, we get
1=1
4 2 Σ exp (2//T)
e
Σ (xp(2:/T)
Σ exp (z/ T) +
Σ 1=L+1
exp (2//TT
Σ cxp(zi/))+
T))
E exp(2;/T)
5=b+1
$
c exp (2i
exp(z:/T) E
exp (z/ 2
Σ
Σ
exp(z)/T)
5=1
$
Σ exp(zz/T)
Σ 1=1 exp (2//T)
Σ 1= exp(2)/T)
j=1 exp (2) TT)
$
exp(2)/T)
exp(2;/ T)
j=1 E exp (z)
Ee exp (zc (T)
as stated in the theorem. Note that the inequality is strict unless L. = C (both sides equal 1) or T = To _... =c (all the pairs are equal).
Corollary A.2. The threshold value qr of APS and RAPS decreases monotonically as the lemperature T increases.
Proof. Let us start with APS. Set T 2 T. For each sample (x, y) in the CP set, we get the post softmax vectors कं and πt, with and without TS, respectively. By Theorem 4.1, applied on the sorted vector π = ( *()))) T with I. = Ly (the index that y is permuted to after sorting), we have that
Σ j=1 TT.
j=1 TT.
(8)
That is, the score of APS decreases universally for each sample in the CP set. This implies that q, the [n + 1)(1 a)]
n
quantile of the scores of the samples of the CP set, decreases as well.
We turn to consider RAPS. In this case, decrease due to TS in the score of each sample (x, v) in the CP set, i.e.,
Ly
Σ 7 (x) + A(Ly- kreg) +
(T (x)
X(Ly Kreg)++
simply follows from adding M(L, - kireg)+ to both sides of (8). The rest of the arguments are exactly as in APS.
Proposition A.3. Let z E Dº such that 21 22 ... =c. क, (z/TT) and म = oz)). Let år E (0, 1] such that if
T> > 1 then q 2 qr and 0 < T then à s âr. Let minfl: Σ
2 q} Lr = minfl E क 2 qr
The following holds:
then Σ #;
Lr--
Mtn- मे. - - बr = L LT
br-
If 0 < T 1 and LT >
99)
1-11
m4 E/1
1-11
If T> 1 and L > 1 then
TT. 2 q dr = 1 5 br
i=1
i-1
13

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
Proof. We start with T 1 branch, for which q 2 Qr:
TT 2 q - dr
t
#r,s + q âr
(10)
i=1
i=1
i=1
Note that L = min(e:
#j q. Thus, if (10) holds, then: !}
i=
L-1
1-1 ==== s=1
N1 + q 9r
qr
Therefore, Lr := min
Tr. 2 } cannot be smaller than L.
We continue with branch, for which êr 2 9. we will take similar steps:
Σ br-1 ls i=1
Ri Σ
का, S q - Gr
lo--1
Σ Ior-1
As + GT á
(II)
i=1
iiml
1- T
Note that LT == min(l: Σ TT, 2 (r)
âr. Thus, if (11) holds, then:
Σ Ju--
tr + 4T
#r.
iml
Therefore, L :- [e: Σ ===
मi 2 4) cannot be smaller than LT.
Theorem A.4. Let C such that 2 22
2c and denote A 21 22. Then, the following holds:
T
T
T
(4(C- }
If <<T<1: Az
max
In
In
V=
T-1
T
(12)
If T>1: A
T
In(4T)
T
In(4T(C - (1))
Vag(zTTMM) <0
T-1
T+1
Proof. Let start with T branch: The gap function is defined follows:
Σ 1 exp(z:)
Expexpl exp(=:/T)
(=(2; TM)
1= exp(z;)
E exp(2;/()
Let us differentiate with respect to
V2 T,M exp(z1)
-==1 exp(2;) - EM exp(zi)
explai/TT
Ee -==1 exp(2;/T) -M exp(z:/T)
DC exp(2,)
ES exp(2; (T)
2
exp(21) Ee De=M41 M+1 exp(2e)
1 exp(21/T) EeMo exp(zc/T)
exp(2e)
Eg exp(ze/TT)
14

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
Therefore,
(= <0 = exp (
21 (2)
1 (1-7) ))
1 E
M+1 M+1 exp(ze/T)
Σ exp(ze)
2
(13)
T
E M exp(ze)
exp(ze/
where we arranged the incquality and used exp(z1)/ exp(x1/T) = exp (* (2)
(1117)
1
According to the auxiliary lemma in Theorem A.I, if we substitute T = 1 we get: for all 24 > 2j and T > 1 we have exp(z;) exp(2,/T) > exp(2:/T) exp(2;)-
Therefore, taking i = MM + 1 and summing both sides of the inequality over j = M + 1,... C, the following holds:
Ec M+1 exp(zofT)
exp(=ar+1/T)
(14)
M+1 exp(ze)
exp(zar+1)
In addition, note that AA such that A > max (2(C 1) exp((A2)))) 2) we get:
A> -C expp -aop--A/TT) + = Aexp(2//T) Σ exp(ze/T)
(15)
where the implication follows from
exp (sp(ze/T))) (xp(zz//)//xppz///// = 1+ Σ expl-(zz - zc)/T) s - (C' - 1) exxep(((( (20pp((21222/1111 z2)
Using the above inequalities ((14) and (15)) we obtain
1 E E exp(ze/T)
c= exp(2c)
1 exp(zM+1/T)
exp(21)
exp (2:1 - 22) (1-7))
2 A2T
T E exp(2e)
Ee exp(zc/T)
T exp(2M+1)
Aexp(z1/7T (T)
where in the last inequality we used
exp(zMr+1/)) /T) exp(zı) exp(2M+1) (xp(21/T)
(1- 2 exp
(1- (1-7) 1--1)
exp (31 - 2M+1)
(as 22)
| (1-7))
1
(221 22) | (1-
Hence, according to (13): exp
21
exp
V2. <0
A2T
Note that
(1-1)
1- T )) <
-1
(221 -22)
1 - )
exp
A2T exp
T
- 1< A²T
exp
((1122
(21 - 22)
(1-1)) =
A >
T
In(A?T)
T-1
And by using the definition of A we obtain:
T
T
In(44TC-111 -19))
Az max
In(47)
V (2; <0
T-1
T+1
We continue with 0 < T < branch: Based on steps we took for the previous branch:
(z. > = exp 11 (1(11)
1)
1 De M1 exp(ze/T)
Σ exp(2e)
2
(16)
T
E E exp(ze)
exp(2e/T)
15

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
Note that according to the auxiliary lemma in Theorem A.I, if substitute π = 1 get: for all 2, 2g and <<< have exp(z;) exp(2z/T) exp(=/T) exp(zz) and therefore the following holds:
E exp(2e)
exp(zar+1)
Σ M+=M+1 exp(2c/T)
exp(zM+1//I)
In addition, note that AA such that A > max (2(C - exp( -e), 2) we get:
C
A> 1) (xp(-)))))-))))----- -0 +1 1= Acxp(21) > Σ exp(ze)
(17)
Using above incqualities we obtain
1 >e M+1 exp(ze/T)
exp(ze)
2
eexp(zM+1/T)
A exx((z1)
2
A²
(1-
T exp
(22) - 22)
2 M+1 exp(ze)
exp(ze!T)
exp(2as+1)
exp(21177
1
2
(1-7) 1
Hence, according to (16): exp
1
exp (221 22)
V
(x >0
T
T
Note that
Aa
(221 - 22) (1- (1-
T
T
exp
21
exp
A
In
1
The sign of the in-equality changed because And by using the definition of A we obtain:
T < O.
T
T
T
44CC [A(C-122)
113 ) = (e > 0
Az z max
(
T-1
4
1
T +
T
Proposition A.5. Let z RC,, a()) be the softmax function, and A C-1 denote the simplex in R Consider Shannon's 's
entropy H : C-1 + R. ie., H(T) Σ m. In(;)) Unless x 1c (then a (2/T) (z)), we have that (z(TT)
1=1
strictly monotonically increasing as grows.
Proof. To prove this statement, let us show that the function S(T) H(o(z/T)) monotonically increases (as T increases, regardless of z). To achieve this, we need to show that f'(T) df ((o(zz)T 2 0.
d By the chain-rule, ((( (H(o(z/T)))
(H))) (z) à
do dz aT (z/ T). Let compute each term:
dr
@H(o) - In(o;) - - = - In(o;) - -1 - @H(o) -n(o)) T_ - 1
1
0o:
do
(z, (z(z) (1(i= = j} az(2)) 0z diag(o - (Z))o((2)
dz,
dz
0
(z/T) T2
0T
16

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
where in (n( the function operates entry-wise and [ = is the indicator function (equals if = j and 0 otherwise).
Next, observe that 1 (diag(o) oo ooo 0 Consequently, we get
1
dT (H(a(z/T))
T In(o(())) (iaz)))T(din (diag(o(z)) - ozz)o(z)) )
1
(z - s<))cc) (diag(o ( ( )
T
T (diag((o(z)) oz(z)o(z))
where in the second equality we used (z)]]; = In exp(zi)
2===1 exp(z;) ) = 21 - s()).
d (H(((z/T)) 0, we can show that (diag(o)) ooo positive semi-definite
Therefore, for establishing that JT
matrix. Let o, exp(2;) and notice that 0g
Indeed, for any RC
{0} we have that
2j=1 oj
C
C
2
T (diag(or) oo ))u-) =
Σ
U
0; Σ
Ui
i=1
1=1
Σ upā:: E/=
U;O0
2
2 0,
where the inequality follows from Cauchy- -chhha- inequality:
Σ u²ō =
Σ 1=1
U4 uu R 0.Vo.
0.5
U
oj.
Bauchy-Schward inequality is attained with equality iff u, ō. cvō; with the constant for i 1, I,..., C,i.e when u = cla. Recalling that dT JT (H(a(z/T)) = -5² (diag(or) - oo T )2, this implies that
dT (H(o(z/T)) =
21 Ec. and otherwise dT (H(a(z/T))) > 0.
A.1. On the link between (3) and (5)
With our assumption that and fr are associated with the same quantile sample z", we have that - år in (3) can be written as g T, L?), where L denotes the rank of the true label of z², The left-hand side in (3) be written as g(z; T, Ly 1) for T < 1 and g(z; T, 1) for T > 1. In this section, prove exponential decay of this distance function of Az. According to Figure 2, this justifies the assumption of g(2". T, M) = g(z". ,(z",T,L)) L?),, which leads to studying (5).
theoretically show the similarity between g(z; T, M) and
g(2;T; T, Ly) VM e [C]. Specifically,
Proposition A.6. Letz I such that 21 22
zc- Consider the following functions:
d(z; exp(z;)
exp(=:/7) (T)
Σ exp(z;)
Σ5 exp(2,/TT (T)
ΣM M - exp(z1)
M M exp(2x/T)
g(z;T,M)
Σ d(z;T,i)
ΣS= exp(z;)
ΣS=1 exp(z,/T)
Let 8T>1 = max (i E [C] : d(z; 7.f) > 0}, iee, the last index where d(z, T) is positive. Similarly, let 8T<1
17

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
2.4
Index of the last bin increased
Index of the last bin decreased after TS with T
2.2
after T5 with 7
2.0
1.8
1.6
1.4
12-
10
4000
10000
2000 4000
6000
8000
10000
Samples indexes sorted by score value
Samples indexes sorted by score value
Figure 5. Considering CIFARI00-ResNece with sorted logits vectors. Left: the last index increased by TS with T 0.5; Right: the last index decreased by TS with = 1.5, for each sample sorted by score value. We see that sт<1 = and 8T>1 = 1 for the quantile sample.
max E [C] :d(z]T,) < 0}, ie., the last index where díz, T..) is negulive. The following holds:
(C - 1) exp (-Az)
0<T<<< <1: ST<1 =1 V4 L4 e [C]
I, g1
(C: - 1) exp (-Dz) +
(C - 1) exp Az/TT)
T>1: sT>1=1 VM [C]
T, g(z; T, L인)
(C - 1) exp (-A=/T) +1
When applying the proposition on the quantile sample z", since it has A= >>>>>>>>>>>>> we get small upper bounds.
As can be seen in the proposition, the bounds require that ST<1 and sT=1 equal 1. Recall also that the logits vector is sorted. For T (resp. T < 1), this means that the TS attenuates (resp. amplifies) only the maximal softmax bin. This is expected to hold for the quantile sample, due having very dominant entry in z". We show it empirically in Figure 5, which presents the curves for ST<1 and ST>1 across samples sorted by their scores for SHFAR100-ReeNethe Both curves indicate that approximately the first third of the samples correspond to s > 1, while higher-score samples align with 1. Notably, for the quantile sample z² characterized by scores exceeding 90% of the samples - we observe that 8 = 1 and 8T>1 1.
Proof of Proposition A.6
We start by stating and proving Lemmas A.7 and A.8, which serve as auxiliary to Proposition A.6.
Lemma A.7. Let π be a descendingly sorted softmax vector and WT the same vector after temperature scaling Define the difference vector := π лт. Then, there exists an index such that the vector d is partitioned into two segments, where all elements in one segment have the opposite sign to those in the other segment.
Proof. First, let us formulate this proposition:
Let E RC such that 21 2 2 2c- Consider the following function difference:
exp(z;)
exp(21/T) m: AT,
d(z; T.i) "g=1exp(a exp(2)) E exp(2;/T")
The following holds:
d(z;Ti)) > Ah>i d(z;T,k)> > If T>1 and Di E s.t. d(z:T,i) 0 Ai> d(z;T,k) < 0
If 0<T<1 and
3i eie [C]
1
Denote A ==
and B = Σ ==1 exp(2,/T)
Notice that z and T are constants and only is changing in the
Σ exp(2;)
18

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
theorem condition. Let rewrite d(zz Ti)):
( = d(zi) = A exp (z:) - B exp (2/TT)
Notice that d(2;) is discrete function because E z. To understand when is it possible that d(2;) = 0 we convert the function to continuous, i.e., zj can be any value. The continuous function (that gets single variable instead of vector) is as follows:
d(x) = Aexp (I) - B exp (x/T)
T
In(B/A) == x", We will now divide the rest of the There a single solution the equation d(x) = 0 which is x --1 proof into two temperature branches:
Assume 21 <x",, d(x1) < 0 then, Vz < <a we have d(xx) < 0. In our discrete case, we know that d(z)) > 0 (substitute L = 1 in Theorem A.1), and that d(z;) < 0 (therefore 2 < a* in our analogy to the continuous case), consequently Vzg < 22 d(zx) < 0. Overall, because k > i, 2 < 24 and we get that d(2; T, k)) < O.
T 1 branch:
T <1 branch: Assume x2 > xx, d(x2)= > 0 then, Vx > 22 we have d(x) > O. In our discrete case, we know that d(2)) < 0 (substitute in analogy to the continuous case), consequently Vzz < 24z d(zx) > Overall, because k > 2k 24 and get that d(2; k)) > 0.
Theorem A.1), and that d(2+) > (therefore 2 < 2*
Lemma A.8. Let z C such that 21 2 22 2 ... 2 zc. Consider the following functions:
exp(21)
exp(21 (T)
d(i: ΣS= exp(zz)
ΣS= exp(zz/T)
g(2; olz M) = - exp(z;)
"xpeex/1// Eÿ exp(2,/TT) = Σ d(z; T,
M
ΣS= exp(z;)
Then, if we denote by ST>1 max {i E [C] d(z; T,j) > 0} the last index where d(z, T) is positive and similarly by
8T<1 = max (i E [C] d(z; T,j) < 0} the last index where d(z, .) is negative, the following holds:
If 0 <<T<1: VM [C] ( g(2;T,L9) < Σ d(2":T,0)
If T>1: VM. [C] 1s(2; - u(; (0MM)))g(277..0
Σ
Proof. Let us begin by rewriting [g(z; T, T.M) M) g(z; T, L.), assume M > L without loss of generality:
ΣM exp(zi)
M exp(2./T)
E exp(z))
E cxp(z/T)
- -(.;T))) =
ES exp(z;)
Σ5= exp(2,/T)
1= exp(z;)
exp(ey/TT
M
L
M Σ L1+1
Σ
d(z: T,i)
Σ d(z;T,
d(zz
Henceforth we denote 8 == 8T>1 or = 8T<1 depending on the temperature. By Lemma A.7, Vi < 8 we have
M Σ 1341
d(z; T, > and Vi we have d(z; T, < O, therefore we can divide the analysis of
d(z; T,i)
into cases:
19

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
+12 in this case, sum only differences with the same sign, and create upper bound for this by summing all the differences with the same sign:
C d(z; T.i)
d
S
Σ
M I s in this case like previous case, we sum only on differences with the same sign, and we can create an upper bound for this case by summing all the differences with the same sign, note that in any case the starting index the summation is higher (L +13 2), therefore we can exclude the first difference:
Σ 1=
d(z; T
Σ 1=2
d(z: T,
L +1< and M > s in this case, we sum both on positive and negative differences, and we can create an upper bound for this case by taking the maximum between previous cases bounds:
M
Σ 1=
Σ
< max
Σ d(z: T,,)
i)
d(z I,
1
1=0+1
M Σ
Overall, we can write the upper bound of
as in case 3:
i=b++1
M
d(z; T,i)
S max
Σ 1=++1
d(2; T2
Σ d(2; T,
\=+++1
Note that the summation of all differences zero, Le.
Σ
d(z: T, = 0, therefore,
iml
Σ d(z;T,i)= - Σ d(2; 1) and Σ d(z; T,
d(z; T,
because both summations contain differences
1=1 i=++1
k=++1
with the same sign, subtracting elements from them decrease them, therefore.
d(z;T,0)
Σ
d(z; T,
1=2
i
C
C
max
Σ d(z;Tri)
Σ
d(z; 12i)
Σ
d(z; Tii)
=
Σ
1=041
1=2
i=0+1
And overall we get:
VM e [C]
[d(z:T, - g(x:T,L9)]
Σ
d(z; T.
1=1
We now turn to proving Proposition A.6.
Proof. T 1 branch:
Substituting 1 in Lemma A.8 we get:
VM, L E [C] (g(z;T. g(z;T, L
< Σ
d T,
Id(z; T, 1)) = d(z; T.1)
20

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
We remove the absolute value because d(z; 0 for T > We continue by bounding d(z; T. 1):
exp(21) exp(21/T) (x=((xp(z))
1 exp(21/T)
d(z;T,1)=
Σj=1 exp(2z/T)
2==1 exp(z)/T)
exp(21/T)
1
=1- exp(=1/T) _S expl -(1 - z;)/T) +1
=1
ΣS expl (00p -2)//T) +1
1
(C' 1) exp( -
(C- (C-1)exp(-Az
-C- expl -A) +1
Overall, we get:
(C - cxp (-A:/T)
MM, L E [0] l9(z; g(z; <
(C' (- (-Az/T)
< < 1 branch:
Substituting in Lemma A.8 get:
LM E [C]
M) - g(z;
Σ
d(z; T, T.
=Id(z;T, = -
We remove the absolute value and add minus sign because d(z;T,1) for T 1.
We continue by bounding -d(z; T, 1):
exp(21/T)
exp(21)
1 exp(21)
(d2zTT1)) -(iz T,
jj= exp(2)/T)
2Ø=1 exp(2))
> j=1 exp exp(z)) (zg)
exp(21)
=1
=
exp(z1) -S (exp(-(=1 23)) +1
レド= exp( ( -2,)) +1
(C 1) expeppppp -A2)
< (C'- cxpp( 1x) +1
(C - 1) exp( - A=)) +
Overall, get:
(C - exp
exp (-Az)
AM, 17 E [
Ig(z: T, M) ,(z:TT < (C - 1) exp (-Az) +1
21

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
B. Additional Experimental Details and Results
B.1. Training details
For ImageNet models, we utilized pretrained models from the FORCHVISION.MODENN sub-package. For full training details, please refer the following link:
seferences/classiicctte
For CIFAR-100 and CIFAR-I0 10 models, we use: Batch size: 128; Epochs: 300; Cross- -ross-Entropy loss; Optimizer: SGD; Learning rate: 0.1; Momentum: 0.9; Weight decay: 0.0005.
B.2. Experiments compute resources
We conducted our experiments using an NVIDIA GeForce GTX 1080 Ti. Given the trained models, each experiment runtime is within a range of minutes.
B.3. Temperature scaling calibration
As mentioned in Section 2.1, two popular calibration objectives are the Negative Log-Likelihood (NLL) (Hastie et al., 2005) and the Expected Calibration Error (ECE) (Naeini et al., 2015).
NLL, given by L C=
(n (x:)), measures the cross-entropy between the true conditional distribution of data (one-hot vector associated with y:) and (x:).
ECE aims to approximate E (xX) = Y ((x)() = p) - -r]]- Specifically, the confidence range [0, I] is divided }. Each sample (x,, y/) is assigned to a bin B₁ according to Ty. (x.). The objective is
into L equally sized bins
given by ECE Σ Bi |aco(B) conf where acc(Bi)
[B| Σ NEB H8((,) yi and conf(B:)
n
Σ ie
ne. (x,). Here, (-) denotes the indicator function.
Bj
B.3.1. ECE NLL MINIMIZATION
Above, we defined the two common minimization objectives for the TS calibration procedure. Throughout the paper, employed the ECE objective. Here, we justify this choice by demonstrating the proximity of the optimal calibration temperature 7 for both objectives.
Table 3. Optimal Temperature for NLL and ECE objectives
Dataset-Model
T NLL loss T ECE loss
CIFAR- 100, ResNet50
1.438
1.524
CIFAR-100, DenseNet121
1.380
1.469
ImageNet, ResNet152
1.207
1.227
ImageNet, DenseNet121
1.054
1.024
ImageNet, ViT-B/16
1.18
1.21
CIFAR-10, ResNet50
1.683
1.761
CIFAR-10, ResNet34
1.715
1.802
Using both objectives, we obtain similar optimal calibration temperatures T resulting in minor changes to the values in Tables and 2, presented in Section 3.2. Furthermore, in Section 3.3, we examine the effect of TS over a range of temperatures, which naturally includes both optimal temperatures.
22

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
B.3.2. RELIABILITY DIAGRAMS
The reliability diagram (DeGroot & Fienberg, 1983; Niculescu-Mizil & Caruana, 2005) is a graphical depiction of a model before and after calibration. The confidence range [o, 1] is divided into bins and the validation samples (not used in the calibration) are assigned to the bins according to (x)) The average accuracy (Top-1) is computed per bin. In the case of perfect calibration, the diagram should be aligned with the identity function. Any significant deviation from slope of I indicates miscalibration.
Below, we present reliability diagrams for dataset- dataset-model pairs examined in our study. We divided the confidence range into 10 bins and displayed the accuracy for each bin as a histogram. The red bars represent the calibration error for each bin.
ImageNet, ResNet152
CIFAR-100, ResNet50
CIFAR-10, ResNet50
a & Costidacca
1
Cuecfaace Toog
Cantibera Bange
Accency
Acsuagg
Costideaca Blenge
Coutleeca Bonge
Cantibeen Bonge
Figure 6. Reliability diagrams before (top) and after (bottom) TS calibration with ECE objective.
ImageNet, DenseNet121
CIFAR-100, DenseNet121
CIFAR-10, ResNet34
Gualabanse
Gsaltbonce
Figure? 7. Reliability diagrams before (top) and after (bottom) TS calibration with ECE objective.
23

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
B.4. The effect of TS calibration on CP methods
As an extension Section 3.2, we provide additional experiments where we examine the effect of TS calibration on CP methods with different settings of hyper-parameters (a and CP set size). The tables below present prediction set sizes and coverage metrics before and after TS calibration for different CP set sizes and additional CP coverage probability value.
Table Prediction Set Size. AvgSize metric along with 7 and accuracy for Lataset-mode pairs using LAC, APS, and RAPS algorithms with 0.1, CP set size 5%, pre- and post-TS calibration.
Accuracy(
AvgSize APS
AvgSize after
Datasct-Model ImageNet, R5s
T
Top-1
Top-5
LAC
RAPS
LAC
APS
RAPS
1.227
78.3
94.0
1.94
7.24
3.20
1.95
10.3
4.35
ImageNet, 1
1.024
74.4
91.9
2.70
10.1
4.71
2.77
11.2
4.91
ImageNet, ViT-B/16
1.180
83.9
97.0
2.75
10.18
2.33
19.19
2.41
CIFAR-100, ResNet50
1.524
80.9
95.4
1.62
5.75
2.78
1.57
9.76
4.93
CIFAR- 100, 1
1.469
76.1
93.5
2.10
4.30
2.99
2.08
6.61
4.38
CIFAR-I0. 10, ResNet50
1.761
94.6
99.7
0.92
1.05
0.95
0.91
1.13
1.01
CIFAR-10, ResNet34
1.802
95.3
99.8
0.91
1.03
0.94
0.93
1.11
1.05
Table Prediction Size. AvgSize metric along with T and accuracy for datasct-model pairs using LAC, APS, and RAPS algorithms with 0.1, CP set size 20%, pre- and post-TSS calibration.
Accuracy(
AvgSize APS
AvgSize after
Dataset-Model ImageNet. ResNet152
T
Top-1
Top-5
LAC
RAPS
LAC
APS
RAPS
1.227
78.3
94.0
1.95
7.34
3.30
1.92
12.5
4.40
ImageNet, DenseNet121
1.024
74.4
91.9
2.73
13.1
4.70
2.76
13.3
4.88
ImageNet, ViT-B/16
1.180
83.9
97.0
2.69
10.03
1.89
2.24
19.05
2.48
CIFAR-100, ResNet50
1.524
80.9
95.4
1.62
5.35
2.68
1.57
9.34
4.96
CIFAR- 100, 1
1.469
76.1
93.5
2.13
4.36
2.95
2.06
6.81
4.37
CIFAR- 10, ResNet50
1.761
94.6
99.7
0.91
1.04
0.98
0.91
1.13
1.05
CIFAR-10, ResNet34
1.802
95.3
99.8
0.91
1.03
0.94
0.93
1.11
1.05
Table Coverage Metries MarCovGap and TopCovGap metrics for dataset-model pairs using LAC, APS, and RAPS algorithms with 0.1, CP set size 5%, pre- and post-TS calibration.
MarCovGapt %)
MarCov ISt %
TopCovGapt %
Gap IS(
Dataset Model
APS
APS
RAPS
LAC
RAPS
LAC
APS
RAPS
ImageNet, ResNet152
0.1
0
23.5
13.7
16.9
24.1
13.6
15.0
ImageNet, DenseNetl21
0.1
24.9
15.7
18
25.2
14.9
17.6
ImageNet, V/T
0.1
0.1
24.1
14.9
14.5
24.8
12.4
12.6
CIFAR-100, ResNet50
0.1
0.1
0
0.1
0
13.9
11.9
10.7
12.5
8.2
7.5
CIFAR-100 DenseNet121
0.1
9.0
7.8
7.7
CIFAR- 10, ResNe150
0
0,1
0
11.1
5.0
4.8
11.2
2.4
2.6
CIFAR-I0, ResNer34
0.1
9.5
2.8
9.1
2.2
2.2
Table 7. Coverage Metries. MarCovGap and GopCovGap metries for datasct-model pairs using LAC, APS, and RAPS algorithms with CP size 20%, pre- and post- calibration.
MarCovGap(SS
MarCovGap TS(
TopCovGapt
TopCorGap IS(%)
Dataset Model
LAC
APS
RAPS
LAC
APS
RAPS
LAC
APS
RAPS
LAC
APS
RAPS
ImageNet, ResNet152
0.1
0.1
0.1
0
23.6
16.3
17.5
23,6
13.9
15.6
ImageNet, 2e
0.1
0
24.9
15.7
18
25.2
14.9
17.6
ImageNor. VIT-B/16
0.1
0.1
0
23.9
15.2
14.1
24.3
12.7
12.6
CIFAR-IU ResNet50
CIFAR 100, DenseNet121
0.1
11.5
9.5
9.7
12.2
7.8
8.0
CIFAR-IO 10,
4.6
2.1
2.6
CIFAR 10, ResNei34
0.1
9.1
3,1
2.6
9.3
2.1
2.3
We from the above tables that the results for different CP sizes very similar. The goal of the CP set is be large enough to represent the rest of the data from the same distribution. In our experiments, see that even 5% of the validation set is sufficient for this purpose.
The increased coverage probability is reflected in both prediction set sizes and the conditional coverage metric. We observe an increase in prediction set sizes compared to Table 1, which is expected due to the stricter coverage probability requirement. Note that the tendency for prediction sizes increase with remains. Regarding the coverage metrie TopCovGap, we
24

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
Table 8. Prediction Set Size. AvgSize metric along with " and accuracy for dataset- dadel pairs using LAC, APS, and RAPS algorithms with a 0.05, CP set size 10%, pre- and post-TS calibration.
Accuracy($) (+)
AvgSize APS
AvgSize after TS
Dodes
T
Top-1
Top-5
LAC
RAPS
LAC
APS
RAPS
ImageNet, ResNet152
1.227
78.3
94.0
3.28
14.9
4.10
3.22
24.1
3.1
ImageNet, DenseNet121
1,024
74.4
91.9
3.33
20.1
5.02
3.61
22.8
5.88
ImageNet, VIT VIT-B/16 B/16
1.180
83.9
97.0
2.91
22.80
4.51
3.02
39.8
5.55
CIFAR-100, ResNer50
1.524
80.9
95.4
3.97
11.10
3.98
2.21
16.2
6.80
CIFAR-100, DenseNet121
1.469
76.1
93.5
4.89
8.81
5.01
4.23
12.16
6.11
CIBAR-I0, ResNe150
1.761
94.6
99.7
1.02
1,08
1.02
CIFAR-I0, RosNer34
1.802
95.3
99.8
1.01
1.06
1.19
1.01
1,06
1.19
Table 9. Coverage Metries. MarCovGap and TopCovGap metries for dataset-model pairs using LAC, APS, and RAPS algorithms with a = 0.05, CP set size 10%, pre- and post-TS calibration.
MarCovGapl %)
MarCoxGap TS( %
TopCov %)
Cap TS(
Dodaset-Model
LAC
APS
RAPS
LAC
APS
RAPS
LAC
APS
RAPS
LAC
APS
RAPS
ImugeNet, R5
0.1
0
0
0
0
0
16.1
11.5
14.3
16.5
10.1
12.4
ImageNet, 21
0.1
0
15.5
12.0
15.0
16.0
11.7
14.3
ImugeNet. B/T6
0,1
0,1
0,1
14.6
11.6
11.5
15.0
9.27
9.78
CIFAR-100, ResNe150
0.1
0
0
0.1
0
7,51
8.81
6.82
7.28
4.9
4.48
CIFAR- 100. DenseNet121
0.1
6.72
5.91
6.50
6.50
5,41
5.41
CIFAR-10, ResNet50
0
0
0.1
0
6.50
4.22
4.22
7.03
2.13
1.98
CIFAR- 10, ResNei34
4.12
2.71
2.73
4.17
1.29
see an improvement (lower values), which can be explained by the increase in prediction set sizes. Here, the tendency for the metrics to improve as T "'nccreses also remains.
B.4.1. "MICROSCOPIC" ANALYSIS
In addition to the tables, in order verify that the increase in the mean set size for APS and RAPS is not caused by a small number of extreme outliers, we "microscopically" analyze the change per sample. Specifically, for each sample in the validation set, we compare the prediction set size after the CP procedure with and without the initial TS calibration (i.e., set size with TS calibration minus set size without). Sorting the differences in a descending order yields a staircase-shaped curve. The smoothed version of this curve, which is obtained after averaging over the 100 trials, is presented in Figure 8 for several modaset-model pairs, both for APS and for RAPS.
For the discussion, let us focus on the CIFAR- DIFAR-100-ReR R05-Res dataset-model pair in Figure 8. For this pair, we see that approximately one third of the samples experience negative impact on the prediction set size due the TS calibration. For about half of the samples, there is no change in set size. Only the remaining small minority of samples experience improvement but to a much lesser extent than the harm observed for others. Interestingly, the existence of samples (though few) where the TS procedure causes a decrease in set size indicates that we cannot make a universal (uniform) statement about the impact of TS on the set size of arbitrary sample, but rather consider a typical/average case.
ImageNet, ResNet152
ImageNet, DenseNet121
CIFAR-100, ResNet50
C-00R- DenseNet121
Setoo Luitory
Figure& 8. Mean sorted differences in prediction set sizes before and after TS calibration for APS (top) and RAPS (bottom) CP algorithms with a = 0.1 and CP set size 10%.
25

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
B.5. TS beyond calibration
As an extension of Section 3.3, we provide additional experiments with different settings examine the effect of TS beyond calibration on CP methods. The figures below present prediction set sizes and conditional coverage metrics for range of temperatures for additional dataset-model pairs, different CP set sizes and an additional CP coverage probability value. Overall, the temperature T allows to trade off between AvgSize and TopCovGap, as discussed in the paper.
B.5.1. PREDICTION SET SIZE
ImageNet, ViT-B/16
CIFAR-100, ResNet50
CIFAR-10, ResNet50
Nout prediction wit size
Xoaan producton
Temperaturn
Tomperaburro T
Tenczerraure
ImageNet, ResNet152
CIFAR-100, DenseNet121
CIFAR-10, ResNet34
Mour prodectiin not 3230
Moue prodiction set su0
prodiciion
Figure Prediction Set Size. AvgSize using LAC, APS and RAPS with 0.1 and CP size 10% the temperature T for additional datasct-model pairs.
ImageNet, ViT-B/16
CIF0R-100, ResNet50
CIFAR-10, ResNet50
Moan prodiction sot 5090
Mean prediction set
Mess predictioo
Temperature
ImageNet, ResNet152
CIFAR-100, DenseNet121
CIFAR-10, ResNet34
Meen prediction sct stoe
Moan prodiction set stzee
50con prediction
Temnerature
Tempersiure
Figure Prediction Set Size. AvgSize using LAC, APS and RAPS with a = CP set size 5%, versus the temperature for additional dataset-model pairs.
26

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
ImageNet, ViT-B/16
CIFAR-100, ResNet50
--0, ResNet50
Doeats predictin sot stoe
bdeus prediction
Tempenntaae
Temoerrature
ImageNet, ResNet152
CIFAR-100, DenseNet121
CIFAR-10, ResNet34
30ren preticton oct stDe
Meeut prediction set
prediction set
Tecacersfaree
Tomperetuure
Tennerdiur
Figure 11. Prediction Set Size. AvgSize using LAC, APS and RAPS with 0.05 and CP set size 10% versus the temperature T for additional dataset-model pairs.
B.5.2. TOPCOvGAP METRIC
ImageNet, ViT-B/16
CIFAR-100, ResNet50
CIFAR-10, 10, ResNet50
TopCorCap
Temoecature
Temperatonn
ImageNet, ResNet152
CIFAR-100, DenseNet121
CIFAR-10, ResNet34
TopCorCap
CopCorCCpp
Temueerturre
Temserature
Temmerttore
Figure 12. Conditional Coverage Metric. TopCov Gap using LAC, APS and RAPS with a 0.1 and CP set size 10% versus the
temperature T for additional dataset-model pairs.
27

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
ImageNet, ViT-B/16
CIFAR-100, ResNet50
CIFAR-10, ResNet50
DepCovCoo
TopDorCaa
TopCooCap
Temmerature
Temoerstore
ImageNet, ResNet152
CIFAR-100, DenseNet121
CIEAR-10, ResNet34
ToyCovCap
TeyCowCop
impuory Lap
Temperature
Figure 13. Conditional Coverage Metric. TopCovGap using LAC, APS and RAPS with = 0.05 and CP set size 10% versus the temperature T for additional dataset-model pairs.
B.5.3. AVGCOVGAP METRIC
In this subsection, we define and examine the average class-coverage gap (AvgCovGap) a metric that measures the deviation from the target a coverage, averaged across classes:
AvgCovGap 1 C
Σ Σ iCT,
{u! (oal) EC (z((al)) }
(1 a))
where l, (i E [Neat] Mi with the TopCovGap metric.
(val)
y). The following experimental results for AvgCovGap closely mirror those observed
ImageNet, VIT-B/16
CIFAR-100, ResNet50
C-FAR-10, 10, ResNet50
Temperature T
Tomnerature
Temoerature
ImageNet, ResNet152
CIFAR-100, DenseNet121
CIFAR-10, ResNet34
Temnerature
Comperature T
Tempersiuze
Figure 14. Conditional Coverage Metric. AvgCovGap using LAC, APS and RAPS with 0.1 and CP set size 10% versus the lemperature T for additional dataset-model pairs.
28

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
ImageNet, ViT-B/16
CIFAR-100, ResNet50
CIFAR-10, ResNet50
A0gLonnuap
6000
Aug 1000
Temperature
Timperatom
Temperatare
ImageNet, ResNet152
CIFAR-100, Dense
CIFAR-10, ResNet34
Augg 1000
Temmerature
Temmerafure
Temoerabure
Figure 15. Conditional Coverage Metrie. AvpCovGap using LAC, APS and RAPS with 0.05 and CP size 10% the temperature 7 for additional model pairs.
B.5.4. CP THRESHOLD VALUE
ImageNet, VIT-B/16
CIFAR-100, ResNet50
CIFAR-10, ResNet50
Moan 3
Mosun
Mesa
Temoerature T
Temperature T
Temoerstare
ImageNet, ResNet152
CIFAR-100, DenseNet121
CIFAR-10, ResNet34
Mean a
Moun
Tenaberatare
Tommeniimn:
Temperature
Figure 16. CP threshold value q using LAC, APS and RAPS with 0.1 and CP set size 10% versus the temperature T for additional dataset-model pairs.
29

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
ImageNet, ViT-B/16
CIFAR-100, ResNet50
CIFAR-10, ResNet50
Mean
Moan
Thenoerature
Tempersture
Temnerature
ImageNet, 155
CIFAR-100, Dense
CIFAR-10, ResNet34
Moai Q
Temneraturs
Tempersture
Figure 17. CP threshold value using LAC, APS and RAPS with a 0.05 and set size 10% versus temperature T for additional dataset-model pairs
B.5.5. THE IMPACT OF TS AT EXTREMELY LOW TEMPERATURES
Mean prediction set size
100
In our experiments presented in Section 1, we lower bound the temper- ature range at T = 0.3. This choice was motivated by the deviation from the desired marginal coverage guarantee observed at extremely small temperatures. Specifically, we observe that for too small T the threshold value reaches maximal value, 4 1, and, presumably due to numerical errors, this leads to impractical CP procedure, with significant over-coverage and excessively large prediction set sizes, demonstrated in Figure 18 for CIFAR100-DenseNeet/0 a 0.1 and CP set size 10%.
130
006
013
0.55
033
1.5
8.30
040
Temperature T
Figure 18. Prediction Set Size. Mean prediction set size for LAC, APS and RAPS versus low temperatures, for CIFAR 100--e0seeeNe111...
B.6. Illustrating the similarity between z and zi
Figure 2 in Section 4.2 shows strong correlation between the APS score and Az z(1) - =(2). indicating that quantile samples typically have dominant entry in their logit vectors (2(1) >> z(22). This behavior persists under TS, i.e., for the logits vector z/ of the quantile sample when TS has been applied with some temperature T. Recall that the softmax operation applied the logits vectors is invariant to constant shifts and further amplifies the dominance of larger entries (T, exp(2.)). Thus, having 2),,1) 27,02) x z/1 2 1, practically implics that z z².
Here, we demonstrate this similarity using concrete examples of the largest 5 elements of these vector for various dataset-model pairs and temperature settings. To simplify the comparison, we report the values after applying softmax. That is, we report the values of (z²)).
CIFARIO-ResNecG4
[z 5] =
(2/=T-) )|: 5] = 010.:60-0000..2222-0000066666660
8.00 8.0810c-066 2...01000-00,,.0099-----000000 06,
(z)=0s)) 5) 11,999e--01,.1150 4. 4.195e. 2.0069c-06, 1.9213e-06
30

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
(1)=2)11 5] 09.997e--01, 2.0271e 2.0588e-06,
-.0525ee--0 1.7960e 1.6457e-06) 1.4477---000
CIFARR000DenseNee122.
= [000000000044444.000000111100110
(z/=r-)l )]: 5] 110---90-0000000000-0----------0
(9.9997e-01,
IeageNettReeNNet152::
a 0-0.000000....333000000000111100
o (2)=T-) || 5] 00.9922--00.3333333--00,,00----0
o oz)
o (2)= 5] = 01g0::00084422220022222222-01010
1.5007 6.11000007
Across all these dataset-model pairs and temperature settings, the quantile samples exhibit strong similarity. This observation further supports the technical assumption in in Section 4.2 that both q and @r correspond to the same underlying sample.
C. Approximating the Trade-off via the Calibration Set
As discussed Section 5, exploring the trade-off between prediction set size and class-conditional coverage through the temperature parameter 7' beneficial when using adaptive CP algorithms.
We propose using TS with separate temperature parameters distinet branches: T*, optimized for TS calibration, and 7', which allows for trading prediction set sizes and conditional coverage properties of APS/RAPS to align better with task requirements. One limitation is that the metrics' values for different T are not known priori. However, since we decouple calibration from the CP procedure, the calibration set can be used to evaluate CP algorithms without compromising exchangeability.
The curves Figure were generated by evaluating the CP methods on the entire validation set (excluding the calibration and CP set) and averaging over 100 trials. Both are feasible in practice, where the practitioner only has the calibration set and the CP set of single trial. Here, show that these curves can be approximated using only the calibration set for evaluation. In Figure 19, we plot the curves using calibration set + CP set, which together are 20% of the validation set (as in the main body of the paper). Specifically, 10% of the validation set is used for the CP operation (computing the threshold), and the remaining 10% (the original calibration set) serves as a "validation set" to evaluate the CP performance. Therefore, no additional samples are used compared to the common practice of performing calibration and CP calibration sequentially rather than in parallel.
Unlike the curves in Figure 1, the curves in Figure 19 averaged over 100 trials but are based on a single trial. Due to randomization, the curves will vary between runs, better refleet the practitioner's 's experience, from 3 separate runs. In each of the runs in Figure 19 the marginal coverage is preserved and the curves of AvgSize and TopCovGap closely resemble the averaged ones shown in Figure 1, demonstrating the user's 's ability to select T based on these single-trial graphs generated using small amount of data. Additionally, note that the procedure required to produce these approximated curves exccuted offline during calibration and has a negligible runtime.
present results
31

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
Table 10. Comparison between MCP TS with T arg min TopCovGap(T) (enhancing conditional coverage), both based RAPS.
for CIPAR 100- Re-Ree ImageNet-Vill Recall that lower metrics imply hetter performance.
CIFAR100-ResNett
ImageNet- ViT
MCP
TS with T
MCP
TS with f
CP set size (%)
10%
20%
10%
20%
10%
20%
10%
20%
Metric
AvgSize
4.5
8.1
2.9
3.0
7.4
6.2
2.9
3.0
MarCovGap
0.04
0.05
0.04
0.06
0.09
0.07
0.06
0.06
TopCovGap
0.29
0.125
0.12
0.11
0.62
0.31
0.10
0.11
D. Advantages of the Proposed Guidelines over Mondrian Conformal Prediction
In this section, we consider the case of user that prioritizes consiiooniiional coverage. In this case, our study recommends applying TS with the temperature of the minimum in the approximated TopCovGap curve, i.e. T arg min TopCovGap(T))
where TopCovGap is approximated explained in Appendix C followed by adaptive CP method like RAPS. Recall that using TS with such temperature yield high AvgSize, due the trade-off, but the emphasize here is on prioritizing low TopCovGap.
An existing alternative is to use the Mondrian Conformal Prediction (MCP) approach (Vovk, 2012). MCP aims to construct prediction sets with group-conditional coverage guarantees. Considering the groups to the classes, the method based on partitioning the data used for calibration (i.e., the CP set) by classes and obtaining a threshold per class. At deployment, the thresholds used in a classwise manner. However, major drawback of MCP is its limited applicability classification tasks with many classes, since its performance degrades when the number of samples used for calibrating CP per class is small (Ding et al., 2023).
Note that in our experiments, we consider C10 that has 100 classes and CP set (used to calibrate the CP) of size up to 2000 (20% of the validation set), and ImageNet that has 1000 classes and CP set of size up to 10000 (20% of the validation set). This means that, approximately, we have up to 20 samples per class to calibrate CP for CIFAR-100 and up to 10 samples per class calibrate CP for ImageNet.
Table 10 presents the metrics AvgSize, MarCovGap and TopCovGap for proposed approach and for MCP, when both utilize RAPS, for the dataset-model pairs CIFAR (00-Reeee and ImageNet-ViT. The results demonstrate the superiority of using TS with f arg min (opCovGap(T)) (computed based on approximated curve) over MCP across all metrics.
T
Recall that we consider the case where class-conditional coverage is preferred, and indeed, TS with such temperature constantly yields better Top than MCP. Yet, interestingly. it outperforms MCP also at AvgSize.
To conclude, the experiments presented this section further shows the practical significance of our guidelines.
32

---

On Temperature Scaling and Conformal Prediction of Deep Classifiers
Run
MarCovGap
AvgSize
TopCovGap
Moan prediction set size
TopCCo
Tempersture
Tennersbure
Temperature
Run 2
MarCovGap
AvgSize
TopCovGap
Mean predites ion sot 4toe
Temoeratarre
Temseratirr
Temnectiur
Run 3
MarCovCap
AvgSize
TopCovGap
Moan prodictoo sert 5U00
TrpprreCCap Copp
Temperstare
Tomnecature
Figure Performance evaluation with small data. Examining the performance of APS and RAPS with small evaluation data for ImageNet-VV DioeeNee-ViT-/IG ViT-B/I6 with 0.1 and CP set size 10%. Each row displays the marginal coverage, prediction size and conditional coverage metrics that are computed over trial using 10% of validation set.
33


Reviews:
[{"review_id": "pGWZOifs6s", "reviewer": "Reviewer_doKf", "summary": "The paper aims to study the interplay between conformal prediction (CP) and temperature scaling (TS) calibration. They study the effect of TS on conformal prediction using extensive empirical evaluation with three different CP methods. They present a theoretical analysis to explain the effect of TS on APS and RAPS conformal prediction methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and insightful review. We are pleased that you recognized the extensive empirical evaluation of our experimental design. \n\nBelow, we carefully respond to all of your comments.\n\n> Regarding the technical assumption\n\nThe core of this technical assumption is the strong similarity between the softmax vectors of the \"quantile samples\" associated with $\\hat{q}$ and $\\hat{q}_T$, as discussed in the paragraph in lines 332-357.\n\nDue to the limitation of characters, we refer you to the response \"Explanation for $\\pi^q \\approx \\pi^q_T$ in Line $353$\" to Reviewer 8jYt, where we explain and illustrate the proximity between $\\pi^q $ and $ \\pi^q_T$. To highlight the strong similarity, in the revised version we will include concrete examples of these vectors for various dataset-model pairs.\n\nGiven that $\\pi^q \\approx \\pi^q_T$, we believe that it is reasonable to make the theoretical derivation tractable by the technical assumption that $\\hat{q}$ and $\\hat{q}_T$ correspond to the same sample. We do not claim that it holds in every possible setting. Nevertheless, the fact that our theory provides insights that are aligned with the empirical behavior along many models and datasets serves as a justification for this technical assumption.\n\n> Metrics for class-conditional coverage\n\nEvaluating conditional coverage across groups using the worst-case coverage metric is both informative and relevant, as demonstrated by previous works, e.g., (Gibbs et al., 2023). Our TopCovGap metric is averaged over the worst 5\\% due to high variance observed when considering only the single worst-case coverage, as explained in Appendix B.4.\n    \nFollowing your comment, we will report also the AvgCovGap metric that has been used in (Ding et al., 2023) (denoted CovGap there). For example, we present in https://postimg.cc/1fCkf8CH this metric for the three dataset-model pairs shown in Figure 1. Notably, this metric exhibits similar behavior to TopCovGap, displaying a comparable trend of achieving a minimum at temperatures $T>1$.\n\n> Missing references\n\nWe thank you for bringing these related papers to our attention. Both papers will be cited and discussed in the revised version.\n\nPaper [1] (arXiv preprint) is a concurrent work (our paper was uploaded to arXiv at the same time).\nThe TS results in [1] are only a small subset of our results. They do not consider conditional coverage and use a limited range of T, which masks the non-monotonic effect on the prediction set size of APS and RAPS. Moreover, they also do not compare APS and RAPS to LAC. On the other hand, our paper provides a complete picture of the effect of TS with a wide range of temperatures on both the prediction set size and the class-conditional coverage of APS, RAPS, and LAC. This complete picture teaches practitioners that tuning the temperature for APS and RAPS introduces a trade-off effect (overlooked in [1]), and we provide a practical way to control it.\n\n\nPaper [2] presents important theoretical results on prediction sets and calibration, which are indeed related to our work. \nNote though, that [2] is limited to binary classification and do not provide explanation to the fact that calibration (e.g., TS calibration) affects CP methods differently and in a non-monotonic way, as empirically shown and analyzed in our paper.\n\n> Practical utility of the findings\n\nOur work investigates the effect of TS on CP as a function of temperature $T$. Through our analysis, we identify a trade-off between two essential properties of adaptive CP methods: mean prediction set size and proximity to conditional coverage. Our guidelines, introduced in Section 5, enable practitioners to navigate this trade-off effectively, which is a novel contribution to the CP literature. The guidelines are further explored in Appendix C and D.\n\nAppendix C demonstrates how, with a limited amount of calibration data, users can approximate the curves in Figure 1. Based on these trends, they can select an appropriate $\\hat{T}$ that aligns best with their objectives. For instance, a practitioner working with CIFAR100-ResNet50 who prioritizes prediction set size can achieve over a 50\\% reduction in the AvgSize of APS using our guidelines.\n\nAppendix D further illustrates the practical relevance of our findings. Specifically, we show that applying TS at the temperature corresponding to the minimum of the *approximated* TopCovGap curve, followed by RAPS, outperforms Mondrian CP (Vovk, 2012) in both TopCovGap and AvgSize metrics.\n\n> Regarding the runtime\n\nIn the revised version we will discuss more about the runtime of the proposed guidance. We would like to emphasize that this procedure is done offline during the calibration phase and its runtime is within a range of minutes.\n\n> Improving readability\n\nWe will include the definitions of the reported metrics (currently appear in the appendix) in the main body of the revised version leveraging the extra page allowed in the final version.", "claims_and_evidence": "The paper presents theory to support their claims. I am unsure about the assumption that $\\hat{q}$ and $\\hat{q}_T$ correspond to the same sample based on Figure 2. How generalizable is this claim? There can always be adversarial sequences that violate this, which puts the assumption in question. If not justifiable theoretically, at the very least more extensive empirical analysis to demonstrate this holds for datasets included in the paper and beyond is important.", "methods_and_evaluation_criteria": "While the paper presents extensive empirical evaluation and the datasets make sense, I am concerned about the metrics included. The justification behind evaluating class-coverage gap on top-5% classes is not provided; moreover, 5% seems an arbitrary choice. Why would you not consider evaluation on average class coverage gap as defined in Ding et al., 2023? At the very least, both metrics could be included. Also, ablation on the 5% choice seems important to make general claims about class-conditional coverage. My other concern is regarding absence of standard error reporting in the experiments.", "theoretical_claims": "I went over proofs A.1-A.4.", "experimental_designs_or_analyses": "I checked the soundness of experiments in the main paper.", "relation_to_broader_scientific_literature": "The paper aims to study the interplay between conformal prediction and temperature scaling calibration – methods that have been often studied individually in literature. TS is usually applied initially in conformal prediction methods. They report their findings on the effect of TS on conformal prediction sets in terms of set size and class-conditional coverage.", "essential_references_not_discussed": "While the paper claims this interplay has not been investigated yet, the paper misses an important reference [1] that studies this very impact of confidence calibration on conformal prediction. From a first glance, the analysis and empirical observations of [1] are close to this work. The paper also does not cite [2] who study the connection between calibration and prediction sets, although in the binary classification setting. These are just a few examples and not an exhaustive list! Authors should acknowledge these works and include a discussion on relationship with these works at the very least. The authors are also suggested to do a thorough review of existing literature to contextualize their work better.\n\n[1] Xi, H., Huang, J., Liu, K., Feng, L., and Wei, H. (2024). Does confidence calibration improve conformal prediction?\n\n[2] Gupta, C., Podkopaev, A., and Ramdas, A. (2020). Distribution-free binary classification: prediction sets, confidence intervals and calibration. NeurIPS.", "other_strengths_and_weaknesses": "The writing of the paper, especially the technical writing and notation in theorems can be improved for greater clarity. I am also unsure about the practical utility of the findings presented – temperature scaling is not a core component of conformal methods and can be done away with. While the authors present some guidelines, it does not seem convincing in the current context. Additionally, the authors mention the runtime of their procedure (pg 8). I have two comments here – (i) please discuss the runtime of your procedure in the paper, (ii) I believe offline training of DNNs should not be compared here given the post-hoc nature of methods.", "other_comments_or_suggestions": "The definition for reported metrics (pg 4) should be included in the main paper for improved readability.", "questions_for_authors": "No specific questions, please refer to individual comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C5OFQG90yE", "reviewer": "Reviewer_8jYt", "summary": "Calibration and Conformal Prediction (CP) are two popular approaches to solving the overconfidence problem in modern DNN classifiers. In this paper, the authors studied the effects of temperature scaling (TS), which is effective for calibration, on the efficiency of CP. The authors first designed extensive experiments to show how the TS affects the sizes of the prediction sets and class-conditional coverage. Then, the authors conducted theoretical analyses for the empirical observations and finally provided practical guidelines for choosing the temperature parameter.\n\n## update after rebuttal\nThe authors have addressed my concerns, so I would support acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your constructive  and thorough review of our theoretical work. We appreciate your acknowledgment of our novel analysis, supported by extensive empirical studies and theoretical investigation. We have carefully addressed your comments and suggestions.\n\nBelow, we provide a point-by-point response to your comments.\n\n***\n\n> Regarding the proof for Lemma for Theorem 4.1 (A.1)\n\n***\n\nThank you for highlighting this issue. You are correct — there was an error and a simple proof will be constructed in the revision based on the monotonic decrease of $\\exp(c/t)$ with respect to $t$ when $c>0$. Indeed,\n\n$\\\\exp(z_{i}/\\\\tilde{T}) \\\\cdot \\\\exp(z_{j}/T) \\\\geq \\\\exp(z_{i}/T) \\\\cdot \\\\exp(z_{j}/\\\\tilde{T}) \\iff \\\\exp((z_{i}-z_j)/\\\\tilde{T}) \\\\geq \\\\exp((z_{i}-z_j)/T)$\n\nand since $z_{i}-z_j \\geq 0$ we have that this inequality holds as $\\tilde{T} \\leq T$.\n\n***\n\n> Typo in Proposition 4.3\n\n***\n\nThank you for pointing out this typo.\n    Indeed, in the branch $0<T<1$,  $M\\in\\left[L\\right]$ should be changed to $M\\in\\left[L_T\\right]$, and in the branch $T>1$, $M\\in\\left[L_T\\right]$ should be changed to $M\\in\\left[L\\right]$.\n    Similarly, in the proof, in lines 685 and 688, $L$ should be replaced with $L_T$ (3 instances).\n\nAs for the sum symbol, we intended to save space by using \n$\\\\sum_{i}^{M} \\\\hat{\\\\pi}_i - \\\\hat{\\\\pi}^T_i $, which we read as $\\sum_i^{M} \\left( \\hat{\\pi}_i - \\hat{\\pi}^T_i \\right)$, rather than $\\\\sum_i^M \\\\hat{\\\\pi}_i - \\\\sum_i^M \\\\hat{\\\\pi}_i^T$. \n\nIn the revision, we will ensure consistent phrasing in the main body of the paper and the appendix.\n\n***\n\n> Explanation for $\\pi^q \\approx \\pi^q_T$ in Line 353.\n\n***\n\nThe paragraph in lines 332-357 analyzes the structure of the \"quantile sample\" of APS. Figure 2 shows a strong correlation between the samples' scores and $\\Delta z = z_{(1)} - z_{(2)}$ (the difference between the largest and second largest entries of the logits vector), both with and without temperature scaling. Since the quantile sample corresponds to a high score ($1 - \\alpha$ quantile of all scores), it consistently has a large $\\Delta z$. As evidenced in Figure 2, the $1 - \\alpha = 0.9$ quantile sample had $\\Delta z^q \\approx 11$. According to the relation $\\pi_i \\propto \\exp{z_i}$, this implies $\\pi^q_{(1)} \\gg \\pi^q_{(2)}$, specifically, after some approximations we get $\\frac{\\pi^q_{(1)}}{\\pi^q_{(2)}} \\propto 10^4$. Similarly, when temperature scaling calibration was applied,  the $1 - \\alpha = 0.9$ quantile sample again resulted in $\\Delta z_T^q \\approx 11$, leading to $\\pi_{T,(1)}^{q} \\gg \\pi_{T,(2)}^{q}$. Thus, due to the overwhelmingly dominant entry in both quantile samples, we have that $\\pi_{(1)}^{q}$ and $\\pi_{T,(1)}^{q}$ are nearly 1, and the associated sorted softmax vectors obey $\\pi^q \\approx \\pi^q_T$. \n\nFollowing this comment, to further illustrate the strong similarity between sorted $\\pi^{q}$ and $\\pi^{q}_T$, we will add in the revised version concrete examples for these vectors for several dataset-model pairs. For example, the first 5 elements in $\\pi^{q}$ and $\\pi^{q}_T$ (after sorting) for ImageNet-ViT (with $T = T^*$ optimal for calibration): \n\n$\\pi^q[:5] = [9.9697e-01, 4.9439e-04, 2.1028e-04, 2.0065e-04, 1.6687e-04]$\n\n\n$ \\pi^{q}_T[:5] = [9.9755e-01, 7.6261e-04, 8.7435e-05, 7.8683e-05, 7.6373e-05]$\n\n***\n\n> Regarding the bound in Theorem 4.4.\n\n***\n\nThank you for pointing out this typo. Indeed, there is a factor of $\\frac{T}{T+1}$ in the $T>1$ branch of Theorem 4.4 and not $\\frac{T-1}{T+1}$. \nThe same correction applies to lines 787 and 828 in the proof.  \n\nWe emphasize that the proof remains valid except for the final line of each branch (787, 828), where $T-1$ was mistakenly written instead of $T$ in the numerator of the bound. This will be fixed. \nAfter this minor correction, the bounds are still aligned with the empirical trends.\n\nThe complete substitution of $A$ from line 756 into the inequality in line 782, detailed in the following link: https://postimg.cc/hXXHxKNq confirms the correctness of $\\max \\left( \\frac{T}{T-1}\\ln(4T),\\frac{T}{T+1}\\ln(4T(C-1)^2) \\right)$ for the branch $T>1$.\n\nFollowing this comment, we will include this substitution explicitly in the revised version.", "claims_and_evidence": "The main claim made in this work is that TS has little effect on LAC but strongly affects the prediction set's size and conditional coverage of APS and RAPS. Besides, the effects of the temperature parameter are non-monotonic. These claims were empirically observed and then were analyzed theoretically.", "methods_and_evaluation_criteria": "The evaluation criteria used in the empirical studies, including marginal coverage gap, average set size, and top-5% class coverage gap, are meaningful.", "theoretical_claims": "The overall theoretical claims seem correct, but some details should be corrected or might be more clearly clarified.\n\n- The proof for the Lemma for Theorem 4.1(A.1) seems not correct, or more specifically, Line 575 to 578 is not reasonable. In fact, this lemma can be more simply proved by investigating the monotonicity of function $\\exp\\left(\\frac{z_i-z_j}{t}\\right)$ with respect to $t$.\n\n- According to the proof for Proposition 4.3 in Appendix A, it seems that, in Line 322 ‘M∈[L]' should be changed to ‘M∈[L_T]' if 0<T<1, and in Line 325 ‘M∈[L_T]' should be changed to ‘M∈[L]' if T>1. Besides, summation signs are missing in Lines 322 and 325, comparing Proposition 4.3 and A.3, which are indeed the same proposition.\n\n- Could the authors give more explanations for $\\pi^q\\approx\\pi^q_T$ in Line 353?\n\n- In the proof of Theorem 4.4, should $\\frac{T-1}{T+1}\\ln\\frac{4(C-1)^2}{T}$ be changed to $\\frac{T}{T+1}\\ln\\frac{4(C-1)^2}{T}$? And consequently the results in the theorem.\n\nI suggest that the authors should carefully check all the details of the theoretical parts.", "experimental_designs_or_analyses": "The experiments in Section 3 are sufficient to support the authors’ empirical analysis of the effect of TS on CP. The experiments in Section 4 and the appendix are also enough to verify the authors' claims.", "relation_to_broader_scientific_literature": "The prior related studies only use initial TS calibration before applying CP methods [1, 2, 3, 4]. None of them explored the effect of TS on CP, which is analyzed in detail in this paper.\n\n[1] Angelopoulos, A. N., Bates, S., Jordan, M., and Malik, J. Uncertainty sets for image classifiers using conformal prediction. In International Conference on Learning Representations, 2020.\n\n[2] Lu, C., Yu, Y., Karimireddy, S. P., Jordan, M., and Raskar, R. Federated conformal predictors for distributed uncertainty quantification. In International Conference on Machine Learning, pp. 22942–22964. PMLR, 2023.\n\n[3] Gibbs, I., Cherian, J. J., and Cand` es, E. J. Conformal prediction with conditional guarantees. arXiv preprint arXiv:2305.12616, 2023.\n\n[4] Lu, C., Yu, Y., Karimireddy, S. P., Jordan, M., and Raskar, R. Federated conformal predictors for distributed uncertainty quantification. In International Conference on Machine Learning, pp. 22942–22964. PMLR, 2023.", "essential_references_not_discussed": "The related references are properly cited and discussed.", "other_strengths_and_weaknesses": "Strength:\n\n- The effect of TS on the performance of CP is discussed for the first time.\n\n- The empirical studies are extensive.\n\n- Theoretical analyses are provided.\n\nWeakness:\n\n- Some details of the theoretical analyses should be refined.", "other_comments_or_suggestions": "In Line 1087, $d$ in the LHS of the inequality should be $g$.", "questions_for_authors": "My questions are provided in the above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r96GTwSiD8", "reviewer": "Reviewer_t6t7", "summary": "In this work, the authors studied the effect of the widely used temperature scaling calibration on the performance of conformal prediction techniques for deep neural network classifiers. A wide range of experiments are conducted and a theoretical framework is proposed to explain the effects.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable suggestions. We appreciate your recognition of our comprehensive experiments and clear presentation.\n\nBelow, we provide a point-by-point response to your comments.\n\n***\n\n> Improving readability of the figures.\n\n***\n\nFollowing this comment, in the revision we will make every effort to improve the readability of the figures. The extra page allowed in the final version will provide additional space to further enhance readability.\n\n***\n\n> Practicality of the proposed theory.\n\n***\n\nThe primary aim of our theoretical study is to provide mathematical reasoning for the surprising empirical behavior that we discovered. We believe that addressing the \"why\" question behind non-intuitive results is of high scientific importance. \n\nThat being said, our theory also provides practical implications. For example, if a practitioner cares only about AvgSize and wants to use APS/RAPS, the $0<T<1$ branch of our theory suggests that they pick small $T$ (which is aligned with the empirical observations).\nConversely, practitioners can leverage the $T>1$ branch of our theory, where monotonicity in CP properties breaks. Specifically, it justifies defining a finite range for tuning $T$ to balance prediction set sizes and class-conditional coverage, as discussed in Section 5.    We refer the reviewer to Appendix C, where we demonstrate how a small calibration dataset can approximate the curves for AvgSize and TopCovGap versus $T$ (shown in Figure 1). Based on these approximate trends, users can select an optimal $\\hat{T}$ that aligns with their needs. Additionally, in Appendix D, we highlight the advantages of this approach over existing methods.\n\n***\n\n> Applicability to other domains.\n\n***\n\nOur work deals with temperature scaling and conformal prediction applied to multiclass classification.\nWe used image classification datasets (ImageNet, CIFAR-100, CIFAR-10) as they are the benchmark datasets in the related literature.\nNevertheless, we expect our findings to be beneficial to classification tasks in domains other than images.\n\nIn particular, the practical relevance of our paper lies in empowering users to effectively apply CP in multiclass classification, based on their specific needs, by following our proposed guidelines. The applications extend to domains other than images where CP has been used, such as medical diagnosis [1],[2] and NLP [3],[4].\n\n[1] Lu, Charles, et al (2021). \"Fair conformal predictors for applications in medical imaging.\" Proceedings of the AAAI Conference on Artificial Intelligence.\n\n[2] Vazquez, J., Facelli, J. C. (2022). \"Conformal Prediction in Clinical Medical Sciences.\" Journal of healthcare informatics research,\n\n[3] Kumar, Bhawesh, et al (2023). \"Conformal prediction with large language models for multi-choice question answering.\" ICML 2023.\n\n[4] Campos, Margarida, et al (2024). \"Conformal prediction for natural language processing: A survey.\"", "claims_and_evidence": "The authors have provided a theoretical analysis to show how temperature values influence the properties of prediction sets. With the theoretical results, researchers can understand why temperature scaling can affect the conformal prediction. Empirically, the experiments are strong and extensive.", "methods_and_evaluation_criteria": "The experimental settings are strong, covering various datasets, neural network backbones, etc.  The descriptions of the validations are clear.", "theoretical_claims": "Because this work falls beyond my research comfort zone, a serious assessment of the proposed theories in detail is a challenging task for me. Therefore, this innitial evaluation may be conservative. I am eager to actively follow the discussions with the authors, other reviewers, and ACs.", "experimental_designs_or_analyses": "No concerns are raised.", "relation_to_broader_scientific_literature": "Broad literature, including the recent achievements, are duly included in the paper.", "essential_references_not_discussed": "Seems not applicable.", "other_strengths_and_weaknesses": "Additional strength: Basically, the paper is well written and organized. The appendices are extensive.", "other_comments_or_suggestions": "Some suggestions:\n\n1. It would increase the readability if the font size and line width in the figures are increased.\n\n2. Providing clearer insights/explanations to link some proposed theories and their practical values would further enhance the practicality or be attractive for the practitioners in the ML community. \n\nAdditional questions:\n\nCan the authors provide some insights on the applicability of the findings in other tasks not limited to image classification?", "questions_for_authors": "No further comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zi7H6z8s9E", "reviewer": "Reviewer_paq4", "summary": "This paper focuses on a popular calibration technique known as temperature scaling (TS) and investigates its effect on major conformal prediction (CP) methods (LAC, APS, RAPS). They show that TS improves class-conditional coverage of adaptive CP but increases prediction set sizes; the effect on LAC is negligible. They uncover a trade-off between prediction set size and conditional coverage when modifying the temperature in TS, with some theoretical analysis.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. We appreciate your recognition of our extensive empirical study and clear experimental design.  \n\nBelow, we address your comments on the theoretical analysis point by point.\n\n***\n\n> Theory for conditional coverage.\n\n***\n\nOur work develops a comprehensive mathematical framework to explain the non-intuitive effects of TS on prediction set sizes in CP. We rigorously formulate conditions for changes in prediction set size and clarify the underlying theoretical reasons for these effects. While extending this analysis to conditional coverage is valuable, it would require an entirely new theoretical framework with different assumptions and tools, making it beyond the scope of this study. Nonetheless, establishing theory for the impact of TS on conditional coverage is indeed a promising direction for future research.\n\n***\n\n> Regarding Proposition 4.3.\n\n***\n\nLet us clarify the scope of our theoretical analysis and particularly the contribution of Proposition 4.3. This proposition focuses on how TS affects prediction set size of APS. It provides a condition, dependent on the value of $ T $, for a decrease or an increase in the prediction set size after applying TS, denoted by $L_T$, compared to the size without applying TS (equivalently, TS with $T=1$), denoted by $L$. It is important to note that this result covers all positive values $ T > 0 $.\n\n\nWe would like to clarify that this is not the primary finding of our work. Theorem 4.4, along with its implications, specifically addresses this \"kink point\" of the non-monotone dependency. Theorem 4.4 shows that applying TS with temperature $T$ affects the prediction set size of a sample $z$ based on $ \\Delta z = z_{(1)} - z_{(2)}$ and a bound $ b(T) $. For example, for $ T > 1 $, if $ \\Delta z > b_{T>1}(T) $, the prediction set size increases ($ L_T \\geq L $). Figure 3 reveals a non-monotonic curve for $ b(T) $ at $T>1$, and since lower (resp. higher) $ b(T) $ implies that more (resp. less) samples obey the condition this provides an explanation to the non-monotonic pattern of the mean prediction set size (AvgSize): it increases for $ 1 < T < \\tilde{T}_c $ and decreases for $ T > \\tilde{T}_c $ where $\\tilde{T}_c$ is the temperature above 1 where $ b(T) $ is minimal. This critical temperature $ \\tilde{T}_c $ decreases with $ C $, aligning with empirical results.", "claims_and_evidence": "* Empirical\n  * The paper claims to have an extensive empirical study on DNN classifiers to demonstrate how TS affects CP. Numerical experiments are conducted on three CP methods over a range of classification dataset. Several findings are summarized from the experiments, and the appendix contain more numerical details regarding hyperparameter settings and furture comparison. The experiments are pretty thorough and well documented. \n* Theoretical analysis\n  * Compared to the empirical part, the theoretical side is a bit straightforward but is compatible with the main finding, in part. It does not concern the conditional coverage, but mainly focus on the prediction set size. Regarding the non-monotone structure, Proposition 4.3 does have some theory regarding T=1, but this is more indirect rather than the true kink point in the non-monotone relationship.", "methods_and_evaluation_criteria": "* The evaluation metrics of conditional coverage (impossible in general) and prediction set size are very well recognized in conformal prediction literature. \n\n* The numerical experiments are conducted in a range of data sets.", "theoretical_claims": "* The theoretical claims seem intuitive and correct but I only had limited review of the proofs. On the other hand, the result is not strong and more auxiliary to support the empirical findings.", "experimental_designs_or_analyses": "The experimental design and analyses appear to be sounds. The authors use relevant datasets, metrics, and comprehensive comparison. The theoretical analysis provides insights into the empirical observations. Limitations and future directions are appropriately discussed.", "relation_to_broader_scientific_literature": "Conformal prediction provides guarantee for high-risk decision making and warrants further development. This paper's finding helps to refine calibration technique of CP.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents the first systematic study of how temperature scaling interacts with adaptive conformal prediction, combining an extensive empirical survey with a supporting mathematical analysis that yields practical tuning guidelines. During the discussion, the authors corrected proof slips, supplied extra examples for an assumption, committed to add average class‑coverage metrics, error bars and runtime figures, and agreed to cite concurrently posted related work. There are a significant number of points that would still need attention in the camera‑ready version, such as: notation and metric definitions require clearer presentation, the technical assumption on sample similarity should be explicitly scoped and backed by the promised additional evidence, and the new evaluation material must be incorporated as agreed. With all of these changes the paper would provide a useful contribution. In consultation with the SAC we recommend weak accept.
Decision: Accept (poster)