Conformity Score Averaging for Classification
Authors: Rui Luo
Zhixin Zhou
Venue: ICML 2025 poster (2025)
Keywords: conformal prediction
model averaging
VC dimension
Primary area: general_machine_learning
TL;DR: A model averaging method that uses subclass of functions in Vapnik-Chervonenkis theory that achieves state-of-the-art prediction set efficiency in conformal prediction.

Abstract: Conformal prediction provides a robust framework for generating prediction sets with finite-sample coverage guarantees, independent of the underlying data distribution. However, existing methods typically rely on a single conformity score function, which can limit the efficiency and informativeness of the prediction sets. In this paper, we present a novel approach that enhances conformal prediction for multi-class classification by optimally averaging multiple conformity score functions. Our method involves assigning weights to different score functions and employing various data splitting strategies. Additionally, our approach bridges concepts from conformal prediction and model averaging, offering a more flexible and efficient tool for uncertainty quantification in classification tasks. We provide a comprehensive theoretical analysis grounded in Vapnik–Chervonenkis (VC) theory, establishing finite-sample coverage guarantees and demonstrating the efficiency of our method. Empirical evaluations on benchmark datasets show that our weighted averaging approach consistently outperforms single-score methods by producing smaller prediction sets without sacrificing coverage.

Conformity Score Averaging for Classification
Rui Luo! Zhixin Zhou
training data, ensuring that the true label included with Conformal prediction provides robust frame- work for generating prediction sets with finite- sample coverage guarantees, independent of the underlying data distribution. However, existing methods typically rely on a single conformity score function, which can limit the efficiency and informativeness of the prediction sets. In this pa- per, we present novel approach that enhances conformal prediction for multi-class classifica- tion by optimally averaging multiple conformity score functions. Our method involves assigning weights to different score functions and employ- ing various data splitting strategies. Addition- ally, our approach bridges concepts from confor- mal prediction and model averaging, offering a more flexible and efficient tool for uncertainty quantification in classification tasks. We provide a comprehensive theoretical analysis grounded in Hapnik-Chervonenkee (VC) theory, establish- ing fimite-sample coverage guarantees and demon- strating the efficiency of our method. Empirical evaluations benchmark datasets show that weighted averaging approach consistently outper- forms single-score methods by producing smaller prediction sets without sacrificing coverage. Our code is available at https: / //ggt/ttt//////// com/ luo luo-lorry/Wee www.loorr//iigiinnggg...
Abstract
a probability of at least 1 a. This coverage assurance holds regardless of the specific point prediction algorithm employed, making conformal prediction a versatile tool for uncertainty quantification in machine learning, In the split conformal prediction framework (Papadopoulos et al., 2002; Lei & Wasserman, 2014; Vovk et al., 2018), the training data is partitioned into a training set and a calibration set. The predictive model trained on the training subset, while the score functions are evaluated on the calibration sub- set. The conformal prediction set then comprises all labels whose conformity scores exceed specific quantile, with the quantile determined by the coverage level.
The choice of score function is critical in determining the efficiency of the resulting prediction sets, especially for multi-class classification. Well-chosen score functions can lead to more informative and precise predictions. This flex- ibility allows conformal prediction adapt the specific characteristics of the data and distribution. Consequently, developing score functions that optimize informativeness and efficiency for various problem settings, including re- gression (Papadopoulos et al., 2008; 2011; Romano et al., 2019; Kivaranovic et al., 2020; Guan, 2023; Colombo, 2023; 2024) and multi-clat classification (Sadinle et al., 2019; Ro- mano et al., 2020; Angelopoulos al., 2021; Huang et al., 2024; Luo & Zhou, 2024: 2025c), remains an active area of research. This work focuses on enhancing the efficiency of prediction sets for classification tasks. While the underlying idea can extend to regression, this paper will focus on the tasks of multi-class classification. We will also extend the method and theory to basic conformal regression problem.
1. Introduction
Conformal prediction (Vovk et al., 2005; Manokhin, 2022) robust framework that generates prediction sets with finite-sample coverage guarantees, irrespective of the derlying data distribution (Angelopoulos et al., 2023), The fundamental principle of conformal prediction is to con- struct a prediction set for a new test instance based on the
Our approach assumes the availability of multiple score functions for the same classification task, each differing due to variations in the classification algorithm or the defini- tion of the score. We propose assigning optimal weights to aggregate these score functions. Using a validation set, we determine a threshold to achieve the desired coverage and identify the weight combination that minimizes the prediction set size. The final prediction is then based on this weighted score function. Our aim is to find the op- timal weights for linear combinations of score functions, thereby fully leveraging the strengths of existing score func- tions. While our approach shares similarities with (Yang & Kuchibhotla, 2024), stands out in three key aspects:
'Department of System Engineering, City University of Hong Kong, China Alpha Benito Research, Los Angeles, USA. Cor- respondence to: Rui Luo <ruiluo@cityu.cu..k edu.hk>, Zhixin Zhou <zzhou@ caapuu@@@pphabbnto.com.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Conformity Score Averaging for Classification
1. Weighted Averaging of Score Functions: Instead of selecting the single best-performing score function, our approach combines multiple score functions through optimal weighting. This averaging can yield more effi- cient prediction sets than any individual score function while maintaining the desired coverage guarantees.
Algorithm 1 Split Conformal Prediction
input Labeled data (z) M4) E Iusin L )
unlabeled data [a;; i E I
significance level or
output Prediction set C(1:) for i E Trst
1: Train a model p(x) on ( V1) HiE
2. Novel Data Splitting Strategies: We explore and cat- egorize several data spliting methods to determine the optimal weights for combining score functions. In ad- dition to Validity First Conformal Prediction (VFCP) and Efficiency First Conformal Prediction (EFCP) dis- cussed in (Yang & Kuchibhotla, 2024), we introduce Data Leakage Conformal Prediction (DLCP) and its variant DLCP+, which utilize all available data hance weight determination.
2: 9--a + +[(1+ [Taal) - 1-t largest score s(2,, for E Teal.
for i Iuee do
C(x) qI-a) f ty E [K] s(1;,)) 2 9I-a3
5: end for
for E [K]. While our method and theoretical analysis do not depend on the accuracy of this approximation, beneficial assume that higher values of (1( indicate a greater likelihood of sample x: having label y. We consider this training procedure to be performed on a separate dataset, ensuring that Py (2)) is independent of the dataset used in this paper.
3. Theoretical Foundations Using VC Theory: We pro- vide theoretical analysis of our method that lever- ages Vapnik-Chervonenkk theory to establish cover- age guarantees and expected prediction set sizes. This solid mathematical foundation underscores the validity and efficiency of our approach across different data splitting strategies.
We begin by defining a conformity score function, s(x, y), which quantifies the agreement between a sample x and a potential label y. Higher values of s(x, y) indicate a stronger belief that y is the correct label for A common choice for this score is the estimated class probability, s(x, y) Py (x) (this approach is related to scores used in methods like (Sadinle et al., 2019), though original definitions may vary, e.g., using non-conformity scores).
Beyond its novel contributions to the conformal prediction literature, our method is closely related to model averaging (Claeskens & Hjort, 2008), a well-established technique in machine learning. Unlike traditional model averaging, which assigns weights to different models to improve predic- tion accuracy, our method assigns weights to score functions. This distinction requires the development of specific data splitting techniques to ensure the desired coverage guaran- tees. Consequently, our work can be viewed as an innovative adaptation of model averaging principles to the conformal prediction framework.
The conformal prediction procedure for classification can then be outlined. Initially, predictive model (x) is trained a dedicated training set Tarain Subsequently, threshold ęı-a is determined using separate labeled cal- ibration set Teat. This threshold is chosen such that the conformity scores of true labels, s(1;; 3i) for (xi,, yi) ) E Teal. satisfy s(z,, y4) 2 91-a for at least a 1 aa proportion of the calibration samples. Finally, for any new test sample Ty, this threshold q1-a is used to construct the prediction set as the collection of labels whose conformity scores meet exceed it: Ö(æ,) = {y s(zj, 3) 41-aa. ). This set represents the upper level set of the scoring function s(z,,)).
The remainder of the paper is organized as follows. In Section 3, we detail our weighted averaging approach and the various data splitting strategies employed. Section 3 presents the theoretical analysis, establishing coverage guar- antees and expected prediction set sizes. In Section 4, we demonstrate the effectiveness of our method through ex- periments. Related works are discussed in Section 5. We conclude in Section 6 and outline future research directions.
This algorithm constructs prediction sets C(.,, q1-a) for each i Iests based on the conformity scores and thresh- old determined by the desired coverage probability 1 a. In conformal prediction, we assume the samples in Zeal and Irest are exchangeable. The threshold gI-o is chosen to ensure the desired coverage probability under the exchange- ability assumption.
Notations. [K] denotes the set (1...., K} for positive integer K. I₁ I2 denotes the union of the disjoint sets Z₁ and Iz. (a, denotes the inner product of vector a and b. [C| denotes the set size of finite set C.
2. Methodology
2.1. Conformal Prediction for Classification
2.2. Various Score Functions for Classifications
We start by assuming that a K-class classification algorithm
In conformal prediction, the choice of score function s y) critically influences the prediction sets. These functions act
provides Py (x)) which approximates P(Y x|Xx = x)
2

---

Conformity Score Averaging for Classification
((1) y1 hEZm
((z, Wiliiee
((r Bi) hez,
((1s, Vile
-i ( (20) by (2)
(C(xi, w, vorgiaa ( by (3)
w by (4)
(2) Qi-a by (5)
(Clri, W 20, Te
by (6)
Figure 1. This example illustrates framework for data splitting into I,, Iz, Iz, and Twes Algorithm 2 presents the complete procedure. Briefly, In and I2 are used in Steps 1-2 to select the optimal weight w, while Iş is used in Step 3 as the calibration set for Tres predictions. We present four options: VFCP, EFCP, DLCP, and DLCP+ Their coverage and size properties are discussed theorctically in Section 3 and empirically in Section 4.
as conformity measures that quantify how appropriately label matches input.
Given vector of score functions s(z, y) (s1 (a, v),-, sa(x, v)) instead of simply choosing the score that provides the smallest prediction set on calibration set, we propose assigning weights w E W S Rd to the conformity scores and defining the weighted score function:
Score functions are determined by two primary factors: (1) The predictive model's 's quality in estimating (x) = P(Y = X = x), and (2) The specific methodology for converting these probability estimates into scores. We will consider approaches:
(wn s (xx
Σ w,
(1)
1. Threshold (THR) (Sadinle et al., 2019): STHR (2,9)= (x)
By leveraging different scores to create a completely new score function, we expect the weighted score to outperform any individual score. This approach allows for more flexi- bility in combining the strengths various score functions, potentially leading to improved prediction set efficiency.
2. Adaptive Prediction Sets (APS) (Romano al., 2020):
SAPs(I, 3) Eee Py (2) (I) S (x))
3. Rank-based (RANK) (Luo & Zhou, 2024):
SRANK (I, 11y E [K] By (x) < (x))
These score functions integrate directly with Algorithm 1. While the original formulations include additional refine- ments to achieve exact 1 coverage guarantees, these re- finements do not materially affect our combination method- ology and are omitted for simplicity. Our theoretical anal- ysis remains invariant to specific score function design choices. Other notable approaches like RAPS (Angelopou- los et al., 2021) and SAPS (Huang et al., 2024) demonstrate alternative methodologies for deriving scores from Py (z).
2.4. The Optimal Weight and the Threshold
In this section, we present detailed procedure for deter- mining the optimal weight vector w in the context of the weighted score approach introduced (1). This procedure extends Algorithm by incorporating additional step to minimize the expected size of the prediction set. The following steps outline the process:
1. Train the model p using samples in Iarano and obtain the score functions $1
2.3. Averaging Score Functions
In (Yang & Kuchibhotla, 2024), the authors propose their
2. Extract IT Iz, In from the Iral- We can first assume they are partitions of Ival- More options will be de- scribed in Section 2.5.
method with multiple score functions. Their approach se- lects the score function that yields the smallest average pre- diction set size among the available score functions. They also introduce two data splitting methods: Efficiency First Conformal Prediction (EFCP) and Validity First Conformal Prediction (VFCP).
3. Let W C Rd be the set for the candidates of w can- didates. A typical choice is the a discretized simplex W Ad-1 We systematically explore candidate
3

---

Conformity Score Averaging for Classification
Algorithm 2 Conformal Score Averaging
Function 1 Calibration
input Labeled data (z,, 3t) i ] unlabeled ( i I, significance level at, score functions (.1,..., 8d), set of possible weights: W _ Rd output Prediction set C(z.)) for E Iws
input Labeled data ((z M1) : E I),
score functions $ = ,.)) weight w E W,
significance level a
output 91-a
I: 01-a + [(1 + [T])(1 -th largest score
1: Decide I, C Inal, I2 S Tnal Tress, and Ia S Tyal.
(w, s(1;, v:)) from scores computed for i E I.
2: for w W do
2: Return Q1-a-
3: Compute quantile for every w:
4: (l-a(w) t Calibration 8, w, a))
Function 2 Evaluation
5: for E I2 do
imput Unlabeled data x, score functions = (51, (51,...,,3), weight w E W,
6:
Find a temporary prediction set:
7:
C(a,; w) <- Evaluation 8, w, vf-a(w)
D Compute average prediction set size: 9: S(w) = p Eiez, [C(r;; w]]
8:
quantile output C(x)
10: end for
1: [(z) - {u E [K]: (w, s(, v)) 2 n-aa-
11: end for
Return C(x)
12: >> Find the most efficient w:
@ -arg miw S(w)
14: > Compute quantile for W:
7. For given w, I3 is treated as the calibration set Algo- rithm 1, and we obtain:
15: 4i t CCllibration w Ia, a:)
(2)
>> Find the final prediction set:
17: for i E Trsst do
(2) h- [(1+ a)1-th largest
(5)
Clz w) < Evaluation 8, io q a
value of { s v:)) i E Zz}.
end for 20: Return C(z,, () for i e Irstt
[; Iz. then this quantile has been computed in (2), (2) (1) i.e., 91-a ( (w).
8. The final output is the confidence set for samples i E Ireat in the test set:
weights via grid search with step size 0.01, as detailed in Appendix A.
C(a,, aD) = {v E [K]: (w. s u))
a1-a). (6)
(2)
4. Calculate the threshold for every W IT In the set I, calculate the threshold for every E W:
This procedure comprises two primary steps: Threshold Calibration: Determining appropriate thresholds using Equations (2) and (5) based on the calibration subsets Z, and Ig. Prediction Set Evaluation: Constructing the prediction sets for In and Zest as outlined in Equations (3) and (6). These steps are encapsulated in Function and Function 2, respectively. The entire procedure of the proposed method is summarized in Algorithm 2.
_ (w) = [(1 + [ a)] largest value of {(w, s(,, y:)) : }.).
(2)
5. Determine the prediction set for data in Iz. For a given w and threshold ((-a(w)) (w), define the prediction set for each sample F,, i E Iz as:
Ö(,,, w) = (y E [K] (w, s(x,, v)) 2 91 (f-a(w)) (.)).
2.5. Data Splitting
(3)
These are the prediction sets for all w W. Our goal is to minimize the prediction set size. This intuitively leads to the next step.
In Algorithm 2, the method for splitting the data into I,, Iz, and Σ3 has not been specified. To explore potential data splitting approaches, we first highlight the following two key observations. Firstly, After determining w. the calibration procedure (5) and prediction set construction (6) mirrr the step of finding quantile and prediction set in Algorithm 1. Prior steps aim to identify a weight vector w (4) optimizing algorithm performance. Secondly, (2) and (5) find quantiles, requiring sample labels. Equations (3) and (6)
6. The optimal weight vector à is obtained by minimiz- ing the empirical prediction set size:
Σ [[(x, w)).
① E arg min
WEW I
1EI2

---

Conformity Score Averaging for Classification
define prediction sets, needing only feature xc Thus, the test set can be included in (3) and (6).
w" is the weight vector that minimizes the expected predic- tion set size while ensuring that, together with an appropriate threshold g, the prediction set maintains desired coverage rate of 1 - a. Equivalently, we can define
Based on the observations above, we introduce the following four possible ways of data splitting.
Q1 a (w) supfq E R P(Y C(X, w, 2 1 a}
(a) Validity First Conformal Prediction (VFCP) (Yang & Kuchibhotla, 2024): I = I2 S Iyal, and Z3 = Ival IT. Divides Tuain into two partitions. Marginal coverage probability is guaranteed to be at least 1 a under exchangeability of samples in Za and Zest (b) Efficiency First Conformal Prediction (EFCP) (Yang & Kuchibhotla, 2024): I I2 Ia Iumin- Uses all training data to determine w. resulting in more accurate estimation of optimal This method equires stronger assumptions for coverage guarantee.
and using this definition, we define
w arg min E ||Ō(X, w, 10,01-a0 11 [n)aw])]]] weW
Since w" and F1 (w=" are deterministic once and the true distribution are given, we can succinctly denote the optimal prediction set for x::
(x) Õ(z, w., (t)"
(c) Data Leakage Conformal Prediction (DLCP): I = Is Tumins and I- Tust- Minimizes prediction set Lest in finding i in (4). Called "data leakage" as it uses test data in training procedure of w.
Our analysis of efficiency focuses the difference between the size of the prediction set produced by the proposed methods using to and the expected size of Či-a (1)). In other words. we analyze the discrepancy between the pre- diction set sizes generated by t and the optimal w". We will demonstrate that this difference is negligible given a sufficiently large dataset.
(d) Data Leakage Conformal Prediction+ (DLCP+): I = Ig Turain and I2 Turain Tres- Uses all available data in each step, including Iuss in Zg to maximize sample size for finding in (4).
We note that this analysis differs from verifying whether w converges to w*, which would require assumptions about the smoothness of the function mapping the weight to the prediction set size. Instead, our analysis makes fewer as- sumptions on the optimal prediction set size.
There are various possible methods for data splitting. In the following sections, we focus on these four specific ap- proaches, examining their theoretical properties and evalu- ating their performance through experiments.
3.2. Results by Vapnik-Chervonenkis Theory
3. Theoretical Analysis
We begin by defining the following events on the prob- ability space of pairs of exchangeable random variables (Xi, Y:),E. The event Q(I,, 11) defined as:
3.1. Overview of the Results
We investigate the theoretical properties of the proposed methods in terms of validity and efficiency.
Σ 1( s(X,, XJ) 2 /}
sup wERd .yE
Validity: We assess whether the output prediction set achieves the desired coverage rate - a. For the VFCP method, the coverage rate is guaranteed under the exchange- ability assumption. However, for the other three methods, establishing validity is more complex due the selection bias introduced by w. Consequently, our aim is to demon- strate that the optimization over w E W has small impact on validity.
[
- [Exy((I( X, Y)) 2 /) S n
where (X,Y) has the same distribution as (X,, Yi) and the expectation is for the joint distribution of (X, Y). Let g( denotes the event
sup wEBd .gER T
Σ
Σ
((w,s(X,,W)) 2 2
Efficiency: We observe that the prediction set attains the smallest possible expected size only if the true conditional probabilities p(yla (x) are known. In this section, let us define
iEZ yE[A]
Ex
Σ
1[ s(X, (X, w)) 2 9} SE
C(., w. 4) = {y [K] (w, s(x, v)) 2 q)-
WE K
(8)
We define the population level optimal weight
Here, the expectation is taken over X only.
arg min min E [|Ō(x, . wEW qER
S(I, 17) is about the uniform concentration of the coverage. and ((I, is about the uniform concentration of the pre-
s.t. P ((Y Ō(X, w.)) 2 -
diction set size. This is because by the definition of Ci
5

---

Conformity Score Averaging for Classification
in (2) or (S),
3.3. Consistency of VFCP
1(1, s(x,0)) 2 9) 1 (u Ci w, and Σ ((, v)) 2 q) - 00 C1-a (x, xE[K]
The statistical guarantees of VFCP requires a few additional assumptions, e.g., the continuity between the quantile and the prediction set. To get rid off such assumptions and present the result in a neater way, we will only prove a modified version of VFCP. We will change a in (2) to a slightly smaller a'. The condition for a' will be specified in the theorem.
The following lemma shows that these two events hold with high probability.
Lemma 1. Suppose the samples I are ii.dd, then
Let CY (x) be the output of Algorithm with the setting of VFCP, and modified as mentioned above. Sup- pose the samples in {(X;, YE)JieT are exchangeable, then for (X, Y) in the test set, the coverage probability
Theorem 1.
(a) Ω (I, (2.8//1 / ( 111(
+ 5) hold with probability
[2]
121622
at llesss 1 - eexp
(b) r (I,SKV (1+1) I 1]
+ K8) hold with proba-
P E (x)c 2 a1-a. a.
[2|
bility at least 1- - exp
Moreover, if [I 11)) ((Is, 175) and T(Iz, {2) are satis- fied, and a' + 711 + 113 S a, then for test sample X,
The proof of this lemma appears in Section B in the ap- pendix. This lemma establishes that S?(I, 7) and [(I. E) hold with high probability provided that v,. Z / 2log [2| [2
E $1 [x)]] S [[ii (x)]] + 262.
where a1 178 [[1 [Ts)(11 a')l nss.
The proof, presented in the appendix, applies subgraph classes from Sapnik-Chervonenkis (VC) theory. Addition- ally, if the inequalities in Equations (7) and (8) hold for w E Rdd they naturally extend to w E W C Ré.
The proofs of all theorems appear in Section C and Sec- tion D and the appendix.
The validity of VFCP requires minimal assumption (ex- changeability), while it is less efficient than other method. We will verify this fact in empirical studies.
Remark. Part (b) of Lemma I limits the generalization of our current theoretical results to classification tasks. Com- pared to the bound in Equation (7) (for coverage), the bound in Equation (8) (for prediction set size) involves summa- tion over the K classes. In our proof. this summation is handled using a union bound over y E [K] when analyzing the concentration of individual terms 1 (w, s(X, 9)) 2 4). However, for regression tasks where prediction sets are typically intervals and their size measured by Lebesgue measure, simple union bound over discrete classes is not directly applicable. Therefore, our primary theoretical anal- ysis focuses on classification.
3.4. Consistency of EFCP
In the setting of EFCP, Zi = In I3 Ival- We denote this set by Ival-
Theorem 2. Let CFFCP (x) be the output of Algorithm with
the setting of EFCP. Suppose the data in I,ul and Irea are independent and (TIral That) is satisfied. Then for (X, Y) in the test set, the coverage probability
IP (Y E CEFCC 2 a1 where
Nonetheless, our framework shows promise for certain re- gression settings. Consider the class of prediction sets A := {l : (w, s(x,y) 2 t} : w E Rd, t E R}. If the individual score functions 8j 83 y) are concave in y (for non-negative weights wj 2 0), then the weighted score (w, sz, v)) is also concave in y. In such cases, the super- level sets {y: (w, s(z, y)) 2 t are intervals (or empty, or the whole real line). The class of all intervals in RR has a VC dimension of 2. If the weighted score functions in a regres- sion context produce such -atuual-valued prediction sets, the analysis of prediction set size concentration might sim- plify considerably, potentially avoiding the K K-dependency. This suggests that our method could be extended to handle weighted averages of concave score functions in regression, such as those used in Conformalized Quantile Regression (CQR) (Romano et al., 2019), a direction we leave for future research.
1-01 = [Irat] 1 Inatl - ax]]= - -hall
1
Moreover, (Iv Eval) is satisfied, then for any X in the test set.
E [[(FFE((X)) E EI (X)]] + 2Eval-
3.5. Consistency of DLCP
In the setting of DLCP, I, = I and I2 = Irest- We will denote this by Ival and Iesst respectively. In this case, the test set and w become dependent. The prediction set size is presented differently than previous results.
Theorem 3. Let CPLCP (x)) be the output of Algorithm 2 for a test sample x with the setting of DLCP. Suppose
6

---

Conformity Score Averaging for Classification
MiInal: Vrai) and (Tiess Thess) hold, then the coverage pro- portion satisfies
Coverag
Σ iE Iuce
1yy E CP- (2:) 2 -ax cx1
Then where
Tire
0505
1 [(1 + [Tmrl) (1 - ai] - Mrtl
0302
Tnal
11.
In addition, if I [Inee Eres) holds, then
2331
1ÔPLar (xia)
(xi) SE E [ (x)] + Seu-
Figure 2. Boxplot comparison of different score functions at a significance level of a = 0.01 on CIFAR-100. Our weighted combination method achieves the guaranteed coverage of 99% while maintaining the smallest prediction set size.
[rest
3.6. Consistency of DLCP+
Similar as the DLCP setting, in depends on the test set, so the result of prediction set size is presented in a similar way. In the following theorem, we use Iz to denote Ival J Just-
cifar10
sfar100
3.00
Curs
2.75
APS
THR
110
Theorem 4. Let CPLOP+ (I, w)) be the oulput of Algo- rithm? with the setting of DLCP+. Suppose M[ Tval) and (T Then) hold, then the coverage proportion
2.50
RANK
MARS
2.25
2,00
1.75-
1.50
1 Σ 1y E CPLCP* (I:, })} 2 1 a' Thest
335
0.36
039 Coverage
1.00
Chiverage
where 1 a 1
[(1 + [nal (1 a)] Mhat-
Figure 3. Comparison of size vs. coverage for various score func- tions and our proposed method across a values (0.01-0.05). Our weighted combination method (red) consistently outperforms the other baseline methods by achieving the desired coverage rate with smaller prediction set sizes.
Inail
In addition, IfI E2) holds, then for test sample X.
BllOPLCPt (X)] S E [(i + 282.
3.7. Conclusion of Theoretical Results
In the experiments on CIFAR-10 and CIFAR-100 testing images, which were not used during the pretraining of the model, were used as the Train and Trest sets. The experiments were performed for different significance levels ranging from 0.01 to 0.05. 100 runs with different index splits were conducted to ensure robustness. We have additional experiments on data splitting ratio, which can be found in Section in the supplementary file.
We integrate the results of Lemma I with our main theo- rems. The Lemma 1 indicates that as the size of the set approaches infinity, the events 22(I, 17) and [II, {) hold with probability 1 - o(1) even if n, - 0 is at sufficient slow rate. Consequently, the terms ?/s and & in our theo- rems become negligible. This suggests that the proposed methods achieve coverage rates close to 1 and exhibit near-optimal efficiency.
The primary objective of experiments was evaluate the performance of the proposed weighted score function in comparison with three foundational base score functions: APS, THR, and RANK. Additionally, we compared our method against two competitive baseline score functions, RAPS (Angelopoulos et al., 2021) and SAPS (Huang et al., 2024).
4. Experiments
4.1. Score Weighting
We conducted an experiment to compare the performance of various score functions score functions discussed in Sec- tion 2.2: THR (Sadinle et al., 2019), APS (Romano al., 2020), and RANK (Luo & Zhou, 2024) given by a pretrained classifier Pyy (x)) Additional details about implementation can be found in Section E in the appendix. Throughout the experiment, we assumed that a pretrained classifier P,, (z) was available, and the split of the dataset unseen during the training of the pretrained classifier.
Figure 2 compares coverage and prediction set sizes across methods at 0.01 on CIFAR- while Figure 3 eval- uates performance across E [0.01, 0.05). Using VFCP splits for consistency, our method achieves guaranteed cov- erage with the smallest prediction sets overall. The advan- tages are particularly pronounced on CIFAR-10 across all a values and on CIFAR-100 for a S 0.02.
7

---

Conformity Score Averaging for Classification
0.01
0.05
selected by our proposed method. The results are visualized in Figures 4 and 5, which demonstrate the effectiveness
Method
Coverage
Size
Coverage
Size
VFCP
0.990 (0.003)
13.782 (1.114)
0,950 (0,005)
3,890 (0.266)
EFCP
0.989 (0.003)
13.306 (0.464)
0.949 (0.005)
3.754 (0.096)
of our weighted combination method in terms of weighing scores of different models.
DLCP
0.989 (0.003)
13:298 (0.461)
0.949 (0.005)
3.752 (0.097)
DLCP+
0.989 (0.003)
13.299 (0.459)
0.949 (0.005)
3.753 (0.097)
APS
0.990 (0.003)
40.217 (1.786)
0.949 (0.006)
19.545 (1.022)
THR
0.989 (0.003)
26.949 (2.198)
0.949 (0. 0D6)
4.519 (0.381)
cifar10 (APS)
cifar (RAPS)
RANK
0.990 (0.003)
27.161 (2.421)
0.950 (0.006)
6,298 (0.395)
Dure
3.00
RAPS
0.990 (0.003)
26.403 (2.505)
0.950 (0.005)
5,581 (0.263)
renno66
2.75
resnet56
SAPS
0.990 (0.003) 25.096 (2.298)
0.950 (0.006)
5.036 (0.302)
1.8
vgpll..hn
2.50
vp916.bn
2.25
Table 1. Coverage and size of different methods for a = 0.01 and a = 0.05 on CIFAR-100 dataset. Results are shown as mean (standard deviation). The first four methods corresponds to the splitting methods in Section 2.5.
2.00
1.4-
1.75
1.50
1.2
1:25
1.02
0.55
0.56
0.37
0.98
0.99
0.55
0.96
0.37
0.38
0.95
Coverage
Goverage
cifar10 (SAPS)
cifar10 (THR)
2.0
Notably, THR exhibits strong performance on less chal- lenging tasks (CIFAR-10), where our weighting scheme naturally assigns it dominant weights. In these cases, score averaging provides limited improvement. However, for complex multi-class scenarios (CIFAR- or stringent sig- nificance levels (a < 0.05), our method demonstrates clear superiority by optimally combining score functions.
Gura
resnet56
ressent
shifflenes2__1___
1.8
shuffbeetv2_s1__
2.50
v9926, bi
vopl6 on
2.25
1.6
0
2.4
1.2
1.0
0:35
0.36
0.97
0.36
0.36
0.37
0.30
0.36
Coverage
Coverage
Furthermore, to explore the influence of different data split- ting strategies (as discussed in Section 2.4), we conducted additional comparisons of our weighted score function using various split approaches. We also included the performance results of the five individual score functions when utilizing the VFCP split method. Table provides a comprehensive summary of the coverage and size metrics significance levels of = 0.01 and = 0.05. These results highlight the effectiveness of the different data splitting strategies. It is important to note that while APS aims to achieve conditional coverage, the other methods, including our approach, do not specifically target conditional coverage. Consequently, is not surprising that the alternative methods exhibit bet- ter efficiency compared to APS. Additional experimental results comparing our method with baseline approaches and Synergy Conformal Prediction on MNIST, Fashion-MNIST, and ImageNet- Val datasets are provided in Appendix F.
Figure 4. Across various score functions, our weighted combina- tion of models outperformed any individual model and achieved optimal size on the CIFAR-10 dataset across values (0.01-0.05
5. Related Work
Conformal prediction (CP) (Vovk et al., 2005) is a method- ology designed to generate prediction regions for variables of interest, facilitating the estimation of model uncertainty by providing prediction sets rather than point estimates. CP has been successfully applied to both classification (Luo & Zhou, 2024; Luo & Colombo, 2024) and regression tasks (Luo & Zhou, 2025c;d). Its flexibility allows adap- tation to various real-world scenarios, including segmenta- tion (Luo & Zhou, 2025a), games (Luo et al., 2024; Bao et al., 2025), time-series forecasting (Su et al., 2024), and graph-based applications (Luo et al., 2023; Tang et al., 2025; Luo & Zhou, 2025b; Wang et al., 2025; Luo & Colombo, 2025; Zhang et al., 2025).
4.2. Model Weighting
In addition to choosing the score functions in Section 2.2, we further conducted an experiment to compare the per- formance of various models and their weighted combi- nations for prediction set construction on CIFAR-10 and C000R-100. For CIF we used the models ResNet- 56, ShuffleNet22 (1.0x), and VGGI6-BN. For CIFAR- 100, used the models VGG16-BN, RepVGG-A2, and Mo- bileNetV2 (1.0x). The performance of the weighted combi- nation method was compared against individual models.
Our work builds upon advances in conformal prediction, par- ticularly in model averaging and calibration. In model aggre- gation, Yang et al. (Yang & Kuchibhotla, 2024) introduced two selection algorithms minimize the width of prediction intervals by aggregating and selecting from multiple regres- sion estimators. Various VC dimension techniques have been employed in (Yang & Kuchibhotla, 2024) and (Candès et al., 2023): in Section D.2 of (Yang & Kuchibhotla, 2024), the uniform probability bound is established over a finite set, whereas (Candès et al., 2023) considers maximization over real numbers. However, neither approach incorporates
Specifically, for each of the four score functions: THR, APS, RAPS, and SAPS, we generated prediction sets by combining the scores of different models using weights
8

---

Conformity Score Averaging for Classification
tifar100 (APS)
cifarl DO (RAPS)
score combinations.
Durs
Gurs
35
19916_ on
repvog a2
reprigg. az
v9016,bn bn
These results suggest several directions for future work:
mob 1ene
mobi enervo_xL_
1. Regression Extension: Algorithmic adaptation to re- gression requires new theoretical tools due to contin- uous output spaces, where finite-class union bounds become inapplicable. Potential approaches include metric entropy analysis or covering number techniques.
20
0.97
0.35
ci (SAPS)
cifar100 (THR)
2. Optimization Enhancement:
Developing gradient- based alternatives to grid search, such differentiable conformal objectives or online weight adaptation dur- ing model training, would improve scalability with many score functions.
Our
35-
vag15 bn
Ou'S
25
20
A Acknowledgment
0.95
0.95
0.57 Coverage
0.08
0.50
0.95
0.35
0:37
0.95
0.30
Coverrage
This work was partially supported by Hong Kong RGC and City University of Hong Kong grants (Project No. 9610639 and 6000864), DFG grant No. 389792660, and Volkswa- genStiftung Grant AZ 98514. Zhixin Zhou's 's research was supported by the Genesis Award for Scientific Breakthrough from Alpha Benito LLC.
Figure 5. Across various score functions, our weighted combina- tion of models outperformed any individual model and achieved op- timal size on the CIFAR-IOK dataset across a values (0.01-0.05).
subgraph theory techniques as in our method. Additionally, the model aggregation methods in (Carlsson al., 2014; Li- nusson et al., 2017) combine predictions from multiple models for conformal prediction, but they do not emphasize optimizing efficiency.
Impact Statement
This work contributes to the broader goal of improving ma- chine learning models' reliability and uncertainty quantifi- cation, which has the potential for positive socictal impact across various domains.
concurrent preprint (Liang et al., 2024) examines parametrized score functions sx (x y) for À E within regression settings. While their general framework supports arbitrary parameter spaces, it faces challenges in control- ling Rademacher complexity, except in certain specialized cases. In contrast, our approach linearly combines pre- defined score functions, introducing a structural constraint that allows theoretical guarantees to depend solely on the number constituent scores, rather than the complexity of the function class.
References
Angelopoulos, N., Bates, S., Jordan, M., and Malik, J. Uncertainty sets for image classifiers using conformal prediction. In International Conference Learning Representations, 2021. URL htttp:://ppeereeee//e net/ ffr:/formm?? Sideeddiu_eddbbbaaeee _DMM____DMMMM_
Angelopoulos, A. N., Bates, S., al. Conformal predic- tion: A gentle introduction. Foundations and Trends@ in Machine Learning, 16(4):494-59)
6. Conclusion and Discussion
Bao, J., Dang, C., Luo, R., Zhang. H., and Zhou, Z. En hancing adversarial robustness with conformal prediction: A framework for guaranteed model reliability. In Pro- ceedings of the Forty-second International Conference Machine Learning (ICML), 2025. to appear.
Our weighted score aggregation method enables efficient and valid prediction set construction for multi-class classifi- cation through optimized combinations of score functions and strategic data splitting. Theoretically, we establish finite- sample coverage guarantees and oracle inequalities quanti- fying the efficiency gap between our method and optimal weights. Empirically, experiments demonstrate consistent maintenance of coverage requirements with minimal pre- diction set sizes compared to single-score baselines. This work bridges model averaging and conformal prediction, providing a flexible framework for uncertainty quantifica- tion that adapts to dataset characteristics through optimal
Candès, E., Lei, I.., and Ren, 7.. Conformalized survival analysis. Journal of the Royal Statistical Society Series B: Statistical Methodology, 85(1):24-45, 2023.
Carlsson, L.., Eklund, M., and Norinder, U. Aggregated con- formal prediction. In Artificial Intelligence Applications and Innovations: AIAI 201 Workshops: CoPA, MHDW.

---

Conformity Score Averaging for Classification
IIV,, and MT4BD, Rhodes, Greece, September 19-21, 2014. Proceedings 10, pp. 231-240. Springer, 2014.
Luo, R. and Zhou, Z. Conditional conformal risk adaptation. arXiv preprint arXiv:2504.07611 2025a.
Claeskens, and Hjort, L.. Model selection and model averaging. Cambridge books, 2008.
Luo, R. and Zhou, Conformalized interval arithmetic with
symmetric calibration. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 39, pp. 19207- 19215, 2025b.
Colombo, N. On training locally adaptive cp. In Conformal and Probabilistic Prediction with Applications, pp. p 384- 398. PMLR, 2023.
Luo, R. and Zhou, Z. Conformal thresholded intervals for ef- ficient regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 19216-19223, 2025c.
Colombo, N. Normalizing flows for conformal regression. In The 40th Conference on Uncertainty in Artificial Intel- ligence, 2024. URL https: /wwp/reveveeeeeeeeeeeeeeeeeeeeee ew, net./ forum? ddw.::dwwwawwwacccccccc..
Luo, R. and Zhou, Z. Volume-sorted prediction set: Efficient conformal prediction for multi-target regression. arXiv preprint arXiv:2503.cm 022005 2025d.
Gauraha, N. and Spjuth, O. Synergy conformal prediction. In Conformal and Probabilistic Prediction and Applica- lions, pp. 91-110. PMLR, 2021.
Luo, R., Nettasinghe, B., and Krishnamurthy, V. Anoma- lous edge detection in edge exchangeable social network models. In Conformal and probabilistic prediction with applications, pp. 287-310. PMLR, 2023.
Guan, L. Localized conformal prediction: A generalized in- ference framework for conformal prediction. Biometrika, 110(1):33-50, 2023.
Luo, R., Bao, J., Zhou, Z., and Dang, C. Game-theorete defenses for robust conformal prediction against ad- versarial attacks in medical imaging. arXiv preprint arXiv:241.003776, 2024.
Huang. J., Xi, H., Zhang, L.. Yao, H., Qiu, Y., and Wei, H. Conformal prediction for deep classifier via label ranking, 2024. URL https:://// http://ppeeeeeeeeeeeeeem / forum? 1d=zzVm3JqJ7s..
Manokhin, Awesome conformal prediction. you use Awesome Conformal Prediction. please cite it as below, 2022.
Kivaranovic, D., Johnson, K. D., and Leeb, H. Adaptive, distribution-free prediction intervals for deep networks. In International Conference on Artificial Intelligence and Statistics, pp. 4346 4356. PMLR, 2020.
Papadopoulos, H.,, Proedrou, K., Vovk, V., and Gammerman, Inductive confidence machines for regression. Ma- chine Learning: ECML 2002: 13th European Conference Machine Learning Helsinki, Finland, August 19-23, 2002 Proceedings 13, pp. 345-356. Springer, 2002.
Lei, J. and Wasserman, L. Distribution-fiee prediction bands for non-pane parametric regression. Journal of the Royal Sta- tistical Society: Series Statistical Methodology, pp. 71-96, 2014.
Papadopoulos, H., Gammerman, A., and Vovk, Normal- ized nonconformity measures for regression conformal prediction. In Proceedings of the IASTED International Conference on Artificial Intelligence and Applications (AIA 2008), pp. 64 19, 2008
Liang, R., Zhu, W., and Barber, R. F. Conformal prediction after efficiency-oriented model selection. arXiv preprint arXiv:2408.0 070666 2024.
Linusson, H., Norinder, U., Boström, H., Johansson, U., and Löfström, T. On the calibration of aggregated conformal predictors. In Conformal and probabilistic prediction and applications, pp. 154 173. PMLR, 2017.
Papadopoulos, H., Vovk, V., and Gammerman, A. Regres- sion conformal prediction with nearest neighbours. Jour- nal of Artificial Intelligence Research, 40:815-844 201
Luo, R. and Colombo, N. Entropy reweighted conformal classification. The 13th Symposium on Conformal and Probabilistic Prediction with Applications, pp. 264-276. PMLR, 2024.
Romano, Y., Patterson, E., and Candes, E. Conformalized quantile regression. Advances in neural information pro- cessing systems, 32, 2019.
Romano, Y., Sesia, M.. and Candes, E, Classification with valid and adaptive coverage. Advances in Neural Infor- mation Processing Systems, 33:3581-3591, 2020.
Luo, R. and Colombo, N. Conformal load prediction with transductive graph autoencoders. Machine Learning, 114 (3):1 2025.
Sadinle, M., Lei, J., and Wasserman, L. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Staristical Association, 114(525):223- 234, 2019.
Luo, R. and Zhou, Z. Trustworthy classification through rank-based conformal prediction sets. arXiv preprint AXXv:2407.044477, 2024.
10

---

Conformity Score Averaging for Classification
Su, X., Zhou, Z.,, and Luo, R. Adaptive conformal inference by particle filtering under hidden markov models. arXiv preprint wrxiv:241..01588, 2024.
Tang, L., Luo, R., Zhou, Z., and Colombo, N. Enhanced route planning with calibrated uncertainty set. Machine Learning, 116,5):1-1,, 2025.
Vovk, V., Gammerman, A., and Shafer, G. Algorithmic learning random world, volume 29. Springer, 2005.
Vovk, V., Nouretdinov, I., Manokhin, V., and Gammerman, A. cross-conformal predictive distributions. In conformal and probabilistic prediction and applications, pp. 37-51. PMLR, 2018.
Wainwright, M. J. High-dimensional stafistics: A non- asymptotic viewpoint, volume 48. Cambridge university press, 2019.
Wang. I., Zhou, Z., and Luo, R. Enhancing trustworthiness of graph neural networks with rank-based conformal train- ing. Proceedings of the AAAI Conference on Artificial Intelligence, (1221)21221121221 2025.
Yang, Y. and Kuchibhotla, A. K. Selection and aggregation of conformal prediction sets. Journal of the American 2024.
Statistical Association, pp. 2024.
Zhang, Z., Bao, J., Zhou, Z., Colombo, N., Cheng, L., and Luo, R. Residual reweighted conformal prediction for graph neural networks. In Proceedings of the 41st Conference on Uncertainty in Artificial Intelligence (UAI), 2025. to appear.

---

Conformity Score Averaging for Classification
A. Grid Search Implementation
To solve (4), we discretize the probability simplex:
d w E
A dd1
R
==1
using a grid resolution E = 0.01. Candidate weights are generated as:
w (Kie,..... ..l.,,...................... kads) W
Kj ENE, N, Σk₃ = [1/6]
j=1
yielding (1//)++-1) distinct weight vectors. For d 3 scores, this produces 5,151 candidates, ensuring comprehensive
coverage of the parameter space while remaining computationally tractable through parallel evaluation.
B. Proof of Lemma 1
B.1. VC Dimension of the Subgraph Classes
Proposition Both of the following classes of functions
{(,, v) + 1 (((,, ft: : E ERAtER)
(9)
and
{x+ ( (w,,s,,,)) v)) 2 f w w tERR).
(10)
where y [K] is fixed. have VC-dimension at most d + 1.
Proof. Both the classes are the subgraph classes of vector space of functions with dimension d + 1. By Proposition 4.20 of (Wainwright, 2019), these subgraph classes have dimension at most + This is also a direct result Example 4.21 in the book.
B.2. Proof the Lemma
This proof basically follows from "apnik-Chervonenkis theory. We will use theorems and lemmas in (Wainwright, 2019) as reference. The proof of both parts of the lemma are almost identical. We will focus on part (a). Let be i..d. symmetrie random variables take value -1 1, i.c. IP(:, 1) PPee 1) 0.5. Then we define the Rademacher complexity for the function class in (9),
1
R(I)
E
sup
fl(( s(X,, Yi)) 2
I
By Theorem 4.10 in the book,
I
(I, + 6)) exp
(11)
By Proposition the VC dimension of the function class in (9) Then altogether with Lemma 4.14 and Proposition
1..8 in the book,
R(T)= < (d + 1) log( +
(12)
I
12

---

Conformity Score Averaging for Classification
We combine (11) and (12) to obtain the result of part (a). The proof of part (b) of the lemma follows analogously, using the VC dimension result for the function class in (10). Let define
T (I,n))
sup WER
2 2 -0------
13) 5n.
EE
For I C Tarain U Trest
(d+ 1) log( [ + 1) + 6)) 1)
21 exp (-
(-1216²)
[2]6²
IP Ty
I, 1.
(-2
I
2
clear that
CT(Z, DE[K]
U
Taking the union bound on E [K] implies the result of part (b).
C. Preliminary Lemmas for the Theorems
C.1. Bounds for Coverage Probability
The following result requires the data for optimizing w are independent with the calibration set and the test The algorithm satisfying these conditions have the most reliable coverage probability.
Lemma 2. Let (z)a(x) be the output of Algorithm 2. Suppose
(1) ((X;,Y)))iEt, YE) are exchangeable.
(ii) ((X,, Yi)iez,u, and {(Xi, Yi) Mezs are independent.
Then for (X in the test set, the coverage probability
P(Y E (x)=
Proof. t only depends on {(X,, Ye)),,,, so it is independent of {(X,, Y.) Hez The weighted score function {(00, s(X,, K.)) are also exchangeable. For any (X,Y) in the test set, the rank of s(X, Y) is smaller than (1 In) with probability [(1+|2,]]1110 2 aaa
Lemma 3. Lei (a) be the oulput of Algorithm 2. Suppose
(i) ((Ts, nt) is satisfied.
(ii ((X,,Y))) and ((Xi, YOJEE are independent.
Then for (X, Y) in the test set, the coverage probability
1
P(Y E (i-)
[11 + [T()(1 aa] - 773
Izl
Proof. Let write the threshold in (5) h-a (w, Iz) to emphasize that this quantile depends i and the samples in Iz. By the procedure of the algorithm, we have
(2)
P p( E CI-a(X)) = P (((0, s(x, 2 (w. Iz))
13

---

Conformity Score Averaging for Classification
Assuming the event [([3, 113).
[(1+ [a)((1 -01
Q1 Q1,a sup
t R
Σ
((
I3
[3
1
(13)
2 sup
t E P( s(, Y)) 2
(1+ [sl) a)1 - 173
Ta
QQ- (w)
1 where a'= = [7g
(1 + [sl)(1 - a)] - 1/3- By the definition of the quantile,
P((0, sXX, Y)) 2 91-a (0,
(w. Iz) (( s(X, 2 Q1-or 11-d
(14)
The proof is complete.
Lemma 4. Let Č1-a w) be the output of Algorithm 2. Suppose [2(Is; 17)) and S2 Thea) hold, then the coverage
proportion satisfies
1
Σ
((ai)} 2
[11 [Ta))(1 ox)] 9/3 Qeer--
Trest
[z
iEIum
Proof. Under the event De(Ta mea)) for all w E W.
1(00, (X,, Vi) 2 (0, ((x,, ((x, 2 (w, Is)}]
Treal
iETia
Σ s(x,, 2 t (x,, 2 t}} S nest
sup EER
Irest
WER
sEZua
where (X, Y) is an i.i.d. copy of (X,, Yj) in the test set. particular, we let and 91-a (w, Iz), we have
(in s( (X Y)) Q1 [s)} in = P(( s s(X, Y)) 2 (m, Ia) i a.
The remaining proof has similar argument as (13) and (14), and is omitted here.
C.2. Bounds for Prediction Set Size
Lemma 5. Suppose the samples in In and I2 satisfy Q(I,, m)) and [I {2) respectively, and suppose Q 1 (10, (i) s
Q1-a(1; Iz), then for X in the test set, the expected prediction set size satisfies
1
Σ
1f
((0,s(XX,, s(x,, v)) 2 (w. E
Σ [E(
1f(w" s(X,))) 2 Q1-an (w*)} + +
[2
TETa yE|K]
where 1- a-a1= = + (L [1)(1 a')1 7/1
Proof. Under the assumption Qi-a! (w,Ir) < 91-a Iz), for w E W, we have
Σ Σ s((,, w) 2 - (w,
1
Σ
1(( s(X,))) (w.)}.
LZ
Iz
K)
iEI,
WE(K]
t is obtained from
w E arg min NEW I
Σ
Σ
1((w,s(X,))) 2 (w,
iET, yE[K]
14

---

Conformity Score Averaging for Classification
Therefore,
Σ Σ 1f ,(),))) Q1-a iE WE[K]
1 T
Σ
Σ
1f(w", s(X,))) 2 Q1-ar(w", (I
TT yE[K]
Given the event (2(I,, m), for W and t e E,
P((w, sX, Y)) 2 1) ( 24)
6 2 20 m-
I
EEI
The lower bound the LHS is also a lower bound of the RHS, so we have
1
1
Qw-a'(w II, sup
E ::
(( s(,,, 242
[(1 (1
I1
(15)
2 sup {
R (( s(X, Y)) 21)
[[1
(1 - d')1+ +m)
[
Q: a- (w"),
where! 1 - 01
[11 + [1)(1 - a')1 + 1 and Q1-ai (w") is the (1 - qua-quae of the population distribution
(w, s( X, Y)). Now we can conclude that
Σ
1(( s(X. w)) 2 Q1-a' (w= [1)]
s
Σ Σ
If(w0 ,(X,))) 2 Q1 (w")).
IEZZ WE (K(
.ETz DE(K)
On the event E2),
Σ Σ ((10",(Xxx)) (Xx,0) 2 Q1 Q SB
Σ
(((w",((xx)) 2 Q1-an (w*)} + E2.
L
BEIN (E(K]
The proof is complete.
D. Proof of the Theorems
D.1. Proof of Theorem
Under the assumption of the theorem, the setting of VFCP satisfies the condition of Lemma 2. This proves the coverage probability in the theorem. For the expected prediction set size, under the event I E2), by Lemma 5, for X in the test set,
E
Σ [VE[K]
2 (w,
Σ
Σ
H( s(,, w) 2 2-- (w0, + Eo.
Iz
IEIa yE[K]
This implies the coverage probability of the theorem.
D.2. Proof of Theorem
Under the assumption the theorem. the setting of VFCP satisfies the condition Lemma This proves the coverage probability in the theorem. For the expected prediction set size, since I₁ I3 and I(I2, (2) is satisfied, by Lemma 5, for X in the test set,
Σ ((,,, 2 (w, Iz)l [VE[K]
Σ
Σ
1((60, s(x,,)) 2 a1 (w, [s)] } + E2-
TeTa yE[K]
This implies the coverage probability of the theorem.
15

---

Conformity Score Averaging for Classification
D.3. Proof of Theorem 3
Since 1 -3 Irrain and Σ2 Tesst one can verify that the theorem is the direct result of Lemma and Lemma 5,
D.4. Proof of Theorem
Similar as the setting of DLCP, the coverage probability is the direct result of Lemma 4. For the prediction set size, suppose [[I, {2) holds, then for X in the test set,
B B (X)I
(X)]
Σ
( Iz)} + E2
I
iEIa yE|K]
S E
Σ
1(100
,(X,")) w)) 2 Q (1-0r(10")} + 2f2.
E. Additional Discussion on Score Functions
The three basic score functions used in our experiment were:
Least Ambiguous Set Values Classifier (THR) (Sadinle et al., 2019).
The score function of THR is defined as:
STHR (I., v) (x)
This score function is straightforward: it assigns higher score labels with a higher estimated probability, The prediction set includes labels with the highest estimated probabilities. This is the score function have include because if p is the true posterior probability, then the score function itself can achieve the smallest expected prediction set size.
Adaptive Prediction Set (APS) (Romano et al., 2020).
The score function of APS is defined as:
SAPS x y) =
Σ y y'E[K]
(P) - (x)),
where 1 is the indicator function. This score funetion can be interpreted as the complement of the p-value for label k. measures the sum of the estimated probabilities of all labels that have the same or smaller estimated probability than label k.
Rank-based Score Function (RANK) (Luo & Zhou, 2024). The score funetion of RANK is defined as:
SRANK (z, y) I(K' E [K] (x) < Pc(x))
K-1
This score function assigns score based the rank of the estimated probability (1)) among all the estimated probabilities for input æ. The rank is divided by K 1 so that the range of the score is from 0 to 1. The prediction set gives higher priority to labels with larger ranks.
The three score functions, THR, APS, and RANK, employ different strategies for assigning scores based on the estimated probabilities. THR directly utilizes the estimated probabilities as scores. APS, on the other hand, considers the cumulative probability labels that have the same or lower estimated probabilities compared the label interest. RANK, in contrast, focuses on the relative ranking of the estimated probabilities among all possible labels. score functions preserve the order of the labels, which means that, for fixed x, the order of labels based on their estimated probabilities, Py (x), remains the same when ranked according their scores, s(x, y).
important to note that all these
16

---

Conformity Score Averaging for Classification
F. Additional Experiments with Different Datasets
We conducted additional experiments on MNIST, Fashion-MNIST and ImageNet- Val, including comparisons with Synergy Conformal Prediction (SCP) method (Gauraha Spjuth, 2021). suggested by Reviewer bRdZ. Each experiment used 2000 samples, 100 runs with different calibration/test splits (as in Table 1, Section 4.1), the APS score function, and the EFCP split method. Results show method consistently achieves smaller prediction sets while maintaining coverage at 0.01 and a 0.05.
E.1. MNIST
Method
Coverage (a 0.01)
Size (a 0.01)
Coverage (a 0.05)
Size (a 0.05)
Ours
0.988 (0,005)
1,577 (0.061)
0.951 (0.011)
1.001 (0.011)
SVM
0.990 (0.005)
2.323 (0. (3.135)
0.950 (0.011)
1.033 (0.014)
Random Forest
0.990 (0,005)
2.205 (0.108)
0.951 (0.013)
1.181 (0.023)
Logistic Regression
0.990 (0.005)
3.695 (0.123)
0.950 (0.012)
1.557 (0.065)
SCP
0.990 (0.005)
1.771 (0.083)
0.951 (0.012)
1.018 (0.011)
Table 2. Results on MNIST dataset.
F.2. MaShion-MNIST
Method
Coverage (a = 0.01)
Size (a = 0.01)
Coverage (a = 0.05)
Size (a 0.05)
Ours
0.988 (0.006)
2.296 (0.108)
0.948 (0.012)
1.265 (0.031)
SVM
0.991 (0.005)
2.941 (0.156)
0.952 (0.012)
1.449 (0.044)
Random Forest
0.990 (0.006)
3.264 (0.187)
0.949 (0.012)
1.612 (0.035)
Logistic Regression
0.989 (0,006)
3.325 (0.1
0.949 (0.012)
1.841 (0.050)
SCP
0.991 (0.005)
2.446 (0.100)
0.951 (0.013)
1.315 (0.030)
Table 3. Results on Fashion-MNIST dataset.
F33 ImageNet- Val
Method
Coverage (a = 0.01)
Size (a = 0.01)
Coverage (a = 0.05)
Size (a = 0.05)
Ours
0.989 (0.006)
48.264 (4.019)
0.949 (0.013)
6.670 (0.618)
Re
0.990 (0.005)
53.744 (3.940)
0.950 (0.013)
6.798 (0.631)
VGG16
0,991 (0.005)
100.683 (9.445)
0.950 (0.011)
15.149 (0.831)
ResNet18
0.990 (0.006)
110.401 (11.694)
0.949 (0.011)
18.323 (1.422)
SCP
0.991 (0.005)
77.243 (5.667)
0.950 (0.011)
10.661 (0.630)
Table 4. Results ImageNet- Val dataset.
17

---

Conformity Score Averaging for Classification
G. Additional Experiments with Different Data Splitting Ratio
train
Lest
80:20
Coverage
Size
0.960
4.8
4.6
0.955
4.4
0.950
4.2-
4.0
0.945
3.8
0.940
3.6
0.935
3.4
VFCP
EFCP
DLCP
DLCP+
VFCP
EFCP
DLCP
DLCP+
Method
Method
(a) Comparison coverage and size for different data split methods a significance level a 0.05 when Turin Test 80:20. EFCP
DLCP, and DLCP+ exhibit similar size results, but DLCP has the smallest coverage and the largest gap from the desired coverage level a = 0.95. VFCP attains the desired coverage at the cost of having the largest prediction set size.
train
test = 99:1
Coverage
Size
4.75
0.98
4.50
0.96
4.25
0.94
4.00
0.92
3.75
3.50
0.90
3.25
0.88
3.00
VFCP
EFCP
DLCP
DLCP+
VFCP
EFCP
DLCP
DLCP+
Method
Method
(b) Comparison of coverage and size for different data split methods at significance level of a 0.05 when Imain Inse 99:1. DLCP achieves the smallest size among the compared methods. VFCP attains the desired coverage at the cost of having the largest prediction set size.
Figure 6. Comparison of coverage and size for different data split methods a=0.05.
18


Reviews:
[{"review_id": "gAW6lDkLUq", "reviewer": "Reviewer_82f9", "summary": "Existing conformal prediction methods typically rely on a single conformity score function, limiting both their efficiency and informativeness. In this paper, they propose a new approach that enhances conformal prediction by averaging multiple conformity score functions for the same classification task. They also provide a comprehensive theoretical analysis based on VC theory, demonstrating the effectiveness of our method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and helpful suggestions. We also appreciate your careful review of the supplemental materials. Below, we will address your comments.\n\n---\n\n### **1. About the intuition of VC dimension in our proof**\n\nWe agree that providing this intuition would benefit readers. The primary idea relies on the Dvoretzky–Kiefer–Wolfowitz (DKW) inequality. If the supremum in equation (7) or (8) is taken over a single variable, e.g., $q \\in \\mathbb{R}$ or $w_1 \\in \\mathbb{R}$, the DKW inequality can be directly applied.  \n\nUsing the VC dimension generalizes the DKW inequality, allowing the supremum to extend over multiple variables when the function is linear in those variables. This generalization underpins our proof's validity and efficiency. We will add this explanation to the revision.\n\n---\n\n### **2. About the theoretical contribution compared to existing papers**\n\nIn Section 5, we compare our method to two relevant approaches and can elaborate as follows:  \n\n- The method in [1] optimizes over $w \\in \\\\{e_1, \\dots, e_d\\\\}$, where $e_i$ is the $i$-th standard basis in $\\mathbb{R}^d$. In contrast, we optimize over $w \\in \\Delta^d$, considering a much larger set of candidate score functions. Both methods show mild validity loss, but our approach is more efficient.  \n\n- The concurrent preprint [2] defines score functions as $s_w(x, y)$ compared to our $\\langle w, s(x, y) \\rangle$. While both use Rademacher complexity to analyze validity and efficiency, [2] cannot provide implicit bounds for validity or prediction set size via subgraph theory, as we do. This highlights our unique theoretical contribution.\n\n---\n\n### **3. About experiments**\n\nWe conducted additional experiments on MNIST, Fashion-MNIST, and ImageNet-Val, including comparisons with the Synergy Conformal Prediction (SCP) method [3], suggested by Reviewer bRdZ. Each experiment used 2000 samples, 100 runs with different calibration/test splits (as in Table 1, Section 4.1), the APS score function, and the EFCP split method. Results show our method consistently achieves smaller prediction sets while maintaining coverage at α= 0.01 and = 0.05.\n\n#### **MNIST**\n| Method                | Coverage (α=0.01) | Size (α=0.01) | Coverage (α=0.05) | Size (α=0.05) |\n|-----------------------|--------------------|---------------|--------------------|---------------|\n| **Ours**             | 0.988 (0.005)     | **1.577 (0.061)** | 0.951 (0.011)     | **1.001 (0.011)** |\n| SVM                  | 0.990 (0.005)     | 2.323 (0.135) | 0.950 (0.011)     | 1.033 (0.014) |\n| Random Forest        | 0.990 (0.005)     | 2.205 (0.108) | 0.951 (0.013)     | 1.181 (0.023) |\n| Logistic Regression  | 0.990 (0.005)     | 3.695 (0.123) | 0.950 (0.012)     | 1.557 (0.065) |\n| SCP                  | 0.990 (0.005)     | 1.771 (0.083) | 0.951 (0.012)     | 1.018 (0.011) |\n\n#### **Fashion-MNIST**\n| Method                | Coverage (α=0.01) | Size (α=0.01) | Coverage (α=0.05) | Size (α=0.05) |\n|-----------------------|--------------------|---------------|--------------------|---------------|\n| **Ours**             | 0.988 (0.006)     | **2.296 (0.108)** | 0.948 (0.012)     | **1.265 (0.031)** |\n| SVM                  | 0.991 (0.005)     | 2.941 (0.156) | 0.952 (0.012)     | 1.449 (0.044) |\n| Random Forest        | 0.990 (0.006)     | 3.264 (0.187) | 0.949 (0.012)     | 1.612 (0.035) |\n| Logistic Regression  | 0.989 (0.006)     | 3.325 (0.120) | 0.949 (0.012)     | 1.841 (0.050) |\n| SCP                  | 0.991 (0.005)     | 2.446 (0.100) | 0.951 (0.013)     | 1.315 (0.030) |\n\n#### **ImageNet-Val**\n| Method                | Coverage (α=0.01) | Size (α=0.01) | Coverage (α=0.05) | Size (α=0.05) |\n|-----------------------|--------------------|---------------|--------------------|---------------|\n| **Ours**             | 0.989 (0.006)     | **48.264 (4.019)** | 0.949 (0.013)     | **6.670 (0.618)** |\n| ResNet101            | 0.990 (0.005)     | 53.744 (3.940) | 0.950 (0.013)     | 6.798 (0.631) |\n| VGG16                | 0.991 (0.005)     | 100.683 (9.445) | 0.950 (0.011)     | 15.149 (0.831) |\n| ResNet18             | 0.990 (0.006)     | 110.401 (11.694) | 0.949 (0.011)     | 18.323 (1.422) |\n| SCP                  | 0.991 (0.005)     | 77.243 (5.667) | 0.950 (0.011)     | 10.661 (0.630) |\n\n---\n\n### **References**\n\n[1] Yang, Y., and Kuchibhotla, A. K. (2024). Selection and aggregation of conformal prediction sets. _Journal of the American Statistical Association_, 1–13.  \n\n[2] Liang, R., Zhu, W., and Barber, R. F. (2024). Conformal prediction after efficiency-oriented model selection. _arXiv preprint arXiv:2408.07066_.  \n\n[3] Gauraha, N., and Spjuth, O. (2021). Synergy conformal prediction. _Conformal and Probabilistic Prediction and Applications. PMLR_.", "claims_and_evidence": "The proposed research problem is clearly defined, and the discussion around using a single score function is valid. The approach of combining multiple score functions with data splitting to enhance conformal prediction is logically sound. Furthermore, the paper provides comprehensive theoretical support to demonstrate the advantages of the proposed method, and the experiments empirically validate its claims.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are logically well-founded for the stated problem.", "theoretical_claims": "Yes. The theoretical claims are comprehensive and are supported by a detailed proof.", "experimental_designs_or_analyses": "Yes. The overall experimental design makes sense, but expanding the experimental settings would make the work more solid.", "relation_to_broader_scientific_literature": "Conformal prediction is a flexible technique that can be applied in a variety of domains. The methods proposed in this work further advance the development of conformal prediction.", "essential_references_not_discussed": "n.a.", "other_strengths_and_weaknesses": "Strengths:\n1. The research problem is intriguing, as it explores how to apply multiple conformal score functions to a single classification task.\n2. The proposed method is both intuitive and logically sound.\n3. The theoretical analysis is thorough, providing a comprehensive basis for the method’s correctness.\n\nWeaknesses\n1. The rationale for using the VC dimension should be explained in a more intuitive manner, making it easier for readers to understand its relevance.\n2. Emphasize how the proposed theoretical contributions surpass or improve upon existing methods, which will help highlight the significance of the work.\n3. For the experimental evaluation, only two datasets were used, which may limit the persuasiveness of the results. Including additional datasets or varying the experimental settings would further strengthen the evidence for the proposed method.", "other_comments_or_suggestions": "see weakness", "questions_for_authors": "n.a.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u3Zm7yKAid", "reviewer": "Reviewer_j2fr", "summary": "In this paper, the authors presented an approach that enhances conformal prediction for multi-class classification by optimally averaging multiple conformity score functions, and a set of evaluation experiments showed that the weighted averaging approach consistently outperforms single-score methods by producing smaller prediction sets without sacrificing coverage.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and for appreciating the organization and motivation of our paper. Below, we address your questions and comments.\n\n---\n\n### **1. About the complexity of the proposed approach**\n\nIn our algorithm, the main source of complexity lies in finding the optimal weight $\\hat{w}$. Specifically, we need to compute the prediction set size $O(1/\\epsilon^d)$ times, where $\\epsilon$ is the grid size, as detailed in Section A of the appendix.  \n\nWe emphasize that evaluating prediction sets is computationally efficient for each iteration. Although the complexity is exponential in $d$, the process of finding $\\hat{w}$ is still significantly faster than training deep learning models for classification tasks.  \n\nTo address scalability when the number of score functions is small, we suggest the following greedy approach:  \n\n1. Start with $w_1 = 1$.  \n2. Add $w_2$ so that $w_1 s_1 + w_2 s_2$ minimizes the prediction set size.  \n3. Continue iteratively, adding $w_3, w_4, \\dots$, ensuring each step minimizes the prediction set size.  \n\nAdditionally, we observed in our experiments that the prediction set size is often quasi-convex in $w_i$, allowing for optimization via the golden-section search method. However, we cannot guarantee convexity in general, meaning this greedy approach may not yield the theoretically optimal $\\hat{w}$ or satisfy the statistical guarantees in Section 3.  \n\nWe discuss possible additional efficient methods in our response to Reviewer gF9v, who has suggested a useful approach. Kindly refer to that response for further details.  \n\nThe focus of this work is on analyzing the properties of $\\hat{w}$, rather than developing efficient algorithms to compute it. We believe that designing efficient methods to find $\\hat{w}$ is an exciting direction for future work.\n\n---\n\n### **2. About the number of runs in the experiments**\n\nIf we understood your question correctly, the number of runs is 100. This is stated on line 368 of page 7. In each run, the data is randomly partitioned into training, calibration, and test sets.\n\n---\n\n### **3. About the impact statement**\n\nThank you for reminding us about the impact statement. We apologize for its omission. We will add the following statement to the revised manuscript:  \n\n*This work contributes to the broader goal of improving machine learning models' reliability and uncertainty quantification, which has the potential for positive societal impact across various domains.*", "claims_and_evidence": "A set of the experiments using CIFAR10 and CIFAR100 using different significance levels, and the performance comparisons \nof various models and their weighted combinations, *empirically* supported the outperformance of using the optimally averaging approach.", "methods_and_evaluation_criteria": "Some questions are listed as follows:\n\nCould the authors provide insights into the complexity comparison between the proposed approach and single-score methods? A clearer understanding would help the community assess the balance between computational overhead and performance improvement.\n\nIt is not clear in the main body how many runs are conducted for each experimental setting.", "theoretical_claims": "Given that the presented research field extends beyond my area of expertise, thoroughly assessing its theoretical soundness in detail is challenging. My current evaluation may be conservative and I will make my best effort to actively engage in discussions with the authors, other reviewers, and ACs.", "experimental_designs_or_analyses": "Please refer to the questions in the Methods and Evaluation Criteria Section.", "relation_to_broader_scientific_literature": "It looks fine.", "essential_references_not_discussed": "Seems not applicable.", "other_strengths_and_weaknesses": "Additional strengths: The paper structure is well organised, the motivation is clearly described.", "other_comments_or_suggestions": "Additional comment: Impact Statement is missing in the submission.", "questions_for_authors": "No futher questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b4NVkaAqWb", "reviewer": "Reviewer_gF9v", "summary": "The paper proposes a method for improving prediction set efficiency in classification tasks through conformity score averaging. It introduces weighted averaging of multiple score functions and explores various data-splitting strategies to optimize the weight selection process. Theoretical guarantees for coverage and efficiency are established using VC theory, and experiments on CIFAR-10 and CIFAR-100 datasets demonstrate the method’s effectiveness over existing approaches.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive review. Most of your questions are high-level and insightful. We will address them in order, starting with simpler questions and moving to more complex, open-ended ones.\n\n---\n\n### **1. About additional experiments**\n\nWe have conducted additional experiments on MNIST, Fashion-MNIST, and ImageNet-Val, as detailed in our response to Reviewer 82f9. These results will be included in the revised version of the paper.\n\n---\n\n### **2. Coverage vs. set size as a function of the calibration set size**\n\nIf we understand the question correctly, this refers to the relationship between calibration set size, coverage accuracy, and prediction set size. We have not systematically measured this relationship as a function, as this type of experiment is computationally expensive. Such an experiment would require multiple repetitions (e.g., 100 runs) to observe meaningful differences.\n\nIn Section F, we provide related experiments comparing two calibration set sizes. The results show that a larger calibration set provides more accurate coverage. However, the prediction set size also depends on the splitting method used. \n\n---\n\n### **3. About conditional coverage**\n\nOur method can be applied to conditional coverage. The current work focuses on improving the efficiency of conformal prediction, with the prediction set size as the key optimization criterion. However, this criterion can be adapted to metrics related to conditional coverage, such as group-conditional coverage, label-conditional coverage, or ECE. By adjusting the criterion and allowing the algorithm to search for $\\hat{w}$ that optimizes it, our method can generate conformal prediction sets tailored to meet conditional coverage requirements.\n\n---\n\n### **4. About cross-validation or bootstrap**\n\nWe do not believe the data leakage idea can be directly combined with cross-validation or bootstrap. The data leakage method uses the prediction set sizes of test samples to determine the weight $\\hat{w}$. Below, we outline our thoughts on how cross-validation and bootstrap might connect to this idea.\n\nA generalized version of the main algorithm could be considered. In Algorithm 2, $\\mathcal{I}_1$ is used to find $\\hat{w}$, and $\\mathcal{I}_2$ is used to determine the quantile corresponding to $\\hat{w}$. In VFCP, $\\mathcal{I}_1$ and $\\mathcal{I}_2$ are fixed partitions. A possible generalization would involve sampling $\\mathcal{I}_1$ and $\\mathcal{I}_2$ multiple times (e.g., through cross-validation or bootstrap). This process would result in multiple estimates of $\\hat{w}$. Averaging these estimates could yield a final $\\hat{w}$. The question is do we need another dataset to determine the threshold. This brings us back to the choice between VFCP and EFCP: whether to perform calibration on the same set or a separate set.\n\nWe do not have a clear theoretical answer on how these methods should be properly applied. We hypothesize that cross-validation or bootstrap could provide an intermediate solution between VFCP and EFCP.\n\n---\n\n### **5. About efficient methods to find $\\hat{w}$**\n\nWe have explored several methods to improve the efficiency of finding $\\hat{w}$. Some simpler methods are introduced in our response to Reviewer gF9v.\n\nAs you suggested, gradient descent is a potential approach. The main technical challenge is that the prediction set size is not a continuous function of $w$. To address this, we can approximate the indicator functions in the objective function using sigmoid functions as conformal training does. The success of this approach heavily depends on tuning the temperature parameter of the sigmoid function.\n\nWe have also considered stochastic optimization. For instance, starting from a random point $w$ on the grid $\\Delta^d$, we evaluate the prediction set size for $w$ and its neighbors. We then move to a neighboring point with a probability based on the prediction set size. We record the smallest prediction set size encountered and use the corresponding $w$ as the final $\\hat{w}$. Alternatively, discrete steps can be replaced with normal variable on $\\Delta^d$. This energy-based method has shown promising results in some experiments, and we plan to include it in future work.\n\n---\n\n### **6. About extension to regression problems**\n\nOur theoretical results can be extended to some families of score functions. Specifically, Lemma 1(b) relies on the VC dimension of the subgraph class: \n$$\\mathcal{A}:= \\\\{ \\\\{ y : \\langle w, s(x, y) \\rangle \\geq t \\\\} : w \\in \\mathbb{R}^d, t \\in \\mathbb{R} \\\\}.$$\nIf the scores $s(x, y)$ are always concave in $y$, then the weighted score $\\langle w, s(x, y) \\rangle$ is also concave in $y$. In this case, all elements of $\\mathcal{A}$ are intervals, and the VC dimension of $\\mathcal{A}$ is 2. This result suggests that our method can handle any number of concave score functions, including those used in CQR. We believe this result will be useful to some readers and will include it in the revision.", "claims_and_evidence": "The claims are mostly supported by the evidence presented in the paper. More precisely:\n\n- Averaging multiple conformity score functions leads to more efficient (i.e., smaller) prediction sets than using any single score alone.\n- The method retains finite-sample coverage guarantees for multi-class classification.\n- Empirical results on CIFAR-10 and CIFAR-100 show that the weighted method consistently achieves the target coverage while reducing prediction set size.\n- Theoretical proofs based on VC theory support the coverage and efficiency claims.\n- While convincing within the scope of the experiments, extending evaluations to more diverse or real-world datasets could further strengthen the evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense. Some remarks in this regard:\n\n- Weighted averaging of several nonconformity scores using a grid search over a discretized probability simplex.\n- Implementation of different data splitting strategies (VFCP, EFCP, DLCP, DLCP+) to  optimize weight selection. \n- The use of grid search over the probability simplex to determine optimal weights is  sensible, though computationally intensive.\n- Evaluating the methods based on coverage probability and average prediction set size aligns well with the goals of both validity and efficiency in uncertainty quantification (and is standard practice in the CP literature).", "theoretical_claims": "I tried to check all theoretical claims (and proofs thereof) in the paper. However, I did not check the corresponding Appendix line by line (mostly the \"critical\" parts to make sure that everything aligns). \n\nRemarks:\n\n- Finite-sample coverage guarantees are established via a theoretical analysis grounded in VC theory.\n- The paper provides oracle inequalities that quantify the efficiency gap between the proposed weighted method and the ideal (optimal) prediction set.\n- Consistency results demonstrate that the prediction set size approaches that of the optimal predictor as the sample size increases.", "experimental_designs_or_analyses": "The experimental design and analysis is fine.\n\n- Experiments conducted on benchmark datasets such as CIFAR-10 and CIFAR-100.\n- Performance is compared against single-score methods (e.g., THR, APS, RANK) and competitive baselines (e.g., RAPS, SAPS).\n- Multiple experimental runs (e.g., 100 different splits) and varying significance levels ($\\alpha$ values).\n- Key metrics include coverage probability and average prediction set size.\n- Although the experiments on CIFAR datasets are well-conducted, incorporating additional datasets from different domains could enhance the findings.", "relation_to_broader_scientific_literature": "The work builds on the established framework of conformal prediction and extends it by integrating ideas from model averaging. It compares and contrasts with earlier methods such as APS, RAPS, and SAPS, highlighting improvements in efficiency.", "essential_references_not_discussed": "The related works in the paper is fine. While not strictly missing from the discussion, the following paper might be interesting to consider:\n\nGasparin, Matteo, and Aaditya Ramdas. \"Merging uncertainty sets via majority vote.\" arXiv preprint arXiv:2401.09379 (2024).", "other_strengths_and_weaknesses": "Strengths:\n\n- The linear combination of base scores is straightforward and can unify multiple advanced scoring rules.\n- They rely on standard VC arguments to guarantee coverage and near-optimal size.\n- They examine a variety of splits (VFCP, EFCP, DLCP, DLCP+), clarifying how coverage vs. efficiency can shift.\n- In CIFAR tasks, the method consistently matches or outperforms single-score baselines, achieving the nominal coverage with smaller sets.\n\n\nWeaknesses:\n\n- The extension to regression remains open because the union-bounding trick used to handle classification does not trivially extend to real intervals (this limitation is acknowledged, though). \n- The method does a full grid search over $\\Delta^d$, which can be expensive if the number of base scores or the resolution is large.\n- The paper’s experiments focus on CIFAR-10 and CIFAR-100. Additional or larger datasets (e.g., ImageNet) or real-world tasks could better test the method’s scalability.", "other_comments_or_suggestions": "Some suggestions:\n\n- Implementations might consider more direct/continuous optimization of the average set size, e.g., using subgradients w.r.t $\\omega$.\n- Extending to the label-conditional coverage or group-conditional coverage would be an interesting next step.\n- The authors might systematically measure coverage vs. set size as a function of the calibration set size or the fraction used for weight selection.", "questions_for_authors": "Questions:\n\n-  Is it possible combine the data leakage idea with cross-validation or bootstrap to reduce coverage biases while still leveraging more data to pick $\\hat{\\omega}$?\n- Have you considered applying a subgraph approach to real intervals for regression tasks? Perhaps bounding the capacity by restricting the family of linear transformations.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "276WeIBNW0", "reviewer": "Reviewer_bRdZ", "summary": "This paper proposes to improve conformal prediction by optimally averaging multiple conformity score functions. This papers explores various data splitting methods and optimal weights for aggregating the score functions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback and constructive suggestions. \n\n---\n\n### **1. About why our method can improve performance**\n\nIn our setting, there exist $d$ score functions corresponding to weights $w = e_1, \\dots, e_d$, where $e_i$ is the $i$-th standard basis in $\\mathbb{R}^d$. The optimal weight $w^* \\in \\Delta^d$ yields the most efficient score function. Since $\\{e_1, \\dots, e_d\\} \\subset \\Delta^d$, the score function $\\langle w^*, s(x, y) \\rangle$ is at least as efficient as any single score function. The improvement of our method is quantified by the difference in prediction set size between $\\langle w^*, s(x, y) \\rangle$ and the best single-score function.\n\n---\n\n### **2. About novelty in the ensemble method literature**\n\nWe agree that ensemble methods are common in machine learning and statistics, and this motivates our proposal to apply model averaging to conformity scores.  \n\nUnlike standard ensemble methods, our approach aggregates scores in the final step during conformal prediction. This allows ensemble methods to be applied even when only one model is trained, by aggregating different score functions. To the best of our knowledge, this is the first work that enables ensemble techniques at the conformity score level.  \n\nTheoretically, our paper is the first to show that averaging conformity scores has a mild effect on validity while achieving near-optimal efficiency.\n\n---\n\n### **3. About inefficiency of grid search**\n\nWe acknowledge that grid search may not be efficient and view efficient weight optimization as future work. We have introduce some alternative method in the response to Reviewer j2fr and Reviewer gF9v. The methods include greedy search, golden section search, gradient descent after smoothing and stochastic optimization. \n\nWhile we acknowledge the importance of efficient optimization, the focus of this work is on analyzing the properties of $\\hat{w}$ and providing its statistical guarantees. \n\n---\n\n### **4. About why we consider different splitting methods**\n\nWe respectfully disagree with the assertion that introducing different splitting methods blurs the main focus of the paper. As described in Section 2.5, we use only 34 lines to introduce these methods, which we believe is a reasonable length. It is necessary to present the splitting methods here, as the statistical guarantees in the next section depend on them.  \n\nAt least one splitting method is essential for finding $\\hat{w}$. The main takeaway from introducing different splitting methods is to demonstrate that our conformity score averaging method is versatile and can be applied in various splitting scenarios.  \n\nAs pointed out by Reviewer gF9v, examining a variety of splits and clarifying how coverage versus efficiency tradeoffs shift is one of the strengths of our work. Therefore, we will retain the splitting methods, along with their theoretical analysis and experiments, in the paper.\n\n---\n\n### **5. About lack of datasets, such as variants of MNIST**\n\nWe have conducted additional experiments to address this concern. Please see the updated experimental results in our response to Reviewer 82f9.\n\n---\n\n### **6. About interpretation of theoretical claims**\n\nThank you for confirming the correctness of our theoretical results. Section 3 provides a detailed interpretation of our theorems, with the key takeaway being that $\\eta$ and $\\xi$ are small as long as the dataset sizes are large.  \n\nAs suggested by Reviewer 82f9, we will add additional intuition about the VC dimension and its connection to the DKW inequality to make the theoretical results more accessible.\n\n---\n\n### **7. About comparison with other alternative ensemble methods**\n\nThe most relevant ensemble method is in [1], which outputs the most efficient prediction set among a finite set of score functions. Our experiments already compare against single scores and models, including the most efficient ones. Please see Table 1 and Figures 3–5, where our method consistently outperforms these baselines.\n\n---\n\n### **8. About the suggested reference, \"Synergy conformal prediction\"**\n\nThank you for suggesting this reference. This paper also proposes an ensemble method for conformal prediction. However, our method differs in two key ways:  \n\n1. Our averaging occurs at the conformity score level, while their method aggregates models.  \n2. Our method optimizes the weights, whereas theirs does not.  \n\nWe will cite this paper in the revised manuscript and clarify the distinctions between the two methods. Please see the experiment result in our response to Reviewer 82f9. \n\n---\n\n### **References**\n\n[1] Yang, Y., and Kuchibhotla, A. K. (2024). Selection and aggregation of conformal prediction sets. *Journal of the American Statistical Association*, 1–13.", "claims_and_evidence": "The main claim of this paper is that by optimally weighting multiple score functions, the resulting threshold can achieve better performance.   This claim is validated both theoretically and numerically. However, in terms of theoretical results, this paper shows that optimizing weights has small impact on the validity. However, the validity of proposed threshold does not directly support the claim of improving performance. The numerical experiments are limited to two datasets.", "methods_and_evaluation_criteria": "* The proposed method makes sense but lacks principled guidance. \n  * Ensemble method is common in machine learning. Despite the claim, it is unclear where the novelty lies.\n  * The optimal weights are determined by grid search, which is potentially very inefficient. Instead of investigating further on the optimal weights, this paper go into different splitting methods, which blur the main focus and the main take away on different splitting is also not clear.\n* In terms of evaluation, the benchmark datasets are not sufficient and missing out on others like variants of MNIST.", "theoretical_claims": "Limited review. Appears to be correct in terms of statements. But it lacks interpretation in the main texts of the paper.", "experimental_designs_or_analyses": "The experiment metric makes sense, but it lacks a comprehensive comparison with other alternative ensemble methods.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "Gauraha, Niharika, and Ola Spjuth. \"Synergy conformal prediction.\" Conformal and Probabilistic Prediction and Applications. PMLR, 2021.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose an ensemble approach to conformal prediction, which is based on the idea of averaging multiple conformity score functions. The weights of the individual scores (in the weighted average) are optimised so as to maximise efficiency (i.e., minimise expected prediction set size) while keeping the validity guarantee. A theoretical analysis establishes finite-sample coverage guarantees. Moreover, empirical studies on benchmark data show that weighted averaging systematically outperforms single-score methods.

Overall, this is a convincing paper with a solid contribution to conformal prediction, although the idea as such is very simple. The reviewers had some comments and made some minor suggestions for improvement, which could be addressed by the authors in their rebuttal.
Decision: Accept (poster)