Valid Selection among Conformal Sets
Authors: Mahmoud Hegazy
Liviu Aolaritei
Michael I. Jordan
Aymeric Dieuleveut
Venue: NeurIPS 2025 poster ()
Keywords: Conformal Prediction
Selection after Inference
Uncertainty Quantification
Primary area: general_machine_learning
TL;DR: When multiple conformal predictors are available, we allow data-dependent selection without the loss of coverage.

Abstract: Conformal prediction offers a distribution-free framework for constructing prediction sets with coverage guarantees. In practice, multiple valid conformal prediction sets may be available, arising from different models or methodologies. However, selecting the most desirable set, such as the smallest, can invalidate the coverage guarantees. To address this challenge, we propose a stability-based approach that ensures coverage for the selected prediction set. We extend our results to the online conformal setting, propose several refinements in settings where additional structure is available, and demonstrate its effectiveness through experiments.

Valid Selection among Conformal Sets
Mahmoud Hegazy!.2 1.2
Liviu Aolaritei
Michael I. Jordan 2.3
Aymeric Dieuleveut
CMAP, École polytechnique, Institut Polytechnique de Paris
Inria, Ecole Normale Supérieure, PSL Research University
Department of Electrical Engineering and Computer Sciences, University of California, Berkeley
mahmoud. hegazygpolytechmiquu.. edu liviu. hiviuu.allartteiibbee.elll edu j jordan@cs berkeley edu aymeric. oit. edu edu
Abstract
Conformal prediction offers distribution-fre framework for constructing predic- tion sets with coverage guarantees. In practice, multiple valid conformal prediction sets may be available, arising from different models methodologies. However, selecting the most desirable set, such as the smallest, can invalidate the coverage guarantees. To address this challenge, ensures coverage for the selected prediction set. We extend our results to the online conformal setting, propose several refinements in settings where additional structure is available, and demonstrate its effectiveness through experiments.
propose stability-based approach that
Introduction
Conformal Prediction (CP) provides principled framework for uncertainty quantification by con- structing prediction sets with guaranteed marginal coverage [1-3]. For desired coverage level a, a conformal predictor outputs set that on average contains the true label with at least this probability. The appeal of conformal prediction lies in its minimal assumptions about the data distribution and the underlying predictive model. In practice, multiple conformal prediction algorithms may be available for given task, arising from variations underlying models or data splits. This multiplicity motivates the choice of the most desirable set, often the smallest. To illustrate, consider a prediction problem with feature and K sets (X)) each generated by a different conformal predictor. Suppose each C7 (X) satisfies the marginal coverage guarantee P (Y C (X)3 1 a, for all i [K]. where Y denotes the label associated to X. Although each (X) individually meets the coverage guarantee, selecting the smallest set generally invalidates the guarantee due to dependencies on the data introduced by the selection.
To address this issue, we first introduce novel perspective on the selection process based on algorithmic stability [4, 5J. The core idea is to employ a stable randomized selection mechanism, meaning its output is robust to small input perturbations. Such stability then allows us to transfer the marginal coverage of individual conformal predictors to the selected set. We introduce several stable selection rules, in particular MinSE, which we prove to be optimal. We further extend approach, by introducing an adaptive and a derandomized variant. These contributions given in Section 3, after recalling preliminaries on conformal prediction in Section 2.
Contributions and outline.
Furthermore, we extend in Section 4 work the online conformal setting [6-9], where data arrives sequentially, and predictions are made in real-time. We first demonstrate how our stability- based approach integrates seamlessly with existing online conformal methods, particularly with the approach of [10], to enable more adaptable selection among online predictors. Finally, in Section 5, we explore methods to optimize the implementation of our stable selection framework practice. In particular, we take a closer look at the split conformal setting, where the stability based bound can be overly conservative, and propose a recalibration mechanism that can achieve better empirical performance. Lastly, we validate approaches multiple experimental settings in Section 6.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Related Work. The motivation to select the smallest conformal prediction set has spurred series of recent works introducing principled selection methods that retain coverage guarantees while favoring smaller sets. For example, Liang et al. [11] proposes to merge multiple predictors by selecting the one with smallest average set size on the calibration data. Moreover, [12] proposes split-conformal methods that either inflate quantiles or use independent splits to preserve coverage after selection. Building on this approach, for the classification setting, Luo and Zhou [13] proposes a method for constructing the best possible conformal score, which may be expressed as weighted average of different scores. In contrast to such methods, our work enables pointwise selection depending on each X while maintaining validity, i.e. aim is not to select the predictor with best average performance, but rather to pointwise select a predictor producing a small set for each realization of X.
In the online setting, Gasparin and Ramdas [10] proposed a method for conformal online model aggregation, which adapts model weights over time based on past performance. Their method combines prediction sets using weighted majority vote with the weights learned in an online fashion. In another recent contribution, Hajihashemi and Shen [14] handles distribution shifts but focuses on temporal adaptation to distribution shifts, our approach addresses the distinct challenge of ensuring valid coverage when selecting across multiple predictors in static or online settings.
Our approach builds on algorithmic stability, a concept with roots in generalization properties of algorithms [15, 16] and differential privacy [17, 18]. Originally introduced to ensure privacy- preserving data analysis, differential privacy has been adapted for other tasks, such as adaptive data analysis [19], where it addresses the challenges of reusing data for multiple adaptive queries. Zmic and Jordan [4] applied baabee techniques to establish statistical validity after selection processes. Building on their work, we extend these ideas to the conformal prediction setting, developing stability-baser methods for both batch and online conformal frameworks. Within the conformal predictions literature, different notions of algorithmic stability have been leveraged. For example, Barber et al. [20] proved coverage properties of the Jackknife method under a notion of stability albeit very different from the one we use.
While the aforementioned works focus on selecting conformal predictors for single prediction task, other research has explored related but distinct problems in the context of conformal inference. Conformalized selection methods aim to identify subset of data points whose unobserved labels exceed a given threshold while controlling the False Discovery Rate (FDR) [21]. In a recent work, Bai and Jin [22] introduced a framework that allows data reuse for both training and selection while maintaining inite-sample FDR control. Although addresses different problem than ours, the focus on managing data reuse aligns conceptually with our goal of ensuring valid coverage despite
dependencies introduced by the selection.
2 Preliminaries in Conformal Prediction
We consider CP in two key scenarios: the batch setting, which assumes i.i.d. or exchangeable samples, and the online setting, where data arrives sequentially under minimal distributional assumptions.
Batch Setting. We consider dataset D = Y.)....).....6 (X., Y.)] E (X' x 3/)", where the points in D. along with any test sample (X, E x Vi are assumed to either i.i.d. or exchangeable. In the i.i.d. case, we denote by P the distribution from which they are drawn, and by Px and Py its marginals over X and Y, respectively. Without any further assumptions on the data generating process, conformal prediction allows to construct (random) prediction set (*() with the guarantee
P (Y E (X)) Z 1-a a.
(1)
Here, the probability is taken over (X, Y) as well as the randomness employed in the construction
of C". Arguably, the most common approach for batch conformal prediction is the split conformal procedure, which first partitions the dataset D into two disjoint subsets: D {X,,, )")" used to train the underlying predictor f, and Deal = ((X,, ) reserved for calibration. Then, for any nonconformity score function 8, which quantifies the error between the predictor f and the true output, it estimates the empirical [(1 am + 1) ////m-quantile, denoted qa. of the set {s; = s(X Y,, } Finally, for the test point X, the split conformal prediction set is defined as (a(X)= = (y E y 8 s(X, f) < Ja.. which satisfies (1) through a rank statistic argument.
Online Setting, In this setting, observations (X,, Yt) arrive sequentially for L 1, At each time step t, observe X, and aim to cover Y, using prediction set C())(
(Xe), which is constructed
2

---

based on a base model trained on all past data prediction, the true label Y, is revealed, and the process continues to the next time step. Unlike the classical conformal prediction setting, where data is assumed to be exchangeable, the online setting allows for the data to be non-stationary or even adversarial. As result, classical coverage guarantees no longer hold, and alternative notions of asymptotic coverage are required [6]. Specifically, we say that {C(t) Fre achieves asymptotic coverage if
(X-,,,,,((_,,, After making the
lim i + + 1 EY C (x,) a
(2)
The limit (2) ensures that, in the long run, the fraction of instances where the true label Y, falls within the prediction set (X,) meets or exceeds the desired coverage level, even under non-stationary or adversarial data. Stronger notions of asymptotic coverage can be considered, for example [23].
3 Smallest Confidence Set Selection
This section considers the batch setting without imposing additional restrictions, such as those in split conformal methods. Specifically, we focus on the problem of selecting the smallest among a collection of K conformal prediction sets {C, (X)) While such a selection is appealing, it inevitably invalidates the marginal coverage guarantee (1) since the selection process depends on the data. To address this issue, we develop strategy based on algorithmic stability, ensuring adjusted coverage guarantees even after the data-dependent selection process.
3.1 Valid Selection via Algorithmic Stability
We first recall the notion of algorithmic stability from [4] and extend their framework to tackle the data-dependent selection in conformal prediction. We start by introducing indistinguishability.
Definition 1 (Indisinguishabiiitt)) A random variable (r.v.) S is (1, T) indiistiinguuihable from r.v. So, denoted Rg, So, if for all measurable sets O, holds that P(S E O) S e"P(S E 0} + T.
l This definition extends to the conditional case, denoted by S ~,, Sg. if the inequality holds almost surely with respect to the conditioning variable ç, that is, IPS E O {} S (S, E O {} + т. In essence, the parameter 17 measures the degree of similarity between the distributions of S and So. with smaller values of allowing for greater similarity. Leveraging indistinguishability, we can define notion of stability for randomized algorithms. For more precision, define randomized algorithm as a deterministic mapping from E x & into S, where is typically the data space, and e describes the inner randomness of the algorithm. We also note that the randomness of an algorithm S may be either implicitly or explicitly parameterized by E. Nonetheless, we keep the dependence on & explicit in order to more precisely separate different sources of randomness in our statements.
Definition (Stability). randomized algorithm $
+ (1, T, u)-stable w.r.l. a measure P on E if there exists r.v. So. possibly dependent P, such that P P(S(E.e) 2 So 2 1-v.
In words, a randomized algorithm Š is stable there exists reference r.v. Sg such that, for almost any inputs Ç E E (up to a probability 1/) sampled from the distribution P, the distribution of Š(g,e) resembles that of S.- Essentially, this means that, for most inputs &, the algorithm's 's output (that randomly depends on E) behaves as if governed by a fixed distribution, independent of the specific input. We now examine how stability can be leveraged for selection among confidence sets.
Let Ç E and 5 be two random variables with arbitrary dependence. Suppose that there exists a set of (possibly random) confidence intervals {CTo|s E S). each correlated with & (for instance, may be vector of size (S] containing the size of all sets CI such that, for all E S, we have
P [C & CI2) < a,
for some a E (0, 1). Moreover, let $(5,s) define an arbitrary selection algorithm. For example, Š(C. might be biased towards selecting smaller size confidence sets. Without further assumptions, individual guarantees (3) do not translate to the selected interval CI i.. P(C & CI S a is not guaranteed to hold. Nonetheless, in the following theorem, we show that if S is (11 t, v) -ttsbbbility, then an adjustment of the confidence level in (3) is sufficient to account for the effects of selection.
3

---

Theorem 1 (Valid stable selection). Assume that P & C I a holds for all s E S. S E E - S an (n, T, -taable selection algorithm, then,
P(c & CT ((SE)) S ae" +
3.2 Application to Conformal Prediction
In the context of conformal prediction, S = (l,..., } and [CIg)ees are the conformal prediction sets at X, i.c. (X)) where each set C (X) assumed satisfy the coverage guarantee (I). The notion of (n), T, -)-staillity enables one to tackle the challenge of favoring the smallest among the K conformal prediction sets (X)) The r.v. Ç represents the output and we define
(C) (X)),,
(5)
where X(C;(()) represents "size" (for example, scaled Lebesgue's measure, counting measure, or more generically, any notion of set desirability) of set C (X). for all i [K]. Now note that introduced Definition 2 a function of the distribution of ç. In line with conformal prediction methods, which benefit from distribution-free guarantees, which v = 0, and will actually obtain results that hold almost surely on Ç. With slight abuse of notation, we will call these algorithms (17, --ttbblee (or simply yy-stable if, additionally, 0). We are ready to specialize Theorem to conformal prediction.
will focus selection algorithms for
Corollary (Smallest conformal set selection). Let S be an (n, -)stable selection algorithm (e.g.. for approximating arg [ A(C (X))). Then, we have
p F(y Y E C S(e.e) (x) 2 2 ae" 1---ee--0 TT
(6)
Note that the standard 1 coverage can be achieved by simply adjusting the confidence level of the K individual sets to (a - r)e" In what follows, inspired by the differential privacy literature [18, 24], we provide several examples of easily implementable stable selection algorithms.
Lemma 1 (Stability via Laplace noise). Assume X(C; E [0,1], for all i E [K]. (Lap @L/)))@k arg min (A (X)) + ei} is n-stable.
1 &. then the selection algorithm S such that S(C,E)
Lemma 2 (Stability via exponential mechanism). Assume A(C. (X)) E [0, 1], for all i E [K]. Then, the selection algorithu S with
P ((((E.)) = expl])epp((1 (C7)))
ONP(-RA(C (X))
is stable Note that we do not need to make the distribution of or the mapping Š explicit here.
We note that the assumption that X(C (X)) E [o, 1] for any i E [n] was used alleviate notations and may be relaxed up to appropriate scaling of the mechanisms. More importantly, while the two selection algorithms discussed above satisfy the stability requirement, they are adapted from literature of differential privacy, which is strictly stronger than our required notion of stability. Thus, this additional strength can lead overly conservative behavior when only stability is required.
To address this, we propose new Minimum Stable Expectation (MinSE) selection mechanism, designed to achieve stability as tightly as possible. MinSE relies on a prior b A K K- that encodes prior knowledge on which interval to select, before observing the different predictive intervals at X.
Lemma 3 (Minimum stable expectation). Let n7, t 2 0, and a fixed b E A K-1, and consider the following linear program
K Σ XP:A(C
p² (b, arg min
P
(MinSE)
s.t. DEAK-I SERE, p: S b + 8is
Σ 8: S
KE[K]
Then, the selection algorithm S({,E) with P(Š(E,E) = = (b, ) is (7, stabble

---

is the All three lemmas propose randomized selection algorithms which assign the highest probability to the smallest set. However, different from the first two, which assign nonzero probability to all the sets, MinSE assigns zero probability to the largest, whenever feasible.
To develop further intuition of MinSH, consider the case where b- (1/K) [/[K] [E]]] represents uniform prior. Then, we set e = K, T = O, MinSE reduces to arg min, {A (C (X)) ; - l,..., K). thus selecting deterministically the smallest set. This is reasonable, as then, for 1 - after selection, the original confidence sets should have - a a/K coverage, which corresponds to a Bonferroni correction [25]. Now consider less extreme example with e" 2, O, and b still uniform prior. By direct checking of the feasibility conditions, one can show that MinSE will never choose any of the [K/2] largest sets. In addition, it is possible to show that MinSE achieves notion of optimality among all stable selection mechanisms.
Proposition 1 (Optimality of MinSE), Let A E + [K] be an (n, -)ttbblee algorithm w.r.t measure P on E. Then there exists prior vector b E A K-1 such that it holds P-almost surely that
Σ P (b, 5)A (CF S E (E)
where & = =I(CT(X)) ;.. ,.... X (X)), the vector p" (b,g)
output of MinSE with parameters 17, 7 2 0 and prior b, and of (E) P.
In words, for any (77 -ttbbe algorithm, there exists prior vector such that the distribution of the output of MinSE achieves the smallest expected size.
3.3 MinSE examples, tightness, and extensions
Example 1 (Worst-case Oracles). Consider K oracle confidence interval methods such that for any X E X, an index j & [K] is chosen uniformly at random, and oracle j outputs (X) = 0. while all other oracles i * output (X) = y. This setup provides scenario to analyze the tightness of the stability guarantee. For simplicity, let X())) > O and A(0) = 0. For any datapoint (X, Y). each oracle individually has marginal miscoverage P (Y & (x)X) = 1/K. Let exp (1) /K + S applying MinSE with parameters (1), r) and uniform prior b. we examine the post-selection miscoverage P(Y & Cg). Miscoverage occurs only if the empty oracle is selected. By construction, one and only one set miscovers and has minimal size. MinSE assigns the maximum possible probability p7 = exp (n) / K + r to the zero-size set j. Thus, the miscoverage probability exp (17) /K + т. This result exactly matches the upper bound from Corollary I, demonstrating that the stability bound indeed tight in worst-case scenarios. Arguably, this is a pathological setting due to the dependence structure of the confidence sets. Nonetheless, we empirically show in the next example that the stability coverage bound is tight, even with independent confidence sets.
Example 2 (Coin flips). Let y [0, 1] and each of the K sets be constructed using an independent coin flip: for each i E [K] CT() is equal to y with probability 1 - a and to @ with probability a. In this example, we appropriately adjust the coverage of the original coin flips to achieve coverage of 1-a a after st-stable selection. Contrary to Example 1, here the different oracles are independent. As shown in Figure I, the rtssaalle selection using MinSE almost exactly achieves 1 a coverage across different stability levels 7, particularly as K grows. This suggests that while the stable selection mechanism inflates the probability of yielding the full set V, selecting the set with the minimum size effectively counterbalances this inflation. Therefore, even with oracle independence, without additional assumptions, the inflation of the original sets is necessary.
(Toy regression model). Let Y X + (0, 0.25), with X ~ Uniform([-1, 1)), and consider the following two predictors (X) = X and fx(X) = -X. This setup could arise, for example, when the data is split during training, e.g., due to privacy or design constraints, and is meant to be favouralle for our method. Indeed, on each half-space (X 2 0 or X < O), one confidence interval is much smaller than the other one. Numerical results, given to the right of Figure 1, show that despite the inflation in set size, stable selection leads to an improvement in the average set size, in comparison to relying on any individual predictor, while guaranteeing the same coverage. Numerically, the individual sets are about 30% wider on average. This improvement highlights the advantage of using stable selection in settings where different predictors have complementary strengths, i.c. are accurate on different subsets of X.
Example 3
Remark (Adaptive version-Adas AdaMinS................ A practical consideration when using MinSE is the selection of the stability parameters 17 and т. Given K conformal predictors, each achieving at
5

---

Coverage: y= 0.1
Coverage y= 2
Model: f(X) X
Model ((X) X
1 mn(2)
1000
3.5
350
0.20
1000
0.35
0.06
The Tata
Teta CLBS
Patt
0.1
0.1
CIISRR
3.85
50
100
50 100
Figure (Left-example 2) Coverage of the coin flips example after selection using MinSE with parameters E 20.1.2} and r 0. Before selection, the oracle returns the full set [0, 1] with probability 1- a exp (-n). (Right-example 3) Second and third figures reflecting adaptive conformal intervals, with miscoverage a 0.05, obtained using the models f(X) = X and f(X) -X. Fifth figure shows the stable selection applied the conformal sets in the first two figures, adjusted to have miscoverage = 0.025, using MinSE with 17 = In (2).
least 1 - a' coverage, and a desired post-selection coverage of 1 - a, any pair (1, satisfying a' S (a ()e is theoretically valid. This presents choice, as the trade-off between 7) and is not always immediately obvious. To address this, we introduce an adaptive version of MinSE, AdaMinSE, in Appendix A. This method automatically optimizes the (17, r) trade-off achieve the target 1 - a coverage after selection, thereby alleviating the need for manual parameter tuning.
Remark 2 (Derandomization). The randomized output of the stable selection mechanisms, such as MinSE, may be undesirable in certain applications where deterministic prediction sets are required. To address this limitation, it is possible to derandomize the selection process. Building upon techniques from [26], one can construct single, deterministic prediction set from the output probabilities of the stable selection rule, while still preserving (correspondingly adjusted) coverage guarantee. The precise results are deferred to Appendix A.2.
Remark 3 (Conditional Coverage). For clarity and ease of exposition, our main analysis has focused on marginal coverage guarantees as defined in (I), However, the stability-basee selection framework naturally extends to scenarios where the underlying conformal predictors satisfy stronger guarantees, such as conditional coverage. We provide further details on this extension in Appendix A.3.
4 Extension to Online Conformal Prediction
Next, we show how the framework based on algorithmic stability introduced above extends naturally to the online setting. Consider a collection of K online conformal prediction algorithms that, at each time step t E N, produce K prediction sets {c{(² (X,))
[= for the label Y,. As noted in Section 2,
each prediction set depends on the entire history {(x,, Y Similarly to Section 3.2, we define
& = (A(C(() (Xt)). X(C*) (X+))l, where A(C!!) (X,)) denotes the size of the set (i (X.), for each i E [K]. The following corollary specializes Theorem I to this scenario, showing that an adjusted coverage guarantee can be obtained a stable selection is applied at each time step t.
Corollary 2 (Smallest online conformal set selection). At each time E N, let S(E,, ex) be (n,T)- stable selection algorithm (e.g.. for approximating arg min X(C! (Xt))). Assume each 0 (.) satisfies the guarantee (2). Moreover, let the elements of the sequence (Et)teN be independent. Then,
lim in 0o & AEm IP C(()
(t) (
(x,) 2 1 ae -. -
(7)
In essence, given multiple online conformal prediction algorithms, applying a stable selection
mechanism provides a practical and systematic way combine them. It worth noting that while the
coverage guarantee in (7) matches the one established for the batch setting in (6), the guarantee in (7) is achieved only in the long run. This distinction aligns with the nature of online conformal prediction, as discussed in Section 2. Finally, we emphasize that, unlike (2), which ensures coverage for the
empirical average 1 ΣT, 1 (Yr E ((t) (X.)). the guarantee in (7) is in probability. However, the only source of randomness in this setting is the selection noise e, in the algorithm Š(E,, Ex), as all other quantities can be treated as adversarial or fixed through conditioning.

---

Algorithm Adaptive COMA (AdaCOMA)
Input: K conformal algorithms (, stability parameter (n, initial weights w(l)
(1/k,...,1/k))
For: 1- 1, ...
Compute wi(t) using COMA.
Compute p" ((w(()) Et) E K- using MinSE with b w(t) and parameters (11 T)
Output: Any of the following two options:
Option 1: Combined set c( (X,) equal to E ) Σ ps ( { E c, (X,) 2 2//
Option 2: Combined predictor leading to C(
(Xi), with ((er.e)) - i|6o
- pē (w((),
4.1 Adaptive Conformal Online Model Aggregation.
Conformal Online Model Aggregation (COMA) [10] extends online conformal prediction by address- ing the challenge of model aggregation. It combines prediction sets from multiple algorithms through a voting mechanism, where weights are dynamically adjusted over time based on past performance. Formally, at each time step 6, COMA assigns weights w(()) [w[") ..... w K (t) A which reflect the relative importance of each of the K underlying conformal predictors according to the following rule w exp (-ht- E X(c! (X,))). where nr is the adaptive learning rate from the AdaHedge algorithm [27], an adaptive version of the Hedge algorithm [28]. COMA then outputs the aggregated prediction set (*(e) = {v y Σ w
(0) 1 c. (Xx)) (X))) }), which can be interpreted the aggregated set obtained by selecting the i-th conformal set with probability w!").
Non-adaptiveness COMA. Crucially, at time i, the COMA framework assigns weights to the K conformal algorithms using only the observations up to time t 1, without access to X₃ or the prediction sets { (X)) R Furthermore, due its AdaHedge formulation, COMA optimizes the weights on average over time, without adapting to each individual X₃.
AdaCOMA. To achieve the best of both worlds, incorporate COMA into our stable selection algorithm. Specifically, at iteration I, we use COMA's weights w(() the prior for stable selection mechanism, which selects after observing both X, and the sets (C,(0) (Xx)) allowing for pointwise adaptability. The combined procedure, termed AdaCOMA, is detailed in Algorithm
COMA does not have assumption-frn freee coverage guarantees for fixed target level. However, letting
B: = EK
K w 1 ( o.0 (x.)]
(8)
Gasparin and Ramdas [10] show that the following holds
P pLr. c00) 23.
In the remainder of this section, we analyze the coverage guarantees of AdaCOMA in comparison to the bound in (9) for COMA. Detailed bounds on 3r under additional assumptions, are given in Gasparin and Ramdas [10].
Proposition 2 (Adaptive COMA). Consider Algorithm 1, and let 3, be defined as in (8). Then,
the set cit) comb(X))
(X,) satisfies {Yo E c(t) commb(X,) (x.) 2 1 - 1-2( + TT,
* the set (t) $
(Xi) satisfies P {ye E (t) Sieved (x)) 2- --1--,e B,e"
Proposition demonstrates that AdaCOMA inherits the flexibility of COMA while improving its adaptability to current observations through the stable selection mechanism.
7

---

Post-selection Calibration in Split Conformal Prediction
The coverage bounds derived from our stability-based approach (e.g., Corollary 1) are distribution- free and hold under minimal assumptions, allowing them to extend to even the adversarial online case. While potentially tight in worst-case scenarios, these bounds can be conservative when additional structure is available, partieularly in the batch setting. By leveraging the inherent rank structure of the split conformal method, this section develops a recalibration procedure specifically for split conformal prediction, aiming to achieve tight finite-sample coverage guarantees after selection.
We operate within the standard split conformal setup introduced in Section 2, with calibration dataset Deal {(X,, YJ) and test point (X, Y). We assume the sequence of m + 1 data points {(X1, Yi)...., ..... (Xm: Ym). (X, Y)} to be exchangeable. We consider base predictors .......fx. and corresponding non-conformity score functions $....... 8K. For each base predictor & E [K] and datapoint E [m]] denote the non-conformity scores as Sk.i == Sy (X,, fx). We use sk,(r) to denote the r-th order statistic of $k.1..... Sk,m- Finally, for any & and i, denote the rank of score Sk,i as R = Σy= 1 [sk._ I $..i). Using these definitions, we can parameterize the conformal prediction sets using ranks, i.e., for any rank index R E [m], we define
Cx, = y E V (x y. ft) S $n.(R))
which recovers the classical set C recalled in Section 2 for Ra = [(1 - am + 1)1, satisfying, for any ls [K], P (Y Cx(X, Ra)} 2 }}
Calibration After Selection using Effective Ranks. We now introduce an arbitrary (stochastic) selection rule S. Our goal is to determine an effective rank Rs such that a similar property holds, but for the selected interval, i... P[Y E Cs(x,e)(X Ra)) 2 1 a. To that end, we use a recalibration process after selection, that uses the effective ranks meta-scores. For each point E [m], we apply the selection rule using its feature vector X, and independent randomness E; Pe- We now define the effective rank (or the meta-score) for the i-th point as:
R, R
that is, the rank of the i-th point's score calculated using the selected predictor $(X,, ez). Subse- quently, we define the sequence of effective ranks R = (Rı,..., Rm). Using uniform random tiebreaks between equal ranks in R. for E [ml) we use Rio to denote the t-th order statistics of R.
Theorem 2. Assume that S is independent of Deal. Let Ta [(1 a) 1)] I Then,
P(y CSCxx.e) ((T)
a.
Theorem provides a method to maintain coverage guarantees after selecting among multiple split conformal predictors. The use of effective ranks acts as a meta-score, allowing for the calibration of predictors even if they utilize different non-conformity score functions 8& (.), offering flexibility in model-specific score design. The theorem states that by selecting the appropriate order statis- tic Pir..) of these effective ranks, derived using independent selection rule Š, the conformal set Cssx.e)(X (X, R(T.)) for the chosen predictor 5(X, s) achieves the desired coverage.
Constructing an Independent S. Theorem 2 mandates the selection rule Š to be independent of the calibration data Deat. This independence plays a critical role in the proof as it ensures that the effective ranks Ri,..., Ru, are exchangeable with the unobserved effective rank of the test point. Consequently, if S aims to select the smallest set, it cannot use set sizes derived from D based quantiles due to the induced dependency. To ensure independence, we employ an auxiliary dataset Dmux. disjoint from and independent of Deal- For each predictor & E [K] and test point X, proxy quantiles Pai computed from Daux (at a preliminary miscoverage rate ă), define proxy conformal sets C. (X) and their corresponding sizes ((CIE(()) The vector of these proxy sizes, ..... A(CRI(()) ]. is then independent of Deat. A selection rule Š(X, e) based solely on X and this gaux (X) (e.g., employing stable mechanisms from Section 3.2, such as MinSE) thus satisfies the independence condition. This allows Theorem 2 to be applied directly for
(X)

---

YK Base
YK- MK
ModSel
Recal
AdaMiinS of 0.50
MasSE y= 1.0
Resi Data: Choseaus
Real Date: Interval Lengths
Stuthetic Data: Coving
Sunladi Duti: Interval Length
1.30
1,00
0.35
0.08
0.30-
0,90
0.00
0.55
0.85
0.55
30 Sharing
Calitonia Housing
Shoring
Aballue Caliornia Houainng
01:00
Number of Conformaal Prediccons
Figure 2: Marginal coverage and average lengths, for real datasets (Bike Sharing, Abalone, California Housing) and synthetic data; Error bars represent twice the standard error of mean estimation using multiple seeds for real datasets and 2 s.d. for the synthetic data.
1 1-a coverage, avoiding the inflation factors inherent in stability- -bsed bounds that depend Deal- The practical effectiveness of such selection hinges on how accurately the proxy information from Daux reflects the true characteristics based on Deal.
6 Experiments
To illustrate our approach in practice, we present here two simple experimental setups, one on synthetic and one on real data. We defer online experiments, additional batch experiments, and further experimental details to Appendix C. We compare our approach to Yang and Kuchibhotla
[12] and Liang al. [11]. denoted as YK and ModSel, respectively. As Yang and Kuchibhotla [12] proposed multiple algorithms, adopting the following naming convention from Liang et al. [11], we compare against Yd-Adjust and YK-Base. YK-Adjust adjusts the underlying conformal predictors to ensure valid coverage after selecting the best-on-and best-on-average conformal predictor on the calibration dataset. YK-Base simply selects the rest-on-average conformal predictor and does not have coverage guarantees. We report the performance of MinSE with parameters (1) = T 0) and AdaMinSE with a' = 0.05. In addition, we report the performance of Recal, which is based on AdaMinSE with a' 0.02, followed by the recalibration procedure of Section 5. For Recal, the calibration dataset is further split into two blocks to construct Deal and Daux, so that the selection satisfies the independence requirement of Theorem 2. For the experiments presented here, we target a miscoverage level of a = 0.1. Except for YK-Base, all methods guarantee miscoverage a = 0.1 after selection.
Throughout the experiments, we aim to design a scenario where the performance of individual predictors varies across the input space. As such, using clustering, we split the feature space into disjoint sets and train each predictor exclusively on randomly selected subset. We provide additional experiments without such data splitting in Appendix C. Moreover, the code to reproduce the experiments is available in the supplementary material'.
Synthetic Regression. We generate n data points, {(X,, Y:) with X, NN (before feature space splitting), and the response variable defined as Y, sin( X,)) + O.LA(O,, 1). where 33 is the vector {1/d]ie[d]- The feature dimension is set to d = 10, and the training data is split into two blocks. In the first block, we train K distinct regression models, fi...., fK, using the Kernel Ridge Regression model from seikit-learn [29]. For each model, we randomly sample the kernel function (either linear or radial basis function (RBF)) and the regularization parameter (uniformly chosen between 0.1 and 1). For each model i, we use the second block of training data to train random forest model ys that predicts the absolute residuals (Xx(X) YI. enabling us to use the nonconformity score, defined as (X, = (X) (X).. We use 400 datapoints for the calibration dataset.
Real Datasets. In this experiment, we aim model a more typical data analysis scenario. We conduct experiments on three standard regression datasets: Abalone, California Housing, and Bike Sharing [30, 31]. For each dataset, leveraging scikit-optimize for hyperparameter tuning [32], we used the following scikit-lean models: AdaBoostRegressor, Decision TreisionnrreRRegressoor GradientBoost-
'Code also available at Valid Salid-Selection-amonne Celeetioo-moonneCCffoommllle

---

ingRegressor, ElasticNet, RandomForessi Foresreegrssssoooror..o and Linear We used 80% of the data for trainin, 10% for calibration, and 10% for testing. Similar the synthetic experiments, we used the same adaptive score function, with FandoomoonsstRegressor trained to predict the residuals. Then, up to adjusting coverage to ensure post-selection coverage of a = 0.1, we used the same conformal predictors for all selection methods. We normalized the labels across datasets to keep the size of conformal sets comparable.
Both synthetic and real data results are reported in Figure 2. For both experiments, Yd-Adjust, MinSE, and AdaMinSE overcover, while Recal and ModSel achieve similar coverage to the target marginal coverage of 0.9. YK-Base undercovers in the synthetic setting. Differences emerge in the resulting average interval lengths. Our proposed Recal consistently performs the best on both the synthetic and real data settings. On real datasets, MinSE and AdaMinSE are competitive with baselines on Abalone and California Housing but produce larger sets on Bike Sharing. Meanwhile, for the synthetic setting, Ad and MinSE beat the benchmarks.
7 Conclusion
In this paper, we introduced a stability-based framework for selecting among multiple conformal predictors while preserving coverage. By casting selection as an (17 -ttbbe randomized mechanism, we established distribution
guarantees that transfer the validity of individual predictors to the post-selection set, enabling pointwise with practical mechanisms and proposed the MinSE mechanism, which is optimal among stable selectors, along with adaptive and derandomized variants. We further extended the framework the online setting; combined with online aggregation, this yields AdaCOMA, which uses COMA weights prior for stable per-time-step selection based the current features and sets, thereby adapting over time and across each input. Finally, for split conformal prediction, we introduced post-selection recalibration via effective ranks that mitigates the conservativeness of worst-case stability bounds. Empirically, our methods meet the target coverage and often reduce set sizes relative to existing selection approaches across synthetic and real datasets in heterogeneous settings. Nonetheless, the stability- -ased guarantees are worst-case and can be conservative in benign regimes; our recalibration reduces this conservativeness but requires an auxiliary dataset, which independent from calibration, and its effectiveness depends on the quality of proxy quantiles. Moreover, randomized selection and the choice of stability budget introduce utility/validity trade-offs, and extending the framework by deriving tighter instant-dependent coverage bounds under additional assumptions remains an open direction.
selection. We instantiated this principle
Acknowledgement
The work of Aymeric Diculeveut and Mahmoud Hegazy is supported by French State aid managed by the Agence Nationalc de la Recherche (ANR) under France 2030 program with the reference ANR-23-PELA-000 (REDEEM project), and ANR-23-IACL-000S in particular HilParis FLAG chair. Liviu Aolaritei acknowledges support from the Swiss National Science Foundation through the Postdoc. Mobility Fellowship (grant agreement P500PT 222215). Additionally, this project was funded by the European Union expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. This publication is part of the Chair "Markets and Learning", supported by Air Liquide, BNP PARIBAS ASSET MANAGEMENT Europe, EDE, Orange and SNCF, sponsors of the Inria Foundation.
2---0022SYG-CEEAANCO 101071601). Views and opinions
References
[1] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world, volume 29. Springer, 2005.
[2] Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008.
10

---

[3] Anastasios N Angelopoulos, Rina Foygel Barber, and Stephen Bates. Theoretical foundations of conformal prediction. arXiv preprint arXiv:2411.11824, 2024.
[4] Tijana Zmnc and Michael Jordan. Post-selection inference via algorithmic stability. The Annals of Statistics, 51(4): (119)::666611660 1691, 2023.
[5] Raef Bassily and Yoav Freund. Typical stability. arXiv preprint arXiv:1604.03336, 2016.
[6] Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34:1660-1672, 2021.
[7] Isaac Gibbs and Emmanuel J Candès. Conformal inference for online prediction with arbitrary distribution shifts. Journal of Machine Learning Research, 25(162):1-36, 2024.
[8] Margaux Zaffran, Olivier Féron, Yannig Goude, Julie Josse, and Aymeric Dieuleveut. Adaptive conformal predictions for time series. In International Conference on Machine Learning, pages 25834-25866. PMLR, 2022.
[9] Shai Feldman, Liran Ringel, Stephen Bates, and Yaniv Romano. Achieving risk control in online learning settings. arXiv preprint arXiv:2205.0000 09095, 2022.
[10] Matteo Gasparin and Aaditya Ramdas. Conformal online model aggregation. arXiv preprint arXiv:2403. 15527, 2024.
[11] Ruiting Liang, Wanrong Zhu, and Rina Foygel Barber. Conformal prediction after efficiency- oriented model selection. arXiv preprint 1xXiv:2408.07066 2024.
[12] Yachong Yang and Arun Kumar Kuchibhotl.. Selection and aggregation of conformal prediction sets. Journal of the American Statistical Association, pages 1-13, 2024.
[13] Rui Luo and Zhixin Zhou. Conformity score averaging for classification. In Forty-second International Conference on Machine Learning, 2025. URL https://openreviert heete/eeeeeeee//eeeeeeee net/ forum?:d-PvEd77MWSS.
[14] Erfan Hajihashemi and Yanning Shen. Multi-model ensemble conformal prediction dynamic environments. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[15] Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research, 2:499-526, 2002.
[16] Andre Elisseeff, Theodoros Evgeniou, Massimiliano Pontil, and Leslie Pack Kaelbing, Stability of randomized learning algorithms. Journal of Machine Learning Research, 6(1), 2005.
[17] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Founda- tions and Trends@ in Theoretical Computer Science, 9(3-4):211-407) 2014.
[18] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity private data analysis. In Theory of Cryptogruphy: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pages 265-284. Springer, 2006.
[19] Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Leon Roth. Preserving statistical validity adaptive data analysis. In Proceedings of the forty-seventh anmual ACM symposium on Theory of computing, pages 117-126, 2015.
[20] Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Predictive inference with the jackknife. The Annals of Statistics, 49(1):4865007 2021.
[21] Ying Jin and Emmanuel Candès. Selection by prediction with conformal p-values. Journal of Machine Learning Research, 24(244):1-41, 2023.
[22] Tian Bai and Ying Jin. Optimized conformal selection: Powerful selective inference after conformity score optimization. arXiv preprint arXiv:2411.17983 2024.
11

---

[23] Aadyot Bhatnagar, Huan Wang, Caiming Xiong, and Yu Bai. Improved online conformal prediction via strongly adaptive online learnin.. In International Conference on Machine Learning, pages 2337-2363. PMLR, 2023.
[24] Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS pages 94-103. IEEE, 2007.
[25] Eric W Weisstein. Bonferroni correction. https://matwworld. wolfram, com/, 2004.
[26] Matteo Gasparin and Aaditya Ramdas. Merging uncertainty sets via majority vote. arXiv
preprint arXiv:2407. 00309, 2024.
[27] Steven De Rooij, Tim Van Erven, Peter Grünwald. and Wouter M Koolen. Follow the leader if you can, hedge if you must. The Journal of Machine Learning Research, 15(1):1281-1316, 2014.
[28] Yoav Freund and Robert Schapire. A decision-theorrtic generalization of on-line learning and application to boosting. Journal of computer and system sciences, 55(1):119-13)) 1997.
[29] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
[30] R Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33(3):291 1997.
[31] Markelle Kelly, Rachel Longjohn, and Kolby Nottingham. The UCI machine learning repository. attps://archive.. ics. uci edu, 2023. Accessed: 10-3-
[32] Tim Head, MechCoder, Gilles Louppe, Iaroslav Shcherbatyi, fcharras, Zé Vinícius, emmalone, Christopher Schröder, ne1215, Nuno Campos, Todd Young, Stefano Cereda, Thomas Fan, rene rex, Kejia (KJ) Shi, Justus Schwabedal, carlosdanielcsantos, Hvass-Labs, Mikhail Pak, SoManyUsernnees Taken, Fred Callaway, Loic Estève, Lilian Besson, Mehdi Cherti, Karlson Pfannschmidt, Fabian Linzberger, Christophe Cauet, Anna Gut, Andreas Mueller, and Alexander Fabisch. scikitt-ppimize//kkiii-ppppmmp 5281/zenodo. 1207017.
v0.5.2, March 2018. URL https: //doi. org/10.
[33] Isaac Gibbs, John Cherian, and Emmanuel J Candès. Conformal prediction with conditional guarantees. Journal of the Royal Statistical Society Series B: Statistical Methodology, page qkaf008. 2025.
[34] Raef Bassily, Kobbi Nissim, Adam Smith, Thomas Steinke, Uri Stemmer, and Jonathan Ullman. Algorithmic stability for adaptive data analysis. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pages 1046-1059, 2016.
[35] Paul S Bradley, Kristin P Bennett, and Ayhan Demiriz. Constrained k-means clustering. Microsoft Research, Redmond, 20(0):0, 2000.
[36] Josh Levy-Kramer. k-mmanss-onstraiiee, April 2018. URL http:pss//gtthu...cc//aallalllll k-means- constrained.
[37] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255, Ieee, 2009.
[38] Ross Wightman. PyTorch Image Models. URL httpp:///gtttub..omm//ggggggggeg pytorch-image-mon
[39] Michael Harries, New South Wales, et al. Splice-2 comparative evaluation: Electricity pricing. 1999.
[40] Ruey S Tsay. Analysis of financial time series. John wiley & sons, 2005.
12

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction accurately reflect the scope and contributions of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: While the paper does not summarize all limitations in single section, discusses the limitations of the proposed methods at various points. For instance, the potential conservativeness of the stability-based coverage guarantees (e.g.. Corollary 1) is acknowledged at the beginning of Section 5, which motivates the development of a specific recalibration approach for split conformal prediction setting. The inherently randomized nature of the primary stable selection mechanisms (like MinSE) is noted as potentially undesirable in some applications (remark after Example 3), leading to the proposal of a derandomization technique in Appendix A.2, though this comes with degraded coverage guarantee. Furthermore, the recalibration method presented in Section itself has the limitation of requiring the selection rule to be independent of the main calibration data, necessitating an auxiliary dataset, and its effectiveness hinges on the quality of this proxy data, discussed at the end of that section.
achieve tighter bounds in that
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
13

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Detailed assumptions and complete proofs for all propositions, theorems, and corollaries provided in Appendix B. All results properly numbered and referenced, with assumptions clearly stated alongside the formal statements.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: All information necessary to reproduce the main experimental re- sults- includinn data generation, evaluation procedures, and implementation details is provided in Section 6 and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., s., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
14

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e-g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Datasets used are either synthetic open-source under permissive licenses. The code to reproduce all experiments and guide on how to use it can be found in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/ public/guided htt////ooeeemiiiinnnnnoooe
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//nps..cc htt.cc//ppbb.. ccc//uulcc//aeeeccceeeceecceeeee http//gideeeeeeeeee Maaees//ooeeummiii CcodeSubmission/e wwtssbbmisssioni ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the details of the experimental setup in Section 6 and Ap- pendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental
material.
7. Experiment statistical significance
15

---

Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We provide error bars for all our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The computational resources, including hardware type (CPU/GPU) and run- time, are described appendix
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https :/tps:////eerrr //neurnrs.....// httpp..cc/pppee cc/public/ hetpt/tthhhiiiitiee ceGuidelines"
Answer: [Yes]
Justification: This research adheres to the NeurlPS Code of Ethics. It does not involve human subjects. sensitive data, or real-world deployments with potential societal impact. All results are reproducible, and the code will be released upon acceptance.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
16

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
This work is foundational and does involve specific applications or deploy- ments. While robust prediction can support decision-making in domains such as healthcare or autonomous systems, the paper does not target any particular use case and therefore does not entail direct societal impacts.
Justification:
Guidelines: This paper is not directly tied to particular application.
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e-g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deg.oyeemm of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This work does not involve the release of pretrained models, large-scale datasets, tools that pose significant risk of misuse. The research is theoretical and algorithmic in nature, and all released code for reproducibility of the experiments.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
17

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All external assets used in this work-including code and datasets- are properly cited in the paper, and their licenses have been respected in accordance with the stated terms of use (see Section 6). No proprietary or restricted-acces, data was used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce new code for our method and experiments, which is documented, included in the supplementary material, and will be made publicly available upon acceptance. The code includes clear instructions for reproducing all results and is structured to facilitate ease of use.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
18

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work does not involve human subjects crowdsourcing and therefore does not require IRB approval.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: This research does not involve the use of large language models (LLMs) in any part of the core methodology. Any LLM usage was limited to minor writing assistance and had no impact on the scientific content or originality of the work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: httppp:///nuuis.... http://rppp.cccmmeeeeeeeecccccce cc//onfferecces/22555 for what should or should not be described.
19

---

Contents
1
Introduction
Preliminaries in Conformal Prediction
3 Smallest Confidence Set Selection
3.1 Valid Selection via Algorithmic Stability
3.2 Application to Conformal Prediction
3.3 MinSE examples, tightness, and extensions
Extension to Online Conformal Prediction
4.1
Adaptive Conformal Online Model Aggregation.
5
Post-selection Calibration in Split Conformal Prediction
Experiments
7
Conclusion
10
Deferred Content Section 3
21
A.1
Adaptive Minimum Stable Expectation
21
A.2
Derandomizing the Prediction Set
22
A.3 Conditional Coverage
22
B Proofs
23
B.1 Proofs of Section
23
B.2 Proofs of Section
25
B.3 Proofs of Section
25
Additional Experiments
27
C.1 Batch Experiments
27
C.1.1
Homogeneous vs Heterogeneous Data Preprocessing
27
C.1.2
Varying Data Generation in Synthetic Experiments
"ipp.meeei......................
27
C.1.3
Comparison against Average Model Baseline
29
C.1.4
Effect of the Number of Calibration Points
29
C.1.5
Additional Classification Experiments
29
C.2 Online Experiments:
29
20

---

Outline of the Appendix
This appendix provides supplementary material to the main paper. The first part details deferred content from Section 3 of the main paper: Appendix A. introduces and proves the Adaptive Minimum Stable Expectation (AdaMinSH) mechanism; Appendix A.2 presents and proves derandomization technique for prediction sets; and Appendix A.3 discusses and proves an extension to conditional coverage guarantees. Appendix B provides the proofs for the theoretical results presented in Section (on stable selection), Section 4 (on online conformal prediction), and Section 5 (concerning post- selection calibration in split conformal prediction). Finally, Appendix C dedicated to additional experimental results. These results include further batch experiments under varied settings, online experiments evaluating AdaCOMA, and detailed setup information for these experiments.
A Deferred Content Section 3
A.1 Adaptive Minimum Stable Expectation
One difficulty in using MinSE is tuning the stability parameters 1) and r. For instance, assume a user has access K conformal predictors, each with coverage least - a', and wishes to apply MinSE to select among them such that the coverage after selection is at least 1 - a. Then, they may choose any values of r and 7 satisfying a' S (ax - T)e-. In particular, the utility tradeoff between and not immediately clear.
To address this, propose AdaMinSE, an adaptive version of MinSE, which also optimizes over the choice o and ry. Similarly to MinSE, AdaMinSE also makes use a prior b E A A (which can be the past as in Algorithm 1). However, instead of requiring the parameters (17, r) as input, it simply takes the current level of miscoverage a' and the desired miscoverage level a after selection. The following proposition introduces AdaMinSE, together with its coverage guarantee.
chosen depending
Proposition (Adaptive Minimum Stable Expectation). Let a', a (0, 1) with a' S a, and let be A K- be fixed. Consider the following linear program
K
d' (b, arg min ye (C+ (X))
Σ
d
1=1
s.t.
de A K-1
SE SERE RK
T, 2 0,
(AdaMinSE)
d S b, + Si.
Σ s S + 5 K
L E or (x)) 2 1- a., for all i E [K] Moreover, consider the selection algorithm $(5,E)
with ((((().)) = -[g] = d' (b, Then, P { C (X)] 2 -
Proof. The result can be recovered follows
y CC
(x) E
Σ d' (b.))) o. (x)
K
ed y $ C (x)) t ee + aa
where the first and last inequalities follow from the two constraint d, e + 8; and ed + S a in AdaMinSE, respectively.
Proposition 3 ensures that AdaMinSE achieves the desired coverage. By optimizing over (1, r), removes the need for manual tuning, making it practical and reliable approach for selection.
21

---

A.2 Derandomizing the Prediction Set
One potential limitation the stable selection algorithms presented in Section 3.2 is that they produce random confidence set, which may be undesirable in certain applications. In such cases, the stable selection process can be derandomized using techniques from [26]. This formalized in the following proposition.
Proposition (Derandomized smallest conformal set). Let S(g,e) be an ( selection algorithm, and define pı(E) == TP {((,.) = =ll). Then, consider the derandomized confidence set
K Car (x E V: Ep,(S) tu E CF 2 2
1
1=1
If O (Di...., 0 ())x satisfy (1), holds that IP (Y Cx, 2 2aae" + +).
Proof. The proof builds upon the reasoning in [26]. Since S e) is (n, T) -stabilii, we know that there exists a fixed point b E A K-1 such that pe(E) S exp (n) b, + s and Di 8 S T. It follows that
K P.(E) & (2(0)] S (
E
Moreover, using the Markov inequality, we have that
1
(( & (x,)) =P Σ p:(()1 (Y & Cx 2 S 2E
Σ p:(() (Y & C2(X))
from which the result follows immediately.
We highlight that while the set (-r(E is still random with respect to the randomness of {C, (.)) the derandomization here refers to the fact that the stable selection process does not introduce additional randomness due to e.
A.3 Conditional Coverage
A stronger guarantee than marginal coverage (1) is conditional coverage. Let G X - function that maps an input X to group attribute G(X) E G. A conformal predictor C(X) is said to satisfy (1 ar) conditional coverage with respect to G if,
G be
P (Y C(X)) 2 21 -
(10)
This ensures that the coverage guarantee holds not just on average over all X, but also when restricted to specific subpopulations defined by G. An example is the case where G finite, which then corresponds to Crtoup-Conditionss validity [33]. Such guarantees are crucial for added reliability in many applications.
Our stability-based selection framework can be extended to preserve conditional coverage. Suppose we have K conformal predictors (C)) each satisfying (1 -aaa a) conditional coverage with respect to G. That is, for each E [K].
P[Y E (X)G((X) 21 a.
(11)
We can now state the conditional coverage guarantee for the selected set. Proposition 5 (Conditionally valid stable selection). Assume that each conformal predictor (x) satisfies (1 a) conditional coverage with respect to G. for all E [K]. $ E E - [K] is an (n, stable selection algorithm (with & = [X(C((X)),
X(C2 (X))), then almost surely
P P
E aa Co. S(E,e) (X) (X) [(X) 2 -ae" + T).
(12)
22

---

Proof. We want to bound the conditional miscoverage probability
miscover (G() : P P $ C (X) G(X)).
By the law of total expectation, conditioning on X we have
mmiscover|G(X) Ex(G(X) [ {r & ((a()) (X) (x) xXx(
Since G(X) is determined by X, the inner probability is P {r & C ()) (X) x} randomness in S(E(X), s) is independent of Y given X, we have
Since the
P Y $ C ((e()),)))
|x] ] - Σ PIY & ( 1XP ((E())),s) = =
Using (n. r--sbbbllity (conditional on {(X)) property of S, we have
K
K
PY & (T()) IIP. (S(E(X), = =s xx} S e
Σ
P(Y $ x
s e Σ
P (2()) ()P 5) +
for some random So variable with support [K]. It follows that
K
P mmiscover|G(X)
e
Σ
P (So i} Exjo()) [P & CP()) +
S ta
This result shows that if the original conformal predictors provide conditional coverage, the stable
selection mechanism allows for selecting among them while retaining a (correspondingly adjusted) conditional coverage guarantee. The required level for the initial predictors would be 1 (a T)e to achieve 1 - a conditional coverage post-selection.
B Proofs
B.1 Prools of Section 3
The proof of Theorem requires the following lemma from Zrnic and Jordan [4], which is inspired by the approach in [34].
Lemma 4. [4, Lemma Let S & - be an (n,T,, v)-stablee selection algorithm and So be the corresponding random variable. Then, (E, (()) B (E, So).
Proof of Theorem I. In this proof, similar to analogous results Zrnic and Jordan [4], lot of the heavy lifting is done by Lemma 4. Nonetheless, the selection dependence on the data on the confidence sets themselves introduces some subtleties, making direct application the steps Zmic and Jordan [4] non-straightforwarrd Thus, we take different starting step by defining a shadow algorithm.
Let us define S": : x - such that S" e) Š((, e), for all (E. e x x Z. Then, if S (1) v, u) stable with respect to P on E, we also have that S' (1,, t, stable with respect to the product distribution P& Q, where Q is the distribution of Ç. Combining this with Lemma 4, we obtain
23

---

(,,E, S(E,E)) En,++v (s So). Now, defining the event O₃ = ((C.
S(E.s) ). we have that
P F(s
CI (6 ") S(5.c) S S PPP ((C,5, E 03) + +
= =
{ $ CIS.
} + - v t0
where the first inequality follows from the definition of indistiggushaabiliy, and the second inequality follows from the assumption that P {< & C & a holds for all s S.
Proof of Corollary I. The result follows immediately from Theorem with Y ( and 0.
Proof of Lemma I. Let So arg |i] Ci. with & lid. Lap (1/n). Moreover, note that ((, = if and only e S minzæi (ez + A (X)( - (CF(())) & min Ej + 1, where we used the fact that X(CT(X) E [o, 1]. Finally, for all E [K]
E S mine, + 1/0(E. (esb+t) T ((( = oGs (esbttt) S
s#
S e E, S min E, l, (eylimi)
P (So (E, {
where the second inequality uses the fact that the densities ratio ps: 1/Pss is upper bounded by e". The result now follows by taking expectation on both sides.
Proof of Lemma 2. Let Sy be a uniform r.v. with P(S. = i} = 1/k. Then,
S(g,E) =
kexp (-niA (X)))
P 1(n =i}
EielK] exp (Sp(-n) (C;(())
k exp (1)
exp 27)
k eexp (-1))
where we used the fact that (XC)(X)) [0.
Proof of Lemma 3. The optimal solution pb,bb {) satisfies pf E) S e + st. with KK S T. for all E k. Therefore, letting So be a.v. with PSS,- i} b,. Then, for all S S [K], we have
+
P($(C.c) < eTIP (So E S) +
which concludes the proof.
Proof of Proposition 1. By the assumption that A is (1, -tstbble w.r.t. P, Definition 2 guarantees the existence of a r.v. So such that A(E, R So holds P-almost surely. This implies that for any G' C [K]. (A(E,e) Glş) < e"P (So E G} + holds P-almost surely.
We the prior vector 6 e A K-1 as the distribution of So by choosing b, = P i). Let
(E) (E) P. (A(E,e) il(). Define s;(E) max (0. (E) e²b.). Clearly, si(E) 2 and
(E) S ebb, + sE( for all i. S*) {i p/((E) > ebb,). Then, P-almost surely,
Σ sx(E) = Σ (14 - eb) P. {A(E, e) E st (S)1E) e'P (So E ({))
(ES+(E)
< (e" (So St (E)} + e'I (So S+{))) =
Since (E) is probability distribution, y AA) -- Thus, r satisfies the constraints of the MinSE linear program with prior for P-almost surely all &.
24

---

The MinSE algorithm finds the solution pr (b) that minimizes the objective function E Ei=P, (x)(C) over the set of all feasible distributions satisfying these constraints. Since (E) is feasible solution P-almost surely, its objective value must be greater than equal to the minimum objective value achieved by the optimal solution p" (). Therefore, P-almost surely,
K
K
Σ (0 (C, S
p (CAA
This completes the proof.
B.2 Proots of Section 4
Proof of Corollary 2. Since ()),, is (1, (y.a))stabilitt, Using the same starting derivation as the proof of Proposition 1, we know that there exists fixed point E A AK-1 such that pr(E:) s exp (n) by + 8; and Σ I T. Thus,
K
P Yz & c(0 (x.) = E [1 {v, & c
Slen.)) (x.)
S(eL..) (x:)
Σ
p (E)
(v. & (x,)
==1
K
eb11 Y, $ c. (x,) +
from which we have that
T
K
lim sup T-000
Σ T
1 Yo $
c(0)
lim sup T+00
Σ
e
bb
1
Y c. (x.) +
1=1
K e
T
b;
lim T-+00
1
Σ
fx
c. c."0 (x))) +
T 1=1
S +
where the second inequality follows from Fubini's theorem, which allows us to interchange the two sums, and the subadditivity of lim sup. This concludes the proof.
Proof of Proposition 2. Using the definition of ((t) in Algorithm 1, together with Markov's in- equality, we have that
P p(x Ye & ciet }
pl ( EI {r, (x.) 2
-P
p (w()) ,,) (x c. ci"0 (x.))
<2 E
Moreover, from MinSE, we know that p. (w(()),, (x) s e (t) s; and D 8 S Therefore,
K
p (yr & cl } 2 e
Σ (0) {vr $ cx +T) (c*3+++)
where the equality follows from (8). This concludes the proof.
B.3 Proof' of Section 5
Notation reminder. Before providing the proof of Theorem 2, we introduce some additional notation and recall some notation the main text. We consider calibration dataset Drat = (X,, Ei) } and test point (X,Y). Y). We interchangeably denote (X, as (Xm+1, Ym+1) and
25

---

define D {x,,, xe)emm+1 In addition, for each base predictor [K], we compute the non-conformity scores the calibration data, for [m]]
Sh = 5% ( ft.
and denote the score of the test point as Sk,m+1 = 8k (x Ym+1, Let Ty: == f.k.1....... Skmml denote the set of these calibration scores for model k. We also define 7 = Ti U ( We use (k,(r) to denote the r-th order statistic in Th and T Furthermore, we denote the ranks of
$R,(r) and s+
score Sk,i within the two sets as follows R = ンドー [(sk.j S sn,i and R = ΣØ=1 [ S $4,r). Similarly to Section 5, we parameterize the conformal prediction sets using ranks. For rank index R E [m], define, respectively:
Cx(X, = (y Eye:: (x fx) S Sn,(R))
C (X, = y V: sx fk) ))==) S st.11))
Both set families are monotonic: (x) S Cx(X, if < R2; similarly for C The relationship SR,(R) 2 + *R,(R) holds for R E [m], implying the set inclusion C (X, R) C C&(X, To relate with the standard perspective split conformal threshold, note that for i' E [m]
P R Ri 28) = P ) P ( S $k,60)
(Ym+1 Ca(Xm=1, 2'))
where { { S { 2 l'(mm + 1) by exchangeability. Note that this is equivalent up to reparameterization the split conformal method introduced in Section 2, by setting i' (1 + 1)7.
For each point E [m 1], apply the selection rule using its feature vector X; and independent randomness 6; ~ Pe. Let k; S(X,, be the index of the predictor selected for the i-th data point. By our assumption, k; independent of Deat, conditionally to X,. We now define the effective rank for the i-th point. This is the rank of the i-th point's score calculated using the predictor k; that was selected specifically for X₁:
R = R = R
S(X.)))))
Similarly, for i E [m]] we define
R2 == Re Rs(x,.e))
Finally, we define the following sequences R+ (R R and R = (R1, ........ Rm)) with (i) and Repy denoting their i-th and i'th order statistics respectively, for E [m + 1] and i' T [m].
Proof of Theorem 2. Notice that R+ forms an exchangcable sequence [3, Lemma 2.2]. This follows from the exchangeability of the original data pairs {(X,, Y m++1 and the fact that the procedure to obtain रिद is symmetric w.r.t. D In addition, R. C R+ implies that R(m I R(m-- As direct consequence, for E [m]. we get:
<P A m++
R+ RR)
(1)
SP
s Km 1. (B/
S Rm
(R (Att))
m +1
kmm 1.++1 S "mm+i (Attg)
km km-1.m+1 S 2km-1s(Pee
P Yn++ E Chemrs (Xm+11 R(0)
26

---

Here, the first line follows from the monotonicity of the order statistics of (s) kmm=
Rm+1.
and the definition of Rm+1 The second line is by the fact that for any & E [K], i' E [m + 11, we have at k.( - The third by Rit) S (1) for any i [m]] and the fourth by the monotonicity of the split conformal set w.r.t. increasing score. Finally, choosing i = [(1 - am + 1)], we get
P {Ym+1 E CX +1,
R (()))
=P P( (( Ckmms (X R())} 2 1 a.
and recover the theorem.
Additional Experiments
C.1 Batch Experiments
In Section 6, we provided some results in the batch setting. Here, using the same selection algorithms as in Section 6, we extend the experimental setting detailed in the following subsections. We first provide additional information on the batch experiments.
For the real dataset experiments (Abalone, Bike Sharing [31, CC BY 4.0), California Housing [30, BSD License]). the hyperparameters of several base regression models were optimized prior to their use the main conformal prediction experiments. This tuning was performed using "BayesSearhhCV" from the scikit-optimin subjected this tuning process included RandommFoorestRerressoo GradieenBooostinggegrrsss ElasticNet, Decision TreeRegressor, AdaaBosttRegressor, and Reerrarr All experiments took approximately 200 CPU hours using 16 cores of Intel Xeon CPU Gold 6230 and 32 GB of system memory. For the implementation of ModSel and the structure of our code, we based our implementation on the version open-soures by Liang et al. [11]. Nonetheless, we significantly deviated from their implementation to allow for more efficient parallel processing.
library [32, BSD-2 license], as mentioned in Section 6. The models
Both X and Y were standardized. The BayesSearchCV process configured to for 25 itera- tions (n iter=25) with 3-fold cross-validation (cv_folds=3) for each model and hyperparameter setting. The nonconformity score function also utilized a RandomForessReggee and its hyperparameters were the same as the tuned parameters for FonndomforsstRegressor for prediction.
predict residuals,
Finally, for the synthetic experiments to follow, we note that YK-Adjust produced infinitely large sets in some runs, such it is not plotted.
C.1.1 Homogeneous vs Heterogeneous Data Preprocessing
In Section 6, we preprocessed the dataset, both synthetic and real, by splitting them to 5 disjoint equal subsets using constrained K-means [35, 36, Code under BSD 3-Clause License]. Here, we provide additional results, without this splitting step. We call the setting with no splitting, homogeneous, and the setting with splitting heterogeneous, The homogeneous setting may represent a more challenging environment for our approach and can be more favorable to the methods of Liang et al. [11] and Yang and Kuchibhotla [12]. particular, this homogeneous setting, one conformal predictor can typically be superior to all other predictors, as such simply selecting the best-on- rest-on-average predictor can yield very competitive results, specially that the stability-bar -ased coverage guarantees can be conservative.
Using the same synthetic data setting as in Section 6, we report the homogeneous results in Figure 3. In addition, we report the homogeneous results on the real data experiments in Figure 4. For real data experiments, Recal stays competitive with baselines. Nonetheless, for the synthetic experiments, our approach performs worse than baselines.
C.1.2 Varying Data Generation in Synthetic Experiments
To further compare with baselines, we adjust the data generation procedure in the synthetic experiment to match one of the settings in [11]. In particular, we repeat exactly the same steps the synthetic
27

---

YiC-Buae
ModSel
Recal
AdaMinSE o' 0.50
MisE y 1.0
1.00
0.825
0.0
0.000
Interaa Lmmeth
0,273
Covner
0.80
0.70
0.225
0.200
0.00
0.175
0.00
00
30
01
70
0.100
10
20
30
60
Number Confictiol Prodictors
Numter Condormal Predettos
Figure 3: Homogeneous synthetic results. Each plot shows coverage (left) and interval length (right)
YK-Adjus
ModSei
Rocal
AdaMin8F.
MinSE v -
1.00
3.0
0.80
Interaal Lesulls
2.0
1.5-
0.80
0.20
0.5
Biee Sharing
Abelone
California Housing
Blue Shottng
Abaloue
Califoruia Houeing
Figure 4: Comparison of UCI dataset results under homogeneous data processing.
experiments of Section 6, but replace the data generation procedure step by
X ~ .N(0, Ia),
E E~ c~,(0, 0, (i mod 20 = 0}
Y = x +
where d = 300. This matches the sparse normal setting with Gaussian noise in Liang et al. [11]. Given the linear dependency, this setting behaves more similarly to the homogeneous experiments; thus the globally optimal predictor may be inferred by predictors learning on any subset of the data. We report the results in Figure 5. We observe that Recal achieves very similar results ModSel. K-Base produces smaller conformal sets at the cost of higher miscoverage.
YicBase
YKcAjust
ModSel
Recal
AdaMin SE
MinSE
1.00
24
22
0.90
20
80
$0.80
1.
16
0.60
12
10
Conformal
Conformal
Figure Comparison with baselines in the sparse linear setting with Gaussian noise
28

---

C.1.3 Comparison against Average Model Baseline
Furthermore, we repeat the experiments of synthetic experiments of Section 6, but add an additional baseline. In particular, we consider the average model baseline, which averages all predictors trained on the data and conformalizes the average predictor. This baseline may be interpreted as a simple model averaging approach. We report the results in Figure 6, denoting this additional baseline as AvgSplit. We observe that Recal, AdaMinSE, and MinSE perform better than the average model baseline.
YKBase
Yk-Ajjust
ModSel
Recal
AdaMinSE
MinSE
Avg
1.00
0.60
0.90
0.55
0.50
60.80
0.45
0.35
0.60
0.25
0.50
16
20
40
40
50
Number Predictorr
Prodictors
Figure 6: Comparison with average model baseline
C.1.4
Effect of the Number of Calibration Points
In addition, we repeat the experiments on synthetic data, while varying the number datapoints in the calibration dataset. This mainly affects the results of YK-Adjust as its performance improves with larger calibration datasets. We report the corresponding results in Figure and Figure 8.
C.1.5 Additional Classification Experiments
We complement our regression studies with a compact ImageNet-Ik classification [37, Non- Commercial Use] experiment intended to emulate heterogeneous setting. Concretely, we construct two ViT-Base models from the same pretrained model: one kept clean, and one degraded by randomly shuffling the top-k logits fraction of examples (we moderate corruption fraction of 10% and 20). Then, each of the two models used to construct a conformal predictor. For the models, used the pretrained timm checkpoints [38].
Then, we construct 10 conformal predictors by mixing the two base conformal models in different ways, i.e. each model outputs the predictions of the clean for 50% samples and degraded for the remaining. We use this structure to where different predictors perform better/worst on different samples.
Using this construction, we compare YK-Base, MinSE, and AdaMinSE, which operate purely the set level - inspecting only the resulting conformal sets without requiring access to the underlying scores or split-conforme internals. We also evaluate a averraieg baseline of Luo and Zhou [13] that linearly combines per-class scores via simplex weights learned on held-out indices. Because this baseline requires score access, we perform the mixing at the logit/score level rather than the set level. We report the results in Figure 9. We note that this setting is one where the performance of different predictors varies across the input space, which precisely where our approach is most beneficial.
C.2 Online Experiments:
Online Setting Experiments: tested online algorithm, AdaCOMA, by constructing online analogue of our heterogeneous batch setting, where the performance of different predictors vary across time. The key advantage of AdaCOMA lies in its ability condition selection the current features X,, via the observed interval sizes & used in the stable selection mechanism. In contrast, COMA relies solely historical performance in determining its weights (((() (Algorithm 1).
To evaluate this, we designed an environment in which both COMA and AdaCOMA track and assign weights to K 10 distinct forecasters. These forecasters are not fixed models; instead,
29

---

Muid
MasB LD
13
0.70
Nacler Chaberaa Tradiies
N
300
YSL-Dmm
YKoAdpe
MedBal
Docal
AdaMedE of 1150
MusE
0250
0.20
0220
Ntader
Prolete
Ntader Chakarnll loteet
(b) N 400
V52 Dtaoe
+ Mortel
MAMMSS of 0
Mhsh 1.
130
0.30
0.
022
8.5 33
0.16
(c) N 500
Figure 7: Homogeneous synthetic results. Each plot shows coverage (left) and interval length (right) for a different number of calibration examples (N).
they dynamically generate prediction intervals by drawing from smaller pool of M 6 diverse base online conformal algorithms. Each base algorithm an instance of Adaptive Conformal Inference (ACD) [6] applied to online learning model. For each of the K forecasters, we simulate heterogeneous environment where the optimal conformal predictor varies over time by partitioning the input data stream conceptually as follows: the start of the experiment, each of the K forecaster pick one model from the smaller subset M models. Then each 7 = 50 timesteps, the forecasters pick a different models, and This setup, where forecasters change their models each timesteps, aims to simulate an environment, which requires the selection algorithm choosing among the forecasters to be strongly adaptive.
Base Online Conformal Algorithms. The M = 6 base algorithms were instances of Adaptive Conformal Inference (ACI) [6]. Default ACI parameters were: initialization period of 100 timesteps, used adaptive ACI stepsize adapted from the original code of [10]. The underlying online learning models were:
Two SGD regressors: one with L1 penalty (Lasso. = 0.001, penalty = 0.1) and one with L.2 penalty (Ridge, ?/- = 0.001, penalty a = 0.1).
Two SGD regressors (no penalty) with learning rates %o E (0.001, 0.005).
Two Rolling Linear Regression models with window sizes of 50 (retrain frequency 12) and
100 (retrain frequency 25).
30

---

YR
VioAdjo
MorSal
Doca
AdAMesE
Mudli L.
13
Stadier Cafarrad Podaaces
Ntader Coobiraa Predetaa
N -300
Bace
VicMjjt
MosS
MisSE L.D
133
0.90
3.50
8:5
23
20
(b) N = 400
Moiid
MisBB LD
1.30
33
0.70
Naabor of Cuadirad Pradicss
N 500
Figure 8: Heterogeneous synthetic results. Each plot shows coverage (left) and interval length (right) for different number of calibration examples (N).
YK-Base
MinSE
A.daMiiSE:
Score Averaging
10
0.80
Interval Lengtta
Covosuge
0.00
0.30
YK-Bum
MinSE
AdaMiSSE
Store Averuging
YK-
MiNSE
AdaMinSE Sccce Averaging
Figure 9; ImageNet classification results
31

---

We conducted experiments on two datasets, used for evaluation in Gasparin and Ramdas [10]. ELEC [39, CC-BY 4.0] and synthetic data generate according the ARMA(1,1) model [40, Chapter 21. For the precise data generation procedure for 1RMA(1,1) model, refer to [10, Page 18.j. For both, AdaCOMA and COMA, we used two algorithms for the weights w(t) (Algorithm 1) over the forecasters: AdaHedge and Hedge with learning rate 0.1. We repeated the experiment for 50 seeds. For ARMA(1.1) dataset, 4 runs experienced numerical instability producing interval length multiple orders of magnitude above the rest for both COMA and AdaCOMA. We excluded those runs in calculating the reported results. The results are presented in Table 1. For COMA, we ran the underlying predictors using ACI with nominal miscoverage rate 0.1. For AdaCOMA, we ran ACI with nominal miscoverage rate of 0.09 and used AdaMinSE: for the selection. For AdaMinSE selection, we tuned the selection such that COMA and AdaCOMA achieve similar coverage. The results are reported in Table 1. For ELEC dataset, AdaCOMA significantly outperformed COMA. For ARMA(I.1), both methods performed similarly with a small advantage to AdaCOMA.
Table 1: Comparison of COMA and AdaCOMA with different underlying aggregation algorithms (AdaHedge. Hedge) on ELEC and ARMA(1.1) datasets. Values are mean standard deviation (divide by the root of the number of seeds (1/v50) for the standard error). The target miscoverage is 0.1.
Dataset
Method
Avg. Miscoverage
Avg. Length
COMA (AdaHedge)
0.0942 + 0.002
0.69 + 0.06
AdaCOMA (AdaHedge)
0.0959 = 0.001
0.32 = 0.09
ELEC
COMA (Hedge)
0.0942 = 0.002
0.69 = 0.06
AdaCOMA (Hedge)
0.0963 = 0.001
0.31 0.01
COMA (AdaHedge)
0.102 = 0.001
4.40 0.91
AdaCOMA (AdaHedge)
0.101 0.001
4.23
ARMA(1,1)
COMA (Hedge)
0.102 = 0.001
4.40
AdaCOMA (Hedge)
0.101 + 0.001
4.31 + 1.06
32


Reviews:
[{"review_id": "MkjsHT8O2J", "reviewer": "Reviewer_MLMo", "summary": "This paper takes up an interesting challenge of comparing conformal prediction (CP) sets that are obtained via different learning methods. To this end, the paper proposes a stability-based approach that focuses on coverage for the selected prediction set. It introduces a few stable selection rules, of which MinSE is proved to be optimal. The paper also shows the applicability of this approach to online CP and validates the contributions through a reasonable suite of experiments. It also improves the empirical performance on split CP through a recalibration step.", "questions": "Please answer the concerns in the weaknesses above. Below are some additional questions:\n* In Corollary 1, how does one find $\\hat{S}$ in practice?\n* It is interesting to see that when $b$ is a uniform prior (in Sec 3.2), MinSE reduces to deterministically selecting the smallest set. What would this be if $b$ were a prior based on the class frequencies?\n\nAlthough I have marked the score as BR, I am on the borderline on this paper. The approach is interesting and nicely done; however, the lukewarm results and lack of discussion around the size of prediction sets in later sections of the paper make the final takeaways not as consistent as may be required. I look forward to seeing the authors' responses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+ The idea of using algorithmic stability to get the smallest prediction set in CP that satisfies coverage is novel and interesting.\n+ The case studies in Sec 3.3 was nice to see.\n+ The additional perspectives on extension to online CP, formulation of adaptive MinSE and recalibration for split CP show a good depth of analysis of the proposed approach.\n\nWeaknesses:\n- The paper largely relies on [4] for their work on algorithmic stability and applies those ideas to CP. However, I do think the application to CP is non-trivial and done well.\n- For the contribution in Sec 3, a limitation of this work seems to be that you needs to run CP multiple times and get multiple sets/intervals, and then apply this method to find the smallest one. What is the computational overhead of this approach, and what is its practicality? Considering the paper largely focuses on adopting algorithmic stability to CP, it may be important to also comment on the additional complexity introduced. \n- Continuing with the above point, the empirical results for this contribution may need to be compared against methods that also have this advantage of seeing multiple prediction sets. Would there be a more fair comparison where the proposed method is compared against other approaches that find the smallest prediction set given multiple CP predictors? \n- In Line 139, the paper refers to $\\lambda$ as the \"size\" of set $C_i$, and in the subsequent analysis, $\\lambda$ is assumed to be $\\in [0,1]$. How would one define such $\\lambda$s in practice that actually capture the size of a set (considering the objective of this work is to find the smallest prediction set)? This seems to be key to the stable selection procedure, and I could not find how this was defined in the results section of the main paper too.\n- AdaCOMA (Sec 4.1) does not make any comments on the size of the prediction set in this setting. Is it possible to comment on this?\n- In the main results (Fig 2), YK-Base seems to do as well/outperform the proposed method on the interval length in Fig 2. While the paper says YK-Base does not guarantee coverage, if $\\alpha=0.1$ (as stated in Line 343), YK-Base seems the most calibrated on Bike Sharing and California Housing.\n- Also, it is concerning that while the paper compares with references [11] and [12], the datasets used are different from what is used in those papers. This makes it hard to know if the datasets used herein are just favorable for the proposed results, or if these generalize to the datasets used in [11] and [12].\n- (Editorial point, but relevant here) It would have been nice to specify $\\alpha$ in the caption of Fig 2 itself. It was hard to parse the figure by looking through the text for this value. This issue is exacerbated in the appendix where it was hard to find the confidence level, without which it is hard to interpret coverage results.", "was_revised": true, "final_justification": "I thank the authors for their responses; they clarified most of my questions. I am increasing my score to a Borderline Accept. \n\nI am also satisfied with the responses from the clarifications in the discussion period, and believe that the paper makes a useful contribution. \n\n(I also want to appreciate the authors for stating and clarifying the code bug issue in a transparent and honest manner.)", "rebuttal": "We thank the reviewer for their thoughtful feedback. Below, we address your valuable comments and questions.\n\n**Response to Weaknesses:**\n\n*  **Regarding Novelty:** We thank you for acknowledging the novelty and non-trivial nature of our contribution.\n\n*  **Regarding Computational Overhead:** We appreciate this practical question. The complexity has two parts: generating $K$ candidate sets, which is somewhat inherent to ensembling-like methods and scales linearly in $K$, and our selection step. The selection, using our algorithms, is efficient. For example, although, we formulated MinSE as a linear program (generally requiring approximately $O(K^3)$ steps to solve), it actually admits a greedy solution with $O(K \\log K)$ complexity. This is often negligible compared to the complexity of model inference. \n\n3.  **Regarding Fairness of Empirical Comparison:** We chose methods from Yang et al. (2024) and Liang et al. (2024) as they are prominent recent works that address the same challenge of selecting from a pool of conformal predictors. As such, the benchmarks are also methods that see and select among a pool of conformal sets. \n\n4.  **Regarding the Definition of Set Size $\\lambda$:** The assumption that $\\lambda(C_i^{\\alpha}(X)) \\in [0,1]$ is made only for notational convenience in Lemma 1 and 2, as noted on Line 157, and is not required for MinSE (and its extension AdaMinSE). As shown by Proposition 1, MinSE algorithm represents the optimal stable algorithm. As such, within the context of this paper, there is generally no motivation to use Lemma 1 and 2 for the selection step and we mainly included them to illustrate different examples of stable selection algorithms. Nonetheless, even for Lemma 1 and 2, the requirement that $\\lambda(C_i^{\\alpha}(X)) \\in [0,1]$ can be relaxed up to a simple rescaling preprocessing step. For all of our experiments in the paper, we used $\\lambda$ to be the size the set as measured by the Lebesgue measure. For the experiments in the classification setting, $\\lambda$ can be set to the counting measure of different confidence sets. \n\n5.  **Regarding AdaCOMA and Prediction Set Size:** You are correct that the main text focuses on coverage. The practical advantage in set size is shown in our online experiments in the Appendix, where AdaCOMA produces smaller sets than the COMA baseline by adapting to the current data point $X_t$, while maintaining target coverage.\n\n6.  **Regarding Comparison with Baselines:** YK-Base indeed performs reasonably well empirically. Nonetheless, it lacks a formal theoretical coverage guarantees. Moreover, on figure 1, Recal systematically provides better results (same coverge with much smaller intervals) and MinSE and AdaMinSE provide shorter or similar intervals with higher coverage on the two of the 3 examples. On the third one YB-base actually undercovers. In addition, in [11], the authors showed an experimental settings where YK-Base undercovers. \n\n7.  **Regarding Dataset Choice:** This is a fair point. The authors of [11] have open-sourced their code. To facilitate a direct comparison, we commit to add experiments on their synthetic data setting in the final version of our manuscript. \n\n8.  **Regarding Editorial Point** We will specify the target miscoverage $\\alpha=0.1$ in all figure captions.\n\n\n\n**Response to Questions:**\n\n1.  **How to find $\\hat{S}$ in practice?** Corollary 1 is a general result. For implementation, Lemmas 1-3 provide concrete algorithms. Our experiments use the MinSE algorithm from Lemma 3, which is efficient and optimal (Proposition 2). As such, we used MinSE for all of our experiments. \n\n2.  **What if the prior $b$ in MinSE is non-uniform?** A non-uniform prior $b$ allows incorporating prior knowledge. For a trusted predictor (producing smaller sets) $j$, a higher $b_j$ allows our MinSE algorithm to select it with higher probability. On other hand, a higher $b_j$ does necessitate decreasing another coordinate $b_i$, as $b$ is constrained to the simplex. As such, MinSE will pick a predictor $i$ with less probability (or even never pick it up if $b_i=0$ and $\\tau=0$). Thus, a non-uniform prior can be seen as blending prior expert belief with the data-driven results observed during inference time (i.e., after observing the conformal set sizes).  Within this paper, we did not explore how to tune the prior $b$. This is partially due our belief that further tuning of our method may cause unfair comparisons with the benchmark that required no tuning. \n\n\nFinally, we thank you for the paper formatting remarks. We will revise the manuscript to better frame and transition between parts that focus on coverage and parts that focus set size. We are grateful for your feedback and remain eager to answer any follow-up questions or provide further details. \n\n\n---\n\n**Erratum and recap on experiments**\nWe wanted to openly disclose that we detected after submission a typo in the code, in file `batch_setting/rescale_residual.py` line 162, `cal_sigma` should be `cal_sigma[m]`. This changes the performance of ModSel, that is ultimately on-par with YK-base (yet never better, in fact YK-base lower bounds the confidence set size of ModSel). Other methods are unaffected. It thus does not change the quality of the \"best competitor\".\n\nThis also doesn't change the conclusion of our experiments, summarized below:\n- on **bike sharing**, our methods MinSE and AdaMinSE  improve coverage by 2% while having a average length 10% below YK-Base. Recal achieves nominal coverage with set length more than 30% smaller.\n- on **abalone**, differences are more minor: both YK-base and ModelSel undercover by 1%,  Recal is valid and reduces set size by 5% against all competitors, while MinSE and AdaMinSE suffer from conservatism, overcovering by 2% and increasing set size by 10% w.r.t. YK-base\n- on **California housing**, YK-base, ModSel, MinSE, AdaMinSE achieve the same length, but with more than 2% of extra coverage for MinSE and AdaMinSE. Recal has nominal coverage with length 25% smaller than any other competitor."}, {"review_id": "WWzLclca85", "reviewer": "Reviewer_CuX3", "summary": "This paper presents a compelling framework for selecting among multiple predictive sets based on any criteria, such as selecting the smallest set, while maintaining the coverage guarantees via algorithmic stability. The proposed procedure has coverage guarantees in both batch and online settings. Notably, their method encourages selection on a pointwise basis.", "questions": "Could you clarify how AdaMinSE automatically optimises the parameters? I've already looked at the proof, but I might be missing something. It seems to me that the approach is mainly bounding the hyperparameters to eliminate them and using $\\tilde{\\alpha}$ instead (Update after further reading: It seems that the parameters become variables rather than inputs to the linear program. Still, I'd appreciate your take if you have a clearer or more accurate explanation.)\n\nAdditionally, if AdaMinSE is truly optimising the parameters, why does MinSE sometimes perform better? Along the same lines, can you prove that your proposal will outperform any single confidence set?\n\nDo you have any intuition on how to select the prior, especially in the batch setting?\n\nI saw in the appendix that you extended your idea for conditional coverage. Could it be extended for PAC coverage?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well-written and enjoyable to read. I believe it makes a nice contribution, the core idea and the results are elegant, especially the pointwise selection aspect, which makes this work distinct from other approaches.  \n\nHowever, despite the nice derivations and theoretical results, the proposed approach does not appear to meaningfully outperform the baseline methods, particularly YK-BASE. From what I understand, the method alone does not surpass the baseline unless the recalibration step described in Section 5 is applied. This makes the comparison somewhat unfair, as it relies on an additional calibration set.\n\nHowever, it seems that the method demonstrates its true strength primarily in the online setting. Could you confirm if this is the case? If it is true, could you give more intuition on this difference?", "was_revised": true, "final_justification": "The authors have addressed all my concerns, and I still think this is a strong paper. I continue to support acceptance.", "rebuttal": "We are very grateful for your review and constructive feedback. We proceed to address your questions below.  \n\n**Regarding experimental performance**\n\nThe stability-based coverage bound (with the $e^\\eta$ factor) can be seen as a worst-case guarantee. As shown in our synthetic examples, this bound is tight in pathological cases but can be conservative in more benign settings. This generality and robustness are precisely what make the framework applicable to the challenging online setting, with no distributional assumptions. However, it can result in overcoverage in less pathological cases.\n\nTo address this potential conservativeness in the more structured batch split-conformal setting, we introduced the recalibration procedure (Recal). It circumvents the stability bound by leveraging the rank structure of split-conformal prediction. We want to emphasize that Recal does *not* use an additional calibration set. We split the *single, existing* calibration data budget. Thus, the comparison with baselines is fair in terms of the total data used. \n\nRecal thus indeed systematically outperforms competing methods, achieving valid coverage with up to 30% smaller intervals. However, MinSE and AdaMinSE do also provide some improvement on the validity/efficiency tradeoff. On 2 datasets, they reduce or maintain the set size (by 10% on Bike Sharing) while significantly reducing miscoverage (by more than 2% out of 10%). On the third dataset (Abalone), their average set size is indeed 10% above YK-base, but YK-base simply does not achieve validity, while both MinSE and AdaMinSE achieve coverage 91+%. See detailed recap on experiments at the end.\n\n\n**Regarding batch vs. online settings:** For our online setting, indeed, AdaCOMA can adapt to shifts more quickly than alternative methods that are purely history-based.\n\n**Response to Questions:**\n\n1.  **How AdaMinSE optimizes parameters:** Your understanding is perfectly correct. Instead of taking $\\eta$ and $\\tau$ as fixed inputs, AdaMinSE treats them as optimization variables within the linear program. The key intuition is that for any initial miscoverage level $\\alpha'$ and a target level $\\alpha$, there is a trade-off curve defined by $e^\\eta \\alpha' + \\tau \\le \\alpha$. For each test point, AdaMinSE automatically finds the point on this curve that minimizes the expected set size for that specific instance, thus adaptively choosing the best tradeoff between $\\eta$ and $\\tau$.\n\n2.  **Why MinSE sometimes outperforms AdaMinSE:** This is a subtle but important point. While AdaMinSE is adaptive, its performance depends on the initial miscoverage $\\alpha'$ provided. MinSE, in our experiments, uses a fixed $(\\eta=1, \\tau=0)$. Theoretically, if we were to run AdaMinSE with an initial level $\\alpha' = \\alpha / e^1$, it would be guaranteed to perform at least as well as this specific MinSE instance. The small performance differences observed in our experiments arise because the $\\alpha'$ we chose for AdaMinSE may not have perfectly corresponded to this optimal point for the given dataset, making the fixed-parameter MinSE appear slightly better by chance.\n\n3.  **Proof of outperforming a single confidence set:** We cannot prove that our method will produce a smaller set than the best single predictor for *every* point. However, our framework is designed to be superior *on average* in heterogeneous settings where no single predictor is uniformly best (as illustrated in our Example 3). In such cases, being able to adaptively switch between predictors pointwise will naturally lead to a smaller average set size compared to being locked into any single predictor.\n\n4.  **How to select the prior $b$:** We investigated this direction. A principled way to set $b$ is via cross-validation, which leads to a bilevel optimization problem. The outer loop optimizes $b$ to minimize the average selected set size on a validation set, while the inner loop is the MinSE optimization itself. To make the problem easier, we found that adding a negative entropy regularizer to the inner problem makes it strongly convex and much more tractable, admitting efficient solution methods. For simplicity and because the paper was already dense, we used a uniform prior in our experiments. We will add a brief discussion of this approach for setting the prior to the appendix.\n\n5.  **Extension to PAC coverage:** Thank you for this insightful question. Our current framework provides guarantees on the *expected* miscoverage. PAC-style guarantees, which provide high-probability bounds on the empirical miscoverage rate (e.g., $P(\\text{Miscoverage} > \\alpha + \\epsilon) < \\delta$), are indeed a very interesting and powerful extension. This is a non-trivial next step, as it would require analyzing how the randomness from our selection mechanism interacts with the sampling randomness of the test data to affect the concentration of the miscoverage rate. Nonetheless, we believe this extension is achievable.\n\nThank you again for your feedback, which helps us to improve the paper. We remain available to answer further questions during the discussion period.\n\n\n---\n\n**Erratum and recap on experiments**\nWe wanted to openly disclose that we detected after submission a typo in the code, in file `batch_setting/rescale_residual.py` line 162, `cal_sigma` should be `cal_sigma[m]`. This changes the performance of ModSel, that is ultimately on-par with YK-base (yet never better, in fact YK-base lower bounds the confidence set size of ModSel). Other methods are unaffected. It thus does not change the quality of the \"best competitor\".\n\nThis also doesn't change the conclusion of our experiments, summarized below:\n- on **bike sharing**, our methods MinSE and AdaMinSE  improve coverage by 2% while having a average length 10% below YK-Base. Recal achieves nominal coverage with set length more than 30% smaller.\n- on **abalone**, differences are more minor: both YK-base and ModelSel undercover by 1%,  Recal is valid and reduces set size by 5% against all competitors, while MinSE and AdaMinSE suffer from conservatism, overcovering by 2% and increasing set size by 10% w.r.t. YK-base\n- on **California housing**, YK-base, ModSel, MinSE, AdaMinSE achieve the same length, but with more than 2% of extra coverage for MinSE and AdaMinSE. Recal has nominal coverage with length 25% smaller than any other competitor."}, {"review_id": "bA6FhvlGhT", "reviewer": "Reviewer_6BVL", "summary": "Assume k different conformal predictors (e.g. score taken from an ensemble of models). Directly picking the smallest set among the prediction sets, results in breaking the marginal guarantee in conformal prediction. Same applies to any selection that tunes some objective as it becomes data-dependent. The paper addresses this by valid stable selection. The authors show a set of selections that are stable, and they bound the coverage guarantee of a selection parameterized by the stability parameters. Furthermore they discuss the same setup for time-series data.", "questions": "1. What is the probability in Eq. 3 over?\n2. Why the evaluation on real datasets are so limited? Why didn’t the authors also discuss classification. From what I understand application on classification setup is not a limit of their framework. The models here are not as complicated as a deep model. \n3. If I understood correctly, your definition of stability is based on a reference (intuitively center) distribution. But can we also redefine this based on two random samples of the distribution. Intuitively I think if two samples are not too far with a high probability, they are also both close to a central reference sample.\n4. What prevented authors to evaluate their result on a more complicated setup like ensemble image classification?\n5. Also since the paper defines stable way to predict time-series data, why didn’t the authors try the algorithm on some time series task; e.g. stocks forecast, electricity demand forecast, etc.\n6. I did not fully understood your definition of adversarial setup, and solution for it. To me adversarial setup is defined by a threat model including the set of possible perturbations, and the objective the adversary can achieve. Can you elaborate on this definition?\n\nIf my questions are answered and mentioned weaknesses are addressed, I’ll review the work again and try to increase my score.", "limitations": "No limitations mentioned. But I do not see specific limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strength.**  I think the paper is mathematically solid, and also the contributions are noteworthy, despite their limited evaluations, I was convinced to accept the paper even around Section 3.3.\n\nAlso the paper addresses an interesting problem in CP. Basically they define a lower bound on the prediction set defined from an ensemble of CPs while it is data dependent tuning for a specific goal; e.g. smaller set size.\n\n**Weakness: Empirical evaluation.**  I believe in contrast with the complicated theory, there could be evaluation done on larger datasets, and more complicated model. Of course, authors can easily examine their method on an ensemble of classifiers for the CIFAR10 dataset. \n\nI suggest some rephrasing, however they are only for the readability; feel free to ignore them.\n\n**Line 29.** It is better to reorder the words: In section 4, we extend our work …\n\n**Line 115.** It makes it more understandable if you use an example for the inner-randomness of an algorithm. To me, one example is the uniform random value in the APS score function.\n\n**Line 117.** I suggest to use another notation for algorithms that is dissimilar to random variable, unless there is an inherent similarity in the nature of both entities.\n\n**Line 130.** The random algorithm is stable not stability. Right?\n\n**Examples.** I think the authors have used a block similar to theorem for their examples. The readers eye gets tired as there is an italic text for several paragraphs. I suggest the authors to try some other type of environment that has font in normal style.\n\n**No conclusion, and limitations.** Also why there is no conclusions section?", "was_revised": true, "final_justification": "I read the response by the authors. It addressed my comments. As mentioned while writing the review, I find the contribution considerable and the mathematical justifications to be solid.", "rebuttal": "We thank the reviewer for their constructive review. Your feedback is very valuable in improving our work. We address your questions and suggestions below.\n\n\n* **Regarding Empirical Evaluation:** We thank you for this constructive suggestion. Your feedback prompted us to run new experiments on an ImageNet-1k classification task to better demonstrate the generality and scalability of our framework. Our experimental setup was designed to create a *heterogeneous* and challenging selection environment. We began with two pre-trained base models with different performance characteristics—one with a top-1 accuracy around 90% and another around 80%—and converted each into a standalone conformal predictor. These models naturally exhibit varying performance across the diverse samples of ImageNet. For the conformal scores, we used an adaptive score similar to the one used in the Appendix of [11]. From these two base predictors, we constructed *eight  composite predictors*. Each composite model creates a unique \"risk profile\" by outputing the conformal set either from the high-accuracy model or from the lower-accuracy model, depending on the specific input $X$. These eight models use different probability distributions for this choice, creating a diverse pool of predictors for the selection algorithms to manage. All the composite predictors achieve marginal coverage guarantee. \n\nWe compared the following selection methods `YK-Base`, `MinSE`, and `AdaMinSE`. The results, averaged over 10 runs, are presented below.\n\n| Method   | Coverage (Target: 0.90) | Avg. Set Size |\n| :------- | :----------------------- | :------------ |\n| YK-Base  | 0.895                      | 3.93  |\n| MinSE    |  0.962     | 3.36  |\n| AdaMinSE | 0.949     | 3.27  |\n\nAs shown, our proposed methods (`MinSE`, `AdaMinSE`) successfully achieve higher coverage with smaller sets. This is due to their pointwise selection capability, which allows them to produce smaller average prediction sets than the non-pointwise `YK-Base` baseline.\n\n   \n* **Time-Series Evaluation:** As you correctly noted, our method is well-suited for time-series data. We would like to highlight the online experiments already present in the appendix, where AdaCOMA algorithm is evaluated on electricity forecasting and a synthetic task.\n\n* **Regarding Readability,  Presentation, and Formatting:** Thank you for these helpful suggestions. We will incorporate all of them in the revised manuscript to improve clarity. We also believe that making the example non-italic is easier on the eye. Using the extra page allowed in the camera-ready version, we will a conclusion section, which discusses the limitation. \n\n\n* **Response to Other Questions:**\n\n    *  **What is the probability in Eq. (3) over:** The probability is taken over $\\zeta$ and the randomness of CI itself. The statement holds for all $s$ and $\\alpha$ is assumed to be fixed. \n    * **On distributional notion of stability:** This is an insightful question. Our definition of stability is asymmetric (bounding an output distribution relative to a fixed one) and it is the minimal condition required for our coverage proofs to hold. I.E., the stability statement is only required with the reference distribution on the right-hand side and the output distribution on the left-hand side, but not vice-versa. We believe it would be possible to use the notion of stability you proposed to get our coverage guarantees. Nonetheless, due to symmetry, it may be stronger than necessary for the proofs. \n  \n    3.  **\"adversarial\" setup:** We use the term \"adversarial\" in the standard sense from online learning theory. It means the sequence of data points $(X_t, Y_t)$ can be chosen by an adversary, which allows us to prove worst-case robustness to arbitrary distribution shifts over time. This is distinct from the concept of adversarial examples (i.e., perturbed inputs). We will clarify this distinction in the text.\n\nWe hope these responses and our new experiments fully address your concerns. We are happy to answer any further questions during the discussion period.\n\n\n---\n\n**Erratum and recap on experiments**\nWe wanted to openly disclose that we detected after submission a typo in the code, in file `batch_setting/rescale_residual.py` line 162, `cal_sigma` should be `cal_sigma[m]`. This changes the performance of ModSel, that is ultimately on-par with YK-base (yet never better, in fact YK-base lower bounds the confidence set size of ModSel). Other methods are unaffected. It thus does not change the quality of the \"best competitor\".\n\nThis also doesn't change the conclusion of our experiments, summarized below:\n- on **bike sharing**, our methods MinSE and AdaMinSE  improve coverage by 2% while having a average length 10% below YK-Base. Recal achieves nominal coverage with set length more than 30% smaller.\n- on **abalone**, differences are more minor: both YK-base and ModelSel undercover by 1%,  Recal is valid and reduces set size by 5% against all competitors, while MinSE and AdaMinSE suffer from conservatism, overcovering by 2% and increasing set size by 10% w.r.t. YK-base\n- on **California housing**, YK-base, ModSel, MinSE, AdaMinSE achieve the same length, but with more than 2% of extra coverage for MinSE and AdaMinSE. Recal has nominal coverage with length 25% smaller than any other competitor."}, {"review_id": "yVjoQpdvpY", "reviewer": "Reviewer_bYvi", "summary": "This paper studies the problem of constructing a conformal set when K different conformal predictors are given. Both the batch setting (i.e. standard split conformal prediction) and the online setting (without stationarity assumptions) are considered. A central point of departure is the notion of algorithmic stability, which allows for constructing valid conformal sets when an optimization step over the K existing conformal sets is needed (Theorem 1 and Corollary 1). In Lemma 3 a concrete algorithm that has theoretical coverage guarantees is derived. In Section 4 the online setting is considered, and a theoretical comparison with the existing COMA algorithm is made. In Section 5 the method is further improved by calibration after selection using effective ranks. This leads to Theorem 2, which shows that the method achieves marginal coverage after optimizing for the best among K conformal predictors. In Section 6 the new methods are experimentally compared to a few baselines on synthetic data and three real-world regression datasets.", "questions": "See weaknesses.", "limitations": "Limitations of the presented methods are not discussed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 4, "strengths_and_weaknesses": "For me the main strength of the paper is that it introduces quite some concepts that are new in the area of conformal prediction (according to my knowledge of the literature). These concepts can be useful when aggregating different conformal predictors in an ensemble learning setting. The authors clearly show that they have a strong knowledge of the theory in this field. The paper is written in a mathematically rigorous way. \n\nThe main weakness of the paper is that it is hard to follow the story (even for researchers with prior knowledge of conformal prediction I would say). The paper is written in a very dense manner, and many design choices are not really justified. I have read the paper two times. The first time I had a hard job to understand Section 3-6. The second time it was better, but I still have a lot of questions. Here I am listing the points that were not clear to me. \n\n- It is not really discussed in which ML setting the proposed algorithms would be useful. I would guess that the main application domain is ensemble learning, either ensembles constructed using bootstrapping, or ensembles that involve randomizing model parameters (such as deep ensembles). Please be more specific in this. \n- For such a setting, an obvious baselines would be performing conformal prediction on the mean of the ensemble. Why is this not a useful approach and inferior to the complicated aggregation that the authors propose?\n- In the experiments on synthetic data, an ensemble is created by randomizing the kernel function and the regularization parameter in kernel ridge regression. Why would that be useful? I would argue that one should first optimize the hyperparameters using grid search, and run conformal prediction on the best performing model in terms of MSE. Would such an approach not perform better? Why not consider bootstrapping or another more useful technique to construct an ensemble?\n- In the experiments the baselines that are implemented are references [11] and [12]. For [11] it is more or less clear to me what it does, but for [12] the authors only say that it is the \"best-on-average\" conformal predictor. I have no idea what this means without reading carefully [12]. More information is needed for readers that don't know [12].  \n- From Corollary 1 I conclude that the theoretical coverage highly depends on the hyperparameters \\nu and \\tau. In the experiments they are just set to one and zero. Isn't this like a very loose coverage bound? (Instead of 1-\\alpha coverage one obtains 1-\\alpha\\epsilon coverage, which is much lower?  \n- In the introduction the authors write: \"In contrast to [11] and [12] our method allow pointwise selection depending on each X\". I don't see why this claim would be the case. The way I interpret the MinSE optimization problem in Lemma 3 is that the best convex combination of model is found. The same p_i values are used for all X, so the convex combination is not specific for each X? \n- In the experiments on real data, what are the K models? Are these the six different regression models from scikit-learn, so K=6? Not 100% clear. \n- The absolute residuals are considered as nonconformity scores, leading to prediction intervals that have the same length for all instances. This might be a clear disadvantage when interval size is a criterion, so it should be relatively easy to outperform baselines using this nonconformity score. I would be interested to see experiments where estimated quantiles are used as nonconformity scores (i.e. conformalized quantile regression). I expect that it becomes much more difficult to outperform the baselines using such a nonconformity score.    \n- Would the proposed method also be useful in classification settings? For such settings, specific nonconformity scores that optimize set size have been proposed, such as the adaptive prediction sets (APS) and its variants. A comparison to such methods could also be useful.\n\nA lot of these questions are related to explaining the experiments better and comparing to different baselines. The experiments were in general a bit underwhelming, but I did not check the additional experiments in the appendix.", "was_revised": true, "final_justification": "I think that the arguments given by the authors in favor of their approach compared to the mean ensemble as a baseline can make sense, but this all still has to be tested empirically. However, I completely understand this is not possible within a one week rebuttal period. In summary, I think that the method has interesting novel components, but I am not sure whether it performs as good as the authors claim. I will stick to my score of a \"borderline accept\".", "rebuttal": "We thank the reviewer for their detailed feedback and for recognizing our work's mathematical rigor and originality. We are glad you found our contributions noteworthy. Your comments on the paper's clarity and experimental justification are very helpful, and we address them below. They are roughly by the order of first mention of the review.\n\n1. **On the paper's clarity and flow:**\n    We agree that the paper is dense and thank you for taking the time for a second reading. We will use the extra page for the camera-ready version to revise the manuscript, focusing on improving the flow and better motivating our design choices to make the story easier to follow.\n\n2. **Application Setting and Ensembling:** Thank you for this question, which allows us to clarify a key aspect of our contribution. There are two ways to use ensembles for uncertainty quantification: (1) average the model predictions and then conformalize the final predictor, or (2) generate a conformal set for each model and then select among the sets. Our work focuses on the second, which can be seen as \"ensembling in the space of conformal predictors.\" This is a general concept that applies even with a **single** underlying prediction model, which itself can be an ensemble. For instance, one could have a single model but multiple different nonconformity scores. Our framework provides a valid way to select the among the different conformal prediction strategies. As such, even with the mean of an ensemble, it is possible to leverage it to build multiple conformal predictors. Thus, even in this case, our approach can still be valuable to select among different conformal predictors. \n\n\n3. **Experimental Motivation:** The synthetic experiment aimed to illustrate this core idea by creating a diverse pool of conformal predictors. To make the application setting clearer, we will add a new experiment in the revision where we fix the underlying regression model but vary the conformalization procedures themselves. As such, we will have different conformal predictor build on the same single model.\n\n\n4. **On the baselines from references [11] and [12]:** The baseline from [12] is the *\"best-on-average\"* conformal predictor in the sense that all $K$ conformal predictors are reevaluated on the full calibration set. Then, a single predictor (among the $k$ predictors) with best average performance (e.g., smallest average set size) is identified and then use that predictor for all future test points. This contrasts sharply with our method, which makes a dynamic, **pointwise** selection for each new point. We are pointwise in the sense that for every new test point $X$, we evaluate the conformal sets at $X$, and try to select the smallest set at $X$. To refer to example 3 in the paper, a best-on-average predictor will try to find the better performing conformal predictor among the third and fourth images ($f(X)=X$ and $f(X)=-X$) based on the average set size. Then, this predictor will be used for all future test points. Meanwhile, a pointwise mechanism makes the choice based on the specific realization of $X$ and as such produces the fifth figure ($\\eta=\\ln(2)$). \n\n5. **On the looseness of the coverage bound:** The bound involving the $e^\\eta$ factor, while potentially conservative, is **tight**. We show in Example 1 that there exist pathological (but valid) settings where the miscoverage is exactly $e^\\eta \\alpha$. This is the theoretical price for a guarantee that holds under minimal assumptions, even in non-stationary online settings. For practical use, we propose two solutions that address this: `AdaMinSE`, which adaptively tunes the stability parameters, and our `Recal` procedure, which completely circumvents this factor in split-conformal settings by performing recalibration after the selection procedure.\n\n6. **On the \"pointwise selection\" claim.**  This is an important point: the MinSE optimization is solved **separately for each test point $X$**. The optimization's input is the vector of set sizes $\\xi = [\\lambda(C_1(X)),\\ldots, \\lambda(C_k(X))]$, which explicitly depends on $X$. Consequently, the optimal probabilities $p_i(\\xi)$ are a function of the current test point, enabling true pointwise selection. We will revise the text to emphasize this mechanism and clarify this aspect in the paper. \n\n7. **Real data models:** Indeed, those are six regression models from scikit-learn and $K=6$. \n\n8. **On the nonconformity score and interval length:** We do *not* use absolute residuals, which would lead to constant-length intervals. Our nonconformity score is $s_i(X, Y) = |f_i(X) - Y|/g_i(X)$, where $g_i(X)$ is a model trained to predict the residuals. This creates *adaptive* prediction intervals whose lengths vary with the input $X$, allowing each method to produce tighter intervals for \"easier\" instances.\n\n9. **On the details of the real-data experiments.**\n    For the real-data experiments, we used $K=6$ models, and we will make this explicit. \n    \n10. **On the applicability to classification:** Yes, our method is directly applicable to classification. To demonstrate this, we conducted a new set of experiments on the ImageNet_1k, which we will include in the revised manuscript. We believe this new experiment provides a compelling use case for our proposed selection framework.\n\n    The experimental setup was designed to create a **heterogeneous** and challenging selection environment. We began with two pre-trained base models with different performance characteristics—one with a top-1 accuracy around 90% and another around 80%—and converted each into a standalone conformal predictor. These models naturally exhibit varying performance across the diverse classes in ImageNet. For the conformal scores, we used an adaptive score similar to the one used in the Appendix of [11].\n\n    From these two base predictors, we constructed *eight  composite predictors*. Each composite model creates a unique \"risk profile\" by outputing the conformal set either from the high-accuracy model  or from the lower-accuracy model, depending on the specific input $X$. These eight models use different probability distributions for this choice, creating a diverse pool of predictors for the selection algorithms to manage.\n\n   We compared the following selection methods `YK-Base`, `MinSE`, and `AdaMinSE`. The results, averaged over multiple 10 runs, are presented below.\n\n| Method   | Coverage (Target: 0.90) | Avg. Set Size |\n| :------- | :----------------------- | :------------ |\n| YK-Base  | 0.895                      | 3.93  |\n| MinSE    |  0.962     | 3.36  |\n| AdaMinSE | 0.949     | 3.27  |\n\nAs shown, our proposed methods (`MinSE`, `AdaMinSE`) successfully achieve higher coverage with smaller sets.  Furthermore, their pointwise selection capability allows them to produce smaller average prediction sets than the non-pointwise `YK-Base` baseline, demonstrating their advantage in more heterogenous settings.\n\n\n* **On the missing Limitations section.* We tried to discuss limitation in the checklist and throughout the paper. Nonetheless, we agree that a dedicated limitations section is of value and we will add one to the revised manuscript.\n\n\n\nWe thank the reviewer for his support and feedback. We remain available for any further discussion.\n\n\n---\n\n**Erratum and recap on experiments**\nWe wanted to openly disclose that we detected after submission a typo in the code, in file `batch_setting/rescale_residual.py` line 162, `cal_sigma` should be `cal_sigma[m]`. This changes the performance of ModSel, that is ultimately on-par with YK-base (yet never better, in fact YK-base lower bounds the confidence set size of ModSel). Other methods are unaffected. It thus does not change the quality of the \"best competitor\".\n\nThis also doesn't change the conclusion of our experiments, summarized below:\n- on **bike sharing**, our methods MinSE and AdaMinSE  improve coverage by 2% while having a average length 10% below YK-Base. Recal achieves nominal coverage with set length more than 30% smaller.\n- on **abalone**, differences are more minor: both YK-base and ModelSel undercover by 1%,  Recal is valid and reduces set size by 5% against all competitors, while MinSE and AdaMinSE suffer from conservatism, overcovering by 2% and increasing set size by 10% w.r.t. YK-base\n- on **California housing**, YK-base, ModSel, MinSE, AdaMinSE achieve the same length, but with more than 2% of extra coverage for MinSE and AdaMinSE. Recal has nominal coverage with length 25% smaller than any other competitor."}]

Author rebuttal:

Meta-review: This paper presents a novel and well-motivated framework for selecting among multiple conformal prediction sets while maintaining coverage guarantees. The reviewers found the proposed stability-based approach to be elegant, technically sound, and a valuable contribution to the field. Initial concerns regarding computational overhead, empirical comparisons to baselines, and certain methodological details were effectively addressed by the authors during the rebuttal period. The authors' clarification of their method's performance and their transparent handling of a minor code bug in a baseline comparison were appreciated. Following the discussion, the reviewers were satisfied with the authors' responses and acknowledged the paper's strengths, viewing it as a solid contribution to the literature.
Decision: Accept (poster)