Conformal Prediction for Ensembles: Improving Efficiency via Score-Based Aggregation
Authors: Yash Patel
Eduardo Ochoa Rivera
Ambuj Tewari
Venue: NeurIPS 2025 poster ()
Keywords: uncertainty quantification
conformal prediction
conformal aggregation
ensembling
Primary area: probabilistic_methods
TL;DR: Rather than performing conformal aggregation in the space of prediction regions, we propose a vector extension of the conformal score, through which aggregation can be performed more efficiently.

Abstract: Distribution-free uncertainty estimation for ensemble methods is increasingly desirable due to the widening deployment of multi-modal black-box predictive models. Conformal prediction is one approach that avoids such distributional assumptions. Methods for conformal aggregation have in turn been proposed for ensembled prediction, where the prediction regions of individual models are merged as to retain coverage guarantees while minimizing conservatism. Merging the prediction regions directly, however, sacrifices structures present in the conformal scores that can further reduce conservatism. We, therefore, propose a novel framework that extends the standard scalar formulation of a score function to a multivariate score that produces more efficient prediction regions. We then demonstrate that such a framework can be efficiently leveraged in both classification and predict-then-optimize regression settings downstream and empirically show the advantage over alternate conformal aggregation methods.

Conformal Prediction for Ensembles:
Improving Efficiency via Score-Based A Aggregation
Eduardo Ochoa Rivera" Department of Statistics University of Michigan Ann Arbor, MI 48 104 eochoaßumi ch. edu
Yash Patel"
Ambuj Tewari
Department of Statistics University of Michigan Ann Arbor, MI 48 104 yppatel@ume ch. edu
Department of Statistics University of Michigan Ann Arbor, MI 48 104 tewari iaarria@uiiee ch, edu
Abstract
Distribution-free uncertainty estimation for ensemble methods is increasingly desir- able due to the widening deployment multi-modal black-box predictive models. Conformal prediction is approach that avoids making strong distributional assumptions. Methods for conformal aggregation have been proposed for ensem- bled prediction, where the prediction regions of individual models are merged retain coverage guarantees while minimizing conservatism. Merging the prediction regions directly, however, miss on opportunities further reduce conser- vatism by exploiting structures present in the conformal scores. We, therefore, propose novel framework that extends the standard scalar formulation of score function to multivariate score that produces more efficient prediction regions. We then demonstrate that such framework can efficiently leveraged in both classi- fication and predict-then then-optinnzee regression settings downstream and empirically show the advantage over alternate conformal aggregation methods.
Introduction
Ensemble methods reduce variance or improve predictive accuracy []] Such methods are increasingly being coupled
oft-used class of statistical modeling techniques due their ability
with complex, black-box models, such in multi-modal language models 4 6 g) Couplings of
this sort are seeing ever-widening deployment in safety-criticll settings, such as medicine (
and robotics
Increasing interest is, therefore, now being placed on quantifying uncertainty for such models 16 17 18 19] Towards this end, methods of uncertainty quantification have arisen, such deep ensembles and committee estimation 220 21 22]. Such methods, however, sacrifice generality with the imposition of distributional assumptions, motivating the need for distribution-fie uncertainty quantification for ensemble methods.
One method for performing distribution-fiee uncertainty quantification is conformal prediction, which provides a principled framework for producing distribution-free prediction regions with marginal frequentist coverage guarantees 24] 24
By using conformal prediction on user-defined score function, prediction regions attain marginal coverage guarantees. While calibration is guaranteed from this procedure, predictive efficiency, i.e., the size of the resulting prediction regions, can be large for poorly chosen score functions.
As a result, methods have arisen to perform conformal model aggregation, which both provide uncertainty estimates of the ensembled predictions and do so in ways as minimize the prediction region size [2 26 27 28 29). While such approaches succeed in reducing the prediction region
Denotes alphabetic ordering indicating equal contributions.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

8 {(5) (24), s2(x;)))
II um (S)
Q n -= m=1
H H(u Umm 7m)
Figure 1: CSA provides a principled extension to the standard conformal prediction pipeline by leveraging ideas from nigher-dimensionss quantile regression to define quantile envelopes Q instead of scalar quantiles q. It does so by evaluating a collection of score functions (here 8) and $2) over the calibration dataset to define S. finding quantiles {îm} over set of projection directions {um J. and taking Q to be the intersection the resulting half-planes H (um: T.)) These quantile envelopes result in more informative prediction regions that can be used in downstream tasks.
size over naive aggregation, they all aggregate the separately conformalized prediction regions of the predictors in the ensemble. In doing so, they forgo the possibility of automatically leveraging shared structure amongst the scores of the individual predictors, resulting in conservative prediction regions.
We instead propose to perform aggregation in score space by extending traditional conformal prediction to consider multivariate score function and defining prediction regions using "quantile envelopes" place of scalar quantiles. Doing so enables efficient, data-driven, automated conformal model aggreegation. We demonstrate that this formulation retains the desired distribution-fii coverage guarantees typical of standard conformal prediction and that the resulting prediction regions can be used efficiently in both classification and regression settings. Our contributions
Providing multivariate extension conformal prediction, dubbed "conformal score ag- gregation" (CSA), that leverages quantile envelopes to enable data-driven, informative uncertainty estimation for model ensembles while retaining coverage guarantees.
Demonstrating how the prediction regions resulting from CSA can be efficiently leveraged in downstream predict-then-optimiz regression tasks.
Demonstrating the empirical improvement of the CSA framework over alternate conformal aggregation strategies across classification and regression settings.
Background
2.1 Conformal Prediction
Coverage guarantees of uncertainty quantification methods generally rely distributional assump- tions, often via asymptotics or explicit specification. To alleviate the need for such restrictive assumptions, interest in finite-sample, distribution-free uncertainty quantification methods has risen. Conformal prediction is one such method 23
Conformal prediction serves as wrapper around such predictors, producing prediction regions C(x) that have formal guarantees of the form Px, (Y & C(X)) a for some prespecified level a. To achieve this, "split conformal" partitions the dataset calibration set Dc. The former serves as the data used to fit f. Users of conformal prediction must then design "score function" sxx, y). which should quantify "test error", often in domain-specific
{ 3:) Wi) into training set Dy and
manner. For instance, a simple score function for regression setting would be s(x, y) [lf(x)) vll-
This score function is then evaluated across the calibration set to define Sc = {s(z. v) (x. E De).
For a desired coverage of a. we then take : to be the [[Decl + 1)(1 - a) a)]/|Pel quantile of Sc. with which prediction regions for future test queries can be defined as C(xr {v s(x, v) < T
Under the exchangeabilityy of the score of a test point s(X', Y') with Sc, we have the desired
fimite-sample probabilistic guarantee that - Px C(X)).

---

While this guarantee holds for any s(z,y), y), the informativeness of the resulting prediction regions, quantified as the inverse expected Lebesgue measure across X, i.c. (E]C(C(X)))) is intimately tied to its specification [24]. Thus, much of the challenge of conformal prediction relates to choosing score function that retains coverage while minimizing region size.
2.2 Quantile Envelopes
Generalizations of quantiles have long history in statistics 30 31] Unlike univariate data, mul- tivariate data do not lend itself to an unambiguous definition of a quantile, as there is no canon- ical ordering in higher dimensional spaces. The notion of "directional quantile" for random variable X E R can, however, be directly defined given some direction u E S namely as Q(X,a, a, u) inf E R P((u' X S 4) ax) [32
When there is ambiguity, we just denote as Q(a, u). For any given u, notice the choice of quantile defines a corresponding halfplane
33 34
(u, ((a, u)) = {ir E X uuu x S Q(a, w)}. The quantile envelope is then the intersection thereof:
D(a) n (u, Q(a, wESt-1
Notably, while each individual (u, Q(, u)) captures 1 of the points, D(a)) does not, as it is the intersection thereof and hence captures < a of the mass. If combined coverage is sought, correction, such Bonferroni adjustment, is used for the individual planes.
2.3 Predict-Then-Optinie
In the case of classification, conformal prediction regions simply constitute a subset of the label space, making their direct use by end users straightforward [35]. In high- diggh-dmenssinal regression settings, however, prediction regions become harder to directly; for this reason, recent works have started shifting focus to using them in their implicit forms.
One such application is 36 where conformal prediction was leveraged in redict-then-optimize setting. As the name suggests, predict-ther predict-then-optimg then-optimize problems are two-stage problems, which take observed contextual information and predict the parametric specification of downstream problem of interest 2 : g(x) with some trained predictor g. The final result is then a decision made with this specification, w min,. [(w, Z). An example of such setting is if an optimal labor allocation w" sought based on predicted demand from transactions æ in a delivery platform.
While the predicted ē is often trusted, this approach is inappropriate in riss-sensitive settings, where misspecification of the map g : X + C could lead suboptimal decision-making, For this reason, recent interest has been placed on studying a "robust" formulation [37 38 33]. Following this line of work, (36) proposed studying w" (ar) := min, maXzeC(s) f (w, 2), with C(x) being produced by conformalizing the predictor g.
2.4 Related Works
Ensemble methods consist of K predictors f& X, - Y: notably, such predictors need not map from the same set of covariates. naive approach for uncertainty quantification would then be to conformalize the ensembled predictor. That is, for an ensembling algorithm F y - Y, a score function a(F(J (x), ..., JK (z)), y) would be defined. Denoting the [[Ne + 1)(1 a/)/Ne quantile of the score distribution over Dc as T()), C(x) {u= s(x, v) & {(a}} would then be calibrated.
Such an approach, however, lacks some desirable properties. In particular, prediction regions C(x) should have the quality that, if a particular predictor has less uncertainty in its predictions, as is frequently true of ensemble settings where the predictors span multiple input data modalities, upon routing to that predictor, the corresponding size of the prediction region should be smaller than if it had been routed to a different predictor. While the naive approach does, in principle, support this property, it ultimately relies on defining uncertainty- typical form, however, F simply takes point predictions fi (x:), ..., J in as input, meaning any aawerees would need to be baked in priori into the definition of J through domain knowledge of the uncertainties of the predictors f1,..., JK, which can seldom be specified precisely, sacrificing the predictive efficiency of C(x).
ensembling algorithm F.. In its
3

---

Conformal model aggregation, thus, seeks to mitigate these deficiencies by aggregating the prediction
regions C (z),..., (x(x) rather than the individual point predictions [25 27
28 29 While there are
several methods in this vein, they can be categorized into one of two general approaches. The first line of work seeks to perform model selection, in which a single conformal predictor selected Ca-,
typically based on the criterion of minimizing region size k* == arg ming IE [C[ (X)) (27 28
Generally, however, methods leveraging the full collection of predictors produce less conservative regions 22 29]. Such works aggregate the individual prediction regions into a final region by defining
C(x) = {y Σ> == [y E (x)x 2 ) for weights {wx} E [0. 1] such that Σk= W/ = 1 and a threshold a. Methods then differ in the procedure by which {ure) } and ā are prescribed, several of which were prescribed by (29]. whose detailed presentation is deferred to Appendix for space reasons. We note that the methods of [25] are designed for a different setting than that considered herein, namely that in which conformal coverage is sought adaptively over data streams.
In this vein, [40] have recently proposed a vector- extension as that discussed herein, in which candidate weight vectors { {wmm J E R are searched over for score aggregation. That is, vector s(x) := (s) (z. .)................... Sx (x(2, y)) E R of scores (I. y) corresponding to each predictor (x) is predicted and its aggregate prediction region defined on the projection (wmm=, 8) for Wm- the weight resulting in the smallest prediction region. This method, however, has two shortcomings addressed herein. The first is that their method can only be applied in classification settings, whereas our method can be leveraged across both regression and classification problems. The second is that their approach only uses a single weighted projection in the end, resulting in suboptimal aggregation and, therefore, conservative prediction regions.
3 Method
3.1 Multivariate Score Quantile
We consider the setting typical of conformal model aggregation, as discussed in Section 2.4 in which predictors fi ( fK(x) and corresponding scores 81 (1 y); ..., 8K (1r, y) are defined. We assume similar premise [40], in which the scores are stacked into multivariate score s(z, y) == (81 (z, y),... S (z, v)). A naive approach would then leverage standard conformal prediction over a pre-defined map g RK + R. e.g., g(s) = Ek= k=j Sk. Similar to the naive conformalization of an ensembled predictor discussed in Section 2.4
using a fixed g fails adapt to any disparities in
uncertainties present across predictors or requires intimate knowledge of such uncertainties. We instead wish to provide a aata-vdaptive pipeline to automatically produce such a g.
Importantly, we hereafter assume the score functions are non-negative, i.e., 8k : : x y +R_, which typically the case the score serves as a generalization of the residual. We highlight that many of the details of the method presented below are geometrie in nature and are more easily understood with the supplement of diagrams. We have, thus, provided an accompanying visual walkthrough of the procedure in Appendix A to clarify its presentation.
3.1.1 Score Partial Ordering
Intuitively, our method seeks to directly generalize the approach of split conformal, by "ordering" the collection of multivariate calibration scores and taking the a score under such an ordering to be a threshold @ with which prediction regions are then implicitly defined. Formally, the multivariate "ordering" established as pre-ordering & over KR pre-ordering differs from a total ordering in that it need not satisfy the antisymmetric axiom of a total ordering. Roughly speaking, an "acceptance region," " so called as it serves as the criterion used to ultimately decide which y are accepted into the prediction region, is then defined as Q == {s s S 9) where q is the - a empirical quantile of Sc under Z. Such a Q naturally generalizes the standard scalar acceptance interval of [0, q in the case of neg-tegat score functions. We briefly highlight the distinction between acceptance regions and prediction regions. The former are subsets of the space C IR. of multivariate scores that ultimately define the criteria for retaining particular y values in the prediction region. The latter are the subsets of the output space )' and it is these that ultimately have coverage guarantees. The two, however, are directly related; particular, for fixed score a(x, y). a larger acceptance region will result in a more conservative prediction region.

---

Crucially, therefore, the problem of choosing this pre-ordering closely parallels that of choosing 9, where a poorly chosen pre-ordering will result in overly large acceptance regions and, hence, conservative prediction regions. For instance, using a lexicographical ordering Stex will result in axis-aligned hyper-rectangular acceptance regions. As a result, rather than manually prescribing a pre-ordering, we define S in data-driven fashion by prescribing an indexed family of nested sets {Ac hER, such that As, C At for 1 S t2 and stating 81 X 52 if Vt, $2 E A. = $1 E A..
For family of sets {At (ERR we take each Ar to be the region of the positive orthant R bounded by the coordinate axes and an "outer frontier" parameterized by t. The shape of this outer frontier remains fixed over the family and is merely scaled outward from the origin with t. Under this choice, comparing $1 $2 E I. K i.e., checking if $1 2 $2, amounts to checking if ts1) S t(s2). where t(s) is the smallest t for which the outer frontier of A, intersects s. Notably, t(s) is precisely the aforementioned data-driven score fusion function g(s) of interest. Defining a data-adaptive g(s), therefore, reduces to having a data-driven approach for defining the outer frontier of At. We restrict this outer frontier to be such that A a convex set; if A, were permitted to nonconvex, computing t(s) == min(t E R: : } would potentially be computationally expensive. The benefits of such convexity are highlighted, for example, in Section 3.2
To have tight acceptance regions, we formally wish for the pre-ordering to have the property that the acceptance region given by Q has minimal Lebesgue measure and captures 1 points of Sc. The problem of discovering an optimal pre-ordering can, thus, be equivalently stated as secking to define the outer frontier of A, to match that of the tightest 1 - convex cover of Sc.
This motivates selecting the outer frontier to be the 1 a quantile envelope of Sc. Using Se to define A, and in turn S. however, sacrifices the exchangeability of its points with test scores s', as the very nature of ordering would change in swapping s' with any s E Sc. The goal follows as seeking to define the outer frontier as the 1 a quantile envelope of Sc without directly using Sc. For this
reason, we partition Sc s U US? where we define X using St and compute q over s(?) Such split is predicated on the assumption that the - quantile envelope defined over se resembles that of s(e implying the ((() should be sufficiently large as to capture this structure accurately.
We now focus attention on defining the quantile envelope over o using a technique paralleling that described in Section 2.2 In particular, we start by selecting the projection directions {um) of Equation since 8 E _ we similarly restrict Um E S == SK n RK To best approximate Equation we wish for {um to be uniformly distributed over S however, exactly finding an evenly distributed set of points over hyperspheres in arbitrary n-dimensional spaces is classically
difficult problem [4]] If K = 2, we can solve this exactly; for K > 2, we generate directions stochastically such that U ~ Unif(S -1) by drawing Vi..., Vas ~ (O, {K*K) and defining
U, = / + + Vir. where ol denotes the component wise absolute values.
We now wish to define the quantile thresholds } J for the selected directions to optimally capture 1 a-a of S() Naively taking the 1 quantile per projection direction um results in joint coverage by ē= - m= H(um; Tm) of s. to be < - a. A straightforward fix is to replace the 1 quantile a/M quantile. While valid, this approach
per direction instead with its Bonferroni-corrected produces overly conservative prediction regions. We, therefore, instead tune separate / E (a/M, a) parameter via binary search, finding the maximum 3* such that using the 3 quantile per direction provides the overall desired coverage, i.e., 1 Im= H( Im (1- (1-3")1) ()sss for some fixed, small e > O. With this choice of ((um, Tm) we have a defined pre-ordering, whose coverage guarantees are formally stated below and proven in Appendix B
E (1-a, 1-a+t)
(X', are exchangeable, where De == {(X,, Ne, Assume further that K non-negative maps : X x y + 14, have been defined and a composite s(X,Y)= == Y)., (X, (XYY Y)) is defined.
Theorem 3.1. Suppose
Let a = (oi,..., ,NN)) be a random permutation of the indices and independently of De and (X)Y"). Let the calibration set 'De be partitioned into D +Ne,
drawn uniformly
{(Xo,, 1o,)1 No; and D(2) == ((Xo, Yo, ) I=NNo,+ +1, where Nc == Nc, + Nc,, Let the corre-
5

---

sponding score sets be Se and S Let (1:5)) IR4 - R be a deterministic function for any given realization of s(3)
For some Or E (0, 1), let be the [(Nc, -1)(1-a)]- smallest value of the set of transformed scores (t)
S Assume that ties among the transformed scores occur with probability zero. Then, denoting by C(X') = ty E y T(s(X',)) S) s(1) S 03, P(Y' E E((X)) 2 1 1--0 a. where the probability is defined over the joint draw of the data Dc, (X'Y"), and the permulation
3.1.2 Score Quantile Threshold
To then compute 7. we find (*) for each s E S(2) defined to be min(l E R
Am-1 (um, t77m). )}. This can be efficiently computed as t" (s) = max I.... ( Denot- ing the [(Ne, + 1) a:]]-th largest (*))) as I, Tm == Mm and @ = Nm=1 H(um, Tm) the tightest quantile envelope was already discovered over Sc this adjustment factor 1 R 1. Critically, such calculations can be computed efficiently in vector form. Due to space restrictions, we defer this
discussion to Appendix [E] We additionally there empirically validate the efficiency of the procedure under this vectorized implementation. We present the full algorithm in Algorithm [
Importantly, while this procedure will result in convex regions C, this does not mean the downstream prediction regions in y will be convex, discussed in Section 3.2 However, it is unsurprising such flexibility exists, as even a single scalar score 81 (z, y) can produce nonconvex prediction regions. One additional notable property of the CSA prediction regions is that their sizes vary across Z even if such variability is not baked into the constituent scores. For instance, using $3 (z. y) == (f( y with standard, scalar conformal prediction yields intervals of length 27j for any x, yet (COSA (a:) even with such (#x (2. y)} will vary with æ. This variability is desirable, as predictive uncertainty is seldom uniform across the covariate space. See Appendix F for full illustration of this.
Algorithm 1 CSA: UNIFHYPERSPPHERE(K (K) is an assumed subroutine that samples ~ Unif(S*-1)
1: Inputs: Score functions 81, 8K X -> y, Calibration set De, Desired coverage 1
2: [Stnn Sui] < [a/M, a), ē 0
3: a (nif(Peertttttions of [1....,Nc))
4: Se (S) + ((5x( Vo(e) )) -= {um < UNIFHYPPERSPHEEEE((K) m=
No, No,
5: while (s(0) @/Ne,, $ 1 e
6:
B + (B10 Sni)/2
M Im + (1 - 93) empirical quantile of (u (,5 sm
7:
8:
e- M
--m=1 H(um Fm)
9:
if (s n @l/Ne, 1 then 310 3 else Bai +- 8
10: end while
11: Z t- (1 - a) empirical quantile of (maxme[M] (u (T 82/ (m 2
Return {um, {4m))]m m=
Notably, this algorithm achieves the aforementioned coverage guarantee as direct corollary of Theorem 3.1 stated below and proven in Appendix Intuitively, the proof proceeds by demonstrating that the 1 scoring function defined implicitly by Algorithm [ satisfies those conditions posited in Theorem from which the posited coverage immediately follows.
Corollary 3.2. Let De. (X',Y") Y'). {sk) and a be as defined in Theorem 3.1 Let a, (S ((2)
and U = {um L
mmm1 be as defined by lines 3-4 of the call CSA((sk) De, a:) of Algorithm
Denote by {gm mm=1 the parameters defined by lines 4-9 of Algorithm and by T. the scoring
function T(5; 5(1) (1) U)= = maxm=1.... M....M (UTT s/Qm) for any score vector & E TR Then, denoting
by C(X') = {y E y T(x(X', -y:;
(ils(("));: Se) E DJ, P(Y' E C(X)) 2 at where the probability is defined over the joint draw of the data De. (X', Y'), and the permutation

---

3.2 Predict-Then-Optimizz
With this generalization of the score function, natural question is how leverage the result- ing prediction regions C(x). For both classification and regression, C(z) = Nm= Cm (x) where (z) == {y (x, y) S }. For classification, where | E N. explicit construction of C(x) is straightforward: for any z, explicitly constructing C(x) can be done by iterating through y ) and checking if s(x, y) Q by comparing s(x, y) against each one of the thresholds 7m after projection.
In the case of regression, however, the prediction region cannot be explicitly constructed in the general case, since Y' contains uncountably many elements. In fact, explicit construction is generally not of interest for downstream regression applications. We, therefore, focus on one particular application, namely that of (36) discussed in Section 2.3] and demonstrate the CSA prediction regions can be leveraged in their framework for problems studied therein. For instance, the authors demonstrated the utility of their method in robust traffic routing setting with being predicted traffic from probabilistic weather model gC X)) for weather covariates X. An ensembling approach emerges with multiple predictive models, such as (2 X) predicting traffic based on historical trends.
we note that the below described algorithm is better understood with a visual accom- paniment, which provide in Appendix D [36] demonstrated that solving the robust problem variant " (x) == min... maxaeC(a) fw, C) in a computationally efficient manner is feasible by performing gradient-based optimization on w, where the gradient V Vw (w) of <(w) == maX fww, 2) can be computed by leveraging Danskin's is Theorem so long as max fww 2) is efficiently com- putable for any fixed w. We focus on demonstrating that this remains the case for CSA, specifically considering the case where individual view score functions take the form of the "GPCP" score considered therein. In this setup, each constituent predictor is generative model qs(C' X) from which {Gk 1/= (C' X) samples are drawn. Note that Jg need not be constant across k. The GPCP score, used to define the score components, is
As in Section 3.1
$k c) = min [| c||2]- jeL.....hk
(2)
Notably, this framework subsumes many standard regression settings, e.g., for deterministic predictor, one can take qu (C X) = (X)). To compute maxeec(s) f(w, C), we first let
i J = (j.,... be an indexing tuple, where each ja E Jg.. That is, each is a vector that "selects" sample per predictor. Notably then, the projection un S0z, c) is convex in c, since the projection directions are all restricted to sK-1 Thus,
of = arg max f(w, c)
s.t.
ue ((e 9m Vm E (1,..., M}
remains standard convex optimization problem. The final maximum can then be found by aggre- gation, namely c* arg maxje.7 f(w, ct). While |J] = II Jk. in certain cases of ensemble prediction, such as multi-view prediction, there tend to be limited number of predictors in practice, typically K = or K 3. This coupled with the trivial parallelizability of computing over indices means this approach is still computationally tractable. The full procedure is outlined in Algorithm 2
Algorithm 2 Thedict-Then--Opiimiz Under CSA
1: Inputs: Context Predictors (C) x)){= Optimization steps T, Sample counts {Ju)
CSA quantile {(mm, 4m, } M
2: (e ~ (C | x) =I
[JA]
K
3: ())) U(W)
4: for E (1,..... do
5: for j E I do of t arg max, f(ol!),))
s.t. Vm E 1,...,M M
uJ sT c) S am
6: c arg max,; f(w(l),c()
7: ((() (w( 11 v/ (10((1)) ((())))))))
8: end for
9: Return (T)

---

Table I: Classification results are shown across tasks for a 0.10, 0.05, and 0.01, with coverages in the top (grey) and average prediction set sizes (white) in the bottom of each row. Both assessed over a batch of i..dd test samples (15% of the validation set from ImageNet). Standard deviations and means were computed across 10 randomized draws of the calibration and test sets.
Dotasetio
Denoeeet 0,902 00 (EE)))
IrsageNct
0.003 (8,05))
0:502 (0.00))
0,899 (0004)
0,558 (LDK))
0L09 00.04:
0,299 e0 054)
0.9 (9003)
[sr 0.100
137 (L95)
156 12 396
120/% (2.27)
66 1885)
$1.337 (1.50)
82. 1921
131,856 (2.57)
60.12 (1.317)
34 (0.24))
220022 (2002)
229 (3.07)
205,59 (2.018)
78.109 (2004)
166.233 (2.57)
141.223 (2932)
220.491 (23)))
112.8) (2.115)
99,574 (2.32)
[sr 0.01)
ILS0 (0,001)
0:591 (0,00))
0.980 01 (92))
0.99 (0.0011
0.997 (00001)
1.001 (0.002
I.00 0000
0.59 (0000)
10:99 (0) (917)
(7.11)
(700)
4 Experiments
We now study CSA empirically across several tasks, demonstrating its coverage guarantees with reduced conservatism. We demonstrate improvements in ImageNet classification task in Sec- tion 4.1 across real-data regression benchmark tasks in Section 4.2 as proposed by [42). and in a downstream predict-then-optimize task in Section 4.3 We additionally assess the robustness of CSA
to imbalanced ensembles and perform an ablation study of the two-stage calibration.
We note that the predictors and calibration and test sets were fixed across choices of calibration procedure for each experiment, meaning care had to be taken in partitioning Dc Dc U De for CSA, where insufficiently large (C would result in poor estimation of the a-quantile envelope and hence require large adjustment 7 factor and an insufficiently large De in the classical reduced predictive efficiency from conformal prediction. We note the splits in each of the sections that follow.
We compare against the methods presented in Section 2.4 viz. the model selection of [27]. the aggregation methods of [25). and the single weighted score projection (VFCP) of [40] We addi- tionally include the initial strategy discussed in Section 2.4 in which the ensemble predictor is directly conformalized, using a natural aggregate "ensemble" score, given in the following sec- tions. From the work of (25). consider the following methods: the standard majority-vor CMM, partially randomized thresholding CR, and fully randomized thresholding CV approaches (see Ap- pendix N Notably, these methods do not lend themselves for use in the tredict-then-optimiie setting, so we eliminate them from consideration therein. VFCP can only be applied in classifi- cation settings; we, thus, do not compare it across the regression tasks. Code is available at https: //tp::////tttutt http:://ggtabbb.o//appptttoooo/ ccnm//raapaaee1140//ffssioooon
4.1 Classification Tasks
We first study the predictive efficiency of the aforementioned methods on the ImageNet classification task [43]. In particular, an ensemble was constructed from three separately trained deep learning architectures, namely ResNet-50, VGG-11, and DenseNet-121. Conformalization on the individual models was performed using the standard classification score function across all approaches, namely s(x,y) = E/=1 [(x))x(()) where (+) and (<x) is the permutation of (1,..., [ that sorts I (x) from most to least likely. Here, the "Ensemble" score was computed with the same s(1, u). replacing (x) with the ensemble average probability, i.e. u(=), = [=(J<(=)))/ Calibration was performed using 85% of the ImageNet test set and assessment of the coverage and interval lengths on the remaining 15%, with 10 trials conducted over randomized draws of these calibration and test sets. 25/75% split was used for (1)_ (2), The results are presented Table | the full results across additional a is given in Appendix G We see that all the approaches exhibit the desired coverages across a. However, CSA consistently produces significantly smaller prediction regions than both the individually conformalized models and alternate aggregation strategies.
4.2 Regression Tasks
We now similarly study the predictive efficiency of CSA across suite of regression tasks from [42 The data for each task were split with 50/45/5% for training, calibration, and testing for coverage and interval lengths, with five trials conducted over randomized selections of such sets. A 5/95% split was used for De (2) methods were ensembled, namely an OLS model, LASSO linear model, random forest (RF),
(2), The problem setup was replicated from (2), which four prediction
8

---

Table 2: The results for five distinct tasks shown below for = 0.05 (top five rows) and a = 0.025 (bottom five rows). For each, the average coverages (grey rows) and prediction set lengths (white rows) with standard deviations are given, both assessed over 5 randomized draws of the training. calibration, and test sets. In cases where the method failed to achieve sufficient coverage (i.e. .93 for a 0.05 and < 0.96 for a 0.025), we do not include it in comparison for set length.
Datandii
0LS
LASSO
RV
XtiBoost
2"
Enscradee
Atngle-Sere
CSA
350234
0
00554
0:35
(o. (115)
4675:0 140
6.05 (3.5))
10 10:00:00 1806
V.357 01C57)
1196-00 01 1235
700 05 1166
2.50
(5.156
17 757 (1 3795
7ash (9871)
TASK (U.0T)
0.06)
00:55
3- 34141 00.246
10.86600558 (8.
367 (9-53)
7.282 (0/57))
16:248 (0. 11C
15.333 (1 3730
760 (9-450)
1 1 (5,300
(.06)
(1:2330
1J80s (8.6622)
6.304 (1 32x3)
4.554 (L.00-3)
1064 (LIMN-)
464 (8.12.2)
9.1644 (2.45.5)
5,10N (2.49c3)
8 3.34- (1.25-3)
351337
(8.0)
(115
27 055651
46534
S1.65
7093
23 1996
353241
0.506 (33001)
0087 (D003))
(N0031
0L923 M0L50
0.852/0.0
HGU0001
(0.0211
359234
0.99 13UMAY
09977 (U955))
MS7 (8308)
(5905 (800))
RUPTS BUDOMI
(n - (.925)
11.99 60. 31)
11.87 00 08.44
<5 166.
972 (3,09)
1.59 (0L1
11.59 (0/96))
11:65 00050:
25 974)
0.2599
8.855 (M.55)
0966
B:58 (HOON)
0.97600
DIRR (1-00))
Iu-m (U123)
24.905
(0.225)
26. 641 (1-125)
11.801
(0.922)
257 (3.750)
14472 (0L722)
12.278 (UXON)
19223 00356:
7.715 (2.467)
12.565 (L.66)
369336
[n = (U125)
4.9964 (4.0440
1.80a (5.4
L.50ll (T.5
Thee (8 31
S766 (1 CLANGS
(2 (7.0064)
3,29% (3.31m0)
3 3.eh (1 (1.05-35)
353221
(n 0015)
47.73 5433
140.0 096)
93 (3 0077
56.306 (1,87))
30 954 (D 901
37.122 (USKET)
67 660 1666
18.34 (8. (TG
26807 (U.11))
(023)
21,722 (0:35)
23.699.0 vO.
21 11.43-20 (3.00) D00)
21,454 (3,109)
26,902 (8V95))
19,291 (8/00))
21.905 (0.411
40 083 045)
17,763 (L.29)
J98077 (LU02)
and an XGBoost model. A residual function was used as the score across all methods, namely s(x, v) = [f(x) - y/- Here, the "Ensemble" score was the standard s(x, y) : I where (u(#), (x)) are the ensemble mean and standard deviation. Prediction intervals could be analytically constructed for the ,h,,C*, and CU methods. To assess CSA, however, a discretized grid Gy C y of coarseness Ay was considered, and an interval length estimate given by [(((z)) & Ay Ky y E We also present an ablation, labeled "Single-Stage," to demonstrate the two-stage calibration is necessary to retain coverage; this single-stage approach does not split Sc and instead directly computes {âm) on Sc per Section 3.1.1 For CSA, M = 1000 was used. Intuitively, as M 00, we would expect to recover the 1 - tightest cover and, thus, that the prediction region size should be roughly decreasing in M, with some plateau. This is explicitly shown in Appendix [
a(xr)
gy, <(., y) E HH
We provide the results for a 0.05 and a 0.025 to demonstrate the consistency of the method performance. A subset of the results is given in Table the full set of results is deferred to Appendix[H] As in the results of Section 4.1 we see that CSA retains the coverage guarantees typical of conformal prediction yet produces significantly smaller prediction intervals than both the individual models and the alternate aggregation strategies. We additionally see that the "Single-Stage" approach fails to retain coverage, demonstrating the necessity of the two-stage calibration. We provide a visual comparison of the prediction regions resulting from these methods in Appendix
We additionally assessed the robustness of our method imbalanced ensembles. The experiments of 29 were conducted on a UCI benchmark task 44 with an ensemble of an OLS model, LASSO linear model, a random forest, and an MLP, and they found the conformalized random forest to outperform all the proposed aggregation strategies, due to the lack of orthogonal information in considering the other predictors. We find that, in these degenerate cases, where the best decision is to simply choose a single predictor, our method outperforms other aggregation methods and nearly matches the performance of the best conformalized predictor in hindset; the results are presented in Appendix K across number of UCI benchmarks.
4.3 CSA Predict- Predict-Then-Oppiee Then-Optimize
We now study a real-world prdicc-then-optimizz traffic routing task, from (36). In this task, a time series of T' preceding precipitations is used to predict future precipitations and, in turn, future traffic, as fully described in Appendix []] We consider the traffic routing problem for fixed source-target pair (s, t) over the graph of Manhattan, where | = 4584 and | = 9867. Formally,
w= (x: min max 2 w s.t. w E [0, 1]s Au P.Px.cCC C(X)) 2 a w ZcC(2)
where x E Hx are the previous precipitation readings, We E R the traffic proportion routed along road e, c e 1l the transit times anticipated across roads, A E | the graph incidence
9

---

matrix, and b E R the vector that specifies the routing problem, in which b, 1, -1, and by for the travel source node, the terminal node, and k & {s,t) all other nodes.
We then consider two probabilistic models for traffic prediction, namely one based on the classical probabilistic Lagrangian integro-difference approach (STEPS) of [55 and
the modern latent diffusion model (LDM) approach of [46]. As a result of the higher inference cost of the latter, we consider the setup where > J2. specifically with J1 4 and J2 flexibility of non-uniform sampling from predictors discussed in Section 3.2 As discussed in Section
highlighting the
4 the alternate aggregation strategies do not lend themselves for use this setting. We, therefore. only compare CSA to the separate conformalizations of the two predictors, with the score from Equation (2). We here evaluate the methods using the expected suboptimality gap proportion, A% Ex (X, / min,.. fw, C(X))], where A is defined as discussed in Section 2.3 This measures the conservatism of the robust optimal value and bounded [0.
Experiments were conducted with [Del 200, with 20/80% split used for "D The suboptimality was then computed across 100 i.i.d. test samples. To assess the improvement, we (CSA)
A (STERS) and H1, A
(CSA) < (STErS) and %
conducted two paired t-tests, where Ho A%
similarly for A (CSA) and A (LDM) The results provided in Table from which find that CSA significantly reduces the suboptimality after accounting for Bonferrom multiple testing. We see that, while conformalization of either of the two views individually already produces the desired coverage, CSA produces more informative prediction regions, and hence less conservative robust upper bounds.
Table Coverages for a = 0.05 for the individually conformalized and CSA approach and p-values of the paired t-tests comparing Ax are shown, both computed over 100 i.i.d. test samples.
Coverage
P-values for Hi
STEPS
LDM
CSA
(CSA)
A (STEPS), 3.61 x 10
(OSA) (LDM) 9.50 x 10
0.981
0.962
0.968
Ang
5 Discussion
We have presented a framework for producing informative prediction regions in ensemble predictor pipelines, suggesting many directions for extension. One is in extracting insights of the relative predictor uncertainties from the data-driven relation S. Another is the integration of CSA with [47), which proposed end-to-end extension to (36). Such end-to-end integration may discover more optimal vector comparisons than the quantile envelope partial ordering approach proposed herein. sacrificing statistical efficiency, which may be infeasible in data-sparse regimes: developing a method analogous to full conformal, which forgoes computational efficiency for statistical efficiency, would be another valuable direction for extension. Finally, given the prevalence of sensor fusion in robotics, another avenue is to study the of CSA in robust control. This would extend recent works that have leveraged conformal prediction for robust linear control 48
Additionally, in requiring the data to be split to retain coverage.
Acknowledgements
We acknowledge the support of the College of LSA at the University of Michigan via the "Meet the Moment" initiative.
10

---

References
[1] Robert E Schapire et al. brief introduction to boosting. In Ijeai, volume 99, pages 1401-1406. Citeseer, 1999.
[2] Cha Zhang and Yunqian Ma. Ensemble machine learning, volume 144. Springer, 2012.
[3] Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pages 1-15. Springer, 2000.
[4] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[5] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning. pages 9748-8763. PMLR, 2021.
[6] Shiliang Sun. A survey of multi-view machine learning. Neural computing and applications. 23:2031-2038,, 2013.
[7] Jing Zhao, Xijiong Xie, Xin Xu, and Shiliang Sun. Multi-view learning overview: Recent progress and new challenges. Information Fusion, 38:43-54, 2017.
[8] Xiaoqiang Yan, Shizhe Hu, Yiqiao Mao, Yangdong Ye, and Hui Yu. Deep mueti-view learning methods: review. Neurocomputing, 448:106-12 129, 2021.
[9] Ye Yuan, Guangxu Xun, Kebin Jia, and Aidong Zhang. A multi-view deep learning framework for eeg seizure detection. IEEE journal of biomedical and health informatics, 23(1):83-94, 2018.
[10] Yifeng Li, Xiang Wu, and Alioune Ngom. A review machine learning principles for multi-view biological data integration. Briefings in bioinformatics, (9(2):355-340, 2018.
[11] Ye Yuan, Guangxu Xun, Kebin Jía, and Aidong Zhang. A multi-view deep learning method for epileptic seizure detection using short-time fourier transform. Proceedings of the 8th ACM international conference bioinformatics, computational biology, and health informatics, pages 213-222, 2017.
[12] Ramon F Brena, Antonio A Aguileta, Luis A Trejo, Erik Molino-Minero-Re, and Oscar Mayora. Choosing the best sensor fusion method: machine-learning approach. Sensors, 20(8):3350, 2020.
[13] Erik Blasch, Tien Pham, Chee- Yee Chong, Wolfgang Koch, Henry Leung, Dave Braines, and Tarek Abdelzaher. Machine learning/artificaal intelligence for sensor data fusion opportunities and challenges. IEEE Aerospace and Electronic Systems Magazine, 36(7):60-9)) 930, 2021.
[14] Mary B Alatise and Gerhard P Hancke. A review on challenges of autonomous mobile robot and sensor fusion methods. IEEE Access, 8:39830-39846, 2020.
[15] Mahesh Subedar, Ranganath Krishnan, Paulo Lopez Meyer, Omesh Tickoo, and Jonathan Huang. Uncertainty awware audiovisual activity recognition using deep bayesian variational inference. In Proceedings of the IEEF/CVF international conference on computer vision, pages 6301-6310, 2019.
[16] Junjiao Tian, Wesley Cheung, Nathaniel Glaser, Yen-Cheng Liu, and Zsolt Kira. Uno: Uncertainty- aware noisy-or multimodal fusion for unanticipated input degradation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 57 17223... IEEE, 2020.
[17] John Denker and Yann LeCun. Transforming neural-net output levels to probability distributions. Advances in neural information processing systems, 3, 1990.

---

[18] Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew M Dai, and Dustin Tran. Training independent subnetworks for robust prediction. arXiv preprint arXiv:2010006610, 2020.
[19] Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018.
[20] Rahul Rahaman et al. Uncertainty quantification and deep ensembles. Advances in neural information processing systems, 02220063-2075.2 2021.
[21] Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, et al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information fusion, 76:243-297, 2021.
[22] Jesús Carrete, Hadrián Montes-Campos, sen. Deep ensembles vs committees for uncertainty estimation in neural-network force fields: Comparison and application to active learning. The Journal of Chemical Physics, 158(20), 2023.
Campos, Ralf Wanzenböck, Esther Heid, and Georg KH Mad-
[23] Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
[24] Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008.
[25] Matteo Gasparin and Aaditya Ramdas. Conformal online model aggregation. arXiv preprint arXiv:2403.15522 2024.
[26] Vladimir Trunov and Vladimir V V'yugin. Online aggregation of conformal predictive systems. In Conformal and Probabilistic Prediction with Applications, pages 430-449. PMLR, 2023.
[27] Yachong Yang and Arun Kumar Kuchibhotla. Selection and aggregation of conformal prediction sets. Journal of the American Statistical Association, pages 1-13, 2024.
[28] VV V yugii and VG Trunov. Online aggregation of conformal forecasting systems. Journal of Communications Technology and Electronics, 68(Suppl 2):S239-S225 S253, 2023.
[29] Matteo Gasparin and Aaditya Ramdas. Merging uncertainty sets via majority vote. arXiv preprint arXiv:2401.09377 2024.
[30] Peter Rousseeuw and Anja Struyf. Computing location depth and regression depth in higher dimensions. Statistics and Computing, 8:193-203, 1998.
[31] Robert Serfingg Quantile functions for multivariate analysis: approaches and applications. Statistica Neerlandica, 56(2):214-232, 2002.
[32] Linglong Kong and Ivan Mizera. Quantile tomography: Using quantiles with multivariate data. Statistica Sinica, pages 1589-1610, 2012.
[33] Davy Paindaveine and Miroslav Šiman. On directional multiplc-output quantile regression. Journal of Multivariate Analysis, 102(2):193-21) 2011.
[34] Mare Hallin, Davy Paindaveine, and Marianna Šiman. Multivariate quantiles and multiple- output regression quantiles: From & optimization to halfspace depth. Annals of Statistics, 38:635-669, 2010.
[35] Jesse C Cresswell, Yi Sui, Bhargava Kumar, and Noël Vouitsis. Conformal prediction sets improve human decision making. arXiv preprint arXiv:2401. /3744, 2024.
[36] Yash Patel, Sahana Rayan, and Ambuj Tewari. Conformal contextual robust optimization. arXiv preprint arXiv:2310.10000 2023.
12

---

[37] Shunichi Ohmori. A predictive prescription using minimum volume k-nearest neighbor enclos- ing ellipsoid and robust optimization. Mathematics, 9(2):119, 2021.
[38] Abhilash Reddy Chenreddy, Nymisha Bandi, and Erick Delage. Data-driven conditional robust optimization. Advances in Neural Information Processing Systems, 35:9525-9537, 2022.
[39] Chunlin Sun, Linyu Liu, and Xiaocheng Li. Predict-then-calibrat:: A new perspective of robust contextual lp. arXiv preprint arXiv:2305.15696 2023.
[40] Rui Luo and Zhixin Zhou. Weighted aggregation of conformity scores for classification. arXiv preprint arXiv:2407.10230, 2024.
[41] Stefan Schnabel and Wolfhard Janke. simple algorithm for uniform sampling on the surface of hypersphere. arXiv preprint arXiv:2204. 140004044.4404, 2022.
[42] Sebastian Felix Fischer, Matthias Feurer, and Bernd Bischl. Openml-ctr23-a curated tabular regression benchmarking suite. In AutoML Conference 2023 (Workshop), 2023.
[43] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255 leee, 2009.
[44] Arthur Asuncion, David Newman, et al. Uci machine learning repository. 2007.
[45] Seppo Pulkkinen, Daniele Nerini, Andrés A Pérez Hortal, Carlos Gelasco-Forero Alan Seed, Urs Germann, and Loris Foresti. Pysteps: An open-source python library for probabilistic precipitation nowcasting (v1. O). Geoscientific Model Development, 21100)4185-444 2019.
[46] Zhihan Gao, Xingjian Shi, Boran Han, Hao Wang. Xiaoyong Jin, Danielle Maddix, Yi Zhu, Mu Li, and Yuyang Bernie Wang. Prediff: Precipitation nowcasting with latent diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[47] Abhilash Chenreddy and Erick Delage. End-to-end conditional robust optimization. arXiv preprint arXiv:2403.04670, 2024.
[48] Yash Patel, Sahana Rayan, and Ambuj Tewari. Conformal robust control of linear systems. arXiv preprint arXiv:2405.16250, 2024.
[49] Geoff Boeing. Modeling and analyzing urban networks and amenities with osmnx. 2024.
13

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpippi.ccppulli//uuuii wwwticccguie//eeeeeeeeeii CoodeSubeiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http::://neuriipee wwtph/iww......uu.epeeeeee httpp//ccccppp....tttttttcteeeee
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
18

---

Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
19

---

Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htteas////eerrpppoccc///o/rreers for what should or should not be described.
20

---

A CSA Visual Walkthrough
We walk through visual presentation of the approach below supplement the textual description in the main text. We start with a collection of multivariate calibration scores Sc with E S being RRK, For the purposes of visualization in this section, have K 2. We first partition the score evaluations Se = S US! with subset s()) used to define the pre-ordering and the remainder define the multivariate quantile.
Figure The calibration score evaluations are first split between those used define the pre-ordering (green) 5()) and those used to define the final multivariate quantile (red) S(2)
We first wish to define the pre-ordering over S!) As described in the main text, the goal is to define this using an indexed family of sets A, with index í IR, after which the multivariate quantile approach reduces to the univariate quantile formulation. To ensure the final envelope over 5(2) remains as tight as possible, we wish to define this family
data-driven fashion. Critically, the shape of this tightest envelope around s will vary across a, meaning we must define the family separately for each choice of a. We expect the contour of the tightest a envelope for S will be similar to that over S motivating such choice to define the indexing family. To do this, we project S along number of directions, finding the 3 quantile along each, in turn defining a half half- plane, where 3 is as described Section 3.1
Figure 3: The pre-ordering points are projected across a number of directions, after which the / quantile is used to define direction quantile. This defines half-plane of points that are in the region (blue) and those outside (red).
We then iteratively update 3 in the manner described in Algorithm 1 to obtain 0* namely the minimum value for which the region given by the intersection of the corresponding half salf-planes covers roughly 1 - of s())
21

---

We the intersection of hyperplanes to define the quantile envelope, seeking 9" that achieves the desired coverage.
Figure 4:
Once this 1 - a quantile envelope of s(1) is found, we define Ai to be such an envelope, with which future points can now be partially ordered. That is, for any point E RK notice that we can unambiguously associate it with t(s) min(t R Ae). Intuitively, this the t where the contour "intersects" 8. Notably, now that the partial ordering has been defined, the points of s(1) are longer used. It would be of interest investigate whether concurrent definition of the partial ordering and final calibration is possible without such data splitting in future work.
A1,0
Ao
N075
Figure 5: Using the quantile envelope, the family of nested sets A, is defined, in turn defining partial ordering over k
With this AG, we find the final q simply by mapping the points of S to their corresponding t(s) values in the aforementioned fashion and performing standard conformal prediction. As discussed, if the envelope a similar structure to that found over ( the envelope should be adjusted by only minor amount.
22

---

Figure 6: Using the nested family of sets, expand contract the envelope appropriately using the data of s(2) to find the final adjustment factor.
B Multivariate Score Coverage
The proof of the multivariate extension of conformal prediction follows in precisely the same manner as that of standard conformal prediction with the pre-order S replacing the complete ordering used in traditional conformal prediction.
Theorem B.1. Suppose Ki,,Y),.... (XNe YNe),(X', Y') are exchangeable, where De {(X,, YJ) j- Assume further that K non-negative maps sk X - R., have been defined and composite s(X, Y) := (x,(X)),.. sx (X, Y)) is defined.
Let (o1,...,ONG, be random permutation of the indices (1...., Nc), drawn uniformly and independently of De and (X', Y"). Let the calibration set De be partitioned into D(!) == {(Xo,: Y,,)1 Ne, and D(2) = {(Xo,, Yo, (,,)), Ne, +Noz
1o)NN, +1 where Nc Nc, Ncz- Let the corre- sponding score sets be SC and S Let (1:5()) R + R be a deterministic function for any given realization of s())
For some a E (0, 1), let be the [(Nc, + +1)(1 (1)---)]-tt smallest value of the set of transformed scores (T(:;: ( E ( Assume that ties among the transformed scores occur with probability
zero. Then, denoting by C(X') = ty E T(s(X', ,y); S( s(1) < i}, P(Y" E C((')) 2 1 1--0 ax where the probability is defined over the joint draw of the data Dc. (X,Y'), and the permutation
Proof. The overall probability is taken over the joint distribution of the exchangeable data, De and (X', Y'), and the independent random permutation, σ. We use the law of total probability by first conditioning on a specific realization of the permutation, = RT, and the data in the first split, D()) d Given π and DC d(1), the score set s( is fixed. As result, the function T(: becomes a fixed, deterministic transformation.
By the initial exchangeability of all data points, after conditioning the values of the first split D the remaining Nc, calibration points in D(2) and the test point (X', Y") are still an exchangeable sequence. Applying the fixed transformation T to their scores yields an exchangeable sequence of Nc, + 1 scalar values:
(T(s;: (X,, ( (T((((X,,))))
Under the no-ties assumption, the rank of the test value T(s(X', Y"); s within this sequence is uniformly distributed on NN, + 1}. The test point Y' is covered if its transformed score is
23

---

less than or equal to the threshold f. This occurs if and only if the rank of the test score is at most
m [[No, + +1 (((1-a), a)]. The probability of this event, conditional on π and D d()), is:
[[Ne, + 1)(1 - a)] Nc, +1 2 -
P(Y' E E((x) o - n, D =d
Since this guarantee holds for any realization (I,, ((1), the unconditional probability also holds by the law of total probability:
P(Y' E C(X)) = = D EC(')
(c( EC(X) 10, [,, 2 E o [ - a² = ax
where the expectation taken over the joint distribution of a and D This completes the proof.
C CSA Algorithm Coverage
We now provide the proof of the coverage guarantees of the region produced by Algorithm I As mentioned in the main text, this follows as a direct corollary of Theorem 3.1
Corollary C.1. Let Dc, (X',Y),
[s and be defined in Theorem 3.1 Let (5
and U {um) m== be defined by lines 3-4 of the call CSA({sk) Dc, 1- a) of Algorithm
Denote by {qm m==1 the parameters defined by lines 4-9 of Algorithm and by T the scoring function Tss;
vector RRK. Then, denoting by C(X') = {y E y T(x("", 1y) S(D) S i}, P(Y' E C(X')) 2 1- a, where the probability is defined over the joint draw of the data De. (X', Y"), and the permutation a.
U) max ( (u s/ /m)m) for any
Proof. To prove the corollary, we must show that this specific function T satisfies the conditions of Theorem The overall probability is taken over the joint draw of the data (Dc, (X', Y")), the random permutation a, and the random directions U. We use the law of total probability by conditioning on specific realizations of the random elements
D!! d(1), and U
Given these fixed realizations, the score set S. and the projection directions (um) are fixed. The procedure in Algorithm to find the base quantiles (Zm) } via binary search is deterministic operation on this fixed data. Therefore, the function T(s; S( U) becomes a fixed, deterministic function s. The conditions of Theorem I met (again assuming no ties in T',, and its proof implies that the conditional probability of coverage is at least 1 o:
P(Y' E C(X)) a= n, D dd U = u) 1-
Since this guarantee holds for any realization (ι,, (d u), the unconditional guarantee follows from the law of total probability:
P(Y' E C(X') = E. " P(Y' eC(x') 10, DE" .y] 2 -
Thus, the guarantee holds for the specific procedure in Algorithm
D CSA Predict-Then-
Visual Walkthrough
We now present a visual accompaniment of the gredict-then-opiine algorithm presented in Sec- tion We once again take K for visual clarity in this walkthrough, where the predictors are as discussed in Section 3.2, namely assumed to be generative predictors (C' X)) where the number of samples per predictor are fixed to be {Jk]. For illustration, we assume J1 = 5 and J/2 meaning predictions with the first model are made by drawing 5 samples and 3 for the second. We assume the CSA calibration of Section projection directions and quantiles {(um. 9m))))))) ) are available that implicitly define an acceptance
already been performed, from which collection of
region& Q. We further assume the individual predictor score functions all the GPCP score given in Equation (2). with de from Equation 2 specifically here taken to simply be the standard Euclidean 2-norm, giving
34 9x c) min | cll jeI....Ju
(4)
24

---

We now wish to compute c* ma f(w, 2). To do so, we must start by defining this region C(x) for the test point æ, which we do by drawing the respective number of samples from the two models, producing samples {čır}]=1 and {c2)]/=1 as shown in Figure 7
O23
6z
012
014
611
G15
613
621
Figure Samples drawn from the two generative models {ī5} (C x)) (blue) and [e2))}=1 (c(C)) æ)) (green). Note that this is a visualization in the C space, i.e. not the space of multivariate scores.
By definition, Ve E C(x),
T min ||a,, cll. min ||e2 -ell S qm 1=.1....3
Vm = 1..... M.
(5)
I....5
As result, we must have that, Ve C(z), 3 j1 1,.... 5 and je = 1...., such that um (lii, oll. [e - cll) S Om Vm = 1,... M. Solving for e", therefore, amounts to con- sidering each pair j = (j1,32) J, where J = {1....,5} 5) x f1, and solving
of = arg max f(w,e) c)
(6)
s.t. un (122; cl1, 1122 -cl) s amn Vm E (L....,
Notice that, for any fixed J, this is standard convex optimization problem with convex feasible region. We illustrate how the feasible region would be constructed for fixed j in Figure 8 Note
that the construction of this feasible is never explicitly done in practice and is only implicitly used by convex solver routines in practice. We can, therefore, then solve Equation (6) over all possible j T and aggregate the maxima compute e*.
Figure 8: A candidate C is in prediction region if the projections of its distances (d,, d2) from at least one pair of points indexed by = (f,, j22 is in the acceptance region Q. Here, illustrate point e that lies the feasible region from its proximity the j := (4, 1) pair of points. Note that the left is again a visualization over the C. space, whereas the right is of the score space.
25

---

E
Computational Efficiency
E.1 Vectorized Score Computation
We now discuss the vectorized form of the computations discussed in Section 3.1.2 In particular, putting the scores into a matrix Se [si,..,, NNc. I E R&o,, x and directions into a matrix
U = [ui,.., ..., war] E RKXM all the projections & can be computed as S E RRNe, xM where [S] U k. is precisely & q E TRM == {9m == quantile([s{ U - a)} is then the quantile per row.
For any test point E we can then very efficiently check if it falls into the region by checking it satisfies U''' S q component-wise. Each iteration of the loop to find 3*, therefore, is very fast, and we find the search typically converges 5-10 iterations.
The final step is computing f. Computing this follows similarly above, where take the scores 5 compute projections s E RNe,, xM find T := UU/ E NN, xM where division is interpreted being defined component- along the rows, computing the maxima similarly along the rows [ max Ti.,, and finally computing 8 as the quantile of T
E.2 Empirical Efficiency Validation
To demonstrate the computational efficiency of this vectorized approach, we reran the experiment on the "Parkinsons" UCI task with both varying numbers of predictors (K) and projection directions (M); for each combination, we measured the total time taken to compute the quantile (i.e. to run Algorithm I) and to perform the projection to assess coverage for the test points. The additional predictors were taken to be random forests with different numbers of trees. K is given in the left column and M in each column heading, with the entry for each (K, M) pair being reported in seconds. As expected, by the vectorized nature of the computations, as discussed in the main paper, the performance scales gracefully over M roughly (MM) and remains roughly constant in K.
Table 4: Performance values for varying K (number of predictors) and M (number of projection directions). All values are reported in seconds.
Af
K
10
100
1000
10000
0.373029
2.32503
37.2561
8
0.0961056
0.327051
2.16211
36.7216
0,117146
0.373464
2,73875
37.2603
12
0,123772
0.384527
2.35386
37.0735
F Prediction Region Intuition
We now consider a simplified setting of the general procedure gain insight into the efficiency resulting prediction regions. In particular, we consider a scalar regression setting with K predictors fi...., fK X' + R. We further suppose. with slight abuse of notation, f(x) - y N (0, Σ) for f(x) = (x) fx (æ]] and that the scores are taken to be su(I, 8) == (f) y) Then, as Q is precisely the region with minimal volume that captures - density, it is precisely given by Xk. the a quantile of the x² distribution with K degrees of freedom. That is, the prediction
regions are (C) = ty : (f(x) v)
y) x Notably, if Σ diag((o)) )),
K
fu(z) Tx
C CSA diag (x) =
2 XK,1-
(7)
iml
Two insights arise from this expression. The first is that, unlike in the prediction regions for the case of the univariate score function sx y) == (fx(z)) y)² the size ICUSA (issA-ia((()) (z)) is dependent on x. In the univariate case, the size is 27x across all x. Here, however, the feasible set of becomes smaller the more distinct the values [x(x) are. The second insight, therefore, is that, under such independence of residuals, prediction region sizes are minimized in having well-separated predictions, which suggests that efficiency optimized by having an ensemble of predictors that learn distinct maps from X Y, such as those that focus distinct views of the covariate space.
26

---

G Full ImageNet Results
Here we present the complete collection of results for the classification task across additional a than those that could fit in the main paper.
Table 5: Average coverages across different coverage levels are shown in the top rows and average prediction set sizes in the bottom rows. Both were assessed over batch of i.i.d. test samples (15% of the validation set from ImageNet). Standard deviations and means were computed across 10 randomized draws of the calibration and test sets.
Coveags
23.0033
13.011
0030 (UOK)
01354 (3.300
0,586
0.548 801
0.23 13.000
0923 (0012)
11100-6559
(5.362
Leuch
220 (2,72)
123333 (76)
28.638 CL01
78.106 (2004)
196 2157)
145522 (2.02))
122
11216/22.15
(1874
$00-000
Teagh
187487 (5.66))
361376 (4.
(5--17 (5460)
377.356(5990)
311.52 (4.55)
325.754 (6,25))
365 (33)
200 033 (3.653)
133 (5
121,77 (16.71)
Coveraga
0.99
(1088)
3.99 (2.000)
2:00
0.587
0.99
D39 (0.210201)
10.9
On (000
Leagh
691 (8.53)
(2.1
400 1WW
(96.99 (4.99)
391192/0771134
592. 155/55555 (34
506 (T/0)
199 (8-30
(M153 (59)
281.33144.09)
H Full OpenML Results
Table 6: Average coverages across tasks for a 0.05 are shown in the top row and average prediction set lengths in the bottom row, where both were assessed over batch of i.i.d. test samples (20% of the dataset size). Standard deviations and means were computed across 5 randomizations of draws of the training, calibration, and test sets. In cases where the method failed to achieve sufficient coverage (defined as < 0.93), we do not include it in comparison for set length. Similarly, the single-stage approach fails to achieve coverage due to lack of exchangeability with test points.
Diasc:
Madda
Heal
LAGO
Ruskaa
McEime
Dascaal
CSA
EV p1R
S0NIMED:
ESR
1656
Teigh
L40s 9/W65
M1M01115 145
SUTAA 344
16131 Consue
18.10 5-N21
505
KIGI
2859
(300))
Tengh
1044 (93)
CST KVRE
30400000
15403000
358.3007
01/0091
$1.05
FUR'S
PIRTS
500
AUMA
KRISANO
Teigh
1/8 (1M
1006 9-60
7:00 F100
270 (100)
Table 7: Average coverages across tasks for a 0.025 are shown the top row and average
prediction set lengths in the bottom row, where both were assessed over batch of i.i.d. test samples (20% of the dataset size). Standard deviations and means were computed across randomizations of draws of the training, calibration, and test sets. In cases where the method failed to achieve sufficient coverage (defined as < 0.96), we do not include it in comparison for set length. Similarly, the single-stage approach fails to achieve coverage due to lack of exchangeability with test points.
LASI0
Eserad
CSA
20124
309
KUMB
28.0 6:124
1181 1390
H412 3.12
MA21 BL26
Conrge
300
08503300
WAT 140
METTARTEE 201
AWI 91
290E 4N (79
WITH INTI ASG
X1137 Gonnge
1091 (96)
0381 180
050 9331
307 XUMI
8002 (UB)
057 3.30
0R513/00
64778001
OBE TEX
X134
(1
30%
1997
Tarel
91240
M 2115
MADO 911
81428 211
9A/S L7N
WINE Dewagr
3001 2000
MM
500 S1ETO
050 90
12
100 SRETO
BINE BIETE
DETE 37C
304
0129
CRIVSSOON
Tagel
SAN (VE2
513 AUTTS
3291 FIETE
550 365
3
27

---

I CSA Region Size Over M
Across all the choices of M presented in the below table, the coverages were identical, namely 0.981 for the = 0.025 case and 0.962 for = 0.05 (for task 361237).
Table 8: Comparison of interval lengths for different M values.
M
Length (a 0.025)
Length (a: 0.05)
50
28.037
23.017
100
27.111
22.071
500
25,880
22.621
1000
26.110
22.172
5000
24.943
22.037
J CSA Prediction Region Visualizations
We now visualize some the prediction regions corresponding to some the trials run in Appendix H While find these intervals to be connected across these tasks, we expect visualizations over multivariate output spaces, i.c for 2D regression problems, would reveal sets to be non-connected.
Figure 9: Prediction regions across methods for task 361235 for = 0.05 (left) and 0.025 (right).
28

---

Coennee
Figure 10: Prediction regions across methods for task 361242 for 0.05 (left) and 0.025 (right).
K UCI Results
44 that have at least 1,000 samples. We consider those regression tasks from the UCI repository The complete collection of results is presented in Table 9 As discussed in the main text, across nearly all the UCI benchmark tasks, we find that the conformalized random forest does optimally and that ensembling methods provide no further benefit over simply taking this single predictor. In this degenerate case, we would expect an optimal aggregator to simply then return this optimal single predictor. We then find CSA to consistently significantly outperform other aggregation strategies and to return a prediction region of size comparable to that of the nominal random forest.
Table 9: Average coverages across tasks for a 0.05 are shown in the top row and average prediction set lengths in the bottom row, where both were assessed over a batch of i.i.d. test samples (10% of the dataset size). We are highlighting the robustness compared to other aggregation strategies here and so bold the best performing amongst the aggregation methods. Standard deviations and means were computed across 5 randomizations of draws of the training, calibration, and test sets. Note that, while the single-stage prediction regions are the smallest, they fail to achieve the desired coverage level and are, therefore, precluded from comparison.
Merka
Liwer Model
LASSO
CSA /Stple Suger
1196-00011
0:576 (1011)
0.55 19
p466 (1011))
C/M15 (D477))
0916 (0034)
01 0060
D25S (H-37)
0 0991/90011)
Lengh
19002 (0613))
11.358 (0.)55
11.70000
(0.133)
(50123)
35 (2.022
4.09/6.719)
14.35
bike
Coaage
0.544-0 067)
0.543 (0.04))
0.004 (3002)
0558 (0.03)
0/228 (2,000
0/08 (00)
0.56-0..00
069)
0:555 (00)
0.547 (3000
Lengh
1178/080115
1186-101440
BESE (0000)
8.111 (RUNT))
B882 (4000)
6.14-001147)
Coseagg
0L59 (1808)
0561 18:00
DOLS (0023))
D0N2 (D4))
0U15 (N022)
0.981 (8.00
0U15 (M823)
0.91T (0148)
Teegh
11.7510-474)
10.7H 1761
0605312.11 (44)
(1..111)
32,055 (65)
107700 16 593)
25.303 (3.34)
(60)
1L946 (1503)
(010))
D144 (U95T)
C/R15 (DINEZ)
DARER (0002)
11949 01:0
0 10 DNN)
0.91 (0
Loggl
(0.01)
5.991 (U816)
2.555
(2/00)
3349 (8025)
3,399 (0/05))
S:35 (8 (..00)
6955 (M803)
2.12906000
2 18 (ULN
palitorn
0555 063)
198-:0.000
D853
0.903
0:536-0
8.777 043)
DWS)
Lengh
35.087 (0.323)
15-200 10300
234 (3.42)
11 206 (2 114)
14.922 (1.074)
$1.220 (OURES
24 161 (0380)
4:27 (6.72))
4.564 (RAST)
Tergh
07:41100111 90)
47.771 (0 100
75.000 000
46 572 3DW)
45 016-011333
69117S 10 967)
57 122 6566
24 32 176)
X5.33 (1.56))
Leghh
2331-10.000 000
24112000000 (0.01)
2.151 (3.016)
2 (0/95)
2.134 (0/072)
2669 10 2015
5,50 (0.39))
1,99 18 (000
Legih
3.57 (9.00)
3,599
1325 (3107)
3 (0259)
3,507 (5481)
2.350 (0043)
3344-00.03)
5.35% (0.33)
1.364 (0.09)
1220 (8,776)
Coonge
(3.00)
(0007))
095003101)
1955 (U981)
03A4 (D1MIT)
1.029 01300)
4.535 (1:36)
(8.44))
winc
Coneaag
(0003)
5043 (200)
(2.02)
0:024 (QU035)
0U994 (02255)
0/345 (1605))
L Robust Traffic Routing Setup
We replicate the experimental setup of [3)] namely where graph of Manhattan with corresponding nominal transit times was extracted using OSMMx [49]. Formally, the Manhattan graph is given as tuple EV,E), where the edge weights represent the transit times along the respective city roads.
29

---

Such weights were assigned in a two-step process, namely by first making weather predictions and then using such weather predictions then upweight the nominal transit times. In particular, precipitation forecasts were made from time-series observations of previous precipitations readings, specifically given over a map spatially resolved to H W resolution. Precipitation forecasters, such as those considered in the experiments herein as given in [S] and [46], specifically map such previous observations to potential future trajectories. Formally, they define probabilistic models over some future time horizon T,, from which probabilistic draws Y E I x ~ P( x)) can be made, where a E RT*"=XW Notably, we instead consider the probabilistic forecasts at some future fixed time point T", meaning the outcome of interest Y E IR #W =Yr
From precipitation map, namely spatially resolved reading Y "xx" assign the final edge weights by first associating nodes to the closest pixel coordinate of the precipitation map. That is, denoting the pixel nearest vertex location yg,,y-- To, therefore, assign the edge weight, we average the weights of the edge endpoints and then weigh the nominal transit time, In particular, denoting the nominal transit time along such an edge between nodes (s, t) as Ce. the transit time with traffic was computed
(p% PC)) the node is assigned the value at such spatial
2 .PM
Ce == c. exp
(8)
2
M Compute Details
All OpenML were all run on a standard-grade CPU. The deep learning-based experiments, namely the ImageNet classification and traffic forecasting predict-then-optinize task, were performed on an Nvidia RTX 2080 Ti GPU. Such experiments, however, were conducted with publicly available, pre-trained models provided by the works respectively referenced in the sections describing the experimental setups.
N Conformal Aggregation Methods
We now describe the methods from [25 that were compared against experimentally, specifically the standard majority-vote C.M, partially randomized thresholding CR, and fully randomized thresholding C approaches. As discussed in Section 2.4, these methods all follow the structural form of
C(x)
v
Σ wylyy E Cx 2
(9)
and largely differ in their choice of weights and thresholds. The standard majority-vote CM the most natural choice, defined by
K Cx(x)) > x
1
1 iyy
(M( = v
(10)
The randomized methods differ in that independent randomization is leveraged over the threshold, namely with:
K
1
U
C C
(x:
1 e Cx)))
= 2+ 2
(11)
K
Σ
(12)
K
for U Uni 1)). Notably, all these methods retain the guarantees typical of conformal prediction.
30


Reviews:
[{"review_id": "jEZEPOpKRB", "reviewer": "Reviewer_Sni5", "summary": "This paper aims to improve conformal prediction for ensemble models. While existing work has already proposed aggregating ensemble prediction regions to retain coverage guarantees, this paper introduces a new framework called Conformal Score Aggregation (CSA). CSA leverages quantile envelopes to enable data-driven uncertainty estimation. The method can be directly incorporated into Predict-Then-Optimize tasks. It is evaluated on both regression and classification tasks and demonstrates strong performance.", "questions": "Please see above for the weaknesses. All the questions are there.", "limitations": "yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well-written and well-organized.\n\nThe method provides theoretical guarantees for convergence.\n\nIt is applicable to both regression and classification tasks.\n\nWeaknesses:\n\nI am not very familiar with this line of work, and I found the paper quite difficult to understand. Section 3.1.1 is especially complex, and I am unsure what the main novelty is and why it is important.\n\nI suggest adding illustrative figures or diagrams to help visualize the process of conformal prediction in ensemble models. This would also help clarify the specific contributions and steps introduced in the paper.\n\nThe phrase “improving efficiency” in the title is unclear to me. Does it refer to optimization speed or something else? Clarification would be helpful.\n\nI also found the evaluation results hard to interpret. For instance, what evaluation metrics are used for the regression and classification tasks?\n\nWhile the abstract mentions uncertainty quantification, the paper does not clearly demonstrate how uncertainty is estimated, nor does it provide an evaluation of the uncertainty quantification quality.\n\nI may improve my score based on further discussion.", "was_revised": true, "final_justification": "Most of my concerns are addressed. I will raise my score.", "rebuttal": "We thank the reviewer for their time and questions. We address each of the points successively below. Two of the questions are related, so we address these concerns first and then address the remaining ones.\n\n- **The paper does not clearly demonstrate how uncertainty is estimated, nor does it provide an evaluation of the uncertainty quantification quality.**\n\n- **The phrase “improving efficiency” in the title is unclear to me.**\n\nIn the conformal prediction community, the goal is to construct prediction regions $\\mathcal{C}(x)$ that achieve some pre-determined level of marginal coverage $1-\\alpha$, i.e. $\\mathcal{P}_ {X,Y}(Y\\in\\mathcal{C}(X))\\ge 1-\\alpha$. This is the sense in which the conformal prediction community does “uncertainty estimation”: the replacement of point predictions with $\\mathcal{C}(x)$ gives a sense of how that prediction could be wrong (i.e., by any deviation that remains in the prediction region). To make such uncertainty estimates maximally informative, we wish to achieve this coverage guarantee while minimizing the *average size* of such regions, which is referred to as the “predictive efficiency” of the procedure. This is precisely the “efficiency” being referenced in the title. Evaluating different methods of uncertainty quantification, therefore, occurs in two stages: demonstrating the coverage guarantee holds and, if so, then comparing the average prediction region sizes.\n\n**I am unsure what the main novelty is and why it is important.**\n\nAs elaborated on in our Related Works discussion, there is now a significant amount of interest in producing these prediction regions for model ensembles. The naive strategy is just to conformalize the ensembled prediction; however, several previous works have demonstrated improvements (in the sense of creating smaller prediction regions) possible by instead conformalizing the models first and then aggregating the individual models’ prediction regions. \n\nThese aggregation strategies, however, perform poorly (producing overly large prediction regions) when the uncertainty across models varies significantly. Our method takes a completely different route for aggregation to tackle this issue. Specifically, our core contribution was to define a **multivariate-score** $(s_1(x,y), …, s_K(x,y)$ for an ensemble with $K$ models and to *both* demonstrate that the probabilistic coverage guarantees of conformal prediction could be extended by introducing a multivariate **quantile envelope** in place of the classical **scalar** “threshold” $\\widehat{q}$ (from which a partial ordering over $\\mathbb{R}^{K}_ +$ could be defined and used to generalize the definition of $\\mathcal{C}(x)$) **and** that the resulting prediction regions defined by this procedure were more “efficient” than alternate aggregation strategies, in both classification and regression settings. Extending the coverage guarantees was highly non-trivial and is what is presented in Section 3.1.1, which involved defining a partial ordering using ideas from quantile envelopes and a non-obvious data-splitting step to ensure this partial ordering produces prediction regions that are as tight as possible (which was confirmed to be necessary for coverage in the ablation shown in Section 4.2). \n\nAdditionally, using high-dimensional prediction regions in regression settings is often not possible to do directly. One common use case is performing robust decision making downstream of having a conformalized predictor. Section 3.2, therefore, was another significant contribution to highlight that this approach we proposed can be used in decision-making frameworks. This very critically relies on some of the modeling choices of how we constructed the quantile envelopes above and would not trivially follow if these details changed, in particular, in ensuring the constraints can be expressed within the framework of convex optimization, as $u_{m}^{\\top} s(\\widehat{c}_ {\\vec{j}}, c) \\le \\widehat{q}_ {m} $ $\\forall m\\in\\\\{1,...,M\\\\}$. Notably, *none* of the other aggregation strategies lend themselves to use in this fashion, as they do not take on a similarly explicitly decomposable form, rendering it not possible to efficiently solve the resulting problems using convex solvers.\n\n**For instance, what evaluation metrics are used for the regression and classification tasks?**\n\nAs discussed above, the metric of interest in the conformal prediction literature is, assuming a method achieves the desired coverage, how small the average prediction region size is, although how this is measured in regression settings is sometimes non-obvious. In the classification case, one can simply count, for a given $x$, how many of the classes of $\\mathcal{Y}$ comprise $\\mathcal{C}(x)$. This is what is shown in Table 1: we demonstrate that, for three distinct choices of $\\alpha$ (0.1, 0.05, and 0.01), our method attains the theoretically demonstrated coverage guarantee and produces significantly smaller prediction regions than the competitor aggregation strategies and the strategy of conformalizing the ensembled predictor.\n\nIn **scalar** regression settings (i.e. $\\mathcal{Y} = \\mathbb{R}$), prediction region sizes can similarly be assessed by discretizing the output space and checking what subset of such discretized points falls into $\\mathcal{C}(x)$. The coverages and region sizes as measured by summing the covered discretized space, therefore, are again what are shown in Table 2, across a wide collection of benchmark tasks, again for two different values of $\\alpha$ (0.05 and 0.025). \n\nAs mentioned earlier, however, many regression problems are over multi-dimensional output spaces. In these cases, two issues arise. One is that the ability to even estimate the volume of the prediction region (other than some simple analytically computable cases) becomes infeasible due to the exponentially large grids necessary. More importantly, however, it is often not of practical interest to create a discrete collection of points that lie in the prediction region in these high-dimensional cases. These two points, therefore, point to *indirect* measures of the prediction region size as being of greater interest in these high-dimensional settings.\n\nAs discussed in the first answer, these high-dimensional regression regions are often useful when employed in robust decision-making pipelines. The standard conformal decision-making framework seeks to solve $w^{\\*}_ {\\mathrm{rob}} := \\mathrm{arg}\\min_w \\min_{\\widehat{c}\\in\\mathcal{C}(x)} f(w, \\widehat{c})$ as a robust proxy to solving the problem under a true, but *unknown*, problem parameter $c$, i.e. $w^\\* := \\mathrm{arg}\\min_w f(w, c)$. In robust decision making, a **larger prediction region** will lead to a **more conservative decision**, which results in a higher suboptimality gap $\\Delta(x, c) := \\min_{w} \\max_{\\widehat{c}\\in\\mathcal{C}(x)} f(w, \\widehat{c}) - \\min_{w} f(w, c)$. A normalized version of this suboptimality gap is *precisely* the indirect measure that is of interest to compare methods for high-dimensional settings and is what we use to assess and demonstrate improvements in the extremely high-dimensional ($\\mathbb{R}^{9867}$) problem of Section 4.3.\n\n**I suggest adding illustrative figures or diagrams to help visualize the process of conformal prediction in ensemble models.**\n\nWe were hoping the visual accompaniment provided in Appendix A would clarify the textual description on Section 3.1.1 but would gladly want to extend this to further clarify parts of the method that remain opaque."}, {"review_id": "xVtX1yDc1F", "reviewer": "Reviewer_MGCG", "summary": "Summary:\n\nThe paper introduces a novel method called Conformal Score Aggregation (CSA) to enhance uncertainty estimation in ensemble models using conformal prediction. While existing approaches aggregate prediction regions after conformalization, potentially causing overly conservative regions, CSA instead aggregates predictons in the score space itself. The main contribution is using a multivariate score function and defining prediction regions via quantile envelopes. This enables tighter, more data-driven prediction regions. The authors demonstrate that their method achieves distribution-free coverage guarantees. It is less conservative, and improves efficiency in both classification and predict-then-optimize regression tasks through empirical evaluations.", "questions": "Questions for Authors:\n\nHow does computational complexity scale with increasing ensemble size (K) or calibration dataset size (NC)? Could the method become infeasible at larger scales?\n\nHow sensitive is CSA to the choice of projection directions {u_m}? Would alternative direction-sampling methods improve performance further?\n\nHow should practitioners optimally select the calibration split ratio (DC^(1):DC^(2)) in practice, and how sensitive are results to variations in this choice?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe introduction of multivariate score functions and quantile envelopes in conformal prediction is innovative and addresses limitations in existing methods.\n\nThe method leverages quantile envelopes and concepts of higher-dimensional quantiles for aggregation, enhancing predictive efficiency.\n\nAuthors conduct experimentation, including classification (ImageNet) and regression benchmarks, providing strong evidence of CSA's practical advantages.\n\nCSA can be applied across both classification and regression tasks, unlike some existing methods restricted to specific scenarios.\n\nWeaknesses:\n\nThe approach involves multiple stages (splitting calibration sets, directional quantile computation). Authors should comment on the computational overhead, especially with larger ensemble sizes.\n\nThe necessity of tuning (e.g., binary search for optimal quantiles) could limit practical scalability.\n\nPerformance is likely sensitive to the calibration set split (e.g.,\nand\n); suboptimal splitting could substantially affect predictive efficiency and coverage.\n\nWhile the paper extensively compares to previous methods, authors should comment extensively on the computational cost relative to these alternatives.", "was_revised": true, "final_justification": "Initially I mistakenly uploaded review of another paper. And added the review of the submitted paper in the official comment because I was unable to edit the main review at the time. I am able to edit the main review now. Have not received any rebuttal by authors yet on the new review. Will edit this soon after  I receive their rebuttal.", "rebuttal": ""}, {"review_id": "yZfqY0F1oB", "reviewer": "Reviewer_xX66", "summary": "The authors propose a method to merge prediction sets from different predictors, each being a valid prediction set with at least $1-\\alpha$ coverage rate. They take advantage of the quantile envelope idea but make substantial changes. Most importantly, they develop a nontrivial method to correct the coverage rate for the intersection of many prediction sets, which seems to be a promising alternative to Bonferroni. Performance on real datasets is shown.", "questions": "1) What are the typical values of $\\beta^\\ast$ in your numerical experiments? I guess it might be very close to $\\alpha$ in many cases as the random forest is one of the predictors and its performance is almost as good as that of the proposed method (Appendices H and I). What about $\\hat t$?\n\n2) What are the typical correlations between scores from different predictors in your experiments? In general, the random forest and the XGBoost could provide very similar scores. \n\n3) Overall, LASSO or OLS is not as good as more advanced algorithms, so if we drop them from the ensemble, does it really matter? In other words, under what situations, including such ``simple'' predictors will help? (In terms of coverage and/or size of the resulting prediction set.)\n\n4) Is it possible to establish some proper concentration inequality that quantify the fluctuation of a finite number of random directions $M$?  Or, could you provide some empirical evaluation on the influence of $M$ on coverage and size of the prediction set? (Similar to Table 4.)", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strength: It is impressive to see the idea of splitting the calibration set into two (disjoint) parts, with the first part used to calculate a $\\beta^\\ast$ for coverage correction and the second part for providing a further correction $\\hat t$ for constructing the final prediction set. In my opinion, this is the most important contribution of this work to the study of conformal prediction.\n\nWeaknesses: \n1) I do not think Theorem 1 precisely covers the setting of Algorithm 1. A complete proof of the correctness of Algorithm 1 must be provided. Theoretical guarantees are perhaps the most valuable feature of conformal prediction. Also, in the statement of the theorem, as well as in its proof (Appendix B) and elsewhere in the paper, the authors should carefully and clearly specify with respect to which random variables the probability is taken. I will raise my rating once these are done satisfactorily.\n\n2) The readability of some parts of the paper needs to be improved, especially on page 5. This is important, as this page contains the most essential idea of the paper. For example, $\\mathcal{S}^{K-1}$ is not defined; just above Theorem 1,  what does it mean by ``with this choice of $\\{ (u_m, \\widetilde{q}_m) \\}$''? Does $\\widetilde{q}_m$ already contain the information of $\\beta^\\ast$?", "was_revised": true, "final_justification": "The authors propose a theoretically justified method for aggregating multiple valid prediction regions, addressing a timely topic that has received considerable attention recently. The approach is novel and interesting. I believe it substantially advances the research in this direction.", "rebuttal": "We thank the reviewer for their time and questions. We address each of the points successively below.\n\n**A complete proof of the correctness of Algorithm 1 must be provided. Theoretical guarantees…**\n\nWe consider the modified form of the theorem statement and proof below, where we make explicit note of the data splitting procedure and the random variables over which the probabilities are defined. In this presentation, we first provide the general theorem and proof for multivariate scores. We then state a corollary for the choice of $T$ (defined in the theorem below) defined with quantile envelopes used in the paper.\n\n**Theorem**\n\nSuppose $(X_1, Y_1), \\dots, (X_{N_C}, Y_{N_C}), (X', Y')$ are exchangeable, where $D_C := \\\\{(X_i,Y_i)\\\\}_ {i=1}^{N_{C}}$. Assume further that $K$ non-negative maps $s_{k} : \\mathcal{X}\\times\\mathcal{Y}\\rightarrow\\mathbb{R}_ +$ have been defined and a composite $s(X,Y) := (s_1(X,Y),...,s_{K}(X,Y))$ is defined.\n\nLet $\\sigma = (\\sigma_1, \\dots, \\sigma_{N_C})$ be a random permutation of the indices $\\\\{1, \\dots, N_C\\\\}$, drawn uniformly and independently of $D_C$ and $(X',Y')$. Let the calibration set $D_C$ be partitioned into $D_C^{(1)} := \\\\{(X_{\\sigma_j}, Y_{\\sigma_j})\\\\}_ {j=1}^{N_{C_1}}$ and $D_C^{(2)} := \\\\{(X_{\\sigma_j}, Y_{\\sigma_j})\\\\}_ {j=N_{C_1}+1}^{N_{C_1} + N_{C_2}}$, where $N_{C} := N_{C_1} + N_{C_2}$. Let the corresponding score sets be $\\mathcal{S}_ C^{(1)}$ and $\\mathcal{S}_ C^{(2)}$. Let $T(\\cdot; \\mathcal{S}_ C^{(1)}): \\mathbb{R}_ +^K \\to \\mathbb{R}$ be a deterministic function for any given realization of $\\mathcal{S}_ C^{(1)}$.\n\nFor some $\\alpha\\in(0,1)$, let $\\hat{t}$ be the $\\lceil(N_{C_{2}}+1)(1-\\alpha)\\rceil$-th smallest value of the set of transformed scores $\\\\{T(s_i; \\mathcal{S}_ C^{(1)}) \\mid s_i \\in \\mathcal{S}_ C^{(2)}\\\\}$. Assume that ties among the transformed scores occur with probability zero.  Then, denoting by $\\mathcal{C}(X') = \\\\{ y \\in \\mathcal{Y} \\mid T(s(X', y); \\mathcal{S}_ C^{(1)}) \\le \\hat{t} \\\\}$,\n\n$$ \\mathcal{P}(Y' \\in \\mathcal{C}(X')) \\ge 1-\\alpha, $$\n\nwhere the probability is defined over the joint draw of the data $D_C$, $(X',Y')$, and the random permutation $\\sigma$.\n\n**Proof**\n\nThe overall probability is taken over the joint distribution of the exchangeable data, $D_C$ and $(X',Y')$, and the independent random permutation, $\\sigma$. We use the law of total probability by first conditioning on a specific realization of the permutation, $\\sigma = \\pi$, and the data in the first split, $D_C^{(1)} = d^{(1)}$. Given $\\sigma = \\pi$ and $D_C^{(1)} = d^{(1)}$, the score set $\\mathcal{S}_ C^{(1)}$ is fixed. As a result, the function $T(\\cdot; \\mathcal{S}_ C^{(1)})$ becomes a fixed, deterministic transformation.\n\nBy the initial exchangeability of all data points, after conditioning on the values of the first split $D_C^{(1)}$, the remaining $N_{C_2}$ calibration points in $D_C^{(2)}$ and the test point $(X', Y')$ are still an exchangeable sequence. Applying the fixed transformation $T$ to their scores yields an exchangeable sequence of $N_{C_2} + 1$ scalar values:\n\n$$ \\\\{T(s_i; \\mathcal{S}_ C^{(1)}) \\mid (X_i, Y_i) \\in D_C^{(2)}\\\\} \\cup \\\\{T(s(X', Y'); \\mathcal{S}_ C^{(1)})\\\\} $$\n\nUnder the no-ties assumption, the rank of the test value $T(s(X', Y'); \\mathcal{S}_ C^{(1)})$ within this sequence is uniformly distributed on $\\\\{1, \\dots, N_{C_2}+1\\\\}$. The test point $Y'$ is covered if its transformed score is less than or equal to the threshold $\\hat{t}$. This occurs if and only if the rank of the test score is at most $m = \\lceil(N_{C_2}+1)(1-\\alpha)\\rceil$. The probability of this event, conditional on $\\sigma=\\pi$ and $D_C^{(1)} = d^{(1)}$, is:\n\n$$\n\\mathcal{P}(Y' \\in \\mathcal{C}(X') \\mid \\sigma = \\pi, D_C^{(1)} = d^{(1)}) = \\frac{\\lceil(N_{C_2}+1)(1-\\alpha)\\rceil}{N_{C_2}+1} \\ge 1-\\alpha.\n$$\n\nSince this guarantee holds for any realization $(\\pi, d^{(1)})$, the unconditional probability also holds by the law of total probability:\n\n$$\n\\mathcal{P}(Y' \\in \\mathcal{C}(X')) = \\mathbb{E}_ {\\sigma, D_C^{(1)}}\\left[\\mathcal{P}(Y' \\in \\mathcal{C}(X') \\mid \\sigma, D_C^{(1)})\\right] \\ge \\mathbb{E}_ {\\sigma, D_C^{(1)}}[1-\\alpha] = 1-\\alpha,\n$$\n\nwhere the expectation is taken over the joint distribution of $\\sigma$ and $D_C^{(1)}$. This completes the proof.\n\n**Corollary**\n\nThe coverage guarantee of Theorem 1 holds for the $T$ implicitly defined by Algorithm 1. Let $U = \\\\{u_m\\\\}_ {m=1}^M$ be the set of projection directions, drawn randomly and independently of the data $(D_C, (X', Y'))$ and the random permutation $\\sigma$. Given a realization of the first score split, $\\mathcal{S}_ C^{(1)}$, and the directions $U$, the parameters $\\\\{\\tilde{q}_ m\\\\}_ {m=1}^M$ are determined by the procedure in lines 4-9 of Algorithm 1. The function is then explicitly defined for any score vector $s \\in \\mathbb{R}_ +^K$ as $ T(s; \\mathcal{S}_ C^{(1)}, U) = \\max_{m=1,\\dots,M} (u_m^\\top s / \\tilde{q}_ m) $\n\n**Proof**\n\nTo prove the corollary, we must show that this specific function $T$ satisfies the conditions of Theorem 1. The overall probability is taken over the joint draw of the data ($D_C, (X',Y')$), the random permutation $\\sigma$, and the random directions $U$. We use the law of total probability by conditioning on specific realizations of the random elements $\\sigma=\\pi$, $D_C^{(1)}=d^{(1)}$, and $U=u$.\n\nGiven these fixed realizations, the score set $\\mathcal{S}_ C^{(1)}$ and the projection directions $\\\\{u_m\\\\}$ are fixed. The procedure in Algorithm 1 to find the base quantiles $\\\\{\\tilde{q}_ m\\\\}$ via binary search is a deterministic operation on this fixed data. Therefore, the function $T(s; \\mathcal{S}_ C^{(1)}, U)$ becomes a fixed, deterministic function of $s$. The conditions of Theorem 1 are met (again assuming no ties in $T$), and its proof implies that the conditional probability of coverage is at least $1-\\alpha$:\n\n$$\n\\mathcal{P}(Y' \\in \\mathcal{C}(X') \\mid \\sigma = \\pi, D_C^{(1)} = d^{(1)}, U = u) \\ge 1-\\alpha.\n$$\n\nSince this guarantee holds for any realization $(\\pi, d^{(1)}, u)$, the unconditional guarantee follows from the law of total probability:\n\n$$\n\\mathcal{P}(Y' \\in \\mathcal{C}(X')) = \\mathbb{E}_ {\\sigma, D_C^{(1)}, U}\\left[\\mathcal{P}(Y' \\in \\mathcal{C}(X') \\mid \\sigma, D_C^{(1)}, U)\\right] \\ge 1-\\alpha.\n$$\n\nThus, the guarantee holds for the specific procedure in Algorithm 1.\n\n**The readability of some parts of the paper needs to be improved…**\n\nWe apologize for this lack of clarity; we were hoping the Appendix A accompaniment would clarify the exposition. For your specific notational comments,  $\\mathcal{S}^{K-1}$ denotes the unit sphere in $\\mathbb{R}^K$, which we will make explicit where it is introduced in Section 2.2. For the clarification on $(u_m, \\tilde{q}_ m)$, **yes**, these parameters are defined once $\\beta^\\*$ has been determined.\n\n**What are the typical values of $\\beta$ and $\\widehat{t}$ in your numerical experiments?**\n\nIn general, $\\beta^\\*$ should be fairly close to $\\alpha$, and, if the $1-\\alpha$ quantile envelopes for $\\mathcal{S}_ C^{(1)}$ and $\\mathcal{S}_ C^{(2)}$ are similar, $\\widehat{t}=1$; $\\widehat{t}>1$ if the $\\mathcal{S}_ C^{(1)}$ envelope is overly tight and $\\widehat{t}<1$ if overly conservative. We show these below for the experiments that were run in the paper. We also highlight that, while Appendix I suggests that the random forest achieved comparable performance to aggregation, this was only true in the simple, UCI baseline experiments. In the more complex settings considered in the main text (Appendix G), our aggregation strategy outperformed the random forest predictor.\n\n$\\alpha = 0.05$ -- $\\beta^\\* = 0.04125$ across all tasks here\n\n| Task ID | $\\widehat{t}$ |\n| :--- | :--- |\n| 361234 | 1.093155 |\n| 361235 | 1.818106 |\n| 361236 | 1.272447 |\n| 361237 | 1.575767 |\n| 361238 | 1.172660 |\n\n$\\alpha = 0.025$ -- $\\beta^\\* = 0.020625$ across all tasks here\n\n| Task ID | $\\widehat{t}$ |\n| :--- | :--- |\n| 361234 | 1.029694 |\n| 361235 | 1.779467 |\n| 361236 | 1.297137 |\n| 361237 | 1.433094 |\n| 361238 | 1.148173 |\n\n**What are the typical correlations between scores from different predictors in your experiments?**\n\nHere are some examples of correlations of scores from the experiments. As expected, the linear and LASSO models can be highly correlated; CSA is most useful when predictor scores are uncorrelated.\n\n**Task 361234**\n\n$$\n\\begin{bmatrix}\n1& 0.998 & 0.664 & 0.844 \\\\\\\\\n0.998 & 1& 0.672 & 0.856 \\\\\\\\\n0.664 & 0.672 & 1& 0.808 \\\\\\\\\n0.844 & 0.856 & 0.808 & 1\n\\end{bmatrix}\n$$\n\n**Task 361235**\n\n$$\n\\begin{bmatrix}\n1& 0.567 & 0.297 & 0.444 \\\\\\\\\n0.567 & 1& 0.25 & 0.301 \\\\\\\\\n0.297 & 0.25 & 1& 0.593 \\\\\\\\\n0.444 & 0.301 & 0.593 & 1\n\\end{bmatrix}\n$$\n\n**Under what situations does including such “simple” predictors help?**\n\nYou are correct that including such simple predictors should not help. However, we wanted to highlight that the method would not be “thrown off” by the presence of these extra predictors, which we confirm in the ablations for LASSO below. This is because, in a real use case, it may not be a priori clear which predictors are “overly simple” or not. \n\n**Task 361234**\n\n| $\\alpha$ | With | Without |\n| :--- | :--- | :--- |\n| 0.05 | 8.0534 | 8.1016 |\n| 0.025 | 9.0303 | 9.0375 |\n| 0.01 | 10.2253 | 10.1544 |\n\n**Could you provide some empirical evaluation on the influence of $M$ on coverage and size of the prediction set?**\n\nIntuitively, as $M\\to\\infty$, we would expect to recover the $1-\\alpha$ tightest cover and, thus, that the prediction region size should be roughly decreasing in $M$, with some plateau. This is borne out below. Across all the choices of $M$, the coverages were identical, namely 0.981 for the $\\alpha=0.025$ case and 0.962 for $\\alpha=0.05$ (for task 361237).\n\n| $M$ | Length ($\\alpha=0.025$) | Length ($\\alpha=0.05$) |\n| :--- | :--- | :--- |\n| 50 | 28.037 | 23.017 |\n| 100 | 27.111 | 22.071 |\n| 500 | 25.880 | 22.621 |\n| 1000 | 26.110 | 22.172 |\n| 5000 | 24.943 | 22.037 |"}, {"review_id": "h2uCy0IGQy", "reviewer": "Reviewer_RKaG", "summary": "Authors propose conformal score aggregation (CSA) a new method for adapting conformal prediction to ensembles. Instead of averaging the scores of each model, they construct a data-driven ordering of the score vectors. This pre-ordering is achieved using a set of nested sets, each defined by a threshold parameter. Once the conformal scores are ordered, the coverage guarantee can be computed. Then, they go over a binary search over a separate parameter to obtain the desired coverage based on that ordering. They prove the theoretical consistency of CSA and provide experiments demonstrating the improvements it offers.", "questions": "Q1. The conclusions section appears as Discussion, it should be self-contained and have more detail.\n\nQ2. The paper can be improved with a comment on the limitations of CSA. \n\nQ3. We are not sufficiently familiar with this specific field to provide a confident assessment, however the papper seems technically sound and consistent. This is spetially the case for sections 3.2 and 4.2.", "limitations": "No, the authors should comment on limitations of the proposed method.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nS1. The CSA framework is novel and takes a different approach on the adaptation of conformal prediction to ensembles. (Originality).\n\nS2. CSA experimentally proves to yield significantly less conservative regions than previous methods. (Significance).\n\nS3. Computational efficiency.\n\nS4. The writing is clear and the exposition and methods are sound. (Quality, Clarity).  \n\nWeaknesses:\n\nW1. There is no comment on the limitations of the proposed method.\n\nW2. The article lacks a conclusion section.\n\nMinor comments:\nE1. Line 230 grammatical error.", "was_revised": true, "final_justification": "The concerns raised in the review were properly addressed by the authors. Since these were minor comments, I will keep my rating as it was.", "rebuttal": "We thank the reviewer for their time and questions. We address each of the points successively below.\n\n**The conclusions section appears as Discussion, it should be self-contained and have more detail.**\n\nWe will expand the discussion in the final submission to include more details about extensions that can be made on the current CSA framework.\n\n**The paper can be improved with a comment on the limitations of CSA.**\n\nWe will extend the conclusions to note some limitations. Some current ones were highlighted in the discussion section, where it would be of interest to see further use cases of CSA being integrated into regression settings beyond the predict-then-optimize setting we considered in the paper. Another limitation might be in the loss of data efficiency in needing to further split the calibration data into $D_ {C_1}$ and $D_ {C_2}$ for determining the quantile envelope shape without sacrificing coverage guarantees. Similar to how there is full conformal prediction as a way to augment split conformal use cases when it is infeasible to hold out a separate calibration set, it could likely be possible to generate a similar “full CSA” procedure, where the set is not split and where we sacrifice computational efficiency for data efficiency."}]

Author rebuttal:

Meta-review: This paper presents a method for aggregating prediction regions based on  scores produced by individual models in their ensemble. Earlier work merges prediction sets, each of which is generated by each individual model. On the other hand, the current paper considers a vector of scores and leverages quantile envelope to construct a final prediction set. All of reviewers feel that the paper is well written and the method referred to as CSA enhances predictive efficiency. The authors did a good job in responding to reviewers’ comments, leading that two of reviewers raised their overall scores. I believe that this work is one promising research toward conformal prediction for ensembles.
Decision: Accept (poster)