False Coverage Proportion Control for Conformal Prediction
Authors: Alexandre Blain
Bertrand Thirion
Pierre Neuvial
Venue: ICML 2025 poster (2025)
Keywords: Conformal prediction
multiple testing
False Discovery control
Primary area: general_machine_learning
TL;DR: We develop an aggregation-based method to achieve tight control of the False Coverage Proportion in a transductive conformal inference framework.

Abstract: Split Conformal Prediction (SCP) provides a computationally efficient way to construct confidence intervals in prediction problems. Notably, most of the theory built around SCP is focused on the single test point setting. In real-life, inference sets consist of multiple
points, which raises the question of coverage guarantees for many points simultaneously. While *on average*, the False Coverage Proportion (FCP) remains controlled, it can fluctuate strongly around its mean, the False Coverage Rate (FCR). We observe that when a
dataset is split multiple times, classical SCP may not control the FCP in a majority of the splits. We propose CoJER, a novel method that achieves sharp FCP control in probability for conformal prediction, based on a recent characterization of the distribution of conformal $p$-values in a transductive setting. This procedure incorporates an aggregation scheme which provides robustness with respect to modeling choices. We show through extensive real data experiments that CoJER provides FCP control while standard SCP does not. Furthermore, CoJER yields shorter intervals than the *state-of-the-art method* for FCP control and only slightly larger intervals than standard SCP.

False Coverage Proportion Control for Conformal Prediction
Alexandre Blain 123
Bertrand Thirion 142 Pierre Neuvial 3
Abstract
about the underlying data distribution. These methods are used across various fields such as medicine, finance, and engineering, where decision-makinn requires quantifiable estimates of uncertainty.
Split Conformal Prediction (SCP) provides a computationally efficient way to construct confi- dence intervals in prediction problems. Notably, most of the theory built around SCP is focused on the single test point setting. real-life, infer- ence sets consist of multiple points, which raises the question of coverage guarantees for many points simultaneously. While on average, the False Coverage Proportion (FCP) remains con- trolled, it can fluctuate strongly around its mean, the False Coverage Rate (FCR). We observe that when a dataset is split multiple times, classi- cal SCP may not control the FCP in a major- ity of the splits. We propose CoJER, novel method that achieves sharp FCP control in prob- ability for conformal prediction, based on re- cent characterization of the distribution of con- formal p-values in a transductive setting. This procedure incorporates an aggregation scheme which provides robustness with respect to mod- eling choices. We show through extensive real data experiments that CoJER provides FCP con- trol while standard SCP does not. Furthermore, CoJER yields shorter intervals than the state-of- the-art method for FCP control and only slightly larger intervals than standard SCP.
Split conformal prediction (SCP) is the standard practical approach within the conformal prediction framework, of- fering improved computational efficiency and scalability, In this method, the dataset is divided into two parts: a train- ing set and a calibration set. The model is first trained on the training set, and then the calibration set is used to compute non-conformity scores and ultimately confidence intervals. This approach simplifies the computation com- pared to full conformal prediction, as the calibration set re- mains fixed while the model parameters are tuned. Split conformal prediction ensures marginal coverage, meaning that on average, the prediction intervals will contain the true value a specified proportion of the time - e.g.. 90% or 95% - providing reliable measure of uncertainty.
Many real-world applications, such as healthcare (predict- ing outcomes for multiple patients), finance (forecasting ranges for multiple assets), and manufacturing (quality control across batches), inherently require predictions for multiple data points simultaneously. However, SCP en- counters limitations when applied to multiple test points si- multaneously, as discussed by Vovk, 2013. A key concern in this transductive setting is to control the false coverage proportion (FCP), that is, the proportion of prediction in- tervals that fail to contain the true value. In applications involving a large number of simultaneous predictions, is important to ensure that the FCP remains within acceptable bounds. While SCP controls the expected proportion of non-covered points (FCR; False Coverage Rate), it does not guarantee control over the actual proportion of non-covered intervals across multiple test points with high probability.
1. Introduction
In the realm of uncertain predictions, conformal prediction methods provide robust approach to obtaining confidence intervals that maintain a predetermined coverage with sta- tistical guarantees. Introduced by Vovk et al., 2005, con- formal prediction methods are nonparametric techniques that can be applied to any predictive model to generate pre- diction intervals that are valid under minimal assumptions
To address this, we introduce method to control the FCP using conformal p-vluues within a Joint Error Rate (JER, Blanchard et al., 2020) control framework and cumula- tive distribution function (CDF) formulations (Gazin al., 2024). This approach allows controlling the FCP with high probability across multiple predictions, ensuring that the proportion of incorrect intervals does not exceed a pre- specified level. While the approach of Gazin et al., 2024 yields valid FCP bounds, the latter can be conservative, as
'INRIA Uniiversité Paris-Saclay 'Institut de Mathématiques de Toulouse. Université de Toulouse; CNRS; UPS. F-31062 Toulouse Cedex 9, France *CEA. Correspondence to: Alexandre Blain <alexandre.be blain @nria.fr..
Proceedings of the 1nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

realistic measure of the performance of the trained model. We consider arbitrary non-conformity scores -1.--- Sn. where the only requirement is that S; = 8(X,, Y.) for e [n]. where & depends on the observations only via Duain. Typically, the non-conformity score S; is an increasing function of the distance between Y; and the prediction at point X,, For example, a regression framework, the non- conformity scores can be obtained as S Y - (X,)), where j(x)) is a point prediction of Y; given X; learned False Coverage Proportion Control for Conformal Prediction
discussed by the authors. We propose a novel method that relies on nonparametric calibration obtain sharper FCP bounds.
Furthermore, to mitigate the impact of modeling deci- sions, we extend this procedure to aggregate conformal prediction intervals given by different models. Notably, we manage to obtain a fully nonparametric method valid for any aggregation scheme, which ensures valid coverage while maintaining relatively small prediction intervals.
Dimin Alternative non-conformity scores in the same framework include normalized residuals or model-based uncertainties.
Through extensive experimental validation on OpenML datasets, we demonstrate that our proposed methods effec- tively control the FCP and produce sharp prediction inter- vals. We also show that the proposed aggregation proce- dure retains FCP control while yielding more informative intervals than existing methods. Our results highlight the practical utility of these contributions in improving the re- liability and interpretability of conformal prediction meth- ods.
Relying on non-conformity scores avoids making assump- tions about data distributions or model characteristics. To obtain valid intervals, non-conformity scores have be ex- changeable across the calibration set and test point. For- mally, random vector (Zi,..., Zx) is said to be ex- changeable (see e.g. Vovk et al., 2005) if for any permu- tation T E 6x.
(Z1,... 7x) d (Z,(1):- ...., (r)))
2. Refresher on Conformal Prediction
Notation. The input data are denoted by
Provided that the vector of non-conformity scores (Si,..., Sn-1)) )) is exchangeable, SCP yields the following valid interval:
(X1,Y) Yi) ..... (X,, Yn) with Yi) E IR x R. where is the number of samples and p the number of variables. For any set S, |S] denotes the cardinality of S.
C. ( (((++1)0-)))
For any positive integer k, [&]] denotes the set k.). For a vector z = (Tz) and S C [p], we denote by P(y:s) (or z()) when there is no ambiguity) the j'h smallest value the sub-vector (za)aEs. Equality in distribution is denoted by d =, G, is the symmetric group of degree n, and (0001 1] denotes the uniform distribution on [0, 1].
This construction is detailed and thoroughly studied in Pa- padopoulos et al., 2002; Lei et al., 2018. Alternatively, this interval can also be obtained by thresholding a so-called conformal p-value (Vovk et al., 2005; Lei et al., 2018). The conformal p-value test point (Xn+1, Ya+1) is defined as:
Problem setup. The goal is to build confidence interval given predictive model and new sample Xn+1 E RP. The associated true outcome Yo+1 is not observed but should lie inside the confidence interval with high proba- bility. Formally, the goal to build confidence interval Ca such that:
n
1+ Σ
18 S SS,}
p
n
1
Intuitively, this p-value quantifies how unlikely the non- conformity score Sx+1 = (.Xn+1.Ynntl is, given the ob- served non-conformity scores (St) computed on the calibration set. our setting, since Yu+1 is unobserved, so are 5x+1 and the associated conformal p-value. This naturally leads the following definition of conformal p-value function.
PfYme E Ca (Xn+1))
(1)
2.1. Split Conformal Prediction
Split Conformal Prediction (SCP) offers a simple and flex- ible approach to constructing reliable prediction intervals. This method offers the desired guarantee in model- agnostic way at the cost of splitting the observations into training set Drrain and a calibration set Drat. For notation simplicity, we assume that the calibration set Deat consists of all n observations, and that an external training data set Dirain (say (X,, -1)=415000)) is also available.
Definition (Conformal p-value function, (Barber et al., 2021)). The conformal p-value function associated to the score function & is P Rp x R - [0. with
P(X,)) n+1
1+ 1 11 v) $ 8(X X3)1
j=1
The intuition behind SCP that the calibration set provides
With this definition, the conformal p-value (2) corresponds
2

---

False Coverage Proportion Control for Conformal Prediction
p P(Xn+1; Yn+1). In turn, the split conformal pre- diction confidence interval Co can be written as
However, this does not guarantee that FCP(C(a)) with high probability, noted by Gazin et. 2024. This is akin to the distinction between False Discovery Proportion control and False Discovery Rate control highlighted in the multiple testing literature (Genovese & Wasserman, 2006).
(X,+t)))) {y e s.t. P(Xn+1,; y) > a),
which also sometimes abbreviated as C, {p > o).
With this notation, the coverage property (1) is equivalent to
To ensure informative and interpretable control, our aim to build FCP upper bounds that hold with high probability. For any pre-specified level å O, this amounts to building (FCP a.,) ae(0..) a) such that:
P
which is indeed the definition of valid valuue. This du- ality between confidence intervals and p-values in the con- text of conformal prediction is the basis of the main con- tributions of the present paper. Reformulating split confor- mal prediction as a p-value thresholding procedure unlocks tools of the rich literature on p-value-basee based error control.
IP (Va [0, 1], FCP(C(a)) s FCP..)) 2 d. (5)
3. Tight FCP Control for Conformal Prediction
3.1. FCP Control and Conformal p-values
2.2. Conformal Prediction for Multiple Test Points
In this section, we recall the approach introduced by Gazin et al. (2024) obtain FCP control. This approach stems from close link between FCP control and the empir- ical Cumulative Distribution Function (CDF) of confor- mal p-values. Given non-conformity scores (S.) the conformal p-value of the test point (Xn-i, Yn+4) for S S m is defined as:
In practical applications, the test set for which we wish to obtain confidence intervals may contain many points. We consider the transductive setting introduced by Vovk (2013), in which m test points .,,t are ob- served while the corresponding outcomes Y..1...., Yutm are not observed. Performing split conformal prediction yields m confidence intervals C(a:) = (Cia)e with
Ci [A (Xn+s)) + ((((+1((-0)))
(3)
Pr = ++1
1
1 (S [s)
Split conformal inference for multiple test points relies on the assumption of exchangeability of the vector non- conformity scores:
Note that these p-values are unobserved, because the def- inition of p; involves the unobserved value Ynt+ via the corresponding non-conformity score Sn+i. These p-values can also be written as p; P(Xn+é; Yn+i), where P iss the conformal p-value function introduced in Definition 1. The SCP confidence intervals can then be written as
(S, [5)mmm)+]] is exchangeable
(Exch)
Under (Exch), the following marginal guarantee holds:
E [m],
P P(Yn+s E Cra (Xnti) 2
(4)
C,, = {ps S aa..
Lemma 1 (Empirical CDF of p-values and FCP, Gazin et al., 2024). Denote by F,, the empirical CDF of the joint distribution of .Pm). For any a [0, 1] denote by C(a) the split conformal intervals. Then:
To provide quantitative measure of coverage the of points using intervals I (It) [mm] define the False Coverage Proportion (FCP) and False Coverage Rate (FCR):
FCP(C(a))= = =F.
FCP(T): == Σ 1 (Yats $ I).
Lemma implies that the FCP only depends on the joint distribution of the conformal p-values. key result of Gazin al., 2024 is the characterization of this distri- bution under (Exch). This characterization involves the definition of a discrete distribution denoted by pe on 1+ l E [n + 11} associated with any fixed vector U = (U....., Un) E [0, 1]", as follows:
FCR(Z): == EFCP(Z)
By (4). FCR control holds at level a for standard split con- formal prediction:
m
pC ([t((n + 1))) = U(o - -Ue-1) LE [n + 1], (6)
FCR(C(a)) Σ p {Y (Xa(ii))) Sa.
where U/) S Un) ... SUm) S U((+1) = =1
S.
3

---

False Coverage Proportion Control for Conformal Prediction
Proposition (Joint distribution of conformal p-values, Gazin al., 2024). Assume that the non-conformity score vector (St) satisfies (Exch) and has ties. Then the m conformal p-values are distributed as (q) (q)...., ),)....)mm) where (q1,...,9m U) itd. pu, with U =
Proof. We write JER(t) as a function of Fr::
JER(t) P (7j [m]] PG)
< <11))
m
P 3j [m]] Σ
1
{or s 1
(U,....,U,, Un) Utd. u((0,1) 1J.
P 3j [m] Fnn(tt) 2 -
m
The approach of Gazin et al., 2024 to FCP control consists in using this distribution to obtain Dvoret- Aky-Kiefee-WolloiiaaMsssss
Then, we note that if for some E [0.1] we have FCP(C(a)) 2 jo(a)/m, then by definition of jo(a) com- bined with Lemma 1, j == jo(a) E [m]] is such that P... (tg) (o)(o) = FCP(C(a)) 2 2/m
like inequality (DKWM; Massart, 1990). The original DWKM inequality only holds under independence and therefore cannot be used in this context. The obtained inequality bounds the gap between the empirical CDF and the true CDF with high probability. FCP bounds can in turn be obtained using the CDF formu- lation of Lemma 1.
As an immediate consequence of Proposition 2, bounds of the form (5) with FCP.,s jo(a)/m are obtained di- rectly from JER controlling families. Indeed, assuming that t controls the JER at level 8 > O, then by Proposition 2 the associated (a)) = min{j E [m]] : a S [ty} satisfies:
3.2. FCP and Joint Error Rate Control
To obtain FCP control, we instead rely on Joint Error Rate control as introduced by Blanchard al. (2020). For kmaz E [m]] we define a threshold family of size kmax I tkmas I 1. The Joint Error Rate (JER) is formally defined as follows:
P Va E [0, FCP(C(a)) < }8(a))
2 1 - 8. (8)
as vector t = (ty))e[[mmez] such that 0 t1
Note that the dependence of jā in 6 comes from the fact that t is assumed to control JER at level 6. As a conse- quence of the above results, we obtain confidence intervals with uniform in in-probability FCP control associated to any on a JER controlling family.
Definition 2 (Joint Error Rate, Blanchard et al., 2020). De- note by P(j) the jth smallest value among (P1, (Pi....,PP.. P.m. The JER associated with t = (ty)je[ml is:
Corollary 1. Assume that controls the JER at level 8 > For a E [0, 1]. let = [Lam]- Then the associated intervals C(ā) (Cr,a)telmm) with Či.
(ER(()= = P (3j E [kmaz] P() < t2)
(7)
[ (X,+t) (((6++))(-)))) satisfy:
P(va E (0, s aa) 21 -
The threshold family t is said to control the JER at level 5 JE) S 5.
To prove Corollary 1, it is sufficient to note that the choice â = 'Lams] in (8) yields (ā)mm [am]/mn //m S a. Corol- lary states that the intervals C(ā) have FCP below a with high probability, uniformly in a. In order to build these intervals in practice, only remains to obtain JER con- trolling family
A threshold family t is a (vector) parameter, which has to be chosen in order to provide JER control. In the multiple hypothesis testing literature, JER control has been success- fully used to derive bounds on the False Discovery Pro- portion in various settings (Blanchard et al., 2020; Blain et al., 2022; Durand et al., 2020). Here, intend to obtain JER control for conformal p-values and derive FCP bounds from this control. We first derive from Lemma 1 tight link between FCP control and JER control.
3.3. Building a JER Controlling Family
We follow the approach of Blain et al., 2023, which have exploited related ideas to reach False Discovery control in for Knockoff inference (Candès et al., 2018). As already noted, we can view the threshold family t as tuning pa- rameter for JER control. In a nutshell, we first provide an algorithm to estimate the JER associated to a given t, and then use this algorithm to choose a family achieving the tightest possible JER control among a set of candidates (which will be called template).
Proposition 2 (FCP and JER). Lef t be arbitrary thresh- old family. Denote jo(a) = min{j e [m]] a S tt} for E [0, 1]. Then:
IP 3a E 10, FCP(C(a)) > jo(a) m < (t.
4

---

False Coverage Proportion Control for Conformal Prediction
We start by introducing Algorithm 1, which uses Propo- sition to draw B Monte-Carlo samples of the joint dis- tribution of conformal p-values, and sorts each of them to obtain order statistics. These order statistics are denoted by
t/ (())(0)), where
1 max b' EBE S.t. JER Ab( B'
b' B'
T
8
Gazin et al. (2024) also discuss (in Appendix B) possible a priori choices for template families in order to obtain FCP control. Here, we propose to derive the template from the joint distribution of conformal p-values itself. Indeed, as observed by Blain et al. (2022). optimal power is reached when the candidate families match the shape of the distri- bution of the p-values under exchangeability. Therefore, we define a template based on the distribution of the con- formal p-values appearing in Proposition 1. In practice, we apply Algorithm to obtain B' x m matrix of B' samples or order statistics of conformal p-values, indepen- dently from the first B Monte Carlo samples in order to avoid circularity biases. By sorting each column of this ma- trix, we obtain a B' m matrix Q of empirical quantiles of these order statistics. This defines a discrete template To in the sense of Definition 4, where T (b /B)) is the b'-th row of Q for b' E [B']. This construction is summarized in Algorithm 3 (Appendix B). We obtain the following result, which is akin to Theorem in Blain et al. (2023):
Algorithm 1 Sampling order statistics of conformal p- values using Proposition 1.
1: Input: B the number of MC draws; n the number of calibration points; m the number of test points
2: Output: II E [0, 10.1)8xm Bxm a matrix of simulated p- values
3: Io < zeros B, m)
4: be {1..... B) do
5: Sample U = [ul...., un] from 4((0, 1))*
6: Sample [7..... Iml from p // using (6)
7: Holb] + [n. [m)
8: end for
9: II, < line_sort sort (Il)
10: Return HI
Pb(1) .... S Pb(m) for each b E [B]. This allows us to evaluate the empirical JER, which estimates the actual JER.
Proposition 3 (JER control for conformal p-values). Con- sider the threshold family defined by t = ()n(())- Then, as B - t00,
Definition 3 (Empirical JER). Let t be threshold family of size kmaz- The empirical JER of t associated to set of B samples of order statistics of conformal p-values (as obtained from Algorithm 1) is defined as:
Proposition 3 is a consequence of the fact that JER < 8 (which holds by the definition of t,²), combined with the fact that the difference between JER and its Monte-Carlo approximation JER, is uniformly bounded in probability by 1/vB as B +oo. The proof of Proposition 3 is given in Appendix A for completeness.
112 13k
B
(t)
E1
Σ 1
[kmaal] Pb(k) }
In practice, we compute the empirical JER using Algo- rithm (Appendix B). In order to ensure JER control, it remains to choose t such that (ER(t)= < 6. To this end, we consider a sorted set of candidate threshold families called a template:
The number B of Monte-Carlo samples in Proposition 3 can be chosen arbitrarily large to obtain JER control. This leads to a valid FCP bound by Corollary 1. Algorithm describes all the steps needed to compute t. The resulting FCP bound is fully nonparametrie and therefore expected to yield tighter intervals than Gazin et al., 2024's approach. We call the resulting approach CoJER (Conformal - JER).
Definition& 4 (Template (Blanchard et al., 2020).. A lem- plate is component-wise non-decreasing function T [0, 1] + R that maps parameter X E [0, to threshold family T()) E RP.
This definition is naturally extended to the case of tem- plates containing a finite number of threshold families. The template corresponding to B' threshold families is then de- noted by (T (b/ ((//B") B")EEB
4. Aggregated Conformal Prediction
famisiel
While conformal prediction coverage guarantees are distri- bution free, the confidence interval output by the method can strongly depend on the chosen model ja in practice. Mitigating the consequences of such modeling decisions motivates the use of aggregation schemes to obtain more stable and generalizable confidence intervals.
Once template is specified, the calibrution procedure (Blanchard et al., 2020) can be performed. This con- sists in finding, among the template, the least conservative threshold family that controls the empirical JER at level Formally, we consider the threshold family defined by
Let us assume that we have K models fi....., fxx fitted on D Dtrain- The goal of aggregation is to build a valid confi-
5

---

False Coverage Proportion Control for Conformal Prediction
dence interval C(a) that takes into account the information provided by each model. Aggregating schemes for con- formal prediction have been introduced in Lei et al., 2018; Barber et al., 2021. Lei et al., 2018 propose a Bonferroni- type construction, where the confidence interval of each of the K models is built at level a K. An union bound argu- ment shows that the intersection of these intervals is valid level a, therefore yielding FCR control at level a.
Calibration can be performed in the same way as in the non-aggregated case. Note that we perform calibration af- ter aggregation; therefore, JER control is ensured directly on aggregated p-values and is not result of aggregat- ing JER controlling families. Importantly, this approach holds without additional assumptions on the aggregation scheme f. In particular, contrary to the approaches of Lei et al., 2018: Barber et al., 2021, our aggregated p-values are not required be valid p-values for the corresponding intersection hypothesis. We consider the threshold family B to = T(AB(0)), where
Barber et al., 2021 propose method that relies on p- value aggregation result which states that twice the arith- metic mean of valid p-values is a valid p-value see e.g. Vovk & Wang (2020). This results in a FCR controlling procedure. Therefore, existing solutions require the con- struction of valid aggregated p-values. Here, we define a generic aggregation scheme that yields FCP control using the same tools as in Blain et al., 2023. In particular, our proposed construction does not require the result of aggre- gation to be a valid p-value. First, let us define an aggrega- tion procedure:
1 ( = B
{ max b' [B' S.L. JERg
T B'
With T template composed of B' candidate curves that match quantiles of the distribution of aggregated conformal p-values p, obtain the following result:
Proposition 4 (JER control for aggregated conformal p-values), Consider the threshold family defined by t. = Tºd (AB(0)). Then, as B + 400,
Definition 5. Given K models, an aggregation procedure is a function ʃ TR& + IR. that maps a vector of {<))]]]]] statistics to a scalar statistic p.
B
JER B) S Op+0/(I
In practice, since we have m test points, aggregation is per- formed for each test point, i.e.:
Proof. The proof is identical to that of Proposition 3 using the empirical aggregated JER.
Vj [m]]
Pj [w) -'.
Then, inference is performed the vector of aggregated p-values (P......PPp)
The calibrated aggregated threshold family yields valid FCP upper bounds via Corollary 1. We therefore achieve a fully nonparametric aggregation scheme for conformal pre- diction, along with guarantees on the FCP.
For fixed aggregation scheme f we can naturally extend the calibration procedure of the preceding section. Instead of drawing a single B x p matrix of conformal p-values containing Ps E RP for each b E [B], we draw K such matrices. Given is [K] each matrix contains r E RRP for each b E [B] We then perform aggregation: Pb f ((P)) The JER in the aggregated case is defined as:
5. Experiments
Setup. We use 17 OpenML (Vanschoren et al., 2014) datasets from (Grinsztajn et al., 2022). Each dataset is ran- domly split (Nsplit = 30 times) into a train, calibration and test set. The latter is of size m and denoted by Diest- We fit 5 regression models on the training sets': Random For- est (RF) (Breiman, 2001), Multi-Layer Perceptron (MLP) (Hinton, 1990), Support Vector Regression (SVR) (Platt et al., 1999) K-Nearest Neighbors (KNN; Cover & Hart, 1967) and Lasso (Tibshirani, 1996).
(ER(t) = P [jj E [kmma]] Pt) < <t))
We obtain the aggregated template following the same pro- cedure, i.e. drawing K templates and aggregating them. For each 1/ E [B']. the aggregated threshold family is writ- ten:
FCP control. We consider three methods for comparison: classical Split Conformal Prediction, the method proposed by Gazin et al., 2024 to obtain FCP control via DK W-type bounds (Massart, 1990) and the proposed approach. We use 0.1 for all methods. For FCP controlling methods, we set 5 = 0.1 and use SCP with the largest level a' such that FCP a < a.
((T)
5' B'
(TM
T
-
/
B'
kE[K]
We can then write the empirical JER in the aggregated case as:
B
1
1/
JER
T
3je E [Rimall: Pb(j)
T
All experiments were performed using 40 CPUs, Intel(R) Xcon(R) CPU ES-2660 v2 2.20GHz
B'
B
B'
6=11
6

---

False Coverage Proportion Control for Conformal Prediction
For each dataset, we compute for each split the empirical FCP for each model and conformal prediction method. For- mally, for given data set, denoting by C* = ((:))
CV+, the Bonferroni-like construction of Lei et al., 2018 and the proposed method. For the proposed method, we use the harmonic mean as the aggregation scheme. In Ap- pendix C we provide a comparison to other possible ag- gregation schemes (arithmetic mean, geometric mean and quantile aggregation). This comparison shows that the har- monic mean consistently outperformed the other aggrega- tion schemes considered in terms of interval tightness.
it the confidence intervals obtained for the s-th split for given method, the associated empirical FCP is given by:
Σ Ox iED
FCP(C*)
As in the first experiment, compute the FCP coverage of each method and the relative interval length. The left panel of Figure 2 shows that all three methods control the FCP at level 8 = 0.1. While CoJER offers a theoretical guarantee on this control, this is not the case for CV+ and Bonferroni. These two methods likely control the FCP due to excessive conservativeness, as the FCP event is controlled 100% of the time for most datasets using CV+ and Bonferroni inter- section.
m
Then for each dataset, we compute the associated empiri- cal coverage as the proportion of splits for which the FCP control event holds:
[FCC((C)<< <
Naplits
The right panel of Figure 2 shows that CoJER yields the most informative intervals across all datasets. The intervals yielded by the CV+ procedure are ~ 75% larger on average than those of CoJER. The conferroni-intersectiee are 20% larger on average than those of CoJER. Overall, these experiments show that the proposed nonparametric aggregation scheme achieves sharp FCP control while pro- viding tighter intervals that state-of oftte--f-hheeeee the-aar arr methods.
We also compute the interval length of each method for each dataset. We report the relative length to the shortest interval found among all methods, averaged across all splits for each dataset. This allows having a comparable metric for interval informativeness across all datasets.
intervals
The left panel of Figure 1 shows that across all models and datasets, standard Split Conformal does not guarantee FCP control at level a - this is consistent with theory, as Split Conformal prediction only guarantees FCR control. Strikingly, the proportion of splits for which FCP < a for Split Conformal can be as low as 35% for certain models and datasets. Both the proposed method and the method of Gazin et al., 2024 control the FCP as expected. Concretely, this means that for all datasets, the proportion of splits for which FCP < a is indeed superior to 1 - 8.
6. Discussion
In this paper, we have proposed a novel method that allows sharp control of the false coverage proportion in confor- mal prediction. This method builds upon JER control in- troduced by (Blanchard et al., 2020) and on characteriza- tion of the joint distribution of conformal p-values recently established by (Gazin et al., 2024). Our main contribution of this paper with respect to these papers is twofold. First, we build tighter FCP confidence bounds than (Gazin al., 2024) by deriving the shape of the template family from the distribution of conformal p-values itself. Second, we in- troduce an aggregation scheme which yields an integrated method which provides robustness with respect to model- ing choices, and yields tighter intervals than other existing aggregation schemes.
The right panel of Figure shows that SCP yields the short- est intervals in all settings. This is expected, as FCR con- trol is less stringent than FCP control, leading to shorter in- tervals. Among the two FCP controlling methods, the pro- posed method is less conservative than the method of Gazin et al., 2024. On average across all models and datasets, the proposed method yields intervals that are only ~ 15% larger than standard SCP. In worst-case scenarios, the pro- posed method yields intervals ~ 25% larger than SCP, while intervals obtained using the method of Gazin et al., 2024 are 80% larger than SCP intervals. Overall, the proposed method yields sharp FCP control at modest cost in terms of interval length compared to SCP.
The computational cost of this method is comparable to classical SCP. For given sizes of calibration and test sets, sampling conformal p-values from Algorithm I can be done once and for all. Calibration using Algorithm 3 is per- formed via binary search of complexity Oloog (l))) Com- puting the empirical JER of a threshold family using Algo- rithm has a computational complexity of O OBKkmar).
Aggregation. We use the five regression models men- tioned above and consider three aggregation methods for comparison: the method based on the arithmetic mean of p-value functions proposed by Barber et al., 2021 labeled
Additionally, once calibration is performed, the bound of Corollary 1 holds simultaneously for all values of a. In practice, users can try different values of post hoc while
7

---

False Coverage Proportion Control for Conformal Prediction
FCP control
Interval lengths
2.0
1.0
Relative length (vs best)
CoJER
FCP event control
1.8
Gazin et al. SCP
0.8
1.6
0.6
1.4
1.2
0.4
1.0
RF
Lasso
MLP
SVR
KNN
RF
Lasso
MLP
SVR
KNN
Figure 1: Coverage and relative interval length for 5 models. We 17 OpenML (Vanschoren et al., 2014) datasets from (Grinsztajn al., 2022). Each dataset is split (30 times) into train, calibration and test set. We fit 5 regression models on the training sets: Random Forest (RF), Multi-Layer Perceptron (MLP), Support Vector Regression (SVR), K- Nearest Neighbors (KNN) and Lasso. Calibration sets are used to compute SCP intervals and conformal p-values. For each method and dataset, we report the FCP coverage, i.. the proportion of test splits for which the event FCP S a was realized. We also report the interval length, relative the smallest valid interval found among all methods. Notice that standard SCP does not guarantee FCP control level a: for certain datasets and models, FCP event coverage can be as low 30%. Both the proposed approach and Gazin al., 2024 obtain FCP control. However, the proposed approach is much less conservative.
retaining valid FCP bounds without needing to relaunch the complete procedure.
github. Satpsssouui--ooo///J-R--a--- to- gether with the code to reproduce the numerical results of this paper.
Notably, valid p-values are not needed to obtain FCP con- trol. Since this control is direct consequence of JER con- trol aggregated p-values, gregation scheme f. In particular, mean to aggregate p-values leads to valid FCP control, even if the harmonic mean does not always yield valid p-values (Chen et al., 2024).
be obtained for any ag-
Acknowledgments
of the harmonic
This research has received funding from the KARAIB AI chair
the H2020 Research In- frastructures Grant EBRAIN-Health 101058516, and the VITE SANN-23-CE23-000 Grant.
While we have focused on the regression setting for the nu- merical experiments reported in this paper, our proposed method inherits the genericity of SCP and could thus also be applied to classification tasks. This method could also be extended to other uncertainty quantification frameworks such as bootstrap or resampling based methods like the jackknife+ (Barber et al., 2021). Since valid p-values are not needed, characterizing the distribution of statistics quantifying uncertainty is sufficient to apply the proposed method.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Barber, R. F., Candes, E. J., Ramdas, A., and Tibshirani, J. Predictive inference with the jackknifet. Annals of Statistics, 2021.
Another interesting avenue of work is to study the impact of controlling the FCP rather than the FCR for downstream decisions taken using confidence intervals. This type of analysis has been conducted in Vovk & Bendtsen, 2018 8 in classical conformal prediction framework and in Perez- Lebel al., 2024 in the field of model calibration.
Blain, A., Thirion, B., and Neuvial, P. Notip: Non- parametric true discovery proportion control for brain imaging. Neurolmage, 260: 260:119492, 19492, 2022.
An implementation of CoJER is available https://
Blain, A., Thirion, B., Grisel, O., and Neuvial, False
8

---

False Coverage Proportion Control for Conformal Prediction
FCP control
Interval Lengths
3.5
1.0
Relative length (vs best)
CoJER
FCP event control
3.0
CV+
0.8
Bonferroni
2.5
0.6
2.0
1.5
0.4
1.0
CoJER
CV+
Bonferroni
CojER
CV+
Bonferroni
Figure 2: Coverage and relative interval length for aggregation-based methods using 5 models. We 17 OpenML (Vanschoren et al., 2014) datasets from (Grinsztajn et al., 2022). Each dataset is split into train, calibration and test set. We fit 5 regression models on the training sets: Random Forests, Multi-Layer Perceptron, Support Vector Regression, K-Nearest Neighbors and Lasso. Calibration sets are used to compute Split Conformal prediction intervals and conformal p-value functions. These functions are used to compute confidence intervals for the proposed method. For each method and dataset, we report the FCP coverage, i.e. the proportion of test splits for which the event FCP I a was realized. We also report the interval length, relative to the smallest valid interval found among all methods. The proposed method achieves the expected FCP coverage while providing the most informative intervals.
discovery proportion control for aggregated knockoffs. NeurlPS 2023, 2023.
tional Conference Artificial Intelligence and Statis- tics, pp. 1504-1512. PMLR, 2024.
Blanchard, G., Neuvial, P., Roquain, H., et al. Post hoc con- fidence bounds on false positives using reference fami- lies. Annals of Statistics, 48(3):1281-1333, 2020.
Genovese, C. R. and Wasserman, L. Exceedance control of the false discovery proportion. Journal of the American Statistical Association, 101 401(476;:408-44117 2006.
Breiman, L. Random forests. Machine learning, 45:5-32, 2001.
Grinsztajn, L., Oyallon, E., and Varoquaux, G. Why do tree-based models still outperform deep learning on typ- ical tabular data? Advances in neural information pro- cessing systems, 35:507-520, 2022.
Candès, E., Fan, Y., Janson, L., and Lv, J. Panning for gold:' mmodel-'' knockoffs for high dimensional con- trolled variable selection. Journal of the Royal Statisti- cal Society: Series B (Statistical Methodology), 80(3): 551-577, 2018.
Hinton, G. E. Connectionist learning procedures. In Ma- chine learning, pp. 610- Elsevier, 1990.
Lei, J., G'Sell, M., Rinaldo, A., Tibshirani, R. J., and Wasserman, I.. Distribution predictive inference for regression. Journal of the American Statistical Associa- tion, 1094333100944111,, 2018.
Chen, Y., Wang, R. Wang, Y., and Zhu, W. Sub- uniformity of harmonic mean p-values. arXiv preprint arXiv:2405.01366 2024.
Massart, P. The tight constant in the dvoretzky-kiete wolfowitz inequality. The annals of Probability, pp. 1269-1283, 1990.
Cover, T. and Hart, Nearest neighbor pattern classifica- tion. IEEE transactions on information theory, 13(1): 21-27, 1967.
Meinshausen, N., Meier, L., and Bühlmann, P. P-values for high-dimensional regression. Journal of the American Statistical Association, 104(488):161111661 2009.
Durand, G., Blanchard, G., Neuvial, P., and Roquain, E. Post hoc false positive control for structured hypothe- ses. Scandinavian journal of Statistics, 47(4):1114 1148, 2020.
Papadopoulos, H., Proedrou, K., Vovk, V., and Gammer- man, A. Inductive confidence machines for regression. Machine learning: ECML 2002: 13th European con- ference on machine learning Helsinki, Finland, August
Gazin, U., Blanchard, G., and Roquain, E. Transductive conformal inference with adaptive scores. In Interna-

---

False Coverage Proportion Control for Conformal Prediction
19-23, 2002 proceedings 13, pp. 345-356. Springer, 2002.
A. Proofs
Lemma For any threshold famity t, we have
Perez-Lebel, A., Varoquaux, G., Koyejo, S., Doutreligne, M., and Morvan, M. L. Decision from suboptimal clas- sifiers: Regret pre- and post-calibration. Under Review, 2024.
JER (t) (ERBs()==O((1/ =
Proof of Lemina 2. Let VB JER (t) ( (E)R Theorem, we have
Zz(t)
Platt, J. et al. Probabilistic outputs for support vector ma- chines and comparisons to regularized likelihood meth- ods. Advances in large margin classifiers, 10(3):11-74, 1999.
By the Central Limit
(t
Z(t),
Tibshirani, R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological)) 58(1):267-288 1996.
B-+00
where Z(t) is centered Gaussian random variable with
variance (²()) = JER (t)(1 JER (t)). As such, for any M O, we have
Vanschoren, J.. Van Rijn, J. N., Bischl, B., and Torgo, L. Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15(2):99-60, 2014.
P(]ZB(()) M)
P(IZ(t) 2 M).
B-+20
Vovk, V. Transductive conformal predictors. In Artifi- cial Intelligence Applications and Innovations: 9th IFIP WG 12.5 International Conference, AIAI 2013, Paphos, Cyprus, September 30-October 2013, Proceedings 9, pp. 348-360. Springer, 2013.
Since JER. (t) 1, have o²( S 1/4 for any t, so that (t) is stochastically dominated by N (0. 1/4), which does not depend on the threshold family t. As such, we have PP (t) 2 M) = 2I (Z(t) 2 M) S (2M), where & denotes the tail function of the standard normal distribution. Since (xx) tends
Vovk, V. and Bendtsen, C. Conformal predictive decision making. In Conformal and Probabilistic Prediction and Applications, pp. 62-62. PMLR, 2018.
x - +00, we have
proved that Bt (1)).
Proposition 3 (JER control for conformal p-values). Con- sider the threshold family defined by t = ())(())) Then, as B > +o0,
Vovk, V. and Wang, R. Combining p- p-values via averaging. Biometrika, 107(4):791-800) 2020.
Vovk, V., Gammerman, A., and Shafer, G. Algorithmic learning in random world, volume 29. Springer, 2005.
S0+
Proof. We treat the case where t. is well defined for all B, i.e. that there exists a threshold family among T controls the empirical JER for B draws. If this is not the case for some B, then B is set to the null family and the result holds.
We can write:
JER (t) JERn(t) + (JER (t) (E)
-JERR(() + Op(1/VB)
by Lemma 2. Applying the above to = toe yields the desired result since Jt s 8 by definition.
U
B. Algorithms
In this section, we provide Algorithm 2 to compute the em- pirical JER Definition and Algorithm to perform the calibration procedure on conformal p-values.
10

---

False Coverage Proportion Control for Conformal Prediction
Algorithm Computing the Empirical JER. The empir- ical JER is computed for given threshold family and matrix of conformal p-values. This algorithm is similar to Algorithm of (Blain et al., 2022).
We use the setup described in Section 5 - i.e. 0.1, 0.1. Importantly,
first check that the FCP controlled for all types of aggregation by reporting the FCP event cov- erage. This value is expected to be above 1 - = 90%. We also compute the (relative) interval width for each aggrega- tion scheme, averaged across 20 splits.
1: Input: II a matrix of conformal p-values; a threshold family; kmaz the size of the threshold family
2: Output: JER, the empirical JER of threshold family
FCP event
Interval width
3: (B, m) < -II)
4: JER +-
coverage
increase (vs best)
Harmonic mean
94%
0%
5: for b {1...., } do
6: for E (1..... kmaz do
Geometric mean
100%
+24%
7: diff[i]] + [b] t|i // Check JER
Arithmetic mean
100%
+230%
Quantile aggregation
100%
+54%
control at rank 8: end for
Table 1: Comparing four possible aggregation schemes on 17 OpenML datasets. We report the empirical FCP cover- age (for target coverage of 1 8 90%) and relative interval width for harmonic mean, arithmetic mean, geo- metric mean and quantile aggregation (Meinshausen et al., 2009). The FCP is controlled for all four aggregation schemes. Harmonic mean aggregation consistently outper- forms the other aggregation schemes.
9: if (diff < then
+ B // Increment risk if JER control event is violated end if 12: end for Return JER
13:
Algorithm 3 Performing calibration on conformal p- Algorithm 1 to build suitable tem-
values. First, plate and estimate the JER of each candidate threshold fam- ily. Then, we perform calibration to select the least conser- vative possible threshold family that controls the JER at given level 5.
The results presented in Table 1. Coherently with the thcoretical guarantees obtained in Proposition 4, the FCP is controlled for all four aggregation schemes. In terms of interval tightness, harmonic mean aggregation outperforms arithmetic mean, geometric mean and quantile aggregation consistently.
1: Input: 5 the desired coverage; B the number of MC draws for JER estimation; B' the number candidate threshold families
Output: to the calibrated threshold family at level 8 ((B, m)) // Algo.
3: II < sraw_conformal Baatioormalpp_BB.,,__
4: II' +- draw draw_conformal_pe (B', m) // Algo.
5: for b' E (1....,,'"" B'} do
6: Q(6/] + quantiles(II', B/BB)) // Build
template
7: [' < empirical_jer/ee (II Q(6") // Apply
Algorithm to each family
8: end for
max(b E (L...., B'} s.t. JER[6] 8) // 9: 6
Perform calibration
10: to + QBaal
11: Return to
C. Comparison Between Aggregation Schemes
We performed an additional experiment on the 17 OpenML. datasets used in Section 5. In this experiment, we com- pare four possible aggregation schemes: harmonic mean, arithmetic mean, geometric mean and quantile aggregation (Meinshausen al., 2009).


Reviews:
[{"review_id": "RFD6rfAC4B", "reviewer": "Reviewer_DTau", "summary": "This paper addresses the challenge of controlling the False Coverage Proportion (FCP) in split conformal prediction (SCP). While SCP provides computationally efficient confidence intervals, it only guarantees marginal coverage over multiple test points. The authors highlight that in real-world scenarios, where multiple predictions are made simultaneously, the FCP of standard conformal prediction algorithms fluctuates significantly. This work proposes CoJER, a novel Joint Error Rate (JER) control-based method that achieves tight and reliable FCP control using a refined characterization of conformal p-values in a transductive setting. The authors also extend this procedure to provide FCP control under any pre-specified aggregation scheme for using knowledge from multiple prediction models simultaneously.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and insightful comments. Please find our answers to the points raised below.\n\n> The paper is very dense and hard to read. Some toy examples / walk-throughs of the procedure could be quite useful for understanding.The template-building procedure is not described well.\n\nTo improve the clarity of the paper, we will add a pedagogical figure that explains the procedure in two steps in the updated manuscript. The first panel illustrates the concept of JER control with an example of template and of calibrated threshold family. The second panel illustrates the intervals obtained with CoJER using the adjusted risk level $\\widehat{\\alpha} = t_{\\lfloor \\alpha m\\rfloor}$ with $t$ the calibrated threshold family. We will also add pedagogical interpretations for the notions of threshold families and templates.\n\n>The matrix dimensions are mentioned to be n x p, but p is never mentioned before. This is quite confusing. (It might be a typo making, and it should be n x m)\n\nThanks for pointing this out, this is indeed a typo -- the matrix dimensions are $n \\times m$. We have corrected this in the updated version of the manuscript.\n\n>The authors don't theoretically or experimentally explore the relative tightness of the finite sample FCP bounds depending on aggregation mechanism used.\n\nTo address the reviewer's concern, we performed an additional experiment on the 17 openML datasets used in the paper. In this experiment, we compare four possible aggregation schemes: harmonic mean, arithmetic mean, geometric mean and quantile aggregation [1].\n\nWe use the setup described in the main text - i.e. $\\alpha = 0.1, \\delta = 0.1$. Importantly, we first check that the FCP is controlled for all types of aggregation by reporting the FCP event non-coverage as described in the main text. We also compute the (relative) interval width for each aggregation scheme, averaged across 20 splits.\n\n|                                   | Harmonic mean | Geometric mean | Arithmetic mean | Quantile aggregation |\n|-----------------------------------|---------------|----------------|-----------------|----------------------|\n| FCP event coverage                | 94%           | 100%           | 100%            | 100%                 |\n| Interval width increase (vs best) | **0%**            | +24%           | +230%           | +54%                 |\n\n\nCoherently with the theoretical guarantees obtained in Proposition 4, **the FCP is controlled for all four aggregation schemes.** In terms of interval tightness, harmonic mean aggregation outperforms arithmetic mean, geometric mean and quantile aggregation consistently.\n\n### References\n[1] Meinshausen, N., Meier, L., & Bühlmann, P. (2009). P-values for high-dimensional regression. Journal of the American Statistical Association, 104(488), 1671-1681.", "claims_and_evidence": "The claims made in the paper are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The datasets, baselines, and evaluation metrics considered for the problem are appropriate.", "theoretical_claims": "The proofs of the theoretical claims are accurate.", "experimental_designs_or_analyses": "The experimental design and analysis is sound.", "relation_to_broader_scientific_literature": "This work provides a way to control the FCP over a given set of test points with high probability. It also extends the guarantees to aggregate the knowledge from multiple models to provide more efficient conformal intervals. While there has been previous work on both of these topics, this work offers a new perspective by using the p-value interpretation of conformal intervals to better use the rich literature on Joint Error Control using p-values. The algorithms mentioned in the paper improve the applicability of the conformal prediction framework to more real-world scenarios.", "essential_references_not_discussed": "The authors have provided a detailed overview of all relevant related works.", "other_strengths_and_weaknesses": "Strengths\n- The paper addresses an important issue.\n- The JER procedure and the aggregation procedure provide strong theoretical guarantees without any strong assumptions beyond the standard exchangeability assumption.\n- The experimental evidence shows a clear advantage of using the proposed algorithms over the existing state-of-the-art.\n\nWeaknesses\n- The paper is very dense and hard to read. Some toy examples / walk-throughs of the procedure could be quite useful for understanding.\n- The template-building procedure is not described well.\n- The matrix dimensions are mentioned to be n x p, but p is never mentioned before. This is quite confusing. (It might be a typo making, and it should be n x m)\n- The authors don't theoretically or experimentally explore the relative tightness of the finite sample FCP bounds depending on aggregation mechanism used.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please refer to the Weaknesses section above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "obRIxmdUUu", "reviewer": "Reviewer_8Xty", "summary": "This paper examines the limitations of Split Conformal Prediction (SCP) in controlling the False Coverage Proportion (FCP) across multiple predictions. While SCP ensures control over the False Coverage Rate (FCR), it does not provide high-probability guarantees on the actual proportion of non-covered intervals across multiple test points.  To address this gap, the authors propose CoJER, a method specifically designed for FCP control in multi-point prediction settings. By reformulating SCP as a p-value thresholding procedure, they derive conformal p-values within a Joint Error Rate (JER) control framework, avoiding strong parametric assumptions. This results in a more adaptive and sharper bound compared to the prior work of Gazin et al. (2024).   Additionally, the authors extend their method to aggregate conformal prediction intervals across different models, enhancing robustness and reducing sensitivity to specific modeling choices. Their results demonstrate that CoJER effectively controls FCP under any aggregation scheme*, offering a principled approach to multi-point uncertainty quantification.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and insightful comments. Please find our answers to the points raised below.\n\n>However, the evaluation setup raises concerns. Since SCP does not ensure FCP control, it should not be included in interval length comparisons [...] including additional baseline methods or alternative strategies for FCP control, would strengthen the evidence supporting this statement.\n\nTo the best of our knowledge, the only existing approach that explicitly targets FCP control in CP is the work of Gazin et al. We would be glad to include additional baselines if others are brought to our attention, and we welcome any suggestions in that regard.\n\nWe agree that a fair comparison of interval lengths should ideally be restricted to methods that control the false coverage proportion (FCP). We chose to include SCP, despite it not controlling the FCP, to illustrate that CoJER provides substantially stronger coverage guarantees with only a modest increase in interval length.\n\n> The paper lacks a dedicated related work section [...]  other risk-control methodologies should have been explored—particularly conformal risk control [1] and risk-controlling prediction sets [2].\"\n\nFor a new test point $(X_{n+1},Y_{n+1})$, SCP provides a confidence interval $C_{\\alpha}\\left(X_{n+1}\\right)$ with the following guarantee:\n$\\mathbb{P}\\left[Y_{n+1} \\notin C_{\\alpha}\\left(X_{n+1}\\right)\\right] \\leq \\alpha$.\nThe two approaches mentionned by the referee extend SCP as follows:\n- Conformal risk control replaces marginal miscoverage by a general loss $\\ell$ , aiming for the guarantee: $\\mathbb{E} \\left[\\ell(C(X_{n+1}, Y_{n+1})\\right] \\leq \\alpha$ \n- The approach in [2] provide the same guarantees as SCP, in the case of set-valued prediction.\n\nAs such, these approaches still provide *marginal risk control for a single test point*. Our paper focuses on the transductive setting, where $m$ such test points are available. In this setting, while the marginal guarantees could be leveraged to control the False Coverage Rate (FCR), they do not provide FCP contol in probability. Therefore, they are not more relevant than SCP as competitors for our method. We have added text to the section 2.1 \"Split conformal prediction\" of the manuscript to clarify this important point. \n\n\n> Line 62 (Page 2): How do you define \"prediction intervals with a size close to the optimal length\"? What constitutes the optimal length in this context?\n\nIn this context, we meant that CoJER produces prediction intervals with lengths close to those of SCP, while additionally providing formal FCP control—something SCP does not guarantee. We agree that the term \"optimal\" could be misleading and have reformulated this point in the updated version of the paper to avoid ambiguity.\n\n>What is the impact of $\\delta$ on performance? Could setting  to a very low value be beneficial?\n\nFCP is controlled with probability greater than $1 - \\delta$. As $\\delta$ becomes smaller, FCP control becomes increasingly stringent. In terms of performance, this means that decreasing $\\delta$ increases interval width. Setting $\\delta$ to a very low value (e.g. $\\delta = 0.001$) would ultimately lead to a non-informative statement, with FCP control holding with overwhelming probability, for very wide intervals.\n\n>The title is too broad; \"Tight and reliable conformal prediction\" is a common goal in this field and does not clearly distinguish the contribution.\n\nWe agree that the title is too vague and doesn't clearly distinguish the contribution. We propose to rename the paper **False Coverage Proportion control for Conformal Prediction**. To this end, we have sent a message to the PCs of the conference.\n\n>Lacks sufficient intuitive explanations for key steps, with some missing motivations (e.g., the choice of the harmonic mean for the aggregation scheme).\n\nRegarding the choice of the harmonic mean, we have performed an additional experiment to compare four possible choices: harmonic mean, arithmetic mean, geometric mean and quantile aggregation. Please see our answer to reviewer DTau for all experimental details.\n\n>Limited comparison, as it considers only two methods, one of which does not ensure FCP control, restricting the strength of the empirical evaluation.\n\nPlease see our answer above: to the best of our knowledge, the work of Gazin et al. is the only existing approach that explicitly targets FCP control in CP. We would be glad to include additional baselines if others are brought to our attention, and we welcome any suggestions in that regard. \n\n> Could you provide relevant examples of multiple test point settings?\n\nA common example of multiple test point setting in CP are real-time systems with batches: In applications like fraud detection or content recommendation, models may process incoming data in batches for efficiency. Another common setup is the offline evaluation of predictive models: before deploying a model, it is often evaluated on a fixed test set.", "claims_and_evidence": "The claim that \"CoJER yields shorter intervals than the state-of-the-art method for FCP control and only slightly larger intervals than standard SCP.\" is empirically validated using benchmark datasets (OpenML).\n\nHowever, the evaluation setup raises concerns. Since SCP does not ensure FCP control, it should not be included in interval length comparisons, as its intervals may be shorter at the expense of failing to meet the desired coverage guarantees. Consequently, the evaluation effectively compares CoJER against a single state-of-the-art method, limiting the strength of the claim. A more comprehensive comparison, potentially including additional baseline methods or alternative strategies for FCP control, would strengthen the evidence supporting this statement.", "methods_and_evaluation_criteria": "Using 17 OpenML datasets provides a solid foundation for evaluation. Additionally, reporting the relative interval length compared to the shortest interval across all methods, averaged over multiple splits per dataset, offers a clear basis for comparison. However, incorporating domain-specific real-world datasets from fields such as medicine or finance would have further strengthened the study, given the stated relevance of multi-point prediction in these areas.", "theoretical_claims": "The proofs for Lemma 2, Proposition 2, and Proposition 3 appear sound.", "experimental_designs_or_analyses": "I verified the validity of the experimental design. They used 17 OpenML datasets, performed 30 dataset splits, and tested five different regression models, providing a reasonable basis for confidence in the results. The choice of $ \\alpha = \\delta = 0.1 $ is appropriate based on the literature.", "relation_to_broader_scientific_literature": "The paper lacks a dedicated related work section, which would help contextualize previous methods and enhance understanding. While the connection to SCP and the approach by Gazin et al. is well-established, a broader comparison with other risk-control methodologies should have been explored—particularly *conformal risk control* [1] and *risk-controlling prediction sets* [2]. Integrating these perspectives would provide a more comprehensive view of how CoJER fits within the broader landscape of uncertainty quantification and risk control.\n\n\n[1] Angelopoulos, Anastasios N., Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. 2022. “Conformal Risk Control.”\n[2] Bates, Stephen, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. 2021. “Distribution-Free, Risk-Controlling Prediction Sets.”", "essential_references_not_discussed": "The only two works essential for understanding this paper are Gazin et al. [3] and the research on joint error rate [4], both of which are extensively discussed.", "other_strengths_and_weaknesses": "### *Strengths:* \n- Solid mathematical foundation with a well-structured flow of ideas.  \n- Strong empirical results that clearly demonstrate the superiority of the proposed method.  \n\n### *Weaknesses:*  \n- The title is too broad; *\"Tight and reliable conformal prediction\"* is a common goal in this field and does not clearly distinguish the contribution.  \n- Lacks sufficient intuitive explanations for key steps, with some missing motivations (e.g., the choice of the harmonic mean for the aggregation scheme).  \n- Limited comparison, as it considers only two methods, one of which does not ensure FCP control, restricting the strength of the empirical evaluation.", "other_comments_or_suggestions": "The title of the paper is too broad, as achieving tight and reliable confidence intervals is a fundamental goal of any conformal prediction method. A more precise title should reference *CoJER* and its role in *FCP control* to better reflect the paper's specific contribution.  \n\n### *Questions:*\n- *Line 62 (Page 2):* How do you define *\"prediction intervals with a size close to the optimal length\"*? What constitutes the optimal length in this context?  \n- What is the impact of $\\delta $ on performance? Could setting $\\delta $ to a very low value be beneficial?  \n\n### *Minor Suggestions:* \n- Could you provide relevant examples of multiple test point settings?  \n- Most equations lack numbering. While this improves clarity, adding numbers would enhance readability and referencing.  \n- *Line 225:* The reference to *\"Appendix B\"* is not clickable.  \n- The legends in *Figures 1 and 2* would be more readable if placed above the plots.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1AYiMJ4aMt", "reviewer": "Reviewer_FJKa", "summary": "The paper introduces CoJER (Conformal Joint Error Rate), a novel method designed to improve the reliability of split conformal prediction (SCP) by controlling the False Coverage Proportion (FCP). While traditional SCP ensures marginal coverage over multiple test points, it does not guarantee that the proportion of test points outside their confidence intervals remains controlled with high probability. This limitation is crucial in real-world settings, where multiple predictions are made simultaneously, such as in healthcare and finance.\n\nThe authors propose CoJER, which leverages conformal p-values and Joint Error Rate (JER) control to obtain tight FCP bounds. Unlike prior work (e.g., Gazin et al., 2024), which uses fully parametric bounds, CoJER provides a nonparametric calibration procedure that ensures sharper confidence intervals. The method is extended to model aggregation, allowing for robustness across different predictive models. Extensive experiments on 17 OpenML datasets demonstrate that CoJER achieves better FCP control while maintaining shorter confidence intervals than existing methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and insightful comments. Please find our answers to the points raised below.\n\n>However, one potential limitation is that the paper primarily evaluates performance on tabular regression datasets. The applicability of CoJER to classification problems or high-dimensional deep learning models is not explored in depth.\n\nWhile our experiments focus on tabular regression datasets, we would like to emphasize that **CoJER is fundamentally agnostic to the specific predictive setting** as long as the transductive assumption holds (i.e., access to many test points). This stems from the fact that CoJER operates purely on conformal p-values inherited from the CP framework. As such, the method is applicable to classification tasks and other model families, including high-dimensional deep learning models, provided conformal $p$-values are available.\n\n\n> No ablation studies to test different parameter choices. \n\nThe main parameter of CoJER is the aggregation function. We have performed an additional experiment to compare four possible choices: harmonic mean, arithmetic mean, geometric mean and quantile aggregation. In this setting, harmonic mean aggregation outperforms arithmetic mean, geometric mean and quantile aggregation consistently. Please see our answer to reviewer DTau for all experimental details.\n\n> Monte Carlo estimation reliance—more discussion on computational trade-offs is needed. How does CoJER perform on classification problems?\n\nIn our setting, we see the reliance on Monte Carlo estimation as a strength rather than a weakness, since it allows JER control with arbitrary precision at a small computation cost. Sampling from $P_{n,m}$ is done in $O(n + m)$. Moreover, this is done once and for all for given values of $n$ and $m$.\n\n> What is the computational overhead compared to standard SCP? \n\nIn the transductive setting considered in this paper with $m$ test points, both SCP and CoJER require $O(m n \\log(n))$ to obtain $m$ conformal $p$-values (each of them requires sorting $n$ conformity scores). For FCP control with $B$ MC samples, CoJER additionally requires $O(B m \\log(m))$ for template generation and sorting, and $O(B m (\\log(m) + \\log(B)))$ for calibration using binary search.\n\nTherefore, neglecting the logarithmic terms for simplicity, the complexity of standard SCP is $O(m n)$ and that of CoJER is $O(m (n+B))$, where $B$ is the number of MC samples (which is user-defined does not depend on $n$ or $m$). In particular, the complexities are of the same order if $B$ is chosen to be of the same order as $n$. \n\n> The paper provides a comprehensive review of prior work, but could benefit from including: Adaptive Conformal Inference: Angelopoulos et al. (2023) propose an adaptive method for controlling conformal prediction widths, which may be relevant for understanding how CoJER adapts to different datasets. Resampling-based Conformal Methods: Recent work on Jackknife+ (Barber et al., 2021) explores alternatives to split conformal prediction, which could be useful for comparison.\n\nBoth of these approaches provide marginal risk control, i.e. *for a single test point*. As argued above in our reply to reviewer 8Xty, such approaches could be leveraged to control the False Coverage Rate (FCR) but they do not provide FCP control in probability. Therefore, they are not more relevant than SCP as competitors for our method.", "claims_and_evidence": "The claims made in the paper are generally well-supported by theoretical and empirical evidence. The theoretical guarantees for FCP control are rigorously derived, and the connection between conformal p-values, JER control, and FCP bounds is well-established. The proposed method is thoroughly compared to existing SCP-based approaches, particularly the parametric method of Gazin et al. (2024), which is known to be overly conservative.\n\nEmpirically, the extensive evaluation across multiple datasets and models confirms that CoJER:\n\nAchieves valid FCP control while standard SCP fails.\nProduces shorter confidence intervals compared to existing FCP-controlling methods.\nRemains robust across different model choices through the aggregation framework.\nHowever, one potential limitation is that the paper primarily evaluates performance on tabular regression datasets. The applicability of CoJER to classification problems or high-dimensional deep learning models is not explored in depth.", "methods_and_evaluation_criteria": "Yes, the methodology and evaluation criteria are well-aligned with the problem. The paper focuses on realistic tabular prediction tasks where ensuring tight and reliable confidence intervals is critical. The choice of OpenML datasets provides a diverse set of benchmarks, making the results more generalizable.\n\nThe evaluation metrics—FCP control, interval length, and empirical coverage rates—are appropriate for assessing the effectiveness of the proposed method. However, additional benchmarks on structured datasets (e.g., time-series, NLP tasks) could further strengthen the paper’s claims about general applicability.", "theoretical_claims": "The proofs presented in Sections 3 and 4 appear to be mathematically sound. The derivation of JER-based FCP bounds follows from standard techniques in conformal inference and multiple testing. Specifically:\n\nProposition 1 (characterization of the joint distribution of conformal p-values) is correctly cited from Gazin et al. (2024).\nProposition 2 (link between FCP and JER) follows logically from Lemma 1 and prior work on multiple hypothesis testing.\nProposition 3 & 4 (nonparametric JER control) leverage Monte Carlo approximations, which are empirically validated.\nI did not find any major issues in the proofs, but a more detailed discussion on the asymptotic behavior of CoJER for large-scale datasets would be beneficial.", "experimental_designs_or_analyses": "The experimental design is robust and methodologically sound:\n\nMultiple datasets: The evaluation on 17 OpenML datasets ensures that results are not dataset-specific.\nMultiple models: The experiments consider Random Forest, MLP, SVR, KNN, and Lasso, providing insights into method robustness across different modeling paradigms.\nMultiple baselines: The comparison includes standard SCP, the Gazin et al. (2024) method, and CoJER, making it comprehensive.\nOne area for improvement is the lack of ablation studies on:\n\nThe effect of different choices of transformation functions in CoJER.\nThe impact of varying JER thresholds on FCP control.\nA deeper exploration of these factors could provide more insights into when and why CoJER outperforms existing methods.", "relation_to_broader_scientific_literature": "This paper builds upon two key strands of research:\n\nConformal Prediction: The work extends Split Conformal Prediction (Lei et al., 2018; Vovk et al., 2005) to handle False Coverage Proportion (FCP) control.\nMultiple Testing & JER Control: The approach adapts Joint Error Rate control (Blanchard et al., 2020) to derive nonparametric FCP bounds.\nThe primary novelty lies in:\n\nReformulating FCP control using conformal p-values and JER-based calibration.\nProviding tighter, nonparametric bounds compared to prior parametric approaches.\nExtending conformal aggregation techniques to improve robustness across models.", "essential_references_not_discussed": "The paper provides a comprehensive review of prior work, but could benefit from including:\n\nAdaptive Conformal Inference: Angelopoulos et al. (2023) propose an adaptive method for controlling conformal prediction widths, which may be relevant for understanding how CoJER adapts to different datasets.\nResampling-based Conformal Methods: Recent work on Jackknife+ (Barber et al., 2021) explores alternatives to split conformal prediction, which could be useful for comparison.", "other_strengths_and_weaknesses": "Strengths:\nWell-motivated problem: FCP control is a crucial extension to standard SCP.\nMathematically rigorous: The derivations are well-grounded in statistical theory.\nEfficient and practical: CoJER remains computationally comparable to SCP.\nWeaknesses:\nLimited exploration of classification problems.\nNo ablation studies to test different parameter choices.\nMonte Carlo estimation reliance—more discussion on computational trade-offs is needed.", "other_comments_or_suggestions": "Figure 1 & 2: Labels could be clearer—mentioning \"Relative Interval Length\" explicitly in the y-axis would help.\nSection 5: Define \"$$\\delta$$\" earlier to improve readability.", "questions_for_authors": "How does CoJER perform on classification problems?\nWhat is the computational overhead compared to standard SCP?\nCan the method be extended to structured prediction tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eIKej1rh7T", "reviewer": "Reviewer_ExJ2", "summary": "The authors propose using the Joint Error Control (JER) framework of Blanchard et al. (2020) to control the false coverage proportion (FCP) across multiple conformal prediction intervals. This approach leverages the exact joint distribution of conformal $p$-values derived in Gavin et al. (2024). They introduce a specific instantiation of the JER method, selecting a particular \"template\" and \"threshold function,\" which they argue is optimal or tight, though this claim is not formally proven. Additionally, they propose a method for aggregating conformal $p$-values and intervals while ensuring that both the FCP and JER remain controlled. In experiments, the proposed methods outperform uncorrected approaches and existing, more conservative, methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and insightful comments. Please find our answers to the points raised below.\n\n>In Lemma 1, how are ( p_1, \\dots, p_m ) defined? Is ( p_j ) equal to ( P(Y_j) ), where ( P(\\cdot) ) represents the p-value function?\n\nFor each $j \\in [[m]]$, we define\n$p_j := \\frac{1}{n+1} \\left(1 + \\sum_{i=1}^n \\mathbf{1}[S_j \\le S_i]\\right)$ as per Definition 1. We have added this to Lemma 1 in the updated version of the manuscript.\n\n> The concepts of threshold families and templates may be abstract for those unfamiliar with this literature. Providing some context on why these notions are introduced and their practical applications could be helpful.\n\nWe will add a pedagogical figure that explains the procedure in two steps in the updated manuscript -- we detailed this in our answer to reviewer DTau. In a nutshell, the threshold family can be viewed as a (vector) parameter, and templates are sets of candidate parameters, from which a candidate ensuring JER control (and hence FCP coverage) is chosen by the calibration algorithm.\n\n**Regarding the \"parametric\" nature of the Gazin et al. bound**, we agree that the use of \"parametric\" can be confusing here. Indeed, the paper of Gazin et al. (2024) considers distribution-free conformal prediction settings. We wanted to emphasize that the **confidence envelope of $\\widehat{F}_m$ they propose has a fixed shape**. Theorem 2.3 of Gazin et al. (2024) is a Dvoretzky–Kiefer–Wolfowitz (DKW) type inequality, proved to hold for the specific (known) dependence between conformal $p$-values. As noted in that paper (see Remark 2.6 therein), this approach can be conservative, which can be addressed by the JER calibration framework. CoJER is fully non-parametric in the sense that the shape of the tempate is itself derived from the joint distribution of $p$-values, and not chosen *a priori*. Our experimental results show that CoJER is much less conservative than the approach of Gazin et al. (2024) while preserving FCP control.\n\n**Explicit construction of the confidence intervals:** we thank the reviewer for pointing this out. Once Algorithm 3 is run, we obtain a JER controlling family $t$. Corollary 1 states that, with probability at least $1-\\delta$, $\\forall \\alpha \\in[0,1], \\quad \\operatorname{FCP}(\\mathcal{C}(\\alpha)) \\leq \\frac{j(\\alpha, \\delta)}{m}$ with $j(\\alpha, \\delta) = \\min \\{j \\in [[m]] : \\alpha \\leq t_j\\}$. In turn, to obtain FCP control at level $\\alpha$ with probability $\\geq 1 - \\delta$ we choose $\\widehat{\\alpha} = t_{\\lfloor \\alpha m\\rfloor}$. Therefore, the FCP-controlling intervals can be explicitly written $\\mathbf{\\mathcal{C}(\\widehat{\\alpha})} = (C_{i, \\widehat{\\alpha}})_{i \\in [[m]]}$ with $C_{i, \\widehat{\\alpha}} = \\left[\\hat{\\mu}\\left(X_{n+i}\\right) \\pm S_{(\\lceil(n+1)(1-\\widehat{\\alpha})\\rceil)}\\right]$. The confidence intervals only depend on the threshold family via $\\widehat{\\alpha}$. We have added this explanation to the updated version of the manuscript.\n\n**Template optimality**: while our experiments unambiguously show that CoJER leads to tighter FCP control than the bounds of Gazin et al, we currently do not have theoretical support for this result. In fact, even the formal definition of an optimal shape is not trivial, and we have left this exciting perspective for future work. \n\n> Does the model aggregation scheme preserve the coverage properties of CP? Does this follow from JER control?\n\nCoJER does not ensure a marginal coverage guarantee for each test observation when $m>1$. Indeed, we argue that this type of guarantee is not interpretable in the transductive case considered here. Instead, CoJER offers a strong probabilistic guarantee (FCP control) over the entire set of $m$ test observations.  However, note that for a single observation ($m=1$), FCP control is equivalent to the marginal coverage guarantee offered by SCP.\n\n**Significance level and comparison to the Benjamini-Hochberg procedure**: indeed, the reviewer's intuition is correct. CoJER builds a JER controlling family $t$ and outputs an adjusted level $\\widehat{\\alpha} = t_{\\lfloor \\alpha m\\rfloor}$ for which the confidence intervals $C(\\widehat{\\alpha})$ controls the FCP at level $\\alpha$. This is markedly different from the Benjamini-Hochberg (BH) procedure. First, BH outputs a **rejection set** for which there is a statistical guarantee. In the transductive setting of conformal prediction, we are interested in **obtaining a guarantee for all test points simultaneously** and not only for a certain subset. This renders BH and other similar methods unapplicable in this context. Second, BH offers a guarantee on the **expected** proportion of False Discoveries and not in probability.  \n\n**Title**: we agree that the title is too vague and doesn't clearly distinguish the contribution. We propose to rename the paper **False Coverage Proportion control for Conformal Prediction**. To this end, we have sent a message to the PCs of the conference.", "claims_and_evidence": "The authors claim that their approach is expected to be tighter than that of Gazin et al. (2024). Why is this the case? Can you provide a formal result? \n\nAdditionally, Section 1 states that “while the approach of Gazin et al. (2024) yields valid FCP bounds, they are fully parametric, which can entail conservativeness, as discussed by the authors.” However, upon reviewing their paper, they also consider distribution-free conformal prediction settings, and I do not see any explicit dependence on parametric assumptions. Can the authors clarify what they mean by \"parametric\" in this context and explain why they consider the approach of Gazin et al. (2024) to be conservative?", "methods_and_evaluation_criteria": "NA", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experiments appear sound and comprehensive.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "### Strengths\n\n- The application of the Joint Error Control (JER) framework of Blanchard et al. (2020) to conformal prediction is novel\n- The proposed aggregation scheme for conformal $p$-values and intervals that ensures control of bot FCP and JER is novel and practically useful. \n\n\n### Weaknesses\n\n- It is not explained how calibrated conformal prediction intervals are derived from the calibrated $p$-values and the threshold family. How should the threshold family output by Algorithm 3 be used to construct conformal intervals with JER control? Without explicitly defining these intervals, the utility of Proposition 3 is unclear.  Am I supposed to apply Corollary 1 with the adjusted interval? If so, this interval does not appear to be explicitly defined.\n\n\n- The writing is somewhat unclear, and the paper could benefit from additional intuition and background on multiple testing methods. For instance, the motivation for introducing threshold families and templates, as well as their utility, may not be immediately clear to readers.  \n\n \n- The theoretical results primarily build on or modify existing work, and some key formal results are missing. Specifically:  \n  1. There are no optimality properties established for the template proposed for JER control in Section 3.3.  \n  2. There are no formal results justifying the claims of tightness and sharpness of the proposed intervals compared to existing methods (e.g., Gazin et al., 2024).  \n  3. There are no formal or informal results demonstrating that the model aggregation method preserves the coverage properties, although JER control is established.  If marginal coverage follows from JER control this should be stated.", "other_comments_or_suggestions": "- In Lemma 1, how are \\( p_1, \\dots, p_m \\) defined? Is \\( p_j \\) equal to \\( P(Y_j) \\), where \\( P(\\cdot) \\) represents the p-value function? \n\n- The concepts of threshold families and templates may be abstract for those unfamiliar with this literature. Providing some context on why these notions are introduced and their practical applications could be helpful. \n\n-  To facilitate broader adoption and improve accessibility within the CP community, it might be useful to present a less abstract version and intuition of the algorithm, e.g., introduce the method for a simple template and threshold function. Is the general idea to decrease the rejection thresholds of the p-values monotonically, thereby effectively lowering the significance (alpha) levels of the CP intervals? How does this procedure compare to and differ from the Benjamini-Hochberg method, which may be more familiar to readers?\n\n- For the template constructed using Monte Carlo simulation described above in Proposition 3, is this template optimal in power when the number of MC replicates ($B$) approaches infinity? It would be beneficial to formally define the optimal template function, as the informal statement that it must \"match the shape of the distribution of the $p$-values under exchangeability\" is unclear. Does the proposed template approximate such an oracle template? Formal results on the optimality of the procedure or an oracle variant of it would be appreciated.\n\n\n- Does the model aggregation scheme preserve the coverage properties of CP? Does this follow from JER control?\n\n- The title does not seem appropriate, as there are no tightness guarantees and \"reliability\" is vague and CP is arguably reliable as is. The title should reflect the actual contributions of the paper, e.g.,  something like \"Controlling the FCP/JER of CP\".", "questions_for_authors": "- Once we have obtained the calibrated p-values using Algorithm 3, how do we derive the adjusted conformal prediction intervals? Are the sequence of \\( t \\)'s the new alpha values for the corresponding sequence of prediction intervals? Does Algorithm 3 provide a calibrated p-value function that can be used to construct the new interval following the approach in Definition 2? It would be helpful to include an explicit end-to-end algorithm for constructing the calibrated intervals.\n\n\n- In proposition 2, does the definition of the interval C(alpha) depend on the threshold family? In Section 3.3, it sounds like the idea of the proposed method is to find a threshold family whose JER is controlled at a desired level. However, as mentioned above, it is not clear how this threshold family is mapped to new intervals.\n\n-  It would be beneficial to present a formal result in Section 3.3 on the JER control of the proposed method by combining Proposition 3 with Corollary 1. Specifically, for the method proposed by the authors, what is the value or order of \\( j(\\alpha, \\delta) \\)? How does the bound behave when \\( \\delta = 1/\\sqrt{n} \\) or \\( \\delta = 1/n \\)?\n\n# Post review\nI have raised my score from a 2 to a 3 and am positive of the work.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The overall recommendations are: 3, 4, 3, 3

Paper summary: this paper proposes a method CoJER to control FCP (false coverage portion) by controlling JER (joint error rate), so that it concentrates around FCR (false coverage rate). The underlying JER controlling framework is to characterize the distribution of p-values in a transductive setting.

Reviewers agree that this paper considers an important and well-motivated problem. The proposed method has strong theoretical and empirical results. Every reviewer gives positive ratings to support accepting this paper.
Decision: Accept (poster)