Class conditional conformal prediction for multiple inputs by p-value aggregation
Authors: Jean-Baptiste Fermanian
Mohamed Hebiri
Joseph Salmon
Venue: NeurIPS 2025 poster ()
Keywords: Conformal Prediction
multiclass classification
aggregation
conformal p-value
Pl@ntnet
Primary area: general_machine_learning
TL;DR: We aggregate the conformal p-values of multiple inputs of the same instance to get a class conditionnal conformal set.

Abstract: Conformal prediction methods are statistical tools designed to quantify uncertainty and generate predictive sets with guaranteed coverage probabilities. This work introduces an innovative refinement to these methods for classification tasks, specifically tailored for scenarios where multiple observations (multi-inputs) of a single instance are available at prediction time. Our approach is particularly motivated by applications in citizen science, where multiple images of the same plant or animal are captured by individuals.
Our method integrates the information from each observation into conformal prediction, enabling a reduction in the size of the predicted label set while preserving the required class-conditional coverage guarantee. The approach is based on the aggregation of conformal p-values computed from each observation of a multi-input. 
By exploiting the exact distribution of these p-values, we propose a general aggregation framework using an abstract scoring function, encompassing many classical statistical tools. Knowledge of this distribution also enables refined versions of standard strategies, such as majority voting.
We evaluate our method on simulated and real data, with a particular focus on Pl@ntNet, a prominent citizen science platform that facilitates the collection and identification of plant species through user-submitted images.

Class conditional conformal prediction for multiple inputs by p-value aggregation
Jean-Baptiste Fermanian IMAG, IROKO, Univ Montpellier, Inria, CNRS, Montpellier, France jean-baptiste j fermanian@inria.ff
Mohamed Hebiri
LAMA, Université Gustave Eiffel, Paris, France
Mohamed www.ieeeeeibeeiiiiiieeeeee
Joseph Salmon
IMAG, IROKO, Univ Montpellier, Inria, CNRS,
Montpellier, France joseph. salmon@inria.ff
Abstract
Conformal prediction methods are statistical tools designed quantify uncertainty and generate predictive sets with guaranteed coverage probabilities. This work introduces an innovative refinement to these methods for classification tasks, specif- ically tailored for scenarios where multiple observations (multi-inputs) of single instance are available at prediction time. Our approach is particularly motivated by applications in citizen science, where multiple images of the same plant animal are captured by individuals. Our method integrates the information from each observation into conformal prediction, enabling reduction in the size of the predicted label set while preserving the required class-conditional coverage guarantee. The approach is based on the aggregation of conformal p-values com- puted from each observation of multi-input. By exploiting the exact distribution of these p-values, propose general aggregation framework using abstract scoring function, encompassing many classical statistical tools. Knowledge of this distribution also enables refined versions of standard strategies, such majority voting. We evaluate our method on simulated and real data, with particular focus Pl ntNet, prominent citizen science platform that facilitates the collection and identification of plant species through user-submitted images.
Introduction
As modern algorithms and the data they process become increasingly complex, the importance of quantifying their uncertainty has grown significantly. One systematic method to meet this requirement (split) conformal prediction, which leverages scoring function and labeled calibration data to generate set of potential outputs for new instance (Vovk et al., 2005). For the users, the size of this prediction set is intended to reflect the algorithm's 's uncertainty. This approach requires the assumption exchangeable, (i.e., invariant by permutations)
that the calibration data and new observations get a prescribed coverage. In classification tasks, the conformal set usually comprises a subset of labels that, with a high degree of prescribed confidence, includes the true label of the new observation (see Angelopoulos et al., 2023; Fontana et al., 2023 for overviews of conformal prediction and Chzhen al., 2021 I for set valued classification).
In this work, we focus on a setting where each input to be classified (at testing) consists multiple observations, each of which can be independently processed prior aggregation. This methodology commonly employed in machine learning-drit driven citizen science applications, such as @t@ntNet
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(Joly et al., 2014), where users are prompted to submit one or more images of plant for identification. Similar multi-input samples are also prevalent iNaturalist (Van Hom et al., 2018) or in eBird (Sullivan et al., 2009), the latter managing multi-input sounds of birds, rather than images. Although multiple images may be submitted, algorithms often process them independently and only aggregate the resulting (softmax) predictions. A naive conformal prediction step based on such aggregated scores severely disrupts the exchangeability. This results in excessively large prediction sets, rendering them impractical (see Appendix B).
Simultaneously, we aim class conditional coverage calling which would need a sufficiently large number of calibration examples for each class to reliably estimate how aggregation behaves. In datasets with highly imbalanced class distribution, such as those commonly found in citizen science datasets with many underrepresented classes (see, for example, Garein et al., 2021), achieving this requirement is not feasible.
To overcome this limitation, we propose to perform conformal prediction at the level of individual observations, and to aggregate the resulting conformal prediction sets. We frame this problem as one of p-value aggregation, where the p-values are the conformal p-values computed for each individual observation. Using recent results on the distribution of these p-values, we develop general framework for aggregating them using score functions, which encompasses classical techniques such as p-value correction, majority voting, or template matching.
Setting. We consider the scenario of multi-class classification with K classes. We have at disposal an already trained classifier A X - [0, 1| a calibration set of n labeled points {(X,, r:) and multiple observations {Xn+1) 1m of the same item from the (unknown) class Yn+1- Our objective is to build a set C., depending on the classifier, the calibration set, and the new points satisfying for a fixed (0, 1) and for all y [K]]
Ply C. (A. ((2, YODL (Xn+s)f=a) Yn-1 2
To get an informative set, C, is intended be small as possible.
Notation. For two integers S m. we denote [n mm] == ............... and [m] [1 nl We set s :: [K] - R to be a score function and S. = s(x,, Y,) for i E [m]]] are the calibration scores. For label y [K], we denote ny Hi E [[>]]] Y, y} the number of calibration points of class y (supposed fixed in most of our results) and SV ...... SW, the scores of these points. We denote So+i(y) == ((nnti,)) y) the score of a label at the test point Xn+i for i E [m].
Contributions. We develop a framework for constructing conformal prediction set that is class conditionally valid (1), by aggregating multiple observations from the same class. This approach can be viewed as stemming from tests that assess the exchangcability between the observations and the points within each class. It has been made possible through the adaptation of recent findings on the joint distribution of conformal p-values, as discussed (Gazin et al., 2023). Our contributions are summarized below:
The prediction set is built from class conditional conformal p-values for which we give the exact distribution. Notably, our construction explicitly handles exchangeability and, in particular, ties between the conformal scores-a frequent occurrence with softmax outputs from neural networks, yet often overlooked in the literature. This probabilistic result serves as a key ingredient in the contributions that follow (Section 3).
2. We relate our method to the prediction set aggregation framework of Gasparin and Ramdas (2024), and propose an enhancement tailored to the setting where the aggregated sets are class-conditional conformal sets constructed from shared calibration set (Section 4).
3. We propose
efficient way to aggregate class-conditional conformal p- p-values by constructing a region of reject based on the construction of a score u on the p-values. (Section 5).
4. We test our methods on classification on synthetic dataset of mixture of Gaussian distri- butions and on LifeCLEF Plant Identification Task 2015 (Goëau et al., 2015). In all the experiments, the classifiers are neural networks with softmax output (Section 6).

---

2 Background and related works
2.1 Overview on classical conformal prediction
Conformal Prediction (CP) is framework introduced by Vovk et al. (2005) to construct distribution- free prediction sets quantifying the uncertainty in the predictions of an algorithm. this context, E (0. 1) represents the significance level, and 1 denotes the targeted coverage probability. Formally, given a calibration set f(X,, helll of points in X' x [K] and a trained classifier, a CP method constructs for a new point Xx+1 E A' a marginally valid set Ca. i.e. set containing the unobserved outcome ,, E [A] with high probability:
P (x( 21
The idea behind CP methods is to first build non-conformity scores S; := s(x,, Y;) on the calibration set, for a given score function s X x [K] - I intended to quantify the error of the algorithm at given point. Many score functions have been considered to catch different type of information, see egg, Angelopoulos et al. (2023) for a review. A common choice in classification is to use (one minus) the estimated probability of the class (e.g.. the softmax output of neural network). In this paper, we will focus on the case where an already trained algorithm is provided, which corresponds to the case of split CP. Then, for a score function 8, the prediction set
(x)x) y E [K]]
v) < $(ra-a)(a+1)))
is marginally valid (as defined in Eq. (2)) where S(n)) is the k-th smallest score of Si...., Sn, co (Vovk et al., 2005).
In certain applications, it is desirable to construct prediction sets that satisfy a class conditional coverage guarantee; that is, for every class y, the following condition holds:
P(y E C. (X.+1) Y = v) 2 1 -
While this condition implies marginal coverage, the converse does not hold. In scenarios with highly imbalanced heterogeneous classes, marginal split CP can lead prediction sets with significant variability in class-wise coverage (Sadinle et al., 2019; Ding et al., 2023). To enforce conditional coverage as specified in (4), natural approach is to apply split CP separately for each class (Vovk et al., 2005). The resulting prediction set is given by
Cad (x) = {u= s(x,8) S S (((1-a)(n++])
(5)
which is conditionally valid, with {5 [e[), being the scores of the ny points with label in the calibration set. A limit of this approach is that it requires to have at least 1 points for each class (Ding et al., 2023). Otherwise, such class would always be predicted which can lead to uninformative prediction set.
2.2 P-value point of view of conformal prediction
The conformal set (3) can be interpreted as the outcome of family of two-sample tests, each assess- ing-for every y E [ whethher ((xn+1.) y) is exchangeable with the scores of the calibration. The predicted classes y E Ca are those not rejected by sample of size and of size n, Wilcoxon, 1945; Mann and Whitney, 1947). One can associate a so-called split conformal p p-value (Vovk et al., 2005) to split CP, defined as
Manooooo-Man-Whinnne test (between
(v)
1
Σ 1=1
18(X S S.) +
(6)
+
such that the condition s(Xn+1, 8) S S((a-a)(x+t)) in Eq. (3) is equivalent to Px+1 > For the true label Yn+1, assuming the scores have no ties, the p-value of the true label p(Yn+1) follows a uniform distribution over (i/(n- + 1) i [n- + 1]} This distribution is explicit and independent of the underlying data distribution, which allows for effective identification of unlikely candidate labels. The construction of this distribution-free quantity forms the basis of conformal prediction (see Barber and Tibshirani, 2025 for recent p p-value point of view).
3

---

Similarly, for class-condie conditional prediction sets (5), class- cladscconiiiional p-value can be defined as (v) Ny+I |> 1(s(Xn-1..) y) I { 1]. As in the marginal case, this set can be inter- preted as arising from a test of exchangeability between the test score and the calibration scores for the corresponding class.
2.3 Joint control of the p-values.
In our setting, where multiple images
with the same label Yn+1 are observed, our
approach relies on constructing vectors of conformal p-values pyy) = (pi(y), (v),... Pm (3)),, where
Pj py()) the class-conditional p-value associated with X,+; for class y. One question to find set E C 00, m that contains the vector p(Y=+1) with high probability:
P(p(Y E F. 2 a
(7)
For m = 1, conformal prediction chooses E = [a, 1], but, for m larger, wide range of choices become possible. This question has been extensively studied across various contexts and for multiple purposes. In the realm of multiple testing, numerous methods have been developed to control the false discovery proportion (FDP), often leading to criteria interpretable as statistical envelope: For instance Bonferroni (1936); Rüger (1978); Rüschendorf (1982); Vovk et al. (2022) address scenarios involving arbitrarily dependent p-values.; Fisher (1925); Pearson (1934); Simes (1986); Benjamini and Hochberg (1995) focus on settings with independent p-values. Additionally, Sarkar (1998); Benjamini and Yekutieli (2001) provide results under positive dependence assumptions while the exchangeable case has been recently considered by Gasparin et al. (2025). Blanchard et al. (2020) (see also Li et al., 2024) introduce the notion of template, which most closely aligns with the envelope perspective adopt. In the framework of conformal prediction, controls of the form (7) have been employed across various applications, including outlier detection using conformal scores (Bates al., 2023; Gazin et al., 2023), batch prediction (Gazin et al., 2024; Lee al., 2024), aggregation of prediction intervals (Gasparin and Ramdas, 2024), and conformal prediction for ranking (Fermanian al., 2025). All these methods can be summarized constructing a score function v of the p-values such that E E' + v()) 2 QE. where is a threshold specific to each method. Table I provides an overview of these methods, the associated score functions v, and a concise summary of the required dependencies among the p-values to ensure the guarantee (7).
Set E c [0, 1)
v vp))
Dependence
Bonferroni
{p Vj. Pj 2 a/m}
min Pj
Simes
(p: Vj. P() 2 ja/m3
miny mPO//
PRDS
Template
{p: Vj. PU) 2 (Na))
min, (7'(PG)
{p: Σド= (ps2a/2) 2 m/2]
(1/2)
Majority vote
(p : Vk, E5=1 1 (p,20/2) 2 k/2] ming (1/2) Exchangeability
Table 1: Examples of sets satisfying (7) and the associated score function. Fx denotes the empirical CDF of (pi....,t)) Template method refers to Blanchard et al. (2020) where łj denotes a family of functions and Aa 0 is parameter. Majority vote refers the methods analyzed by Gasparin and Ramdas (2024). PRDS denotes Positive Regression Dependence on a Subset (Benjamini and Yekutieli, 2001).
3 Joint distribution of the conditional p-values
The present section introduces fundamental mathematical tools that will be required to derive conformal predictors (based aggregation on p-values) with the right validity in Section 4 and Section 5. We first present some key lemma about a uniform distribution over the ranks. We then link this distribution the joint distribution of conditional p-values associated each observation X,+i- Finally, we derive the distribution of some key statistics.
Key lemma.
Let Anm for n, E N be the set of ordered discrete p p-values:
Anm = a :0sisn) :0 .... .... S am
(8)

---

Lemma 3.1. Let m E N and P (An,,))) where U denotes the uniform distribution. Then
* n .n.(G)) BettBBeeaaannaae.eeeaeeaaaa j.m 1) for [m]]
. E (n-P(12)) ~ BetaBin(m,,
for E [0:
Remark 3.2. The beta-binomial distribution BetaBin b) is the distribution of a binomial random variable B(m. Q) where Q drawn independently as beta distribution B(a, b). If a and are positive integers, then for & E [0 m],
BeeeaBin(mm@u a, = (k (k+a- & m
m
I+++b
In this case, the distribution BetaBin(m, a, b) is a negative hypergeometric distribution which can be easily simulated using Polya's urns. For 0 (resp. b 0), the distribution defined as Dirac in 0 (resp. in m).
Joint distribution of the conditional p-values. We focus here the joint distribution of the ordered vector of p-values p+ = (P(1)()).... ..mm)))......))((( where for j [m]] and y E [K]
1 my
(y())
Σ 1(s(Xn+j,0) S
(9)
Thy t=1
The guarantees we obtain on this vector are under the condition that the scores of the new points evaluated in the true label are exchangeable with the scores of the same class: Assumption 3.3. For all E [K] and conditionally to Yn+1 (SY..... SW Sat1(y)......... Sn+m(() is exchangeable.
the vector of scores
This hypothesis is fundamental to our work. It assumes that the scores of the observed images behave similarly to the calibration scores of the same class. This assumption is more discussed in Section Remark 3.4. The number of multi-inputs m is formally assumed be fixed throughout the paper. However, the method remains directly applicable when m varies. In that case, if Assumption 3.3 holds conditionally on m, then our results remain valid conditionally on m.
The distribution of the unordered vector of p-values has been deeply studied by Gazin al. (2023); Gazin (2024) in the general case where the p-values are the marginal ones, the test points do not all share the same class, and the scores have ties. The following proposition can be seen as extension of these results conditionally to the class, which includes the possibility of ties between scores. To this purpose, we introduce randomized version p of the vector of conformal p-values p.
Theorem 3.5. Under Assumption 3.3, for y E [K] and [m]]] let us introduce
15
((S4z) S + 1{S,+, St)1,,
S UN}
(10)
Tl
where (Ui)ieTm) and (U ordered vector of randomized p-values pT* (w) (v7i(()) -.......--.. (7mm (w)) conditionally Y..1 y. is uniformly drawn from Anymm m
) for y E [K] are i.i.d. uniform random variables on [0, 1]. Then the
p (Y,+, (1 (Y = y) - ~ ~ (An,.)))
(11)
Moreover. for E [m]] we have ny P (Yn+1) 1 (Yn+1 = y) ~ BetaBinnnn j.m. - +++ 1).
The proof is given Appendix A.3. As minor remark, notice that when scores have no ties, the original is preserved. More important, the advantage of the above result is that the knowledge of the exact distribution of the conformal p-values will allow us estimate more precisely the quantiles of statistics constructed from this vector. As consequence, the resulting prediction sets will be more informative (smaller sets) than those based on bounds for these quantiles.
Remark 3.6. The definition (10) of the conformal p-values differs bit from the usual definition (different normalization and no additional +1, can be compared with (6)) but provides an equivalent tool. With this definition, the ordered vector of p-values has a symmetric distribution around the diagonal: 1 - (p(m) (omm)(Yzz (Yn+1),.... ....)).............. (Yn+(Y)+)) has the same distribution p (Y,++))).
5

---

Other important law derivations. Combining this first result with Lemma 3.1, we can derive the distribution of critical quantities such as the empirical coverage or the marginal distribution of the ordered vector.
Corollary 3.7. Under Assumption 3.3, let By = Σj= {y E Ca (Xu+s)} be the number of class conditional sets containing the label y. Thenforr E [K] and & E [m]
P[Br..1 = =Y = u| 2 P = [],
(12)
where B follows the distribution BetaBin(m, (ny + 1) - kg.a: kyya) with Ryg. = [(n + 1)xx] Moreover, if the scores have ties, Eq. (12) is equality.
The distribution of empirical coverage for conformal prediction sets has been first studied by Vovk (2012) and Angelopoulos al. (2023). For the exchangeable case with ties, we will refer to Marques F. (2025) who obtains similar result. This result will be used to enhance the aggregation of conformal prediction sets by majority voting in our setting.
4 Refinement of majority voting for multiple inputs
Thanks to these probabilistic results, we are in position to build new conformal predictors tailored for scenarios where multiple predictive observations of single instance are available. As this problem can be seen as the aggregation of the prediction set associated to each instance, we focus in this section the adaptation to our setting of the general majority voting methods proposed by Gasparin and Ramdas (2024). They propose to keep the labels appearing in at least one half of the sets, in our setting consists to consider the majority voting set CM {y > 1/y E Cod (Xn+s) 2 m/2].
These authors also propose exchangeable majority voting sets defined as CMr. n= C. usable in our setting as the sets Cad (Xn+z) are exchangeable. Both methods achieve class-conditional coverage of 1 - 2o, replacing a by a/2 leads to the correct coverage (see Appendix C).
C, = Cal( (x++),, then thanks to Proposition 3.7, the distribution the number of sets containing the true label Yz+1 is known. The threshold m chosen in the majority vote method can then be improved: we replace it by the quantile of a Beta- Binomial distribution.
Proposition 4.1. Let (0,1), assume Assumptioo 3.3, let
CBB
y 1(yE Cal (Xn+i)) BB(()))
(v))
(13)
where BB (y) is the a-quantile of the distribution BetaBin(m, [[ny 1)(1 a) 1mn 1)a)). Then, for all y E [K]:
P P[Yn+1 E CBB |Y++1= = 11/32116 21-a.
For each class, the set C.B requires computing or storing quantiles of a Seta-Biioomial distributions, which depends on the number of calibration points belonging to that class. To avoid performing this computation for each class individually, for example if the number of classes is too large, one can approximate the Beta-Binomiial distribution by Binomial distribution (see Lemma F. This approximation induces loss in coverage as presented in the following proposition.
Proposition 4.2. Let a E (0, 1), assume Assumption 3.3 holds, let
BBin : {{::I(u e cal (Xn-s)) 2 da
Bin
(14)
where Rii is the a- a-quantile of the binomial distribution B(m, 1 a.). Then all [K]]
P[ E CBii cBiillnn11 = y, y, 2 (m 1) min(1, amm)
a.s.
(15)
ny +1
Moreover, the labels Yi,..., Yat1 are i.i.d., then
K(m 1) min(1,, om)
P[Ynt1 E c.Bin] 2 1 1
(16)
1

---

The loss of coverage remains negligible when the number of repetitions is small (m & n/K for the marginal one). In the setting of Pi Pi@ttNet data, the number repetitions is effectively low, but some classes have also small numbers of examples in the calibration set which makes this approximation only usable for large classes or when only a marginal coverage is targeted.
Remark 4.3. The possibility to use a binomial distribution quantile has been already proposed by Gasparin and Ramdas (2024) merge independent prediction sets. The assumption of independence a bit restrictive and this result proves that it can effectively be used for prediction sets correlated by the calibration (so not independent) for small enough number of sets m.
5 p-value aggregation methods
Voting methods presented above aggregate the conformal sets constructed for each observation point by only considering the number sets that contain given label. Although this procedure can be interpreted as an aggregation of the conformal p-values and as forming an envelope around them (see Figure 1), we observe, however, that some information is lost in the process. In this section, we exploit the structure of the ordered vector of p-values to aggregate the observations. These methods decisively outperform the previously discussed voting strategies.
5.1 General method
The method consists in building (non-) joonfooonfrrm score function e [0, x [K] + R directly on the p-values vector. Then, exploiting the result Theorem 3.5, we calibrate this new score by simulating samples P ..... P7 of the distribution U (An..,m) for each class, using for example Algorithm 1. Here, T can be seen budget for approximating the score's 's distribution by Monte Carlo sampling. The method is summarized in the following result which gives guarantee on the conditional coverage.
Proposition 5.1. Let [0, 1|Th x [K] + R and T N. for y E [K] let P Lid. u(An,,)) and Vi :- v(P for t [T]] Then, for a E (0, 1), if Assumption 3.3 is satisfied, the set
C = {u v u(( (v). v) 2 VV Vor++1oD)
(17)
conditionally valid: Ply E C2 = vl 2 a for all y [K]]
This result is direct application of conformal pre- diction using the score function v, and follows as a consequence of Theorem 3.5. While the choice of the score function e discussed below, it is worth noting retain the low scores in order
Algorithm Simulation of P ~ U(Am.))
1: Input: m.
2: Draw Ry...., Rm w/o. replacement
that for given to remain within the standard framework of envelope
from [n m].
3: P. + R(g for 1 I is m.
methods: shown in Table 1, the scores associated with envelopes tend to reject p-values corresponding to low scores.
4: P. + P- - for 1 < m.
5: Output: (Pi/n,.... Pm/nn)
Remark 5.2. Recall that the guarantees provided by conformal prediction in (2) and (4) hold with high probability, including with respect to the calibration set. Formally, the sample (P.") )t., should be redrawn at each execution. However, is also possible fix T sufficiently large and store the value Vi ()()+)))o)) for each class, interpreting it as quantile estimate of the score distribution.
5.2 Construction of score functions w
We propose in the following different envelopes- associated to a score function u [0, 11 - IR on the p-values- that can either be used for the vector p-values p or its randomized version p as in the above result.
Quantile envelope. The first envelope that we consider relies on quantiles of the beta-binomial distribution. For a clear presentation, let us first remark that as presented in Theorem 3.5, the marginal distributions of the sorted vector of p-values p pzd((,, (Yn..) are known. Indeed, conditionally to
Yn+1 y and ny, the random variable ny (v) follows the distribution BetaBin ny - 1+.1..
7

---

Code for reproducing wtwpatteeeeaaaaaanaaaaseeeeeeeee To capture the geometry of the trajectory of p-values. we propose to choose an envelope of the trajectories constructed as the vector of quantiles of level À of each marginal (see Figure 1). Consider {Ri) a general family of edf and let us remark that
(p Vj Pú) 2 F)) = (p (ppmpp min (P())) 2 }}
The envelope of quantile of marginal associated to the quantile score Vi is defined as for p E [0, 1)m"""""'" E [K]
UP v) = min Fa,n, (nl P())u)), sE[m]
(18)
where F.. is the cdf of the distribution BetaBin(n, j. m + 1). This kind of enve- lope, constructed the true quantile of the marginal with a specific level, appears for instance in Genovese and Wasserman (2006); Blanchard al. (2020) and in Fermanian et al. (2025) in an empirical version.
Quanti
Bonterone
Mayariy
1.0
3.8
3.6
(9-Area envelope. A simple idea to reject the classes whose p-values are too small could be sup- ported by the fact that the area of the trajectory too small. For q > O, we define the area score: Arearea (P. v)
||lll where Il-ll,, for 9 2 is
the ly fy-norm in RT.. For q 1, the procedure that consists in rejecting a trajectory or not can be viewed variant of the Vilcoxon-Maan-Whing test (Wilcoxon, 1945; Mann and Whitney, 1947).
Figure 1: Simulation of 150 samples of P (A for m = 10 and n = 100 (black lines). Quantile is the envelope associated to UQ. and Majority to the majority vote.
(9 (9-envelope. As asymptotically, the vector (Y,-) converges to the identity vector Id = (00m++))//))))
our last score only considers
the distance of p-value vector to this vector. In particular, [r (Yn+1)] - (ef. Theorem 3.5) which motivates the score Upv defined as: Ue- (P, := - Idll,. The resulting set (17) will then keep p-values close Id. For q = co, bounds of the exact quantile and the asymptotic distribution
are provided respectively by Gazin et al. (2023) and Gazin (2024).
Experiments
A key requirement of conformal approach is its ability to accurately quantify uncertainty. For an efficient classifier, this should lead to small prediction sets; for less accurate one, the conformal set should still contain the correct label, at the cost of a larger set. To this purpose, compare the methods, by comparing the size the prediction sets returned for a fixed coverage.
Multiple observations of the same instance are particularly valuable in challenging tasks where single observation does not permit informative classification. we will demonstrate, our aggregation techniques lead to reduction in prediction set size in such cases resulting in gain of interpretability. Throughout our experiments, we report the accuracy the base classifiers. The score used for all the following experiments is the softmax score which is one minus the softmax output of the class. Additional results using the APS score (Romano et al., 2020) are presented in Section D.2.
The tested prediction sets are the Majority C -Eq. (22), Exchangeable Majority CMe_Eq. (23), BetaBinomial CBB Eq. (13), Binomial Bin Eqq. (14), and the set C" Cn_Eq. (17) with the score functions UQ (Quantile), U1-Area (Wilcoxon), Ureare (f2 Area) and Vt, (62).
experiments is available
https://gittul/ com/ _P_frrrr_rr______________rr___d_ forfforruutttt Multi, Inputs.
6.1 Synthetic data
We choose as distribution of the points, a mixture of Gaussian distributions with randomly drawn centers. Formally, Y ~ p(y6 and X| y) NNy,, L2))) where K 10, d = 6, the
8

---

Coverage
Averape leugh
Conditivmal Coverago
Winst Conditioal Choraag
1.00
10
1.00
Miggial covengg
0.55
0.95
Size for
1200
40
00
50
100
20
40
60
80
100
20
80
80
100
2 Nanmbbrr of oloerrvationn volios
Nomber of observations
N Nammber of observations
Magerity
Binvariaa
Rotr Rimeniial
Exchongodble majority
Wilnosers
Ea Area
Figure 2: Synthetic data: marginal coverage, average length and minimum of the class conditional coverages in function of the number of observations for a = 0.1, computed on 5000 repetitions. The confidence region is + (empirical stdl/v50000
classes are bit imbalanced p(y) x (1 + +3y 1)/K) and x² = 3. The centers of the classes My are drawn independently as (0, I2). The strength of the noise makes this problem difficult with only one observation as the distance between two centers clusters is relatively close to the distance of a point to its cluster's center: E||| Jy II] = Bill,, - -=1]]- The model is a ReLu neural network with three hidden layers of width 15. It is trained with 2000 observations and achieves top accuracy of 0.49. We apply our conformal procedure with calibration set of size n = 1000 and repeat 5000 times our procedure for number of observations E (1,...,100)
We observe in Figure 2 that the methods based on the p-values perform best as compared to majority vote methods. For instance, for m 20 and éz-envelope method, we observe significant improve- ment with set of average size of 3.6 as compared to an original value of 6.6. They provide the smallest sets while maintaining marginal coverage above 1 a, and exhibit conditional coverage close to a. The majority vote methods appear to overly conservative. The proposed refinement of the majority vote improves on the two original versions. As announced in Proposition 4.2, for small m, the binomial quantile approximates the beta-binomial one well. The marginal coverage is not significantly affected as long as m/n remains small. However, in the third figure, the conditional coverage for some classes is impacted by this approximation when m is large as the numbers of points of some classes in the calibration set are smaller than m. The over-coverage observed with the approach arises from the potential non-existence of a quantile at the exact level a. This issue can be addressed by introducing additional randomization.
6.2 The LifeCLEF Plant Identification Task 2015
We apply our method to LifeCLEF Plant Identification Task 2015'
Averge Longth (Tomp-
Avergg Length (Temp 20)
(Goëau et al., 2015) dataset, a dataset of 113,205 images of plants spread between IK species (classes), with high heterogeneity. We split equally between train, calibration and test sets the original full dataset thanks stratified approach to maintain the original classes distribution. The classifier considered is ResNet50, trained on @I@ ntNet-300K Garcin al. (2021)2, for which we have fine tuned the last layer on our training set. achieves a top-1 accuracy of 43% (and top-5 of 63%, see Section D.I for others top-k accuracy). As a pre-processing step, we restrict ourselves to the classes (species) for which we have at least 20 examples in the calibration set, hence keeping K = 688 classes. This is needed for having informative class conditional CP set. We group randomly images of the same classes to simulate multi-view observations of the same plant with a number of repetitions of m E E(1, (1...., 10).
Size of the set
10a
10:
109
2.5
5.0
7.5
10.0
2.5
5.0
7.5
10.0
Number observations
Number of observations
Majotit
Easmiial
Wilcoson
majority
Bes--ineniil
Quantile
Figure 3: LifeCLEF Plant Identification Task 2015: average length in log-scale for 0.1 and two difffernnt choice of temperature in function of the number of observations.
'httpa:/ ptttpa://www.. http://wwwwimmmceeeeeeee wtt.:ooo/eeeeee///155555
see code and models https: com/ wwt.pp/anettaaatttttttteeeee

---

In Figure 3, the prediction set lengths of various methods for two temperature settings (Temp=1 and Temp=20), all methods achieve a valid coverage of at least 0.9, with detailed results deferred to Section D.2. The temperature refers to the standard re-scaling of the output of the network before the application of the softmax function. Our interpretation that, due to the numerical approximation of the softmax function, a low temperature can lead to fies between scores (many are then numerically 0 or 1), which may hinder the performance conformal procedures (see, e.g., Dabah and Tirer, 2024 for discussion on the relationship between temperature scaling and conformal prediction). Specifically, we observe that when using the raw network outputs without temperature adjustment (Temp=1), the majority vote struggles and tends to predict nearly all classes. In contrast, methods based on conformal p-values consistently produce narrower prediction sets after just a few observations and are less sensitive to this parameter. Our refinements to the majority vote method do lead to improvements m over the original approach, but they are still outperformed by the p-value aggregation methods.
7 Conclusion and limitations
We introduced new
prediction framework for the problem of classification when multiple observations of the same instance at prediction time are available. Assuming the scores of the new points evaluated in the true label are exchangeable with those of the same class, developed an analysis of the distribution of the vector of the underlying ordered conformal p-values. We built two novel aggregation strategies of these p-values and showed how effective is the method that captures the best the structure of the conformal p-values.
Exchangeability of scores vector seems to be the bedrock of our strategy and more generally of conformal prediction (see Section E). We ensured this condition by sampling the multi-inputs randomly from the same class. It would be interesting to investigate the more realistic case of dependent multi-inputs, for which our method constitutes a foundational step.
Acknowledgments and Disclosure of Funding
The authors acknowledge Ulysse Gazin, Pierre Humbert and Etienne Roquain for helpful discussion. This work was funded by the French National Research Agency (ANR) through the grant Chaire IA CtMeLOt ANNR20-CCHAA-0001001,
References
Anastasios N Angelopoulos, Stephen Bates, et al. Conformal prediction: A gentle introduction. Foundations and Trends@ in Machine Learning, 16(4):494-51, 2023.
Rina Foygel Barber and Ryan J Tibshirani. Unifying different theories of conformal prediction. arXiv preprint arXiv:2204.022: 022992, 2025.
Stephen Bates, Emmanuel Candès, Lihua Lei, Yaniv Romano, and Mattco Sesia. Testing for outliers with conformal p-values. Ann. Statist, 51(1):149-178, 2023.
Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. Ser. B Stat. Methodol,, 57(1)2289-300 1995.
Yoav Benjamini and Daniel Yekutieli. The control of the false discovery rate in multiple testing under dependency. Ann. Statist., pages 1 1165-1188, 2001.
Gilles Blanchard, Pierre Neuvial, and Etienne Roquain. Post hoc confidence bounds false positives using reference families. Ann. Starist., 48(3): 128--330, June 2020. doi: 10.1214/19-AOS18
Carlo Bonferroni. Teoria statistica delle classi e calcolo delle probabilita. Pubblicazioni del R istituto superiore di scienze economiche commericiali di firenze, 8:3 62, 1936.
Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, and Titouan Lorieul. Set-valued classification- overview via a unified framework. arXiv preprint arXiv:2102.12318, 2021.
Lahav Dabah and Tom Tirer. On temperature scaling and conformal prediction of deep classifiers.
arXiv preprint arXiv:2402.05886 2024.
10

---

Tiffany Ding, Anastasios Angelopoulos, Stephen Bates, Michael Jordan, and Ryan J Tibshirani. Class-condittonal conformal prediction with many classes. NewrIPS, 36:64555-6453 64576, 2023.
Tiffany Ding, Jean-Baptiste Fermanian, and Joseph Salmon. Conformal prediction for long-tailed classification. arXiv preprini vrXii:2507.06867 2025.
Jean-Baptiste Fermanian, Pierre Humbert, and Gilles Blanchard. Transductive conformal inference for ranking. arXiv preprint arXiv:250111388, 2025.
R.A. Fisher. Statistical methods for research workers. Edinburgh Oliver & Boyd, 1925.
Matteo Fontana, Gianluca Zeni, and Simone Vantini. Conformal prediction: a unified review of theory and new challenges. Bernoulli, 29(1):1-23, 2023.
Camille Garcin, Alexis Joly, Pierre Bonnet, Antoine Affouard, lean-Christophe. Lombardo, Mathias Chouet, Maximilien Servajean, Titouan Lorieul, and Joseph Salmon. @ ntnet-300k: a plant image dataset with high label ambiguity and a long-tailed distribution. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Matteo Gasparin and Aaditya Ramdas. Merging uncertainty sets via majority vote. arXiv preprint arXiv:2401.09379 2024.
Matteo Gasparin, Ruodu Wang, and Aaditya Ramdas. Combining exchangeable p-values. Proc. Natl. Acad. Sci., (22(11):c2418841222 2025.
Ulysse Gazin. Asymptotics for conformal inference. arXiv preprint 24X1v:2409.12019, 2024.
Ulysse Gazin, Gilles Blanchard, and Étienne Roquain. Transductive conformal inference with adaptive scores. In AISTATS, 2023.
Ulysse Gazin, Ruth Heller, Etienne Roquain, and Aldo Solari. Powerful batch conformal prediction for classification. arXiv preprint 2..02::2411.022.9, 2024.
Christopher R Genovese and Larry Wasserman, Exceedance control the false discovery proportion. J. Amer. Statist. Assoc., 1414446)1448811411 -4177 2006.
Hervé Gočau, Pierre Bonnet, and Alexis Joly. LifeCLEF Plant Identification Task 2015. In Working Notes of CLEF 2015 Conference and Labs of the Evaluation Forum, volume CEUR Workshop Proceedings of Working Notes of CLEF 2015 Conference and Labs of the Evaluation forum, Toulouse, France, 2015.
Alexis Joly, Hervé Goëau, Pierre Bonnet, Vera Bakić, Julien Barbe, Souheil Selmi, Itheri Yahiaoui, Jennifer Carré, Elise Mouysset, Lean-Frangee Molino, Nozha Boujemaa, and Daniel Barthélémy, Interactive plant identification based on social image data. Ecological Informatics, 23:22-34, 2014.
Yonghoon Lee, Eric Tchetgen Tchetgen, and Edgar Dobriban. Batch predictive inference. arXiv preprint arXiv:24009 13990, 2024.
Jinzhou Li, Marloes Maathuis, and Jelle J Goeman. Simultaneous false discovery proportion bounds via knockoffs and closed testing. J. R. Stat. Soc. Ser. B Stat. Methodol, 86(4):966-99)) 2024.
Henry B Mann and Donald R Whitney. On a test of whether one of two random variables is stochastically larger than the other. Ann. Math. Stat., pages 50-60, 1947.
Paulo C. Marques F. Universal distribution of the empirical coverage in split conformal prediction. Statistics & Probability Letters, 219:1 10350, 2025. ISSN 0167-7152.
Karl Pearson. On new method of determining" goodness of fit". Biometrika, 26(4):425 442, 1934.
Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive coverage. NeurlPS, 00::351133991 2020.
Bernhard Rüger. Das maximale signifikanzniveau des lests:" o ab, wenn unter n gegebenen tests zur ablehnung führen". Metrika, 25:171-178, 1978.

---

Ludger Rüschendorf. Random variables with maximum sums. Ann. Appl. Probab., 14(3):233-63 1982.
Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least ambiguous set-valued classifiers with bounded error levels. J. Amer: Statist. Assoc., 1145555)223-234, 2019.
Sanat K Sarkar. Some probability inequalities for ordered mtp 2 random variables: proof of the simes conjecture. Ann. Statist., pages 494-504, 1998.
John Simes. An improved bonferroni procedure for multiple tests of significance. Biometrika, 73 (3):551-754, 1986.
Brian L Sullivan, Christopher L Wood, Marshall J Iliff, Rick E Bonney, Daniel Fink, and Steve Kelling. ebird: A citizen-based bird observation network in the biological sciences. Biological conservation, 142(10):2282-223) 2009.
Kanint Teerapabolam. bound on the binomial approximation to the beta binomial distribution. In International Mathematical Forum, volume 3, pages 1355-1358. 2008
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, pages 8769-8778, 2018.
Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian conference on machine learming, pages 475-490. PMLR, 2012.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in random world, volume 29, Springer, 2005.
Vladimir Vovk, Bin Wang, and Ruodu Wang. Admissible ways of merging p-values under arbitrary dependence. Ann. Statist., 50(1):351-375 2022.
Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-8) 83, 1945.
12

---

A Proofs
A.1 Preliminary results
This first result is derived from Theorem A.1 of Gazin et al. (2023).
Corollary A.1. Let f1....,,a4t, exchangeable random variables with no ties (S; + S almost surely). For j E [m]] ler
1 - Σ 1{ 2 Sth. Pj
(19)
=11
and P1 = (P(1) .(mm)) be the vector of ordered p-values. Then
Pt U()
where An, is defined in (8).
Proof. This result a direct consequence of Theorem A.1 of Gazin et al. (2023). According to this work, let M [0.m]] m|" " be the histogram of p. i.e., for [0,
{ P: 2 M2 {
i
Then, M ~ ((H,m where Hn,m = {h [0.m]" " :E h2 m) is the set of histogram (Gazin al., 2023, Theorem A.I). We can now remark that Hnm in bijection with the set of ordered
trajectories Anm = {a a :osisnn :n s... S am For example 6 Hn,m + An,
defined for h E Hnm by:
1
( 0 0. n
((h)
11
times
times
such a bijection. Then pt = ((M) is uniformly drawn into A Anm
A.2 Proof of Lemma 3.1
Distribution of the marginal. Let Let ((Am.))) k E [m]] and n.. The probability of the event {P(j) = k/n} is the probability that the sub-trajectories P1:(j-1) = (P(1), P(j. 1))
belong (i/n, E [0:
[ AI]
and the sub-trajectories P(tt1)mm = (P(j 1). ...., ......,,mm)). belong
to (i/n, i E [A: : n]] m There are respectively (*++)))) and (" (n-k+m-) 3) of such trajectories which
the numbers of non-decreasing sequences of (resp. m j) elements of k] (resp. [|k nl). Then,
PP(()) k/n] = # # E An. : inP.ag-1) E [0: kBp 1 nP(j) = k. nP DP E [k: nmmm-)
#An.
# A
# Anmm
(a-k-1) (n+m-
BetaBin(n,j,m--- 1)
Distribution of the empirical cdf. The event Em 1 In))eee= is satisfied if the first values of P are strictly below e/n and the lo last ones are above f/n.. That means that the sub- trajectories Pl:(m-N) (P(1). ..., P(m - k))) belong to {i/n, [00 1]} and the sub- trajectories P(m-k+1)m (P(m k + 1), ..... P(m)) belong to (i/n, n]]. There
13

---

respectively (--++--) k and ("**+)) of such trajectories Then:
# E An.m n -PImmm---- k) E [0:
- 1] 1]m-> ; nn (m-b+1)cm E [e: n}*
1 P())2
# #An.
# Ag x An-
# #AAm
( (t-1+ (-1+m-- (n-btt)) [tk)
( =k]-
A.3 Proofs of Theorem 3.5 and Corollary 3.7
In this section we prove Theorem 3.5 and Theorem 3.7 simultaneously. Let first remark that if the scores have no ties, under the Assumption 3.3, all the results are obtained by applying Corollary A.1 and Lemma 3.1 conditionally to (Yz+1 y) to the scores
S., (3)..,., Sm (v)).
Let us now consider the case where there are potentially ties between the scores. The proof uses some common idea with the proof of Proposition of Fermanian et al. (2025). Let 5 be the minimum non null distance between the new scores of the true class and those evaluated in the calibration set:
5 = min miin(1
{1s
s' * E (Stheem U (Sm+s (Yn+)() [))m))[])]]])
If all the scores equal, 5 can be set to any non null value (for example 1). We now define some proxy scores $ by:
Suty: == : + U,8, for E [m]] (20) where U,, U, for E [K], i E [ny] and j [m]] are i.idd uniform random variables in [0, 1]. The proxy scores S satisfy Assumption 3.3 and have no ties. We then apply Corollary A. the proxy scores (SY.... n. (v), 3n+m(y) conditionally to (Yn+1 y) to get (11). Indeed, the class conditional p-values associated the proxy scores S equal prd:
5 S² + U,5, for y [K] i [ny];
(,(3) = Tly 1=1
Σ
1f (w) s S.
11Sm+s()) < < + 1(S S/}1{88, s SUN}
Rg
11Sx+5()) < S (V) 1T s UP]] 15
Then the distribution of the marginal P is a consequence of Lemma 3.1.
To get Corollary 3.7, we can remark that the proxy scores preserve the order as Sn-z(y) S. Snts(y) < Sy. Thus for j E [[]] and y E [K]:
y E cad (Xnts) = Snts(y) ((()++)(--))(()))
Σ (( - 2S ( 1)oe)
1(Sn+5(0) < S 1(Sx+5()) 8 2 [nny + 1)al
ny
S Sn+ < <5t) 2 1(n, 1)a]
Σ
3 (2) S 555 2 1(n + 1)a}
y End (Xn+i)
14

---

where Cad is the class conditional prediction set (5) constructed with the proxy scores S. We have therefore shown that (Xx3)) C Cal (Xn+j), using that the randomization preserves the order and that the scores "tilde" have no ties. This implies that almost surely, the empirical coverage of the true scores is lower bounded by the empirical coverage of conformal sets constructed with the proxy scores, i.e.
Σ 1fYx+1 E Ca (Xn+s))
Σ ==1
1 Yn+1 E Ca (X(+s))
We conclude again using Lemma 3.1.
A.4
Proof of Proposition 4.1
The result is direct consequence of Corollary 3.7.
A.5 Proof of Proposition 4.2
Let us denote By = -ドー 1/y E Cal (Xnts)) First observe that:
(Yn+1 & C.in) (BY < (B 1 a))} c (BY44s < Qa ayn,)))}
where 0Y = ( +1]a] < and Qa (Q) denotes the quantile of the distribution Q. Let ny
Zy B(m, 1 ag) and By ~ BetaBin(m, By + ky ky) where ky = La(ny + 1)J. Then, for
y E [K]:
P( & C Yut1 8) $ P(y < Qa ay))|
< P(3 (B(m, 1 -1,)))s
using Corollary 3.7. It follows that:
P[Yn+1 & CBin (Y+1 = v] <PIZy < Qa(B(m, a.y))
+ drv B((m, aly) BetaBin + - Ky kyy
<a+ (m - 1) min(1,
(21)
ny
where we have used Corollary F.2 in the last inequality bound the total variation distance between the Binomial and the Deta-Binomic distributions. We have then obtained the conditional coverage guarantee.
Assume now that the labels YI,.... Ya+1 are i.i.d.. Then ny = Σド 1(Y; = y is independent of Yn+1 and follows Binomial distribution B(n, (())) for q(v) := P|Y y). Then:
q
(1))
K
E
E
E
ny
Tly
(n)
m+1
where we have used Lemma F.3 for the inequality. That gives the marginal coverage after taking the expectation relatively to YA+1 into (21).
B Failure of the naive approaches
We discuss in this section the challenges that naive approaches may face when dealing with the case of aggregated predictions.
B.1 Synthetic dataset
We first consider two naive methods that we view as natural. We illustrate them in the case of mean aggregation and evaluate their performance the synthetic dataset presented in Section 6.1.
First method: directly using the calibration set. first naive idea is to construct the prediction set using the original scores and to use it directly with the aggregating scores, i.e., when the aggregation
15

---

Distributions of the aggregated scores
1.0
0.6
0.0
Calibration
Figure 4: Distribution of the averaged softmax scores for synthetic data. 5000 averaged scores are computed and sorted for m 7,9.
Marginal coverage
Average length
0.98
9
Coverage
0.96
Mean set, size
8-
7-
0.92
0.90
20
40
60
80
100
20
40
60
80
100
Number observations
Number of observations
Direct calibration
Mean calibration
Figure 5: Empirical coverage and average lengths for class-conditional approach using two naive calibration sets for synthetic data. The calibration set is of size 5000, the coverage and the length are evaluated on 300 multi-inputs repeated 1000 times. The confidence region is + (empirical std)/ /5000
the mean, considering the set {u: S (v) m Em Sn+s(y) S (( +1,(
Formally, this set lacks any guarantee since the new (aggregated) score S (Y,-1) follows different distribution than the calibration scores. In Figure 4, we compare the distribution of this averaged scores for 1 to 9 observations to the distribution of calibration scores. A clear change in distribution can be observed even for small value of m. practice, this method produces conservative prediction sets, whose size increases with the number of repetitions, as it can be seen in Figure 5.
Second method: aggregating the calibration scores. A second idea obtain a calibration set exchangeable with the new score is, for a given number of observations m. to aggregate the calibration scores by class, i.e., by averaging them in groups of points, in order to mimic the distribution of ag- gregated scores. These new aggregated calibration scores will be exchangeable under Assumption 3.3 with the new aggregated score. However, by doing so, the calibration n/m points (more precisely. Σ [n./mm]. Recall that the number of points per class must at least a in order to obtain informative class- cans-condiiionally valid prediction sets, which no longer holds when m is large. shown Figure this method yields valid prediction sets whose size decreases for small values of m. but increases again when the number of observations becomes too large. This strategy may be sufficient cases involving limited number of inputs, or for classes with sufficient number of points, i.e., when < Ny Note however that the methods based on
reduced to approximately
16

---

p-values aggregation outperform this strategy, with an average size quickly below 4 elements (see Figure 2).
Remark B. Figure 4 also highlights the occurrence of tied scores produced by the softmax output of a neural network. When examining the calibration curve, constructed from one minus the softmax outputs corresponding to the true classes, we observe that many of these values are either close equal to 0 or 1 (plateaus at the left or the right). It indicates that the classifier often returns Dirac, sometimes the true class (scores of 0) but also often in a wrong class (scores of 1). This leads score ties, therefore caused by combination of classifier overconfidence and numerical approximation.
B.2 LifeClef dataset
Our approach assumes that, at the time of calibration, only individual data points are observed, rather than multi-inputs. If multi-inputs are available at calibration time, an alternative to our methods would be to apply class-conditions setting, each data point corresponds to a multi-input. and each calibration score computed as the aggregated score of that multi-input. However, with the "second method" presented above for the synthetic dataset, such an approach requires a sufficient number of multi-inputs per class, specifically at least a- In the case of the LifeCLEF 2015 dataset, these conditions are not satisfied, and consequently, this method yields prediction sets that include nearly all classes. In comparison, the p-value aggregation methods does not suffer this issue it constructed from the calibration scores taken individually. To overcome this issue, it could be worthwhile to explore class-conditional conformal prediction methods specifically adapted to long-tailed distributions, such Ding al. (2023, 2025) but this remains out of the scope of this work. Nevertheless, we emphasize that if a sufficient number of multi-inputs is available, we can expect this method to behave better.
conformal prediction approach to the multi-inputs. In this
This is illustrated in Table 2, which reports the results this naive method applied to the LifeCLEF dataset constructed as described in Section 6.2 in average multi-inputs. The aggregation strategy remains a simple average of the softmax outputs. However, as previously discussed, the large prediction set sizes are due primarily to the limited amount of available data, rather than to the quality of the aggregation itself.
all sizes and for different sizes of
m
1
3
5
7
9
In average
Coverage (%) Length
88.2 + 0.5
96.3 + 0.6
97.5 = 0.7
95.5 + 1.5
99.0 = 1.0
91.9 + 0.3 646 +
623 +
665 +
673 +
675 +
677 +
Table 2: Evaluation of class-conditional conformal prediction using multi-inputs structure conditional of different size m of multi inputs and marginally over the different sizes. Recall that K = 688.
C Majority voting.
We present in this section the majority voting strategies proposed by Gasparin and Ramdas (2024) for aggregating conformal sets and the guarantee we get in our setting. The purpose of these methods is to aggregate marginally valid prediction sets C...... C. coming from some blackbox method. We have compared to two of their methods that we consider of interest for our application. The majority vote set CM contains the classes selected by at least one half of the sets. Formally:
1 m
CM - CM=
y
Σ m j=1
1y E C,) 2 1/2].
(22)
the sets Cj have a coverage able to compare this method to sets of canssioondiii coverage 1 a, we apply it to the sets
a, this set has a marginal coverage of at least 2a. To be
C, = =c (Xn+5) of class-conditional coverage - a/2. Then, the resulting majority vote set has a class conditional coverage of at least 1 - i.e., for all [K]]
m
Ply y E C. CY = =)] = m
vECad (X,, 1) 2 Ynt1 = 2 21-0
17

---

Top-k Accursey
1.0
0.8
Top-k Accuracy
0.6
0.4
0.2
0.0
100
400
Figure 6: Top-k accuracy of the model for the LifeClef dataset. The dot line indicates the top-k accuracy of 0.9: at least & = 380 classes needed to reach this level.
This method can be improved when the sets are exchangeable (Gasparin and Ramdas, 2024): that situation the intersection Cade of the majority vote with increasingly more "voters" defined by:
&
C Me = {y: y: <y:Vk Vk E [m]
1 k
E i aod (X 2 1/2 ==1
(23)
ensures class conditional coverage of at least - In fact, Assumption 3.3 guarantees that the sets 2 (Xn+s) for i [m]] are exchangeable given Y,+1-
D Further experiments and technical details for Section 6.2
D.1 Technical details
The classifier considered is ResNet50, pre-trained Pl@ntNe--00KK Garcin al. (2021) (see code and models at https:: //gi tttp//ttt..u.... cwwcclmmmaaaccccco plantnet/PPl Plaatteeeettteeeee -00k//), for which we have fine tuned the last layer on our training set. The system used for this training phase equipped with 4x Intel Xcon Gold 6142 (64 cores/122 128 threads total @ 2.6-3.7 GHz, 88MB L3 cache) while the GPUs 2x NVIDIA A10 (24GB VRAM each) and 2x NVIDIA RTX 2080 (IIGB VRAM each), for a total of 70GB GPU VRAM. Obtaining the softmax scores required about 10 hours on our GPU resources fine tune this pre-trained model. The conformal method runs on a standard machine. The codes of the experiments are provided in the supplementary materials.
Figure presents the top-k accuracy of the model for different values of k: To achieve accuracy of 0.9, one needs to consider at least approximately k = 380 classes. This information can be compared with the average sizes of the conformal prediction significantly smaller for the methods we introduce, when few additional observations are available. Even for m 1 with large temperature parameter (Temp=20), e-value-basee methods produce smaller prediction sets, whose size is less than 100. This is likely due to the randomization involved in the p-vuuue computation.
shown in Figure 3. These sets tend to be
D.2 Additional experiments
In Figure 7, report the marginal coverage of the prediction sets, well as the average class- conditional coverage. Due to the small number of observations available for some classes (e.g., only 14 test points in the smallest class, which yields at most one multi-input sample of size this class), evaluating the conditional coverage per class becomes infeasible. this case, the average coverage serves proxy for the true conditional coverage. It is also worth noting that it closely aligns with the marginal coverage, which is expected given the limited number of test points.
10 for
In Figure 8, we present the results (coverage and size) our methods applied using the APS score (Romano al., 2020) instead of the softmax score used in the rest of our experiments. Let
18

---

Coverage (Temp-
Coverage (Temp 20)
1.000
Marginal coverage
0.977
0.925
0.900
0.873
10
Average Conditional Coverage (Temp
Average Conditional Coverage (Temp
20)
1.000
Coverage
0.975
0.925
0.900
10
Number of obeervations
Number observacions
Magoity
majonty
Figure 7: Coverage the methods for the LifeClef dataset and different values of temperature.
Coverage (APS, Temp
Coverage (APS, Temp 20)
Mangital coverage
0.975
0.950
0.925
0.900
Average LeagthiAPS Temp 1)
Average Leugth (APS, Temp 20)
Averug Sten
10
Number obeervations
Number observatios
Magenty
majenty
Figure 8: Coverage and size (in log scale) of the methods for the LifeClef dataset with APS score and different values temperature.
#: + [0, 11 be the classifer (softmax output the neural network), it is defined as:
SA = Σ 7(x) 1a()) + +UR(x)y
(24)
y'y
where U ~ ~0 1) is uniform random variable. The results are similar to those obtained using the softmax score: the p- p-value aggregation methods offer again substantial improvement terms of size compared to the majority-bar baseed methods.
19

---

E Limitations of the assumption of exchangeability.
The methods proposed in this work rely on the assumption that the multi-input samples are exchange- able with data from the same class. In the context of Pl@ ntNet, this means that multiple images of the same plant behave similarly to images of different plants from the same species. In the experiments presented above, this assumption is satisfied by construction of the dataset, the multi-inpus sampled within the same class.
The LifeCLEF Plant Identificathe Task 2015 dataset also features a multi-input structure, derived from images jointly submitted by users. However, we observed that the exchangcability assumption does not hold for the multi-inputs in this dataset. The correlation between images within the same input is strong enough to break Assumption 3.3, resulting in prediction sets that are no longer valid.
Our p- p-value-based methods implicitly rely on the idea that each new observation brings additional information, thereby enabling more refined class selection. When images within the same input are too similar, the resulting p-values tend to be overly close, which can lead to the rejection of the correct class.
In contrast, majority vote-based methods sets with appropriate coverage, although often at the cost of larger size. The Binomial and Beta- Binomial methods still improve upon majority vote strategies (with reduction in size of around 100 for m 10), but, like the p-value aggregation approach, they no longer offer theoretical guarantees when the exchangeability assumption violated.
robust this effect and typically yield prediction
Coverage
Average Length
1.0
Marginal coverage
600
0.8
400
0.6
10
10
Number of observations
Number of observations
Majocit
micelty
Biaonial
Figure 9: Empirical coverage and average lengths for the LifeClef dataset with multi-input breaking exchangeability.
The coverage and the average length of Figure 9 are evaluated for number of multi-input observation from 4102 for m = 1 to 112 m = 10. The temperature parameter equal to The loss of exchangeability causes clear decrease in coverage of methods based p-values aggregation.
Probabilistic results
F.1 Total variation distance between Binomial and Beta-Binomial distributions
The following Lemma gives bound on the total variation distance between a Binomial and Deta-Binomias distribution.
Lemma F.1 (Teerapabolamm, 2008). Let m N, a, E Rs,, ifp σ and - p. then:
m(m -1) (m + 1) + + +9)
drv B(m, BetaBBa(m(mma. s m+ 9mh ml
Let us apply this bound the conformal setting where BetaBin has parameters m, ki, and n + kin
with h. [n 1) -a]l and E 1).
20

---

Corollary F2. Let ka = [(n + 1) - a)]. then
drv (((m, Mo/n+1), BetaBin(m. (m, kia + 1 - -kaaa))) (m 1) min(1,
n +
Proof. It directly deduced from Lemma F.1. Just lower bound by and
Na n+1
by a to get
that:
m (m 1)
drv (B(m, ho/n+t), BetaBinnnninnn h + - ka)) 1 (1 (1 - (1-a)
m + n 2
Then let us just remark that:
(m- 1)m (1 - (1 a)m S (m 1)m m +1
min(1, (m + 1)a) & (m 1) min(1,am)
m +
F.2 Technical lemma
Lemma E.3. Let N B(n, p). then:
1
E
(1 - -p)
+1
(n + 1)p
Proof. By direct computation:
E
p² (1 - n
k
k=0
n
px(1 (1 -pp)
k + 1/
n n
n+1
+ 1
p² (1 m)1
(n + 1)p k=1
k
1
(1 (1 n)))
(n+ 1)p
where the last equality holds because the sum is almost the full binomial expansion, except for the first term.
21

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions mostly methodological, and the methods introduced well explained, with theoretical results as well as experiments showing their interest.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have raised some issues about the excheangeability assumption that not necessarily holding in practice. For instance for the LifeCLEF Plant Identifi iatictiiiiiii catiinn Task 2015 dataset (Goëau al., 2015), we comment clearly that the assumption only partially met. Other limit include the classwise setting that is possibly leading to poor results for long-tail label distribution. We discuss that in the conelusioo-lmmaation section, and provide some results in Appendix E.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specificattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
22

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The proof are mostly in Appendix A meet the size constraint, but are all given. The main assumption of the paper is Assumption 3.3 provided in the main paper.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The Python code is provided in the supplementary material, and the datasets are either simulated or open data like the the LifeCLEF Plant Identification Task 2015' (Goëau et al., 2015). Overall the method is simple and could be reimplemented from scratch by interested reader.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce
the model (e.g., with open-soure source dataset or instructions for how to construct the dataset).
platnt
23

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The Python code is provided in supplementary materials, and the datasets are either simulated or open data like the the LifeCLEF Plant Identification Task 2015 (Goëau et al., 2015). Overall the method is simple and could be reimplemented from scratch by an interested reader.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/ public/ cabblc/ggiies... Coleee//doSSSubisstcee www.abmissionoooiiioooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips... wwcpcccccccc///ccccccccccccb..lt /ttee/uuuieeeeeeeemmimieeee Coc///ooddeessseeeeeeeeeee wwwassnnoolllle onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experiments are rather straightforward, as the paper is mostly methodolog- ical, and the full details are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes]
Question:
http://fc/////elee/e// httpp//////yyr...gggoo.ooooo.oor plant
24

---

Justification: We provide errois bar on the average size of our sets and report 95% confidence interval (with Monte Carlo simulations of 5000 samples for synthetic data). For the coverage. as the variable are binary, the emror bar where not included being less informative. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The experiments were short (less than one hour for the CPU experiments) once the scores computed. Obtaining the softmax scores required about 10 hours GPU resources to fine tune model pre-trained on the PI@ NtN dataset" (Garcin et al., 2021) for the the LifeCLEF Plant Identification Task 2015 experiments. The system is equipped with a 4x Intel Xeon Gold 6142 (64 cores/128 threads total @ 2.6-3.7 GHz, 88MB L3 cache) while the GPUs are 2x NVIDIA A10 (24GB VRAM each) and 2x NVIDIA RTX 2080 Ti (11GB VRAM each), for total of 70GB GPU VRAM. These informations are provided Appendix D.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: pettps:/nneurigee httplpp....cppppl/ cctppccccccc/eeeeeecccccccccccct wtw.bl/i///hiisduddell.e....
Answer: [Yes]
Justification: The contribution centered on statistical methodology for classification, and respect all the code of Ethics.
httpp://ggita wwg.hhww.../plaatttt///aaateeeee
25

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g,, if there is special consider- ation due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
The direct contribution has the potential to benefit society in the mid to long term. Specifically, our methodological advancements could enhance citizen science platforms like Pl ntNet, benefiting both the general public and ecologists by improving how these systems handle multi-inputs, stated in introduction.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
This is irreleavant for a methodological paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
26

---

12. Licenses for existing assets
Are the creators or original owners of assets (e.g, code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: cite the datasets we used precisely. For the code, apart from standard Python code (numpy, pytorch, matplotlib, etc.) we rely mostly on our own code. It will be release under MIT License.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thoode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The research code with document associated zip file in the supplementary material.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
27

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.M policy (https:/ hattps://neurds wwtpccc///oonoiaeaeeeeee http://p.......o.eeeeeeee/////// for what should or should not be described.
28


Reviews:
[{"review_id": "QNFKhyddNq", "reviewer": "Reviewer_4nuN", "summary": "This paper develops a conformal prediction framework for multiple observations of the same instance, motivated by citizen science applications like plant identification. The authors aggregate conformal p-values from individual observations rather than predictions directly, preserving exchangeability assumptions. Key contributions include deriving the exact joint distribution of class-conditional conformal p-values and proposing two aggregation strategies: refined majority voting using Beta-Binomial quantiles and p-value aggregation methods with various score functions.", "questions": "**Missing Baseline:** The experiments lack comparison with multi-view models combined with standard conformal prediction. This baseline would involve end-to-end multi-view architectures applying standard conformal prediction to their outputs. Without this comparison, the relative advantages of p-value aggregation versus direct multi-view modeling remain unclear. Could you provide guidance on when each approach is preferable?\n\n**Exchangeability:** How sensitive are coverage guarantees to violations of Assumption 3.3? What diagnostic tools could practitioners use to assess this assumption?\n\n**Score Function Selection:** What guidance exists for choosing between quantile, area-based, and ℓq-envelope score functions under different conditions?", "limitations": "The authors adequately address several important limitations, particularly the exchangeability assumption and its empirical violations. The computational requirements discussion is appropriate. However, more comprehensive baseline comparisons and specific practical implementation guidance would strengthen the work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Quality:** The theoretical development is rigorous with proper handling of exchangeability assumptions and tied scores. However, the experimental evaluation lacks a critical baseline comparing against multi-view models with standard conformal prediction.\n\n**Clarity:** Well-written with clear mathematical exposition, though some technical details could be better integrated into the main text.\n\n**Significance:** Addresses a practically important problem with meaningful theoretical extensions to conformal prediction. Empirical results show substantial improvements within the tested framework.\n\n**Originality:** The p-value aggregation approach represents a novel contribution with original theoretical analysis of joint p-value distributions.", "was_revised": true, "final_justification": "The authors' rebuttal partially addresses my concerns, particularly regarding their experimental setting where multi-inputs are only available at test time, which is reasonable given the Pl@ntNet application context. However, the missing baseline comparison with multi-view models combined with standard conformal prediction remains a significant limitation that weakens the evaluation. While the authors acknowledge this gap and express willingness to include such comparisons, the current work lacks comprehensive baselines to demonstrate when p-value aggregation is preferable over direct multi-view modeling. Despite these limitations, the work addresses a practically important problem with novel theoretical insights, and the authors show a willingness to address the evaluation gaps. Given the theoretical rigor and potential impact, I raise my score.", "rebuttal": "We thank the reviewer for their feedback and understand their concern. We would like to point out that the comparison with standard conformal prediction using aggregated scores is discussed in Appendix B, and the limitations of the exchangeability assumption in Appendix E. Thanks to the reviewer’s feedback, we will further expand these discussions.\n\n**Missing Baseline:** *The experiments lack comparison with multi-view models combined with standard conformal prediction. This baseline would involve end-to-end multi-view architectures applying standard conformal prediction to their outputs. Without this comparison, the relative advantages of p-value aggregation versus direct multi-view modeling remain unclear. Could you provide guidance on when each approach is preferable?*\n\nWe would like to draw the reviewer's attention to the fact that in our setting, multi-inputs are only present at test time. During both training and calibration, the available data consists solely of individual images. This design choice is motivated by the application context of Pl@ntNet, where the verified data primarily consists of single images. We refer the reviewer to the Pl@ntNet website, where it can be observed that the number of available images ($3.4 \\times 10^7$) is very close to the number of multi-input observations ($2.8 \\times 10^7$) so with an average number of multi-inputs of $1.2$. Nevertheless, our objective was thus to develop a method capable of providing meaningful prediction sets for multi-input sizes that may not have been encountered during training.\nOf course, this does not prevent us from comparing our approach to a standard prediction method that has access to multi-inputs. The advantage of our aggregation methods lies in the number of data points required for calibration. As a reminder, class-conditional conformal prediction methods typically require at least $\\simeq \\alpha^{-1}$ samples per class to yield non-trivial results. If the user has access to this number of multi-view samples per class, standard conformal prediction methods are expected to perform adequately.\n\nHowever, in practice, we have very few available calibration samples for certain classes, possibly even fewer than $m$, the size of the multi-input. This situation typically arises in long-tail datasets, of which the Pl@ntNet dataset is a representative example.\nBy comparing prediction errors individually and aggregated, our methods are capable of handling such cases in a non-naive manner. For instance, our methods perform well even when the multi-input size exceeds the total number of calibration points available for a given class.\n\nWe discussed this advantage in Appendix B (second naive approach: mean calibration), and indeed observed that standard conformal prediction methods tend to fail when $m$ becomes too large relative to the available calibration data. This comparison is carried out using a naive aggregation method (averaging the predictions), but the observation extends to more sophisticated approaches, as the limitation stems more from the number of available calibration samples than from the specific aggregation strategy used for the multi-view inputs.\n\nWe are, however, open to comparing our method with standard CP applied to more sophisticated aggregation strategies. We would be particularly interested in any references to aggregation methods, especially those that are adaptive in $m$.\n\n**Exchangeability:** *How sensitive are coverage guarantees to violations of Assumption 3.3? What diagnostic tools could practitioners use to assess this assumption?*\n\nThis assumption is central to our methodology and is further examined in Appendix E, where we apply our approach to data that do not satisfy it. We observe a loss in coverage of p-values based methods.\nIf a few labeled multi-input samples are available, one may attempt to assess the exchangeability of the data, for example using Mann–Whitney–Wilcoxon tests.\nWhile relaxing this assumption is essential for broader applicability, we view the present work as a meaningful and necessary first step to this problem.\n\n**Score Function Selection:** *What guidance exists for choosing between quantile, area-based, and $\\ell_q$-envelope score functions under different conditions?*\n\nThese score functions yield comparable results across the different settings in which we have used them. We do not have strong recommendations in favor of one over the others. The quantile score has sometimes performed slightly better than the others in some of our experiments. However, we did not find these results to be significant relative to the simplicity of the area-based and $\\ell_q$ score functions.\n\nWe hope we have addressed your concerns and are happy to discuss further if not."}, {"review_id": "BdiDMkT5rF", "reviewer": "Reviewer_Joh5", "summary": "This paper proposes a framework to integrate multi-input information into conformal prediction to further improve the predictive interval.", "questions": "Your results for the LifeCLEF dataset show a significant difference in performance based on the temperature scaling of the softmax outputs. This suggests a strong interplay between the quality of the base classifier's scores and the effectiveness of your aggregation. Could you expand on this? For example, would applying a more sophisticated calibration method (like Ensemble Temperature Scaling) to the base classifier's scores before computing p-values further improve the reduction in set size?", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strength: original; I also enjoyed reading this paper.\n\nWeakness. If I understand correctly,Section 5 p-value score aggregation requires a Monte Carlo simulation to calibrate the quantiles of the score v for every class? For datasets with thousands of classes, this could be computationally prohibitive. The paper does not discuss the computational complexity or potential optimizations for this critical step.", "was_revised": true, "final_justification": "My score remains unchanged. Overall this is a solid paper.", "rebuttal": "We thank the reviewer for the positive feedback and provide answer to your questions below.\n\n**Weakness.** *If I understand correctly, Section 5 p-value score aggregation requires a Monte Carlo simulation to calibrate the quantiles of the score v for every class? For datasets with thousands of classes, this could be computationally prohibitive. The paper does not discuss the computational complexity or potential optimizations for this critical step.*\n\nThe quantile of the score $v$ must indeed be computed for each class. However, we do not consider this computationally limiting for several reasons:\n- The p-values simulation relies solely on a draw without replacement (Algorithm 1), which is both easily parallelizable and computationally inexpensive ($O(n_y + m)$ for class $y$) where $n_y$ is the number of points of calibration of class $y$..\n- As for an application, the number of points by class $n_y$ in the calibration is fixed, although this is not formally valid; these quantiles can be precomputed and stored for different values of $m$. Since the distribution depends only on $n_y$, the number of points per class, and $m$, the number of multi-inputs, these quantiles can even be reused for other applications.\n- For experiments on PlantClef, the number of classes is around 700, and the simulation can be run on a laptop. Moreover, in practice, the number of repetitions $m$ required is typically small.\n\n*Your results for the LifeCLEF dataset show a significant difference in performance based on the temperature scaling of the softmax outputs. This suggests a strong interplay between the quality of the base classifier's scores and the effectiveness of your aggregation. Could you expand on this? For example, would applying a more sophisticated calibration method (like Ensemble Temperature Scaling) to the base classifier's scores before computing p-values further improve the reduction in set size?*\n\nIn general, a well-performing classifier is expected to produce prediction sets of small size. For a perfect classifier, the prediction set would contain only the true class. In our case, temperature scaling does not improve top-$k$ performance, since the transformation preserves the ordering of the scores. However, it appears that the smoothing effect of temperature scaling helps break ties between class scores caused by numerical approximations (by $0$ or $1$) in the softmax function . Such ties can negatively impact conformal prediction methods by producing overly large prediction sets.\n\nIn scenarios where the classifier is overly confident yet incorrect, the softmax scores may degenerate into binary values ($0$ or $1$), which leads to excessively large prediction sets (see, for instance, the curve “calibration” in Figure 4 representing the values of the scores of the synthetic data.).\n\nTo summarize:\n- In our setting, the main effect of temperature scaling seems to break ties between scores caused by numerical approximation.\n- More generally, any calibration method that improves the classifier’s performance is likely to be beneficial for the construction of prediction sets.\n\nWe remain available during the discussion phase if the reviewer wishes to further discuss these points."}, {"review_id": "hq5ACEVnaG", "reviewer": "Reviewer_RZJ6", "summary": "This paper presents a novel framework for reducing uncertainty in class-conditional conformal prediction, specifically tailored to scenarios where multiple observations (e.g., images) of a single instance are available at test time. This setting is well-motivated by citizen science applications such as Pl@ntNet. The core idea is to perform class-conditional conformal prediction on each individual observation to generate p-values, and then aggregate these p-values to make a final prediction.\n\nThe authors first propose a refinement of majority voting by replacing the heuristic threshold with a quantile from the Beta-Binomial distribution. Second, they introduce a more powerful class of aggregation methods based on score functions defined directly on the vector of p-values. These methods—including scoring based on quantiles or the geometric area of the p-value trajectory—better capture joint information from multiple observations.\n\nThe authors validate their methods on synthetic data and the LifeCLEF 2015 plant identification dataset, demonstrating a significant reduction in prediction set size compared to baselines, while maintaining the desired class-conditional coverage guarantees.", "questions": "- **Line 49**: In your setting description, is *m* (the number of observations) considered fixed or a random variable? In practice, users may submit different numbers of images. Does the framework still hold if *m* varies per test instance, as long as calibration for the p-value scores (Algorithm 1) is done for that specific *m*? This should be clarified.\n\n### Typos\n- **Line 161**: “is” should be “are.”  \n- **Line 143**: “conditionnal” should be “conditional.”", "limitations": "The authors are transparent about the main assumption underpinning their framework (Assumption 3.3), and they strengthen the paper by providing an empirical analysis in Appendix E where this assumption is violated.\n\nA second, less discussed limitation is the potential computational cost of the p-value scoring methods (Section 5), which require separate Monte Carlo calibrations for each class. This may become prohibitive in settings with a very large number of classes (K), although the calibration is a one-time offline process.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### Strengths\n- The paper addresses uncertainty quantification in a setting where multiple inputs are available at test time, a scenario where standard conformal prediction is not directly applicable (as discussed in Appendix B).\n\n- The derivation of the joint distribution of ordered conditional p-values (Theorem 3.4) provides a solid foundation for Sections 4 and 5.\n\n- The paper offers two methodological contributions: (1) a refined majority voting scheme using the Beta-Binomial distribution (Section 4), which is a direct and elegant improvement over existing heuristics; and (2) a general framework for scoring p-value vectors based on simulation (Section 5), which flexibly captures the structure of joint p-values.\n\n- Experiments on both synthetic and real-world data show that the proposed methods, particularly the p-value scoring envelopes, produce significantly smaller prediction sets than baselines, while maintaining valid coverage.\n\n### Weaknesses\n- A major concern is the strong assumption of exchangeability among the multiple inputs (i.e., views) of a test instance. In real-world multi-view settings—such as images taken from different angles, lighting conditions, or crops—this assumption is often violated due to inherent correlations between views. For example, an image and its 90° rotation are not identically distributed and thus not exchangeable. The theoretical guarantees of the proposed framework rest on Assumption 3.3, which requires exchangeability between test-time inputs and the calibration data from the same class. To sidestep this issue, the experimental setup simulates multi-view data using independently drawn samples from the same class, thereby artificially restoring exchangeability. While this limitation is acknowledged, it raises significant doubts about whether the method can be reliably applied to realistic multi-view scenarios, particularly in cases where users submit similar or low-quality images.\n\n- The evaluation primarily compares the proposed methods to majority voting—a simple heuristic that does not take into account the scale of p-values. For a fairer and more comprehensive comparison, it would be valuable to benchmark against more advanced aggregation techniques.\n\n- Most plots comparing coverage and prediction set sizes (except for Figure 5) do not include any indication of statistical uncertainty (e.g., error bars or confidence intervals). This limits the reader's ability to assess the robustness and significance of the reported improvements.\n\n- While the framework is general, the empirical validation is limited to basic softmax scores. Demonstrating the method’s effectiveness with stronger conformity scores, such as Adaptive Prediction Sets (APS) [1], would reinforce the claims—especially since APS has been shown to improve conditional coverage.\n\n- In Figure 2, the refined voting methods (Beta-Binomial and Binomial) consistently exhibit over-coverage, providing coverage significantly above the nominal 90% level. A discussion of the causes and potential corrections for this behavior would strengthen the analysis in Section 4.\n\n- Some methodological choices lack justification. For instance, the choice of quantile level $\\lambda$ in the quantile envelope method (line 271) is not explained. Similarly, the decision to approximate the Beta-Binomial distribution with a Binomial (Proposition 4.2) is unclear, particularly given that Beta-Binomial quantiles are efficiently computable (e.g., via `scipy.stats.betabinom`).\n\n- While the main results are strong, additional metrics could provide further insights. Reporting the median prediction set size (e.g., in the Appendix) would offer robustness to outliers. Furthermore, evaluating coverage conditional on the input features (e.g., worst-slab coverage, as reported in [1]) could show whether the aggregation methods improve this stricter form of conditional validity.\n\n[1] Romano, Yaniv et al. \"Classification with valid and adaptive coverage.\" NeurIPS 2020.", "was_revised": true, "final_justification": "The authors addressed the main concerns, notably by adding experiments with APS scores and committing to include confidence intervals in the plots. Their explanations regarding methodology and evaluation were satisfactory, and the remaining limitations do not significantly affect the overall contribution. Based on this, I maintain my recommendation to accept.", "rebuttal": "We thank the reviewer for the positive feedback and detailed review. Below, we provide clarifications on the points that remain unclear and further discuss the weaknesses raised. We have revised the paper accordingly in light of the suggestions provided by the reviewer.\n\n**Weaknesses**\n\n*A major concern is the strong assumption of exchangeability among the multiple inputs (i.e., views) of a test instance. In real-world multi-view settings—such as images taken from different angles, lighting conditions, or crops—this assumption is often violated due to inherent correlations between views. For example, an image and its 90° rotation are not identically distributed and thus not exchangeable. The theoretical guarantees of the proposed framework rest on Assumption 3.3, which requires exchangeability between test-time inputs and the calibration data from the same class. To sidestep this issue, the experimental setup simulates multi-view data using independently drawn samples from the same class, thereby artificially restoring exchangeability. While this limitation is acknowledged, it raises significant doubts about whether the method can be reliably applied to realistic multi-view scenarios, particularly in cases where users submit similar or low-quality images.*\n\nWe are aware of this limitation and have made an effort to be transparent about it. We are really interested in addressing it, but believe that this challenge warrants a dedicated study in future work.\n\n*The evaluation primarily compares the proposed methods to majority voting—a simple heuristic that does not take into account the scale of p-values. For a fairer and more comprehensive comparison, it would be valuable to benchmark against more advanced aggregation techniques.*\n\nWe chose to compare our method to majority voting, as it has been used in conformal prediction for aggregating prediction sets. We considered that standard p-value aggregation methods, such as Bonferroni or Simes, would yield similar results based on the observations in Figure 1. However, we are open to testing these methods and welcome any suggestions from the reviewer regarding alternative p-value aggregation strategies.\n\n*Most plots comparing coverage and prediction set sizes (except for Figure 5) do not include any indication of statistical uncertainty (e.g., error bars or confidence intervals). This limits the reader's ability to assess the robustness and significance of the reported improvements.*\n\nConfidence intervals have so far been computed only on synthetic data. For real datasets, we plan to include them in the camera-ready version by repeatedly shuffling the calibration and test sets.\n\n*While the framework is general, the empirical validation is limited to basic softmax scores. Demonstrating the method’s effectiveness with stronger conformity scores, such as Adaptive Prediction Sets (APS) [1], would reinforce the claims—especially since APS has been shown to improve conditional coverage.*\n\nWe previously tested the use of the APS score and obtained results similar to those with softmax scores. This score is also implemented in the provided code (see supplementary material). We will include these experiments in the camera-ready version.\n\n*In Figure 2, the refined voting methods (Beta-Binomial and Binomial) consistently exhibit over-coverage, providing coverage significantly above the nominal 90% level. A discussion of the causes and potential corrections for this behavior would strengthen the analysis in Section 4.*\n\nThis phenomenon arises from the discrete nature of the statistic that counts the number of sets containing each label. As a result, an exact quantile at level $1- \\alpha$ may not exist, leading to coverage that exceeds the threshold. In the case of the Beta-Binomial quantile, which corresponds to the exact distribution of the statistic, this issue can be addressed through additional randomization to achieve exact coverage at level $1- \\alpha$. For the Binomial quantile, however, the situation is less straightforward, as it is an approximation of the true distribution and does not achieve a marginal coverage for large $m$.\n\n*Some methodological choices lack justification. For instance, the choice of quantile level in the quantile envelope method (line 271) is not explained. Similarly, the decision to approximate the Beta-Binomial distribution with a Binomial (Proposition 4.2) is unclear, particularly given that Beta-Binomial quantiles are efficiently computable (e.g., via scipy.stats.betabinom).*\n\nThese choices stem from different aggregation methods proposed in the literature. As briefly discussed in Remark 4.3, the use of a Binomial quantile is connected to the aggregation method of Gasparin and Ramdas (2024) in the case of independent sets. We show that this choice also remains valid when the sets are correlated through the calibration dataset. \nThe quantile envelope, sometimes referred to as a template, is commonly used in multiple testing procedures (see, e.g., Blanchard et al., 2024; Li et al., 2024) or in conformal prediction for ranking tasks (Fermanian et al., 2025), see discussion in Section 2.3. This envelope is intended to capture the behavior of each marginal by choosing for each coordinate the quantile of the marginal distribution at a common level $\\lambda_\\alpha$. We link this envelope to the score function $v_Q$ and this level $\\lambda_\\alpha$ is seen as a threshold of the score chosen using a conformal step.\n\n*While the main results are strong, additional metrics could provide further insights. Reporting the median prediction set size (e.g., in the Appendix) would offer robustness to outliers. Furthermore, evaluating coverage conditional on the input features (e.g., worst-slab coverage, as reported in [1]) could show whether the aggregation methods improve this stricter form of conditional validity.*\n\nThank you for the recommendation. We have computed the median of lengths for the different methods and number of multi-inputs $m$, and observed results very similar to the behavior of the means. We will include this metric in the Appendix.\n\n**Questions**\n\n*Line 49: In your setting description, is m (the number of observations) considered fixed or a random variable? In practice, users may submit different numbers of images. Does the framework still hold if m varies per test instance, as long as calibration for the p-value scores (Algorithm 1) is done for that specific m? This should be clarified.*\n\nThank you for the remark. We will clarify this point in the revised version. We would like to point out that our motivated application is a case where $m$ effectively varies at test time. In Pl@ntNet, users may submit a different number of images. However, Pl@ntNet has access to only a (relatively) small number of trustworthy labeled multi-input and relies only on lonely labeled images of plants, which motivates the setting considered. In our method, the thresholds (exact quantiles of simulated ones) of all our methods depend on $m$ and are then recomputed (or loaded) depending on the size of the new multi-input.\n\nTheoretically, we have supposed that $m$ is fixed. However, as long as Assumption 3.3 holds conditionally on $m$, all the theoretical results remain valid, can be stated conditionally on $m$, and then are marginally valid with respect to the randomness of $m$.\n\n**Typos**\n\nThanks for the typos. We have updated the paper accordingly.\n\n**Limitations:**\n\n*A second, less discussed limitation is the potential computational cost of the p-value scoring methods (Section 5), which require separate Monte Carlo calibrations for each class. This may become prohibitive in settings with a very large number of classes (K), although the calibration is a one-time offline process.*\n\nThe quantile of the score $v$ must indeed be computed for each class. However, we do not consider this computationally limiting for several reasons:\n- The p-values simulation relies solely on a draw without replacement (Algorithm 1), which is both easily parallelizable and computationally inexpensive ($O(n_y + m)$ for class $y$) where $n_y$ is the number of points of calibration of class $y$..\n- As for an application, the number of points by class $n_y$ in the calibration is fixed, although this is not formally valid; these quantiles can be precomputed and stored for different values of $m$. Since the distribution depends only on $n_y$, the number of points per class, and $m$, the number of multi-inputs, these quantiles can even be reused for other applications.\n- For experiments on PlantClef, the number of classes is around 700, and the simulation can be run on a laptop. Moreover, in practice, the number of repetitions $m$ required is typically small.\n\nWe remain available during the discussion phase if the reviewer wishes to further discuss these points.\n\n[1] Romano, Yaniv et al. \"Classification with valid and adaptive coverage.\" NeurIPS 2020."}, {"review_id": "e1Gf4KEBfl", "reviewer": "Reviewer_WJtv", "summary": "This paper addresses class-conditional conformal prediction with multiple input covariates. Building upon the perspective of p-values and mathematical tools for their joint distribution, the paper proposes techniques to merge multiple prediction sets by aggregation methods including and beyond majority voting, which corresponds to different choices of non-conformity score functions, and reduces the size of the uncertainty set compared to the original majority voting. Under the critical assumption that the scores within the same class are exchangeable, class-conditional coverage is guaranteed. The technique is validated by experiments on both synthetic data and realistic data from citizen science, with simulated multiple inputs.", "questions": "1. Is it possible to empirically validate the technique with natural datasets containing multiple inputs, instead of synthetic construction of multiple inputs?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n1. The paper has unified aggregation methods for prediction sets as forming an envelope around their p-values. Subsequently, it is well-motivated from the observation that the envelope for majority voting is not tight enough. The paper generalizes aggregation methods through various choices of score functions taking p-values as input. The resulting framework reduces the size of the prediction set by design, and is also empirically validated.\n\n### Weaknesses\n1. The critical assumption that the scores within the same class are exchangeable (Assumption 3.3) weakens the applicability of the technique. In the context of citizen science, it implies an impractical assumption that images of the same plant are exchangeable with images of different plants from the same species. The paper also reports that the exchangeability assumption does not hold for the LifeCLEF Plant Identification Task 2015 dataset. Therefore, exchangeability is enforced by random grouping of images of the same class. Empirical results on natural datasets without synthetic transformation would be more convincing to test the robustness of the technique when the assumption is violated.\n\n### Comments\n1. The core mathematical tool for the exact distribution of multiple values is mostly based on Gazin et al, though the results have been extended to handle ties between scores, and directly extended to be conditioned on classes.", "was_revised": true, "final_justification": "Considering the feedback from the authors, I maintain my vote for acceptance (marginal above the threshold). \n\nThe major contribution of the paper is to address conformal prediction with multi-view inputs.  The proposed framework unifies and generalizes aggregation methods for prediction sets through various choices of score functions taking p-values as input.\n\nThe limitation remains over the class-conditional exchangeability assumption. Thank the author for their clarification and reference to the appendix. In fact, the exchangeability assumption does not hold for the LifeCLEF Plant Identification Task 2015 dataset.", "rebuttal": "We thank the reviewer for the feedback and positive review. As discussed below, we would like to point out that we have evaluated our method on natural dataset in Appendix E to observe the limitation of the exchangeability assumption. \n\n**Comments**\n\n*The core mathematical tool for the exact distribution of multiple values is mostly based on Gazin et al, though the results have been extended to handle ties between scores, and directly extended to be conditioned on classes.*\n\nWe would nonetheless like to emphasize that, although our main result builds on that of Gazin et al. (2024), its application in the context of conformal prediction for multi-input settings is entirely novel. As the reviewer rightly pointed out, our theoretical contributions lie in the treatment of ties and the class-conditional use of the tool.\n\n**Questions:**\n\n*Is it possible to empirically validate the technique with natural datasets containing multiple inputs, instead of synthetic construction of multiple inputs?*\n\nWe have evaluated our method on the LifeClef dataset without artificially sampling multi-inputs and observed that the exchangeability assumption is violated in this case, as detailed in Appendix E. As our method works well under this assumption, our focus is now more on relaxing the exchangeability assumption than on identifying datasets that satisfy it. We believe we have been transparent about this limitation and intend to address it in future work.\n\nWe remain available during the discussion phase if anything remains unclear."}]

Author rebuttal:

Meta-review: This paper provides a framework for reducing uncertainty in class-conditional conformal prediction, tailored to scenarios with multiple observations of a single instance. Contributions include deriving the exact joint distribution of class-conditional conformal p-values and proposing refined majority voting using Beta-Binomial quantiles and p-value aggregation methods with various score functions as aggregation strategies. The authors conduct experiments evaluating their approach on synthetic data and the LifeCLEF 2015 plant identification dataset and observe how prediction set size is affected by their techniques.

The reviewers appreciated the rigorous theoretical development and how the framework addresses a practically important problem, and the authors’ rebuttals clarified some questions reviewers had. In revising, I recommend the authors consider adding experimental comparisons to more sophisticated aggregation approaches than majority vote, as well as application of standard conformal prediction to a multiview system as described by reviewer 4nuN to demonstrate when the aggregation of p-values that the framework uses has an advantage.
Decision: Accept (poster)