Conformal Prediction for Long-Tailed Classification
Authors: Tiffany Ding
Jean-Baptiste Fermanian
Joseph Salmon
Venue: ICLR 2026 Poster (2026)
Keywords: conformal prediction
uncertainty quantification
long tail
class imbalance
fine-grained image classification
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: Many real-world classification problems, such as plant identification, have extremely long-tailed class distributions. In order for prediction sets to be useful in such settings, they should (i) provide good class-conditional coverage, ensuring that rare classes are not systematically omitted from the prediction sets, and (ii) be a reasonable size, allowing users to easily verify candidate labels. Unfortunately, existing conformal prediction methods, when applied to the long-tailed setting, force practitioners to make a binary choice between small sets with poor class-conditional coverage or sets that have very good class-conditional coverage but are extremely large. We propose methods with marginal coverage guarantees that smoothly trade off set size and class-conditional coverage. First, we introduce a new conformal score function called prevalence-adjusted softmax that optimizes for macro-coverage, defined as the average class-conditional coverage across classes. Second, we propose a new procedure that interpolates between marginal and class-conditional conformal prediction by linearly interpolating their conformal score thresholds. We demonstrate our methods on Pl@ntNet-300K and iNaturalist-2018, two long-tailed image datasets with 1,081 and 8,142 classes, respectively.

Published as a conference paper at ICLR 2026
CONFORMAL PREDICTION FOR LONG-TAILED CLAS- SIFICATION
Tiffany Ding' Jean-Baptiste Fermanian"
Joseph Salmon
University of California, Berkeley, UUniversity of Montpellier, Inria, CNRS ti siffany.ding@berre wwwggeereeeeeeee.e.e..e... edu (jean-baptiste. wtt.riisstteeeeiarrraaaeaaa fermanian, joseph salmmmeemmmmmnia..ff
ABSTRACT
Many real-world classification problems, such as plant identification, have ex- tremely long-tailed class distributions. In order for prediction sets to be useful in such settings, they should (i) provide good class-conditioaal coverage, ensur- ing that rare classes are not systematically omitted from the prediction sets, and (ii) be a reasonable size, allowing users to easily verify candidate labels. Unfor- tunately, existing conformal prediction methods, when applied to the long-tailed setting, force practitioners to make a binary choice between small sets with poor class-conditional coverage sets that have very good class-conditional coverage but are extremely large. We propose methods with marginal coverage guaran- tees that smoothly trade off set size and class-conditional coverage. First, we introduce new conformal score function called prevalence-adjuster softmax that optimizes for macro-coverage, defined the average class-conditional coverage across classes. Second, we propose new procedure that interpolates between marginal and caas-coondiionall conformal prediction by linearly interpolating their conformal score thresholds. We demonstrate our methods on Pl@ ntNet-300KK and iNaturalist-2018 two long-tailed image datasets with 1,081 and 8, 8.142 classes, re- spectively.
INTRODUCTION
Prediction sets are useful because they replace single fallible point prediction with a set that is likely to contain the true label. In classification, prediction sets are most useful in settings with many classes, as they narrow down the label space to a set of labels that human decision makers can then verify. Consider amateur plant enthusiast who wants to identify plant. The enthusiast struggles to identify plants on their own, but when presented with a short list of possible matches, is easy for them to go through the list and select the correct species (e.g., by comparing their plant with images of potential matches).
Another key characteristic of plant identification is its extremely long-tailed class distribution. As shown in Figure 1, there are thousands of images of common plants but only a handful for rare plants. Such skewed distributions also appear in animal identification and disease diagnosis. An added challenge is that we often care even more about identifying instances of the rare classes than the popular ones. In botany, scientists may want prioritize the acquisition of examples of endangered plant species, which fall in the tail (Figure I). In medicine, catching the few cases of an aggressive cancer early matters more than perfectly classifying common benign lesions. In collaborative iuman-AAl systems where a human generates labels based on Al recommendations and these labels are used to improve the predictive model in future training rounds, neglecting niche classes can accelerate "model collapse" (Shumailov et al., 2024), shrinking the model's effective label space over time and degrading accuracy. This motivates pursuing prediction sets where all classes have high probability of being correctly included in the prediction set. Beyond training good predictive models in this setting, an additional challenge for post-hoe uncertainty quantification that most available examples of rare classes used for model training, leaving these classes with few or zero holdout examples to use for calibrating uncertainty quantification methods.
Formally, let X X' be features with unknown label Y Y 21, D']). Our goal is to construct a set-generating procedure C : X + 2>, where 2 denotes the set all subsets of V, with good class-conditional coverage. For y E Y, the class-conditional coverage for class

---

Published as a conference paper at ICLR 2026
Net 301K Species by IUC Status
oxamples
6,000
The Schiiw
4,000
Loct
Retvepeal Polegsst
2,000
200
4000
600
800
1003
(sosted by prevalenoe)
Figure 1: The number of train examples of each species in Pl@ntNet-300K (Garein al., 2021). We highlight threatened species, as defined by the International Union for Conservation of Nature (https: / /uuen, org), which are particularly important to identify for biodiversity monitoring purposes. Note that most of these species are in the tail of the distribution.
is CondCov(C,y) = P(Y C(X) YY= = -y). This can be contrasted with marginal coverage, which is simply MarginalCov((C) P( C(X)). To ensure that our prediction sets are useful for identifying instances of all classes, including rare ones, we aim for high class- cass--onniitioal coverage across all classes. In addition to coverage, set size also crucial, as large sets are often impractical. For instance, in plant identification, users lack the time to review prediction sets containing hundreds of species. Ideally, we would generate prediction sets with both high class-conditional coverage and small size. Unfortunately, there is an inherent trade-off between small set sizes and class-conditional coverage in the long-tailes tailed setting.
Conformal prediction (CP) provides methods guaranteed to achieve marginal class-conditional coverage under no distributional assumptions. STANDARD CP, the most basic conformal prediction method, yields small sets but only guarantees marginal coverage and often has poor class-conditional coverage for some classes. However, approaches targeting class-conditional coverage struggle in scenarios where many classes have few examples. In such settings, CLASSWISE CP (an instantiation of Mondrian conformal; see Vovk et al., 2005) and rank-calibrated class-condittonal CP (Shi et al., 2024) produce large sets, and Clustered CP (Ding et al., 2023) effectively defaults STANDARD CP for rare classes that it is unable to assign to a cluster.
Objective. Our goal is produce prediction sets that maintain marginal coverage while striking more useful trade-off between set size and class conditional coverage compared to STANDARD or CLASSWISE CP. We approach this in two ways.
APPROACH I: Target relaxed notion of class conditioon coverage. Motivated by the multi-class classification concept of "macro- accuroccc which is the average of class-wise accuracies (Lewis, 1991), we aim to construct sets with high macro-coverage, which is the average of class-conditional coverages:
MacroCov(C) =
Σ P(Y E(()) Y =y) BEy
Σ yEy
CondCov(C, v).
(I)
D
D
In contrast, marginal coverage is weighted average of class-conditional coverages where the weight of class y is its prevalence p(y):
MarginalCov(C=>>> = =Erou CondCov(C,y).
DEY
This (over)emphasizes coverage of more frequent classes. We derive the form of the prediction set that optimally trades off set size and macro-coverage under oracle knowledge of the underlying distribution and design a conformal score function, called prevalence-adjusted softmax (PAS), which approximates these oracle optimal sets given imperfect classifier p(ylæ:) and estimated label distribution p(y).
APPROACH II: Target class-conditional coverage, then back off (until the size is reasonable). We propose a simple procedure called INTERP-Q that interpolates between CLASSWISE CP and STANDARD CP in literal sense by linearly interpolating their quantile thresholds, This method allows the user to choose their position on the trade-off curve between set size and class-conditional coverage via the interpolation parameter.
The choice between these two approaches depends on the setting. Targeting macro-coverage (APPROACH 1) implies that care equally about the coverage of all classes, but it is acceptable
2

---

Published as a conference paper at ICLR 2026
if a few classes have poor coverage, so long as the average class-conditions coverage is high. On the other hand, starting from CLASSWISE and softening (APPROACH II) implies that we want all classes to have good coverage. This approach also comes with parameter that the user vary depending on their preference between small sets and class-conditional coverage.
1.1 RELATED WORK
Class-conditional conformal prediction. Conformal prediction provides a way to construct pre- diction sets with coverage guarantees given calibration dataset that is exchangeable with the test point (Vovk et al., 2005; Angelopoulos & Bates, 2023). We are specifically interested in class- conditional coverage, which is difficult achieve in a useful way when many classes have very few calibration examples, as is the case in long-tailed settings. Previous works on class-conditional conformal focus on easy settings with most ten classes (Shi et al., 2013; Löfström et al., 2015; Hechtlinger et al., 2018; Sadinle et al., 2019), or the harder setting of many classes with some class imbalance but still lacking a truly long tail (Ding et al., 2023; Shi al., 2024).
Optimally trading off set size and coverage. Prediction sets should achieve the desired coverage while being as small as possible, so as to maximally informative. In general, the set-generating procedures that optimally navigate the coverage-size trade-off depend on the density of the under- lying distribution (Lei et al., 2013; Lei & Wasserman, 2014; Vovk et al., 2016; Sadinle et al., 2019). Although the true density is unknown in practice, these theoretically optimal sets serve as guidelines for designing effective conformal score functions or new conformal procedures for various target quantities, such as X X-conditional coverage (e.g., APS from Romano et al., 2020, RAPS from An- gelopoulos et al., 2021, SAPS from Huang et al., 2023, CQC from Cauchois et al., 2021) or size (Denis& & Hebiri, 2017; Kiyani et al., 2024). In APPROACH I, construct score function inspired by the oracle sets that optimally trade-off set size and macro-coverage, a coverage target that has not been previously explored in this context.
Learning from long-tailed data. Many real-world classification problems exhibit long-tailed dis- tributions, a challenge particularly pronounced in
visual categorization, field that has recently attracted substantial interest, especially in biodiversity (Van Horn et al.. 2015; 2018; Garein et al., 2021; Wang et al., 2022). The goal is to classify images into highly specific subcategories (e.g., plant species), which often differ only subtly. While extensive research has focused on im- proving conditonndiional top-1 or top-k accuracy for such tasks (Russakovsky et al., 2015; Lapin et al., 2015; Liu et al., 2019; Garcin et al., 2021; Zhang et al., 2023), less attention has been devoted to constructing high-quality prediction sets long-tailed settings, beyond the naive approach of selecting the top-k predictions. While methods like logit adjustment for long-tail learning (Menon et al., 2021) and focal loss for dense object detection (Lin et al., 2017) address class imbalance through reweighting or output adjustment to improve classification performance, designing robust prediction sets under long-tailed distributions remains understudied. Addressing this gap is critical for systems like ntNe (Joly et al., 2014), a plant identification app where users upload images and receive candidate species matches.
1.2 PRELIMINARIES
Let Deal = {(,,, Y2) for i E [m]} be a calibration set, where (X,, Y.) for 1,..., are exchangeable with the test point (Xn+1, Yn+1). where the label Yu+ is unknown. For a class E V, we use I, = {i E [n] Y; = y) to denote the set of calibration points with label y and n, μZ,l the number of calibration points with label y. Let a E [0, 1] be user-specified probability of miscoverage. We use s: X' x y + R to denote a conformal score function, where smaller values of s(x, y) indicate that the pair (sr y) conforms better with previously seen data.
Notation. For a positive integer n, let [n] :=
Score function. We focus on the split conformal setting in which the score function s con- structed using predictive model trained on data separate from the calibration set Deat- We use the softmax conformal score function, defined as Ssoftmax (x, 3) = 1 - p(yl x)). where p(y|z) is the predicted probability for class y for input æ obtained from the softmax output of trained neural network. This is also known as the Least Ambiguous Classifier (LAC) score (Sadinle et al., 2019). Our methods can be readily adapted to other scoring functions but we focus this score because alternatives like APS (Romano et al., 2020) and its variants produce larger sets and are primarily designed for X cooodittinnal coverage, which is outside our scope.
3

---

Published as a conference paper at ICLR 2026
Algorithm 1 Conformal prediction
Require: Score function X x + R, miscoverage level [0, 1]. calibration set Deal {(,, Y)) test point Xx+1, threshold function q : R" [0, D*
Compute scores: S; < s(X,, Y2) for 1,...,,
Compute thresholds: <- ((
return Prediction set ((x++1) = {Vr : ((X++1))) S Jy where oy is the y-th entry of
Conformal prediction as thresholded sets. Let q = (71, 92. 91r)) be | vec- tor of score thresholds. Define the q-thresholded set q) = {u E y sXX, .x, y) S dy)
(3)
Conformal prediction provides principled ways set q as a function of the calibration data Deal and the chosen miscoverage level a so as to achieve coverage guarantees.
STANDARD conformal prediction constructs sets as CSTAND. (X) C(X; (STAND.) for ÂSTAND. (4....,q) where
Quantile, (n + Σ 5xxxN n 500)
1
(4)
and where Quantile, (P) (n E R: PPVV S <x) 2 2 denotes the level-y quantile of a random variable V P and 6, is the Dirac measure at point s (this follows the notation of Tibshirani et al., 2019). By setting the thresholds in this way, STANDARD conformal prediction sets achieve marginal coverage guarantee (Vovk et al., 2005):
P(Y E C(X; (STAND.)
CLASSWISE conformal prediction constructs sets CCLASSWISE(X) = C(X: (CLASSWISE) where the y-th entry of QCLASSWISE
CW
Quantila (
Σ
6x(x.))
ny + 820).
(6)
My +1
iEI,
CLASSWISE conformal prediction sets achieve class-conditional coverage guarantee (Vovk al., 2005):
IP C(X; (CLASSWISE) Y = y) 2 1 1-10 a, for all y E By marginalizing over y. this implies that CCLASSWISH (X) also achieves - a marginal cover- age.
We explicitly describe the meta-algorithm for conformal prediction in Algorithm 1. The existing methods, STANDARD and CLASSWISE, and the methods will propose in the next section are instantiations of this meta-algorithm for the score functions and threshold functions described in Table
METHODS
We take two approaches to the problem of simultaneously achieving reasonable set sizes and reasonable class-conditional coverage in the long-tailed classification setting, each lead- ing method. APPROACH I targets the weaker objective of macro-coverage. From this, we derive an oracle set that optimally balances set size and macro-coverage We then define a conformal score function (PAS) by replacing the true conditional density with its estimate. We also consider an exten- sion (WPAS) that prioritizes coverage of user- specified classes. APPROACH II addresses the trade-off by interpolating between class- conditional and marginal score quantiles, leading to a simple new procedure (INTERP-O) QQ). We defer the formal statements of propositions and their proofs Appendix B.
Table 1: Summary of the conformal methods considered. MargCov refers to the marginal cov- erage guarantce of the method.
Score function
Threshold function
MargCov
STANDARD
any
ASANDD (4)
a
CLASSWISE
any
Gc.wss (6)
a
PAS*
SPAS (1))
[STAN.. (4)
a
WPAS*
sweAS (14)
(STAND. (4)
1
INTERP-O Q
any
10 (15)
2a
"Our methods
4

---

Published as a conference paper at ICLR 2026
2.1 APPROACH I: TARGETING (WEIGHTED) MACRO-COVERAGE VIA NEW SCORE
We consider the population optimization problem of minimizing the expected set size subject to macro-coverage constraint,
min C:K~22
[X)]
subject to MacroCov(C) 2 3
(8)
and its dual version maximizing macro-coverage subject to an expected set size constraint,
max MacroCov(C) subject to E|IC(X)]] S
C:R+~2
where MacroCov(C) is defined (1) and B 2 0 and A. 2 0.
Proposition 1 (Informal). The solutions of (8) and (9) are of the form
(x) = {y E y plul=)/((x) 2 1},
(10)
for some threshold I. that depends on or K, respectively.
The key takeaway from this proposition is that thresholding on p(y pp)())) optimally balances macro-coverage and expected set size. Specifically, among set-generating procedures with given expected set size, none achieve better macro-coverage than thresholding on p(y //p(v)) Similarly, for a fixed macro-coverage, no other procedure yields smaller expected set size. We contrast this with the solution to the more classical problem of minimizing expected set size subject to marginal or classccondiional coverage, which is given by thresholding on p(y|æ) (Neyman & Pearson, 1933; Sadinle et al., 2019).
Although we do not have access to p(y[z) and p(y) practice, we have estimates ((y(x) and p(y) from our classifier and the distribution of empirical training labels, respectively. By creating prediction sets as C(x) = {y E y p(yl=x//(v) 2 t) for a threshold t, we approximate an oracle Pareto-optimal set. We choose & to achieve 1 - marginal coverage in the following way: Observe
that C can be rewritten as C(x) ) = {u E y (x, y) I -t, where
SPAS y) -p())=//(()
(11)
and PAS stands for prevalence-adiin adjjustee softmax. By setting -t as the STANDARD CP & from (4) using sPAS the score, C. inherits the marginal coverage guarantee of STANDARD CP. In summary, the first method we propose simply running STANDARD CP with the PAS score function (which will refer to STANDARD with PAS), as this achieves the desired marginal coverage guarantee while (approximately) optimally trading off set size and macro-coverage. We emphasize that PAS aims better handle this trade-off but does not directly target macro-coverage guarantee.
Extension to weighted macro-coverage. Recall that macro-coverage is the unweighted average of the class-conditional coverages, and the PAS score function designed to optimize macro- coverage among all set-generating procedures with a certain expected set size. However, in some settings, we may instead wish to optimize for a weighted average of the class-conditional cover- ages (e.g., because it is more important to cover some classes than others). Given user-chosen class weights w(y) for y E y that sum to one, we can similarly define the w-weighted macro-coverage
Mc) = Σ (W())((Y& E C(X) Y y)
(12)
DEY
For (y)) | recover MacroCov and for w(y) p(y) we get MarginalCov. Proposition 2 (Informal). The solutions of (8) and (9) when MacroCov is replaced with MacroCov., are of the form
(x) w(y) p(yl)//(() 2 2t,
(13)
for some threshold t that depends on and 3 K, respectively.
We can approximate these optimal sets by running STANDARD CP with the weighted prevalence- adjusted softmax (WPAS).
WWPAs(x, v) == -w(y p(y xr)
(14)
7(u)
the score function.
5

---

Published as a conference paper at ICLR 2026
2.2 APPROACH II: SOFTENING CLASSWISE CP VIA INTERP-Q
In the previous section, we presented our first solution, a new conformal score function; here we present our second solution, which simple way to interpolate between the behaviors of CLASSWISE and STANDARD by linearly interpolating their quantile thresholds. We call this proce-
dure INTERP-Q for "interpolated quantile" because it constructs sets as CINTERR-O (X) == C(X; 91o)
where the y-th entry of qio is weighted average of q and g defined in (4) and (6), for some weight E [0, 1]. That is,
10 y
T4y
+ (1 T)
for all y E V.
(15)
For classes where G oc due to small ny. replace it with one (the maximum possible value of the Ssoftmax conformal score) before interpolating.
Proposition 3. If q and G.w (for y E Y) are the STANDARD and CLASSWISE conformal quantiles for a E [0, 1). then CINTERE-Q achieves marginal coverage of at least 1 - 2a.
Theoretically, this lower bound is almost tight; we can construct a pathological example where INTERP-Q Q achieves coverage of 1 2a + a², which is close to 1 2ox for small (see Appendix B.3). Empirically, however, we find that INTERP-Q achieves coverage close to aa This is because the real data we test on do not exhibit the pathologies from our example (which uses discrete score distributions that differ greatly between classes).
In Appendix A, we consider an alternative way of instantiating the interpolation idea used by INTERP-Q Q via a method we call Fuzzy Classwise CP, which interpolates by computing weighted quantiles using class dependent weights determined by some notion of class similarity.
EXPERIMENTS
Overview, Code for reproducing our experiments is available at https::/ggt / com/ tiffanyding/ loapiinggoogg-talll--eee wtt.raaiooooooaaalloooaaaaoaaaaa We consider two datasets for long-tailed classifi- cation: P1@KtNet-300K (Garcin et al., 2021) and iNaturalist-2018 (Van Horn et al., 2018). Fig- ure 2 shows the class distributions of the datasets INaturalist-200 is that their test sets are also long-tailed, hindering reliable class-conditional evalu- ation for rare classes. To address this, we create truncated versions (see Appendix C.1 for details) that preserve some of the long-tail structure while allowing for robust estimation of class-conditional metries: Classes in the truncated datasets have 100 test examples each, but we assume the test distri- bution of interest is equivalent to the (long-tailed) train distribution, so we compute marginal metrics by computing a weighted average of the conditional metrics where the weight for class y is the prevalence p(v) computed on train. When it comes to aggregated class-conditional metrics (such as movro-coveragg), the full and truncated datasets produce similar results, so we defer the truncated results to Appendix D. However, when reporting unaggregated class-conditional metrics (as is the case in Section 3.3 below), it is necessary to use the truncated versions.
use. A key challenge of Pl @ #t@ntNet330KK and
iKaturalit 2015
iNacumalist-20 2015
10,000
* vrnagkk
1,000
100
100
2500
Cluss
Figure 2: Class distributions (sorted by prevalence), plotted using a logarithmic scale, of the classical train, val, and test sets in the datasets we experiment on. We further randomly split 30% of val to use for model validation and use the remaining 70% the calibration set Deal. We use the truncated versions to obtain good estimates of class-conditional metrics.
Unless otherwise stated, we use the Ssoftmax score function described in Section 1.2. The base model is ResNet-50 (He et al., 2016) trained using the standard cross-entropy loss (see Appendix D.3 for similar results using focal loss (Lin et al., 2017), a loss designed for class-imbalanced settings). More details about the experimental setup are available in Appendix C.
169% of classes in Pl@mtNet-300K and 90% of classes in iNaturalist have fewer than 10 test examples.

---

Published as a conference paper at ICLR 2026
Metrics. We evaluate each set-generating procedure C: X' -2 that is separate from the data used for model training (and validation) and calibration. Let Jy - [ be the set of indices of the test examples with label y. and define
2 on test dataset {(X7, Y.7
By = 1
Σ 10, EC()))
(16)
BEJ
as the empirical class-conditional coverage of class y.
We consider several ways of aggregating c across y to obtain a scalar metric: (i) the fraction of classes with coverage below threshold (50% in our ex- periments), (ii) the undercoverage gap, defined as the average undercoverage across classes (zero for classes with coverage at least 1 a)), and (iii) the av- erage of & 's, yielding the empirical macrocoverage. This is summarized in the first three rows of Table 2. Which of these metrics is most natural depends on the goal of the practitioner. Furthermore, we con- sider the standard prediction set metrics of marginal coverage and average set size, detailed in the last two rows of Table
Table 2: Our evaluation metrics, by is the empirical coverage for class y on the test set of N points, - a is the target coverage level, and |J] is the number of classes.
Metric name
Definition
FracBelow50%
Σ
2/, $ 0.5}
2c
UnderCovGap
Σ
max(1 - a Cv. 0)
MacroCov
2
yEy
Methods. We STANDARD with the PAS score function from Section 2. and INTERP-Q from Sec- tion 2.2 with weights 7 E {0. 0.25, 0.5, 0.75, 0.9, 0.95, 0.975, 0.99. 0.999, }} on the CLASSWISE thresholds. These weights are chosen to effectively trace out the trade-off between set size and class-condition coverage in our experiments. We com- pare against the following baseline conformal prediction methods: STANDARD, as described (4); CLASSWISE, as described in (6); and CLUSTERED (Ding et al., 2023), method that targets class- conditional coverage in the many-classe setting by grouping together classes with similar score distributions and computing single score threshold for each cluster. Additional baselines with weaker performance, such as RC3P from Shi et al. (2024), are deferred to Appendix D.
MarginalCov
I(Y EC(X)))
N
Average aet size
3.1 EVALUATING THE SIZE-COVERAGE TRADE-OFF
Figure visualizes the trade-off between set size and various notions of coverage calass-condiiioal, macro- and marginal) achieved by each method. We describe some high-level takeaways:
(i) When targeting set size and class-conditional or macro-coverage, it is more effective to optimize for this trade-off directly than trading off set size and marginal coverage. Adjusting a in STANDARD CP is plausible way to target consiconniiional coverage but the results show that this does not optimally navigate the set aize/class-conditiona coverage trade-off, which is our goal. In compar- ison, our methods, which explicitly optimize for class-conditional macro-coverage, consistently achieve better trade-offs than STANDARD CP.
(ii) CLASSWISE should generally be avoided, comparable class-conditional and macro-coverage can be achieved with significantly smaller sets using our proposed methods.
(iii) INTERP-Q produces reasonable set sizes even for large values of Linearly interpolating between the STANDARD quantile and CLASSWISE quantile does not linearly interpolate the average sizes of the two methods: at
INTERP-Q coincides with CLASSWISE and consequently
has very large average set size (780 for Pl Pl@ntNet-300M and 7430 for iNaturalist-201 a = 0.1), but decreasing 7 only slightly to T = 0.99 results in much more reasonable average set sizes of 7.6 on @I@ ntNet-300K and 55.8 on N This nonlinear relationship is likely due to the fact that the 8softmax distribution of rare classes is highly skewed towards one because the classifier consistently assigns them predicted probabilities near zero.
(iv) STANDARD with PAS is Pareto optimal, in the sense that at any marginal coverage level, there is no method that simultaneously achieves better set size and class-conditional/mmeso suggests that STANDARD with PAS is a good starting place for practitioners due to its simplicity and strong performance all metries. However, INTERP-Q Q is also of practical value since its tunable
coverage. This
7

---

Published as a conference paper at ICLR 2026
1,0000
Average:
10
0.00
0.25
0.50
0.0
0.2
0.4
0.50
0.75
1,00
0.8
0.9 MarginalCov
1.0
50%%5eloo::0%%
UnderCovCap
MarroCov
INatmraaist-2018
Avergee sime
10,000
1,000
10
0.0
0.1
0.0
0.1
0.8
1.0
0.8
0.9
1.0
FracBelow50%%
UnderCovGap
MacroCov
Margins
Stundeed
0.21
Sandlnd
jo
(a- 0.00)
Clocraion
[r 0.2)
Cheosiie
(6 01)
Chansrie
(o 0.00)
Clanaree
(o 0.1)
Clasend
[ir 0.21
Caatored
Chalaxed
0.06)
Chasaed
(a 0.0))
Steadaed
Sandlad
Stardand
PA8
Standand
Q
Qneep-Q
kamp-Q
(o-
Trrep-Q
Figure 3: Average set size vs. FracBelow550% UnderCovGap, MacroCov, and MarginalCov for various methods on the two datasets. For INTERP-Q. lines are used to trace out trade-off curve achieved by rumning the method with different values for fixed a. For FracBelow:00% and UnderCovGap, it better to be closer to the bottom left. For MacroCov, the bottom right better. For MarginalCov, we want to be to the right of the dotted line at for the which the method run.
parameter allows practitioners to choose where they want to be on the trade-off curve between set size and class-conditional coverage while maintaining marginal coverage. Note that the methods can also be combined, as presented in Appendix D.2, Figure 9.
Pl@ntNet-300K case study. Suppose we want sets with 90% marginal coverage on ntet- 300K. STANDARD has small average set size of 1.57 but 421 of the 1081 plant species have coverage below 50%. Conversely, the CLASSWISE sets have zero classes with coverage below 50%, but average produces sets that have an average size only slightly bigger than STANDARD (2.57) but more than halves the number of classes with coverage below 50% 180. INTERP-Q behaves similarly, with the added bonus that the trade-off between set size and class-conditional coverage can be tuned by adjusting We provide table of the metric values plotted in Figure in Appendix D.1 and extended PI@K case study in Appendix D.4.
size of 780. Our methods provide middle ground: STANDARD with PAS
3.2 TARGETING ENDANGERED SPECIES
Motivated by plant conservation, we use the weighted prevalence-adjusted softmax (WPAS) score to target coverage of at-risk species in Pl@ntNet-3000 We will weight the coverage of at-risk species À 2 times more than the coverage of other species, so
Let Va-ris C be the set of at-risk species.
if E Varrisk
wv))
W
otherwise,
where W aI |) Parrisk is normalizing factor to ensure Evey ((y) =
The results are shown in Figure 4. We observe that STANDARD with WPAS improves the class- conditional coverage of at-risk classes relative to STANDARD with softmax or PAS. Comparing that increasing X, the amount
WPAS PAS, improvements in the class-conditional coverage of at-risk classes, as expected. These increases are "paid for" in terms of mild increase in average set size and have no discernible effect on the class- conditional coverage of not-at-risk classes, which is appealing from a practical perspective.
upweight at-risk classes, leads larger
We consider species with an IUCN status of "endangered", "vulnerable". "near threatened", "critically endangered", or "lower risk" at risk. Of the 1,081 total species in Pl@mtNet-3006 33 species qualify as at-risk.
8

---

Published as a conference paper at ICLR 2026
PlömtNet-300K
Avrnge axt size
20-
15
x
*
10-
0.22
0.50
0.75
1.00
0.44
03.6
00.8
0.4
0.6
00.8
0.8
0.9
1.0
At-risk meerage Gi
Not-at-risk average 6x
MucroCor
MarginalCov
Standard
[o 0.20)
Standind
[o - 0
x
Standied
[o 0.00]
Stazdted
[a 0,31]
Standand wr. PAS
(o 0.20)
Standand w. PAS
(o 0.10)
Standant w. PAS
[o 0.06]
Standhnt w. to 0.31]
Standaad WPAS
(o 0.00)
Stardinnd w. WPAS
[o
Stardant w. WPAS
to 0.00)
Stardant w. WPAS to 0.31]
Figure 4: Results for running STANDARD on Pl@ntNct-300K with different conformal score func- tions: softmax, PAS, and WPAS with À {1, 10, 102, 10 Increasing À in WPAS improves the class-conditional coverage of at-risk classes, which is measured using Cy. the empirical class- conditional coverage of class y. "At-risk average by" is computed as (1/[Darrisk) Eyelin by and "not-at-risk average 6y" is computed analogously. Note that here the y-axis is on linear scale.
3.3 SIMULATED HIMMAN DECISION-MAKER
We now examine how coverage and set size can jointly influence human decision accuracy. Human interpretations of prediction sets can vary (Zhang et al., 2024; Hullman et al., 2025), and we focus on two models of human decision-making that impacted by coverage and set size: an expert verifier Hexpert and a random guesser H Hrandom (equivalent to the uncertainty suppressing decision maker in Hullman et al., 2025). Let (CX)), Y) E y be a human's 's chosen label given prediction set C(X) when the true label is Y. The probability (((() label after sceing prediction set C(X) L(Y C(X)) H Hexpert and 1(Y e C(X))/ if H = Hrandom- Hexpert is only affected by coverage and not set size, whereas random guessers are highly sensitive set size, as they choose label uniformly at random from the prediction set. We also consider mixture decision- makers. Hmixtures who act as Hexpert with probability Texp. and Hyandom with 1 Texp. for Texp. E [0. 1]. We are interested in how prediction sets affect the probability that human correctly labels an instance of given class, which we formalize as the class-conditonall decision accuracy for class y under procedure C, defined as ((H(C(X),
= Y) that the human chooses the correct
Y Y y). Note that, due to the definition of Hexpert, the class-conditional decision accuracy of Hexpert is equivalent to the class- conditional coverage.
Figure shows the class-conditional decision accuracies for various types of decision makers under STANDARD with PAS as compared to baseline methods (the plots for INTERP-Q are similar; see Appendix D.6). We report results on the truncated versions of P1@ntNet-300K and iNaturalist-2018 that only include classes with enough test examples to reliably estimate class-conditional decision accuracy (see Appendix C.I for more details). We observe that STANDARD and STANDARD with PAS achieve strong performance regardless of Texp.; whereas CLASSWISE only does well when Texp. is high. However, compared to STANDARD, STANDARD with PAS has an additional benefit that its performance is more balanced across classes. The worst decision accuracies for STANDARD with PAS are higher than those of STANDARD, which is paid for by only a slight decrease the best decision accuracies.
DISCUSSION
In this paper, we consider the problem of producing good prediction sets in long-tailes settings, which heretofore has been understudied problem despite its relevance in many applications, such as species or disease classification. We develop two approaches to better navigate the trade-off be- tween set size and class-conditional coverage in long-tailed settings. First, we introduce prevalence- adjusted softmax (PAS) and its weighted variant (WPAS), score functions inspired by oracle solu- tions that optimally balance expected set size and (weighted) macro-coverage. Second, we propose INTERP-QQ Q. a simple procedure that interpolates between STANDARD and CLASSWISE CP to pro- vide tunable compromise between small sets and strong class-conditional coverage.
Limitations and future work. To use INTERP-Q, we select a desired average set size and choose the parameter value that achieves this size on the calibration set, this will produce reasonable results (despite the exchangeability violation). Another
must choose a value of In practice, if

---

effort to search Published as a conference paper at ICLR 2026
Plkint trrmett--t
Decision accuraey
Nepp 0%
7ep
25%
Teep
50%
Nesp. 75%
Teeps
100%
200
Class
Class
Class
Cluss
Cluss
iN Iasturalist-aa Sesturalist-21--tuuaar 201mmmtmmatdd
Dociiton Mccuracy
Tesp. 0%
Tex..
25%
Top.
50%
Tesp. 75%
Teaps
100%
300
500
500
Class
Cluss
Cluss
Clussi
Cliesi
Cherrvii
Standand
Standard wt PAB
Figure 5: Class-conditional decision accuracies for range of decision makers when presented with 0.1 Classes ordered by
sets from STANDARD, CLASSWISE STANDARD with PAS decreasing decision accuracy of Hexpert under each method. Note that the decision accuracy when Texp. 100% equivalent the class-conditional coverage.
promising future research direction that we touched upon is the interaction between set sizes and coverage in determining the utility of prediction sets. In particular, important aspect of decision- making not addressed by the decision accuracy discussed in Section 3.3 is the concept of "effort": if human identify the correct label within set, larger sets require through.
Broader impacts. The methods we introduce have the potential to benefit society in the mid- to long-term by improving uncertainty quantification citizen science platforms like Pl@ntNet. Our methods increase probability that prediction sets for rare or endangered species contain true label while keeping the average set size under control. This has important implications for biodiversity monitoring and has the potential to produce virtuous cycle: As more images of rare species identified. they be used retrain the classifier to better identify such species, which in turn improves the probability that citizen scientists will correctly classify future images these species.
ACKNOWLEDGMENTS
We thank Pierre Bonnet, Alexis Joly, Anastasios Angelopolous, and Ryan Tibshirani for insightful conversations and Margaux Zaffran for making this project possible. This work was funded by the French National Research Agency (ANR) through the grant Chaire IA CaMeLOt (ANR-20-CHIA- 0001-01). TD additionally acknowledges support from the National Science Foundation Graduate Research Fellowship Program under grant no. 2146752.
REFERENCES
Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. Foundations and Trends in Machine Learning, 16(4): 494-591, 2023.
Anastasios Angelopoulos, Stephen Bates, Jitendra Malik, and Michael Jordan. Uncertainty sets for image classifiers using conformal prediction. In ICLR, 2021.
Rina Foygel Barber and Ryan J Tibshirani. Unifying different theories of conformal prediction. arXiv preprint arXiv:2504.02290 2025.
Rina Foygel Barber, Emmanuel Candès, Aaditya Ramdas, and Ryan J Tibshirami. Conformal prediction beyond exchangeability. Ann. S1atis,,5 51(2):816-44 2023.
George Casella and Roger Berger. Statistical Inference. Duxbury Resource Center, June 2001. ISBN 0534243126.
10

---

Published as a conference paper at ICLR 2026
Maxime Cauchois, Suyash Gupta, and John C Duchi. Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction. Mach. Learn. Res., 22(81):1-44 2021.
Christophe Denis and Mohamed Hebiri. Confidence sets with expected sizes for multiclass classifi- cation. J. Mach. Learn. Res., 18( 18(102):128, 2017.
Tiffany Ding, Anastasios Angelopoulos, Stephen Bates, Michael Jordan, and Ryan Tibshirani. Class-conditional conformal prediction with many classes. In NeurlPS, volume 36, pp. 64555- 64576, 2023.
Camille Garein, Alexis Joly, Pierre Bonnet, Jean- Christophe Lombardo, Antoine Affouard, Mathias Chouet, Maximilien Servajean, Titouan Lorieul, and Joseph Salmon. Pl@ntNet-300K: a plant image dataset with high label ambiguity and long-tailed distribution. In NeurlPS Datasets and Benchmarks 2021, 2021.
Camille Garein, Maximilien Servajean, Alexis Joly, and Joseph Salmon. Stochastic smoothing of the top-k calibrated hinge loss for deep imbalanced classification. In JCML, volume 162, pp- 7208-7222, 2022.
Charles Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg. Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array program- ming with NumPy, September 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pp. 770-778, 2016.
Yotam Hechtlinger, Barnabás Póczos, and Larry Wasserman. Cautious deep learning. arXiv preprint 1XXi:18805009460 2018.
Jianguo Huang, Huajun Xi, Linjun Zhang, Huaxiu Yao, Yue Qiu, and Hongxin Wei. Conformal prediction for deep classifier via label ranking. arXiv preprint arXiv:2310 0606630, 2023.
Jessica Hullman, Yifan Wu, Dawei Xie, Ziyang Guo, and Andrew Gelman. Conformal prediction and human decision making. arXiv preprint arXiv:2503. 2025.
Rob J Hyndman and Yanan Fan. Sample quantiles in statistical packages. The American Statistician, 50(4):361-365, 1996.
Alexis Joly, Hervé Goëau, Pierre Bonnet, Vera Bakić, Julien Barbe, Souheil Selmi, Itheri Yahiaoui, Jennifer Carré, Elise Mouysset, FrannFianoois Molino, Nozha Boujemaa, and Daniel Barthélémy. Interactive plant identification based social image data. Ecological Informatics, 23:22-34, 2014.
Shayan Kiyani, George Pappas, and Hamed Hassani. Length optimization conformal prediction. NewrIPS, 77:99519-99561 2024.
Maksim Lapin, Matthias Hein, and Bernt Schiele. Top-k multiclass SVM. In NeurlPS, pp. 325-333, 2015.
Jing Lei and Larry Wasserman. Distribution- free prediction bands for non-parametric regression. J. R. Stat. Soc. Ser. B Stat. Methodol., 76(1):71-96 2014.
Jing Lei, James Robins, and Larry Wasserman. Distribution-free prediction sets. J. Amer. Statist. Assoc., 108(501):288-27,. 2013.
David D. Lewis. Evaluating text categorization I. In Speech and Natural Language, Proceedings of a Workshop held at Pacific Grove, California, USA, February 19-22. 1991. Morgan Kaufmann, 1991.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pp- 2980-2988, 2017.
11

---

Published as a conference paper at ICLR 2026
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Tuve Löfström, Henrik Boström, Henrik Linusson, and Ulf Johansson. Bias reduction through conditional conformal prediction. Intelligent Data Analysis, 1906:1355-1375,200 2015.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In ICLR, 2021.
Jerzy Neyman and Egon Sharpe Pearson. IX. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London, Series A., 231(694-706): 289-337, 1933.
Aleksandr Podkopaev and Aaditya Ramdas. Distribution-free uncertainty quantification for classi- fication under label shift. In UAI, pp. $44-853. PMLR, 2021.
Yaniv Romano, Matteo Sesia, and Emmanuel Candès. coverage. In NeurlPS, 2020.
Classification with valid and adaptive
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeey Satheesh, Scan Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vision, 1153))211-252, 2015.
Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least ambiguous set-valued classifiers with bounded error levels. J. Amer. Statist. Assoc., 1146525):223-234, 2019.
Jun Shao. Mathematical statistics. Springer Science & Business Media, 2008
Fan Shi, Cheng Soon Ong, and Christopher Leckie. Applications of class-conditional conformal predictor in multi-class classification. In ICML, 2013.
Yuanjie Shi, Subhankar Ghosh, Taha Belkhouja, Jana Doppa, and Yan Yan. Conformal prediction for class-wise coverage via augmented label rank calibration. NeurlPS, 37:13333-311111 2024.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot Ross Anderson, and Yarin Gal.
Al models collapse when trained on recursively generated data. Nature, 631(8022)755-755,
2024.
Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candès, and Aaditya Ramdas. Conformal pre- diction under covariate shift. In NeurIPS, volume 32, 2019.
Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In CVPR, pp. 595-604, 2015.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The iNaturalist species classification and detection dataset. In CVPR, pp. 9769-8778. 2018.
Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, 2005.
Vladimir Vovk, Valentina Fedorova, Ili Nouretdinov, and Alexander Gammerman. Criteria of efficiency for conformal prediction. In Conformal and Probabilistic Prediction with Applications: 5th International Symposium. pp. 23-39. Springer, 2016.
Jingdong Wang, Zhuowen Tu, Jianlong Fu, Nicu Sebe, and Serge Belongie. Guest editorial: Intro- duction to the special section on fine-grained visual categorization. IEEE Trans. Pattern Anal. Mach. Intell., 44(02):560-562. 2022.
Dongping Zhang, Angelos Chatzimparmpas, Negar Kamali, and Jessica Hullman. Evaluating the utility of conformal prediction sets for Al-advised image labeling. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pp. 1-19, 2024.
12

---

Published as a conference paper at ICLR 2026
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long- bong-tailed learning: A survey. IEEE Trans. Pattern Anal. Mach, Intell., 45(9):10795- 10816, 2023.
13

---

Published as a conference paper at ICLR 2026
A FUZZY CLASSWISE CONFORMAL PREDICTION
The INTERP-Q method introduced Section 2.2 shrinks dy in a naive way that disregards rela- tionships between classes; in this section, propose more sophisticated approach called fuzzy classwise CP (FUZZY for short) that assigns weights to the score of each calibration example based on the "similarity" between their class and class and then takes the quantile of this weighted dis- tribution. Notably, this approach comes with
a marginal coverage guarantee (rather than the 1 2a guarantee of INTERP-Q), Q). Empirically, we find that it does not do better than the simpler INTERP-Q Q procedure, but we still believe it is useful to present the method, as it may be preferable in settings where marginal coverage guarantee is desired. Proofs are deferred to Appendix B.
This method relates to work on weighted conformal prediction, which generalizes standard con- formal prediction by computing a weighted quantile of calibration scores (Tibshirani et al., 2019; Barber et al., 2023; Barber & Tibshirani, 2025). Methodologically closest to our work is Podkopaev & Ramdas (2021) I), which uses label-dependent weighting to ensure marginal coverage under label shift.
A.1 PRELIMINARIES
Recall the STANDARD CP and CLASSWISE CP procedures introduced in Section 1.2. We now consider a generalization of these two methods. Let w y x R2o be a weighting function where w(y', y) is the weight assigned to point with label y' when computing the weighted quan- tile threshold for class y. The wa-abel--nn weibbe--eighted conformal prediction set (see, e.g., Podkopaev & Ramdas, 2021) is CLw (X; w)) C(X; qw)) where the y-th entry of qu is
w/ (Yi,)) Wy
w(y. v) W
b Quantile
0x(X.)9)+
800
(17)
and Wy = E (Y, v) ww y) is normalizing factor to ensure the weights sum to one.
STANDARD corresponds label- conformal with equal weights for all classes regardless of y - that is, w(y' y) x 1. CLASSWISE corresponds to label-weighted conformal where nonzero weights are assigned only to other calibration points of class y. i.e. wyyy,3) yy) o {y/ y
A.2 THE FUZZY METHOD
We now present the fuzzy classwise conformal prediction (FUZZY) procedure. We begin by present- ing a simpler method, Raw FUZZY, that FUzZY builds upon.
Raw FUzZY. The large sets of CLASSWISE result from not hav- ing enough data from each class when computing the score quan- tiles. A natural solution to this problem is to borrow data from other classes, where more data is borrowed from classes that are more similar.
CLASSWISE
FUZZY STANDARD
We define the Raw FUZZY prediction set to be the habel-weighter conformal prediction set obtained using WPuzzy y x y 220 as the weighting function in (17), i.e., CRAWFUZZY (X) CLw (X; (Fuzzy).
II(v)
Figure 6: The weighting
function w of each CP method, when viewed as weightwe conformal prediction.
To construct this weighting function, we use class mapping II y + C Rd for some small functions he A A + R20 parameterized by bandwidth o 0.
(e.g.. A = (0, ()) and kernel
A good II should map classes with similar score distributions close together. The kernel hg takes in two mapped classes and outputs non-negative scalar that is decreasing with the distance between the two inputs, and it is allowed to depend on y. For example, the kernel bandwidth could be rescaled to decrease with the number of examples have from class y so that classes with more examples "borrow less" from other classes.
We then define the weighting function as WFuzzy (n' (y'.u) = I()))
(18)
Figure 6 visualizes how Raw FUZZY can be viewed as interpolating between STANDARD and CLASSWISE CP by setting weights that are in between the two extremes.
14

---

Published as a conference paper at ICLR 2026
The following proposition tells us that for most reasonable kernels, such as the Gaussian kernel h" hu(u, v) x exp (-(v (- /)²/(20")) Raw FUZZY can recover both STANDARD and CLASSWISE by setting the bandwidth appropriately.
Proposition 4 (Informal), Assume IT maps each class to a unique point. If. for all u, v E A with F v, the kernel h satisfies ho(u, + 0 as - O and he (u, ho(u, u) as a - oc, then for sufficiently small a we have CRAWFUZZY (X) = CCLASSWISE (X) and sufficiently large a we have CRAWFUZZY (X) CSTANDARD(X).
FUZZY= = Raw FUZZY + reconformalization for marginal coverage. To obtain the FUZZY pro- cedure, we add a reconformalization wrapper around Raw FUZZY to ensure marginal coverage. In order to do this reconformalization, we use holdout dataset. In practice, this holdout dataset can be created by setting aside a relatively small part of the calibration dataset (here, 5000 examples) since we only use it to estimate single parameter. The intuition behind our procedure is as follows. When reconformalizing, we hope to equally affect all class-conditional coverages. One way to do this is find the á 2 0 such that running Raw FUZZY at level 1 â achieves the desired 1 - coverage. This can be formulated as an example of STANDARD CP with a special score function 8(1, v) = Fg (5(x,,)) v)), where
WFuzzy UYi v)
(t) = Σ w (ss <t} and w
(19)
Di=1 WFuzzy (Y,, y) + WFuzzr (y.
Then, we have the equivalence y E CRAWFUZZY (x) interpreted as (one minus) weighted conformal p-value (Vovk et al., 2005; Barber & Tibshirani, 2025), which we re-calibrate. To achieve the desired marginal coverage, the method FUZZY does STANDARD CP with the new score function and held-out data calibration set.
5(x,8) y) < 11-a. a. The score 3 can be
Proposition 5. Let Deal = {X,,, Y)) be a calibration set and Dhold = {(XH, ) be held-out examples. Define â be the STANDARD CP threshold obtained by applying 8 on the holdout set:
1 1 Quantile,
) + mn m+1 5o0)
1
(20)
Then, if the scores evaluated on the held-out dataset 5(X Y²H) are exchangeable with the score of the test point, the set Cruzzr (x) = {y 5(2,8) S ā} will achieve
marginal coverage.
The score function 8 depends on the calibration set Deat. but the assumption of exchangeability is easily satisfied, c.g., if the points of the calibration set and the held-out set and the test point are exchangeable. A careful reader may also notice that Cruzzy is not exactly equivalent to apply Raw FuzZY at level as strict inequality has been replaced by a non-strict one to get the coverage guarantee. As a result, we implement FUZZY as Raw FUZZY at level â c for small perturbation 8 > 0 (specifically, < ,,
Note that, if desired, we could use full conformal techniques (Vovk et al., 2005) instead of data splitting (i.e., let Deat = Dnota), but this incurs higher computational costs that are undesirable for practical applications (see Proposition 8 for details).
Choosing a mapping II. To instantiate FUZZY, we must define a mapping II from y to a space in which we can compute distances between classes. What makes good mapping? Intuitively, we want classes with similar score distributions to be mapped close together. This is similar to the motivation behind the clustering procedure from Ding et al. (2023). However, in long-tailed settings, many classes do not have sufficient calibration examples for us to estimate their score distributions. A "zero-shot" way of attempting to group together classes with similar score distributions is to group together classes based on their prevalence, with the intuition that the underlying classifier is likely to assign small softmax scores to infrequently seen classes and large scores to more frequently seen classes. Specifically, we map each class to its number of t examples ny nirain, normalize, and then add a small amount of random noise to ensure that the uniqueness condition of Proposition is satisfied with probability one: [lprevalence (v) cntrain + E, where Ey Unif([-0.01, 0.01]) independently for each for y E V We normalize using c = 1/ (maxx' e n traain) to ensure that Ipprvaleece (1) E [0, 1] so that the same bandwidth o has similar effects on different datasets. This is just one possible choice for II and is what we use in our main experiments; other options are explored in Appendix D.2.
15

---

Published as a conference paper at ICLR 2026
B THEORETICAL GUARANTEES
B.1 OPTIMAL PREDICTION SETS FOR (WHIGHTED) MACRO-COVERAGE
In this section, we state and prove more formal version of Proposition 2, which Proposition is special case with uniform weights. Proposition 6 (Formal version of Proposition 2). Let w y + [0, 1] be non-negative weighting function summing one. For R, define
wly) p(ylx)
- :: 2) where 3(0,y) v)
(21)
p(u)
and plyla) denotes the conditional density of given = x and ply) is the marginal density of Y.
(a) Let a E [0, 1]. If there exists la such that MacroCov, (Ct.)) = 1-a, then Cra is the optimal solution to
min [[C(()]] subject to C:
Y 9)
(22)
NEY
(b) Let n. 2 O. If there exists tx such that EE[[ n, then C₁ is the optimal solution to
Σ [w) E C(X) IY = =y) subject to E[[C(X)]] S K. C:X-+2
max
(23)
VEY
Remark 1. If thresholds ta f. achieving exact equality do not exist, the optimal set remains of the form (21) but must be combined with randomization to achieve the optimal solution (that has weighted macro-coverage of exactly 1 or expected set size of exactly k). See, for instance, Shao (2008, Theorem 6.1) for a formal statement of Lemma B. below for this case.
Proof. This result is consequence of the Neyman-Pearson Lemma, which we state below using the formulation of Sadinle al. (2019) (see also Casella & Berger, 2001, Theorem 8.3. We provide proof of this lemma in Appendix B.2.
Lemma B.1 (Neyman & Pearson, 1933). Let μ be measure on X x and let f.g : XXX + Rzo be two non non-negative functions. For v 2 O, consider the problem
min C.X JXXY
Lly E C(x)))((a, v)
(24)
subject to
Hy (o Jxxy
If there exists by such that
fr t. (0) =
(25)
then
-xxx=(yE)) ((x,1) 2 lu g(a,u))
(26)
the optimal solution to (24); Any other minimizer C of (24) is equal to C μt and μ almost E C(x))) = where A
everywhere, i.e., for E {..g). ps ({(x,y) y E ((xx)) denotes the symmetric distance between the sets and μn is defined as (n(A) = SA hdp.
Remark 2. An analogous statement of Lemma B.1 holds where we replace the ">" with ">" in (25) and (26).
PROOF OF PROPOSITION 6(a). We assume that has density p with respect the Lebesgue mea- sure X, but the proof can be adapted for any measure. Define the functions f(x, y) == w(v)p(=ly)
and g( y) p(x) and the measure À x (Enevd,) 5u)) which the product measure between
16

---

Published as a conference paper at ICLR 2026
the Lebesgue measure (denoted by A) and the counting measure on y. Let plugging these values f. 9. p. and y into (24) yields (22), because
Observe that
SM 1y E (z) v)
/ 1 E (())))(()))
Σ w(y)
yEy
Σ VEy
w()) C(X) ( =)
and
J 1/ E xxy
1000E11 1 (()))p(x)dr
NEY x
J Σ 1lv (()))(dd))
x wEY
/ (c())p(x)dd
=E(I(())
By Lemma B. follows that the optimal solution (22) given by (26), which for our choice of and can be rewritten as Ci-a (x) ty EV: -(y)puulz)
2 1-aa by observing that
ply)
(xx v) w(y)p(xrl) w())(a,,) w(y)p(ula) (z, y) p(x)
p(z)p(u)
p(v)
PROOF OF PROPOSITION 6(b). Let us now derive the optimal solution to the dual problem. To do so, we must rewrite the problem in a form where Lemma B.1 can be applied. First, observe that
Σ NEY
(y E C(X) = y) = Σ w(y) Pv $ C(X) Y y-y))
yEy
= 1
Σ WW) = yEy
where (=XX) : y C(X) denotes the complement of C(X). Similarly, observe that expected set size can be written in terms of the complement as E[(C(())) |>l| (X)I). Thus, we can obtain the optimal solution to (23) by taking the complement of the optimal solution to
min Σ ENU)) (y E E(X) Y =v) subject to B(((())) > D - (27)
C:N'+2
DEY
Applying Lemma B.I, combined with Remark 2, with f(x,y) p(z), g(z, y) w(y)p(r(g) v)) the same measure as in the proof of Proposition 6(a), and v = [)]] tells us that if there exists f, such that
f JAXY (ff(x,)) > Ehe g(x (ol. = | - K
(28)
then the optimal solution (27) is
c(l=){(e> v y: f(,yy) g(x,y)
p(v) & - --YEE yey:
w(y)p(yla) p(y)
Ere fn y
w(y)p(y|x)
Thus, the optimal solution to our original problem (23) is
{ yy w())(()lz) tx p(v)
(= = (x))(x) =
17

---

Published as a conference paper at ICLR 2026
B.2 PROOF OF LEMMA B.1
For completeness we give a proof of the Neyman-Pearson Lemma, as stated in Lemma B.I.
Proof. We will show that C* (x) = {y f f(x, v) 2 (z. y)} is the optimal solution to (24). We first demonstrate that it is optimal solution and then that it is unique.
Optimality. By definition of tu we have
Sen (x))(()) u)dp(z v) =
(29)
which is trivially greater than or equal to v, so (z) is indeed feasible solution. To show that (xx is optimal, we must argue that it achieves smaller objective value than any other feasible solution. Let C X + + 2d be any other set- getgeenrrating procedure that satisfies the constraint in (24). We want show that
[
v) S 1v ((x))a((x v)
1y (*
We prove this by showing their difference is negative:
f E ("))(c))))a) - E ((o))ol
IxxY
- -J (yECC))((W))ttt
1 (W
-xxy
JxXY
$ 1 I C
1
to xxx H (
Jxxy
1 U 1 (VECC xxx
1 (()))
- (v- 50.
y). The second inequality comes from applying the equality stated (29) to the first integral and then using the definition of C as satisfying the constraint of (24) to lower bound the second integral.
The first inequality follows from E (xx = g(z,8) S fz.
Uniqueness. Let C: : X + 2 another optimal set-generating procedure, so achieves the same objective value as C*
/ xxy
1y E (()))((() = Sr xxx
16 E (=))))(2,
(30)
and it is a feasible solution,
fr 10 E (c,)))( vv.
Let us first note the following non-negativity relationship:
(=)) 1yy e C(x))) (f(z, y) tzg(z,))) 20, E(x,) Integrating (31) over V, then applying (30), we get
(31)
fr 1fy C* (x))
S 1(v E (()) v) 2 0.
Combining this with the definition of C. and the feasibility of C, we must have Sxxy 1sy e C(z))f(x, y)dp(æ, y) Thus, (31) integrates to zero. Since we also know that (31) is non- negative, must be true that, y-almost everywhere,
(v C* (x)) 11 E ((x))) (f(a.v) tog(a, v))
18

---

Published as a conference paper at ICLR 2026
For (z, y) X' x D such that f(., v) * tzg(z.)) y), then 1y C²(x)] 1s C(x)). which
implies, using the definition of C* (x) = fy f(z,y) 2 tzg(z, y)}. that p-almost everywhere, ((x) S (z) and (x) S ((x) U f(z,y) tog(a,)) y)}. It remains show that the sets are equal almost everywhere by showing that the set
= { (a, v) y & ((x) and y E (²)) {(,, : $ C(x) and f(x, v) - (g((2,)) 8)}
of measure under ptg. By definition of C²
V S xxx
1ly E C (2))
Sea 1y E (ls))
Sexy 1((,,)) (dd(a))
= S. fdu.
Thus must have (D( = 0. Since tug f on D, we also get that Pg 0.
The version of the Neyman- Pearsso Lemma described in Remark can be proven in the same way.
B.3 PROOF OF PROPOSITION 3
We present Proposition 3 in two parts: the lower bound on coverage and the extreme case achieving coverage close to this lower bound. Proposition 3 (restated, first part). Using calibration data {(x,, Y;) let q be the STANDARD quantile threshold (4) computed for [0, and, for y Y, let q be the CLASSWISE quantile threshold (6) computed for the same If the test point (Xn+1, Yn+1) is exchangeable with the calibration data {(X;, Y.)) then the INTERP-Q P prediction sets satisfy
P(Yn+1 E CINTERR-Q (Xxnx)))((+1)))))))))))) 2 20.
(32)
Observe that for the test score Sn-1 s(xn+1, Vnt1), we have the following inclusion of events:
Proof.
{sut1 S STQEW + (1 (1-r)/g 3 {Sati S (ai }
Thus,
IP (Su+1 S Too + (1 (1-r)
2 P SS S min 9)
CW
1 - P Sntt >
Pra1 or Sut1 > 9)
CW
2 1 - P Sn+1
(Y PP (S > g)
using union bound for the last inequality. By the coverage guarantees for STANDARD and CLASSWISE, which follow from classical conformal prediction arguments using exchangeability
(see, e.g., Vovk al., 2005), we have IP (Sm+1 (Y S and IP (S ②) a. which concludes the proof.
The following result provides an example of a score distribution for which a coverage of (1 - a attained. For simplicity, the example is constructed as if the distribution and the quantile are exactly known, i.c., in the case of an infinite calibration set (n = oo).
Proposition 3 (restated, second part). There exists joint distribution IP of tuple (score, label) over [o, 1] x y with D 2 2, such that for any E (0, 1). we have
PP (S < T4Y
CW + (1 - 7)4) = (1 aa)
(33)
where q = (P) is the (exact) (1 a:) -qanttiee of S and for y E v. a CW = 9 CW (W(P)) is the (exact) (1 quaaniie of S|Y = y.
19

---

Published as a conference paper at ICLR 2026
Proof. Let us define explicitly the distributions IP for which the upper bound is achieved. The distribution of (S,Y) ~ is defined by:
Y ~ (1 a00 + add
S vo) (1 a)do add1/2. y1) 61,
for yo F y1 Then 0 1/2, 4y0 0 and By)
1. It follows that, for E (0,
P (S S Tqp CW + (1 7)9)
P ($ - 7y0
CW (1 r)glY y0) (1 a)
+ P (S Σ T4y: + (1 (1-T)9)) -///////-- = y1)
CW
= P (S < (1 (1-r) = yo) (1 - a)
+ (8 < T (1-r)/2|Y 11)a.
By plugging in the exact distributions of S = yo and S|Y y/, we get
PP (S
CW + (1 - r) = P (S = 0|Y =0o) (1 a) +0 (1 aa) S T4Y
where we have used that (1 - T)/ < 1/2 and r + (1 /2 <
B.4 RAw FUZZY RECOVERS STANDARD AND CLASSWISE CP
We first more formally state the result of Proposition 4. Proposition 7. Let E [0, 1] and assume a satisfies ma & N for all m E [n + 17. Also assume
II: A maps each class to a unique point.
If. for all u, v A with u # v, the kernel h satisfies
v) as a + and
no(u, - ho (u, u) as a + oo,
then, for sufficiently small a, for any x E X and calibration set, we have
CRAWFUZZY (1) = CCLASSWISE(()) (T).
and, for sufficiently large a. for any x E X and calibration set, we have
CRAWFUZZY (xx) CSTANDARD(I)..
Proof. Let us first recall that CRAWFUZeY (X) CLW (X; WFUZZY). In the rest of the proof, we will
write wa to refer to the weighting function WFuzzy with bandwidth > 0. As described in (17), the weighting function w, in the second argument of C-w(-) is used obtain the class-specific thresholds Q=" and the weightee conformal prediction set is constructed as (y E ) s(x, y) <
For all E V/ we will show that as a - O, we eventually have q.- q. and as - oc, we eventually have 4 aCW where q and ,cW the STANDARD and CLASSWISE quantiles from (4) and (6).
" }.
To show that scalar x is equal to the a quantile of distribution Q, we will show
Step 1: P P (X < x) a.
SStep 2: For any < æ, we have Px~Q (X t) <
In the remainder of this proof, will use two ways of indexing the calibration scores. First, for
i E [n], we let S; := s(X,, Yt). Second, we index the calibration scores by class: for each class
y Vi we use S for i E [My] denote the calibration scores for class y. Before beginning, we
also note that the assumption that ma & IN for all m E [n + 1] ensures [[,, + 1) a)l >
(ny + 1)(1 a) [(n. 1) a)] for all y E y. This will be used below.
'If the desired miscoverage level a does not satisfy this assumption, note that there exists an à that is infinitesimally close to a that does satisfy the assumption. Stated formally, for any a E [0, 1] and any >> 0, there exists à such that lá al S and må g N for all [n + 1). This is due to the density of inrationals in real numbers and the fact that any irrational á satisfies the assumption.
20

---

Published as a conference paper at ICLR 2026
CONVERGENCE TO CCLASSWISE FOR SMALL We will show that for sufficiently small o, have 4y Bu for all y E V/ where
CW
E Wa (Y,, v),, + wo (v. (y.28)600 Ei=1 wo (Yr, 4) wo (v.
Ruzz Quantile By
Q
In other words, 4y the 1 a quantile of Qy. We now apply the two-step procedure outlined above to show that on is equal to the - a quantile of Qy when a is sufficiently small. Step I. We begin by observing
Eis wo (Y,,)) S g
IP Ps-Qu (S (
EL Wo (1.9) v) + Wir (y: y)
We (y) v) Ee 1 ( S 1y (WW) } + E We (2(2, E 1(5 S dy CW)
(ny + W (y: y) + > nzw, (2,y)
Wa (v, [1 (n + 1)1
(34)
(ny + 1 y) > n. (z,y)*
where we have used for the last inequality the definition of the CLASSWISE quantile and have lower bounded the second term of the numerator by O. By assumption, we know that wo (z, y) + 0 as + 0 for all classes = + y. Thus, for a small enough, have
Wo (V..33) 3) 1 (I(ng + 1)(1 a)l - -(ny +1)(1 - ai)
Σ
m, S
For such a, (34) becomes
Ps (S (5 2 1 a2,
which implies that GCW 2 0g
Step 2. We now show the converse inequality. Given any < gy we want to show Ps-q, (S & t) < - a. Similar to above, we begin by writing
Ps-Q, tt wo v) En 11S S {t + Exer(ii wo v) Ei 11s St}
(ny + 1)w, (v, 3) + Exer(v) no (2,3)
wo v) (ny 1)(1 a)] + E no (2,4)
(35)
(ny + (y.
The inequality is obtained by (i) removing a positive term from the denominator and (ii) observing that t < g.W then at most (n, + 1)(-)))) scores of class are smaller equal to Otherwise, would be higher than the class-conditional quantile ic By assumption, we know that wz y) - O as - O for all classes = + y. Thus, for a small enough, we have
Σ (w)
ug (2, < wo (2. (o..))(C ((1 (n + +1 - [(n + 1)(1 - +))(--)))).
For such a, (35) becomes
Ps-Q, (S < a
Thus, CCW & Together, Step I and Step 2 tell us that for small enough, we have q gueeze a which concludes the proof for the convergence of Cruzzy Cc ASSWISE
CONVERGENCE TO CSTANDARD FOR LARGE a. Recall that we assume as a - co, we have wo wo(z, (z.y) + wa (g. y) for all y, D. Then for any = there exists a large enough such that
Wo (2. y)
max B.
S
(36)
Wa (y. v)
21

---

Published as a conference paper at ICLR 2026
Step 1. As in the previous case, write
Di= W/ (Y, 1) - =
Qs~Q, (S ()
2i=1 wo (,,, 8) + wa (v.
Eey wir (z, y) n 1 1 S = 11 + Wo (V,
Eeey n,
W/ (v. (1 E) Σ i= 1 9)
(n(1 + 8) + 1) Wo (v.
[nn + 1) a)]
I+E
The first inequality is obtained by lower bounding wa (2. y) by =) (v, y) in the numerator and upper bounding w y) by (1 + E) (y. y) in the denominator. The last inequality comes from the fact that q is the empirical quantile of the scores. By choosing small enough, specifically,
(n + 1)(1 a)] (n + 1)(1 a) [nn + 1)(1 ai] (n 1)(1 ax)
get that for a large enough, Qs-Q, ( n) 2 1 a. Thus for such Ø, we have á 2
9 yure for
all y E y.
Step 2. We now show the converse inequality. Given any
q. we want show Ps-Q, (S st)
1 - a. We start by observing
wo (2, Eo
Σ (5 S
PsnQ, (8
Evey n2 20 (2, + wo (8, y)
1+ 11s, tt]
1+E (n + 1)(1 - a)]
I 1-8
1-8
n +
+
We have used (36) again for the first inequality. For the last inequality, since q. there are most [nn + 1)(1 a) scores which are smaller or equal t. Otherwise, t would be higher than q. Choosing small enough such that
1+ [(n 1)(++)(111))) a)]
< a
1 1-E n 1+1
yields Pss-q, (S S < 1 which implies that á S g gireer. Combining the results from Step and Step 2, we get that dy guoweer = ÿ for a large enough, so Cruzzy (X) = CSTANDARD
INFINITE QUANTILES CASE. The above proof assumes that the STANDARD and CLASSSWSSE quan- tiles not infinite. In the infinite case where q = Do &eW co, we directly get Step then PS-Q, (S < 4) = 1 2 a for a E (ů. (CW), The rest of the proof remains similar to before.
B.5 RECONFORMALIZZATION OF RAW FUZZY
Proof of Proposition 5. This follows from the exchangcability of the held-out set Dhold and test point (X,+1: 1) We first link the set CRAWFUZZY to the score function š. For x E X, y E y, let wy WPizy (y, w)/ (E WFozeY (Y,, v) WFuzzy (y, w)). then:
E CRAWFUZZY (x) + s(x, v) < Quantilej-a
Σ
w S bo(x)))) + wo/000
wo Y.) < s( 5) < -1-a a + 3(2, v) < a. (37)
By exchangeability, the set (y: $(z < Quantile, -a (m Σ & Sx y.a m=1100) has marginal coverage of 1
22

---

Published as a conference paper at ICLR 2026
We now briefly present possible adaptation of the reconformalization step for FUZZY CP that avoids the need for an additional dataset Dhold- To do so, we adapt the score á from (19) to recali- brate the weighted quantile with the same dataset Deal using full conformal techniques (Vovk al., 2005).
Proposition 8 (Reconformalization with Deal). Let define for (x. v) and finite subset 'D of N' x y the score
Stall ((2, D) = E(Froee Wrezzy (1', ( <<<,,)))))
E(ewDeD WFUzzY (3', ,y)
Given calibration set Deal {(X,, Y.) let (, Srull ((Xi, Yi), Deal U (z, (x. Then the set
(FUI {u EJ: Srull ((x, Deat (xr, S Quantile --a
8s (s + )oo) }
has marginal coverage of for the test point, long as it is exchangeable with Deal-
Proof. As long as the test point (Xa+1, Yx+1) and the points Deat are exchangeable, the scores S((N++1, Ya+1) and S,+1 (Xn+1, Yn+1) == Stall (Xa+1, Y Yn+1), Deat U (Xa+1, Yn+1)) are also exchangeable, which yields the marginal validity of CFULL.
cases where the amount of calibration data is limited, this full-conformal adaptation allows us to avoid data-splitting. It is also computationally feasible in such scenarios, the burden of recalcu- lating the ensemble thresholds for each test point is tolerable when Deal is small.
C EXPERIMENT DETAILS
C.I DATASET PREPARATION
Overview. Our dataset preparation has two steps. We first identify or create train/val/test splits to replicate the standard machine learning pipeline. After obtaining tzain/val/test splits, we further split val into two subsets: the first for selecting the number of epochs (containing a random 30% of val) and the second for use the calibration set Deat (containing the remaining 70%). This split is necessary to ensure that Deal is exchangeable with the test points; if we reuse val for both epoch selection and calibration, this would violate exchangeability.
We now describe the train/val/test splits for each of the datasets we use.
P1@ntNer-300K. We use the provided train/val/tes: splits of Pl@ntNet-300k. The creation of the dataset is described in Garcin et al. (2022).
iNtturalist We use the 2018 version of iNaturalist" because it has the most "natural" class distribution (i.e., not truncated). Unfortunately. the provided train/val/ test splits are not well- suited to applying and evaluating conformal prediction methods for two reasons: First, the provided test set is not labeled and cannot be used for evaluation purposes. Second, the provided val l set class-balanced (with three examples per class), which means that it not a representative sample of the test distribution. If we used this validation set our conformal calibration set, this would violate the key ussumption that the calibration points are exchangeable with the test points. To remedy these two problems, same distribution. Specifically, we aggregate the t n and val data, then for each class randomly select 80% of examples to put in train, and then put 10% each in va 1 and test.
create own tr n/val/ 1/ test splits where all splits have the
Truncated versions. A key challenge of the long-tailed datasets we work with is that their test sets also long-tailed, which hinders reliable class-conditional evaluation for rare classes. To address this, we create truncated versions of the datasets by removing classes with fewer than 101 examples, resulting in 330 Pl e ntNNet-000K classes and 857 iNaturalist classes. For each remaining class, we assign 100 examples to test, then divide the remainder 90%/10% into trai train/val. For rare classes, we prioritize training set allocation, which may lead to having zero calibration examples.
'nttps:/ http://gthoob..ommpaannnnetettee 'htpps: /i thub. comb..com//// com/vii si pedi hat_nnaatttteeeeeeeeeeeeee /2018
23

---

Published as a conference paper at ICLR 2026
Specifically, to obtain MeentNtt-300K-trrnee we apply the above procedure the combined train, val, and test splits of Pl@ntNet-300K. To obtain iNaturalist- 2018-trumcated, we apply the procedure to the combined train and val splits of the original iNaturalist dataset."
Classes with zero calibration examples. Due to randomness in the data splitting, some classes end up with no calibration examples: 105 (out of 1081) in Pl@ntNet-300K, 606 (out of 8142) in INaturalist 2018, 12 (out of 330) in truncated Pl0ntNet-300K, and 47 (out of 857) in truncated Naturalistt2018.
C.2 MODEL TRAINING
After following the procedures described in Appendix C.I to obtain train, val, and test splits of each dataset, we train a ResNet-50 (He al., 2016) initialized ImageNet pretrained weights for 20 epochs using a learning rate of 0.0001. We use train for training the neural network and the randomly selected 30% of val for computing the validation accuracy. We then select the epoch number that results in the highest validation accuracy (up to 20 epochs).
C.3 COMPUTATIONAL RESOURCES
The system we use is equipped with 4x Intel Xeon Gold 6142 (64 cores/128 threads total @ 2.6-3.7 GHz, 88MB L3 cache) while the GPUs are 2x NVIDIA A10 (24GB VRAM each) and 2x NVIDIA RTX 2080 Ti (11GB VRAM each), for a total of 70GB GPU VRAM.
D ADDITIONAL EXPERIMENTAL RESULTS
Overview. In this section, we extend the experimental results from the main paper in the following ways:
1. We include additional methods: To ensure that we are evaluating fairly against existing procedures, include two additional baseline methods. The first is called EXACT CLASS- WISE, a randomized version of classwise conformal that is designed to achieve exact (rather than at least) a coverage, as described in Appendix C.3 of Ding et al. (2023). The sec- ond is rank calibrated class- caas--ooniiionall conformal prediction (RC3P), proposed by Shi et al. (2024). We also include the Raw FuzzY and FuzzY methods we propose Appendix A.
2. We evaluate on the truncated versions of PI@ nt@ntNKt3300K and INaturalist-201,, as de- scribed in Appendix C.I.
In Appendix D.1, provide additional results complement Figure 3 in the main paper. In Appendix D.2, we provide results for additional methods. In Appendix D.3, we recreate Figure using a ResNet-50 trained using focal loss rather than cross-entropy loss. In Appendix D.4, we do case study of the implications of our methods for plant identification. In Appendix D.5, we visualize the score thresholds we obtain from our methods demonstrate how they interpolate between the STANDARD and CLASSWISE thresholds.
D.1 RESULTS FOR MAIN METHODS
To aid interpretation of Figure 3, we extract the metric values for select methods for = 0.1 and present them Table 3 for Pl Pt@ttNet-30KK and Table for iNaturalitt20018 We also include a comprehensive visualization of all methods on all datasets, including the truncated versions, in Fig- ure 7. We remark that the two existing conformal prediction methods for many-class classification perform poorly in the long-tailed setting. The CLUSTERED CP method from Ding et al. (2023) defaults to the score threshold from STANDARD CP for classes that have insufficient examples to confidently assign to a cluster, resulting in poor class-conditional coverage. Meanwhile, the RC3P method from Shi et al. (2024) suffers from a similar data splitting problem as CLASSWISE CP, as it strongly relies on the estimation of the conditional score quantile, which can be very bad for classes with few calibration examples. Furthermore. it requires the estimation of the conditional top-k error for all classes, which is also hard when classes have few calibration examples. Moreover, as these
'For truncated datasets, we must compute marginal metrics differently when doing evaluation: due to the uniform class distribution of the test splits of these datasets, simple average over all test examples does not reflect marginal performance on the true distribution. We estimate the marginal coverage as E,ey P( where p(v) is estimated using r in and é is the empirical class-conditional coverage defined in (16). A similar weighting procedure is used estimate the average size.
24

---

Published as a conference paper at ICLR 2026
quantities are estimated the calibration the exchangeability assumption is violated which explains why the marginal coverage sometimes below 1 a. The methods we propose strictly dominate RC3P; for given average set size, get much better class-conditional coverage.
Table 3: Set size and coverage metrics for Pl@ntNet-300K using the Ssoftmax score and a 0.1. The arrows next to the coverage metric names indicate whether it is better for the metric to be smaller (L) larger (t).
MarginalCov (desired 0.9)
Method
FracBelow50 %
UnderCovGap 4
MacroCov 1
Avg. set size +
STANDARD
0.389
0.398
0.525
0.907
1.57
CLASSWISE
0,000
0.006
0.976
0.912
780.00
CLUSTERED
0.398
0.406
0,513
0,882
1.57
STANDARD W. PAS
0.167
0.193
0,755
0,902
2.57
INTERP-Q Q (T 0.9)
0.248
0.265
0.671
0.901
2.24
INTERP-Q Q (T 0.99)
0.151
0.168
0.785
0.905
3.95
INTERP- Q (T 0.999)
0.098
0.109
0,856
0.908
7.58
Table 4: Set size and coverage metrics for iNaturalist-201s using the soofmax score and a 0.1.
MarginalCov (desired 2 0.9)
Method
UnderCovGap FracBelow50%%
MacroCov f
Avg. size !
STANDARD
0.058
0.116
0.849
0.902
10.9
CLASSWISE
0.000
0.002
0.992
0,954
7430.0
CLUSTERED
0.059
0.118
0.845
0.880
8.4
STANDARD PAS
0.042
0.093
0.875
0.900
11.3
INTERP-Q Q (r 0.9)
0.034
0.081
0.891
0,907
16.8
INTERP-Q
Q (T 0.99)
0.020
0.055
0.924
0.923
31.1
INTERP-Q (T 0.999)
0.010
0.037
0.947
0.934
55.8
D.2 RESULTS FOR ADDITIONAL METHODS
FUzZY with other class mappings. Recall that in order to instantiate FUZZY CP, we must choose a mapping IT that maps each class y E y to some low-dimensional space in which we can compute distances meaningful way. Depending on the setting, some mappings may work better than others. The mapping we proposed in the main paper, 11 pprevalence is a simple mapping that works well in the long-tailed settings considered. In this section, we present results for two additional mappings. We describe all three mappings here.
1. Prevalence: This is the mapping presented before, which maps each class to its popularity in the train set:
Ilprevalence (v) = cn,y
train +
where Ey Unif([-0.01. 0.01]) independently for each for Y. We normalize using c = 1/(maxy'eD rair to ensure that [prevalence E [0, 1] so that the same bandwidth a has similar effects different datasets.
2. Random. As a baseline, we try mapping each class to a random value. Specifically,
Hrandom (y) U, where U, lid Unif([o, 1)) for E V.
3. Quantile. Recall the intuition described in the last paragraph Section 2, which says that we want mapping that groups together classes with similar score distributions. To further develop this intuition, suppose that when computing ÿy. we assign non-zero weights only to classes with the same 1 score quantile as class y. Taking the 1 weighted quantile would then recover the - a quantile of class as the number of samples with non-zero weight grows. This is because the mixture of distributions with the same
25

---

Published as a conference paper at ICLR 2026
Nee
Average: size
1,0000
10
0.00
0:25
0.50
0.0
0.2 Cowpe
0.4
0.50
0.75
1.00
0.8
0.9
1.0
500
MarginalCovv
iNaturalist-2U0
Averaee set sbe
10,000
1,000
0.0
0.1
0.0
0.1 UnderCovGapp
0.2
0.8
1.0
0.8
0.9
1.0
FracBelow30%%
MacroCor
MarginalCov
P N 100-330--trtmeatee
Average
10:
0.0
0.2
0.4
0.0
0.1 Covp
0.2
0.3
0.6
1.0
0.8
0.99
1.0
PracBelow50%
MacroCov
MarginalCov
INaturalis-2011-tmmma
Average set sime
1,000
100
0.0
0.0
0.1
0.8
1.0
0.8
1.0
FracBelow 50%
Cov
MacroCov
MarginalCovv
Stancacd
(a 0.4)
* Standars
(o
Standart
[a= [.06)
Standant
(o [.0))
Cluscaiio
Chasseise
Chansric
Clowarer
0.9)
0.1)
Lludbeeed
(a
(o (.06)
Clostreed
(.01)
(a
BC3P
(a (.1)
HCMP
(a (.00)
RCSP
[a (U02)
Standaadd
Shardand PAS
Standard PAS
Standard
Inherp-S
Puncy
0.2)
Plazy
Pazzy
(.06)
Fancy
Piagy
[1 0.21
Facer
(0 0.11
Facoy
(a- 0.06)
Fucy
(6 (.01)
Figure Average set size vs. FracBelow550% UnderCovGap, MacroCov, and MarginalCov for various methods on full and truncated datasets. For methods with tunable parameters, lines used to trace out the trade-off curve achieved by running the method with different values for fixed a. For FracBelow50% and UnderCovGap. it is better be closer to the bottom left. For MacroCov, the bottom right better. For MarginalCov, we want to at the bottom and to the right of the dotted line 1 for the at which the method is run.
quantile has that same 1 a quantile. Motivated by this idea, we map each class the linearly interpolated empirical is similar ccw but not the same due to the linear interpolation. We chose to linearly interpolate to avoid the problem of rare classes being mapped to co, which happens we apply no interpolation and directly apply Quantile, as it is defined in Section 1.2. Linear interpolation of quantiles is described in Definition of Hyndman & Fan (1996) and is the default interpolation method implemented in numpy quant le () (Harris et al., 2020). Given finite set A C R and level E [0, 1]. let LinQuantile, (A) denote the linearly interpolated 7 quantile of the elements of A or smax if A is empty, where $max = max s(X,, Y)) is the maximum observed calibration score. The quantile projection is given by
a quantile of its scores the calibration set. This
I Iquantite LinQuantile ((s( ())X,, Vi)),ez,)-
26

---

Published as a conference paper at ICLR 2026
Results from applying FUZZY CP with Iandom and Iqquatiie shown in Figure 8. We observe that perevaennce achieves better trade-off than Ilaaddom- which is expected, but also performs better than quantiil which is perhaps less expected. We believe that the reason that the quantile projection does not do well, despite being intuitively appealing, is that is very sensitive to noise due the low number of calibration examples per class. This likely causes classes that do not actually have similar score distributions to be mapped to similar values.
PlaintNet-300k
Average sime
1,000
10
0.00
0.25
0.50
0.0
0.2 Cndeeror
0.1
0.50
0.75
1.00
0.8
0.9 MarginalCoo
1.0
310
MacroCov
INuturalist-0118
Avergge size
10,000
1,000
10
0.0
0.1
0.2
0.0
0.1 Cap
0.2
0.8
1.0
0.8
0.9
1.0
5raceellow00%
MacroCov
MarginalCov
Stancaad
(a 0.1)
Standand
(a (12)
Standard
[a= (16))
Standart
[o (40))
Churniee
(a 0.13
(a 022
Claneriee
(a = (.06)
Clussrac
[a (U3))
Baw Fasay
[ux 0.41
Ruz Plaay
Rue Fizcy
0.06)
Ruy Fancy
(a 0.1)
Funs--llusion
0.11
Bun Bladedl
0.06)
(a 0.11)
Piagy-Jllxdaas
(t 0.17
Fazer-llasti
0.2)
(a- 0.06)
Fucoy
[o (.01)
Raw Funny Riguetsa 0.11
Run Pancy T1
0.20
Row Fincy-Tl
Row Faccy
0.01)
0.9)
Pansy Pigaett
(o-
Tanny Popnetts
Figure 8: The performance of FuzZY CP under different class mappings II.
FUZZY and INTERP-Q with PAS. Recall that proposed two solution approaches in the main paper. One led to a conformal score function, PAS, and the other led to new procedures, INTERP-Q and FuzZY. One may wonder if combining the two approaches provides additional benefit over using just one, so we test this idea. In Figure 9, we plot the results of running FUZZY and INTERP-Q using PAS as the conformal score function. The thick blue x's correspond to STANDARD with PAS (previously shown as gold triangles in the main text), and we observe that the interpolation methods, FUZZY with PAS and INTERP-Q with PAS, do provide some additional benefit at appropriately chosen parameter values by more optimally trading off set size and UnderCovGap, while doing no worse than STANDARD with PAS in terms of other metrics.
D.3 RESULTS USING FOCAL LOSS
To understand how proposed conformal prediction methods be combined with existing strategies for dealing with long-tailed distributions, we run additional experiments where we replace the cross-entropy loss with the focal loss in our ResNet-50 training. The focal loss was proposed by Lin et al. (2017) to improve model accuracy on rare classes by modifying the cross-entropy loss. We use the PyTorch implementation of focal loss from ht : /tt:://gtttttt htw.hh...ooomaaae www./deellll pytorch-mults mimiiiii. lassssssee wwwssalocaalaaaaaaaaaaaaaaaaaaaa The results are shown in Figure 10 and are qualitatively similar to the cross-entropy results in Figure 3 in the main paper.
with the default parameter values of = 2 and a = 1.
D.4 PL@NTNET CASE STUDY
In this section, we highlight the importance of conformal prediction for settings like Pl@ntNet, a phone-based app that allows users to identify plants from images (Joly et al., 2014). Conformal prediction provides value to such applications by generating sets of possible labels instead of point predictions. These prediction sets should balance several desiderata:
Marginal coverage. For the general public. prediction sets improve the user experience relative to point predictions by offering multiple potential identifications, increasing the
27

---

Published as a conference paper at ICLR 2026
1,0000
Average:
10
0.00
0.25
0.50
0.0
0.2
0.4
0.50
0.75
1,00
0.8
0.9 MarginalCov
1.0
50%4Beloow:0%%
UnderCovCap
MaeroCov
INatmrraist-2018
Avergee sime
10,000
1,000
0.0
0.1
0.0
0.1
0.8
1.0
0.8
0.9
1.0
51%%Below30%
UnderCovGap
MacroCov
Margins/Cov
Steadaed
Sauddaed (Boluadd
(Solans))
Stuskand (Solmess)
0.11
Clocraion
[r 0.1]
Cheosiie
Chanorie
(a 0.00)
Clansree
(o 0.11)
Clasend
[x 0.13
Cantered
Chutazed
0.06)
Chasaed
(a 0.01)
Qaterp=
Priny
Piazy
Dlooy
0.33
Fanoy
(6- 0.5)
Funcy
(o- (.00)
Func
(o- 0.01)
Figure 9: This plot is similar to Figure 3 except here we use PAS as the conformal score function instead of softmax. Note that the thick blue x's in this plot are equivalent to the gold triangles in Figure 3.
PlantNet-300K (Focal Loss]
1,000
100
Avengge
10
0.00
01:25
0.50
0.0
0..2 UnderCovCapp
0.4
0.50
0.75
1.00
0.8
0.9 MarginalCon
1.0
PracBelow-50%
NNaturalist-20 2018 (Focal Loss)
Antrage sipe
10,000
1,000
10
0.0
0.1
0.0
0.1
0.8
1.0
0.8
0.9 Margina/oov
1.0
PracBeloow50%
UnderCavGap
MacroCo
Standard
(a 0.4)
Standaod
(a
(L66)
Standant
b01)
Chuuniee
(a 0.21
Claisbbe
(a 0.1
Clanerbe
(o (u06)
Claasrac
[a (U33)
Steadace PA8
[x 0.2)
Sanilad
PA8
Sundand PAS
0.06)
Stazsand PAS
(a 0.1)
Figure 10: This figure is identical to Figure 3 except the base model is trained using focal loss (Lin al., 2017).
likelihood of accurate identification even with ambiguous or low-quality images. In order for users to have a good chance of making the correct identification most of the time, we want the marginal coverage be high.
coverage (especially in the tail). Ecologists would like to focus more 2. identifying (near) endangered or less commonly observed species for the purpose of scientific data collection. This calls for improving coverage of classes in the tail of the label distribution.
28

---

Published as a conference paper at ICLR 2026
3. Set size. For both the general public and ecologists, maintaining reasonable set sizes is crucial, as sifting through large sets is impractical.
An additional challenge in plant identification is that visually similar species often have an im- balanced number of labeled images, with one species significantly more represented. Standard conformal methods can suffer from occlusion, where the dominant species overshadows the rarer one at prediction time, that the classifier always assigns low probability to the rare class. If the rare species is never included in the prediction sets, this can lead to a vicious cycle of increasing imbalance, users simply confirm the classifier's suggestions.
For this case study, we focus on comparing our proposed method STANDARD with PAS against STANDARD and CLASSWISE when run at the a = 0.1 level. All methods have a marginal coverage guarantee, so focus comparing cladsiconditional coverage and set size between the methods. Figure 11 shows the class-conee conditional coverage and average set size for the three methods. Species that are considered endangered by the International Union for Conservation of Nature (IUCN) highlighted in red. We observe that CLASSWISE results in huge prediction sets. On the other hand, STANDARD with PAS enhances the coverage of classes that have low coverage under STANDARD with softmax without producing huge sets. We provide visual examples of some endangered species from the @l@ nONet-300K dataset in Figure 12.
Classwise
Standard
Standard PAS
PleintNer-3000 Combined Analysis (a 0.1)
Class coverage
200
400
600
800
1000
1,000
100
Av. sot
10-
200
400
600
800
1000
# exanples
5000
Noa- Uhreateged
Threstnned (CR/EN/VU)
0
200
400
600
800
1000
Class (sorted by prevalence)
Figure 11: A detailed look at results by class on Pl@ntNet-3000 dataset for three methods: STANDARD (with softmax), CLASSWISE (with softmax), and STANDARD with PAS. All meth- ods are run at the a = 0.1 level. Species are ordered according to the prevalence (computed on train), and the ones considered "threatened" according to the IUCN are highlighted in red.
D.5 UNDERSTANDING MOVEMENTS IN Qy
Figure 13 visualizes the Ó vectors that result from each of our methods. Recall that STANDARD has a single score threshold q. which we plot as horizontal line (ăy = 9 for all classes y). As a reminder, smaller value of Gy means that the class is less likely to be included in the prediction set, whereas classes with d, = (or oc) are always included in the prediction set. For STANDARD with PAS, plot the effective by by observing that the class-uniform ğ threshold in terms of the PAS score implies classwise i thresholds in terms of the softmax score. Specifically, observe that
SPAS (x, v) < q p(ylx)
p(y) < q
definition of spAS
p(y(z) < 1 + (p(y)
Soottma (I,y) v) < 1 + gp(v)
29

---

Published as a conference paper at ICLR 2026
Species: Metasequoia glyp- toxtroboides Hu & W.C. W.C.Cheng # of examples: 410
Species: Vanilla plamifolia Jacks.
Species: Abeliophyllum distichum
Andrews
Nakui
of examples:
35
# of examples:
Method
Coverage
Size
Method
Coverage
Size
Method
Coverage
Size
Standard 0.00
1.3
Standard
0.60
3.2
Standard
0.00
4.0
Classwise
1.00
781.3
Classwise
1.00
782.2
Classwise
1.00
784.0
Std w. PAS
1.00
7.3
Std w. PAS
0.80
10.0
Std w. PAS
0.00
6.0
Figure 12: Examples of three species in Pl@ntNet-300K considered "'endangered" by the IUCN. Each table reports the empirical class-conditional coverage and the average size of the prediction sets when the given species is the true label for baseline methods (STANDARD with softmax and CLASSWISE with softmax), as well as one of our proposed methods (STANDARD with PAS).
where Ssoftmax denotes the softmax score function described in Section 1.2. Thus, for a class y E V, we refer to 1 + (p(y) as the effective îy of STANDARD with PAS.
All of our methods are intended to "interpolate" between STANDARD (with softmax) and CLASSWISE (with softmax). INTERP-Q interpolates in a very literal sense by linearly interpolating
q and Gy The other three methods appear to interpolate in an alternative geometry and allow the ay for each class to be adjusted in a different way. These plots also reveal why CLASS WISE yields such large sets in this setting: there are many classes for which icw (or oo), and all of these classes are always included the prediction set.
D.6 ADDITIONAL DECISION ACCURACY PLOTS
In the main text we present decision accuracy plots for only one of our methods, STANDARD with PAS. In Figure 14, we present results for our other method, INTERP-Q.
30

---

Published as a conference paper at ICLR 2026
Standard 6
Claznize 4
1.0
1.0
0.5
Stanshed PA8
Lo-30)
Pracy (e Io-30
Incppp fr
1000
500
Class (sorted by 4.90
1.0
1.0
0.5
Te:30
Ister--Q
50n
1000
0.5
0.5
Paacy
Interp-92
0.5
68
Pinryy
Interp-Q
0.0
0.5
Jaterp-2 0.20
0.0
1000
1000
0.50.
Raw Piezy
Flees 1300)
0.0
0.0
500
1000
1000
Class (sorted by 20W
Class (sorsed by 25th
Class (sorted by "
Figure 13:
Score thresholds G, of our proposed methods (STANDARD with PAS, Raw FUZZY, FUZZY, and INTERP-Q) on PI@ mtNet-300K for a = 0.1. For visualization purposes, infinite values of ây are replaced with one, the maximum possible softmax score value. Furthermore, for case of comparison, we sort the classes in ascending value of the CLASSWISE thresholds gew (plotted as dashed red line).
31

---

Published as a conference paper at ICLR 2026
Pliitt Prantteer
Docisioc arater
Kog 0%
Yosp. 25%
Vesp. 50%
Tosp. 75%
Tex
100%
200
200
200
Class
Class
Class
Claes
Claes
INaturalist INtturalis-2211--rruum 201-mmmatttd
Decsion accuraty
Teop. 0%
Tesp. 25%
Tep. 50%
Teg.. 75%
Top.
100%
500
Class
Cluss
Class
Clues
Claes
3,0
0.25
0.75
0.86
0.00
1.0
Figure 14: with sets from INTERP-Q run with different values of r. Recall that = recovers STANDARD and recovers CLASSWISE. Classes are ordered by decreasing decision accuracy of H Hexpert under each method.
Class-conditional decision accuracies for range of decision makers when presented
32


Reviews:
[{"review_id": "bmJM9IvSwt", "reviewer": "Reviewer_mYkq", "summary": "Existing conformal prediction methods often trade off between poor class-conditional coverage and overly large prediction sets. This paper proposes a new non-conformity score, PAS (and its weighted variant WPAS), which aims to interpolate between marginal and class-conditional conformal prediction. The goal is to achieve better coverage–efficiency balance under long-tailed label distributions.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tAddresses the practically relevant problem of conformal prediction under long-tailed label distributions.\n\n2.\tThe proposed PAS/WPAS scores are simple and easy to implement.\n\n3.\tThe experiments offer preliminary evidence that the proposed method improves coverage fairness under long tailed setting.", "weaknesses": "1.\tTruncated dataset setup: The test datasets are balanced by truncating rare classes and retaining those with more than 100 samples per class. The choice of this threshold is not explained. Why 100? In more realistic scenarios where both calibration and test sets are long tailed, how would the proposed method perform?\n\n2.\tMotivation: The new non-conformity score (PAS) is motivated by an oracle analysis showing that the optimal set depends on p(y|x)/p(y), but this only characterizes an ideal solution rather than a provably better practical formulation. Overall, the motivation appears ad-hoc, and it remains unclear what concrete limitation of existing scores (e.g., APS, LAC) PAS resolves under long-tailed distributions. \n\n3.\tBassline: The paper does not include [1], which also targets class-conditional coverage under imbalanced settings, which appears highly relevant to this work. In addition, standard non-conformity scores such as APS [2] and LAC [3] are omitted. \n\n4.\tScore comparison: Since APS emphasizes X-conditional coverage and LAC minimizes the expected set size, including them would provide a more informative comparison of the adaptiveness–efficiency tradeoff that PAS/WPAS aims to improve. It would also be helpful to clarify whether a weighted combination of APS and LAC could already achieve a similar trade-off, and how PAS/INTERP-Q compares in that respect.\n\n5.\tEvaluation Metric: The study reports FracBelow50% and UCG but omits the more intuitive Under-Coverage Ratio (UCR) used in [1], which reflects the fraction of classes that fail to meet the target coverage and would provide a more informative evaluation.\n\n[1] Yuanjie Shi, Subhankar Ghosh, Taha Belkhouja, Jana Doppa, and Yan Yan. Conformal prediction for class-wise coverage via augmented label rank calibration. NeurIPS 2024.\n\n[2] Yaniv Romano, Matteo Sesia, and Emmanuel J. Candes. Classification with valid and adaptive coverage. NeurIPS, 2020.\n\n[3] Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least ambiguous set-valued classifiers with bounded error levels. J. Amer. Statist. Assoc 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HoT0518dJA", "reviewer": "Reviewer_pAFA", "summary": "The paper applies conformal prediction (CP) to an extremely long-tailed classification setting. The standard CP tends to ensure coverage of common classes and under/miss coverage of the rare ones, whereas classwise CP covers each class but produces unusable prediction sets. The authors address this challenge by 2 approaches: a) PAS / WPAS -> the idea here is to, instead of trusting the raw model probabilities, recalibrate label scores by their rarity, so that rare classes get a boost. The authors claim this results in smaller prediction sets compared to classwise CP, and is also fairer to rare classes compared to standard CP. b) INTERP-Q -> instead of score recalibration, this approach uses a controllable parameter that provides a cutoff threshold that balances between standard CP and classwise CP thresholds. It gives the user a controllable knob that can decide how much to protect tail classes versus how large a prediction set can be.\nEmpirically, they compare against standard baselines and the evidence supports their claims.", "questions": "Please refer weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Strength\n\na) The main paper is well motivated, mostly clear and easy to follow.\n\nb) It tackles conformal prediction in the extreme long-tailed scenario, which is practically important.\n\nc) The class coverage vs prediction set size tradeoff as a problem formulation itself seems novel. The two proposed approaches also appear reasonably original. \n\nd) Empirical studies are convincing, and their human decision maker simulation experiment seems interesting to me.", "weaknesses": "Weaknesses\n\na) I could understand the working of PAS/WPAS and INTERP-Q, but I couldn't clearly find the motivation for utilizing either/or both of them. The two methods seem to address different parts of the pipeline, but the paper does not clearly explain a practical guideline when a practitioner should pick PAS/WPAS, INTERP-Q, or use them together.\n\nb) The experimental details in the appendix mention utilizing a truncated version with n-core filtering with n = 101. I am curious: doesn't this contradict the problem the authors are trying to solve? The paper is motivated by the extreme long tail, but the analysis is done only on this truncated subset. It would be helpful to clarify whether the main conclusions still hold for the truly rare classes that were filtered out.\n\nc) Based on my understanding, PAS / WPAS is motivated by the hypothesis that reweighting by class prevalence is optimal for trading off average per-class coverage and set size. However, in the implementation, PAS simply recalibrates standard split CP scores, which only guarantees marginal coverage, not class-conditional coverage. So the claim that there are fewer under-covered long-tailed classes seems to be empirical rather than a strict guarantee to me. Similarly, INTERP-Q appears to be only loosely bound to the standard finite-sample marginal guarantee. Neither approach provides a strict class-conditional guarantee, which makes the strength of claims less clear.\n\nd) The paper only evaluates using splits where calibration and test share the same long-tailed label frequencies. It would be interesting to see whether the proposed methods hold up when the test distribution differs from calibration.  I believe stress-testing robustness to even basic shifts would better support the practicality claims of the paper.\n\ne) Comparison with standard CP, classwise CP, and the clustered variant is good. But I would like to see how the method fares against some recent methods that seem to have similar motivation/methodologies [1][2].\n\n\n[1] Liu, Shuqi & Huang, Jianguo & Ong, Luke. (2025). Conformal Prediction Meets Long-tail Classification.\n\n[2] Yuanjie Shi, Subhankar Ghosh, Taha Belkhouja, Janardhan Rao Doppa, and Yan Yan. (2024). Conformal prediction for class-wise coverage via augmented label rank calibration (RC3P). NeurIPS 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9x3c0RcBgz", "reviewer": "Reviewer_5apV", "summary": "The paper studies conformal prediction for very imbalanced multi class problems where a few classes have many examples and many classes have very few. The goal is to produce prediction sets that keep marginal coverage guarantees while improving coverage for rare classes without exploding set size. The authors offer two main ideas. First, a new score called prevalence adjusted softmax and its weighted version that aim directly at macro coverage, with an option to upweight classes of interest such as endangered species. Second, an interpolation procedure called INTERP Q that blends classwise and standard conformal thresholds to trade off set size and class conditional coverage, with a conservative marginal guarantee. They test on Pl@ntNet 300K and iNaturalist 2018 and report better coverage for tail classes at small or moderate set sizes, plus a study with simple human decision models.", "questions": "How sensitive is PAS to misspecified prevalence. If the true p of y differs from the training estimate, can you still expect macro coverage gains.\n\nCan you provide a practical scheme to choose tau on the calibration fold to meet a specific marginal coverage while improving class conditional coverage.\n\nHave you tried teaching a small correctness predictor on the calibration set and using that as the score inside PAS or as a rank corrector.\n\nMacro coverage can look good while a handful of classes are still far below target. Can you add plots of the full distribution of per class coverage, not just the fraction below fifty percent.\n\nFor a practitioner who wants sets of average size at most three while lifting average coverage of at risk species above a target, can you give a small recipe that picks alpha and lambda and, for INTERP Q, tau.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Targeting macro coverage with a simple change to the score is a neat idea. It connects the oracle form of the optimal set for macro coverage to a practical score based on p hat of y given x divided by the estimated prevalence. The weighted version lets users push coverage toward special subsets like at risk species.\n\nThe paper is easy to follow. The problem is well motivated with plant identification. The two approaches are separated and labeled. Table 1 is a good map of methods and guarantees.\n\nSelective set prediction in long tailed regimes is common in biodiversity and medicine and also in open world recognition. A method that improves tail coverage while keeping sets short is directly useful for real labeling workflows and can slow collapse of rare labels in human in the loop systems.", "weaknesses": "PAS relies on p hat of y given x and an estimate of label prevalence. In real systems there is often label shift between train, calibration, and test. The paper does not test robustness under such shift, even though label shift directly changes the p of y term that PAS divides by.\n\nThe 1 minus 2 alpha lower bound is likely conservative, as the authors note, but the paper does not quantify the realized marginal coverage gap across settings or give a simple correction to hit a target level.\n\nMost results use softmax scores from a standard ResNet trained with cross entropy, with one mention of focal loss in the appendix. Since the approach is driven by score quality, the work would benefit from a broader check across stronger long tail learners such as logit adjusted training and from alternative scores such as APS or label ranking scores, even if set sizes grow\n\nThe human models are an expert verifier and a random guesser, plus mixtures. That is a useful first look, but real users are neither.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UpuzbDYXfi", "reviewer": "Reviewer_mwex", "summary": "This paper addresses the challenge of constructing useful prediction sets using conformal prediction (CP) in long-tailed classification settings, such as plant identification, where rare classes are underrepresented. Standard CP achieves marginal coverage but often fails to cover rare classes adequately, while class-conditional CP methods produce excessively large sets. The authors propose two approaches: (1) a new conformal score function called prevalence-adjusted softmax (PAS) and its weighted variant (WPAS) to target macro-coverage (the unweighted average of class-conditional coverages), derived from oracle-optimal sets that balance set size and coverage; (2) INTERP-Q, a simple interpolation between standard and classwise CP quantiles to trade off set size and class-conditional coverage. They evaluate these methods on long-tailed image datasets, demonstrating improved trade-offs compared to baselines.", "questions": "1. Can this method be combined with other scores, such as RAPS and SAPS?\n2. In figure 2, there are cases where test label distribution is different from training and validation distribution, how this affects the results, especially considering PAS that is based on class normalization.\n3. Can this method be extended to regression problems with class imbalance, where the continuous target variable has a highly skewed distribution with certain value ranges, much more frequent than others?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper introduces novel conformal score functions (PAS and WPAS) specifically tailored for macro-coverage in long-tailed settings, grounded in theoretical derivations of oracle-optimal prediction sets. This extends prior work on CP beyond marginal or basic class-conditional guarantees, addressing a practical gap in applications like biodiversity monitoring where rare classes are critical. The paper is clear, with well-defined notation, algorithms, and metrics, making the contributions accessible.", "weaknesses": "The interpolation in INTERP-Q is simple but lacks deeper analysis of why linear weighting works well, and why it can only guarantee a conservative coverage bound ($1-2\\alpha$) while being close to $1-\\alpha$ in practice. \n\nAdditionally, comparisons could include more recent long-tail methods (e.g., Ding et al. (2023)).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)