Singleton-Optimized Conformal Prediction
Authors: Tao Wang
Yan Sun
Edgar Dobriban
Venue: ICLR 2026 Poster (2026)
Keywords: Conformal Prediction; Uncertainty Quantification
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We design a new non-conformity score that encourages conformal prediction sets to output singletons (one-element sets) in classification.

Abstract: Conformal prediction can be used to construct prediction sets that cover the true outcome with a desired probability, but can sometimes lead to large prediction sets that are costly in practice. The most useful outcome is a singleton prediction---an unambiguous decision---yet existing efficiency-oriented methods primarily optimize average set size. Motivated by this, we propose a new non-conformity score that is motivated by minimizing the probability of producing non-singleton sets while maintaining coverage. Starting from a non-convex constrained optimization problem as a motivation, we provide a convex-geometric reformulation and associated algorithm for computing the non-conformity score and associated split conformal prediction sets in $O(K)$ time for $K$-class problems. Using this score in split conformal prediction, we introduce Singleton-Optimized Conformal Prediction (SOCOP). We evaluate our method in experiments on image classification and LLM multiple-choice answering, comparing with standard non-conformity scores such as the (negative) label probability estimates and their cumulative distribution function; both of which are motivated by aiming to optimize average length. The results show that SOCOP increases singleton frequency (sometimes by over 20\%) compared to the above scores, with minimal impact on average set size.

Published as a conference paper at ICLR 2026
SINGLETON-OPTIMZER CONFORMAL PREDICTION
Tao Wang' * Yan Sun Edgar Dobriban'
University of Pennsylvania New Jersey Institute of Technology
(tawan, dobribar)@whartenn upenn. edu {yan. sunjenjitejjd
ABSTRACT
Conformal prediction can be used construct prediction sets that cover the true outcome with a desired probability, but can sometimes lead to large prediction sets that are costly in practice. The most useful outcome is singleton prediction an unambiguous decision-yee existing efficiency-oriented methods primarily opti- mize average set size. Motivated by this, we propose a new nonconformity score that aims minimize the probability of producing non-singleton sets. Starting from a non-convex constrained optimization problem as a motivation, we provide geometric reformulation and associated algorithm for computing the noncon- formity score and associated split conformal prediction sets in O(K) time for K-class problems. Using this score in split conformal prediction leads to proposed Singleton-Optimized Conformal Prediction (SOCOP) method. We eval- uate our method in experiments on image classification and LLM multiple-choice question- answering.. comparing with standard nonconformity scores such as the (negative) label probability estimates and their cumulative distribution function: both of which are motivated by optimizing length. The results show that SOCOP increases singleton frequency (sometimes by over 20%) compared to the above scores, with minimal impact on average set size.
INTRODUCTION
Reliable uncertainty quantification is often needed for deploying predictive models settings of importance. While standard single point predictions can be very useful if models are accurate, they can be problematic if model accuracy drops. Prediction sets address this limitation by providing a subset of possible labels, C(x) C V. for given input
X. The primary requirement for such sets usually a form of coverage. Formally, given features X E X' with some distribution, and multi-class label Y V, seek sets C(X) C y satisfying the marginal coverage guarantee a. Conformal prediction (see e.g., Vovk et al. 1999 Gammerman et al.) 1998 Vovk al. 2005 etc) offers methodology for constructing prediction sets that satisfy this guarantee under the mild assumption of data exchangeability.
P(Y E C(X)} >
While validity is essential, the practical utility of prediction set is determined by its efficiency. For instance, trivial set containing all labels is valid but uninformative. In practice, efficiency often evaluated by the expected size of the sets Ex]]C(X]]] A variety of works have studied how to achieve small sets on average, ranging from choosing suitable nonconformity scores to ex- plicit optimization approaches (see e.g., Takeuchi] 2020 Sadinle et al.) 2019 Romano et al. 2020
Angelopoulos al. 2021: Kiyani et al. 2024 etc).
However, average size not necessarily the ideal measure of efficiency. Often, the most desirable outcome is an unambiguous prediction, singleton sei ser]connaining only one label. A set of size two or more may require additional human intervention or changing the workflow when used in down- stream analysis, and thus brings outsized cost. This motivates alternative efficiency criterion, first conceptualized Vovk et al. (2005) as the M-criterion, which seeks to minimize the probability
Coorreppnndence to: tawan&wharton upenn. edu, dobribane wharton. upenn edu 'Strictly speaking. singleton set refers to a cardinality of exactly one ([C(X) = 1). In this work, we use the term "singleton objective" to broadly refer to the goal of minimizing the probability of returning multiple labels ((C()) 1). As we discuss below in our experiments, zero sets occur extremely rarely, and so the two objectives effectively coincide.

---

Published as a conference paper at ICLR 2026
of producing non-singleton set, Px|[C(X) > 1]. We refer to this the singleton objective. To our knowledge, practical conformal prediction methods that aim to optimize the singleton objective have not yet been developed.
In this work, we bridge this gap by developing conformal prediction sets motivated by optimizing a combination of the singleton objective and the expected length for classification problems, subject to coverage. We begin by formulating this as an optimization problem over prediction sets (which discrete variables). Our main contributions are then follows:
1. Nonconformity score inspired by singleton objective. We use the singleton objective as inspiration to define a nonconformity score aiming to enhance singleton probability, Since the original optimization problem constrained, we consider its Lagrangian, which show is separable across v. We show that for each fixed v, the optimal prediction set is the set of top-few labels, and that the prediction sets are nested as the Lagrangian penalty parameter increases. This motivates us to define a nonconformity score based on nested 2022).
conformal prediction (Vovk al. 2005 Gupta
2. Efficient algorithm to compute nonconformity score:
We derive a highly efficient algo- rithm to compute the nonconformity score, through geometric perspective. We show that this problem reduces to finding the lower convex hull of for K-class classification problems, which has O(K) complexity per instance. We show that split conformal prediction sets can computed with the same complexity.
K two-dimensional points
3. Empirical validation: We conduct detailed experiments on three image classification datasets (two versions of ImageNet and TissueMNIST) and LLM multiple-choice question answering. The results demonstrate that our method, which we call Conformal Prediction (SOCOP), achieves a favorable balance between minimizing average set size and maximizing the frequency of singleton predictions compared to state-of-the-ar baselines. Often, we can reduce the non-singleton probability by a large fraction (such as 20%) while only incurring small increase in expected set size.
Notation. For a positive integer K, we denote [K] == (1....,K]) K) We denote the (K 1)- dimensional simplex of probabilities by AK-1 { ... =k) E A, we write |A] for its cardinality. The indicator of set A is denoted by (A).
2, 1}. For finite set
1.1 RELATED WORK
The origins of distribution-free prediction sets date back to the early works of Wilks 1941), Wald
[144),, [Scheffe & Tukey [1945). and Tukey [1947] 1948). Distribution-free inference and conformal
prediction has been extensively studied in recent works (see, e.g.. Saunders et al. 1999 Vovk et al.,
1999
Papadopoulos
al. [002]] Vovk et al. 2005 Vovk 2013 Lei al. [013]] Lei & Wasserman
2014
Romano et [0.20 etc). Overviews of the field are provided by Vovk et al. 2005 Shafer & Vovk 2008). and Angelopoulos Bates (2023).
Lei et | 2018
Recent research has started investigating ways to improve the efficiency of prediction sets. (Sadinle al. 2019) have shown that the true probability of the labels given the features is the conformity score that leads to prediction sets that minimize expected length. Adaptive scoring schemes (Ro- mano et al. 2020 Angelopoulos et al., (02T)) have similar motivation, but are derived from conditional coverage perspective. These works related to ours in that also derive new non- conformity score. However, taking into account the singleton probability or M-criterion (Vovk et al. 2005), our work requires addressing prediction sets. Recent work aims to directly optimize the length, possibly with conditional coverage guarantees (Kiyani et al., 2024). Other work has explored different notions of efficiency, through direct optimization (Stutz et al. 2022, Shi et al. (2025), computational shortcuts (Liang et al.
technical challenges in terms of efficiently computing the
2023
or other approaches. see e.g., [Liagg et al. (2025), Le Bars & Humbert (2025); Braun et al.
2025
Behboodi et al. (2025), etc. Additional related work is discussed in Appendix
2 A SINGLETON- NONCONFORMITY SCORE
We begin with population-level optimization formulation that motivates our nonconformity score.
2

---

Published as a conference paper at ICLR 2026
2.1 PROBLEM FORMULATION
We consider a classification problem with labels y E = (1...., , K} and features x E X = Rd_ Our goal is construct prediction sets C(x),, for all x. satisfying the coverage guarantee P(Y C(X)) > -aa a. Let M be the collection of a (measurable) prediction sets C: X' - 2 Our
motivating problem to find prediction sets that are optimal with respect linear combination of the singleton objective and length, subject to coverage:
min CEM
FA = Px [IC(X) > 1] + XEx [IC(X)]
S.L. G(C) IPYY E C(X) - (1 0.
where X > 0 is regularization parameter that we will set later. This objective balances the probabil- ity of non-singletons Px [|C(X)] > 1] and the expected size Ex [[C(X)]]- We will argue that this leads to a favorable trade-off, whereby increasing one by small amount results in large decrease in the other.
This optimization problem is defined over prediction sets, which belong to discrete, discontinuous space (e.g., the linear combination of two sets is undefined), and so standard gradient- opti- emphasize that this problem will merely serve as
mization methods are not applicable. However, a motivation for us to define a useful nonconformity score. We will not attempt to solve this problem exactly, but rather use it as starting point, transforming it into form that allows to derive our nonconformity score.
Our first step towards defining the nonconformity score is to study the dual of the above problem. This will allow us to use separability in the solution, and thus derive a nonconformity score. Let Px be the distribution of X. The Lagrangian with dual variable 7) > 0 is:
Lx(C,n) = | x [(([()) 1) A|C(z)1 -
Σ #EC(())
Prix( Px(dx) + t1111 - a).
(1)
Since ()(),m) FX(C) < (C) for every feasible C' and > 0, minimizing Lx n) gives a lower bound on the original problem
A key observation is that the minimization of (C, 17) over C' is separable in x, i.e., it can be solved by optimizing over each x separately. Denote, for all x E X, the per-instance loss
(p(1m)) (C(z):n) I(C(()) > 1) X[C(2)) 17
Σ p(vlz).
(EC(x)
Then, we can write Lx Jx a (C(+);;) Px (dx) + 17(1 - a) as an integral of the
per-instance loss. Thus, ① can be minimized over C M by minimizing fp(lln). (C(z); for each x E X separately. Since l)) (C) can be viewed as an instance-level cost associated with the prediction set C(z) and the probabilities of the labels p(-1z), this motivates us to leverage it to construct our nonconformity score.
Continuing with the general approach of leveraging the theoretically optimal prediction set for the construction of the nonconformity score, we study the minimization of f. For any probability distri- bution E AK-I, and Lagrange multiplier 7) > O, we consider solving for the following singleton- optimized set Sny C [K], defined by the optimization problen
S Sp E arg min (x.x SEY
We will endow X' with the Borel o-algebra. All quantities considered in this paper will be measurable with respect appropriate o-algebras; this will not be mentioned further.
'A optimal solution C" to this problem minimizes the original objective F subject to the constraint P(Y E C(X)) P(Y E (X)).. For this reason, it would be reasonable consider the original opti- mization problem subject to the constraint G(C) = 0, in which case, the Lagrange multiplier approach could provide a certificate of optimality or near-optimality quite directly. However, ultimately, we will not solve the above problem directly but rather only use it as way to define a nonconformity score, which we will then in conformal prediction. Therefore, certifying the optimality of our intermediate solution to the original optimization problem is a central goal of our research.
'f there are multiple solutions, we choose any set that has a minimal size. The same holds for the definitions in the following text. Our claims will hold for all optimizing sets, and for simplicity will refer to "the" optimizer.
3

---

Published as a conference paper at ICLR 2026
Then, all solutions of minimizing | can be writter C2(2)) pp,p((12)-
2.2 DEFINITION OF THE NONCONFORMITY SCORE
From now on, without loss of generality, we order the probabilities such that Yyn 2 Typ
Tyx > 0, where K = | Fortunately, the structure of the prediction sets Sn,y can be characterized. A starting point is the following simple result, whose proof (with all proofs) is provided in the appendix. For any j E (0, I,.... K}, let F, denote a set of the top j labels, breaking ties arbitrarily; where Fo is the empty set.
Lemma 2.1 (The structure of singleton optimal sets). For any vj 2 0 and 7 E Ax-1 Sny is the set of top-j labels for some j that depends on y and 7,
The next and crucial observation that the sets Snn from 2 are nested as a function of the Lagrange multiplier 1).
Lemma 2.2 (Nested Sets Property). For 0 & 77 < 12. we have Sm S &main
This motivates us to define nonconformity score via nested conformal prediction (Vovk et al, 2005| Gupta et al. 2022). where we aim to find the smallest 7nd thus the smallest set Sn that contains the true label.
In practice, the true conditional probability p(-(z) is typically unknown; and instead, we only have access to an estimated probability p(-(z)). By plugging in the estimated probabilities in lieu of the true ones and using nested conformal prediction (Vovk et al. 2005: Gupta al. 2022). we define the singleton-optimized nonconformity score:
Definition 2.3 Singleton-optimizee nonconformity score). For an input x E X with label y E D, for probabilistic predictor P such that C(- z) is probability distribution over V, and regularization parameter X 3 0, define the singleton-optimize
nonconformity score
r(a, v) == (x = inf {r 2 0::0 y E Sr,((-m),))
(3)
where the singleton-optimal set Snn, X defined in (2) for a Lagrange multiplier rj > 0.
In principle, this nonconformity prediction, including split conformal prediction (Papadopoulos et al.) (002), cross-conformal pre- diction [Vovk] 2015), Mondrian and label conditional conformal prediction (Vovk et al. 2005). etc, to construct prediction sets. The method of choice depends on the type of data and guarantee de-
be used with variety of techniques from conformal
sired. However, the practical use of the nonconformity score first requires efficient algorithm to compute it. As we will see below, a naive search over т can be expensive when the number of classes is large. In what follows, we discuss how to compute the nonconformity score r efficiently. Readers more interested in experimental results may skip to Section 3
2.3 GEOMETRIC APPROACH TO COMPUTING THE NONCONFORMITY SCORE
In order to develop a method to compute the nonconformity score, we first study the problem of computing the prediction set S,,y for given vector of probabilities Y E AK-1- This is used directly in the nonconformity score. By Lemma 2.1 the optimal prediction sets from (2) are equal to the top few labels. Specifically, S, Fe(nn)) where x(1)7) is the optimal subset size (or optimal index), defined via the optimization problem:
<< 7) arg min OSkSK
(k. == T(k > 1) Als 1)
Jo.
(4)
For fixed value of n, the optimal index x(m7) can be found in time O(K) by observing that, for > 3, the gaps dn == (k 7) (r - 1,7) A 7/7s, are non-decreasing in due to the ordering Ty Tya 2 ... Hence, to find the optimum, it is enough to find the smallest index k* 3 such that 0g- < 0 < 6 if such an index exists; otherwise setting ks K. Then, we compare the objective value at A with those for k 0, 1 and choose the best. This immediately leads to an O(K) algorithm for computing the prediction set S.
'hhen the value of is fixed clear from the context, we will often omit it from our notation.
4

---

Published as a conference paper at ICLR 2026
Next, by leveraging the reduced problem 4 the nonconformity score in 3 can be equivalently written as:
(5) A direct approach might be to search over values of r. checking x (T: > i for each case, until we find a value that approximates the true value within a certain desired accuracy. However, this direct approach becomes computationally challenging for large values of K, because computing the optimal index takes linear time O(K) for each т. Therefore we propose a fast alternative computa- tional method, which relies on studying the optimal index for different values simultaneously, and can be viewed through geometric perspective.
r(z, v1) in > 0: x (T: > i}.
first step observation is that the nested sets property immediately implies that yj + k(173 7) is monotone step function.
Corollary 2.4 (Properties of optimal index function). For any 7 AK-L. <(())) [0, oo] -
{0. 1, 1,.., K} is a monotonically non-decreasing, left-continouus step function with k(0; 7) = 0 and
7( = 1im x((1))= K.
Next, we aim to characterize the specific points where the jumps of k. happen. Denote r₂ E Ts. and 9k = [[k > 1) + Ak for con- ciseness. For each k 0, 1, ..... we consider the point Pj- = (T 9k) in TR... This yields a set of K points P = (Po..... Pk). Our al- gorithm will leverage the convex hull of P, i.e., {E/ P;P; :3; > O, E B = 1}. which is convex polygon in R². The lower convex hull is the lower boundary of this polygon, starting from Po (0, to Px- = (1,1 XK).
2.00
Point Px (fx, ge)
14.7
1.75
Lower Convex Hull of
3.2
3.7
1.50
P:
6 1.00
0/ 0.2
0.75
0.50
Let the ordered sequence of vertices of the lower convex hull of P be {P P......, Pr.. ). where Um are indices from {0. (.. Kj. By construction, we have = 0 and Um = K, since I'x are strictly increasing and Jk are non-decreasing with k. For i 1,..., m, define the slope of the edge connecting the ver- tices P Po._, and P... as 1): == (g., (r.-,)/(TT (T To: .). To unify the analysis, we define Vo- == 0 and mm = +00 The following theorem (with proof in Appendix characterizes the jumps and slopes of k. Figure [ shows an example of lower convex hull of point set P for probability vector with K = 1
0.00 P
01
Pg
0.5
0.0
0.2
0.4
0.6
0.8
1.0
Fk
Yo
< 01
Figure 1: Lower convex hull for a simulated proba-
bility vector with K = 10.
Theorem 2.5 (Characterizing the optimal index function k). The range of k(T7)) for E (0)
is precisely the set (vo, v Jm of indices of the vertices of the lower convex hull. Moreover, the discontinuity points of <([); 7) are the slopes T) i 1,...,m, of the edges of the vertices. Specifically,
for 1) [0,m]
Vi. K,
for 1 E (Mi, 16+1), 1gigm-1
for E (1m, oo).
Computing the nonconformity score. With Theorem 2.5 we can efficiently compute the noncon- formity scores and the final prediction sets. The form r(x, y1) = inf(n) 2 Yi E Sn,((12)) = inf(n > 0: <<m p(-(x)) > i) is equivalent finding the smallest slope nj that leads to a prediction set of size Uj > i. Intuitively, the slope 7) represents the "price" per unit of coverage relative to the set-size penalty. Navigating the lower convex hull corresponds to finding the minimum price required to "purchase" enough coverage to include the target label y: in the set. To compute this, we can first find the vertices of the lower convex hull (which can be done with standard approach, Algorithm 2). and identify the correct slope; these can be performed in single loop. Having an efficient algorithm to compute the nonconformity score is useful in a variety of conformal prediction
'Red points indicate the hull vertices, and di denote the corresponding slopes. The nonconformity scores r(z, 21) nh, r(x, v2) r(x,, UT) = 72, r(z, ys) na. r(2, yo) 174, and (zz, y1o) 75-
5

---

Published as a conference paper at ICLR 2026
methods, such as split conformal prediction (Papadopoulos et al. 2002), cross-conformal prediction [Vovk] 2015). Mondrian conformal prediction Vovk et al. 2005 etc. In this paper, we will focus on split conformal prediction, which is one of the most popular and widely applicable methods.
To run split conformal prediction given a set of n calibration data points and a desired target coverage level 1 in [0, 1], we can compute q, the (1 - (1 + 1/n) /n) th quantile of the nonconformity scores over the calibration set; see Algorithm in the Appendix.
Coverage guarantees. Naturally, the guarantees of conformal prediction are inherited here. Specif- ically, if our calibration and test data point are exchangeable, then we have that P(Ya+1 E C(X++1)) > 1 - o, where the randomness is taken jointly over the calibration and test data.
Computing the prediction set. Consider a new data point In++1 for which we aim compute the prediction set ((n+1)). The range of r( ./),, ... r(I++1+ +1; 3KK is the set of the discontinuity points of s(m; p(-[[x++1) 1). Therefore, due to the monotonicity of k. in r7, do not need to compute the score for each candidate label individually. Instead, we can directly search for the maximal slope
along the lower convex hull that falls below the quantile ÿ; see Algorithm
(Split) Conformal Prediction; with Require: Pre-trained model: p. test point: X penalty: À > 0, (1 - a) 1/n)-th quantile of calibration
Algorithm SOCOP: Singleton-Optinng
Score
set nonconformity scores: 4-
Ensure: A prediction set ((X441) with coverage 1 1- a.
1: Sort f(- (x+1) to get (- (x+11) ) and associated lahels idx
2: (V,T,9) < Find lower convex hull using Algorithm 2 with input (Peccted (- xx+1);
3: kimal
4: for j to V 1 do
5: e_ <- Víj 1]: f- Vbi: Ti <- (go, go ) (T., r
6: 1f & q then kinnal <- v., else break from for loop
7: end for
8:
C(X.m1) + i [k]
k E kinna
9:
return C(X++1)
2.4
THE SCOPE OF OUR FRAMEWORK
In this section, we discuss certain important special cases and extensions of our methodology.
Our nonconformity score was derived starting from linear combination of the singleton probability and the expected length. Therefore, it would be reassuring to know that our solution can indeed provably interpolate between the two by recovering them in certain limiting cases. the next result, show that this is indeed true and that nonconformity score reduces to the corresponding nonconformity scores for these two cases. Recall below that we consider the labels to be sorted such that p(y1 (z) > p(V2(z) >
Corollary 2.6 (Recovery of singleton objective optimization and least ambiguous sets). (1) When A 0c, the nonconformity scores have the limit rus (x. Us) 1/)(y, The resulting split confor- mal prediction sets have the form ty E E ppy [n--)) 1) 2 c). some quantity recovering least ambiguous set-valued classifiers (Sadinle el al. 2019).
(2) When A = O, the nonconformity score becomes Tsingleton (I, = 1(i 2 2) - p(y1 | x)) xx)) The resulting split conformal prediction sets are either the top-1 label {y) } if p(yn (= 3 C, for some quantity c, or the whole set y otherwise.
The proof of Corollary 2.6 is provided in Appendix B The solution to the pure singleton objective has an intriguing structure. The prediction sets are either the top-label or the full-label set. This is intuitively reasonable: in the singleton objective, we are not paying any cost for the first label included in the set, so makes sense to always include the most confident label. Moreover, we are paying full cost for any additional label included, and thus to ensure coverage, it is reasonable to include all labels into the prediction set.
However, this dichotomous behavior may not provide enough granularity in practice and may often output large prediction sets. This motivates our approach of taking a linear combination between the singleton and the length objectives. Our empirical results demonstrate that the nonconformity

---

Published as a conference paper at ICLR 2026
scores derived from this linear combination offer a favorable trade-off, significantly reducing the probability of non-singletons compared, while only increasing the length by a little.
Extension to P(size > An). Beyond controlling the probability of non-singletons, in appli- cations might instead be more desirable to control the probability of sets larger than some other number, such as two, three, or ten. For instance, we might have two employees check one output each, and thus we might tolerate prediction sets of size two. Therefore, it is desirable to extend our framework control the probability Fx [[C()] ko] of sets size larger than ko E fl,...,, 1...
Fortunately, it turns out that our methods extend seamlessly to this case. We now seek to minimize Fx [x[IC(X)] > Ko] + X Ex [EC[((X)]] subject to the same coverage constraint. The corresponding Lagrangian and separability arguments proceed similarly, with the difference that in the problem ( ge in the cost function becomes gi I(k > kn) + Ak, k 0, ........A. K. The remaining steps are identical. The nonconformity score for the case À O from Corollary 2.6 becomes < (z, Ui)
I(fi ko))/(1 E x)). The corresponding sets consist of either the top ko indices or the full set.
3 EXPERIMENTS
In this section we report experiments comparing our SOCOP method with several prediction sets. The first one uses the probabilities output by the classifier directly, sorts them in decreasing order, and outputs the smallest set of classes whose predicted probabilities sum to at least 1 - a; we call this the P Plug-Ix sets We also report results with split conformal prediction sets using variety of nonconformity scores such as RAPS (Angelopoulos et al.) 2021); Pure Singleton on Sets (Sadinle al. 2019), corresponding to the nonconformity scord (z, y) + pLv z) which recovers the case À > Do in our method. We additionally evaluate the CPL method proposed by Kiyani et al. 2024). This approach employs the same nonconformity Set.s, but is conceptually different, as it replaces split conformal prediction with a training procedure optimize prediction set length, Results for this method are reported in Table 17 in Appendix D.3
(A 0); Least Ambiguous
score as Least
For our proposed SOCOP method, the hyperparameter is selected by aiming to find "knee point" of the size-singleton probability curve on the tuming subset, as detailed in Section 3.1.1 The evalu- ation metrics we use are Coverage, Average Size, and (size>1)
3.1 IMAGE CLASSIFICATION ON IMAGENET
First, we consider image classification on the ImageNer- Val and ImageNet-V2 datasets, with several models, including ResNet 152 v2, EfficientNet -22--eetttt22--- -2- and ViT-h-14,
Evaluation on ImageNet- Val. In this experiment, we randomly sample three subsets of ImageNet- Val over 100 trials: one tuning subset of size 10K, one conformal calibration subset of size 20K and one evaluation subset of size 20K. For the RAPS baseline, we employ the hyperparameter tuning method from Algorithm 4 of Angelloooss et al. (2021). The details of the hyperparameter grid are provided in Appendix F.
The averaged results for ResNet and along with standard errors, are reported and Swin-v2 are in Ap- pendix D.I and show similar trends. All methods achieve the target coverage of 0.95. Our method SOCOP outperforms Plug-Iu In and RAPS in both Average Size and P (si si>)).>.. Compared to Least Ambiguous Sets, SCCOP maintains a good balance: produces sets nearly as small as Least Ambiguous Sets while significantly reducing the probability of non-singletons.
in Table Results for Efficienti E.fficient.Net-v22-11 ConwNeet-baas
"This strategy is theoretically guaranteed attain the nominal level of coverage. However, it can be viewed as a reasonable empirically motivated baseline that practitioners might use by default.
"In Corollary 2.6 we wrote this non-cone conformity score 1/p(y|z); These are equivalent since any strictly monotone transformation of nonconformity score induces the same prediction sets.
'We also evaluated the empty set rate, P(C(()) = 0). across all experiments. We observed that empty sets occur in fewer than 0.01% % of test instances for all methods, with the exception of RAPS (where they can he slightly larger but still insignificant, reaching & 0.1 %).. Consequently, omit this metric from the results its impact negligible.

---

Published as a conference paper at ICLR 2026
Table 1:
Performance on ImageNet- -all, a Coverage of a 0.95; Methods compared: Plug-In.
RAPS (Angelopoulos et al. 2021), Pure Singleton (A 0). Least Ambiguous Set (A = oo) (Sadinle et ||]]] Kiyani et al. 2024) and our method SOCOP. Results are averages over 100 random splits. The smallest values in each column are highlighted in green, while all results worse than our method are highlighted in red. For method SOCOP, the Avg Size and P(size 1) highlighted in light green
facilitate comparison across models.
Model
Method
Coverage
Avg Size
P(size > 1)
? Plug-In
0,968 + 0.003
44.955 + 5.558
0,460 + 0.019
RAPS
0.950 + 0.002
3.158 = 0.101
0.603 + 0.154
ResNet152-v2
Pure Singleton
0.949 0.002
249.453 + 4.960
0.249 0.005
least Ambi Set.s
0.950 0.002
2.274 0.046
0.466 0.007
SOCOP (ours)
0.950 + 0.002
2.477 + 0.048
0.370 + 0.006
PLug-In In
0,976 + 0.001
8,529 1 0.805
0.356 0,008
RAPS
0.950 + 0.002
1.380 + 0.020
0.314 + 0.031
ViT-h-14
Pure Singleton
0.950 + 0.002
136. 236.219-4 + 4.980
0.135 + 0.005
Least Ambiguous Sets
0.950 + 0.002
1.291 + 0.011
0.224 0.006
SOCOP (ours)
0.950 0.002
1.356 1 0.017
0.175 0.006
Evaluation on ImageNet- V2. We apply the same evaluation pipeline to ImageNet- V2 (Recht et al., 2019). which more challenging test dataset. This dataset was constructed by re-collecting im- ages with new sampling pipeline, introducing a natural distribution shift that typically results in a significant drop in accuracy for models trained on the original ImageNet dataset. Since this is smaller dataset, we randomly sample three subsets of Imagenet- Val over 100 trials: one tuning subset of size IK, one conformal calibration subset of size 4K, and one evaluation subset of size 4K. Table I in Appendix D.2 reports the results for all five models. The empirical findings are consis- tent with those on ImageNet- Val. Notably, the advantages SOCOP are even more pronounced on this more challenging dataset. The variance of the coverage is higher due having less data.
3.1.1 EFFECT OF AND HYPERPARAMETER TUNING
SOCOP: Effect of
SOCOP Tradeoff
3.0
0.42
000
Tuned Point
0.45
0.40
0.50
4.5
0.38
030
0.40
Average
4.0
Pisize > 1) (SOCOP)
Plaze
0.36
0.20
3.5
(LAS)
0.35
0.34
Aug Size (SOCOP)
0.32
Aug Size (LAS)
0.30
0.30
0.03
0.02
2.5
0.28
0001
2.5
4.0
5.0
Average
Figure 2: Visualizing the evaluation results of ResNet 152-v2 on ImageNet- Val from Table LAS denotes Least Ambigueus Sets. Left: Average size and P(size > 1) varying with X; Right: Visualization of (Average size, P(size > 1)), each point corresponding rameter X selected by the kneedle algorithm (Satopaa et al. 2011) are highlighted.
specific X. Results corresponding the hyperpa-
Next, we study the effect of the regularization parameter on Average Size and (size>1). See Figure for the trade-offs on the ResNet 152-v2 model evaluated over ImageNet- Val. Results are averaged over 100 random splits of Imagenet-Vai each of size 20K for calibration and 20K for evaluation. As X goes from 0 to the Average Si ze decreases from the level of the Pure Singleton (A = 0) and converges to Least Amb Ambiguous guous Sets limit (A = oo); while (size>>1) follows an opposite trajectory.
The right panel of Figure summarizes this trade-off by plotting P (size>11 against Average Size. In practice, one can choose À according their own preference by drawing the tradeoff plot (the right panel of Figure?? 2 on their tuning dataset. For illustration, in our experiments from Section 3.1 choosing points along a trade-off curve that come with favorable trade-offs. All five models exhibit the same pattern on both ImageNet- Val and ImageNet- V2, Tables 16 in the Appendix.
the kneedle algorithm (Satopaa al. 2011), which is a popular method for
8

---

Published as a conference paper at ICLR 2026
0.1
À 1.0
SOCOP
SOCOP
LAS
LAS
à 0.0495
0.0089
Density
0.4
0.2
0.0
6
8
10
12
14
16
18
20
0
2
10 size
12
14
16
18
20
Set size
Figure 3: sizes produced with ResNet152-v2
ImageNet- Val. LAS denotes Least Ambi Ambiguou se Bars indicate empirical probabilities of set sizes, and shaded bins mark non-singlcton set sizes where SOCOF assigns higher mass. Reported A values denote the cumulative probability difference on shaded bins. The x-axis truncated at 20 clarity.
We investigate the effect of regularization in more detail. For two values of X, we collect the set sizes produced by SOCOP and Least Ambiguous Sets, and report their histograms in Figure 3 The figure shows that SOCOP yields more singleton sets and fewer small set sets (as desired), but produces slightly more sets with a large size sets (as expected due to the tradeoff). To quantify this shift toward larger sets, calculate the cumulative excess probability mass A of SOCOP
Least Ambiguous Sets on non-singleton sizes, i.e., A == Σ 1 ( (SSCCOP
fLAS), where SSSCOPP SLLS are empirical frequencies of prediction size for the methods. We observe that this value is small, meaning that our method only leads to slightly more large sets.
3.1.2 ADAPTIVENESS ON IMAGENET
In this experiment, we evaluate the size-stratified coverage violation (SSCV) introduced by An- gelopoulos et al. (202T) as measure of adaptiveness and conditional coverage violation. Following Angelopoulos et al. (202I), we adopt the same set-size strata 0-1, 2-3, 4- 10, 11-100, and 101- 1000, and to maximize adaptiveness, we choose the hyperparameter À minimize SSCV on the tuning set for RAPS and SOCOP. The details of the hyperparameter grid are provided Appendix The results reported in Table 2
Table 2: Evaluation results for the SSCV metric on ImageNet- Val, with the same protocol as in Table
Method
Coverage
P(size Avg Size
SSCV
ResNet 152-v2 52-v2
Plug-In RAPS
0.969 + 0.003
47.362 + 7.188
0.469 - 0.025
0.046 0.001
0.950 + 0.002
8.568 + 1.580
0.448 = 0.012
0.031 - 0.011
Pure Singleton
0.950 0.002
250.539 4.554
0.250 0.005
0.050 + 0.000
Least Ambiguous Sets
0.950 0.002
2.279
0.467 1 0.007
0.197
1 0.026
SOCOP (ours)
0.950 1 0.002
3.372 1 0.198
0.304 1 0.008
0.039 1 0.009
Vir-h-14
Plug-In
0.976 0.001
8.529 0.805
0.356 0.008
0.048 = 0.002
RAPS
0,950 + 0.002
7.652 + 2,259
0.319 = 0.007
0.047 0.003
Pure Singleton
0.950 + 0.003
136.219 4.980
0.135 + 0.005
0.050 + 0.000
Least Ambi guous Sets
0.950 + 0.002
1.291 + 0.011
0.224 + 0.006
0.126 + 0.119
SOCOP (ours)
0.950 0.002
1.519 + 0.068
0.155 0.006
0.041 + 0.016
Our method SOCOP and RAPS achieve the smallest SSCV among the methods compared. However, the average size of RAPS increases drastically (from se 3.2 to z 8.6 for Resnet.12-v2 and from 1.4 to 2 7.7 for (1T-h-14) 4), while our SOCOP method maintains reasonably small average size and a significantly lower non-singleton probability, demonstrating that SOCOP can achieve adaptivity without sacrificing efficiency.

---

Published as a conference paper at ICLR 2026
3.2 IMAGE CLASSIFICATION ON TISSUEMNIST
We further evaluate on a medical image classification problem. We use the TissueMNIST dataset, a subset of MedMNIST Yang et 2023), which contains microscopy images of human kidney cor- tex cells categorized into eight classes. We use ResNet-50 (224) model released by the dataset authors. We perform 100 random splits into 10K/15K/15K for tuning, calibration, and evaluation. The results are summarized in Table 3 As in previous experiments, we observe that our method can significantly reduce the non-singleton probability (by about 15%), while increasing the average size only slightly. This again validates the efficiency of our method.
Table 3: Evaluation results TisSueMNIST using ResNet- (224) Yang et al. 2023 with the same
protocol as in T
Method
Coverage
Avg Size
P(iize > 1)
Piug-In
0.973 0.002
3.294 0.030
0.866 0,005
RAPS
0.950 + 0.003
2.844 + 0.031
0.844 0,006
Pure Singletor
0,950 + 0.003
4.931 +: 0.053
0.562 1: 0.008
Least Ambiguous Sets
0.950 = 0.003
2.647 + 0.028
0.788 + 0.005
SOCOP (ours)
0.950 0.003
2.847 0.037
0.638 0.009
3.3 MULTIPLE CHOICE QUESTION ANSWERING
We also evaluate on MMLU (Hendrycks et al. 2021). multiple-choice question answering dataset. Following the same evaluation pipeline, we perform 100 random splits into 4K/5K/5K for tun- ing, calibration, and evaluation. We use 3.1ma-3.11 3.1-BBBBBBBBBBBB ------nnsttt (Dubey al. (024). and following Kiyani et al. (2024), we input the fixed prompt: "This is a 4-choice question that you should answer: {question) (coocee) The correct answer to this question is: ". We then extract the logits of the first output token corresponding to the answer options A, B, C, and D. Applying the softmax function yields probabilities over the four choices. The results are summarized in Table 4 As in previous experiments, we observe that our method can reduce the probability that the set size is greater than one by significant amount (about 10%), while only increasing the average size by negligible amount. This further reinforces that our approach provides favorable trade-off between size and non-singleton probability.
Table 4: Evaluation on MMLU using amamm-33 1-81118881111 ruct, with the same protocol as in Table
Method
Coverage
Avg Size
P(size 1)
Plug-In
0.965 + 0.002
2.648 + 0.013
0.745 + 0.005
RAPS
0.950 0.004
2.601 0.032
0.779 = 0.025
Pure Singleton
0.950 0.004
2.633 + 0.029
0.544 = 0.010
Least Ambiguous Sets
0.950 + 0.004
2.426 +: 0.030
0.675 + 0.008
SOCOP (ours)
0.950 + 0.004
2.177 + 0.034
0.587 + 0.016
3.4 DISCUSSION
SOCOP reframes efficiency in conformal classification around the goal of producing singletons, de- riving nonconformity score from geometric analysis of Lagrangian relaxation of the singleton objective. This yields an O(K) per-instance algorithm, enabling split conformal sets that preserve marginal coverage while substantially increasing singleton frequency with minimal impact aver- age size. Empirically, over image classification and LLM multiple-chote benchmarks, this reduces non-singleton rates significantly relative suggesting that our method could be broadly useful in practice.
-ntthh-pttmmzed baselines near-identical set sizes;
In future work, it would be of interest to extend this method to more advanced conformal pre- diction methods, such as label-conditional or Mondrian conformal prediction (Vovk et al.) 2005). Furthermore, a challenging but interesting theoretical direction is to design a nonconformity score intrinsically targeted for conditional coverage. This would likely entail retracing the derivation of SOCOP starting from a conditional-awan (2025). Finally, while current protocol uses separate tuning set maintain validity, future work could investigate data-dependent selection of À using the calibration set directly to improve data efficiency, while accounting for the resulting tuning bias (Zeng al. 2025
optimization objective, such as the one from Gibbs et al.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
All experimental details, including dataset information and evaluation protocols, are provided in Section and appendix D A GitHub repository, containing the implementation of SOCOP, baseline methods, and code to reproduce all experiments, is available thit repository All theoretical results and assumptions stated in Section 2.1 with complete proofs provided in Appendix B
ACKNOWLEDGGMETS
This work was partially supported by the NSF, ARO, AFOSR, ONR, and the Sloan Foundation.
LLM USAGE
LLMs did not play a significant role in this work and were only used for grammar polishing in writing.
REFERENCES
Alex M Andrew. Another efficient algorithm for convex hulls in two dimensions. Information processing letters, 9())216-2119 1979.
Anastasios N Angelopoulos and Stephen Bates. Conformal prediction: A gentle introduction. Foun- dations and Trends in Machine Learning, 16(4):494-59) 2023.
Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan, and Jitendra Malik. Uncertainty sets for image classifiers using conformal prediction. In International Conference on Learning Representations, 2021. URL http::t:/pteeee//////// http::/pppeepeeeeee.eeeererrreea http://oormmmmmdddiiia BDMM DbM9
Meshi Bashari, Roy Maor Lotan, Yonghoon Lee, Edgar Dobriban, and Yaniv Romano. Synthetic- powered predictive inference. arXiv preprint arXiv:2505. 13:52 2025.
Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan. Distribution-free, risk-controlling prediction sets. Journal of the ACM (JACM), 68(6))1-33, 2021.
Arash Behboodi, Alvaro HC Correia, Fabio Valerio Massoli, and Christos Louizos. Fundamental bounds efficiency-confidence trade-off for transductive conformal prediction. arXiv preprint arXiv:2509.04633 046331 2025.
Sacha Braun, Liviu Aolaritei, Michael Jordan, and Francis Bach. Minimum volume conformal sets for multivariate regression. arXiv preprint arXiv:2503. 0ate:25333.966. 19068, 2025.
Kwan Ho Ryan Chan, Yuyan Ge, Edgar Dobriban, Hamed Hassani. and René Vidal. Con- formal information pursuit for interactively guiding large language models. arXiv preprint arXiv:2507.03279, 2025.
Rafael Correa, Abderrahim Hantoute, and Marco A López. Fundamentals of convex analysis and optimization. Springer, 2023.
Edgar Dobriban. Statistical methods in generative ai. arXiv preprint arXiv:2509.07050 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
A Gammerman, V Vovk, and V Vapnik. Learning by transduction. In Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence, 1998.
Scymour Geisser. Predictive inference: an introduction. Chapman and Hall/CRC, 2017.
Isaac Gibbs, John Cherian, and Emmanuel Candès. Conformal prediction with conditional guar- antees. Journal of the Royal Statistical Society Series B: Statistical Methodology, pp. gkaf008, 2025.
11

---

Published as a conference paper at ICLR 2026
Chirag Gupta, Arun K Kuchibhotla, and Aaditya Ramdas. Nested conformal prediction and quantile out-of-bag ensemble methods. Pattern Recognition, 127:108496, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- lional Conference on Learning Represenlations (ICLR), 2021.
Eliahu Horwitz and Yedid Hoshen. Conffusion: Confidence intervals for diffusion models, arXiv preprint 1tti::21110099990 2022.
Sunay Joshi, Shayan Kiyani, George Pappas, Edgar Dobriban, and Hamed Hassani. Conformal in- ference under high-dimensional covariate shifts via likelihood-ratio regularization. arXiv preprint arXiv:2502.11030. 2025.
Shayan Kiyani, George Pappas, and Hamed Hassani. Length optimization in conformal prediction. Advances in Neural Information Processing Systems, 37:99519-99563, 2024.
Batiste Le Bars and Pierre Humbert. On volume minimization conformal regression. In Forty-
second International Conference Machine Learning, 2025. UtL/tttts::/opperrrrri
feet/forummidd=22lnnne fdinwwnDDD
Yonghoon Lee, Edgar Dobriban, and Eric Tchetgen Tchetgen. Conditional predictive inference for 04613
missing outcomes, 2025. URL
Jing Lei and Larry Wasserman. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Statistical Society: Series (Statistical Methodology). 76(1):71-99) 2014.
Jing Lei, James Robins, and Larry Wasserman. Distribution-fiet prediction sets. Journal of the American Statistical Association, 108(501):278-287, 2013.
Jing Lei, Max Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution- free predictive inference for regression. Journal of the American Statistical Association, 113 1524):109411111, 2018.
Ruiting Liang, Wanrong Zhu, and Rina Foygel Barber. Conformal prediction after efficiency- oriented model selection. arXiv preprint arXiv:2408. 07066, 2024.
Ruiting Liang, Wanrong Zhu, and Rina Foygel Barber. Conformal prediction after data-dependent model selection, 2025. hnttpssss////rttt http://axxivv..or///////////// 07066
Ziyi Liang. Yanfci Zhou, and Matteo Sesia. Conformal inference is (almost) free for neural networks trained with early stopping. In International Conference on Machine Learning, pp. 208 10-20555. PMLR, 2023.
Christopher Mohri and Tatsunori Hashimoto. Language models with conformal factuality guaran- tees. In Forty-first International Conference on Machine Learning, 2024.
Joseph o' Rourke. Computational geometry in C. Cambridge university press, 1998.
Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence machines for regression. In Machine learning: ECML 2002: 13th European conference ma- chine learning Helsinki, Finland, August 19-23, 2002 proceedings 13, pp. 345-356. Springer, 2002.
Sangdon Park, Edgar Dobriban, Insup Lee, and Osbert Bastani. PAC prediction sets under covariate shift. In International Conference on Learning Representations, 2022a.
Sangdon Park, Edgar Dobriban, Insup Lee, and Osbert Bastani. PAC prediction sets for meta- learning. In Advances in Neural Information Processing Systems, 2022b.
Hongxiang Qiu, Edgar Dobriban, and Eric Tchetgen Tchetgen. Prediction adaptive unknown covariate shift. Journal of the Royal Statistical Society Series B: Statistical Methodology, pp- qkad069, (07 2023.
12

---

Published as a conference paper at ICLR 2026
Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S Jaakkola, and Regina Barzilay. Conformal language modeling. In The Twelfth International Conference on Learning Representations, 2024.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp. 5389-538-00 54000 PMLR, 2019.
Tyrrell Rockafellar. Convex Analysis. Princeton University Press, 1997.
Yaniv Romano, Mattco Sesia, and Emmanuel Candes. Classification with valid and adaptive cover- age. Advances neural information processing systems, 33:3581-3591, 2020.
Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association, 114(525):223-2355 2224, 2019.
Ville Satopaa, Jeannie Albrecht, David Irwin, and Barath Raghavan. Finding a "kneedle" in
haystack: Detecting knee points in system behavior. In 2011 31st international conference on distributed computing systems workshops, pp. 166-171. IEEE, 2011.
Craig Saunders, Alexander Gammerman, and Volodya Vovk. Transduction with confidence and credibility. In IJCAI, 1999.
Henry Scheffe and John W Tukey, Non-parametric estimation. I. Validation of order statisties. The Annals of Mathematical Statistics, 16(2):187-192, 1945.
Matteo Sesia, Stefano Favaro, and Edgar Dobriban. Conformal frequency estimation using discrete sketched data with coverage for distinct queries. Journal of Machine Learning Reseurch, 24(348): 1-80, 2023.
Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(Mar)3371-22, 2008
Yuanjie Shi, Hooman Shahrokhi. Xuesong Jia, Xiongzhi Chen, Janardhan Rao Doppa, and Yan Yan. Direct prediction set minimization via bilevel conformal classifier training. arXiv preprint (rXiiv2506.06599 2025.
Wenwen Si, Sangdon Park, Insup Lee, Edgar Dobriban, and Osbert Bastani. PAC prediction sets under label shift. International Conference on Learning Representations, 2024.
David Stutz, Krishnamurthy Dj Dvijotham, Ali Taylan Cemgil, and Amaud Doucet. Learning opti- mal conformal classifiers. In International Conference on Learning Representations, 2022. URL
https::/ httpeeewwwwwwweeeeeeeeee
Kei Takeuchi. Contributions theory mathematical statistics. Springer, 2020.
Jacopo Teneggi, Matthew Tivnan, Web Stayman, and Jeremias Sulam. How to trust your diffusion model: A convex optimization approach to conformal risk control. In International Conference on Machine Learning. pp. 11940-33960) PMLR, 2023.
John W Tukey. Non-parametric estimation II. Statistically equivalent blocks and tolerance regions- the continuous case. The Annals of Mathematical Statistics, 18(4):529-539 1947.
John W Tukey. Nonparametric estimation, III. Statistically equivalent blocks and multivariate toler- ance regions-the discontinuous case. The Annals of Mathematical Statistics, 19()):0033, 1948.
Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian Conference Machine Learning, 2013.
Vladimir Vovk. Cross-conformal predictors. Annals of Mathematics and Artificial Intelligence, 74 (1):9-28, 2015.
Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic learning in a random world. Springer Science & Business Media, 2005.
13

---

Published as a conference paper at ICLR 2026
Volodya Vovk, Alexander Gammerman, and Craig Saunders. Machine-learning applications of al- gorithmic randomness. In International Conference on Machine Learning. 1999.
Abraham Wald. An extension of wilks' method for setting tolerance limits. The Annals of Mathe- matical Statistics, 14(1):45-55, 1943.
S.S. S. Wilks. Determination of sample sizes for setting tolerance limits. The Annals of Mathematical Statistics, 12(1):91-96, 1941.
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
Yachong Yang and Arun Kumar Kuchibhotla. Selection and aggregation of conformal prediction sets. Journal of the American Statistical Association, 120(549):435 447, 2025.
Hao Zeng, Kangdao Liu, Bingyi Jing, and Hongxin Wei. Parametric scaling law of tuning bias in conformal prediction. arXiv preprint arXiv:2502.03003, 2025.
A ADDITIONAL RELATED WORK
The non-parametric techniques which have been studied in conformal prediction belong much broader tradition of predictive inference in statistics which over the years have been developed both under parametric and canaaaamettic assumptions. See for instance Geisser (2017) and more recent
works such as Bates et al. (2021) Park et al. 2022a b): Sesia et al.
(2023) Qiu et al. (2023); (Si et al.
(2024) Lee al. (2025); Bashari al. (2025) Joshi et al. (2025). which concern problems under
variety of assumptions.
Regarding the efficiency and optimality of conformal prediction, early work by Takeuchi in the 1970s-reviewed in [Takeuchi] (2020)-has established fundamental results, such as the fact that conformal prediction with conformity score equal to a particular density f is optimal-in terms of minimizing the expected length at the distribution with density f-among all methods of predictive inference that have marginal coverage over all distributions. Modern work has revisited optimality questions from variety of different angles, discussed in the main paper.
Conformal-tyre type techniques have been developed to be used beyond standard classification and re- gression problems, for instance in sampling from large semantic spaces with generative AI models, see e.g., Horwitz & Hoshen (2022); Teneggi et al. (2023): Quach et al. (2024): Mohri & Hashimoto
(2024): Chan et al. (2025), etc; and see Dobriban (2025) for review. In work, we provide
illustration for language model multiple-choice question answering, which becomes conventional classification problem.
Recent frameworks (Liang et al. 2024 Yang & Kuchibhotla 2025) optimize efficiency by selecting
a nonconformity score from pre-specified candidate set that minimizes target loss. This differs fundamentally from our approach, which uses the loss to derive the score directly. Applying our composite loss within these selection frameworks faces two practical difficulties: (1) the resulting performance is strictly bounded by the pre-defined candidate pool; and (2) this would introduce the additional difficulty of selecting the hyperparameter X, and their framework would have to be potentially extended to allow the selection of not just non-conformity scores but also loss functions.
B AUXILIARY RESULTS AND PROOFS
B. PROOF OF LEMMA 2.1
Proof. Clearly, Su,- = 1 Fu. Now let 1) > O. If S is empty, then the claim holds; therefore, only need to consider the case where Sn is non-empty. Assume Sn,y includes yi, but Viz where T9:2 > Yo: Then we can construct S" = (Snr' (3r,) U [Wig} such that
(S') = (S +n Σ 7v
Σ yv (S + (7wa, (wiz) << E(Sn.in)) DES"
yES
14

---

Published as a conference paper at ICLR 2026
which contradicts with the optimality of S.y Therefore, Sn, must be the set of top-j labels for some j that depends on 17 and 7
B.2 PROOF OF LEMMA 2.2
Proof. For any set S [K], we define g(8) - I([S] > 1) + X|S) and -(S) -yes T9- We will first prove that for v/ < 12, (( S (()))))) Then by Lemma we must have Sm.7 Smar
By the optimality of each set, we have
(m m.) < m m., i.., g(Sm))) m1 (S & g(Sman) 171 I (Smam))
. E((mm.) & (S 12), i.c., (S 72 (S & g(Smin) 1/2 I [()
Combining these inequalities gives
(T r))m.))) < g(Sm))) gSS (m,)))) < m (T(Sm.)) I(Sma.))
Since 7/2 7 O, have T [()m))) ( (p.>) & 0. By Lemma 2.1 for any and E A K-11
Snyy is the set of top-j labels for some j. Then ((Sm))) $ (((m) implies that Sm C Sma.r- L
B.3 PROOF OF THEOREM 2.5
To simplify notation, let us denote e (1) = (F(F:: 9s k- Then the minimization
problem in (4) can be rewritten <( 7) = arg min: e (1)) The optimal value of the objective is given by
f = min (T)-
ugk
This problem can be analyzed from two viewpoints, also Figure
Dual Space L) For each l E: {0,... (O...., K}. function of 7) for (n, () E 2². For a fixed value 7, the optimal value (n)) corresponds to finding the lowest point among the intersections of the K + lines with the vertical line = 17. As shown in the left plot of Figure 4 the function Ex forms the lower envelope of this family of K + 1 lines. The vertices of this lower envelope correspond to the values of 1) where the optimal index <(7; 7) transitions from one value to another.
view e (n) gh. nI'ss linear
Primal Space (T, g): Since Os = nTT a + el))
(n), e(*t (v) can be viewed as the intercept of line with slope 77 that passes through the point Ps (I',, kx)... For a fixed n, in the space of (T,g), {g = Ty[ + e, E IR] family of parallel lines. Minimizing ( (v) over amounts to finding the first point in (Po... Px} } that is "hit" by such a line the intercept raises from -O.
Mathematically, the duality between these two perspectives can be formalized using convex conju- gacy. Define primal function Φ: 1] + [0, oo] based on the point set P {P...... Pk
ifT A OSkSK
(TT)
+oo
otherwise.
The convex conjugate (see e.g., Rockafellar 1997 of & is
sup o(T)) max {nI'x - 9gk E,,
(6)
FER
OSkE
Furthermore, the biconjugate of defined the conjugate of is
(T)
sup
{1ī - (n)) sup {np (n))
HER
7E
By the Mocehee--MoauuuRoode the closed convex hull of the original function d: 6= = co( where co(o) denotes the closed convex hull of d. Thus, suffices characterize co((), which will do through its epigraph.
theorem (see e.g. Theorem 3.2.2 in Correa et al. 2023 dt
15

---

Published as a conference paper at ICLR 2026
Dual Space (n. Lower of Lines
Primal Space (G.g): Lower Convex Hull Points
2.5
2.00
Point Pg = (T)
P% 14.7
Lower Convex Hull of
Pa
P 3.7
76.1
0.0
0.5
332
2.2
Pr
Po.
f1s
-2.5
3.7
1.50
1.25
(n
kn
0.1.00
01.7
(n)
0.75
(n)
(n))
0..10 10 14.7
0.25
15.0
Inactive lines
P
Lower Enwelope (7()))
0.00
15.0
0.0
Figure 4:
Primal and dual views, for À 0.1 and the example probability vector is 0.077, 0.057, 0.081, 0.027, 0.007].
Let epi(o) denote the epigraph of c, defined epi(o) U ((Tk,w) w 2 ge Using that epi(co(()) = co(epi(o)), any point (I, co(o) can be expressed as:
(T,w) Σ k=1
3R wx) for some Bx 2 O. Σ 8x and (Tk,wk) E epi(o).
K Since DK Bkwu Ek=l Bagk. and the minimum is attained, we know that
K r
K Σ
x²T(T) inf Σ
BaDk
BR 2 0, Σ Bx =1]
(8)
k=1
which is precisely the lower convex hull of the point P.
We now continue with the proof of Theorem 2.5
the set of optimal solutions for some no (ie., 6.9 (no) if and only if its corresponding point Ps lies on the graph of the lower convex hull function o= (T)
Lemma B.1. An index
(i.e., = 60 (T))).
Proof. ( (-) ) Assume E( (no) (16). By definition, this implies 9)- 1ol'' =1 (Ta). From the biconjugate (*x(Tx) sup,, nn + E, (n)) T&mo + (m))) = 9k- Since 6t* the lower convex hull function of P. we must also have 6² (T&)) gk- Therefore, Ik (Txx).
(+) Assume that the point Ps = (T's.t)) lies on the graph of the lower convex hull, i.e., Th =0*=(Tx), By the Supporting Hyperplane Theorem (Rockafellar) 1997). Pa being on the lower boundary of convex hull implies that there exists supporting line to the function 6** at the point Tg. Let the slope of this supporting line be пe. Then for all r in the domain, we have 6 (T) (T=( + Th (T - Tx).
For any j E (O...., K). P3 = (T),9;) must lie on or above the lower convex hull, i.e., 9j > (T))) Applying this to the inequality above for T = J we find:
の0 (T₁) > (T&) + (T, - -T))
By our initial assumption 8² (Tx) ge. then have 9j > 9h + TI (T Tx). This inequality ......,,,, j E (0...., K). Therefore, e( (mx) C, (m²) i.e., is optimal index for n Tk-
holds for all j E .O...., K}. Thus. g; - mel; > gs Ix. that is, c04) (Tx) > g(4) (m) for all
n
16

---

Published as a conference paper at ICLR 2026
Recall that the vertices of the lower convex hull of P are (Pre- Por..... P. Pem. where 0 2u V1 < ..... < Um = are indices from (0.....K), Recall from Section 2.3 that for i = 1..... m,
5v,, 0---1 where we define the slope of the edge connecting vertex Po._ and P. is defined 7/2 Tv, -
No 0 and 7m +o0. From the definition of convexity, it follows that these slopes are strictly increasing:
? < 112 < ... < 7m < xc. Our next result is the following:
Lemma B.2 (Unique Optimality Vertex Intervals). For any yj E (Mi 7e11), we have elai
( (i))fo for all k + 0;.
Proof. Let 7 E (%, T441) ) for a given E f0. m 1}. We need to show that 9k Io, > nT' ). for any k # U4- By Lemma B.1 any point not the lower convex hull cannot be optimal, so it suffices to check this for other vertices P., where + i.
Case 1: j i (i.e.,) n, r.;). Since 6tx is convex, for any j i, have
904+
Dv.
9v, 9e,
M7+1
Fr
T
To
By our choice of 17, we have 17 < 7-+1- Combining these gives 17 < have vT(T,, v, - (o:)) < In; - Io. so that (l (11 < e)))))))
gy; Bu, Tv,-Ie,
As T - -r, > 0, we
Case 2: j < i (i.e., F, < v. Similarly, for any j i, by the convexity of o²", we have:
go: Ous
Dv. -v--1 1/6-
T. To
T,, r P
which implies elrr) (n) < (n) as above.
This finishes the proof.
Proof of Theorem 2.5 From Lemma B.2 for any v in the open interval (M); ()-)), the unique min-
imizer is Us. so x(1)7) = V;. At boundary points v) T for i .l....,m), we have ( (TK) = ( (T)) by definition. The proofs in Lemma B.2 show that for any other vertex Ugs (l (m.) is strictly greater. Thus, the set of optimal indices is {u--1, v. By the tie-breaking rule we have x(mi?) = U1-1-
Combining these observations, for i 1,..., 1, and for any 1) E (m, 1)). the optimal index is x(m 7) = U, is clear that rc(0; = Therefore, for E [0, mı]. x(m))) By Lemma B.2 for 17 (7m,, 1m++1) = (Ams 00), k(T) 7) Um. This finishes the proof.
B.4 PROOF OF COROLLARY 2.6
Let p(-(x) be the probability prediction from some pre-trained model. We order the probabilities such that p<)1/=) 2 p())2=) 2 p(yx (x))
(1) When dividing the Lagrangian [1] by A, the problem remains equivalent by changing variables from to 7 = n/ The resulting k(Π p(-[x)) becomes:
1(k > 1)
+ - 7j Σ piiules)
x(f:(-1xx) = arg min DSASK
X
b(u)(1)) 1(k21] + +k). We first con- sider the simpler case where p(y)(a) p(aaaa)) > ... p(yx(z). When X + J0, Pe
The corresponding set of points P becomes Pe = -t
E bkul) Pt. The slope between two consecutive points is: 1/6(s441/x), which is
10hhen there multiple solutions, we choose any set that has minimal size, which corresponds to choosing the smallest index for x(7) 7)
17

---

Published as a conference paper at ICLR 2026
Ble-Breaking [offl-Contiuous]
Tie-Breaking (Right-Continous)
10
14.7
14.7
V)
2.2
2.2
0.5
11
- 0.
0
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Figure 5: Optimal Set Size x(m; 7) with the same parameters as Figure
The tic-breaking rule does not affect
the value of nonconformity score.
strictly increasing in Hence, every point Pa is vertex of the lower convex hull. Thus x(īi: p(-|z)) becomes:
0g for r) E
Pla1 20)
R -(;;((-1x))
i for E
IEiEK- K-1
p(ulzj) Plu:+1
K, for E
[[uxc [=]
Therefore, the nonconformity score is Pas (x, vi) = 1/p(vs Ir).
Now, suppose there is tie, e.g., (yk x) = (yk+1 2))
(yk+m x)) for some 2 1, m 2 Then, the points Pk-1, Pk...... Pa+m are collinear, with vertices PA-1 and Pa+m and slope "( The function x(::((-1z)) exhibits single jump from k - 1 to + as î crosses this
slope value. For any label yı with l: i < ks m, the nonconformity score is
Mas (z) in 0: x p (x)) i} (yb x) 1/6 (y4 x),
as desired.
(2) When 0, the of points (Po..... Pk3 becomes
Pg = 0), Pı = (p (yn z), Ps = [ Emull)))))) (k 2), Pk = (1,
As & is strictly increasing in k. the vertices of the lower convex hull are (Po. P1. Px). The corre- 0-0
sponding slopes are 171 Flyl
and
9K
91
1 1 ( x))
122 =
Σ K
p())(z) p(y1(z)
Hence, x(m;((1xx) becomes:
0. for 0
1, for E
0,
((mp((-1z) =
P (y/ x)
K, for E
11 p (v1 x)) 20
Therefore, by definition of the nonconformity score r(z, 9:) = inf in 2 0: (=mm > (1). we
have singletoon (zr, y:) T(i 2 (1 p(mn x)) -1
18

---

Published as a conference paper at ICLR 2026
C ADDITIONAL ALGORITHMS
We leverage the monotone chain algorithm (Andrew lower convex hull, as detailed in Algorithm
1979: Rourke 1998) to find the vertices of
Algorithm Compute Lower Convex Hull via Monotone Chain Algorithm
Require: Sorted probability vector p. penalty O.
Ensure: A tuple (V, (. g) where is the list of vertex indices of lower convex hull, T are cumulative sums, and are objective values.
1: Compute cumulative sums F& <- - Pj for k O...., K
So - 0
2: Compute objective values gk 1(k > + Ak for k ...... K
3: Define CrossProduct(j,, i, k; g)= (I'; (g) 9e) - (g: -9: 9T' -T))
4: Initialize empty list of indices V.
5: for = to K do
D Monotone Chain
6: while [V] > 2 and CrossProduct(|1122/) V[- k, do
Last points
7: Remove the last index from V.
8: end while
9: Append index k to V.
end for
11:
return (V,
For calibration, we adopt standard split conformal prediction, see Algorithm 3
Algorithm SOCOP Conformal Calibration
Require: Pre-trained model ý. calibration data {(X,, Y)) level a E (0, 1), penalty À >> 0.
Ensure: Calibrated threshold q.
1: for i= to n do
2: Sort p(-|XX,) get Psonted (-omee(-(X.)) (X,,)
3: Let frunk be the 1-based rank of the true label Y;
4:
(V,T.g) < Algorithm] 2 Puwrted (X))
5:
Find the smallest index j E {1,
[| 1} such that Vij] > Lank
6:
V_ 1 - 1]; 04 4- V[i
Ti (g+, 9o_) (Tu+ To.)
8: end for
9: / < the [(1 a1 + n)] largest value in {ri)
10: return ;
D ADDITIONAL EXPERIMENTS RESULTS
D.1
IMAGENET- VAL
Results for EfficientNet-v22 -, ConvNeXt base, and Swin-v2 on the ImageNet-Val dataset are reported in T
For this dataset, the effect of À our SOCOP across all five models are reported in Table respectively.
I0
D.2 IMAGENET-V2
Results for all five models the ImageNet- V2 dataset are reported in Table II]
For this dataset, the effect of À on our SOCOP across all five models are reported Table 12 16 respectively.
19

---

Published as a conference paper at ICLR 2026
Table 5: Continuation of the results in Table 1 with the same protocol used.
Model
Mothod
Coverage
Avg
P(size
Plug Io
0.970 + 0.002
16.606 + 2.023
0.401 0.013
RAPS
0.950 = 0.002
1.909 = 0.077
0.769 + 0.076
LfficientNet-v22
Pure Singleton
0.950 1: 0.002
188.942 4 4.836
0.188 0.005
Least Ambiguous Sets
0.950 0.002
1.512
0.329 0.007
80C0P (ours)
0.950 + 0.002
1.659 + 0.023
0.262 + 0.006
Plug In
0.967 = 0.003
27.137 + 5.790
0,441 + 0.030
RAPS
0.950 +: 0.003
2.546 0.096
0.843 0.234
Xonviictt
Singletin
0,950 1: 0.002
226.991
4.935
0.226 0.005
Teast Ambiguous Sets
0.950 + 0.002
1.807 + 0.034
0.398 + 0.007
SOCOP (ours)
0.950 - 0.002
2.086
0.046
0.316 0.007
Plug-Tn
0.968 + 0.003
19.646 + 3.701
0.423 0.023
RAPS
0,950 1: 0.002
2.314 0.062
0.477 0.121
Swin-v2-o
Pure S1ng letoo
0.950 + 0.002
225.685 1 4. 4.909
0.225 + 0.005
Least Ambiguous Secs
0.950 0,002
1.881 0.033
0.396 0.007
SOC0P (ours)
0.950 + 0.002
2.068 = 0.038
0,316 + 0.007
Table 6: Performance of reaNett152-25 on ImageNet-Val with different values (a 0.05). Results are averaged over 100 data splits.
Method
Coverage
Avg Size
P(size > 1)
Puze Singletin
0.949 + 0.002
249.453
+ 4.960
0.249 + 0.005
0.01
SOCOP (ouns)
0.950 1: 0.002
5.078
1 0,200
0.279 1: 0.005
0.02
S0C0P (ours)
0.950 0.002
3.932
0.125
0.293 1 0.005
0.03
80C0P (ours)
0.950 + 0.002
3.508 = 0.103
0.302 + 0.006
8000P (ours)
0.950 0.002
3.267
+ 0.006
0,05
SOCOP (ours)
0.950 1: 0.002
3.110 1 0.092
0.315 1 0.006
S0C0P (ouns)
0.950 0.002
3.002
0.081
0.321 1 0.006
0.07
SUCOP (ours)
0.950 + 0.002
2.916 + U.074
0.325 + 0.006
0.08
S0COP (ouns)
0.950 1 0.002
2.847 0.071
0.329 1 0.006
0.09
8000P (ours)
0.950 0.002
2.795 0.069
0.332 + 0.006
0.10
80C0P (ours)
0.950 + 0.002
2.719 = 0.068
0.336 + 0.006
S000P (ours)
+
0.30
SOCOP (ours)
0.949 1 0.002
2.461 1 0.058
0.376 1 0.006
8000P (ours)
0.950 0.002
2.430 0.059
0.388 + 0.007
0.50
SOCOP (ouns
0.950 1 0.002
2.406
0.054
0.396 1 0.006
0.60
S0COP (ouns)
0.950 1: 0.002
2.388 1 0.051
0.403 1 0.006
0.70
8000P (ours)
0.950 0.002
2.373 0.051
0.408 + 0.006
0,80
80C0P (ours)
0.950 + 0.002
2.361 = 0.051
0.112 + 0.006
SUDOP (ouns)
0.002
2.355
0.416 1
1.00
80COP (ours)
0.950 1: 0.002
2.349 1 0.054
0.419 1 0.007
00
Least
Ambi guous Set
0.950 0.002
2.274
0.046
0.466 + 0.007
Table 7: Performance of EifieiiettNee--2211 ImageNet-Val Val with different À values 0.05).
Results are averaged over 100 data splits.
Method
Coverage
Avg Size
P(siz >> 1)
Puze Singleton
0.950 + 0.002
188.9 + 4.444.00 4.836
0.188 + 0.005
30C0P (ours)
0.950 0.002
2.873
0.091
0.205 0.005
0.02
SUCOP (ouss
0.950 1: 0.002
2.326
1 0.060
0.212 1 0.005
S0C0P (oues)
0.950 0.002
2.122
0.053
1: 0.006
0.04
SOCOP (ours)
0.950 : 0.002
2.012 + U.042
0.221 + 0.005
0.05
S0C0P (ouns)
0.950 1 0.002
1.946
1 0.038
0.226 1 0.005
0.06
8000P (ours)
0.950 0.002
1.808
0.035
0.229 1 0.005
0.07
80C0P (ours)
0.950 + 0.002
1.859 = 0.034
0.232 + 0.006
3000P (ours)
0.09
SOCOP (ours)
0.950 1 0.002
1.802 1 0.031
0.236 1 0.006
S0C0P (ours)
0.950 0.002
1.782 0.028
0.238 + 0.005
0.20
80C0P (ours)
0.950 + 0.002
1.078 = 0.023
0.255 + 0.005
0.30
30COP (ours)
0.950 1: 0.002
1 1.640 1 0.022
0.265 1 0.006
0.40
8000P (ours)
0.950 0.002
1.621 0.021
0.274 + 0.006
0.50
80C0P (ours)
0.950 + 0.002
1.608 + 0.023
0.279 + 0.006
SU00R (ours)
0.002
1
0.70
S0C0P (ours)
0.950 1 0.002
1.887 1 0.022
0.286 1 0.007
8000P (ours)
0.950 0.002
1.581 0.022
0.289 + 0.006
0.90
80C0P (ours)
0.950 + 0.002
1.575 0.022
0.291 + 0.007
1.00
30COP (ouns)
0.950 1: 0.002
1,571 + 0.022
0.293 4 0.007
Least
Ambi Subug SeLs
0.950 0.002
1.522 0.018
0.329 - 0.007
20

---

Published as a conference paper at ICLR 2026
Table 8: Performance of ConvNeXt-baase on ImageNet-Val with different À values (a = 0.05). Results are averaged over 100 data splits.
Method
Coverage
Avg Size
P(size
Pure: Singlaton
0.950 0.002
226.901 + 4.935
0.226 0.005
0.01
30C0P (ours)
0.950 + 0.002
4.074 + 0.155
0.244 + 0.005
0.02
SOC0P (ours)
0.950 + 0.002
3.194
0.097
0.254 0.005
0.03
SOCOP (ours)
0.950 1: 0.002
2.866
0.081
0.262 0.005
0.04
80C0P (ours)
0,950 + 0.002
2.683 + 0.068
0.268 + 0.005
0.05
8OCOP (ours)
0.950 0.002
2.567 0.067
0.274 0.005
0.06
80COP (ours)
0.950 +: 0.002
2.485 1: 0.065
0.278 1: 0.006
0.07
SOCOP (ours)
0.950 0.002
2.429 0.062
0.283 0.006
0.08
80C0P (ours)
0.950 + 0.002
2.383 0.063
0.287 + 0.006
0.09
SOCOP (ours)
0.950 + 0.002
2.344 0.060
0.290 + 0.006
0.10
SOCOP (ours)
0.949 1 0.002
2.307 0.056
0.293 0.006
0.20
80C0P (ours)
0.950 + 0.002
2.121 + 0.045
0.310 + 0.006
0.30
80COP (ours)
0.950 0.002
2.044 0.036
0.321 0.006
0.40
8000P (owrs)
0.950 + 0.002
2.008 0.036
0.330 + 0.006
0.50
SOCOP (ours)
0.950 + 0. 0.002
1.983 0.035
0.336 0. 0.006
0.60
8OCOP (ours)
0.950 + 0.002
1.961 0,033
0.341 0.005
0.70
8OCOP (ours)
0.950 1: 0.002
1.951 0.035
0.345 1 0.006
0.80
S0COP (owrs)
0.950 0.002
1.943 0.035
0.349 0.006
0.90
8OCOP (ours)
0.950 + 0.002
1.937 1 0.034
0.352 + 0.006
8OCOP (ours)
0.950 0.002
1.932 0.033
0.355 0.006
00
Leas: Amb Ambiguoua
Setta
0.950 + 0.002
1.897 0.034
0.398 + 0.007
Table 9: Performance of Swin-v2-b on ImageNet-Val Val with different À values (a 0.05). Results are
averaged over 100 data splits.
A
Method
Coverage
Avg Size
P(size > 1)
0
Puze Singleton
0.950 + 0.002
225. 685
+ 4.909
0.225 + 0.005
0.01
S0C0P (ours)
0.949 0.002
3.844
0.121
0.245 0.004
0.02
80COP (ours)
0.949 + 0.002
3.089 : 0.087
0.256 + 0.005
0.05
SOCOP (ours)
0.949 0.002
2.775 0.070
0.262 0.005
0.04
80C0P (ours)
0.949 + 0.002
2.610 0.060
0.268 = 0.005
0.05
SOCOP (ours)
0.949 + 0.002
2.502 0.054
0.273 1 0.005
0.06
(owrs)
0.949 0.002
2.426
0.052
0.278 0.005
0.07
SOCOP (ours)
0.949 + 0.002
2.369 + 0.056
0.281 + 0.005
0.08
8OCOP (ours)
0.949 0.002
2.324 0.053
0.284 0.005
0.09
3000P (owrs)
0.949 + 0.002
2.285 0.051
0.287 + 0.005
0.10
80COP (ours)
0.949 0. 002
2.253 0.051
0.289 0. 0.006
0.20
SOCOP (ours)
0.949 + 0.002
2,096 0.045
0.309 0.006
0.30
80COP (ours)
0.949 1 0.002
2.030 0.038
0.320 1 0.006
0.40
50COP (owrs)
0.949 0.002
1.997 0.036
0.329 0.006
0.50
8OCOP (ours)
0.949 1 0.002
1.976 1 0.035
0.336 1 0.006
0.60
S0COP (ours)
0.950 0.002
1.961 0.035
0.341 0.006
0.70
80C0P (ours)
0.950 0.002
1.949 0.036
0.345 + 0.006
0.80
80C0P (ours)
0.950 + 0.002
1.912 0.037
0.349 + 0.007
0.90
SOCOP (ours)
0.930 : 0.002
1.935
0.037
0.352 0.007
1.00
SOCOP (ours)
0.950 1 0.002
1.931 0.037
0.355 1: 0.007
Leas: Amb Ambiguoua Seta
0.950 0.002
1.881 0.033
0.396 0.007
Table 10: Performance of vir-h-14 on ImageNet- Val with different À values (a = 0.05). Results are
averaged over 100 data splits.
À
Method
Coverage
Avg Size
P(size >> 1)
0
Pure Singleton
0.950 +: 0.002
136.219
:: 4.980
0.135 1: 0.005
0.01
SOCOP (ours)
0.950 1: 0.003
2.061
0.062
0.141 1: 0.005
0.02
30COP (ours)
0.950 0.003
1.761
0.040
0.145 0.005
0.03
80C0P (ours)
0.950 + 0.002
1.611 + 0.030
0.148 + 0.005
0.04
SOCOP (ours)
0.950 0.002
1.572
0.151 0.005
0.05
80COP (ours)
0.950 1 0.002
1.529 1 0.023
0.153 1 0.005
0.06
S0COP (ours)
0.950 0.002
1.498 0.023
0.155 0.005
0.07
SOCOP (ours)
0.950 1 0.002
1.474
0.022
0.156 0.005
0.08
SOCOP (ours)
0.950 + 0.002
1.457 + 0.021
0.158 1: 0.005
0.09
30COP (ours)
0.950 0.002
1.444 0.021
0.159 0.005
0.10
80C0P (ours)
0.950 + 0.002
1.432 + 0.022
0.161 + 0.005
0.20
80COP (ours)
0.950 0.002
1.368 0.016
0.171 0.005
0.30
30C0P (ours)
0.950 + 0.002
1.344 1 0.015
0.177 + 0.005
0.40
S0COP (ours)
0.950 0.002
1.332 0.014
0.183 + 0.005
0.50
80C0P (ours)
0.950 + 0.002
1.321 0.014
0.187 + 0.005
0.60
80C0P (ours)
0.950 + 0.002
1.319 + 0.013
0. + 0.005
0.70
30C0P (ours)
0.950 0.002
1.315 0.012
0.193 0.005
0.80
SOCOP (ours)
0.950 + 0.002
1.312 1 0.013
0.195 1 0.006
0.90
SOCOP (ours)
0.950 0.002
1.309 0.014
0.197 0.006
1.00
30COP (ours)
0.950 + 0.002
1.307 0.014
098 + 0.006
00
Leas: Amb inbigu Seta
0.950 + 0.002
1.291 0.011
0.224 0.006
21

---

Published as a conference paper at ICLR 2026
Table 11: Performance on ImageNet- with protocol identical to that in Table
Model
Method
Coverage
Avg Size
P(size 1)
Plug-Tn
0.975 0.004
190.453 +: 23.837
0.839 0.035
RAPS
0.950 0.004
11.521 + 0.793
1.000 = 0.000
BesNet1522-8
Pure Singleton
432.673
Least Ambiguous Sets (ours)
0.949 0.005
9.067 1 0.677
0.798 1 0.011
0.950 0.005
+:
Plug-in
0.963 0. 0.004
70.999 1: 7.761
0.661 1 0.022
RAPS
0.950 0.005
5.947 0.485
0.920 0.069
ECficientNe:-w2111
Pure Bingleton
0.950 0. 0.005
309.720 726 + 14. 14.194
0.369 = 0.014
Least: ambiguosa
SOCOP (ours)
0.950 0.004
4.736 + 0.354
0.571 = 0.024
Plug-In
0.971 0.005
161.625 1. 22.907
0.845 0.030
RAPS
0.950 0. 0.005
10.380 + 0.8 19
1.000 = 0.000
ConvNeXL
Singletos
004
428. 14.761
Least Ambigueus Sets
0.950 = U 0.005
6.810 + 0.492
0.787 = 0.016
(ours)
Plug-in
0.981 U. 0.004
166.462 +: 26.307
0.941 = 0.032
RAP5
0.951 0.005
9.306 0.860
0.000
Swin-v2-b
Pure Singleton
0.950 = U. 0.005
414.601 + 13. 13.283 283
0.414 = 0.013
Ambiguous
SOCOP (ours
0.950 0.005
7.634 + 0.703
0.626 + 0.021
Plug-in
0.965 0.003
33.017 1: 3.027
0.540 0.013
RAPS
0.951 0.005
3.259 +: 0.264
0.979 + 0.092
ViT-h-14
Singleton
0.950 0.004
304.159 + 13.861
0.304
Least Ambiguous Sets
0.950 = 0. 0.005
2.378 = 0.105
0.539 = 0.018
(ours)
0.950 0.005
0.165
Table 12: Performance of ReaNet152-v2 on ImageNet-Ve V2 with different values (a 0.05). Results are averaged over 100 data splits.
A
Mathod
Covernge
Avg Size
P(size
0
Singleton
0.950 1 0.005
432.673
12.869
0.432 0.013
0.01
SOCOP (ours)
0.950 = 0.006
25.474 + 3.292
0.502 0.013
0.02
(ours)
16.743
1.657
0.531 0.013
0.03
90002 (ours)
0.950 = 0.005
14.088
+ 1.270
0.550 :: 0.012
SOCO2 (ours)
0.05
S0C0P (ours)
0.950 = 0.005
12.527 + 1.047
0.581 = 0.013
80002 (ours)
0.950
12.091
1.020
0.592 0.013
0.07
(purs)
0.950 0.005
11.732 0.974
0.601 0.013
0.08
80C02 (ours)
0.950 = 0.005
11.432 + 0.931
0.009 0.013
(ours)
0.615
0.10
50002 (ours)
0.950 = 0.005
10.920 1 0.826
0.620 0.012
S0COP (ours)
9.949
0.30
50C02 (ours)
0.950 = 005
9.745 0.661
0.684 0.011
0.40
80002 (ours)
0.950 =
9.616 0.623
0.699 0.010
0.50
80C0P (purs)
0.950 0.005
9.555 0.609
0.711 0.010
0.60
80C02 (ours)
0.950 = 0.005
9.508 = 0.579
0.720 + 0.010
(ours)
0.80
S0C02 (ours)
0.950 1 0.005
9.344 1 0.554
0.730 1 0.010
80C0P (ours)
9.290 +
0.735 0.010
1.00
50C02 (ours)
0.950 0.0
9.262 = 0.578
0.739 00001
20
Least Ambiguous Set
0.919 0.005
9.067 = 0.677
0.798 00011
22

---

Published as a conference paper at ICLR 2026
Table 13:
Performance of Results are averaged over 100 data splits.
on ImageNet-V2 with different À values (a = 0.05).
A
Method
Coverage
Avg Size
P(size
2 Puze BingleLon
0,950 = 0.003
309. 726
14.19:
0.369 0.014
0.01
90C02 (ours)
0.951 = 0.005
12.243
1.247
0.422 = 0.014
0.02
SOC0P (ours)
0.950 1 0.004
8.506
1 0.744
0.448 0.015
0.03
80C0P (ours)
0.950 = 0.004
7.266 0.505
0.466 0.013
0.04
80C02 (ours)
0,950 = 0. 004
6.643 = 0.433
0,480 = 0.013
0.05
90002 (ours)
0.950 0.004
0.491
0.06
80C02 (ours)
0.950 = 0. 004
5.910 = 0.398
0.500 = 0.015
0.07
90C02 (ours)
0,950 = 0,004
5.718 = 0.397
0.509 0.016
0.08
50202 (ours)
0.950 = O. 00004
5.559 + 0.399
0.516 0.016
0.09
S0C02 (ours)
0.950 1 0.004
5.429 1 0.405
0.522 1 0.017
0.10
80C02 (ours)
0.950 0.004
5.304 0.385
0.628 0.016
0.20
90002 (ours)
0.950 = O. 004
1.751 = 0.302
0.565 = 0.016
0.30
50002 (ours)
0.950 0.004
4.586 U.285
0.590 0.016
0.40
90C02 (ours)
0,950 = O. 004
4.496 + 0.290
0,606 = 0.016
0.50
90002 (ours)
0.950 = 0.004
4.418 + U.273
0.617 + 0.016
0.60
S0C0P (ours)
0.950 1 0.004
4.372 1 0.258
0.626 0.015
0.70
80C02 (ours)
0.950 - 0.004
4.335 + 0.251
0.634 1 0.015
0.80
80C02 (ours)
0,950 0.004
4.305 0.248
0.639 0.015
0.90
90002 (ours)
0.950 = 0.004
1.284 = 0.250
0.614 = 0.015
1.00
S0CU2 (ours)
0.950 0.004
4,272 0.249
0.649 0.016
00
Teast
Amb1 goouus S
0.950 + 0.004
4.157 1 0.231
0.718 1 0.016
Table 14: Performance of -onsNeXt-base on ImageNet-V2 V2 with different À values (a = 0.05). Results are averaged over 100 data splits.
A
Method
Coverage
Avg Size
P(size
0
Pure Singleten
0.950 + 0.004
428.852 4: 14.761
0.428 0.015
0.01
S0C0P (ours)
0.950 + 0.004
19.963 4: 2.012
0.465 1 0.012
0.02
80C02 (ours)
0,950 0.004
14.093
1.081
0.498 0.011
0.03
90002 (ours)
0.950 = 0. 004
11.929 + 0.907
0.518 = 0.012
0.04
80C02 (ours)
0.950 0.004
10.687 0.818
0.531 = 0.012
0.05
90C02 (ours)
0,950 O. 004
9.976 = 0.750
0.513 = 0.012
0.06
50002 (ours)
0,950 = 0.004
9.448 : U.682
0.552 0.012
0.07
S0COP (ours)
0.950 0.004
9.104 1 0.634
0.560 0.012
0.08
80C09 (ours)
0.950 + 0.004
8.830 = 0.599
0.568 : 0.011
0.09
80C02 (ours)
0,950 0.004
8.649 0.577
0.574 0.011
0.10
90002 (ours)
0.950 = O. 0.004
8.493 = 0.561
0.580 = 0.012
0.20
50002 (cars)
0.950 = 0.003
7.008 + 0.525
0.622 0.013
0.30
90C02 (ours)
0.950 005
7.386 = 0.499
0.617 0.013
0.40
50002 (ours)
0.950 = 0.005
7,245 1 0.505
0.665 1 0.014
0.50
S0CDP (ours)
0.950 0.005
7.182 0.503
0.677 0.014
0.60
80C02 (ours)
0.950 = 0.005
7.148 = 0.513
0.688 + 0.015
0.70
90C02 (ours)
0.950 0.003
7.135
0.697 0.015
0.80
50C02 (ours)
0.950 = 0.005
7.111 + 0.519
0.704 : 0.016
0.90
80C02 (curs)
0,950 = 003
7.108 - 0.512
0.711 0.016
1.00
90C02 (ours)
0.950 005
7.104 = U.509
0.717 0.016
20
Least: Ambi mmbigueus
Sett
0.950 0.005
6.810 1 0.492
0.787 0.016
Table 15: Performance of Swin-v2-b ImageNet-V2 with different À values (a 0.05). Results are averaged over 100 data splits.
Method
Coverago
Avg Size
P(size 1)
0
Pure nenton
0.950 = 0.005
414.604 +: 13.283
0.414 :: 0.013
S0C02 (ours)
0.951 1 0.004
20.127
1.710
0.478 0.011
0.02
80C0P (ours)
0.950 = 0.005
13.318 + 1.226
0.501 = 0.013
0.03
50002 (ours)
0.950 = 0. 0.005
11.647 1.024
0.522 0.014
0.04
90C02 (ours)
0.950 0. 0.004
10.768
+ 0.875
0.537 0.013
0.05
S0CU2 (ours)
0.950 1 0.004
10.036 0.794
0.547 1 0.013
0.06
S0CDP (purs)
0.950 0.004
9.507 1 0.706
0.556 0.013
0.07
80C02 (ours)
0.950 = 0.004
9.113 = 0.620
0.564 = 0.013
0.08
S0C09 (ours)
0.950 : 0.004
8.834 1
0.571 0.012
0.09
50C02 (ours)
0.950 = 0.004
8.640 # 0.548
0.579 = 0.013
0.10
80002 (ours)
0,950 = 0. 4.004
8,150 = 0.329
0.585 0.013
0.20
50C02 (ours)
0.950 = 0.004
7.625 + U.596
0.624 0.015
0.30
S0C09 (ours)
0.950 1 0.005
7.337 1 0.583
0.646 1 0.016
0.40
80C0P (ours)
0.950 0.004
7.183 1 0.570
0.602 0.016
0.50
80C02 (ours)
0.950 = 0.005
7.093 = 0.563
0.674 = 0.017
0.60
80C0P (ours)
0.950 = 0.005
7.049 : 0.568
0.684 0.017
0.70
50C02 (ours)
0.950 = 0. 005
7.004 = 0.551
0.692 = 0.017
0.80
90C02 (ours)
0.950 = 0.005
6.971 = 0.349
0.699 - 0.017
0.90
S0C02 (ours)
0.950 0.005
6.954 1 0.538
0.705 1 0.017
1.00
80C09 (ours)
0.950 - 0.005
6.929 1 0.527
0.710 0.017
00
Leaat: Ambiguous Sets
0.950 0.004
6.673 0.472
0.777 0.011
23

---

Published as a conference paper at ICLR 2026
Table 16:
Performance ViT-h-14 on ImageNet- V2 with different À values (a 0.05). Results are
averaged over 100 data splits.
A
Methol
Coverage
Avg Size
P(size 1)
0.00
Pure Singleter.
0.950 = 0.004
304.159
13.851
0.304 0.014
0.01
SOCO2 (ours)
0.950 1 0.004
6.672 1 0.461
0.323 0.012
90C02 (ours)
4.803
0.336
0.03
S0C02 (ours)
0,950 + 0.004
4.145 1 0.213
0.346 1: 0.011
0.04
80C02 (ours)
0.004
3.747
0.352 0.012
0.05
80C02 (ours)
0.950 0. 004
3.493 = 0.186
0.357 = 0.012
0.06
90C02 (ours)
0.950 = 0. 0.004
3.329 = 0.176
0.361 0.013
0.07
S0C02 (ours)
0.950 0.004
3.223 0.162
0.367 0.013
0.08
S0C09 (ours)
0.950 1 0.005
3.147 1 0.160
0.372 0.013
50002 (ours)
0.377 0.014
0.10
S0C0P (ours)
0.950 1 0.005
3.027 1 0.167
0.381 1 0.015
0.20
80002 (ours)
0.950 + 0.005
2.743 0.
0.411 0.016
0.30
80C02 (ours)
0,950 O. 003
2.036 0.141
0.129 0.016
0.40
90002 (ours)
0.950 = 0.005
2.375
0.141
0.441 0.017
0.50
S0C02 (ours)
0.950 0.005
2.539
0.135
0.450 0.017
0.60
80C02 (ours)
0.951 = 0.005
2.516 + 0.127
0.458 + 0.017
S0C02 (ours)
2.496
0.464 0.017
0.80
S0CDP (ours)
0.950 + 0.005
2.480
+ 0.117
0.469 1 0.016
0.90
80002 (ours)
0.950 0.003
2.471 0.116
0.474 0.016
1.00
90C02 (ours)
0.950 0. 005
2.461
= 0.116
0.178 0.016
30
Least Ambi Ambiguous guou5 Sets
0.950 0.005
2.378
0.105
0.539 0.018
D.3 CPL METHOD
We report the performance of the CPL method (Kiyani et 2024) under the same experimental protocol as in the main text. Following Kiyani et al. (2024), we implement H as a linear head on top of the pre-trained model, mapping the final hidden-layer representations to a real-valued scalar. The results, shown in Table indicate that this method exhibits slight undercoverage, while attaining similar performance to Least Ambi guous Sets. Notably, these results are different from the ones reported by (Kiyani et al.) 2024), where the CPL method reduced average set sizes. However, the experimental settings considered in the two papers are different, which may explain the experimental differences. In particular, their results use older large language models which perform quite poorly on MMLU, such that the original average set sizes are very large, being for instance equal approximately 3.5 out of 4 in example. This leaves ample opportunity for improving the set sizes by the CPL method. In contrast, in our setting, the language models have a higher performance (leading to smaller sizes with the default least ambiguous sizes method, around 2.5 out of 4), which may leave less opportunity for improvement.
E HYPERPARAMETER GRID
For RAPS, we follow (0.001, 0.01, 0.1, 0.2, 0.5) optimize set size and a grid with smaller values A (0.00001, 0.0001, 0.0008, 0.001, 0.0015, 0.002] to optimize SSCV. For our SOCOP method, use a linearly spaced grid of 15 values over over [0.05, 1.0] optimize the balance between set size and non-singleton rate, and linearly spaced grid of 15 values over [0.005, 0.1] to optimize SSCV.
Angelopoulos et al. (021), using the gird X E
24

---

Published as a conference paper at ICLR 2026
Table 17: Performance of CPL (Kiyani al. 2024 MMLU with the same protocol used (a 0.05).
ImageNet Val, ImageNet-Ve V2, TissueMNIST and
Model
Coverage
Avg Size
P(size 1)
122-2
0.950 + 0.002
2.297 + 0.059
0.463 + 0.006
Rfficiintteet-v2-1
0.949 0.003
1.542 + 0.044
0
ConsNext-base
0.948 + 0.003
1.866 + 0.050
0.392
Swin-v2-b
0.948 1 0.003
1.841 1: 0.039
0.386 0.009
ViT-h-14
0.949 1 0.004
1.292 1 0.030
0.221 0.0017
(a) ImageNet- Val
Model
Coverage
Avg Size
P(size > 1)
ResNct152-v2
0.950 = 0.006
9.295 + 1.167
0.797 0.016
RFficientNNet-v2-1
0.940 + 0.006
3.394 + 0.400
0.675
bass
0.949 + 0.005
6.677 + 0.615
0.773 0.019
Swin-v2-b
0.949 + 0.005
6.493 + 0.673
+
ViT-h-14
0.948 + 0.005
2.400 + 0.154
0.489 + 0.019
(b) ImageNet-V2
Model
Coverage
Avg Size
Písize > 1)
ResNet (224) 0.950 0.003 2.640
0.040 0.791 0.008
(c) TiSST
Model
Coverage
Avg Size 2.400 + 0.046
P(size 1)
L Llama3
0.948 E 0.006
0.644 + 0.012
(d) MMLU
25


Reviews:
[{"review_id": "4QXsrJxNkU", "reviewer": "Reviewer_5adu", "summary": "The paper proposes a new nonconformity score with the goal of reducing the probability of producing non-singleton sets. The score is motivated by an optimization problem that optimizes a combination of the singleton objective and the expected length of the prediction sets subject to coverage constraint. The authors provide an efficient algorithm through a geometric reformulation to compute the nonconformity score. The paper includes empirical evaluation on image classification and LLM multiple-choice question answering that shows the proposed method reduces average set size and increases the frequency of singleton predictions.", "questions": "Comments/typos:\n\n- P4 l201: extra‘that’\n- P5 l231: extra ‘the’", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The motivation behind the problem of singleton sets is important in practical settings. The paper presents a theoretically sound and efficient algorithm for the problem. Empirical evaluation shows that the method reduces the non-singleton probability while maintaining the average set size. Additionally, the extensions discussed in Section 2.2 are appreciated.", "weaknesses": "1. The paper is not written very clearly and it is hard to follow. The notation is extensive and dense in Section 2, with several new symbols introduced with little intuition. Additionally, the definition of nested conformal prediction was deferred for later, while the concepts are used in the paper; e.g., l215 ‘the nested sets property implies that…’ – this hampers the readability of the paper. It is suggested to work on this in the revised version.\n2. While the paper guarantees marginal validity, there is no discussion on the implications of this objective on conditional coverage. Such analysis (theoretical and empirical) is warranted to understand the underlying tradeoffs.\n3. Apart from Coverage and Avg Size, the empirical evaluation should report metrics that measure adaptivity of the sets e.g., worst-slice coverage, SSCV. The current evaluation is limited in its scope of studying the Avg Size and P(Size > 1) that are directly optimized by the method. To understand the usefulness of the sets, the analysis will benefit from additional metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TG2XEnR7VU", "reviewer": "Reviewer_XEKF", "summary": "This paper proposes a new conformal score function designed to produce prediction sets that minimize a weighted sum of the probability of singleton sets and the expected set size, subject to lower bound on marginal coverage. This generalizes the LAC score function proposed in Sadinle et al., 2019, which is a special case of this paper’s score function with the weight on expected set size set to infinity. Computing this score function is nontrivial, and the paper derives a way to compute it efficiently in O(K) time, where K is the number of classes. The method is validated on image and Q&A datasets and we see that the proposed method does a good job balancing average size and P(size > 1).", "questions": "* Can you provide more intuition in the paragraph “Computing the nonconformity score”? My attempt is “The conformal score is the smallest “price” $\\eta$ in terms of the set size metric (singleton + size objective) per unit of “coverage” such that if we “buy” all classes with price less than $\\eta$, we will have purchased the true class. I'm not sure if this is precisely correct, but something like this would be useful. \n* Why do you use Plug-In sets (which do not have a marginal coverage guarantee) instead of APS (which is essentially the same procedure but with the threshold tuned to achieve a marginal coverage guarantee)?\n* “Our method SOCOP outperforms Plug-In and RAPS in both Average Size and P(size > 1). We find this result remarkable […]” — I don’t think the second sentence is warranted. Plug-In/APS and RAPS target X-conditional coverage. How does SOCOP do on this metric? \n* On line 382, you say that ImageNet-V2 is more challenging test dataset. Can you briefly describe why in the text?\n* A class-conditional extension of this method that is mentioned in the Discussion section seems quite important to me, as there are many classification tasks where class-conditional coverage is important. However, I don’t immediately see a way that the current methodology could be extended. Do you have thoughts on how? \n\n\nSmall corrections/suggestions:\n* Line 162: “result;” -> “result,”\n* Line 200: “that, for k >= 3, that” -> “that, for k >= 3,”\n* Line 231: “the the jumps”\n* Line 267: “is set” -> “is the set”\n* Line 337: “Results are reported” -> “Results for this method are reported” (initially I thought this sentence was referring to results for all methods, not just CPL)\n* Line 365: “Imagenet” -> “ImageNet”\n* Line 399: Fig 2 caption “for of”", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is very well written. Some parts I found particularly useful: the discussion of connections to Sadinle et al., 2019 and the comparison in Figs 2-3; the motivation for taking a linear combination of the singleton and size objectives starting on line 311\n* This method can be easily extended to P(size > $k_0$) for any integer $k_0 \\geq 1$. This seems quite practically useful. \n* The experiments are extensive and shed insight into the operating characteristics of the proposed method.", "weaknesses": "There are some places the writing could be improved. See Questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Ob3Cght5Q", "reviewer": "Reviewer_b3RD", "summary": "The paper targets increasing the probability that conformal prediction sets are singletons (sets of size one) in multi-class classification. It proposes a new nonconformity score derived from a Lagrangian relaxation of a joint objective balancing singleton frequency and expected set size, and introduces an O(K) per-instance algorithm via a lower convex hull construction. Empirically, the method increases singleton frequency with small impact on average set size while maintaining marginal coverage.", "questions": "The definition of a singleton set should be more precise. The definition used in the formulas is $|C(X)| \\le 1$, which differs from $|C(X)| = 1$. Empty prediction sets can occur in conformal prediction. For example, if $\\hat p$ can almost perfectly predict the true class, then approximately an $\\alpha$ fraction of least ambiguous sets will be empty. If we use $|C(X)| = 1$ as the definition of a singleton set and change the first term of the loss in line 135 to $I(|C(X)| \\ne 1)$, this may harm the monotonicity of the score function.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Clarity and organization: The paper is well written and easy to follow. The problem setup, the Lagrangian relaxation, and the geometric algorithm are clearly presented.\n\nNovel scoring and efficient computation: The nonconformity score is new and is explicitly motivated by the singleton objective rather than average size alone. The geometric O(K) algorithm based on the lower convex hull is elegant and practically important.", "weaknesses": "1. The statement that “often the most desirable outcome is an unambiguous prediction, a singleton set containing only one label” needs stronger justification. Conformal prediction naturally returns sets of varying sizes; prioritizing singletons is application-dependent rather than universally optimal. Please provide concrete justification and cite papers explicitly values singleton sets.\n\n2. Compared with APS [1], the method provides marginal but not conditional validity when the true $p(y\\mid x)$ is given.\n\n3. While the score is motivated by minimizing $P(|C(X)|>1) + \\lambda E[|C(X)|]$, similar composite losses can potentially be handled within existing efficiency-oriented frameworks (e.g., [2]). Please clarify why a new score is necessary rather than adapting losses within those frameworks, and whether SOCOP can be integrated into them while preserving nestedness, monotonicity, and computational benefits.\n\n[1] Romano, Yaniv, Matteo Sesia, and Emmanuel Candès. Classification with valid and adaptive coverage. NeurIPS 33 (2020): 3581–3591.\n\n[2] Liang, Ruiting, Wanrong Zhu, and Rina Foygel Barber. Conformal prediction after efficiency-oriented model selection. arXiv:2408.07066 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rh54h4yj2Z", "reviewer": "Reviewer_67Nw", "summary": "**Summary**\n\nThis paper addresses a key limitation of conformal prediction: while it creates reliable prediction sets (e.g., a list of possible answers that is 95% likely to contain the true one), these sets often contain multiple items, making them ambiguous for automated decision-making. The authors argue that the most useful outcome is a set with just one answer (a \"singleton\"), as it provides a clear, actionable result. \nThey introduce a method called SOCOP (Singleton-Optimized Conformal Prediction) which is specifically designed to maximize the frequency of these singleton predictions. \nInstead of only trying to make the prediction sets as small as possible on average, SOCOP try to produce unambiguous, single-answer outputs, creating a more practical trade-off for real-world applications where decision clarity is more important than a slightly smaller average set size.\n\n\n**Contribution**\n\nThe main contribution is a novel and highly efficient method to achieve this goal. \nThe authors derive a new \"nonconformity score\" from an optimization objective that directly penalizes non-singleton sets. \nTheir key innovation is a geometric insight that transforms the complex task of calculating this score into a simple computational geometry problem: finding the lower convex hull of a set of 2D points. \nThis allows the score to be computed in fast time, making the method practical even for problems with thousands of classes. \nExperiments on image classification and large language models show that SOCOP significantly increases the rate of singleton predictions (often by over 20%) with only a minimal cost to the average set size, proving its effectiveness and practical value.", "questions": "1.  Could you please make the practical difference between a singleton set from your method and a standard top-1 prediction? For instance, in what specific scenarios would a decision-maker prefer a SOCOP singleton over a top-1 prediction, especially considering SOCOP still produces multi-sets in other cases? A more direct comparison of their respective risk profiles would be helpful.\n\n2.  The performance of SOCOP depends on the hyperparameter $\\lambda$, which is selected on a tuning set. How sensitive is the method's performance to the choice of $\\lambda$ and the size of the tuning set? Could you comment on the potential for tuning bias, as discussed in works like [2], and perhaps provide practical guidance for users on how to select $\\lambda$ robustly?\n\n3.  Could you share your thoughts on potential future directions for this work? For example, could this singleton-optimization framework be extended to other, more complex conformal prediction methods beyond the split-conformal setting, such as those that aim for conditional coverage?\n\n4. For theorem part, is there any insight for the P(size <1)'s increasing under proposed method?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "*   The paper's main strength lies in its novel objective function and the corresponding insight into evaluating conformal prediction. By shifting the focus from minimizing average set size to minimizing the probability of non-singleton sets (`P(size > 1)`), the work introduces a practical and well-motivated criterion for efficiency that directly relates to the goal of automated decision-making. This reframing provides a valuable perspective on what constitutes a \"useful\" prediction set in many real-world applications.\n\n*   This conceptual contribution is supported by a thorough and well-explained computational algorithm. The paper provides a deep discussion on how to make the proposed nonconformity score practical, culminating in an efficient O(K) method based on a geometric reformulation involving the lower convex hull. This ensures the method is not just a theoretical construct but a scalable tool that can be readily applied.\n\n*   The work can be seen as a natural and timely extension of recent research on improving the efficiency of conformal sets, such as the work on length optimization by [1]. While that paper focused on minimizing the expected set size, this paper explores a related but distinct objective—maximizing singleton frequency. This progression shows a thoughtful engagement with the state-of-the-art and pushes the conversation forward on defining and optimizing different notions of efficiency in conformal prediction.\n\n\n[Reference] \n\n[1] Kiyani, Shayan, George J. Pappas, and Hamed Hassani. 2024. “Length Optimization in Conformal Prediction.” In Neural Information Processing Systems. Vol. 37. Curran Associates, Inc.", "weaknesses": "*   The practical significance of the proposed objective could be further elaborated. While the goal of maximizing singletons is intuitive, the paper could benefit from a more direct discussion on the distinction between the method's singleton predictions and simply using the standard top-1 prediction. A clearer explanation of the trade-offs would help readers understand when the statistical guarantees of a conformal singleton justify the method's complexity, especially since it can still produce non-singleton sets.\n\n*   The method introduces a hyperparameter, $\\lambda$, that requires tuning, which can be a practical challenge. The paper could provide a more in-depth discussion on the sensitivity of the results to the choice of $\\lambda$. It might also be beneficial to explore the potential impact of tuning bias on the final performance, a topic investigated in recent work such as [1], and consider if more principled selection methods could be developed.\n\n*   The clarity of the writing in the methodology section could be improved. While the experimental results are presented clearly, the section explaining the method itself consists of large blocks of text that can make it difficult to follow the main ideas. The presentation would be enhanced by breaking down the key concepts more, perhaps by using more structured text or adding a high-level diagram to help illustrate the geometric approach.\n\n[Reference] \n\n[2] Zeng, Hao, Kangdao Liu, Bingyi Jing, and Hongxin Wei. 2025. “Parametric Scaling Law of Tuning Bias in Conformal Prediction.” In Forty-Second International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)