What Does It Take to Build a Performant Selective Classifier?
Authors: Stephan Rabanser
Nicolas Papernot
Venue: NeurIPS 2025 poster ()
Keywords: selective prediction
selective classification
calibration
ranking
abstention
rejection
uncertainty quantification
Primary area: other
TL;DR: We decompose the gap between selective classifiers and the ideal oracle into five measurable sources, showing that only non-monotone scoring methods can reduce it and improve reliability.

Abstract: Selective classifiers improve model reliability by abstaining on inputs the model deems uncertain. However, few practical approaches achieve the gold-standard performance of a perfect-ordering oracle that accepts examples exactly in order of correctness. Our work formalizes this shortfall as the selective-classification gap and present the first finite-sample decomposition of this gap to five distinct sources of looseness: Bayes noise, approximation error, ranking error, statistical noise, and implementation- or shift-induced slack. Our analysis reveals that monotone post-hoc calibration—often believed to strengthen selective classifiers—has limited impact on closing this gap, since it rarely alters the model’s underlying score ranking. Bridging the gap therefore requires scoring mechanisms that can effectively reorder predictions rather than merely rescale them. We validate our decomposition on synthetic two-moons data and on real-world vision and language benchmarks, isolating each error component through controlled experiments. Our results show that (i) Bayes noise and limited model capacity can account for substantial gaps, (ii) only non-monotone or feature-aware calibrators consistently reduce the ranking term, and (iii) distribution shift introduces a separate slack that demands distributionally robust training. Together, our decomposition yields a quantitative error budget as well as actionable design guidelines that practitioners can use to build selective classifiers which approximate ideal oracle behavior more closely.

What Does It Take to Build a Performant Selective Classifier?
Stephan Rabanser" Princeton University
Nicolas Papernot University of Toronto & Vector Institute nicolas www.erotetoooooooo.oooooooo
rabasser@princeton.eed
Abstract
Selective classifiers improve model reliability by abstaining inputs the model deems uncertain. However, few practical approaches achieve the gold-standard performance of perfect-ordering oracle that accepts examples exactly in order of correctness, Our work formalizes this shortfall as the selective-clan cllectiveclasssfficatinn gap and present the first finite-saplle decomposition of this gap to five distinct sources of looseness: Bayes noise, approximation error, ranking error, statistical noise, and implementation- or shift-induced slack. Crucially, our analysis reveals that mono- tone post-hoc calibration- -ofeen believed to strengthen selective classifiers- limited impact on closing this gap, since it rarely alters the model's s underlying score ranking. Bridging the gap therefore requires scoring mechanisms that can effectively reorder predictions rather than merely rescale them. We validate our decomposition on synthetic two-moons data and real-world vision and language benchmarks, isolating each error component through controlled experiments. Our results confirm that (i) Bayes noise and limited model capacity account for substantial gaps, (ii) only richer, feature- aware calibrators mcaningfully improve score ordering, and (iii) data shift introduces separate slack that demands distri- butionally robust training. Together, our decomposition yields a quantitative error budget as well as actionable design guidelines that practitioners use to build selective classifiers which approximate ideal oracle behavior more closely.
Introduction
In high-stakes applications like finance Coenen et al. 2020]. healthcare [Guan et al. 2020 and
autonomous driving Ghodsi et al. 2021] machine learning (ML) models are increasingly tasked with making decisions under uncertainty, where dependable predictions are critical. Selective classifiers [Chow 1957 El- Yaniv et al. 2010) formalize the option abstain on inputs deemed unreliable, reducing the risk of costly errors by refusing to predict when uncertain. Their effectiveness depends on identifying which predictions to trust and which defer. common evaluation metric the
tradeoff, which quantifies how performance degrades as the model accepts broader set inputs. The benchmark hypothetical oracle that ranks inputs by their true likelihood of correctness, yiclding perfect-ordering upper bound [Geifman et al. 2019 Rabanser et al. 2023
While some selective predictors approach this bound, others fall short- revealing persistent gaps and raising open questions about what properties of the learning setup truly govern selective performance.
Classical theory explains selective classification two idealized regimes. In the realizable setting [EI Yaniv et al.] 2010]. where the data is noiseless and the true predictor lies within the hypothesis class, the model can asymptotically achieve the ideal eecuracy-coveragg curve. the more general agnostic setting [Wiener and H|- Yaniv 2011). the classifier competes with the best-in-class predictor, but this benchmark may itself fall well below the oracle bound-an the theory does not isolate
Work done while the University of Toronto and the Vector Institute.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the source of the gap. Yet in practice, we never operate in such asymptotic or idealized conditions: models are often misspecified, the data used for training and evaluation are finite, and asymptotic guarantees offer little actionable insight. As result, even the strongest formal guarantees provide limited guidance, which leaves practitioners with the following question:
For my finite model on finite data, what aspects of the learning setup will actually move my trade-off off curve closer to the perfect-ordering upper bound?
To answer this question, we re-frame selective performance around the selective classification gap (ce): the mismatch between model's accuracy coverage curve and the oracle bound for all coverage levels (see Figure !.. Our work shows that this gap admits finite-sample decomposition:
(c) < Bayes(() (c)
Eapprox (c)
+ Trank (c) + S
+
Emisc
E (0, (1)
irroducible
capacity
ranking
data
optimization shift
Each term corresponds to a distinct-and of- ten setaswrrbbe-souree of looseness. The first
Selective Accuracy
Oeull
aoc, (A. g)
melarra e)
term, Bayes((), reflects irreducible uncertainty: if the true label is inherently unpredictable from the input (e.g., ., due to label noise), even a perfect classifier must abstain on some examples. Next, Eapprox (c) captures limits of the model class: if the function class is too weak to approximate the Bayes-optimal decision rule, the gap widens. The third term, frank(c) quantifies the model's failure to correctly order inputs by their likeli- hood of correctness- lyipicllss-typicalllyy due poor con- fidence estimation or miscalibration. The sta- tistical term s (c) accounts for finite-sample fluctuations that affect both learning and evalua- tion. Finally, S mise (c) aggregates practical imper- fections, such as optimization еrror or test-time distribution shift. Equation (I) thus provides a coverage-unie -nifrrrmmufommmmmmm "error budget" that trans- forms the qualitative question posed earlier into a concrete quantitative diagnosis.
Jd Atejak
0.5
correct points
incorrect points
0.2
0.4
0.6
0.8
Coverage (c)
Figure 1: Visualization of the selective classifi- cation gap Ace). The dashed curve is the oracle frontier acc (Arul c) under which coverage levels left of c Opal (blue) accept all correct predictions first, and rank incorrect predictions last (orange). This constitutes the ideal behavior of selective predictor. On the other hand, the solid curve shows the realized selective accuracy ahc g). The mismatch between acc (qulll c) and acc, g) at coverage is the gap A(c); the gray shaded area visualizes this gap over the full coverage spectrum.
Two key insights, developed further in later sections, are worth previewing. First, show that monotone post-hoc calibration-2 technique that is often thought to improve selective prediction performance-onle possess a limited ability of reducing the ranking term (c)n In contrast, methods that directly yield improved uncertainty scores by leveraging richer feature representations or aggregating diverse model perspectives dominate post-hoc calibration methods. Second, Equation (T) serves as an error budget that identifies cost-effective levers leading to actionable recommendations (i) use additional or repeated labels and noise-robust losses to reduce Bayes (ii) increase capacity or distill from more expressive teacher to shrink Eapprox i (iii) enlarge validation data to lower E and (iv) apply domain adaptation or importance weighting address Emise-
for practitioners:
Contributions. We summarize our main contributions below:
Problem formulation. We recast selective prediction terms of a enverage-uniforey selective classification gap-the the key quantity minimize to approach perfect selective prediction. This framing unifies prior work and highlights which failure modes dominate at each coverage level.
Theoretical analysis. We present the first finite-ammple decomposition of the selective classification gap (Equation (T), dividing it into five terms: Bayes, approximation, ranking, statistical, and miscellaneous errors. Our analysis further shows that monotone calibration is ineffective at reducing the gap, motivating the use of methods that can change the ranking more flexibly.
Empirical validation. Our synthetic and real-world experiments confirm the decomposition: Bayes noise and capacity limits drive large gaps; temperature scaling improves calibration but not ranking: and shirt-aware methods remain essential under distribution shift. These results clarify which factors matter most and how to target them effectively in practice.

---

2 Background & Related Work on Selective Classification
Selective classification extends the standard supervised classification framework as follows:
Definition 1 (Selective Classifier [Chow 1957 EI- Yaniv et al. 2010 ) A selective classifier is pair (h,,) g), where h X y is classifier over covariates X RD. and labels y {1....,K] and g X' x (X' + 3) - R is a selection function that assigns a confidence score.
Given threshold IR, the model abstains when the score falls below the threshold:
-((aa) (h(a)) if gfr 2 h) otherwise
(h, - [1
Intuitively, selective classifier predicts only when confident. The selection score g(x, h) determines whether to accept or abstain: if g(z, h)) 2 r, the model outputs h(zz); otherwise, it returns J.
Many prior works have developed selective classification methods for training competitive pairs (h. A popular method is Maximum Softmax Probability (MSP) [Hendrycks and Gimpel 2017 Geifman and Yaniv) 2017], which uses classifier confidence as the selection score. To improve calibration and reduce predictive variance, ensembling approaches have been explored: Deep En- al. 2017) train multiple models with different initializations, while Selective Classification via Training Dynamics (SCTD) [Rabanser et al. [022] ensembles intermediate checkpoints. Other methods- such as SelectiveNer (SN) Geiffman and EI- Yaniv 2019).
sembles (DE) [Lakshminarayanan
Deep Gamblers (Liu et al.) 2019), and Self-Adaptive Training (SAT) [Huang et al. 2020 alter
the model architecture or loss function ensuring that prediction and rejection are learned jointl..
The efficacy of a selective classifier is evaluated using the empirical accuracy-covert tradeoff.
Definition 2 (Empirical Accuracy Coverrae Tradeoff). Let D (xx,, be dataset. For a selective classifier (h, g) and threshold т, define
1 (i: g(r, 2 73/
(T) N
(3)
{i: M(xa)))) = y and g( h) 2T] {i (i:g(a;))) h) 2 r}
if Ex. (T) > 0,
(T.
(4)
if E 0.
The pair (E, à) as varies is the empirical accuracy-coverage curve.
The score g(r ft) therefore induces a total order over D: æ is accepted before 22 if g(z1, h) > g( h). This ordering governs which inputs are retained as coverage decreases. Effective strategies aim to maximize â at each coverage level ç, often trading off accuracy and coverage.
Accuracy-coverage tradeoff evaluation. The accuracy-coveret tradeoff is often summarized by the area under the accuracy-coverage curve (AUACC), integrating selective accuracy over all Geifmman et al. [2019] show that AUACC favors models already accurate at full coverage. To address this issue, Geifman et al. (0199 and Rabanser al.)
coverage levels [Traub et al. 2024). However,
[023) propose oracle-based bounds, which become loose at low utility Galil et [l.][202) To avoid accuracy bias, Galil et al. [2023] and Pugnana and Ruggieri [2023] recommend using the classifier's AURDC instead. But AUROC is not monotonic in AUACC [Cattelan and Silva 2023 Ding
al. 2020). thus favoring methods tuned for AUROC over selective accuracy. Recently, Traub et al.
[2024]] introduced the Area Under the Goals Reweighte Curve (AUGRC), which multiplies accuracy
by coverage to mitigate bias toward low- cow-coverage regions, while Muesányi al. (2024) provide benchmark disentangling uncertainty sources for fairer comparison. These efforts refine evaluation metrics, whereas our work complements them by analyzing what causes selective performance gaps. Farlier work YEn-va et al.] 2010 Wiener and EI- Yaniv] 2011] characterizes optimal selective classifiers in both realizable and agnostic regimes but focuses on existence rather than practical instantiation- unlike our finite-sample perspective.
3

---

3 Decomposing the Selective Classification Gap
We characterize the optimal performance achievable by selective classifier given its full-coverage accuracy, establishing a reference against which all practical selective classifiers can be evaluated.
3.1 Oracle Bound and Selective Classification Gap
Definition 3 (Perfect Ordering Upper Bound [Geifman et 2019
Rabanser al. 2023 Fix
a base classifier h whose full-coverage (standard) accuracy is Ofull := Pr (r( Y) [0, 1]. For any desired coverage level с E (0, 1], the best selective accuracy achieved by accepting the c-fraction of points with the highest posterior correctness Pr(h(XX Y X) is
1,
< c S dhull
acc ( (full c)
(5)
Ahull <e<l 1.
Assuming Bayes noise-that is, all errors are avoidable given perfect confidence- this piecewise curve (see Figure [ traces the oracle accuracy-coverage frontier based on perfect ranking of examples by correctness probability. Any real selective classifier falls below this bound--poenttially far below, depending on its calibration, expressivity, and sensitivity to noise. To quantify how far given classifier falls short of this ideal, we define the selective classification gap.
Definition 4 (Selective Classification Gap). Let (h, g) be a selective classifier with foll-coverage accuracy Ofull Pr(h(X) Y). For a coverage level e E (0, 1]. let Ta be the threshold satisfying Pr(g( h) 2 To) = c. The selective accuracy at coverage e is acc, (h, g) == Pr(h(X)=Y y(X, h) 2 TT). The selective classification gup at coverage is then defined as the deviation from the perfect-ordering upper bound:
(c)) == acd(aru c) acco(h, g).
(6)
This gap A(c) can be interpreted the excess selective risk given coverage c. We note that integrating A (cc) over the entire coverage spectrum, la (c) is equivalent to the definition of the Excess-AURC (B-AURC) metric proposed by Geifman et al. 2019].
The function (cc) offers a coverage-resolved diagnostic of selective performance. A small gap indi- cates near-oracle behavior-accepting only examples it can confidently and correctly classify-while a large gap suggests limitations estimating correctness or ranking examples reliably. Understanding the magnitude and shape of this gap is key to analyzing and improving selective classifiers.
3.2 Why Is the Upper Bound Loose?
The oracle bound in Definition rellie on two idealized assumptions: perfect prediction all inputs and perfect ranking by the true correctness posterior. practice, selective classifiers deviate in four principal ways, each corresponding to term in later decomposition (EBayes Eapprox Eranks Esta):
Bayes noise (é Bayes)) Even a Bayes-optimal rule errs on intrinsically ambiguous points (where max, Pr(Y- the oracle must accept some of these noisy inputs, lowering the achievable accuracy.
X) < 1), unavoidable in real data (Devroye et al. 2013). As coverage increases,
2. Approximation limits (Capprox). A learned model drawn from restricted hypothesis class may misclassify inputs with high posterior confidence under the Bayes rule [Bishop] 2006). This gap reduces full-coverage accuracy and limits selective performance.
3. Ranking error (Erunk). Let (z) == Pr(h(x) Y X x) denote the true correctness posterior, i.e., the probability that the model's prediction correct given the input. Ideally, the confidence score g h) should rank examples in decreasing order of nı (z)-so so that samples the model is likely to classify correctly (high (x). examples that are "easy") are accepted before those it is likely to misclassify (low T) (æ), examples that are "hard"). When g(X, h)) fails to preserve this ordering, high-confidence errors and low-confidence corrects are interleaved, increasing the selective gap (c))

---

4. Statistical noise (Estat). Estimating the threshold To and selective accuracy from a finite validation set introduces randomness of order O( (10)//))/n) This follows from concentration bounds; see SSalev-Shwarte and Ben-David 2014 for standard applications in learning theory.
Takeaway. The selective classification gap A(c) reflects a mix of irreducible noise, model misspecification, ranking errors, and sampling variability. Addressing cach cleaner labels, stronger models, or improved ranking- tighten selective prediction performance.
In the next subsection, we formalize this decomposition and provide general bound on the total gap.
3.3 Formal Decomposition of the Gap
We now give a principled decomposition of the selective classification gap and provide a correspond- ing finite-sample upper bound. For clarity and notational simplicity, we treat the binary-label case y = (0, 1}; the multiclass extension follows by standard one-vs-rest reduction.
Notation. Let (/x)) Pr(y X .)) be the Bayes posterior. For a fixed classifier h y define its (induced) correctness posterior
mh (x) = Pr(h(x)= = X = =) = (m(x))=11 + (1- (I-n(())[<xxe>===>
(7)
All expectations and probabilities are taken w.r.t the data distribution D. Throughout let g(z, h)) be the confidence score. For a target coverage e E (0, 1] denote by
te s.t. Pr(g(X,h) 2 tz) =
(8)
the population threshold, and write the accepted region A == {ar g(x, h)) tc). The oracle that attains the perfect-ordering bound accepts A = {a: (x)) is among the largest c-fraction):
Error terms. We isolate the following sources of error affecting selective prediction performance:
Bayes (c) E max (nxxx)(X), 1-10),1n())) nx))
1
x A.).
Eapprox (c) = E |xm(X) - nx)] x Ao),
(10)
Erank (c) E6 (xx(X) X A.]
EE[m X A.] (z 0),
(11)
Esta (c) = CV log(1/8)
(12)
n
where n the evaluation-set size, 8 (0, 1) a confidence parameter, and C > 0 an absolute constant. Intuitively, Bayes is the irreducible label noise inside the accepted region; Eapprox measures how far from Bayes-optimal on the selected inputs; rank is a ranking regret measuring the accuracy loss due solely to picking the wrong c-fraction of samples; and aąл captures the sampling uncertainty due evaluating on a finite dataset. Note that we freeze the acceptance set A, defined by the current scoring function and ask how much worse the learned classifier h is than the Bayes-optimal rule.
Remark (Distance to a Perfect Ranker). A natural way to gauge how far the learned acceptance rule is from the oracle the mass mis-ordered
Drank (c) Pr(X E A \ A.) + Pr(X E A. \.\ \A:).
(13)
It equals the total probability of examples that would have to be swapped between A, and A to recover perfect ordering. Hence Drank (c) iff A A in which case Erank (c) also vanishes.
Theorem 1 (Selective Gap Bound). For coverage level E (0, 1] and selective classifier (h, g) the population gap obeys
(cc) = acc (fulll c) - (h. g) I EBayes (c) + Eapprox (c) + Erank (c).
(14)
5

---

Let (c) be the empirical gap on n i..d. test points. Then, with probability at least - 8,
A c) Bayes (c) + Sapprox (c) + Crunk + CV log(1/6)
(15)
Proof. Because acc, (h, g)
[0 (X)
Ae], the gap decomposes as
A (c) =
17h
A
E( Ae]
+ E mm [(--) Ae] + E[1 max (axx(1). 1 n) Ad]
Fownk
Eapproa (c)
This yields the population bound 14
For each expectation in the decomposition apply Hoeffding's s inequality, a union bound over the three terms gives, with probability 1 6,
(A)) - (c)) < <Cy CCClog11//0 (1/ Adding this deviation to (14) establishes (15).
See Appendix B.1 for extended proof with detailed intermediate steps.
A single design choice can shrink multiple error terms. We note that the individual error terms from the decomposition in Equation (T5) can still interact with each other. For example, when the confidence score is the maximum softmax probability (MSP), better approximation of the true conditional 7 not only lowers the approximation term Eappoox (c) but also tends to align MSP more closely with 7), thereby indirectly reducing the ranking error rrank (c). Conversely, a non-monotone calibration head can reduce Eran(() without improving Eapprox (c).
3.4 Calibration and Its (Limited) Effect on the Gap
As shown in Theorem the selective classification gap includes a runking error term frank (c), which captures misalignment between the confidence score and true correctness. Model calibra- tion INiuulecuuMMizi and Caruana assumed to improve this alignment by transforming scores to better reflect correctness likelihood. Yet its effect selective performance remains ambiguous and context-dependent. Prior work has reached conflicting conclusions: Zhu et al. [2022] argue that calibration may degrade abstention behavior, while Galil et al. [2023] find that temperature scaling improve selective prediction in practice. We show that the impact on the gap depends critically on the rype of calibration method used and its influence on the induced ranking. We begin by recalling the formal definition of calibration.
used to reduce
or underconfidence- often
Definition (Perfect Calibration). For each input let a model produce a predicted label ((x) and an associated confidence score s(x) E [0, 1]. We say the model is perfectly calibrated if
Pr(Y x(X) (X)
t) t for every confidence level t E [0, 1].
(16)
Practical estimators approximate (16) via a post-hoc map e such that 8(x) ((s(x)) approaches
prefect calibration. Expected Calibration Error (ECE) [Naeini et al. 2015 quantifies this closeness:
B ECE = Σ
1
1 Σ [((x;)) Tol
yi =
Σ
3(x;)
(17)
n
To
TEL.
where J, is the set of indices in bin b, is the total number of examples, and B is the number of bins.
Monotone score-level calibration leaves the gap intact. Isotonic regression Zadrozny and Elkan
2002) and histogram binning (Zadrozny and Elkan 2001] fit monotone o: 0, + [0, 1 that preserves score ordering. Because monotone maps preserve ordering, the acceptance set A. {a 3(x) 2 JaJ is identical to the one obtained from s(x); hence the selective accuracy acc, (h, g) and the gap (c) = ace (Oaml, c) acc,,(h, (h, g) are unchanged. Monotone calibration thereby reduces the approximation error Eapprox (c) in Section 3.3 but leaves the ranking error Erank(c) untouched.
The effect of temperature scaling on the SC gap. Temperature scaling, the most widely used post-hoc calibration technique, divides every logit vector 2(x) TRKK by scalar > 0.
p (T) exp(2)(x)/T)) exp(((((x))
(18)

---

While this operation leads to a monotone rescaling of the logits, it can lead to non-monotone rescaling of the seftmax probabilities. Since the softmax function is non-linear with respect to the temperature parameter, temperature scaling can therefore change the ranking of samples by confidence. This re-ranking can lead to small but empirically validated improvements in selective classification performance, as measured by metrics like AUROC [Galil et al. 2023 Cattelan and Silva, 2023]. However, the magnitude of this effect is inherently limited (see Appendix B.2 for an extended discussion). While temperature scaling can refine the ordering, it does not fundamentally alter the underlying quality of the model's 's uncertainty estimates.
Moving the gap requires non-monotone scoring. To reduce the selective classification gap Ac)), is not enough to calibrate scores post-hoc using monotone mappings. One must actively change the ranking of accepted examples to better reflect their true likelihood of correctness. Achieving this typically requires non-monotone scoring mechanisms that incorporate richer, instance-specific information-sul -uchi as deep ensembles (DE), self-adaptive training (SAT), or learned correctness heads Io (x) that map hidden representations to confidence estimates. These approaches leverage model diversity, stochasticity, or internal feature structure to distinguish samples that would otherwise receive identical or wrongly ordered confidence values under standard softmax outputs.
Why binning and vector scaling should not be used. Histogram binning [Naeini et al. 2015 and vector/Dirichlet scaling (Kull et al., [019]-whill widely to improve calibration are poorly suited for selective classification. Histogram binning quantizes scores into a small number of bins, mapping wide score intervals the same value and destroying within-bin ordering, which leads to effectively random selection among tied examples. Vector and Dirichlet scaling are post-hoc calibration methods that generalize temperature scaling by learning class-specific transformations of logits-vecce vector scaling applies a linear transformation, while Dirichlet scaling interprets the logits as parameters of a Dirichlet distribution to better model uncertainty. Recent work by Le-Coz et al. [2024) shows that histogram binning and vector/Dirichlet scaling consistently degrade AUROC in selective classification. These results underscore our central claim: improving calibration does not guarantee better ranking. Reducing the selective classification gap requires score functions that explicitly learn to separate easy from hard examples, not just to produce better-calibrated probabilities.
Loss prediction as a multicalibration litmus test. A complementary view on how calibration connects to ranking ability arises from the notion of multicalibration [Hébert-Johnson et al. 2018). which requires that model's confidence be calibrated not only overall but also across many subgroups of inputs. Recent work by Gollakota et al. (2025) shows that achieving strong multicalibration is equivalent to learning an accurate predictor of one's own loss-that is, training auxiliary model to estimate, for each input, the probability that the base predictor will be correct. Viewed this way, reliability becomes self-forecasting problem: if a model (or an auxiliary head) can successfully predict its own 0-1 loss, then its confidence scores must already be well aligned with correctness, leaving little residual ranking error. We formalize this equivalence in Appendix Eland show, both theoretically and empirically, that the degree to which a model's 's loss can be predicted corresponds directly to the magnitude of the ranking-erre term Erank (c). In short, when no auxiliary predictor can outperform the model's own confidence scores at identifying its mistakes, the model is effectively multicalibrated and near the oracle frontier; conversely, any nontrivial foss-prediction advantage exposes where-ane and by how much its internal ranking deviates from perfect ordering.
Takeaway. While post-hoc calibration with temperature scaling can provide modest improve- ments ranking, it is not sufficient close the SC gap. Substantially reducing the ranking error (Erank) ) requires more powerful scoring methods that actively re-rank examples based on richer information, such feature-aware heads, ensembles, or non-monotone transformations.
3.5 Additional Practical Sources of Louseness
The decomposition in Theorem I captures the intrinsic sources of error-Bayes Bayes noise, approximation limits, ranking error, and sampling Slacknforming a principled bound that holds even under perfect optimization, infinite data, and i.i.d. testing. In practical deployments, however, additional imperfec- tions can inflate the empirical gap (c). These stem from implementation details, scoring granularity, and distribution shift-not fundamental limits, but contingent slack terms reducible through better engineering. We summarize them below under single residual slack term Emise (c).
7

---

1. Optimization error Eopt- In practice, gradient-basee solvers rarely attain the empirical risk minimizer. If L(0) denotes the end-to-end training objective encompassngn model architecture, loss (e.g. cross-entropy). and training data- and 0 its final iterate, then Eapt = L(0) - ming L(Ø), which- -ii standard surrogate- to-tte calibration bounds translates into nonzero selective- accuracy loss that persists even under infinite data.
Distribution shift Eshift (c). When the test distribution Prest deviates from the training distribution
Ptrain both calibration and ranking typically degrade. In particular, for a hypothesis class H. the gap due shift can be bounded by an Integral Probability Metric (IPM) [Müller 1997]:
Sshit (c) I IPMx (Ptrains Ptest) := sup preain [f] E peen [f]l
(19)
Hence, larger shifts in distribution (relative to H) lead to wider selective classification gaps.
Residual slack. The dominant practical sources of looseness are optimization error and distribution shift, summarized by (m)se(c) == Eope + (c).n These two terms capture the main drivers of residual deviation between the theoretical and empirical gaps. For completeness, we discuss additional minor contributors such as threshold -seshold-slection noise or score quantization Appendix [B.3] Together, these effects make the bound in Equation (20) sufficient, not merely necessary. for explaining all observed looseness in practical selective classifiers, yielding the streamlined probbabbbbbbby bound.
((c)
(c) + Eapprox (c) + Erank (c) + Estat + E (c).
(20)
intrinsic
Takeaway. Only Bayes reflects irreducible uncertainty; the other intrinsic terms terms-Euppre: Supprex Frank. and -sta-Can can be reduced with better models, calibration, and data. The miscellaneous slack Emisc highlights optimization and shift-robustness as levers for closing the gap to the oracle.
4 Empirical Results
Our experimental study organized around three guiding questions that reflect the theoretical decomposition in Section Unless otherwise specified, all results are averaged over 5 random seeds.
4.1
Q1: How do Bayes error and approximation error shape the gap?
Setup. We conduct both synthetic and real-world experiments. For our synthetic results, which give us precise control over the data generation process, we simulate two sources of intrinsic difficulty on the two-moons dataset: (i) noise o {0.1, 0.33, 0.66, 1.5} controls how much the two moons expand into each other; and (ii) model capacity, varied from logistic regression (low capacity) shallow MLP (high capacity). For our real-world experiments similarly: for (a) we evaluate a trained CIFAR-10 model on the 1IFAR-1ON/1OON (Wei et al. 2022] datasets to assess which data points have large labeling disagreement; and for (b) we vary the model architecture across a simple CNN (details in Appendix D.3). a ResNet-18 (He et al. 2016), and 5es- (Zagoruyko and Komodakis 2016) CIFAR-100 (Krizhevsky et al.
tackle the analysis
2009 and StanfordCars Krause et al. [013)] For each setting, we compute the Excess-AURC: (E-
AURC) Geiffaa et al. 2019 by integrating the empirical gap A(c) across all coverage levels.
Findings. In terms of approximation error, Figure Figure demonstrates that limited model capacity leads to larger gaps, while more expressive models yield tighter alignment with the perfect-ordering bound. This suggests that approximation error is key driver of looseness. In terms of Bayes error, Figure 3 shows that increasing label noise consistently lowers the accuracy-coverage curve, indicating that Bayes error introduces an irreducible component the gap. These results validate the canonical bound (Equation (15)): large Bayes or approximation error can explain substantial looseness.
4.2 Q2: When-and what kind of calabbaian helps?
Setup. We study the same three model classes as before on CIFAR-100: a lightweight CNN, a R8sNet-18, and 5ideResNet-50. dence-scoring variants: (i) maximum softmax probability (MSP) (Hendrycks and Gimpel 2017]:
each backbone we evaluate the following confi-
8

---

Lisear
Mecameyy-Cowera Tradeoff
Accuracy-Cvverap Tradeoff
Jocuracy-Coeerng Tradeaff
1,000
1.0
1.0-
Selectur Accurvcy
Sobccive Accunacy
0..
0,930
R-NN
NN
0.6
StaNat
8,800
Lisan 2.004
0.7
CNS 0.006
0.4
Tacd 0.003
N00 802
WiSelleeNet-55 5.007
0.25
(a) Approximation error with two moons dataset.
(b) CIFAR-IOR -00
(c) StanfordCars
Figure 2: Experiments on approximation error. We find that approximation error is a major contributor to the gap. (a) We show the two moons dataset fitted with models of different degrees of expressiveness as well as the corresponding accuracy-coverage tradeoffs. (b) + (c) Accuracy-coverec tradeoffs for various model architectures on CIFAR-100 and StanfordCars, respectively.
Accuracy Accurasy-Coveree Coveragee
Accuracy- Coverage Tradeoft
Mecurryy-Coverage Tradroff
1.0
1.000
Aecuracy
0.975
0.95
0.9
0.90
0.950
e= 0.66
0-
0.8-
0.925
0.55
0.50
0.7
Co
0:322
0,500
0.055
10%6 0 0.02
25% 0 0.012
50%6 0 010
0.75
0.70
20%
50%: 0.053
0.075
0.065
0.75
1.00
(a) Bayes error with two moons dataset.
(b) CIFAR-1ON
(c) CIFAR-IOOM ROON
Figure 3: Experiments on Bayes error. We find that irreducible noise significantly contributes to the gap. (a) We show the two moons dataset with varying degrees of noise a E (0.1, 0.33, 0.66, 1.5) as well as the corresponding eecuracy-coveraee tradeoffs.. (b) (c) Accuracy-coverage tradeoff's for the 10% (blue), 25% (orange), and 50% (green) most noisy images in CIFAR-1ON11ONN, respectively.
(ii) a temperature-scaled softmax (monotone probability calibration, TEMP) Guo al. 2017]:
: (iii)
self-adaptive training (SAT) (Huang et al. 2020], which implicitly calibrates by relabelling uncertain samples during training; and (iv) deep ensembles (DE) (Lakshminarayanan
al. 2017 of five independently initialised networks (non- monotonee aggregation; improves ranking via variance), Our inclusion of the MSP baseline is motivated by the large-scale study Jaeger al. [2023). who find that MSP, while simple and easy to implement, is often hard-to-bear in practice. For each score we report (a) the weighted Expected Calibration Error (ECE); and (b) the Excess-AURC (E-AURC) [Geifman et al. 2019 metric measuring selective prediction performance.
Findings. We summarize our findings in Table T While temperature scaling (TEMP) consistently improves ECE across model classes relative to MSP, it leaves the selective classification gap largely sechanged-highlighhtee
the limitations of monotone calibration. contrast, SAT slightly improves both ECE and gap by perturbing rankings through relabeling, while deep ensembles (DE) achieve the largest gap reductions by explicitly reordering predictions via averaging. These trends confirm that only methods capable of re-ranking eerraaningiiilpiiitt (SAT) explicitly (DE) can meaningfully improve selective performance. Consistent with this, we find that only SAT and models reliably predict their own loss, reinforcing their stronger alignment with correctness. See Appendix F.55 for details.
4.3
Q3: How does the gap evolve under distribution shift?
Setup. As in QI, we explore this question using both synthetic and real-world distribution shifts. For synthetic experiments, we use the two moons dataset with three types of input shift: shear, rotation, and translation (details in Appendix D.4). For real data with synthetic corruptions, we use CIFAR-10C [Hendrycks and Dietterich, 2019), which applies algorithmic covariate corruptions to the CIFAR-10 test set across five severity levels (1-5). To evaluate under a real distribution shift, we also consider Camelyon 1irroo177-0 7-WILDS (Koh et al. 202T 1 cancer detection dataset where test data collected from different hospital system than the training data.

---

Table 1: Experiments on calibration across model classes on CIFAR-100. Temperature scaling (TEMP) significantly improves ECE over the Maximum Softmax Probability (MSP) baseline but does not help to close the selective classification gap. Self-Adaptive Training (SAT) and Deep Ensembles (DE) improve calibration non-monotonically and also improve selective classification acceptance ordering through re-ranking. A corresponding plot is given in Figure more datasets in Tables
CNN
ResNet-18
WideResNet-50
MSP
TEMP
SAT
MSP
SAT
DE
MSP
TEMP
SAT
DE
E-AURC
0.086
0.085
0.081
0.065
0.033
0.033
0.028
0.026
0.031
0.032
0.028
0.026
ECE
0.142
0.008
0.116
0.019
0.052
0.048
0.026
0.034
0.066
0.050
0.046
0.030
Originel
Sbeer
Accuracy Coverage Tradeoff
Accuracy-Cooe Coveerye Tradeoff
Accuracy-Coveraee Tradeaff
Selocttre Accurscy
0.
Saletteo Becusury
0.9
0.5
Rotation
Transiation
0.8
0.6
Odgeaal
0.7
Recilie:
507
0.8
0.6-
0.150
2.00
0.25
0.50
0.75
1.00
0.00
0.25
0.59
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Coverage
Coverage
Coverage
(a) Distribution shifts with two moons dataset.
(b) CIFAR-IOC
(c) Camelyoml7 1--li/onnn-WWDDS
Figure 4: Experiments on distribution shifts. We find that shifts can also significantly contribute to the gap. (a) Two moons under shear, rotation, and translation with corresponding accuracy-cove coverage
curves. (b) CIFAR-10C across three distinct corruption severities. (c) Camelyon17 OOD shift.
Findings. Figure 4 shows clear trend: as covariate shifts intensify, the cocuracy-coverage coverage curve moves farther below its oracle bound, indicating that abstention no longer isolates easy inputs. Selective classifiers thus become over-confidently wrong, echoing evidence that many uncertainty metrics deteriorate under shift or misspecification [Snoek et al. [019]. As the gap grows with shift severity, deployments must pair selective prediction with robust ranking or shift-detection safeguards.
Conclusion
Building truly performant selective classifier hinges on understanding and closing the gap between practical models and the oracle perfect-orded ordering bound. To answer what takes, we introduce gap and derive the first finite-sample decomposition that pinpoints exactly five limiting factors: three intrinsic sources- Bayes noise, approximation error, and ranking (calibration) error-ane two contingent slack terms- terms-sampling sampling variability and implementation or distribution-shift artifacts. Our experiments show that each component can be individually measured and, importantly, directly improved: stronger model backbones reduce approximation error, non-monotone or aeature-aware scoring shrinks ranking error, and shift-robust training with larger validation sets minimizes residual slack. Together, these insights provide clear recipe for designing and evaluating high-performance selective classifiers.
coverage- uniform delective-classifie
Linitations and future work. While decomposition cleanly bounds the sllectvv-classsifiattin gap, its error budgets can interact-for example, increasing capacity often improves both approxi- mation and ranking-whiin schemes (e.g., SAT, mixup, focal loss) simultancously affect ranking and full-coverage accuracy, confounding the separation of budgets. Our core experiments focus on synthetic and vision bench- marks; extending these insights to large-scale foundation models would be an important direction. We present preliminary exploration on large language models Appendix F2 Finally, because our oracle bound and gap are defined for 0-/ loss, adapting to asymmetric or depended cost functions- required high-stakes decision-making- and its decomposition. Our finite-sample gap decomposition lays the groundwork for more unified reliability framework; extending (i) settings where out-of-distribution inputs must be rejected and (ii) open-ended language generation constitutes promising agenda for future work.
makes unique attribution challenging. Many training-tine calibration
require generalizing both the bound
10

---

Acknowledgements
We acknowledge the following sponsors, who support our research with financial and in-kind contributions: Apple, CIFAR through the Canada CIFAR Al Chair, Meta, NSERC through the Discovery Grant and an Alliance Grant with ServiceNow and DRDC, the Ontario Early Researcher Award, the Schmidt Sciences foundation through the A12050 Early Career Fellow program. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Govermmnnt of Canada through CIFAR, and companies sponsoring the Vector Institute. We thank Relu Patrascu and the computing team at the University of Toronto's Computer Science Department for administrating and procuring the compute infrastructure used for the experiments in this paper. We would also like to thank Andy Wei Liu, Anvith Thudi, David Glukhov, Vardan Papyan, and many others at the Vector Institute for discussions contributing this paper.
References
A. N. Angelopoulos and S. Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. ArXiv preprint, abs/2107.07511, 2021. https://:///xxi org/ abs/2107.00511
A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and Schuster. Conformal risk control. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
C. M. Bishop. Pattern recognition and machine learning. Springer, 2:1122-1122 2006.
M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC. Canada, October 10-17, 2021, pages 9630- 9640. IEEE, 2021. doi: 10. 10.1109994V449222222211 200.00
URL URL https:///oot org 10. 1109/
ICCV48922. 2021 2021.00951 00951
L. F. P. Cattelan and D. Silva. How to fix a broken confidence estimator: Evaluating post-hoc methods for selective classification with deep neural networks, 2023.
T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR. 2020. URL http://pproceeinnss .mm.innn.m..rr......m...mmmm.mmm mlr press/v119/ chen20j html
C.-K. Chow. An optimum character recognition system using decision functions. IRE Transactions Electronic Computers, (4):247-254, 1957.
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafford. Think you have solved question answering? try are, the ai2 reasoning challenge, ArXiv preprint, abs/1803.05457, 2018. URL /attps://arrae org/ abs/1803. 05457
C. J. Clopper and F. Pearson. The use of confidence or fiducial limits illustrated in the case the binomial. Biometrika, 26(4):404-413, 1934.
L. Coenen, A. K. A. Abdullah, and T. Guns. Probability of default estimation, with a reject option. In 2020 IEEE 7th International Conference Data Science and Advanced Analytics (DSAA), pages 439-448, 2020. doi: 10.1 1099/SA44901 2.200000058..
C. Corbière, N. Thome, Bar- Hen, M. Cord, and P. Pérez. Addressing failure prediction by learning model confidence. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Bue Buhh-Buu, E. B. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 2909, 2019. URL https::t.... ptp.:eeeedsppooooongg/ese hww.oeeedinng..euuippspee. www.ipe.ipp... ec/ Tttp:///5hnaa//tteaaaSoec0000000 wtw.r//88---sttttttttacaa

---

E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le. Autoaugment: Learning augmentation policies from data. ArXiv preprint,
2018. URL /attps://arxi.... org/ abs/
E. D. Cubuk, B. Zoph, J. Shlens, and Q. Le. Randaugment: Practical automated data aug- mentation with a reduced search space. In H. Larochelle, M. Ranzato, R. Hadsell, M. Bal- can, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020. URL https: hwt.:/00000000----aaaaaatca
Abstract html
I.. Devroye, I.. Györfi, and G. Lugosi. A probabilistic theory of pattern recognition, volume 31. Springer Science & Business Media, 2013.
A. Dietmüller, A. G. Alcoz, and I.. Vanbever. Fitnets: An adaptive framework to learn accurate traffic distributions. ArXiv preprint, abs/2405.10931, 2024. URL hattps://arx// org/abs..a.a..... /bg/bbs/
2405. 10931
Y. Ding, Liu, J. Xiong, and Y. Shi. Revisiting the evaluation of uncertainty estimation and its application to explore model complexity complexity-uncertainty uncertainny trade-off. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 4-5, 2020.
A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:://pennreview net/ wete/forummiieeo http///rmmmiiiiiccaaaTTaTa
R. El- al. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(5), 2010.
L. Feng, S. Shu, 7.. Lin, F. Lx, I. Li, and B. An. Can cross entropy loss be robust to label noise? In C. Bessiere, editor, Proceedings of the Twenty-Ninhh International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 2206-22 2212. ijcai.org 2020. doi: 10.24963/jccai.20200000 URL https://doi. org/10 10. 24963/1jccc 2020/305
I. Galil, M. Dabbah, and R. El-yaniv. What can we learn from the selective prediction and uncertainty estimation performance of 523 imagenet classifiers? The Eleventh International Conference on Learning Representations. ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
URL httph:ts/pppeeeerree.... net/ httpppfffddddd mid71ddp66622i6...
Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, Laviolette, M. March, and V. Lempit- sky. Domain-adversarial training of neural networks. Journal of machine learning research, 17 (59):1-35, 2016.
Y. Geifman and R. El-ianiv. Selective classification for deep neural networks. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Gar- nett, editors, Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4878 4887, 2017. URL httph::t///ppt htttpa://proceeddnngs neeeeeddiinnne.uiniieee neur meerips... / htwee4444weeeecccbbbbrarttttttte html
Y. Geifman and R. El-Yaniv. Selectivenet: A deep neural network with integrated reject option. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California. USA, volume 97 of Proceedings of Machine Learning Research, pages 2151-2159. PMLR, 2019. URL http! //proceedings.co mlr ppwss/v97 http://v77/eeifmaa...... html
Y. Geifman, G. Uziel, and R. Yaniv. Bias-reduced uncertainty estimation for deep neural classifiers. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LAA USA, May 6-9, 2019. OpenReview.net, 2019. URI. httpp:://ppereeiiew net/ net/forum?fid SJfb5j CqKm
12

---

Z. Ghodsi, S. K. S. Hari, I. Frosio, T. Tsai, A. Troccoli, S. W. Keckler, S. Garg, and A. Anandkumar. Generating and characterizing scenarios for safety testing of autonomous vehicles. ArXiv preprint. abs/2103.07403, 2021. URL httphhttp://araae
org/ httpg//ab/2110...//00 07403
A. Gollakota, P. Gopalan, A. Karan, C. Peale, and Wieder. When does a predictor know own loss? ArXiv preprint, abs/2502.20375, 2025. URL htttttttp/////iiii wrw. httg//ab//502..200 20375
A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf. and A. Smola. A kernel two-sample test. The journal of machine learning research, 13(1):723-773, 2012.
J. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. Á. Pires, Z. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent A new approach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Had- sell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, Decem- ber 6-12. 2020, virtual, 2020. URL https: http::s://p/p/eedd/p//eeee neurips... cc/ cc/paper///eeccccccccccccpeea (aaperr/020//10 2020/hash/ Abstract html
H. Guan, Y. Zhang, H.-D. Cheng, and X. Tang. bounded-abstaining classification for breast tumors in imbalanced ultrasound images. International Journal of Applied Mathematics and Computer Science, 30(2), 2020.
C. Guo. G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning. ICML 2017, Sydney, NSW. Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1321-1330. PMLR, 2017. URL http:ttt//ppooceediia mlr 2oss//770/guo1760
P. E. Hart, D. G. Stork, and R. Duda. Pattern classification. Wiley Hoboken, 2001.
K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NY, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2016. doi: 10.1109/CPP..2016. URL https://doi. org/10. 10. 1109/ /tt.//CVR........... 2016.90
U. Hébert-Johnson M. P. Kim, O. Reingold, and G. N. Rothblum. Multicalibration: Calibration for the (computaiioaallyddenttiaae)))) masses. G. Dy and A. Krause, editors, Pruceedings of the 35th International Conference Machine Learning, ICML 2018, Stuckholmsmässan, Stockholm, Swe- den, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1944-1953. PMLR, 2018. URL htttt:tt/proceeciinngge//////// mlr, prwps/pssssseeeee wwtppppwssseeeeppeppppppp/pppeet wtt.vv00/veeeeee wwwpettbbetteeeeeeeeeeeeeeeeeeee johnson18a. html
D. Hendrycks and T. G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In 7th International Conference Learning Representaations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://openreevier net/
Horum?id-H[Jzsi.c.
D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openreevie. hwtpeevwwwwieeeeeeeeee www. font//ooum??? wewoorridddhhheeeoooo 2id=Hkg41199IX
D. Hendrycks, K. Lee, and M. Mazeika. Using pre-training can improve model robustness and uncertainty. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2712-27211 PMLR, 2019. URL http://proceeddngs. mlr, wwwppresssvvv /97 http//////hnnncckssee/ html
D. Hendrycks, C. Burns, Basart, A. Zou, M. Mazeika, Song, and J. Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Rep- resentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreviee net wtwp:e/wweeeettottnneeeeee (ttp://ooummm.ddddkjiiiimmm
13

---

G. Hinton, O. Vinyals. and J. Dean. Distilling the knowledge in neural network. ArXiv preprint, abs/1503.02531, 2015. URL /ttpp:://///rr http:/..rrgaab///5///.
aba 1503, 02531
G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pages 646-661. Springer, 2016.
G. Huang, Y. Li, G. Pleiss, Z. Liu, J. E. Hopcroft, and K. Q. Weinberger. Snapshot ensembles: Train 1, get M for free. In 5th International Conference on Learning Representations. ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
htttp://opperrevee wwt. met//fouue htt//morrmmmi/i// BJWwY911
J. Huang, A. J. Smola, A. Gretton, K. M. Borgwardt, and Schölkopf. Correcting sample selection bias by unlabeled data. In B. Schölkopf, J. C. Platt, and T. Hofmann, editors, Advances in Neural Information Processing Systems 19, Proceedings of the Twentieth Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 4-7, 2006, pages 601-608. MIT Press, 2006. URL baatts::/proceeediie www.ipeuriiicom /cepippeecc/pppee/2006/// hash/a2188
Abstract html
L. Huang. C. Zhang, and H. Zhang. Self-adaptive training:
beyond empirical risk minimization. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Sys- tems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL httph:tp://// httpps://proceeddngg has.aeeeee-brttatt...a.
neurips. ce/paper/2022 (ttp:////aea02221111116611611100
P. F. Jaeger, C. T. Lüth, L. Klein, and T. J. Bungert. A call to reflect on evaluation practices for failure detection in image classification. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreeiiet
E. Jones, S. Sagawa, P. W. Koh, A. Kumar, and P. Liang. Selective classification can magnify disparities across groups. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL/attps://opencee
net/forum?id=NOM -
S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, et al. Language models (mostly) know what they know. ArXiv preprint, abs/2207 005211, 2022. URL [ttt:tttt///rrrtt
org/ abs abs/ 05221
P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. L. Phillips, I. Gao, Lee, E. David, Stavness, W. Guo, B. Earnshaw, S. Haque, S. M. Beery, J. Leskovec, A. Kundaje, E. Pierson, S. Levine, C. Finn, and P. Liang. WILDS: A benchmark of in-the-wild distribution shifts. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Leurning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 5637-5664. PMLR, 2021. URL http://proceeedings mlr, aalrpprsse wttpps//we///eeeeeepeeeeeppeeeee hww.13999/hh2aa.... html
R. Kohavi et al. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Ijcai, volume 14, pages 1137-1145. Montreal, Canada, 1995.
J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
M. Kull, M. Perelló-Nieto M. Kängsepp, T. de Menezes e Silva Filho, H. Song, and P. A. Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Achhé Buc. E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 122252122300 2019. URL http:tttt///ooeeediiiii wwwrroceedinssnneuiii.n. www.ieuripp.. cc/paper/
2019 o011:haa//ooll0s0000000072222222
html
14

---

B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estima- tion using deep ensembles. I. Guyon, U. von Luxburg, S. Bengio. H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. pages 3413, 2017. URL / cc/ ht//aaper/001// /ta////////////////// htt.:///cceeoee000efffffffasaea.
neurips. html
A. Le-Coz, S. Herbin, and F. Adjed. Confidence calibration of classifiers with many classes. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems 38: Anmual Conference ral Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, Decem- ber 10 15, 2024, 2024. http: //tppers/ppper..... nips cttccc/ppppco wacc/pppr__teee files www.sseaaeeeereeeeeeeeeeeaaeeeee wwwppaaww//////appaappppppp Bdff0c//115aa568811110033 a2476//2244baaac otgp://aaaaaaaattooooeeeeeaaaaaa -obstracttConfffe
Neu-
html
Y. Li, N. Wang, J. Shi, J. Liu, and X. Hou. Revisiting batch normalization for practical domain adap- tation. ArXiv preprint, abs/ (bs//603.047773 2016. httlhttps///rxiii
hts///b////ta //obb//////.... 04779
Z. Liu, Z. Wang, Liang. R. Salakhutdinov, L. Morency, and M. Ueda. Deep gamblers: Learning to abstain with portfolio theory. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Bue E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrlPS 20/9, December 8-14, 2019, Vancouver, BC, Canada, pages 10622-106332 2019. URL https//can//// www.diinnnnnnnnnn.nnee neurips cc/ cc/paper/200
html
Z. Liu, H. Mao, C. Wu, C. Feichtenhoferr T. Darrell, and S. Xie. A convnet for the 2020s. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 11966-11976, IEEE, 2022. doi: 10. 2WW9CCCVPR2228822222016666 URL
https: / /oii org/ 10. 1109/ 2022. 01167
I. Loshchilov and F. Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th In- ternational Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017. URL forum/id=Skq39Scxx
net/
I. Loshchilov, F. Hutter, et al. Fixing weight decay regularization in adam. ArXiv preprint, abs/1711.05101, 2017. URL http:////axxivoo/aa///////ss 05101
P. Massart. The tight constant in the dworetzayy-kefferwwwloowii inequality. The annals of Probability. pages 1269-1283, 1990.
P. Micikevicius, S. Narang, J. Alben, G. F. Diamos, E. Elsen, D. García, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu. Mixed precision training. 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3. 2018, Con- ference Track Proceedings. OpenReview.net, 2018. URL id=rlgsJggRZ
B. Mucsányi, M. Kirchhof, and S. J. Oh. Benchmarking uncertainty disentanglement: Specialized uncertainties for specialized tasks. In A. Globersons, L.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024, URL http:hhtt::p::pt nips. wttpniipsscppppefiiiile.. files/ and_
Baachmarks_Track...
A. Müller. Integral probability metrics and their generating classes of functions. Advances in applied probability, 29(2):429 443, 1997.
Z. Nado, N. Band, M. Collier, Djolonga, M. W. Dusenberry, S. Farquhar, Q. Feng, A. Filos, M. Havasi, R. Jenatton, et al. Uncertainty baselines: Benchmarks for uncertainty & robustness in deep learning. ArXiv preprint, abs/2106.04015, 2021. URL. aattps://arxxs /ttp//riiv..or/bb///////aea 04015.
15

---

M. P. Naeini, G. F. Cooper. and M. Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. B. Bonet and S. Koenig, editors, Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, pages 2901-2907. AAAI Press, 2015. StRL/http://wette www. wwwp:///wwwwwaa.a.aa.aaaaaaaaaa. .rgg ocs/ index.
9667
A. Prulescu-MMizii and R. Caruana. Predicting good probabilities with supervised learning. In L. D. Raedt and Wrobel, editors, Machine Learning, Proceedings of the Tienty-Secone International Conference (ICML 2005), Bonn, Germany, August 7-11, 2005, volume 119 of ACM International Conference Proceeding Series, pages 625-632. ACM, 2005, doi: 10. 10.1144/110351.111 1002430. URL https: org/ 10. 1111/11111211
1102430
C. Northcutt, L. Jiang, and I. Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373 1411, 2021.
J. C. Peterson, R. M. Battleday, T. L. Griffiths, and O. Russakovsky. Human uncertainty makes classification more robust. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South). October 27- November 2, 2019, pages 9616 96259625..220 IEEE, 2019. doi: 10. 10.110///CCV22019.000999
URL htthhttps:: //doi org/ 10. 1109/ ICCV 2019. 00971
A. Pugnana and S. Ruggieri. Auc-based selective classification. In F. J. R. Ruiz, J. G. Dy, and van de Meent, editors, International Conference on Artificial Intelligence and Statistics, 25- 27 April 2023, Palau de Congressos, Valencia, Spain, volume 206 of Proceedings of Machine mlr. press//ss/
Learning Research, pages 2494 2514. PMLR, 2023. URL httptttts///proeeeeiie #206/pugnanaa htmmlll html
S. Rabanser, S. Günnemann, and Z. C. Lipton. Failing loudly: An empirical study of methods for detecting dataset shift. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' A'ché- Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NewrlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 1394- 1406, 2019. URL https: pttp:/edooceece..ccooopo.o. neurips wew.diiinnnnnnnisss.eee.. cc/ (6c88 Abstract .ttml
paper/2019/hant 2ttpe///00///aah//66660////////1 faaaah//666220077556664444444442
S. Rabanser, A. Thudi, K. Hamidieh, A. Dziedzic, and N. Papernot. Selective classification via neural network training dynamics. ArXiv preprint, 0bbs/2055.15332 2022. URL URL https://axxxx org/ abs/ 13532
S. Rabanser, A. Thudi, A. G. Thakurta, K. Dvijotham, and N. Papernot. Training private models that know what they don't know. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, IA, USA, Decem- ber 10- 16, 2023, 2023. URL, http::///ppprssss wwp.:iiiss..p.ppppco.. lalppaappellsssslllllllllleseeee files/ wwwes/papeee//eeeeeeeeeeaaaa (aaper22023/ash/ html
S. Rabanser, A. S. Shamsabadi, O. Franzese, X. Wang, A. Weller, and N. Papernot. Confidential guardian: Cryptographically prohibiting the abuse of model abstention. In Proceedings of the 42nd International Conference on Machine Learning, 2025.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of mmlr
Machine Learning Research, pages 8748-8763. PMLR, 2021. htw.tttttt///// press/vi33 w139/radfordd111... html
S. Reed, H. Lee, D. Anguelov, C. Szegedy, D. Erhan, and A. Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint
2014.
R. Salakhutdinov. Deep learning. S. A. Macskassy, C. Perlich, Leskovee, W. Wang, and R. Ghani, editors, The 20th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '4, New York, NY, USA August 24 27, 2014, page 1973. ACM, 2014, dois 10.1 10.114552623330226666 2630800 URL https: /oi. org/ 10. 1 1145/ 200555/223330.022 2630809
16

---

S. Shalev-Shwartz and S. Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
L. N. Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV), pages 464-472. IEEE, 2017.
I. N. Smith and N. Topin. Super-convergence: Very fast training of neural networks using large learn- ing rates, In Artificial intelligence and machine learning for multi-domain operations applications, volume 11006, pages 369-386. SPIE, 2019.
J. Snock, Y. Ovadia, E. Fertig, B. Lakshminarayanan, S. Nowozin, D. Sculley, J. V. Dillon, J. Ren, and Z. Nado. Can you trust your model's ' uncertainty? evaluating predictive uncertainty under dataset shift. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver. BC, Canada, pages 13969 13980, 2019. URL Abstractb html
neur wwwerrps.cc/ppppeeeeeeeeeeeeeeee cc/paper/
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):
1929-1958, 2014.
G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ramé, M. Rivière, et al. Gemma 3 technical report. ArXiv preprint, abs/2503, 19786, 2025. URL https://arxiv org/ httgg/ab//2000.. 19786
R. J. Tibshirani and B. Efron, An introduction the bootstrap. Monographs on statistics and applied probability, 57(1):1-436, 1993.
J. Traub, T. J. Bungert, C. T. Lüth, M. Baumgartner, K. H. Maier-Hein, L. Maier-Hein, and P. F. Jaeger. Overcoming common flaws in the evaluation of selective classification systems. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomezak, and C. Zhang, editors, Advances Neural Information Processing Systems 38: Annual Conference on Neu- ral Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, Decem- ber 10 15, 2024, 2024. http:///:///ptee www.iii..pppppeiaecim fecd/pppr__f_iiee files/ wiles/paper//ee/ pappr/2024/haas 0447744c500bd8e22344400 o c64ff
html
V. Vovk, A. Gammerman, and G. Shafer. Algorithmic learning in random world. Springer, 2005.
D. Wang, E. Shelhamer, S. Liu, B. A. Olshausen, and T. Darrell. Tent: Fully test-time adaptation by entropy minimization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.nee, 2021. URL Forumpid-
net/
Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen. Mmlu-pro: A more robust and challenging multi multi-task language understanding benchmark. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomezak, and C. Zhang, editors, Ad- vances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NearIPS 2024, Vancouver, BC, Canada, December htcp/ppperfilees/_/eeeeeeeeeeeee Abstract Datasets_ wanddddeehaarkse...aaaaaaaaaaaae _
10 15, 2024, 2024. URL aa23edd554333335611222ee
nips. cc/ cctpppprpppppeeeeeeeeeeeeeeeeeee filerrfiies//eeee www.ffilleseeeeee halppppp////// 2024/hash/
html
J. Wei, Z. Zhu, H. Cheng, T. Liu, G. Niu, and Y. Liu. Learning with noisy labels revisited: A study using real-worid human annotations. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https://openrevieet
Y. Wiener and R. El- Caniy. Agnostic selective classification. In Shawe-Taylor, R. S. Zemel, P. L.. Bartlett, F. C. N. Pereira, and K. Q. Weinberger, editors, Advances in Neurul Infor- mation Processing Systems 24: 25th Annual Conference on Neural Information Process- ing Systems 2011. Proceedings of meeting held 12-14 December 2011, Granada, Spain,
17

---

pages 1665-1673, 2011 URL Abstracc..ta
Xie, R. B. Girshick, P. Dollár, Tu, and K. He. Aggregated residual transformations for deep neural networks. In 2017 IEEE Conference Computer Vision and Puttern Recognition, CVPR 2017, Honolulu, HI. USA, July 21-26, 2017, pages -995,, IEEE Computer Society, 2017. doi: 00.1009/CVP.2201..665 6344. URL htthttss: httpp:://dooioooo org/ http//11.119110 1109/ CVPR. 2017 .644
S. Yun, D. Han, S. Chun, S. Oh, Y. Yoo, and J. Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In 2019 IEEE/CVE International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2019, pages 6022-6031. IEEE, 2019. doi: 10. 10911CCV22009
URLhttps: https: https::/ddoi.ooo org. 10. 1119/ICCCV 2019 006122
B. Zadrozny and C. Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In C. Brodley and A. Danyluk, editors, Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001). Williams College, Williamstown, MA, USA, June 28- July 1, 2001, pages 609-616. Morgan Kaufmann, 2001.
B. Zadrozny and C. Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694-699, 2002.
S. Zagoruyko and N. Komodakis. Wide residual networks. In R. C. Wilson, E. R. Hancock, and W. A. P. Smith, editors, Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016. BMVA Press, 2016. URL htth:tt////ei bmva. org/oogg///ooo org/bmvc/ html
2016 2016
H. Zhang, M. Cissé, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https /openreview net/ htr//oorummm/ rormmp--
F. Zhu, Z. Cheng, X.-Y. Y. Zhang, and C.-L. Liu. Rethinking confidence calibration for failure prediction. In European conference on computer vision, pages 518-536. Springer, 2022.
18

---

A Broader Impact
This work introduces a decomposition of the selective-classifcation gap into measurable com- ponents Bayes noise, approximation error, ranking error, statistical noise, and deployment slack-offering offering practical guidance for improving abstaining classifiers. By diagnosing which source dominates in given setting, our method supports more targeted model design and evaluation.
Positive implications. Our decomposition improves transparency and supports safer deployment in high-stakes domains by helping practitioners understand whether their model underperforms due to ranking, capacity, or robustness. Because each gap component explicitly quantified, our approach can serve tool for model debugging, monitoring, and fairer benchmarking.
Potential risks. Selective classifiers may disproportionately defer on certain groups, amplifying disparities- risk previously observed by Jones et al. [202T] Additionally, institutions may exploit uncertainty estimates to justify strategic abstention deliberately deferring on individuals they prefer not serve [Rabanser al. 2025]. While our framework identifies which part of the gap drives poor performance, it does not control how deferred inputs are handled.
Mitigations. We recommend reporting gap components disaggregated by sensitive attributes, auditing scoring functions for spurious correlations, and documenting fallback policies. These steps essential to ensure that abstention mechanisms improve reliability without undermining fairness.
Outlook. We hope this work encourages more precise evaluations of selective classifiers, shifting focus from aggregate calibration to interpretable, component-wis gap analysis that can inform both technical improvements and policy safeguards.
B Methods Extension
B.1 Detailed Proof of Theorem 1
We restate the theorem for convenience.
Theorem 2 (Selective classification Gap; detailed), Fix coverage level (0, 1]. score
function g(: h), and its associated population threshold satisfying Pr(g(X, X, h) 2 tc) =
Define the accepted region A. := {u g(x, h) 2 te) and the oracle region A = {x Ti (z) is among the largest c-fraction). With the error terms
Baaee (c) E [1 max(n(X), 7(X) | Acl,
(21)
Eapprox (c) (X) - 7X) X Ad),
(22)
Erank (c) E (x) X A E (X) X E Adl (2 0),
(23)
the population gap satisfies
(Cc) = acc bace(auull, e) (h. (1.00 g) 1 Baaeas + Eapprox (c) + Erank (c).
(24)
Moreover, let A(c) be the empirical gap computed on n independent test samples. Then for any 8 E (0, 1), with probability at least - 8,
log 1/0)
A <
Bayes (c) + Eapprox (c) + (c)nk(c) + C.
(25)
where C is absolute constant.
Proof. We split the argument into four self-contained steps.
Step O. Oracle upper bound revisited. For completeness we justify the piecewise form of acc acc(affll e) in Definition Because @tull Pr(h(X) Y) E[n, (X)), the
{a T/ (x) 1} has probability mass at least amit. Hence oracle that retains the
19

---

highest-cone coneestnec points achieves perfect accuracy for all coverages e I Afull. For e > dfull the best can do include only those perfect points plus a (c Grun) (hun--faction of the remaining examples, which contribute at worst 7ero accuracy. Therefore
Gfull
acull acc Afull c)
Afull < cc 1.
(26)
Step 1. Algebraic decomposition of the gap. Recall that acco(h, - [l) | We repeatedly add and subtract the same quantity:
X E A.].
(c)) acd(anal) e) acce (h, g)
aco( (fulll c) E(my, A
+ B[m AA]] E[ma Ac]
I 1m A - E[m. A.
(rank)
+ E[m [(n=)] Ael
(approx+Bayes)
(rank(c) (c) + Eapprox (c) + Bayes (c).
(27)
Explanation of the two labelled inequalities.
1. (rank) isolates the ranking error, Erank (c) := EE [7, A4 Ein A,)] The inequality holds because the remaining term from the previous line, acc(amm) c) E(m At], is non-negative quantity that bounded by the error sources introduced next.
2. (approx+Bayes) adds and subtracts (x)) inside the expectation, then splits the abso- lute value:
Tih - (n=Y) = (m n/) (17 - [(h=y))-
(28)
The second summand satisfies the deterministic bound 1n(X)-Ipp) 7) [(k=r) S - max(n) 7), yielding exactly Bayes (c). The first summand contributes Eapprox(c).
- -I(h-ryl = max(1, 1
Step 2. Non-negativity of (c). Because (X) E |oo, 1] and A contains the c-fraction of points with the largest 7), -alues,, E(mn. A E[m A..] hence Srank(c) (c) 2 0 as stated.
Step 3. Finite-sample deviation. Let f be any empirical average of [0, I] 1]-valued random
variable with expectation μ. Hoeffding's inequality gives Pr([ū μ4 e) < 2e - Apply this bound separately the three empirical estimates that constitute A(c), and take a union bound with log(6/8)
2n This yields, with probability at least 5, A (c) A(c) <
Colog(1/8)f/ for an absolute constant C. Combining with 24 proves 25
Step 4. Connection to ranking distance. Define the mass of mis-ordered points Drank(c)
Pr(X E A Ac) + Pr(X E A \ A²). Because 7) E [0, 1].
Erank (c) E[m A
[m.. A.]
(29)
< mh lloo Drank (c)
(30)
< D Drank (c).
(31)
Hence (rank(C) 0 if and only A. A
This completes the proof.
Multiclass remark. For K > 2 labels, define 1)(x) (Pr(Y = 2 Pr(Y = K x))
and its complement confidence y) (Ir = maxı 7h (2). Then the inequality myy - [(==)] <
1 - max replaces the binary bound above, and the rest of the argument goes through verbatim. The
approximation term becomes [[[lm 7 As]; all other quantities are unchanged.
20

---

B.2 When Can Temperature Scaling Re-rank Confidence Scores?
Temperature scaling multiplies every logit by the same factor 1/T (T> 0) before the softmax,
exp(z)(x)/T)
(T) (x) En exp(2x(x)/T)
(32)
Although the predicted label arg max, 2, (x) is invariant to T, the confidence score sI sr(x) max; D) (z) (x) can change its cross-sample ordering.
General form. Le arg max, 2y(x) and ry(x)= exp(2z(x)- Zj. (x)) + j.). Then
sT (z) = 1+ 1ts. Ejtt,
(33)
For binary classification, the sum has a single term and (33 collapses to the familiar logistic form 8T(x) 1/(1 + A/T) with = Zj. 23-j.-
Two-sample condition. every rij (T(11) > ST (x2) can change exactly those temperatures T where S (T) S2(T).
For two inputs r1, let S,(T) - E To Because each Tíj S 1,
1/T" is monotone non-decreasing in T' (strictly increasing unless there is tie), and the ordering
Illustrative example (K = 3).
(1) (-2, (2) (0, 0.1,
(34)
At T 1 one finds 81 (x1) = 0.576 > 0.512 = $1 (I22), while at 3 we see that sy(x1) = 0.411 < 0.428 83 (22)) temperature scaling would now accept 22 before 21-
How likely a swap? Equation 33 shows that swap requires the one-dimensional curves (T) and (2)T) to intersect. Since the curves are continuous and monotone, the intersection occurs- if at all-at isolated temperatures and only when the competing logit patterns are finely tuned.
Practical implication. Temperature scaling can in principle tighten the selective-classifcatio gap, but only for the vanishingly small subset of inputs whose non-maxinn mo--maximmm logits happen to satisfy S (T*) (7*).. To obtain meaningful re-ordering one must therefore adopt non-monotone calibration strategies.
B.3 Additional Contingent Slack
In the main text (Sec. 3.5 we folded all implementation-level imperfections into a single residual term (mise retaining only optimization error and distribution shift explicitly. Here we list two further slack terms omitted there:
-heestood-selectoon noise Sahr When the coverage threshold i, chosen on validation set of size m, the realized coverage deviates from the target by
(
(35)
inducing a corresponding vertical shift in selective accuracy.
4. Tie-breaking score quantization (c).
Discrete confidence values (e.g. low-precision logits) create equivalence classes of samples with identical scores. If 6 denotes the maximum number of tied samples at any score level, then
Gie (c)
S K
(36)
where is the size of the evaluation set.
Residual slack revisited. Together with optimization error Spp and shift (c),()), these yield
Emise (c) Eopt Eshifi (c) + Sttr(c) + (ne
(37)
21

---

C Practitioner Checklist for Tightening the Selective-Cassiiiattion Gap
Below is an expanded, actionable checklist to help practitioners systematically tackle each component of the selectiv-classificatie gap. For each item, we list concrete steps, recommended tools, and pointers to reduce the corresponding error term.
E approox Shrink Approximation Error
Model capacity; Upgrade to deeper wider architectures like ResNeXt, VIT. ConvNeXt to better approximate complex functions and reduce base error [Xie et al. 2017 Dosovitskiy
et al. 2021 Liu et al. 2022, Kadavath et al. [022]]
Pre-traiming: Initializing with rich features from self-supervised methods (SimCLR, BYOL) or foundation models (CLIP, DINO) can improve out-of-the-box performance, convergence.
and uncertainty scores [Chen al., 2020 Grill al. 2020 Radford al.) 2021 Caron et al.)
2021 Hendrycks et al. 2019]. However, pre-training can also sometimes negatively affect
selective classification performance [aali al. 2023
Distillation: Use teacher-student training with logit matching or feature hints to inherit
accuracy from a larger model at lower cost [Galil et al. 2023 Hinton et al. 2013 Dietmüller
et al. 2024).
Data augmentation: Augmentations can often improve generalization with policy- based (Au- toAugment, RandAugment) mixing-based (MixUp, CutMix) augmentations to regularize the learner [Cubuk et al. 2018, 2020 Zhang et al. 2018, Yun et al., 2019]. However, strong augmentations may also degrade selective classification performance for certain minority classes [Jones et al. 2021].
rank - Improve Ranking Calibration:
Feature-awar aware scoring:
Train auxiliary heads like ConfidNet learn correctness scores using both logits and input features [Corbière et al., 2019). often improving uncertainty estimates. Self-Adaptive Training (SAT) further enhances this by encouraging internal representations to separate correct and incorrect predictions through contrastive regularization or supervised
signals (Huang et al. 2020).
Deep ensembles; Use the disagreement or predictive entropy across multiple independently trained models to estimate uncertainty [Lakshminarayaaan et al. 2017).
Connorral methods: Generate conformal p-values or risk-controlled selection sets that respect desired coverage guarantees (Vovk et al. 2005 Angelopoulos et al. 20224..
Use caution with vector/Dirichlet scaling: While previous work has shown that vector, matrix, or Dirichlet transformations can be beneficial to reshape confidence distributions [Guo et al.)
2017 Kull et al. 2019), Le-Coz et al. 2024 shows that these techniques can harm ranking
under a large number of classes.
Copt Reduce Optimization Error:
Convergence diagnostics: Track training/validation loss curves to detect underfitting and determine optimal stopping points [Salakhutdinov] [2014).
LLarring- rate schedules: Employ dynamic LR strategies like cosine decay, OneCycle, or CLR
reach better optima more consistently [Loshchilov and Hutter 2017 Smith and Topin 2019
Smith 2017).
Early stopping / checkpoints: Save and average late-stage checkpoints or use snapshot ensem-
bling to smooth optimization variance (Huang et al. 2017 Lakshminarayanan al. 2017 Rabanser et al.) 2022).
Regularization: Use dropout, weight decay, or stochastic depth to prevent overfitting and
stabilize training (Srivastava et al. 2014 [Huang et al. 2016 Loshchilov et al. 2017
Bayes Quantify Irreducible Noise:
Repeated labels: Collect multiple annotations (e.g.. CIFAR 10H) to estimate human-
disagreement and the Bayes error floor (Peterson et al.
2019
Wei et al. 2022].
-Noise-robust training: Mitigate label noise using bootstrapped or Taylor-truncatee loss func- tions that temper reliance on hard labels [Reed et al.
2014 Feng al. 2020
-Dataset curation: Apply confident learning to flag likely label emrors or use active learning for data relabeling [Northcutt al. 2021
22

---

stat
Control Statistical Slack:
Validation set size: Use a sufficiently large holdout set to estimate thresholds and calibrate
uncertainty reliably (Hart et al., (200]].
Confidence intervals: Use DKW or Clopper- bounds
conservative thresholds with statistical guarantees on coverage Massart 1990 Clopper and Pearson 1934].
Cross-validation: Average selection thresholds over folds to reduce their variance and avoid overfitting single validation set Kohavi et al. 1995].
Shift Mitigate Distribution Shift:
Shift detection: Detect covariate shift via statistical two-sample tests such as MMD or KL divergence between feature distributions [Gretton et al., 2012 Rabanser et al. 2019].
Importance weighting: Correct mismatched data distributions with density ratio weighting, e.g., using kernel mean matching [Huang al.] [0066.
Domain adaptation: Finetune with in-domain examples use unsupervised techniques like
AdaBN or domain-adversariis training (DANN) [Ganin et al. 2016 Li et al. 2016].
Test-time adaptation: Adapt models at inference using entropy minimization (Tent) or batch norm recalibration restore accuracy under shift (Nado al. 2021 Wang et al. 2021].
thr Threshold-Selection Noise:
Bootstrap resampling: Estimate variability in the selection threshold by computing its standard error across bootstrap samples [Tibshirani and Efron 1993].
Smooth thresholds: Interpolate between adjacent scores or accept a random subset at the threshold to reduce coverage discontinuities (Angelopoulos and Bates 2021]
tie -ii-BBra & Score Quantization:
Higher precision: Use higher float precision (e.g.. FP32 or FP64) or more logits bits to distinguish close scores and avoid ties [Micikevicius al. 2018).
Dithering: Add tiny random noise scores before thresholding stochastically resolve ties and reduce instability.
Refrain from binning: Histogram binning (HQ) or Bayesian Binning into Quantiles (BBQ) often improve calibration but not selective classification performance (Naeini et al. 2015
Le-Coz et al. 2024..
Putting it all together. After addressing each bullet above, recompute your selective accu- racy-coverage curve and compare the oracle bound (Def. 3). Iterating over these steps will systematically shrink ((c) toward its irreducible floor.
D Experimental Details
D.1 Computational Resources
Our experiments were conducted on mix of GPU-cquipped compute nodes with varying hardware configurations. Some machines are equipped with Intel Xeon Silver CPUs (10 cores, 20 threads) and 128GB RAM, each hosting 4x NVIDIA GeForce RTX 2080 Ti GPUs with 11GB VRAM. Others feature AMD EPYC 7643 processors (48 cores, 96 threads), 512GB of RAM, and 4x NVIDIA A100 GPUs, each with 80GB VRAM.
D.2 Hyper-Parameters
We follow standard
training settings across all datasets. For each architec- ture dataset pair, we use fixed learning rate, weight decay, and batch size as detailed below:
SimpleCNN:
Learning rate: 0.01
Weight decay: 1 10-4
Batch size: 128
ResNet-18:
23

---

Learning rate: 0.1 for CIFAR datasets; 0.01 for Stanford Cars, Camelyon11
Weight decay: 5 x 10-4
Batch size: 128
NideResNet-5002:
Same settings as ResNet-18
Epochs:
200 epochs for all datasets except Camelyon17, which uses 10
Optimization: SGD with momentum 0.9, Nesterov enabled, and cosine annealing learning rate schedule.
Selective prediction methods:
MSP: Standard cross-entropy training
SAT: Cross-entropy pretraining for half of training epochs, followed by Self-Adaptive Training (momentum 0.9) with extra abstain class
All experiments use fixed random seeds for reproducibility and standard data augmentation per dataset (random crops, flips, normalization).
D.3 SimpleCNN Architecture
The SimpleCNN model is compact convolutional neural network used for experiments on lower- resolution image datasets. The architecture defined by the following sequence of layers:
A 3 convolution with 32 filters and padding followed by ReLU and x max-pooling.
A second 3 x 3 convolution with 64 filters and padding 1, followed by ReLU and 2 x max -ooolio..
A flattening layer, followed by fully connected layer with 128 hidden units and ReLU activation.
A final fully connected layer projecting to the number of output classes
Let s = iinput_sizee// / denote the spatial resolution after two pooling layers. Then, the full
model is:
SimpleCNN(x) Linear (128 - num coum_classes, ReLUo
Linear (64 s² 128) Flatteno
(64
MaxPool2d ReLU Conv2d(32 64)0
MaxPool2d ReLU Conv2d(3 + 32)
The number of output classes is set as follows:
10
for CIFAR-10,
100
for CIFAR-100,
num_clase clasess
with an optional extra class if extra_ class is True.
196
for Stanford Cars,
2
for Camelyon17,
The input size dataset-dependent and set to:
32
for CIFAR-10 and CIFAR-100,
input_size
224
for Stanford Cars, Camelyon
The model structure is summarized below:
SimpleCNN (
(net): : Sequential
(0): Conv2d (3, 82, kernel.size- (3, 3), stride-(1, 1), padding-1)
(1): ()

---

(2) MaxPool2d (xernel_size=2, stride=2)
(3) Conv2d (32, 64, s3zeel_ 3), stride=(1, 1), padding=1)
(4) ReLU()
(5) : MaxPool2d (kernel_size-2, stride-2)
(6) Flatten (start_dim-1)
(7): Linear (in__eatures 4096, out_
(8) ReLU()
12
(9): Linear (in_feeiel features=118, out out_features-nnn
13
D.4 Synthetic Distribution Shifts on Two Moons
To evaluate robustness under controlled covariate shifts, we apply a series of synthetic affine transfor- mations to the test set of the standard two moons dataset. Each transformation simulates distinet type of distribution shift:
Original: No transformation; the unperturbed test set.
Shear: shear transformation along the z-axis defined by:
1.25
x + 1.25y
Shear matrix S =
so that = Sx
(38)
y/
Rotation: A rotation by 30 degrees counterclockwise, using:
cos
sin θ
R
(39)
sin
cos
Translation: A shift of the input space by fixed vector:
1.0
x'
where
(40)
0.5
Each transformation is applied the test data matrix Xlest via matrix multiplication or translation, yielding the following test sets:
Original:
Xies
Shear: Xues ST
(41)
Rotation: Xueu R
Translation:
Xies + f.
These transformations create meaningful distribution shifts while preserving label semantics, enabling precise evaluations of model robustness under shift.
D.5 CIFAR-10C Severity Levels
For the CIFAR- -FC severity levels (1-5), we aggregate all 15 corruption types at given severity to form single validation set. For severity level I, we collect all corruptions labeled as severity / across following categories:
Noise: caussian_noise shot_ impulse sec_lse_noooe noise
Blur: defocus_blur, glass_bblullu blur, motiinnnllluu zoom blur
Weather: snow, frost, fog. brightness
Digital: contrast, #lastic_transfon pixelate, jpeg_ ppeg_coopression
This results single validation set per severity level 1, where each image sampled from one of these 15 corruptions applied at the specified severity.
25

---

SAT
DE
TEMP
1.0
1.0
1.0
1.0
Salectiv Accursey
0.9
Sallettiv Arcurary
0.9
Salediive Aucuracy
Saledtive Aucurary
0.9
0.8
0.7
Ra
0.6
M.5.RaNe552 EA20
0.8
M1d4RaaNe:d0. 3832
0.6
0.50
0.50
Covwrage
Comerage
Covwrage
Coverage
MSP
SAT
DE
TEMP
Raale-Ja L8 (ECO
ReaNe-Ji 1BCD-1026
TaaNe-JI IECE-0224
CNN (SCE-O.JA (42)
CNN (SCE=6.110)
CNO (2CD-60012)
CNN (DCE-0 5OE
0.8
0.8
0.8
Hidshalle::3
0.5
Widshaalleddd
Accurses
0.6
0.6
Aucurarr
0.5
Aucuraccy
0.6
0.4
0.00
0.50
1.00
0.00
0.50
1.00
0.00
0.50
1.00
0.00
0.50
0.75
1.00
Confidence
Confidence
Confidence
Confidence
Figure 5: Comparison between gap and calibration on CIFAR-100. Top: selective accuracy curves across four training methods and three architectures. Bottom: corresponding reliability diagrams (ECE shown in parentheses). Temperature scaling (TEMP) consistently improves calibration but does reduce the gap. By contrast, SAT and DE reduce the gap more models by improving the ranking.
for larger
E
Loss Prediction, Multicalibration, and Ranking Error
This appendix offers alternative perspective on the ranking error term Crank (c) by framing it as challenge of per-example loss prediction. Instead of building directly on the calibration discussion in Section 3.4 we show how the ability to forecast one's 's own 0-1 loss tightly controls the selective- classification gap. We formalize this connection through the recent theory of loss prediction [Gollakotal al.) 20255 and multicalibration (Hébert-Johnson et al., 2018). Throughout we adopt the binary-label conventions of Section 3.3 Extensions to multiclass losses likewise follow by one-vs-rest reduction.
E.1 Loss-Predictton Preliminaries
Let ((((r), = I({(x) * y} denote the 0-1 loss of fixed classifier h. A loss predictor LP: 1b - R maps auxiliary features o(x, fi) d estimate of E(h(x), y). The canonical baseline is the self-entropy predictor SEP(2) == [[[(h(r),,) h(x)] (which equals min 1 11137 p} for probabilistic p Mx)).
Definition 6 (Advantage over the self-entropy predictor). The (squared-error) advantage of loss predictor LP
Adv(LP)= == - SEP)) [[(e- LP)3].
(42)
A positive advantage means T.P forecasts the instance-wise loss better than the model itself.
Depending on Φ we obtain a hierarchy of predictors: prediction-only (ø = h(x))) input-aware (ø (h)):, and representation-awarr (Ø ((=)), refer to Gollakota et al. 2025 for a detailed taxonomy.
E.2 Multicalibration Background
Multicalibration is fine-grained notion of reliability that asks not just for global calibration, but for calibration conditional rich class of subpopulations a high level, a model is multicalibrated if its predicted scores match outcomes not only on average, but also across large collection of subsets defined by auxiliary variables or internal representations.
features [Hébert-Johnson al. 2018). At
26

---

Definition (Multicalibration Error), Let C be a class of weighting functions e: a - |-1, 1]. and let h: X [0. be classifier. The multicalibration error of with respect to is defined as
MCF(C,,) == max eEC
[[(Y c(6(x,
(43)
Each function C defines subpopulation slice of the input space via its support. The quantity MCE(C, f)) measures how well the model's predicted scores h(x) match the true label Y when weighted over these slices. When C' consists of indicator functions over discrete demographic sub- groups, small MCE(C, h)) implies groupwise calibration. More generally, if C includes continuous or date-dependent functions (e.g., based on internal features), low multicalibration error guarantees alignment between predicted and true outcomes across flexible set of conditions.
In our selective classification setting, (( h) may include the model's output confidence, the input or hidden representations from the network. The class C can be constructed accordingly to enforce calibration in feature-dependent risk-sensitive regions of the input space.
E.3 Loss Prediction > Multicalibration
now describe how ability to predict one's own 0-1 loss deeply connected to multicalibration. This perspective stems from the work of Gollakota et al. (2025). who characterize when model "knows its own loss" in terms multicalibration violations.
Let F be class of loss predictors LP: (xx,, h) + E [0, 1J. which estimate the 0-1 loss ((h(x). y) = (h((a))) * y} of a fixed classifier h. As discussed Section E. loss predictor considered good it has significant squared-error advantage over the model 's self-estimate SEP(z).
Remarkably, Gollakota al. 2025 show that this predictive advantage is tightly characterized by the multicalibration еттor of the model-measureed over derived weight class C' that depends on the predictors in F. The following theorem formalizes this connection:
Theorem 3 (Gollakota et 2025
Thm. 4.1 adaaadpped). For any function class F of loss predictors and the associated weight class C = =[ - -EPP) H((h(())) : F},
max Adv(LP) I MCE(C, h) S max Adv(LP),
(44)
LPEF
LPEP
where F" augments F with linear mixtures of SEP and elements of F. Thus non-trivial advantage is possible iff' h exhibits a multicalibration violation of similar magnitude.
This result bridges two domains: learning to predict loss (a regression task) and satisfying a general- ization constraint (calibration under distributional conditions). In the selective classification setting, this insight underpins Corollary I which shows that the ranking error and hence the gap oracle performance is tightly controlled by the model's 's ability to forecast its own mistakes.
E.4 Bounding the Ranking-Error Term Erank(c)
Theorem translates into bound on the ranking emrrr that drives the selective-classifiatioo gap.
Corollary 1 (Loss prediccre advantage controls mis-ranking). Fix coverage E (0, 1] and let Adv" := maXLPeF Adv(LP) for some input- class F. Then the ranking-crror term in
Theorem satisfies Erank (c) < Adv
Proof. Recall that A for (x) is in the top e-mass) and A, = (a : g(x, 1.). Write the difference indicator dx(x)) = (x) (x) E {-1,0, 0, 1} so Pr(5. =1) =)=Pr(dd 1) = c and E[5_] = 0.
27

---

Step 1: Express ranking error as a covariance.
With r(z) == 1(h(2) Y we have
rrak (c) = Bir A Bir Ad] = E [[(())x((X)]-
(45)
Step 2: Replace correctness by residual Y - -h(X). Because r = =1 and t= (Y - h1) for binary labels,
rd (1- (1- - - 1i (since E[6.] 0).
(46)
Hence
rank (c) | (xx)))((X)]
(47)
Step 3: Bound the covariance by multicalibration error. Define the bounded weight
function c²(x) := 6x(x); then [c'(x) S 1, so c* E (the weight class in Theorem 3). By
definition of multicalibration error,
E[ M(X)) c (X)]
MCE(C, h).
(48)
Combining 47 and 48 with S yields
rank (c) S MCE(C,
(49)
Step 4: Invoke the loss-prediction bound. Theorem
3 states MCE(C, h)
<
/maxpPer" Adv(LP). Since F C F' and vi monotone, finally have
(rank()) < Adv""
(50)
where the factor 2 absorbs the two-sided ++ F' constant in Theorem 3
Interpretation. Let 2 == maxLPer Adv(LP) be an upper bound on loss-prediction advantage. no loss predictor can beat self-entropy by more than e², then the selective classifier is within O(e) of the oracle at every coverage level. Conversely, a large loss-prediction advantage is a certificate of poor ranking and therefore of wide gap (cc).
Takeaway. Loss prediction and multicalibration offer principled lens on selective prediction: if you cannot beat your own self-entropy predictor, you are already close to the oracle frontier. Otherwise, the loss predictor pinpoints exactly which inputs are being mis-ranked and by how much, providing both a diagnostic and blueprint for tightening the caletive-classifieee gap.
E.5 Empirical Evaluation
To illustrate and validate our gap-decomposition framework, compared four selective-classfficttion strategies on C1, CIFAR- 100, and StanfordCars:
MSP: standard maximum sifimmmm-oftmaxxxc softmaxx-robabilitty abstention.
TEMP: MSP with post-hoc temperature scaling.
SAT: self-adaptive training, which co-trains abstain class.
DE: a deep ensemble of five MSP models.
For each method, we first trained a ResNet-18 on 80% of the training set (using the usual data augmentations and held-out 20% for LP fitting). At each epoch then:
1. Extract the 512-dim "penultimate" feature vector ((x) from the ResNet backbone (or its ensemble average).
2. Compute the model's self-entropy score
SEP(x) = max (z( with P)(x) = softmax, (lggits())/T))
28

---

CIFAR-10
CIFAR-100
StanfordCars
Becitvee Advuntep (ikdrou
0.025
0.020
MSP
0.010
TEMP
0.005
D3
10
15
Epoch
Epoch
Figure 6: Relative LP advantage over training epochs across datasets. For each method, plot the shift in test-set advantage Ad (t) relative to epoch 1, indicating how much additional ranking signal the loss predictor learns over time, Larger values imply greater misalignment between the model's s confidence and correctness.
Train small MLP ((xx) + غ [0, to minimize B[(ē- - TT((()) + v}) 20% split.
on the held-out
Measure the LP advantage on the fest set,
AdVtest E[(e SEP) E [{(e LP) f= 1(((z) * y}
and record its shift relative to the first epoch A Advvestt (1) Advtost (1) AdV (1).
LossePredintion Network. Below is the Py PyTorch representation of our two-hidden-layer LP head. takes the ResNet features (optionally concatenated with SEP) and regresses the per-example ()-1 loss via mean-squared error.
LossPredicts ctor
(net): : Sequential
(0) Linear (1t_ out out_features-128, bias-True)
(1) ReLU()
(2) : propout(p=0.55
6
(3): Linear (in_features=128, out out_features-f4, bias=True)
(4) ReLU()
(5) propout(p=0.5)
(6) Linear (in_ featureee,, out_features-l, bias=True)
10
11
Key observations. On CIFAR-10 10 (left panel of Figure 6). all methods stay close to zero AAdv indicating that the model's own confidence scores already capture most of the available ranking signal. On CIFAR-100 100 (middle panel), MSP and TEMP exhibit large positive shifts in I.P advantage, suggesting that a dedicated loss predictor can substantially improve canking-consice gap from the oracle. By contrast, SAT and DE remain near zero, indicating that their confidence scores are already well aligned with correctness. On StanfordCars (right panel), the gap widens even further: both MSP and TEMP allow for significant gains via loss prediction, and even SAT leaves nontrivial room for improvement. Only DE consistently resists such gains, implying that deep ensembling is uniquely effective preserving reliable ranking in high- nigh-variance domains.
with larger
Conclusion. These results match our theory perfectly: whenever the LP head cannot improve on self-entropy, the selective classifier is effectively oracle-optimal; whenever it can, the size of that advantage precisely quantifies the remaining ranking error and the gap from the ideal frontier.
Additional Results
F11
Calibration Experiments
E-AURC vs ECE We provide additional comparisons on more datasets (CIFAR-10 and Stanford-
Cars) the relationship between the selective classification gap and the model's expected calibration
29

---

error. See Tables and for exact results. In general, our conclusions from Section 4.2 hold here as well: while temperature scaling (TEMP) improves ECE over MSP, does not reduce the selective classification gap- gap-underscoorin the limits of monotone calibration. In contrast, SAT and deep ensembles (DE) improve both ECE and gap by altering the ranking, confirming that only re-ranking methods yield meaningful gains in selective performance.
Table 2: Experiments on calibration across model classes CIFAR-10. Similar Table
CNN
ResNet-18
WideRes
MSP
TEMP
SAT
DE
MSP
TEMP
SAT
DE
MSP
TEMP
SAT
DE
E-AURC
0.024
0.023
0.019
0.016
0.004
0.004
0.003
0.002
0.003
0.003
0.002
0.002
ECE
0.075
0.025
0.035
0.010
0.025
0.014
0.016
0.007
0.027
0.022
0.019
0.010
Table 3: Experiments calibration across model classes on Stanford Similar as Table
CNN
ResNet-18
WideResNNt-500
MSP
TEMP
SAT
DE
MSP
TEMP
SAT
DE
MSP
TEMP
SAT
DE
B-AURC
0.176
0.177
0.166
0.159
0.030
0.029
0.26
0.022
0.026
0.026
0.23
0.020
ECE
0.110 0.025
0.058
0.025
0.040
0.027
0.037
0.025
0.017
0.017
0.015
0.015
F2
Extension to Large Language Models
Our primary experiments focus on vision and synthetic datasets, where uncertainty and selective prediction have well-established definitions and evaluation metrics. In these domains, notions such as confidence calibration, abstention rates, and oracle coverage curves provide a clear framework for measuring reliability. Extending the same analysis to large language models, however, presents new difficulties as outlined in particlar by the following two challenges:
1. Uncertainty for generative models remains ill-defined. Even for classification-stylee prompts, the community has not fully converged on how to translate sequence-level proba- bilities into abstention scores.
2. Prompting artefacts add variance. Small changes in in-context examples or decoding settings can swamp the effects we wish isolate.
Despite these challenges, we have added a focused set of LLM experiments to demonstrate that our five-term decomposition still diagnoses the gap.
F.2.1 Approximation Error - Scaling from 4B + 12B
We evaluate Gemma 3-IT 4B and 12B [Team et al. 2025 on ARC-Challenge (ARC-C, 25-shot) (Clark]
al. 2018 and MMLU (5-shot, top- 1) Henddyykks et al. 2021 using the standard MSP score on the first answer token (no further fine-tuning).
Table 4: Accuracy comparison across model scales.
Model
ARC-C Accuracy MMLU Accuracy
Gemma 3-IT 4B
56.2%
59.6%
Gemma 3-IT 12B
68.9%
74.5%
Observation. Consistent with our vision experiments, increasing capacity reduces the gap.
F.2.2 Bayes Error - Separating Easy vs. Noisy Questions
Following the MMLU-Pro protocol [Wang et al. 2024), J, we partition the validation set into the easiest 25% and noisiest 25% questions (based on human LLM agreement).
30

---

Table 5: Selective-Classffictiion gap area (lower better).
Model
ARC-C E-AURC
MMLU E-AURC
Gemma 3-IT 4B
0.114
0.107
Gemma 3-IT 12B
0.091
0.082
Table 6: B-AURC across data difficulty levels on Gemma 3-IT 4B.
Split
E-AURC
Full MMLU
0.107
Easiest quartile
0.018
Noisiest quartile
0.316
Observation. When intrinsic Bayes noise is low (easy questions), the gap nearly vanishes; when noise is high, the gap widens.
F2..3 Ranking Quality - Calibration Re-ranking
We keep the backbone fixed Gemma 3-IT 4B and compare the ranking quality of the following uncertainty scores:
MSP,
TEMP (scalar 7 fitted on held-out validation split),
DE of five LoRA-fine-tur replicas.
Table Calibration and gap performance across ranking methods on Gemma 3-IT 4B.
ARC-C
MMLU
MSP
TEMP
DE
MSP
TEMP
DE
E-AURC
0.127
0.126
0.087
0.122
0.122
0.079
ECE
0.171
0.092
0.056
0.135
0.084
0.052
Observation. Temperature scaling lowers ECE yet leaves the gap untouched; the ensemble both calibrates and improves ranking, shrinking the gap.
Summary. These results confirm that decomposition extends LLMss capacity, Bayes noise, and ranking quality each contribute measurable terms. A full generative-text study (e.g., free-form question answering code synthesis) will require new abstention semanties and we leave more thorough treatment for future work.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract and intro reflects the contributions accurately.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section S
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

We provide short proof in the main paper and a more extensive analysis and proof in Appendix B
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See Appendix
D
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
33

---

Answer: [Yes]
Justification:
We include our full experimental suite and details for reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipp/.cc/pullc//uuuii wwwiii/gguie///eeeeeeoecc CoodeSubeiiie ssi ssionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Appendix D.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
All of our reported results are reported as mean values over random runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
34

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix D.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htttps://euur: httppss:///uips...o/m http://........oc...c..ccciieeaa
Answer: [Yes]
Justification:
The paper conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
35

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
We not releasing any assets that require any specific safeguards. Guidelines:
Justification:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited related work appropriately.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, snperswwihcoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
36

---

Answer: [Yes]
Justification:
We are releasing our codebase to aid reproducibility.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We did not conduct any crowdsourcing and/or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We did conduct any studies requiring IRB approval.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
37

---

Justification:
We only used LLMs to help with paper editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http:ses//eerrpppocccc//o/rrrees for what should or should not be described.
38


Reviews:
[{"review_id": "r8vxDkt78e", "reviewer": "Reviewer_VZwS", "summary": "This paper tackles the problem of selective classification: a model can choose to abstain from making a prediction on a subset of examples. They first define a formalism for studying this problem: given a classifier h and a scoring function g, we abstain if g is lower than a threshold tau. They then define the central quantity of interest: the selective classification gap \\Delta(c) (given a coverage level c) which measures the gap between the performance of the system (h,g) compared to the optimal selective classification decisions. The main contribution is to decompose this gap into 5 factors (irreducible gap, capacity, ranking, data, and misc) formally and study each of these terms and the impact of common solutions (calibration) on this gap. They also provide empirical results to help illustrate the theory.", "questions": "- Can you expand the experiments to include newer models and datasets, notably with LLMs?\n\n- How does this analysis and how does the gap Delta(c) relate to a gap we can compute in practice based on the oracle selective classification rule? Or is there other ways to compute this gap?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Quality: This paper is very solid overall. The theoretical results are sound. The empirical methodology is sound, however, the experiments rely on old datasets and models. Not to be cliche, but it would have been good to scale the experiments with LLM based classifiers (one can still extract logits from those). \n\nClarity:\nOverall the paper is very easy to read and the presentation is excellent.\n- please make Figure 1 bigger so that its easier to read\n\nOriginality:\nWhile the presentation of the results is very well done and thoughtful, I am not sure there is much novelty in the decomposition presented. The main contribution here is framing of the selective classification gap as a quantity, the decomposition and the bound is then immediately obtained from prior work in learning theory. In practice, we can't compute the true gap but have to resort an upper bound that relies on whether the model got each example right or wrong (abstain from only wrong examples as much as needed based on coverage level c). Therefore, I am not sure how the bound can be used to inform practice. \n\nSignificance:\nI think this paper is very enjoyable to read and will help future researchers in more easily framing the area. However, I do not believe the contributions of this paper make much of a contribution to the theory or practice of selective classification.", "was_revised": true, "final_justification": "I am keeping my score to a borderline reject after reading the author's rebuttal. Overall, I think the theory contribution is too weak to stand on it's own and the empirical evaluations even after the added experiments are too limited to supplement the theory enough. For future submissions, I encourage the authors to make the paper stronger with a much more comprehensive evaluation or innovations on the theory side.", "rebuttal": "We thank the reviewer for their thoughtful feedback and **kind words about the paper’s clarity, presentation, and theoretical soundness**. We appreciate their **suggestions regarding experimental scope, perceived novelty, and practical utility of the decomposition**. Below, we clarify our contributions and address these concerns—most notably by **adding results on LLM-based models** and **elaborating on how the gap informs practice**.\n\n---\n\n``1) Novelty and relevance of the presented decomposition.``\n\nWe thank the reviewer for their kind words regarding our exposition. We respectfully contend, however, that our work offers both novel theoretical insight and a practical diagnostic that was not available before. We address each of your concerns in turn.\n\n- **Coverage-uniform finite-sample bound**. To our knowledge, no previous result provides a decomposition that *simultaneously* (i) holds **for every coverage level $c$**, (ii) is stated **at finite sample size n**, and (iii) partitions the gap into four interpretable sources. Existing selective-risk PAC bounds fix $c$ *a priori* and collapse ranking and approximation into a single term.\n- **Separation of ranking error**. By isolating $\\varepsilon_\\text{rank}$​ we show that any **monotone post-hoc calibration** (temperature scaling, isotonic, histogram) cannot shrink the gap (§4.4). This clarifies empirical disagreement reported in recent studies.\n- **Interpretable formalization**. Through formalizing the gap decomposition in Equations (14) and (15) we provide the first error budget for abstaining systems. Researchers can now ask *which* piece limits current methods and design targeted algorithms, just as bias-variance decompositions shaped classical supervised learning.\n- **Controlled experiments for each bound quantity**. We provide a set of controlled experiments that (i) verifies our theoretical insights; and (ii) allows us to *independently* measure and target each of the contributing bound components.\n\nWe hope this concise clarification demonstrates the novelty and practical value of our contribution and addresses your concern about significance. If the reviewer is aware of prior work that offers the same combination of formal decomposition, tight upper bounds, and empirical disentanglement, we would be grateful for the references and will gladly acknowledge and cite them in our revised version.\n\n---\n\n``2) Can you expand the experiments to include newer models and datasets, notably with LLMs?``\n\nWe agree that large language models (LLMs) are an important test-bed for our framework.  Achieving the same level of *controlled* experimentation as in our synthetic and vision experiments, however, is inherently harder for three main reasons:\n\n1. **Uncertainty for generative models remains ill-defined.**  For classification-style prompts but even more so for generative tasks in particular, the community has not fully converged on how to translate sequence-level probabilities into abstention scores.\n2. **Prompting artefacts add variance.**  Small changes in in-context examples or decoding settings can swamp the effects we wish to isolate.\n3. **Training pipelines are opaque and inconsistent.** Many LLMs are trained with undisclosed data mixtures, objectives, or fine-tuning procedures—introducing uncertainty about what distributions the model has implicitly seen and how it generalizes.\n\nDespite these challenges, we have added a focused set of LLM experiments—summarized below and plotted in the revised manuscript—to demonstrate that our decomposition still diagnoses the gap.\n\n**1. Approximation error — scaling from 4B → 12B**\n\nWe evaluate Gemma 3-IT 4B and 12B [1] on ARC-Challenge [2] (ARC-C, 25-shot) and MMLU [3] (5-shot, top-1) using the standard MSP score on the *first answer token* (no further fine-tuning).\n\n| Model           | ARC-C Acc. | MMLU Acc. |\n| --------------- | ---------- | --------- |\n| Gemma 3-IT 4 B  | 56.2 %     | 59.6 %    |\n| Gemma 3-IT 12 B | 68.9 %     | 74.5 %    |\n\n| Model           | ARC-C Gap Area | MMLU Gap Area |\n| --------------- | -------------- | ------------- |\n| Gemma 3-IT 4 B  | 0.114          | 0.107         |\n| Gemma 3-IT 12 B | 0.091      | 0.082     |\n\n*Observation.*  Increasing capacity reduces the gap, mirroring the vision results on model expressiveness.\n\n**2. Bayes error — separating easy vs. noisy questions (4B)**\n\nFollowing the MMLU-Pro protocol [4], we partition the validation set into the *easiest 25 %* and *noisiest 25 %* questions (based on human–LLM agreement).\n\n| Split             | Gap Area  |\n| ----------------- | --------- |\n| Full MMLU         | 0.107     |\n| Easiest quartile  | 0.023 |\n| Noisiest quartile | 0.316 |\n\n*Observation.*  When intrinsic Bayes noise is low (easy questions), the gap nearly vanishes; when noise is high, the gap widens.\n\n**3. Ranking quality — calibration vs. re-ranking (4B)**\n\nWe keep the backbone fixed (Gemma 3-IT 4B) and compare:\n\n- MSP (raw logits)\n- Temp. scaling (scalar $T$ fitted on a held out validation split)\n- Deep ensemble of five LoRA-fine-tuned replicas\n\n| Metric       | ARC-C MSP | ARC-C TEMP | ARC-C DE | MMLU MSP | MMLU TEMP | MMLU DE |\n| ------------ | --------- | ---------- | -------------- | -------- | --------- | ------------- |\n| ECE      | 0.171     | 0.092  | 0.056          | 0.135    | 0.084 | 0.052         |\n| Gap Area | 0.127     | 0.126      | 0.087     | 0.122    | 0.122     | 0.079     |\n\n*Observation.*  Temperature scaling lowers ECE yet leaves the gap untouched; the ensemble both calibrates *and* improves ranking, shrinking the gap—recapitulating our CIFAR-100 findings.\n\n**Overall, these results confirm that our decomposition extends to LLMs**: capacity, Bayes noise, and ranking quality each contribute measurable terms.  A full generative-text study (e.g. free-form question answering or code synthesis) will require new abstention semantics and is left for future work.\n\n[1]: Team, Gemma, et al. \"Gemma 3 technical report.\" arXiv preprint arXiv:2503.19786 (2025).\n\n[2]: Clark, Peter, et al. \"Think you have solved question answering? try arc, the ai2 reasoning challenge.\" arXiv preprint arXiv:1803.05457 (2018).\n\n[3]: Hendrycks, Dan, et al. \"Measuring massive multitask language understanding.\" ICLR 2021.\n\n[4]: Wang, Yubo, et al. \"Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.\" NeurIPS 2024.\n\n---\n\n``3) How does this analysis and how does the gap Delta(c) relate to a gap we can compute in practice based on the oracle selective classification rule? Or is there other ways to compute this gap?``\n\nOur analysis is built around the *empirical* gap that **practitioners already compute**: for any held-out dataset with ground-truth labels, we obtain **(i)** the model’s empirical accuracy–coverage curve by thresholding its confidence scores, and **(ii)** the oracle curve by retroactively ranking the *same* examples in descending order of realized correctness (i.e., 1 for correct, 0 for incorrect)—exactly the “oracle selective-classification rule” used in prior work.  The coverage-uniform gap $\\widehat{\\Delta}(c)$ we decompose is simply the point-wise difference between these two curves, or, when desired, its integral over $c$.  Thus every term in our bound upper-bounds a quantity that is **directly observable from data**; no additional assumptions or simulations are required.  Alternative oracle choices (e.g. Bayes posterior ordering when noise estimates are available) plug into the *same* recipe and our decomposition still applies verbatim—only the definition of the benchmark curve changes.  In short, the gap we analyze is exactly the gap researchers and practitioners already plot, and our results explain *why* that observable gap arises and which interventions (capacity, non-monotone calibration, shift-robustness, etc.) can provably shrink it.\n\n---\n\n``4) Please make Figure 1 bigger so that its easier to read``\n\nWe have fixed this in our updated paper draft.\n\n---\n\n**We hope that our rebuttal has addressed the reviewer’s questions and concerns and that the reviewer considers raising their individual and overall score(s). Our work has already significantly improved thanks to their feedback. We are happy to further engage with them during the discussion period!**"}, {"review_id": "8BVwWFiEis", "reviewer": "Reviewer_2rvR", "summary": "The paper analyzes the problem of learning selective classifiers, which are classifiers that have the option to abstain on inputs deemed unreliable, reducing the risk of costly errors. Their analysis is based on the gap between the accuracy-coverage tradeoff of the classifier, which quantifies the degradation of performance as the model accepts a broader set of inputs, and a perfect-ordering upper bound, which assumes an oracle that orders the example according to the likelihood of error by the classifier. They show that this gap can be decomposed into five (not necessarily non-overlapping) components that measure different potential sources for this gap: Bayes noise, approximation error, ranking error, statistical noise and implementation of shift-induced slack. Through controlled experiments with both synthetic and real data, they isolate these components, showing the effect of different design choices for selective classifiers that impact the accuracy coverage tradeoff gap. In particular, their results show that calibration methods that are monotonic (i.e. do not affect ranking) do not improve the accuracy coverage tradeoff gap, which was an open question in previous research with contradictory results.", "questions": "How is the selective classification gap affected by overfitting? What component does it affect? Have you seen cases in practice where increasing model capacity resulted in a worse accuracy coverage tradeoff?\n\nIn the abstract you say that you provide concrete design guidelines for building selective classifiers. Could you list exactly are these concrete guidelines? (I haven’t read the Appendix).", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n-The paper is well-written, with a good description of the area and recent research, positioning well the contributions of the paper, the conclusions derived from their theoretical and empirical validation and the limitations of the work. \n-The experimental analysis both on synthetic and real data is very well designed to show the effect of the different sources of errors \n-The paper presents an original analysis of the selective classifier learning problem, introducing new concepts (the selective classification gap) and different ways to understand the impact of different design choices.\n\n\nWeaknesses:\n-Although the formulation is clean and elucidates the problem, it is a little hard to see practical, non-obvious, design guidelines for building selective classifiers coming out of this formulation. Even monotonic calibration could in principle improve approximation error, so we can only find out whether it works through controlled experiments (as done in the paper). \n-The paper does not mention overfitting and the effects that this may cause on the selective classification gap — all the experiments show increased performance when the capacity of the model increases in terms of the accuracy-coverage tradeoff gap, but we know that in practice this is not always true because of overfitting.", "was_revised": true, "final_justification": "I had originally recommended (4) accept. After reading the other review and rebuttals, I saw that some of the other reviewers had lower scores due mainly to considerations about novelty and clarity. In particular, they requested clarifications about the relationship between the proposed theoretical formulation and previous formulations. While I agree that these were weaknesses in the paper that I had overlooked, I believe that the authors have addressed the concerns satisfactorily in their rebuttal and can improve the final version, so I will keep my score.", "rebuttal": "We thank the reviewer for their thoughtful and constructive positive review. We are glad the reviewer finds the **paper is well-written, the analysis original, and the experiments carefully designed**. We appreciate the **concerns raised regarding the practical design takeaways and the role of overfitting**, and we address both points in detail below.\n\n---\n\n``1) It is hard to see practical, non-obvious, design guidelines for building selective classifiers coming out of this formulation?``\n\nWe want to clarify that our **contribution goes beyond establishing yet another bound**: by decomposing the gap into five *interpretable* terms—Bayes noise, approximation, ranking, statistical, and misspecification slack—we expose concrete levers that practitioners can manipulate. In particular, the new ranking term clarifies *why* monotone calibration alone cannot close the gap: it leaves the score ordering intact, so any benefit is limited to full-coverage accuracy rather than selective performance. Conversely, non-monotone or feature-aware calibrators and model-ensemble methods *do* tighten the ranking term, while increasing capacity or distilling from larger teachers targets approximation error, and robust-loss or repeated-label strategies reduce Bayes noise. We distilled these insights into a step-by-step practitioner checklist in Appendix C, which, to our knowledge, is the first set of explicit, empirically validated design guidelines for approaching oracle-level selective classification—transforming our theoretical formulation into a practical toolkit.\n\n---\n\n``2) What is the role of overfitting on the SC gap?``\n\nOur results are reported exclusively on unseen test data, so the kind of **overfitting to the training set** that inflates in-sample accuracy is automatically folded into the finite-sample decomposition: if a larger model memorizes training idiosyncrasies, its *test-time* full-coverage accuracy plateaus while its confidence scores sharpen, thereby enlarging the *ranking* term $\\varepsilon_{\\text{rank}}(c)$ and, when optimization becomes unstable, the *miscellaneous* slack $\\varepsilon_{\\text{misc}}(c)$. Overfitting to a **narrow training distribution** manifests similarly under shift, and we isolate that effect in §4.3 where $\\varepsilon_{\\text{misc}}$ grows with corruption severity. Overall, we observe that capacity improves the gap only when it also preserves good ranking and generalization—otherwise the decomposition correctly attributes the performance loss to ranking or optimization slack. We are happy to include this discussion in the paper.\n\n---\n\n``3) Where are the promised concrete design guidelines for selective classifiers?``\n\nWe include these concrete guidelines in Appendix C.\n\n---\n\n**We hope that our rebuttal has addressed the reviewer’s questions and concerns and are happy to further engage with them during the discussion period!**"}, {"review_id": "2cWbFhLkLm", "reviewer": "Reviewer_9vHG", "summary": "They formulate the performance difference of a selective classifier to its perfect ordering scoring (optimality) measured with the *AUACC* (Area Under the Accuracy–Coverage Curve) as the *coverage-uniform selective-classification gap*. Further they differentiate the finite-sample decomposition of the gap dividing it into five parts: 1)“Bayes noise/irreducible”, 2)“appoximation error/capacity”, 3)“ranking error/ranking”, 4)“statistical noise/data” and 5)“implementation or shift-induced slack/optimization & shift”.\n\nThey provide a theoretical analysis of closing the gap and show that only calibration that modifies the ranking improves selective classification, with a specific focus on temperature scaling and the reason for its limited effect on the AUACC.\n\nFinally they support their theoretical analysis by means of empirical experiments on: the Toy Two Moons dataset, CIFAR10 & CIFAR100, Stanford Cars and Camelyon 17-WILDS.", "questions": "For questions, I refer to the Weaknesses, where I already detailed ways to improve the quality of the work. In general, I am positive about the value of this work, but see it more as introducing a thought framework and guiding beginners in the field, which could be greatly improved by aggregating findings of previous work, as I see the overall novelty and scale of their experiments as limited.", "limitations": "yes, all limitations except the ones mentioned in the Weaknesses section are addressed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- The paper provides a clear theoretical thought framework for silent failure cases in selective classification, which the selective classifier does not catch due to the five aforementioned sources of errors.\n- The paper is well structured and beautiful if I may say so. Especially, the takeaway boxes substantially improve parsing of the text.\n- The appendix is well organized and, in combination with the code provided in the supplementary material, which is also well structured, seems to provide all the information necessary to replicate their empirical results.\n- All of this combined can make it a nice starting point for people working themselves into the field of selective classification and on which areas to focus to improve selective classifiers, especially in combination with Appendix C, the “Practitioner Checklist for Tightening the Selective-Classification Gap”.\n\n## Weaknesses\n**Formulation**:\n- Given the stated purpose in (lines 35-36): “For my finite model on finite data, what aspects of the learning setup will actually move my trade-off curve closer to the upper bound?” \n- The algebraic decomposition of the selective classification gap (here a) in equation 27 boils down to an inequality of the form $a <= a + b$ without a clear reasoning why we should care about $b$ if the goal is to improve $a$ as the upper bound appears to be defined as the perfect ranking upper bound, which is defined as $a$.\n    - Please clarify what the exact goal of the finite-sample decomposition is in a simple manner, and especially explain the reason for using this decomposition.\n- As the formulation of the selective classification gap seems to be very similar –if not identical– to that of the e-AURC [D], this should be discussed.\n- Overall, the theoretical formulations could be greatly improved by clarifying the relations to existing literature and metrics.\n\n\n**Related Works**:\n- Currently the results of this paper are not discussing some recent works which have been highly prominent in the community like [A] (ICLR Oral), [B] (NeurIPS Spotlight) and [C] (NeurIPS). This substantially reduces the quality of the text and reduces its value as starting point for people to familiarize themselves with the topic of selective classification.\n    - Specifically [B] introduces the AUGRC which should be discussed in Section 2 paragraph “Accuracy-coverage tradeoff evaluation.” as it essentially reweights the AURC with the coverage which reduces the impact of the very low coverage area.\n    - The finding of [A] based on a large-scale empirical study reveals that “None of the prevalent methods proposed in the literature is able to outperform a softmax baseline across a range of realistic failure sources,” which would help motivate the selection of MSP in Section 4.\n        - Further, the reasoning in [A] goes against the reasoning here because the goal is to reduce the selective classification gap instead of the best overall classifier. This is worth discussing what your view means for the practitioner.\n    - A detailed discussion of the difference to the e-AURC [D] is missing where it is analyzed based on formulas.\n- See last point in “Formulation”.\n\n\n**Novelty**:\n- Playing devils advocate here: That label noise (Bayes), overall model performance (capacity), sample size (data) and optimization errors or distribution shifts (optimization & shifts) influence the overall model performance and that uncertainty estimation (ranking) influences the selection process is hardly new to anyone in the field.\n    - I see the main value in the thought framework and handing solutions. Based on this I believe that more citations to the work referenced as solutions in App. C would greatly increase the value of this work as well as the works which indicate that this is a reliable solution for the use case of selective classification to make it more valuable for future reference.\n- Further the novelty that methods which induce changes in the ranking lead to substantial improvements is rather limited as this is true for basically any metric which is based on ordering of scores (e.g. AUROC, AUPRC…)\n\n\n**Evaluation**:\n- I do not agree with the authors that they perform a “large scale” evaluation. The evaluation works like [A] and [B] spans more methods and more datasets while [C] uses more methods and ImageNet instead of the small scale CIFAR, Stanford and Camelyon datasets. \n    - Solutions: (1) rephrase this, (2) In an optimal scenario I encourage the authors to reinterpret the results from [A], [B] and [C] in their framework.\n- General weaknesses with regard to the evaluation: They only evaluate four different selective classification methods 1) MSP, 2) Temp. Scaled Softmax, 3) Self-Adaptive Training, 4) Deep Ensemble. All of which are simple to implement and 1 & 2 are essentially identical with regard to the overall setup. Further many of the Practical Guidelines in App. C are not explicitly shown (e.g. how RandAugment improves the model [see point above for a solution not necessitating extension of the evaluation]).\n\n\n**Minor points**:\n- In Sections 4.1 and 4.2 in Setup: there are too many enumerations with (i) to (iii) I would appreciate to do (i)...(iii) and (a)... (z) to make the distinction across them clearer.\n\n\n\n### Citations\n[A] Jaeger, P. F., Lüth, C. T., Klein, L., & Bungert, T. J. (2023). A call to reflect on evaluation practices for failure detection in image classification. The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=YnkGMIh0gvX\n\n[B] Traub, J., Bungert, T. J., Lüth, C. T., Baumgartner, M., Maier-Hein, K., Maier-hein, L., & Jaeger, P. F. (n.d.). Overcoming common flaws in the evaluation of selective classification systems. The Thirty-Eighth Annual Conference on Neural Information Processing Systems. https://openreview.net/forum?id=2TktDpGqNM&noteId=MJx9cPynNy\n\n[C] Mucsányi, B., Kirchhof, M., & Oh, S. J. (2024). Benchmarking uncertainty disentanglement: Specialized uncertainties for specialized tasks. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, & C. Zhang (Eds.), Advances in neural information processing systems (Vol. 37, pp. 50972–51038). Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2024/file/5afa9cb1e917b898ad418216dc726fbd-Paper-Datasets_and_Benchmarks_Track.pdf\n\n[D] Yonatan Geifman, Guy Uziel, and Ran El-Yaniv. Bias-reduced uncertainty estimation for deep neural classifiers. arXiv preprint arXiv:1805.08206, 2018", "was_revised": true, "final_justification": "The authors made a great rebuttal; therefore, I increased my score. Several issues with the work crystallized during the rebuttal, which were addressed by the authors. While doing that, however, potentially novel issues became clear due to the multitude of changes. Especially, their main derivation (eq.27) is skipping some important steps and was not adequately described in the submitted version.\n\nDespite my leaning towards acceptance, the multitude of changes makes it hard for me to grasp the manuscript in its updated form. Therefore, I believe another round of reviewing might be beneficial.", "rebuttal": "We sincerely thank the reviewer for their in-depth feedback. We are especially grateful for their recognition of the paper’s **structured presentation, clear theoretical framing,**  and its potential as a **starting point for newcomers to SC**. At the same time, we acknowledge the reviewer’s **concerns regarding the novelty, formulation, and scope of evaluation**. We have taken these suggestions seriously and provide clarifications, additional related work discussion, and rewordings throughout our rebuttal to address them in detail.\n\n---\n\n``1.1) Purpose of lines 35-36.``\n\nWe would appreciate additional clarification, as we are unsure what the reviewer is specifically asking.\n\n---\n\n``1.2) Algebraic decomposition of Equation 27.``\n\nWe apologize for the confusion—Eq. 27 is *not* intended as the tautological inequality $a \\le a + b$. Rather, **by adding and subtracting $\\mathbb{E}[\\eta_h \\mid A_c^{\\star}]$, we obtain an exact identity**\n\n$$\n\\Delta(c) = \\varepsilon_{\\text{rank}}(c) + \\varepsilon_{\\text{approx}}(c) + \\varepsilon_{\\text{Bayes}}(c),\n$$\n\nwhere each non-negative term isolates a distinct failure mode:  \n**(i)** $\\varepsilon_{\\text{Bayes}}$ is the irreducible label noise among the accepted points;  \n**(ii)** $\\varepsilon_{\\text{approx}}$ captures errors that persist even if ranking were perfect (e.g., due to limited model capacity or training); and  \n**(iii)** $\\varepsilon_{\\text{rank}}$ quantifies the loss due solely to mis-ordering examples.\n\n**This factorization is therefore diagnostic, not merely algebraic**: by measuring the three components in Sec. 4, we show, for instance, that ranking error dominates on CIFAR-100, pointing practitioners toward improving uncertainty estimation rather than enlarging the model. We have added a concise explanation of this objective immediately before Eq. 27 to prevent further misunderstanding.\n\n---\n\n``1.3) Relationship to e-AURC [D].``\n\nThe reviewer is correct that our formulation of the SC gap shares similarities with the definition of the Excess-AURC (E-AURC) introduced in [D], as both quantify deviations from an ideal abstention policy. However, **the two differ in both granularity and perspective**. Our gap measures pointwise accuracy shortfall at a fixed coverage level relative to a perfect oracle with the same base accuracy, whereas E-AURC aggregates excess risk across all rejection levels relative to an optimal ranking function. To clarify this connection, we have explicitly acknowledged the similarity between our gap and E-AURC directly following Definition 4 in the updated draft. We would also be happy to include a more detailed comparison of the two performance metrics in the appendix.\n\n---\n\n``2.1) Discussion of recent works [A,B,C].``\n\nWe thank the reviewer for pointing us to these influential recent works. **We have incorporated their insights in the revised manuscript** as follows:\n\nIn §2 “Accuracy–coverage trade-off evaluation” we added:\n\n> Recently, [B] introduced the Area Under the Goals-Reweighted Curve (AUGRC), which multiplies accuracy by coverage before integration, thereby down-weighting the very-low-coverage tail that can dominate the traditional AURC.\n\n> Complementary to metric design, [C] provides a comprehensive benchmark that disentangles sources of uncertainty, underscoring the need for task-aware evaluation of selective classifiers.\n\nIn §4.2 “Setup” we added:\n\n> Our inclusion on the maximum-softmax-probability baseline is motivated by the large-scale study of [A], who find that MSP proves to be a hard-to-beat baseline.\n\n---\n\n``2.2) [A] goes against our reasoning.``\n\nCertainly—here is a more concise version that integrates your collaborator’s feedback while keeping the overall length close to the original:\n\n---\n\nFirst, we note that **\\[A] did not evaluate Self-Adaptive Training (SAT) or Deep Ensembles (DE)**. Instantiated to support our theoretical formulation, our empirical results in §4.2 and Appendix E.5 show that **SAT and DE consistently narrow the selective-classification gap better than MSP**—by improving ranking quality—and are also more reliable at predicting their own loss. Second, we agree that practitioners ultimately care about the best possible performance at the chosen coverage. In that sense, **minimizing the selective-classification gap complements accuracy maximization**: it reveals whether remaining error stems from noise, ranking, or capacity limits, and thus which lever to pull (e.g., stronger backbone vs. better uncertainty scoring). This disentanglement also clarifies whether improvements from compute-heavy methods like DE come from gap reductions or generic accuracy boosts—helping practitioners identify more efficient alternatives. Please let us know if we have misinterpreted your concern.\n\n---\n\n``3.1) Role of isolated gap quantities is hardly new to anyone in the field.``\n\nWe agree that practitioners often recognize each factor qualitatively. However, to our knowledge, no prior work **(i)** formalizes *all five* influences in a single finite-sample, coverage-uniform inequality, **(ii)** proves that every SC gap can be *exactly upper-bounded* by these terms, and **(iii)** demonstrates—via controlled experiments—that each term can be *independently* measured and targeted. Our decomposition thus **turns familiar intuitions into a quantitative error budget that guides design**: it shows why monotone post-hoc calibration cannot improve ranking, when added capacity yields diminishing returns due to Bayes noise, and how shift-induced slack can limit SC gains. If the reviewer knows of prior work with this combination of decomposition, bounds, and empirical disentanglement, we would be grateful for the reference and acknowledge it. See extended discussion for Reviewer VZwS.\n\n---\n\n``3.2) More citations in App. C and in the main body.``\n\nThis is a good suggestion; we agree with the reviewer that the **checklist would benefit from additional citations** to prior work that has proposed or validated the listed solutions. In the revised version, we will enrich Appendix C by explicitly citing the relevant methods alongside each entry, including both original sources and follow-up studies that confirm their utility in selective classification. At the same time, **we will also include relevant citations in the main body**. We hope that this will help readers trace the origins of each recommendation and better understand the contexts in which they have been shown to be effective.\n\n---\n\n``3.3) Ranking insight and connection to AUROC, AUPRC.``\n\nWe appreciate the reviewer’s observation that metrics like AUROC/AUPRC improve with better score ordering. **Our goal is not to reiterate this general fact, but to make it operational in the context of selective classification**. Specifically, we isolate the ranking error term and show that **(i)** monotone calibration methods cannot reduce it, while **(ii)** reranking techniques can. Unlike threshold-averaged metrics, **SC directly evaluates accuracy at fixed coverage levels, where a poor ranking can have a more immediate cost**. By embedding ranking error alongside Bayes noise, approximation error, and slack in a single inequality—and validating each term through controlled experiments—we provide practitioners with a diagnostic toolkit to identify what limits performance and how to improve it.\n\n---\n\n``4.1) “Large scale” evaluation appears to be overclaiming.``\n\nOur paper devotes substantial space to **(i)** developing a theoretical decomposition of the selective-classification gap, **(ii)** identifying which calibration mechanisms can or cannot tighten it, and **(iii)** connecting selective prediction to loss prediction. **The experiments are designed as controlled diagnostics that vary one interpretable factor at a time to isolate its effect on each bound term**. That said, we agree that the phrase “large-scale evaluation” overstates the current scope. We have revised the paper to the more accurate wording **“systematic, controlled evaluation on synthetic and vision benchmarks.”** We also note our response to Reviewer VZwS, who requested **additional LLM results**. \n\n---\n\n``4.2) Only four different SC methods are implemented: 1) MSP 2) TEMP 3) SAT 4) DE.``\n\nWe chose these four methods not for sheer breadth but as **controlled, representative instantiations of the key scoring strategies** in our decomposition: MSP/TEMP to illustrate that monotone recalibration only affects approximation error, SAT as a feature‐aware reranker, and DE as an ensemble‐based reranker. This ablation‐style setup lets us cleanly attribute gap reductions to each mechanism. Although the implementations are simple, these methods are widely used because of their ease of implementation and the insights into how each method maps to our theoretical error terms are novel. We will clarify this in §4.2 and, if desired, include a brief appendix study of additional scoring approaches.\n\n---\n\n``4.3) Many recommendations from App. C are not explicitly experimented on.``\n\nIndeed, the recommendations in Appendix C consist of a combination of insights from our work as well as best practices in the SC literature. In the revised Appendix C we now **distinctly label each recommendation** as either **(i)** an original insight from our theoretical insights and controlled experiments; or **(ii)** a best practice drawn from the SC literature. As previously mentioned, we have added targeted citations so that readers can immediately see the evidence supporting each guideline.\n\n---\n\n``5) Enumerations in 4.1 & 4.2.``\n\nWe have fixed this as per the reviewer's suggestion.\n\n---\n\n**We hope that our rebuttal has addressed the reviewer’s questions and concerns and that the reviewer considers raising their individual and overall score(s). Our work has already significantly improved thanks to their feedback. We are happy to further engage with them during the discussion period!**"}, {"review_id": "kyhWjACuAB", "reviewer": "Reviewer_7Yt4", "summary": "This paper presents an analytical framework for selective classification that decomposes the gap between a selective classifier and the optimal \"oracle\" into a number of well-defined components: \n\n1. Bayes (Aleatoric uncertainty) resulting in natural label ambiguity.\n2. Approximation error of the model to the true conditional distribution.\n3. Ranking error of the confidence score.\n4. Statistical error from using empirical estimates of expectations\n5. Miscellaneous covering \"the rest\" such as optimisation imperfections.\n\nThe authors then demonstrate empirically the impact of each component on model performance.\n\nThroughout the paper, the authors aim to provide practical takeaways to the reader for improving the gap.", "questions": "See weaknesses, I am keen to raise my score if the authors are able to sufficiently address the issues I've presented above. In fact, I am generally positive on the paper at a high level, and would like to see it improve.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The knowledge presented could be of practical use to a range of readers and practictioners, facilitating better real-world systems for selective classification. I think there is real value in being able to disentangle the contributing causes to a problem in order to build better systems in a more informed way. For example, in existing research aimed at improving selective classification, especially *training* based approaches, it is not always clear whether improved selection is due to better ranking (as is typically how methods are motivated) or actually simply due to better accuracy due to improved generalisation performance. \n- Attractive and clear presentation. The authors structure the paper in a way that is easy to follow, with important information visually highlighted.\n- Good balance of theory and practical information. Although the paper is theory heavy, practical takeaways are presented intuitively in natural language, reducing the barrier-to-entry for the reader. Thumbs up especially for the inclusion of Appendix C.\n\n## Weaknesses\n\nAs a reviewer, I can certainly make mistakes. Due to the large reviewing load for NeurIPS I may have missed information in the paper, or misinterpreted details. I welcome the authors to correct me if that is the case. \n\n\n1. **Missing nuance in the presentation of the oracle**: I believe there are some presentation issues here \n    1. It is not made clear in Definition 3 that the upper bound is achieved under no Bayes noise. \n    2. It is not clear whether the oracle is the bayes-optimal ranker (in the presence of Bayes noise) or a ranker with \"oracle\" knowledge of the true sampled label and thus the correctness of the classifier.\n    3. It is not made clear that lower approximation error (higher mean accuracy $a_\\text{full}$) also raises the upper bound rather than just decreasing the selective gap. How this interacts with Eqs. 9-12 is also not discussed. In fact, this is clearly shown in Fig. 2 but not emphasised.\n2. **Overstated contribution on monotone calibration**. The fact that *monotone calibration cannot improve selective classification* should indeed be emphasised, but to claim this result as a contribution is odd to me as it seems to obviously arise from basic knowledge of binary classification. I would recommend the authors temper this claimed contribution.\n3. **Space for improvement presenting Eqs. 9-12**. Although I have previously praised the presentation, I feel like Sec. 3.3 would benefit from guiding the reader a bit more. For example:\n    1. Making it clear that Eq. 10 compares the learned classifier to the optimal classifier, *whilst keeping the selection function constant*. Note that this is not necessarily intuitive as in practice changing the classifier generally leads to changing the selection function. \n    2. Bringing in discussion (mentioned in the limitations section) about how a single design choice might affect multiple components. E.g. Typically, MSP is used by default, so improving the approximation error of the model $P_\\theta(y|x)$ to the true conditional will simultaneously improve both $\\varepsilon_\\text{rank}$ and $\\varepsilon_\\text{approx}$. This sort of clarification would be especially useful to practitioners.\n4. **Numerous issues with calibration discussion**\n    1. The claim that *temperature scaling rarely reorders* is supported by incorrect calculations in Appendix B.2 (simply sanity checking the provided logits in Eq. 34 gives different softmax outputs, and Eq. 32 seems to be wrong). Besides, existing research [1,2] (mentioned by the authors) clearly demonstrates empirically that temperature scaling has a large effect on selective classification. \n    2. Poor discussion on scoring. Histogram binning, which explicitly introduces ties by many-to-one mappings, is claimed to preserve ordering (even though ties are later discussed in the paper). Recent work [2,3] that both empirically demonstrates and analytically explains the effectiveness of post-training logit normalisation for improving selection, as well as investigates the effects of the calibration approach label smoothing, are not discussed.\n5. **Various missing discussion**. I believe a number of recent research directions should be at least briefly discussed as being relevant (e.g. for future work).\n    1. Unified selective classification [4,5,6] including OOD data to be rejected. How could the gap be extended in this case?\n    2. Uncertainty for open-ended language generation [7,8]. What challenges exist as our models are no longer directly trained to model the Bayes posterior?\n7. **Advice not backed by empirical evidence**. Although I have praised the presence of Appendix C, much of the advice given is not based on experiments, but simply the authors' understanding of existing approaches. This could lead to potential issues. For example, [3] explains how label smoothing, an approach that superficially should improve both approximation error and calibration, in fact degrades selective classification. Additionally, they recommend binning approaches, even though binning introduces ties.  Authors ought to roll back advice or provide additional experiments/references to other works with results for specifically selective classification, to support the advice given.\n6. **Minor complaints**.\n    1. The use of colour, although aesthetic, is a little arbitrary. I initially was confused as I thought definitions and takeaways were the same colour.\n    2. Sec. 2 Accuracy-coverage tradeoff is missing a reference [9].\n    3. Both MSP and SR are used to refer to the same thing.\n\n**Note**: I do not see the paper's lack of a novel method/algorithm as a weakness, as I believe this sort of framework/analysis paper can be just as, if not more impactful than a paper that proposes new solutions.\n\n[1] Galil et al. What Can we Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers? *ICLR 2023*\n\n[2] Cattelan and Silva How to Fix a Broken Confidence Estimator: Evaluating Post-hoc Methods for Selective Classification with Deep Neural Networks *UAI 2024*\n\n[3] Xia et al. Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It *ICLR 2025*\n\n[4] Jaeger et al. A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification *ICLR 2023*\n\n[5] Xia and Bouganis Augmenting the Softmax with Additional Confidence Scores for Improved Selective Classification with Out-of-Distribution Data *IJCV 2024*\n\n[6] Narasimhan et al. Plugin estimators for selective classification with out-of-distribution detection *ICLR 2024*\n\n[7] Kuhn et al. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation *ICLR 2023*\n\n[8] Kang et al. Scalable Best-of-N Selection for Large Language Models via Self-Certainty 2025\n\n[9] Traub et al. Overcoming Common Flaws in the Evaluation of Selective Classification Systems *NeurIPS 2024*", "was_revised": true, "final_justification": "(copied from another comment in the review discussion thread)\n\nAfter rereading my discussion with the authors as well as their discussion with other reviewers, I would like to summarise my thoughts on the submission. \n\n- My views on the strengths of the paper remain unchanged from my initial review.\n- The authors (for the most part) have \"patched\" the various issues mentioned by myself and other reviewers, as well as taken in feedback relating to paper presentation/communication. I am appreciative of their efforts as the discussion has been extensive.  \n\nAs such, **I do not have a strong reason to reject the submission**; however, due to the *extensive changes* undergone during rebuttal, the inability for reviewers to clearly view the current (post-rebuttal) state of the submitted manuscript, and the issues that exist in the original manuscript if viewed as is, I **strongly believe the paper would benefit from an additional round of peer review** (in agreement with reviewer 9vHG). This would allow the manuscript to be properly evaluated in its current, evolved state, and potentially give it the chance to receive a higher recommendation (I reiterate my positive view on the *potential* of the paper).\n\nAs such I will **raise my score to borderline accept**, however, I **urge the AC to consider the above when making their final recommendation**.", "rebuttal": "We thank the reviewer for their in-depth feedback, and for **highlighting the clarity, practical value, and balance of our paper**. The reviewer raises important points about the **presentation of the oracle bound, the calibration discussion, and the need for clearer exposition and empirical support in parts**. We address each of these concerns below and have revised the paper to incorporate the suggested corrections.\n\n---\n\n``1) Missing nuance in the presentation of the oracle.``\n\n``Definition 3 assumes no Bayes noise.``\n\nWe agree with the reviewer that this assumption was indeed not stated explicitly. We have revised the sentence immediately following Definition 3: \n\n> Assuming no Bayes noise—that is, all errors are avoidable given perfect confidence—this piecewise curve (see Figure 1) traces the oracle accuracy–coverage frontier based on a perfect ranking of examples by correctness probability.\n\n``What kind of oracle?``\n\nAfter rephrasing the previous sentence, this distinction becomes immediate under the no-Bayes-noise assumption: when every example’s posterior correctness $\\Pr\\bigl(h(x)=Y \\mid x\\bigr)\\in\\{0,1\\}$, ranking by the Bayes-optimal posterior is identical to ranking by the true realized label. Hence, **in the no-Bayes-noise setting the two oracle interpretations coincide**.\n\n``State that lower approximation error also raises the upper bound.``\n\nAs Definition 3 makes explicit, the oracle curve $\\overline{\\mathrm{acc}}(a_{\\text{full}}, c)$ depends directly on the full-coverage accuracy $a_{\\text{full}}$. Thus, improving approximation error (i.e., raising $a_{\\text{full}}$) not only reduces the selective classification gap $\\Delta(c)$, but also lifts the upper bound itself across all coverage levels. This interaction is indeed visible in Figure 2 but was indeed not emphasized in the text. **We have now clarified this point directly after Definition 4** and revised the figure caption to highlight how changes in approximation accuracy affect both the bound and the observed gap.\n\n---\n\n``2) Overstated contribution on monotone calibration.``\n\nWe thank the reviewer for pointing out this issue and have revised the draft to clarify our intent. While it is well understood that monotone transforms preserve score ordering and therefore cannot improve ranking-based risk–coverage curves, **this point is often muddled in the literature**—especially given empirical studies [1, 2] that appear to show large selective-classification gains from temperature scaling. Our contribution is to **(i)** formalize this invariance result in the multi-class case, **(ii)** identify rare logit-margin ties as the only way monotonicity can fail to preserve ranking, and **(iii)** highlight that performance improvements must stem from calibration methods that actively rerank examples. We now present this section as a clarification **aimed at resolving apparent contradictions in prior work**, not as a standalone result. We hope this helps readers more clearly distinguish between calibration techniques that change confidence scores and those that change rank order.\n\n---\n\n``3) Space for improvement presenting Eqs. 9-12``\n\n``Clarify that Eq. 10 keeps the selection function constant.``\n\nThank you for pointing out the potential ambiguity. To make the role of each term crystal-clear, we have **added an additional clarification** on this point after Eq. (9)-(12) that explicitly states: \n\n> Here we freeze the acceptance set $A_c$ defined by the current scoring function and ask how much worse the learned classifier $h$ is than the Bayes-optimal rule on this *same* set. \n\nWe then remind the reader that any shifts in the acceptance region caused by replacing $h$ are *not* charged to $\\varepsilon_{\\text{approx}}(c)$ but are instead captured by the ranking-error term $\\varepsilon_{\\text{rank}}(c)$ in Eq. (11). This clarification makes it evident that the two terms partition the total gap without double-counting.\n\n``Foreshadow discussion about how a single design choice might affect multiple components.``\n\nThis is a great suggestion. We have now **added a short “cross-term interactions” paragraph** at the end of §3.3 that makes this explicit.\n\n> **A single design choice can shrink multiple error terms.** When the confidence score is the maximum softmax probability (MSP), a better approximation of the true conditional $\\eta$ not only lowers the *approximation* term $\\varepsilon_{\\text{approx}}(c)$ but also tends to align MSP more closely with $\\eta_h$, thereby *indirectly reducing* the ranking error $\\varepsilon_{\\text{rank}}(c)$. Conversely, a non-monotone calibration head can slash $\\varepsilon_{\\text{rank}}(c)$ without improving $\\varepsilon_{\\text{approx}}(c)$. \n\nWe believe this forward-reference clarifies how model-capacity upgrades, richer pre-training, or loss-function tweaks can yield compound benefits, and we direct readers to the limitations section for a fuller discussion of such interactions.\n\n---\n\n``4) Numerous issues with calibration discussion.``\n\n`` Incorrect calculations in Appendix B.2.``\n\nWe thank the reviewer for catching the errors in Appendix B.2. We have **thoroughly revised this section to correct both the analytic expression for temperature-scaled confidence scores and the illustrative example**. The previous equation for $s_T(x)$ incorrectly assumed a binary logistic form; we now provide the correct expression for arbitrary $K$:\n\n$$\ns_T(x) = \\frac{1}{1 + \\sum_{j \\ne j_\\star} \\exp[(z_j - z_{j_\\star})/T]} = \\frac{1}{1 + \\sum_{j \\ne j_\\star} r_j^{1/T}}.\n$$\n\nWe also replaced the earlier example, which wrongly claimed a ranking swap, with corrected logits that do exhibit a swap at higher temperature: $s_1(x_1) = 0.576 > 0.512 = s_1(x_2)$ at $T = 1$, but $s_3(x_1) = 0.411 < 0.428 = s_3(x_2)$ at $T = 3$. Finally, we clarified the condition for reordering: that the sums over non-max logits, $S_1(T) = S_2(T)$, cross for some $T$.\n\n``Existing research [1,2] demonstrates that TEMP has a large effect on SC.``\n\nWe thank the reviewer for pointing to prior work suggesting that temperature scaling (TS) improves selective prediction. **The discrepancy arises from differences in evaluation setup**. Our analysis (§3.4, App. B.2) focuses on ranking-based selective classifiers, where prediction is accepted based on the top-\\$c\\$ fraction ranked by \\$s(x)\\$. In this setting, monotonic transformations like TS preserve the ordering and thus do not affect the selective classification gap \\$\\Delta(c)\\$ or selective accuracy at fixed coverage. In contrast, [1] uses a fixed score threshold \\$\\tau\\$ (e.g., “accept if \\$s(x) > \\tau\\$”), where TS can shift scores so that more correct predictions exceed \\$\\tau\\$, improving metrics like AUROC. However, **AUROC mixes overall accuracy and ranking quality**, whereas our decomposition isolates the selective component by conditioning on rejection. As shown in §4.2, while TS may improve calibration, it does not affect \\$\\Delta(c)\\$ unless it changes the ranking. We have clarified this in the revised draft.\n\n``Poor discussion on scoring/binning.``\n\nWe appreciate the reviewer’s observation and agree that our original discussion of histogram binning was too brief. While histogram binning fits a monotone mapping, it does induce score ties by mapping wide score ranges to the same value; **we now clarify in §3.4 and Appendix B.2 how such ties can—and in rare cases do—affect the selection ordering**. We have also added reference to the mentioned works [2,3] on post-training logit normalization and label smoothing.\n\n---\n\n``5) Discussion on future directions, including (i) connections to OOD; and (ii) uncertainty in open-ended generative (sequence) models.``\n\nWe appreciate the reviewer’s pointer to several fast-moving research threads that lie just beyond the scope of our current study. While we think that these directions should be more thoroughly investigated in future work, we want to acknowledge these connections. Hence, we have **created a new Appendix F** (“Beyond the Closed-World Setting”) that provides two self-contained subsections (§F.1 and §F.2) outlining how our decomposition might be generalized and which technical obstacles remain. We also include the following summary and pointer in our conclusion section: \n\n> Our finite-sample gap decomposition lays the groundwork for a more unified reliability framework; extending it to (i) settings where out-of-distribution inputs must be rejected and (ii) open-ended language generation—where correctness is semantic rather than categorical—constitutes a promising agenda for future work that we outline in Appendix F.\n\nWe are happy to provide a more detailed overview of these sections upon the reviewer’s request.\n\n---\n\n``6) Advice given in Appendix C.``\n\nWe have clarified in Appendix C **which checklist items derive from our theoretical and empirical results versus established best practices**, and we’ve **supported each recommendation with appropriate citations** (including the reviewer’s suggestions). To avoid confusion, we now explicitly note [3] cautioning against the use of label smoothing (which notably was not part of our original recommendations). For techniques with mixed effects—such as binning—we cross‐reference tie‐breaking remedies and advise tuning bin granularity to prevent coverage cliffs.\n\n---\n\n``7) Minor issues: Definition vs takeaways color; Sec. 2 accuracy-coverage tradeoff reference; MSP and SR are both used.``\n\nWe thank the reviewer for bringing these issues to our attention. We have adjusted the colors to be more distinctive between definitions and take-aways, have added a reference to [9], and have replaced the single mention of SR in §2 with MSP.\n\n---\n\n**We hope that our rebuttal has addressed the reviewer’s questions and concerns and that the reviewer considers raising their individual and overall score(s). Our work has already significantly improved thanks to their feedback. We are happy to further engage with them during the discussion period!**"}]

Author rebuttal:

Meta-review: (a) This paper investigates what is needed to build a performant selective classifier. To that end, a generalization bound is introduced, that covers multiple different components. These components provide insight how the performance of an selective classifier can be improved for practitioners. 

(b) Strengths: The reviewers mention that the paper is solid, theoretically and empirically, and is easy, enjoyable to read and has excellent presentation. Reviewers mention that the appendix is clear, all information is there to replicate the study, and applaud the guidelines of the checklist. While the paper does not provide a new method, the reviewers think the paper is insightful, and can help newcomers in the field. 

(c) Weaknesses: One reviewer mentioned that the amount of methods is limited. 

(d) Accept. While some reviewers remark that this paper has undergone some large changes and recommend another round of peer review, I think that this paper has survived the intense scrutiny extremely well, and I disagree with some reviewers - I believe they are perhaps too strict. I think the amount of peer review and scrutiny this paper has received is exceptional and surely above average, and the authors are extremely rigorous - I trust them to carry out the necessary changes. 

(e) Some old models and datasets were used, and the authors responded to this by providing a small LLM study. Some of the reviewers mention that some of the conclusions regarding temperature scaling are obvious - but the authors clearly indicate that empirical studies are not aware of this, and as such, their framework helps clarify empirical results. There have been several in depth discussions regarding technical details and proofs; with some mistakes that have been fixed - from what I can tell most issues have been adequately addressed.
Decision: Accept (poster)