Know When to Abstain: Optimal Selective Classification with Likelihood Ratios
Authors: Alvin Heng
Harold Soh
Venue: ICLR 2026 Poster (2026)
Keywords: selective classification
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose likelihood ratio-based selective classification methods and evaluate them under vision and language covariate shifts tasks.

Abstract: Selective classification enhances the reliability of predictive models by allowing them to abstain from making uncertain predictions. In this work, we revisit the design of optimal selection functions through the lens of the Neyman–Pearson lemma, a classical result in statistics that characterizes the optimal rejection rule as a likelihood ratio test. We show that this perspective not only unifies the behavior of several post-hoc selection baselines, but also motivates new approaches to selective classification which we propose here. A central focus of our work is the setting of covariate shift, where the input distribution at test time differs from that at training. This realistic and challenging scenario remains relatively underexplored in the context of selective classification. We evaluate our proposed methods across a range of vision and language tasks, including both supervised learning and vision-language models. Our experiments demonstrate that our Neyman-Pearson-informed methods consistently outperform existing baselines, indicating that likelihood ratio-based selection offers a robust mechanism for improving selective classification under covariate shifts.

Published as a conference paper at ICLR 2026
KNOW WHEN TO ABSTAIN: OPTIMAL SELECTIVE CLASSIFICATION WITH LIKELIHOOD RATIOS
Alvin Heng' & Harold So
'Department of Computer Science, National University of Singapore Smart Systems Institute, National University of Singapore (alvinh, harold] eomp. nus edu.sg sg
ABSTRACT
Selective classification enhances the reliability of predictive models by allowing them to abstain from making uncertain predictions. In this work, we revisit the design of optimal selection functions through the lens of the Neyman- lemma, a classical result in statistics that characterizes the optimal rejection rule a likelihood ratio test. We show that this perspective not only unifies the behavior of several post-hoc selection baselines, but also motivates new approaches to selective classification which we propose here. A central focus of our work is the setting of covariate shift, where the input distribution at test time differs from that training. This realistic and challenging scenario remains relatively underexplored in the context of selective classification. We evaluate our proposed methods across range of vision and language tasks, including both supervised learning and vision-language models. Our experiments demonstrate that our Neyman- Pearson Pearson-informed methods consistently outperform existing baselines, indicating that likelihood badi- selection offers a robust mechanism for improving selective classification under covariate shifts. Our code is publicly available https::/ //:::///iit Ehub. com/ C eelear-nnt euur-uus/ sc- kike 1hood- rat patioss
INTRODUCTION
Machine learning models are inherently fallible and can make erroneous predictions. Unlike humans, who can abstain from answering when uncertain, e.g., by saying "I don't ton't know". predictive models typically produce a prediction for every input regardless confidence. Selective classification aims to address this limitation by enabling models abstain uncertain inputs, thereby improving overall performance and robustness, for instance by deferring ambiguous cases to human experts.
A wide range of methods have been proposed to determine whether model should accept reject an input. Common post-hoe approaches rely on heuristic confidence estimates, such as the maximum softmax probability (Geifman & Elnianiv 2017 [eeddrycks & Gimpell (017), logit margins (Liang
itt al. 2024). or Monte Carlo dropout (Geifiman & El- Yaniv/ [201]] Gal & Ghahramani 2016). Other techniques assess a sample's 's proximity to the training distribution Leee al. 2018 Sun et 2022) under the assumption that samples farther from the data manifold are more likely to be misclassified.
separate line of work trains models with explicit abstention mechanisms, such as a rejection logit or head (Geifman & EI- Yaniv 2019 Liu et al., [2019] Huang et al.) (020). In this work we focus on post-hoc methods, which are model-agnosee and do not require specialized training.
Despite the rich literature, two important gaps remain. First, while foundational results (such as Chow (1770); Geifman & Yaniv (2017)) provide theoretical underpinnings for selective classification, there a lack of general, principled guidance for designing effective selector functions in the context of modern deep networks. Sccond, most evaluations are conducted in the i.i.d. setting, where test data assumed to follow the training distribution. Few works have begun exploring selective classification under distribution shifts (Xia & Bouganis] [022] Narasimhan et al. 2024). but focus on the common semantic shifts, neglecting the covariate shift setting that is becoming increasingly relevant.
To address these challenges, we propose new perspective rooted the Neyman- Pearson lemma, a classical result from statisties that defines the optimal hypothesis test in terms of likelihood ratio.

---

Published as a conference paper at ICLR 2026
We show that existing selectors can be interpreted approximations to this test, and this insight to derive two new selectors, A-MDS and A A-KNN, as well as a simple linear combination strategy. We evaluate our methods covariate shift, where the input distribution changes while the label space remains fixed. We focus on covariate shift for two key reasons: first, it is underexplored relative to semantic shifts (Heng & Soh 2025) (which is well studied in the context of out-of-distribttion detection (Hendrycks & Gimpel 2017 Ming et al., 2022, Lee et al. 2018, Heng et al. 2024)); second, it is increasingly relevant in modern applications such as vision- Saiion- models (VLMs), where the label set is large and variable, rendering most practical shifts in deployment covariate nature. Our results demonstrate that the proposed selectors outperform existing baselines and provide robust performance across distribution shifts, including on powerful VLMs like CLIP.
comprehensive suite of vision and language benchmarks under
In summary, our key contributions are:
We introduce for the first time a Neyman- Pearson--ea framework for defining optimality in selective classification via likelihood ratio tests.
2. We unify several existing selector methods and propose two new selectors and a linear combination approach under this framework.
3. We conduct thorough evaluation under distribution shifts, both covariate and semantic, across vision and language tasks and demonstrate superior performance across both VLMs and traditional supervised models.
2 BACKGROUND
Selective Classification. Consider a standard classification problem with input space X' C Rd, label space y/ {1, ...; K). and data distribution Dx.y over X' x V. A selective classifier is pair 9f. where X' + EK is base classifier, and g : X {0, 1} is selector function that determines whether to make a prediction or abstain. Formally.
f(x) if g()) =1,
(z) -
Jabstain if g()) = O.
That is, the model abstains on input æ when g(x)) = 0. In practice, g is typically implemented by thresholding a veal-valued confidence score:
Is,7 (a) = 1 (xx > 7],
where : R is a confidence scoring funetion (often adapted from OOD detection methods; see below), and is tunable threshold. The performance of selective classifier is typically evaluated using two metrics:
Coverage:
ØN,7 E.EDD ( (a:) E ((a.)) [f(((()) v) (x)]
(3)
Selective Risk: Ra,y
where {(f(x). y) is the 0/1 loss GGeifman & El- Yaniv 2017). Selective classification aims to optimize the tradeoff between selective risk and coverage by ideally reducing risk while maintaining high coverage. Improvements can stem from enhancing the base classifier f, or from refining the selector g to better identify error-prone inputs. In this work, we fix f to be strong pretrained model and focus on designing more effective selector functions g.
Covariate shift refers to a scenario where the marginal distribution over inputs, p(xx), changes between training and testing, while the support of the label distribution p(y) remains unchanged. For example, model trained on photographs of cats may face covariate shift when evaluated on paintings of cats, as the input appearance changes but the semantic categories are preserved. This is in contrast semantic shift, where both p(x) and p(y) change, typically due to the introduction of unseen classes. In this work, we focus on covariate shifts, which are increasingly relevant in modern applications such as VLMs, where the label set is large and can be adjusted to suit a given task. In such settings, distributional changes primarily manifest as covariate shifts, making them critical yet underexplored challenge for robust selective classification.
Covariate Shift.
2

---

Published as a conference paper at ICLR 2026
Duttrf-Distributtio (OOD) Detection. OOD detection is closely related to selective classification, as many selector functions s(x) are derived from or inspired by OOD scoring methods. Given an in-distribution (ID) data distribution prn, the goal of OOD detection is to construct scoring function 8 X' + R such that <(z) indicates the likelihood that æ originates from prp- A higher s(x)) corresponds to higher confidence that is in-distribution. In selective classification, these scores thresholded to determine whether to accept or abstain on a given input, as formalized in Eq. 2
3 SELECTIVE CLASSIFICATION VIA THE NEYMAN-
LEMMA
We begin by framing selective classification within the paradigm of hypothesis testing. Let Ho : denote the hypothesis that the classifier makes a correct prediction, and Hi CC that it makes an incorrect one. Selective classification then reduces to deciding, for each input, whether to accept Ho or reject in favor of H1. i.e., binary decision problem. This perspective a natural fit for a foundational result from statistics: the Neyman- Pennannneerronnnooo (NP) lemma (Neyman & Pearson 1933; Lehmann et al. 1986), which characterizes the optimal decision rule between two competing hypotheses.
Lemma (Neyman-Pearson (Neyman & Pearson
1933 Lehmann al. 1986)). Let Z Rd be
random variable, and consider the hypotheses:
Ha : Z Po
vs. H1 2 P1,
where Po and P1 have densities Pu and p1 that strictly positive shared support Z C R.. For any measurable acceptance region C Z under Ho, define the rype error (false rejection) as ({A) Po 17 & A), and type II error (false acceptance) as ((A) (z A).
Fix a rype error tolerance og E [0, 1]. Let ((0)) be the threshold such that
Po(2)
A' == 2 Z
2 7(a0)
(2)
satisfies a(A") = 000- Then A (oo) minimizes the type II error:
((A) Asa(A)=ao
min B(A).
In other words, among all decision rules with the same false rejection rate, the likelihood ratio test minimizes the false acceptance rate.
Applied selective classification, Lemma suggests the optimal selection score is likelihood ratio:
Pc(x) (x)
s(x)
where (x( and (x) denote the probability density associated with the classifier making a correct and wrong prediction respectively. Thresholding this score yields the lowest possible selective risk for any given coverage level.
Corollary (Informal). Any selector score s(x) that is a monotonic transformation of the likelihood
ratio po(a)) is also optimal under the Seyyann-Peasson criterion.
Corollary follows directly from the lemma, since monotonic transformations (e.g., logarithmic or affine maps) preserve the ordering of scores and hence do not alter the resulting acceptance region. We define a score function sax) to Neyman-Peurson optimal if a monotonic transformation of the likelihood ratio pe (=(x)) k (xr).. In practice, the true likelihood ratio is not accessible, so we approximate or construct monotonic proxy that captures the posterior odds of correct versus incorrect prediction for given input.
Note that Pe (a( and pw, (x) are general and naturally accounts for distribution shifts; pa(x) (pw (x)) includes all samples that the classifier classifies correctly (wrongly), regardless of whether they ID or distribution shift. Therefore, this simplifies our framework compared to prior works that consider ID and OOD distributions separately (Xia & Bouganis 2022 Narasimhan al. 2024).
In what follows, we first reinterpret existing selection scores from the literature as implicit approxima- tions this likelihood ratio and show conditions under which they NP optimal. We then introduce
3

---

Published as a conference paper at ICLR 2026
two new distance-based selection functions inspired by the NP framework. Finally, we propose a hybrid score that linearly combines multiple selectors which we find performs well in practice. Throughout, the assumptions used in our theoretical results are meant to clarify the connection to NP optimality and the structure of an optimal selector, rather than to prescribe conditions that must hold in practice. Introduction and brief details of the scores discussed in this section are provided in Appendix A
3.1 LOGIT-BASED SCORES AS APPROXIMATIONS TO LIKELIHOOD RATIOS
We consider two popular confidence scores, Maximum Softmax Probability (MSP) (Hendrycks & Gimpell 2017) and Raw Logits (RLog) (Liang et al.) 2024), and interpret them as approximations to likelihood ratio tests in the NP sense. This provides theoretical justification for their empirical success selector scores in selective classification.
Let (11) 2 ((2) 2 2 (KK denote the logits output by a classifier for a sample æ, sorted in
descending order. Define the corresponding softmax probabilities d( softmax(((0)) The MSP
score is given by (x)
d(1), while the RLog score is defined as SKLog (x) ( 7(2)
MSP has become standard baseline for OOD detection and selective classification (Hendrycks &
Gimpel 2017 Geifman & EI- Yaniv| (017). and RLog has been recently proposed as a strong score for selective classification (Liang et al. 2024).
Theorem 1. Let ((x) = arg (I..e(I...... Poly = x)) be the predicted label and define the event = (x( = Y) that the classifier is correct. Assume the classifier is calibrated for top-1 I correctness, i.e. P(C X= x) max; po = k x) =: (x)). Then MSP is Neyman- Pearson optimal for selective classification. Moreover, under the additional assumption that the softmax distribution is concentrated on the top two classes, i.e., L == Eizs d < d the RLog score is also Neyman-Pearson optimal.
The proof provided in Appendix B shows that under these assumptions, both MSP and RLog monotomic transformations of the likelihood ratio pe/ Pw.. and therefore is NP optimal by Corollary Of course, these assumptions has shown that modern neural classifiers tend to be poorly calibrated, and has proposed post-hoc calibration methods such temperature scaling. Notably, RLog has been shown to be invariant to temperature scaling (Liang et al., (024), making it robust to miscalibration and a compelling choice in practice. This aligns with our empirical findings in Sec. where RLog generally outperforms MSP (which corresponds to temperature scaling with T
not always satisfied in practice. Prior work (Guo al. 2017
While the effect of calibration is an
important factor in logit- based methods (Cattelan & Silva scope of this work.
2023
Fisch et al. 2022). lies beyond the
3.2 NEYMAN-PEARSON OPTIMAL DISTANCE SCORES
The logit-based scores discussed the previous section rely on classifier logit calibration, a condition often violated in practice (Guo et al. (017). To avoid this dependency, we consider distance-batt based methods that make alternative assumptions independent of calibration. As we show below, these methods approximate the likelihood ratio Pe/P,, /Po by leveraging spatial relationships in feature space.
Two distance methods widely used in OOD detection are the Mahalanobis distance (MDS) (Lee) et al. 2018) and k-- Neearestt Neighbors (KNN) (Sun al., 2022). Both rely computing distances between a test sample and training features (see Appendix A for details). Brieily, MDS is defined 14),, where ((x) denotes the extracted feature of x, typically from the penultimate or final layer of a trained deep network, μ4; is the empirical mean feature of class i, and Σ is a shared covariance matrix. In contrast, KNN scores inputs by the negative distance to the k-th nearest training feature vector. We introduce A-MDS and ANKN,, which are modified versions of these scores that explicitly incorporate insights from the NP lemma by estimating separate distributions for correctly and incorrectly classified training samples. Figure [ gives an overview of our approach, and we provide pscudocode for our proposed methods in Appendix
as (M))( max; -(((z) m4) Σ --'((())
A-MDS. Instead of estimating single distribution per class, maintain two sets of statistics per class: {μę, E and {n," y 1K Ji=1 corresponding to the mean and shared covariance of features for training samples that the classifier predicts correctly and wrongly, respectively. These quantities
4

---

Published as a conference paper at ICLR 2026
xx(x)
Pu(x)
D((s))
(u(xs)
D((x))
D
a(x) ((D,(x), (x())
<(xa) > a(xa)
Figure 1: Illustration of our proposed Neyman- optimal listance-based selective classification methods. We estimate the likelihoods of correct and incorrect predictions (pe and p..) a function of distances to training sets consisting of correctly and incorrectly classified samples: s(x) = (x),, D (x)), where here denotes function. For example, æ1 is "closer" Pe and "farther" from Pw than 22, and should therefore receive higher score.
are easily estimated as the true labels are known. We then define the A-MDS score as the difference in Mahalanobis distances between the two distributions:
SA-MDS (x) = DMDS( (z; pc, Σ DMDS( ,,"
(5)
(4:) the standard Mahalanobis score. The score intuitively increases when the input is closer (in Mahalanobis sense) to the "correctly classified" region and farther from the "wrongly classified" region in feature space. now formalize this intuition:
where DMDS [2; [4g. Σ) max; -(p(x) 14) T (d(x))
Theorem 2.
Let =Z E Bd be the feature representation of input x. Let C be the event the classifier makes correct prediction and -C its negation. Assume Z Pe = NNp. Ed) and 21-C Pi = =N E.). Then the A-MDS score MDM(()))) is Neyman-Peurson optimal for selective classification.
The proof is provided in Appendix B which shows that SA MDS is monotonic transformation of the likelihood ratio Pe/Pw. and thus is NP opttiiall as per Corollary The Gaussian assumption on feature representations is supported both empirically and theoretically via connections between Gaussian Discriminant Analysis and softmax classifiers (Lee et al.) 2018). making A-MDS well-suited for modern deep classifiers trained on standard supervised learning objectives.
Next we introduce A-KNN, non-parametric distance-based score inspired by the NP framework. Let A, = (xr)) ... (a.)) and A, (dh (x1),..... (w denote the feature representations of training samples that the classifier predicted correctly and wrongly respectively, and No |Ael and N., = |Aw.). Let z = ((x) be the feature vector of test input æ. Define (z) and F (2) as the Euclidean distances from
A-KNN.
its k-th nearest neighbors in A, and Ag. We define the A ANNN score as the difference in log- dig-ditaances:
SA-KNN (x) DKNN (x: Ac) DRNN (x; Aw)
(6)
where DKNN (z; Aa) - log (ogl))((((())) and DRNN Aw) ) = (o([v((((x))) This score measures how much closer a test point is to the region correctly classified samples compared incorrectly classified ones. We now show that A-KNN is asymptotically NP optimal:
Theorem 3. Let 2 = 6(x) E Rd be the feature representation of input x. and let C denote the event that the classifier makes correct prediction. Suppose Z Pe and -C Pw are arbitrary continuous densities bounded away from zero. Let N. = |Ael and N,, = A If k - oo while k/N. + and k/N... 0 as No, N + oo, then sANNN( x)) is Neyman-Pearsn optimal selector.
The proof is provided in Appendix As in previous cases. it relies showing that SA-KNN is monotonic transformation of the likelihood ratio Pa/ Pa/Pu. Importantly, this result does not require parametric assumptions the form of pa Pws unlike A-MDS. However, it does depend
5

---

Published as a conference paper at ICLR 2026
asymptotic properties of the k--earest neighbor density estimator, and the required conditions on k, Ne. and Nº may be difficult to satisfy in finite-sample settings. As such, both methods have their tradeoff's in terms of modeling assumptions.
In practice, we replace the single k-th neighbor distance with the average log-distance to the top k neighbors. Specifically, we use: Dxnw(z; = - E log[u:(((z)) and DKNN(æ; Aw)
k - log loglu; (o(z))) We find that this smoother version improves empirical performance, as shown in our ablation studies in Sec. [5.3 While this modification deviates from the form in Theorem we include a discussion Appendix suggesting NP optimality holds for the averaged log-distance
formulation under standard assumptions.
3.3 LINEAR COMBINATIONS OF DISTANCE AND LOGIT- BASED SCORES
The selector scores we discussed rely on different modeling assumptions and exhibit complementary strengths, discussed in Wang al. (2022). Logit-based based scores utilize the classifier's 's learned boundaries, while Ristance-based methods depend on geometric structures in feature space defined by training samples. We are thus motivated to leverage their respective advantages by proposing simple yet effective solution: linearly combining selector scores. Intuitively, this allows each score to compensate for the limitations of the other. The following lemma formalizes the NP optimality of such a linear combination:
Lemma 2. Lef 81 (ac) E IR and sar)) E IR be two selector scores. Assume both are Deyman-Pearsee (1) /pe and s2(x) is a monotone transform of (2) (2) Then for any scalar X R. t(xr) (1() + (82(()) is monotonic transformation of Pe /
optimal; that is, $1 (1(x) is a monotone transform of Pe
Pe (pe (p) /p= //pp"(p"))) {
The proof provided in Appendix B follows by expressing ((x) as a log-product of likelihood ratios. Thus, ((x) remains NP optimal under the assumption that the density for each hypothesis takes the form of multiplicative (or "tilted") product: Pe normalization constant. In practice, we find that combining a distance-based score (e.g., A-MDS) with based score (e.g.. RLog) leads to the best performance. We refer to such combinations by concatenating their names, e.g., A-MDS- RLog. We discuss fitting parameters like À in Sec.
(1) ()(())//// /Ze and similarly for pas where Ze is
4 RELATED WORKS
The study of classification with a reject option has a long history, beginning with cost-based formula- tions (Chow 1970) and extensions to classical models like SVMs (Fumera & Roli 2002 Bartlett & Wegkamp| 2008 and nearest neighbors (Hellman) 1970). In deep learning. [LeCnn et al. (189) ex-
plored rejection via top logit activations. Later, the covercoe and classifier-selcctor Trameworks were formalized (El- Yaniy et al. 2010 Geifman & EI- Yaniv 2017). with methods like MSP and Monte Carlo dropout proposed to provide confidence-based selection (Gal & Ghahramani 2016).
Subsequent works have extended this direction by studying popular logit and distance- scores, many originally developed for OOD detection. Examples include MSP (Hendrycks & Gimpel 2017) MaxLogit (Hendrycks et af. 2019). Energy (Liu et al., 2020). MDS (Lee et al. 2018). and KNN (Sun al. 2022). A limitation of logit-based methods is their reliance on classifier calibration. Although calibration is not the focus of our work, several studies have examined its impact on selective classification performance (Cattelan & Silva 2023 Galil al. 2023). An alternative line of research uses conformal prediction to construct calibrated prediction sets with formal guarantees (Vovk et al.
2005 Angelopoulos & Bates 202T Bates et al. 202T Angelopoulos et al. 2024). While such methods could be adapted for selective classification, they differ fundamentally from our goal of designing scoring functions optimized for selective risk. Diang et al. (2018) proposes a trust score by comparing the model's 's prediction with class-conditiona. KNN distances to estimate if a sample will be correctly classified, but it was shown to be ineffective on high-dimensional images.
Some methods incorporate rejection directly into training. For instance, SelectiveNet (Geifman & Yanivy 2019) adds a dedicated rejection head, while Deep Gamblers (Liu et al. 2019) and Self-Adaptive Training (Huang al.) (2020) introduce reject class and train the model to abstain. These methods require architectural modifications and joint training. In contrast, our work focuses on post-hoc methods that can be applied pretrained classifiers without retraining.

---

Published as a conference paper at ICLR 2026
Table I: DFN CLIP AURC (A) and NAURC (N) results on ImageNet and its covariate shifted variants at full coverage. Lower is better. AURC results are on the 10 scale. Bold and underline denotes the best and second best result respectively. "Avg (IK)" denotes average results over datasets with full K-claass coverage, while "Avg (all)" is the average result over all datasets.
Im-I K
Tm-R
Im-A
ON
Im-V2
Im-S
Im-C
Avg (IK)
Avg (all)
Method
A
N
A
A
N
A
N
A
N
A
A
N
MSP
9.08
0.542
2.00
0.344
2.29
0.179
8.87
0.268
9.39
0.521
12.3
0.524
15.1
0.328
11.5
0.479
8.43
0.387
Maxt agin
9.08
0.542
2.21
0.385
2,96
0.249
9.15
0.518
17.7
0.423
13.9
0.502
11.3
0.440
Energy
14.2
0
5.81
1.06
12.2
1.2
33.3
1.43
14.9
0.882
20.8
0.975
49.2
1.59
24.8
1.09
21.5
1.15
MDS
11.3
0.699
3.41
0.608
3.21
0.274
16.4
0.624
11.7
0.672
15.2
0.68
17.6
0.426
13.9
0.619
11.3
0.569
KNN
10.5
0.643
2.51
0.439
2.67
0.219
11.4
0.390
10.9
0.618
14.1
0.619
16.7
0.389
13.1
0.567
9.83
0.474
RLog
4.83
0.246
0.808
0.122
1.59
1.108
7.73
0.214
5.27
0.250
6,84
0.234
12.6
0.226
7.39
0.239
5.67
0.200
SIRC
16.3
1.04
3.90
0.701
8.48
0.817
15.1
0.564
17.1
1.03
20.7
0.968
22.1
0.612
19.05
0.913
14.8
0.819
A-MSS
5.00
0.257
2.19
0.380
2
0.194
0.304
5.43
0.260
0.311
12.5
0.224
7.81
0.263
6.50
0.276
A-KNN
0,230
1.42
1,99
0.149
4.99
0.231
12.1
0.207
7.32
5,89
0.225
MDS RLog
4.13
0.197
1.09
0.175
1.60
0.109
7.13
0.185
4.52
0.200
6.27
0.204
11.1
0.170
6.51
0.193
5,12
0.177
A KNN. RLog
3.98
0.187
0.770
0.115
1.45
0.093
7.14
0.186
4.36
0.190
6.13
0.196
11.3
0.175
6.43
0.187
5.01
0.163
Related to work are approaches that combine selective classification and OOD detection, termed SCOD (Xia & Bouganis [022]] Narasimhan et al.] 2024). Such methods consider the ID classification and OOD distributions separately and seek to combine them into single score function. These approaches are typically designed for semantic shifts and need adaptation for covariate shift. Our formulation avoids this by representing all distribution shifts through the general pair Pw.), which does not require distinguishing between shift types.
Finally, most closely related to our work is Liang et al. (024), who study selective classification under both semantic and covariate shifts and introduce the Raw Logit (RLog) score. Our work differs in several ways: 1) we focus on covariate shifts, which we argue more relevant in modern settings where large and variable label sets (c.g.. from vision-language models) mitigate label drift; 2) we introduce a unified theoretical framework grounded the Neyman-Pearson Pearson lemma, from which we derive new selector scores with formal optimality guarantees; and 3) we evaluate our methods on broader class of models, including VLMs, whereas Liang et al. (2024) focus exclusively on standard supervised learning paradigms.
5 EXPERIMENTS
Datasets. We evaluate our methods across vision and language domains, with a primary focus on the former: For vision tasks, we use ImageNet- IK (Im-1K) and suite of covariate-shifted variants:
1) ImageNet-Rendition (Im-R) (Hendrycks et al., 2020), 2) ImageNet-A (Im-A) (Hendrycks et al. 2021). 3) ObjectNet (ON) (Barbu al.) 2019). 4) ImageNetV2 (Im-V2) (Recht et al. [019)) 5)
ImageNet-Sketch (Im-S) (Wang et al.) 2019), and 6) ImageNet-C (Im-C) (Hendrycks & Dietterich 2019 We group these datasets based on label coverage: full 1000-class coverage (Im- IK, Im-V2, Im- S. Im-C) and subsets of classes (Im-R, Im-A, ON). For language tasks, we evaluate on the Amazon Reviews dataset (Ni et al., 2019: Koh al. 2021). To simulate realistic deployment scenarios involving distribution shift, following Liang et al. (0024) we evaluate on mixed test sets that combine in-distribution and covariate-shifted samples. For example, results reported Im-C are computed on a combined test set of Im- 1K and Im-C.
Classifiers and Baseline Selector Scores. We consider two families of classifiers for vision exper- iments, namely CLIP zero-shot VLMs (Radford et al. 202T)) and supervised classifiers. Specifically. we use the CLIP model from Data Filtering Networks (DFN) (Fang et al., (024) and EVA (Fang et al. 2023) for supervised learning, chosen for their state-of-the-art accuracy on ImageNet. Our focus is not on model training but on evaluating selector scores applied post-hoc. Note that for EVA, we restrict evaluation to datasets with full IK class coverage as the model is trained on the complete ImageNet label set only. In contrast, CLIP can be adapted at inference time to arbitrary label subsets, so evaluate it across all datasets. For language tasks, fine-tune DistilBERT (Sanh et al.) 2019) model using LISA (Yao et al. 2022) on the Amazon Reviews training set and evaluate selective classification performance the full test set.
7

---

Published as a conference paper at ICLR 2026
Table 3: Supervised learning AURC (A) and NAURC (N) results with the EVA model at full coverage.
Lower is better. AURC results are on the 10-2 scale. Bold and underline denotes the best and second best result respectively.
Im-1K
Im- V2
Im-S
Im-C
Avg (1K)
Method
A
N
A
A
N
A
N
MSP
3.32
0.256
3,85
0.266
8.15
0.319
6.41
0.215
5.43
0.264
Maxi
4.53
0.371
5.16
0.379
10.3
0.437
7.98
0.301
6.99
0.372
Energy
6.82
0.590
7.58
0.589
14.0
0.641
11.1
0.474
9.89
0.573
MDS
4,01
0.322
4.32
0.307
7.26
0.271
6.80
0.236
5.60
0,284
4.00
0.321
4.31
0.306
6.77
5.56
0.282
RLog
2.33
0.161
2.72
0.168
5.90
0.197
5.50
0.163
4.11
0.172
SIRC
3.68 0.290
4,23
0.299
8.71
0,350
6.84
0.240
5.87
0.295
A MDS
2.56
0.183
2,90
0.183
5.76
0.189
5.50
0.164
4.18
0.180
A-KNN
2.60
081
2.99
0.191
5.91
0.197
5.74
0.177
4.31
0.188
--DS-RLog
2.26
0.155
2.61
0.158
5.45
0.172
5.12
0.143
3.86
0.157
KNN Rlog
2.31 0.159
2.69
0.165
0.182
5.38
0.157
4.00
0.166
For baseline scores, we compare our proposed A-MDS, A-KNN, and their linear combinations with common OOD detection and uncertainty-basee scores: MSP (Hendrycks & Gimpel 2017). MCM
(for CLIP), Logit (Hendrycks et al. 2019), Energy (Liu et al. 2020). MDS (Lee et al. 2018).
KNN (Sun et al. 2022), and RLog (Liang et 2024) as well as SIRC (Xia & Bouganis 2022). As
they are functionally similar, we abbreviate MCM as MSP when presenting CLIP results. Details of the baseline are provided in Appendix A
Evaluation Metries. We evaluate performance using two metrics: the Area Under the Risk- Coverage Curve (AURC) and the Normalized AURC (NAURC) (Cattelan & Silva 2023). The AURC captures the joint performance of the classifier and selector across coverage levels. NAURC normalizes AURC to account for the classifier's base error rate, providing a fairer comparison across models with different accuracies. Formally, NAURC is defined as:
NAURC().9) = AURC(f.)) AURC((..))
R()) - AURC(A
where 9" denotes an oracle confidence function achieving optimal AURC, and R(f) is the risk of f. The oracle can be computed in practice using the ground-truth labels of the evaluation set. Intuitively. NAURC measures how close the selector 9 gets to the optimal, normalized by the classifier's ss total error. Thus, while AURC is useful for understanding overall performance in the context of a specific model, NAURC enables fair selector comparisons across models by factoring out baseline classifier accuracy.
Selecting À and k. Both À and k can be selected on val- idation set. We found that the simplest recipe to fitting À is to balance the magnitudes of 81 (x) and syx)), so that neither overpowers the other. For k in KNN-based scores, we find that k e [25, 50] is sweet spot. Full experimental settings are pro- vided in Appendix E and hyperparameter sensitivity analysis for X and l is presented in appendix Fig.
Table 2: Results on Amazon Re-
views and its covariate shifted test set at full coverage using Distil- BERT trained with LISA.
In-D
Cov Shift
Method
N
5.1
IMAGE EXPERIMENTS
MSP
12.2
0.368
13.9
0.401
MaxLogit
12.6
0,384
14.3 0.416
We report full selective classification results for CLIP and EVA models in Table [ and 3 respectively. First, let us consider CLIP results. We see that going from MDS and KNN their NP-informed variants, A-MDS and ANNN,, leads to roughly 50% reduction in average AURC and NAURC, showing that the assumptions made in the Np-optimality theory hold well in practice. The best average performance is achieved by the linear combinations A A-KNN--LL -LNN and A-MDS- with - RLog leading overall in both AURC and NAURC. RLog score ranks third on average, highlighting its strength standalone
Energy
12.89
0.397
14.6 0.428
MDS
20.6
0.739
22,2 0.750
KNN
19.4
0.686
21.3 0.711
Rl.ng
12.4
0.410
SIRC
12.3
0.370
14.0 0.403
A-MDS
12.7
0.389
14.4 0,422
ANNNN
12.4
0.374
14.2 0.412
A-MDS- RLog
12.2
0.368
13.9 0.401
A-KNN-RLog
12.0
0.358
13.8 0.394
MSP
11.9
A-KNN-MSP
12.0
0.359
13.8
0.396
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation experiments on DFN CLIP.
Im-lK
Im-R
Im-A
ON
Im-V2
Im-S
Im-C
Arg (IK)
Avg (all)
Method
A
A
A
Ahlations on A ANKNN
A-KNN no avg
0.167
8.87
0.268
7.63
0.276
0.216
7,45
0.241
KNN w/ avg
7.55
0.272
7.32
Ablations on lincar combinations
A-DSS-A-NN
9.66
746
MSP-RLog
4.82
0.245
0.800
0.120
1.56
0.105
7.51
0.204
5.26
0.249
6.81
0.232
0.222
7.35
0.237
5.61
0.197
A-KNN-MSP
4.57
0.228
1.22
0.199
1.82
0.131
7.58
0.207
4.94
0,228
7.40
0.264
11.8
0.110
7.18
0.244
5.62
0.253
A A-KNN-RLog
3.98
0.187
0.770
0.115
1.49
0.993
7.14
0.186
4.36
0.190
6.13
0.196
11.3
0.175
6.43
0.187
5.01
0.163
logit-based selector. Motivated by this strong performance, we use RLog in combination with our distance-based scores. We plot the risk-coverage curves for selected datasets Fig. 3f the appendix, showing our methods consistently demonstrate the most favorable trade-off across all coverage levels, remaining stable
low coverage.
For practitioners aiming to identify the best overall selective classification setup that considers both the base classifier and the selector, one approach is compare performance using the AURC metric. On the IK-class datasets, EVA paired with A-MDS-RR RLoog achieves an AURC of 3.86, outperforming the DFN CLIP model with A A-KNN-RLog 5.01. Despite similar NAURC values, EVA's higher Im- IK base accuracy (84.3%% vs. DFN CLIP's 80.39%) makes it the preferred choice when considering both components. Intuitively, the optimal setup involves pairing the best selector (here, A-MDS- RLog) with the most accurate base classifier.
For EVA, the ranking reversed: A-MDS-RLog achieves the best overall performance, followed by A A-KNN-RLog. This supports hypothesis that MDS-based methods particularly effective for supervised models due to the close connection between softmax classifiers and Gaussian Discriminant Analysis (Lee et al. 2018). which justifies the Gaussian assumptions used in MDS. In contrast, CLIP models trained with contrastive learning (Radford et al.] (021)) do not satisfy these assumptions, making the nonparametrie A A-KNN combination more suitable. The bottom
of Fig. 3 of the appendix confirms that A-MDS-RL-g yields the best risk-coverage behavior for supervised learning across all coverage levels. Comparing average NAURC the full K-Cllass datasets, A-MDS-RLAA with EVA is the top performing selector score, with slightly better score (0.157) than the best performer on CLIP (0.163).
To verify that the performance gains of our methods stem from actual algorithmic improvements, rather than lcrge-scale pretraining of CLIP or EVA potentially mitigating distribution shifts, we also evaluate on ResNet50 trained solely on ImageNet-IK, The results in appendix Table 7hoow that our methods perform the best, consistent with earlier findings.
Semantic Shift Experiments. For completeness, we also report results for experiments on datasets that are semantic shifts to ImageNet-1K in Appendix Table 8 In agreement with the covariate shift experiments, our proposed methods achieve the best performance on this benchmark.
5.2 LANGUAGE EXPERIMENTS
Table 2 presents results on the Amazon Reviews dataset. Unlike the vision tasks, the best-performing method is A-MDS-MSP. followed closely by A-MDS-RLog and A-KNN-MSP. Since LISA (Yao) al. 2022 uses softmax classification objective, the superiority of MDS-based selectors supports our hypothesis about their suitability for supervised models. Interestingly, MSP outperforms RLog in this domain, resulting in better performance when combined with AMDS. This highlights another important practical insight that the best linear combination often involves pairing the top-performing standalone distance-based and logit-based score.

---

Published as a conference paper at ICLR 2026
Table Ablation results using DFN-CLIP on ImageNet-1K where the fraction of labeled samples used in feature computation for our proposed methods are varied.
0.1% %
1%
10%
50 %
100 %
Method
A
N
A
N
A
N
A
N
A
N
A-MDS-RLog
10.5
0.638
4.19
0.202
4.14 0.198
4.13
0.197
A A-KNN-RLog 4.81 0.245 4.58
0.229
4.17
0.200
3.98
0.188
3.98
0.187
5.3 ABLATIONS
Design Choices. Table summarizes several ablation experiments on the design choices of our proposed methods. First, we justify averaging the top-k nearest neighbor distances in A-KNN rather than using the k-th distance alone. This modification yields measurable gains, where average AURC improves from 6.03 5.89 and NAURC improves from 0.233 to 0.225. We also investigate various combinations of selector scores. For CLIP, A-KNN-RLogg remains the best across all configurations, outperforming both double-distance combinations (e.g.. S-MMDSA--KNN and double- logit combinations (e.g., MSP-RLog). Notably, pairing A-KNN with Rlog significantly outperforms pairing it with MSP, further validating Rlog''s 's role as a strong logit-based complement.
Sample Efficiency. Although our methods require a one-time feature computation step, this cost is amortized over all future inference runs as the resulting features can be cached. Nevertheless, to evaluate performance low-data or low-computation resource regimes, we conducted ablations limiting the amount of labeled samples used. The results in Table show that both methods are surprisingly stable. A-KNN is especially robust, maintaining strong performance with as little as 0.1% of labeled data. As expected, A-MDS degrades the 1% level due to the difficulty of estimating per-class statistics with so few samples, and is not applicable at 0. 0.1% (roughly image per class). Importantly, A J--KNN-RLL RLog continues outperform RLog I 1% and matches it at 0. % (see Table []. indicating that our method is still preferable whenever even a small amount of labeled data is accessible.
CONCLUSION
We presented framework for designing selector functions for selective classification, grounded the Pearssn lemma. This reveals that the optimal selection score is monotonic transformation of likelihood ratio, which unifies several existing methods. We proposed two novel distance-basee scores and their linear combinations with logit-based baselines. Experiments across vision and language demonstrate that our methods achieve state-of-the-art performance across diverse settings.
Limitations and Future Work. While our focus has been on classification, the Neyman Pearson framework is general and broadly applicable to other predictive tasks. Exploring selective prediction in settings where uncertainty plays a critical role, such as semantic segmentation and time series forecasting, presents promising future directions. Additionally, extending these ideas to generative models such as LLMs is another exciting avenue for future work.
Neyman-Pearson
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This research project is supported by STAR under its National Robotics Programme (NRP) (Award M23NBK0053), The authors also acknowledge support from Google (Google South Asia and South-East Asia Award).
ETHICS STATEMENT
This work does not involve human subjects, and it relies solely on publicly available models and datasets with all attributions provided. Based on the scope of our methods and results, we do not identify ethical issues that require special attention, and we are not aware of immediate harmful applications arising from this research.
REPRODUCIBILITY STATEMENT
We include the theoretical and experimental details necessary to reproduce our findings. Proofs and supporting discussions for all theoretical claims appear in appendix Sec. B and Sec. C For
implementation clarity, we provide pseudocode for our methods in appendix Sec. and report experimental setup and hyperparameters appendix Sec.
LLM USAGE
We used large language models to assist formulating and checking mathematical proofs and to
improve grammar and writing clarity. All outputs from the LLMs were reviewed by the authors before inclusion, and the authors take full responsibility for the paper's content and the accuracy of the presented results.
REFERENCES
Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 2021.
Anastasios Nikolas Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Con- formal risk control. In The Twelfth International Conference on Learning Representations, 2024.
URL
net /ttee/forne forum 3xt.mmmmm1d3333G322223
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Peter Bartlett and Marten Wegkamp. Classification with reject option using a hinge loss. Journal of Machine Learning Research, 98), 2008.
Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan. Distribution- free, risk-controlling prediction sets. Journal of the ACM (JACM), 68(6):1-34, 2021.
Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. International conference on muchine learning, pp. 1059-1071. PMLR, 2021.
Luis Felipe P Cattelan and Danilo Silva. How fix a broken confidence estimator: Evaluat- ing post-hoc methods for selective classification with deep neural networks. arXiv preprint 1rXiv:2305.15500 2023.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-imann learning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2818-2820 2023.

---

Published as a conference paper at ICLR 2026
C Chow. On optimum recognition error and reject tradeoff. JEEE Transactions on information theory, 16(1):41-6,, 1970.
Ran El- Yaniv et al. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(5), 2010.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. In The Twelfth International Conference Learning Represen- tations, 2024,
net/ fttr:/ooummmmmmm (orrum?id=KAk6ng2222
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19358- 19369, 2023.
Adam Fisch, Tommi S. Jaakkola, and Regina Barzilay. Calibrated selective classification. Transac- tions on Machine Learning Research, 2022. ISSN 2835-8856. URL /tR//tttpp://oprreveee net / Eorum? ? dd= 2FNBB
Giorgio Fumera and Fabio Roli. Support vector machines with embedded reject option. In Pattern Recognition with Support Vector Machines: First International Workshop. SVM 2002 Niagara Falls, Canada, August 10, 2002 Proceedings, pp. 68-82. Springer, 2002.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
Ido Galil, Mohammed Dabbah, and Ran El- Yani.. What can we learn from the selective prediction and uncertainty estimation performance of 523 imagenet classifiers? In The Eleventh International Conference on Learning Representations, 2023.
4ddp66A2216XXmm
Yonatan Geifman and Ran Yaniv. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017.
Yonatan Geifman and Ran EI- Yaniv.. Selectivenet: deep neural network with an integrated reject
option. In International conference machine learning, pp. 2151-2159, PMLR, 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Martin E Hellman. The nearest neighbor classification rule with reject option. IEEE Transactions on Systems Science and Cybernetics, 6(3):179-185, 1970.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint 1rXiv:1903.12200 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and oftto--distritttion examples in neural networks. In International Conference Learning Representations, 2017.
URL http:t/::: / http///ppp//vviieee net. feeee//ooormiiiddkkff441
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. arXiv preprint arXiv: 1911. 1132, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Lixuan Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of -ut-t--distiibuttinn generalization. 2021 ieee. In CVF International Conference on Computer Vision (ICCV), volume 2, 2020.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 15262-15271, 2021.
12

---

Published as a conference paper at ICLR 2026
Alvin Heng and Harold Soh. Detecting covariate shifts with vision-language foundation models. In ICLR 2025 Workshop on Foundation Models in the Wild, 2025. URL /R//stpss:////eeerrreew
Alvin Heng, Harold Soh, al. model. Advances in Neural Information Processing Systems, 37:43952-43974, 2024.
detection with single unconditional diffusion
Lang Huang. Chao Zhang, and Hongyang Zhang. Self-adaptive training: beyond empirical risk
minimization. Advances in neural information processing systems, 33: 33:19665-19376, 2020.
Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. To trust or not to trust a classifier. Advances in neural information processing systems, 31, 2018.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of wn-the-wild distribution shifts. In International conference machine learning, pp. 5637-5664. PMLR, 2021.
Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with a back-propagation network. Advances in neural information processing systems, 2, 1989.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting disto--dist samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Erich Leo Lehmann, Joseph Romano, et al. Testing statistical hypotheses, volume 3. Springer, 1986.
Hengyue Liang, Le Peng, and Ju Sun. Selective classification under distribution shifts. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URLhtttps://ppeerrreie n
forum? Somum?id=-e 1ewpdddddmmxmwwwm00m
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based of-distribution detection. Advances in neural information processing systems, 33:21 46:21464-22475,, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF international conference computer vision, pp. 10012-10022, 2021.
Ziyin Liu, Zhikang Wang, Paul Pu Liang, Russ R Salakhutdinov, Louis-Philippe Morency, and Masahito Ueda. Deep gamblers: Learning to abstain with portfolio theory. Advances in Neural Information Processing Systems, 32, 2019.
Don O Loftsgaarden and Charles P Quesenbeny. A nonparametric estimate of multivariate density function. The Annals of Mathematical Statistics, 1049)100491051,, 1965.
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of- distribution detection with -ision-language representations. Advances in neural information processing systems, 35:35087-35102. 2022.
Harikrishna Narasimhan, Aditya Krishna Menon, Wittawat Jitkrittum, and Sanjiv Kumar. Plugin estimators for selective classification with -fitt---dst detection. In The Twelfth International Conference on Learning Representations, 2024. URL htlpsttt::// /ttp:://ppeerveeee ew net. forum?
IddDDASh7rJ7gg
Jerzy Neyman and Egon Sharpe Pearson. Ix. on the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, (111699-70))2299-3333 1933.
Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantlylaabeldd reviews and fine- -naggraneed aspects. In Proceedings of the 2019 conference empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNIP-)JCNIIP), pp. 188-197,7 2019.
13

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PmLR, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? International conference on machine learning, pp. 5389-5400. PMLR. 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: DeXXiv:190.000 1910.01 108, 2019.
Bernard W Silverman. Density estimation for statistics and data analysis. Routledge, 2018.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-diitributioon detection with deep nearest neighbors. International Conference on Machine Learning. pp. 20827-20840. PMLR, 2022.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning random world, volume 29, Springer, 2005.
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances Neural Information Processing Systems, 32, 2019.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual- logit matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4921-4930, 2022.
Guoxuan Xia and Christos-Savwar Bouganis. Augmenting softmax information for selective classifi- cation with ooltt--distribttion data. In Proceedings of the Asian Conference on Computer Vision, pp. 1995-2012, 2022.
Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of-distribution robustness via selective augmentation. In International Conference on Machine Learning, pp. 25407-25437. PMLR, 2022.
Puning Zhao and Lifeng Lai. Analysis of knn density estimation. IEEE Transactions on Information Theory, (8(12))7971-7999 2022.
14

---

Published as a conference paper at ICLR 2026
Appendix for "Know When to Abstain: Optimal Selective
Classification with Likelihood Ratios"
A DESCRIPTION OF BASELINES
In this section provide brief description of each baseline considered in this work.
Maximum Softmax Probability (MSP) (Hendrycks & Gimpel, 2017). Given an input æ, let the classifier output logits be { jk=1 and corresponding softmax probabilitiess po(y k/æ)
softmax(1(k)). The MSP score is defined
(MSP( = max Po(y = klar).
(e().....K)
MSP is commonly used OOD/confidence score: larger values indicate the model places high probability mass on single class, and thus the input treated as more "in-distribution or "confident". For selective classification, threshold this score (and subsequent scores) form the selector Is.7 (x) ([) > 7].
Maximum Logit (MaxLogit) (Hendrycks et al. 2019).
The MaxLogit score is defined
MMaxLogit (x)
max ke (K.
(()
Intuitively, larger values indicate that least one class is assigned large unnormalized score, and thus the model is more confident in its prediction.
Energy (Liu et al. 2020). The energy score is defined from logits as the Helmholtz free energy
K
E(x) = -T log
exp (16) (1))TT)
where T' > is a temperature parameter (we T 1). For selective classification we use the negative energy (so that larger values indicate more in in-distribution-l inputs):
K
8Energy (x)) = log Σ
exp ((K))
Mahalanobis Distance (MDS) (Lee et al. 2018). Let Φfx:) denote the feature representation of x (e.g., penultimate features of a deep network). Using training data, we estimate the empirical mean feature 14 for each class i (1...., K) and shared (tied) covariance matrix Σ across classes. The MDS score is then
SMDS (x)
-(2 1G)=))-T-1 14),
se (E.....,KK
i.c., the negative squared Mahalanobis distance to the closest class centroid. Intuitively, SMDS (œ) large when æ lies high-density region of some class under the fitted Gaussian discriminant model, and small when æ is far from all class clusters.
k-Nearest Neighbors (KNN) (Sun et al. 2022). Let {2,}}_1 denote the set of training features (we normalize the features). Define (z) the Euclidean distance from to its k-th nearest neighbor among {x,} ). The KNN score
BKNN( rz(2),
so that inputs lying in denser regions of the training feature manifold (smaller neighbor distances) receive higher scores.
15

---

Published as a conference paper at ICLR 2026
Raw Logits (RLog) (Liang et al., 2024).
RLog uses the confidence margin between the top two
logits as scale-robust confidence score. Let {(1)
2 ((2) 2 ... 2 ((4) be the logits sorted in descending order for given input m. We define
SRLog (z) (0) ((2)
Intuitively, SRLog (a) is large when there is clear "winner" class, and small when the classifier is uncertain between competing labels. Liang et al. (2024) argue that using a logit-space margin yields more robust score under classifier miscalibration and post-hoc logit transformations (e.g.. temperature scaling), since it depends only on the relative separation between the top classes.
Softmax Information Retaining Combination (SIRC) (Xia Bouganis 2022 Let S (x) be primary softmax-derive confidence score, and let (a) be an auxiliary feature based score. We choose S₁ to be MSP S2 to the L1 norm, in line with Xia Bouganis 2022). SIRC combines these via
(2) - -S S (z))
exp exp(<b(sa(z)
where STnax is the maximum attainable value of S₁ (e.g., STmax for MSP), and a, control how strongly S2 influences the score. We follow Xia & Bouganis (2022) and set and using in-distribution statistics of S2 (a = Usa 30 and 1/05.))
B PROOFS
Theorem 1. Let ((x) arg maxke(I..... K) Poly x)) be the predicted label and define the event == (x() } that the classifier correct. Assume the classifier is calibrated for top-1 correctness, i.e., P(C =x) max; poly = k. x) =: d (x)) Then MSP is Neyman-Pearson optimal for selective classification. Moreover, under the additional assumption that the softmax
distribution concentrated on the top two classes, i.e., == Eizs d() 2) the RLog score
also Neyman-Pearson optimal.
Proof. Recall that we denote ((1) 2 ... 2 [(K) as the logits predicted by the classifier for given input æ (sorted in descending order) and d()) = softmax(1(0)) the corresponding softmax probabilities. Then SMSP (x) = (x) and SRLog (x) (x))
Define
(r( = p( |C) and (x) = p( -C),
i.., the (test-time) input densities conditioned the classifier being correct wrong, respectively. Let == P(C) and define the posterior correctness probability g(æ) := P(C X = =x).
MSP Optimality.
By Bayes' rule,
p( P(C)
((x) p C)) P(C) p( PC)
(e(x))
Pc(x) π + Pw (x) (1 TT)
Equivalently, the posterior odds satisfy
Pr(at) m 1 (xx)
((x))
(= 1-
Since the classifier is (perfectly) calibrated for top-1 correctness such that (x)) P(C æ:) = y(z).
x=
(=( (x) (z) g(x) 1 1- (w))
(8)
P=(w)
Pa
The mapping h(2)
I+z is strictly increasing for z 2 0 since h'(z) [لक > 0, and scaling by
the positive constant preserves monotonicity. Hence SMSP (æ:) is monotone transformation of the likelihood ratio Pu (x))
P=l and is -eaasonn optimal by Corollary
16

---

Published as a conference paper at ICLR 2026
RLog Optimality.
We can express RLog the logarithm of the ratio of the top two softmax values:
(m) ( (x))
d()) (x)
= (za) = enL.og(a))
d (a) (())
thus SRLop (x) log
Observe that
(1) = 1 - (x)
(x)
[(x:).
1+ d(2)
(9)
(2)
where L(x) 1 (a) d(2) (x) Eizg (x) 2 0. In the binary classification case,
((x) and hence
g(x)) = log Po(x)) 8RLog (z) = log d(1) 1 g(x) Pro(ar)
(x)
π
log
+ log
pe(a) which differs from log pa( only by an additive constant. Since log(-) is strictly increasing and additive constants do not change ordering, SRLog is Neyman-Pearson optimal by Corollary [
In the multiclass case, the extra term log [1
L(w) (x)) may vary across samples and can in principle
affect ordering. Under the stated assumption L(x) < (x), which is empirically supported by
high top-5 classification accuracies in prior works (Liu et al. 2021 Brock et al. (02T), this term
small and varies little, so SRLoR (x) ss log [=](
q(æ) and remains an approximately monotone proxy for
(a( log Pu
yielding (approximate) Neyman-Pearson optimality.
Theorem 2. Let Z = <(x)) E & be the feature representation of inpul æ. Lei C be the event the classifier makes a correct prediction and -C negation. Assume ZCC ~ Pe = NNe ET) and -I-C Pw = =N(u,, ).). Then the A-MDS score $L-MDS(x) is Neyman-Pearsee optimal for selective classification.
Proof. The likelihood of multivariate
Gaussian in Rd is p(z;,E
d/2 (2T) (d/) det((2) 112 exp (-)
We see that the Mahalanobis distance D(2; Σ) is proportional to the log-likelihood of the multi- variate Gaussian, As such, assuming that the underlying Pe and pw follow multivariate Gaussians of the form (Mg, Ed) and (u4" E") respectively.
MDS (x) DMDS [z; u1, Σ²) DMDD( pi Σ") Pe(2; n4, Eº) Pw (=; p. Σ")
det E
2og
+ log det E
Therefore, (D) is monotone transform of Pe/Pw and is Neyman Pearson optimal by Corol- lary I
Theorem 3. Let = 6(x) E Rd be the feature representation of input x. and let C denote the event that the classifier makes correct prediction. Suppose Z Pe and Z -C Pw are arbitrary continuous densities bounded away from zero. Let N. = |Acl and N,, = -A If + co while Pearson optimal selector.
k/N. + and k/N as Ne, N + 00, then 8 (x) is
Proof. The empirical likelihood of the KNN density estimator (Silverman 2018 Zhao & Lai given by
2022
k
pc(z) (2))
(z)
(10)
where & 2 2, ux(x) and (2) are the Euclidean distances from to its k-th nearest neighbor from A, and A,, and Va is the bait-ball volume in Rd. A classic result of non non-parametria nearest neighbor density estimation (Loftsgaarden & Quesenberry 1965) states that k oc but k/Na -
17

---

Published as a conference paper at ICLR 2026
k/N - 0, then P=(z) - P=(z) and (z) Pw) for every In other words, the empirical KNN density estimator converges to the true density under the stated asymptotic conditions.
One can that the difference in log- likelihoods is
(o) log (=)
= log (x(2) + ux()) + log Nu No
(II)
Therefore
SL.KNN(z) - log (x(2)) ) + log uk (z)
log (z( 1 N
log
(12)
Pw d
No
Since the last term is constant, (3) is a monotone transform of pe(#)
[=( Under the stated
conditions on k Ne and Nw. the empirical likelihoods converge to the true likelihoods pe and p thus $N-KNN z) is also Neassn-Pearson optimal.
Lemma Let 81 (a)) E I and (a)) E I two selector scores. Assume both are
optimal; that is, (1(x) is monotone transform of pe (1) /pw and sz(x) is a monotone transform of
(2) (2) Then for any scalar R, t(x:) 81(x) + (02(2)) is monotonic transformation of Pe /p
Pe (pe (x ) /Pw (p
Proof. Let
L (xx Pe (x) Pw (x)
Pe
(2)
(x)
L2(x)
(13)
Dio (x)
(2)
Since each score is already strictly monotone transform of L(x), are free re-express the scores
in any other convenient monotone scale without affecting relative ordering and thus Neyman- Pearson
p (z)
optimality. Without loss of generality, we will let 8; (a) = log
pe (z) i = 1,2, which are identical to
the original scores in terms sample acceptance and rejection patterns. Given I,
t(xx
si (x) + (2)
(14)
= log (z(x) + X Alog L2(x)
(15)
(x)
(16)
Pe (pe
(1) (2)
log (2)
(17)
Pw (pw
In other words, t(x) is monotone transform of the tilted likelihood ratio pe" (p( /App/////p (pe))
(2)
Recall Ho and Hi represent the hypotheses that the classifier will make a correct and wrong prediction respectively, Assuming that the density of Ho takes the form of tilted likelihood (1) (p())) /Z..
where Z, is a normalizing constant, and vice-versa for H4, then t(x: is Sayyan--Paassee optimal by Corollary
C AVERAGE TOP l A-KNN MODIFICATION
Here we discuss how Neyman- Pearson optimality of the average log-distance formulation of A-KNN can hold as described in the main text. Recall that we DRNN z; Ad) = - E log (z)))) and vice-vert versa for A,,-
For conereteness, let us consider distances to the correct set; the derivation is identical for the wrong set. In the asymptotic limit where N, is large and & No. the ball centered that just encloses its
18

---

Published as a conference paper at ICLR 2026
kth nearest neighbor (i.e.. with volume Va(ux(2) is small that the true density is essentially constant over it, so the radii of the first & neighbors are conditionally i.i.d. uniform the ball.
As such, let define the normalized variable
24 d
U₁+ =
i i= 1==1,-. L,...,,. k.
(18)
24 (z)
Note that U, E [0, 1] for all i. Since the joint distribution of U/ depends only on k, we know from the i-th order statistics of lo i..d Uniform(0. 1) variables that each U. is Beta-distributed, U3
Beta(i, -- 0 S U1 S ... Un= = 1. With some algebra, log u; (z) log Us + log (z). Then,
(o)( log (2) + 1
log U;
(19)
k: 1=1
kd
The second term converges almost surely to
1 kd log U, - à. l. log dr 1
(20)
it is sum of & ii.d. Uniform(0, 1) random variables. In other words, in the asymptotic limit the average log-distance is a monotone transform of the log-distance itself. By substituting Eq. 19]aack into Eq. for distances both correct and wrong sets, the modified A-KNN formulation remains a monotone transform o Po/Pw. thus suggesting Neyman-Pearson optimality under Corollary
as l
D ALGORITHM PSEUDOCODE FOR PROPOSED SCORES
Pseudocode for Algorithms [ MMD and its linear combination) and (Scoring with A-KNN and its linear combination) are shown on the next page.
19

---

Published as a conference paper at ICLR 2026
Algorithm 1 Scoring with A-MDS and its linear combination
Input: Trained classifier f. feature extractor á (typically penultimate or final layer of f), training set Dunin {(x, }i)), test set Diest = { optional logit-based score function (logi(())
combination weight À
Output: Selector scores s(x) for each a Diest
1: Initialize A 0 and A 0 for i to K & Class-wise correct and incorrect feature sets
2: for each (x. y) in Dwain do
3:
<- f(x)
4: z + ((x)
5:
if ÿ y then
Add z to A!
6:
7:
else
8: Add x to A
9: end if
10: end for
Compute {ug, JO Ji=1 and {ut") 2 JV=1 from Ae and Aw
12: for each x in Drest do
= +x))
14:
de t max; -2z - 14) (Σ9) - (z 141)
15:
du <- max; - (z - (4 (- 11)5)
$A-MDS(2) + de dw
17:
if' using linear combination then
s(x:) t MDs (x) X Stagil (2)
19: else
20:
s(x) + SA-MDS(2))
21: end if
22:
end for
23:
return {s(z)) for each æ E Diest
20

---

Published as a conference paper at ICLR 2026
Algorithm 2 Scoring with A KNN and its linear combination
Input: Trained classifier f, feature extractor é (typically penultimate or final layer of f). training set Dinin {( }. test set Dise = { number of neighbors k. optional logit based score function (x)) combination weight À
Output: Selector scores s(x) for each a E Dea
I: Initialize A, +- 0, Aw + 8
b Global sets of correct and incorrect features
2: for each (v. v) in Durin do
3:
f(x) +(
5: if g = then
6:
Add to A.
7: else
Add Aw
9: end if
10: end for
11: for each a in Dica do
12:
<(x:)
13: Compute U/ ik + distances from # to nearest neighbors in A.
Compute (2: i=1 &- distances from z nearest neighbors in A...
15: de +-
log(u:)
16:
duo Σ
SA-KNN( f- 1 log(v:)
du
18:
if using linear combination then
19:
s(x) + SA.XNN (ω) + X (zn
20: else
21:
<(x) SA KNN (x)
22:
end if
23:
end for
24:
return {s(w)} for each Diea
21

---

Published as a conference paper at ICLR 2026
E EXPERIMENTAL DETAILS
Image Experiments. This section outlines the models and datasets used in the vision experiments in Section For classifiers, use the ViT-H/14 variant for DFN CLIP and the "Giant" variant of EVA, both with patch size 14. Pretrained weights are obtained from the CpenCLIE Cherti et al. and t. timm libraries, respectively.
2023
ImageNet and its covariate-shifted variants are downloaded from their respective open-source repos- For ImageNetV2, we use the MalchedFrequency test to match the frequency distribution of the original ImageNet. For ImageNet-C, we evaluate using corruption level 5 to simulate the most challenging conditions. The dataset includes four corruption categories: 1) blur, 2) digital, 3) noise, and 4) weather. Each category contains multiple corruption types (e.g., Gaussian, impulse, and shot noise under the noise category). ensure balanced evaluation, we first average results within each category, then average across the four categories to give equal weight to each corruption type.
itories
For logit-based scores with CLIP, dot product between the text embedding (e)c) of class concepts с and the image embedding, o(x), where To is the text encoder of CLIP. Given class label y. construct the class concept with the template "a real, high-quality, clear and clean photo of a {y} when computing the confidence scores for logit-based selectors. We found this improve scores slightly as opposed using the default template in the original work (Radford et al., 2021). We attribute this to the hypothesis that CLIP should produce lower confidence scores when faced with covariate-shifted inputs, such sketches or corrupted images, as the eтror rate on covariate shifts are much higher (Heng & Soh 2025) than on Im-lK, which are generally clear and well-lit photographs.
required to construct logits over class concepts by taking the
Distance- basen methods such as MDS, KNN, and our proposed variants compute distances in the feature space. For CLIP, we use the output of the final layer of the vision encoder; for EVA, we use the penultimate layer output.
The hyperparameters (for linear combinations) and & (for KNN-based scores) used in the experi- ments reported in Table and 3 are summarized in Table 6 All experiments are conducted on a single NVIDIA A6000 GPU with 48GB of memory.
Language Experiments. For language experiments, fine-tune DistilBERT model on the Amazon Reviews dataset using the training pipeline provided in the official LISA repository [T]] with default hyperparameters. For distance-based selectors, we extract features from the penultimate layer of DistilBERT, consistent with the EVA setting. The values of À and & used in Table 2 are also listed All language experiments are run on single NVIDIA A6000 48GB GPU.
in Table 6
F ADDITIONAL RESULTS
The following provides additional experimental results showing risk-coverage trade-off's, hyperpa- rameter sensitivity, and performance results on both semantic and covariate shift.
httss://bbiectnet.dde/
attp:://inagnnetv2.oor
22

---

Published as a conference paper at ICLR 2026
Table Values of and for results reported in Sec.
Method
A
A
DEN CLIP
KNN
50
A-KNN
25
A-MDS- A-MDS-RLog
RLog
10000
A-KNN-RLog
10
25
Eva
KNN
50
A-KNN 1000
25
A-MDS-RLog
3-KNN-RLog
0.5
25
DistiBBERT
KNN
50
A A-KNN
25
A MDD-RLog
1000
A KNN-RLog
0.05
25
A MDS MDS-MSP
1000
A KKNN-MP
0.5
25
AURC
NAURC
Varying
Varying k
4.4
0.215
6.0
0.325
0.210
0.300
5.5
AURC
0.205
NAURC
AURC
0.275
NAURC
4.2
0.200
5.0
0.250
4.1
0.195
0.225
0.190
0.200
4.0
4.0
12
5
10
20
5
10
25
50
100
X
k (log scale)
Figure 2: Hyperparameter sensitivity plots for and for A-KNN-RLog with DFN on ImageNet The results for this combination are not highly sensitive X, while for k results plateau at around 25.
mm-1K
Im/
Im-C
MCM/MSP
Maxt
Selective Risk
Energy
MDS
1.5
1.5
1.5
1.0
03
1.0
0.
2.0
0.35
5.15
3.25
0.25
RLog
MDS
607
0.075
0.5
0.35
MDDS-RLag
&-NNN-Ling
Coverage
Figure 3: Risk-coverage curves of various selector methods for CLIP (top row) and EVA (bottom row). Our proposed methods consistently achieve the best risk-coverage tradeoff and remain stable at low coverage levels.
23

---

Published as a conference paper at ICLR 2026
Table Additional results under covariate shift using ResNet50 trained on ImageNetIK from the official Py repository. Consistent with the findings on larger models in the main text, our proposed methods outperform all baselines. These results affirm that the gains stem from algorithmic improvements in selective classification, rather than from large-scale pretraining potentially mitigating distribution shifts.
Im-1K
Im-V2
Im-S
Im-C
Avg (IK)
Method
N
A
N
N
A
N
A
MSP
9.67 0.248
10.32
0.257
23.4
0.196
22.6
0.203
16.5 0.226
MaxLogit
12.7
0.379
13.5
0.394
0.248
24.5 0.260
19.0 0.320
Energy
16.3
0.539
17.6
0.567
28.1 0.328
27.8 0.358 22.5 0.448
MDS
21.2 0.752
21.6
0.736 58.4 1.19 58.3 1.23 39.9 0.977
KNN
28.8 1.08
30.4
1.11 40.2
0.673
46.2 0.900 36.4 0.940
RLog
8.61 0.201
9.14
0.207
24.5 0.225
22.4 0.193 16.2 0.207
SIRC
18.9 0.648
19.6
0.652 46.2 0.844 40.5 0.729 31.3 0.718
A-MDS
13.0 0.392
13.7
0.399 30.3 0.390 27.3 0.338 21.1 0.380
A-KNN
15.3
0.493
16.2
0.509 36.0 0.552
33,9 0.529
25.3 0.521
A-MDS-RL.og
8.33
0.189
8.86
0.195
24.1 0.215
21.9 0.180
15.8 0.195
A-KNN-RLog
8.48
0.195
9.02
0.202
24.3
0.221
22.2 0.189
16.0 0.202
Table 8: Additional results semantic shift datasets, namely ImageNet-O,, iNaturalist, SUN and Places. The ID distribution is ImageNet-IK.
ImageNet-O
iNaturalist
SUN
Places
Avg
A
N
A
N
A
N
A
N
A
N
DFN CLIP
MSP
9.70
0.459
11.7
0.273
12.2
0.293
12.8
0.319
11.6
0.336
MDS
11.8
0.584
14.1
0.370
14.8
0.398
15.2
0.414
14.0
0.442
MaxLogit
9.75
0.462
11.9
0.282
12.6
0.311
13.4
0.343
11.9
0.350
Energy
17.7
0.930
28.1
0.935
33.2
1.14
32.9
1.13
28.0
1.03
KNN
11.3
0.552
13.2
0.334
13.9
0.360
13.9
0.362
13.1
0.402
RLog
6.21
0.253
10.0
0.205
11.2
0.254
11.5
0.267
9.73
0.245
SIRC
18.2
0.958
32.7
1.12
30.2
1.02
32.3
1.11
28.4
1.05
A A-KNN
5.93
0.237
9.42
0.181
10.6
0.228
10.4
0.220
9.09
0.217
A A-KNN-BE K-NN-LLo
5.20
0.194
8.60
0.148
9.69
0.192
9.76
0.195
8.31
0.182
EVA
MSP
4.94
0.285
8.46
0.214
9.29
0.251
10.1
0.289
8.20
0.260
MDS
4.58
0.258
6.64
0.133
7.41
0.167
7.78
0.184
6.60
0.186
MaxLogit
6.37
0.392
10.8
0.319
11.5
0.349
12.6
0.399
10.3
0.365
Energy
8.80
0.572
14.5
0.487
14.5
0.484
15.8
0.545
13.4
0.522
KNN
4.63
0.262
6.70
0.135
7.43
0.168
7.70
0.180
6.62
0.186
RLog
3.82
0.201
6.92
0.145
8.25
0.205
8.21
0.203
6.80
0.189
SIRC
5.41
0.320
9.14
0.245
10.2
0.293
11.1
0.331
8,96
0.297
A-MDS
3.49
0.177
5.63
0.087
6.72
0.136
6.95
0.146
5.70
0.137
A-MSS- MDDS-LLog
3.37
0.202
5.64
0.130
6.90
0.187
7.08
0.186
5.75
0.176
24


Reviews:
[{"review_id": "E8BQtv6lRj", "reviewer": "Reviewer_Vjb9", "summary": "This work studies the problem of selective classification where a classifier is allowed to abstain from making predictions if the model is not confident enough. This work proposed new approaches to selective classification  based on the Neyman-Pearson lemma and also unifies several existing approaches to this problem. They also provide experiments to support their theoretical results and show that their method outperforms various baselines under covariate shifts where the test input distribution is different form the train input distribution.", "questions": "- The authors in lemma 2 assume that density for each distribution takes a tilted form. Could the authors please elaborate that?\n- The authors mention in line 305 that knn-distance based classifiers are ineffective on high dimensions. Why do they work for this work?\n- Are the values of lambda and k similar across datasets and classifiers? Or, they have to be tuned separately for each setting?\n- How is g^* computed in eqn 7?\n- The authors have not defined SIRC method which is used as comparison in the results.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The authors using NP lemma to combine several existing baseline methods is simple and intuitive. \n- The authors proposed method - linear combination of distance based and logic based methods is simple and interesting.", "weaknesses": "- Theorem 2 relies on strong assumptions that the covariance distribution conditioned on the prediction is a gaussian. Theorem 3 relies on  k tending to infinity which is not practical. \n- The authors do not provide intuitive understanding of in which cases, their proposed method should perform well compared to the baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y7sgyPqq7a", "reviewer": "Reviewer_z5TG", "summary": "This paper tackles the problem of selective classification, where one has to decide whether to predict or abstain. The authors leverage the Neyman-Pearson lemma which sets up the problem as a hypothesis test between H0 (the classifier makes a correct prediction) and H1 (the classifier makes an incorrect prediction). The authors can cast existing methods such as RLog, MDS, and KNN into the Neyman-Pearson framework to show optimality. In the end, the method takes a linear combination of classifier logit scores and distance, which produce strong results on image and text classification benchmarks.", "questions": "Methods like MDS require estimating the covariance, which could be statistically expensive (require many samples) compared to RLog? The methodology in the experiments could be a bit more transparent: how many examples are required? In the end, the hybrid methods improve over existing methods, but I want to make sure I understand the resources / additional tuning that's required for each one?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper provides a unified framework based on the Neyman-Pearson lemma that captures existing methods (which are often treated as ad-hoc).\nThe paper is fairly well-written and uses proper mathematical notation.\nThe empirical results are strong.", "weaknesses": "I think the optimality of Neyman-Pearson is a bit overstated, since optimality depends crucially on the distributional assumptions being valid.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bzHnBnsp0s", "reviewer": "Reviewer_55hp", "summary": "The paper considers the problem of abstaining from making a classification decision. Abstaining can be useful when the classifier is likely to make a wrong prediction. The authors consider a formalism where abstaining decision is made by thresholding a confidence scoring function. Next, the authors review the Neyman–Pearson (NP) lemma on the optimal decision, and observe that some of the popular scoring functions can be viewed as approximations of the optimal rule. The authors then introduce two new scoring methods and describe conditions under which the methods are NP optimal. Effectiveness of the proposed scoring functions is supported by experiments on vision and language data.", "questions": "1. What amount of labelled data (e.g., relative to the amount of training data) is needed for the method to work reliably?\n2. What do assumptions of Theorem 2 mean in practice?\n3. What are the practical implications of Lemma 2?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The problem of abstaining rather than making incorrect predictions is an important practical problem\n2. The authors offer a framework to unify previous and newly proposed confidence scoring functions. Relevance to the NP lemma is an insightful observation\n3. The paper provides formal arguments (i.e., proofs) on optimality of different scores\n4. Evaluation on different datasets shows usefulness of the proposed scores\n5. The paper is clearly presented. There are minor issues, but overall the paper is easy to follow", "weaknesses": "1. On several occasions, justification of assumptions and theoretical constructs is not clear. First, it is not clear why p(y) should remain unchanged. It changes if relative frequencies of classes change. Also, it is not clear why exactly this assumption is required. Second, the practical implications of Lemma 2 are not clear. Third, Theorem 1 uses symbol \"<<\", which informally means \"much smaller\", but does not have any formal meaning\n2. The newly introduced scores are not fundamentally new, since MDS and KNN scores have already been considered\n3. On the practical side, it is not clear what amount of labelled data (e.g., relative to the amount of training data) is needed for the method to work reliably. This can be an issue, because modern classifiers can be constructed from pre-trained models with a minimum amount of training data (e.g., using few shot learning).\n4. In terms of presentation, it would be useful to define AURC within the paper. Also, I'm not sure how NP Lemma implies that \"thresholding this score yields the lowest possible selective risk for any given coverage level\"\n5. The experiments do not provide confidence intervals or p-values", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)