Plugin estimators for selective classification with out-of-distribution detection
Authors: Harikrishna Narasimhan
Aditya Krishna Menon
Wittawat Jitkrittum
Sanjiv Kumar
Venue: ICLR 2024 poster (2024)
Keywords: Selective classification
Learning to reject
Abstention
OOD detection
SCOD
Loss functions
Plug-in estimators
Statistical consistency
Primary area: learning theory
TL;DR: Plugin estimators that are theoretically grounded, effective, and generalise approaches for selective classification and OOD detection

Abstract: Real-world classifiers can benefit from the option of abstaining from predicting on samples where they have low confidence. Such abstention is particularly useful on samples which are close to the learned decision boundary, or which are outliers with respect to the training sample. These settings have been the subject of extensive but disjoint study in the selective classification (SC) and out-of-distribution (OOD) detection literature. Recent work on selective classification with OOD detection (SCOD) has argued for the unified study of these problems; however, the formal underpinnings of this problem are still nascent, and existing techniques are heuristic in nature. In this paper, we propose new plugin estimators for SCOD that are theoretically grounded, effective, and generalise existing approaches from the SC and OOD detection literature. In the course of our analysis, we formally explicate how naïve use of existing SC and OOD detection baselines may be inadequate for SCOD. We empirically demonstrate that our approaches yields competitive SC and OOD detection trade-offs compared to common baselines.

Published as a conference paper at ICLR 2024
PLUGIN ESTIMATORS FOR SELECTIVE CLASSIFICA- TION WITH
DETECTION
Harikrishna Narasimhan, Aditya Krishna Menon, Wittawat Jitkrittum, Sanjiv Kumar
Google Research
(hnarasimhan, adityakmenon, wittawat, sanjivkJagoogle.co
ABSTRACT
Real-world classifiers can benefit from optionally abstaining from predicting on samples where they have low confidence. Such abstention is particularly useful on samples which are close to the learned decision boundary, or which are outliers with respect to the training set. These settings have been the subject of extensive but disjoint study in the selective classification (SC) and duittr-diisribuutin (OOD) detection literature. Recent work selective classification with OOD detection (SCOD) has argued for the unified study of these problems; however, the formal underpinnings of this problem still nascent, and existing techniques are heuristic in nature. In this paper, we propose new plugin estimators for SCOD that are theoretically grounded, effective, and generalise existing approaches from the SC and OOD detection literature. In the course of our analysis, we formally explicate how naïve use existing SC and OOD detection baselines may be inadequate for SCOD, We empirically demonstrate that our approaches yields competitive SC and OOD detection trade-offs compared common baselines.
INTRODUCTION
Given a training sample drawn i.i.d. from a distribution Pin (e.g.. images of cats and dogs), the standard classification paradigm concerns learning classifier that accurately predicts the label for test samples drawn from Pin- However, in real-world deployment, one may encounter out-of out-of- distribution (OOD) test samples, i.e., samples drawn from some Pout * Pinn (e.g., images of aeroplanes). Out-of- distribution detection is the problem of accurately identifying such OOD samples, and has received considerable recent study (Hendrycks and Gimpel, 2017; Lee et al., 2018; Hendrycks et al., 2019; Ren et al., 2019; Huang et al., 2021; Huang and Li, 2021; Thulasidasan et al., 2021: Wang et al., 2022a; Bitterwolf et al., 2022; Katz-Samucla et al., 2022; Wei et al., 2022; Sun et al., 2022; Hendrycks et al., 2022). An accurate OOD detector allows one to abstain from making a prediction on OOD samples, rather than making egregiously incorrect prediction; this enhances reliability and trust-worthiness.
The quality of an OOD detector is typically assessed by its ability to distinguish in in-distribution (ID) versus OOD samples. However, some recent works (Kim et al., 2021; Xia and Bouganis, 2022; Cen et al., 2023; Humblot-Renaux et al., 2024) argued that to accurately capture the real-world deployment of OOD detectors, it important consider distinguishing correctly-classifee ID versus OOD and misclassified ID samples. Indeed, it is intuitive that a classifier not only abstain on OOD samples, but also abstains from predicting on "hard" (e.g., ambiguously labelled) ID samples which are likely to be misclassified. This problem is termed unknown detection (UD) in Kim et al. (2021), and selective classification with OOD detection (SCOD) in Xia and Bouganis (2022); we adopt the latter in the sequel. One may view SCOD as unification of OOD detection and the selective classification (SC) paradigm (Chow, 1970; Bartlett and Wegkamp, 2008; El-iv and Wiener, 2010; Cortes et al., 2016b; Ramaswamy et al., 2018; Ni et al., 2019; Cortes et al., 2023; Mao et al., 2024).
Both OOD detection and SC have well-established formal underpinnings, with accompanying principled techniques (Bitterwolf et al., 2022; Cortes et al., 2016b; Ramaswamy et al., 2018); however, by contrast, the theoretical understanding of SCOD is relatively nascent. Prior work on SCOD employs heuristic design to construct the rejection rule (Xia and Bouganis, 2022). Specifically, given confidence scores for correct classification, and scores for OOD detection, the mechanism heuristically combines the two scores decide whether abstain sample. It remains

---

Published as a conference paper at ICLR 2024
Table 1: Summary of two different settings for SCOD. Our goal is to learn classifier capable of rejecting both out-of-distribttion (OOD) and "hard" in-distribution (ID) samples. We present a plug-in estimator for SCOD, and apply it two settings: one with access only ID data, the other with additional access to unlabeled mixture of ID and OOD data (Katz-Samuels et al., 2022). In both cases, we reject samples by suitably combining scores that order samples based on selective classification (SC) OOD detection criteria. The former setting leverages any off-the-shelff scores for these tasks, while the latter minimises a joint loss functions to estimate these scores.
Black-box SCOD
Loss-based SCOD
Training data SC score 8sc
ID data only
ID OOD data Minimise (10), obtain maxye(e] fu()))
Any off-the-shelf technique, e-g., maxi- mum softmax prohability (Chow, 1970)
OOD score Sood
Any off the-shelf technique, c.g., gradi- ent norm (Huang et al., 2021)
Minimise (10), obtain s(x)
Rejection rule
Combine Sac, Sood via (8)
unclear if there settings where this approach may fail, and what the optimal combination strategy would look like. We aim to address these questions in this paper.
More concretely, we provide a statistical formulation for the SCOD problem, and derive the Bayes- optimal solution. Based on this solution, we propose a plug-in approach for SCOD: this takes confidence estimates for SC and density estimates for OOD detection, and optimally combines them to output a rejection decision. We then show case how our plug-in in approach can be applied under two different assumptions on available data during training (Table 1). The first is the challenging setting where one has access to only II) data, and we leverage existing techniques for SC and OOD detection in a black-box manner. The second is the setting of Eatz-Samuels et al. (2022), where one additionally has access to unlabeled "wild" sample comprising mixture of both ID and OOD data, and one can use techniques such as Thulasidasan et al. 1); Bitterwolf et al. (2022) to jointly estimate scores for SC and OOD detection. In summary, our contributions are:
(i) We provide a statistical formulation for SCOD that unifies both the SC and OOD detection prob- lems ($3), and derive the corresponding Bayes-optimal solution (Lemma 3.1), which combines scores for SC and OOD detection. Intriguingly this solution is a variant of the popular maximum softmax probability baseline for SC and OOD) detection (Chow, 1970; Hendrycks and Gimpel, 2017), using a sample-dependen rather than constant threshold.
(ii) Based on the form of the Bayes-optimal solution, we propose a plug-in approach for SCOD (84), and showcase how it can be applied to a setting with access to only ID data ($4. 1), and the setting of Katz-Samuels et al. (2022) with access to a mixture of ID and OOD) data ($4.2).
(iii Experiments on benchmark image classification datasets ($5) show that our plug-in in approach yields competitive classification and OOD detection performance at any desired abstention rate, compared to the heuristic approach of Xia and Bouganis (2022), and other common baselines.
2 BACKGROUND AND NOTATION
We focus on multi-class classification problems: given instances X., labels y = [L], and a training sample S = (II, yn yn) INENN] (X x y) comprising N i.i.d. draws from training (or inlier) distribution Pin. the goal is to learn a classifier h: X - y with minimal misclassification error Pio(y * h(x)) for a test distribution PPee- By default, it is assumed that the training and test distribution coincide, i.c., Pre = Pin. Typically, h(z) = argmax (EL)
f (z). where f: X + g.. scores the affinity of each label to a given instance. One may learn f via minimisation of the empirical surrogate risk R( S.
|S| Σ E(ennnes Ely. ((x,)) for loss function E: [L] x RL + R..
The standard classification setting requires making a prediction for all test samples. However, as we now detail, it is often prudent to allow the classsifer to abstain from predicting on some samples.
Selective classification (SC). In selective classification (SC) (Geifman and El- Yaniv, 2019), closely related to the learning to reject or learning with abstention (Bartlett and Wegkamp, 2008; Cortes
2

---

Published as a conference paper at ICLR 2024
et al., 2016a; Gangrade et al., 2021) problem, one may abstain from predicting samples where a classifier has low-confidence. Intuitively, this allows for abstention on "hard" (e.g., ambiguously labelled) samples, which could be forwarded to expert (e.g.. human labeller). Formally, given budget beej E (0. ) on the fraction of samples that can be rejected, one learns a classifier h: X -> y and rejector r: X {0, to minimise the misclassification error on non-rejected samples:
min (y & r(x) = 0): Pin (r(z) = 1) S brej-
(I)
hrr
The original SC formulation in Geifman and EI- Yaniv (2019) conditions the misclassification error on samples that are not rejected; as shown in Appendix B, both formulations share the same optimal so- lution. The simplest SC baseline is confidence-bused rejection (Chow, 1970; Ni et al., 2019), wherein r thresholds the maximum of the softmax probability Pa( ix (ap(J(())) Alternatively, one may modify the training loss & (Bartlett and Wegkamp, 2008; Ramaswamy et al., 2018; Charoenphakdee al., 2021; Gangrade al., 2021), or jointly learn an explicit rejector and classifier (Cortes et al., 2016a; Geifman and EI- El-iv, 2019; Thulasidasan et al., 2019; Mozannar and Sontag, 2020).
OOD detection. In out-of distribution (OOD) detection, one seeks to identify test samples which are anomalous with respect to the training distribution (Hendrycks and Gimpel. 2017; Bendale and Boult, 2016; 6; Bitterwolf et al., 2022). Intuitively, this allows one to abstain from predicting on samples where it is unreasonable to expect the classifier to generalise. This is closely related to the problem of detecting whether a sample is likely to be misclassified (Granese ct al., 2021).
Formally, suppose Pte - Tin PPn (1- - for (unknown) distribution Pout and The E (0, 1). Samples from Put may be regarded as outliers or out-of-distributtin with respect to the inlier distribution (ID) Pi,. Given budget bipe E (0, 1) on the false positive rate (i.e., the fraction of ID samples incorrectly predicted as OOD), the goal is to learn an OOD detector r: X - {0, via
min, (r)t((()) = 0): (r(xr) = 1) S bipr-
(2)
Labelled OOD detection (Lee al., 2018; Thulasidasan et al., 2019) additionally accounts for the accuracy of h. OOD detection is natural task in the real-world, as standard classifiers may produce high-confidence predictions even on completely arbitrary inputs (Nguyen et al., 2015; Hendryeks and Gimpel, 2017), and assign higher scores to OOD compared to ID samples (Nalisnick et al., 2019).
Analogous to SC, a remarkably effective baseline for OOD detection that requires only ID samples is the maximum softmax probability (Hendrycks and Gimpel, 2017). possibly with temperature scaling and data augmentation (Liang et al., 2018). Recent works found that the maximum logit (Vaze et al., 2021; Hendrycks et al., 2022; Wei et al., 2022), and energy-based variants (Liu et al., 2020b) may be preferable. These may be further improved by taking into account imbalance in the ID classes (Jiang et al., 2023) and employing watermarking strategies (Wang et al., 2022b). More effective detectors can be designed in settings where one additionally has access to an OOD sample (Hendrycks et al., 2019; Thulasidasan al., 2019; Dhamija al., 2018; Katz-Samuels et al., 2022).
Selective classification with OOD detection (SCOD). SC and OOD detection both involve abstain- ing from prediction, but for subtly different reasons: SC concerns in-distribution but difficult samples, while OOD detection concerns out-of of distribution samples. In practical classifier deployment mentt one is likely to encounter both types of samples. To this end, selective classification with OOD detection (SCÓD) (Kim et al., 2021; Xia and Bouganis, 2022) allows for abstention on each sample type, with a user-specified parameter controlling their relative importance. Formally, suppose as before that Pie- = - Inin + (1 - - Poout Given budget brej E (0, 1) on the fraction of test samples that can be rejected, the goal is to learn a classifier h: X + y and a rejector r: X {0, 1} to minimise:
ming,r (1 ciu) Pin + h(x), r(x) = 0) + emn Pout (r(x) = 0): Pre(r(z) = 1) < beej. (3)
Here, Cin E 1] is user-specified cost of not rejecting an OOD sample. In Appendix C, we discuss alternate formulations for SCOD, and explain how our results seamlessly extend to such variants.
Contrasting SCOD, SC, and OOD detection. Before proceeding, it is worth pausing to emphasise the distinction between the three problems introduced above. All problems involve learning a rejector to enable the classifier from abstaining on certain samples. Crucially, SCOD encourages rejection on both ID samples that are likely to be misclassified, and OOD samples; by contrast, the SC and OOD detection problems only focus one of these cases. Recent work has observed that standard OOD detectors tend to reject misclassified ID samples (Cen et al., 2023); thus, not considering the latter can lead to overly pessimistic estimates of rejector performance.
3

---

Published as a conference paper at ICLR 2024
Given the practical relevance of SCOD, it is of interest design effective techniques for the problem, analogous to those for SC and OOD detection. Surprisingly, the literature offers only a few instances of such techniques, most notably the SIRC method of Xia and Bouganis (2022). While empirically effective, this approach is heuristic in nature. We seek to design theoretically grounded techniques that are equally effective. To that end, we begin by investigating fundamental property of SCOD.
Concurrent to this paper, we became aware the highly related work of Franc et al. (2023), who provide optimality characterisations for SCOD-like formulations. In another concurrent work, Chaudhuri and Lopez-Paz (2023) seek to jointly calibrate a model for both SC and OOD detection.
BAYES-OPTIMAL SELECTIVE CLASSIFICATION WITH OOD DETECTION
We begin our formal analysis of SCOD by deriving its associated Bayes-optimal solution, which we show combines confidence scores for SC and density ratio scores for OOD detection.
3.1 BAYES-OPTIMAL SCOD RULE: SAMPLE-DEPENDERS DEPENDENT CONFIDENCE THRESHOLDING
Before designing new techniques for SCOD, it is prudent ask: what are the theoretically optimal choices for h, r that we hope to approximate? More precisely, we seek to explicate the population SCOD objective (3) minimisers over all possible classifiers h: X + y, and rejectors X {0, 1}. These minimisers will depend on the unknown distributions Pin, Pte. and are thus not practically realisable as-is; nonetheless, they will subsequently motivate the design of simple, effective, and theoretically grounded solutions to SCOD. Further, these help study the efficacy of existing baselines.
Under mild distributional assumptions. one can apply standard Lagrangian analysis (detailed in Appendix D) to show that (3) is equivalent to minimising over h, r:
Larod (hi, r)
= =(1 Gin - Cout) Pin (y / r(), = 0) + Cin (r(x) 1) + Cout - Pouu (r(x) 0). (4)
Here, Cin: Cout E [0, 1] are distributionndeeendeet constants which encode the false negative outlier cost Cin abstention budget brej, and the proportion Tin of inliers in Pie. We shall momentarily treat these constants as fixed and known; return to the issue of suitable choices for them in $4.3. Furthermore, this formulation is fairly general and can be used to capture a variety of alternate constraints in (3) for specific choices of Gin and Cout (details in Appendix C). Note that obtain a soft-penalty version of the SC problem when Cout = 0, and the OOD detection problem when Gin + Cout 1. In general, we have the following Bayes-optimal solution for (4).
Lemma 3.1. Let (h²", r") denote any minimiser of (3). Then, for any x with Pin (a) > 0: Font(w) Pin(z)
(x) 1 (1- Cin Cout) (1 maxye[4) Pin z) + Cout
Cin
(5)
Further, r" (x: = 1 when (x) = O, and h² (x) = yrgmax,eerr) Pin (x) when r" (x) 0.
The optimal classifier h² has an unsurprising form: for non-rejected samples, we predict the label y with highest inlier class-probability Im (y r). The Bayes-optimal rejector is more interest- ing, and involves a comparison between two key quantities: the maximum inlier class-probability max [z] Pin yy] I)), and the density ratio P.n(a)
Po.)(2) These respectively reflect the confidence in the most likely label, and the confidence in the sample being inlier. Intuitively, when either of these quantities is sufficiently small, a sample is a candidate for rejection.
We now verify that Lemma 3.1 generalises existing Bayes-optimal rules for SC and OOD detection.
Special case: SC. Suppose Cout = and Cn < 1. Then, (5) reduces to Chow's 's rule (Chow, 1970; Ramaswamy et al., 2018):
(z)
= max Pin(y) x)
Gin
(6)
I-cio
Thus, samples with high uncertainty in the label distribution are rejected.
Special case: OOD detection. Suppose Cn + Cout = 1 and en < 1. Then, (5) reduces to density- based rejection (Steinwart et al., 2005; Chandola et al., 2009) when P (x) 0:
(*)
(owe(a) Gin (=)
Can
Thus, samples with relatively high density under P. Pout rejected.
4

---

Published as a conference paper at ICLR 2024
3.2 IMPLICATION: EXISTING SC AND OOD BASELINES DO NOT SUFFICE FOR SCOD
Lemma 3.1 implies that SCOD cannot be readily solved by existing SC and OOD detection base- lines. Specifically, consider the confidence-based rejection baseline, which rejects samples where max Pn (y 2:) is lower than a fixed constant. This is known as Chow's 's rule (6) in the SC litera- ture (Chow, 1970; Ramaswamy et al., 2018; Ni et al., 2019), and the maximum softmax probability (MSP) in OOD literature (Hendrycks and Gimpel, 2017); for brevity, we adopt the latter terminology. The MSP baseline does not suffice for the SCOD problem in general: even if ma Pin (y/ x) it may be optimal to reject an input x E X if IP out (x) > (I)
In fact, the MSP may result in arbitrarily bad rejection decisions. Surprisingly. this even holds in a special cases of OOD detection, such as open-set classification, wherein there strong relationship between Pin and Fout that a-priori would appear favourable to the MSP (Scheirer et al., 2013; Vaze et al., 2021). We elaborate on this with concrete examples in Appendix I.1.
One may ask whether using the maximum logit rather than softmax probability can prove successful in the open-set setting. Unfortunately, as this similarly does not include information about Pout. can also fail. For the same reason, other baselines from the OOD and SC literature can also fail; see Appendix I.3, Rather than using existing baselines as-is, we now consider more direct approach to estimating the Bayes-optimal SCOD rejector in (5), which has strong empirical performance.
4 PLUG-IN ESTIMATORS TO THE BAYES-OPTIMAL SCOD RULE
The Bayes-optimal rule in (5) provides a prescription for how to combine estimates of confidence
scores ste (z:) = maX (v æ) and density ratios Sand (1) =
(ar) to make optimal rejection
(2n
decisions for SCOD. Of course, obtaining reliable estimates of both quantities can be challenging-
Our focus in this paper is not to offer new approaches for estimating either of these quantities; rather, we seek leverage existing selective classification and OOD detection techniques to estimate sg. (x) and Sood (z), and demonstrate how optimally combining the two scores leads to improved SCOD performance in practice. We also show theoretically that the efficacy of the resulting solution would indeed depend on the quality of the individual estimates (Lemmas 4.1 and 4.2), as is also the case with prior SCOD approaches (Xia and Bouganis, 2022).
To this end, we show how this combination strategy can be applied to two popular settings in the OOD detection literature: one where there access to only samples from P.n. and the other where there is also access to unlabeled mix of ID and OOD samples (Katz-Samuels et al., 2022).
4.1 BL.AC--BOX SCOD USING ONLY ID DATA
The first setting we consider assumes access to ID samples from Pm.. One may use any cxisting SC score e.g., the maximum softmax probability estimate of Chow (1970) to obtain estimates Sec: X + R of the SC score. Similarly, we can leverage any existing OOD detection score Sood X RR that computed only from ID data, e.g., the gradient norm score of Huang et al. (2021). Given these scores, we propose the following black-box rejector:
(8) where /BB = 1- 2- Gi - Cout, and 0: ト z Observe that (8) exactly coincides with the optimal rejector (5) when Ssc, Sood equal their optimal counterparts (x) = maxve[L] (in x) and (x) = P.,(2)
(BB(Z) (z) ((1 Cia Couu) $xc(x) Cout - ((soa( tBB),
[o) Thus, as is intuitive, "BB will perform well when 8sc; 8ood perform well on their respective tasks. Below, we bound the excess risk for TBB in terms errors in the estimated scores (which can be further bounded if, e.g., the scores are result of minimising a surrogate loss).
Lemma 4.1. Suppose we have estimates Pin(y x) of the inlier class probabilities Pin(y | x)). estimates (x) of the density ratio Pin(ar) Fous(z); and SC scores (z) = max,v(L)) Pir x). Let I),, and îBB be a rejector defined according to (8) from 8 (x) and
h(x) E argmax Bood (x). Let Ip (x) = 0.5 (Pin(2)) + (r)) Then, for the SCOD-risk (3) minimizers (h" ()";:)):
Liood (he, Pnn) - Lscodd (h",r") < Exap- Ey Pin(y x) - Pim (y 2x)
(a)(
Band (x)
I+8oad
5

---

Published as a conference paper at ICLR 2024
Sub-optimality of SIRC method (Xia and Bouganis, 2022). Interestingly, this black-box rejector can be seen as a principled variant of the SIRC method of Xia and Bouganis (2022). As with TBB. SIRC works by combining rejection scores $zc(z), Snnd (2) for SC and OOD detection respectively. The key difference is that SIRC employs a multiplicatiie combination:
(2) = =
(#so(z) -a))- e( Sood (z) + aa) tSIRC.
for constants 0 02, 43, threshold ÈSTRC. and monotone transform @: 2 + t Intuitively, one rejects samples where there is sufficient signal that the sample is both near the decision boundary, and likely drawn from the outlier distribution. While empirically effective, it is not hard to see that the Bayes-optimal rejector (5) does not take the form of (9); thus, in general, SIRC may be sub-optimal. We note that this also holds for the objective considered in Xia and Bouganis (2022), which is a slight variation of (3) that enforces constraint on the ID recall.
4.2 LOSS-BASED SCOD USING ID AND OOD DATA
The second setting we consider is that of Satzela et al. (2022), which assumes access to both ID data, and a "wild" unlabeled sample comprising a mixture of ID and OOD data. As noted by Katz-Samuels et al., unlabeled "wild" data is typically plentiful, and can be collected, for example, from a currently deployed machine learning production system.
In this setting, the literature offers different loss functions (Hendrycks et al., 2019; Thulasidasan et al., 2021; Bitterwolf et al., 2022) to jointly estimate both the SC and OOD scores. We pick an adaptation of the decoupled loss proposed in Bitterwolf et al. (2022) due to its simplicity. We first describe this loss, assuming access to "clean" samples from Fout and then explain how this loss can be applied to more practical settings where we have access to only "wild" samples.
Specifically, we learn scorers f: X + R² and s: X + R. with the goal of applying a suitable transformation to (z) and s(x) to approximate (1 I) and Pin(z) We propose to minimise: (x))
(xs, Lyn fxx)]] + EEwFi [bb s(:))] + ExwZon ((bc(_1, s(x)),
(10)
where lm : [L] x RL + E. and lbc: {+1} x R + E, are strictly proper composite (Reid and Williamson, 2010) losses for multi-class and binary classification respectively. Canonical instantiations are the softmax crosspentropy (yc fx)) = log [Evetj e - (x), and the
sigmoid cross-entropy lee (2, s(x)) = log(1 + =-=((()) ). In words, we use a standard multi-class classification loss on the ID data, with an additional loss that discriminates between the ID and OOD data. Note that in the last two terms, we do not impose separate costs for the OOD detection errors.
Lemma 4.2. Let TP' (xx 2) = (x) 1(2 = 1) + (z) 1(2 = -1) denote a joint ID-OOD distribution, with z = 1 indicating an OOD sample. Suppose Eme, bo correspond to the softmax and sigmoid cross-entropy. Let (8*, 5²) ) be the minimizer of the decoupled loss in (10). For any explsx(x)) and p (x) I+oxp(-<(a])
scorers f. 8, with transformations Py(xr) = Σy explig(o))
EywPin Eveiey (pb(2)) - PPn m)]
S v2 Bea.i ( (V. (x)) Ela,)) [m (V, 8 (x:))
BEwae [p)(x)) P(((t)+Fou((z)
(ar)
V2 Beere)o [lee s(x)) - B(ere) [Cel 8* (x))).
See Appendix E for a detailed generalization bound. Thus the quality of the estimates Py (z) and P. (Ir) depend on how well we are able to optimize the classifcation loss lm and the rejector loss be
in (10). Note that eu uses only the classification scores fy(x), while bbe uses only the rejector score s(x). The two losses are thus decoupled. We may introduce coupling implicitly, by parameterising
Fyr (x) = w. (xx) and s(x) = u ((x) for shared embedding Q; or explicitly, as follows.
Practical algorithm: SCOD in the wild. The loss in (10) requires estimating expectations under P. Pout. While obtaining access to a sample drawn from P. Pont may be challenging, we adopt a similar strategy to Katz-Samucla et al. (2022), and assume access to two sets of unlabelled samples:
(A1) Smix. consisting of a mixture of inlier and outlier samples drawn i.idd. from a mixture Pmix = Tmix Pin + (1 Tmix) P. of samples observed in the wild (e.g., during deployment)
(A2) S consisting of samples certified to be strictly inlier, i.e., with P (x) = 0, Vrr St

---

Published as a conference paper at ICLR 2024
Algorithm 1 Loss-based SCOD using an unlabeled mixture of ID and OOD data
Input: Labeled Sin Pin. Unlabeled Smix ~ P mixx. Strictly inlier Sh, with Pout (2) =
2: Parameters: Costs Cins Cout (derived from Otn and brej specified in (3))
3: Surrogate loss: Find minimizers f X + Re and $ X + R of the decoupled loss:
Ei,ynss... ( + [8mm| Σ (s.D))ss (be (bc(+1,8(z)) + [Smixl Exes lbel (bl--1,((1))
4: Inlier class probabilities: Pin ())()) = 1 z ex (z)), where Z = -Ey exp(fx(x))
5: Mixture proportion: fmix 1 Exesp
6: Density ratio: (x) = Meii (exp(-$(x)) fmix)) )
7: Plug-in: Plug estimates Pin (y12),, Bood (x))
and costs Cin, Cout. into (8), and construct (h, F)
8: Output: h,
Assumption (A1) was employed in Katz- Stiizel et al. (2022), and may be implemented by collecting samples encountered "in the wild" during deployment of the SCOD classifier and rejector. Assumption (A2) merely requires identifying samples that are clearly not OOD, and is not difficult to satisfy: it may be implemented in practice by either identifying prototypical training samples', or by simply selecting random subset of the training sample. We follow the latter in our experiments.
Equipped with Smix. following Katz-Samuels al. (2022), we propose to use to approximate expectations under Fout. One challenge is that the rejection logit will now estimate Pu.(w)
Fmix rather
than (a))(ar) To resolve this, it is not hard to show that by (A2), one can estimate the latter via simple
Pan(2)
transformation (see Appendix G). Plugging these estimates into (8) then gives us approximation to
the Bayes-optimal solution. We summarise this procedure in Algorithm for the decoupled loss.
In Appendix F, we additionally discuss "coupled" variant of the loss in (10), and explain how the proposed losses relate to existing losses for OOD detection.
4.3 CHOOSING Cu AND Cout
So far, we have focused on minimising (4), which requires specifying Cin. Cout These costs need to be chosen based on parameters specified in the primal SCOD formulation in (3), namely: the abstention budget brej and the cost Ctn for non-rejection of OOD samples. The latter is a trade-off parameter also required in Xia and Bouganis (2022), and indicates how risk-averse user is to making predictions on OOD samples (a value close to 1 indicates almost no tolerance for predictions on OOD samples).
Using the Lagrangian for (3), one may set dout = Cin - - (1 - min)) and dn = À Tine where is the Lagrange multiplier and Tin is the proportion of ID samples in the test population'; the resulting
rejector takes the form r" (=:)
(1 - (m)) (1
maxye[4] PPn (y ()) cout Pont(2))
Tan(x)
We prescribe treating À as the lone tuning parameter, and tuning it so that the resulting rejector in (5) satisfies the budget constraint specified by Drej. See Appendix H for further details.
EXPERIMENTAL RESULTS
We demonstrate the efficacy of our proposed plug-in approaches to SCOD on a range of image classification benchmarks from the OOD detection and SCOD literature (Bitterwolf et al., 2022; Katz-Samuels et al., 2022; Xia and Bouganis, 2022). We report results with both pre-trained models and models trained from scratch: the latter are averaged over 5 random trials.
Datasets. We use CIFAR- 100 (Krizhevsky, 2009) and ImageNet (Deng et al., 2009) as the in- distribution (ID) datasets, and SVHN (Netzer et al., 201 1), Places365 (Zhou et al., 2017), LSUN (Yu et al., 2015) (original and resized), Texture (Cimpoi et al., 2014), CelebA (Liu et al., 2015), 300K Random Images (Hendrycks et al., 2019), OpenImages (Krasin al., 2017), OpenImages-O (Wang
1 AAs a practical example, we were classify images as either cats or dogs, is not hard to collect images that clearly show either a cat or a dog, and these would constitute strictly ID samples. In industry production settings, one may he able to estimate कं through inspection of historical logged data.
7

---

Published as a conference paper at ICLR 2024
Table 2: Area Under the Coverage Curve (AUC-RC) for methods trained with CIFAR-100 as
the IID sample and a mix of CIFAR-100 and either 300K Random Images or Open Images as the wild sample (cmn = 0.75). The wild set contains 10% ID and 90% OOD. Base model is ResNet-56. A * against a method indicates that it uses both ID and OOD samples for training. The test set contains 50% ID and 50% OOD samples. Lower is better.
ID + OOD training with P
RRndom300K
ID +OOD training with Po
-Opellmagos
Method/ /out Fout
SVHN
Places
LSUN
LSUN-R
Texture
SVHN
Places
LSUN
LSUN-R
Texture
MSP
0.307
0.338
0.323
0.388
0.344
0.307
0.338
0.323
0.388
0.344
MaxLogit
0.281
0.327
0.302
0,368
0.332
0.281
0.327
0.302
0,368
0.332
Energy
0.282
0.328
0.302
0.327
0.282
0.328
0.302
0.370
0.327
DOCTOR
0.306
0.336
0.322
0.384
0.341
0.306
0.336
0.322
0.384
0.341
SIRC 1
0.279
0.334
0.302
0,385
0.316
0.279
0.334
0.302
0,385
0.316
SIRC [Res]
0.258
0.333
0.289
0.383
0.311
0.258
0,333
0.289
0.383
0.311
OCE*
0.287
0.314
0.254
0.212
0.257
0.303
0.209
0.246
0,210
0.277
DCE*
0.294
0.325
0.246
0
0.258
0.352
0.213
0.263
0,214
0.292
OE
0.312
0.305
0.260
0.204
0.259
0.318
0.202
0.259
0.204
0.297
Plug-in BB [L,]]
0.223
0.318
0.240
0.349
0.245
0.223
0.318
0.240
0,349
0.245
Plug-in BB [Res]
0.205
0.324
0.240
0,321
0.264
0.205
0.324
0.240
0,321
0.264
Plug-in I.B
0.289
0.305
0.243
0.187
0.249
0.31.5
0.182
0.267
0.186
0.292
ct al., 2022a), iNaturalist-O (Huang and Li, 2021) and Colorectal (Kather et al., 2016) as the OOD datasets. For training, we use labeled ID samples and (optionally) an unlabeled "wild" mixture of ID and OOD samples (Fimxx = miix Pi + (1 - Tmixx .Pou.). For testing, we use OOD samples (Pitt) that may be different from those used in training (P (Po.t) We train ResNet-56 on CIFAR, and use parameter details in Appendix J.1).
pre-trained BiT ResNet-101 on ImageNet
In experiments where use both ID and OOD samples for training, the training set comprises of equal number of ID samples and wild samples. We hold out 5% of the original ID test set and use it as the "strictly inlier" sample needed to estimate Tmix for Algorithm 1. Our final test set contains equal proportions of ID and OOD samples; we report results with other choices in Appendix J.5.
solve the constrained SCOD objective in (3). One way to measure performance with respect to this objective is to measure the area under the risk-coverage curve (AUC-RC), as considered in prior work (Kim et al., 2021; Xia and Bouganis, 2022). Concretely, we plot the joint risk in (3) as function of samples abstained, and evaluate the area under the curve. This summarizes the rejector performance on both selective classification and OOD detection. For a fixed fraction brej = [Sanl Exesan 1(((x) = 1) of abstained samples, we measure the joint risk as:
Evaluation metries. Recall that our goal
( - ((n)- E(aw)ss.... 1yy # ((x), = 0) + cin Eves... 1(((x) = 0)), (11)
where Z = EZES 1(r(x)= = 0) conditions the risk on non-rejected samples, and Sall = {x= (xc, v) E S.n) Sout is the combined ID-OOD dataset. See Appendix H for details of how our plug-in estimators handle this constrained objective. We set Ct. = 0.75 here, and explore other cost parameters in Appendix J.6.
Baselines. Our main competitor SIRC from the SCOD literature (Xia and Bouganis, 2022). We compare with two variants of SIRC, which respectively use the Li-norm of the embeddings for sood. and residual score (Wang al., 2022a) instead. For completeness, we also include representative baselines from the OOD literature to show that a stand-alone OOD scorer can be sub-optimal for SCOD. We do not include an exhaustive list of OOD methods, as the task at hand is SCOD, and not stand-alone OOD detection. We include both methods that train only on the ID samples, namely, MSP (Chow, 1970; Hendryeks and Gimpel, 2017), MaxLogit (Hendrickx et al., 2021), energy-based scorer (Liu et al., 2020b), DOCTOR (Granese et al., 2021), and those which additionally use OOD samples, namely, the coupled CE loss (CCE) (Thulasidasan et al., 2021), the de-coupled CE loss (DCE) (Bitterwolf et al., 2022), and the outlier exposure (OE) (Hendrycks et al., 2019). In Appendix J, we also compare against cost-sensitive softmax (CSS) loss (Mozannar and Sontag, 2020), a representative SC baseline, neighst scorers (Sun et al., 2022), and ODIN (Liang et al., 2018).
Plug-in estimators. For a fair comparison, we implement our black-box rejector in (8) using the same Sond scorers Xia and Bouganis (2022), namely their (i) L, scorer and (ii) residual scorer;
8

---

Published as a conference paper at ICLR 2024
Table 3: AUC-RC (L) for CIFAR-100 as ID, and "wild" comprising of 90% ID and only 10% OOD.
The OOD part of the wild set is drawn from the same OOD dataset from which the test set is drawn.
ID + OOD training with P
Pon
Method / 1pte
SVHN
Places
LSUN
LSUN-R
Texture
Opentmages
MSP
0.307
0.338
0.323
0.388
0.344
0.342
MaxLogii
0,281
0.327
0.302
0.368
0.332
0.351
Energy
0,282
0.328
0.302
0.370
0.327
0.351
DOCTOR
0,306
0,336
0.322
0.384
0,341
0,342
SIRC [7.1]
0,279
0.334
0,302
0.385
0,316
0.340
STRC [Res]
0.258
0.333
0.289
0.383
0.311
0.341
CCE
0,238
0.227
0.231
0.235
0.239
0.243
DCE
0.235
0.220
0.226
0.230
0.235
0.241
OE
0,245
0.245
0.254
0.241
0.264
0.255
Plug-in in BB [f.g]
0.223
0.318
0,240
0.349
0.245
0.334
Plug-in BB [Res]
0,205
0.324
0.240
0.321
0.264
0.342
Plug-in LB"
0.221
0.199
0.209
0.215
0.218
0.225
Table 4: AUC-RC (4) for methods trained on ImageNet (inlier) with no OOD samples. The base model is a pre-trained BiT R00 Lower values are better. Additional results in App. J.9.
ID-ooly training
Method/ / Pout
Places
LSUN
CelebA
Colorectal
INaturalist-O
Texture
OpenImages-O
ImageNet-O
MSP
0.227
0.234
0.241
0.218
0.195
0.220
0.203
0.325
Maxl.ogit
0.229
0.239
0.256
0.204
0.195
0.223
0.202
0.326
Energy
0.235
0.246
0.278
0.204
0.199
0,227
0.210
0.330
DOCTOR
0,220
0.233
0.235
0.220
0.193
0.226
0.202
0,331
SIRC [1,]
0.222
0.229
0.248
0,220
0.196
0,226
0.200
0,313
SIRC [Res]
0
0.198
0.178
0.161
0.175
0,219
0.201
0.327
Plug-in BB [L,]]
0.261
0.257
0.337
0.283
0.219
0.270
0.222
0.333
Plug-in BB [Res]
0.191
0.170
0.145
0.149
0.162
0.252
0.215
0.378
use their MSP scorer for Sac. We also include our (iii) loss-based rejector based on the de-coupled (DC) loss in (10). (i) and (ii) use only ID samples; (iii) uses both ID and wild samples for training.
Tuning parameter. Each baseline has single threshold cost parameter that needs to be tuned to achieve a given rate of abstention byej (details in Appendix J.1); we aggregate performance across different abstention rates. Our plug-in method also uses single tuning parameter (details in $4.3).
Results. Our first experiments use CIFAR-100 as the ID sample. Table 2 reports results for a setting where the OOD samples used (as part of the wild set) during training are different from those used for testing (Par * ). Table 3 contains results for a setting where they are the same (Par = IPte ) In both cases, one among the three plug-in in estimators yields the lowest AUC-RC. Interestingly, when out * Ipte, the two black box (BB) plug-in estimators that use only ID-samples for training often fare better than the loss-based (LB) one which uses both ID) and wild samples for training. This is likely due to the mismatch between the training and test OOD distributions resulting in the decoupled loss yielding poor estimates of P.n (u)
When Far
the LB estimator often performs the best. Pit but
Pout (x)
Table 4 presents results with ImageNet as ID, and no OOD samples for training. The BB plug-in estimator (residual) yields notable gains on 5/8 OOD datasets. On the remaining, even the SIRC baselines are often only marginally better than MSP: this is because the grad-norm scorers used by them (and also by our estimators) are not very effective in detecting OOD samples for these datasets.
We have thus provided theoretically grounded plug-in estimators for SCOD that combine scores of SC and OOD detection, and demonstrated their efficacy on both settings that train with only ID samples, and those that additionally use a noisy OOD sample. A key element in our approach is the Sood scorer for estimating the ID-OOD density ratio, for which we employed the grad-norm based scorers ( (Wang et al., 2022a) used by prior SCOD methods (Xia and Bouganis, 2022). In the future, wish to explore other approaches for estimating the density ratio such Ren et al. (2019); Sun et al. (2022). It is also of interest to consider unifying the statistical formulation of SCOD with the problem of identifying misclassified samples (Hendrycks and Gimpel, 2017; Granese et al., 2021).

---

Published as a conference paper at ICLR 2024
REFERENCES
Peter L. Bartlett and Marten H. Wegkamp. Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 1833)1882-1884, 2008.
Abhijit Bendale and Terrance Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1563-1572, 2016.
Julian Bitterwolf, Alexander Meinke, Maximilian Augustin, and Matthias Hein. Breaking down out-of diittii-ittiittiion detection: Many methods based on OOD training data estimate a combination of the same core quantities. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 2041-2074. PMLR, 17-23 Jul 2022.
Jun Cen, Di Luan, Shiwei Zhang, Yixuan Pei. Yingya Zhang. Deli Zhao, Shaojie Shen, and Qifeng samples: Towards unified open-set recognition. Chen. The devil in the Conference on Learning Representations, 2023. URL https: In The Eleventh International /oopenreview.noe net/ /oof//oidddxxoro......o
Varun Chandola, Arindam Banerjee. and Vipin Kumar. Anomaly detection: A survey. ACM Comput. Surv., 41()), jul 2009. ISSN 0360-0300. doi: 10.1145/154 1541188 URL https::
//doi.org/ 10. 11e11111111111
1541882.
Nontawat Charoenphakdee, Zhenghang Cui, Yivan Zhang, and Masashi Sugiyama. Classification with rejection based on cost-sensitive classification. In Marina Meila and Tong Zhang, editors, Pro- ceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1507-1517. PMLR, 18-24 Jul 2021.
Kamalika Chaudhuri and David Lopez-Paz. Unified uncertainty calibration. arXiv:2310.01202, 2023.
arXiv preprint
C. Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16(1):41-46, 1970. doi: 10.1 109/TTT. 1et...997........ 1054406.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Boosting with abstention. Advances in Neural Information Processing Systems, 29: 1660-1666, 2016a.
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In ALT, 2016b.
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Theory and algorithms for learning with rejection in binary classification. Annals of Mathematics and Artificial Intelligence, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Ieee, 2009.
Akshay Raj Dhamija, Manuel Günther, and Terrance E. Boult. Reducing network agnostophobia. In Proceedings of the 32nd International Conference Neural Information Processing Systems, NIPS' 18, page 9175-9186, Red Hook, NY, USA, 2018. Curran Associates Inc.
Ran El-Yanir and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(53):1605-164) 2010. URL http::ttttp:...../t http://iiiirrrrr.////// org.papers/ html.
Charles Elkan. The foundations of cost-sensitive learning. In In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, pages 973-978, 2001.
Vojtech Franc, Daniel Prusa, and Jakub Paplham. Reject option models comprising out-of-distributin detection. arXiv preprint arXiv:2307.055 00000.00/99999 2023.
10

---

Published as a conference paper at ICLR 2024
Aditya Gangrade, Anil Kag, and Venkatesh Saligrama. Selective classification via one-sided prediction. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th In- ternational Conference Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 2179-2187. PMLR, 13-15 Apr 2021. URL https: //proceedings.mlt wtt.sssm.....r.sss.ee.e https////vo///gggaaaeeeeee html.
Saurabh Garg, Yifan Wu, Alexander J Smola, Sivaraman Balakrishnan, and Zachary Lipton. Mixture proportion estimation and pu learning: A modern approach. Advances in Neural Information Processing Systems, 34:8532- 00552-88544, 2021.
Yonatan Geifman and Ran El-Yaniv, SelectiveNet: A deep neural network with an integrated reject option. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2151-2159. PMLR, 09-15 Jun 2019.
Federica Granese, Marco Romanelli, Daniele Gorla, Catuscia Palamidessi, and Pablo Piantanida. DOCTOR: A simple method for detecting misclassification errors. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https: //t:://oppeee http//ppeeeveeeeeeeeeeeeee net/ -d
Kilian Hendrickx, Lorenzo Perini, Dries Van der Plas, Wannes Meert, and Jesse Davis. Machine learning with reject option: A survey. CoRR, abs/2107.11277 2021.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. arXiv preprint arXiv: 12261, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations, 2017.
URL https:t///// http://pppprreeeeeeeeeeeee
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In S. Bengio, H. Wallach, H. Larochlle,, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceediingg hww.pssscccipppccccpppppeepeeeee wccppppee_.fllls//ppee/////e//// files/paper/2018/ httl://d...Bccb06667777772422242
Paper. .df.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. Proceedings of the International Conference on Learning Representations, 2019.
Dan Hendryeks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mosta- jabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distributiin detection for real-world settings. In Kamalika Chaudhuri, Stefanie Jegelka, Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 8759-8773, PMLR, 17-23 Jul 2022. URL wtt...pree/////////eddreeeeee htmm.
https: //proce//p
Rui Huang and Yixuan Li. Mos: Towards scaling out-o--dist detection for large semantic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan. editors, Advances Neural Information Processing Systems, 2021. URL https://openrevieer htt.fforummid--miwl00oee
Galadrielle Humblot-Renane Sergio Escalera, and Thomas B Moeslund. A noisy elephant in the room: Is your out-of-distributinn detector robust to label noise? arXiv preprint 44X.v:2404.01775, 2024.
Xue Jiang, Feng Liu, Zhen Fang, Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Detecting oftttf--iittibutiinn data through in-distribution class prior. In International Conference on Machine Learning, pages 15067-15088. PMLR, 2023.
11

---

Published as a conference paper at ICLR 2024
Erik Jones, Shiori Sagawa, Pang Wei Koh, Ananya Kumar, and Percy Liang. Selective classification can magnify disparities across groups. In International Conference on Learning Representations. 2021.
Jakob Nikolas Kather, Cleo-Aron Weis, Francesco Bianconi, Susanne M Melchers, Lothar Schad, Timo Gaiser, Alexander Marx, and Frank Gerrit Zöllner. Multi-class texture analysis in colorcetal cancer histology. Scientific reports, 6(1):1-11, 2016.
Julian Katz-Samuels, Julia B Nakhleh, Robert Nowak, and Yixuan Li. Training OOD detectors in their natural habitats. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 10848-10865. PMLR, 17-23 Jul 2022.
Jihyo Kim, Jiin Koo, and Sangheum Hwang. A unified benchmark for the unknown detection capability of deep neural networks, 2021.
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-Fl-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, et al. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://githu. con/openimages, 2(3):18, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting out out-of-distribution samples. In International Conference on Learning Representations, 2018. URL https: http:///pppeeieeeeeeeeeeee http://perrreeeeeeeeeeeerrrrrrre wtt.:e//oorumnddryyyyaaaa...
Shiyu Liang Yixuan Li, and R. Srikant. Enhancing the reliability of out-o--distribution image detection in neural networks. In International Conference on Learning Representations, 2018. URL htttps://openrviiiie httpeeiwwwwweeeeeeeeeeeeeeee HarrummmidHHVGkKKK.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, and Qi Ju. FastBERT: a self-distiling bert with adaptive inference time. In Proceedings of ACL 2020, 2020a.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 21464-21475. Curran Asso- ciates, Inc., 2020b. URL ht htRLhttss///prrceaiinng neurips cc wwtpppppp/po///fiiiicccc Paper. pddf.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Anqi Mao, Mehryar Mohri, and Yutao Zhong. Theoretically grounded loss functions and algorithms for score-based multi-class abstention. In 7wenty-sixth Conference on Artificial Intelligence and Statistics, 2024.
Hussein Mozannar and David Sontag. Consistent estimators for learning defer an expert. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7076-7087. PMLR, 13-18 Jul 2020.
Eric T. Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Görür, and Balaji Lakshminarayanan. Do deep generative models know what they don't 't know? In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, fert/forum?? -ewiimmiddddiiiii H11xxhhCCYmm
2019. URL https:://pperreiiett
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
12

---

Published as a conference paper at ICLR 2024
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks easily fooled: High confidence predictions for unrecognizable images. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 427-436, 2015. doi: 10.11090CVPR222155729998
Chenri Ni, Nontawat Charoenphakdee, Junya Honda, and Masashi Sugiyama. On the calibration of multiclass classification with rejection. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence Alché-Bue, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pages 2382,2592,22 2019.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust label noise: loss correction approach. In Computer Vision and Pattern Recognition (CVPR), pages 2233-2241, 2017.
Harish G. Ramaswamy, Ambuj Tewari, and Shivani Agarwal. Consistent algorithms for multiclass classification with an abstain option. Electronic Journal of Statisticx, 12(1):530 554, 2018. doi: 10. 10.1214/17-ESS1388
Mark D. Reid and Robert C. Williamson. Composite binary losses. Journal of Machine Learning Research, 1:2387-2422, 2010.
Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, and Balaji Lakshminarayanan. Likelihood Ratios for Out-of-Distribution Detection, pages 14707- 147718. Curran Associates Inc., Red Hook, NY, USA, 2019.
Walter J. Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E. Boult. Toward open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(7): 1757-1772, 2013. doi: 10.1 10.1009/PAMM1.202.22555
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
Ingo Steinwart, Don Hush, and Clint Scovel. A classification framework for anomaly detection. Jour- nal of Machine Learning Research, 6(8):211-232, 2005. URL http::/jmlrr.rrr http:///ijmmtttoggppprr///////// v6/s seteeiie wwweewwwattttoee www.wattos html.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 20827 20840. PMLR, 17-23 Jul 2022. URL https:: ptttp:://proceedingg
mwr. pwwps/eessssse v122//112///// 001622/sun220 sun22d. html.
Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd- Yusof. Combating label noise in deep learning using abstention. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6234-6243, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Sunil Thulasidasan, Sushil Thapa, Sayera Dhaubhadel, Gopinath Chennupati, Tanmoy Bhat- tacharya, and Jeff A. Bilmes. An effective baseline for robustness to distributional shift. CoRR, abs/2105.07107, 2021. URL https:
org 07107
Junjiao Tian, Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Exploring covariate and concept shift for detection and confidence calibration of out-of-distribution data, 2022. URL ttttps://ppe /tttp:://openrevieet www.:neefforuuii hwt.frruummiiiiiaaabooooeeeee i3abvoMoeZZ.
Alexandre B. Tsybakov. Introduction to Nonparametric Estimation. Springer- Verlag, 2009.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier all you need. arXiv preprint arXiv:21 10.00277 2021.
Rajeev Verma and Eric Nalisnick. Calibrated learning to defer with one-vs-all classifiers. arXiv preprint arXiv:2202.0366) 2022.
13

---

Published as a conference paper at ICLR 2024
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: logit matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4921-4930, 2022a.
with virtual-
Qizhou Wang, Feng Liu, Yonggang Zhang, Jing Zhang, Chen Gong, Tongliang Liu, and Bo Han. Watermarking for out-of-distribution detection. Advances in Neural Information Processing Systems, 35:15545-15577. 2022b.
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 2-631-23333 122644. PMLR, 17-23 Jul 2022. URL https://proceedingg p v162/we we i 222.. html.
Guoxuan Xia and Christos-Savvas Bouganis. Augmenting softmax information for selective classifi- cation with out-of-distribution data. ArXiv, abs/2207.075066 2022.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans loop. arXiv preprint arXiv: 015.500663.3. 2015.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6)) 2017.
14

---

Published as a conference paper at ICLR 2024
Appendix
A PROOFS
Lemma 3.1. Let r*) denote any minimiser of (3). Then, for any x E X with Pin Pin(x) > 0:
(x) = ((1- Cin Cout) (1- max Pin (y 1)) + Cout wE[L]
Pout(x)
> Cin (12)
Pin (ar)
Further, (x) = when (x) = O, and h" (x) = ( ax) when r" (a) = 0.
Proof of Lemma 3.1. We first define a joint marginal distribution commm that samples from I' (x) and Pour(2) (x) with equal probabilities. We then rewrite the objective in (4) in terms of the joint marginal distribution:
Lacod (h, r) = Bag.P comb (Ti(H()),, r(x)) + T2(h(x),,(x)))
Pin(x)
1 + (())) = =0]
Ti (1((x)))(x)) = Cin Cout)) EyyaawP
Peomb (x))
Pin(2)
(1 Cin Cout)
Pin(y/or)
Peomb (z) 1 # r(x) = 0)
VE[L]
Pin (x)
T2(h(z),(zz) Gn comb
((((x) 1) Cout 1(r(x) = 0).
The conditional risk that classifier h incurs when abstaining (i.e., predicting r(x) = 1) on fixed instance given by:
Pin(a) Cin (x)
The conditional risk associated with predicting a base class E [L] on instance æ is given by:
Pin (2)()
IP Pout (x)
- Cin Cout)
TP Promb (r) (1 (i)(())))) + Cout
P Promb (x)
The Bayes-optimal classifier then predicts the label with the lowest conditional risk. When P' (I) this amounts to predicting abstain (r(x) = 1). When Pin (x) > 0, the optimal classifier predicts r(x) when:
(1) Cin Peomb (x) < (1 Cin Cout)
(in(a:)
Pout (x)
min (1 (M)(2) + Cout
Promb (xx)
yE[L]
P Poomb (xr)
Cn PIn( < (1 Cin Cout)
(z)
min (1 (in(()):)) + Cout Poot (2)
sE[L]
Cin (in(x)) < Cin Cout) Pin(x) (1 (1-
[1-max max Pin (ul(vl)))) + Cout Pout (x) wE[L]
(1- (1-max max Pin (y)(z)) yE[L]
Pont(2)
Cin < <1 -Cut)
+ Cout
(in())
Otherwise, the classifier does not abstain (r(x:) 0), and predicts argmax (in(y) as desired.
Lemma 4.1. Suppose we have estimates Pin x)) of the inlier class probabilities Pin (v/ x). estimates Food (x) of the density ratio Pin(x)) (aa[ and SC scores (x) = max)ye(L] Pin(y | x)) Let Mx) E argmax wE() Pin(y x)), and ÂBB be a rejector defined according to (8) from (xc( and
Good (x). Let p (z) (Pin(x) (x)) Then, for the SCOD-risk (3) minimizers (h* r*):
Lscod(h, (h, fBB) Lecod(h",
< 2. Exupr
Eye[ta] Pin(y x) - Pim w)) + 4
Bnod
P I+ Food (2)
15

---

Published as a conference paper at ICLR 2024
Proof of Lemma 4.1. Let P denote the joint distribution that draws sample from In and Pout with The joint risk in (4) can be written as:
equal probability. Denote "in (x) (aa
(1)(2)
Lycod(h, (h,
= Gin - Cout) Py # h(x), =0) + Cin (rn((()) = =1 + Cout Pout(r(x) =
= Enwpr
(1 - Cin Cout) Vin (x)
Σ
Pin ()) 1(r(x) =0)
+in (2) 1 ((((x) = 1) + Cout (1 - (-n((()))))) 1(((x) =0]
For class probability estimates Fy x)) ~ Pin (y | x), and scorers (ao(z) = max Pn x) and (2))(x)= Rd
(x) (w) we construct a classifier h(x) argmax argmma and black-box rejector:
(BB(x) =1
Cin Cout) - -1--ape $xc(x)) + Cout
(Bo0a(z))
>
Cin-
(13)
Let (h","") r"*) denote the optimal classifier and rejector as defined in (5). We then wish to bound the following regret:
Locod (h, (nn) - Locod (h" ,r*)= = Lsennah, (h, fBA) - Lsena (hi" (BB) + Locood (h", (nn) - Locond (h r"))))
temm
We first bound the first term:
term = Egap=
(1 - Cin - Couu) (mm())) ((fBB(r)=00 =0)
())
Σ
Pin x)) - Σ
Pin (a((lz))) x)) (x1
wot((a)
= Ezwg=
w(x) (
Σ
Pinly x))
Σ
Fin 1x))
vth((x)
(x)
where we denote w(x) (1 Cin Cout) nm(x)) (fBB (x) 0).
Furthermore, we can write:
term
Egap
w(x)
Pinty x)
Σ
(n (x) +
Σ
Pin x))
Σ
Pin at)
vzh((z)
(x)
(x)
< B
w(x) Σ
Pinly x)) Σ
Pin x)
Σ
Pnn
Σ
Pinly x))
vzt((()))
vth(ar)
(x)
vth
2 2xzpp
w(x)
Σ
Pin (y x)) iP.n (v [=)
vE[r.]
< 2- -2-EE-p
Σ
Pin x)) - Pi w0)
WELL]
where the third step uses the definition of and the fact that (xx > O; the last step uses the fact that w(x) <
We bound the second term now. For this, we first define:
Teros (r) = REr (11 Gin Cout) (x)) (1 max Pinn(y x:)) + Cout. (1 7in (x)) ((((x)= =0)
vE[(]
16

---

Published as a conference paper at ICLR 2024
+ Gn (nn 1(((x) = =1)
and (r) = EENB. (11-ca) Cin Cour) (a(()) (1
max Pin [x)) + Cout (1 ni (x)) ((((x) - 0) vE[L]
+ Gn 1 (x) 1(r(x) = 11
where we denote (x)
Rood (ar)
1+ Sx)d
Notice that r" minimizes [(r) over all rejectors r: : - {0, 1}. Similarly, note that fB minimizes [rr) over all rejectors r X + {0,1}
Then the second term can be written as:
terma-
Liej (fBB) Liej (1*)
= (fBB) - (re)(r))
(r*) + (re)(r")) [re)(r)) s Lue (FBB) Treg (FBB) + Liros(" (ra)
< 2. (1 Gin Cout) max P Pi |x)
max yE(L]]
Pin xx) (1)n(x)) (in (x)
.E(L]]
+ ((1 Cin Cowe) + Cout + cim) (mn - (x))
22 (1 Gin Cout)
(1)-17mn(2) Sin (x)) + 2- (1) -hin(x) (x))
< 4 |/i (x) - (x)(
Pin (z)
Food (2)
4 (x) + Pont (x)
+ Bond (x)
where the third step follows from ÁNB being minimizer of (raj(r), the fourth step uses the fact that
maX (y x) maX (y æ²)
< 1, and the fifth step uses the fact that Cin + Cout S 1.
Combining the bounds on term, and term, completes the proof.
Lemma 4.2. Let 1 (x,, 2) (I 1(2 = 1) + IPuut 1(2 - denote joint ID-OOD distribution, with -1 indicating an OOD sample. Suppose &mes 4n correspond to the softmax and sigmoid cross-entropy. Let sf", be the minimizer of the decoupled loss in (10). For any exp(fy(>))
scorers f. s, with transformations Py = Ey expliy (x))
and Td (x) = T+exp(-s(o))*
Eg-PI.. EvErey (p,(x)) - -Pin x)]
S v2 (r. Em (v. f(x)) E(r,)) Am (v, f (x))
(an( Egpp. [PL(x)) + Poul(()
S v2 v B(e,e)=p= (2, sxx)) B( [lb(2, s (x))].
Proof of Lemma 4.2. We first note that fx) x (n 2)) and s*()) log
(*(==1|>) P-(a=-110)
Regret Bound 1: We start with the first regret bound. expand the multi-class cross-entropy loss to get:
B( (yn [())) = ERNPP
Σ Py (x) log (p)(xr)
VE[L]
Begy)eBn Luncyy S (x))) = EnpPi
Σ Pin (y (x) log (Pinly x))
WE[L]
The right-hand side of the first bound can then be expanded as:
Pin x))
B(ay)~au [u, [(x)) Beaayren ( [*xx)) = En-Pi
Σ vE
Pin (y x)) log
Py
(14)
17

---

Published as a conference paper at ICLR 2024
which the KL-divergence between (v | x)) and Pz(z).
The KL-divergence between two probability mass functions p and q over 24 can be lower bounded by:
2
(L(pll4) 2
Σ Im(w) - ((u))
(15)
via Pinsker's inequality (Tsybakov, 2009, Section 2.8). Applying (15) (14), we have:
Pin x)) (z(
Σ
Pu x)) log
2 2
Σ WELL
Pin x)) p3(x))
yE 21
and therefore:
2
(( [[moly [(x)] - B(aay)e ["x)) 2
1
Σ
Pin x)) Po(cr))
WEIL]
2
1
EnwPi
Σ
Pin x)) - Po(
2
EgwPin
(y x)) po(oo||
S 22 Blo.B (enc [
(aw)) (enc(y.
f (x)))
DE
Regret Bound 2: We expand the binary sigmoid cross-entropy loss to get:
((r,)))B- [Ebo s(x)) Ex-p- [-P (a x) log (p (I)) Ip* -1 x) log (1 p1(x))]
( Uoo(,, s" (x)) En-P- (- x) log (P = 1 z)) 1 (= -1 x) log (1E (2==1 -1 xx)]]
and furthermore
[x,x)~>+ (2, [(c,)))= 5
1=(==11=)) IP
(= = - 1 xt)
Epop-
P (z 1 |x) log
-1 -1|x) -1
log
(1(x)
pl (x)
1 (IP" (z 1 x) (1)) P (2 - x) - (1 p1 (=))))
En
I (IP*(z 1 x) -1 (1)) + (1 (s = 2)) (1 p1(+))))
Ep
ExnP- [II* x) (1)
2 2- (Eg-p-
[|IP = 1 x)) - p
where the second step the bound in (15) and the last step uses Jensen's inequality, Note here that (I) serves as an approximation to Pr (= = 1 x).
Taking square-root on both sides and noting that P" x)) proof.
P=) (.. (a) completes the
B RELATIONSHIP BETWEEN SELECTIVE CLASSIFICATION AND LEARNING TO REJECT
There are two closely related formulations for classification problems with an abstention option: one is selective classification (SC), where uses the conditional error TP(y # Mxx) r(x) 0)
18

---

Published as a conference paper at ICLR 2024
Geifman and El-Yami Yaniv (2019)); the other is learning reject (L2R), where one instead uses the joint error Fy + h(x), r(x) = 0) Ramaswamy et al. (2018).
There is a one-to- amn-to-correspondeme 1) in (I), is not hard to see that:
between the two formulations: owing to the constraint P(r(z)
minP(y + h(z) ((x)) P(r(z) = 1) b
h,"
Ply h(z), 0) P(r(z) 6
=min
h,r
P(((() =0)
P(u # ((x), =0) 0) :(((()) 1) s s P(r(() 0) 2 a
-min h,n.
Ply # h(z), r()) = =0) P(((x) :(((x) 1) - min(b, 1-
-min
R,r,0
Thus, for a fixed a, the problem is equivalent to (I), with modified choice of the constraint on P(r(x) 1). The Bayes-optimal classifier is thus unaffected.
C ALTERNATE FORMULATIONS FOR SCOD
Like us, the prior work of Xia and Bouganis (2022) also formulate SCOD a constrained optimization problem, and consider two types of constraints: (i) a constraint on the total fraction of abstention in (3); and (ii) constraint on the fraction of correctly classified ID samples that were accepted:
min (1 Cia) Pin (y * h(x), r(x) 0) cia Pout (r(z) 0)
hr
(rn)
h(x)) benry
for some budget bynr- As acknowledged by limiting, it only considers abstentions the correctly classified samples (see Section 5. in their paper). They argue that the coverage constraint we employ is more appropriate for SCOD.
Xia and Bouganis (2022), the second constraint can be
Furthermore, the Lagrangian formulation we consider (4) with costs Cin and Cout is fairly general, and captures wide range of SCOD formulations. For example, we can show under mild distributional assumptions that for any SCOD problem of the following form:
min (1 ct.) Pin (y * (xx), r 0) Ctn Pour (r(x) = 0)
h.r
Koo Pin(())) = 0) + K01- Pin (r(x) = 1) K10 Pout (r(x) = 0) + K11 Pout (r(x) = 1) S 6,
where K00, K01, K10, K11, b E R4, we can formulate an equivalent objective of the form in (4), for appropriate choices of costs Gin and Cout-
D LAGRANGIAN ANALYSIS FOR SCOD
Let R)h,r) Pin(y # h(x),((z) 0) + c Four (r(z) = 0). We wish to solve (3), which is re-written below:
min R(h, r): (ro()(x) = 1) s b. 4r
The Lagrangian for this problem is given by:
=) F(h,r, R(h,r) + X ())
where À > 0 is the Lagrange multiplier. We explicate when it is admissible to use the unconstrained Lagrangian to solve the constrained problem (3).
Assumption D.I. Pin(y) x), (x),, Pout (2) and Pto(x) are continuous in s.
Theorem D.2. Under Assumption D.I, there exists X > 0 such that:
(hy, rx) argming, F(h,r,A) F(g
(ha,r3) E argmin (r(r(===1) R(h,r).
19

---

Published as a conference paper at ICLR 2024
We will find it useful to state the following lemma: Lemma D.3. Let (5) be the minimizer of F(h, r, for Lagrange multiplier X 2 0. Then:
R( x - r),,r),
for all rh,r) such that (r(x)) 1) s (r) 1).
Proof. Since (hg,7 r3) minimizes the Lagrangian, for any (h, F(i 12 S FF,,r, i... (r(x) 1) (z) Since À 2 0, for any (h, r) such that (rr((()) = 1) S (t)(r((z) 1), R(A),3)) & R(h,r),
R(h rx) - R),,r)
= =1)).
as desired.
We are now ready prove Theorem D.2.
Proof of Theorem D.2. For fixed À 0, the minimizer of the Lagrangian F(h, r, X) takes the form:
(2) argina
yr Pi a));
Pin (nn(())
(x) =
max
(1 1u)(
< A.
vE|L]
P Pout
The abstention rate for (1) can then be written as:
(ir(((()) = 1) JHCOYA Fie(x)dr.
where (xx) (maxpel Pin (v c
Pin(
Since (y x), PP (z), Puu (x) are continuous in æ, (x) is continuous in x. Furthermore, since the density (t:) is also continuous,
always find À 2 0 for which (x) = 1) b. Applying Lemma D.3 with this choice of X, we then have that R(h3, r I R(h, r) for all (h, such that IP, (r(z) 1) b.
When the underlying distributions are discrete mixed, there may be budgets b for which no equivalent Lagrange multiplier exists. In such cases, one may choose the multiplier with coverage closest to b and solve relaxation to (3).
E GENERALIZZATION ANALYSIS
For labeled set S.n P and unlabeled set Sout Pout. denote:
S* = (xx 1) v). E Sin] U {(x, -1): : Sout).
Let Nin = [8in], Nout = Soou and n = nin + Nout- We denote the expected risks by:
Rme (f) = ( [mme (y. f(x))]; Rbo(s) = E,r,))rr- (())))
and their empirical counter-parts by:
Rine() Thin (Z.
Σ
encly, f(x));
(b)
Σ (b(2,,(z))) (1,
n²
Let JF be hypothesis class of bounded scorers of the form f X + RL and G be class of bounded scorers s X + R. Let((FF denote the covering number for F with the oc-norm, and N e) similarly denote the covering number for G. Let (f. be the minimizer of the decoupled loss (f) (s) over F G. and (f* (") be the minimizers over all measurable functions.
20

---

Published as a conference paper at ICLR 2024
Lemma E.1. Suppose me bb correspond to the softmax and sigmoid cross- cross-entropy losses, with (mc(::)) S Bie and Ebel";)) Bue- Let (f,8) 8) be the minimizer of the empirical decoupled loss in (10), i.e., of Rfc(f) + (s) over F x G. Let Py(a)) Ev exp(fyr(x)) with probability at least 5 over draw of Sin and Sout
exp(fx(x)) and (1(z) "Iexpl- -(x]]
(1
2 log LoggN(G,e/L)
log(1/0)
Ex--Fin
Σ WELL
(0x(x)
Pin x)|
2 inf 5>0
E+ B
+ O
Ilin
+ V2. V Rmn
Pmm (5*);
Exnp- |pl(0)-
Pin (x)
log Nog,NN E Bbo
(V
(og(110)
1/2
2 2
2 inf e>0
+ O
(i) + PP
n
11"
V2. v RBe (s)
The proof uses the following generalization bounds based on uniform convergence (Shalev-Shwartz and Ben-David, 2014).
Lemma E.2. Suppose the losses (mc(::)) S Buc and (ne(:.)) S Bhc- For any 8 (0, 1), with probability at least - 5 over draw of Sin and Sout for any S E and s' E G:
(v
2. log (N(F,e/L)
1og(1/0)
(f') RSn S 2. inf r00
+ BLng
O
Aia
log (N(9, c))
(v log(1/5)
Rno(d') - Rno(dd)
S 2. inf 6s0
+ Bbe
n
11
Proof of Lemma E.2. We prove the second bound. The first bound follows through similar arguments. Let C = {(I, y) + s(x)) s E G} be the class of sigmoid cross-entropy losses induced by hypothesis class G. Let R(C) denote the empirical Radamacher complexity of L. Then standard two-sided Radamacher complexity bound gives us that with probability at least - å (see C. Scott, UMich EECS 598: Statistical Learning Theory, Winter 2014, Topic 10, Theorem 2):
Rsc - Rbc(s') s 2 (C(( + o (V
sup eeEg
n
We next bound R((C) in terms of the covering number of L. Fix under the oo-norm. By our assumption there exists such a cover with size [ğ] = NGGGe)). This implies that for any s E G. there exists a 3 e such that sup, (s(x) - 5(x)] S e. Since the loss be 1-Lipschitz in its second argument, this further implies that for any E G. there exists a E 9 such that (u. (en s(x)) (y, 3(x)) S
Let & be an 6-cover for
We then have:
1
R(G)
E.
sup
oi lbe (Vr, (v(,
sEg
1
= E,
sup sEG 11
the (vi, (x:))) + n
Σ ai (he((n,s(azi) - loelye, 8(1;)))
i=1
< E,
sup
9. tbe (yis 3(x:))
1
Σ |ool
(holyi,s(zz) loc 8(1.)))
sEG 1
n²
< E,
sup
Σ 1=1
o bbe (vi, 3(x:)) +
SEG 7
21

---

Published as a conference paper at ICLR 2024
where is random variable drawn uniformly from {-1, +1]=+ By Massart's lemma (Shalev-
Shwartz & Ben-David, 2014, Lemma 26.8) and using the fact that (be(..)) :) I Bbc. we have:
V2. log (19)
R(G) max SEG
Σ (lbo s(x;))²
+
n
I Bbe Vnc V2 logN(g,e
te
n
Nog
Bbe
+
This holds for any > Taking an infimum over e completes the proof.
Proof of Lemma I. Applying Lemma 4.2 f and 8, we have:
(x)) - Pin m)]]
EgnPin
vE[L]
S V2. Rmo(f) (f*)
S V2. V Bm Rmo (f) + /2. v Rmo(f) - Rmo
V2. Rma( Rin (f) + Rmo(f Rmolf) + /2. v Rmo(f) Rin (S*)
22. Pin(() Rm (f) + Rmo Rmo (f) V2. v Rmm(lF) (mm)
S V2 v /Bmmo (f) Rimo (f) Pmm(f) - (m)( + V2. v Rmoof) - lmm(s*)
< sup V2 /2 22 Rme (S') + /2 +V2 V RRmc(f) Rmc(s*);
PEF
where the third step uses the fact that Rmee (f) Rme (f). This follows from the fact that (f, 8) is minimizer of the empirical decoupled loss in (10), i.e., of Rm)(J) + Rbe(s), over F x Gi since the losses are decoupled, is a minimizer of R Rmo (f) over F. We similarly have:
Pm (x)
E
(1
Pin (x: + F out (z)
sup V2. /2. (so( (s') + V2 v Rmo (3) Rme (s*)
.'EG Substituting the right-hand sides with the bound from Lemma E.2 completes the proof.
F COUPLED LOss FOR SCOD
Our second loss function seeks to learn an augmented scorer f: X > RL+1, with the additional score corresponding to a "reject class", denoted by 1., and based on the following simple observation: define
By (x) = (1 Cin - Cout) Pin x) Powi(w) (1 - 2 Cin Cout) + Cout (mn
y' E [4] if y =1,
and let Cyr(3) (x) 2(x) for 7(x) En"E(AJu(L 2y (x). Now suppose that has estimate
of Ç. This yields an alternate plug-in estimator the Bayes-optimal SCOD rule (5):
f(x)
max (x) (z()).
(16)
>'E(L]
One may readily estimate by with a standard multi-class loss bue, with suitable modification:
E (ai
Lmecyy, f(x)) (1 cm)
E z~ii
[-, + Cout z--Fout [Lmc(L, (17)
Compared to the decoupled loss (10), the key difference is that the penalties on the rejection logit fi (Ir) involve the classification logits well.
22

---

Published as a conference paper at ICLR 2024
F.1 RELATION TO EXISTING LOSSES
Equation 10 generalises several existing proposals in the SC and OOD detection literature. In particular, it reduces to the loss proposed in Verma and Nalisnick (2022), when Pin Pott. i.e., when one only wishes to abstain on low confidence ID samples. Interestingly, this also corresponds the decoupled loss for OOD detection in Bitterwolf et al. (2022); crucially, however, they reject only based on whether fi (z) O, rather than comparing fz) and maX fw (æ). The latter essential match the Bayes-optimal predictor in (5). Similarly, the coupled loss in (17) reduces to the cost-sensitive softmax cross-entropy in Mozannar and Sontag (2020) when Cout 0, and the 1.
OOD detection loss of Thulasidasan et al. (2021) when Cin = 0, Cout
G TECHNICAL DETAILS: ESTIMATING THE OOD MIXING WEIGHT Tmix
To obtain the latter, we apply simple transformation as follows.
Lemma G.1. Suppose P Fmix
Tmix
P + Tmixx) Pout with Tmix < 1. Then, if (x)) 0.
(F Pin(2) (I)
1
P (Pmix(()) Tmix
(x)
Tmix
The above transformation requires knowing the mixing proportion Tmix of inlier samples in the unlabeled dataset. However, it measures the fraction of OOD samples during deployment, Tmix is typically unknown. We may however estimate this with (A2). Observe that for strictly inlier example E Sin we have
Fmix(()
Tmix: i.e., exp(- R Tmix- Therefore, we can estimate
(x)
-1
Sood (1))
(exp(-8(xx) - #mix))
where miix
Σ in FEST
mmix
S
We remark here that this problem is roughly akin to class prior estimation for PU learning (Garg al., 2021), and noise rate estimation for label noise (Patrini et al., 2017). As in those literatures, estimating mmix without any assumptions is challenging. Our assumption on the existence of Striet Inlier set S is analogous to assuming the existence golden label set in the label noise literature (Hendrycks et al., 2018).
Proof of Lemma G.I. Expanding the right-hand side, we have:
P (2) Tmix
mmix Pin(() + (1 Tmixx
Pout (x)
Tmix
Pin(x)
Tmix
Pin (I)
Tix
Pout (xr)
(in(x:)
as desired.
H TECHNICAL DETAILS: PLUG-IN ESTIMATORS WITH AN ABSTENTION BUDGET
Recall that the constrained SCOD objective stated in (3)
min (1 c(a) Pin (y+ * h(z),r(z) 0) Cta Pout (r((((()) 0): Pie(r(x) 1) s brej- (18)
hrr
The corresponding Lagrangian is
min max () br
where
F(h, r; 2) = (1 -rm) P. h(z),r(x) 0) + cm P (r(x) 0) + P.o(((()) = - - -bojjj
= (1 - -aa) Pin + ((z), 0) + cta Pout (r(x) = 0) + A + (in((()) = =1+
= (1 P (r(s) brej
23

---

Published as a conference paper at ICLR 2024
(1 cma) Pin
cm Pout (r(z) 0) + X Aha Pin(((x) 1)+
= =x (1 - mm) - X- - - 0) - A breij
(1 - cma) Pm ( 0) + cm Pm (r(z) 1)+
= (A) Pou Pow(((()) (r(x) = 0) +
(19)
where in the last line we define
Gm (A) X An
Cout (A) = Ctu > (1 min)
1/ A- (1 min) X brej-
Solving (19) requires optimising over both rh,,r) and X. Suppose momentarily that À fixed. Then, F(h, r; A) is exactly a scaled version of the soft-penalty objective (4). Thus, we can use Algorithm 1 to construct a plug-in classifier that minimizes the above joint risk. To find the optimal A, we only need to implement the surrogate minimisation step in Algorithm once to estimate the relevant probabilities. We can then construct multiple plug-in classifiers for different values of X, and perform an inexpensive threshold search: amongst the classifiers satisfying the budget constraint, pick the one that minimises (19).
The above requires estimating "in the fraction of inliers observed during deployment. Following (A2), one plausible estimate is Tmix, the fraction of inliers in the "wild" mixture set Smix- In some industry production settings, it may be reasonable to estimate " through, for example, inspection of logged data. This is the setting we assume in our experiments.
Remark. The previous work of Katz-Samuels al. (2022) for OOD detection also seeks to solve an optimization problem with explicit constraints on abstention rates, However, there are some subtle, but important, technical differences between their formulation and ours.
Like us, Katz-Samuels et al. (2022) also seek to jointly learn classifier and an OOD scorer, with constraints on the classification and abstention rates, given access samples from Fi, and F mixx For joint classifier h X >> [Z.] and rejector r : X + {0, 1). their formulation can be written as: min Pout (r(z) 0)
(20)
s.t.
Pm (r(z) = 1) S
P (h(x) r(x) = 0) S.
for given targets K, T (0,
While Pout is not directly available, Katz-Samuch et al. provide a simple solution to solving (20) using only access to P Pixx and Pm. They show that under some mild assumptions, replacing Pour with Pmix in the above problem does not alter the optimal solution. The intuition behind this is that when the first constraint on the inlier abstention rate is satisfied with equality, we have Fiix (r(z)- = 0) = mix (1 - Cin) + (1 Tmix) - Pout (r(z) = 0), and minimizing this objective is equivalent to minimizing the OOD objective in (20).
This simple trick of replacing Pout with mmix will only work when we have an explicit constraint on the inlier abstention rate, and will not work for the formulation we are interested in (19). This is because in our formulation, we impose a budget on the overall abstention rate (as this is a more intuitive quantity that practitioner may want constraint), and do not explicitly control the abstention rate on Pin.
In comparison to Katz-Samuels et al. (2022), the plug-in based approach we prescribe is more general, and can be applied to optimize any objective that involves weighted combination of the mis-classification error and the abstention rates on the inlier and OOD samples. This includes both the budget-constrainee problem consider in (19), and the constrained problem of Katz-Samuels et al. in (20).
ILLUSTRATING THE FAILURE OF MSP FOR OOD DETECTION
I.1 MSP FAILS FOR OPEN-SET RECOGNITION
We show that MSP may result in arbitrarily bad rejection decisions even for the special case of OOD detection wherein there is a strong relationship between P.,. and Poot that a-priori would
24

---

Published as a conference paper at ICLR 2024
appear favourable to the MSP. Specifically, given some distribution Piee over X U, consider the open-set classification (OSC) setting (Scheirer et al., 2013; Vaze et al., 2021): during training, one only observes samples from distribution Pin over X x Vin where Vin C y. Here, Pin is restriction of Pte to a subset of labels. At evaluation time, one seeks to accurately classify samples possessing these labels, while rejecting samples with unobserved labels
yin.
Under this setup, thresholding m (y x) might appear a reasonable approach. However, we now demonstrate that it may lead to arbitrarily poor decisions. In what follows, for simplicity we consider the OSC problem wherein yin y { , so that there is only one label unobserved in the We have the following.
in-distribution sample. Further, we focus on the setting where Cin + Cout
Lemma I.1. Under the open-sel selting, the Bayes-optimal classifier for the SCOD problem is:
T (x) =1 + Pie 1 Eoer F max (v' 1x)
-mx max Pto x);
J'LL
1- 1-
y'LL
where toxc = F
Gia (Lo=L) for F: + 2/(1 2). Cout Pro
Lemma 1.1 shows that the optimal decision is to reject when the maximum softmax probability (with respect to PPin) is higher than some (sample-dependen) threshold. This is the precise opposite the MSP baseline, which rejects when the maximum probability is lower than some threshold. What is the reason for this stark discrepancy? Intuitively, the issue is that we would like to threshold Fie 2), not Pinyy (y/ z): however, these two distributions may not align, the latter includes normalisation term that causes unexpected behaviour when we threshold. We make this concrete with simple example; see also Figure for illustration.
Example I.2 (Failure of MSP baseline). Consider a setting where the class probabilities (,,(y' xx) are equal for all the known classes y' * L. This implies that P Bayes-optimal classifier rejects sample when [L x))
(y/ xx) T-T Vyy & L. The On the other hand, MSP
Clty mooss rejects a sample iff the threshold tmpp < - Notice that the rejection decision is independent of the unknown class density (Lo x), and therefore will not agree with the Bayes-optimal classifier
in general. The following lemma formalizes this observation.
Lemma 1.3. Pick any Empp E (0, 1), and consider the corresponding MSP baseline which rejects x E X iff m (y z) < tmup. Then, there exists a class-probes croos-probability function Poo(y) for which the Bayes-optimal rejector [L x) tuse disagrees with MSP ttusp (0,
Proof of Lemma 1.1. Recall that in open-set classification, the outlier distribution is Pout (z) Pie(x y = L), while the training distribution is
Pin(x 3) = (te(x] y)
(v) Pin(v)
1 # L) 1 (L)
The (v).
We will find it useful to derive the following quantities.
Pin (I. y) = Tin(y) (r v)
1( + T.)
Tee(y) Poo
1 - (L)
11 $ L)
[T) Pio v)
(in(x)
Σ
Pin Pin(
*E(L]
= Σ
Tin (y) PPin( v)
DE[L]
Σ
Tto(y) Prc( v)
1 Tee
25

---

Published as a conference paper at ICLR 2024
Σ E Pro (to(a))
1 (L) VXL
Puo + I (1x)
P(o(
1 (L)
Pin(e, 8)
Pin(y 22)
Pin (x)
1(y + L.)
1 1-Two (L)
Pio(r,y)
1 - -TTe(L Piely + L x))
Pte(z)
1(y + L)
Pro x).
Pio(y + L [))
The first part follows from standard results in cost-sensitive learning (Elkan, 2001):
r (x) =1
Cin (z) Cout Pout (z) < 0
Cin (x) < Cout Pout
Cin (r * L.)
Cout Pto( y/ I.)
Gi Pro + L x) Pio = L) < Cout. Pro = L 22)- Pr + L)
Gn (y L)
Pue = L x.)
<
Cout Pre (y + L)
Pvo # L 1111 x)
Cin Pio =yybk L))
Prely = L >
Cout Po (y + L)
We further have for threshold ti +
Can Fea[LL))=
Prclyrst.))
Pirly = [1(x) x)) Lusc = Pioly # T, x))
Losc
1 Pioly F L. x))
1
2
1osc
max, 1 Pro(y
x3)
maxy HI Proly' (x)
Pue F
1 tasn
maxy' L Pto x))
max Pin (3' x)) N
V&L
1
10
That is, to reject when the maximum softmax probability is higher than some (sample- dependent) threshold.
Proof of Lemma 1.3. Fix e E 10, We consider two cases for threshold lmppi
Case (i): linep I T-T Consider a distribution where for all instances x, Pto(y = L x)) = 1
and (1 | [x) T- Vy' + L. Then the Bayes-optimal classifier accepts any instance x for all
thresholds & E (0, c). In contrast, Chow's 's rule would compute max Pin(y) x)) --1 and
thus reject all instances
Case (ii): tmsp > Z Consider a distribution where for all instances x, Pie (y L x) = e and I- Pio(y' x) I-I Vy' + L. Then the Bayes-optimal classifier would reject any instance for thresholds t (c, 1). whereas Chow's rule would accept all instances.
Taking c - O completes the proof.
1.2 ILLUSTRATION OF MSP FAILURE FOR OPEN-SET CLASSIFICATION
Figure shows a graphical illustration of the example discussed in Example 1.2, wherein the MSP baseline can fail for open-set classification. Figure 2 has another example setting.
26

---

Published as a conference paper at ICLR 2024
P*(yle)
Pa
04
0.
93
62
(a)
(b)
Figure 1: Examples of two open-set classification settings (a) and (b) with L = 10 classes, where P__(y]a)
the inlier class distributions (i x) Foo[v=10 over the first classes are identical, but the unknown class density IP' (10[z) significantly different. Consequently, the MSP baseline, which relies only the inlier class probabilities, will output the same rejection decision for both settings, whereas the Bayes-optimal classifier, which rejects by thresholding IP* (10)zr), may output different decisions for the two settings.
0.4
Negative
0.6
Inlier samples samples
0.5
Probability
0.2
0.0
0.3
-0.2
0.2
Inlier
Roject
Outtier class
0.0
-2
-0.50
1.00
0.0
Maximum intier probability
(0) Uniform outlier distribution Pout.
(b) Open-set classificution.
Figure 2: Example of two settings where the maximum softmax probability (MSP) baseline fails for OOD detection. Setting (a) considers low-density OOD detection, where positive and negative samples drawn from a one-dimensional Gaussian distribution. Samples away from the origin will have (xx)
and are thus outliers under the Bayes-optimal OOD detector. However, the MSP baseline will deem samples near the origin to be outliers, as these have maximal max, Py x). This illustrates the distinction between abstentions favoured by L,2R (low label certainty) and OOD detection (low density). Setting (b) considers open-sel classification where there are 7. = 4 total classes, with the fourth class (denoted by v) assumed to comprise outliers not seen during training. Each class-conditional is an isotropic Gaussian (left). Note that the maximum inlier class-probability Pin z) scores OOD samples significantly higher than ID samples (right). Thus, the MSP baseline, which declares samples with low max, (i æ)) as outliers, will perform poorly.
1.3 ILLUSTRATION OF MAXIMUM LOGIT FAILURE FOR OPEN-SET CLASSIFICATION
we show in Figure 3 the maximum logit computed over the inlier distribution. As with the maximum probability, the outlier samples tend to get a higher score than the inlier samples.
For the same reason, rejectors that threshold the margin between the highest and the second-highest probabilities, instead the maximum class probability, can also fail. The use of other SC methods such as the cost-sensitive softmax cross-entropy (Mozannar and Sontag, 2020) may not be successful either, because the optimal solutions for these methods have the same form as MSP.
J ADDITIONAL EXPERIMENTS
We provide details about the hyper- ppper-parameters and dataset splits used in the experiments, well as, additional experimental results and plots that were not included in the main text. The in-training experimental results are averaged over 5 random trials.
27

---

Published as a conference paper at ICLR 2024
0.40
oOD
0,30
Probabiity
0.25
0.20
0.25
0.00
Naximum
logit
Figure 3: For the same setting Figure 2. we show the maximum logit computed over the inlier distribution. with the maximum prohability, the outlier samples tend to get higher score than the inlier samples.
Rejector
Tunable
Estimated Parameter
Training Samples
Validation/Test Samples
Parameter
Black-box (BB)
ID samples
ID OOD (tc) samples
(Tab. 2-4, 6-12)
Loss-hased (I.B)
Tmix
ID. Unlabeled mix of + ID + OOD (tc) samples
(Tab. 2-3, 6-10)
OOD (tr), Strietly ID
Table 5: Summary of hyper-parameters and dataset splits for different settings. We assume the practitioner specifies Crn and broj. and that Tin is known.
J.1
CHOICES
We provide details of the learning rate (LR) schedule and other hyper-parameters used in experiments.
Dataset
Model
LR
Schedule
Epochs
Batch size
CIFAR-40/100
CIFAR ResNet 56
1.0
anneal
256
1024
We use SGD with momentum as the optimization algorithm for all models. For annealing schedule, the specified learning rate (LR) is the initial rate, which is then decayed by a factor of ten after each epoch in a specified list. For CIFAR, these epochs 15, 96, 192 and 224.
Furthermore, as noted in $4.3, the proposed plug-in in estimators requires specification cin and Cout. which parameter, and nij is the proportion of ID samples in the test population, which we assume to be known. Interestingly, we find our plug-in estimators to be robust the specification of this parameter.
given by Gn Cin X (1 (1-mī, and Cout X - where À is tunalee
Table 5 summarizes the details for both the black-box ($4.1) and loss-based ($4.2) settings.
J.2 BASELINE DETAILS
We provide further details about the baselines we compare with. The following baselines are trained only the inlier data.
MSP or Chow's 's rule: Train a scorer f - R using CE loss, and threshold the MSP to decide to abstain (Chow, 1970; Hendrycks and Gimpel, 2017).
MaxL Same as above, but instead threshold the maximum logit max (x) (Hendrickx et al., 2021).
Energy score: Same as above, but threshold the energy function - loog E, (x)) (Liu et al., 2020a).
DOCTOR: Same as above, but threshold the scorer 1 - E, (ff (Granese al., 2021).
28

---

Published as a conference paper at ICLR 2024
Table 6: AUC-RC (L) for CIFAR-100 as ID, and "wild" comprising of 90% ID and only 10% OOD. The OOD part of the wild set is drawn from the same OOD dataset from which the test set is drawn. We compare the proposed methods with the cost-sensitive softmax (CSS) carning-to--reeee loss of Mozannar and Sontag (2020) and the ODIN method of Hendrickx et al. (2021). The test set contains 50% ID and 50% OOD samples. We set Cin = 0.75.
ID OOD training with Ptr
Pie
out
Method
/Ple out
SVHN
Places
OpenImages
CSS
0.286
0.263
0.254
ODIN
0.218
0.217
0.217
Plug-in BB [L1]
0.196
0.210
0.222
Plug-in BB [Res]
0.198
0.236
0.251
Plug-in LB*
0.221
0.199
0.225
ODIN:
Train scorer f X' - RC using CE loss, and uses combination of input noise and temperature-scaled MSP to decide when to abstain Hendrickx et al. (2021).
k-NN: Train scorer f: X' - R using CE loss, compute embeddings from the embedding layer of the scorer, and threshold the (negative) 2-norm distance the k-th nearest training sample in the embedding space (Sun et al., 2022).
SIRC: Train scorer f X - TR using CE loss, and compute post-hoc deferral rule that combines the MSP score with either the Li-orrm or the residual score of the embedding layer from the scorer f (Xia and Bouganis, 2022).
CSS: Minimize the cost-sensitive softmax L2R loss of Mozannar and Sontag (2020) using only the inlier dataset to learn scorer f: X + RR++1, augmented with rejection score fi (I:), and abstain iff fi (I) > max Jy (x) + 1, for threshold t.
The following baselines additional use the unlabeled data containing mix of inlier and OOD samples.
Coupled CE (CCE): Train a scorer f: X + R++1, augmented with a rejection score fi (x) by optimizing the CCE loss of Thulasidaaan al. (2021), and abstain iff fi (x) maxy'eLL) Sy (2)+ 1, for threshold I.
De-coupled CE (DCE): Same above but uses the DCE loss of Bitterwolf al. (2022) for training.
Outlier Exposure (OE): Train scorer using the OE loss of Hendrycks et al. (2019) and threshold the MSP.
J.3 DATA SPLIT DETAILS
For the CIFAR-100 100 experiments where we use a wild sample containing a mix of ID and OOD examples, split the original CIFAR-100 training set into two halves, sample and the other half to construct the wild sample. For evaluation, we combine the orignal CIFAR-100 test set with the respective OOD test set. In each case, the larger of the ID and OOD dataset is down-sampled to match the desired ID-OOD ratio. The experimental results are averaged over 5 random trials.
half the inlier
For the pre- ireitrined ImageNet experiments, we sample equal number of examples from the ImageNet validation sample and the OOD dataset, and annotate them with the pre-trained model. The number of samples is set to the smaller of the size of the OOD dataset or 5000.
J.4 COMPARISON TO CSS AND ODIN BASELINES
We present some representative results in Table 6 comparing proposed methods against the cost-sensitive softmax (CSS) of Mozannar and Sontag (2020), a representative learning- ceaming-to--eeeer to-reject baseline, and the ODIN method of Hendrickx et al. (2021), an OOD detection baseline. As expected, the CSS baseline, which does not have OOD detection capabilities is seen to under-perform. The ODIN, baseline, the other hand, is occasionally seen to be competitive.
29

---

Published as a conference paper at ICLR 2024
Table 7: Area Under the Risk-Coverage Curve (AUC-RC) for methods trained with CIFAR-100 100 as the ID sample and mix of CIFAR-100 and 300K Random Images as the wild sample, and with the proportion of OOD samples in test set varied. The wild set contains 10% ID and 90% OOD, The test sets contain 50% ID and 50% OOD samples. Base model is ResNet-56. Cin 0.75. A against a method indicates that it uses both ID and OOD samples for training. Lower values are better.
Test OOD proportion = 0.25
Test OOD propartion = 0.75
Method/ Fout
SVIIN
Places
LSUN
LSUN-R R
Texture
SVIIN
Places
LSUN
LSUN-R
Texture
MSP
0.166
0.185
0.178
0,221
0.188
0.488
0.519
0.507
0.559
0.520
MaxLogit
0.154
0.183
0.166
0.211
0.181
0.461
0.507
0.488
0,544
0.509
Energy
0.156
0.183
0.169
0.211
0.185
0.462
0.508
0.489
0.542
0.511
DOCTOR
0.166
0,184
0.176
0.220
0.189
0.488
0,519
0.505
0.559
0.522
SIRC[4]]
0.147
0.184
0.161
0.219
0.172
0.464
0.515
0.486
0.557
0.507
STRC [Res]
0.133
0.183
0.155
0.219
0. 0.166
0.442
0.516
0.477
0.555
0.494
OCT
0.175
0.191
0.153
0.131
0.154
0.460
0.487
0.425
0.374
0.429
DCE*
0.182
0.200
0.155
0.136
0.162
0.467
0.498
0.414
0.372
0.428
OE
0.179
0.174
0.147
0.117
0.148
0.492
0.487
0.440
0.371
0.440
Plug-in BB [fa]]
0.124
0.180
0.135
0,207
0.139
0.395
0.490
0.412
0.508
0.422
Plug-in [Res]
0.180
0.134
0.194
1.146
0.378
0.503
0.416
0.476
0.451
Plug-in LB*
0.160
0.169
0.133
0.099
0.132
0.468
0.489
0.418
0.351
0.430
Test OOD proportion = 0.01
Test OOD proportion = 0.99
Method/ / Pie
SVHN
Places
LSUN
LSUN- R
Texture
SVHN
Places
LSUN
LSUN-R R
Texture
MSP
0.063
0.064
0.063
U.065
0.063
0.731
0.732
0.731
0.734
0.733
MaxLogit
0.069
0.070
0.070
0,071
0.068
0.727
0.736
0.734
0,734
0.739
Energyy
0.071
0.072
0.071
0.072
0.071
0.727
0,734
0.734
0.736
0.735
DOCTOR
0.062
0.064
0.063
0.065
0.063
0.730
0.731
0.731
0.733
0.733
SIRC |L,1
0.062
0.063
0.062
0.062
0.728
0.731
0.731
0.730
SIRC [Res]
0.062
0.063
0.062
0.065
0.062
0.726
0.731
0.730
0.734
0.731
OCE*
0.105
0.106
0.104
0.103
0.105
0.727
0.735
0.724
0,715
0.727
DCE*
0.115
0.115
0.113
0.113
0.113
0.732
0.735
0.724
0.714
0.729
OE
0.084
0.085
0.084
0.082
0.083
0.730
0.729
0.726
0.715
0.725
Plug-in [4,]
0.062
0.062
0.062
0.065
0.063
0.722
0.733
0.725
0.728
Plug in BB [Res]
0.062
0.064
0.062
0.065
0.062
0.719
0.735
0.727
0.728
0.731
Plug LB
0.065
0.065
0.064
0.062
0.062
0.727
0.729
0.724
0.709
0.725
J.5 VARYING OOD MIXING PROPORTION IN TEST SET
We repeat the experiments in Table on CIFAR-100 and 100K Random Images with varying proportions of OOD samples in the test set, and present the results in Table 7. In each case, we assume that the proportion of OOD samples in the test set is known when computing On and cout ($4.3), although we find our plug-in estimators to be robust to this parameter. We find one among the proposed plug-in methods continues to perform the best.
J.6 VARYING OOD COST PARAMETER
We repeat the experiments in Table 2 on CIFAR- and 100K Random Images with varying values of cost parameter Cin. and present the results in Table 8. One among the proposed plug-in methods continues to perform the best. The lower the value of Cin, the closer the SCOD problem in (3) is to classical OOD detection (i.e., lower is the importance given to classification accuracy on inlier samples). When en. = 1, the AUC-RC metric in (11) solely evaluates the quality of OOD detection (ignoring inlier classification performance).
30

---

Published as a conference paper at ICLR 2024
Table 8: Area Under the Risk-Coverage Curve (AUC-RC) for methods trained with CIFAR-100
as the ID sample and mix of CIFAR-100 and 300K Random Images as the wild sample, and for
different values of cost parameter ein The wild contains 10% ID and 90% OOD. The test sets contain 50% ID and 50% OOD samples. Base model is ResNet-56.
Ct
0.5
0.9
Mothod/ / p
SVHN
Places
LSUN
LSUN-R
Texture
SVHN
Places
LSUN
LSUN-R
Texture
MSP
0.256
0.271
0.266
0.297
0.275
0.336
0.376
0.358
0.442
0.381
MaxLogit
0.253
0.275
0.263
0.294
0.277
0.301
0.359
0.325
0.414
0.359
Energy
0.254
0.276
0.263
0.295
0.279
0.301
0.359
0.325
0.414
0.363
DOCTOR
0.255
0.271
0.263
0.296
0.273
0.335
0.376
0.357
0,440
0.381
[
0.248
0.271
0.259
0.267
0.300
0.372
0.324
0.346
SIRC [Res]
0.240
0.272
0.254
0.295
0.263
0.269
0.372
0.313
0.435
0.333
OCE*
0.296
0.307
0.283
0,269
0.286
0.282
0.318
0.233
0.179
0.240
DCE
0.303
0.317
0.285
0.270
0.292
0.289
0.331
0.225
0.177
0.238
OE
0.287
0.283
0.270
0,255
0.272
0.327
0.315
0.252
0.173
0.251
Plug-in 14,1
0.237
0.270
0.244
0.248
0.208
0.333
0.223
0.237
Plug-in BB [Res]
0.232
0.271
0.244
0.279
0.255
0.187
0.347
0.225
0.305
0.270
Plug-in I.B
0.256
0.265
0.243
0.222
0.245
0.299
0.326
0.234
0.165
0.246
Table 9: Area Under the Risk-Coverage Curve (AUC-RC) for methods trained with CIFAR-100 as the ID sample and a mix of CIFAR-100 and 300K Random Images as the wild sample, with 95%
confidence intervals included. The wild set contains 10% ID and 90% OOD, The test sets contain 50% ID and 50% OOD samples. Base model is ResNet-56. We set Cín 0.75.
Method/ pnr
SVHN
Places
LSUN
LSUN-R
Texture
MSP
0,307 + 0.015
0.335 : 0.017
0.322 + 0.009
0.387 = 0.027
0.340 +: 0.004
MaxLogit
0,282 + 0.014
0.327 = 0.015
0,302 + 0.009
0.368 = 0.030
0.332 + 0.007
Energy
0,282 + 0.013
0.327 = 0.015
0,300 + 0.010
0.369 = 0.031
0.329 + 0.007
DOCTOR
0.305 1 0.014
0.337 1: 0.016
0.324 1: 0.008
0.385 1 0.028
0.341 1: 0.004
SIRC 1411
0,281 +: 0.012
0.334 : 0.018
0,300 + 0.009
0.385 = 0.028
0,318 + 0.005
SIRC [Res]
0,256 0.011
0.336 0.018
0.290 + 0.007
0.382
0.028
0.309 :: 0.005
CCE*
0,288 0.017
0.315 = 0.018
0.252 + 0.004
0.213 = 0.001
0.255 + 0.004
DCE
0,295 0.015
0.326 1: 0.028
0,246 1 0.004
0.212 + 0.001
0.260 1: 0,005
O
0,313 0.015
0.304 1 0.006
0.261 1: 0.001
0.204 : 0.002
0,260 1: 0.002
Plug-in BB [L,]
0,223 + 0.006
0.318 : 0.025
0.237 + 0.008
0.351 = 0.040
0.244 + 0.004
Plug BB [Res]
0.205 0.002
0.020
0.240 1 0.005
0.319 1: 0.026
0.265 1: 0.004
Plug in LB
0,290 1 0.017
0.306 1: 0.016
0.243 1: 0.003
0.186
0.001
0.248 1: 0.006
J.7 CONFIDENCE INTERVALS
In Table 9, report 95% confidence intervals for the experiments CIFAR-100 and 100K Random Images from Table 2. In each case, the differences between the best performing plug-in method and the baselines are statistically significant.
J.8 COVARIATE-SHIFTE OOD SETTING
In Table 10, we present experimental results on covariate-shifted OOD setting, where the ID dataset is CIFAR-100 100, and the OOD dataset we evaluate on during test time is a noise corrupted version of CUFAR-100 (Hendrycks and Dietterich, 2019; Tian et al., 2022), which we refer to as CIFAR-100-C. Since both ID and OOD samples being variants of the same dataset. this task is more challenging than the previous ones. We evaluate both methods that use only ID samples, and methods that are additionally provided images from R00Kom300K, as a part of a "wild" dataset. The latter are seen to fare better, with our proposed loss based plug-in method (Plug-in LB) performing the best.
31

---

Published as a conference paper at ICLR 2024
Table 10: Area Under the Risk-Coverage Curve (AUC-RC) for methods trained with CIFAR-100 as the ID sample and mix of CIFAR-100 and either 300K Random Images as the wild sample (ot 0.75). The OOD dataset we evaluate on during test time is version of CIFAR-I00 corrupted by 15 types of noises (Hendrycks and Dietterich, 2019; Tian et al., 2022), referred to as CIFAR-100-C. The wild set contains 10% ID and 90% OOD. The test sets contain 50% and 50% OOD samples. Base model is ResNet-56. A against a method indicates that it uses both ID and OOD samples for training. Lower values are better.
Method
CIFAR-I00-C
MSP
0.359
MaxLogit
0.356
Energy
0.355
DOCTOR
0.361
SIRC [7.1]
0.357
SIRC [Res]
0.355
CCE'
0.212
DCE
0.214
OB*
0.204
Plug-in BB [fa]]
0.357
Plug-in BB [Res]
0.343
Plug-in LB"
0.185
J.9 ADDITIONAL RESULTS ON PRE- TRA-NED IMAGENET MODELS
Following Xia and Bouganis (2022), we present additional results with pre-trained models with ImageNet-200 (a subset of ImageNet with 200 classes) as the inlier dataset in Table 11. The base model is ResNet-50. In Table 12, we once again present our experiments on ImageNet with the BiT ResNet- 101 base model, with additional comparisons with the nearest-neighbor scorers of Sun et al. (2022).
K LIMITATIONS AND BROADER IMPACT
Recall that proposed plug-in rejectors seek optimize for overall classification and OOD detection accuracy while keeping the total fraction of abstentions within a limit. However, the improved overall accuracy may come at the cost of poorer performance on smaller sub-groups. For example, Jones et al. (2021) show that Chow's rule or the MSP scorer "can magnify existing accuracy disparities between various groups within population, especially in the presence of spurious correlations".. It would be of interest to carry out a similar study with the two plug-in based rejectors proposed in this paper, and understand how both their inlier classification accuracy and their OOD detection performance varies across uub-groups. It would also be of interest to explore variants of our proposed rejectors that mitigate such disparities among sub-groups.
Another limitation of our proposed plug-in rejectors is that they are only as good as the estimators for the density ratio Pin(z)
Four(z) When estimates of the density ratio are not accurate, the plug-in rejectors are seen to often perform worse than the SIRC baseline that use the same estimates.
Exploring better ways for estimating the density ratio is an important direction for future work.
Beyond SCOD, the proposed rejection strategies are also applicable to the growing literature on adaptive inference Liu et al. (2020a). With the wide adoption of large-scale machine learning models with billions of parameters, it is becoming increasingly important that we are able to perform speed up the inference time for these models. To this end, adaptive inference strategies have gained popularity, wherein one varies the amount of compute the model spends on an example, by for example, exiting early on "casy" examples. The proposed approaches for SCOD may be adapted to equip early-exit models to not only exit early on high-confidence "easy" samples, but also exit early on samples that deemed to be outliers. In the future, it would be interesting to explore the design of such early-exit models that are equipped with an OOD detector to aid in their routing decisions.
32

---

Published as a conference paper at ICLR 2024
Table 11: AUC-RC (1) for methods trained with ImageNet-200 as the inlier dataset and withou OOD samples. The hase model is pre-trained ResNet-50 model. Lower values are better.
ID-only training iNaturalist-O
Method pie
Places
LSUN
ColobA
Colorectal
Texture
ImageNet-O
Food32
0.183
0.186
0.156
0.163
0.172
0.217
MaxLogit
0.173
0.184
0.146
0.149
0.166
0.162
0,209
0.218
Energy
0.176
0.185
0.145
0.146
0.172
0.166
0.211
DOCTOR
0.179
0.185
0.152
0.155
0.159
0.170
0.226
0.175
NN 1)
0.234
0.234
0.186
0.136
0.253
0.154
0.199
0.263
NN
0.239
0.252
0.171
0.139
0.222
0.143
0.204
0.285
NN (A 10)
0.252
0.284
0.177
0.140
0.230
0.148
0.184
0.323
SIRC [A,]]
0.185
0.195
0.155
0.165
0.166
0.172
0.214
0.184
SIRC [Res]
0.180
0.179
0.137
0.140
0.151
0.167
0.219
Plug-in BB [1r]
0.262
0.261
0.199
0.225
0.228
0.270
0.298
0.240
Plug-in [Res]
0.184
0.172
0.135
0.138
0.145
0.194
0.285
0.164
ID-only training Caltech65
Method ptee
Nea--naggNNa-200
Places32
Noise
0.209
0.176
0.188
MaxLogit
0.220
0.171
0.170
0.192
Energy
0.217
0.175
0.169
0.190
DOCTOR
0.198
0.170
0.171
0.187
NN (E: 1)
0.252
0.182
0.232
0.1
NN (A:
0.280
0.182
0.227
0.141
NN (A: 10)
0.295
0.190
0.249
0.140
SIRC [T,]]
0.205
0.182
0.174
0.191
SIRC [Res]
0.204
0.177
0.173
0.136
Plug-in BB [Lr]
0.264
0.242
0.256
0.344
Plug-in BB [Res]
0.247
0.202
0.171
0.136
33

---

Published as a conference paper at ICLR 2024
Table 12: AUC-RC (1) for methods trained with ImageNet-200 as the inlier dataset and withou OOD samples. The hase model is pre-trained ResNet-50 model. Lower values are better.
ID-only training iNaturalist-O
Method pie
Places
LSUN
ColobA
Colorectal
Texture
ImageNet-O
Food32
0.183
0.186
0.156
0.163
0.172
0.217
MaxLogit
0.173
0.184
0.146
0.149
0.166
0.162
0,209
0.218
Energy
0.176
0.185
0.145
0.146
0.172
0.166
0.211
DOCTOR
0.179
0.185
0.152
0.155
0.159
0.170
0.226
0.175
NN 1)
0.234
0.234
0.186
0.136
0.253
0.154
0.199
0.263
NN
0.239
0.252
0.171
0.139
0.222
0.143
0.204
0.285
NN (A 10)
0.252
0.284
0.177
0.140
0.230
0.148
0.184
0.323
SIRC [A,]]
0.185
0.195
0.155
0.165
0.166
0.172
0.214
0.184
SIRC [Res]
0.180
0.179
0.137
0.140
0.151
0.167
0.219
Plug-in BB [1r]
0.262
0.261
0.199
0.225
0.228
0.270
0.298
0.240
Plug-in [Res]
0.184
0.172
0.135
0.138
0.145
0.194
0.285
0.164
ID-only training Caltech65
Method ptee
Nea--naggNNa-200
Places32
Noise
0.209
0.176
0.188
MaxLogit
0.220
0.171
0.170
0.192
Energy DOCTOR
0.217
0.175
0.169
0.190
0.198
0.170
0.171
0.187
NN (E: 1)
0.252
0.182
0.232
0.1
NN
0.280
0.182
0.227
0.141
NN (A: 10)
0.295
0.190
0.249
0.140
SIRC [T,]]
0.205
0.182
0.174
0.191
SIRC [Res]
0.204
0.177
0.173
0.136
Plug-in BB [Lr]
0.264
0.242
0.256
0.344
Plug-in BB [Res]
0.247
0.202
0.171
0.136
34


Reviews:
[{"review_id": "M6t8ylBwuL", "reviewer": "Reviewer_nGxu", "summary": "The paper addresses the problem of selective classification with OOD detection (SCOD), where the goal is to learn a classifier, rejector pair that can abstain on “hard” in-distribution samples as well as OOD samples. The unification of the selective classification (SC) and OOD detection areas has recently been explored, but its formal underpinnings have not been fully developed. The paper presents a formal statistical analysis of SCOD and derives the Bayes-optimal solution for the classifier, rejector pair. This solution generalizes the Bayes-optimal solutions for the SC and OOD detection problems considered separately. Based on this solution, they propose plug-in estimators for optimally combining confidence scores for SC and density-ratio scores for OOD detection. This provides a principled way of combining existing scoring methods from the SC and OOD detection areas to address the SCOD problem. They address two settings, the first one being a black-box setting with only in-distribution training data, and the second one being a loss-based setting where one additionally has access to an unlabeled mixture of in-distribution and OOD data.", "questions": "### 1. Conditional probability?\nIn the formulation of selective classification, would it not be better to use the conditional probability of misclassification given the input is accepted, i.e. $P_{in}(y \\neq h(x) \\~|\\~ r(x) = 0)$? Also, under the subsection `Evaluation metrics` on page 8, the joint risk is divided by the total number of accepted inputs, which seems to be consistent with the conditional probability.\n\n### 2. On the Lagrangian\nIt is not clear to me how to arrive at the Lagrangian in Eqn (4) from the objective (3). Based on my simplification of the Lagrangian from the SCOD objective (3), I am getting a slightly different form than that in Eqn (4). Specifically, I get $c\\_{in} = \\lambda \\pi^{\\star}\\_{in}$ and $c\\_{out} = c\\_{fn} - \\lambda (1 - \\pi^{\\star}\\_{in})$. Referring to Section 4.3, it seems like $c_{in}$ and $c_{out}$ specified here may have been swapped? \n\nAlso, it seems to me the multiplier of the first term in Eqn (4) is $(1 \\~-\\~ c\\_{out} \\~-\\~ ((1 - \\pi^{\\star}\\_{in}) / \\pi^{\\star}\\_{in}) \\\\, c\\_{in})$, rather than $1 - c\\_{in} - c\\_{out}$. \n\nIt is certainly possible I missed/messed something, but would appreciate some clarification. \n\n### 3. Need for strictly-inlier dataset\nIt seems to me that the strictly-inlier dataset $S^{\\star}\\_{in}$ is only needed for estimating the mixture proportion $\\hat{\\pi}\\_{mix}$. It is not clear why the labeled inlier dataset $S_{in}$ (with the labels discarded) cannot be used in place of $S^{\\star}\\_{in}$ for estimating $\\hat{\\pi}_{mix}$. Any theoretical reason for this?\n\n### 4. Estimation of $\\pi^{\\star}\\_{in}$\nPlease clarify if $\\pi^{\\star}\\_{in}$ used in the formulation for SCOD Eqn (3) can be estimated using $\\hat{\\pi}_{mix}$? Does it have to be estimated at test time as mentioned in Footnote 3 on page 7? From my understanding, the mixture dataset $S\\_{mix}$ is collected “in the wild” during deployment, which should give a good idea of $\\pi^{\\star}\\_{in}$ as well. \n\n### Suggestions for the Theorems/Lemmas\n- Would help to restate the Lemma/Theorem statements in the appendix.\n- Some comments/takeaways on Lemma 4.1 and Lemma 4.2 would be useful.\n- In Lemma 3.1, I believe it should be: Let $(h^\\star, r^\\star)$ denote any minimiser of (3) (not (2)).\n- In the proof of Lemma 3.1 (Appendix A), it seems like the reject class $\\perp$ is allowed to be part of the classifier $h(x)$ output. However, the classifier is originally defined as $h : \\mathcal{X} \\mapsto [L]$. Please clarify this point. \n- Some pointers could be added to the proofs. For example, Eqn (13) on page 18 follows from Pinsker’s inequality, which is worth mentioning.\n- Lemma 4.2: it should be $s^\\star$ not $r^\\star$.\n- Lemma 4.2: it would be useful to clarify that $p_{\\perp}(x)$ is an approximation for $P^\\star(z = 1 \\~|\\~ x)$. The $\\perp$ symbol is commonly used for rejection, whereas here it corresponds to the probability of accepting. \n- Lemma 4.2: for introducing coupling implicitly it should be $s(x) = u^T \\Phi(x)$ (no dependency on $y’$ here).\n- Typo in the first line of the Proof of Lemma 4.2 on page 17: it should be $s^\\star(x) = \\log(P^\\star(z=1 | x) / P^\\star(z=-1 | x))$.\n\n### Rejector definition\nMight be better to define the rejectors $r^\\star(x)$ and $r_{BB}(x)$ in Eqn (5) and Eqn (8) directly using the indicator function $\\mathbb{1}[\\cdot]$.\n\n### On the Algorithm\n- Line 3 of Algorithm 1: should it be $\\hat{f} : \\mathcal{X} \\mapsto \\mathrm{R}^L$? That is, $\\hat{f}$ predicts a logit for each of the $L$ classes.\n- Line 7 of Algorithm 1: Can directly specify the final classifier $\\hat{h}(x) = \\arg\\max_{y} \\hat{f}_y(x)$ since it does not depend on Eqn (8). Can also specify that $s\\_{sc}(x) = \\max\\_{y} \\hat{f}_y(x)$ in Eqn (8).\n\n### Other points\n1. Under `Baselines` on page 8: reference for energy-based scorer should be (Liu et al., 2020b) and (Hendrycks & Gimpel, 2017) should also be cited for MSP.\n\n2. In Table 2, why do methods such as MSP, MaxLogit, and Energy (which do not use OOD training data) have different performance under the two settings: $P^{tr}\\_{out}=$ Random300K and $P^{tr}\\_{out}=$ OpenImages? \n\n3. From Table 4, it seems that the performance of `Plug-in BB [L_1]` is consistently worse than `SIRC [L_1]`, despite the former (proposed) method being more principled. Please explain this discrepancy. \n\n4. I think it would be useful to provide results of this method using the Deep NN method (Sun et al., 2022), especially on the ImageNet dataset, since it seems like the grad-norm scorers are not providing good estimates of the density ratio.  \n\n5. It might be worth citing the following paper which characterizes the Bayes-optimal detector for mis-classification detection.  \nDoctor: A simple method for detecting misclassification errors, https://proceedings.neurips.cc/paper_files/paper/2021/hash/2cb6b10338a7fc4117a80da24b582060-Abstract.html", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Strong theoretical work that unifies the seemingly disparate literatures of selective classification and OOD detection. The proposed statistical formulation and Bayes optimal solution for the classifier and rejector is a general result that can guide the design of selective classifiers that can reject on both uncertain in-distribution (mis-classified) inputs and OOD inputs. Although the areas of selective classification and OOD detection have been independently studied well, a combined analysis of the problems in a principled setting has been lacking and somewhat heuristic. This paper addresses the gap. \n\n- The proposed plug-in estimators specify how to optimally combine existing confidence scores from the selective classification literature and density-ratio scores from the OOD detection literature. Therefore, it allows researchers in these areas to leverage, in a principled way, existing scoring methods for selective classification and OOD detection. The novelty of the paper does not lie in new approaches to estimate the confidence scores or the density-ratio scores, but rather in how to optimally combine them for the SCOD problem.\n\n- They also propose a loss-based approach for learning the classifier and rejector by leveraging an unlabeled mixture of in-distribution and OOD data “in the wild” (similar to Katz-Samuels et al., 2022). \n\n- Overall, the paper was interesting and insightful to read. There is a lot of discussion and results in the appendices which could be useful to researchers in this area.", "weaknesses": "1. The experiments mainly focus on semantic OOD (or far OOD) inputs, where there is no intersection in the label space of the in-distribution and OOD. It is also important to consider covariate-shifted OOD, e.g. which are caused due to common corruptions, noise weather changes etc. Some results on the covariate-shifted OOD data would strengthen the paper.   \n\n2. The method requires a few constants or hyper-parameters to be set. For instance, the cost of false negatives $c_{fn}$, the maximum rejection rate $b_{rej}$, the proportion of inlier and OOD data in the unlabeled set $S_{mix}$, the choice of training OOD data $P^{tr}_{out}$. The results in the main paper are for specific choices of these parameters (understably due to the page limit), but there is not much discussion or takeaways on how these parameters affect the performance. Some discussion on this would be useful. \n\n3. Minor: there is lack of clarity in some parts of the paper, which could be improved. Please see the Questions section. \n\n4. The code has not been made available and some implementation details are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7PlFaHQ3aF", "reviewer": "Reviewer_ZjGh", "summary": "This work proposes a new view of the framework for selective classification and out-of-distribution (SCOD) detection into a unified classifier, rejector tuple problem. Their framework allows for plugging in any existing OOD similarity score to perform rejection under different scenarios of data access, e.g., only in-distribution or a mixture of unlabeled IND and OOD data. They also showcase a statistical formulation for the problem and derive the Bayes optimal solution from it. They provide an empirical evaluation of the popular image classification benchmarks.", "questions": "1. Could the authors show the steps to go from (3) to (4) to make $c_{in}$ and $c_{out}$ appear as in 4.3?\n2. Could the excessive loss bound in Lemma 4.1 be rewritten by considering the estimation error of $P_{in}(y|x)$, $\\pi^*_{in}$, and $\\hat{s}_{ood}$?\n3. The constraint considered in (3) takes into account the rejection rate on the test distribution. Why not consider the rejection rate only on the in-distribution like (1) and (2)?\n4. Assumption (A2) in page 6 states that $P_{out}(x) = 0$ for $x$ in $S_{in}^{\\*}$. How to guarantee this in practice? I.e., how to build $S_{in}^{\\*}$? The proposed strategy in the footnote considers $(x,y)$ and not simply $x$. The way I see it is that it is impossible to obtain a strict $S^*_{in}$ without full knowledge on $P_{out}$.\n5.  For CIFAR, the proposed SCOD learning in algorithm 1 does not seem to yield better results than training only on the CE loss and using heuristic scores to perform SCOD. Could the authors elaborate on potential limitations on why this is the case?\n6. How does the inlier rejection option perform on this task compared to the proposed method and existing OOD detectors? Please check the references cited in the Weakness section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "An extensive theoretical formulation of the SCOD problem, with derivations for the optimal classifier rejector pair and an alternative surrogate loss.\n\nThe experimental suite is large, with experiments on both CIFAR and ImageNet datasets, comparing to previous SCOD methods (SIRC) and traditional OOD detection metrics, but not selective classification methods.", "weaknesses": "* No new insights are given on better estimating the prediction confidence or the probability ratio between in and out distributions.\n\n* Since no guarantees can be drawn for either $s_{ood}$ or $s_{sc}$, (8) or lemma 4.1 does not inherit any further guarantees. \n\n* Loss-based results are only shown for the CIFAR benchmark, not ImageNet.\n\n* The AUC-RC is compared against OOD methods but not against selective classification/misclassification detection methods such as [1], [2], [3], etc. Adding them to the benchmark would further strengthen the experiments.\n\n**Notation**: sometimes the notation is a little bit confusing. $L$ is the number of classes, the loss function, and the Lagrangian. Also, $[\\cdot]$ is a set, but {${\\cdot}$} is also a set. \n\n**Typo**: the second term of the surrogate loss in Algorithm 1 differs from the one in (10) (sampling over (x,y) instead of x). Notation might become heavy, but maybe introducing the marginal could be helpful. In algorithm 1, line 5 is not a probability if $\\hat{s}$ maps into $\\mathbb{R}$.\n\nReferences:\n\n[1] SelectiveNet. Geifman, Y. \"SelectiveNet: A Deep Neural Network with an Integrated Reject Option.\" ICML 2019. /abs/1901.09192.\\\n[2] ConfidNet. Corbière et al. \"Addressing Failure Prediction by Learning Model Confidence.\" NeurIPS 2019. /abs/1910.04851.\\\n[3] Doctor. Granes et. al. \"DOCTOR: A Simple Method for Detecting Misclassification Errors.\" NeurIPS 2021. /abs/2106.02395.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qymMn8yNq8", "reviewer": "Reviewer_WNPt", "summary": "In this paper, the authors explore a novel setting, selective classification with OOD detection (SCOD), which combines two common settings in machine learning, selective classification (SC) and out-of-distribution (OOD) detection. This investigation hold promise, as recent studies have highlighted the propensity for challenging in-distribution (ID) samples to be mistakenly classified as OOD. The authors provide a theoretical foundation for understanding SCOD, and they introduce a method that adeptly integrates existing SC and OOD detection techniques. The experimental results affirm that the proposed method performs admirably in SCOD settings.", "questions": "1. How to determine the PI_in* through inspection of logged data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The integration of SC and OOD in the setting is intriguing, as it more closely resembles real-world scenarios.\n2. This paper is well-presented, providing a detailed certification process.\n3. The proposed method demonstrates superior results in both settings.", "weaknesses": "1. How to choose hyperparameters is crucial for this combined method, and it would be beneficial if the authors could provide more detail.\n2. Evaluating performance on datasets that are exclusively OOD or SC is meaningful, as the proportion of each may vary across different scenarios.\n3. The performance in the absence of OOD samples is not outstanding, with some results even falling below those of the baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)