Rejection via Learning Density Ratios
Authors: Alexander Soen
Hisham Husain
Philip Schulz
Vu Nguyen
Venue: NeurIPS 2024 poster (2024)
Keywords: Rejection
Distributional Robust Optimization
Variational Inference
Density Ratio
Primary area: optimization
TL;DR: We provide an alternative perspective for classification with rejection by learning density ratios.

Abstract: Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions. 
The predominant approach is to alter the supervised learning pipeline by augmenting typical loss functions, letting model rejection incur a lower loss than an incorrect prediction.
Instead, we propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance.
This can be formalized via the optimization of a loss's risk with a $ \phi$-divergence regularization term.
Through this idealized distribution, a rejection decision can be made by utilizing the density ratio between this distribution and the data distribution.
We focus on the setting where our $ \phi $-divergences are specified by the family of $ \alpha $-divergence.
Our framework is tested empirically over clean and noisy datasets.

Rejection via Learning Density Ratios
Alexander Soen"
Hisham Husain"
Amazon
hisham hisham.husain@pre www.ahnorottooaiil....... com
The Australian National University exander.... soendanu. edu.au
Philip Schulz Amazon phschuul2@mazon.ccom com
Vu Nguyen
Amazon vutngndamazon.com com
Abstract
Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions. The predominant approach is to alter the supervised learning pipeline by augmenting typical loss functions, letting model rejection incur lower loss than incorrect prediction. Instead, propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes pretrained model's performance. This can be formalized via the optimization term. Through this idcalized distribution, rejection decision can be made by utilizing the density ratio between this distribution and the data distribution. We focus the setting where g-divergences specified by the family of divergence. Our framework tested empirically over clean and noisy datasets.
loss's ' risk with a g-ddivergence regularization
Introduction
Forcing Machine Learning (ML) models to always make a prediction can lead to costly consequences. Indeed, in real-world domains such automated driving, product inspection, and medical diagnosis, inaccurate prediction can cause significant real-world harm [16, 29, 53, 50]. To deal with such a proposed to modify the standard
dilemma, selective prediction and classification with rejection supervised learning setting [15, 17, 72]. The idea is to allow for a model to explicitly reject making a prediction whenever the underlying prediction would be either inaccurate and uncertain.
In classification, a confidence-bun based approach can be utilized, where a classifier is trained to out- put "margin" which is used as confidence score for rejection [7, 31, 71, 59, 53]. The model rejects whenever this confidence score is lower than an assigned threshold value. A key aspect of these approaches is that they rely good probability estimates Pr(Y X x) [60], i.e., being calibrated [71, 53]. While some approaches avoid explicit probability estimation, these are typically restricted to binary classification 31, 47]. Empirically, approaches utilizing confidence scores have shown to outperform other methods, even with simple plugin estimates for probabilities [26, 39, 58].
Another elassifier-rejectien approach aims to simultaneously train a prediction and rejection model in tandem [16, 17, 53]. These approaches are theoretically driven by the construction surrogate loss functions, but in the multiclass classification case many of these loss functions have been shown to not be suitable [53]. For the multiclass classification setting, one approach connects classification with rejection to cost-sensitive classification [14]. In practice, these classification-reeelle approaches require models to be trained from scratch using their specific loss function and architecture - there
Work done while interning Amazon. Work done while employed Amazon.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Reject if p(x) (x) T Idealized Dist. Q
Algorithm 1 Density-Raan Rejection
Require: Model h; Divergence a 2 1;
Reg. À 0: Threshold [0, 1].
Data Dist. P
1: Calibrate / if needed.
2: Calculate density ratio P
x
(either Corollary 4.1 or 4.6).
3: Normalize p with Monte-Carlo or
An idealized distribution Q is learned to mini- mizes the loss of model. We then compare Q with the original data distribution P via a density ratio p = dQ/dP. A rejection criteria is defined via threshold value
Figure 1:
Eq. (17).
4: Calculate rejector T, via Def. 3.2.
output
an existing classifier for the dataset, it must be discarded. A recent approach proposes to learn a post-hoc rejector top of a pretrained classifier via surrogate loss functions [48].
In this work, learn rejectors we shift from loss function perspective to distributional perspective. Given a model and a corresponding loss function, we find distribution where the model and loss performs "best" and compare it against the data input distribution to make rejection decisions (see Fig. I and Algorithm 1 with equations boxed). As such, the set of rejectors that we propose creates a rejection decision by considering the density ratio [67] between a "best" case (idealized) distribution and the data distribution, which can be thresholded by different values to provide different accuracy vs rejection percentage trade-offs. To learn these density ratios for rejection, we consider risk minimization problem which is regularized by --divergences [1, 19]. We study particular type of sp-divergences as our regularizer: the family of a-divergences which generalizes the KL-divergence. To this end, one of our core contributions in this work is providing various methods for constructing and approximating idealized distributions, in particular those constructed by a-divergences.
The idcalized distributions that we consider are connected to adversarial distributions examined Distributionally Robust Optimization (DRO) [63] and the distributions learned in Generalized Variational Inference (GVI) [42]; and, as such, the closed formed solutions found for our idealized distribution have utility outside of rejection. Furthermore, when utilizing the KL--divrgeecc and Bayes optimal models, we recover the well known optimal rejection policies, i.e., Chow's 's rule [15, 72]. Our rejectors are then examined empirically on different datasets and under label noise corruption.
In summary, our contributions are the following:
We present new framework for learning with rejection involving the density ratios of idcalized distributions which mirrors the distributions Iearned in DRO and GVI;
We show that rejection policies learned in framework theoretically recover optimal rejection policies, i.e., Chow's 's rule;
We derive optimal idealized distributions generated from a-divergences;
We present set of simplifying assumptions such that our framework can be utilized in practice for post-hoe rejection and verify this empirically.
2 Preliminaries
Notation Let X be domain of inputs and y be output targets. We primarily consider the case where a finite set of N labels [N]. where [N] = {o..... 13. We also consider output domains y' which is not necessarily the same as the output data y, e.g., class probabilities estimates in classification. Denote the underlying (usually unknown) input distribution as Px, E D(X x y). The marginals of joint distribution are denoted by subscript, e.g., Py D(Y) for the marginal on the label space. We denote conditional distributions, e.g., P Px For marginals on D(X), the subscripting of is implicit with P = Px- The empirical distributions of P are denoted by PN. with N denoting the number of samples used to generate it. Denote the Iverson bracket [p] = 1 if the predicate is true and [p] 0 otherwise [43]. The maximum [2]+ max(0, (0, 2} is shorthanded.
Learning with Rejection We first recall the standard risk minimization setting for learning. Suppose that are given (point-wisg loss function E: y x y' + R2o which measures the level of

---

disagreement between predictions and data. Given a data distribution Px,y and a hypothesis set of models JC, we aim to minimize the expected risk w.n.t h: X - y',
argmin E.,,, ((X)) hE3
One way of viewing learning with rejection is to augment the risk minimization framework by learning an additional rejection function r: X + [0, This can be formally defined using regularization term c E R20 which controls the rate of rejection:
argmin (k,))
EP (a..:((11xxx))) r())) h(x)) + Pfr(X) - 1),
(2)
where R denotes a hypothesis set of rejection functions. Once minimized, a combined model f which can return rejection token ① to abstain from predictions can be defined,
R
if ((x) = 1
f(x)
h((x))
if r(x) 0.
Suppose that R is complete (contains all possible rejectors). In such a case, one can see that setting c = oo reduces Eq. (2) to standard risk minimization setting given by Eq. (1). Furthermore, setting c = 0 will result in r always rejecting, i.e., the case where there is no rejection cost. Some values of e E Rz can be redundant. If the loss is bounded above by B, then any c >> B is equivalent to setting e = B- - the optimal r* will be to never reject. classification, where & taken to be the zero-one-loss, c is typically restricted to values in (0, 0.5) otherwise low confidence prediction can be accepted [53] - our work only considers this case. [59] explores the E [0.5, 1] scenario.
So far, the learning task has been left general. By considering Class Probability Estimation (CPE) [61], we recover familiar optimal rejection and classifier pairs.
Theorem 2.1 (Optimal CPE Rejection / Chow'ss 's Rule). Let us consider the binary CPE setting, where {0. 1}. y' [0, 1), and l be any proper' loss function [60] (e.g., log loss). Then w.n.t. Eq. (2), the optimal classifier is given by (x) = P|Y = X x] and the optimal rejector is given by r* (*) (x=)) (x)) 2 cl.
The theorem can easily be generalized to non-binary cases. We note that Theorem 2.1 is generaliza- tion of the well known Chow's rule of classification with rejection [15, 14]. Indeed, taking E to be the zero-one-loss function, we get r" (x) = [|2-PlY = X= x] 2 1 cl. One can further clarify this by noticing that Pr(r(X)= 1] Ep [r())] In the general proper loss case, the optimal rejector is thresholding generalized entropy function (known as the conditional Bayes risk [60]). This would correspond thresholding the class probabilities P[Y = y X x]] (via the Bayes posterior h²), but with different thresholding values per class E y (unless is symmetric loss function).
Although the objective of Eq. (2) follows the cost-based model of rejection [15], other models of rejection exist in the literature. Alternatively, the bounded improvement model of rejection [54, 29] maximizes coverage (non-rejection rate) whilst maintaining constraint the performance of the rejection measured by the selective risk (the first term of Eq. (2) inversely weighted by the coverage). Optimality conditions of the bounded improvement have been explored in [27]. In the bounded abstention model, the constraint and objective is switched - the selective risk is minimized with a constraint on minimum coverage [54]. Our objective function Eq. (2) (and the cost-based model) can be interpreted as a change in the type of risk considered and a change in the hard coverage constraint soft constraint w.r.t. bounded abstention. The optimal strategies of these approaches are explored in [28]. Alongside Section 1, further details about rejection can be found in [34].
Generalized Variational Inference Generalized Variational Inference (GVI) [42] provides
framework for a generalized set of entropy regularized risk minimization problems. In particular,
GVI generalizes Bayes' rule by interpreting the Bayesian posterior as the solution to minimization problem [73]. The Bayes' rule minimization is given by,
N Σ
argmin (E)(()
Egg
Mogp(z: (0) + KL(P || Q),
4
where 0 ~ Q denotes a set of model parameters with likelihood function p(x [)) and { denotes data. Here, P E D() denotes the prior and the optimal Q* denotes the posterior in Bayes' rule.
Prropemess ensures that the true class probability is the minimizer of the loss in expectation.
3

---

For GVI, we generalize a number of quantities. For instance, the 'loss' considered can be altered from the log- likelihood poz 0) an alternative loss function over samples {2₂}. The divergence function KI.(P Q) can also be altered to change the notion of distributional distance. Furthermore, the set of distributions being minimized (()) can also be altered to, e.g., reduce the computational cost of the minimization problem. One can thus alter Eq. (4) to give the following:
argmin L(Q) + X D(P || Q), QEQ
(5)
where > O and L, D, and denote the generalized loss, divergence, and set of distribution, respectively. Here À seeks to act a regularization constant which can be tuned.
In our work, we will consider a GVI problem which changes the loss function and divergence to define an entropy regularized risk minimization problem. Our loss function corresponds to the learning setting. For the change in divergence, we consider ye-divergence [1, 19], which are otherwise referred to as f-divergences [62] or the Csiszár divergence [18].
Definition 2.2. Let y:: IR - (-00, so] then the corresponding --divergence over non-negative measures for P, P.Q is defined as:
be convex lower semi-continuous function with ((1) =
D Q1 = {
dQ x dP
dP,
Q < P:
otherwise,
D& Q) = +00.
(6)
We note that for y-divergences, the regularization constant in Eq. (5) can be absorbed into the sp-divergence generator, i.e., A (P Q) = (P Q).
Distributionally Robust Optimization A related piece of literature is Distributionally Robust Optimization (DRO) [63], where the goal to find a distribution that maximizes (or minimizes) the expectation of a function from a prescribed unceriainty set. Popular candidates for these uncertainty sets include all distributions that are certain radius away from a fixed distribution by some divergence. p-divergences have been used to define such uncertainty set [8, 22, 45]. Given radius E >> 0, define By (P) = {Q D(X x y) : D. (P Q) < e]. Given a point-wise loss function É: y x y' + R. DRO alters risk minimization, per Eq. (I), to solve the following optimization problem:
min max NES
Eq... [(X))
(7)
E..
The max over Q is typically over the target space y. That is, Q(z, v) = Q(v) P(x y) and the max adversarial over the marginal label distribution [74]. Note that converting the e-ball constraint into a Lagrange multiplier, the inner optimization over Q in Eq. (7) mirrors Eq. (5). The connection between GVT and adversarial robustness has been previously noted [37].
Typically in DRO and related learning settings, the construction of the adversarial distribution defined by the inner maximization problem is implicitly solved. For example, when o twice differentiable. it has been shown that the inner maximization can be reduced to variance regularization expression [22, 21]; whereas other choices of divergences such as kernel Maximum Mean Discrepancy (MMD) yields kernel regularization [66] and Integral Probability Metrics (IPM) correspond to general regularizers [36]. Another popular choice the Wasserstein distance which has shown strong connections to point-wise adversarial robustness [10, 11, 64).
The aforementioned work, however, seek only to find the value of the inner maximization in Eq. (7) without considering the form the optimal adversarial distribution takes.
3 Rejection via Idcalized Distributions
We propose learning rejection functions r: X + 10 by comparing data distributions P to a learned idealized distribution Q (as per Fig. 1). An idealized distribution (w.r.t. model h)) is a distribution which when taken as data results in low risk (per Eg. (1)). Q are idealized rather than 'ideal' as they do not solely rely on a model's 's performance. but are also regularized by their distance from the data distribution P. Formally, we define our idealized distribution via a GVI minimization problem.
Definition 3.1. Given a data distribution Px, E D(X x Y) and a s--divergence, an idealized distribution Q E D(X) for fixed model h and loss is distribution given by
arginf L(Q)= + X (P || Q), QED(X)
(8)
where L(Q)= = [x[L'(X)] and L'(z) = Epyx 1e(Y, (())))

---

Given the objective of Eq. (8), an idealized distribution Q will have high mass when L'(x:) is small and low mass when L'(x) is large. The p-divergence regularization term prevents the idealized distributions from collapsing to point mass. Indeed, without regularization the distance from P, idealized distributions would simply be Dirac deltas at values of x E X which minimize T/ (x).
With an idealized distribution Q a rejection can be made via the density ratio [67] w.r.t. P.
Definition 3.2. Given data distribution P and an idcalized distribution Q < P. the r-ratio- rejector w.r.t. Q is given by r, (z) = [p(z) I r]. where p(z) = dQ/ (z).
Definition 3.2 aims to reject inputs where the idcalized rejection distribution has lower mass than the original data distribution. Given Definition 3. for idealized distribution, small values of p(æ) corresponds to regions of the input space X where having lower data probability would decrease the expected risk of the model. Note that we do not reject on regions with high density ratio p as L'(x) would be necessarily small or the likelihood of occurrence P(r) would be relatively small. We restrict the value of to (0, 1] to ensure that rejection is focused regions where L'(x) is high with high probability w.r.t. P(x) - further noting that 0 always rejects.
Although Definitions 3.1 and 3.2 suggests that we should learn distributions Q (and P) separately to make our rejection decision, in practice, we can learn the density ratio dQ/ /PP directly. Indeed, through the definition of Definition 2.2, we have that equivalent minimization problem:
argmin Y-
Epx, [p(X) E(Y, h(X)) + À (x(())
s.t. Ep [p(X)] = 1.
(9)
Such an equivalence has been utilized in DRO previously [23, Proof of Theorem 1]. Notice that the learning density ratio in Eq. (9) is analogous to the acceptor - r(x) in Eq. (2). Indeed, ignoring the normalization constraint IEp [p(X]] = 1, by restricting p(x) {0, 13. letting À c, and letting <p()) = [z = OJ. the objective function of Eq. (9) can be reduced to:
Er [e(X) L(Y, ((X)]( c. P (p(() 0).
(10)
Given the restriction of p to binary outputs {0, 1} (and Definition 3.2), we have that r-)) 1- p(x).
As such, Eq. (10) in this setting is equivalent to the minimization of r in Eq. (2) (with R as all possible functions X - {0. 13). Through the specific selection of o and restriction of r, we have shown that rejection via idealized distributions generalizes the typical learning with rejection objective Eq. (2).
By utilizing form of p-divergences, we find the form of the idealized rejection distributions Q and their corresponding density ratio p used for rejection.
Theorem 3.3. Given Definition 3.1, the optimal density ratio function p of Eq. (9) is of the form,
(x) - (())=)(+)-- [a(z) L'(x)+b
(11)
where a(x) are Lagrange multipliers ensure non-negativity pf() 2 0; and b is Lagrange multiplier to ensure the constraint Ep [p_ (X)] = is satisfied. Furthermore, the optimal idealized rejection distribution given by: (x) = P(x) (x).
Taking h as the Bayes posterior h., L' becomes a function of the ground truth posterior Pr(Y X = x).
Hence taking the output h as neural network plugin estimate of the underlying true posterior Pr(Y) X = x) (see Section 4.3) yields an approach similar to softmax response, i.e., rejection based on the output of h when it outputs softmax probabilities [29]. Theorem 3.3 presents general approach to generating rejectors from these plugin estimates, as a function of l and 4p.
Connections to GVI and DRO The formulation and solutions to the optimization of idcalized distributions has several connections to GVI and DRO. In contrast to the setting of GVI, Eqs. (4) and (5), the support of the idealized distributions being learned in Definitio 3.1 is w.r.t. inputs X instead of parameters. Furthermore, the inner maximization of the DRO optimization problem, Eq. (7), seeks solve a similar form of optimization. For idealized distributions, the maximization is switched to minimization and we consider a distribution over inputs X instead of targets y. Indeed, notice that the explicit inner optimization of DRO (in Eq. (7)) over Q can be expressed as the following via the Fan's minimax Theorem [25, Theorem 2]:
sup inf A QxED(2)
I[Qa)) + (D&(QA P| - e).
(12)
5

---

Notably, the loss _(Q) in Eq. (7) can be simply negated to make the optimization over Q in Eq. (12) equivalent to DRO Eq. (8) (noting the only requirement for Eq. (7) to Eq. (12) that L(Q) is linear functional of Q). This shows that switching the sign of the loss function changes idealized distributions of Definition 3.1 to DRO adversarial distributions. Indeed, through the connection between our idealized distributions and DRO adversarial distributions, the distributions (*(z)) will have the same form as the optimal rejection distributions implicitly learned in DRO.
Corollary 3.4. Suppose Q% denotes the optimal idealized distribution in Theorem 3.3 (switching L(Q)
to [())) for fixed X O. Further let X E arginf x>o { (-L(Q)) X (D (Q P) E)}.
Then the optimal adversarial distribution in the inner minimization for DRO (Eq. (7)) is Q
As such, the various optimal idealized distributions (w.r.. Definition 3.1) for rejection we will present in the sequel can be directly used obtain the optimal adversarial distributions for DRO.
4 Learning Idealized Distributions
In the following section, we explore optimal closed-form density ratio rejectors. We first examine the easiest example - the KL-divergence- and then consider the more general a-divergences.
4.1 KI.-Divergence
Let us first consider the KI-divergence [2] for constructing density ratio rejectors via Theorem 3.3.
Corollary 4.1. Let (2)) = 2 log 2 - 2 and X > 0. The optimal density ratio of Definition 3.1
is,
I' (x) À
where Z = Ep exp -I()))
KL PX (x)
exp
(13)
Z
One will notice that p corresponds to an exponential tilt [24] of P to yield Gibbs distribution QKL The KL density ratio rejectors are significant in few ways. First, we obtain a closed-form solution due to the properties of 'log' and complementary slackness, iee,, a(-) Secondly, due to the properties of exp, the normalization term b has closed form solution given by the typical log-normalizer term of exponential families.
Another notable property of utilizing a KI idealized distribution is that it recovers the previously mentioned optimal rejection policies for classical modelling with rejection via cost penalty.
Theorem 4.2 (Informal). Given the CPE setting Theorem 2. if = h²* is optimal, then there exists r which is equivalent the optimal rejectors in Theorem 2.1.
Theorem 4.2 states that the KL density rejectors (with correctly specified À and r) with optimal predictors recovers the optimal rejectors the typical rejection setting, i.e., Chow's rule.
Until now, we have implicitly assumed that the true data distribution P is accessible. In practice, we only have access empirical PN, defining subsequent rejectors PA.N- We show that for the KI. rejector, PN is enough given the following generalization bound.
Theorem 4.3. Assume have bounded loss [e(---) S B for B 0, Px C P with h.p., and - < +00, then with probability 6, we have that
C SPpp(P). Suppose M =
2
2M
sup (xx((z) - JKL (x SC. EET
log
N
where C = exp sinh (B/)).
Looking at Theorem 4.3, essentially we pay price in generalization M for each element & E T are testing for rejection. For generalization, is useful consider how N, M changes rate in Theorem 4.3. If we assume that the test set 'J is small in comparison to the N samples used to generate empirical distribution P, then the (1/VNN rate will dominate. A concrete example of this case is when | is finite. A less advantaged scenario is when M & N, yielding the scenario where we test approximately the same number of data points as that used to learn the rejector. This rate will still decrease with N + oo, although with a log N price.
(NooggNN///)

---

4.2 Alpha-Divergences
Although the general case of finding idealized rejection distributions for u-divergences is difficult, we examine specific generalization of the KL case, the a-divergences [3]. Definition 4.4. For a E IR, the a-divergence Da defined as the aa -iv-iieegeece, where
1
(2 if a + =1
I-a
(z)=
log + (z -1
if =- i
2 <log2 - (2-1)
We further define Va. where va(z) = غي when a # and (2) = log x when @ = 1.
Note that taking divergences including the Pearson x² divergence (a =3. For the density ratio, a-divergences with f (i.e. not KL) can be characterized the following.
1 recover the KI.-divergence. The a-divergence covers wide range of
Theorem 4.5. Let a + and 0. For Pox the optimal density ratio rejector px is,
(x) -/² 1 (a(x)-
2
L'(x) +b)
(14)
a(x) are Lagrange multipliers for positivity; and b is a Lagrange multiplier for normalization.
One major downside of using general i-divergenccss that solving the Lagrange multipliers for the idealized rejection distribution is often difficult. Indeed, the "log" and "exp" ensures non-negativity of the idealized distribution when the input data is in the interior of the simplex; and also provides a convenient normalization calculation. For a a-divergences, the non-negative Lagrange multipliers a(-) can be directly solved given certain conditions.
Corollary 4.6. Suppose >> 1 and À > O, then Eq. (14) simplifies to,
(
a 1
L' [1(x)) 1()))
(x
(15)
2
A
where take non-integer powers of negative values O.
On the other hand, for a < -1, D. (P Q) = oo whenever QQ is on the boundary whenever is not on the boundary [3, Section 3.4.1]. As such, we can partially simplify Eq. (14) for < -1.
Corollary 4.7. Suppose a - -1, A 0, and P lies in the simplex interior, then a(-) = 0 in Eq. (14).
Both Corollaries 4.6 and 4.7 can provide a unique rejector policy than the KL-divergence variant. Corollary 4.7 can provide similar effect when a(x) * 0 for all z. Nevertheless, having to determine which inputs axx) * and solving these values are difficult practice. As such, we focus on a > If there are values of Tz(x:) with high risk, the max will flatten these inputs to 0 in Corollary 4.6. However, if the original model performs well and L'(z) is relatively small for all x, then it is possible that the max is not utilized. In such case, the a-divergence rejectors can end up being similar - this follows the fact that (as & (1) > 0) locally all a-divergences will be similar to the x" / (a 3)-divergence [56, Theorem 7.20). This ultimately results in a-divergences being similar to, e.g., Chow's 's rule when (x)) Pr(Y = y = x) in classification via Theorem 2.
Among the a cases, examine the ddveieegee (a 3) which results in closed form for bounded loss functions and sufficient large regularizing parameter X.
Corollary 4.8. Suppose that .(..-) S B and X > max, T/(x:) [p][)( Then,
p 3 11 Lp (x) L'(x)
(16)
The condition on À for Corollary 4.8 is equivalent to rescaling bounded loss functions é. Indeed, by fixing À 1, we can achieve similar Theorem with suitable rescaling Nevertheless, Eq. (16) provides a convenient form to allow for generalization bounds to be established.
7

---

Theorem 4.9. Assume we have bounded loss [K(:.)) S B for B > X> 2B, PN C Pith h.p..
and J C Supp(P). Suppose M-
[J] +00, then with probability ō. we have that
2M 8
sup SET
px (z) px (z)
log
Notice that Theorem 4.9's sample complexity is equivalent to Theorem 4.3 up to constant multiplica- tion. Hence, the analysis of Theorem 4.3 regarding the scales of N, M hold for Theorem 4.9.
question pertaining DRO is what would be the generalization capabilities of the corresponding adversarial distributions Q2,N = PN Px.NN (through Corollary 3.4). On finite domain. via Theorems 4.3 and 4.9 and a simple triangle inequality, one can immediately bound the total variation TV(QA.N Qx) & O(///N), see Appendix M for further details.
4.3 Practical Rejection
We consider practical concerns for utilizing the Kl.-or (a: 1)-divergence case, Eqs. (13) and (15), for post-hoe rejection. To do so, we need to estimate the loss [1)) and rejector normalizer Z or b.
Loss: The former is tricky, we require an estimate to evaluate L'(x) Epy,, [&(Y, h(x))] over any possible z E I to allow us to make rejection decision. Implicitly, this requires us to have a high quality estimate of P Pylx- the overall objective that we are trying to learning. ie., predicting target given an input However, in the case of CPE (classification), it is not unreasonable obtain an calibrated estimate of Pylx via the classifier h: X D(y) [55]. In Section 5, we utilize temperature scaling to calibrate the neural networks we learn to provide such an estimate [32]. Hence, we set L'(x)) = Ey~h(a)) [E(Y, h(x)); For proper CPE loss functions, Ey~A(z) [E(Y, /xx))] acts as a generalized entropy function. As such, the rejectors Eqs. (13) and (15) act functions over said generalized entropy functions. It should be noted that simply considering the softmax outputs of neural networks for probability estimation have seen prior success [29, 39]. This equivalent to taking the 0-1-loss function [15, 35, 28] and taking a plugin estimate of probabilities via the neural network's 's output. The study of using plugin estimates have also been explored in the model cascade literature [40].
general learning setting, this can be difficult to obtain in fact it is just
For the latter, we utilize a sample based estimate (over the dataset used to train the rejector) of Ep [pX (X) is utilized to solve the normalizers Z and b. In the case of the KL-divergence rejector, this is all that is required due to the convenient function form of the Gibbs distribution, ie.. the normalizer Z can be simply estimated by sample mean. However, for a 1-divergences b needs to be found to determine the normalization. Practically, we find b through bisection search of
Normalization:
(
Ep
a - 2
L'(x) (1-2 X
0
11=00 0.
(17)
This practically works as the optimization b is over single dimension. Furthermore, we can have that b > ming (A)) As an upper bound over the possible values of b, we utilize heuristic where we multiple the corresponding maximum of L' L'(x)/A with constant.
Threshold 7: In addition to learning the density ratio, an additional consideration is how tune in the rejection decision, Definition 3.2. Given fixed density estimator p. change 7 amounts to changing the rate of rejection. We note that this problem is not limited to our density ratio rejectors, where approaches with rejectors E R via a (surrogate) minimization of Eq. (2) may be required multiple rounds of training with different rejection costs e find an accept rate of rejection. In our case, we have fixed which allows easy tuning of T given validation dataset, similar to other confidence based rejection approaches, e.g., tuning threshold for the margin of a classifier [7].
Experiments
In this section, we evaluate our distributional approach to rejection across number of datasets. In particular, we consider the standard classification setting with an addition setting where uniform label noise is introduced [4, 30]. For our density ratio rejectors, evaluate the KL-divergence based
8

---

Table I: Summary of rejection methods over all baselines and datasets targeting 80% coverage. Each
cell reports the "aceuracy [coverage]" values, bold for most accurate, and s.t.d. reported in Appendix.
Base
Kl,-Rej
(a=3)-Rej
PredRej
CSS
DFFER
GCE
HAR
97.38 [100]
99.93 181]
99.93 [82]
98.86 [85]
99.58 [81|
99. 94.44 [80]
99.31 [82]
Gas Drift
94.10 [100]
99.16 [80]
99.16 [80]
98.12
[80]
98.68
[80]
98.06 [80]
97.62
[80]
Clean
MNIST
98.55 [100]
99.93 [87]
99.93 [88]
99.18 [74]
99.95 [83]
99.93 [80]
99.85 [80]
CIFAR-10
90.20 [100]
97.22 [80]
97.17 [81]
91.40 [74]
95.45 [81]
93.72 [81]
94.25 [81]
OrganMNIST
89.10 [
96.55 180]
96.52 180]
93.79
[82]
94.49 [80]
93.47 [80]
93.68 [80]
OcIMNIST
91.93 [100]
97.08 [81]
97.18 [80]
93.43 [86]
95.40 [81]
94.66 [85]
94.91 [87]
HAR
96,51 [100] 98.56 [81]
98.
[81]
97.22 [80]
97.82 [81]
97.78 [69]
98.85 [80]
Noisy (25%)
Gas Drift
93.84 [100]
97.30 [80]
97.28 [80]
95.87 [82)
98.71 [80]
99.02 [77]
97.52 [75]
MNIST
97.88 [100]
99.89 [80]
99.89 [81]
98.00 [93]
99.94 [80]
99.93 [81]
99.95 [81]
CIFAR-10
85.31 [100]
92.25 [82]
92.50 [81]
85.84 [88]
89.58 [82]
90.93 [80]
92.22 [81]
OrganMNIST
89.10 [100]
95.86 [82]
96.29 [81]
93.40 [84]
96.74 [81]
94.67 [80]
94.48 [80]
OcIMNIST
91.89 [100)
97.17 [80]
97.10 [81]
93.42 [83]
95.49 [81]
94.08 [89]
94.63 [78)
rejector (Corollary 4.1) and (a=3)-based rejectors (Corollary 4.6) with 50 equidistant 5 E (0, [] values. For our tests, we fix À 1. Throughout our evaluation, we assume that a neural network (NN) model without rejection is accessible for all (applicable) approaches. For our density ratio rejectors, we utilize the log-loss. practical considerations in Section 4.3, and Algorithm 1.2
To evaluate our density ratio rejectors and baselines, we compare accuracy and acceptance coverage. Accuracy corresponds to the 0-1 loss in Eq. (1) and the acceptance coverage is the percentage of non-rejections in the test set. Ideally, a rejector smoothly trade-offs between accuracy and acceptance, i.e., higher accuracy can be achieved by decreasing the acceptance coverage by rejecting more data. We study this trade-off by examining multiple cut-off values T and rejection costs c.
Dataset and Baselines We consider 6 multiclass classification datasets. For tabular datasets, consider the gas drift dataset [68] and the human activity recognition (HAR) dataset [5]. Each of these datasets consists of 6 classes to predict. Furthermore, utilize two hidden layer MLP NN model for these datasets. We consider the MNIST image dataset [46] (10 classes), where we utilize convolutional NN. Additionally, we consider 3 larger image datasets with ResNet-18 -8 architecture [33]] CIFAR-I0 [44] (10 classes); and OrgMNIST / OrganSMNIST (11 classes) and OctMNIST (4 classes) from the MedMNIST collection [69, 70]. These prediction models are trained utilizing the standard logistic/ / log-loss without rejection and then are calibrated via temperature scaling [32]. For each of these datasets, we utilize both clean and noisy variants. For the noisy variant, we flip the class labels of the train set with a rate of 25%. We note that the test set is clean in both cases. All evaluation uses 5-fold cross validation. All implementation use PyTorch and training was done on a p3. 2xlarge AWS instance.
We consider 4 different baseline for comparison. Each is trained with 50 equidistant costs c, [0, 0.5), except on OctMNIST which uses 10 equidistant costs (selecting c, T discussed in Appendix P). One baseline used corresponds a modification of [49]'s cross-entropy surrogate approach (DEFER) originally used for the learning to defer literature (see [14, Appendix A.2]). This approach treats the rejection option as a separate class and solves a y 1 classification problem. A generalization of DEFER is considered which utilizes generalized cross-entropy [75] as its surrogate loss (GCE) [13]. We also consider a cost-sensitiee classification reduction (CSS) of the classification with rejection problem [14] utilizing the sigmoid loss function. The aforementioned 3 baselines all learn a model with rejection simultaneously, i.e., a pretrained model cannot be utilized. We also consider a two stage predictor-rejector approach (PredRej) which learns a rejector from a pretrained classifier [48].
Results Table 1 presents tabular summary of accuracy and coverage values when targeting 80% coverage values; and Fig. 2 presents a summary plot of the acceptance coverage versus model accuracy after rejection, focused around the 60% to 100% coverage region. This plot is limited to HAR, Gas Drift, and MNIST due to space limitations however the deferred datasets show curves where the density ratio rejector dominate with better accuracy and coverage in the plotted region. with corresponding extended plots for all datasets in Appendix Q Over all folds for MNIST our density ratio rejectors take approximately 1/2 hour to fit. A single baseline (fixed c) takes upwards of 2 hour for single fold. Overall, given that the underlying model is calibrated, we find that our
Our rejector's code public at: https: : //gith/bb.tttt..//g thub. www.///.//aaaeeee htaa///alxanneesee/eeensssstteie wwt.snstttttttttrjetiettii

---

HAR
Gas Drift
MNIST
Noisy (25%) Accuracy Clean
99
98
99-
98
94
100
1.0
1.00
99
98
94
60
70
80 90
100
60
70
80
90
100
60
70
80
90
100
Acceptance Coverage
KL- Rej (a. 3)
PredRej
.....:
C55 DEFER
GCE
Figure 2: Accuracy vs coverage plots across select datasets and all approaches, with 50 equidistant E (0, 1] and с [0, 0.5) values (sorted by coverage). The black horizontal line depicts base models trained without rejection. Missing approaches in the plots indicates that the model rejects more than 60% of test points or has accuracy below the base model. Shaded region indicates #1 s.t.d. region.
density ratio rejector either competitive superior to the baselines. One might notice that the aforementioned baselines do not or provide poor trade-offs for coverage values > 95% (as per Fig. 2). Indeed, achieve rejection with high coverage (without architecture tuning), approaches which 'wrap' a base classifier seem preferable, i..., PredRej and our density ratios rejectors. Even at lower coverage targets (80%). Table shows that density-ratic methods are comparable or superior in the more complex datasets of CIFAR-I0 10 and the MedMNIST collection. If large models are allowed to be used for the rejector accuracy vs acceptance coverage trade-offs (noisy MNIST). However, this is not always true as per CIFAR-10 where all approaches are similarly effected by noise; or OctMNIST and OrgMNIST where approaches only slightly change with noise. The latter appears to be a consequence of label noise not effecting the Base classifier's 's accuracy (using the larger ResNet- 18 architecture), as per Table 1.
per the MNIST case CSS, DEFER, and GCE can provide superior
Among the approaches which 'wrap' the base classifier h, we find that these approaches have higher variance ranges than the other approaches. In partieular, the randomness of the base model potentially magnifies the randomness after rejection. The variance range of the base model tends to increase as the noise increases (additional ranges of noise for HAR and Gas Drift are presented in the Appendix). The influence on rejection is unsurprising as these 'wrapping' approaches predict via a composition of the original model (and hence inherits its randomness across folds). In general, our density ratio rejector outperforms PredRej. However, it should be noted that PredRej does not require calibrated classifier. Among the density ratio rejectors, between KL and
3. the only variation is in coverage region that the 5 E (0, 1] threshold covers. This follows from the fact that for similar distributions, ep-divergences act similarly [56, Theorem 7.20]. We find this pattern holds for other values of a.
Limitations and Conclusions
We propose new framework for rejection by learning idcalized density ratios. Our proposed rejection framework links typically explored classification with rejection to generalized variational inference and distributionally robust optimization. classification, L'(Q) could in theory be replaced by any other loss functions. In this sense, one could adapt this approach to other learning tasks such as regression, discussed in Appendix N. Furthermore, although we have focused on yp-divergences, there are many alternative ways idealized distribution can be constructed, e.g., integral probability metrics [51, 9]. One limitation of our distributional way of rejecting is the reliance on approximating P|Y X] with model h. In future work, one may seek to approximate the density ratio by explicitly learning densities Q and or via gradient based methods (for the latter, Appendix O).
should be noted that although we have focused on
10

---

Acknowledgments and Disclosure of Funding
We thank the reviewers and the area chair for the various suggestions which improved the paper. A majority of the work was done whilst AS was interning at Amazon.
References
[1] Syed Mumtaz Ali and Samuel D Silvey. general class coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society: Series B (Methodological), 28(1):131-142, 1966.
[2] S.-I. Amari and H. Nagaoka. Methods of Information Geometry. Oxford University Press, 2000.
[3] Shun-ichi Amari. Information geometry and its applications, volume 194. Springer, 2016.
[4] Dana Angluin and Philip Laird. Learning from noisy examples. Machine learning, 2:343-370, 1988.
[5] Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, Jorge Luis Reyes- et public domain dataset for human activity recognition using smartphones. In Esann, volume 3, page 2013.
[6] Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
[7] Peter L Bartlett and Marten H Wegkamp. Classification with reject option using hinge loss. Journal of Machine Learning Research, 9(8), 2008.
[8] Aharon Ben-Tal Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59(2):341-357, 2013.
[9] Jeremiah Birrell, Paul Dupuis, Markos Katsoulakis, Yannis Pantazis, and Luc Rey-Bellet. (J, DDveeeDiveesssss:: interpolating between f-divergences and integral probability metrics. The Journal of Machine Learning Research, 23(():1816-1885, 2022.
[10] Jose Blanchet and Karthyek Murthy. Quantifying distributional model risk via optimal transport. Mathematics of Operations Research, 44(2):565 600, 2019.
[11] Jose Blanchet, Yang Kang, and Karthyek Murthy. Robust wasserstein profile inference and applications to machine learning. Journal of Applied Probability, 56(3):830-857, 2019.
[12] Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford University Press, 2013.
[13] Yuzhou Cao, Tianchi Cai, Lei Feng, Lihong Gu, Jinjie Gu, Bo An, Gang Niu, and Masashi Sugiyama. Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. Advances in Neural Information Processing Systems, 35:521-534, 2022.
[14] Nontawat Charoenphakdee, Zhenghang Cui, Yivan Zhang, and Masashi Sugiyama. Classifi- cation with rejection based on cost-sensitive classification. In International Conference on Machine Learning, pages 1507-1517, 2021.
[15] Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on information theory, 16(1):41-46, 1970.
[16] Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Boosting with abstention. Advances in Neural Information Processing Systems, 29, 2016.
[17] Corinna Cortes, Giulia DeSalvo, and Mchryar Mohri. Learning with rejection. In Algorithmic Learning Theory: 27th International Conference, ALT22066 Bari, Italy, October 19-21, 2016, Proceedings 27, pages 67-82. Springer, 2016.
[18] Imre Csiszár. Eine informaaionstheonrticche ungleichung und ihre anwendung auf beweis der ergodizitaet von markoffschen ketten. Magyer Tud. Akad. Mat. Kutato Int. Koezl., 8:85-108. 1964.

---

[19] Imre Csiszár. Information -tppe measures of difference of probability distributions and indirect observation. studia scientiarum Mathematicarum Hungarica. 2:229-318, 1967.
[20] Imre Csiszár, Paul C Shields, et al. Information theory and statistics: A tutorial. Foundations and Trends@ in Communications and Information Theory, 1(4):417-528, 2004.
[21] John Duchi and Hongseok Namkoong. Variance-basedd regularization with convex objectives. The Journal of Machine Learning Research, 20(1):2450- 2504,, 2019.
[22] John C Duchi, Peter W Glynn, and Hongseok Namkoong. Statistics of robust optimization: generalized empirical likelihood approach. Mathematics of Operations Research, 46(3): 946-969, 2021.
[23] Krishnamurthy Dj Dvijotham, Jamie Hayes, Borja Balle, Zico Kolter, Chongli Qin, Andras Gyorgy, Kai Xiao, Sven Gowal, and Pushmeet Kohli. A framework for robustmess certifica- tion of smoothed classifiers using f-divergences. In International Conference on Learning Representations, 2019.
[24] Bradley Efron. Exponential families in theory and practice. Cambridge University Press, 2022.
[25] Ky Fan. Minimax theorems. Proceedings of the National Academy of Sciences of the United States of America, 39(1):42, 1953.
[26] Leo Feng, Mohamed Osama Ahmed, Hossein Hajimirsadeghi, and Amir H. Abdi. Towards better selective classification. In The Eleventh International Conference on Learning Representations, 2023.
[27] Vojtech Franc and Daniel Prusa. On discriminative learning of prediction uncertainty. In International Conference on Machine Learning, pages 1963-1971, 2019.
[28] Vojtech Franc, Daniel Prusa, and Vaclav Voracek. Optimal strategies for reject option classifiers. Journal of Machine Learning Research, 24(11): 11:1:49994 1-99, 2023.
[29] Yonatan Geifman and Ran El- Yaniv. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017.
[30] Aritra Ghosh, Naresh Manwani, and PS Sastry. Making risk minimization tolerant to label noise. Neurocomputing, 160:93-107, 2015.
[31] Yves Grandvalet, Alain Rakotomamonjy, Joseph Keshet, and Stéphane Canu. Support vector machines with reject option. Advances in neural information processing systems, 21, 2008.
[32] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321-1330, 2017.
[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[34] Kilian Hendrickx, Lorenzo Perini, Dries Van der Plas, Wannes Meert, and Jesse Davis. Machine learning with a reject option: A survey. Machine Learning, 1136)):0033-110, 2024.
[35] Radu Herbei and Marten H Wegkamp. Classification with reject option. The Canadian Journal of Statistics/La Revue Canadienne de Statistique, pages 709-721, 2006.
[36] Hisham Husain. Distributional robustness with ipms and links to regularization and gans. Advances in Neural Information Processing Systems, 33:11816-11827, 2020. 2020.
[37] Hisham Husain and Jeremias Knoblauch. Adversarial interpretation of bayesian inference. In International Conference on Algorithmic Learning Theory, pages 553-572, 2022.
[38] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, In International conference on machine learning, pages 448-456, 2015.
12

---

[39] Paul F Jaeger, Carsten Tim Lüth, Lukas Klein, and Till Bungert. A call to reflect on evaluation practices for failure detection image classification. In The Eleventh International Conference on Learning Reprexentations, 2023.
[40] Wittawat Jitkrittum, Neha Gupta, Aditya K Menon, Harikrishna Narasimhan, Ankit Rawat, and Sanjiv Kumar. When does confidence-basee cascade deferral suffice? Advances in Neural Information Processing Systems, 36, 2024.
[41] Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Representations, 2014.
[42] Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. An optimization-centric view on bayes' rule: Reviewing and generalizing variational inference. The Journal of Machine Learning Research, 23(1):5789-588 2022.
[43] Donald Knuth. Two notes on notation. The American Mathematical Monthly, 9965):403-422, 1992.
[44] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Toronto, ON, Canada, 2009.
[45] Henry Lam. Robust sensitivity analysis for stochastic systems. Mathematics of Operations Research, (444:1244811277 2016.
[46]
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online], 2, 2010.
[47] Naresh Manwani, Kalpit Desai, Sanand Sasidharan, and Ranasubramanian Sundararajan. Double ramp loss based reject option classifier. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 151-163. Springer, 2015.
[48] Anqi Mao, Mehryar Mohri, and Yutao Zhong. Predictor-rejector multi-class abstention: Theo- retical analysis and algorithms. In International Conference on Algorithmic Learning Theory, pages 822-867, 2024.
[49] Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning, pages 7076-7087, 2020.
[50] Hussein Mozannar, Hunter Lang, Dennis Wei, Prasanna Sattigeri, Subhro Das, and David Sontag. Who should predict? exact algorithms for learning to defer to humans. International Conference Artificial Intelligence and Statistics, pages 10520-10545, 2023.
[51] Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):4294433, 1997.
[52] Harikrishna Narasimhan, Wittawat Jitkrittum, Aditya Menon, Ankit Rawat, and Sanjiv Kumar. Post-hoc estimators for learning to defer to an expert. Advances in Neural Information Processing Systems, 35:29292-29304, 2022.
[53] Chenri Ni, Nontawat Charoenphakdee, Junya Honda, and Masashi Sugiyama. On the calibration of multiclass classification with rejection. Advances in Neural Information Processing Systems, 32, 2019.
[54] Tadeusz Pietraszek. Optimizing abstaining classifiers using roc analysis. Proceedings of the 22nd international conference Machine learning, pages 665-672 2005.
[55] John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances large margin classifiers, 10(3):61-74, 1999.
[56] Yury Polyanskiy and Yihong Wu. Information theory: From coding to learning. Cambridge university press, 2024.
[57] Andrea Pugnana and Salvatore Ruggieri. A model-agnostic heuristics for selective classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1461-9469, 2023.
13

---

[58] Andrea Pugnana, Lorenzo Perini, Jesse Davis, and Salvatore Ruggieri. Deep neural network benchmarks for selective classification. Journal of Data-centric Machine Learning Research, 2024.
[59] H. G. Ramaswamy, Ambuj Tewari, and Shivani Agarwal, Consistent algorithms for multiclass classification with an abstain option. Electronic Journal of Statistics, 12:530-554 554, 2018.
[60] Mark D Reid and Robert C Williamson. Composite binary losses. The Journal of Machine Learning Research, 1 1:2387-2422, 2010.
[61] Mark D Reid and Robert C. Williamson. Information, divergence and risk for binary experiments. Journal of Machine Learning Research, 12:731-817, 2011.
[62] Igal Sason and Sergio Verdú. f f-divergence inequalities. IEEE Transactions on Information Theory, (2(11):5973-60
2016.
[63] Herbert E Scarf. A min-max solution of an inventory problem. Technical report, RAND CORP SANTA MONICA CALIF, 1957.
[64] Aman Sinha, Hongseok Namkoong, and John Duchi. Certifiable distributional robustness with principled adversarial training. In International Conference on Learning Representations, 2018.
[65] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15()):1929-195, 2014.
[66] Matthew Staib and Stefanie Jegelka. Distributionally robust optimization and generalization in kernel methods. In Advances Neural Information Processing Systems, pages 9131-9141, 2019.
[67] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
[68] Alexander Vergara, Shankar Vembu, Tuba Ayhan, Margaret A Ryan, Margie L Homer, and Ramón Huerta. Chemical gas sensor drift compensation using classifier ensembles. Sensors and Actuators B: Chemical, 166:320-329, 2012.
[69] Jiancheng Yang, Rui Shi, and Bingbing Ni. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 191-195. IEEE, 2021.
[70] Jiancheng Yang. Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
[71] Ming Yuan and Marten Wegkamp. Classification methods with reject option based on convex risk minimization. Journal of Machine Learning Research, 11(1), 2010.
[72] Ahmed Zaoui, Christophe Denis, and Mohamed Hebiri. Regression with reject option and application knn. Advances in Neural Information Processing Systems, 33:20073-20082. 2020.
[73] Arnold Zellner. Optimal information processing and bayes's theorem. American Statistician, pages 278-280, 1988.
[74] Jingzhao Zhang, Aditya Krishna Menon, Andreas Veit, Srinadh Bhojanapalli,, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. In International Conference on Learning Representations, 2021.
[75] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
14

---

Supplementary Material
This is the Supplementary Material to Paper "Rejection via Learning Density Ratios". To differentiate with the numberings in the main file, the numbering of Theorems is letter-based (A, B.....
Table of contents
Proof
Appendix A: Proof of Theorem 2.1
Pg 16
+ Appendix B: Proof of Theorem 3.3
Pg 16
Appendix C: Proof of Corollary 3.4
Pg 17
Appendix D: Proof of Corollary 4.1
Pg 17
Appendix E: Proof of Theorem 4.2
Pg 17
+ Appendix F: Proof of Theorem 4.3
Pg 18
Appendix G: Proof of Theorem 4.5
Pg 19
+ Appendix H: Proof of Corollary 4.6
Pg 21
Appendix I: Proof Corollary 4.7
Pg 21
Appendix J: Proof of Corollary 4.8
Pg 21
Appendix K: Proof of Theorem 4.9
Pg 22
Deferred Content
+ Appendix L: Broader Impact
Pg 22
4 Appendix M: Distribution Generalization Bounds
Pg 22
4 Appendix N: Rejection for Regression
Pg 23
Appendix O: Gradient of Density Ratio Objective
Pg 24
Appendix P: Finding the Best Rejection Cost
Pg 24
Appendix Q: Additional Experimental Details
Pg 25
15

---

A Proof of Theorem 2.1
Proof. We first rewrite the coverage probability as expectation:
Ep[(1 - r(X)) <Y, h(x))] c. Pr(r(X)= =1 Ep((1 - r(()) EY, (()) + +c.r(x))-
We note that point-wise, the proper loss & is minimized by taking the Bayes optimal classifier 7 (²x) = Pr|Y - +1 X - æ|. Thus taking the argmin over all possible CPE classifiers, ht ny" We note that the point-wise risk taken by the Bayes optimal classifier is typical denoted as the Bayes point-wise risk L(x) = Ep, , [,Y, 77 (z)) [60, 61].
As such, left to minimize over,
Ep (1 r(x)) E(Y, ," (X)) c.. rxr(
Ep, 1 [11 r(X)) e(Y, (X)) +e.r(X))
1Bp.1(1 r(X)) L(X) +(xr(x]]
= [Ep] + Ep. [r(X) (c- L(X))]
Thus, we immediately get the optimal (=(x)= [L(x) 2d.
B Proof of Theorem 3.3
Proof. We use the theory of Lagrange multipliers to make different constraints explicit optimization problems.
Let us first consider the reduction from learing explicit distributions Eq. (8) to density ratios Eq. (9).
First note that the objective Eq. (8) can be written as follows:
L(Q) + X Dpp || Q)
/ I((x)dQ(z) + /
dQ dP
dP(x)
= / (1(x) dQ
dQ
x (x)dP (a) +X
(1) (P
dP
dp
dQ
= Ep L'(x)
x
The reduction Eq. (9) now follows from reduction from minimizing over Q dQ/dP, noting that Q is restricted to be on the simplex. As such, the simplex constraints are transfered to:
dQ
dP 20 0 and f dP(c))
dQ dp (z) = Ep
dQQ
dP (X) 11
where the former is the non-negativity of simplex elements and the latter is the normalization requirement. Hence, taking p Px completes the reduction. (we remove the subscript "X" for the rest of the proof)
dQ
As such we have the optimization problem in Eq. (9), where we will convert the constraints into Lagrange multipliers, defining,
C(p; a, b) Ep [p(X) L'(X) + =(e(X)) / d'(x)((z)dr + b. (1 EE
= Ep [x(X) ['(X) + + A (p(X)) - a(X) p(X) - b. p(X)] + b,
where a(x: -(xx))
We can obtain the first order optimality conditions by taking the functional derivative. Suppose that 8 > 0 and h: X + R is any function. The functional derivative is given by.
c(p 5. h: o, b)
dø
6=0
= EEp (L'(X) - X - (p(X) + 8 A(X)) a(x) - -b]]
6=0
= (I'(X) - - (p()) a(X) - bb]]
16

---

Thus, the first order condition is,
= do
L(p + --- -
= Ep (()) (L'(X) + X o (x(x)) - a(X) - bb],
6=0
for all h: X > R. As the condition must hold for all h for an optimal pt, we have that for all x E X,
L'(x) + X xp (0 (x)) a(x)
o (x)) b a(x) (z)
p'(x) = (4'')-1 ( (b-I'())+
['(x) + =((x))
A
As required.
C Proof of Corollary 3.4
First notice that the inner maximization in the DRO optimization Eq. (7) can be simplified as follows:
Proof.
sup (P)
L(Q) = inf
-T(())
QEB
inf
sup -(Qs)) x. 4 Qa))
QE)(() A20
sup inf -(QN) - x (s- Qx)) A20 QxeD(x)
where the last inequality follows from Fan's minimax Theorem [25, Theorem 2] noting that Q
L(Qx) linear and the selected yp per Definition 2.2 makes Q + D(P Qx) a convex lower semi-continuous function.
Now notice that the inner minimization of this simplification exactly our idealized rejection distribution objective Eq. (8) when we negate the loss. As such, noticing that solutions to Eq. (8) are exactly given by P p yields the result after optimizing for the 'arguments' in the above DRO objective for both À and Qx.
D Proof of Corollary 4.1
We defer the proof of Corollary 4.1 to Appendix G, which covers any including KL (a 1).
E Proof of Theorem 4.2
We breakdown Theorem 4.2 into two sub-theorems Theorem E. for each the settings. Theorem E.1. Given the binary CPE setting presented in Theorem 2.1 and that are given the optimal classifier h (x) = P(Y = 1 X = I), for any X > O there exists a > such that the r rejector generated the optimal density ratio in Corollary 4.1 is equivalent the optimal rejector in Theorem 2.1.
The proofs of each are similar. We first make the observation that the rejector function KT. can be simplified follows:
r [T((x) - (log Z + log [igT)]_
Now we note that given a fixed À > 0 (which also fixes 2), the RHS term has a one-to-one mapping from R,, to R. Thus all that is to verify is that thresholding L'(x)) is equivalent to the rejectors of Theorems 2.1 and N.I.
Proof. For the CPE case, from assumptions, we have that L'(x:) = EY [e(Y, (a)) = Ey [e(Y, (A(Y h" (x:) ht is optimal. Thus thresholding used in KT. is equivalent to thresholding L' L'xx) by keeping À fixed and changine appropriately.
17

---

Proof of Theorem 4.3
To prove the theorem, we will be using the standard Hoeffding's 's inequality [12]. Theorem F.1 (Hoeffding's Inequality [12, Theorem 2.8]). Let XI...., Xn be independent random
variables such that X; takes values in (a,, bi] almost surely for all i S n. Defining X -E -
then for every > 0,
21
Pr X 2 S cxp
(-E=10 -a:j3)
az)²
Proof. Let us denote Z to be the normalizer with the true expectation Lpp and 2 to be the normalizer with the empirical expectation Ep.
As is bounded, we take note of the following bounds,
(-i(x)) X
cxp(-B/A) & max cxp
Z,2
cxp(B/A).
This can be simply verified by taking the smallest and largest values of the exp. Now simply have
1 1
(x) - -x(L(x)) exp
X
Z
2
B
S exp
X
2
21
exp
21
B
2). Notice that IZ _ can be bounded via concentration inequality (bounded) random variable (X)
S exp
1
exp
Thus, we have that by Theorem F.I
(=
L'(xx)
>1
-2.NN N. (Tb-
Pr Ep
exp
Ep exp
S 2exp
where (b a)² = (exp(B/A) -
4sinh (B/A).
Taking union bound over J, we have that
Pr 3x J: Ep exp L'(x))
(-1400)]
L'xz))
- N t²
Ep exp
>1)
< 2M exp
2sinh 2 (B B/A)
Thus taking sinh(B/A) log
2M
have that with probability 1-6 for 6 > for any
E T.
B 3
(xL - KLL (x)l S exp
12
2
π
B
2 V N
2M
< exp
sinh(B/A)
log
As required.
18

---

G Proof of Theorem 4.5
Before proving the theorems, we first state some basic properties of vaa Lemma G.1. For * -1, (u uv) (u) (a(c) and va(1/v) 11) Furthermore, these statements hold when th is replaced with v.a
Lemma G.2.
I
ifa + otherwise
Vo (z)
(18)
exp(x)
The above statements follows directly from definition and simple calculation. The next statement directly connects & to aaa Lemma G.3. For + 1,
(+%)))) a - Va (1/2).
(19)
Proof. We prove this via cases.
=
d de
2
('-1(2)
(-log 2) -2
2 (1/2).
Q
a +1:
(a(z) = de
1 a²
140 (1-
))
1
2 a-1
2
1-0
2
.2
1-a
2
Q
2
1
1/ v (1/2)
We now define the following constants depending on a:
cb 2
i #1
T-a 1 (-1) exp(-1)
(20)
otherwise.
Lemma G.4. For + 1,
1 (2) Ca
(P%a)
(21)
Va (x)
Proof. We prove this via cases.
=-
(2)(2)= = d
(- 1-log 2) = ----1
-1 2 2
-1 -1. Va (1/2).
de
Then,
(4L_ 11 Vo (-1-2)
wー -1
(-1)
Ca.
Va (2)
wa (2)
# +1:
%. (2) dz
d 4
1 a²
(1- 1t0
)
2
a
2
a-1
2
a
1-a
2
a
tba (1/2)
Then.
(v%.)
V/
1-0 2
2)
w/a
1-0
Va
(z)
Ca
V
19

---

Lemma G.5, For a 1,
(x)))+(22 co Va (2).
(22)
Proof. a=1:
(1(()) dz
(=log 2) llog = + 1
Then,
(46-1)- = exp(2 - 1) = exp(-1) exp(2) = - (-1) क) ')'(()) = ca 2a.v% 1/
Thus now via Lemmas G.4 and G.5, we can prove the Theorems.
Proof of Corollary 4. I and Theorem 4.5. The proof follows from utilizing either Lemmas G.4 and G.5 in conjunction with Theorem 3.3.
1: We have that,
px(x) = (000)- (a(x) L'(x) +b +b
X [((x) ['(x) +b) x
= v/a
L'(x) + b b-1 [axx)
= exp
As (x) for already positive by 'exp', by complementary slackness, the Lagrange
multipliers a(.) 0. Hence, we can further simplify the above,
L' x
+
A
(x)) = exp
The normalizer then can be easily calculated, renaming V' = X, we simplify have that
L'(x) + 8
(x()) = exp
A
L'(x)
exp(-8/ exp
A
which by normalization condition and setting 7 = exp( -1//A),
(L'(2))
exp
L'(x) ]
0 =
exp
which completes the case (Corollary 4.1). 1: We have that,
[((x) T'(x:) +
(x)) (00)-1
11 a(xx) L'(x) +
Co
A
[((x) L'(xx) + +1) -1
a
A
(a(x) L'(x) + [10))
a
2
a(x) Tl(x:) b
where (a) we exploit the that (0
(2)) 1o (1/z) via Lemma G.I.
20

---

H Proof of Corollary 4.6
Proof. First simplify the density ratio rejector.
[((x) ['(x) +b) -
Px(x)
a -
a - -102 (aaxx) [1(x)
A +b) 4
2
We suppose that it is possible for
a (az² (((x)
L'(x)
+ +b)
<0
for values of a(x), b, and Otherwise, a(x) 0 and we are done due to the above equation's non-negativity.
Let X be arbitrary. Suppose that a(x) 0. Then by complementary slackness, we have that
(az) a-1 . A + b)
aa-1 127--T
--- [1(
1)
a-1
L'(x) > 0.
By contra-positive, we have that b ('(x)/A S 0 implies that a(x) O. By prime feasibility, in this case we also have p(x) = O. We can solve either case by using the maximum as stated.
I Proof of Corollary 4.7
Proof of Corollary 4.6. The proof directly follows from property of the a-divergence when one of the measure have disjoint support. From [3] we have.
Theorem I.1 ([3, Section 3.4.1 (4))). For a-divergences, we have that
1. For -1, (P || Q) when P(x) + 0 and Q(x) = for some X.
This result immediately gives the result, as otherwise the objective function is co.
J Proof of Corollary 4.8
Proof. We first note simplifying pa= via Corollary 4.6 yields:
(x) max 0, (0.6 L'(x)
Thus our goal is to solve to give Ep j 3 [X)] =. Now, consider the following,
1 Ep [X'()] L'(x) >0 -
x> L'(x) Er [X)),
A
where the latter holds uniformly for all x from assumptions on A.
Er (x))
Thus setting +
we simplify
A
/ max 0,1+
Ep [I'(X)]
E'(X)
dP(x)
A
+ Ep ['(X)]
L'(X)
dP(x)
A
X
1.
Ex (')) yields the Theorem.
As such, b= 1
solves the required normalization. Substituting back into / /==3
21

---

K Proof of Theorem 4.9
Proof. First we note that for any meas R. À > 2B implies that L'(x) ER [L())] (taking largest and smallest values of L.
As such, for both P, P have closed forms Corollary 4.8 Thus, the bound can be simply shown to have,
P
Ep [X()]] Ep [T'
Thus by Hoeffding's inequality Theorem F.l and union bound (see for instance the proof of Theo- rem 4.3), setting & B v log (2M) ). have that with probability 1 for all
B
2
2M 8
10X 1 (x))
3
log
N
As required.
L Broader Impact
The paper presents work which reinterprets the classification with rejection problem in terms of learning distributions and density ratios. Beyond advancing Machine Learning in general. potential socictal consequences include enhancing the understanding of the rejection paradigm and, consequen- tially, the human-in-the-lon toop paradigm. The general rejection setting aims prevent models from making prediction when they are not confident, which can have societal significance when deployed in high stakes real life scenarios allowing for human intervention.
M Distribution Generalization Bounds
In the following, we seek to provide generalization bounds on Qx,N = PN ÂA.N. That is, one seeks to know that as N do how and if QA,N Q. A natural measure of distance for probability measures lotal variation [20].
1 ||p Q11 = -1((0) PPx)) dr.
TV(P, Q)
2
Let us also define
|P- - Ql = sup (Q(z) P-x)|-
=EX
One immediately gets a rate we assume that [X] finite and we have bound for Px.N. Theorem M.1. Suppose that [X] M is finite and bounded density ratio 1pA.N| S B' for B' > O. Then, if we have that with probability å that,
[l{(x) (x) SC (V N
2M log
+ C'
5
for some C, C' > O, then we have that
TV(Q, Qx.)) =0 (1
2 log M
(23)
N
Proof. Noting that the empirical distribution is sum of Dirac deltas (x = EieiNy 8(x 2z). establish simple bound via consequence of Hoeffding's Theorem Theorem F.I (also noting that P x) We have that,
Pr([P(z) (x) 2 f) < 2exp (-2Nt²).
22

---

Thus,
Pr(Var E [N P. 2 1) S 2M exp (-2N1²).
Setting & = V
1 2/V log
2M 8
we have with probability 1 -
2M
||P PN S V 2N
log
We now consider the following:
||Q QA Illo
|P p PP iA
= |P - pX.N) + (P - (Ppxx) PA,,Nloo
3 [PP (p ~x.vx + (1P-Pxx) P2,Noloo
< (1 )x,,)]l + B' [[(P PN)
Taking a union bound of the above inequality and our assumption, we have that, for some C > 0, with probability
||Q Qx S 11pp-
PA,N) + BB. -1(PP- [(P-Px)lle
4M
4M
SC.
log 5
+ + +B'
log
N
2N
8
log M
Converting the bound to TV amounts to simply summing over X, which gives TV(Q, Qx.N) = ((MMM M)/N), as required.
Notably, with appropriate assumptions, Theorems 4.3 and 4.9 can be used with Theorem M.1 get bounds for KKL and Q2,N
N Rejection for Regression
In the main-text of the paper we have focused on classification. However, many of the idea discussed can be extended for the regression setting. For instance, similar to Chow's Rule in Theorem 2.1 we can express the regression equivalent to the optimal solution of Eq. (2).
Theorem N.1 (Optimal Regression Rejection [72]). Let us consider the regression setting, where
y y and e(y, { - 2/)². Then w.r.. Eq. (2), the optimal model is given by (x)x)) Ep = and the optimal rejector is given by r rxxx) = [o²(x) clI, where o² (xx) is the conditional variance of given X x.
For regression, there is clear analogous notion of output "confidence score" unless the model explicitly outputs probabilities. Indeed, rejection method for regression explicitly requires the estimation of the target variable's conditional variance [72].
Similar to the CPE case, our KI. density ratio rejector can provide rejection policy equivalent to the typical case.
Theorem N.2. Given the regression setting presented in Theorem N.1 and that we are given optimal x), for any > there exists a T > such that the KL rejector generated the optimal density ratio in Corollary 4.1 is equivalent to the optimal rejector in Theorem N.I.
regressor (x) Er|Y
Proof. The proof follows similarly to the proof of Theorem E.I. The regression case follows almost identically, by noticing that T.' Tz(xx) = (2() is the variance. This is similar the proof of Theorem N.1 [72].
23

---

Despite the equivalence, there is a difficult in using the density ratio rejectors, per the closed form equations of Section 4, for regression. Estimating L' (2) challenging. Unlike classification where learning calibrated classifiers has variety of approaches, learning a regression model which
explicitly outputs probabilities is quite difficult. As such, approximating P(Y X = x) with the model (ar) cannot be done.
Gradient of Density Ratio Objective
As an alternative to the closed-form rejectors explored in the main-text, one may want to explore method to learn p iteratively. We consider the gradients of the optimization problem in Eq. (9). In practice, we found that we were unable to learn such rejectors via taking gradient updates and thus leave concrete implementation of the idea for future work.
The idea comes from utilizzig the "variational" aspect of the GVI formulation (which was not explored in the main-text) We suppose that the rejectors we are interested in come from parameterized family, In particular, we consider the self self-norm family Pa/ where ZB Ep (X)] normalizes the rejector such that lP [ax [X)] 1. Having the Za normalizing term means that the constraint in Eq. (9) is satisfied for any The only constraint that we must have for me non-negativity, ie., me is a neural network with exponential last activation functions from X - R4- By setting parametric form of po. we implicitly restrict the set of idcalized distributions to Qa P Pa. The gradients of such parametric form can be calculated follows.
Corollary O.1. Let Pa = TT(x)/ ZZa Then the gradient of Eq. (9) w.r.t 0 is given by,
(x())
(0'
(x
+ +X-EEY,,(00)
EP
V
(24)
Zo
Zo
Proof.
The proof is immediate from differentiation of Eq. (9).
alternative form of Eq. (24) can be found by noticing that Vʃ f. Vlog f. This provides an expression in terms of the log density ratio.
Thd (X)
Tt Tx(X))
(0/
To (X)
+ + +X-
Ep 2.5
V log
Za
Za
Zo
One will notice that the gradient is in-fact the log-likelihood of the idealized distribution: noting free of 0, have V (o)(z)(X)/29) gradient Eq. (24) is equivalent to the gradient of a weighted log- log-likelihood
V log(P(X): Tyy (Z) V log Qa. As such, the
Despite the potential nice form of the gradient, found that learning rejector through this was not possible. One limiting factor of computing such gradients is that we need to estimate Zo at each gradient calculation, i.e., this must happen whenever 9 changes. This can be particularly costly when is high-dimensional only tested on simple neural networks and convolutions neural networks, mirroring the architecture of the base classifiers used in our experimental setting.
Secondly, we suspect that the model capacity of Ta was not sufficient: we
P Finding the Best Rejection Cost
Both the baselines and density ratio approaches evaluated in Section 5 require tuning of hyperpa- rameters C, T. Practically, we are more interested in the rejection rate P(r(X) = rather than the abstract choices of c, T. Hence, one often wishes to select c, J according a coverage constraint.
One of the simplest choices of selecting c, is to utilize a calibration set and approximate the rejection rate coverage on this data. Our experiments in Section mirrors this in Tables and by treating the final test set as the calibration data.
Other works in the literature utilize more sophisticated methods for deriving coverage constraints. [29] picks threshold values which provide guarantee on a coverage-modifer risk requirement. Another approach [57] does not require an additional calibration set and instead fits threshold values T by exploiting stacking confidences and (stratified) cross-fitting. Although one could consider such approaches for all basclines, including those which utilize the cost-model of rejection via c, one will
24

---

have to pay price in retraining the learned rejector for each instance of the hyperparameter that is searched. As result, approaches which can trade-off accuracy and coverage via a threshold variable (rather than an expensive retraining) are computationally preferable when exhaustively searching for good e, T.
We leave more sophisticated methods for selecting т for density ratio rejectors for future work.
Additional Experimental Details
Q.I
Training Settings
The neural network architecture used to train the base classifiers and baselines are almost identical. For the baseline approaches which have output dimension which is different than the output of the original neural network, we modify the last linear layer of the base classifier's 's architecture to fit the baseline's requirements, e.g., adding an additional output dimension for rejection in DEFER. Our architectures utilize batch normalization [38] and dropout [65] in variety of places. Training settings are mostly identical, with some baselines requiring slight changes.
The base model's s architecture is as follows.
HAR (CC BY 4.0): We utilize two hidden layer neural network with batch normalization. Both hidden layer is 64 neurons and the activation function the sigmoid function. We take 64 batch size, 40 training epochs and 0.0001 learning rate.
Gas Drift (CC BY 4.0): We utilize two hidden layer neural network with batch normaliza- tion. Both hidden layers are 64 neurons and the activation function the sigmoid function. We take 64 batch size, 40 training epochs and 0.0001 learning rate.
MNIST (CC B-SA 3.0): We utilize a convolutional neural network with two convolutional layers and two linear layers. The architecture follows directly from the MNIST example for PyTorrch We utilize the sigmoid function activation function. We take a 256 batch size, 40 training epochs and 0.0001 learning rate.
CIFAR-10 (CC BY 4.0): We utilize ResNet- 18 classifier. Random cropping and horizontal flipping data augmentation is utilized in training. We take a 256 batch size, 40 training epochs and a 0.0001 learning rate.
OrgMNIST (CC BY 4.0): We utilize ResNet-18 classifier. We take a 256 batch size, 40 training epochs and a 0,0001 learning rate.
OctMNIST (CC BY 4.0): We utilize ResNet-18 classifier. We take 256 batch size, 40 training epochs and a 0.0001 learning rate.
For CSS, noted in [53], batch normalization is needed at the final layer to stabilize training.
All training utilizes the Adam [41] optimizer.
All datasets we consider are the public domain, e.g., UCI [6].
Q.II Extended Table and Plots
Table I presents extended version of Table over coverage targets of 80% and 90%. The standard deviation is additionally reported. One can see the observation from the main text are consistent with this extended table.
Plots Figs. I III show Fig. 2 over and extend region of acceptable coverage percentages. addition, we include a larger range of noise rates for HAR and Gas Drift. For MNIST, we explore a larger range of noise rates for MNIST in Fig. VII for our density ratio rejectors. We find that the findings in the main text are extended to these additional noise rates. In particular, we find that the our density ratio rejectors can be competitive with the various baseline approaches. We find that our density ratio approaches can have more fine-grained trade-offs at higher ranges of acceptance coverage. This is an important region where the budge for rejection may be low (and only a few examples, e.g. < 10,, can be rejected). Indeed, the baseline approaches which do not 'wrap' a base model require lower maximum acceptance coverage as the noise rate increases (the approaches require a higher rejection % for any type of rejection). Nevertheless, we do see downside of the density ratio approach: the
25

---

Table I: Extended Summary performance of rejection methods over all baselines and datasets targeting 80% and 90% coverage. Each cell reports the "accuracy
31 (0.52) (8) 1878 (4 (.699)
07.62 (0. (6) 180.45 (3 3.922)
88 (0.04) (80.02 (3.75))
94.2 25 (0.31) 88.99 (0.83))
93 08 (1) 88,04 (7.48)
94.91 799 (86 686.67 (3.66]]
9% 85 <0 4) 180 18 (3, 11
99.05 (3 (8) 11220 (1:55]
111800055 (00.55(100 (0:41))
(80 880-9 (3.1))
94.63 (1. (4)14)) 77.5777766 (13.79)]
99.3 37 (0 190.27 (0.251)
96. 26 c0 5.) (87 (2,24) (2,24)]
99 84 (0.03) 190.72 (3411)
(0.451)
90.88 r0 75) (00 12/01)
99..8 (2. (211
189.34 (1.231]
97.52 (0. 75 [ (2.76)
( (0851)
(81.9 (1233)
92.1 17 (0 24) ( (0-55)
(039)
97.52 (0.75) 72.31
94.23 24 (0. 08 190
91.54 (0.44)
93.39 (0.89)
12.14 (0
GCE
92 02.22
94.48
56
99
99. 29.44 (0.19) (80) 16 (66.1.. (1.54)
98 06.06/ (0.97) (30.49 (1. (.09)
22.53 (0:01) 300.33.00 (1600)
93.72 (0.94) 180.57 (4.78 78)
94.66 (85.32 (2..32(24))) 4222)
97.74 77.78 (1 (.000 (69.] (6. 36)
99.03 (0:03) (81 45 (0.38)
90.93 (0:45) 1801800222 29 (1 .9
(1 190.39 (5-5))
94.08 (0.64) (89.10 2.42)
99 23 (0. 19) 190344 (0.29))
97.44 46.46 (0 44) (84.5 (200))
99 S0 (0.83) (90,22 (0.32)
(1.65)
92.3 (0 58) (90.6 (.68)
19.18 2.08)
77.78 (1 169.13 (6.36)
99 02 (0 48) 76.54 (2.55(2.11) (2,11)
185 (3.47(0-55)
(1.
(1.17)
91.77 (0 (4) (901 (2,00)
(3.099 (305)
9347 (0:55) (80.16(4.5)
(76.54
89 04 92)
DEPER
93.49 (0.
89
99. 50 (1.31 [8) 39 () 48)
98. 68.68 (0. (9) [80 3. (1. 1.188)
95. (0.95) [80 56 (1.14i
18 05 (2.58)
67)
08
484
77-
(4.24)
09 (0.70)
99 95 (0:02) 190-2 (0. 361
(0.72)
42 (1.78)
1.622
(1.071
(1911)
89.71 (1.01) (83 40 (1-44)
91.92 (1 05)05) (90 22 (1.17)
(0.52) (0.93))
12
94.49) (0:05) 17
100 50
38
19
20
89
50
54
98 71 (0. 27) 8M 32
99:95 (0:03) 183
99 04 (0:03) 880
18
88
(00
91. (1 (9)
18
95. (l 06)
97. 82 (0. 35)
11 29
96.74 (0)68)
95 (1 353
96. 96.99666 (D 56)
92.52 66)
98 (0) 29)
89. 58
92. 92.77
CSS
98.
98.
99
(8:5))
98. (1003) (R 89.099 (6.03))
99. 19 (0.11) 03403 (4.05)
(16.73)
93.43 (0.81 [85 55 08 22)
V7. 72 (0.55) 800 24 (2 93.
98.00 (0 20) 12.995 (10.17) 17]
65 35.84 (1.17) 188. 25 (23.7))
93 33.40 (0.74) 183 78 (5.13))
93.42 01.00 183 14 (1231))
(14.01)
96 64 (.0 (90. 30 (5-65)
98.88 88.87 (0.13) 13 82 (1 11552
(10,55)
44 (1.68) (90-9 (2 (7:6]
92.3 22.23 (0.50) 63 (4.11)
(482)
95.1 (1 1.855 (91. 71 (6.72)
98 (0 200 192 95 (10.17)
(17.0))
92.24 (0.76) 99061 (2 (2.35)
1907 (8.09)
98 (0-44) (84 86
(74.22
77
85 (0.99) (9)
93,77 09)
98. 88.19 74)
96.94 (0.77)
92.41 (0.36)
ProdRc)
91 40 (0
95,87
(8) 82 (0 241
99 16 (0.2)) 180 24 (0.75)
[88 08 (0.58)
97.17 (0.17) 80.53 (0.21))
180.3 (118)
20)
98. 56 (1 66) (8) 13 (15 41)
89.89 (001) (80 64 () 3.)
180 70 11 0.000)
(8099 (1
97.10 (0.21) 880 72.72 (0 781
19 421
9R 12 (1.58) (50.06 (1.37) 37)
99 99.89 (0.4) (90 84 (0.53))
(0.20)
(094) (30 (1 201
95. 17 (0.55) (90 (0.38)
98 29 11:45) 190, 400.40 (7.66))
96 96.68 (2.11) (90.04 (8.34)
(90.43 (0.92)
92.55 (0:44) (00 29 60)
(90 (0.45)
(16.
(50.22
00.03
99.95 (0.00)
99.93 (0.03)
96.52 (0.2)
97.1 17.18 (0.19)
92 50 (.338
99 (0.17)
92.57 (2.57
89 49 (0.39)
95 (0.23)
(o=3)-Rc)
97 27
96.29
94.33
99
99.93 (0.06) (80 .0.74 (0 84)
99. 6 (0. (0.62) [80 880.04 78)]
59.93 (0.03)) 88 07.37700 (7.60))) 66]
97.22 (0. 18) (0.277000 (0.23)
(1 (1.22)
HD (4.994(0)28
98 56.56/1 (1 66) 88 16 (15400)
(80.02 (16.4)))
89 ro 01) 80 84 (1.4)) 1.54]
181 61 981
96 8 82.07 1.55]
97.17 (1.21) H0 29 878
190 (0.38) 38)
[8]
89.89 ( 990 99.57 (0.55]
(90.19
60 (0 87) (90 (90.55 1433
94.98 26 (91 91.55 (0.43)
08 28.28/1 <1. (4) 190.1 11 (8. 101
96 58 22 05) (90.75 (7.6) 7.96]
(0.03) 19001 (0.99)
(0.9))
92.56 (0 62) (90 (1 1.200)
95.22 (0.21) (0.52)
(accuracy s.t. s.t.d.) [coverage (coverage s.t.d.]" values.
08.05 <1 (1) (90
92.25 33)
KL
97.08
97.30
99.3 16.36.00
89-43
96.55
(1)00)
94. 10 11.0111150 (77 1100.000 01 (0.00 (0)
(0)
90.20 /0.22 (0,29) 100.0000 (0.010)
(0.8) 00)
(4) 1100 00 (0 (0)
88 (0.23 .2) [100.00 00 (0 00]
85,33 (0, (8) 1100 ON (0)
(0)
(0.300)
97.35 78 (0.3 1100 00 (O 0001
10 11.8 .7) 1100 00 (0 (0)
9% 55 (0. (9) 0100,000000000000000000000000000 (1.00)
(0.00)
89. 10.10 (1 (06) [100 00 (0 000
91.93 (0. 1100.00 0O (0.00) (2.00)
96.55 (0-44) 1100.00 0O (O. (0.00)
84 (1.8 (8) [100.00 00 (0.00)
85.31 (0. (8) 1100.00 (0)
89. 10 (0. (7,77) [1000 (0 (1000)
91.89 (0.20) [100.00 (0)
97.38 77.38/00 (0.3 .-) [100.00
89. (1.00)))) [100 00
[100.00
91 .9 (0.20 20) 100.000
1100 00
98.55 (0. 19) (1000000
$1.93 (2) [100.00
77 (100.00
Bosc
96.5:
97.08
HAR Gas Drift
MNIST
CIFAR 10
DigIoMNIST
DaMNNSST
HAR
Gas
MNIST
CIFAR 10
DrganMNIST
DaMNIST
BAR
Gas Drift
MNIST
CIFAR
OrganMNIST
OSIMNIST
HAR
Gas Drift
MNIST
CIFAR In
INSTM
OstMNIST
Cleai
Clean
Nosyy (258)
Taigee Cooerrage:. W00
20

---

quality of the density ratio rejectors is dependent on the initial model. As such, at higher levels of noise there can be higher variation the quality of rejection, see Fig. II. Interestingly, for MNIST and CIFAR-10, 10, Figs. III and IV, the base model the density ratio rejectors are more robust across noise rates than other models. This seems to due to the default MNIST architecture being robust against higher noise rates (notice that the s.t.d. range is also quite small at 100% coverage). In OctMNIST and OrganMNIST, Figs. V and VI, we find that there is little to no change in performance, likely due to the Base classifier's own performance only changing slightly with the addition of noise.
Clean
99
98
Noisy (10%)
Noisy (20%)
98
Accuracy
100
Noisy (23%)
100
Noisy (30%)
99
98
97
Noisy (40%)
20
40
80
80
100
Acceptance Coverage
Rcj
[o 3)-Rej
PredRej
CSS
DEFER
6CE
Figure Extended plots for HAR of Fig. 2.
Q.III Smaller models study
In the following, we consider the Gas Drift dataset when models are switched to a base model with only single hidden layer. First make note of the original setting explored in the main text. In Table II, we take note of the number of tunable parameters in all approaches and baselines. Notably, these default parameter arcciteccture sizes similar to [14], with the HAR and Gas Drift setting including an additional hidden layer than previously utilized in the literature.
In Table III, note the setting consider in this subsection. The parameter sizes of the Gas Drift dataset is reduced to the originally explored model sizes in [14]. Notice that both the base models and the bascline approaches have reduced parameter sizes. It should be noted that this smaller parameter
27

---

Gas Drift
Clean
100
Noisy (10%)
98
96
100
Noisy (20%)
%
100
Noisy (25%)
96
100
Noisy (30%)
98
96
94
100
Noisy (40%)
94-
20
40
60
80
100
Acceptance Coverage %
3)-Rcl
PredRe]
CSS
DEFER
GCE
Figure II: Extended plots for Gas Drift of Fig.
Table II: Default parameter sizes of experiments.
Dataset
BaseCIf
DEFER R-Rej
GCE
CSS
PredRej
HAR
40,647
40,648
40,646
40,711
40,711
80,968
Gas Drift
12,935
12,936
12,934
12,999
12,999
25,544
MNIST
1,199,883
1,199,884
1,200,138
1,200,011
1,200,267
2,398,860
CIFAR-10
22,
21,282,124
21,283,146
21,282,635
21,283,659
42,560,652
OcIMNIST
11,169,733
11,169,734
11,170,756
11,170,245
11,171,269
22,338,950
OrganMNIST
11,173,324
11,173.325
11,174.347
11,173,836
11,174,860
22,342,541
size setting be useful in the related learning with deferral setting [52], where having small model to defer to a larger model is needed.
The results are reported in Figs. VIII and IX. We can see that in this setting, PredRej and our density ratio approaches are more competitive. This might indicate that for simple base models, approaches which 'wrap' base model for rejection can be quite effective (especially in higher coverage regimes). In general, it seems with this smaller architecture regime, the monn-waapping baseline approaches only provide rejection options when the acceptance coverage is lower than 70%.
28

---

MNIST
100
Noisy (25%) Accuracy Clean
100
Acceptance Coverage %
Rej
3)-Rej
PredRej
CS5
DEFER
GCE
Figure III: Extended plots for MNIST of Fig. 2.
CIFAR-10
Noisy (25%) Accuracy Clean
95
90
95
85
80
Acceptance Coverage
KL Rej
(o 3)
PredRej
CSS
DEFER
GCE
Figure IV: Deferred extended plots for CIFAR-10 of Fig. 2.
Q.IV Parameter sweeps over and À
The following shows parameter sweeps over a and for our density ratio rejectors. These are given by Figs. X and XI respectively. We find that increasing compresses the trade-off curve from both sides. While decrease extends the trade-off curve on the left side.
29

---

OctMNIST
100
98
Noisy (25%) Acciracy % Ciean
94
92
100
90
92
20
40
60
80
100
Acceptance Coverage %
Rej
(0
3) Ro
PredRej
Figure V: Deferred extended plots for OctMNIST of Fig.
OrganSMNIST
Noisy (23%) Accuraiy Clean
95
90
90
Acceptance Coverage %
KL Rej
3) 3)
PredRej
CSS
DEFER
GCE
Figure VI: Deferred extended plots for OrganMNIST Fig. 2.
Table III: Alternative parameter sizes of experiments.
Dataset
BaseCIf
a Rcj
DEFER
GCE
CSS
PredRoj
HAR
36,487 8.775
36,488
36,486 8,774
36,551 8,839
36,551 8,839
72,648
Gas Drift
8,776
17,224
30

---

MNIST
100.0
Clean
99.0
100.00
Noisy (10%)
99.75
99.50
99.25
99.00
100.0
Noisy (20%)
99.5
Accuracy
99.0
100.0
Noisy (25%)
99.5
99.0
100.0
Noisy (30%)
98.5
100.0
Noisy (40%)
99.0
98.0
60
Acceptance Coverage
[0 3)
Figure VII: MNIST with different noises for density ratio approaches.
31

---

HAR
100
99
Clean
96
97
100
Noisy (10%)
Noisy (20%)
96
Noisy (25%)
100
96
Noisy (30%)
100-
98
100
Noisy (40%)
60
Acceptance Coverage
Roj
[a R)-Rej
ProdRaj
Figure VIII: Plots for HAR with smaller models.
32

---

Gas Drift (Small)
100
Olean
100
Noisy (10%)
100
Noisy (20%)
Accurscy
Noisy (25%)
96
94
Noisy (30%)
96
Noisy (40%)
100
Acceptance Coverage
Re-Rej
Figure IX: Plots for Gas Drift with smaller models.
HAR
Gas Drift
MNIST
Noisy (23%) Accuracy Clean
99
98
99
99
98
98
96
60
80
100
100
70
90
Acceptance Coverage
0 0.0
d-11
0 1.5
0
2.0
0 0.0
0
4.0
Figure a parameter sweep over dataset + noise combinations. S.t.d. shade not utilized, but instead end points of the trade-off curves for E (0, are shown via vertical bars
33

---

HAR
Gas Drift
MNIST
100
100
100
Clean
99-
98
99
100
1000
100
Noisy (23%)
99
99
99
97
98
96
98
80
Acceptance Coverage %
0.5
1.0
A 1.5
A 2.0
2
3
Figure XI: À parameter sweep over dataset + noise combinations for KL rejector. S.t.d. shade is not utilized, but instead end points of the trade-off curves for (0, are shown via vertical bars
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims are justified via formal results and experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are appropriately discussed throughout the paper. Furthermore, the last section of the main-text also explicitly discusses limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
35

---

Justification:
Assumptions are stated and complete proofs are provided in Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Detailed settings of experiments are provided in Appendix. Code is also included.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
36

---

Answer: [Yes]
Justification:
Code and scripts to create small scale experiments are provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
onssiicyn)))) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.cooc wwtppppwlcggcpiieeceeeeeoooceppe gttee/ugiee///eee CCodeSubmiieeeeece
/ttea////ggiideeeeeeuuuubiiiiiii ieriommsssiiooooooiiiooo (eionnllccy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All included the main-text and also the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars are given in plots.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
37

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We state the AWS resources utilized and estimated total compute time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https http///eerrpp.eeeee cc htcp/p/lcc//ttcccccueeeeeeee
didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: Research conforms with the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide broader impact section in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
38

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No release.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Baselines and data are correctly credited (with licenses given in the Appendix).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
39

---

Answer:
[NA]
Justification: No new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
40


Reviews:
[{"review_id": "zQ0eKrdBwH", "reviewer": "Reviewer_m8bi", "summary": "This paper propose a new classification algorithm with abstention by learning a density ratio between an \"idealized\" distribution to the data distribution: if the density ratio is small, the classifier rejects to predict.\nThe proposed learning framework is general, and is a function of the choice of $f$-divergence. \nThe paper studies two concrete choices of divergences: KL- and $\\a$-divergences, which admit a closed form rejector.\nThe paper provides some experimental supports for the proposed ideas with neural-net classifiers.", "questions": "**Questions**\n- What is the meaning of \"The maximization is adversarial over the marginal label distribution\" in line 128?\n- I had hard time to understanding the meaning of the \"idealized rejection distribution\" (as called. in line 157), especially because the following sentence was extremely confusing: \n```\nGiven Definition 3.1 for idealized distribution, small values of $\\rho(x)$ corresponds to regions of the input space $\\mathcal{X}$ where having lower data probability would decrease the expected risk of the model. \n```\nI can't follow this sentence. Can you please elaborate this?\n- Further, I can't understand why does it make sense not to reject where $P(x)$ is relatively small? If the likelihood of occurrence is small, shouldn't we also avoid prediction at that point as the classification will be likely wrong? Or is the argument that we do not need to care the small probability mass in the first place as it won't incur a significant loss in expectation? \n- In the experiments, the proposed algorithms seem to perform well in the high-coverage regime, while dominated by existing methods in the low-coverage regime. Do you have any explanation on this?\n\n**Suggestions**\n- Compared to the quality of math, there are way too many typos and grammatical errors. To spot a very few:\n  - line 23: \"While\" some approaches avoid...\n  - line 25: ... approach aim**s**...\n  - line 74: ... an addition**al**...\nThere are too many of these throughout, so please revise them carefully.\n- Please consider moving lines 84-85 after line 76.\n- Please put subsections in Section 4 for readability.", "limitations": "Limitations of the framework are noted by the authors.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper studies an important problem of classification with abstention.\nThe proposed mathematical framework is quite neat and has nice connections with GVI and DRO.\nThe math is quite crisp throughout the paper.", "weaknesses": "While the proposed method sounds reasonable, I have some concerns/questions in the presentation.\nPlease see questions and suggestions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their praise of the mathematical presentation of the paper and their careful reading. We are happy to change the paper as per the reviewer's suggestions and will ensure that the typos and grammar improves. In what follows, we will answer **Reviewer m8bi** additional questions.\n\n> What is the meaning of \"The maximization is adversarial over the marginal label distribution\" in line 128?\n\nThis line was to clarify that in DRO for label noise (specifically reference \\[60\\]), typically the distribution which is “adversarial” only applies to the marginal label distribution and not the full joint distribution over both inputs and labels. Thus the set of adversarial joint distributions $Q(y) P(x | y)$ are generated by varying the marginal over labels $Q(y)$ whilst keeping the conditional equal to the data $P(x | y)$.\n\n> I had hard time to understanding the meaning of the \"idealized rejection distribution\" (as called. in line 157\\)\n\nThe best intuitive way of thinking of the idealized distribution definition is to remove the regularization term in Definition 3.1. This corresponds to the distribution which minimizes the current model’s loss. Hence its “ideal” for the current model / set of parameters.\n\n> especially because the following sentence \\[…\\]\n\nFor the follow up sentence, we further elaborate.  \nWhen $\\\\rho(x)$ is small, this implies that $Q(x) \\\\ll P(x) \\\\leq 1$. (Recalling that $Q$ is the idealized distribution and $P$ is the data distribution.)  \nFurther note that from Definition 3.1 / Equation (8), $Q(x)$ being small implies that the loss $L’(x)$ cannot be too large (as otherwise we should reallocate mass from other x’s).  \nHence wrt the data distribution $P(x)$, moving mass away for regions of $x$ where $\\\\rho(x)$ is small reduces the loss function.\n\n> Further, I can't understand why does it make sense not to reject where is relatively small? \\[...\\] Or is the argument that we do not need to care the small probability mass in the first place as it won't incur a significant loss in expectation?\n\nThe reviewers comment about the values corresponding to loss probability mass and thus no significant loss penalty in expectation is correct. Our rejection approach focuses on losses at different inputs values, as per Definition 3.1.\n\nOf course, this is from the perspective of our framework, so practically, various rejection rules may be needed to account for different types of uncertainty.  \nA future direction may be to consider alternatives to expected loss which account for low $P(x)$. For example CVaR etc.\n\nAn alternative theoretical reason for this style of thresholding choice is Theorem 4.2 in the paper. With our specific choice of thresholding / rejection on the ratio $\\\\rho(x) \\= Q(x) / P(x)$, we recover Chow’s rule. The alternative of rejecting based on $P(x)$ would not yield this equivalence.\n\n> In the experiments, the proposed algorithms seem to perform well in the high-coverage regime, while dominated by existing methods in the low-coverage regime. Do you have any explanation on this?\n\nGood question. Firstly, we note that the trade-off curves can be extended further by increasing $\\\\tau > 1$. We suspect the difference in performance to be related to the quality of calibration. If the model was perfectly calibrated, then the density approach would be optimally rejecting. As such, deviation / suboptimality would be due to calibration error.  \nThis explanation is partially supported by the drop in performance of our approach on noisy variants of the dataset. Here the training set (and data used for calibration) is not representative of the test set which would result in a model miss-calibrated on the test set.\n\nFor two additional notes:\n\nWe also point the reviewer to the results of the MedMNIST dataset \\-- additionally added in the rebuttal \\-- in the shared response section. Here we find that for the lower 80% regime, our approach can perform better than the baselines for a more complicated dataset.\n\nA reason why the baselines do not provide / perform well in the extremely high-coverage regime (\\~ 95%) is that it is difficult to tune the rejection costs $c$ hyperparameter to allow for fine tuning of the coverage. As the rejection decision of these baseline involve training a whole other model, the cost of tuning requires training several NNs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZH475Yq3mi", "reviewer": "Reviewer_MiAk", "summary": "This paper proposes a novel method for classification with rejection based on density ratio estimation. Density ratio is estimated between the data distribution P and the \"idealized\" distribution Q, where Q is a distribution such that the model can have good prediction performance, while this distribution is still similar to the original data distribution P. The objective function is proposed to formulated as a risk minimization with a $\\phi$-divergence regularization. Theoretical analysis is also provided to justify the soundness of the proposed method. The experiment shows that the proposed method is effective compared with well-known existing approaches.", "questions": "1. Is there any superiority of $\\alpha=3$ divergence over KL-divergence? I might have missed it.\n2. Is it written in the main body of the paper what $\\varphi'$ is (Eq. 11)? I believe it is a derivative of $\\varphi$ over X? I am sorry if I missed it. If it is not written it should be so because for example, \"'\" can mean different things like how L' is used in this paper.\n3. It seems the proposed method is quite weak under noisy data. Is there any possible explanation why this is the case?\n4. The Theorem 4.2 (informal)  states that there exists a threshold to achieve Chow's rule if original $h$ is correct looks quite restrictive to me in the sense that if h is really optimal, then we can just use h to achieve Chow's rule and there is no need to consider density ratio estimation. Can we say anything when h is not optimal? \n\n\nMinor comment: \nI think it is more common to call \"classification-rejection\" approach, a classifier-rejector approach (like Ni et al., 2019 [44]), or predictor-rejector approach (Mao et al., 2024) [39].", "limitations": "Limitations about the need to estimate P(Y|X) with model h were discussed, suggesting that this approach requires estimating class-posterior probability. \n\n(Minor) broader impacts were discussed in Appendix L and perhaps it is better to put them in the main body if possible.\n\nOverall, I found the discussion about limitations is appropriate.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Proposed method is theoretically guaranteed and can support multiclass classification case. To me, it is not that straightforward to use density ratio estimation approach for multiclass classification with rejection and I appreciate this strength of the paper.\n2. The proposed method and formulation discussed in the paper is quite general, and the discussion for Regression case also provided in appendix, suggesting the generality of the proposed method. Divergence that studies in this paper is not only KL-divergence, but different divergences were also studied.", "weaknesses": "1. Unfortunately, I found the experiment section writing and result is not quite well-written compared with other sections. Perhaps there is a better way to analyse and understand practical behavior of the proposed method. \n - The figure shows in the paper does not really show the superiority of the method except in HAR (clean) and Gas Drift (clean). It is also quite hard to see since the starting point of each method is different. Perhaps table representation or different way to show the result might help.\n  - Only three simple datasets were used in this paper and the discussion using these datasets still look difficult to conclude something as several methods are competitive.\n  - In my understanding, KL and $\\alpha$ divergence give almost exactly the same performance in all cases. Is this really the case?\n  The study of hyperparameter sensitivity can also be beneficial to practitioners\n  - Analysis of what kind of dataset characteristic makes the proposed distributional method more effective than the traditional method would be very useful. I feel existing methods such as CSS or the classifier-rejector approach might feel more like a direct approach to solving classification with rejection. Nevertheless, it is great to explore other approaches.\n  -  As a result, I believe revising the experiment section can significantly improve the paper.\n2. I found the writing and organization of this paper can be significantly improved. It takes until page 4 to state the proposed method, leaving only a small space for the experiment section. Perhaps some discussion of theory for $\\alpha-divergence and background of (DVI, DRO) can be wrapped up more compactly and defer to the appendix to make the paper more self-contained and provide sufficient discussion on experiments. \n  - For reading experience, it might also be useful to highlight the final objective function (or use a latex Algorithm environment) to outline what exactly we need to do to use the proposed method so that practitioners can quickly understand how the method works. I found section 4 is quite difficult to follow. In this form, we have to go through the whole paper and mix all the ingredients to get an idea how to implement the proposed method (e.g., Normalization, how to get a(x), inverse function of $\\varphi$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed reading and many suggestions for improving the readability of the paper. We will change “classification-rejection” to “classifier-rejector” as per the reviewer’s suggestion and space-permitting, will include the broader impact section in the main-body (or add a compact part of it in the conclusion). Please find other rebuttal points below.\n\n> Perhaps table representation or different way to show the result might help. \\[...\\] Only three simple datasets were used in this paper \n\nThank you for the suggestion. We will add tables corresponding to different coverage targets to the paper. Please see the shared section to see a demonstration on an additional dataset added in the rebuttal period. We hope that the added dataset from the MedMNIST collection also alleviates the concern that the datasets we consider are only simple.\n\n> KL and $\\\\alpha$ divergence give almost exactly the same performance in all cases \\[...\\] \\+ Q1\n\nWe would first like to note that we do include a hyperparameter sensitivity plot in Figure VII of the appendix. This tests different $\\\\alpha$s greater than 1\\.  \nIn terms of ranges of $(\\\\alpha \\\\geq 1)$-divergences, the change in performance-to-coverage is minor. This can also be further seen in the table provided in the shared response section (which more clearly shows that although the results are not identical, they are not significantly different).  \nOne difference we did find was that the coverage curve “shortens” for $\\\\tau \\\\in \\[0, 1\\]$. Although $\\\\tau$ can be increased in practice, the cut-off in coverage for $\\\\tau \\= 0$ shortens.\n\n> Analysis of what kind of dataset characteristic makes the proposed distributional method more effective \\[...\\] \\+ Q4: Weakness under noise\n\nWe believe that the dataset characteristic and your question about weakness under noise are linked. In general, in the noiseless case, there doesn’t seem to be a specific characteristic between what makes our proposed distributional rejection more effective. In general, it seems that the distributional rejection is slightly better in this clean case (results in tabular format, see general response).  \nHowever, it becomes clear that from the drop in performance in the noisy dataset setting, that distributional rejection using $(\\\\alpha \\\\geq 1)$-divergences with our current proposed implementation is not as robust to noise as other methods. This makes sense as the current approach relies on the calibration of models, which would break under distribution shift.  \nA note here is that future work examining divergences used in robustness may help alleviate this issue.\n\n> \\[...\\] writing and organization of this paper can be significantly improved.\n\nWe thank the reviewer for their suggestions on improving the writing and organization of the paper. The following summarizes the small changes regarding this topic that will be made:\n\n- Additional space provided by the camera ready will be utilized for further experimental discussion (including some of the above discussed points in this rebuttal section) and a table summarizing experiments.  \n  A related work (sub)section dedicated for rejection will be added as suggested by **Reviewer dzAS** (possibly deferred to the appendix).  \n  The content for DRO and GVI will be reduced if needed.  \n- To shortcut the theoretical parts of the paper, we will add an pseudo-code block next to Figure 1 which summarizes the abstract algorithm of distributional rejection. In addition, we will reference specific equations used to define the ratio $\\\\rho$.  \n  Additional text referencing this will also be added in the intro and start of Sec 3\\.  \n- Frame / highlight environments will also be used on equations referenced in the pseudo-code, as suggested by the reviewer.  \n- We believe that the current structure of the Sec 2-4 works well from a theoretical analysis point of view. And the proposed changes above will shortcut and identify key components for implementation.\n\nWe are confident that with the \\+1 page, adding additional discussion, tables, and cosmetic highlights can be complete to improve the paper’s presentation.\n\n> Q2: Clarity on $\\\\phi'$ and $L'$\n\nWe would like to confirm that $\\\\phi'$ is the derivative. Actually, the $L'$ notation also can be interpreted as a functional derivative of $L$ on $Q(x)$. We will mention this notational choice to clarify why notation is set this way. Do feel free to comment if this still makes the notation of $L’$ confusing. We are open to changing this to another symbol if required, eg, via font $\\\\mathrm{L}$ or with a bar $\\\\bar{L}$.\n\n> Q4: Theorem 4.2\n\nWe agree with the reviewers comment that the theorem is restrictive. Its purpose is just to verify that the proposed distributional rejection method produces Chow’s rule under theoretically optimal conditions.  \nWhen the optimal classifier is not optimal, the optimal rejection becomes a thresholding function over the pointwise risk of the model (in the language of CPE classification, see reference \\[48, 49\\]).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CPJFOOiLMy", "reviewer": "Reviewer_MrTu", "summary": "Classification with rejection emerges as a learning paradigm that allows models to abstain from making predictions. Traditional rejection learning methods typically modify the loss function, enabling models to explicitly reject making predictions when they are inaccurate or uncertain. These methods rely on providing good class probability estimates and often require training models from scratch. This paper proposes an interesting perspective to reconsider rejection not from the standpoint of the loss function, but rather from the perspective of distributions. By optimizing the risk of the loss with a $\\varphi$-divergence regularization term, it seeks an idealized data distribution that maximizes the performance of pretrained models.", "questions": "1. Although theoretically estimating the distribution $P(y|x)$ holds more promise than estimating a single probability $\\max P(y|x)$, in practical situations, estimating the distribution seems less feasible compared to single probability estimates, especially given the various calibration methods available for single probabilities.\n2. How did you design the expert in the experiments where you compared methods [1] from Learning to Defer？\n3. Since comparing with [1], it seems more reasonable to compare with [2], which is more relevant to the focus of this study.\n\n[1] Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. ICML 2020.\n\n[2]  Cao Y, Cai T, Feng L, et al. Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. NeurIPS, 2022.", "limitations": "N/A.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper takes a more interesting approach to rejection compared to previous papers.\n2. In theory, approaching from the perspective of distributions holds better prospects compared to considering single probabilities alone.\n3. The theoretical richness of this paper provides strong evidence for its claims.", "weaknesses": "1. For high-dimensional data, estimating density ratios and computing idealized distributions can be very challenging, implying that this method may only address low-dimensional data and its application scenarios are limited.\n2. This method involves a large number of hyperparameters: $\\lambda$, $\\tau$, $\\alpha$, $T$, which pose a significant burden for optimization.\n3. The experiments in the paper may only be conducted on specific datasets and under certain conditions (some simple datasets), which could limit their generalizability to broader scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for praising the interesting nature of our proposed approach and for the insightful questions. We are glad that the Reviewer appreciates the different approach we have taken and hope that the connections to DRO / GVI / distributions can lead to new theories. Please find the per-point rebuttal below.\n\n> \\[...\\] this method may only address low-dimensional data \\[...\\]\n\nWe agree that with a naive application of this approach, there may be problems in high dimensions. This is a weakness shared by other density ratio-style algorithms like importance weighting.  \nWe would however note that there has been success in leveraging representation learning to operate in a lower dimension. This can be used as a potential work around, especially if the model weights of the original classifier is available.\n\n> This is a weakness seen in other density ratio-style approaches like importance weighting.\n\nWe would like to comment on the hyperparameters noted by the reviewer.\n\n- We are unsure what hyperparameter $T$ the reviewer is referring to.   \n- $\\\\tau$ for the threshold can be treated as a hyperparameter. But it can also be used as a value to be found to satisfy a specific risk-coverage requirement. This can be equivalent to $c$ considered in the baseline approaches.  \n  Actually, **Reviewer dzAs** reminded us of some work which picks thresholds according to a coverage-modified-risk requirement \\[a\\]. As future work, this might be an interesting future direction to our work.  \n- We agree that tuning may present some challenges $\\\\lambda$ and $\\\\alpha$. However, we note that picking $\\\\lambda$ and $\\\\alpha$ is equivalent to picking an appropriate divergence. This is equivalent to how picking a surrogate loss for the classifier-rejection 0/1 loss function has many options.\n\nOne additional note that we make about hyperparameter tuning of our approach, is that although there are potentially more individual parameters one may want to pick from, it is substantially cheaper to tune than the other baselines. Indeed, the process of learning the 1D normalization constant $b$ required for density ratio rejectors (Line 287\\) is substantially cheaper than learning typically a whole network as required by the baselines. For our approach, we can keep the base classifier fixed and we only need to learn $b$ (see Appendix P.III for discussion on the number of tunable parameters of the approaches). \n\n\\[a\\] Geifman, Yonatan, and Ran El-Yaniv. \"Selective classification for deep neural networks.\" Advances in neural information processing systems 30 (2017).\n\n> Experiments (Weakness 3\\)\n\nPlease see the shared rebuttal section. Other reviewers also commented about adding other datasets for strong evaluation. We have taken **Reviewer dzAs**’s recommendation of considering datasets in the MedMNIST collection.\n\n> Q1: Calibration\n\nAlthough calibration for the multiclass case is more challenging in practice than perhaps estimating $\\\\max P(y \\\\mid x)$, we shouldn’t dismiss theoretically motivated frameworks due to this reason.  \nFrom a practical point of view, there are still many approaches which have shown reasonable success. For example, the standard temperature scaling approach that we utilize in the paper; or various different ensembling-style methods which we do not explore.\n\n> Q2: Experts for DEFER\n\nIn essence, we are assuming that the experts replicate the ground truth labels.  \nThis is equivalent to the reduction in \\[b, see Appendix A.2\\] which provides a surrogate loss function for rejection. Essentially, the learning to DEFER loss function can be used to learn a general $g \\\\colon \\\\mathbb{R}^d \\\\rightarrow \\\\mathbb{R}^{K+1}$ classifier for $K$ classes with the last $K+1$ output dimension being used for rejection.  \nThis reduction / interpretation was also commented by Cao et al. (reference \\[2\\] of the reviewer) as their method generalizes this instantiation of DEFER.\n\nWe will make sure to clarify how we are using DEFER in the next version of the paper.\n\n\\[b\\] Charoenphakdee, Nontawat, et al. \"Classification with rejection based on cost-sensitive classification.\" International Conference on Machine Learning. PMLR, 2021\\.\n\n> Q3: Comparing with \\[2\\]\n\nApologies, this seems to be a clarity issue. The GCE baseline tested in the paper is actually \\[2\\] (in retrospect, this may not be as clear as we also cited the general cross-entropy loss function’s original paper). Following up on Q2 as well, this GCE baseline can be interpreted as a generalization of the surrogate loss derived from DEFER.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SLaogw87LO", "reviewer": "Reviewer_dzAs", "summary": "The paper proposes a distributional perspective to build abstaining classifiers. By considering an idealized distribution, the authors show that this translates into optimizing a loss risk with a $\\varphi$-divergence regularization term. Moreover, they provide results when considering $\\alpha$-divergences, a specific family of $\\varphi$-divergences. Empirical evaluation is performed over clean and noisy datasets.", "questions": "I have a few questions for the authors:\n\n1) is there a reason why you did not compare with the simplest (and often the best) abstaining classifier, i.e. the softmax response? In [c], the authors show that the classifier built using this strategy can be optimal. Moreover, recent empirical works [d,e] show that score-based approaches outperform other abstaining classifiers.\n\n2) how did you train DEFER, without access to human predictions? As far as I know, the surrogate loss also requires specific human predictions; however, I could not find in the Appendix how this is done in practice.\n\n[c] Franc, Vojtech, Daniel Prusa, and Vaclav Voracek. \"Optimal strategies for reject option classifiers.\" Journal of Machine Learning Research 24, no. 11 (2023): 1-49.\\\n[d] Jaeger, Paul F., Carsten Tim Lüth, Lukas Klein, and Till J. Bungert. \"A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification.\" In The Eleventh International Conference on Learning Representations. (2023)\\\n[e] Pugnana, Andrea, Lorenzo Perini, Jesse Davis, and Salvatore Ruggieri. \"Deep neural network benchmarks for selective classification.\" arXiv preprint arXiv:2401.12708 (2024).", "limitations": "The authors discuss the limitations of their approach.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The main strengths of the paper are:\n\ni) the paper is clearly written, with clear contributions advancing the state of the art in abstaining classifiers\nii) the theoretical contributions seem sound;\niii) the proposed method is original and bridges DROs with Learning to Reject.", "weaknesses": "The main weaknesses of the paper are:\n\ni) there is no related work section explicitly dedicated to learning with a reject option. See, e.g., [a] for a recent survey.\n\n\nii) the empirical evaluation can be improved:\n* for instance, the paper considers only four baselines (mainly from Learning to Defer literature) and ignores popular alternatives such as softmax response [26]\n* the method is tested only on three datasets, which seem easy to solve. In particular, MNIST might be too easy to evaluate abstaining classifiers (an accuracy of 100% is easy to achieve with relatively few rejections). A more interesting dataset collection could be MedMNIST [b], which contains medical images from real data tasks.\n\n\n\n[a] Hendrickx, Kilian, Lorenzo Perini, Dries Van der Plas, Wannes Meert, and Jesse Davis. \"Machine learning with a reject option: A survey.\" Machine Learning 113, no. 5 (2024): 3073-3110.\\\n[b] Yang, Jiancheng, Rui Shi, and Bingbing Ni. \"Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis.\" In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 191-195. IEEE, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for sharing their insights and providing useful suggestions on improving our evaluation. Please find the rebuttal below.\n\n> Related Work\n\nWe thank the reviewer for their various references. We will make sure to include these in the next version of the paper. We would also be happy to add a dedicated related work section. In particular, discussion regarding softmax response and score-based vs abstaining classifiers would be added.  \nWe would add this either to the main text (Sec 2\\) with the additional page / space of the camera ready (alongside some reduction on the GVI / DRO section) or via an additional appendix section.\n\n> Empirical evaluation can be improved (datasets)\n\nThanks for the dataset recommendation. Please see the shared rebuttal response section. We have added some preliminary results utilizing the MedMNIST collection.\n\n> Q1: Softmax Response (SR)\n\nOur work primarily started from examining rejection approaches which were inspired by surrogate loss functions and cost-sensitive losses. These approaches formed the baselines which we considered (many of these papers did not consider softmax response).  \nAlthough we do not include softmax response as a baseline, we note that the density-ratio rejectors have a strong connection to this method.  \nIn the case of the KL-divergence rejectors with the practical trick employed (as noted in Line 271 onwards), our rejector will threshold the model’s predictive entropy. This is similar to the maximum softmax score.  \nThe theoretical framework of prediction using density-ratios presents a generalization of this elucidating further connections to DRO / GVI. We further believe that the sample complexity bounds do not have an equivalent in SR.\n\nEmpirically, as a result, our approach and naive threshold rejection using the softmax scores should be comparable (when both utilize temperature scaling).  \nThe component of SR that we are missing is the calculation of the threshold according to a desired coverage guarantee. This would be a nice direction for future work to automatically prescribe $\\\\tau$.\n\nWe will include this discussion and connection to the SR in the next version of the paper.\n\n> Q2: DEFER without expert\n\nOne can utilize DEFER without human predictions by utilizing a ground-truth expert (one returning the data’s label). In particular, it has been shown that this interpretation provides a surrogate loss function of the 0-1-c loss function. This has been shown in \\[a, see Appendix A.2\\] and \\[b\\] which generalizes this. We are happy to include this discussion to make this baseline choice clear.\n\n\\[a\\] Charoenphakdee, Nontawat, et al. \"Classification with rejection based on cost-sensitive classification.\" International Conference on Machine Learning. PMLR, 2021\\.\n\n\\[b\\] Cao Y, Cai T, Feng L, et al. Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. NeurIPS, 2022\\.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful comments and various suggestions.  
We are grateful to the reviewers for recognizing the novelty of our work, providing a “\[...\] more interesting approach to rejection compared to previous papers” (**Reviewer MrTu**). We appreciate the comments that our work has “\[...\] clear contributions advancing the state of the art in abstaining classifiers.” (**Reviewer dzAs**) including “\[...\] support multiclass classification case \[... which\] is not that straightforward \[...\] (**Reviewer MiAk**); and the recognition that our approach is “\[...\] quite neat and has nice connections with GVI and DRO \[... and the\] math is quite crisp throughout the paper.” (**Reviewer m8bi**).

Please find the per point rebuttal below. In this shared section, we additionally provide results using the MedMNIST dataset collection and provide some samples for presenting the performance of the results via tables.

## Additional datasets (MedMNIST) & Tables

**Reviewer MiAk, MrTu and dzAs** mentioned that a potential weakness of the current paper is the simplicity of the currently tested datasets. Noise was utilized to add complexity to the dataset settings and to provide cases where the base classifier does not reach a high (near 100%) accuracy. To supplement the experiments on these datasets and their noisy label noise variants, in the rebuttal we provide some additional experimental results on the MedMNIST dataset collection recommended by **Reviewer dzAs**.

Additionally, **Reviewer MiAk** mentioned that presenting results in tabular format might enhance the readability of the experimental results settings. In this rebuttal section, we present accuracy tables for the MedMNIST dataset results, where we select hyperparameters (and their coverage) which are closest to a specific coverage goal. In particular, we will look at 90% coverage (high coverage) and 80% coverage (lower coverage) regimes.
We have also included preliminary tables in the attached PDF for Figure 2 of the paper, where the coverage target is 80%.

In the following, we present results for the OrganSMNIST dataset from the MedMNIST dataset collection. This is a 2D 28x28 classification task over grayscale images and 11 classes. We note that for this dataset, the base classifier (which we use a ResNet18 following the MedMNIST paper \[a\]) is not close to 95-100%.  
The general setup is identical to that in the paper (5-fold cross validation \+ grid search over hyperparameters of either $\\tau$ or $c$). The only difference is that there is a slightly reduced hyperparameter search of $c$ for ranges between 0.01 and 0.3 (rather than 0.01 to 0.5; DEFER had some additional hyperparameters tested) to complete these preliminary experiments on time for the rebuttal.

The result tables are given as follows:

### OrganSMNIST (80% Coverage Target)

|	| Model Name        	| Hyperparameter   | Accuracy   	| Coverage    	|
|---:|:----------------------|:-----------------|:---------------|:----------------|
|  0 | Base Clf          	| -            	| 90.119 (0.867) | 100.000 (0.000) |
|  1 | $\rm KL-Rej$      	| 0.64         	| 97.548 (0.359) | 80.512 (0.910)  |
|  2 | $(\alpha=3)\rm{-Rej}$ | 0.6          	| 97.660 (0.400) | 80.144 (0.795)  |
|  3 | $\rm PredRej$     	| 0.2          	| 89.938 (4.094) | 81.662 (13.320) |
|  4 | $\rm CSS$         	| 0.08         	| 96.566 (0.854) | 81.306 (1.255)  |
|  5 | $\rm DEFER$       	| 0.14         	| 94.068 (0.677) | 80.743 (3.946)  |
|  6 | $\rm GCE$         	| 0.1          	| 93.973 (0.850) | 80.068 (2.677)  |

### OrganSMNIST (90% Coverage Target)

|	| Model Name        	| Hyperparameter   | Accuracy   	| Coverage    	|
|---:|:----------------------|:-----------------|:---------------|:----------------|
|  0 | Base Clf          	| -            	| 90.119 (0.867) | 100.000 (0.000) |
|  1 | $\rm KL-Rej$      	| 0.54         	| 93.757 (0.308) | 90.000 (1.241)  |
|  2 | $(\alpha=3)\rm{-Rej}$ | 0.4          	| 93.565 (0.327) | 90.639 (1.088)  |
|  3 | $\rm PredRej$     	| 0.22         	| 91.556 (1.603) | 90.298 (6.066)  |
|  4 | $\rm CSS$         	| 0.19         	| 92.437 (0.313) | 90.361 (1.619)  |
|  5 | $\rm DEFER$       	| 0.41         	| 91.557 (0.907) | 91.916 (1.496)  |
|  6 | $\rm GCE$         	| 0.25         	| 92.203 (0.346) | 90.714 (1.429)  |

### Discussion

From these results, interestingly our density-ratio rejectors are actually superior over these datasets. Thus, our method has practical relevance to datasets beyond those previously tested.

Some other observations include:

- There are minor differences between KL and $\\chi^2$ / $\\alpha \= 3$. However, the differences are not significant.  
- Performance of the density-ratio rejectors are not tied to near perfect accuracy of the base classifier.  
- Our method can perform better at high and lower coverage target regimes.

We will add the new experimental result in the next version of the paper alongside other 2D datasets in the MedMNIST collection.

\[a\] Yang, Jiancheng, et al. "Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification." Scientific Data 10.1 (2023): 41\.

Meta-review: In this work, the authors develop a new algorithm for classification with rejection by learning an idealized distribution using the model, and rejecting a classification if the ratio of the idealized density to the data density is too large. Efficacy is demonstated using both theoretical error bounds and a short series of experiments. Reviews were generally positive, but criticised the limited number of experiments and acknowledged that both density estimation and the ratio estimator are likely to succumb to the curse of dimensionality for larger problems. In their rebuttal, the authors provided a series of additional experiments highlighting potential efficacy for problems of larger size. After the discussion phase, all reviewers agreed the paper was of high quality with the additional results. Subject to these inclusions, I recommend acceptance.
Decision: Accept (poster)