Rejection via Learning Density Ratios

Authors: alexander  soen
hisham  husain
philip  schulz
vu  nguyen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 06:43:22

Abstract: Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions. 
The predominant approach is to alter the supervised learning pipeline by augmenting typical loss functions, letting model rejection incur a lower loss than an incorrect prediction.
Instead, we propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance.
This can be formalized via the optimization of a loss's risk with a $ \phi$-divergence regularization term.
Through this idealized distribution, a rejection decision can be made by utilizing the density ratio between this distribution and the data distribution.
We focus on the setting where our $ \phi $-divergences are specified by the family of $ \alpha $-divergence.
Our framework is tested empirically over clean and noisy datasets.

Main paper: 
 introduction: 
 Forcing Machine Learning (ML) models to always make a prediction can lead to costly consequences. Indeed, in real-world domains such as automated driving, product inspection, and medical diagnosis, inaccurate prediction can cause significant real-world harm [16, 29, 53, 50] . To deal with such a dilemma, selective prediction and classification with rejection were proposed to modify the standard supervised learning setting [15, 17, 72] . The idea is to allow for a model to explicitly reject making a prediction whenever the underlying prediction would be either inaccurate and / or uncertain.In classification, a confidence-based approach can be utilized, where a classifier is trained to output a "margin" which is used as a confidence score for rejection [7, 31, 71, 59, 53] . The model rejects whenever this confidence score is lower than an assigned threshold value. A key aspect of these approaches is that they rely on good probability estimates Pr(Y | X = x) [60] , i.e., being calibrated [71, 53] . While some approaches avoid explicit probability estimation, these are typically restricted to binary classification [7, 31, 47] . Empirically, approaches utilizing confidence scores have shown to outperform other methods, even with simple plugin estimates for probabilities [26, 39, 58] .Another classifier-rejection approach aims to simultaneously train a prediction and rejection model in tandem [16, 17, 53] . These approaches are theoretically driven by the construction of surrogate loss functions, but in the multiclass classification case many of these loss functions have been shown to not be suitable [53] . For the multiclass classification setting, one approach connects classification with rejection to cost-sensitive classification [14] . In practice, these classification-rejection approaches require models to be trained from scratch using their specific loss function and architecture -if there ( †) Work done while interning at Amazon. (*) Work done while employed at Amazon.x Data Dist. P Idealized Dist. Q Reject if ρ(x) = dQ dP (x) ≤ τ Figure 1 : An idealized distribution Q is learned to minimizes the loss of a model. We then compare Q with the original data distribution P via a density ratio ρ = dQ/dP. A rejection criteria is defined via threshold value τ . is an existing classifier for the dataset, it must be discarded. A recent approach proposes to learn a post-hoc rejector on top of a pretrained classifier via surrogate loss functions [48] .In this work, to learn rejectors we shift from a loss function perspective to a distributional perspective. Given a model and a corresponding loss function, we find a distribution where the model and loss performs "best" and compare it against the data input distribution to make rejection decisions (see Fig. 1 and Algorithm 1 with equations boxed). As such, the set of rejectors that we propose creates a rejection decision by considering the density ratio [67] between a "best" case (idealized) distribution and the data distribution, which can be thresholded by different values τ to provide different accuracy vs rejection percentage trade-offs. To learn these density ratios for rejection, we consider a risk minimization problem which is regularized by φ-divergences [1, 19] . We study a particular type of φ-divergences as our regularizer: the family of α-divergences which generalizes the KL-divergence. To this end, one of our core contributions in this work is providing various methods for constructing and approximating idealized distributions, in particular those constructed by α-divergences.The idealized distributions that we consider are connected to adversarial distributions examined in Distributionally Robust Optimization (DRO) [63] and the distributions learned in Generalized Variational Inference (GVI) [42] ; and, as such, the closed formed solutions found for our idealized distribution have utility outside of rejection. Furthermore, when utilizing the KL-divergence and Bayes optimal models, we recover the well known optimal rejection policies, i.e., Chow's rule [15, 72] . Our rejectors are then examined empirically on 6 different datasets and under label noise corruption.In summary, our contributions are the following:• We present a new framework for learning with rejection involving the density ratios of idealized distributions which mirrors the distributions learned in DRO and GVI; • We show that rejection policies learned in our framework can theoretically recover optimal rejection policies, i.e., Chow's rule; • We derive optimal idealized distributions generated from α-divergences; • We present a set of simplifying assumptions such that our framework can be utilized in practice for post-hoc rejection and verify this empirically. 
preliminaries: 
 Notation Let X be a domain of inputs and Y be output targets. We primarily consider the case where Y is a finite set of N labels [N ], where [N ] . = {0, . . . , N -1}. We also consider output domains Y ′ which is not necessarily the same as the output data Y, e.g., class probabilities estimates in classification. Denote the underlying (usually unknown) input distribution as P x,y ∈ D(X × Y). The marginals of a joint distribution are denoted by subscript, e.g., P y ∈ D(Y) for the marginal on the label space. We denote conditional distributions, e.g., P x|y . For marginals on D(X), the subscripting of x is implicit with P = P x . The empirical distributions of P are denoted by PN , with N denoting the number of samples used to generate it. Denote the Iverson bracket p = 1 if the predicate p is true and p = 0 otherwise [43] . The maximum [z] + . = max{0, z} is shorthanded.Learning with Rejection We first recall the standard risk minimization setting for learning. Suppose that we are given a (point-wise) loss function ℓ : Y × Y ′ → R ≥0 which measures the level of disagreement between predictions and data. Given a data distribution P x,y and a hypothesis set of models h ∈ H, we aim to minimize the expected risk w.r.EQUATIONOne way of viewing learning with rejection is to augment the risk minimization framework by learning an additional rejection function r : X → {0, 1}. This can be formally defined using a regularization term c ∈ R ≥0 which controls the rate of rejection:EQUATIONwhere R denotes a hypothesis set of rejection functions. Once minimized, a combined model f which can return a rejection token ® to abstain from predictions can be defined,EQUATIONSuppose that R is complete (contains all possible rejectors). In such a case, one can see that setting c = ∞ reduces Eq. ( 2) to standard risk minimization setting given by Eq. ( 1). Furthermore, setting c = 0 will result in r always rejecting, i.e., the case where there is no rejection cost. Some values of c ∈ R ≥0 can be redundant. If the loss ℓ is bounded above by B, then any c > B is equivalent to setting c = B -the optimal r ⋆ will be to never reject. In classification, where ℓ is taken to be the zero-one-loss, c is typically restricted to values in (0, 0.5) as otherwise low confidence prediction can be accepted [53] -our work only considers this case. [59] explores the c ∈ [0.5, 1] scenario.So far, the learning task has been left general. By considering Class Probability Estimation (CPE) [61] , we recover familiar optimal rejection and classifier pairs. Theorem 2.1 (Optimal CPE Rejection / Chow's Rule). Let us consider the binary CPE setting, whereY = {0, 1}, Y ′ = [0, 1],and ℓ be any proper1 loss function [60] (e.g., log loss). Then w.r.t. Eq. (2), the optimal classifier is given by h⋆ (x) = P[Y = 1 | X = x]and the optimal rejector is given by r⋆ (x) = E Y∼h ⋆ (X=x) [ℓ(Y, h ⋆ (x))] ≥ c .The theorem can easily be generalized to non-binary cases. We note that Theorem 2.1 is a generalization of the well known Chow's rule of classification with rejection [15, 14] . Indeed, taking ℓ to be the zero-one-loss function, we get r⋆ (x) = |2 • P[Y = 1 | X = x] -1| ≥ 1 -c .One can further clarify this by noticing that Pr[r(X) = 1] = E P [r(X)]. In the general proper loss case, the optimal rejector is thresholding a generalized entropy function (known as the conditional Bayes risk [60] ). This would correspond to thresholding the class probabilities P[Y = y | X = x] (via the Bayes posterior h ⋆ ), but with different thresholding values per class y ∈ Y (unless ℓ is a symmetric loss function).Although the objective of Eq. ( 2) follows the cost-based model of rejection [15] , other models of rejection exist in the literature. Alternatively, the bounded improvement model of rejection [54, 29] maximizes coverage (non-rejection rate) whilst maintaining a constraint on the performance of the rejection measured by the selective risk (the first term of Eq. ( 2) inversely weighted by the coverage). Optimality conditions of the bounded improvement have been explored in [27] . In the bounded abstention model, the constraint and objective is switched -the selective risk is minimized with a constraint on minimum coverage [54] . Our objective function Eq. ( 2) (and the cost-based model) can be interpreted as a change in the type of risk considered and a change in the hard coverage constraint to a soft constraint w.r.t. bounded abstention. The optimal strategies of these approaches are explored in [28] . Alongside Section 1, further details about rejection can be found in [34] .Generalized Variational Inference Generalized Variational Inference (GVI) [42] provides a framework for a generalized set of entropy regularized risk minimization problems. In particular, GVI generalizes Bayes' rule by interpreting the Bayesian posterior as the solution to a minimization problem [73] . The Bayes' rule minimization is given by,EQUATIONwhere θ ∼ Q denotes a set of model parameters with likelihood function p(x | θ) and {z i } denotes data. Here, P ∈ D(Θ) denotes the prior and the optimal Q ⋆ denotes the posterior in Bayes' rule.For GVI, we generalize a number of quantities. For instance, the 'loss' considered can be altered from the log-likelihood log p(z | θ) to an alternative loss function over samples {z i }. The divergence function KL(P ∥ Q) can also be altered to change the notion of distributional distance. Furthermore, the set of distributions being minimized D(Θ) can also be altered to, e.g., reduce the computational cost of the minimization problem. One can thus alter Eq. ( 4) to give the following:EQUATIONwhere λ > 0 and L, D, and Q denote the generalized loss, divergence, and set of distribution, respectively. Here λ seeks to act as a regularization constant which can be tuned.In our work, we will consider a GVI problem which changes the loss function and divergence to define an entropy regularized risk minimization problem. Our loss function corresponds to the learning setting. For the change in divergence, we consider φ-divergence [1, 19] , which are otherwise referred to as f -divergences [62] or the Csiszár divergence [18] . Definition 2.2. Let φ : R → (-∞, ∞] be a convex lower semi-continuous function with φ(1) = 0 then the corresponding φ-divergence over non-negative measures for P, Q is defined as:EQUATIONWe note that for φ-divergences, the regularization constant in Eq. ( 5) can be absorbed into the φ-divergence generator, i.e., λ• D φ (P ∥ Q) = D λ•φ (P ∥ Q).Distributionally Robust Optimization A related piece of literature is Distributionally Robust Optimization (DRO) [63] , where the goal is to find a distribution that maximizes (or minimizes) the expectation of a function from a prescribed uncertainty set. Popular candidates for these uncertainty sets include all distributions that are a certain radius away from a fixed distribution by some divergence. φ-divergences have been used to define such uncertainty set [8, 22, 45] . Given radius ε > 0, define B φ ε (P) . = {Q ∈ D(X × Y) : D φ (P ∥ Q) < ε}.EQUATIONThe max over Q is typically over the target space Y. That is, Q(x, y) = Q(y) • P(x | y) and the max is adversarial over the marginal label distribution [74] . Note that converting the ε-ball constraint into a Lagrange multiplier, the inner optimization over Q in Eq. ( 7) mirrors Eq. (5) . The connection between GVI and adversarial robustness has been previously noted [37] .Typically in DRO and related learning settings, the construction of the adversarial distribution defined by the inner maximization problem is implicitly solved. For example, when φ is twice differentiable, it has been shown that the inner maximization can be reduced to a variance regularization expression [22, 21] ; whereas other choices of divergences such as kernel Maximum Mean Discrepancy (MMD) yields kernel regularization [66] and Integral Probability Metrics (IPM) correspond to general regularizers [36] . Another popular choice is the Wasserstein distance which has shown strong connections to point-wise adversarial robustness [10, 11, 64] .The aforementioned work, however, seek only to find the value of the inner maximization in Eq. ( 7) without considering the form the optimal adversarial distribution takes. 
rejection via idealized distributions: 
 We propose learning rejection functions r : X → {0, 1} by comparing data distributions P to a learned idealized distribution Q (as per Fig. 1 ). An idealized distribution (w.r.t. model h) is a distribution which when taken as data results in low risk (per Eq. ( 1)). Q are idealized rather than 'ideal' as they do not solely rely on a model's performance, but are also regularized by their distance from the data distribution P. Formally, we define our idealized distribution via a GVI minimization problem. Definition 3.1. Given a data distribution P x,y ∈ D(X × Y) and a φ-divergence, an idealized distribution Q ∈ D(X) for a fixed model h and loss ℓ is a distribution given by arginf Q∈D(X)EQUATIONwhereL(Q) . = E Q [L ′ (X)] and L ′ (x) . = E P y|x [ℓ(Y, h(x))].Given the objective of Eq. ( 8), an idealized distribution Q will have high mass when L ′ (x) is small and low mass when L ′ (x) is large. The φ-divergence regularization term prevents the idealized distributions from collapsing to a point mass. Indeed, without regularization the distance from P, idealized distributions would simply be Dirac deltas at values of x ∈ X which minimize L ′ (x).With an idealized distribution Q, a rejection can be made via the density ratio [67] w.r.t. P.Definition 3.2. Given a data distribution P and an idealized distribution Q ≪ P, the τ -ratiorejector w.r.t. Q is given by r τ (x) .= ρ(x) ≤ τ , where ρ(x) .= dQ/dP(x).Definition 3.2 aims to reject inputs where the idealized rejection distribution has lower mass than the original data distribution. Given Definition 3.1 for idealized distribution, small values of ρ(x) corresponds to regions of the input space X where having lower data probability would decrease the expected risk of the model. Note that we do not reject on regions with high density ratio ρ as L ′ (x) would be necessarily small or the likelihood of occurrence P(x) would be relatively small. We restrict the value of τ to (0, 1] to ensure that rejection is focused regions where L ′ (x) is high with high probability w.r.t. P(x) -further noting that τ = 0 always rejects.Although Definitions 3.1 and 3.2 suggests that we should learn distributions Q (and P) separately to make our rejection decision, in practice, we can learn the density ratio dQ/dP directly. Indeed, through the definition of Definition 2.2, we have that equivalent minimization problem:EQUATIONSuch an equivalence has been utilized in DRO previously [23, Proof of Theorem 1] . Notice that the learning density ratio ρ in Eq. ( 9) is analogous to the acceptor 1 -r(x) in Eq. ( 2). Indeed, ignoring the normalization constraint E P [ρ(X)] = 1, by restricting ρ(x) ∈ {0, 1}, letting λ = c, and letting φ(z) = z = 0 , the objective function of Eq. ( 9) can be reduced to:EQUATIONGiven the restriction of ρ to binary outputs {0, 1} (and Definition 3.2), we have that r τ (x) = 1 -ρ(x).As such, Eq. (10) in this setting is equivalent to the minimization of r in Eq. (2) (with R as all possible functions X → {0, 1}). Through the specific selection of φ and restriction of r, we have shown that rejection via idealized distributions generalizes the typical learning with rejection objective Eq. (2) .By utilizing form of φ-divergences, we find the form of the idealized rejection distributions Q and their corresponding density ratio ρ used for rejection. Theorem 3.3. Given Definition 3.1, the optimal density ratio function ρ of Eq. ( 9) is of the form,EQUATIONwhere a(x) are Lagrange multipliers to ensure non-negativity ρ φ λ (•) ≥ 0; and b is a Lagrange multiplier to ensure the constraint E P [ρ φ λ (X)] = 1 is satisfied. Furthermore, the optimal idealized rejection distribution is given by: Qφ (x) = P(x) • ρ φ λ (x).Taking h as the Bayes posterior h ⋆ , L ′ becomes a function of the ground truth posterior Pr(Y | X = x).Hence taking the output h as a neural network plugin estimate of the underlying true posterior Pr(Y | X = x) (see Section 4.3) yields an approach similar to softmax response, i.e., rejection based on the output of h when it outputs softmax probabilities [29] . Theorem 3.3 presents a general approach to generating rejectors from these plugin estimates, as a function of ℓ and φ.Connections to GVI and DRO The formulation and solutions to the optimization of idealized distributions has several connections to GVI and DRO. In contrast to the setting of GVI, Eqs. ( 4) and ( 5), the support of the idealized distributions being learned in Definition 3.1 is w.r.t. inputs X instead of parameters. Furthermore, the inner maximization of the DRO optimization problem, Eq. ( 7), seeks to solve a similar form of optimization. For idealized distributions, the maximization is switched to minimization and we consider a distribution over inputs X instead of targets Y. Indeed, notice that the explicit inner optimization of DRO (in Eq. ( 7)) over Q can be expressed as the following via the Fan's minimax Theorem [25, Theorem 2] :EQUATIONNotably, the loss L(Q) in Eq. ( 7) can be simply negated to make the optimization over Q in Eq. ( 12) equivalent to DRO Eq. ( 8) (noting the only requirement for Eq. ( 7) to Eq. ( 12) is that L(Q) is a linear functional of Q). This shows that switching the sign of the loss function changes idealized distributions of Definition 3.1 to DRO adversarial distributions. Indeed, through the connection between our idealized distributions and DRO adversarial distributions, the distributions Q φ (x) will have the same form as the optimal rejection distributions implicitly learned in DRO. Corollary 3.4. Suppose Q φ λ denotes the optimal idealized distribution in Theorem 3.3 (switchingL(Q) to -L(Q)) for a fixed λ > 0. Further let λ ⋆ ∈ arginf λ>0 {-L(Q φ λ ) + λ • (D φ (Q φ λ ∥ P) -ε)}.Then the optimal adversarial distribution in the inner minimization for DRO (Eq. ( 7)) is Q φ λ ⋆ . As such, the various optimal idealized distributions (w.r.t. Definition 3.1) for rejection we will present in the sequel can be directly used to obtain the optimal adversarial distributions for DRO. 
learning idealized distributions: 
 In the following section, we explore optimal closed-form density ratio rejectors. We first examine the easiest example -the KL-divergence -and then consider the more general α-divergences.Let us first consider the KL-divergence [2] for constructing density ratio rejectors via Theorem 3.3.Corollary 4.1. Let φ(z) = z log z -z + 1 and λ > 0.The optimal density ratio of Definition 3.1 is,EQUATIONOne will notice that ρ KL λ corresponds to an exponential tilt [24] of P to yield a Gibbs distribution Q KL . The KL density ratio rejectors are significant in a few ways. First, we obtain a closed-form solution due to the properties of 'log' and complementary slackness, i.e., a(•) = 0. Secondly, due to the properties of exp, the normalization term b has a closed form solution given by the typical log-normalizer term of exponential families.Another notable property of utilizing a KL idealized distribution is that it recovers the previously mentioned optimal rejection policies for classical modelling with rejection via cost penalty. Theorem 4.2 (Informal). Given the CPE setting Theorem 2.1, if h = h ⋆ is optimal, then there exists a r KL τ which is equivalent to the optimal rejectors in Theorem 2.1. Theorem 4.2 states that the KL density rejectors (with correctly specified λ and τ ) with optimal predictors h ⋆ recovers the optimal rejectors of the typical rejection setting, i.e., Chow's rule.Until now, we have implicitly assumed that the true data distribution P is accessible. In practice, we only have access empirical PN , defining subsequent rejectors ρλ,N . We show that for the KL rejector, PN is enough given the following generalization bound. sup x∈T ρ KL λ (x) -ρKL λ,N (x) ≤ C • 2 N log 2M δ ,where C = exp (B/λ) 3 • sinh (B/λ).Looking at Theorem 4.3, essentially we pay a price in generalization M for each element x ∈ T we are testing for rejection. For generalization, it is useful to consider how N, M changes our rate in Theorem 4.3. If we assume that the test set T is small in comparison to the N samples used to generate empirical distribution P, then the O(1/ √ N ) rate will dominate. A concrete example of this case is when |X| is finite. A less advantaged scenario is when M ≈ N , yielding O(log(N )/ √ N )the scenario where we test approximately the same number of data points as that used to learn the rejector. This rate will still decrease with N → ∞, although with a log N price.Although the general case of finding idealized rejection distributions for φ-divergences is difficult, we examine a specific generalization of the KL case, the α-divergences [3] . Definition 4.4. For α ∈ R, the α-divergence D α is defined as the φ α -divergence, whereφ α (z) . =      4 1-α 2 1 -z 1+α 2 -2 1-α (z -1) if α ̸ = ±1 -log z + (z -1) if α = -1 z log z -(z -1) if α = 1.We further define ψ α , where ψ α (z) = z 1-α 2when α ̸ = 1 and ψ α (z) = log z when α = 1.Note that taking α = 1 recover the KL-divergence. The α-divergence covers a wide range of divergences including the Pearson χ 2 divergence (α = 3). For the density ratio, α-divergences with α ̸ = 1 (i.e. not KL) can be characterized as the following. Theorem 4.5. Let α ̸ = 1 and λ > 0. For φ α , the optimal density ratio rejector ρ α λ (x) is,EQUATIONa(x) are Lagrange multipliers for positivity; and b is a Lagrange multiplier for normalization.One major downside of using general φ-divergences is that solving the Lagrange multipliers for the idealized rejection distribution is often difficult. Indeed, the "log" and "exp" ensures non-negativity of the idealized distribution when the input data P is in the interior of the simplex; and also provides a convenient normalization calculation. For α-divergences, the non-negative Lagrange multipliers a(•) can be directly solved given certain conditions.Corollary 4.6. Suppose α > 1 and λ > 0, then Eq. ( 14) simplifies to,EQUATION)where we take non-integer powers of negative values as 0.On the other hand, for α ≤ -1, D α (P ∥ Q) = ∞ whenever Q is on the boundary whenever P is not on the boundary [3, Section 3.4.1] . As such, we can partially simplify Eq. ( 14) for α ≤ -1. Corollary 4.7. Suppose α ≤ -1, λ > 0, and P lies in the simplex interior, then a(•) = 0 in Eq. ( 14).Both Corollaries 4.6 and 4.7 can provide a unique rejector policy than the KL-divergence variant. Corollary 4.7 can provide a similar effect when a(x) ̸ = 0 for all x. Nevertheless, having to determine which inputs a(x) ̸ = 0 and solving these values are difficult in practice. As such, we focus on α > 0.If there are values of L ′ (x) with high risk, the max will flatten these inputs to 0 in Corollary 4.6. However, if the original model h performs well and L ′ (x) is relatively small for all x, then it is possible that the max is not utilized. In such a case, the α-divergence rejectors can end up being similar -this follows the fact that (as φ ′′ α (1) > 0) locally all α-divergences will be similar to the χ 2 / (α = 3)-divergence [56, Theorem 7.20] . This ultimately results in α-divergences being similar to, e.g., Chow's rule when h y (x) ≈ Pr(Y = y | X = x) in classification via Theorem 2.1.Among the α > 0 cases, we examine the χ 2 -divergence (α = 3) which results in closed form for bounded loss functions and sufficient large regularizing parameter λ. Corollary 4.8. Suppose that ℓ(•,EQUATIONThe condition on λ for Corollary 4.8 is equivalent to rescaling bounded loss functions ℓ. Indeed, by fixing λ = 1, we can achieve a similar Theorem with suitable rescaling of ℓ. Nevertheless, Eq. ( 16) provides a convenient form to allow for generalization bounds to be established.Theorem 4.9. Assume we have bounded loss |ℓ(•, •)| ≤ B for B > 0, λ > 2B, PN ⊂ P with h.p., and T ⊂ Supp(P). Suppose M = |T| < +∞, then with probability 1 -δ, we have thatsup x∈T ρ α=3 λ (x) -ρα=3 λ (x) ≤ B λ • 2 N log 2M δ .Notice that Theorem 4.9's sample complexity is equivalent to Theorem 4.3 up to constant multiplication. Hence, the analysis of Theorem 4.3 regarding the scales of N, M hold for Theorem 4.9.A question pertaining to DRO is what would be the generalization capabilities of the corresponding adversarial distributions Qλ,N .= PN • ρλ,N (through Corollary 3.4). On a finite domain, via Theorems 4.3 and 4.9 and a simple triangle inequality, one can immediately bound the total variation TV(Qλ,N , Q λ ) ≤ O(1/ √ N ), see Appendix M for further details.We consider practical concerns for utilizing the KL-or (α > 1)-divergence case, Eqs. ( 13) and ( 15), for post-hoc rejection. To do so, we need to estimate the loss L ′ (x) and rejector normalizer Z or b.Loss: The former is tricky, we require an estimate to evaluate L ′ (x) = E P y|x [ℓ(Y, h(x))] over any possible x ∈ X to allow us to make a rejection decision. Implicitly, this requires us to have a high quality estimate of P y|x . In a general learning setting, this can be difficult to obtain -in fact it is just the overall objective that we are trying to learning, i.e., predicting a target y given an input x. However, in the case of CPE (classification), it is not unreasonable to obtain an calibrated estimate of P y|x via the classifier h : X → D(Y) [55] . In Section 5, we utilize temperature scaling to calibrate the neural networks we learn to provide such an estimate [32] . Hence, we setL ′ (x) = E Y∼h(x) [ℓ(Y, h(x))]. For proper CPE loss functions, E Y∼h(x) [ℓ(Y, h(x))] acts as a generalized entropy function. As such, the rejectors Eqs. ( 13) and ( 15) act as functions over said generalized entropy functions. It should be noted that simply considering the softmax outputs of neural networks for probability estimation have seen prior success [29, 39] . This is equivalent to taking the 0-1-loss function [15, 35, 28] and taking a plugin estimate of probabilities via the neural network's output. The study of using plugin estimates have also been explored in the model cascade literature [40] .Normalization: For the latter, we utilize a sample based estimate (over the dataset used to train the rejector) of E P [ρ φ λ (X)] is utilized to solve the normalizers Z and b. In the case of the KL-divergence rejector, this is all that is required due to the convenient function form of the Gibbs distribution, i.e., the normalizer Z can be simply estimated by a sample mean. However, for α > 1-divergences b needs to be found to determine the normalization. Practically, we find b through bisection search ofEQUATIONThis practically works as the optimization b is over a single dimension. Furthermore, we can have that b > min x {L ′ (x)/λ}. As an upper bound over the possible values of b, we utilize a heuristic where we multiple the corresponding maximum of L ′ (x)/λ with a constant.Threshold τ : In addition to learning the density ratio, an additional consideration is how to tune τ in the rejection decision, Definition 3.2. Given a fixed density estimator ρ, change τ amounts to changing the rate of rejection. We note that this problem is not limited to our density ratio rejectors, where approaches with rejectors r ∈ R via a (surrogate) minimization of Eq. ( 2) may be required multiple rounds of training with different rejection costs c to find an accept rate of rejection. In our case, we have a fixed ρ which allows easy tuning of τ given a validation dataset, similar to other confidence based rejection approaches, e.g., tuning a threshold for the margin of a classifier [7] . 
experiments: 
 In this section, we evaluate our distributional approach to rejection across a number of datasets. In particular, we consider the standard classification setting with an addition setting where uniform label noise is introduced [4, 30] . For our density ratio rejectors, we evaluate the KL-divergence based To evaluate our density ratio rejectors and baselines, we compare accuracy and acceptance coverage. Accuracy corresponds to the 0-1 loss in Eq. ( 1) and the acceptance coverage is the percentage of non-rejections in the test set. Ideally, a rejector smoothly trade-offs between accuracy and acceptance, i.e., a higher accuracy can be achieved by decreasing the acceptance coverage by rejecting more data.We study this trade-off by examining multiple cut-off values τ and rejection costs c.We consider 6 multiclass classification datasets. For tabular datasets, we consider the gas drift dataset [68] and the human activity recognition (HAR) dataset [5] . Each of these datasets consists of 6 classes to predict. Furthermore, we utilize a two hidden layer MLP NN model for these datasets. We consider the MNIST image dataset [46] (10 classes), where we utilize a convolutional NN. Additionally, we consider 3 larger image datasets with ResNet-18 architecture [33] : CIFAR-10 [44] (10 classes); and OrgMNIST / OrganSMNIST (11 classes) and OctMNIST (4 classes) from the MedMNIST collection [69, 70] . These prediction models are trained utilizing the standard logistic / log-loss without rejection and then are calibrated via temperature scaling [32] . For each of these datasets, we utilize both clean and noisy variants. For the noisy variant, we flip the class labels of the train set with a rate of 25%. We note that the test set is clean in both cases. All evaluation uses 5-fold cross validation. All implementation use PyTorch and training was done on a p3.2xlarge AWS instance.We consider 4 different baseline for comparison. Each is trained with 50 equidistant costs c, τ ∈ [0, 0.5), except on OctMNIST which uses 10 equidistant costs (selecting c, τ discussed in Appendix P). One baseline used corresponds to a modification of [49] 's cross-entropy surrogate approach (DEFER) originally used for the learning to defer literature (see [14, Appendix A.2] ). This approach treats the rejection option as a separate class and solves a |Y| + 1 classification problem. A generalization of DEFER is considered which utilizes generalized cross-entropy [75] as its surrogate loss (GCE) [13] .We also consider a cost-sensitive classification reduction (CSS) of the classification with rejection problem [14] utilizing the sigmoid loss function. The aforementioned 3 baselines all learn a model with rejection simultaneously, i.e., a pretrained model cannot be utilized. We also consider a two stage predictor-rejector approach (PredRej) which learns a rejector from a pretrained classifier [48] .Table 1 presents a tabular summary of accuracy and coverage values when targeting 80% coverage values; and Fig. 2 presents a summary plot of the acceptance coverage versus model accuracy after rejection, focused around the 60% to 100% coverage region. This plot is limited to HAR, Gas Drift, and MNIST due to space limitations however the deferred datasets show curves where the density ratio rejector dominate with better accuracy and coverage in the plotted region, with corresponding extended plots for all datasets in Appendix Q . Over all folds for MNIST our density ratio rejectors take approximately ≈ 1/2 hour to fit. A single baseline (fixed c) takes upwards of 2 hour for a single fold. Overall, given that the underlying model is calibrated, we find that our density ratio rejector are either competitive or superior to the baselines. One might notice that the aforementioned baselines do not or provide poor trade-offs for coverage values > 95% (as per Fig. 2 ). Indeed, to achieve rejection with high coverage (without architecture tuning), approaches which 'wrap' a base classifier seem preferable, i.e., PredRej and our density ratios rejectors. Even at lower coverage targets (80%), Table 1 shows that density-ratio methods are comparable or superior in the more complex datasets of CIFAR-10 and the MedMNIST collection. If large models are allowed to be used for the rejector -as per the MNIST case -CSS, DEFER, and GCE can provide superior accuracy vs acceptance coverage trade-offs (noisy MNIST). However, this is not always true as per CIFAR-10 where all approaches are similarly effected by noise; or OctMNIST and OrgMNIST where approaches only slightly change with noise. The latter appears to be a consequence of label noise not effecting the Base classifier's accuracy (using the larger ResNet-18 architecture), as per Table 1 .Among the approaches which 'wrap' the base classifier h, we find that these approaches have higher variance ranges than the other approaches. In particular, the randomness of the base model potentially magnifies the randomness after rejection. The variance range of the base model tends to increase as the noise increases (additional ranges of noise for HAR and Gas Drift are presented in the Appendix). The influence on rejection is unsurprising as these 'wrapping' approaches predict via a composition of the original model (and hence inherits its randomness across folds). In general, our density ratio rejector outperforms PredRej. However, it should be noted that PredRej does not require a calibrated classifier. Among the density ratio rejectors, between KL and α = 3, the only variation is in coverage region that the τ ∈ (0, 1] threshold covers. This follows from the fact that for similar distributions, φ-divergences act similarly [56, Theorem 7.20] . We find this pattern holds for other values of α. 
limitations and conclusions: 
 We propose a new framework for rejection by learning idealized density ratios. Our proposed rejection framework links typically explored classification with rejection to generalized variational inference and distributionally robust optimization. It should be noted that although we have focused on classification, L ′ (Q) could in theory be replaced by any other loss functions. In this sense, one could adapt this approach to other learning tasks such as regression, discussed in Appendix N. Furthermore, although we have focused on φ-divergences, there are many alternative ways idealized distribution can be constructed, e.g., integral probability metrics [51, 9] . One limitation of our distributional way of rejecting is the reliance on approximating P[Y | X] with model h. In future work, one may seek to approximate the density ratio ρ by explicitly learning densities Q and P or via gradient based methods (for the latter, see Appendix O). 
 
 Appendix: Proof. We first rewrite the coverage probability as an expectation:E P [(1 -r(X)) • ℓ(Y, h(X))] + c • Pr[r(X) = 1] = E P [(1 -r(X)) • ℓ(Y, h(X)) + c • r(X)]. We note that point-wise, the proper loss ℓ is minimized by taking the Bayes optimal classifier η ⋆ (x) = Pr[Y = +1 | X = x]. Thus taking the argmin over all possible CPE classifiers, h ⋆ = η ⋆ . We note that the point-wise risk taken by the Bayes optimal classifier is typical denoted as the Bayes point-wise risk L(x)= E P y|x=x [ℓ(Y, η ⋆ (x))] [60, 61].As such, we are left to minimize r over,E P [(1 -r(X)) • ℓ(Y, η ⋆ (X)) + c • r(X)] = E Px E P y|x=X [(1 -r(X)) • ℓ(Y, η ⋆ (X)) + c • r(X)] = E Px [(1 -r(X)) • L(X) + c • r(X)] = E Px [L(X)] + E Px [r(X) • (c -L(X))] .Thus, we immediately get the optimal r ⋆ (x) = L(x) ≥ c .Proof. We use the theory of Lagrange multipliers to make different constraints explicit optimization problems.Let us first consider the reduction from learing explicit distributions Eq. ( 8) to density ratios Eq. ( 9). First note that the objective Eq. ( 8) can be written as follows:L(Q) + λ • D φ (P ∥ Q) = L ′ (x)dQ(x) + λ • φ dQ dP dP(x) = L ′ (x) dQ dP (x)dP(x) + λ • φ dQ dP (x) dP(x) = E P L ′ (x) • dQ dP (x) + λ • φ dQ dP (x)The reduction to Eq. ( 9) now follows from a reduction from minimizing over Q to dQ/dP, noting that Q is restricted to be on the simplex. As such, the simplex constraints are transfered to:dQ dP ≥ 0 and dP(x) • dQ dP (x) = E P dQ dP (X) = 1,where the former is the non-negativity of simplex elements and the latter is the normalization requirement. Hence, taking ρ = ρ λ .= dQ dP completes the reduction. (we remove the subscript "λ" for the rest of the proof) As such we have the optimization problem in Eq. ( 9), where we will convert the constraints into Lagrange multipliers, defining,L(ρ; a, b) = E P [ρ(X) • L ′ (X) + λ • φ(ρ(X))] -a ′ (x)ρ(x)dx + b • (1 -E P [ρ(X)]) = E P [ρ(X) • L ′ (X) + λ • φ (ρ(X)) -a(X) • ρ(X) -b • ρ(X)] + b, where a(x) = a ′ (x) • P(x).We can obtain the first order optimality conditions by taking the functional derivative. Suppose that δ > 0 and h : X → R is any function. The functional derivative is given by,d dδ L(ρ + δ • h; a, b) δ=0 = E P [h(X) • (L ′ (X) + λ • φ ′ (ρ(X) + δ • h(X)) -a(X) -b)] δ=0 = E P [h(X) • (L ′ (X) + λ • φ ′ (ρ(X)) -a(X) -b)] .Thus, the first order condition is,0 = d dδ L(ρ + δ • h; a, b) δ=0 = E P [h(X) • (L ′ (X) + λ • φ ′ (ρ(X)) -a(X) -b)] ,for all h : X → R. As the condition must hold for all h for an optimal ρ ⋆ , we have that for all x ∈ X,0 = L ′ (x) + λ • φ ′ (ρ ⋆ (x)) -a(x) -b ⇐⇒ φ ′ (ρ ⋆ (x)) = b + a(x) -L ′ (x) λ ⇐⇒ ρ ⋆ (x) = (φ ′ ) -1 b -L ′ (x) + a(x) λ .As required.Proof. First notice that the inner maximization in the DRO optimization Eq. ( 7) can be simplified as follows:supQ∈Bε(P) L(Q) = - inf Q∈Bε(P) -L(Q) = - inf Q∈D(X) sup λ≥0 -L(Q λ ) -λ • (ε -D φ (P ∥ Q λ )) = -sup λ≥0 inf Q λ ∈D(X) -L(Q λ ) -λ • (ε -D φ (P ∥ Q λ )) ,where the last inequality follows from Fan's minimax Theorem [25, Theorem 2] noting that Q → L(Q λ ) is linear and the selected φ per Definition 2.2 makes Q → D(P ∥ Q λ ) a convex lower semi-continuous function.Now notice that the inner minimization of this simplification is exactly our idealized rejection distribution objective Eq. ( 8) when we negate the loss. As such, noticing that solutions to Eq. ( 8) are exactly given by P • ρ yields the result after optimizing for the 'arguments' in the above DRO objective for both λ and Q λ .We defer the proof of Corollary 4.1 to Appendix G, which covers any α including KL (α = 1).We breakdown Theorem 4.2 into two sub-theorems Theorem E.1 for each of the settings. Theorem E.1. Given the binary CPE setting presented in Theorem 2.1 and that we are given the optimal classifier h ⋆ (x) = P(Y = 1 | X = x), for any λ > 0 there exists a τ > 0 such that the r KL τ rejector generated the optimal density ratio in Corollary 4.1 is equivalent to the optimal rejector in Theorem 2.1.The proofs of each are similar. We first make the observation that the rejector function r KL τ can be simplified as follows:r KL τ = L ′ (x) ≤ -λ (log Z + log τ ) , Now we note that given a fixed λ > 0 (which also fixes Z), the RHS term has a one-to-one mapping from R + to R. Thus all that is to verify is that thresholding L ′ (x) is equivalent to the rejectors of Theorems 2.1 and N.1.Proof. For the CPE case, from assumptions, we have thatL ′ (x) = E Y∼P(Y|X=x) [ℓ(Y, h ⋆ (x))] = E Y∼h ⋆ (X=x) [ℓ(Y, h ⋆ (x)) as h ⋆ is optimal. Thus thresholding used in r KL τ is equivalent to thresholding L ′ (x) by keeping λ fixed and changine τ appropriately.To prove the theorem, we will be using the standard Hoeffding's inequality [12] .Theorem F.1 (Hoeffding's Inequality [12, Theorem 2.8] ). Let X 1 , . . . , X n be independent random variables such that X i takes values in [a i , b i ] almost surely for all i ≤ n. Defining X = i X i -EX i , then for every t > 0,Pr (X ≥ t) ≤ exp - 2t 2 n i=1 (b i -a i ) 2 .Proof. Let us denote Z to be the normalizer with the true expectation E P and Ẑ to be the normalizer with the empirical expectation E P.As ℓ is bounded, we take note of the following bounds,exp(-B/λ) ≤ max exp -L ′ (x) λ , Z, Ẑ ≤ exp(B/λ).This can be simply verified by taking the smallest and largest values of the exp.|ρ KL (x) -ρKL (x)| = exp -L ′ (x) λ • 1 Z - 1 Ẑ ≤ exp B λ • 1 Z - 1 Ẑ = exp B λ • |Z -Ẑ| |Z • Ẑ| ≤ exp B λ 3 • |Z -Ẑ|.Notice that |Z -Ẑ| can be bounded via concentration inequality on (bounded) random variable exp -L ′ (X) λ .Thus, we have that by Theorem F.1Pr E P exp -L ′ (x) λ -E P exp -L ′ (x) λ > t ≤ 2 exp -2 • N • t 2 (b -a) 2 ,where (b -a) 2 = (exp(B/λ) -exp(-B/λ)) 2 = 4 sinh 2 (B/λ).Pr ∃x ∈ T :E P exp -L ′ (x) λ -E P exp -L ′ (x) λ > t ≤ 2M exp -N • t 2 2 sinh 2 (B/λ).Thus taking t = sinh(B/λ) • 2 N • log 2Mδ , we have that with probability 1 -δ for δ > 0, for any x ∈ T,|ρ KL (x) -ρKL (x)| ≤ exp B λ 3 • |Z -Ẑ| ≤ exp B λ 3 • sinh(B/λ) • 2 N • log 2M δ .As required.Before proving the theorems, we first state some basic properties of ψ α . Lemma G.1.For α ̸ = -1, ψ α (u • v) = ψ α (u) • ψ α (v) and ψ α (1/v) = 1/ψ α (v).Furthermore, these statements hold when ψ α is replaced with ψ -1 α . Lemma G.2.EQUATION)The above statements follows directly from definition and simple calculation. The next statement directly connectsEQUATION)Proof. We prove this via cases.• α = -1:φ ′ -1 (z) = d dz (-log z) = -z -1 = -1 • z -1-α 2 = 2 α -1 • ψ α (1/z).• α ̸ = ±1:φ ′ α (z) = d dz 4 1 -α 2 • 1 -z 1+α 2 = - 2 1 -α z α-1 2 = - 2 1 -α • z -1-α 2 = 2 α -1 • ψ α (1/z)We now define the following constants depending on α:EQUATION)Lemma G.4. For α ̸ = 1,EQUATION)Proof. We prove this via cases.• α = -1:φ ′ -1 (z) = d dz (-log z) = -z -1 = -1 • z -1-α 2 = -1 • ψ α (1/z).Then,(φ ′ -1 ) -1 = 1 ψ -1 α (-1 • z) = ψ -1 α 1 -1 • z = ψ -1 α (-1) • 1 ψ -1 α (z) . = c α • 1 ψ -1 α (z).• α ̸ = ±1:φ ′ α (z) = d dz 4 1 -α 2 • 1 -z 1+α 2 = - 2 1 -α z α-1 2 = - 2 1 -α • z -1-α 2 = - 2 1 -α • ψ α (1/z)Then,EQUATIONProof.• α = 1:φ ′ 1 (z) = d dz (z log z) = log z + 1 Then, (φ ′ -1 ) -1 = exp(z -1) = exp(-1) • exp(z) = ψ -1 1 (-1) • ψ -1 1 (z) = c α • ψ -1 α (z)Thus now via Lemmas G.4 and G.5, we can prove the Theorems.Proof of Corollary 4.1 and Theorem 4.5. The proof follows from utilizing either Lemmas G.4 and G.5 in conjunction with Theorem 3.3.• α = 1: We have that,ρ φ λ (x) = (φ ′ ) -1 a(x) -L ′ (x) + b λ = c α • ψ -1 α a(x) -L ′ (x) + b λ = exp a(x) -L ′ (x) + b -1 λ .As ρ φ λ (x) for α = 1 is already positive by 'exp', by complementary slackness, the Lagrange multipliers a(•) = 0. Hence, we can further simplify the above,ρ φ λ (x) = exp -L ′ (x) + b -λ λ .The normalizer then can be easily calculated, renaming b ′ = b -λ, we simplify have thatρ φ λ (x) = exp -L ′ (x) + b ′ λ = 1 exp(-b ′ /λ) • exp -L ′ (x) λ ,which by normalization condition and setting Z .= exp(-b ′ /λ),1 = E 1 Z • exp -L ′ (x) λ ⇐⇒ Z = E exp -L ′ (x) λ ,which completes the case (Corollary 4.1).• α ̸ = 1: We have that,ρ φ λ (x) = (φ ′ ) -1 a(x) -L ′ (x) + b λ = c α • ψ -1 α a(x) -L ′ (x) + b λ -1 (a) = ψ -1 α 2 α -1 • ψ -1 α a(x) -L ′ (x) + b λ -1 = ψ -1 α 2 α -1 • ψ -1 α a(x) -L ′ (x) + b λ -1 = ψ -1 α 2 α -1 • a(x) -L ′ (x) + b λ -1, where (a) we exploit the that (ψ-1 α (z)) -1 = ψ -1 α (1/z) via Lemma G.1.Proof. First we simplify the density ratio rejector.EQUATION.We suppose that it is possible forα -1 2 • a(x) - L ′ (x) λ + b α-1 2 < 0for values of a(x), b, and λ. Otherwise, α(x) = 0 and we are done due to the above equation's non-negativity.Let x ∈ X be arbitrary. Suppose that a(x) = 0. Then by complementary slackness, we have thatα -1 2 • - L ′ (x) λ + b α-1 2 > 0 ⇐⇒ b - L ′ (x) λ > 0.By contra-positive, we have that b -L ′ (x)/λ ≤ 0 implies that a(x) ̸ = 0. By prime feasibility, in this case we also have ρ(x) = 0. We can solve either case by using the maximum as stated.Proof of Corollary 4.6. The proof directly follows from a property of the α-divergence when one of the measure have disjoint support. From [3] we have.Theorem I.1 ([3, Section 3.4.1 (4)]). For α-divergences, we have that 1. For α ≤ -1, D α (P ∥ Q) = ∞ when P(x) ̸ = 0 and Q(x) = 0 for some x ∈ X.This result immediately gives the result, as otherwise the objective function is ∞.Proof. We first note simplifying ρ α=3 λ via Corollary 4.6 yields:ρ α=3 λ (x) = max 0, b - L ′ (x) λ .Thus our goal is to solve b to giveE P [ρ α=3 λ (X)] = 1.Now, consider the following,1 + E P [L ′ (X)] λ - L ′ (x) λ > 0 ⇐⇒ λ > L ′ (x) -E P [L ′ (X)] ,where the latter holds uniformly for all x from assumptions on λ.Thus setting b = 1 + EP[L ′ (X)] λ, we simplifymax 0, 1 + E P [L ′ (X)] λ - L ′ (X) λ dP(x) = 1 + E P [L ′ (X)] λ - L ′ (X) λ dP(x) = 1. As such, b = 1 + EP[L ′ (X)] λsolves the required normalization. Substituting b back into ρ α=3 λ (x) yields the Theorem. K Proof of Theorem 4.9Proof. First we note that for any meas R, λ > 2B implies that λ > L ′ (x) -E R [L ′ (X)] (taking largest and smallest values of ℓ.As such, for both P, P have closed forms Corollary 4.8 Thus, the bound can be simply shown to have,|ρ α=3 λ (x) -ρα=3 λ (x)| = 1 λ • E P [L ′ (X)] -E P [L ′ (X)] .Thus by Hoeffding's inequality Theorem F.1 and union bound (see for instance the proof of Theo-rem 4.3), setting t = B • 2 N log 2Mδ , we have that with probability 1 -δ for all x ∈ T,|ρ α=3 λ (x) -ρα=3 λ (x)| ≤ B λ • 2 N log 2M δ .As required.The paper presents work which reinterprets the classification with rejection problem in terms of learning distributions and density ratios. Beyond advancing Machine Learning in general, potential societal consequences include enhancing the understanding of the rejection paradigm and, consequentially, the human-in-the-loop paradigm. The general rejection setting aims to prevent models from making prediction when they are not confident, which can have societal significance when deployed in high stakes real life scenarios -allowing for human intervention.In the following, we seek to provide generalization bounds on Qλ,N .= PN • ρλ,N . That is, one seeks to know that as N → ∞ how and if Qλ,N → Q. A natural measure of distance for probability measures is total variation [20] ,TV(P, Q) . = 1 2 • ∥P -Q∥ 1 = |Q(x) -P(x)| dx.Let us also define∥P -Q∥ ∞ = sup x∈X |Q(x) -P(x)|.One immediately gets a rate if we assume that |X| is finite and we have a bound for ρλ,N . Theorem M.1. Suppose that |X| = M < ∞ is finite and bounded density ratio|ρ λ,N | ≤ B ′ for B ′ > 0.Then, if we have that with probability 1 -δ that ,∥ρ(x) -ρN,λ (x)∥ ∞ ≤ C • 1 N • log 2M δ + C ′ ,for some C, C ′ > 0, then we have thatEQUATIONProof. Noting that the empirical distribution is a sum of Dirac deltasPN (x) = i∈[N ] δ(x -x i ),we can establish a simple bound via a consequence of Hoeffding's Theorem Theorem F.1 (also noting that P = E PN ). We have that,Pr(|P(x) -PN (x)| ≥ t) ≤ 2 exp -2N t 2 .Thus,Pr(∀x ∈ X : |P N (x) -PN (x)| ≥ t) ≤ 2M exp -2N t 2 . Setting t = 1 2N log 2M δ , we have with probability 1 -δ ∥P -PN ∥ ∞ ≤ 1 2N log 2M δ .We now consider the following:∥Q -Qλ,N ∥ ∞ = ∥P • ρ -PN • ρλ,N ∥ ∞ = ∥P • (ρ -ρλ,N ) + (P -PN ) • ρλ,N ∥ ∞ ≤ ∥P • (ρ -ρλ,N )∥ ∞ + ∥(P -PN ) • ρλ,N ∥ ∞ ≤ ∥(ρ -ρλ,N )∥ ∞ + B ′ • ∥(P -PN )∥ ∞ .Taking a union bound of the above inequality and our assumption, we have that, for some C > 0, with probability 1 -δ,∥Q -Qλ,N ∥ ∞ ≤ ∥(ρ -ρλ,N )∥ ∞ + B ′ • ∥(P -PN )∥ ∞ ≤ C • 1 N log 4M δ + C ′ + B ′ • 1 2N log 4M δ = O log M N .Converting the bound to TV amounts to simply summing over X, which gives TV(Q, Qλ,N ) = O( (M 2 log M )/N ), as required.Notably, with appropriate assumptions, Theorems 4.3 and 4.9 can be used with Theorem M.1 to get bounds for QKL λ,N and Qα=3 λ,N .In the main-text of the paper we have focused on classification. However, many of the idea discussed can be extended for the regression setting. For instance, similar to Chow's Rule in Theorem 2.1 we can express the regression equivalent to the optimal solution of Eq. ( 2). Theorem N.1 (Optimal Regression Rejection [72] ). Let us consider the regression setting, where Y = Y ′ = R and ℓ(y, y ′ ) = 1 2 (y -y ′ ) 2 . Then w.r.t. Eq. ( 2), the optimal model is given by h⋆ (x) = E P [Y | X = x]and the optimal rejector is given by r⋆ (x) = σ 2 (x) ≤ c , where σ 2 (x) is the conditional variance of Y given X = x.For regression, there is no clear analogous notion of output "confidence score" unless the model explicitly outputs probabilities. Indeed, rejection method for regression explicitly requires the estimation of the target variable's conditional variance [72] .Similar to the CPE case, our KL density ratio rejector can provide a rejection policy equivalent to the typical case. Theorem N.2. Given the regression setting presented in Theorem N.1 and that we are given optimal regressor h ⋆ (x) = E P [Y | X = x], for any λ > 0 there exists a τ > 0 such that the r KL τ rejector generated the optimal density ratio in Corollary 4.1 is equivalent to the optimal rejector in Theorem N.1.Proof. The proof follows similarly to the proof of Theorem E.1. The regression case follows almost identically, by noticing that L ′ (x) = σ 2 (x) is the variance. This is similar to the proof of Theorem N.1 [72] .Despite the equivalence, there is a difficult in using the density ratio rejectors, as per the closed form equations of Section 4, for regression. Estimating L ′ (x) = σ 2 (x) is challenging. Unlike classification where learning calibrated classifiers has a variety of approaches, learning a regression model which explicitly outputs probabilities is quite difficult. As such, approximating P(Y | X = x) with the model h(x) cannot be done.As an alternative to the closed-form rejectors explored in the main-text, one may want to explore a method to learn ρ iteratively. We consider the gradients of the optimization problem in Eq. ( 9). In practice, we found that we were unable to learn such rejectors via taking gradient updates and thus leave a concrete implementation of the idea for future work.The idea comes from utilizing the "variational" aspect of the GVI formulation (which was not explored in the main-text). We suppose that the rejectors we are interested in come from a parameterized family. In particular, we consider the self-normalizing family ρ ϑ /Z ϑ , where Z ϑ = E P [π ϑ (X)] normalizes the rejector such that E P [ρ ϑ (X)] = 1. Having the Z ϑ normalizing term means that the constraint in Eq. ( 9) is satisfied for any ϑ. The only constraint that we must have for π ϑ is non-negativity, i.e., π ϑ is a neural network with exponential last activation functions from X → R + . By setting a parametric form of ρ ϑ , we implicitly restrict the set of idealized distributions to Q ϑ = P • ρ ϑ . The gradients of such a parametric form can be calculated as follows.Corollary O.1. Let ρ ϑ = π ϑ (x)/Z ϑ . Then the gradient of Eq. (9) w.r.t. ϑ is given by,EQUATIONProof. The proof is immediate from differentiation of Eq. ( 9).An alternative form of Eq. ( 24) can be found by noticing that ∇f = f • ∇ log f . This provides an expression in terms of the log density ratio.E Px,y π ϑ (X) Z ϑ • ∇ ϑ log π ϑ (X) Z ϑ • φ ′ π ϑ (X) Z ϑ + λ • ℓ(Y, h(X)) .One will notice that the gradient is in-fact the log-likelihood of the idealized distribution: noting P free of ϑ, we have ∇ ϑ log(π ϑ (X)/Z ϑ ) = ∇ ϑ log(P(X) • π ϑ (X)/Z ϑ ) = ∇ ϑ log Q ϑ . As such, the gradient Eq. ( 24) is equivalent to the gradient of a weighted log-likelihood.Despite the potential nice form of the gradient, we found that learning rejector through this was not possible. One limiting factor of computing such gradients is that we need to estimate Z ϑ at each gradient calculation, i.e., this must happen whenever ϑ changes. This can be particularly costly when X is high-dimensional. Secondly, we suspect that the model capacity of π ϑ was not sufficient: we only tested on simple neural networks and convolutions neural networks, mirroring the architecture of the base classifiers used in our experimental setting.Both the baselines and density ratio approaches evaluated in Section 5 require a tuning of hyperparameters c, τ . Practically, we are more interested in the rejection rate P[r(X) = 1] rather than the abstract choices of c, τ . Hence, one often wishes to select c, τ according to a coverage constraint.One of the simplest choices of selecting c, τ is to utilize a calibration set and approximate the rejection rate / coverage on this data. Our experiments in Section 5 mirrors this in Tables 1 and I by treating the final test set as the calibration data.Other works in the literature utilize more sophisticated methods for deriving coverage constraints. [29] picks threshold values which provide a guarantee on a coverage-modified risk requirement.Another approach [57] does not require an additional calibration set and instead fits threshold values τ by exploiting stacking confidences and (stratified) cross-fitting. Although one could consider such approaches for all baselines, including those which utilize the cost-model of rejection via c, one will have to pay a price in retraining the learned rejector r for each instance of the hyperparameter c that is searched. As a result, approaches which can trade-off accuracy and coverage via a threshold variable τ (rather than an expensive retraining) are computationally preferable when exhaustively searching for a good c, τ .We leave more sophisticated methods for selecting τ for density ratio rejectors for future work.The neural network architecture used to train the base classifiers and baselines are almost identical.For the baseline approaches which have output dimension which is different than the output of the original neural network, we modify the last linear layer of the base classifier's architecture to fit the baseline's requirements, e.g., adding an additional output dimension for rejection in DEFER.Our architectures utilize batch normalization [38] and dropout [65] in a variety of places. Training settings are mostly identical, with some baselines requiring slight changes.The base model's architecture is as follows.• For CSS, as noted in [53] , batch normalization is needed at the final layer to stabilize training.All training utilizes the Adam [41] optimizer.All datasets we consider are in the public domain, e.g., UCI [6] . Plots Figs. I to III show Fig. 2 over and extend region of acceptable coverage percentages. In addition, we include a larger range of noise rates for HAR and Gas Drift. For MNIST, we explore a larger range of noise rates for MNIST in Fig. VII for our density ratio rejectors. We find that the findings in the main text are extended to these additional noise rates. In particular, we find that the our density ratio rejectors can be competitive with the various baseline approaches. We find that our density ratio approaches can have more fine-grained trade-offs at higher ranges of acceptance coverage. This is an important region where the budge for rejection may be low (and only a few examples, e.g. < 10%, can be rejected). Indeed, the baseline approaches which do not 'wrap' a base model require a lower maximum acceptance coverage as the noise rate increases (the approaches require a higher rejection % for any type of rejection). Nevertheless, we do see a downside of the density ratio approach: the quality of the density ratio rejectors is dependent on the initial model. As such, at higher levels of noise there can be higher variation in the quality of rejection, see Fig. II . Interestingly, for MNIST and CIFAR-10, Figs. III and IV, the base model the density ratio rejectors are more robust across noise rates than other models. This seems to be due to the default MNIST architecture being robust against higher noise rates (notice that the s.t.d. range is also quite small at 100% coverage). In OctMNIST and OrganMNIST, Figs. V and VI, we find that there is little to no change in performance, likely due to the Base classifier's own performance only changing slightly with the addition of noise. In the following, we consider the Gas Drift dataset when models are switched to a base model with only a single hidden layer. First we make note of the original setting explored in the main text. In Table II , we take note of the number of tunable parameters in all approaches and baselines. Notably, these default parameter / architecture sizes are similar to [14] , with the HAR and Gas Drift setting including an additional hidden layer than previously utilized in the literature.In Table III , we note the setting we consider in this subsection. The parameter sizes of the Gas Drift dataset is reduced to the originally explored model sizes in [14] . Notice that both the base models and the baseline approaches have reduced parameter sizes. It should be noted that this smaller parameter OrganMNIST 11, 173, 324 11, 173, 325 11, 174, 347 11, 173, 836 11, 174, 860 22, 342, 541 size setting can be useful in the related learning with deferral setting [52] , where having a small model to defer to a larger model is needed.The results are reported in Figs. VIII and IX. We can see that in this setting, PredRej and our density ratio approaches are more competitive. This might indicate that for simple base models, approaches which 'wrap' a base model for rejection can be quite effective (especially in higher coverage regimes). In general, it seems with this smaller architecture regime, the 'non-wrapping' baseline approaches only provide rejection options when the acceptance coverage is lower than 70%. The following shows parameter sweeps over α and λ for our density ratio rejectors. These are given by Figs. X and XI respectively. We find that increasing α compresses the trade-off curve from both sides. While decrease λ extends the trade-off curve on the left side. Properness ensures that the true class probability is the minimizer of the loss in expectation.Our rejector's code public at: https://github.com/alexandersoen/density-ratio-rejection. 


Decision: In this work, the authors develop a new algorithm for classification with rejection by learning an idealized distribution using the model, and rejecting a classification if the ratio of the idealized density to the data density is too large. Efficacy is demonstated using both theoretical error bounds and a short series of experiments. Reviews were generally positive, but criticised the limited number of experiments and acknowledged that both density estimation and the ratio estimator are likely to succumb to the curse of dimensionality for larger problems. In their rebuttal, the authors provided a series of additional experiments highlighting potential efficacy for problems of larger size. After the discussion phase, all reviewers agreed the paper was of high quality with the additional results. Subject to these inclusions, I recommend acceptance.

Reviews: review: paper_summary: paper_summary: This paper propose a new classification algorithm with abstention by learning a density ratio between an "idealized" distribution to the data distribution: if the density ratio is small, the classifier rejects to predict.
The proposed learning framework is general, and is a function of the choice of $f$-divergence. 
The paper studies two concrete choices of divergences: KL- and $\a$-divergences, which admit a closed form rejector.
The paper provides some experimental supports for the proposed ideas with neural-net classifiers.
strength_weakness: strength_weakness: While the proposed method sounds reasonable, I have some concerns/questions in the presentation.
Please see questions and suggestions below.
questions: questions: **Questions**
- What is the meaning of "The maximization is adversarial over the marginal label distribution" in line 128?
- I had hard time to understanding the meaning of the "idealized rejection distribution" (as called. in line 157), especially because the following sentence was extremely confusing: 
```
Given Definition 3.1 for idealized distribution, small values of $\rho(x)$ corresponds to regions of the input space $\mathcal{X}$ where having lower data probability would decrease the expected risk of the model. 
```
I can't follow this sentence. Can you please elaborate this?
- Further, I can't understand why does it make sense not to reject where $P(x)$ is relatively small? If the likelihood of occurrence is small, shouldn't we also avoid prediction at that point as the classification will be likely wrong? Or is the argument that we do not need to care the small probability mass in the first place as it won't incur a significant loss in expectation? 
- In the experiments, the proposed algorithms seem to perform well in the high-coverage regime, while dominated by existing methods in the low-coverage regime. Do you have any explanation on this?

**Suggestions**
- Compared to the quality of math, there are way too many typos and grammatical errors. To spot a very few:
  - line 23: "While" some approaches avoid...
  - line 25: ... approach aim**s**...
  - line 74: ... an addition**al**...
There are too many of these throughout, so please revise them carefully.
- Please consider moving lines 84-85 after line 76.
- Please put subsections in Section 4 for readability.
limitations: limitations: Limitations of the framework are noted by the authors.
paper_summary: paper_summary: This paper proposes a novel method for classification with rejection based on density ratio estimation. Density ratio is estimated between the data distribution P and the "idealized" distribution Q, where Q is a distribution such that the model can have good prediction performance, while this distribution is still similar to the original data distribution P. The objective function is proposed to formulated as a risk minimization with a $\phi$-divergence regularization. Theoretical analysis is also provided to justify the soundness of the proposed method. The experiment shows that the proposed method is effective compared with well-known existing approaches.
strength_weakness: strength_weakness: 1. Unfortunately, I found the experiment section writing and result is not quite well-written compared with other sections. Perhaps there is a better way to analyse and understand practical behavior of the proposed method. 
 - The figure shows in the paper does not really show the superiority of the method except in HAR (clean) and Gas Drift (clean). It is also quite hard to see since the starting point of each method is different. Perhaps table representation or different way to show the result might help.
  - Only three simple datasets were used in this paper and the discussion using these datasets still look difficult to conclude something as several methods are competitive.
  - In my understanding, KL and $\alpha$ divergence give almost exactly the same performance in all cases. Is this really the case?
  The study of hyperparameter sensitivity can also be beneficial to practitioners
  - Analysis of what kind of dataset characteristic makes the proposed distributional method more effective than the traditional method would be very useful. I feel existing methods such as CSS or the classifier-rejector approach might feel more like a direct approach to solving classification with rejection. Nevertheless, it is great to explore other approaches.
  -  As a result, I believe revising the experiment section can significantly improve the paper.
2. I found the writing and organization of this paper can be significantly improved. It takes until page 4 to state the proposed method, leaving only a small space for the experiment section. Perhaps some discussion of theory for $\alpha-divergence and background of (DVI, DRO) can be wrapped up more compactly and defer to the appendix to make the paper more self-contained and provide sufficient discussion on experiments. 
  - For reading experience, it might also be useful to highlight the final objective function (or use a latex Algorithm environment) to outline what exactly we need to do to use the proposed method so that practitioners can quickly understand how the method works. I found section 4 is quite difficult to follow. In this form, we have to go through the whole paper and mix all the ingredients to get an idea how to implement the proposed method (e.g., Normalization, how to get a(x), inverse function of $\varphi$.
questions: questions: 1. Is there any superiority of $\alpha=3$ divergence over KL-divergence? I might have missed it.
2. Is it written in the main body of the paper what $\varphi'$ is (Eq. 11)? I believe it is a derivative of $\varphi$ over X? I am sorry if I missed it. If it is not written it should be so because for example, "'" can mean different things like how L' is used in this paper.
3. It seems the proposed method is quite weak under noisy data. Is there any possible explanation why this is the case?
4. The Theorem 4.2 (informal)  states that there exists a threshold to achieve Chow's rule if original $h$ is correct looks quite restrictive to me in the sense that if h is really optimal, then we can just use h to achieve Chow's rule and there is no need to consider density ratio estimation. Can we say anything when h is not optimal? 


Minor comment: 
I think it is more common to call "classification-rejection" approach, a classifier-rejector approach (like Ni et al., 2019 [44]), or predictor-rejector approach (Mao et al., 2024) [39].
limitations: limitations: Limitations about the need to estimate P(Y|X) with model h were discussed, suggesting that this approach requires estimating class-posterior probability. 

(Minor) broader impacts were discussed in Appendix L and perhaps it is better to put them in the main body if possible.

Overall, I found the discussion about limitations is appropriate.
paper_summary: paper_summary: Classification with rejection emerges as a learning paradigm that allows models to abstain from making predictions. Traditional rejection learning methods typically modify the loss function, enabling models to explicitly reject making predictions when they are inaccurate or uncertain. These methods rely on providing good class probability estimates and often require training models from scratch. This paper proposes an interesting perspective to reconsider rejection not from the standpoint of the loss function, but rather from the perspective of distributions. By optimizing the risk of the loss with a $\varphi$-divergence regularization term, it seeks an idealized data distribution that maximizes the performance of pretrained models.
strength_weakness: strength_weakness: 1. For high-dimensional data, estimating density ratios and computing idealized distributions can be very challenging, implying that this method may only address low-dimensional data and its application scenarios are limited.
2. This method involves a large number of hyperparameters: $\lambda$, $\tau$, $\alpha$, $T$, which pose a significant burden for optimization.
3. The experiments in the paper may only be conducted on specific datasets and under certain conditions (some simple datasets), which could limit their generalizability to broader scenarios.
questions: questions: 1. Although theoretically estimating the distribution $P(y|x)$ holds more promise than estimating a single probability $\max P(y|x)$, in practical situations, estimating the distribution seems less feasible compared to single probability estimates, especially given the various calibration methods available for single probabilities.
2. How did you design the expert in the experiments where you compared methods [1] from Learning to Defer？
3. Since comparing with [1], it seems more reasonable to compare with [2], which is more relevant to the focus of this study.

[1] Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. ICML 2020.

[2]  Cao Y, Cai T, Feng L, et al. Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. NeurIPS, 2022.
limitations: limitations: N/A.
paper_summary: paper_summary: The paper proposes a distributional perspective to build abstaining classifiers. By considering an idealized distribution, the authors show that this translates into optimizing a loss risk with a $\varphi$-divergence regularization term. Moreover, they provide results when considering $\alpha$-divergences, a specific family of $\varphi$-divergences. Empirical evaluation is performed over clean and noisy datasets.
strength_weakness: strength_weakness: The main weaknesses of the paper are:

i) there is no related work section explicitly dedicated to learning with a reject option. See, e.g., [a] for a recent survey.


ii) the empirical evaluation can be improved:
* for instance, the paper considers only four baselines (mainly from Learning to Defer literature) and ignores popular alternatives such as softmax response [26]
* the method is tested only on three datasets, which seem easy to solve. In particular, MNIST might be too easy to evaluate abstaining classifiers (an accuracy of 100% is easy to achieve with relatively few rejections). A more interesting dataset collection could be MedMNIST [b], which contains medical images from real data tasks.



[a] Hendrickx, Kilian, Lorenzo Perini, Dries Van der Plas, Wannes Meert, and Jesse Davis. "Machine learning with a reject option: A survey." Machine Learning 113, no. 5 (2024): 3073-3110.\
[b] Yang, Jiancheng, Rui Shi, and Bingbing Ni. "Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis." In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 191-195. IEEE, 2021.
questions: questions: I have a few questions for the authors:

1) is there a reason why you did not compare with the simplest (and often the best) abstaining classifier, i.e. the softmax response? In [c], the authors show that the classifier built using this strategy can be optimal. Moreover, recent empirical works [d,e] show that score-based approaches outperform other abstaining classifiers.

2) how did you train DEFER, without access to human predictions? As far as I know, the surrogate loss also requires specific human predictions; however, I could not find in the Appendix how this is done in practice.

[c] Franc, Vojtech, Daniel Prusa, and Vaclav Voracek. "Optimal strategies for reject option classifiers." Journal of Machine Learning Research 24, no. 11 (2023): 1-49.\
[d] Jaeger, Paul F., Carsten Tim Lüth, Lukas Klein, and Till J. Bungert. "A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification." In The Eleventh International Conference on Learning Representations. (2023)\
[e] Pugnana, Andrea, Lorenzo Perini, Jesse Davis, and Salvatore Ruggieri. "Deep neural network benchmarks for selective classification." arXiv preprint arXiv:2401.12708 (2024).
limitations: limitations: The authors discuss the limitations of their approach.

Comments: title: Thank you for your feedback
Response
Response to the Rebuttal
comment: Thank you to the authors for providing the additional experiments. I believe that after including these extra experiments, this paper is quite good, and I would be happy to raise my score to 5.
We thank the reviewer for their support of the paper and for providing some additional context on the empirical weakness of the other baselines. We briefly considered cross-fitting to find thresholds values similar to [c], however did not go that route due to increases in computational costs for larger datasets. Nevertheless, we will include this discussion in to next revision and also add citations for previous work which have successful utilized this approach.
We thank the reviewer for raising their score. We are also interested in seeing new algorithms which exploit our proposed perspective of rejection. On the topic of high-dimensional data, the reviewer may be interested in the CIFAR-10 setting proposed by Reviewer MrTu. We have presented experimental results mirroring those provided for MedMNIST / OrganSMNIST and have reached similar conclusions to those expressed in the original global discussion.
We thank the reviewer for their response. We will ensure we incorporate the provided feedback and response into the next revision.
We thank the reviewer for their additional comment. We are happy to include results on CIFAR10. In the following, we have mirrored the setting of the additional MedMNIST / OrganSMNIST experiments presented in our global response.  
Of these settings, we note that this uses a Resnet18 base model for the classifier and baselines; and that the specific hyperparameter grid for $c$ is reduced to ensure that the experiments completed in time for the discussion deadline. Random cropping and horizontal flipping data augmentation are utilized in training for all approaches.

We present an additional 85% coverage table as the coarseness of the $c$ grid provided better comparison for some of the baselines.

### CIFAR-10 (80% Coverage)

|    | Model Name            | Hyperparameter   | Accuracy       | Coverage        |
|---:|:----------------------|:-----------------|:---------------|:----------------|
|  0 | Base Clf              | -                | 90.095 (0.244) | 100.000 (0.000) |
|  1 | $\rm KL-Rej$          | 0.74             | 97.285 (0.222) | 80.165 (0.428)  |
|  2 | $(\alpha=3)\rm{-Rej}$ | 0.74             | 97.236 (0.212) | 80.433 (0.350)  |
|  3 | $\rm PredRej$         | 0.1              | 90.271 (0.415) | 94.792 (6.079)  |
|  4 | $\rm CSS$             | 0.15             | 95.610 (0.274) | 84.077 (0.616)  |
|  5 | $\rm DEFER$           | 0.2              | 93.639 (0.964) | 82.530 (3.919)  |
|  6 | $\rm GCE$             | 0.25             | 93.807 (0.738) | 82.710 (1.115)  |


### CIFAR-10 (85% Coverage)

|    | Model Name            | Hyperparameter   | Accuracy       | Coverage        |
|---:|:----------------------|:-----------------|:---------------|:----------------|
|  0 | Base Clf              | -                | 90.095 (0.244) | 100.000 (0.000) |
|  1 | $\rm KL-Rej$          | 0.62             | 95.966 (0.283) | 85.313 (0.490)  |
|  2 | $(\alpha=3)\rm{-Rej}$ | 0.58             | 96.059 (0.305) | 85.077 (0.469)  |
|  3 | $\rm PredRej$         | 0.1              | 90.271 (0.415) | 94.792 (6.079)  |
|  4 | $\rm CSS$             | 0.2              | 94.339 (0.158) | 87.783 (0.251)  |
|  5 | $\rm DEFER$           | 0.25             | 93.423 (0.698) | 87.837 (1.206)  |
|  6 | $\rm GCE$             | 0.3              | 93.295 (0.781) | 85.720 (0.682)  |


### CIFAR-10 (90% Coverage)

|	| Model Name        	| Hyperparameter   | Accuracy   	| Coverage    	|
|---:|:----------------------|:-----------------|:---------------|:----------------|
|  0 | Base Clf          	| -            	| 90.095 (0.244) | 100.000 (0.000) |
|  1 | $\rm KL-Rej$      	| 0.52         	| 94.314 (0.240) | 90.070 (0.445)  |
|  2 | $(\alpha=3)\rm{-Rej}$ | 0.38         	| 94.179 (0.264) | 90.420 (0.406)  |
|  3 | $\rm PredRej$     	| 0.1          	| 90.271 (0.415) | 94.792 (6.079)  |
|  4 | $\rm CSS$         	| 0.25         	| 93.134 (0.641) | 90.025 (0.705)  |
|  5 | $\rm DEFER$       	| 0.3          	| 93.656 (0.393) | 88.352 (2.164)  |
|  6 | $\rm GCE$         	| 0.3          	| 93.295 (0.781) | 85.720 (0.682)  |

### Discussion

We make a few comments on the reported results.

- For this higher dimensional dataset, the same patterns emerge when comparing to MedMNIST and the paper’s original datasets in the noiseless case. Our approach can perform better than the baselines.  
- An additional observation on the best $c$ taken for the baseline approaches: the optimal cost of rejection per coverage target depends greatly on the approach with no consistency across all baselines. This demonstrates that tuning of $c$ can be difficult in practice (especially as each $c$ requires an entire model to be trained).  
- The increase in dimension from OrganSMNIST/MNIST (1x28x28) to CIFAR-10 (3x32x32) does not seem significant enough to cause any practical issues.

Additional to these direct comments, we would also like to make some other notes. Although the general framework we propose requires the learning of density ratios to determine the rejector, the closed-form solutions and practical trick (Line 271\) that we utilize does not require explicit learning of a density ratio. Instead, transforming a base model’s calibrated output is exploited (with only the normalization constant $b$ or $Z$ requiring approximation). As such, issues of high dimensional data for the current implementation would mainly be linked to difficulties in calibration in high dimension (which is a valid concern).

Nevertheless, we hope these additional results address the reviewer’s concerns for at least this CV dataset. We will of course make a note of this limitation and the corresponding discussion in the next revision. This would be particularly useful in future work where different divergences or modifications do require an explicit distribution or density-ratio to be learned.
I appreciate the authors' rebuttal.
Please carefully incorporate the answers in the revision.
I believe that it is important to elaborate the reasoning behind using the idealized distribution.
I will keep the score as is.
Thank you for the author's rebuttal. I also have read other reviews.

Given the fact that the organization will be modified as mentioned in the rebuttal and additional experiments provided, I would like to raise the score to 6 (Weak accept). I am aware of the concern of the high-dimensionality problem suggested by other reviewers. I believe the approach taken by this paper is innovative and it is also interesting to design other algorithms that follow this principle of identifying idealized distribution to use it as a criterion for rejection, which could be preferable to the current method in the high dimensionality regime.
Thank you for your elaborated response to my review.

I understand that the author's work is interesting and meaningful, but I still maintain my concerns about its validity on high-dimensional datasets (despite the addition of experiments using MedMNIST). As we are now gradually entering the era of large models, could you provide some more experiments on high-dimensional data, such as CIFAR10, regardless of the results? I hope this will help to better understand the limitations of the method.
I thank the authors for their rebuttal, and I hope they found the references to the MedMNIST collection helpful.
I would like to add a small comment on the following:

> The component of SR that we are missing is the calculation of the threshold according to a desired coverage guarantee. This would be a nice direction for future work to automatically prescribe $\tau$.

Some approaches, as noted by the authors in the response to Reviewer MrTu, add a coverage constraint to the loss that is minimized. However, empirical evidence does not support using these models vs. SoftMax Response (see again e.g. [a,b]).
In any case, given a target coverage $c$, both approaches require a calibration set and estimate the threshold $\tau$, considering the $(1-c)$ quantile of the confidence on the calibration set. 
Another option is resorting to cross-fitting to estimate the $\tau$ without the extra calibration set but exploiting test folds and stacking confidences (see, e.g., [c]).

To conclude, I think this paper should be considered for acceptance.

[a] - Feng, Leo, Mohamed Osama Ahmed, Hossein Hajimirsadeghi, and Amir H. Abdi. "Towards Better Selective Classification." In The Eleventh International Conference on Learning Representations.
[b] - Pugnana, Andrea, Lorenzo Perini, Jesse Davis, and Salvatore Ruggieri. "Deep neural network benchmarks for selective classification." arXiv preprint arXiv:2401.12708 (2024).
[c] - Pugnana, Andrea, and Salvatore Ruggieri. "A model-agnostic heuristics for selective classification." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 8, pp. 9461-9469. 2023.