Title: Realizable $H$-Consistent and Bayes-Consistent Loss Functions for Learning to Defer
Venue: NeurIPS 2024
Accepted: yes
Keywords: learning to defer

consistency

realizable H-consistency

learning theory
Abstract: We present a comprehensive study of surrogate loss functions for learning to defer. We introduce a broad family of surrogate losses, parameterized by a non-increasing function $\Psi$, and establish their realizable $H$-consistency under mild conditions. For cost functions based on classification error, we further show that these losses admit $H$-consistency bounds when the hypothesis set is symmetric and complete, a property satisfied by common neural network and linear function hypothesis sets. Our results also resolve an open question raised in previous work [Mozannar et al., 2023] by proving the realizable $H$-consistency and Bayes-consistency of a specific surrogate loss. Furthermore, we identify choices of $\Psi$ that lead to $H$-consistent surrogate losses for *any general cost function*, thus achieving Bayes-consistency, realizable $H$-consistency, and $H$-consistency bounds *simultaneously*. We also investigate the relationship between $H$-consistency bounds and realizable $H$-consistency in learning to defer, highlighting key differences from standard classification. Finally, we empirically evaluate our proposed surrogate losses and compare them with existing baselines.

Realizable H -Consistent and Bayes-Consistent Loss Functions for Learning to Defer

Abstract

We present a comprehensive study of surrogate loss functions for learning to defer. We introduce a broad family of surrogate losses, parameterized by a non-increasing function Ψ , and establish their realizable H -consistency under mild conditions. For cost functions based on classification error, we further show that these loss functions admit H -consistency bounds when the hypothesis set is symmetric and complete, a property satisfied by common neural network and linear function hypothesis sets. Our results also resolve an open question raised in previous work [Mozannar et al., 2023] by proving the realizable H -consistency and Bayes-consistency of a specific surrogate loss. Furthermore, we identify choices of Ψ that lead to H -consistent surrogate losses for any general cost function , thus achieving Bayesconsistency, realizable H -consistency, and H -consistency bounds simultaneously . We also investigate the relationship between H -consistency bounds and realizable H -consistency in learning to defer, highlighting key differences from standard classification. Finally, we empirically evaluate our proposed surrogate losses and compare them with existing baselines.

1 Introduction

In many practical scenarios, combining expert insights with established models can yield significant enhancements. These experts can be human domain specialists or more complex, albeit resourceintensive, models. For example, modern language and dialogue models are prone to producing hallucinations or inaccurate information. The quality of their responses can be significantly enhanced by delegating uncertain predictions to more specialized or advanced pre-trained models. This problem is particularly crucial for large language models (LLMs), as noted in [Wei et al., 2022, Bubeck et al., 2023]. The same principle applies to other generative systems, like those for images or videos, and to learning models in diverse applications such as image classification, annotation, and speech recognition. Thus, the task of learning to defer (L2D) with experts has become increasingly critical across a wide array of applications.

Directly optimizing the deferral loss function, which is the target loss in L2D, is computationally intractable for many choices of the hypothesis set. Therefore, a common approach is to optimize a surrogate loss that facilitates the optimization of the deferral loss function. Recent work in L2D has proposed several surrogate losses [Mozannar and Sontag, 2020, Verma and Nalisnick, 2022, Mozannar et al., 2023, Mao et al., 2024a] and studied their consistency guarantees, including Bayes-consistency, realizable H -consistency, and H -consistency bounds (see definitions in Section 3.2). In particular, Mozannar and Sontag [2020] proposed the first Bayes-consistent surrogate loss by generalizing the cross-entropy loss for L2D. Verma and Nalisnick [2022] proposed an alternative Bayes-consistent surrogate loss by generalizing the one-versus-all loss for L2D. Mozannar et al. [2023] showed that these surrogate losses are not realizable H -consistent. They proposed an alternative surrogate loss

that is realizable H -consistent, but they were unable to prove or disprove whether the proposed surrogate loss is Bayes-consistent. All the surrogate losses mentioned above and their consistency guarantees hold only for cost functions based on classification error. Mao et al. [2024a] generalized the surrogate loss in [Mozannar and Sontag, 2020] to incorporate general cost functions and any multi-class surrogate losses. They provided H -consistency bounds for the novel family of surrogate losses, offering a stronger guarantee than Bayes-consistency.

However, none of these surrogate losses satisfies all these guarantees simultaneously. In particular, a recent AISTATS notable award paper by Mozannar et al. [2023] left open the problem of finding surrogate losses that are both Bayes-consistent and realizable H -consistent when the cost function for the expert is its classification error. The problem becomes even more challenging when considering more general and realistic cost functions.

We present a comprehensive analysis of surrogate loss functions for L2D. Our contributions address the limitations of previous approaches and provide a unified framework for designing surrogate losses with strong theoretical guarantees. In Section 4, we first introduce a broad family of surrogate losses for L2D, derived from first principles (Section 4.1). This family is parameterized by a non-increasing function Ψ , which provides some flexibility in tailoring the loss function to specific requirements. We establish that under mild conditions on Ψ , these surrogate losses achieve realizable H -consistency, a key guarantee for many applications (Section 4.2).

Next, for cost functions based on classification error, we further establish that our surrogate loss functions admit H -consistency bounds when the hypothesis set is symmetric and complete (Section 4.3). This result holds for commonly used neural network and linear function hypothesis sets, further strengthening the applicability of our results. Additionally, our results resolve an open question raised by Mozannar et al. [2023] by proving the realizable H -consistency and Bayes-consistency of their proposed surrogate loss, which the authors had left as an open question (Section 4.4).

In Section 4.3, we further identify specific choices of Ψ , such as the one corresponding to the mean absolute error loss, that lead to H -consistent surrogate losses for any general cost function . These loss functions are adapted to general cost functions and benefit from Bayes-consistency (Section 4.4), realizable H -consistency, and H -consistency bounds simultaneously .

In Section 5, we also study the relationship between H -consistency bounds and realizable H -consistency in the context of L2D, highlighting key distinctions from the standard classification setting. Finally, we further report the results of experiments with our new surrogate losses and their comparison with the baselines in different settings (Section 6).

We discuss the related work in Section 2 and then begin with the preliminaries in Section 3.

2 Related work

The approach of single-stage learning to defer , where a predictor and a deferral function are trained together, was pioneered by Cortes, DeSalvo, and Mohri [2016a,b, 2023] and further developed in subsequent studies on abstention, where the cost is constant [Charoenphakdee et al., 2021, Cao et al., 2022, Li et al., 2023, Cheng et al., 2023, Mao et al., 2024c,b, Mohri et al., 2024] and on deferral, where the cost can vary depending on the instance and the label [Mozannar and Sontag, 2020, Verma and Nalisnick, 2022, Mozannar et al., 2023, Verma et al., 2023, Cao et al., 2023, Mao et al., 2023a, 2024a]. In this approach, the deferral function determines whether to defer to an expert for each input. This approach has been shown to be superior to confidence-based approaches, where the decision to abstain or defer is based solely on the magnitude of the predictor's value [Chow, 1957, 1970, Bartlett and Wegkamp, 2008, Yuan and Wegkamp, 2010, 2011, Ramaswamy et al., 2018, Ni et al., 2019, Jitkrittum et al., 2023]; and to selective classification approaches, where the selection rate is fixed and a cost function modeled by an expert cannot be taken into account [El-Yaniv et al., 2010, El-Yaniv and Wiener, 2012, Wiener and El-Yaniv, 2011, 2012, 2015, Geifman and El-Yaniv, 2017, 2019, Acar et al., 2020, Gangrade et al., 2021, Zaoui et al., 2020, Jiang et al., 2020, Shah et al., 2022].

Madras et al. [2018] initiated the learning to defer (L2D) problem scenario, which integrates human expert decisions into the cost function. This approach has been further explored in subsequent studies [Raghu et al., 2019, Wilder et al., 2021, Pradier et al., 2021]. Mozannar and Sontag [2020] introduced the first Bayes-consistent surrogate loss for L2D, which was further refined in [Raman and Yee, 2021, Liu et al., 2022]. Verma and Nalisnick [2022] proposed an alternative Bayes-consistent surrogate loss,

the one-versus-all loss, which was later examined within a broader family of loss functions [Charusaie et al., 2022]. Cao et al. [2023] proposed an asymmetric softmax function, which can induce a valid probability estimator for learning to defer. Mozannar et al. [2023] showed that the surrogate losses in [Mozannar and Sontag, 2020, Verma and Nalisnick, 2022] are not realizable H -consistent. They proposed an alternative surrogate loss that is realizable H -consistent, but they were unable to prove or disprove whether the proposed surrogate loss is Bayes-consistent. All the surrogate losses mentioned above and their consistency guarantees hold only for cost functions based on classification error. Mao et al. [2024a] generalized the surrogate loss in [Mozannar and Sontag, 2020] to incorporate general cost functions and any multi-class surrogate losses. They provided H -consistency bounds for the novel family of surrogate losses, offering a stronger guarantee than Bayes-consistency.

Additional studies have focused on post-hoc methods, with Okati et al. [2021] suggesting an alternative optimization technique between the predictor and rejector, and Narasimhan et al. [2022] offering corrections for underfitting surrogate losses [Liu et al., 2024], and Charusaie and Samadi [2024] providing a unifying post-processing framework for multi-objective L2D based on a generalization of the Neyman-Pearson Lemma [Neyman and Pearson, 1933]. The L2D framework or variations thereof have found applications in diverse scenarios, spanning regression, reinforcement learning, and human-in-the-loop systems, among others [De et al., 2020, 2021, Straitouri et al., 2021, Zhao et al., 2021, Joshi et al., 2021, Gao et al., 2021, Mozannar et al., 2022, Hemmer et al., 2023, Chen et al., 2024, Palomba et al., 2024]. More recently, the problem of learning to defer with multiple experts has been analyzed in several publications [Hemmer et al., 2022, Keswani et al., 2021, Kerrigan et al., 2021, Straitouri et al., 2022, Benz and Rodriguez, 2022, Verma et al., 2023, Mao et al., 2023a, 2024a,g, Tailor et al., 2024]. Meanwhile, Mao et al. [2023a] also proposed a two-stage learning to defer framework. They introduced two-stage surrogate losses that are both Bayes-consistent and realizable H -consistent with constant costs. However, realizable H -consistency does not hold for cost functions based on classification error. As with [Mozannar and Sontag, 2020, Verma and Nalisnick, 2022, Mozannar et al., 2023], our work focuses on the single-stage and single-expert setting, and we plan to explore a similar approach in a multi-expert/two-stage setting in the future.

3 Preliminaries

We start with the definitions and notations used in the learning-to-defer scenario considered in this paper. We will then introduce consistency guarantees, including Bayes consistency , Realizable H -consistency , and H -consistency bounds . Finally, we will review existing consistent surrogate losses for L2D.

3.1 Learning to defer: problem setup

Let X be an input space and Y = [ n ] ∶= { 1 , . . . , n } be the label space in the standard multi-class classification setting. We study the learning to defer (L2D) scenario, where a learner can either predict a label from Y or defer to an expert.

To model this, we introduce an augmented label space Y = { 1 , . . . , n, n + 1 } , where the label n + 1 corresponds to deferral. An expert is a fixed predictor g ∶ X × Y → R . The goal of L2D is to select a predictor h out of a hypothesis set H of functions mapping from X × Y to R with small expected deferral loss . Let h ( x ) denote the prediction of h on input x ∈ X , defined as h ( x ) = argmax y ∈ Y h ( x, y ) , that is the label in the augmented label space Y with the highest score, with an arbitrary but fixed deterministic strategy for breaking ties. Then, the deferral loss function L def is defined as follows:

where c ( x, y ) is the the cost of deferring on input x with true label y . If the deferral option is selected, that is h ( x ) = n + 1 , the deferral cost c ( x, y ) is incurred. Otherwise, the prediction of h is within the standard label space, h ( x ) ∈ [ n ] , and the loss incurred coincides with the standard zero-one classification loss, 1 h ( x )≠ y .

The choice of the cost function c is flexible. For example, the cost can be defined as the expert's classification error: c ( x, y ) = 1 g ( x )≠ y , as in previous work [Mozannar and Sontag, 2020, Verma and Nalisnick, 2022, Mozannar et al., 2023]. Here, g ( x ) = argmax y ∈[ n ] g ( x, y ) is the prediction made by

the expert g . More generally, it can incorporate the inference cost for the expert [Mao et al., 2024a]: c ( x, y ) = α 1 g ( x )≠ y + β , with α,β > 0 . We assume, without loss of generality, that the cost is bounded by 1: 0 ≤ c ( x, y ) ≤ 1 , which can be achieved through normalization in practice.

3.2 Consistency guarantees

Directly optimizing the deferral loss function, which is the target loss in L2D, is generally computationally intractable for complex hypothesis sets H . Therefore, a common approach is to optimize a surrogate loss that facilitates the optimization of the deferral loss function. A natural learning guarantee for such surrogate losses is Bayes-consistency [Zhang, 2004a, Bartlett et al., 2006, Zhang, 2004b, Tewari and Bartlett, 2007, Steinwart, 2007]:

Definition 3.1 (Bayes-consistency) . A surrogate loss L is Bayes-consistent with respect to L def , if minimizing the surrogate loss over the family of all measurable functions leads to the minimization of the deferral loss:

Here, given a distribution D over X × Y and a loss function L ∶ H × X × Y → R , we denote by E L ( h ) the generalization error of a hypothesis h ∈ H , E L ( h ) = E ( x,y )∼ D [ L ( h, x, y )] , and by E ∗ L ( H ) the best-in-class generalization error , E ∗ L ( H ) = inf h ∈ H E L ( h ) . Bayes-consistency assumes that the optimization occurs over the family of all measurable functions, H all . However, in practice, the hypothesis set of interest is typically a restricted one, such as a family of neural networks. Therefore, a hypothesis-dependent learning guarantee, such as H -consistency bounds [Awasthi et al., 2022a,b] (see also [Awasthi et al., 2021a,b, 2023, 2024, Mao et al., 2023b,e,f, Zheng et al., 2023, Mao et al., 2023c,d, 2024h,e,d,f, Cortes et al., 2024]) and realizable H -consistency [Long and Servedio, 2013, Zhang and Agarwal, 2020], is more informative and relevant. Realizable H -consistency, defined as follows, requires that a minimizer of the surrogate loss over the given hypothesis set H also minimizes the target loss, provided that the underlying distribution is realizable.

Definition 3.2 (Realizable H -consistency) . Asurrogate loss L is realizable H -consistent with respect to L def , if for any distribution over which there exists a predictor h ∗ ∈ H achieving zero deferral loss, E L def ( h ∗ ) = 0 , minimizing the surrogate loss also leads to a zero-error solution:

Note that realizable H -consistency does not imply Bayes-consistency, even if we set H = H all in Definition 3.2, since Bayes-consistency requires that the relationship holds for all distributions, not just realizable ones. H -consistency bounds , on the other hand, always imply Bayes-consistency. Given a hypothesis set H , a surrogate loss L admits an H -consistency bound , if for some nondecreasing concave function Γ ∶ R + → R + with Γ ( 0 ) = 0 , a bound of the following form holds for any hypothesis h ∈ H and any distribution:

where M L ( H ) is the minimizability gap , defined as the difference between the best-in-class generalization error and the expected pointwise infimum loss: M L ( H ) = E ∗ L ( H )-E x [ inf h ∈ H E y ∣ x [ L ( h, x, y )]] . The minimizability gap can be upper-bounded by the approximation error and vanishes when H = H all [Awasthi et al., 2022a,b]. Thus, an H -consistency bound implies Bayes-consistency. The relationship between the two hypothesis-dependent learning guarantees-realizable H -consistency and H -consistency bounds-depends on the target loss adopted in the specific learning scenario. In Section 5, we will demonstrate that in the standard multi-class classification setting, an H -consistency bound is a stronger notion than realizable H -consistency. However, in L2D, these guarantees do not imply one another.

3.3 Existing surrogate losses

Here, we will review several consistent surrogate losses used in L2D. For convenience, we use ̃ c ( x, y ) = 1 g ( x )≠ y to denote the cost when it specifically represents the expert's classification error, and use c ( x, y ) when it represents a general cost function.

Mozannar and Sontag [2020] proposed the first Bayes-consistent surrogate loss by generalizing the cross-entropy loss for L2D, with cost functions based on classification error, which is defined as

Verma and Nalisnick [2022] proposed an alternative one-vs-all surrogates loss with cost functions based on expert's classification error, that is Bayes-consistent as well:

where Φ is a strictly proper binary composite loss [Reid and Williamson, 2010], such as the logistic loss t ↦ log ( 1 + e -t ) . L CE and L OvA are not realizable H -consistent. Instead, Mozannar et al. [2023] proposed the following loss function that is realizable H -consistent when H is closed under scaling:

However, they were unable to prove or disprove whether the surrogate loss L RS is Bayes-consistent. All the surrogate losses mentioned above and their consistency guarantees hold only for cost functions based on the classification error: ̃ c ( x, y ) = 1 g ( x )≠ y . Mao et al. [2024a] generalized the surrogate loss L CE to incorporate general cost functions and any multi-class surrogate losses:

Here, glyph[lscript] is a Bayes-consistent surrogate loss for the multi-class zero-one loss over the augmented label set Y . In particular, glyph[lscript] can be chosen as a comp-sum loss [Mao et al., 2023f], for example, the generalized cross entropy loss (see Section 4.1). As shown by Mao et al. [2024a], L general benefits from H -consistency bounds, which implies its Bayes-consistency.

4 Novel surrogate losses

In this section, we introduce a new family of surrogate losses for L2D that benefit from Bayesconsistency, realizable H -consistency and H -consistency bounds, starting from first principles.

4.1 Derivation from first principles

Observe that for any ( x, y ) ∈ X × Y , we have 1 h ( x )= n + 1 = 1 h ( x )≠ y 1 h ( x )= n + 1 , since h ( x ) = n + 1 implies h ( x ) ≠ y . Thus, using additionally 1 h ( x )∈[ n ] = 1 h ( x )≠ n + 1 , the deferral loss can be rewritten as follows for all ( x, y ) ∈ X × Y :

Next, we will derive the new surrogate losses for L2D by replacing the indicator functions in (2) with smooth loss functions. The first indicator function 1 h ( x )≠ y is just the multi-class zero-one loss. Thus, a natural choice is to replace it with a surrogate loss in standard multi-class classification. We will specifically consider the family of comp-sum losses [Mao et al., 2023f], defined as follows for any ( h, x, y ) ∈ H × X × Y :

where Ψ ∶ [ 0 , 1 ] → R + ∪{+∞} is a non-increasing function. For example, by taking Ψ ( t ) = -log ( t ) , 1 q ( 1 -t q ) with q ∈ ( 0 , 1 ) , 1 -t , we obtain the logistic loss [Verhulst, 1838, 1845, Berkson, 1944,

1951], the generalized cross entropy loss [Zhang and Sabuncu, 2018], and the mean absolute error loss [Ghosh et al., 2017], respectively:

For any ( h, x, y ) ∈ H × X × Y , the confidence margin ρ h ( x, y ) is defined by ρ h ( x, y ) = h ( x, y ) -max y ′ ∈ Y ,y ′ ≠ y h ( x, y ′ ) . Thus, the second indicator function 1 h ( x )≠ y ∧ h ( x )≠ n + 1 can be expressed as follows in terms of the confidence margin:

Note that the first indicator function can also be written in terms of margin: 1 h ( x )≠ y = 1 ρ h ( x,y )≤ 0 . Unlike the first indicator function, which presses h ( x, y ) to be the largest score among Y , that is the margin ρ h ( x, y ) to be positive, the second indicator function only enforces h ( x, y ) or h ( x, n + 1 ) to be the largest score among Y , that is the maximum of two margins, max { ρ h ( x, y ) , ρ h ( x, n + 1 )} , to be positive. This condition can be further strengthened by requiring the sum of two margins, ρ h ( x, y ) + ρ h ( x, n + 1 ) , to be positive. In view of this observation, we adopt the following modified comp-sum surrogate loss for the second indicator function:

where Ψ ∶ [ 0 , 1 ] → R + ∪{+∞} is a non-increasing function. In other words, ̃ glyph[lscript] comp replaces the term e h ( x,y ) in the softmax function in glyph[lscript] comp with the sum e h ( x,y ) + e h ( x,n + 1 ) . The effect is to encourage the sum of the two margins, ρ h ( x, y ) + ρ h ( x, n + 1 ) , to be positive, rather than just the single margin ρ h ( x, y ) . Following this principle, we derive the following expression for a new family of surrogate losses, L RL2D , dubbed realizable L2D :

For the choices of Ψ ( t ) = -log ( t ) , 1 q ( 1 -t q ) with q ∈ ( 0 , 1 ) and 1 -t , we obtain the new surrogate losses for L2D in Table 1. In the next sections, we will prove both realizable H -consistency guarantees and H -consistency bounds for this family of surrogate losses, which imply their excess error bounds and Bayes-consistency as well.

4.2 Realizable H -consistency

Here, we show that L RL2D is realizable H -consistent with respect to L def . We say that a hypothesis set H is closed under scaling if, h ∈ H /Leftrightline⇒ αh ∈ H for any α ∈ R .

Theorem 4.1. Assume that H is closed under scaling. Suppose that Ψ is non-increasing, Ψ ( 2 3 ) > 0 and lim t → 1 Ψ ( t ) = 0 . Then, the surrogate loss L RL2D is realizable H -consistent with respect to L def .

The proof, detailed in Appendix A, begins by establishing an upper bound on the deferral loss in terms of the comp-sum loss: L def ≤ L RL2D Ψ ( 2 3 ) . Letting ˆ h be the minimizer of L RL2D and α be any real number, we then show that E L def ( ˆ h ) ≤ 1 Ψ ( 2 3 ) E L RL2D ( αh ∗ ) . The generalization error is then split by conditioning on whether h ∗ ( x ) is the deferral class ( n + 1 ) or not. Finally, we demonstrate that each conditional term converges to zero as α tends to +∞ , and apply the monotone convergence theorem to complete the proof.

4.3 H -Consistency bounds

Here, we show that L RL2D admits an H -consistency bound with respect to L def , which implies its Bayes-consistency as well. We say that a hypothesis set is symmetric if there exists a family F of functions f mapping from X to R such that {[ h ( x, 1 ) , . . . , h ( x, n + 1 )]∶ h ∈ H } = {[ f 1 ( x ) , . . . , f n + 1 ( x )]∶ f 1 , . . . , f n + 1 ∈ F } , for any x ∈ X . We say that a hypothesis set H is complete if for any ( x, y ) ∈ X × Y , the set of scores generated by it spans across the real numbers: { h ( x, y ) ∣ h ∈ H } = R . Common neural network and linear function hypothesis sets are all symmetric and complete. We first consider the case where the cost is expert's classification error.

Theorem 4.2. Assume that H is symmetric and complete and that c ( x, y ) = 1 g ( x )≠ y . Then, for all h ∈ H and any distribution, the following H -consistency bound holds:

where Γ ( t ) = √ 2 t when Ψ ( t ) = -log ( t ) and Γ ( t ) = √ 2 ( n + 1 ) q t when Ψ ( t ) = 1 q ( 1 -t q ) with q ∈ ( 0 , 1 ) .

The proof, detailed in Appendix B.3 and B.4, establishes strong consistency guarantees for our new surrogate loss L RL2D (Theorem 4.2). We first introduce y max = argmax y ∈ Y p ( x, y ) , the label with the highest conditional probability. We then show that for any hypothesis h and input x , if y max is not the predicted label h max , the conditional error of h is lower bounded by a modified hypothesis h (obtained by swapping the scores of y max and h max ). Next, for hypotheses where y max = h max , we lower bound their conditional regret in terms of the conditional regret of the deferral loss using a new hypothesis h µ . This proof is novel and significantly different from existing approaches for establishing H -consistency bounds in either the standard or deferral settings [Mao et al., 2023f, 2024a].

The next result further shows that when Ψ ( t ) = 1 -t , our surrogate losses benefit from H -consistency bounds for any general cost function.

Theorem 4.3. Assume that H is symmetric and complete. Suppose that Ψ ( t ) = 1 -t . Then, for all h ∈ H and any distribution, the following H -consistency bounds hold:

The proof is included in Appendix B.2. Theorem 4.2 provides stronger consistency guarantees for our new surrogate loss L RL2D with Ψ ( t ) = 1 -t since it holds for any general cost function. The proof idea is similar to that of Theorem 4.2, albeit with more cases to analyze due to the general cost function. This occurs when lower bounding the conditional regret of a hypothesis h , which satisfies y max = h max , in terms of the conditional regret of the deferral loss by introducing a new hypothesis h µ . The additional cases necessitate a more stringent condition for the guarantee, such that the functions Ψ ( t ) = -log ( t ) and Ψ ( t ) = 1 q ( 1 -t q ) do not apply.

4.4 Excess error bounds and Bayes-consistency

For the family of all measurable functions H = H all , the minimizability gaps vanish. In this case, Theorems 4.2 and 4.3 imply the following excess error bounds and Bayes-consistency guarantees.

Corollary 4.4. Suppose that c ( x, y ) = 1 g ( x )≠ y . For all h ∈ H all and any distribution, the following excess error bounds hold:

where Γ ( t ) = √ 2 t when Ψ ( t ) = -log ( t ) and Γ ( t ) = √ 2 ( n + 1 ) q t when Ψ ( t ) = 1 q ( 1 -t q ) with q ∈ ( 0 , 1 ) . Furthermore, the surrogate loss L RL2D is Bayes-consistent with respect to L def in these cases.

Corollary 4.5. Suppose that Ψ ( t ) = 1 -t . For all h ∈ H all and any distribution, the following excess error bounds hold:

Furthermore, the surrogate loss L RL2D is Bayes-consistent with respect to L def in this case.

Therefore, Theorem 4.1 and Corollary 4.4 show that L RL2D is both realizable H -consistent and Bayes-consistent with respect to L def . This solves the open problem raised by Mozannar et al. [2023].

In particular, for cost functions based on classification error, c ( x, y ) = 1 g ( x )≠ y , our surrogate loss L RL2D with Ψ ( t ) = -log ( t ) coincides with the surrogate loss L RS in [Mozannar et al., 2023], modulo a constant. This affirmatively answers the question of whether their surrogate loss is Bayes-consistent when c ( x, y ) = 1 g ( x )≠ y . However, their surrogate loss cannot be shown to be Bayes-consistent for a general cost function. In contrast, our surrogate losses L RL2D with Ψ ( t ) = 1 -t are adaptable to general cost functions and benefit from both H -consistency bounds and realizable H -consistency guarantees. We also provide a more general family of comp-sum loss functions with Ψ ( t ) = 1 q ( 1 -t q ) that benefit from both H -consistency bounds and realizable H -consistency when c ( x, y ) = 1 g ( x )≠ y .

4.5 Summary

Here, we summarize the consistency properties of existing surrogate losses and ours. As mentioned earlier, most surrogate losses proposed in previous work, except for L general , are analyzed under the condition c ( x, y ) = 1 g ( x )≠ y . This naturally leads to a summary of these surrogate losses in this context, as presented in Table 1. Additionally, we provide analyses and the consistency properties of our surrogate loss, L RL2D , with general cost functions.

More specifically, our surrogate losses L RL2D satisfying Theorem 4.1 perform better in realizable scenarios than the surrogate losses L CE , L OVA , and L general from prior work, as ours are realizable H -consistent while theirs are not. This will be illustrated by our experiment results in the realizable case (Figure 1a). Our surrogate losses L RL2D satisfying Theorem 4.2 and Corollary 4.4 are comparable to the surrogate losses in prior work in non-realizable scenarios when the cost is the expert's classification error, as all of them are Bayes-consistent and supported by H-consistency bounds. This is demonstrated by our experiment in the non-realizable case with the cost function being the expert's classification error (Table 3). Our surrogate losses L RL2D satisfying Theorem 4.3 and Corollary 4.5 are superior to the surrogate loss L RS in non-realizable scenarios with general cost functions, as ours are supported by H-consistency bounds and Bayes-consistency while theirs are not. This is evidenced by our experiment in the non-realizable case with general cost functions (Figure 1b).

5 Relationship between H -consistency bounds and realizable H -consistency

Here, we discuss the relationship between H -consistency bounds and realizable H -consistency. First, realizable H -consistency does not imply H -consistency bounds, since H -consistency bounds require that the relationship holds for all distributions, not just realizable ones. Moreover, H -consistency bounds provide non-asymptotic guarantees, while realizable H -consistency provides only asymptotic guarantees. Second, H -consistency bounds imply realizable H -consistency in the standard multi-class classification setting. This is because minimizability gaps vanish under the realizable assumption in standard case. In particular, for comp-sum losses, the following holds (see Appendix C for proof).

Theorem 5.1. Assume that there exists a zero error solution h ∗ ∈ H with E glyph[lscript] 0 -1 ( h ∗ ) = 0 and H is closed under scaling. Assume that lim t → 1 Ψ ( t ) = 0 . Then, the minimizability gap of comp-sum loss glyph[lscript] comp vanishes: M glyph[lscript] comp ( H ) = 0 .

However, in the deferral setting, this relationship no longer holds: H -consistency bounds cannot imply realizable H -consistency. In particular, Mao et al. [2023f] showed that L CE benefits from H -consistency bounds, while Mozannar et al. [2023] showed that it is not realizable H -consistent. The loss function in [Madras et al., 2018] is not Bayes-consistent, and thus does not have H -consistency bound guarantees, but is actually realizable H -consistent [Mozannar et al., 2023].

6 Experiments

In this section, we empirically evaluate our proposed surrogate losses and compare them with existing baselines.

Experimental settings. We follow the setting of Mozannar et al. [2023] and conduct experiments on a synthetic dataset: Mixture-of-Gaussians [Mozannar et al., 2023], and three real-world datasets: CIFAR-10H [Battleday et al., 2020], HateSpeech [Davidson et al., 2017], and COMPASS [Dressel and Farid, 2018]. For these three datasets, we adopt the same model class as that in [Mozannar et al., 2023, Table 1]. Each dataset is randomly split into 70%, 10%, and 20% for training, validation, and testing, respectively. For the Mixture-of-Gaussians, we adopt the exact realizable setting from [Mozannar et al., 2023, Section 7.2], which is realizable by linear functions: there exists a linear hypothesis h ∗ ∈ H achieving zero deferral loss, E L def ( h ∗ ) = 0 .

As with [Mozannar et al., 2023], we choose the cost function to be the expert's classification error: c ( x, y ) = 1 g ( x )≠ y . We compare our surrogate to four baselines as described in Section 3.3: the cross-entropy surrogate L CE from [Mozannar and Sontag, 2020], the one-vs-all surrogate L OvA from [Mozannar and Sontag, 2020], the realizable surrogate L RS from [Mozannar et al., 2023], and the general surrogate L general from [Mao et al., 2024a]. For L OvA , we choose Φ as the logistic loss, following [Verma and Nalisnick, 2022]. For L general , we choose glyph[lscript] as the generalized cross entropy loss with q = 0 . 7 , following [Mao et al., 2024a]. For our Realizable L2D surrogate L RL2D , we consider two choices: glyph[lscript] as the generalized cross entropy loss with q = 0 . 7 , following [Zhang and Sabuncu, 2018, Mao et al., 2024a], and glyph[lscript] as the mean absolute error loss ( q = 1 ). Among these, L CE , L OvA and L general are Bayes-consistent but not realizable H -consistent; L RS , L RL2D with q = 0 . 7 and L RL2D with q = 1 are both Bayes-consistent and realizable H -consistent, as shown in Sections 4.2 and 4.4. Note that in this case, L RS is a special case of L RL2D when Ψ is chosen as t ↦-log ( t ) . We use the same optimizer, learning rate, and number of epochs as chosen in [Mozannar et al., 2023], and we select the model that achieves the highest system accuracy , that is average [ 1 -L def ( h, x, y )] , on a validation set.

500

1000

1500

2000

2500

3000

3500

4000

Training sample size

0.86

0.88

0.90

0.92

0.94

0.96

0.98

System Accuracy

LCE

LOvA

Lgeneral

LRL2D,

q

=0.7

LRL2D,

q

=1

0.65

0.70

0.75

0.80

0.85

Coverage

0.87

0.88

0.89

0.90

0.91

0.92

System Accuracy

LRS

LRL2D,

q

=1

Lgeneral

(a) Comparison of system accuracy versus training sample size on a realizable synthetic dataset. (b) Comparison of system accuracy versus coverage on the HateSpeech dataset with general cost functions.

Evaluation. For the three real-world datasets, we report the system accuracy , that is average value of [ 1 -L def ( h, x, y )] on the test data. For completeness, we also include the accepted accuracy , that is the average value of [ 1 h ( x )≠ y 1 h ( x )∈[ n ] ] . This metric considers only incorrect predictions ( h ( x ) ≠ y ) and measures the fraction of those where the system's output ( h ( x ) ) falls within the valid range of possible outputs ( [ n ] ). We also report the coverage , that is the average value of [ 1 h ( x )∈[ n ] ] on the test set, or the fraction of test instances where the system's prediction falls within the valid range ( [ n ] ). For each metric, we average results over three runs and report the mean accuracy along with the standard deviation for both our proposed methods and the baseline approaches. For the realizable Mixture-of-Gaussians, we plot the system accuracy of various methods on a held-out test dataset consisting of 5,000 points as we increase the size of the training data.

Results. Table 3 shows that for the real-world datasets, L RL2D with q = 0 . 7 , and L RL2D with q = 1 either outperform or are comparable to the best baseline in terms of system accuracy on each dataset. This performance is supported by our H -consistency bounds and Bayes-consistency results for our Realizable L2D surrogate with respect to the deferral loss L def , as shown in Sections 4.3 and 4.4. Table 3 also shows that L RL2D achieves reasonable coverage and acceptable accuracy. The system accuracy, coverage, and standard deviations of the baselines match those in [Mozannar et al., 2023]. Moreover, L RS , L RL2D with q = 0 . 7 , and L RL2D with q = 1 perform differently across various datasets: L RL2D with q = 0 . 7 outperforms the others on HateSpeech and CIFAR-10H, while L RL2D with q = 1 outperforms the others on COMPASS. Note that in this case, L RS is a special case of L RL2D when Ψ is chosen as t ↦-log ( t ) . These results show that Realizable L2D can benefit from the flexibility in the choice of Ψ .

Figure 1a shows system accuracy versus training samples on the realizable Mixture-of-Gaussians distribution. Our surrogate loss L RL2D with q = 0 . 7 and q = 1 are realizable H -consistent, while L CE , L OVA and L general are not. This verifies our theory.

Figure 1b shows system accuracy versus coverage on the HateSpeech dataset by varying β in the general cost functions c ( x, y ) = 1 g ( x )≠ y + β . As β increases, deferral algorithms yield solutions with higher coverage and decreased system accuracy. This is because β controls the trade-off between expert's inference cost and accuracy. L RL2D with q = 1 performs comparably to the surrogate loss L general , as both are supported by H -consistency bounds and Bayes-consistency with general cost functions. Our surrogate loss L RL2D with q = 1 outperforms L RS because the latter does not benefit from Bayes-consistency with general cost functions.

7 Conclusion

We introduced a broad family of surrogate losses and algorithms for learning to defer, parameterized by a non-increasing function. We established their realizable H -consistency properties under mild conditions and proved that several of these surrogate losses benefit from H -consistency bounds for cost functions based on classification error and general cost functions, which also imply their Bayes-consistency. This research not only resolves an open question posed in previous work but also lays the groundwork for comparing various consistency notions in learning to defer and standard classification. Looking forward, our approach offers a promising avenue for analyzing multi-expert and two-stage settings.

References

A. E. Acar, A. Gangrade, and V. Saligrama. Budget learning via bracketing. In International Conference on Artificial Intelligence and Statistics , pages 4109-4119, 2020.

Awasthi, N. Frank, A. Mao, M. Mohri, and Y. Zhong. Calibration and consistency of adversarial surrogate losses. Advances in Neural Information Processing Systems , pages 9804-9815, 2021a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. A finer calibration analysis for adversarial robustness. arXiv preprint arXiv:2105.01550 , 2021b.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. H -consistency bounds for surrogate loss minimizers. In International Conference on Machine Learning , 2022a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. Multi-class H -consistency bounds. In Advances in neural information processing systems , 2022b.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. Theoretically grounded loss functions and algorithms for adversarial robustness. In International Conference on Artificial Intelligence and Statistics , pages 10077-10094, 2023.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. DC-programming for neural network optimizations. Journal of Global Optimization , 2024.

L. Bartlett and M. H. Wegkamp. Classification with a reject option using a hinge loss. Journal of Machine Learning Research , 9(8), 2008.

L. Bartlett, M. I. Jordan, and J. D. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association , 101(473):138-156, 2006.

M. Battleday, J. C. Peterson, and T. L. Griffiths. Capturing human categorization of natural images by combining deep networks and cognitive models. Nature communications , 11(1):5418, 2020.

L. C. Benz and M. G. Rodriguez. Counterfactual inference of second opinions. In Uncertainty in Artificial Intelligence , pages 453-463, 2022.

Berkson. Application of the logistic function to bio-assay. Journal of the American Statistical Association , 39:357--365, 1944.

Berkson. Why I prefer logits to probits. Biometrics , 7(4):327--339, 1951.

Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023.

Cao, T. Cai, L. Feng, L. Gu, J. Gu, B. An, G. Niu, and M. Sugiyama. Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. In Advances in neural information processing systems , 2022.

Cao, H. Mozannar, L. Feng, H. Wei, and B. An. In defense of softmax parametrization for calibrated and consistent learning to defer. In Advances in Neural Information Processing Systems , 2023.

Charoenphakdee, Z. Cui, Y. Zhang, and M. Sugiyama. Classification with rejection based on cost-sensitive classification. In International Conference on Machine Learning , pages 1507-1517, 2021.

M.-A. Charusaie and S. Samadi. A unifying post-processing framework for multi-objective learn-todefer problems. arXiv preprint arXiv:2407.12710 , 2024.

M.-A. Charusaie, H. Mozannar, D. Sontag, and S. Samadi. Sample efficient learning of predictors that complement humans. In International Conference on Machine Learning , pages 2972-3005, 2022.

Chen, X. Li, C. Sun, and H. Wang. Learning to make adherence-aware advice. In International Conference on Learning Representations , 2024.

Cheng, Y. Cao, H. Wang, H. Wei, B. An, and L. Feng. Regression with cost-based rejection. In Advances in Neural Information Processing Systems , 2023.

Chow. An optimum character recognition system using decision function. IEEE T. C. , 1957.

Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on information theory , 16(1):41-46, 1970.

Cortes, G. DeSalvo, and M. Mohri. Learning with rejection. In International Conference on Algorithmic Learning Theory , pages 67-82, 2016a.

Cortes, G. DeSalvo, and M. Mohri. Boosting with abstention. In Advances in Neural Information Processing Systems , pages 1660-1668, 2016b.

Cortes, G. DeSalvo, and M. Mohri. Theory and algorithms for learning with rejection in binary classification. Annals of Mathematics and Artificial Intelligence , pages 1-39, 2023.

Cortes, A. Mao, C. Mohri, M. Mohri, and Y. Zhong. Cardinality-aware set prediction and topk classification. In Advances in neural information processing systems , 2024.

Davidson, D. Warmsley, M. Macy, and I. Weber. Automated hate speech detection and the problem of offensive language. In Proceedings of the international AAAI conference on web and social media , volume 11, pages 512-515, 2017.

De, P. Koley, N. Ganguly, and M. Gomez-Rodriguez. Regression under human assistance. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 2611-2620, 2020.

De, N. Okati, A. Zarezade, and M. G. Rodriguez. Classification under human assistance. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 5905-5913, 2021.

Dressel and H. Farid. The accuracy, fairness, and limits of predicting recidivism. Science advances , 4(1):eaao5580, 2018.

El-Yaniv and Y. Wiener. Active learning via perfect selective classification. Journal of Machine Learning Research , 13(2), 2012.

El-Yaniv et al. On the foundations of noise-free selective classification. Journal of Machine Learning Research , 11(5), 2010.

Gangrade, A. Kag, and V. Saligrama. Selective classification via one-sided prediction. In International Conference on Artificial Intelligence and Statistics , pages 2179-2187, 2021.

Gao, M. Saar-Tsechansky, M. De-Arteaga, L. Han, M. K. Lee, and M. Lease. Human-ai collaboration with bandit feedback. arXiv preprint arXiv:2105.10614 , 2021.

Geifman and R. El-Yaniv. Selective classification for deep neural networks. In Advances in neural information processing systems , 2017.

Geifman and R. El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In International conference on machine learning , pages 2151-2159, 2019.

Ghosh, H. Kumar, and P. S. Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence , 2017.

Hemmer, S. Schellhammer, M. Vössing, J. Jakubik, and G. Satzger. Forming effective human-ai teams: Building machine learning models that complement the capabilities of multiple experts. arXiv preprint arXiv:2206.07948 , 2022.

Hemmer, L. Thede, M. Vössing, J. Jakubik, and N. Kühl. Learning to defer with limited expert predictions. arXiv preprint arXiv:2304.07306 , 2023.

Jiang, Y. Zhao, and Z. Wang. Risk-controlled selective prediction for regression deep neural network models. In 2020 International Joint Conference on Neural Networks (IJCNN) , pages 1-8, 2020.

Jitkrittum, N. Gupta, A. K. Menon, H. Narasimhan, A. Rawat, and S. Kumar. When does confidence-based cascade deferral suffice? In Advances in Neural Information Processing Systems , 2023.

Joshi, S. Parbhoo, and F. Doshi-Velez. Pre-emptive learning-to-defer for sequential medical decision-making under uncertainty. arXiv preprint arXiv:2109.06312 , 2021.

Kerrigan, P. Smyth, and M. Steyvers. Combining human predictions with model probabilities via confusion matrices and calibration. Advances in Neural Information Processing Systems , 34: 4421-4434, 2021.

Keswani, M. Lease, and K. Kenthapadi. Towards unbiased and accurate deferral to multiple experts. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , pages 154-165, 2021.

Li, S. Liu, C. Sun, and H. Wang. When no-rejection learning is optimal for regression with rejection. arXiv preprint arXiv:2307.02932 , 2023.

Liu, B. Gallego, and S. Barbieri. Incorporating uncertainty in learning to defer algorithms for safe computer-aided diagnosis. Scientific reports , 12(1):1762, 2022.

Liu, Y. Cao, Q. Zhang, L. Feng, and B. An. Mitigating underfitting in learning to defer with consistent losses. In International Conference on Artificial Intelligence and Statistics , pages 4816-4824, 2024.

Long and R. Servedio. Consistency versus realizable H-consistency for multiclass classification. In International Conference on Machine Learning , pages 801-809, 2013.

Madras, E. Creager, T. Pitassi, and R. Zemel. Learning adversarially fair and transferable representations. arXiv preprint arXiv:1802.06309 , 2018.

Mao, C. Mohri, M. Mohri, and Y. Zhong. Two-stage learning to defer with multiple experts. In Advances in neural information processing systems , 2023a.

Mao, M. Mohri, and Y. Zhong. H-consistency bounds: Characterization and extensions. In Advances in Neural Information Processing Systems , 2023b.

Mao, M. Mohri, and Y. Zhong. H-consistency bounds for pairwise misranking loss surrogates. In International conference on Machine learning , 2023c.

Mao, M. Mohri, and Y. Zhong. Ranking with abstention. In ICML 2023 Workshop The Many Facets of Preference-Based Learning , 2023d.

Mao, M. Mohri, and Y. Zhong. Structured prediction with stronger consistency guarantees. In Advances in Neural Information Processing Systems , 2023e.

Mao, M. Mohri, and Y. Zhong. Cross-entropy loss functions: Theoretical analysis and applications. In International Conference on Machine Learning , 2023f.

Mao, M. Mohri, and Y. Zhong. Principled approaches for learning to defer with multiple experts. In International Symposium on Artificial Intelligence and Mathematics , 2024a.

Mao, M. Mohri, and Y. Zhong. Predictor-rejector multi-class abstention: Theoretical analysis and algorithms. In International Conference on Algorithmic Learning Theory , pages 822-867, 2024b.

Mao, M. Mohri, and Y. Zhong. Theoretically grounded loss functions and algorithms for scorebased multi-class abstention. In International Conference on Artificial Intelligence and Statistics , pages 4753-4761, 2024c.

Mao, M. Mohri, and Y. Zhong. Enhanced H -consistency bounds. arXiv preprint arXiv:2407.13722 , 2024d.

Mao, M. Mohri, and Y. Zhong. H -consistency guarantees for regression. In International Conference on Machine Learning , pages 34712-34737, 2024e.

Mao, M. Mohri, and Y. Zhong. Multi-label learning with stronger consistency guarantees. In Advances in neural information processing systems , 2024f.

Mao, M. Mohri, and Y. Zhong. Regression with multi-expert deferral. In International Conference on Machine Learning , pages 34738-34759, 2024g.

Mao, M. Mohri, and Y. Zhong. A universal growth rate for learning with smooth surrogate losses. In Advances in neural information processing systems , 2024h.

Mohri, D. Andor, E. Choi, M. Collins, A. Mao, and Y. Zhong. Learning to reject with a fixed predictor: Application to decontextualization. In International Conference on Learning Representations , 2024.

Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of Machine Learning . MIT Press, second edition, 2018.

Mozannar and D. Sontag. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning , pages 7076-7087, 2020.

Mozannar, A. Satyanarayan, and D. Sontag. Teaching humans when to defer to a classifier via exemplars. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 5323-5331, 2022.

Mozannar, H. Lang, D. Wei, P. Sattigeri, S. Das, and D. Sontag. Who should predict? exact algorithms for learning to defer to humans. In International Conference on Artificial Intelligence and Statistics , pages 10520-10545, 2023.

Narasimhan, W. Jitkrittum, A. K. Menon, A. S. Rawat, and S. Kumar. Post-hoc estimators for learning to defer to an expert. In Advances in Neural Information Processing Systems , pages 29292-29304, 2022.

Neyman and E. S. Pearson. Ix. on the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character , 231(694-706):289-337, 1933.

Ni, N. Charoenphakdee, J. Honda, and M. Sugiyama. On the calibration of multiclass classification with rejection. In Advances in Neural Information Processing Systems , pages 2582-2592, 2019.

Okati, A. De, and M. Rodriguez. Differentiable learning under triage. Advances in Neural Information Processing Systems , 34:9140-9151, 2021.

Palomba, A. Pugnana, J. M. Alvarez, and S. Ruggieri. A causal framework for evaluating deferring systems. arXiv preprint arXiv:2405.18902 , 2024.

F. Pradier, J. Zazo, S. Parbhoo, R. H. Perlis, M. Zazzi, and F. Doshi-Velez. Preferential mixtureof-experts: Interpretable models that rely on human expertise as much as possible. AMIA Summits on Translational Science Proceedings , 2021:525, 2021.

Raghu, K. Blumer, G. Corrado, J. Kleinberg, Z. Obermeyer, and S. Mullainathan. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv:1903.12220 , 2019.

Raman and M. Yee. Improving learning-to-defer algorithms through fine-tuning. arXiv preprint arXiv:2112.10768 , 2021.

G. Ramaswamy, A. Tewari, and S. Agarwal. Consistent algorithms for multiclass classification with an abstain option. Electronic Journal of Statistics , 12(1):530-554, 2018.

D. Reid and R. C. Williamson. Composite binary losses. The Journal of Machine Learning Research , 11:2387-2422, 2010.

Shah, Y. Bu, J. K. Lee, S. Das, R. Panda, P. Sattigeri, and G. W. Wornell. Selective regression under fairness criteria. In International Conference on Machine Learning , pages 19598-19615, 2022.

Steinwart. How to compare different loss functions and their risks. Constructive Approximation , 26(2):225-287, 2007.

Straitouri, A. Singla, V. B. Meresht, and M. Gomez-Rodriguez. Reinforcement learning under algorithmic triage. arXiv preprint arXiv:2109.11328 , 2021.

Straitouri, L. Wang, N. Okati, and M. G. Rodriguez. Provably improving expert predictions with conformal prediction. arXiv preprint arXiv:2201.12006 , 2022.

Tailor, A. Patra, R. Verma, P. Manggala, and E. Nalisnick. Learning to defer to a population: A meta-learning approach. In International Conference on Artificial Intelligence and Statistics , pages 3475-3483, 2024.

Tewari and P. L. Bartlett. On the consistency of multiclass classification methods. Journal of Machine Learning Research , 8(36):1007-1025, 2007.

F. Verhulst. Notice sur la loi que la population suit dans son accroissement. Correspondance mathématique et physique , 10:113--121, 1838.

F. Verhulst. Recherches mathématiques sur la loi d'accroissement de la population. Nouveaux Mémoires de l'Académie Royale des Sciences et Belles-Lettres de Bruxelles , 18:1--42, 1845.

Verma and E. Nalisnick. Calibrated learning to defer with one-vs-all classifiers. In International Conference on Machine Learning , pages 22184-22202, 2022.

Verma, D. Barrejón, and E. Nalisnick. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. In International Conference on Artificial Intelligence and Statistics , pages 11415-11434, 2023.

Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus. Emergent abilities of large language models. CoRR , abs/2206.07682, 2022.

Wiener and R. El-Yaniv. Agnostic selective classification. In Advances in neural information processing systems , 2011.

Wiener and R. El-Yaniv. Pointwise tracking the optimal regression function. Advances in Neural Information Processing Systems , 25, 2012.

Wiener and R. El-Yaniv. Agnostic pointwise-competitive selective classification. Journal of Artificial Intelligence Research , 52:171-201, 2015.

Wilder, E. Horvitz, and E. Kamar. Learning to complement humans. In International Joint Conferences on Artificial Intelligence , pages 1526-1533, 2021.

Yuan and M. Wegkamp. Classification methods with reject option based on convex risk minimization. Journal of Machine Learning Research , 11(1), 2010.

Yuan and M. Wegkamp. SVMs with a reject option. In Bernoulli , 2011.

Zaoui, C. Denis, and M. Hebiri. Regression with reject option and application to knn. In Advances in Neural Information Processing Systems , pages 20073-20082, 2020.

Zhang and S. Agarwal. Bayes consistency vs. H-consistency: The interplay between surrogate loss functions and the scoring function class. In Advances in Neural Information Processing Systems , 2020.

Zhang. Statistical behavior and consistency of classification methods based on convex risk minimization. The Annals of Statistics , 32(1):56-85, 2004a.

Zhang. Statistical analysis of some multi-category large margin classification methods. Journal of Machine Learning Research , 5(Oct):1225-1251, 2004b.

Zhang and M. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems , 2018.

Zhao, M. Agrawal, P. Razavi, and D. Sontag. Directing human attention in event localization for clinical timeline creation. In Machine Learning for Healthcare Conference , pages 80-102, 2021.

Zheng, G. Wu, F. Bao, Y. Cao, C. Li, and J. Zhu. Revisiting discriminative vs. generative classifiers: Theory and implications. In International Conference on Machine Learning , pages 42420-42477, 2023.

Contents of Appendix

A Proof of realizable H -consistency

Theorem 4.1. Assume that H is closed under scaling. Suppose that Ψ is non-increasing, Ψ ( 2 3 ) > 0 and lim t → 1 Ψ ( t ) = 0 . Then, the surrogate loss L RL2D is realizable H -consistent with respect to L def .

Proof. We first prove that for every ( h, x, y ) ∈ H × X × Y , the following inequality holds:

We will analyze case by case.

Case I : If h ( x ) ∈ [ n ] (deferral does not occur):

(a) If 1 h ( x )≠ y = 1 , then we must have

(b) If 1 h ( x )≠ y = 0 , then we must have

Case II : If h ( x ) = n + 1 (deferral occurs): then we must have

This concludes that L def ( h, x, y ) ≤ L RL2D ( h,x,y ) Ψ ( 2 3 ) . Next, we prove that L RL2D is realizable H -consistent under the assumptions. Consider a distribution and an expert under which there exists a zero error solution h ∗ ∈ H with E L def ( h ∗ ) = 0 . Let ˆ h be the minimizer of the surrogate loss: ˆ h ∈ argmin h ∈ H E L RL2D ( h ) . Let α be any real number. Then, the following inequality holds:

For the first term conditional on h ∗ ( x ) = n + 1 , we must have h ∗ ( x, n + 1 ) > max y ∈ Y h ∗ ( x, y ) and c ( x, y ) = 0 since the data is realizable. Therefore,

For the second term conditional on h ∗ ( x ) ∈ [ n ] , we must have h ∗ ( x, y ) > max y ′ ∈ Y ,y ′ ≠ y h ( x, y ′ ) since the data is realizable. Therefore, and monotone convergence theorem)

Combining the two analyses, we conclude that E L def ( ˆ h ) = 0 and thus L RL2D is realizable H -consistent with respect to L def .

B Proof of H -consistency bounds

Before delving into the proof, we first establish some essential notation and definitions. Let L represent a deferral surrogate loss and H denote a hypothesis set. We define the conditional error as C L ( h, x ) = E y ∣ x [ L ( h, x, y )] , the best-in-class conditional error as C ∗ L ( H , x ) = inf h ∈ H C L ( h, x ) , and the conditional regret as ∆ C L , H ( h, x ) = C L ( h, x ) -C ∗ L ( H , x ) . We proceed to present a general theorem demonstrating that, to establish H -consistency bounds (1) with a concave function Γ , it suffices to lower bound the conditional regret of the surrogate loss by that of the deferral loss, using the same function Γ .

Theorem B.1. If the following holds for all h ∈ H and x ∈ X , for some concave function Γ :

then, for all hypotheses h ∈ H and for any distribution,

Proof. We can express the expectations of the conditional regrets for L def and L as follows:

Then, by using (4) and taking the expectation, we obtain:

Thus, the proof is complete.

Next, to prove H -consistency bounds using Theorem B.1, we will characterize the conditional regret of the deferral loss L def in the following section.

B.1 Auxiliary lemma

To simplify the presentation, we introduce the following notation. For any y ∈ Y , define p ( x, y ) = P ( Y = y ∣ X = x ) as the conditional probability that Y = y given X = x . For brevity, we will omit the dependency on x in our notation. We denote by h y = h ( x, y ) for any y ∈ Y . We also denote by p y = p ( x, y ) and q y = p ( x, y ) c ( x, y ) for any y ∈ Y , and p n + 1 = ∑ y ∈ Y p ( x, y )( 1 -c ( x, y )) .

Note that p ( x, y )( 1 -c ( x, y )) = p y -q y , ∀ y ∈ Y . Let p h = p h ( x ) = { p h ( x ) h ( x ) ∈ [ n ] p n + 1 h ( x ) = n + 1 . . Let y max = argmax y ∈ Y p y and h max = argmax y ∈ Y h y . Note that both y max and h max are in the label space Y , while h ( x ) is in the augmented label space Y . We characterize the conditional regret of the deferral loss L def as follows.

Lemma B.2. Assume that H is symmetric and complete. Then, the conditional regret of the deferral loss L def can be expressed as follows: ∆ C L def , H ( h, x ) = max { p y max , p n + 1 } -p h .

Proof. We can write the conditional error of the deferral loss as follows:

Since H is symmetric and complete, for any x ∈ X , { h ( x )∶ h ∈ H } = Y . Then, the best-in-class conditional error of L def can be expressed as follows:

Next, we will present the proofs separately in the following sections, by lower bounding the conditional regret of the surrogate loss L by that of the deferral loss L def using Lemma B.2.

B.2 Ψ ( t ) = 1 -t

Theorem B.3. Assume that H is symmetric and complete. Then, for all h ∈ H and any distribution, the following H -consistency bound holds:

Proof. We can write the conditional error of the surrogate loss as follows:

By Lemma B.2, the conditional regret of the deferral loss can be expressed as

Next, we will show that the conditional regret of the surrogate loss can be lower bounded as follows:

We first prove that for any hypothesis h and x ∈ X , if y max ≠ h max , then the conditional error of h can be lower bounded by that of h , which satisfies that h ( x, y ) = ⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ h h max y = y max h y max y = h max h y otherwise . . Indeed,

Therefore, we only need to lower bound the conditional regret of hypothesis h satisfying y max = h max . Next, we will analyze case by case. Note that when ( p y max -p n + 1 )( h y max -h n + 1 ) > 0 , we have ∆ C L def , H ( h, x ) = max { p y max , p n + 1 } -p h = 0 .

1.

Case I : If p y max -p n + 1 ≥ 0 and h y max -h n + 1 ≤ 0 : we define a new hypothesis h µ such that h µ ( x, y ) = ⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ log ( e h n + 1 + µ ) y = y max log ( e h y max -µ ) y = n + 1 h ( x, y ) otherwise . , where e h y max ≥ µ ≥ 0 . Then, we can

lower bound the conditional regret of L RL2D by using ∆ C L RL2D , H ( h, x ) ≥ C L RL2D ( h ) -C ∗ L RL2D ( h µ ) for any e h y max ≥ µ ≥ 0 :

⎪ ⎩

Case II : If p y max -p n + 1 ≤ 0 and h y max -h n + 1 ≥ 0 : we define a new hypothesis h µ such that h µ ( x, y ) = ⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ log ( e h n + 1 -µ ) y = y max log ( e h y max + µ ) y = n + 1 h x, y otherwise . , where e h n + 1 ≥ µ ≥ 0 . Then, we can lower bound

the conditional regret of L RL2D by using ∆ C L RL2D , H ( h, x ) ≥ C L RL2D ( h ) -C L RL2D ( h µ ) for any e h n + 1 ≥ µ ≥ 0 :

⎪ ⎩ ( ) ∗

This proves the inequality (6). By Theorem B.1, we complete the proof.

B.3 Ψ ( t ) = -log ( t )

Theorem B.4. Assume that H is symmetric and complete. Assume that c ( x, y ) = 1 g ( x )≠ y . Then, for all h ∈ H and any distribution, the following H -consistency bound holds:

Proof. We can write the conditional error of the surrogate loss as follows:

By Lemma B.2, the conditional regret of the deferral loss can be expressed as

Next, we will show that the conditional regret of the surrogate loss can be lower bounded as follows:

We first consider the case where g ( x ) ≠ y max . Otherwise, it would be straightforward to see that the bound holds. In the case where g ( x ) ≠ y max , we have q y max = p y max . We first prove that for any hypothesis h and x ∈ X , if y max ≠ h max , then the conditional error of h can be lower bounded by that

Therefore, we only need to lower bound the conditional regret of hypothesis h satisfying y max = h max . Since c ( x, y ) = 1 g ( x )≠ y , we have p y max ≥ p n + 1 = p g ( x ) . Note that when ( p y max -p n + 1 )( h y max -h n + 1 ) > 0 , we have ∆ C L def , H ( h, x ) = max { p y max , p n + 1 }-p h = 0 . When h y max -h n + 1 ≤ 0 , we define h n + 1

a new hypothesis h µ such that h µ ( x, y ) = ⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ log ( e + µ ) y = y max log ( e h y max -µ ) y = n + 1 h ( x, y ) otherwise . , where e h y max -e h n + 1 ≤

µ ≤ e h y max . Then, we can lower bound the conditional regret of L RL2D by using ∆ C L RL2D , H ( h, x ) ≥ C L RL2D ( h ) -C ∗ L RL2D ( h µ ) for any e h y max -e h n + 1 ≤ µ ≤ e h y max :

By differentiating with respect to µ , we obtain that

achieves the maximum. Plugging it into the expression, we have

This can be further lower bounded by taking the minimum over h ∈ H , where the minimum is attained when e h y max = e h n + 1 Therefore,

By applying Pinsker's inequality [Mohri et al., 2018, Proposition E.7], we obtain

This proves the inequality (7). In the case where g ( x ) = y max , we have p n + 1 = p y max . By Lemma B.2, the conditional regret of the deferral loss can be expressed as ∆ C L def , H ( h, x ) = p n + 1 -p h . If h ( x ) = n + 1 , then we have ∆ C L def , H ( h, x ) = 0 . Otherwise, when h ( x ) ≠ n + 1 , we can proceed in the similar

Then, we can lower bound the conditional regret of L RL2D by using ∆ C L RL2D , H ( h, x ) ≥ C L RL2D ( h )-C ∗ L RL2D ( h µ ) , by applying the same derivation as above, modulo replacing y max with h ( x ) . This leads to the inequality (7) as well. By Theorem B.1, we complete the proof.

way as above, by defining a new hypothesis h µ such that h µ ( x, y ) = ⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ log ( e h n + 1 + µ ) y = h ( x ) log ( e h h ( x ) -µ ) y = n + 1 h ( x, y ) otherwise .

Theorem B.5. Assume that H is symmetric and complete. Assume that c ( x, y ) = 1 g ( x )≠ y . Then, for all h ∈ H and any distribution, the following H -consistency bound holds:

Proof. We can write the conditional error of the surrogate loss as follows:

By Lemma B.2, the conditional regret of the deferral loss can be expressed as

Next, we will show that the conditional regret of the surrogate loss can be lower bounded as follows:

We first consider the case where g ( x ) ≠ y max . Otherwise, it would be straightforward to see that the bound holds. In the case where g ( x ) ≠ y max , we have q y max = p y max . We first prove that for any hypothesis h and x ∈ X , if y max ≠ h max , then the conditional error of h can be lower bounded by that

⎪ ⎩ ( )

Therefore, we only need to lower bound the conditional regret of hypothesis h satisfying y max = h max Since c ( x, y ) = 1 g ( x )≠ y , we have p y max ≥ p n + 1 = p g ( x ) . Note that when ( p y max -p n + 1 )( h y max -h n + 1 ) > 0 , we have ∆ C L def , H ( h, x ) = max { p y max , p n + 1 } -p h = 0 . When h y max -h n + 1 ≤ 0 , we define a new hypothesis h µ such that h µ ( x, y ) = ⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ log ( e h n + 1 + µ ) y = y max log ( e h y max -µ ) y = n + 1 h x, y otherwise . , where e h y max -

.

e h n + 1 ≤ µ ≤ e h y max . Then, we can lower bound the conditional regret of hypothesis h by using ∆ C L RL2D , H ( h, x ) ≥ C L RL2D ( h ) -C ∗ L RL2D ( h µ ) for any e h y max -e h n + 1 ≤ µ ≤ e h y max : ∆ C L RL2D , H ( h, x )

By differentiating with respect to µ , we obtain that

achieves the maximum. Plugging it into the expression, we have

This can be further lower bounded by taking the minimum over h ∈ H , where the minimum is attained when e h n + 1 = e h y max = e h y for all y ∈ Y . Therefore,

This proves the inequality (8). In the case where g ( x ) = y max , we have p n + 1 = p y max . By Lemma B.2, the conditional regret of the deferral loss can be expressed as ∆ C L def , H ( h, x ) = p n + 1 -p h . If h ( x ) = n + 1 , then we have ∆ C L def , H ( h, x ) = 0 . Otherwise, when h ( x ) ≠ n + 1 , we can proceed in the similar

⎪

⎧

⎪

⎪

⎨

⎪

⎪

way as above, by defining a new hypothesis

h

µ

such that

h

µ

(

x, y

) =

log log

(

e

e

(

h

h

h

(

x, y

n

h

+

(

)

1

)

x

+

-

µ

)

µ

)

y

y

=

=

h

(

n

x

+

)

1

otherwise

.

⎩ Then, we can lower bound the conditional regret of L RL2D by using ∆ C L RL2D , H ( h, x ) ≥ C L RL2D ( h )-C ∗ L RL2D ( h µ ) , by applying the same derivation as above, modulo replacing y max with h ( x ) . This leads to the inequality (8) as well. By Theorem B.1, we complete the proof.

C Proof of Theorem 5.1

Theorem 5.1. Assume that there exists a zero error solution h ∗ ∈ H with E glyph[lscript] 0 -1 ( h ∗ ) = 0 and H is closed under scaling. Assume that lim t → 1 Ψ ( t ) = 0 . Then, the minimizability gap of comp-sum loss glyph[lscript] comp vanishes: M glyph[lscript] comp ( H ) = 0 .

Proof. By definition and the Lebesgue dominated convergence theorem, we have

D Future work

While we presented a comprehensive study of surrogate loss functions for learning to defer, our work focused on the standard single-expert and single-stage setting, aligning with previous work [Mozannar et al., 2023]. However, an interesting direction is to extend our approach to multi-expert [Verma et al., 2023] and two-stage settings [Mao et al., 2023a], which we have left for future work.

⎪

⎪

⎪

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Section 4, Section 5, Appendix A, Appendix B and Appendix C. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See Section 6.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Table 3.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: For each model training, we use an Nvidia A100 GPU.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The machine learning community has started to address the fairness implications of involving downstream decision-makers. This represents a broader impact for any learning to defer (L2D) methods.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: See Section 6.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies learning to defer, focusing on the single-stage and single-expert setting. It introduces a family of surrogate losses based on comp-sum losses [Mao et al., 2023b] and establishes their realizable H-consistency (under mild conditions). In addition, when the base loss is the logistic loss $\\Psi_{log}$ or the generalized cross entropy loss $\\Psi_{gce}$, H-consistency bounds are proven (under mild conditions) when the cost function of deferring is classification error. When the base loss is the mean absolute error loss $\\Psi_{mae}$, H-consistency bounds are proven (under mild conditions) when the cost function of deferring is general. Note that H-consistency implies Bayes consistency. The results also close an open question raised by Mozannar et al., 2023. The relationship between realizable H-consistency and H-consistency bounds is then analyzed for learning to defer. Finally, the proposed surrogates are evaluated empirically and compared with existing baselines.\n\nSTRENGTHS:\n**Originality**\n\n- A new family of surrogate losses based on comp-sum losses (focusing on $\\Psi_{log}$, $\\Psi_{gce}$, and $\\Psi_{mae}$) [Mao et al., 2023b] for learning to defer (the single-stage and single-expert setting).\n- Conditions for realizable H-consistency are identified (Theorem 4.1). In particular, $\\Psi_{log}$, $\\Psi_{gce}$, and $\\Psi_{mae}$ all satisfy the conditions. This result is more general than Mozannar et al. [2023]\n- When the cost function of deferring is classification error, H-consistency bounds are proven for base losses $\\Psi_{log}$ and $\\Psi_{gce}$ (Theorem 4.2).\n- When the cost function of deferring is general, an H-consistency bound is proven for base loss $\\Psi_{mae}$ (Theorem 4.3).\n\nThe results also close an open question raised by Mozannar et al., 2023 (Section 4.4). The relationship between realizable H-consistency and H-consistency bounds is then analyzed for learning to defer (Section 5). Related work is adequately cited and compared.\n\n**Quality**\n\nThe submission is technically sound. Claims are well supported by proofs.\n\n**Clarity**\n\nThe submission is generally clearly written and well organized.\n\n**Significance**\n\nThe work closes an open question raised in previous work. Its work to connect both realizable H-consistency and H-consistency bounds might be useful in other learning settings.\n\nWEAKNESSES:\n**Originality**\n\n- The realizable H-consistency result (Theorem 4.1) only applies to a subset of comp-sum losses.\n- The H-consistency results (Theorems 4.2 and 4.3) require case-by-case analysis. Is it possible to prove such results for all comp-sum losses?\n\n**Quality**\n\n- Some experiments in realizable settings can help confirm the realizable H-consistency result.\n\n**Clarity**\n\n- Section 5 is a bit unclear. Lines 300-302: Do you mean that in the realizable setting, all surrogate minimizability gaps vanish? Or is it only for comp-sum losses?\n\n**Significance**\n\n- The applicability of this work might be limited to learning to defer.\n\nQUESTIONS:\nBesides my concerns above, here are some other questions:\n\n1. Can constrained losses [1] be used as the base loss?\n\n2. Lines 273-274: Can negative results be proven formally?\n\n\n[1] Y. Lee, Y. Lin, and G. Wahba. Multicategory support vector machines: Theory and application to the classification of microarray data and satellite radiance data. Journal of the American Statistical Association, 99(465):67–81, 2004.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
This paper considers the problem of learning to defer (L2D),  where a classifier is allowed to defer a decision to an expert (possibly expensive to query) and trained to accurately predict while minimising the expert cost. 

A major contribution of this paper is establishing consistency guarantees for surrogate losses. 
Specifically, the authors derive a loss that encompasses existing surrogate losses. 
The authors provide sufficient conditions for the proposed surrogate loss to have realisable H consistency and for H consistency bounds. 
From the derived H consistency bounds, the authors show that certain choices of the expert cost leads to a surrogate loss that is both Bayes and realisable H consistent.

STRENGTHS:
The paper is very well written and does not have major issues in clarity.

This paper seems to provide a general  framework for analysing surrogate losses for L2D, and the established results appear to be novel. 
I am not actively working on L2D and my evaluation might not be accurate.

WEAKNESSES:
Due to my lack of experience in the field, I find it challenging to adequately adjudicate the significance of the work. 
Realisable H consistency effectively assumes that there is a perfect classifier that does not need an expert. 
This preposition seems to be strong and diminishes the point of having an expert.

QUESTIONS:
1. Why is H consistency important in L2D?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provide a framework of surrogate loss functions for learning to defer under the multi-class classification problem. By examining the deferral loss function and choosing different surrogates for the indicator functions, the authors provide a novel class of surrogate loss functions for learning to defer and prove the realizable $\\mathcal{H}$-consistency and $\\mathcal{H}$-consistency for some cases. The authors also verify their proposed loss with numerical results.\n\nSTRENGTHS:\n1. The analysis is novel and provides new insights for learning to defer literature;\n2. The paper addresses the problem of achieving $\\mathcal{H}$-consistency and realizable $\\mathcal{H}$-consistency simultaneously.\n\nWEAKNESSES:\n1. (Major) The proposed loss is not practical in many critical settings when the cost to consult an expert is high; Both terms of the derived loss function contain the exact value of $c(x, y)$ while knowing $c(x, y)$ itself needs consulting the expert if $c(x, y)$ is related to the expert's response. Thus, training a model using the proposed surrogate loss requires querying not only the true label but also the expert's response to every single sample, which is somewhat against the motivation of deriving the problem of learning to defer. See the derivation of equation (2). (Addressed, improve my rating to 6)\n2. (Minor) Some of the results are restricted to the case $c(x,y) = 1_{g(x) \\neq y}$, which is limited compared to the vast problem settings of learning to defer;\n3. (Minor) The presentation is a little confusing: there are three consistencies (Bayes consistency, $\\mathcal{H}$- consistency, and realizable $\\mathcal{H}$-consistency), while their relationships should be formally summarized into some propositions; the existing surrogate loss functions' consistencies should also be summarized into one table with indicator \"yes\", \"no\", or \"not proved\" for each consistency.\n4. (Moderate) The numerical experiments don't specify the expert algorithm; the experiments are also very restricted to the case of $c(x,y) = 1_{g(x) \\neq y}$.\n\nQUESTIONS:\n1. The major concern of mine is that the training under the proposed loss function requires the full responses of the expert to the entire training dataset, which is undesirable for the L2D problem. Previous surrogate loss functions (nominated in Section 3.3) only require $c(x, y)$ if the model chooses to defer. The learning to defer problem is proposed to ease the pain of consulting experts, while I think the loss function proposed works against that principle. Is it possible to mitigate the training cost of the model?\n2. All the remaining questions are addressed in the Weakness part and do not matter unless the first question is addressed.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes considers the setting of learning to defer: a machine learning system can choose to either classify an instance or defer the decision to an expert which incurs a variable cost. The objective is to minimize the deferral loss of the system. To solve this problem, prior work has proposed surrogate losses with certain theoretical guarantees with respect to the original deferral loss. This work proposes the first surrogate loss RL2D that is realizable H-consistent, satisfies an H-consistency bound and thus is bayes consistent. This resolves an open problem proposed from prior work. Empirically, the authors showcase that the proposed surrogate exceeds or matches prior surrogates on three different datasets.\n\nSTRENGTHS:\nOriginality: The surrogate loss proposed in the work is novel as well as the proof technique for the theoretical properties. The derivation of the surrogate loss is different from prior work, however, it is not clear how the technique can be generalized to other settings.\n\nQuality: I have verified a good portion of the theoretical derivations and they seem sound. The experimental setup follows similar protocol to prior work and is sound. The paper is very strong theoretically.\n\nClarity: very well written, clearly stating contributions of prior work and setting the stage for readers unfamiliar with the setting and the literature. Derivation and theoretical properties very well stated and easy to follow along.\n\nSignificance: The paper settles an open problem from prior work at AISTATS and in turns I believe concludes (barring any breakthroughs) a line of work on deriving surrogate losses for learning to defer. I think this is important because now the community can focus on other settings and other considerations beyond theoretical consistency properties. However, I don't think the paper has a lot to offer in terms of techniques/methods for the community beyond the learning to defer problem as the derivation rely on some algebra of the deferral loss. Therefore, I don't expect this to be widely read by the community, but will be instead read in great detail by the community working on learning to defer and related problems.\n\nWEAKNESSES:\nThere are no weaknesses with regard to the theory in this paper beyond the generalizability of the approach taken to related problem settings.\n\nHowever, the experimental setting is quite limited in terms of showcasing the behavior empirically of the newly proposed method. For the use of the surrogate in practice, it is not clear in which scenarios (if any) is the surrogate superior to prior work.\n\nMoreover, it is not clear how do the theoretical properties help in practice.\n\nQUESTIONS:\n- In which settings is the new proposed surrogate better than prior work?\n\n- How do the theoretical properties manifest in practice?\n\n\nFollow-up after author response:\n- the authors have done a good job answering the concerns, I believe this paper merits acceptance but I will maintain my original score as I am unsure about the level of impact of the paper.\n\nLIMITATIONS:\nYes they have.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}