Title: Two-Stage Learning to Defer with Multiple Experts
Venue: NeurIPS 2023
Accepted: yes
Keywords: learning to defer

learning theory
Abstract: We study a two-stage scenario for learning to defer with multiple experts, which is crucial in practice for many applications.  In this scenario, a predictor is derived in a first stage by training with a common loss function such as cross-entropy.  In the second stage, a deferral function is learned to assign the most suitable expert to each input.  We design a new family of surrogate loss functions for this scenario both in the score-based and the predictor-rejector settings and prove that they are supported by $H$-consistency bounds, which implies their Bayes-consistency.  Moreover, we show that, for a constant cost function, our two-stage surrogate losses are realizable $H$-consistent. While the main focus of this work is a theoretical analysis, we also report the results of several experiments on CIFAR-10 and SVHN datasets.

Two-Stage Learning to Defer with Multiple Experts

Abstract

We study a two-stage scenario for learning to defer with multiple experts, which is crucial in practice for many applications. In this scenario, a predictor is derived in a first stage by training with a common loss function such as cross-entropy. In the second stage, a deferral function is learned to assign the most suitable expert to each input. We design a new family of surrogate loss functions for this scenario both in the score-based and the predictor-rejector settings and prove that they are supported by H -consistency bounds, which implies their Bayes-consistency. Moreover, we show that, for a constant cost function, our two-stage surrogate losses are realizable H -consistent. While the main focus of this work is a theoretical analysis, we also report the results of several experiments on CIFAR-10 and SVHN datasets.

1 Introduction

Large language models (LLMs) have achieved a remarkable performance on diverse tasks across multiple domains, as reported in recent surveys [Wei et al., 2022, Bubeck et al., 2023]. However, their practical application faces two critical challenges: the occurrence of hallucinations , that is the generation of incorrect or misleading content, and an inefficient inference. Leveraging multiple experts can address both issues. To reduce hallucinations, one can refrain from using the original predictor in uncertain instances and defer to one of the more complex and more accurate experts. To enhance efficiency, one can derive models of different sizes distilled from the original complex model and use one of these more streamlined versions, while deferring to the more complex and less efficient ones for suitable contexts. Both problems require assigning each instance to the most suitable expert. This motivates the problem of learning to defer in the presence of multiple experts .

The scenario of single-stage learning to defer has been studied by several publications, starting with the foundational framework introduced by Cortes, DeSalvo, and Mohri [2016a,b, 2023] for learning to reject and followed by a series of studies on abstention and deferral [Madras et al., 2018, Raghu et al., 2019, Mozannar and Sontag, 2020, Wilder et al., 2021, Pradier et al., 2021, Keswani et al., 2021, Raman and Yee, 2021, Liu et al., 2022, Verma and Nalisnick, 2022, Charusaie et al., 2022, Cao et al., 2022, Verma et al., 2023, Mao et al., 2023f,b,c, Mozannar et al., 2023]. In the single-stage scenario, a predictor and a deferral function are learned simultaneously, with the deferral function determining the best expert assigned to each input instance. However, in practice, a predictor such as an LLM is already available and retraining one in conjunction with a deferral function could be prohibitively costly: depending on its size and the amount of data used, retraining could take several weeks or months. Thus, the single-stage learning to defer scenario and its solutions often do not align with the practical challenges encountered in real-world applications.

Alternative post-hoc methods have been proposed to address the learning to defer problem. Okati et al. [2021] proposed an iterative approach optimizing a predictor and a rejector over multiple epochs. Within each epoch, first the predictor is trained on points where its loss is lower than that of a human expert; second, the rejector is fitted to predict which of the predictor or the human expert has a lower loss. Narasimhan et al. [2022] suggested a post-hoc correction to the single-stage learning to defer surrogate losses, specifically the cost-sensitive softmax cross-entropy (CSS) surrogate loss in [Mozannar and Sontag, 2020] and the one-versus-all (OvA) surrogate loss in [Verma and Nalisnick, 2022] for cases where they suffer from underfitting. However, as with the single-stage learning to defer solutions, post-hoc approaches do not apply to scenarios where an existing predictor, pre-trained using a standard classification loss function such as cross-entropy, is already available.

Can we derive a principled algorithm for learning to defer with multiple experts in such scenarios? Which surrogate loss should we adopt and what consistency guarantee can we rely on? This paper deals precisely with these questions.

A key criterion for surrogate losses in learning to defer is Bayes-consistency [Zhang, 2004, Bartlett et al., 2006, Steinwart, 2007], that is minimizing the surrogate loss over the family of measurable functions leads to the minimization of the deferral loss. The surrogate losses proposed in [Mozannar and Sontag, 2020, Verma and Nalisnick, 2022] have been shown to be Bayes-consistent for deferral. However, Bayes-consistency is not relevant in learning tasks since the hypothesis set used, for example that of some family of linear functions or neural networks, never includes all measurable functions. Long and Servedio [2013] proposed a notion of realizable H -consistency, that is consistency associated with a specific hypothesis set in the realizable scenario. Mozannar et al. [2023] recently showed that existing Bayes-consistent surrogate losses in [Mozannar and Sontag, 2020, Verma and Nalisnick, 2022] are not realizable H -consistent for learning with deferral, which can pose significant challenges when learning with a restricted hypothesis set H , even for simple linear models. Instead, they proposed a new surrogate loss that is realizable H -consistent when H is closed under scaling. However, they also observed that the loss function of Madras et al. [2018], which is not Bayes-consistent, is actually realizable H -consistent. They acknowledged their inability to prove or disprove whether their proposed surrogate loss is Bayes-consistent. Consequently, it has remained an open problem to identify a surrogate loss that is both consistent and realizable-consistent.

In recent work, Verma et al. [2023] proposed the first Bayes-consistent surrogate losses in the scenario of single-stage learning to defer with multiple experts [Hemmer et al., 2022, Keswani et al., 2021, Kerrigan et al., 2021, Straitouri et al., 2022, Benz and Rodriguez, 2022]. This scenario is more attractive and significant in applications such as large language models, where multiple models are often available for deferral. However, the surrogate losses proposed by the authors do not benefit from realizable H -consistency, even in the single-expert setting, since they are a straightforward generalization of those of Mozannar and Sontag [2020] and Verma and Nalisnick [2022].

Bayes-consistency, or even realizable H -consistency for a specific hypothesis set H , is an asymptotic property, and provides no guarantee for approximate minimizers since convergence could be arbitrarily slow. More favorable guarantees, known as H -consistency bounds , were recently introduced for standard classification settings [Awasthi, Mao, Mohri, and Zhong, 2022b,a]. These guarantees are upper bounds on the target estimation loss expressed in terms of the surrogate estimation loss. They are stronger and more informative guarantees than Bayes-consistency and H -consistency because they are both hypothesis set-specific and non-asymptotic. More recently, Mao et al. [2023b] introduced a new family of surrogate losses and algorithms for the general problem of single-stage learning to defer with multiple experts that benefit from strong H -consistency bounds.

Our contributions. We study a two-stage scenario for learning to defer with multiple experts that is crucial in practice for many applications. In this scenario, a predictor is derived in a first stage by training with a common loss function such as cross-entropy. In the second stage, a deferral function is learned to assign the most suitable expert to each input. We design a new family of surrogate loss functions for this scenario both in the score-based setting (Section 3) and the predictor-rejector setting (Section 4) and prove that they are supported by H -consistency bounds, which implies their Bayes-consistency. Moreover, we show that, for a constant cost function, our two-stage surrogate losses are realizable H -consistent. While the main focus of this work is a theoretical analysis, we also report the results of several experiments on CIFAR-10 and SVHN datasets (Section 5). We give a comprehensive discussion of related work in Appendix A. We begin by providing some basic definitions and notation (Section 2).

2 Preliminaries

We consider the standard multi-class classification setting with an input space X and a set of n ≥ 2 labels Y = [ n ] , where we use the notation [ n ] to denote the set { 1 , . . . , n } . We study the scenario of learning to defer with multiple experts , where the label set Y is augmented with n e additional labels { n + 1 , . . . , n + n e } corresponding to n e pre-defined experts h 1 , . . . , h n e . In this scenario, the learner has the option of returning a label y ∈ Y , which represents the category predicted, or a label y = n + j , j ≥ 1 , in which case it is deferring to expert h j . This setting is referred to as the score-based setting [Mozannar and Sontag, 2020, Cao et al., 2022, Mao et al., 2023f], since the deferral corresponds to extra n e scoring functions. An alternative setting is the predictor-rejector setting [Cortes et al., 2016a, 2023, Mohri et al., 2023, Mao et al., 2023c], where the deferral function is selected from a separate family of functions R . We introduce that setting and include the corresponding results in Section 4 for completeness.

We denote by Y = [ n + n e ] the augmented label set and consider a hypothesis set H of functions mapping from X × Y to R . The prediction associated by h ∈ H to an input x ∈ X is denoted by h ( x ) and defined as the element in Y with the highest score, h ( x ) = argmax y ∈[ n + n e ] h ( x, y ) , with an arbitrary but fixed deterministic strategy for breaking ties. We denote by H all the family of all measurable functions.

The deferral loss function L def is defined as follows for any h ∈ H and ( x, y ) ∈ X × Y :

Thus, the loss incurred coincides with the standard zero-one classification loss when h ( x ) , the label predicted, is in Y . Otherwise, when h ( x ) is equal to n + j , the loss incurred is c j ( x, y ) , the cost of deferring to expert h j . Let ¯ c j ( x, y ) = 1 -c j ( x, y ) . We will denote by c j ≥ 0 and c j ≤ 1 finite lower and upper bounds on the cost ¯ c j , that is ¯ c j ( x, y ) ∈ [ c j , c j ] for all ( x, y ) ∈ X × Y . There are many possible choices for these costs. Our analysis for Theorem 1, Corollary 2, Theorem 6 is general and requires no assumption other than their boundedness. One natural choice is to define cost c j as a function of expert h j 's inaccuracy, for example c j ( x, y ) = α j 1 h j ( x )≠ y + β j , with α j , β j > 0 , where h j ( x ) is the prediction made by h j th for input x . Typically, the hyperparameter α j has two potential values: zero or one. When α j is set to one, the first term of the formulation pertains to the inaccuracy of expert expert h j . Conversely, with α j set to zero, the first term vanishes, focusing solely on the inference cost. Theorems 5 and 7 are analyzed under this assumption. The β j in the second term corresponds to the inference cost incurred by expert h j .

Given a distribution D over X × Y , we will denote by E L def ( h ) the expected deferral loss of a hypothesis h ∈ H , E L def ( h ) = E ( x,y )∼ D [ L def ( h, x, y )] , and by E ∗ L def ( H ) = inf h ∈ H E L def ( h ) its infimum or best-in-class expected loss. We will adopt similar definitions for other loss functions.

Given a hypothesis set H , an H -consistency bound [Awasthi et al., 2021a,b, 2022b,a, 2023a,b, Mao et al., 2023h,d,e, Zheng et al., 2023, Mao et al., 2023a,g] for a surrogate loss glyph[lscript] 1 of a target loss function glyph[lscript] 2 is an inequality of the form

where Γ ∶ R + → R + is a non-decreasing function with Γ ( 0 ) = 0 and where M glyph[lscript] ( H ) is the minimizability gap for the hypothesis set H and loss function glyph[lscript] . M glyph[lscript] ( H ) is defined as the difference of the best-in-class expected loss and that of the expected pointwise infimum loss: M glyph[lscript] ( H ) = E ∗ glyph[lscript] ( H ) -E x [ inf h ∈ H E y ∣ x [ glyph[lscript] ( h, x, y )]] . By the super-additivity of the infimum, the minimizability gap is always non-negative. The minimizability gap vanishes when the best-in-class error E ∗ glyph[lscript] ( H ) coincides with the Bayes error E ∗ glyph[lscript] ( H all ) , in particular when H = H all [Awasthi et al., 2022a,b].

Thus, the H -consistency bound (2) relates the minimization of the estimation error for the surrogate loss glyph[lscript] 1 to that of the target loss glyph[lscript] 2 in a quantitative way. It is a stronger and more informative guarantee than Bayes-consistency which implies Bayes-consistency, as can be seen by setting H = H all .

3 Two-stage H -consistent surrogate loss

In this section, we consider an important two-stage scenario for learning to defer with multiple experts. This is a critical scenario in practice for many applications where a predictor is already available, as a result of training with a loss function glyph[lscript] supported by H -consistency bounds, such as the logistic loss (first stage). The logistic loss coincides with the cross-entropy loss when a softmax activation is applied to the output of a neural network. The problem then consists of learning a deferral function (second stage) to assign the most suitable expert to each input instance.

We first design a new family of surrogate losses for this two-stage scenario (Section 3.1). Next, we show that our surrogate losses benefit from H -consistency bounds (Section 3.2). As a by-product, we prove H -consistency bounds in standard multi-class classification, where H denotes hypothesis sets with a fixed scoring function (Section 3.3). These bounds have not been studied before and can be of independent interest in other consistency studies. Moreover, we show that, for a constant cost function, our two-stage surrogate losses are realizable H -consistent (Section 3.4).

3.1 General surrogate losses

Ahypothesis set H of functions mapping from X ×[ n + n e ] to R can be decomposed as H = H p × H d , where H p denotes the hypothesis set spanned by the first n scores, used for prediction, and H d the hypothesis set spanned by the final n e scores, used for deferral. Thus, any h ∈ H can be written as a pair h = ( h p , h d ) with h p ∈ H p and h d ∈ H d .

Let glyph[lscript] be a surrogate loss for standard multi-class classification with n classes. We consider the following two-stage scenario: in the first stage, h p is learned using the surrogate loss glyph[lscript] 1 ; in the second stage, h d is learned using a surrogate loss L h p that depends on the prediction function h p learned in the first stage.

To any h d ∈ H d , we associate a hypothesis h d defined over ( n e + 1 ) classes { 0 , 1 , . . . , n e } by h d ( x, 0 ) = max y ∈ Y h p ( x, y ) , that is the maximal score assigned by h p to its predicted label, and h d ( x, j ) = h d ( x, j ) for j ∈ [ n e ] . We can then define our suggested surrogate loss for the second stage as follows:

where glyph[lscript] 2 ( h d , x, j ) is a surrogate loss for standard multi-class classification with ( n e + 1 ) categories { 0 , 1 , . . . , n e } . Intuitively, the indicator term 1 h ( x )≠ n + j in the deferral loss (1) penalizes h d ( x, j ) when it has a small value. Similarly, for a standard surrogate loss glyph[lscript] 2 ( h d , x, j ) such as the logistic loss, it penalizes h d ( x, j ) when it has a small value as well. In Table 2, we present a summary of examples of such second-stage surrogate losses, where glyph[lscript] 2 is selected from common surrogate losses in standard multi-class classification defined in Table 1. A detailed derivation is presented in Appendix B.

From the point of view of the second stage, x ↦ h d ( x, 0 ) = max y ∈ Y h p ( x, y ) is a fixed function. We will denote by H d the family of hypotheses h d ∶ X ×{ 0 , 1 , . . . , n e } → R whose first scoring function, h d (⋅ , 0 ) , is fixed and not to be learned in the second stage.

Our formulation bears some similarity with the design of a surrogate loss function for rejectors in [Cortes et al., 2016a, 2023] for learning with rejection in binary classification, where the cost is a

constant. However, our surrogate loss is tailored to accommodate a general cost function depending on both x and y for deferral, in contrast with a constant one, and it allows for multiple deferral options, as opposed to only one rejection option.

3.2 H -consistency bounds for two-stage surrogate losses

In this section, we provide strong guarantees for two-stage surrogate losses, provided that the firststage loss function glyph[lscript] 1 admits an H p -consistency bound, and the second-stage surrogate glyph[lscript] 2 admits an H d -consistency bound.

Theorem 1 ( H -consistency bounds for score-based two-stage surrogates ) . Assume that glyph[lscript] 1 admits an H p -consistency bound and glyph[lscript] 2 admits an H d -consistency bound with respect to the multi-class zero-one classification loss glyph[lscript] 0 -1 respectively. Thus, there are non-decreasing concave functions Γ 1 and Γ 2 such that, for all h p ∈ H p and h d ∈ H d , we have

Then, the following holds for all h ∈ H :

Furthermore, constant factors ( 1 + ∑ n e j = 1 c j ) and 1 ∑ ne j = 1 c j can be removed when Γ 2 is linear.

The proof is given in Appendix D. It consists of expressing the conditional regret of the deferral loss as the sum of two regrets, first by minimizing h d for a fixed h p and then by minimizing h p . Subsequently, we show how each regret can be upper-bounded in terms of the conditional regret of each stage's surrogate loss, leveraging the H p -consistency bound of glyph[lscript] 1 and H d -consistency bound of glyph[lscript] 2 with respect to the zero-one loss. This, in conjunction with the concavity of functions Γ 1 and Γ 2 , establishes our H -consistency bounds.

Thus, the theorem provides a strong guarantee for the two-stage surrogate losses. A specific instance of Theorem 1 holds for the case where E ∗ glyph[lscript] 1 ( H p ) = E ∗ glyph[lscript] 1 ( H all ) and E ∗ L hp ( H d ) = E ∗ L hp ( H all ) , ensuring that the Bayes-error coincides with the best-in-class error and, consequently, M glyph[lscript] 1 ( H p ) = M L hp ( H d ) = 0 . Given Theorem 1 and the non-negativity property of M L def ( H ) , we can derive the following corollary.

Corollary 2. Assume that glyph[lscript] satisfies the same assumption as in Theorem 1. Then, for all h ∈ H and any distribution such that E ∗ glyph[lscript] 1 ( H p ) = E ∗ glyph[lscript] 1 ( H all ) and E ∗ L hp ( H d ) = E ∗ L hp ( H all ) , we have

where the constant factors ( 1 + ∑ n e j = 1 c j ) and 1 ∑ ne j = 1 c j can be removed when Γ 2 is linear.

Corollary 2 implies that when the estimation error of the first-stage surrogate loss, E glyph[lscript] 1 ( h p )-E ∗ glyph[lscript] 1 ( H p ) , is reduced to glyph[epsilon1] 1 , and the estimation error of the second-stage surrogate loss, E L hp ( h d ) -E ∗ L hp ( H d ) , is reduced to glyph[epsilon1] 2 , the estimation error of the deferral loss, E L def ( h ) -E ∗ L def ( H ) , is upper-bound by

The common surrogate losses mentioned earlier all satisfy the first-stage requirement; however, it was unclear if they would meet the second-stage criterion since the H d -consistency bound is for hypothesis sets H d with a fixed first scoring function. This has not been previously studied in the literature. In the next section, we prove for the first time that common multi-class surrogate losses,

glyph[lscript]

2

L

h

p

glyph[lscript]

exp

1

hp

(

x

)=

y

∑

n

e

i

=

1

e

h

(

x,n

+

i

)-

y

∈

Y

h

(

x,y

)

+ ∑

n

e

j

=

1

c

j

(

x, y

)[∑

n

e

i

=

1

,i

≠

j

e

h

(

x,n

+

i

)-

h

(

x,n

+

j

)

+

e

y

∈

Y

h

(

x,y

)-

h

(

x,n

+

j

)

]

glyph[lscript]

log

-

1

hp

(

x

)=

y

log

(

e

y

∈

Y

h

(

x,y

)

e

y

∈

Y

h

(

x,y

)

+∑

ne

i

=

1

e

h

(

x,n

+

i

)

) - ∑

n

e

j

=

1

c

j

(

x, y

)

log

(

e

h

(

x,n

+

j

)

e

y

∈

Y

h

(

x,y

)

+∑

ne

i

=

1

e

h

(

x,n

+

i

)

)

glyph[lscript]

gce

1

hp

(

x

)=

y

1

α

[

1

-[

e

y

∈

Y

h

(

x,y

)

e

y

∈

Y

h

(

x,y

)

+∑

ne

i

=

1

e

h

(

x,n

+

i

)

]

α

] + ∑

n

e

j

=

1

c

j

(

x, y

)

1

α

[

1

-[

e

h

(

x,n

+

j

)

e

y

∈

Y

h

(

x,y

)

+∑

ne

i

=

1

e

h

(

x,n

+

i

)

]

α

]

glyph[lscript]

mae

1

hp

(

x

)=

y

[

1

-

e

y

∈

Y

h

(

x,y

)

e

y

∈

Y

h

(

x,y

)

+∑

ne

i

=

1

e

h

(

x,n

+

i

)

] + ∑

n

e

j

=

1

c

j

(

x, y

)[

1

-

e

h

(

x,n

+

j

)

e

y

∈

Y

h

(

x,y

)

+∑

ne

i

=

1

e

h

(

x,n

+

i

)

]

such as the logistic loss, satisfy this requirement and can be incorporated into both the first and second stage. Hence, based on [Mao et al., 2023h, Theorem 1] and Theorem 3 in Section 3.3, when using logistic loss in both stages, the concave functions are Γ 1 ( t ) = Γ 2 ( t ) = √ 2 t , and thus Corollary 2 yields the following H -consistency bound:

In particular, the bound implies the Bayes-consistency of the two-stage surrogate loss when glyph[lscript] 1 = glyph[lscript] 2 = glyph[lscript] log . Similarly, for other choices of glyph[lscript] 1 and glyph[lscript] 2 defined in Table 1, the two-stage surrogate loss benefits from an H -consistency bound and is also Bayes-consistent.

3.3 H -consistency bounds for standard surrogate loss functions

In this section, we seek to derive H -consistency bounds for common surrogate losses defined in Table 1 in the standard multi-class classification scenario. Recall that the first scoring function of hypotheses in H d is the function max y ∈ Y h p (⋅ , y ) . Here, for any given function λ mapping from X to R , we define the hypothesis set H augmented by λ in a similar way, that is to any h ∈ H we associate a hypothesis h ∈ H defined by h ( x, 0 ) = λ ( x ) and h ( x, j ) = h ( x, j ) for j ≥ 1 . These H -consistency bounds offer strong guarantees when the loss functions in Table 1 are used in the second stage of the two-stage learning to defer surrogate losses (3) instantiated in Table 2. We believe that these results are of independent interest and can admit other applications in the study of H -consistency bounds. As with [Mao et al., 2023h], we assume that the hypothesis set H is symmetric and complete . A hypothesis set is said to be symmetric if there exists a family F of functions f mapping from X to R such that {[ h ( x, 1 ) , . . . , h ( x, n )]∶ h ∈ H } = {[ f 1 ( x ) , . . . , f n ( x )]∶ f 1 , . . . , f n ∈ F } , for any x ∈ X . A hypothesis set H is said to be complete if the set of scores it generates spans R , that is, { h ( x, y )∶ h ∈ H } = R , for any ( x, y ) ∈ X × Y .

Note that for a symmetric and complete H , the associated H is not symmetric and complete. Therefore, the proof of Mao et al. [2023h] cannot be generalized to our setting. Our proofs are presented in Appendix E. We give a new method for upper bounding the conditional regret of the zero-one loss by that of a surrogate loss. To achieve this, we upper bound the minimal conditional surrogate loss by the conditional loss of a carefully constructed hypothesis in H denoted by h µ . The resulting softmax S µ of this hypothesis only differs from the original softmax S corresponding to h on exactly two of the labels.

Theorem 3 ( H -consistency bounds ) . Assume that H is symmetric and complete. Then, for any function λ mapping from X to R , hypothesis h in the associated hypothesis set H and any distribution, the following inequality holds:

Let us underscore that our proof technique is novel and distinct from the approach used in [Mao et al., 2023h]. Their method is tailored for hypothesis sets where each score can span across R . This is not applicable in our context where the hypothesis set adheres to a predefined scoring function.

In their proof, to set an upper bound on the estimation error of the zero-one loss using that of the surrogate loss, they select an auxiliary function h µ for any hypothesis h . This function is contingent on the distinct scores of h . Subsequently, the authors choose an optimal µ to set these bounds. Nevertheless, if any of h 's scores are fixed, an optimal µ does not exist, preventing the establishment of a meaningful bound. Instead, our new proof method overcomes this limitation by choosing h µ based on the softmax, as the softmax corresponding to the label zero can still vary due to the influence of changes in other scores, even when the scoring function on label zero is fixed.

3.4 Realizable H -consistency

Recently, Mozannar et al. [2023] showed that even in the straightforward single-expert setting, existing Bayes-consistent single-stage surrogate losses [Mozannar and Sontag, 2020, Verma and Nalisnick, 2022] are not realizable H -consistent [Long and Servedio, 2013, Zhang and Agarwal, 2020] for learning with deferral. This can pose significant challenges when learning with a restricted hypothesis set H , even for simple linear models. Instead, they proposed a new surrogate loss that is realizable H -consistent when H is closed under scaling , meaning that it satisfies the condition h ∈ H ⇒ τh ∈ H for all τ in the set of real numbers. However, they stated that they could not prove or disprove whether their proposed surrogate loss is Bayes-consistent. Consequently, it has become crucial to identify a surrogate loss that is both consistent and realizable-consistent, which has remained an open problem.

Definition 4 ( Realizable H -consistency ) . A surrogate loss L is considered a realizable H -consistent loss function for the deferral loss L def if, for any distribution that is H -realizable , that is, there exists a zero loss solution h ∗ ∈ H with E L def ( h ∗ ) = 0 , optimizing the surrogate loss results in obtaining the zero-error solution:

In the following result, we show that our two-stage surrogate losses are realizable H -consistent. Combined with their Bayes-consistency properties, which have already been established in Section 3.2, we effectively find surrogate losses that are both Bayes-consistent and realizable consistent in the multi-expert setting, including the single-expert setting as a special case. For simplicity, here, we study the case where glyph[lscript] 1 = glyph[lscript] 2 = glyph[lscript] log , a similar proof holds for other choices of glyph[lscript] 1 and glyph[lscript] 2 defined in Table 1. The proof is included in Appendix F.

Theorem 5 ( Realizable H -consistency for score-based two-stage surrogates ) . Assume that H is closed under scaling and c j ( x, y ) = β j , ∀( x, y ) ∈ X × Y . Let glyph[lscript] 1 and glyph[lscript] 2 be the logistic loss. Let ˆ h p be the minimizer of E glyph[lscript] 1 and ˆ h d be the minimizer of E L ˆ hp such that E L ˆ hp ( ˆ h d ) = min h E L hp ( h d ) . Then, the following equality holds for any ( H , R ) -realizable distribution,

Theorem 5 suggests that when the estimation error of the first-stage surrogate loss, E glyph[lscript] 1 ( h n p ) -E ∗ glyph[lscript] 1 ( H p ) n →+∞ /leftrightline /leftrightline /leftrightline→ 0 , and the estimation error of the second-stage surrogate loss, E L hp ( h n d ) -E ∗ L hp ( H d ) n →+∞ /leftrightline /leftrightline /leftrightline→ 0 , the estimation error of the deferral loss, E L def ( h n ) -E ∗ L def ( H ) n →+∞ /leftrightline /leftrightline /leftrightline→ 0 . This result demonstrates that our two-stage surrogate losses are not only Bayes-consistent, but also realizable H -consistent when only the inference cost ( β j ) exists.

4 Predictor-rejector setting

The results of the previous sections were all given for the score-based setting. We note that another popular setting in learning with deferral/abstention is the predictor-rejector setting [Cortes et al., 2016a, 2023], where the deferral corresponds to a separate function R instead of extra scores. For completeness, we introduce this setting as well. Here too, we design a family of two-stage surrogate losses benefiting from both ( H , R ) -consistency bounds and realizable consistency. For simplicity, we overload the notation as with score-based setting based on the context.

Let H be a hypothesis set of prediction functions mapping from X × Y to R . The label predicted for x ∈ X using a hypothesis h ∈ H is denoted by h ( x ) and defined as one with the highest score,

h ( x ) = argmax y ∈ Y h ( x, y ) , with an arbitrary but fixed deterministic strategy for breaking ties. Let R be a family of deferring functions mapping from X to R n e , where n e is the number of experts. A deferral r = ( r 1 , . . . , r n e ) ∈ R is used to defer the prediction on input x to the j th expert h j if r j ( x ) ≤ 0 and r j ( x ) < min n e i = 1 ,i ≠ j r i ( x ) , in which case a cost c j ( x, y ) = 1 -¯ c j ( x, y ) ∈ [ 1 -c j , 1 -c j ] is incurred with 0 < c j ≤ c j ≤ 1 . A natural choice of the cost is c j ( x, y ) = α j 1 h j ( x )≠ y + β j , where α j , β j > 0 and h j is the prediction of the j th expert. The β j in the second term corresponds to the inference cost incurred by expert h j . Let r 0 = 0 and define r ( x ) = 0 if r 0 ( x ) < min j ∈[ n e ] r j ( x ) ; otherwise, r ( x ) = argmin j ∈[ n e ] r j ( x ) , with an arbitrary but fixed deterministic strategy for breaking ties. The learning to defer loss L def with n e experts is defined as follows for any ( h, r ) ∈ H × R and ( x, y ) ∈ X × Y :

Given a distribution D over X × Y , we will denote by E L def ( h, r ) the expected deferral loss of a predictor h ∈ H and a deferral r ∈ R , E L def ( h, r ) = E ( x,y )∼ D [ L def ( h, r, x, y )] , and by E ∗ L def ( H , R ) = inf h ∈ H ,r ∈ R E L def ( h, r ) its infimum or best-in class expected loss. We will adopt similar definitions for other loss functions. We denote by M L ( H , R ) = E ∗ L ( H , R ) -E x [ inf h ∈ H ,r ∈ R E y ∣ x [ L ( h, r, x, y )]] the minimizability gap for hypothesis sets ( H , R ) and a loss function L .

Let glyph[lscript] 1 be a surrogate loss for standard multi-class classification with n classes. We consider the following two-stage scenario: in the first stage, a predictor h is learned using the surrogate loss glyph[lscript] 1 ; in the second stage, r is learned using a surrogate loss L h that depends on the prediction function h learned in the first stage.

To any r ∈ R , we associate a hypothesis r defined over ( n e + 1 ) classes { 0 , 1 , . . . , n e } by r ( x, 0 ) = 0 , that is zero scoring function, and r ( x, j ) = -r j ( x ) for j ∈ [ n e ] . We can then define our suggested surrogate loss for the second stage:

Here, glyph[lscript] 2 ( r, x, j ) is a surrogate loss for standard multi-class classification with ( n e + 1 ) categories { 0 , 1 , . . . , n e } . Intuitively, the indicator term 1 r ( x )≠ j in the deferral loss penalizes r j ( x ) when it has a large value. However, a standard surrogate loss glyph[lscript] 2 ( r, x, j ) such as the logistic loss penalizes r ( x, j ) when it has a small value. This is why we use a negative sign in the definition of r to maintain consistency between the definitions of L h and L def . In Table 3, we present a summary of examples of such second-stage surrogate losses, where glyph[lscript] 2 is selected from common surrogate losses in standard multi-class classification defined in Table 1. A detailed derivation is presented in Appendix C.

From the point of view of the second stage, we will denote by R the family of hypotheses r ∶ X × { 0 , 1 , . . . , n e } → R whose first scoring function, r (⋅ , 0 ) , is zero function and will not be learned in the second stage. We will provide strong guarantees for two-stage surrogate losses, provided that the first-stage loss function glyph[lscript] 1 admits an H -consistency bound, and the second-stage loss function glyph[lscript] 2 admits an R -consistency bound.

Theorem 6 ( ( H , R ) -consistency bounds for predictor-rejector two-stage surrogates ) . Assume that glyph[lscript] 1 admits an H -consistency bound and glyph[lscript] 2 admits an R -consistency bound with respect to the multi-class zero-one classification loss glyph[lscript] 0 -1 respectively. Thus, there are non-decreasing concave

functions Γ 1 and Γ 2 such that, for all h ∈ H and r ∈ R , we have

Then, the following holds for all h ∈ H and r ∈ R :

where the constant factors ( 1 + ∑ n e j = 1 c j ) and 1 ∑ ne j = 1 c j can be removed when Γ 2 is linear.

As with the score-based setting, a specific instance of Theorem 6 holds for the case where E ∗ glyph[lscript] 1 ( H ) = E ∗ glyph[lscript] 1 ( H all ) and E ∗ L h ( R ) = E ∗ L h ( R all ) , ensuring that the Bayes-error coincides with the best-in-class error and, consequently, M glyph[lscript] 1 ( H ) = M L h ( R ) = 0 . In these cases, when the estimation error of the first-stage surrogate loss, E glyph[lscript] 1 ( h ) -E ∗ glyph[lscript] 1 ( H ) , is reduced to glyph[epsilon1] 1 , and the estimation error of the second-stage surrogate loss, E L h ( r ) -E ∗ L h ( R ) , is reduced to glyph[epsilon1] 2 , the estimation error of the deferral loss, E L def ( h, r ) -E ∗ L def ( H , R ) , is upper bounded by

Next, we show that our two-stage surrogate losses are realizable ( H , R ) -consistent. We say that the distribution is ( H , R ) -realizable , if there exists a zero error solution ( h ∗ , r ∗ ) ∈ H × R with E L def ( h ∗ , r ∗ ) = 0 .

Theorem 7 ( Realizable ( H , R ) -consistency for predictor-rejector two-stage surrogates ) . Assume that H and R is closed under scaling and c j ( x, y ) = β j , ∀( x, y ) ∈ X × Y . Let glyph[lscript] 1 and glyph[lscript] 2 be the logistic loss. Let ˆ h be the minimizer of E glyph[lscript] 1 and ˆ r be the minimizer of E L ˆ h . Then, the following holds for any ( H , R ) -realizable distribution,

The proof is included in Appendix H. Theorem 7 suggests that the two-stage surrogate loss is realizable consistent: when the estimation error of the first-stage surrogate loss E glyph[lscript] 1 ( h n )-E ∗ glyph[lscript] 1 ( H ) n →+∞ /leftrightline /leftrightline /leftrightline→ 0 , and the estimation error of the second-stage surrogate loss E L h ( r n ) -E ∗ L h ( R ) n →+∞ /leftrightline /leftrightline /leftrightline→ 0 , the estimation error of the deferral loss, E L def ( h n , r n ) -E ∗ L def ( H , R ) n →+∞ /leftrightline /leftrightline /leftrightline→ 0 . By Theorem 6 and Theorem 7, in the predictor-rejector setting, we also effectively find both Bayes-consistent and realizable consistent surrogate losses with multiple experts when only the inference cost ( β j ) exists.

Note that while Sections 3 and 4 both propose new two-stage algorithms based on H -consistent surrogate losses, they differ in an important way. Section 3 learns with deferral in a score-based framework, where deferral is associated with extra scores. In contrast, Section 4 learns with deferral in a predictor-rejector setting, where deferral corresponds to a separate function. These represent two distinct learning frameworks that have been studied in the literature. Deriving consistent surrogate losses in the predictor-rejector setting has historically been challenging for traditional single-stage scenarios, leading many to opt for the score-based approach.

We should also highlight that our H -consistency bounds in Theorems 1 and 6 can be used to derive finite sample estimation bounds for the minimizer of the surrogate loss over a hypothesis set H . This is achieved by upper bounding the estimation error of the minimizer of the surrogate loss using standard Rademacher complexity bounds (see [Mao et al., 2023h]).

5 Experiments

In this section, we report the results of our experiments on CIFAR-10 [Krizhevsky, 2009] and SVHN [Netzer et al., 2011] datasets to test the effectiveness of our proposed algorithms for two-stage

learning to defer with multiple experts. We evaluated the overall accuracy of the learned pairs of predictor and deferral model across different scenarios involving varying the number of experts, where the predictor is pre-learned in the first stage and the deferral is subsequently learned using our proposed surrogate loss. We find that as the number of experts increases, the overall accuracy of the learned pairs also increases, in both scenarios with zero and non-zero base costs. This observation highlights the significance of using a multiple expert framework in our approach and the effectiveness of our surrogate loss within the framework.

We used ResNet architectures [He et al., 2016] for the prediction model, the deferral model and expert models. More precisely, we used ResNet4 for both the predictor and the deferral. We adopted three expert models: ResNet10 , ResNet16 , ResNet28 with increasing capacity. For training, we used the Adam optimizer [Kingma and Ba, 2014] with a batch size of 128 and weight decay 1 × 10 -4 . Training was run for 15 epochs for SVHN and 50 epochs for CIFAR-10 with the default learning rate. No data augmentation was used in our experiments. We used our two-stage surrogate loss (3) with the logistic loss glyph[lscript] = glyph[lscript] log to train the deferral model ResNet4 , with a pre-learned predictor ResNet4 trained using logistic loss. A check mark indicates the presence of a base cost in the cost function, whereas a cross mark signifies its absence. We first set the cost function to be 1 h j ( x )≠ y without a base cost. Next, for the experimental results shown in the last two row of Table 4, we chose base costs β j associated with each expert model as: 0 . 1 , 0 . 12 , 0 . 14 increasing with model capacity for SVHN and 0 . 3 , 0 . 32 , 0 . 34 increasing with model capacity for CIFAR-10. A base cost value that is close to the misclassification loss can strike a balance between improving accuracy and maintaining the ratio of deferral. We observed that other neighboring values lead to similar results. Note that the accuracy here refers to the overall accuracy of the learned pairs of predictor and deferral model. It is related to the deferral loss. Specifically, in the absence of the base cost, the accuracy aligns precisely with one minus the expected deferral loss. The results of Table 4 demonstrate the effectiveness of our proposed algorithms for two-stage learning to defer with multiple experts.

To the best of our knowledge, our study pioneers the exploration of a two-stage learning approach for deferral, a framework that is essential in numerous practical applications. Thus, we are unaware of any established baselines within this context.

It is important to underscore the differences between our learning scenario and those presented in [Okati et al., 2021, Narasimhan et al., 2022]. While both of them involve two phases, their methodologies are considerably different from ours. Okati et al. [2021] required conditional probabilities paired with loss estimates from the expert-a component not available in our framework, as emphasized by Mozannar et al. [2023]. On the other hand, Narasimhan et al. [2022] proposed a post-hoc correction for single-stage learning to defer surrogate losses. This approach, however, is not applicable to a pre-trained predictor from the standard multi-class classification. In contrast, our work focuses on enhancing the pre-trained predictor within the standard framework.

A limitation of our study is that the cost function used within the deferral loss is not fixed, and is typically determined through cross-validation in practice. There exists potential to introduce a principled method for selecting the cost function, which we have reserved for future research.

6 Conclusion

We introduced a novel family of surrogate loss functions and algorithms for a crucial two-stage learning to defer approach with multiple experts. We proved that these surrogate losses are supported by H -consistency bounds and established their realizable H -consistency properties for a constant cost function. This work paves the way for comparing different surrogate losses and cost functions within our framework. Further exploration, both theoretically and empirically, holds the potential to identify optimal choices for these quantities across diverse tasks.

References

A. E. Acar, A. Gangrade, and V. Saligrama. Budget learning via bracketing. In International Conference on Artificial Intelligence and Statistics , pages 4109-4119, 2020.

Awasthi, N. Frank, A. Mao, M. Mohri, and Y. Zhong. Calibration and consistency of adversarial surrogate losses. In Advances in Neural Information Processing Systems , 2021a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. A finer calibration analysis for adversarial robustness. arXiv preprint arXiv:2105.01550 , 2021b.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. Multi-class H -consistency bounds. In Advances in neural information processing systems , 2022a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. H -consistency bounds for surrogate loss minimizers. In International Conference on Machine Learning , 2022b.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. Theoretically grounded loss functions and algorithms for adversarial robustness. In International Conference on Artificial Intelligence and Statistics , pages 10077-10094, 2023a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. DC-programming for neural network optimizations. Journal of Global Optimization , 2023b.

Bansal, B. Nushi, E. Kamar, E. Horvitz, and D. S. Weld. Is the most accurate ai the best teammate? optimizing ai for teamwork. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 11405-11414, 2021.

L. Bartlett and M. H. Wegkamp. Classification with a reject option using a hinge loss. Journal of Machine Learning Research , 9(8), 2008.

L. Bartlett, M. I. Jordan, and J. D. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association , 101(473):138-156, 2006.

L. C. Benz and M. G. Rodriguez. Counterfactual inference of second opinions. In Uncertainty in Artificial Intelligence , pages 453-463. PMLR, 2022.

Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023.

Cao, T. Cai, L. Feng, L. Gu, J. Gu, B. An, G. Niu, and M. Sugiyama. Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. In Advances in neural information processing systems , 2022.

Charoenphakdee, Z. Cui, Y. Zhang, and M. Sugiyama. Classification with rejection based on cost-sensitive classification. In International Conference on Machine Learning , pages 1507-1517, 2021.

M.-A. Charusaie, H. Mozannar, D. Sontag, and S. Samadi. Sample efficient learning of predictors that complement humans. In International Conference on Machine Learning , pages 2972-3005, 2022.

Chow. An optimum character recognition system using decision function. IEEE T. C. , 1957.

Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on information theory , 16(1):41-46, 1970.

Cortes, G. DeSalvo, and M. Mohri. Learning with rejection. In International Conference on Algorithmic Learning Theory , pages 67-82, 2016a.

Cortes, G. DeSalvo, and M. Mohri. Boosting with abstention. In Advances in Neural Information Processing Systems , pages 1660-1668, 2016b.

Cortes, G. DeSalvo, and M. Mohri. Theory and algorithms for learning with rejection in binary classification. Annals of Mathematics and Artificial Intelligence , to appear, 2023.

De, P. Koley, N. Ganguly, and M. Gomez-Rodriguez. Regression under human assistance. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 2611-2620, 2020.

El-Yaniv et al. On the foundations of noise-free selective classification. Journal of Machine Learning Research , 11(5), 2010.

Gangrade, A. Kag, and V. Saligrama. Selective classification via one-sided prediction. In International Conference on Artificial Intelligence and Statistics , pages 2179-2187, 2021.

Gao, M. Saar-Tsechansky, M. De-Arteaga, L. Han, M. K. Lee, and M. Lease. Human-ai collaboration with bandit feedback. arXiv preprint arXiv:2105.10614 , 2021.

Geifman and R. El-Yaniv. Selective classification for deep neural networks. In Advances in neural information processing systems , 2017.

Geifman and R. El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In International conference on machine learning , pages 2151-2159, 2019.

Grandvalet, A. Rakotomamonjy, J. Keshet, and S. Canu. Support vector machines with a reject option. In Advances in neural information processing systems , 2008.

He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Hemmer, S. Schellhammer, M. Vössing, J. Jakubik, and G. Satzger. Forming effective human-ai teams: Building machine learning models that complement the capabilities of multiple experts. arXiv preprint arXiv:2206.07948 , 2022.

Hemmer, L. Thede, M. Vössing, J. Jakubik, and N. Kühl. Learning to defer with limited expert predictions. arXiv preprint arXiv:2304.07306 , 2023.

Herbei and M. Wegkamp. Classification with reject option. Can. J. Stat. , 2005.

Joshi, S. Parbhoo, and F. Doshi-Velez. Pre-emptive learning-to-defer for sequential medical decision-making under uncertainty. arXiv preprint arXiv:2109.06312 , 2021.

T. Kalai, V. Kanade, and Y. Mansour. Reliable agnostic learning. Journal of Computer and System Sciences , 78(5):1481-1495, 2012.

Kamar, S. Hacker, and E. Horvitz. Combining human and machine intelligence in large-scale crowdsourcing. In AAMAS , pages 467-474, 2012.

Kerrigan, P. Smyth, and M. Steyvers. Combining human predictions with model probabilities via confusion matrices and calibration. Advances in Neural Information Processing Systems , 34: 4421-4434, 2021.

Keswani, M. Lease, and K. Kenthapadi. Towards unbiased and accurate deferral to multiple experts. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , pages 154-165, 2021.

P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Kleinberg, H. Lakkaraju, J. Leskovec, J. Ludwig, and S. Mullainathan. Human decisions and machine predictions. The quarterly journal of economics , 133(1):237-293, 2018.

Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Toronto University, 2009.

Kuznetsov, M. Mohri, and U. Syed. Multi-class deep boosting. In Advances in Neural Information Processing Systems , pages 2501-2509, 2014.

Liu, B. Gallego, and S. Barbieri. Incorporating uncertainty in learning to defer algorithms for safe computer-aided diagnosis. Scientific reports , 12(1):1762, 2022.

Long and R. Servedio. Consistency versus realizable H-consistency for multiclass classification. In International Conference on Machine Learning , pages 801-809, 2013.

Madras, E. Creager, T. Pitassi, and R. Zemel. Learning adversarially fair and transferable representations. arXiv preprint arXiv:1802.06309 , 2018.

Mao, M. Mohri, and Y. Zhong. H-consistency bounds: Characterization and extensions. In Advances in Neural Information Processing Systems , 2023a.

Mao, M. Mohri, and Y. Zhong. Principled approaches for learning to defer with multiple experts. arXiv preprint arXiv:2310.14774 , 2023b.

Mao, M. Mohri, and Y. Zhong. Predictor-rejector multi-class abstention: Theoretical analysis and algorithms. arXiv preprint arXiv:2310.14772 , 2023c.

Mao, M. Mohri, and Y. Zhong. H-consistency bounds for pairwise misranking loss surrogates. In International conference on Machine learning , 2023d.

Mao, M. Mohri, and Y. Zhong. Ranking with abstention. In ICML 2023 Workshop The Many Facets of Preference-Based Learning , 2023e.

Mao, M. Mohri, and Y. Zhong. Theoretically grounded loss functions and algorithms for scorebased multi-class abstention. arXiv preprint arXiv:2310.14770 , 2023f.

Mao, M. Mohri, and Y. Zhong. Structured prediction with stronger consistency guarantees. In Advances in Neural Information Processing Systems , 2023g.

Mao, M. Mohri, and Y. Zhong. Cross-entropy loss functions: Theoretical analysis and applications. In International Conference on Machine Learning , 2023h.

Mohri, D. Andor, E. Choi, M. Collins, A. Mao, and Y. Zhong. Learning to reject with a fixed predictor: Application to decontextualization. arXiv preprint arXiv:2301.09044 , 2023.

Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of Machine Learning . MIT Press, second edition, 2018.

Mozannar and D. Sontag. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning , pages 7076-7087, 2020.

Mozannar, A. Satyanarayan, and D. Sontag. Teaching humans when to defer to a classifier via exemplars. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 5323-5331, 2022.

Mozannar, H. Lang, D. Wei, P. Sattigeri, S. Das, and D. Sontag. Who should predict? exact algorithms for learning to defer to humans. In International Conference on Artificial Intelligence and Statistics , pages 10520-10545, 2023.

Narasimhan, W. Jitkrittum, A. K. Menon, A. S. Rawat, and S. Kumar. Post-hoc estimators for learning to defer to an expert. In Advances in Neural Information Processing Systems , 2022.

Narasimhan, A. K. Menon, W. Jitkrittum, and S. Kumar. Learning to reject meets ood detection: Are all abstentions created equal? arXiv preprint arXiv:2301.12386 , 2023.

Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems , 2011.

Ni, N. Charoenphakdee, J. Honda, and M. Sugiyama. On the calibration of multiclass classification with rejection. In Advances in Neural Information Processing Systems , pages 2582-2592, 2019.

Okati, A. De, and M. Rodriguez. Differentiable learning under triage. Advances in Neural Information Processing Systems , 34:9140-9151, 2021.

F. Pradier, J. Zazo, S. Parbhoo, R. H. Perlis, M. Zazzi, and F. Doshi-Velez. Preferential mixtureof-experts: Interpretable models that rely on human expertise as much as possible. AMIA Summits on Translational Science Proceedings , 2021:525, 2021.

Raghu, K. Blumer, G. Corrado, J. Kleinberg, Z. Obermeyer, and S. Mullainathan. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv:1903.12220 , 2019.

Raman and M. Yee. Improving learning-to-defer algorithms through fine-tuning. arXiv preprint arXiv:2112.10768 , 2021.

G. Ramaswamy, A. Tewari, and S. Agarwal. Consistent algorithms for multiclass classification with an abstain option. Electronic Journal of Statistics , 12(1):530-554, 2018.

Steinwart. How to compare different loss functions and their risks. Constructive Approximation , 26(2):225-287, 2007.

Straitouri, A. Singla, V. B. Meresht, and M. Gomez-Rodriguez. Reinforcement learning under algorithmic triage. arXiv preprint arXiv:2109.11328 , 2021.

Straitouri, L. Wang, N. Okati, and M. G. Rodriguez. Provably improving expert predictions with conformal prediction. arXiv preprint arXiv:2201.12006 , 2022.

Tan, J. Adebayo, K. Inkpen, and E. Kamar. Investigating human+ machine complementarity for recidivism predictions. arXiv preprint arXiv:1808.09123 , 2018.

Verma and E. Nalisnick. Calibrated learning to defer with one-vs-all classifiers. In International Conference on Machine Learning , pages 22184-22202, 2022.

Verma, D. Barrejón, and E. Nalisnick. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. In International Conference on Artificial Intelligence and Statistics , pages 11415-11434, 2023.

Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus. Emergent abilities of large language models. CoRR , abs/2206.07682, 2022.

Wiener and R. El-Yaniv. Agnostic selective classification. In Advances in neural information processing systems , 2011.

Wilder, E. Horvitz, and E. Kamar. Learning to complement humans. In International Joint Conferences on Artificial Intelligence , pages 1526-1533, 2021.

Yuan and M. Wegkamp. Classification methods with reject option based on convex risk minimization. Journal of Machine Learning Research , 11(1), 2010.

Yuan and M. Wegkamp. SVMs with a reject option. In Bernoulli , 2011.

Zhang and S. Agarwal. Bayes consistency vs. H-consistency: The interplay between surrogate loss functions and the scoring function class. In Advances in Neural Information Processing Systems , 2020.

Zhang. Statistical behavior and consistency of classification methods based on convex risk minimization. The Annals of Statistics , 32(1):56-85, 2004.

Zhao, M. Agrawal, P. Razavi, and D. Sontag. Directing human attention in event localization for clinical timeline creation. In Machine Learning for Healthcare Conference , pages 80-102, 2021.

Zheng, G. Wu, F. Bao, Y. Cao, C. Li, and J. Zhu. Revisiting discriminative vs. generative classifiers: Theory and implications. arXiv preprint arXiv:2302.02334 , 2023.

Ziyin, Z. Wang, P. P. Liang, R. Salakhutdinov, L.-P. Morency, and M. Ueda. Deep gamblers: Learning to abstain with portfolio theory. arXiv preprint arXiv:1907.00208 , 2019.

Contents of Appendix

A Related work

The scenario of single-stage learning to defer has been extensively explored in previous research. The initial studies focused on the problem of abstention and introduced various approaches such as confidence-based abstention [Chow, 1957, 1970, Herbei and Wegkamp, 2005, Bartlett and Wegkamp, 2008, Grandvalet et al., 2008, Yuan and Wegkamp, 2010, 2011, Ramaswamy et al., 2018, Ni et al., 2019], selective classification [El-Yaniv et al., 2010, Wiener and El-Yaniv, 2011, Kalai et al., 2012, Geifman and El-Yaniv, 2017, 2019, Ziyin et al., 2019, Acar et al., 2020, Gangrade et al., 2021], a predictor-rejector framework for abstention [Cortes et al., 2016a,b, Charoenphakdee et al., 2021, Cortes et al., 2023, Mohri et al., 2023, Mao et al., 2023c], and a score-based setting for abstention [Mozannar and Sontag, 2020, Raman and Yee, 2021, Liu et al., 2022, Verma and Nalisnick, 2022, Charusaie et al., 2022, Cao et al., 2022, Mao et al., 2023f, Verma et al., 2023, Mao et al., 2023b, Mozannar et al., 2023].

Another line of research is centered around the joint learning of prediction and deferral functions. Several publications by Madras et al. [2018], Raghu et al. [2019], Wilder et al. [2021], Pradier et al. [2021], Keswani et al. [2021] delve into this topic, considering single-stage learning to defer and its variants. Additionally, the concept of learning to defer has been explored in different scenarios, including combining human and machine predictions, investigating human preferences, regression problems, reinforcement learning, and more [Kamar et al., 2012, Tan et al., 2018, Kleinberg et al., 2018, Bansal et al., 2021, De et al., 2020, Straitouri et al., 2021, Zhao et al., 2021, Joshi et al., 2021, Gao et al., 2021, Mozannar et al., 2022, Hemmer et al., 2023, Narasimhan et al., 2023]. However, in practice, a predictor such as an LLM is already available and retraining one in conjunction with a deferral function could be prohibitively costly: depending on its size and the amount of data used, retraining could take several weeks or months. Thus, the single-stage learning to defer scenario and its solutions often do not align with the practical challenges encountered in real-world applications.

Alternative post-hoc methods have been proposed to address the learning to defer problem. Okati et al. [2021] proposed an iterative approach optimizing a predictor and a rejector over multiple epochs. Within each epoch, first the predictor is trained on points where its loss is lower than that of a human expert; second, the rejector is fitted to predict which of the predictor or the human expert has a lower loss. Narasimhan et al. [2022] suggested a post-hoc correction to the single-stage learning to defer surrogate losses, specifically the cost-sensitive softmax cross-entropy (CSS) surrogate loss in [Mozannar and Sontag, 2020] and the one-versus-all (OvA) surrogate loss in [Verma and Nalisnick, 2022] for cases where they suffer from underfitting. However, as with the single-stage learning to defer solutions, post-hoc approaches do not apply to scenarios where an existing predictor, pre-trained using a standard classification loss function such as cross-entropy, is already available.

A key criterion for surrogate losses in the scenario of learning to defer is Bayes-consistency (also known as consistency) [Zhang, 2004, Bartlett et al., 2006, Steinwart, 2007, Mohri et al., 2018]. This property guarantees that minimizing the surrogate loss over the family of measurable functions leads to the minimization of the deferral loss. The surrogate losses proposed in [Mozannar and Sontag, 2020, Verma and Nalisnick, 2022] have shown to be Bayes-consistent for deferral. However, Bayesconsistency is a property associated with the family of all measurable functions, which of course is considerably broader than the hypothesis sets typically used in learning algorithms, including linear hypothesis sets and the family of neural networks.

Instead, Long and Servedio [2013], Kuznetsov et al. [2014], Zhang and Agarwal [2020] proposed a notion of realizable H -consistency, that is consistency associated with a specific hypothesis set in the realizable scenario. Mozannar et al. [2023] recently showed that existing Bayes-consistent surrogate losses in [Mozannar and Sontag, 2020, Verma and Nalisnick, 2022] are not realizable H -consistent for learning with deferral, which can pose significant challenges when learning with a restricted hypothesis set H , even for simple linear models. Instead, they proposed a new surrogate loss that is realizable H -consistent when H is closed under scaling. However, they also observed that the loss function of Madras et al. [2018], which is not Bayes-consistent, is actually realizable H -consistent. They acknowledged their inability to prove or disprove whether their proposed surrogate loss is Bayes-consistent. Consequently, it has remained an open problem to identify a surrogate loss that is both consistent and realizable-consistent.

In recent work, Verma et al. [2023] proposed the first Bayes-consistent surrogate losses in the scenario of learning to defer with multiple experts [Hemmer et al., 2022, Keswani et al., 2021, Kerrigan et al.,

2021, Straitouri et al., 2022, Benz and Rodriguez, 2022]. This scenario is more attractive and significant in applications such as large language models, where multiple models are often available for deferral. However, the surrogate losses proposed by the authors do not benefit from realizable H -consistency, even in the single-expert setting, since they are a straightforward generalization of those of Mozannar and Sontag [2020] and Verma and Nalisnick [2022].

In summary, the problem of learning to defer in a single-stage scenario has been extensively studied, but it is often impractical in real-world applications. Post-hoc methods and surrogate losses have been explored, but the challenge remains to find a surrogate loss that is both consistent and realizableconsistent. Recent research has made progress in the scenario of learning to defer with multiple experts but has not achieved realizable H -consistency even in a single-expert setting.

B Examples of two-stage score-based surrogate losses

Example: glyph[lscript] 2 = glyph[lscript] exp . For glyph[lscript] 2 ( h d , x, y ) = glyph[lscript] exp ( h d , x, y ) = ∑ y ′ ≠ y e h d ( x,y ′ )-h d ( x,y ) , by (3), we have

Example: glyph[lscript] 2 = glyph[lscript] log . For glyph[lscript] 2 ( h d , x, y ) = glyph[lscript] log ( h d , x, y ) = log (∑ y ′ ∈ Y ∪{ 0 } e h d ( x,y ′ )-h d ( x,y ) ) , by (3), we have

L

h

p

(

h

d

, x, y

)

Example: glyph[lscript] 2 = glyph[lscript] mae . For glyph[lscript] 2 ( h d , x, y ) = glyph[lscript] mae ( h d , x, y ) = 1 -e h d ( x,y ) ∑ y ′ ∈ Y ∪{ 0 } e h d ( x,y ′ ) , by (3), we have

C Examples of two-stage predictor-rejector surrogate losses

Example: glyph[lscript] 2 = glyph[lscript] log . For glyph[lscript] 2 ( r, x, y ) = glyph[lscript] log ( r, x, y ) = log (∑ y ′ ∈ Y ∪{ 0 } e r ( x,y ′ )-r ( x,y ) ) , by (5), we have

Example: glyph[lscript] 2 = glyph[lscript] gce . For glyph[lscript] 2 ( r, x, y ) = glyph[lscript] gce ( r, x, y ) = 1 α [ 1 -[ e r ( x,y ) ∑ y ′ ∈ Y ∪{ 0 } e r ( x,y ′ ) ] α ] , α ∈ ( 0 , 1 ) , by (5), we have

Example: glyph[lscript] 2 = glyph[lscript] mae . For glyph[lscript] 2 ( r, x, y ) = glyph[lscript] mae ( r, x, y ) = 1 -e r ( x,y ) ∑ y ′ ∈ Y ∪{ 0 } e r ( x,y ′ ) , by (5), we have

D Proof of H -consistency bounds for score-based two-stage surrogate losses (Theorem 1)

Theorem 1 ( H -consistency bounds for score-based two-stage surrogates ) . Assume that glyph[lscript] 1 admits an H p -consistency bound and glyph[lscript] 2 admits an H d -consistency bound with respect to the multi-class zero-one classification loss glyph[lscript] 0 -1 respectively. Thus, there are non-decreasing concave functions Γ 1 and Γ 2 such that, for all h p ∈ H p and h d ∈ H d , we have

Then, the following holds for all h ∈ H :

Furthermore, constant factors ( 1 + ∑ n e j = 1 c j ) and 1 ∑ ne j = 1 c j can be removed when Γ 2 is linear.

Proof. If h ( x ) ∈ [ n ] , then h ( x ) = h p ( x ) . Thus, the learning to defer loss can be expressed as follows:

Let ¯ c 0 ( x, y ) = 1 hp ( x )= y . Since h = ( h p , h d ) , we can rewrite E L def ( h ) -E ∗ L def ( H ) + M L def ( H ) as

Let p ( x, j ) = E y [ ¯ c j ( x,y )] E y [∑ ne j = 0 ¯ c j ( x,y )] for any j ∈ { 0 , . . . , n e } . Note that p ( x, ⋅) is the probability vector on the label space { 0 , . . . , n e } . For any h ∈ H , we define h as its augmented hypothesis: h ( x, 0 ) =

max y ∈ Y h ( x, y ) , h ( x, 1 ) = h ( x, 1 ) , . . . , h ( x, n e ) = h ( x, n e ) . By the assumptions, we have

and

Therefore, by (6), we obtain

which completes the proof.

E Proof of H -consistency bounds for standard surrogate loss functions (Theorem 3)

Recall that for a hypothesis h ∶ X × Y → R , we define h as its augmented hypothesis: h (⋅ , 0 ) = λ, h (⋅ , 1 ) = h ( x, 1 ) , . . . , h (⋅ , n ) = h ( x, n ) with some constant λ ∈ R . We define H as the hypothesis set that consists of all such augmented hypotheses of H : H = { h ∶ h ∈ H } . The prediction associated by h ∈ H to an input x ∈ X is denoted by h ( x ) and defined as the element in Y ∪{ 0 } with the highest score, h ( x ) = argmax y ∈ Y ∪{ 0 } h ( x, y ) , with an arbitrary but fixed deterministic strategy for breaking ties. For any x ∈ X and label space Y ∪ { 0 } , we will denote, by H ( x ) the set of labels generated by hypotheses in H : H ( x ) = { h ( x )∶ h ∈ H } . By [Awasthi et al., 2022a, Lemma 3] with label space Y ∪ { 0 } and a conditional probability vector p ( x, ⋅) on Y ∪ { 0 } , the minimal conditional glyph[lscript] 0 -1 -loss and the corresponding calibration gap can be characterized as follows.

Lemma 8. For any x ∈ X , the minimal conditional glyph[lscript] 0 -1 -risk and the calibration gap for glyph[lscript] 0 -1 can be expressed as follows:

E.1 Multinomial logistic loss

Theorem 9 ( H -consistency bound for multinomial logistic loss ) . Assume that H is symmetric and complete. Then, for any λ ∈ R , hypothesis h ∈ H and any distribution,

Proof. For the multinomial logistic loss glyph[lscript] log , the conditional glyph[lscript] log -risk can be expressed as follows:

where we let S ( x, y ) = e h ( x,y ) ∑ y ′ ∈ Y ∪{ 0 } e h ( x,y ′ ) ∈ [ 0 , 1 ] for any y ∈ Y ∪ { 0 } with the constraint that ∑ y ∈ Y ∪{ 0 } S ( x, y ) = 1 . Let y max = argmax y ∈ Y ∪{ 0 } p ( x, y ) , where we choose the label with the same deterministic strategy for breaking ties as that of h ( x ) . For any h ∈ H such that h ( x ) ≠ y max and x ∈ X , by the symmetry and completeness of H , we can always find a family of hypotheses

{ h µ ∶ µ ∈ [-S ( x, y max ) , S ( x, h ( x ))]} ⊂ H such that S µ ( x, ⋅) = e hµ ( x, ⋅) ∑ y ′ ∈ Y ∪{ 0 } e hµ ( x,y ′ ) take the following values:

Note that S µ satisfies the constraint:

Let h ∈ H be a hypothesis such that h ( x ) ≠ y max . By the definition and using the fact that H ( x ) = Y ∪{ 0 } when H is symmetric, we obtain

Differentiating with respect to µ yields the optimum value µ ∗ = p ( x, h ( x )) S ( x, h ( x ))-p ( x,y max ) S ( x,y max ) p ( x,y max )+ p ( x, h ( x )) . Plugging that value in the inequality gives:

Differentiating with respect to S to show that the minimum is attained for S ( x, h ( x )) = S ( x, y max ) , which implies

By Pinsker's inequality, we have, for a, b ∈ [ 0 , 1 ] , a log 2 a a + b + b log 2 b a + b ≥ ( a -b ) 2 2 ( a + b ) . Using this inequality, we obtain:

Since the function t 2 2 is convex, by Jensen's inequality, we obtain for any hypothesis h ∈ H and any distribution,

which leads to

E.2 Sum exponential loss

Theorem 10 ( H -consistency bound for sum exponential loss ) . Assume that H is symmetric and complete. Then, for any λ ∈ R , hypothesis h ∈ H and any distribution,

Proof. For the sum exponential loss glyph[lscript] exp , the conditional glyph[lscript] exp -risk can be expressed as follows:

where we let S ( x, y ) = e h ( x,y ) ∑ y ′ ∈ Y ∪{ 0 } e h ( x,y ′ ) ∈ [ 0 , 1 ] for any y ∈ Y ∪ { 0 } with the constraint that ∑ y ∈ Y ∪{ 0 } S ( x, y ) = 1 . Let y max = argmax y ∈ Y ∪{ 0 } p ( x, y ) , where we choose the label with the highest index under the natural ordering of labels as the tie-breaking strategy. For any h ∈ H such that h ( x ) ≠ y max and x ∈ X , by the symmetry and completeness of H , we can always find a family of hypotheses { h µ ∶ µ ∈ [-S ( x, y max ) , S ( x, h ( x ))]} ⊂ H such that S µ ( x, ⋅) = e hµ ( x, ⋅) ∑ y ′ ∈ Y ∪{ 0 } e hµ ( x,y ′ ) take the following values:

Note that S µ satisfies the constraint:

Let h ∈ H be a hypothesis such that h ( x ) ≠ y max . By the definition and using the fact that H ( x ) = Y ∪{ 0 } when H is symmetric, we obtain

Differentiating with respect to µ yields the optimal value

Plugging that value in the inequality gives:

Differentiating with respect to S to show that the minimum is attained for S ( x, h ( x )) = S ( x, y max ) = 1 2 , which implies

By the concavity of the square-root function, for all a, b ∈ [ 0 , 1 ] , we have 1 2 ( √ a + √ b ) ≤ √ 1 2 ( a + b ) , thus we can write

Since the function t 2 2 is convex, by Jensen's inequality, we obtain for any hypothesis h ∈ H and any distribution,

which leads to

E.3 Generalized cross-entropy loss

Theorem 11 ( H -consistency bound for generalized cross-entropy loss ) . Assume that H is symmetric and complete. Then, for any λ ∈ R , hypothesis h ∈ H and any distribution,

Proof. For the generalized cross-entropy loss glyph[lscript] gce , the conditional glyph[lscript] gce -risk can be expressed as follows:

where we let S ( x, y ) = e h ( x,y ) ∑ y ′ ∈ Y ∪{ 0 } e h ( x,y ′ ) ∈ [ 0 , 1 ] for any y ∈ Y ∪ { 0 } with the constraint that ∑ y ∈ Y ∪{ 0 } S ( x, y ) = 1 . Let y max = argmax y ∈ Y ∪{ 0 } p ( x, y ) , where we choose the label with the same deterministic strategy for breaking ties as that of h ( x ) . For any h ∈ H such that h ( x ) ≠ y max and x ∈ X , by the symmetry and completeness of H , we can always find a family of hypotheses { h µ ∶ µ ∈ [-S ( x, y max ) , S ( x, h ( x ))]} ⊂ H such that S µ ( x, ⋅) = e hµ ( x, ⋅) ∑ y ′ ∈ Y ∪{ 0 } e hµ ( x,y ′ ) take the following values:

Note that S µ satisfies the constraint:

Let h ∈ H be a hypothesis such that h ( x ) ≠ y max . By the definition and using the fact that H ( x ) = Y ∪{ 0 } when H is symmetric, we obtain

Differentiating with respect to µ yields the optimal value

Plugging that value in the inequality gives:

Differentiating with respect to S to show that the minimum is attained for S ( x, h ( x )) = S ( x, y max ) = 1 n + 1 , which implies

By using the fact that for all a, b ∈ [ 0 , 1 ] , 0 ≤ a + b ≤ 1 , we have ( a 1 1 -α + b 1 1 -α 2 ) 1 -α -a + b 2 ≥ α 4 ( a -b ) 2 , thus we can write

Since the function t 2 2 ( n + 1 ) α is convex, by Jensen's inequality, we obtain for any hypothesis h ∈ H and any distribution,

which leads to

E.4 Mean absolute error loss

Theorem 12 ( H -consistency bound for mean absolute error loss ) . Assume that H is symmetric and complete. Then, for any λ ∈ R , hypothesis h ∈ H and any distribution,

Proof. For the mean absolute error loss glyph[lscript] mae , the conditional glyph[lscript] mae -risk can be expressed as follows:

where we let S ( x, y ) = e h ( x,y ) ∑ y ′ ∈ Y ∪{ 0 } e h ( x,y ′ ) ∈ [ 0 , 1 ] for any y ∈ Y ∪ { 0 } with the constraint that ∑ y ∈ Y ∪{ 0 } S ( x, y ) = 1 . Let y max = argmax y ∈ Y ∪{ 0 } p ( x, y ) , where we choose the label with the same deterministic strategy for breaking ties as that of h ( x ) . For any h ∈ H such that h ( x ) ≠ y max and x ∈ X , by the symmetry and completeness of H , we can always find a family of hypotheses { h µ ∶ µ ∈ [-S ( x, y max ) , S ( x, h ( x ))]} ⊂ H such that S µ ( x, ⋅) = e hµ ( x, ⋅) ∑ y ′ ∈ Y ∪{ 0 } e hµ ( x,y ′ ) take the following values:

Note that S µ satisfies the constraint:

Let h ∈ H be a hypothesis such that h ( x ) ≠ y max . By the definition and using the fact that H ( x ) = Y ∪{ 0 } when H is symmetric, we obtain

Differentiating with respect to µ yields the optimum value µ ∗ = -S ( x, y max ) . Plugging that value in the inequality gives:

Differentiating with respect to S to show that the minimum is attained for S ( x, h ( x )) = 1 n + 1 , which implies

Therefore, we obtain for any hypothesis h ∈ H and any distribution,

which leads to

F Proof of realizable consistency for score-based two-stage surrogate losses (Theorem 5)

Theorem 5 ( Realizable H -consistency for score-based two-stage surrogates ) . Assume that H is closed under scaling and c j ( x, y ) = β j , ∀( x, y ) ∈ X × Y . Let glyph[lscript] 1 and glyph[lscript] 2 be the logistic loss. Let ˆ h p be the minimizer of E glyph[lscript] 1 and ˆ h d be the minimizer of E L ˆ hp such that E L ˆ hp ( ˆ h d ) = min h E L hp ( h d ) . Then, the following equality holds for any ( H , R ) -realizable distribution,

Proof. First, by definition, it is straightforward to see that for any h, x, y , L h p ( h d , x, y ) upper bounds the deferral loss L def . Consider a data distribution and costs under which there exists h ∗ ∈ H such that E L def ( h ∗ ) = 0 .

Let ˆ h p be the minimizer of E glyph[lscript] 1 and ˆ h d the minimizer of E L ˆ hp Then, using the fact that L h upper bounds the deferral loss L def , we have E L def ( ˆ h ) ≤ E L ˆ hp ( ˆ h d ) .

Next we analyze two cases. If for a point x , deferral occurs, that is there exists j ∗ ∈ [ n e ] , such that h ∗ ( x ) = n + j ∗ , then we must have c j ∗ = 0 for all x since the data is realizable and c j ∗ is constant. Therefore, there exists an optimal h ∗∗ deferring all the points to the j ∗ th expert. Then, by the assumption that H is closed under scaling and the Lebesgue dominated convergence theorem, for glyph[lscript] 2 being the logistic loss, E L def ( ˆ h ) ≤ E L ˆ hp ( ˆ h d ) ≤ lim τ →+∞ E L h ∗∗ p ( τh ∗∗ d ) = 0 , where we used the fact that in the limit of τ →+∞ the logistic loss term glyph[lscript] 2 ( h ∗∗ d , x, j ) corresponding to j ≠ j ∗ is zero.

On the other hand, if no deferral occurs for any point, that is h ∗ ( x ) ∈ [ n ] for any x , then we must have 1 h ∗ p ( x )≠ y = 0 for all ( x, y ) since the data is realizable. Using the fact that H is closed under scaling and that the logistic loss is realizable H -consistent in the standard classification, we obtain 1 ˆ h p ( x )≠ y = 0 for all ( x, y ) . Then, by the assumption that H is closed under scaling and the Lebesgue dominated convergence theorem, for glyph[lscript] 2 being the logistic loss, E L def ( ˆ h ) ≤ E L ˆ hp ( ˆ h d ) ≤ lim τ →+∞ E L h ∗ p ( τh ∗ d ) = 0 , where we used the fact that in the limit of τ →+∞ the logistic loss term glyph[lscript] 2 ( h ∗ d , x, j ) corresponding to j ≠ 0 is zero.

Therefore, the optimal solution from minimizing score-based two-stage surrogates leads to a zero error solution of the deferral loss, which proves that the score-based two-stage surrogate loss is realizable consistent.

G Proof of ( H , R ) -consistency bounds for predictor-rejector two-stage surrogate losses (Theorem 6)

Theorem 6 ( ( H , R ) -consistency bounds for predictor-rejector two-stage surrogates ) . Assume that glyph[lscript] 1 admits an H -consistency bound and glyph[lscript] 2 admits an R -consistency bound with respect to the multi-class zero-one classification loss glyph[lscript] 0 -1 respectively. Thus, there are non-decreasing concave functions Γ 1 and Γ 2 such that, for all h ∈ H and r ∈ R , we have

Then, the following holds for all h ∈ H and r ∈ R :

where the constant factors ( 1 + ∑ n e j = 1 c j ) and 1 ∑ ne j = 1 c j can be removed when Γ 2 is linear.

Proof. By definition,

Let p ( x, j ) = E y [ ¯ c j ( x,y )] ∑ ne j = 0 E y [ ¯ c j ( x,y )] for any j ∈ { 0 , . . . , n e } . Note that p ( x, ⋅) is the probability vector on the label space { 0 , . . . , n e } . For any r ∈ R , we define r as its augmented hypothesis: r ( x, 0 ) = 0 , r ( x, 1 ) = -r 1 ( x ) , . . . , r ( x, n e ) = -r n e ( x ) . By the assumptions, we have

and

Therefore, by (7), we obtain

which completes the proof.

H Proof of realizable consistency for predictor-rejector two-stage surrogate losses (Theorem 7)

Theorem 7 ( Realizable ( H , R ) -consistency for predictor-rejector two-stage surrogates ) . Assume that H and R is closed under scaling and c j ( x, y ) = β j , ∀( x, y ) ∈ X × Y . Let glyph[lscript] 1 and glyph[lscript] 2 be the logistic loss. Let ˆ h be the minimizer of E glyph[lscript] 1 and ˆ r be the minimizer of E L ˆ h . Then, the following holds for any ( H , R ) -realizable distribution,

Proof. First, by definition, it is straightforward to see that for any h, r, x, y , L h ( r, x, y ) upper bounds the deferral loss L def . Consider a data distribution and costs under which there exists h ∗ ∈ H and r ∗ ∈ R such that E L def ( h ∗ , r ∗ ) = 0 .

Let ˆ h be the minimizer of E glyph[lscript] 1 and ˆ r the minimizer of E L ˆ h Then, using the fact that L h upper bounds the deferral loss L def , we have E L def ( ˆ h, ˆ r ) ≤ E L ˆ h ( ˆ r ) .

Next we analyze two cases. If for a point x , deferral occurs, that is there exists j ∗ ∈ [ n e ] , such that r ∗ ( x ) = j ∗ , then we must have c j ∗ = 0 for all x since the data is realizable and c j ∗ is constant. Therefore, there exists an optimal r ∗∗ deferring all the points to the j ∗ th expert. Then, by the assumption that R is closed under scaling and the Lebesgue dominated convergence theorem, for glyph[lscript] 2 being the logistic loss, E L def ( ˆ h, ˆ r ) ≤ E L ˆ h ( ˆ r ) ≤ lim τ →+∞ E L ˆ h ( τr ∗∗ ) = 0 , where we used the fact that in the limit of τ →+∞ the logistic loss term glyph[lscript] 2 ( r ∗∗ , x, j ) corresponding to j ≠ j ∗ is zero.

On the other hand, if no deferral occurs for any point, that is r ∗ ( x ) = 0 for any x , then we must have 1 h ∗ ( x )≠ y = 0 for all ( x, y ) since the data is realizable. Using the fact that H is closed under scaling and that the logistic loss is realizable H -consistent in the standard classification, we obtain 1 ˆ h ( x )≠ y = 0 for all ( x, y ) . Then, by the assumption that R is closed under scaling and the Lebesgue dominated convergence theorem, for glyph[lscript] 2 being the logistic loss, E L def ( ˆ h, ˆ r ) ≤ E L ˆ h ( ˆ r ) ≤ lim τ →+∞ E L ˆ h ( τr ∗ ) = 0 , where we used the fact that in the limit of τ →+∞ the logistic loss term glyph[lscript] 2 ( r ∗ , x, j ) corresponding to j ≠ 0 is zero.

Therefore, the optimal solution from minimizing predictor-rejector two-stage surrogates leads to a zero error solution of the deferral loss, which proves that the predictor-rejector two-stage surrogate loss is realizable consistent.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies generalization bounds for learning to defer, with its main focus on \"two-stage\" algorithms that first fit a classifier using a surrogate classification loss, then separately fit a rejector that decides which expert, if any, to defer the classification to.\nBy and large, these two-stage algorithms have less theory than _joint_ algorithms that fit the classifier and rejector together, so this paper fills a gap in the theory literature for these algorithms. Bounds are provided for both score-based methods and the more general predictor/rejector setting.\n\nSTRENGTHS:\n- Relatively well-presented and explained theoretical results\n- Fills a gap in the existing theory literature for learning to defer, especially in the multi-expert setting.\n\nWEAKNESSES:\n- The empirical results are missing baselines / comparisons to other surrogate methods in the literature.\n\n- I'm unable to follow the proofs of Theorems 5 and 7, which seem to be missing several steps (perhaps they're obvious?). In any case, the conclusions of these theorems seem to contradict the following simple 2D counterexample based on XOR.\nSuppose $\\mathcal{D}$ is uniform over $[-1,1] \\times [-1,1]$. Set the label $Y=1$ when $X_1 \\cdot X_2 \\ge 0$, and set $Y=-1$ elsewhere (i.e., the label is positive in quadrants 1 and 3, and negative in quadrants 2 and 4). Suppose there is just one human expert, and the human is always correct when $X_1 < 0$ and always incorrect when $X_1 \\ge 0$. This is clearly $(\\mathcal{H},\\mathcal{R})$-realizable for $\\mathcal{H}$ and $\\mathcal{R}$ both linear: $r^*$ chooses the human whenever $X_1 < 0$, and $h^*$ predicts $\\mathrm{sign}(X_2)$, which is the correct classification whenever $X_1 \\ge 0$.\nHowever, the conclusion of the theorem seems false in this case. There is no halfspace that achieves a nontrivial accuracy on all of $\\mathcal{D}$. However, we can still find an $h$ where $\\varepsilon_{l_1}(h) - \\varepsilon^*_{l_1}(\\mathcal{H})$ is arbitrarily small. For a fixed $h$ (which will have a poor loss $l_1(h)$), we can still fit an optimal deferrer, i.e., we can still reduce $\\varepsilon_{L_h}(r) - \\varepsilon^*_{L_h}(\\mathcal{R})$ towards 0. In this case, the RHS of the theorem statement can approach 0, but clearly, this combination has a poor loss $L_{def}$.\nEssentially, the issue is that the right-hand-side of the theorem statement is not sensitive to whether two-stage optimization is suboptimal, but $(\\mathcal{H}, \\mathcal{R})$-realizability allows for the case where it is.\n\n- Without the above, the results are still fairly interesting, since they provide some theory for two-stage methods, but the paper can't claim to resolve the open problem of Mozannar et al. 2023.\n\nQUESTIONS:\n- If I'm mistaken about the above (if so, I will increase my score), how do the results of Theorem 5 and 7 avoid examples where two-stage optimization is very suboptimal and joint optimization is required, like the one I presented above? Can the authors provide more verbose and rigorous proofs of Theorems 5 and 7?\n\n- It might be useful to see if/how the proposed surrogate reduces to other ones in the literature in the single-expert case, depending on the choices of $l$.\n\nLIMITATIONS:\nI could not find a discussion of the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper studies a two-stage scenario for learning to defer with multiple experts. For both the score-based setting and the predictor-rejector setting, the authors design a new family of surrogate loss function . The authors show that the surrogate losses are realizable H-consistent and  Bayes-consistency of our surrogate losses. The authors provide theoretical analysis and empirical results to justify the proposed framework.

STRENGTHS:
1. The problem setup is interesting and realistic. The two stage deferring scenario is suitable when predictor is a pretrained LLM, and has promising real applications.

2. The surrogate loss function is well-motivated and new as far as I am concerned. The theoretical results are strong and comprehensive. I do not fully check the proof, but the results seem reasonable to me.

WEAKNESSES:
I am satisfied with the theoretical part. However, I would like to remark that I am not an expert in this field. I will be happy to follow up in the discussion period, if there is any additional questions raised up by other reviewers. 

For the experiments, the accuracy on CIFAR10 and SVHN is significantly lower than the standard results of ResNet. Moreover, the authors only implement the proposed loss function. I would be better to compare with different baselines in previous works.

QUESTIONS:
See weakness part above.

LIMITATIONS:
I do not see the discussion of limitations, but I think this is not a major problem. I do not see potential negative societal impact as this paper mainly focuses on the theoretical side.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides bounds on the difference between the target loss and its surrogate loss, for two-stage learning to defer to multiple experts. The bounds guarantee that a small surrogate risk leads to a small target risk. The authors also show that in realizable cases, achieving the minimum surrogate risk results in the zero target risk, which affirmatively answers an open question. The paper studies two setups of two-stage learning to defer with multiple experts and provides similar theoretical results for both. Finally, the paper presents experiments using CIFAR-10 and SVHN, showing that having more experts improves performance.\n\nSTRENGTHS:\n- The paper studies the practically important task of learning to defer to experts.\n- The results seem to be non-trivial and critical theoretical guarantees for the approach studied.\n- The writing is overall well-written and easy to follow, except for some technical parts.\n- The paper provides a good review of previous work and clearly explains the contributions.\n- The experiment is interesting and shows the usefulness of the approach of learning to defer.\n\nWEAKNESSES:\n- Some technical parts were a little unclear. (See the questions below.)\n- Section 3 and Section 4 are repetitive. I don't know how much the setups are different from each other.\n- I think some math expressions could be simplified. (See the Questions section below.)\n- The experiments are not extensive. I expected to see some experiments confirming the theoretical results, but the focus of the experiments presented in the paper seems to be the improvement by increasing the number of experts, which is quite independent of the theory part.\n- The paper stresses the novelty of the proof technique and there are brief explanations, but it is difficult for me to understand them without reading the appendix. (Unfortunately, I did not have time to read the appendix.)\n\nQUESTIONS:\n- In line 21, the paper says, \"However, this is not relevant in practice: the cost of retraining the predictor would be prohibitive for many applications.\" I think saying it is \"not relevant\" is a bit too strong if there are still some scenarios to which it is applicable.\n- For the minimizability gap $\\mathcal{R}^*_l - \\mathbb{E}\\_{x}[\\inf\\_{h\\in \\mathcal{H}} \\mathbb{E}\\_{y | x}[\\texttt{L}(h, x, y)]]$, why do we want to consider the point-wise infimum for the second term? I think it is more natural to consider $\\inf\\_{h\\in \\mathcal{H}} \\mathbb{E}\\_{x, y}[\\texttt{L}(h, x, y)]$.\n- In lines 137-140, it is strange to \"decompose\" $\\mathcal{H}$ as $\\mathcal{H} = \\mathcal{H}_p \\times \\mathcal{H}_d$ when $\\mathcal{H}$ is a set of functions. ($\\mathcal{H}_p \\times \\mathcal{H}_d$ is a set of function pairs.)\n- I think it's nicer to express, e.g., Eq. (3) as $L_{h_p}(h_d, x, y) = \\sum\\_{j=0}^{n_e} c\\_j(x, y) \\ell_2(\\bar{h}\\_d, x, j)$, with $c_j$ extended by $c_0(x, y) = \\mathbb 1\\_{h\\_p(x) \\neq y}$. Equations are unnecessarily long and hard to read especially in Table 2.\n- Is Definition 4 correct? (I think there is something wrong with the organization of the sentences.)\n- What do the symbols in the \"Base cost\" column in Table 4 mean?\n- What does Accuracy in Table 4 mean? They are not related to the deferral loss?\n- In ll. 231-234, I could not understand this part: \"In their proof, the authors choose $\\bar h\\_{\\mu}$ based on individual scores $h(x, y)$, rather than the softmax. Consequently, if $h(x)$ or the label $y_{\\text{max}}$ that achieves the maximum conditional probability corresponds to the label zero, $\\mu$ cannot be optimized since the score $h(x, 0)$ is fixed at $\\lambda$.\"\n\nLIMITATIONS:\nThis paper focuses on bounds regarding population quantities. The results do not directly apply to empirical risk minimization although this extension does not seem very hard.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes two stage learning algorithms for learning to defer framework. Learning to defer framework consists of two components, a rejector and a classifier. Previous literature has focussed on jointly training both the classifier and the rejector. This paper, instead, focuses on the two stage setup in designing learning to defer systems where a pre-trained classifier is provided and the rejector is to be trained in the second stage. This is a promising and more practical use case scenario of the deferral systems. To this end, the paper makes a series of contributions: 1. a simple and elegant formulation of the the said two stage setup 2. Proposing a family of surrogate loss functions for this setup as well as providing risk control guarantees based on $\\mathcal{H}$-consistency bounds that builds upon the consistency bounds for the loss employed in both the stages. 3. The paper then prove a more general result on the $\\mathcal{H}$ consistency bounds for general surrogate losses for classification problem which is augmented with some additional instance dependent hypothesis. 4. The paper uses this result to establish $\\mathcal{H}$ consistency as well as the Bayes consistency of the proposed two stage surrogates. Additionally, the paper also establishes the realizable $\\mathcal{H}$ consistency of the two stage surrogates which is a more stronger property. The focus of the paper is general where multiple experts are possible.\n\nSTRENGTHS:\n1. The paper investigates a question of practical relevance to the machine learning community. Deferral systems have appealing properties as they foster trust and transparency, as well as monitoring behaviour by introducing experts into the system. However, it was not possible based on the prior work how to adapt any pre-trained system to experts. This paper fills this gap by devising theoretically sound algorithms.\n\n2. While the paper builds heavily on the previous works on rejection learning, the proposed algorithms are sufficiently novel. The paper also fills a gap in the literature for deriving $\\bar{\\mathcal{H}}$-consistency bounds for a general rejection learning problem. As a plus point, the proposed surrogates are simple and intuitive which makes it easier for adoption in the community. \n\n3. The paper proposes first of its kind $\\mathcal{H}$-realizable consistent algorithms for learning to defer with multiple experts. I believe this is a significant contribution as generally building deferral systems considering multiple experts from scratch would be prohibitively expensive. \n\n4. Although there are some minor typos in the text, the paper is clearly written and is super easy to read.\n\nWEAKNESSES:\n1. While theoretical results is the main focus of this paper, I believe experimental section could have been more insightful. Currently, the experimental section says that the overall system performance gets better on inclusion of more experts. However, this is intuitive and already known. Could authors add one more column in Table 4 to mention the performance of the originally trained classifier? I am curious to learn how the second stage adoption affects the originally trained classifier. While one can lower the estimation error of the first classifier by $\\epsilon_{1}$, it is not clear (to me) where such bound still holds True after the adoption in the second stage. I agree that this is certainly not the focus of the paper, I believe it is important to know if this second-stage training degrades the already trained classifier. I assume not, right?\n\n2. This might be a misunderstanding on my side, but I believe the proposed formulation can only work when $\\alpha_{j}, \\beta_{j} > 0,  \\forall j$ (line 104). However, there are no guidelines on how to define these two quantities, or what is their nature. \n\n3. One of the most promising appeal of the two stage training is preventing costly design from scratch. However, it is not clear whether it also has some disadvantages or not. Consider a single stage training approach where one wants to bound the estimation error by $\\epsilon$? Would it be cheaper / expensive to achieve the same bound using the two-stage approach? Could some quantitative results be given to this end to guide the practitioners?\n\nQUESTIONS:\n1. $\\mathcal{H}$ realizable consistent surrogates have not been proposed in the literature for single stage learning to defer with multiple experts. Is it possible to utilise the results in this paper for that? For example, maybe using iterative training procedure.\n\nLIMITATIONS:\nNo glaring limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}