SlowGuess/NeurIPS_2023_OCR huggingface dataset:
aqmao@cims.nyu.edu
Mehryar Mohri
Google Research & CIMS
New York, NY 10011
mohri@google.com
Christopher Mohri
xmohri@stanford.edu
Yutao Zhong
yutao@cims.nyu.edu
We study a two-stage scenario for learning to defer with multiple experts, which is crucial in practice for many applications. In this scenario, a predictor is derived in a first stage by training with a common loss function such as cross-entropy. In the second stage, a deferral function is learned to assign the most suitable expert to each input. We design a new family of surrogate loss functions for this scenario both in the score-based and the predictor-rejector settings and prove that they are supported by  $\mathcal{H}$ -consistency bounds, which implies their Bayes-consistency. Moreover, we show that, for a constant cost function, our two-stage surrogate losses are realizable  $\mathcal{H}$ -consistent. While the main focus of this work is a theoretical analysis, we also report the results of several experiments on CIFAR-10 and SVHN datasets.
Large language models (LLMs) have achieved a remarkable performance on diverse tasks across multiple domains, as reported in recent surveys [Wei et al., 2022, Bubeck et al., 2023]. However, their practical application faces two critical challenges: the occurrence of hallucinations, that is the generation of incorrect or misleading content, and an inefficient inference. Leveraging multiple experts can address both issues. To reduce hallucinations, one can refrain from using the original predictor in uncertain instances and defer to one of the more complex and more accurate experts. To enhance efficiency, one can derive models of different sizes distilled from the original complex model and use one of these more streamlined versions, while deferring to the more complex and less efficient ones for suitable contexts. Both problems require assigning each instance to the most suitable expert. This motivates the problem of learning to defer in the presence of multiple experts.
The scenario of single-stage learning to defer has been studied by several publications, starting with the foundational framework introduced by Cortes, DeSalvo, and Mohri [2016a,b, 2023] for learning to reject and followed by a series of studies on abstention and deferral [Madras et al., 2018, Raghu et al., 2019, Mozannar and Sontag, 2020, Wilder et al., 2021, Pradier et al., 2021, Keswani et al., 2021, Raman and Yee, 2021, Liu et al., 2022, Verma and Nalisnick, 2022, Charusaie et al., 2022, Cao et al., 2022, Verma et al., 2023, Mao et al., 2023f,b,c, Mozannar et al., 2023]. In the single-stage scenario, a predictor and a deferral function are learned simultaneously, with the deferral function determining the best expert assigned to each input instance. However, in practice, a predictor such as an LLM is already available and retraining one in conjunction with a deferral function could be prohibitively costly: depending on its size and the amount of data used, retraining could take several weeks or months. Thus, the single-stage learning to defer scenario and its solutions often do not align with the practical challenges encountered in real-world applications.
Alternative post-hoc methods have been proposed to address the learning to defer problem. Okati et al. [2021] proposed an iterative approach optimizing a predictor and a rejector over multiple epochs. Within each epoch, first the predictor is trained on points where its loss is lower than that of a human expert; second, the rejector is fitted to predict which of the predictor or the human expert has a lower loss. Narasimhan et al. [2022] suggested a post-hoc correction to the single-stage learning to defer surrogate losses, specifically the cost-sensitive softmax cross-entropy (CSS) surrogate loss in [Mozannar and Sontag, 2020] and the one-versus-all (OvA) surrogate loss in [Verma and Nalisnick, 2022] for cases where they suffer from underfitting. However, as with the single-stage learning to defer solutions, post-hoc approaches do not apply to scenarios where an existing predictor, pre-trained using a standard classification loss function such as cross-entropy, is already available.
Can we derive a principled algorithm for learning to defer with multiple experts in such scenarios? Which surrogate loss should we adopt and what consistency guarantee can we rely on? This paper deals precisely with these questions.
A key criterion for surrogate losses in learning to defer is Bayes-consistency [Zhang, 2004, Bartlett et al., 2006, Steinwart, 2007], that is minimizing the surrogate loss over the family of measurable functions leads to the minimization of the deferral loss. The surrogate losses proposed in [Mozannar and Sontag, 2020, Verma and Nalisnick, 2022] have been shown to be Bayes-consistent for deferral. However, Bayes-consistency is not relevant in learning tasks since the hypothesis set used, for example that of some family of linear functions or neural networks, never includes all measurable functions. Long and Servedio [2013] proposed a notion of realizable  $\mathcal{H}$ -consistency, that is consistency associated with a specific hypothesis set in the realizable scenario. Mozannar et al. [2023] recently showed that existing Bayes-consistent surrogate losses in [Mozannar and Sontag, 2020, Verma and Nalisnick, 2022] are not realizable  $\mathcal{H}$ -consistent for learning with deferral, which can pose significant challenges when learning with a restricted hypothesis set  $\mathcal{H}$ , even for simple linear models. Instead, they proposed a new surrogate loss that is realizable  $\mathcal{H}$ -consistent when  $\mathcal{H}$  is closed under scaling. However, they also observed that the loss function of Madras et al. [2018], which is not Bayes-consistent, is actually realizable  $\mathcal{H}$ -consistent. They acknowledged their inability to prove or disprove whether their proposed surrogate loss is Bayes-consistent. Consequently, it has remained an open problem to identify a surrogate loss that is both consistent and realizable-consistent.
In recent work, Verma et al. [2023] proposed the first Bayes-consistent surrogate losses in the scenario of single-stage learning to defer with multiple experts [Hemmer et al., 2022, Keswani et al., 2021, Kerrigan et al., 2021, Straitouri et al., 2022, Benz and Rodriguez, 2022]. This scenario is more attractive and significant in applications such as large language models, where multiple models are often available for deferral. However, the surrogate losses proposed by the authors do not benefit from realizable  $\mathcal{H}$ -consistency, even in the single-expert setting, since they are a straightforward generalization of those of Mozannar and Sontag [2020] and Verma and Nalisnick [2022].
Bayes-consistency, or even realizable  $\mathcal{H}$ -consistency for a specific hypothesis set  $\mathcal{H}$ , is an asymptotic property, and provides no guarantee for approximate minimizers since convergence could be arbitrarily slow. More favorable guarantees, known as  $\mathcal{H}$ -consistency bounds, were recently introduced for standard classification settings [Awasthi, Mao, Mohri, and Zhong, 2022b,a]. These guarantees are upper bounds on the target estimation loss expressed in terms of the surrogate estimation loss. They are stronger and more informative guarantees than Bayes-consistency and  $\mathcal{H}$ -consistency because they are both hypothesis set-specific and non-asymptotic. More recently, Mao et al. [2023b] introduced a new family of surrogate losses and algorithms for the general problem of single-stage learning to defer with multiple experts that benefit from strong  $\mathcal{H}$ -consistency bounds.
Our contributions. We study a two-stage scenario for learning to defer with multiple experts that is crucial in practice for many applications. In this scenario, a predictor is derived in a first stage by training with a common loss function such as cross-entropy. In the second stage, a deferral function is learned to assign the most suitable expert to each input. We design a new family of surrogate loss functions for this scenario both in the score-based setting (Section 3) and the predictor-rejection setting (Section 4) and prove that they are supported by  $\mathcal{H}$ -consistency bounds, which implies their Bayes-consistency. Moreover, we show that, for a constant cost function, our two-stage surrogate losses are realizable  $\mathcal{H}$ -consistent. While the main focus of this work is a theoretical analysis, we also report the results of several experiments on CIFAR-10 and SVHN datasets (Section 5). We give a comprehensive discussion of related work in Appendix A. We begin by providing some basic definitions and notation (Section 2).
We consider the standard multi-class classification setting with an input space  $\mathcal{X}$  and a set of  $n\geq 2$  labels  $\mathcal{Y} = [n]$ , where we use the notation  $[n]$  to denote the set  $\{1,\dots ,n\}$ . We study the scenario of learning to defer with multiple experts, where the label set  $\mathcal{Y}$  is augmented with  $n_e$  additional labels  $\{n + 1,\ldots ,n + n_e\}$  corresponding to  $n_e$  pre-defined experts  $h_1,\ldots ,h_{n_e}$ . In this scenario, the learner has the option of returning a label  $y\in \mathcal{Y}$ , which represents the category predicted, or a label  $y = n + j$ ,  $j\geq 1$ , in which case it is deferring to expert  $h_j$ . This setting is referred to as the score-based setting [Mozannar and Sontag, 2020, Cao et al., 2022, Mao et al., 2023f], since the deferral corresponds to extra  $n_e$  scoring functions. An alternative setting is the predictor-rejection setting [Cortes et al., 2016a, 2023, Mohri et al., 2023, Mao et al., 2023c], where the deferral function is selected from a separate family of functions  $\mathcal{R}$ . We introduce that setting and include the corresponding results in Section 4 for completeness.
We denote by  $\overline{\mathcal{Y}} = [n + n_e]$  the augmented label set and consider a hypothesis set  $\mathcal{H}$  of functions mapping from  $\mathcal{X} \times \overline{\mathcal{Y}}$  to  $\mathbb{R}$ . The prediction associated by  $h \in \mathcal{H}$  to an input  $x \in \mathcal{X}$  is denoted by  $\mathsf{h}(x)$  and defined as the element in  $\overline{\mathcal{Y}}$  with the highest score,  $\mathsf{h}(x) = \operatorname{argmax}_{y \in [n + n_e]} h(x, y)$ , with an arbitrary but fixed deterministic strategy for breaking ties. We denote by  $\mathcal{H}_{\mathrm{all}}$  the family of all measurable functions.
The deferral loss function  $\mathsf{L}_{\mathrm{def}}$  is defined as follows for any  $h \in \mathcal{H}$  and  $(x,y) \in \mathcal{X} \times \mathcal{Y}$ :
$$
\mathsf {L} _ {\text {d e f}} (h, x, y) = \mathbb {1} _ {\mathsf {h} (x) \neq y} \mathbb {1} _ {\mathsf {h} (x) \in [ n ]} + \sum_ {j = 1} ^ {n _ {e}} c _ {j} (x, y) \mathbb {1} _ {\mathsf {h} (x) = n + j} \tag {1}
$$
Thus, the loss incurred coincides with the standard zero-one classification loss when  $\mathsf{h}(x)$ , the label predicted, is in  $\mathcal{Y}$ . Otherwise, when  $\mathsf{h}(x)$  is equal to  $n + j$ , the loss incurred is  $c_{j}(x,y)$ , the cost of deferring to expert  $h_j$ . Let  $\bar{c}_j(x,y) = 1 - c_j(x,y)$ . We will denote by  $\underline{c}_j\geq 0$  and  $\overline{c}_j\leq 1$  finite lower and upper bounds on the cost  $\bar{c}_j$ , that is  $\bar{c}_j(x,y)\in [\underline{c}_j,\overline{c}_j]$  for all  $(x,y)\in \mathcal{X}\times \mathcal{Y}$ . There are many possible choices for these costs. Our analysis for Theorem 1, Corollary 2, Theorem 6 is general and requires no assumption other than their boundedness. One natural choice is to define cost  $c_{j}$  as a function of expert  $h_j$ 's inaccuracy, for example  $c_{j}(x,y) = \alpha_{j}\mathbb{1}_{\mathsf{h}_{j}(x)\neq y} + \beta_{j}$ , with  $\alpha_{j},\beta_{j} > 0$ , where  $\mathsf{h}_j(x)$  is the prediction made by  $h_j$ th for input  $x$ . Typically, the hyperparameter  $\alpha_{j}$  has two potential values: zero or one. When  $\alpha_{j}$  is set to one, the first term of the formulation pertains to the inaccuracy of expert expert  $h_j$ . Conversely, with  $\alpha_{j}$  set to zero, the first term vanishes, focusing solely on the inference cost. Theorems 5 and 7 are analyzed under this assumption. The  $\beta_{j}$  in the second term corresponds to the inference cost incurred by expert  $h_j$ .
Given a distribution  $\mathcal{D}$  over  $\mathcal{X} \times \mathcal{Y}$ , we will denote by  $\mathcal{E}_{\mathsf{L}_{\mathrm{def}}}(h)$  the expected deferral loss of a hypothesis  $h \in \mathcal{H}$ ,  $\mathcal{E}_{\mathsf{L}_{\mathrm{def}}}(h) = \mathbb{E}_{(x,y) \sim \mathcal{D}}[\mathsf{L}_{\mathrm{def}}(h,x,y)]$ , and by  $\mathcal{E}_{\mathsf{L}_{\mathrm{def}}}^{*}(\mathcal{H}) = \inf_{h \in \mathcal{H}} \mathcal{E}_{\mathsf{L}_{\mathrm{def}}}(h)$  its infimum or best-in-class expected loss. We will adopt similar definitions for other loss functions.
Given a hypothesis set  $\mathcal{H}$ , an  $\mathcal{H}$ -consistency bound [Awasthi et al., 2021a,b, 2022b,a, 2023a,b, Mao et al., 2023h,d,e, Zheng et al., 2023, Mao et al., 2023a,g] for a surrogate loss  $\ell_1$  of a target loss function  $\ell_2$  is an inequality of the form
$$
\forall h \in \mathcal {H}, \mathcal {E} _ {\ell_ {2}} (h) - \mathcal {E} _ {\ell_ {2}} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\ell_ {1}} (\mathcal {H}) \leq \Gamma \left(\mathcal {E} _ {\ell_ {1}} (h) - \mathcal {E} _ {\ell_ {1}} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\ell_ {1}} (\mathcal {H})\right), \tag {2}
$$
where  $\Gamma \colon \mathbb{R}_{+} \to \mathbb{R}_{+}$  is a non-decreasing function with  $\Gamma(0) = 0$  and where  $\mathcal{M}_{\ell}(\mathcal{H})$  is the minimizability gap for the hypothesis set  $\mathcal{H}$  and loss function  $\ell$ .  $\mathcal{M}_{\ell}(\mathcal{H})$  is defined as the difference of the best-in-class expected loss and that of the expected pointwise infimum loss:  $\mathcal{M}_{\ell}(\mathcal{H}) = \mathcal{E}_{\ell}^{*}(\mathcal{H}) - \mathbb{E}_x\left[\inf_{h \in \mathcal{H}} \mathbb{E}_{y|x}\left[\ell(h, x, y)\right]\right]$ . By the super-additivity of the infimum, the minimizability gap is always non-negative. The minimizability gap vanishes when the best-in-class error  $\mathcal{E}_{\ell}^{*}(\mathcal{H})$  coincides with the Bayes error  $\mathcal{E}_{\ell}^{*}(\mathcal{H}_{\mathrm{all}})$ , in particular when  $\mathcal{H} = \mathcal{H}_{\mathrm{all}}$  [Awasthi et al., 2022a,b].
Thus, the  $\mathcal{H}$ -consistency bound (2) relates the minimization of the estimation error for the surrogate loss  $\ell_1$  to that of the target loss  $\ell_2$  in a quantitative way. It is a stronger and more informative guarantee than Bayes-consistency which implies Bayes-consistency, as can be seen by setting  $\mathcal{H} = \mathcal{H}_{\mathrm{all}}$ .
Table 1: Common surrogate losses in standard multi-class classification.
Name	Formulation
Sum exponential loss	\( \ell_{\text{exp}}(\overline{h},x,y) = \sum_{y' \neq y} e^{\overline{h}(x,y') - \overline{h}(x,y)} \).
Multinomial logistic loss	\( \ell_{\text{log}}(\overline{h},x,y) = \log \left( \sum_{y' \in \mathcal{Y} \cup \{0\}} e^{\overline{h}(x,y') - \overline{h}(x,y)} \right) \).
Generalized cross-entropy loss	\( \ell_{\text{gce}}(\overline{h},x,y) = \frac{1}{\alpha} \left[ 1 - \left[ \frac{e^{\overline{h}(x,y)}}{\sum_{y' \in \mathcal{Y} \cup \{0\}} e^{\overline{h}(x,y')}} \right]^\alpha \right], \alpha \in (0,1) \).
Mean absolute error loss	\( \ell_{\text{mae}}(\overline{h},x,y) = 1 - \frac{e^{\overline{h}(x,y)}}{\sum_{y' \in \mathcal{Y} \cup \{0\}} e^{\overline{h}(x,y')}} \).
3 Two-stage  $\mathcal{H}$ -consistent surrogate loss
In this section, we consider an important two-stage scenario for learning to defer with multiple experts. This is a critical scenario in practice for many applications where a predictor is already available, as a result of training with a loss function  $\ell$  supported by  $\mathcal{H}$ -consistency bounds, such as the logistic loss (first stage). The logistic loss coincides with the cross-entropy loss when a softmax activation is applied to the output of a neural network. The problem then consists of learning a deferral function (second stage) to assign the most suitable expert to each input instance.
We first design a new family of surrogate losses for this two-stage scenario (Section 3.1). Next, we show that our surrogate losses benefit from  $\mathcal{H}$ -consistency bounds (Section 3.2). As a by-product, we prove  $\overline{\mathcal{H}}$ -consistency bounds in standard multi-class classification, where  $\overline{\mathcal{H}}$  denotes hypothesis sets with a fixed scoring function (Section 3.3). These bounds have not been studied before and can be of independent interest in other consistency studies. Moreover, we show that, for a constant cost function, our two-stage surrogate losses are realizable  $\mathcal{H}$ -consistent (Section 3.4).
3.1 General surrogate losses
A hypothesis set  $\mathcal{H}$  of functions mapping from  $\mathcal{X} \times [n + n_e]$  to  $\mathbb{R}$  can be decomposed as  $\mathcal{H} = \mathcal{H}_p \times \mathcal{H}_d$ , where  $\mathcal{H}_p$  denotes the hypothesis set spanned by the first  $n$  scores, used for prediction, and  $\mathcal{H}_d$  the hypothesis set spanned by the final  $n_e$  scores, used for deferral. Thus, any  $h \in \mathcal{H}$  can be written as a pair  $h = (h_p, h_d)$  with  $h_p \in \mathcal{H}_p$  and  $h_d \in \mathcal{H}_d$ .
Let  $\ell$  be a surrogate loss for standard multi-class classification with  $n$  classes. We consider the following two-stage scenario: in the first stage,  $h_p$  is learned using the surrogate loss  $\ell_1$ ; in the second stage,  $h_d$  is learned using a surrogate loss  $\mathsf{L}_{h_p}$  that depends on the prediction function  $h_p$  learned in the first stage.
To any  $h_d \in \mathcal{H}_d$ , we associate a hypothesis  $\overline{h}_d$  defined over  $(n_e + 1)$  classes  $\{0, 1, \dots, n_e\}$  by  $\overline{h}_d(x, 0) = \max_{y \in \mathcal{Y}} h_p(x, y)$ , that is the maximal score assigned by  $h_p$  to its predicted label, and  $\overline{h}_d(x, j) = h_d(x, j)$  for  $j \in [n_e]$ . We can then define our suggested surrogate loss for the second stage as follows:
$$
\mathsf {L} _ {h _ {p}} \left(h _ {d}, x, y\right) = \mathbb {1} _ {\mathsf {h} _ {p} (x) = y} \ell_ {2} (\bar {h} _ {d}, x, 0) + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \ell_ {2} (\bar {h} _ {d}, x, j), \tag {3}
$$
where  $\ell_2(\overline{h}_d, x, j)$  is a surrogate loss for standard multi-class classification with  $(n_e + 1)$  categories  $\{0, 1, \ldots, n_e\}$ . Intuitively, the indicator term  $\mathbb{1}_{h(x) \neq n + j}$  in the deferral loss (1) penalizes  $h_d(x, j)$  when it has a small value. Similarly, for a standard surrogate loss  $\ell_2(\overline{h}_d, x, j)$  such as the logistic loss, it penalizes  $\overline{h}_d(x, j)$  when it has a small value as well. In Table 2, we present a summary of examples of such second-stage surrogate losses, where  $\ell_2$  is selected from common surrogate losses in standard multi-class classification defined in Table 1. A detailed derivation is presented in Appendix B.
From the point of view of the second stage,  $x \mapsto \overline{h}_d(x,0) = \max_{y \in \mathcal{Y}} h_p(x,y)$  is a fixed function. We will denote by  $\overline{\mathcal{H}}_d$  the family of hypotheses  $\overline{h}_d \colon \mathcal{X} \times \{0,1,\dots,n_e\} \to \mathbb{R}$  whose first scoring function,  $\overline{h}_d(\cdot,0)$ , is fixed and not to be learned in the second stage.
Our formulation bears some similarity with the design of a surrogate loss function for rejectors in [Cortes et al., 2016a, 2023] for learning with rejection in binary classification, where the cost is a
constant. However, our surrogate loss is tailored to accommodate a general cost function depending on both  $x$  and  $y$  for deferral, in contrast with a constant one, and it allows for multiple deferral options, as opposed to only one rejection option.
3.2  $\mathcal{H}$ -consistency bounds for two-stage surrogate losses
In this section, we provide strong guarantees for two-stage surrogate losses, provided that the first-stage loss function  $\ell_1$  admits an  $\mathcal{H}_p$ -consistency bound, and the second-stage surrogate  $\ell_2$  admits an  $\overline{\mathcal{H}}_d$ -consistency bound.
Theorem 1 ( $\mathcal{H}$ -consistency bounds for score-based two-stage surrogates). Assume that  $\ell_1$  admits an  $\mathcal{H}_p$ -consistency bound and  $\ell_2$  admits an  $\overline{\mathcal{H}}_d$ -consistency bound with respect to the multi-class zero-one classification loss  $\ell_{0-1}$  respectively. Thus, there are non-decreasing concave functions  $\Gamma_1$  and  $\Gamma_2$  such that, for all  $h_p \in \mathcal{H}_p$  and  $\overline{h}_d \in \overline{\mathcal{H}}_d$ , we have
$$
\mathcal {E} _ {\ell_ {0 - 1}} \left(h _ {p}\right) - \mathcal {E} _ {\ell_ {0 - 1}} ^ {*} \left(\mathcal {H} _ {p}\right) + \mathcal {M} _ {\ell_ {0 - 1}} \left(\mathcal {H} _ {p}\right) \leq \Gamma_ {1} \left(\mathcal {E} _ {\ell_ {1}} \left(h _ {p}\right) - \mathcal {E} _ {\ell_ {1}} ^ {*} \left(\mathcal {H} _ {p}\right) + \mathcal {M} _ {\ell_ {1}} \left(\mathcal {H} _ {p}\right)\right)
$$
$$
\mathcal {E} _ {\ell_ {0 - 1}} (\bar {h} _ {d}) - \mathcal {E} _ {\ell_ {0 - 1}} ^ {*} (\overline {{\mathcal {H}}} _ {d}) + \mathcal {M} _ {\ell_ {0 - 1}} (\overline {{\mathcal {H}}} _ {d}) \leq \Gamma_ {2} \big (\mathcal {E} _ {\ell_ {2}} (\bar {h} _ {d}) - \mathcal {E} _ {\ell_ {2}} ^ {*} (\overline {{\mathcal {H}}} _ {d}) + \mathcal {M} _ {\ell_ {2}} (\overline {{\mathcal {H}}} _ {d}) \big).
$$
Then, the following holds for all  $h \in \mathcal{H}$ :
$$
\begin{array}{l} \mathcal {E} _ {\mathrm {L} _ {\text {d e f}}} (h) - \mathcal {E} _ {\mathrm {L} _ {\text {d e f}}} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\mathrm {L} _ {\text {d e f}}} (\mathcal {H}) \\ \leq \Gamma_ {1} \Big (\mathcal {E} _ {\ell_ {1}} (h _ {p}) - \mathcal {E} _ {\ell_ {1}} ^ {*} (\mathcal {H} _ {p}) + \mathcal {M} _ {\ell_ {1}} (\mathcal {H} _ {p}) \Big) + \left(1 + \sum_ {j = 1} ^ {n _ {e}} \overline {{c}} _ {j}\right) \Gamma_ {2} \Bigg (\frac {\mathcal {E} _ {\mathrm {L} _ {h _ {p}}} (h _ {d}) - \mathcal {E} _ {\mathrm {L} _ {h _ {p}}} ^ {*} (\mathcal {H} _ {d}) + \mathcal {M} _ {\mathrm {L} _ {h _ {p}}} (\mathcal {H} _ {d})}{\sum_ {j = 1} ^ {n _ {e}} \underline {{c}} _ {j}} \Bigg). \\ \end{array}
$$
Furthermore, constant factors  $\left(1 + \sum_{j=1}^{n_e} \overline{c}_j\right)$  and  $\frac{1}{\sum_{j=1}^{n_e} \underline{c}_j}$  can be removed when  $\Gamma_2$  is linear.
The proof is given in Appendix D. It consists of expressing the conditional regret of the deferral loss as the sum of two regrets, first by minimizing  $h_d$  for a fixed  $h_p$  and then by minimizing  $h_p$ . Subsequently, we show how each regret can be upper-bounded in terms of the conditional regret of each stage's surrogate loss, leveraging the  $\mathcal{H}_p$ -consistency bound of  $\ell_1$  and  $\overline{\mathcal{H}}_d$ -consistency bound of  $\ell_2$  with respect to the zero-one loss. This, in conjunction with the concavity of functions  $\Gamma_1$  and  $\Gamma_2$ , establishes our  $\mathcal{H}$ -consistency bounds.
Thus, the theorem provides a strong guarantee for the two-stage surrogate losses. A specific instance of Theorem 1 holds for the case where  $\mathcal{E}_{\ell_1}^*(\mathcal{H}_p) = \mathcal{E}_{\ell_1}^*(\mathcal{H}_{\mathrm{all}})$  and  $\mathcal{E}_{\mathsf{L}_{h_p}}^*(\mathcal{H}_d) = \mathcal{E}_{\mathsf{L}_{h_p}}^*(\mathcal{H}_{\mathrm{all}})$ , ensuring that the Bayes-error coincides with the best-in-class error and, consequently,  $\mathcal{M}_{\ell_1}(\mathcal{H}_p) = \mathcal{M}_{\mathsf{L}_{h_p}}(\mathcal{H}_d) = 0$ . Given Theorem 1 and the non-negativity property of  $\mathcal{M}_{\mathsf{L}_{\mathrm{def}}}(\mathcal{H})$ , we can derive the following corollary.