SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Corollary 2. Assume that  $\ell$  satisfies the same assumption as in Theorem 1. Then, for all  $h\in \mathcal{H}$  and any distribution such that  $\mathcal{E}_{\ell_1}^* (\mathcal{H}_p) = \mathcal{E}_{\ell_1}^* (\mathcal{H}_{\mathrm{all}})$  and  $\mathcal{E}_{\mathsf{L}_{h_p}}^* (\mathcal{H}_d) = \mathcal{E}_{\mathsf{L}_{h_p}}^* (\mathcal{H}_{\mathrm{all}})$ , we have
$$
\mathcal {E} _ {\mathsf {L} _ {\mathrm {d e f}}} (h) - \mathcal {E} _ {\mathsf {L} _ {\mathrm {d e f}}} ^ {*} (\mathcal {H}) \leq \Gamma_ {1} \big (\mathcal {E} _ {\ell_ {1}} (h _ {p}) - \mathcal {E} _ {\ell_ {1}} ^ {*} (\mathcal {H} _ {p}) \big) + \left(1 + \sum_ {j = 1} ^ {n _ {e}} \overline {{c}} _ {j}\right) \Gamma_ {2} \left(\frac {\mathcal {E} _ {\mathsf {L} _ {h _ {p}}} (h _ {d}) - \mathcal {E} _ {\mathsf {L} _ {h _ {p}}} ^ {*} (\mathcal {H} _ {d})}{\sum_ {j = 1} ^ {n _ {e}} \underline {{c}} _ {j}}\right),
$$
where the constant factors  $\left(1 + \sum_{j=1}^{n_e} \overline{c}_j\right)$  and  $\frac{1}{\sum_{j=1}^{n_e} \underline{c}_j}$  can be removed when  $\Gamma_2$  is linear.
Corollary 2 implies that when the estimation error of the first-stage surrogate loss,  $\mathcal{E}_{\ell_1}(h_p) - \mathcal{E}_{\ell_1}^* (\mathcal{H}_p)$ , is reduced to  $\epsilon_{1}$ , and the estimation error of the second-stage surrogate loss,  $\mathcal{E}_{\mathrm{L}_{h_p}}(h_d) - \mathcal{E}_{\mathrm{L}_{h_p}}^* (\mathcal{H}_d)$ , is reduced to  $\epsilon_{2}$ , the estimation error of the deferral loss,  $\mathcal{E}_{\mathrm{L}_{\mathrm{def}}}(h) - \mathcal{E}_{\mathrm{L}_{\mathrm{def}}}^* (\mathcal{H})$ , is upper-bound by
$$
\Gamma_ {1} (\epsilon_ {1}) + \left(1 + \sum_ {j = 1} ^ {n _ {e}} \overline {{c}} _ {j}\right) \Gamma_ {2} \left(\frac {\epsilon_ {2}}{\sum_ {j = 1} ^ {n _ {e}} \underline {{c}} _ {j}}\right).
$$
The common surrogate losses mentioned earlier all satisfy the first-stage requirement; however, it was unclear if they would meet the second-stage criterion since the  $\overline{\mathcal{H}}_d$ -consistency bound is for hypothesis sets  $\overline{\mathcal{H}}_d$  with a fixed first scoring function. This has not been previously studied in the literature. In the next section, we prove for the first time that common multi-class surrogate losses,
Table 2: Examples for score-based second-stage surrogate losses (3).
l2	Lhp
lexp	1h_p(x)=y ∑i=1ne e(h(x,n+i)-maxy∈yh(x,y)+ ∑j=1ne c_j(x,y)[∑i=1n_e h(x,n+i)-h(x,n+j)+ e(maxy∈yh(x,y)-h(x,n+j)]
llog	-1h_p(x)=y log(e_maxy∈yh(x,y)+ ∑i=1ne c_j(x,y) - ∑j=1ne c_j(x,y) log(e^h(x,n+j)/e_maxy∈yh(x,y)+ ∑i=1ne c_j(x,n+i))
lgce	1h_p(x)=y 1/α [1 - (e_maxy∈yh(x,y)/e_maxy∈yh(x,y)+ ∑i=1ne c_j(x,y) + ∑j=1ne c_j(x,y) 1/α [1 - (e^h(x,n+j)/e_maxy∈yh(x,y)+ ∑i=1ne c_j(x,n+i)]^α]
lmae	1h_p(x)=y [1 - e_maxy∈yh(x,y)/e_maxy∈yh(x,y)+ ∑i=1ne c_j(x,y) [1 - e^h(x,n+j)/e_maxy∈yh(x,y)+ ∑i=1ne c_j(x,n+i)]
such as the logistic loss, satisfy this requirement and can be incorporated into both the first and second stage. Hence, based on [Mao et al., 2023h, Theorem 1] and Theorem 3 in Section 3.3, when using logistic loss in both stages, the concave functions are  $\Gamma_1(t) = \Gamma_2(t) = \sqrt{2t}$ , and thus Corollary 2 yields the following  $\mathcal{H}$ -consistency bound:
$$
\mathcal {E} _ {\mathsf {L} _ {\mathrm {d e f}}} (h) - \mathcal {E} _ {\mathsf {L} _ {\mathrm {d e f}}} ^ {*} (\mathcal {H}) \leq \sqrt {2} [ \mathcal {E} _ {\ell_ {1}} (h _ {p}) - \mathcal {E} _ {\ell_ {1}} ^ {*} (\mathcal {H} _ {p}) ] ^ {\frac {1}{2}} + \sqrt {2} [ 1 + \sum_ {j = 1} ^ {n _ {e}} \overline {{c}} _ {j} ] [ \frac {\mathcal {E} _ {\mathsf {L} _ {h _ {p}}} (h _ {d}) - \mathcal {E} _ {\mathsf {L} _ {h _ {p}}} ^ {*} (\mathcal {H} _ {d})}{\sum_ {j = 1} ^ {n _ {e}} \underline {{c}} _ {j}} ] ^ {\frac {1}{2}}.
$$
In particular, the bound implies the Bayes-consistency of the two-stage surrogate loss when  $\ell_1 = \ell_2 = \ell_{\log}$ . Similarly, for other choices of  $\ell_1$  and  $\ell_2$  defined in Table 1, the two-stage surrogate loss benefits from an  $\mathcal{H}$ -consistency bound and is also Bayes-consistent.
3.3  $\overline{\mathcal{H}}$ -consistency bounds for standard surrogate loss functions
In this section, we seek to derive  $\overline{\mathcal{H}}$ -consistency bounds for common surrogate losses defined in Table 1 in the standard multi-class classification scenario. Recall that the first scoring function of hypotheses in  $\overline{\mathcal{H}}_d$  is the function  $\max_{y \in \mathcal{Y}} h_p(\cdot, y)$ . Here, for any given function  $\lambda$  mapping from  $\mathcal{X}$  to  $\mathbb{R}$ , we define the hypothesis set  $\overline{\mathcal{H}}$  augmented by  $\lambda$  in a similar way, that is to any  $h \in \mathcal{H}$  we associate a hypothesis  $\overline{h} \in \overline{\mathcal{H}}$  defined by  $\overline{h}(x,0) = \lambda(x)$  and  $\overline{h}(x,j) = h(x,j)$  for  $j \geq 1$ . These  $\overline{\mathcal{H}}$ -consistency bounds offer strong guarantees when the loss functions in Table 1 are used in the second stage of the two-stage learning to defer surrogate losses (3) instantiated in Table 2. We believe that these results are of independent interest and can admit other applications in the study of  $\mathcal{H}$ -consistency bounds. As with [Mao et al., 2023h], we assume that the hypothesis set  $\mathcal{H}$  is symmetric and complete. A hypothesis set is said to be symmetric if there exists a family  $\mathcal{F}$  of functions  $f$  mapping from  $\mathcal{X}$  to  $\mathbb{R}$  such that  $\{[h(x,1), \ldots, h(x,n)]: h \in \mathcal{H}\} = \{[f_1(x), \ldots, f_n(x)]: f_1, \ldots, f_n \in \mathcal{F}\}$ , for any  $x \in \mathcal{X}$ . A hypothesis set  $\mathcal{H}$  is said to be complete if the set of scores it generates spans  $\mathbb{R}$ , that is,  $\{h(x,y): h \in \mathcal{H}\} = \mathbb{R}$ , for any  $(x,y) \in \mathcal{X} \times \mathcal{Y}$ .
Note that for a symmetric and complete  $\mathcal{H}$ , the associated  $\overline{\mathcal{H}}$  is not symmetric and complete. Therefore, the proof of Mao et al. [2023h] cannot be generalized to our setting. Our proofs are presented in Appendix E. We give a new method for upper bounding the conditional regret of the zero-one loss by that of a surrogate loss. To achieve this, we upper bound the minimal conditional surrogate loss by the conditional loss of a carefully constructed hypothesis in  $\overline{\mathcal{H}}$  denoted by  $\overline{h}_{\mu}$ . The resulting softmax  $S_{\mu}$  of this hypothesis only differs from the original softmax  $S$  corresponding to  $\overline{h}$  on exactly two of the labels.
Theorem 3 ( $\overline{\mathcal{H}}$ -consistency bounds). Assume that  $\mathcal{H}$  is symmetric and complete. Then, for any function  $\lambda$  mapping from  $\mathcal{X}$  to  $\mathbb{R}$ , hypothesis  $\overline{h}$  in the associated hypothesis set  $\overline{\mathcal{H}}$  and any distribution, the following inequality holds:
$$
\mathcal {E} _ {\ell_ {0 - 1}} \left(\overline {{h}}\right) - \mathcal {E} _ {\ell_ {0 - 1}} ^ {*} \left(\overline {{\mathcal {H}}}\right) \leq \Gamma \left(\mathcal {E} _ {\ell} \left(\overline {{h}}\right) - \mathcal {E} _ {\ell} ^ {*} \left(\overline {{\mathcal {H}}}\right) + \mathcal {M} _ {\ell} \left(\overline {{\mathcal {H}}}\right)\right) - \mathcal {M} _ {\ell_ {0 - 1}} \left(\overline {{\mathcal {H}}}\right),
$$
where  $\Gamma(t) = \sqrt{2t}$  for  $\ell = \ell_{\mathrm{log}}$  or  $\ell_{\mathrm{exp}}$ ;  $\Gamma(t) = \sqrt{2(n + 1)^{\alpha}t}$  for  $\ell = \ell_{\mathrm{gce}}$ ; and  $\Gamma(t) = (n + 1)t$  for  $\ell = \ell_{\mathrm{mae}}$ .
Let us underscore that our proof technique is novel and distinct from the approach used in [Mao et al., 2023h]. Their method is tailored for hypothesis sets where each score can span across  $\mathbb{R}$ . This is not applicable in our context where the hypothesis set adheres to a predefined scoring function.
In their proof, to set an upper bound on the estimation error of the zero-one loss using that of the surrogate loss, they select an auxiliary function  $\overline{h}_{\mu}$  for any hypothesis  $h$ . This function is contingent on the distinct scores of  $h$ . Subsequently, the authors choose an optimal  $\mu$  to set these bounds. Nevertheless, if any of  $h$ 's scores are fixed, an optimal  $\mu$  does not exist, preventing the establishment of a meaningful bound. Instead, our new proof method overcomes this limitation by choosing  $\overline{h}_{\mu}$  based on the softmax, as the softmax corresponding to the label zero can still vary due to the influence of changes in other scores, even when the scoring function on label zero is fixed.
3.4 Realizable  $\mathcal{H}$ -consistency
Recently, Mozannar et al. [2023] showed that even in the straightforward single-expert setting, existing Bayes-consistent single-stage surrogate losses [Mozannar and Sontag, 2020, Verma and Nalisnick, 2022] are not realizable  $\mathcal{H}$ -consistent [Long and Servedio, 2013, Zhang and Agarwal, 2020] for learning with deferral. This can pose significant challenges when learning with a restricted hypothesis set  $\mathcal{H}$ , even for simple linear models. Instead, they proposed a new surrogate loss that is realizable  $\mathcal{H}$ -consistent when  $\mathcal{H}$  is closed under scaling, meaning that it satisfies the condition  $h\in \mathcal{H}\Rightarrow \tau h\in \mathcal{H}$  for all  $\tau$  in the set of real numbers. However, they stated that they could not prove or disprove whether their proposed surrogate loss is Bayes-consistent. Consequently, it has become crucial to identify a surrogate loss that is both consistent and realizable-consistent, which has remained an open problem.
Definition 4 (Realizable  $\mathcal{H}$ -consistency). A surrogate loss  $\mathsf{L}$  is considered a realizable  $\mathcal{H}$ -consistent loss function for the deferral loss  $\mathsf{L}_{\mathrm{def}}$  if, for any distribution that is  $\mathcal{H}$ -realizable, that is, there exists a zero loss solution  $h^* \in \mathcal{H}$  with  $\mathcal{E}_{\mathsf{L}_{\mathrm{def}}}(h^*) = 0$ , optimizing the surrogate loss results in obtaining the zero-error solution:
$$
\mathcal {E} _ {\mathrm {L}} \left(h _ {n}\right) - \mathcal {E} _ {\mathrm {L}} ^ {*} (\mathcal {H}) \xrightarrow {n \rightarrow + \infty} 0 \Longrightarrow \mathcal {E} _ {\mathrm {L} _ {\text {d e f}}} \left(h _ {n}\right) - \mathcal {E} _ {\mathrm {L} _ {\text {d e f}}} ^ {*} (\mathcal {H}) \xrightarrow {n \rightarrow + \infty} 0.
$$
In the following result, we show that our two-stage surrogate losses are realizable  $\mathcal{H}$ -consistent. Combined with their Bayes-consistency properties, which have already been established in Section 3.2, we effectively find surrogate losses that are both Bayes-consistent and realizable consistent in the multi-expert setting, including the single-expert setting as a special case. For simplicity, here, we study the case where  $\ell_1 = \ell_2 = \ell_{\log}$ , a similar proof holds for other choices of  $\ell_1$  and  $\ell_2$  defined in Table 1. The proof is included in Appendix F.
Theorem 5 (Realizable  $\mathcal{H}$ -consistency for score-based two-stage surrogates). Assume that  $\mathcal{H}$  is closed under scaling and  $c_{j}(x,y) = \beta_{j},\forall (x,y)\in \mathcal{X}\times \mathcal{Y}$ . Let  $\ell_1$  and  $\ell_2$  be the logistic loss. Let  $\hat{h}_p$  be the minimizer of  $\mathcal{E}_{\ell_1}$  and  $\hat{h}_d$  be the minimizer of  $\mathcal{E}_{\mathsf{L}_{\hat{h}_p}}$  such that  $\mathcal{E}_{\mathsf{L}_{\hat{h}_p}}(\hat{h}_d) = \min_h\mathcal{E}_{\mathsf{L}_{h_p}}(h_d)$ . Then, the following equality holds for any  $(\mathcal{H},\mathcal{R})$ -realizable distribution,
$$
\mathcal {E} _ {\mathrm {L} _ {\mathrm {d e f}}} (\hat {h}) = 0, \text {w h e r e} \hat {h} = (\hat {h} _ {p}, \hat {h} _ {d}).
$$
Theorem 5 suggests that when the estimation error of the first-stage surrogate loss,  $\mathcal{E}_{\ell_1}(h_p^n) - \mathcal{E}_{\ell_1}^*(\mathcal{H}_p) \xrightarrow{n \to +\infty} 0$ , and the estimation error of the second-stage surrogate loss,  $\mathcal{E}_{\mathsf{L}_{h_p}}(h_d^n) - \mathcal{E}_{\mathsf{L}_{h_p}}^*(\mathcal{H}_d) \xrightarrow{n \to +\infty} 0$ , the estimation error of the deferral loss,  $\mathcal{E}_{\mathsf{L}_{\mathrm{def}}} (h^n) - \mathcal{E}_{\mathsf{L}_{\mathrm{def}}}^*(\mathcal{H}) \xrightarrow{n \to +\infty} 0$ . This result demonstrates that our two-stage surrogate losses are not only Bayes-consistent, but also realizable  $\mathcal{H}$ -consistent when only the inference cost  $(\beta_j)$  exists.
4 Predictor-rejector setting
The results of the previous sections were all given for the score-based setting. We note that another popular setting in learning with deferral/abstention is the predictor-rejector setting [Cortes et al., 2016a, 2023], where the deferral corresponds to a separate function  $\mathcal{R}$  instead of extra scores. For completeness, we introduce this setting as well. Here too, we design a family of two-stage surrogate losses benefiting from both  $(\mathcal{H},\mathcal{R})$ -consistency bounds and realizable consistency. For simplicity, we overload the notation as with score-based setting based on the context.
Let  $\mathcal{H}$  be a hypothesis set of prediction functions mapping from  $\mathcal{X} \times \mathcal{Y}$  to  $\mathbb{R}$ . The label predicted for  $x \in \mathcal{X}$  using a hypothesis  $h \in \mathcal{H}$  is denoted by  $\mathsf{h}(x)$  and defined as one with the highest score,
Table 3: Examples for predictor-rejector second-stage surrogate losses (5).
l2	Lh_p
l_exp	1h(x)=y ∑nei=1e-ri(x)+ ∑nej=1cbarj(x,y)[∑nei=1,i≠j e^rj(x)-ri(x)+ erj(x)]
l_log	-1h(x)=y log(1+ ∑nei=1e-ri(x)) - ∑nej=1cbarj(x,y) log(e^-rj(x)/1+ ∑nei=1e-ri(x))
l_gce	1h(x)=y 1/α[1 - [1/1+ ∑nei=1e-ri(x)]^α] + ∑nej=1cbarj(x,y) 1/α[1 - [e^-rj(x)/1+ ∑nei=1e-ri(x)]^α]
l_mae	1h(x)=y [1 - 1/1+ ∑nei=1e-ri(x)] + ∑nej=1cbarj(x,y) [1 - e^-rj(x)/1+ ∑nei=1e-ri(x)]
$h(x) = \operatorname{argmax}_{y\in \mathcal{Y}}h(x,y)$ , with an arbitrary but fixed deterministic strategy for breaking ties. Let  $\mathcal{R}$  be a family of deferring functions mapping from  $\mathcal{X}$  to  $\mathbb{R}^{n_e}$ , where  $n_e$  is the number of experts. A deferral  $r = (r_1,\ldots ,r_{n_e})\in \mathcal{R}$  is used to defer the prediction on input  $x$  to the  $j$ th expert  $h_j$  if  $r_j(x)\leq 0$  and  $r_j(x) < \min_{i = 1,i\neq j}^{n_e}r_i(x)$ , in which case a cost  $c_{j}(x,y) = 1 - \bar{c}_{j}(x,y)\in [1 - \bar{c}_{j},1 - \underline{c}_{j}]$  is incurred with  $0 < \underline{c}_j\leq \overline{c}_j\leq 1$ . A natural choice of the cost is  $c_{j}(x,y) = \alpha_{j}\mathbb{1}_{h_{j}(x)\neq y} + \beta_{j}$ , where  $\alpha_{j},\beta_{j} > 0$  and  $h_j$  is the prediction of the  $j$ th expert. The  $\beta_{j}$  in the second term corresponds to the inference cost incurred by expert  $h_j$ . Let  $r_0 = 0$  and define  $r(x) = 0$  if  $r_0(x) < \min_{j\in [n_e]}r_j(x)$ ; otherwise,  $r(x) = \operatorname*{argmin}_{j\in [n_e]}r_j(x)$ , with an arbitrary but fixed deterministic strategy for breaking ties. The learning to defer loss  $\mathsf{L}_{\mathrm{def}}$  with  $n_e$  experts is defined as follows for any  $(h,r)\in \mathcal{H}\times \mathcal{R}$  and  $(x,y)\in \mathcal{X}\times \mathcal{Y}$ :
$$
\mathsf {L} _ {\text {d e f}} (h, r, x, y) = \mathbb {1} _ {\mathrm {h} (x) \neq y} \mathbb {1} _ {\mathrm {r} (x) = 0} + \sum_ {j = 1} ^ {n _ {e}} c _ {j} (x, y) \mathbb {1} _ {\mathrm {r} (x) = j}. \tag {4}
$$
Given a distribution  $\mathcal{D}$  over  $\mathcal{X} \times \mathcal{Y}$ , we will denote by  $\mathcal{E}_{\mathsf{L}_{\mathrm{def}}}(h,r)$  the expected deferral loss of a predictor  $h \in \mathcal{H}$  and a deferral  $r \in \mathcal{R}$ ,  $\mathcal{E}_{\mathsf{L}_{\mathrm{def}}}(h,r) = \mathbb{E}_{(x,y) \sim \mathcal{D}}[\mathsf{L}_{\mathrm{def}}(h,r,x,y)]$ , and by  $\mathcal{E}_{\mathsf{L}_{\mathrm{def}}}^*(\mathcal{H},\mathcal{R}) = \inf_{h \in \mathcal{H}, r \in \mathcal{R}} \mathcal{E}_{\mathsf{L}_{\mathrm{def}}}(h,r)$  its infimum or best-in class expected loss. We will adopt similar definitions for other loss functions. We denote by  $\mathcal{M}_{\mathsf{L}}(\mathcal{H},\mathcal{R}) = \mathcal{E}_{\mathsf{L}}^*(\mathcal{H},\mathcal{R}) - \mathbb{E}_x\left[\inf_{h \in \mathcal{H}, r \in \mathcal{R}} \mathbb{E}_{y|x}[\mathsf{L}(h,r,x,y)]\right]$  the minimizability gap for hypothesis sets ( $\mathcal{H},\mathcal{R}$ ) and a loss function  $\mathsf{L}$ .
Let  $\ell_1$  be a surrogate loss for standard multi-class classification with  $n$  classes. We consider the following two-stage scenario: in the first stage, a predictor  $h$  is learned using the surrogate loss  $\ell_1$ ; in the second stage,  $r$  is learned using a surrogate loss  $\mathsf{L}_h$  that depends on the prediction function  $h$  learned in the first stage.
To any  $r \in \mathcal{R}$ , we associate a hypothesis  $\overline{r}$  defined over  $(n_e + 1)$  classes  $\{0, 1, \dots, n_e\}$  by  $\overline{r}(x, 0) = 0$ , that is zero scoring function, and  $\overline{r}(x, j) = -r_j(x)$  for  $j \in [n_e]$ . We can then define our suggested surrogate loss for the second stage:
$$
\mathsf {L} _ {h} (r, x, y) = \mathbb {1} _ {\mathrm {h} (x) = y} \ell_ {2} (\bar {r}, x, 0) + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \ell_ {2} (\bar {r}, x, j). \tag {5}
$$
Here,  $\ell_2(\overline{r},x,j)$  is a surrogate loss for standard multi-class classification with  $(n_e + 1)$  categories  $\{0,1,\ldots ,n_e\}$ . Intuitively, the indicator term  $\mathbb{1}_{r(x)\neq j}$  in the deferral loss penalizes  $r_j(x)$  when it has a large value. However, a standard surrogate loss  $\ell_2(\overline{r},x,j)$  such as the logistic loss penalizes  $\overline{r} (x,j)$  when it has a small value. This is why we use a negative sign in the definition of  $\overline{r}$  to maintain consistency between the definitions of  $\mathsf{L}_h$  and  $\mathsf{L}_{\mathrm{def}}$ . In Table 3, we present a summary of examples of such second-stage surrogate losses, where  $\ell_2$  is selected from common surrogate losses in standard multi-class classification defined in Table 1. A detailed derivation is presented in Appendix C.
From the point of view of the second stage, we will denote by  $\overline{\mathcal{R}}$  the family of hypotheses  $\overline{r}:\mathcal{X}\times$ $\{0,1,\ldots ,n_e\} \to \mathbb{R}$  whose first scoring function,  $\overline{r} (\cdot ,0)$ , is zero function and will not be learned in the second stage. We will provide strong guarantees for two-stage surrogate losses, provided that the first-stage loss function  $\ell_1$  admits an  $\mathcal{H}$ -consistency bound, and the second-stage loss function  $\ell_{2}$  admits an  $\overline{\mathcal{R}}$ -consistency bound.
Theorem 6 (( $\mathcal{H}, \mathcal{R}$ )-consistency bounds for predictor-rejector two-stage surrogates). Assume that  $\ell_1$  admits an  $\mathcal{H}$ -consistency bound and  $\ell_2$  admits an  $\overline{\mathcal{R}}$ -consistency bound with respect to the multi-class zero-one classification loss  $\ell_{0-1}$  respectively. Thus, there are non-decreasing concave
functions  $\Gamma_{1}$  and  $\Gamma_{2}$  such that, for all  $h\in \mathcal{H}$  and  $\overline{r}\in \overline{\mathcal{R}}$  we have
$$
\begin{array}{l} \mathcal {E} _ {\ell_ {0 - 1}} (h) - \mathcal {E} _ {\ell_ {0 - 1}} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\ell_ {0 - 1}} (\mathcal {H}) \leq \Gamma_ {1} \left(\mathcal {E} _ {\ell_ {1}} (h) - \mathcal {E} _ {\ell_ {1}} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\ell_ {1}} (\mathcal {H}) \right. \\ \mathcal {E} _ {\ell_ {0 - 1}} (\overline {{r}}) - \mathcal {E} _ {\ell_ {0 - 1}} ^ {*} (\overline {{\mathcal {R}}}) + \mathcal {M} _ {\ell_ {0 - 1}} (\overline {{\mathcal {R}}}) \leq \Gamma_ {2} \big (\mathcal {E} _ {\ell_ {2}} (\overline {{r}}) - \mathcal {E} _ {\ell_ {2}} ^ {*} (\overline {{\mathcal {R}}}) + \mathcal {M} _ {\ell_ {2}} (\overline {{\mathcal {R}}}) \big). \\ \end{array}
$$
Then, the following holds for all  $h \in \mathcal{H}$  and  $r \in \mathcal{R}$ :
$$
\begin{array}{l} \mathcal {E} _ {\mathrm {L} _ {\mathrm {d e f}}} (h, r) - \mathcal {E} _ {\mathrm {L} _ {\mathrm {d e f}}} ^ {*} (\mathcal {H}, \mathcal {R}) + \mathcal {M} _ {\mathrm {L} _ {\mathrm {d e f}}} (\mathcal {H}, \mathcal {R}) \\ \leq \Gamma_ {1} \big (\mathcal {E} _ {\ell_ {1}} (h) - \mathcal {E} _ {\ell_ {1}} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\ell_ {1}} (\mathcal {H}) \big) + \left(1 + \sum_ {j = 1} ^ {n _ {e}} \overline {{c}} _ {j}\right) \Gamma_ {2} \left(\frac {\mathcal {E} _ {\mathrm {L} _ {h}} (r) - \mathcal {E} _ {\mathrm {L} _ {h}} ^ {*} (\mathcal {R}) + \mathcal {M} _ {\mathrm {L} _ {h}} (\mathcal {R})}{\sum_ {j = 1} ^ {n _ {e}} \underline {{c}} _ {j}}\right), \\ \end{array}
$$
As with the score-based setting, a specific instance of Theorem 6 holds for the case where  $\mathcal{E}_{\ell_1}^*(\mathcal{H}) = \mathcal{E}_{\ell_1}^*(\mathcal{H}_{\mathrm{all}})$  and  $\mathcal{E}_{\mathsf{L}_h}^*(\mathcal{R}) = \mathcal{E}_{\mathsf{L}_h}^*(\mathcal{R}_{\mathrm{all}})$ , ensuring that the Bayes-error coincides with the best-in-class error and, consequently,  $\mathcal{M}_{\ell_1}(\mathcal{H}) = \mathcal{M}_{\mathsf{L}_h}(\mathcal{R}) = 0$ . In these cases, when the estimation error of the first-stage surrogate loss,  $\mathcal{E}_{\ell_1}(h) - \mathcal{E}_{\ell_1}^*(\mathcal{H})$ , is reduced to  $\epsilon_1$ , and the estimation error of the second-stage surrogate loss,  $\mathcal{E}_{\mathsf{L}_h}(r) - \mathcal{E}_{\mathsf{L}_h}^*(\mathcal{R})$ , is reduced to  $\epsilon_2$ , the estimation error of the deferral loss,  $\mathcal{E}_{\mathsf{L}_{\mathrm{def}}}(h,r) - \mathcal{E}_{\mathsf{L}_{\mathrm{def}}}^*(\mathcal{H},\mathcal{R})$ , is upper bounded by
$$
\Gamma_ {1} \left(\epsilon_ {1}\right) + \left(1 + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j}\right) \Gamma_ {2} \left(\frac {\epsilon_ {2}}{\sum_ {j = 1} ^ {n _ {e}} \underline {{c}} _ {j}}\right).
$$
Next, we show that our two-stage surrogate losses are realizable  $(\mathcal{H},\mathcal{R})$ -consistent. We say that the distribution is  $(\mathcal{H},\mathcal{R})$ -realizable, if there exists a zero error solution  $(h^{*},r^{*})\in \mathcal{H}\times \mathcal{R}$  with  $\mathcal{E}_{\mathrm{L_{def}}}(h^*,r^*) = 0$
Theorem 7 (Realizable  $(\mathcal{H},\mathcal{R})$ -consistency for predictor-rejector two-stage surrogates). Assume that  $\mathcal{H}$  and  $\mathcal{R}$  is closed under scaling and  $c_{j}(x,y) = \beta_{j}, \forall (x,y) \in \mathcal{X} \times \mathcal{Y}$ . Let  $\ell_{1}$  and  $\ell_{2}$  be the logistic loss. Let  $\hat{h}$  be the minimizer of  $\mathcal{E}_{\ell_1}$  and  $\hat{r}$  be the minimizer of  $\mathcal{E}_{\ell_{\hat{h}}}$ . Then, the following holds for any  $(\mathcal{H},\mathcal{R})$ -realizable distribution,
$$
\mathcal {E} _ {\mathrm {L} \mathrm {d e f}} (\hat {h}, \hat {r}) = 0.
$$
The proof is included in Appendix H. Theorem 7 suggests that the two-stage surrogate loss is realizable consistent: when the estimation error of the first-stage surrogate loss  $\mathcal{E}_{\ell_1}(h_n) - \mathcal{E}_{\ell_1}^*(\mathcal{H}) \xrightarrow{n \to +\infty} 0$ , and the estimation error of the second-stage surrogate loss  $\mathcal{E}_{\mathrm{L}_h}(r_n) - \mathcal{E}_{\mathrm{L}_h}^*(\mathcal{R}) \xrightarrow{n \to +\infty} 0$ , the estimation error of the deferral loss,  $\mathcal{E}_{\mathrm{L}_{\mathrm{def}}}(h_n, r_n) - \mathcal{E}_{\mathrm{L}_{\mathrm{def}}}^*(\mathcal{H}, \mathcal{R}) \xrightarrow{n \to +\infty} 0$ . By Theorem 6 and Theorem 7, in the predictor-rejector setting, we also effectively find both Bayes-consistent and realizable consistent surrogate losses with multiple experts when only the inference cost  $(\beta_j)$  exists.