SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {C} _ {\mathrm {L} _ {\text {d e f}}} (h, r, x) - \inf  _ {r \in \mathcal {R}} \mathcal {C} _ {\mathrm {L} _ {\text {d e f}}} (h, r, x) \\ = \underset {y} {\mathbb {E}} \left[ \mathbb {1} _ {\mathfrak {h} (x) \neq y} \mathbb {1} _ {\mathfrak {r} (x) = 0} + \sum_ {j = 1} ^ {n _ {e}} c _ {j} (x, y) \mathbb {1} _ {\mathfrak {r} (x) = j} \right] - \inf  _ {r \in \mathcal {R}} \underset {y} {\mathbb {E}} \left[ \mathbb {1} _ {\mathfrak {h} (x) \neq y} \mathbb {1} _ {\mathfrak {r} (x) = 0} + \sum_ {j = 1} ^ {n _ {e}} c _ {j} (x, y) \mathbb {1} _ {\mathfrak {r} (\mathfrak {x}) = j} \right] \\ = \mathbb {E} _ {y} \left[ \sum_ {j = 0} ^ {n _ {e}} \bar {c} _ {j} (x, y) \right] \times \left[ \sum_ {j = 0} ^ {n _ {e}} \bar {p} (x, j) \ell_ {0 - 1} (\bar {r}, x, j) - \inf  _ {\bar {r} \in \mathcal {R}} \sum_ {j = 0} ^ {n _ {e}} \bar {p} (x, j) \ell_ {0 - 1} (\bar {r}, x, j) \right] \\ \leq \mathbb {E} _ {y} \left[ \sum_ {j = 0} ^ {n _ {e}} \bar {c} _ {j} (x, y) \right] \times \Gamma_ {2} \left[ \sum_ {j = 0} ^ {n _ {e}} \bar {p} (x, j) \ell_ {2} (\bar {r}, x, j) - \inf  _ {\bar {r} \in \mathcal {R}} \sum_ {j = 0} ^ {n _ {e}} \bar {p} (x, j) \ell_ {2} (\bar {r}, x, j) \right] \\ = \mathbb {E} _ {y} \left[ \sum_ {j = 0} ^ {n _ {e}} \bar {c} _ {j} (x, y) \right] \Gamma_ {2} \left(\frac {\mathbb {E} _ {y} \left[ \mathrm {L} _ {h} (r , x , y) \right] - \inf  _ {r \in \mathcal {R}} \mathbb {E} _ {y} \left[ \mathrm {L} _ {h} (r , x , y) \right]}{\mathbb {E} _ {y} \left[ \sum_ {j = 0} ^ {n _ {e}} \bar {c} _ {j} (x , y) \right]}\right) \\ \left(\bar {p} (x, j) = \frac {\mathbb {E} _ {y} [ \bar {c} _ {j} (x , y) ]}{\sum_ {j = 0} ^ {n _ {e}} \mathbb {E} _ {y} [ \bar {c} _ {j} (x , y) ]}\right. \text {a n d f o r m u l t a t i o n (5)}) \\ \leq \left\{ \begin{array}{l l} \Gamma_ {2} \big (\mathcal {C} _ {\mathsf {L} _ {h}} (r, x) - \mathcal {C} _ {\mathsf {L} _ {h}} ^ {*} (\mathcal {R}, x) \big) & \text {w h e n} \Gamma_ {2} \text {i s l i n e a r} \\ \big (1 + \sum_ {j = 1} ^ {n _ {e}} \overline {{c}} _ {j} \big) \Gamma_ {2} \Bigg (\frac {\mathcal {C} _ {\mathsf {L} _ {h}} (r , x) - \mathcal {C} _ {\mathsf {L} _ {h}} ^ {*} (\mathcal {R} , x)}{\sum_ {j = 1} ^ {n _ {e}} \underline {{c}} _ {j}} \Bigg) & \text {o t h e r w i s e} \end{array} \right. \\ \left(\sum_ {j = 1} ^ {n _ {e}} \underline {{c}} _ {j} \leq \mathbb {E} _ {y} \left[ \sum_ {j = 0} ^ {n _ {e}} \bar {c} _ {j} (x, y) \right] \leq 1 + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} \text {a n d} \Gamma_ {2} \text {i s n o n - d e c r e a s i n g}\right) \\ = \left\{ \begin{array}{l l} \Gamma_ {2} \big (\Delta \mathcal {C} _ {\mathrm {L} _ {h}, \mathcal {R}} (r, x) \big) & \text {w h e n} \Gamma_ {2} \text {i s l i n e a r} \\ \Big (1 + \sum_ {j = 1} ^ {n _ {e}} \overline {{c}} _ {j} \Big) \Gamma_ {2} \Bigg (\frac {\Delta \mathcal {C} _ {\mathrm {L} _ {h} , \mathcal {R}} (r , x)}{\sum_ {j = 1} ^ {n _ {e}} c _ {j}} \Bigg) & \text {o t h e r w i s e} \end{array} \right. \\ \end{array}
$$
$$
\begin{array}{l} \inf  _ {r \in \mathcal {R}} \mathcal {C} _ {\mathsf {L} _ {\mathrm {d e f}}} (h, r, x) - \mathcal {C} _ {\mathsf {L} _ {\mathrm {d e f}}} ^ {*} (\mathcal {H}, \mathcal {R}, x) \\ = \inf  _ {r \in \mathcal {R}} \mathcal {C} _ {\mathrm {L} _ {\text {d e f}}} (h, r, x) - \inf  _ {h \in \mathcal {H}, r \in \mathcal {R}} \mathcal {C} _ {\mathrm {L} _ {\text {d e f}}} (h, r, x) \\ = \inf  _ {r \in \mathcal {R}} \mathbb {E} _ {y} \left[ \mathbb {1} _ {\mathfrak {h} (x) \neq y} \mathbb {1} _ {\mathbf {r} (x) = 0} + \sum_ {j = 1} ^ {n _ {e}} c _ {j} (x, y) \mathbb {1} _ {\mathbf {r} (x) = j} \right] - \inf  _ {h \in \mathcal {H}, r \in \mathcal {R}} \mathbb {E} _ {y} \left[ \mathbb {1} _ {\mathfrak {h} (x) \neq y} \mathbb {1} _ {\mathbf {r} (x) = 0} + \sum_ {j = 1} ^ {n _ {e}} c _ {j} (x, y) \mathbb {1} _ {\mathfrak {r} (x) = j} \right] \\ = \min  \left\{\mathbb {E} _ {y} \left[ \mathbb {1} _ {\mathrm {h} (x) \neq y} \right], \mathbb {E} _ {y} \left[ c _ {j} (x, y) \right] \right\} - \min  \left\{\inf  _ {h \in \mathcal {H}} \mathbb {E} _ {y} \left[ \mathbb {1} _ {\mathrm {h} (x) \neq y} \right], \mathbb {E} _ {y} \left[ c _ {j} (x, y) \right] \right\} \\ \leq \mathbb {E} _ {y} \left[ \mathbb {1} _ {\mathsf {h} (x) \neq y} \right] - \inf  _ {h \in \mathcal {H}} \mathbb {E} _ {y} \left[ \mathbb {1} _ {\mathsf {h} (x) \neq y} \right] \\ = \mathcal {C} _ {\ell_ {0 - 1}} (h, x) - \mathcal {C} _ {\ell_ {0 - 1}} ^ {*} (\mathcal {H}, x) \\ = \Delta \mathcal {C} _ {\ell_ {0 - 1}} (h, x) \\ \leq \Gamma_ {1} (\Delta \mathcal {C} _ {\ell} (h, x)). \\ \end{array}
$$
(By  $\mathcal{H}$ -consistency bounds of  $\ell$  under assumption)
Therefore, by (7), we obtain
$$
\begin{array}{l} \mathcal {E} _ {\mathrm {L} _ {\mathrm {d e f}}} (h, r) - \mathcal {E} _ {\mathrm {L} _ {\mathrm {d e f}}} ^ {*} (\mathcal {H}, \mathcal {R}) + \mathcal {M} _ {\mathrm {L} _ {\mathrm {d e f}}} (\mathcal {H}, \mathcal {R}) \\ \leq \left\{ \begin{array}{l l} \mathbb {E} _ {X} \big [ \Gamma_ {2} \big (\Delta \mathcal {C} _ {\mathrm {L} _ {h}, \mathcal {R}} (r, x) \big) \big ] + \mathbb {E} _ {X} \big [ \Gamma_ {1} \big (\Delta \mathcal {C} _ {\ell} (h, x) \big) \big ] & \text {w h e n} \Gamma_ {2} \text {i s l i n e a r} \\ \Big (1 + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} \Big) \mathbb {E} _ {X} \Bigg [ \Gamma_ {2} \Bigg (\frac {\Delta \mathcal {C} _ {\mathrm {L} _ {h} , \mathcal {R}} (r , x)}{\sum_ {j = 1} ^ {n _ {e}} \underline {{c}} _ {j}} \Bigg) \Bigg ] + \mathbb {E} _ {X} \big [ \Gamma_ {1} \big (\Delta \mathcal {C} _ {\ell} (h, x) \big) \big ] & \text {o t h e r w i s e} \end{array} \right. \\ \leq \left\{ \begin{array}{l l} \Gamma_ {2} (\mathbb {E} _ {X} [ \Gamma_ {2} (\Delta \mathcal {C} _ {\mathsf {L} _ {h}, \mathscr {R}} (r, x)) ]) + \Gamma_ {1} (\mathbb {E} _ {X} [ \Delta \mathcal {C} _ {\ell} (h, x) ]) & \text {w h e n} \Gamma_ {2} \text {i s l i n e a r} \\ \big (1 + \sum_ {j = 1} ^ {n _ {e}} \overline {{c}} _ {j} \big) \Gamma_ {2} \Big (\mathbb {E} _ {X} \Big [ \frac {\Delta \mathcal {C} _ {\mathsf {L} _ {h} , \mathscr {R}} (r , x)}{\sum_ {j = 1} ^ {n _ {e}} \underline {{c}} _ {j}} \Big ] \Big) + \Gamma_ {1} (\mathbb {E} _ {X} [ \Delta \mathcal {C} _ {\ell} (h, x) ]) & \text {o t h e r w i s e} \end{array} \right. \\ \end{array}
$$
$(\Gamma_{1}$  and  $\Gamma_{2}$  are concave)
$$
= \left\{ \begin{array}{l l} \Gamma_ {1} \big (\mathcal {E} _ {\ell} (h) - \mathcal {E} _ {\ell} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\ell} (\mathcal {H}) \big) + \Gamma_ {2} \big (\mathcal {E} _ {\mathsf {L} _ {h}} (r) - \mathcal {E} _ {\mathsf {L} _ {h}} ^ {*} (\mathcal {R}) + \mathcal {M} _ {\mathsf {L} _ {h}} (\mathcal {R}) \big) & \text {w h e n} \Gamma_ {2} \text {i s l i n e a r} \\ \Gamma_ {1} \big (\mathcal {E} _ {\ell} (h) - \mathcal {E} _ {\ell} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\ell} (\mathcal {H}) \big) + \big (1 + \sum_ {j = 1} ^ {n _ {e}} \overline {{c}} _ {j} \big) \Gamma_ {2} \Bigg (\frac {\mathcal {E} _ {\mathsf {L} _ {h}} (r) - \mathcal {E} _ {\mathsf {L} _ {h}} ^ {*} (\mathcal {R}) + \mathcal {M} _ {\mathsf {L} _ {h}} (\mathcal {R})}{\sum_ {j = 1} ^ {n _ {e}} c _ {j}} \Bigg) & \text {o t h e r w i s e}, \end{array} \right.
$$
H Proof of realizable consistency for predictor-rejector two-stage surrogate losses (Theorem 7)
Theorem 7 (Realizable  $(\mathcal{H},\mathcal{R})$ -consistency for predictor-rejector two-stage surrogates). Assume that  $\mathcal{H}$  and  $\mathcal{R}$  is closed under scaling and  $c_{j}(x,y) = \beta_{j}, \forall (x,y) \in \mathcal{X} \times \mathcal{Y}$ . Let  $\ell_{1}$  and  $\ell_{2}$  be the logistic loss. Let  $\hat{h}$  be the minimizer of  $\mathcal{E}_{\ell_1}$  and  $\hat{r}$  be the minimizer of  $\mathcal{E}_{\mathsf{L}_h}$ . Then, the following holds for any  $(\mathcal{H},\mathcal{R})$ -realizable distribution,
$$
\mathcal {E} _ {\mathrm {L} \text {d e f}} (\hat {h}, \hat {r}) = 0.
$$
Proof. First, by definition, it is straightforward to see that for any  $h,r,x,y,\mathsf{L}_h(r,x,y)$  upper bounds the deferral loss  $\mathsf{L}_{\mathrm{def}}$ . Consider a data distribution and costs under which there exists  $h^*\in \mathcal{H}$  and  $r^* \in \mathcal{R}$  such that  $\mathcal{E}_{\mathsf{L}_{\mathrm{def}}}(h^{*},r^{*}) = 0$ .
Let  $\hat{h}$  be the minimizer of  $\mathcal{E}_{\ell_1}$  and  $\hat{r}$  the minimizer of  $\mathcal{E}_{\mathsf{L}_{\hat{h}}}$ . Then, using the fact that  $\mathsf{L}_h$  upper bounds the deferral loss  $\mathsf{L}_{\mathrm{def}}$ , we have  $\mathcal{E}_{\mathsf{L}_{\mathrm{def}}}(\hat{h},\hat{r})\leq \mathcal{E}_{\mathsf{L}_{\hat{h}}}(\hat{r})$ .
Next we analyze two cases. If for a point  $x$ , deferral occurs, that is there exists  $j^{*} \in [n_{e}]$ , such that  $r^{*}(x) = j^{*}$ , then we must have  $c_{j^{*}} = 0$  for all  $x$  since the data is realizable and  $c_{j^{*}}$  is constant. Therefore, there exists an optimal  $r^{**}$  deferring all the points to the  $j^{*}$ th expert. Then, by the assumption that  $\mathcal{R}$  is closed under scaling and the Lebesgue dominated convergence theorem, for  $\ell_{2}$  being the logistic loss,  $\mathcal{E}_{\mathsf{L}_{\mathrm{def}}}(\hat{h},\hat{r}) \leq \mathcal{E}_{\mathsf{L}_{\hat{h}}}(\hat{r}) \leq \lim_{\tau \to +\infty} \mathcal{E}_{\mathsf{L}_{\hat{h}}}(\tau r^{**}) = 0$ , where we used the fact that in the limit of  $\tau \to +\infty$  the logistic loss term  $\ell_{2}(\overline{r}^{**},x,j)$  corresponding to  $j \neq j^{*}$  is zero.
On the other hand, if no deferral occurs for any point, that is  $\mathfrak{r}^*(x) = 0$  for any  $x$ , then we must have  $\mathbb{1}_{\mathsf{h}^*(x) \neq y} = 0$  for all  $(x, y)$  since the data is realizable. Using the fact that  $\mathcal{H}$  is closed under scaling and that the logistic loss is realizable  $\mathcal{H}$ -consistent in the standard classification, we obtain  $\mathbb{1}_{\hat{\mathsf{h}}(x) \neq y} = 0$  for all  $(x, y)$ . Then, by the assumption that  $\mathcal{R}$  is closed under scaling and the Lebesgue dominated convergence theorem, for  $\ell_2$  being the logistic loss,  $\mathcal{E}_{\mathsf{L}_{\mathrm{def}}}(\hat{h}, \hat{r}) \leq \mathcal{E}_{\mathsf{L}_{\hat{h}}}(\hat{r}) \leq \lim_{\tau \to +\infty} \mathcal{E}_{\mathsf{L}_{\hat{h}}}(\tau r^*) = 0$ , where we used the fact that in the limit of  $\tau \to +\infty$  the logistic loss term  $\ell_2(\overline{r}^*, x, j)$  corresponding to  $j \neq 0$  is zero.
Therefore, the optimal solution from minimizing predictor-rejector two-stage surrogates leads to a zero error solution of the deferral loss, which proves that the predictor-rejector two-stage surrogate loss is realizable consistent.