SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where we let  $\mathcal{S}(x,y) = \frac{e^{\overline{h}(x,y)}}{\sum_{y' \in \mathcal{Y} \cup \{0\}} e^{\overline{h}(x,y')}} \in [0,1]$  for any  $y \in \mathcal{Y} \cup \{0\}$  with the constraint that  $\sum_{y \in \mathcal{Y} \cup \{0\}} \mathcal{S}(x,y) = 1$ . Let  $y_{\max} = \operatorname{argmax}_{y \in \mathcal{Y} \cup \{0\}} p(x,y)$ , where we choose the label with the highest index under the natural ordering of labels as the tie-breaking strategy. For any  $\overline{h} \in \mathcal{H}$  such that  $\overline{h}(x) \neq y_{\max}$  and  $x \in \mathcal{X}$ , by the symmetry and completeness of  $\mathcal{H}$ , we can always find a family of hypotheses  $\left\{\overline{h}_\mu : \mu \in [-\mathcal{S}(x,y_{\max}), \mathcal{S}(x,\overline{h}(x))]\right\} \subset \overline{\mathcal{H}}$  such that  $\mathcal{S}_\mu(x,\cdot) = \frac{e^{\overline{h}_\mu(x,\cdot)}}{\sum_{y' \in \mathcal{Y} \cup \{0\}} e^{\overline{h}_\mu(x,y')} }$  take the following values:
$$
\mathcal {S} _ {\mu} (x, y) = \left\{ \begin{array}{l l} \mathsf {S} (x, y) & \text {i f} y \notin \big \{y _ {\max }, \overline {{\mathsf {h}}} (x) \big \} \\ \mathsf {S} (x, y _ {\max }) + \mu & \text {i f} y = \overline {{\mathsf {h}}} (x) \\ \mathsf {S} (x, \overline {{\mathsf {h}}} (x)) - \mu & \text {i f} y = y _ {\max }. \end{array} \right.
$$
$$
\sum_ {y \in \mathcal {Y}} \mathcal {S} _ {\mu} (x, y) = \sum_ {y \in \mathcal {Y}} \mathcal {S} (x, y) = 1, \forall \mu \in [ - \mathcal {S} (x, y _ {\max}), \mathcal {S} (x, \overline {{\mathsf {h}}} (x)) ].
$$
$$
\begin{array}{l} \Delta \mathcal {C} _ {\ell_ {\mathrm {e x p}}, \overline {{\mathcal {H}}}} (\overline {{h}}, x) \\ = \mathcal {C} _ {\ell_ {\exp}} (\bar {h}, x) - \mathcal {C} _ {\ell_ {\exp}} ^ {*} (\overline {{\mathcal {H}}}, x) \\ \geq \mathcal{C}_{\ell_{\exp}}\bigl(\overline{h},x\bigr) - \inf_{\mu \in [-\mathcal{S}(x,y_{\max}),\mathcal{S}(x,\overline{\mathsf{h}} (x))]} \mathcal{C}_{\ell_{\exp}}\bigl(\overline{h}_{\mu},x\bigr) \\ = \sup  _ {\mu \in [ - \mathbb {S} (x, y _ {\max }), \mathbb {S} (x, \overline {{\mathsf {h}}} (x)) ]} \left\{p (x, y _ {\max }) \left[ \frac {1}{\mathbb {S} (x , y _ {\max })} - \frac {1}{\mathbb {S} (x , \overline {{\mathsf {h}}} (x)) - \mu} \right] \right. \\ \left. + p (x, \overline {{\mathsf {h}}} (x)) \bigg [ \frac {1}{\mathcal {S} (x , \overline {{\mathsf {h}}} (x))} - \frac {1}{\mathcal {S} (x , y _ {\max}) + \mu} \bigg ] \right\}. \\ \end{array}
$$
Differentiating with respect to  $\mu$  yields the optimal value
$$
\mu^ {*} = \frac {\sqrt {p (x , \overline {{\mathsf {h}}} (x))} \mathcal {S} (x , \overline {{\mathsf {h}}} (x)) - \sqrt {p (x , y _ {\max})} \mathcal {S} (x , y _ {\max})}{\sqrt {p (x , y _ {\max})} + \sqrt {p (x , \overline {{\mathsf {h}}} (x))}}.
$$
Plugging that value in the inequality gives:
$$
\Delta \mathcal {C} _ {\ell_ {\mathrm {e x p}}, \overline {{\mathcal {H}}}} (\overline {{h}}, x) \geq \frac {p (x , y _ {\mathrm {m a x}})}{\mathsf {S} (x , y _ {\mathrm {m a x}})} + \frac {p (x , \overline {{\mathsf {h}}} (x))}{\mathsf {S} (x , \overline {{\mathsf {h}}} (x))} - \frac {\left(\sqrt {p (x , y _ {\mathrm {m a x}})} + \sqrt {p (x , \overline {{\mathsf {h}}} (x))}\right) ^ {2}}{\mathsf {S} (x , y _ {\mathrm {m a x}}) + \mathsf {S} (x , \overline {{\mathsf {h}}} (x))}.
$$
Differentiating with respect to  $\mathcal{S}$  to show that the minimum is attained for  $\mathcal{S}(x,\overline{\mathsf{h}} (x)) = \mathcal{S}(x,y_{\max}) = \frac{1}{2}$ , which implies
$$
\begin{array}{l} \Delta \mathcal {C} _ {\ell_ {\exp}, \overline {{\mathcal {H}}}} (\bar {h}, x) \geq \left(\sqrt {p (x , y _ {\max })} - \sqrt {p (x , \bar {h} (x))}\right) ^ {2} \\ = \frac {\left(p (x , \overline {{h}} (x)) - p (x , y _ {\max})\right) ^ {2}}{\left(\sqrt {p (x , \overline {{h}} (x))} + \sqrt {p (x , y _ {\max})}\right) ^ {2}}. \\ \end{array}
$$
By the concavity of the square-root function, for all  $a,b\in [0,1]$ , we have  $\frac{1}{2}\big(\sqrt{a} +\sqrt{b}\big)\leq \sqrt{\frac{1}{2}(a + b)}$  thus we can write
$$
\begin{array}{l} \Delta \mathcal {C} _ {\ell_ {\exp}, \overline {{\mathcal {H}}}} (\bar {h}, x) \geq \frac {\left(p (x , \bar {h} (x)) - p (x , y _ {\max})\right) ^ {2}}{2 \left(p (x , \bar {h} (x)) + p (x , y _ {\max})\right)} \\ \geq \frac {\left(p (x , \overline {{\mathsf {h}}} (x)) - p (x , y _ {\max })\right) ^ {2}}{2} \quad \left(p (x, \overline {{\mathsf {h}}} (x)) + p (x, y _ {\max }) \leq 1\right) \\ = \frac {1}{2} \left(\Delta \mathcal {C} _ {\ell_ {0 - 1}, \overline {{\mathcal {H}}}} (\bar {h}, x)\right) ^ {2}. \quad \text {(b y L e m m a 8 a n d} \overline {{\mathsf {H}}} (x) = \mathcal {Y} \cup \{0 \}) \\ \end{array}
$$
$$
\frac {\left(\mathbb {E} _ {X} \left[ \Delta \mathcal {C} _ {\ell_ {0 - 1} , \overline {{\mathcal {H}}}} (\bar {h} , x) \right]\right) ^ {2}}{2} \leq \underset {X} {\mathbb {E}} \left[ \frac {\Delta \mathcal {C} _ {\ell_ {0 - 1} , \overline {{\mathcal {H}}}} (\bar {h} , x) ^ {2}}{2} \right] \leq \underset {X} {\mathbb {E}} \left[ \Delta \mathcal {C} _ {\ell_ {\exp}, \overline {{\mathcal {H}}}} (\bar {h}, x) \right],
$$
E.3 Generalized cross-entropy loss
Theorem 11 ( $\overline{\mathcal{H}}$ -consistency bound for generalized cross-entropy loss). Assume that  $\mathcal{H}$  is symmetric and complete. Then, for any  $\lambda \in \mathbb{R}$ , hypothesis  $\overline{h} \in \overline{\mathcal{H}}$  and any distribution,
$$
\mathcal {E} _ {\ell_ {0 - 1}} (\overline {{h}}) - \mathcal {E} _ {\ell_ {0 - 1}} ^ {*} (\overline {{\mathcal {H}}}) \leq \sqrt {2 (n + 1) ^ {\alpha}} \left(\mathcal {E} _ {\ell_ {\mathrm {g c e}}} (\overline {{h}}) - \mathcal {E} _ {\ell_ {\mathrm {g c e}}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell_ {\mathrm {g c e}}} (\overline {{\mathcal {H}}})\right) ^ {\frac {1}{2}} - \mathcal {M} _ {\ell_ {0 - 1}} (\overline {{\mathcal {H}}}).
$$
Proof. For the generalized cross-entropy loss  $\ell_{\mathrm{gce}}$ , the conditional  $\ell_{\mathrm{gce}}$ -risk can be expressed as follows:
$$
\mathcal {C} _ {\ell_ {\mathrm {g c e}}} (\bar {h}, x)) = \sum_ {y \in \mathcal {Y} \cup \{0 \}} p (x, y) \frac {1}{\alpha} \left[ 1 - \left[ \frac {e ^ {\bar {h} (x , y)}}{\sum_ {y ^ {\prime} \in \mathcal {Y} \cup 0} e ^ {\bar {h} (x , y ^ {\prime})}} \right] ^ {\alpha} \right] = \frac {1}{\alpha} \sum_ {y \in \mathcal {Y} \cup \{0 \}} p (x, y) (1 - \mathcal {S} (x, y) ^ {\alpha})
$$
where we let  $\mathcal{S}(x,y) = \frac{e^{\overline{h}(x,y)}}{\sum_{y' \in \mathcal{Y} \cup \{0\}} e^{\overline{h}(x,y')} \in [0,1]}$  for any  $y \in \mathcal{Y} \cup \{0\}$  with the constraint that  $\sum_{y \in \mathcal{Y} \cup \{0\}} \mathcal{S}(x,y) = 1$ . Let  $y_{\max} = \operatorname{argmax}_{y \in \mathcal{Y} \cup \{0\}} p(x,y)$ , where we choose the label with the same deterministic strategy for breaking ties as that of  $\overline{h}(x)$ . For any  $\overline{h} \in \mathcal{H}$  such that  $\overline{h}(x) \neq y_{\max}$  and  $x \in \mathcal{X}$ , by the symmetry and completeness of  $\mathcal{H}$ , we can always find a family of hypotheses  $\left\{\overline{h}_{\mu} : \mu \in [-\mathcal{S}(x,y_{\max}), \mathcal{S}(x,\overline{h}(x))]\right\} \subset \overline{\mathcal{H}}$  such that  $\mathcal{S}_{\mu}(x,\cdot) = \frac{e^{\overline{h}_{\mu}(x,\cdot)}}{\sum_{y' \in \mathcal{Y} \cup \{0\}} e^{\overline{h}_{\mu}(x,y')}}$  take the following values:
$$
\mathcal {S} _ {\mu} (x, y) = \left\{ \begin{array}{l l} \mathbb {S} (x, y) & \text {i f} y \notin \{\mathsf {y} _ {\max }, \overline {{\mathsf {h}}} (x) \} \\ \mathbb {S} (x, \mathsf {y} _ {\max }) + \mu & \text {i f} y = \overline {{\mathsf {h}}} (x) \\ \mathbb {S} (x, \overline {{\mathsf {h}}} (x)) - \mu & \text {i f} y = \mathsf {y} _ {\max }. \end{array} \right.
$$
Let  $\overline{h} \in \overline{\mathcal{H}}$  be a hypothesis such that  $\overline{h}(x) \neq y_{\max}$ . By the definition and using the fact that  $\overline{\mathsf{H}}(x) = \mathcal{Y} \cup \{0\}$  when  $\mathcal{H}$  is symmetric, we obtain
$$
\begin{array}{l} \Delta \mathcal {C} _ {\ell_ {\mathrm {g c e}}, \overline {{\mathcal {H}}}} (\bar {h}, x) \\ = \mathcal {C} _ {\ell_ {\mathrm {g c e}}} (\overline {{h}}, x) - \mathcal {C} _ {\ell_ {\mathrm {g c e}}} ^ {*} (\overline {{\mathcal {H}}}, x) \\ \geq \mathcal{C}_{\ell_{\text{gce}}}\big(\overline{h},x\big) - \inf_{\mu \in [-\mathcal{S}(x,y_{\max}),\mathcal{S}(x,\overline{\mathfrak{h}} (x))]} \mathcal{C}_{\ell_{\text{gce}}}\big(\overline{h}_{\mu},x\big) \\ = \frac {1}{\alpha} \sup  _ {\mu \in [ - \mathbb {S} (x, y _ {\max })  , \mathbb {S} (x, \overline {{\mathbf {h}}} (x)) ]} \left\{p (x, y _ {\max }) \left[ - \mathbb {S} (x, y _ {\max }) ^ {\alpha} + \left(\mathbb {S} (x, \overline {{\mathbf {h}}} (x)) - \mu\right) ^ {\alpha} \right] \right. \\ \left. + p (x, \overline {{\mathsf {h}}} (x)) \left[ - \mathcal {S} (x, \overline {{\mathsf {h}}} (x)) ^ {\alpha} + \left(\mathcal {S} (x, y _ {\max }) + \mu\right) ^ {\alpha} \right] \right\}. \\ \end{array}
$$
$$
\mu^ {*} = \frac {p (x , \overline {{\mathsf {h}}} (x)) ^ {\frac {1}{1 - \alpha}} \mathcal {S} (x , \overline {{\mathsf {h}}} (x)) - p (x , y _ {\max}) ^ {\frac {1}{1 - \alpha}} \mathcal {S} (x , y _ {\max})}{p (x , y _ {\max}) ^ {\frac {1}{1 - \alpha}} + p (x , \overline {{\mathsf {h}}} (x)) ^ {\frac {1}{1 - \alpha}}}.
$$
$$
\begin{array}{l} \Delta \mathcal {C} _ {\ell_ {\mathrm {g c e}}, \overline {{\mathcal {H}}}} (\bar {h}, x) \geq \frac {1}{\alpha} \left(\mathcal {S} (x, \bar {h} (x)) + \mathcal {S} (x, y _ {\max })\right) ^ {\alpha} \left(p (x, y _ {\max }) ^ {\frac {1}{1 - \alpha}} + p (x, \bar {h} (x)) ^ {\frac {1}{1 - \alpha}}\right) ^ {1 - \alpha} \\ - \frac {1}{\alpha} p (x, y _ {\max }) \mathcal {S} (x, y _ {\max }) ^ {\alpha} - \frac {1}{\alpha} p (x, \overline {{h}} (x)) \mathcal {S} (x, \overline {{h}} (x)) ^ {\alpha}. \\ \end{array}
$$
Differentiating with respect to  $\mathcal{S}$  to show that the minimum is attained for  $\mathcal{S}(x,\overline{\mathsf{h}} (x)) = \mathcal{S}(x,y_{\max}) = \frac{1}{n + 1}$ , which implies
$$
\Delta \mathcal {C} _ {\ell_ {\mathrm {g c e}}, \overline {{\mathcal {H}}}} (\overline {{h}}, x) \geq \frac {1}{\alpha (n + 1) ^ {\alpha}} \Big [ 2 ^ {\alpha} \Big (p (x, y _ {\max}) ^ {\frac {1}{1 - \alpha}} + p (x, \overline {{\mathsf {h}}} (x)) ^ {\frac {1}{1 - \alpha}} \Big) ^ {1 - \alpha} - p (x, y _ {\max}) - p (x, \overline {{\mathsf {h}}} (x)) \Big ].
$$
By using the fact that for all  $a, b \in [0,1]$ ,  $0 \leq a + b \leq 1$ , we have  $\left(\frac{a^{\frac{1}{1 - \alpha}} + b^{\frac{1}{1 - \alpha}}}{2}\right)^{1 - \alpha} - \frac{a + b}{2} \geq \frac{\alpha}{4} (a - b)^2$ , thus we can write
$$
\begin{array}{l} \Delta \mathcal {C} _ {\ell_ {\mathrm {g c e}}, \overline {{\mathcal {H}}}} (\bar {h}, x) \geq \frac {\left(p (x , \bar {\mathsf {h}} (x)) - p (x , y _ {\max })\right) ^ {2}}{2 (n + 1) ^ {\alpha}} \\ = \frac {1}{2 (n + 1) ^ {\alpha}} \left(\Delta \mathcal {C} _ {\ell_ {0 - 1}, \overline {{\mathcal {H}}}} (\bar {h}, x)\right) ^ {2}. \quad \text {(b y L e m m a 8 a n d} \overline {{\mathsf {H}}} (x) = \mathcal {Y} \cup \{0 \}) \\ \end{array}
$$
Since the function  $\frac{t^2}{2(n + 1)^\alpha}$  is convex, by Jensen's inequality, we obtain for any hypothesis  $\overline{h} \in \overline{\mathcal{H}}$  and any distribution,
$$
\frac {\left(\mathbb {E} _ {X} \Big [ \Delta \mathcal {C} _ {\ell_ {0 - 1} , \overline {{\mathcal {H}}}} (\overline {{h}} , x) \Big ]\right) ^ {2}}{2 (n + 1) ^ {\alpha}} \leq \underset {X} {\mathbb {E}} \left[ \frac {\Delta \mathcal {C} _ {\ell_ {0 - 1} , \overline {{\mathcal {H}}}} (\overline {{h}} , x) ^ {2}}{2 (n + 1) ^ {\alpha}} \right] \leq \underset {X} {\mathbb {E}} \Big [ \Delta \mathcal {C} _ {\ell_ {\mathrm {g c e}}, \overline {{\mathcal {H}}}} (\overline {{h}}, x) \Big ]
$$
E.4 Mean absolute error loss
Theorem 12 ( $\overline{\mathcal{H}}$ -consistency bound for mean absolute error loss). Assume that  $\mathcal{H}$  is symmetric and complete. Then, for any  $\lambda \in \mathbb{R}$ , hypothesis  $\overline{h} \in \overline{\mathcal{H}}$  and any distribution,
$$
\mathcal {E} _ {\ell_ {0 - 1}} \left(\overline {{h}}\right) - \mathcal {E} _ {\ell_ {0 - 1}} ^ {*} \left(\overline {{\mathcal {H}}}\right) \leq \left(n + 1\right) \left(\mathcal {E} _ {\ell_ {\mathrm {m a e}}} \left(\overline {{h}}\right) - \mathcal {E} _ {\ell_ {\mathrm {m a e}}} ^ {*} \left(\overline {{\mathcal {H}}}\right) + \mathcal {M} _ {\ell_ {\mathrm {m a e}}} \left(\overline {{\mathcal {H}}}\right)\right) - \mathcal {M} _ {\ell_ {0 - 1}} \left(\overline {{\mathcal {H}}}\right).
$$
Proof. For the mean absolute error loss  $\ell_{\mathrm{mae}}$ , the conditional  $\ell_{\mathrm{mae}}$ -risk can be expressed as follows:
$$
\mathcal {C} _ {\ell_ {\text {m a e}}} (\overline {{h}}, x)) = \sum_ {y \in \mathcal {Y} \cup \{0 \}} p (x, y) \left(1 - \frac {e ^ {\overline {{h}} (x , y)}}{\sum_ {y ^ {\prime} \in \mathcal {Y} \cup 0} e ^ {\overline {{h}} (x , y ^ {\prime})}}\right) = \sum_ {y \in \mathcal {Y} \cup \{0 \}} p (x, y) (1 - \mathcal {S} (x, y))
$$
where we let  $\mathcal{S}(x,y) = \frac{e^{\overline{h}(x,y)}}{\sum_{y' \in \mathcal{Y} \cup \{0\}} e^{\overline{h}(x,y')}} \in [0,1]$  for any  $y \in \mathcal{Y} \cup \{0\}$  with the constraint that  $\sum_{y \in \mathcal{Y} \cup \{0\}} \mathcal{S}(x,y) = 1$ . Let  $y_{\max} = \operatorname{argmax}_{y \in \mathcal{Y} \cup \{0\}} p(x,y)$ , where we choose the label with the same deterministic strategy for breaking ties as that of  $\overline{h}(x)$ . For any  $\overline{h} \in \mathcal{H}$  such that  $\overline{h}(x) \neq y_{\max}$  and  $x \in \mathcal{X}$ , by the symmetry and completeness of  $\mathcal{H}$ , we can always find a family of hypotheses  $\left\{\overline{h}_{\mu} : \mu \in [-\mathcal{S}(x,y_{\max}), \mathcal{S}(x,\overline{h}(x))]\right\} \subset \overline{\mathcal{H}}$  such that  $\mathcal{S}_{\mu}(x,\cdot) = \frac{e^{\overline{h}_{\mu}(x,\cdot)}}{\sum_{y' \in \mathcal{Y} \cup \{0\}} e^{\overline{h}_{\mu}(x,y')} }$  take the following values:
$$
\mathcal {S} _ {\mu} (x, y) = \left\{ \begin{array}{l l} \mathcal {S} (x, y) & \text {i f} y \notin \big \{y _ {\max }, \overline {{\mathsf {h}}} (x) \big \} \\ \mathcal {S} (x, y _ {\max }) + \mu & \text {i f} y = \overline {{\mathsf {h}}} (x) \\ \mathcal {S} (x, \overline {{\mathsf {h}}} (x)) - \mu & \text {i f} y = y _ {\max }. \end{array} \right.
$$
$$
\begin{array}{l} \Delta \mathscr {C} _ {\ell_ {\mathrm {m a e}}, \overline {{\mathscr {H}}}} (\overline {{h}}, x) \\ = \mathcal {C} _ {\ell_ {\text {m a e}}} (\bar {h}, x) - \mathcal {C} _ {\ell_ {\text {m a e}}} ^ {*} (\overline {{\mathcal {H}}}, x) \\ \geq \mathcal{C}_{\ell_{\mathrm{mae}}}\big(\overline{h},x\big) - \inf_{\mu \in [-\mathcal{S}(x,y_{\mathrm{max}}),\mathcal{S}(x,\overline{\mathfrak{h}} (x))]} \mathcal{C}_{\ell_{\mathrm{mae}}}\big(\overline{h}_{\mu},x\big) \\ = \sup  _ {\mu \in [ - \mathbb {S} (x, y _ {\max })  , \mathbb {S} (x, \overline {{\mathrm {h}}} (x)) ]} \left\{p (x, y _ {\max }) \left[ - \mathbb {S} (x, y _ {\max }) + \mathbb {S} (x, \overline {{\mathrm {h}}} (x)) - \mu \right] \right. \\ \left. \right.\left. + p (x, \overline {{\mathsf {h}}} (x)) \big [ - \mathcal {S} (x, \overline {{\mathsf {h}}} (x)) + \mathcal {S} (x, y _ {\max}) + \mu \big ] \right\}. \\ \end{array}
$$
Differentiating with respect to  $\mu$  yields the optimum value  $\mu^{*} = -\mathcal{S}(x,y_{\mathrm{max}})$ . Plugging that value in the inequality gives:
$$
\Delta \mathcal {C} _ {\ell_ {\mathrm {m a e}}, \overline {{\mathcal {H}}}} (\overline {{h}}, x) \geq p (x, y _ {\max }) \mathcal {S} (x, \overline {{\mathsf {h}}} (x)) - p (x, \overline {{\mathsf {h}}} (x)) \mathcal {S} (x, \overline {{\mathsf {h}}} (x)).
$$
Differentiating with respect to  $\mathcal{S}$  to show that the minimum is attained for  $\mathcal{S}(x, \overline{\mathsf{h}}(x)) = \frac{1}{n+1}$ , which implies
$$
\begin{array}{l} \Delta \mathscr {C} _ {\ell_ {\mathrm {m a e}}, \overline {{\mathscr {H}}}} (\bar {h}, x) \geq \frac {1}{n + 1} \left(p (x, y _ {\max }) - p (x, \bar {h} (x))\right) \\ = \frac {1}{n + 1} \left(\Delta \mathcal {C} _ {\ell_ {0 - 1}, \bar {\mathcal {H}}} (\bar {h}, x)\right). \quad \text {(b y L e m m a 8 a n d} \bar {\mathsf {H}} (x) = \mathcal {Y} \cup \{0 \}) \\ \end{array}
$$
Therefore, we obtain for any hypothesis  $\overline{h} \in \overline{\mathcal{H}}$  and any distribution,
$$
\frac {\mathbb {E} _ {X} \left[ \Delta \mathcal {C} _ {\ell_ {0 - 1} , \overline {{\mathcal {H}}}} (\bar {h} , x) \right]}{n + 1} \leq \frac {\mathbb {E}}{X} \left[ \Delta \mathcal {C} _ {\ell_ {\mathrm {m a e}}, \overline {{\mathcal {H}}}} (\bar {h}, x) \right],
$$
$$
\mathcal {E} _ {\ell_ {0 - 1}} (\overline {{h}}) - \mathcal {E} _ {\ell_ {0 - 1}} ^ {*} (\overline {{\mathcal {H}}}) \leq (n + 1) \left(\mathcal {E} _ {\ell_ {\text {m a e}}} (\overline {{h}}) - \mathcal {E} _ {\ell_ {\text {m a e}}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell_ {\text {m a e}}} (\overline {{\mathcal {H}}})\right) - \mathcal {M} _ {\ell_ {0 - 1}} (\overline {{\mathcal {H}}}).
$$
Proof of realizable consistency for score-based two-stage surrogate losses (Theorem 5)
Proof. First, by definition, it is straightforward to see that for any  $h, x, y$ ,  $\mathsf{L}_{h_p}(h_d, x, y)$  upper bounds the deferral loss  $\mathsf{L}_{\mathrm{def}}$ . Consider a data distribution and costs under which there exists  $h^* \in \mathcal{H}$  such that  $\mathcal{E}_{\mathsf{L}_{\mathrm{def}}}(h^*) = 0$ .
Let  $\hat{h}_p$  be the minimizer of  $\mathcal{E}_{\ell_1}$  and  $\hat{h}_d$  the minimizer of  $\mathcal{E}_{\mathsf{L}_{\hat{h}_p}}$ . Then, using the fact that  $\mathsf{L}_h$  upper bounds the deferral loss  $\mathsf{L}_{\mathrm{def}}$ , we have  $\mathcal{E}_{\mathsf{L}_{\mathrm{def}}}\big(\hat{h}\big)\leq \mathcal{E}_{\mathsf{L}_{\hat{h}_p}}\big(\hat{h}_d\big)$ .
Next we analyze two cases. If for a point  $x$ , deferral occurs, that is there exists  $j^* \in [n_e]$ , such that  $\mathsf{h}^*(x) = n + j^*$ , then we must have  $c_{j^*} = 0$  for all  $x$  since the data is realizable and  $c_{j^*}$  is constant. Therefore, there exists an optimal  $h^{**}$  deferring all the points to the  $j^*$ th expert. Then, by the assumption that  $\mathcal{H}$  is closed under scaling and the Lebesgue dominated convergence theorem, for  $\ell_2$  being the logistic loss,  $\mathcal{E}_{\mathsf{L}_{\mathrm{def}}}(\hat{h}) \leq \mathcal{E}_{\mathsf{L}_{\hat{h}_p}}(\hat{h}_d) \leq \lim_{\tau \to +\infty} \mathcal{E}_{\mathsf{L}_{h_p^{**}}}(\tau h_d^{**}) = 0$ , where we used the fact that in the limit of  $\tau \to +\infty$  the logistic loss term  $\ell_2(\overline{h}_d^{**}, x, j)$  corresponding to  $j \neq j^*$  is zero.
On the other hand, if no deferral occurs for any point, that is  $\mathfrak{h}^*(x) \in [n]$  for any  $x$ , then we must have  $\mathbb{1}_{\mathfrak{h}_p^*(x) \neq y} = 0$  for all  $(x, y)$  since the data is realizable. Using the fact that  $\mathcal{H}$  is closed under scaling and that the logistic loss is realizable  $\mathcal{H}$ -consistent in the standard classification, we obtain  $\mathbb{1}_{\hat{\mathfrak{h}}_p(x) \neq y} = 0$  for all  $(x, y)$ . Then, by the assumption that  $\mathcal{H}$  is closed under scaling and the Lebesgue dominated convergence theorem, for  $\ell_2$  being the logistic loss,  $\mathcal{E}_{\mathsf{L_{def}}}(\hat{h}) \leq \mathcal{E}_{\mathsf{L}_{\hat{h}_p}}(\hat{h}_d) \leq \lim_{\tau \to +\infty} \mathcal{E}_{\mathsf{L}_{h_p^*}}(\tau h_d^*) = 0$ , where we used the fact that in the limit of  $\tau \to +\infty$  the logistic loss term  $\ell_2(\overline{h}_d^*, x, j)$  corresponding to  $j \neq 0$  is zero.
Therefore, the optimal solution from minimizing score-based two-stage surrogates leads to a zero error solution of the deferral loss, which proves that the score-based two-stage surrogate loss is realizable consistent.
G Proof of  $(\mathcal{H},\mathcal{R})$  -consistency bounds for predictor-rejector two-stage surrogate losses (Theorem 6)
Theorem 6 (( $\mathcal{H}, \mathcal{R}$ )-consistency bounds for predictor-rejector two-stage surrogates). Assume that  $\ell_1$  admits an  $\mathcal{H}$ -consistency bound and  $\ell_2$  admits an  $\overline{\mathcal{R}}$ -consistency bound with respect to the multi-class zero-one classification loss  $\ell_{0-1}$  respectively. Thus, there are non-decreasing concave functions  $\Gamma_1$  and  $\Gamma_2$  such that, for all  $h \in \mathcal{H}$  and  $\overline{r} \in \overline{\mathcal{R}}$ , we have
$$
\mathcal {E} _ {\ell_ {0 - 1}} (h) - \mathcal {E} _ {\ell_ {0 - 1}} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\ell_ {0 - 1}} (\mathcal {H}) \leq \Gamma_ {1} \left(\mathcal {E} _ {\ell_ {1}} (h) - \mathcal {E} _ {\ell_ {1}} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\ell_ {1}} (\mathcal {H}) \right.
$$
$$
\mathcal {E} _ {\ell_ {0 - 1}} (\overline {{r}}) - \mathcal {E} _ {\ell_ {0 - 1}} ^ {*} (\overline {{\mathcal {R}}}) + \mathcal {M} _ {\ell_ {0 - 1}} (\overline {{\mathcal {R}}}) \leq \Gamma_ {2} \big (\mathcal {E} _ {\ell_ {2}} (\overline {{r}}) - \mathcal {E} _ {\ell_ {2}} ^ {*} (\overline {{\mathcal {R}}}) + \mathcal {M} _ {\ell_ {2}} (\overline {{\mathcal {R}}}) \big).
$$
$$
\begin{array}{l} \mathcal {E} _ {\mathrm {L} _ {\mathrm {d e f}}} (h, r) - \mathcal {E} _ {\mathrm {L} _ {\mathrm {d e f}}} ^ {*} (\mathcal {H}, \mathcal {R}) + \mathcal {M} _ {\mathrm {L} _ {\mathrm {d e f}}} (\mathcal {H}, \mathcal {R}) \\ \leq \Gamma_ {1} \Big (\mathcal {E} _ {\ell_ {1}} (h) - \mathcal {E} _ {\ell_ {1}} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\ell_ {1}} (\mathcal {H}) \Big) + \left(1 + \sum_ {j = 1} ^ {n _ {e}} \overline {{c}} _ {j}\right) \Gamma_ {2} \left(\frac {\mathcal {E} _ {\mathrm {L} _ {h}} (r) - \mathcal {E} _ {\mathrm {L} _ {h}} ^ {*} (\mathcal {R}) + \mathcal {M} _ {\mathrm {L} _ {h}} (\mathcal {R})}{\sum_ {j = 1} ^ {n _ {e}} \underline {{c}} _ {j}}\right), \\ \end{array}
$$
$$
\mathsf {L} _ {\mathrm {d e f}} \big (h, r, x, y \big) = \mathbb {1} _ {\mathsf {h} (x) \neq y} \mathbb {1} _ {\mathsf {r} (x) = 0} + \sum_ {j = 1} ^ {n _ {e}} c _ {j} \big (x, y \big) \mathbb {1} _ {\mathsf {r} (x) = j}.
$$
Let  $\bar{c}_0(x,y) = \mathbb{1}_{\mathfrak{h}(x) = y}$ . We can rewrite  $\mathcal{E}_{\mathsf{L}_{\mathrm{def}}}(h,r) - \mathcal{E}_{\mathsf{L}_{\mathrm{def}}}^{*}(\mathcal{H},\mathcal{R}) + \mathcal{M}_{\mathsf{L}_{\mathrm{def}}}(\mathcal{H},\mathcal{R})$  as
$$
\begin{array}{l} \mathcal {E} _ {\mathrm {L} _ {\text {d e f}}} (h, r) - \mathcal {E} _ {\mathrm {L} _ {\text {d e f}}} ^ {*} (\mathcal {H}, \mathcal {R}) + \mathcal {M} _ {\mathrm {L} _ {\text {d e f}}} (\mathcal {H}, \mathcal {R}) \\ = \mathbb {E} _ {X} \left[ \mathcal {C} _ {\mathrm {L} _ {\mathrm {d e f}}} (h, r, x) - \mathcal {C} _ {\mathrm {L} _ {\mathrm {d e f}}} ^ {*} (\mathcal {H}, \mathcal {R}, x) \right] \\ = \mathbb {E} _ {X} \left[ \mathcal {C} _ {\mathrm {L} _ {\mathrm {d e f}}} (h, r, x) - \inf  _ {r \in \mathcal {R}} \mathcal {C} _ {\mathrm {L} _ {\mathrm {d e f}}} (h, r, x) + \inf  _ {r \in \mathcal {R}} \mathcal {C} _ {\mathrm {L} _ {\mathrm {d e f}}} (h, r, x) - \mathcal {C} _ {\mathrm {L} _ {\mathrm {d e f}}} ^ {*} (\mathcal {H}, \mathcal {R}, x) \right] \tag {7} \\ = \mathbb {E} _ {X} \left[ \mathcal {C} _ {\mathrm {L} _ {\text {d e f}}} (h, r, x) - \inf  _ {r \in \mathcal {R}} \mathcal {C} _ {\mathrm {L} _ {\text {d e f}}} (h, r, x) \right] + \mathbb {E} _ {X} \left[ \inf  _ {r \in \mathcal {R}} \mathcal {C} _ {\mathrm {L} _ {\text {d e f}}} (h, r, x) - \mathcal {C} _ {\mathrm {L} _ {\text {d e f}}} ^ {*} (\mathcal {H}, \mathcal {R}, x) \right] \\ \end{array}
$$
Let  $\overline{p}(x, j) = \frac{\mathbb{E}_y[\bar{c}_j(x, y)]}{\sum_{j=0}^{n_e} \mathbb{E}_y[\bar{c}_j(x, y)]}$  for any  $j \in \{0, \dots, n_e\}$ . Note that  $\overline{p}(x, \cdot)$  is the probability vector on the label space  $\{0, \dots, n_e\}$ . For any  $r \in \mathcal{R}$ , we define  $\overline{r}$  as its augmented hypothesis:  $\overline{r}(x, 0) = 0$ ,  $\overline{r}(x, 1) = -r_1(x), \dots, \overline{r}(x, n_e) = -r_{n_e}(x)$ . By the assumptions, we have
(By  $\overline{\mathcal{R}}$ -consistency bounds of  $\ell_2$  under assumption)