SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac {\mathcal {R} _ {\mathrm {L}} (h) - \mathcal {R} _ {\mathrm {L} , \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathrm {L} , \mathcal {H}}}{n} = \frac {\mathbb {E} _ {X} [ \Delta \mathcal {C} _ {\mathrm {L} , \mathcal {H}} (h , x) ]}{n} \\ = \underset {X} {\mathbb {E}} \left[ \Delta \mathcal {C} _ {\mathrm {L} ^ {\text {c o m p}}, \mathcal {H}} (h, x) \right] \\ = \mathcal {R} _ {\mathrm {L} ^ {\operatorname {c o m p}}} (h) - \mathcal {R} _ {\mathrm {L} ^ {\operatorname {c o m p}}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathrm {L} ^ {\operatorname {c o m p}}, \mathcal {H}}, \\ \end{array}
$$
$$
\mathcal {R} _ {\mathsf {L}} (h) - \mathcal {R} _ {\mathsf {L}, \mathcal {H}} ^ {*} \leq n \Big (\mathcal {R} _ {\mathsf {L} _ {\text {m a e}} ^ {\text {c o m p}}} (h) - \mathcal {R} _ {\mathsf {L} _ {\text {m a e}} ^ {\text {c o m p}}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathsf {L} _ {\text {m a e}} ^ {\text {c o m p}}, \mathcal {H}} \Big) - \mathcal {M} _ {\mathsf {L}, \mathcal {H}}.
$$
F Proofs for structured constrained losses
F.1 Structured constrained exponential loss
Theorem 15 ( $\mathcal{H}$ -consistency bound of  $\mathsf{L}_{\mathrm{exp}}^{\mathrm{cstnd}}$ ). Assume that  $\mathcal{H}$  is symmetric and complete. Then, for any target loss  $\ell$ , hypothesis  $h \in \mathcal{H}$  and any distribution,
$$
\mathcal {R} _ {\mathrm {L}} (h) - \mathcal {R} _ {\mathrm {L}, \mathcal {H}} ^ {*} \leq 2 \sqrt {\ell_ {\max }} \left(\mathcal {R} _ {\mathrm {L} _ {\exp} ^ {\text {c s t n d}}} (h) - \mathcal {R} _ {\mathrm {L} _ {\exp} ^ {\text {c s t n d}}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathrm {L} _ {\exp} ^ {\text {c s t n d}}, \mathcal {H}}\right) ^ {\frac {1}{2}} - \mathcal {M} _ {\mathrm {L}, \mathcal {H}}. \tag {27}
$$
Proof. Denote by  $q(x, y') = \sum_{y \in \mathcal{Y}} p(x, y) \ell(y', y) \in [0, \ell_{\max}]$ . For the constrained structured loss  $L_{\exp}^{\text{cstnd}}$ , the conditional  $L_{\exp}^{\text{cstnd}}$ -risk can be expressed as follows:
$$
\mathcal{C}_{\text{\rm{L}^{cstnd}}}(h,x) = \sum_{y\in \mathcal{Y}}p(x,y)\sum_{y^{\prime}\in \mathcal{Y}}\ell (y,y^{\prime})e^{h(x,y^{\prime})} = \sum_{y^{\prime}\in \mathcal{Y}}e^{h(x,y^{\prime})}q(x,y^{\prime}).
$$
Let  $y_{\min} = \operatorname{argmin}_{y' \in \mathcal{Y}} \sum_{y \in \mathcal{Y}} p(x, y) \ell(y', y)$ , where we choose the label with the highest index under the natural ordering of labels as the tie-breaking strategy. For any  $h \in \mathcal{H}$  such that  $h(x) \neq y_{\min}$  and  $x \in \mathcal{X}$ , by the symmetry and completeness of  $\mathcal{H}$ , we can always find a family of hypotheses  $\{\overline{h}_\mu : \mu \in \mathbb{R}\} \subset \mathcal{H}$  such that  $h_\mu(x, \cdot)$  take the following values:
$$
\bar {h} _ {\mu} (x, y) = \left\{ \begin{array}{l l} h (x, y) & \text {i f} y \notin \left\{y _ {\min }, h (x) \right\} \\ h \left(x, y _ {\min }\right) + \mu & \text {i f} y = h (x) \\ h (x, h (x)) - \mu & \text {i f} y = y _ {\min } \end{array} \right. \tag {28}
$$
Note that the hypotheses  $\overline{h}_{\mu}$  satisfies the constraint:
$$
\sum_ {y \in \mathcal {Y}} \bar {h} _ {\mu} (x, y) = \sum_ {y \in \mathcal {Y}} h (x, y) = 0, \forall \mu \in \mathbb {R}.
$$
Since  $\sum_{y\in \mathcal{Y}}h(x,y) = 0$ , there must be non-negative scores. By definition of  $\mathsf{h}(x)$  as a maximizer, we must thus have  $h(x,\mathsf{h}(x))\geq 0$ . By (28) and using the fact that  $\mathsf{H}(x) = \mathcal{Y}$  when  $\mathcal{H}$  is symmetric, we obtain
$$
\begin{array}{l} \Delta \mathcal {C} _ {\text {L c s t n d}, \mathcal {H}} (h, x) \\ = \mathcal {C} _ {\mathrm {L c s t n d}} (h, x) - \mathcal {C} _ {\mathrm {L c s t n d}} ^ {*} (\mathcal {H}, x) \\ \geq \mathcal{C}_{L^{\text{cstnd}}}(h,x) - \inf_{\mu \in \mathbb{R}}\mathcal{C}_{L^{\text{cstnd}}}\left(\overline{h}_{\mu},x\right) \\ = \sup  _ {\mu \in \mathbb {R}} \left\{q (x, y _ {\min }) \left(e ^ {h (x, y _ {\min })} - e ^ {h (x, \mathsf {h} (x)) - \mu}\right) + q (x, \mathsf {h} (x)) \left(e ^ {h (x, \mathsf {h} (x))} - e ^ {h (x, y _ {\min }) + \mu}\right) \right\} \\ = \left(\sqrt {q (x , h (x)) e ^ {h (x , h (x))}} - \sqrt {q (x , y _ {\min }) e ^ {h (x , y _ {\min })}}\right) ^ {2} \\ \end{array}
$$
(differentiating with respect to  $\mu$  to optimize, optimum  $\mu^{*} = \frac{1}{2}\log \frac{q(x,y_{\min})e^{h(x,h(x))}}{q(x,h(x))e^{h(x,y_{\min})}}$
$$
\begin{array}{l} \geq e ^ {h (x, \mathsf {h} (x))} \left(\sqrt {q (x , y _ {\min })} - \sqrt {q (x , \mathsf {h} (x))}\right) ^ {2} \\ (e ^ {h (x, \mathsf {h} (x))} \geq e ^ {h (x, y _ {\min })} \text {a n d} q (x, \mathsf {h} (x)) \geq q (x, y _ {\min }))) \\ \geq \left(\sqrt {q (x , y _ {\min })} - \sqrt {q (x , h (x))}\right) ^ {2} \quad (h (x, h (x)) \geq 0) \\ = \left(\frac {q (x , h (x)) - q (x , y _ {\min })}{\sqrt {q (x , y _ {\min })} + \sqrt {q (x , h (x))}}\right) ^ {2} \\ \geq \frac {1}{4 \ell_ {\max }} \left(q (x, \mathsf {h} (x)) - q (x, y _ {\min })\right) ^ {2} \quad (0 \leq q (x, y) \leq \ell_ {\max }) \\ = \frac {1}{4 \ell_ {\max }} \Delta \mathcal {C} _ {\mathrm {L}, \mathcal {H}} (h, x) ^ {2}. \quad (\text {b y}) \\ \end{array}
$$
Since the function  $t \mapsto \frac{t^2}{4\ell_{\max}}$  is convex, by Jensen's inequality, we obtain for any hypothesis  $h \in \mathcal{H}$  and any distribution,
$$
\begin{array}{l} \frac {\left(\mathcal {R} _ {\mathrm {L}} (h) - \mathcal {R} _ {\mathrm {L} , \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathrm {L} , \mathcal {H}}\right) ^ {2}}{4 \ell_ {\max }} = \frac {\left(\mathbb {E} _ {X} \left[ \Delta \mathbb {C} _ {\mathrm {L} , \mathcal {H}} (h , x) \right]\right) ^ {2}}{4 \ell_ {\max }} \\ \leq \mathbb {E} _ {X} \left[ \frac {\Delta \mathcal {C} _ {\mathrm {L} , \mathcal {H}} (h , x) ^ {2}}{4 \ell_ {\max}} \right] \\ \leq \mathbb {E} _ {X} \left[ \Delta \mathcal {C} _ {\mathrm {L} ^ {\text {c s t n d}}, \mathcal {H}} (h, x) \right] \\ = \mathcal {R} _ {\mathrm {L} ^ {\text {c s t n d}}} (h) - \mathcal {R} _ {\mathrm {L} ^ {\text {c s t n d}}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathrm {L} ^ {\text {c s t n d}}, \mathcal {H}}, \\ \end{array}
$$
$$
\mathcal {R} _ {\mathrm {L}} (h) - \mathcal {R} _ {\mathrm {L}, \mathcal {H}} ^ {*} \leq 2 \sqrt {\ell_ {\max}} \Big (\mathcal {R} _ {\mathrm {L} _ {\exp} ^ {\mathrm {c s t n d}}} (h) - \mathcal {R} _ {\mathrm {L} _ {\exp} ^ {\mathrm {c s t n d}}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathrm {L} _ {\exp} ^ {\mathrm {c s t n d}}, \mathcal {H}} \Big) ^ {\frac {1}{2}} - \mathcal {M} _ {\mathrm {L}, \mathcal {H}}.
$$
F.2 Structured constrained squared-hinge loss
Theorem 16 ( $\mathcal{H}$ -consistency bound of  $\mathsf{L}_{\mathrm{sq - hinge}}^{\mathrm{cstd}}$ ). Assume that  $\mathcal{H}$  is symmetric and complete. Then, for any target loss  $\ell$ , hypothesis  $h \in \mathcal{H}$  and any distribution,
$$
\mathcal {R} _ {\mathrm {L}} (h) - \mathcal {R} _ {\mathrm {L}, \mathcal {H}} ^ {*} \leq \left(\mathcal {R} _ {\mathrm {L} _ {\mathrm {s q} - \text {h i n g e}} ^ {\mathrm {c s t n d}}} (h) - \mathcal {R} _ {\mathrm {L} _ {\mathrm {s q} - \text {h i n g e}} ^ {\mathrm {c s t n d}}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathrm {L} _ {\mathrm {s q} - \text {h i n g e}} ^ {\mathrm {c s t n d}}, \mathcal {H}}\right) ^ {\frac {1}{2}} - \mathcal {M} _ {\mathrm {L}, \mathcal {H}}. \tag {29}
$$
Proof. Denote by  $q(x, y') = \sum_{y \in \mathcal{Y}} p(x, y) \ell(y', y) \in [0, \ell_{\max}]$ . For the constrained structured loss  $L_{\mathrm{sq - hinge}}^{\mathrm{cstnd}}$ , the conditional  $L_{\mathrm{sq - hinge}}^{\mathrm{cstnd}}$ -risk can be expressed as follows:
$$
\begin{array}{l} \mathcal{C}_{\substack{\text{L}^{\text{cstnd}}\\ \text{sq - hing e}}}  (h,x) = \sum_{y\in \mathcal{Y}}p(x,y)\sum_{y^{\prime}\in \mathcal{Y}}\ell (y^{\prime},y)\max \bigl\{0,1 + h(x,y^{\prime})\bigr \}^{2} \\ = \sum_ {y ^ {\prime} \in \mathcal {Y}} \max  \{0, 1 + h (x, y ^ {\prime}) \} ^ {2} q (x, y ^ {\prime}). \\ \end{array}
$$
$$
\bar {h} _ {\mu} (x, y) = \left\{ \begin{array}{l l} h (x, y) & \text {i f} y \notin \left\{y _ {\min }, \mathrm {h} (x) \right\} \\ h \left(x, \mathrm {y} _ {\min }\right) + \mu & \text {i f} y = \mathrm {h} (x) \\ h (x, \mathrm {h} (x)) - \mu & \text {i f} y = \mathrm {y} _ {\min } \end{array} \right. \tag {30}
$$
$$
\sum_ {y \in \mathcal {Y}} \overline {{h}} _ {\mu} (x, y) = \sum_ {y \in \mathcal {Y}} h (x, y) = 0, \forall \mu \in \mathbb {R}.
$$
Since  $\sum_{y\in \mathcal{Y}}h(x,y) = 0$ , there must be non-negative scores. By definition of  $\mathsf{h}(x)$  as a maximizer, we must thus have  $h(x,\mathsf{h}(x))\geq 0$ . By (30) and using the fact that  $\mathsf{H}(x) = \mathcal{Y}$  when  $\mathcal{H}$  is symmetric, we obtain
$$
\begin{array}{l} \Delta \mathcal {C} _ {\mathrm {L} ^ {\text {c s t n d}}, \mathcal {H}} (h, x) \\ = \mathcal {C} _ {\mathrm {L} ^ {\text {c s t n d}}} (h, x) - \mathcal {C} _ {\mathrm {L} ^ {\text {c s t n d}}} ^ {*} (\mathcal {H}, x) \\ \geq \mathcal{C}_{\mathrm{L}^{\mathrm{cstnd}}}(h,x) - \inf_{\mu \in \mathbb{R}}\mathcal{C}_{\mathrm{L}^{\mathrm{cstnd}}}(\overline{h}_{\mu},x) \\ = \sup  _ {\mu \in \mathbb {R}} \left\{q (x, y _ {\min }) \left(\max  \left\{0, 1 + h (x, y _ {\min }) \right\} ^ {2} - \max  \left\{0, 1 + h (x, \mathrm {h} (x)) - \mu \right\} ^ {2}\right) \right. \\ \left. + q (x, \mathfrak {h} (x)) \left(\max  \{0, 1 + h (x, \mathfrak {h} (x)) \} ^ {2} - \max  \{0, 1 + h (x, y _ {\min }) + \mu \} ^ {2}\right) \right\} \\ \geq \left(1 + h (x, \mathrm {h} (x))\right) ^ {2} \left(q (x, y _ {\min }) - q (x, \mathrm {h} (x))\right) ^ {2} \quad (\text {d i f f e r e n t i a t i n g w i t h r e s p e c t t o} \mu \text {t o o p t i m i z e}) \\ \geq \left(q (x, h (x)) - q (x, y _ {\min })\right) ^ {2} \quad (h (x, h (x)) \geq 0) \\ = \left(\sum_ {y \in \mathcal {Y}} p (x, y) \ell (\mathsf {h} (x), y) - \sum_ {y \in \mathcal {Y}} p (x, y) \ell (y _ {\min }, y)\right) ^ {2} \\ = \Delta \mathcal {C} _ {\mathrm {L}, \mathcal {H}} (h, x) ^ {2}. \quad (\text {b y} \\ \end{array}
$$
Since the function  $t \mapsto t^2$  is convex, by Jensen's inequality, we obtain for any hypothesis  $h \in \mathcal{H}$  and any distribution,
$$
\begin{array}{l} \left(\mathcal {R} _ {\mathrm {L}} (h) - \mathcal {R} _ {\mathrm {L}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathrm {L}, \mathcal {H}}\right) ^ {2} = \left(\underset {X} {\mathbb {E}} \left[ \Delta \mathcal {C} _ {\mathrm {L}, \mathcal {H}} (h, x) \right]\right) ^ {2} \\ \leq \mathbb {E} _ {X} \left[ \Delta \mathcal {C} _ {\mathrm {L}, \mathcal {H}} (h, x) ^ {2} \right] \\ \leq \mathbb {E} _ {X} \left[ \Delta \mathcal {C} _ {\mathrm {L} ^ {\text {c s t n d}}, \mathcal {H}} (h, x) \right] \\ = \mathcal {R} _ {\mathrm {L c s t n d}} (h) - \mathcal {R} _ {\mathrm {L c s t n d}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathrm {L c s t n d}, \mathcal {H}}, \\ \end{array}
$$
$$
\mathcal {R} _ {\mathsf {L}} (h) - \mathcal {R} _ {\mathsf {L}, \mathcal {H}} ^ {*} \leq \left(\mathcal {R} _ {\mathsf {L} _ {\mathrm {s q - h i n g e}} ^ {\mathrm {c s t n d}}} (h) - \mathcal {R} _ {\mathsf {L} _ {\mathrm {s q - h i n g e}} ^ {\mathrm {c s t n d}}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathsf {L} _ {\mathrm {s q - h i n g e}} ^ {\mathrm {c s t n d}}, \mathcal {H}}\right) ^ {\frac {1}{2}} - \mathcal {M} _ {\mathsf {L}, \mathcal {H}}.
$$
F.3 Structured constrained hinge loss
Theorem 17 (H-consistency bound of  $\mathsf{L}_{\mathrm{hinge}}^{\mathrm{cstnd}}$ ). Assume that  $\mathcal{H}$  is symmetric and complete. Then, for any target loss  $\ell$ , hypothesis  $h \in \mathcal{H}$  and any distribution,
$$
\mathcal {R} _ {\mathrm {L}} (h) - \mathcal {R} _ {\mathrm {L}, \mathcal {H}} ^ {*} \leq \mathcal {R} _ {\mathrm {L} _ {\text {h i n g e}} ^ {\text {c s t n d}}} (h) - \mathcal {R} _ {\mathrm {L} _ {\text {h i n g e}} ^ {\text {c s t n d}}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathrm {L} _ {\text {h i n g e}} ^ {\text {c s t n d}}, \mathcal {H}} - \mathcal {M} _ {\mathrm {L}, \mathcal {H}}. \tag {31}
$$
Proof. Denote by  $q(x, y') = \sum_{y \in \mathcal{Y}} p(x, y) \ell(y', y) \in [0, \ell_{\max}]$ . For the constrained structured loss  $L_{\text{hinge}}^{\text{cstnd}}$ , the conditional  $L_{\text{hinge}}^{\text{cstnd}}$ -risk can be expressed as follows:
$$
\begin{array}{l} \mathcal{C}_{\text{\rm l}_{\text{\rm hinge}}}^{\text{\rm cstnd}}(h,x) = \sum_{y\in \mathbb{Y}}p(x,y)\sum_{y^{\prime}\in \mathbb{Y}}\ell (y^{\prime},y)\max \{0,1 + h(x,y^{\prime})\} \\ = \sum_ {y ^ {\prime} \in \mathcal {Y}} \max  \left\{0, 1 + h (x, y ^ {\prime}) \right\} q (x, y ^ {\prime}). \\ \end{array}
$$
Let  $y_{\min} = \operatorname*{argmin}_{y' \in \mathcal{Y}} \sum_{y \in \mathcal{Y}} p(x, y) \ell(y', y)$ , where we choose the label with the highest index under the natural ordering of labels as the tie-breaking strategy. For any  $h \in \mathcal{H}$  such that  $h(x) \neq y_{\min}$  and  $x \in \mathcal{X}$ , by the symmetry and completeness of  $\mathcal{H}$ , we can always find a family of hypotheses  $\{\overline{h}_\mu : \mu \in \mathbb{R}\} \subset \mathcal{H}$  such that  $h_\mu(x, \cdot)$  take the following values:
$$
\bar {h} _ {\mu} (x, y) = \left\{ \begin{array}{l l} h (x, y) & \text {i f} y \notin \left\{y _ {\min }, \mathrm {h} (x) \right\} \\ h \left(x, \mathrm {y} _ {\min }\right) + \mu & \text {i f} y = \mathrm {h} (x) \\ h (x, \mathrm {h} (x)) - \mu & \text {i f} y = \mathrm {y} _ {\min } \end{array} \right. \tag {32}
$$
Since  $\sum_{y\in \mathcal{Y}}h(x,y) = 0$ , there must be non-negative scores. By definition of  $\mathsf{h}(x)$  as a maximizer, we must thus have  $h(x,\mathsf{h}(x))\geq 0$ . By (32) and using the fact that  $\mathsf{H}(x) = \mathcal{Y}$  when  $\mathcal{H}$  is symmetric, we obtain
$$
\begin{array}{l} \Delta \mathcal {C} _ {\mathrm {L} ^ {\text {c s t n d}}, \mathcal {H}} (h, x) \\ = \mathcal {C} _ {\mathrm {L} ^ {\text {c s t n d}}} (h, x) - \mathcal {C} _ {\mathrm {L} ^ {\text {c s t n d}}} ^ {*} (\mathcal {H}, x) \\ \geq \mathcal {C} _ {\mathrm {L} ^ {\text {c s t n d}}} (h, x) - \inf  _ {\mu \in \mathbb {R}} \mathcal {C} _ {\mathrm {L} ^ {\text {c s t n d}}} (\bar {h} _ {\mu}, x) \\ = \sup  _ {\mu \in \mathbb {R}} \left\{q (x, y _ {\min }) \left(\max  \{0, 1 + h (x, y _ {\min }) \} - \max  \{0, 1 + h (x, \mathrm {h} (x)) - \mu \}\right) \right. \\ \left. \right.\left. + q (x, h (x)) \left(\max  \{0, 1 + h (x, h (x)) \} - \max  \{0, 1 + h (x, y _ {\min }) + \mu \}\right)\right\} \\ \geq \left(1 + h (x, \mathrm {h} (x))\right) \left(q (x, \mathrm {h} (x)) - q (x, y _ {\min })\right) \quad (\text {d i f f e r e n t i a t i n g w i t h r e s p e c t t o} \mu \text {t o o p t i m i z e}) \\ \geq q (x, \mathrm {h} (x)) - q (x, y _ {\min }) \quad (h (x, \mathrm {h} (x)) \geq 0) \\ = \sum_ {y \in \mathcal {Y}} p (x, y) \ell (\mathsf {h} (x), y) - \sum_ {y \in \mathcal {Y}} p (x, y) \ell (y _ {\min }, y) \\ = \Delta \mathcal {C} _ {\mathrm {L}, \mathcal {H}} (h, x). \quad (\text {b y}) \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {R} _ {\mathrm {L}} (h) - \mathcal {R} _ {\mathrm {L}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathrm {L}, \mathcal {H}} = \underset {X} {\mathbb {E}} \left[ \Delta \mathcal {C} _ {\mathrm {L}, \mathcal {H}} (h, x) \right] \\ \leq \mathbb {E} _ {X} \left[ \Delta \mathcal {C} _ {\mathrm {L} ^ {\text {c s t n d}}, \mathcal {H}} (h, x) \right] \\ = \mathcal {R} _ {\mathrm {L} ^ {\text {c s t n d}}} (h) - \mathcal {R} _ {\mathrm {L} ^ {\text {c s t n d}}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathrm {L} ^ {\text {c s t n d}}, \mathcal {H}}, \\ \end{array}
$$
$$
\mathcal {R} _ {\mathsf {L}} (h) - \mathcal {R} _ {\mathsf {L}, \mathcal {H}} ^ {*} \leq \mathcal {R} _ {\mathsf {L} _ {\mathrm {h i n g e}} ^ {\mathrm {c s t n d}}} (h) - \mathcal {R} _ {\mathsf {L} _ {\mathrm {h i n g e}} ^ {\mathrm {c s t n d}}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathsf {L} _ {\mathrm {h i n g e}} ^ {\mathrm {c s t n d}}, \mathcal {H}} - \mathcal {M} _ {\mathsf {L}, \mathcal {H}}.
$$
F.4 Structured constrained  $\rho$ -margin loss
Theorem 18 ( $\mathcal{H}$ -consistency bound of  $\mathsf{L}_{\rho}^{\mathrm{cstnd}}$ ). Assume that  $\mathcal{H}$  is symmetric and complete. Then, for any target loss  $\ell$ , hypothesis  $h \in \mathcal{H}$  and any distribution,
$$
\mathcal {R} _ {\mathrm {L}} (h) - \mathcal {R} _ {\mathrm {L}, \mathcal {H}} ^ {*} \leq \mathcal {R} _ {\mathrm {L} _ {\rho} ^ {\mathrm {c s t n d}}} (h) - \mathcal {R} _ {\mathrm {L} _ {\rho} ^ {\mathrm {c s t n d}}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathrm {L} _ {\rho} ^ {\mathrm {c s t n d}}, \mathcal {H}} - \mathcal {M} _ {\mathrm {L}, \mathcal {H}}. \tag {33}
$$
Proof. Denote by  $q(x, y') = \sum_{y \in \mathcal{Y}} p(x, y) \ell(y', y) \in [0, \ell_{\max}]$ . For the constrained structured loss  $L_{\rho}^{\mathrm{cstnd}}$ , the conditional  $L_{\rho}^{\mathrm{cstnd}}$ -risk can be expressed as follows:
$$
\begin{array}{l} \mathcal {C} _ {\underset {\rho} {\mathrm {l c s t n d}}} (h, x) = \sum_ {y \in \mathbb {Y}} p (x, y) \sum_ {y ^ {\prime} \in \mathbb {Y}} \ell (y ^ {\prime}, y) \max  \{0, 1 + h (x, y ^ {\prime}) \} \\ = \sum_ {y ^ {\prime} \in \mathcal {Y}} \max  \{0, 1 + h (x, y ^ {\prime}) \} q (x, y ^ {\prime}). \\ \end{array}
$$
$$
\bar {h} _ {\mu} (x, y) = \left\{ \begin{array}{l l} h (x, y) & \text {i f} y \notin \left\{y _ {\min }, \mathrm {h} (x) \right\} \\ h \left(x, y _ {\min }\right) + \mu & \text {i f} y = \mathrm {h} (x) \\ h (x, \mathrm {h} (x)) - \mu & \text {i f} y = y _ {\min } \cdot \end{array} \right. \tag {34}
$$
Since  $\sum_{y\in \mathcal{Y}}h(x,y) = 0$ , there must be non-negative scores. By definition of  $\mathsf{h}(x)$  as a maximizer, we must thus have  $h(x,\mathsf{h}(x))\geq 0$ . By (34) and using the fact that  $\mathsf{H}(x) = \mathcal{Y}$  when  $\mathcal{H}$  is symmetric, we obtain
$$
\begin{array}{l} \Delta \mathcal {C} _ {\text {L c s t n d}, \mathcal {H}} (h, x) \\ = \mathcal {C} _ {\mathrm {L} ^ {\text {c s t n d}}} (h, x) - \mathcal {C} _ {\mathrm {L} ^ {\text {c s t n d}}} ^ {*} (\mathcal {H}, x) \\ \geq \mathcal {C} _ {\mathrm {L} ^ {\text {c s t n d}}} (h, x) - \inf  _ {\mu \in \mathbb {R}} \mathcal {C} _ {\mathrm {L} ^ {\text {c s t n d}}} (\bar {h} _ {\mu}, x) \\ = \sup  _ {\mu \in \mathbb {R}} \left\{q (x, y _ {\min }) \left(\min  \left\{\max  \left\{0, 1 + \frac {h (x , y _ {\min })}{\rho} \right\}, 1 \right\} - \min  \left\{\max  \left\{0, 1 + \frac {h (x , h (x)) - \mu}{\rho} \right\}, 1 \right\}\right) \right. \\ \left. + q (x, \mathsf {h} (x)) \left(\min  \left\{\max  \left\{0, 1 + \frac {h (x , \mathsf {h} (x))}{\rho} \right\}, 1 \right\} - \min  \left\{\max  \left\{0, 1 + \frac {h (x , y _ {\operatorname* {m i n}}) + \mu}{\rho} \right\}, 1 \right\}\right)\right) \\ \geq q (x, \mathrm {h} (x)) - q (x, y _ {\min }) \quad (\text {d i f f e r e n t i a t i n g w i t h r e s p e c t t o} \mu \text {t o o p t i m i z e}) \\ = \sum_ {y \in \mathcal {Y}} p (x, y) \ell (\mathsf {h} (x), y) - \sum_ {y \in \mathcal {Y}} p (x, y) \ell (y _ {\min }, y) \\ = \Delta \mathcal {C} _ {\mathrm {L}, \mathcal {H}} (h, x). \quad (\text {b y} \quad \mathrm {L e m m a 3 a n d} \mathrm {H} (x) = \mathcal {Y}) \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {R} _ {\mathrm {L}} (h) - \mathcal {R} _ {\mathrm {L}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathrm {L}, \mathcal {H}} = \underset {X} {\mathbb {E}} \left[ \Delta \mathcal {C} _ {\mathrm {L}, \mathcal {H}} (h, x) \right] \\ \leq \mathbb {E} _ {X} \left[ \Delta \mathcal {C} _ {\mathrm {L} ^ {\text {c s t n d}}, \mathcal {H}} (h, x) \right] \\ = \mathcal {R} _ {\mathrm {L c s t n d}} (h) - \mathcal {R} _ {\mathrm {L c s t n d}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathrm {L c s t n d}, \mathcal {H}}, \\ \end{array}
$$
$$
\mathcal {R} _ {\mathsf {L}} (h) - \mathcal {R} _ {\mathsf {L}, \mathcal {H}} ^ {*} \leq \mathcal {R} _ {\mathsf {L} _ {\rho} ^ {\mathrm {c s t n d}}} (h) - \mathcal {R} _ {\mathsf {L} _ {\rho} ^ {\mathrm {c s t n d}}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathsf {L} _ {\rho} ^ {\mathrm {c s t n d}}, \mathcal {H}} - \mathcal {M} _ {\mathsf {L}, \mathcal {H}}.
$$
G Efficient gradient computation and inference
Here, we describe efficient algorithms for the computation of the gradients for the loss functions  $\mathsf{L}_{\log}^{\mathrm{comp}}$  and  $\mathsf{L}_{\exp}^{\mathrm{comp}}$ . We also briefly discuss an efficient algorithm for inference.
G.1 Efficient gradient computation for  $\mathsf{L}_{\mathrm{log}}^{\mathrm{comp}}$
We first present an efficient algorithm for the computation of the quantities  $\mathsf{L}(\mathbf{z}, s)$  in the important case of rational losses, next in the case of Markovian losses.
Rational losses. Rational losses form a general family of loss functions based on rational kernels [Cortes et al., 2004] that includes, in particular,  $n$ -gram losses, which can be defined for a pair of sequences  $(y, y')$  as the negative inner product of the vectors of  $n$ -gram counts of  $y$  and  $y'$ .
Our algorithm bears some similarity to that of Cortes et al. [2018] for the computation of the gradient of the VCRF loss function. It is however distinct because the structured prediction loss function we are considering and our definition of rational loss are both different. We will adopt a similar notation and terminology. Recall that for any sequence  $y$ , we denote by  $y_{i}$  the symbol in its  $i$ th position and by  $y_{i}^{j} = y_{i}y_{i + 1}\dots y_{j}$  the substring of  $y$  starting at position  $i$  and ending at  $j$ . We denote by  $\mathsf{E}_{\mathcal{A}}$  the set of transitions of a WFA  $\mathcal{A}$ .
Let  $\mathcal{U}$  be a weighted finite-state transducer (WFST) over the  $(+, \times)$  semiring over the reals, with  $\Delta$  as both the input and output alphabet. Then, we define the rational loss associated to  $\mathcal{U}$  for all  $y, y' \in \Delta^*$  by  $\bar{\ell}(y, y') = \mathcal{U}(y, y')$ .
Figure 3: Illustration of the WFA  $\overline{\mathcal{Y}}$  for  $\Delta = \{a,b\}$  and  $l = 3$  and the WFA  $\mathcal{Y}_i$  where  $y_{i} = aba$ .
Figure 4: Illustration of the WFA  $\mathcal{N}$  for  $\Delta = \{a,b\}$ ,  $p = 2$  and  $l = 2$ .
$$
\mathsf {E} (\mathbf {z}, s) = \left\{\left(\left(q _ {\mathcal {N}}, q _ {\mathcal {M}}\right), z _ {p}, \omega , \left(q _ {\mathcal {N}} ^ {\prime}, q _ {\mathcal {M}} ^ {\prime}\right)\right) \in \mathsf {E} _ {\mathcal {N} \circ \mathcal {M}}: q _ {\mathcal {N}} = \left(\mathbf {z} _ {1} ^ {p - 1}, s - 1\right) \right\}. \tag {35}
$$
The WFA  $\mathcal{N} \circ \mathcal{M}$  is deterministic as a composition of two deterministic WFAs. Thus, there is a unique path labeled with a sequence  $y \in \Delta^l$  in  $\mathcal{N} \circ \mathcal{M}$  and  $y$  admits the substring  $\mathbf{z}$  ending at position
$s$  iff that path goes through a transition in  $\mathsf{E}(\mathbf{z}, s)$  when reaching position  $s$ . Therefore, to compute  $\mathsf{L}(\mathbf{z}, s)$ , it suffices for us to compute the sum of the weights of all paths in  $\mathcal{N} \circ \mathcal{M}$  going through a transition in  $\mathsf{E}(\mathbf{z}, s)$ . This can be done straightforwardly using the forward-backward algorithm or two single-source shortest-distance algorithm over the  $(+, \times)$  semiring [Mohri, 2002a], one from the initial state, the other one from the final states. Since  $\mathcal{N} \circ \mathcal{M}$  is acyclic and admits  $O(l|\Delta|^p)$  transitions, we can compute all the quantities  $\mathsf{L}(\mathbf{z}, s)$ ,  $s \in [l]$  and  $\mathbf{z} \in \Delta^p$ , in time  $O(l|\Delta|^p)$ .
Markovian loss. We consider adopting a Markovian assumption, which is commonly adopted in natural language processing [Manning and Schütze, 1999]. We will assume that  $\overline{\ell}$  can be decomposed as follows for all  $y, y' \in \Delta^l$ :  $\overline{\ell}(y, y') = \prod_{t=1}^l \overline{\ell}_t(y_{t-p+1}^t, y')$ . Thus, we can write:
$$
\mathsf {L} (\mathbf {z}, s) = \sum_ {y: y _ {s - p + 1} ^ {s}} \prod_ {t = 1} ^ {l} \overline {{\ell}} _ {t} (y _ {t - p + 1} ^ {t}, y _ {i}).
$$
To efficiently compute  $\mathsf{L}(\mathbf{z}, s)$ , we will use a WFA representation similar to the one used by Cortes et al. [2016, 2018] and, for convenience, will adopt a similar notation.  $\mathsf{L}(\mathbf{z}, s)$  coincides with a flow computation in a WFA  $\mathcal{A}$  that we now define.  $\mathcal{A}$  has the following set of states: