SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: A new family of surrogate losses for structured prediction: structured comp-sum losses.
Φ1(v)	Name	Formulation
log(v)	Structured logistic loss	Lcomp log = - ∑y' ∈ y √(l(y', y)) log[ (e^h(x, y')) / ∑y'' ∈ y e^h(x, y'')]
v-1	Structured sum-exponential loss	Lcomp exp = ∑y' ∈ y √(l(y', y)) ∑y'' ≠ y' e^h(x, y'') - h(x, y')
1/α[1 - 1/vα]	Structured generalized cross-entropy loss	Lcomp gce = ∑y' ∈ y √(l(y', y)) 1/α [1 - [ (e^h(x, y')) / ∑y'' ∈ y e^h(x, y'')]α]
1 - 1/v	Structured mean absolute error loss	Lcomp mae = ∑y' ∈ y √(l(y', y)) [1 - (e^h(x, y')) / ∑y'' ∈ y e^h(x, y'')]
generalized cross-entropy loss [Zhang and Sabuncu, 2018]; and  $\mathsf{L}_{\mathrm{mae}}^{\mathrm{comp}}$  coincides with the mean absolute error loss [Ghosh et al., 2017].
We will show that these structured comp-sum losses benefit from  $\mathcal{H}$ -consistency bounds in structured prediction, when  $\mathcal{H}$  is a symmetric and complete hypothesis set. A hypothesis set  $\mathcal{H}$  is symmetric if there exists a family  $\mathcal{F}$  of real-valued functions such that  $\{[h(x,1),\ldots ,h(x,n)];h\in \mathcal{H}\} = \{[f_1(x),\ldots ,f_n(x)];f_1,\ldots ,f_n\in \mathcal{F}\}$  for any  $x\in \mathcal{X}$ . Thus, the choice of the scoring functions does not depend on the order of the categories in  $\mathcal{Y}$ . A hypothesis set  $\mathcal{H}$  is complete if it can generate scores that span  $\mathbb{R}$ , that is,  $\{h(x,y):h\in \mathcal{H}\} = \mathbb{R}$  for any  $(x,y)\in \mathcal{X}\times \mathcal{Y}$ . As shown by Awasthi et al. [2022b] and Mao et al. [2023h], these assumptions are general and hold for common hypothesis sets used in practice, such as the family of linear hypotheses and that of multi-layer feed-forward neural networks, and of course that of all measurable functions.
Theorem 6 ( $\mathcal{H}$ -consistency bound of  $\mathsf{L}^{\mathrm{comp}}$ ). Assume that  $\mathcal{H}$  is symmetric and complete. Then, for any target loss  $\ell$ , any hypothesis  $h \in \mathcal{H}$  and any distribution, we have
$$
\mathcal {R} _ {\mathrm {L}} (h) - \mathcal {R} _ {\mathrm {L}, \mathcal {H}} ^ {*} \leq \Gamma \left(\mathcal {R} _ {\mathrm {L} ^ {\text {c o m p}}} (h) - \mathcal {R} _ {\mathrm {L} ^ {\text {c o m p}}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathrm {L} ^ {\text {c o m p}}, \mathcal {H}}\right) - \mathcal {M} _ {\mathrm {L}, \mathcal {H}}, \tag {9}
$$
where  $\Gamma(t) = 2\sqrt{t}$  when  $\mathsf{L}^{\mathrm{comp}} = \mathsf{L}_{\mathrm{log}}^{\mathrm{comp}}$  or  $\mathsf{L}_{\mathrm{exp}}^{\mathrm{comp}}$ ;  $\Gamma(t) = 2\sqrt{n^\alpha t}$  when  $\mathsf{L}^{\mathrm{comp}} = \mathsf{L}_{\mathrm{gce}}^{\mathrm{comp}}$ ; and  $\Gamma(t) = nt$  when  $\mathsf{L}^{\mathrm{comp}} = \mathsf{L}_{\mathrm{mae}}^{\mathrm{comp}}$ .
Theorem 6 represents a consolidated result for the four structured comp-sum losses, with the proofs for each being presented separately in Appendix E. The key step of the proof is to upper bound the conditional regret of the target loss (Lemma 3) by that of a surrogate loss. To achieve this, we upper bound the best-in-class conditional error by the conditional error of a carefully selected hypothesis  $\overline{h}_{\mu} \in \overline{\mathcal{H}}$ . The resulting softmax  $\overline{S}_{\mu}$  of this hypothesis only differs from the original softmax S corresponding to  $\overline{h}$  on two labels. Theorem 6 admits as special cases the  $\mathcal{H}$ -consistency bounds of Mao et al. [2023h] given for standard multi-class classification  $(\ell = \ell_{0-1})$  and significantly extends them to the general structured prediction scenario.
Let us emphasize that our proof technique is novel and distinct from the approach used in [Mao et al., 2023h], which only applies to the special case where  $\ell$  is the zero-one loss and cannot be generalized to any target loss  $\ell$ . In their proof, the authors choose  $\overline{h}_{\mu}$  based on individual scores  $\overline{h}(x,y)$ , rather than the softmax. Consequently, when  $\ell \neq \ell_{0-1}$ , as is common in structured prediction, the resulting optimization problem of  $\mu$  can be very intricate and a closed-form expression of the optimization solution cannot be derived. However, our new proof method overcomes this limitation. By viewing the softmax of hypothesis as a unit and introducing a pseudo-conditional distribution  $\overline{q}$ , we are able to solve a simple constrained optimization problem on  $\mu$  within structured prediction scenario.
By Steinwart [2007, Theorem 3.2], the minimizability gaps  $\mathcal{M}_{\mathrm{L}^{\mathrm{comp}},\mathcal{H}}$  and  $\mathcal{M}_{\mathrm{L},\mathcal{H}}$  vanish for the family of all measurable functions. Therefore, when  $\mathcal{H} = \mathcal{H}_{\mathrm{all}}$ , the  $\mathcal{H}$ -consistency bounds provided in Theorem 6 imply the Bayes-consistency of these structured comp-sum losses.
Corollary 7. The structured comp-sum loss  $\mathsf{L}^{\mathrm{comp}}$  is Bayes-consistent for  $\mathsf{L}^{\mathrm{comp}} = \mathsf{L}_{\log}^{\mathrm{comp}}, \mathsf{L}_{\exp}^{\mathrm{comp}}, \mathsf{L}_{\mathrm{gce}}^{\mathrm{comp}},$  and  $\mathsf{L}_{\mathrm{mae}}^{\mathrm{comp}}$ .
In fact, Theorem 6 provides stronger quantitative bounds than Bayes-consistency when the minimizability gaps vanish, which suggests that if the estimation error of the structured comp-sum loss  $\mathcal{R}_{\mathsf{L}^{\mathrm{comp}}}(h) - \mathcal{R}_{\mathsf{L}^{\mathrm{comp}},\mathcal{H}}^{*}$  is reduced to  $\epsilon$ , the estimation error of the target loss  $\mathcal{R}_{\mathsf{L}}(h) - \mathcal{R}_{\mathsf{L},\mathcal{H}}^{*}$  is upper bounded by  $2\sqrt{\epsilon}$  for structured logistic loss and structured sum-exponential loss,  $2\sqrt{n^{\alpha}\epsilon}$  for structured generalized cross-entropy loss, and  $n\epsilon$  for structured mean absolute error loss.
Table 2: A new family of surrogate losses for structured prediction: structured constrained losses.
Φu(v)	Name	Formulation (Σy∈y h(x,y) = 0)
ue-ν	Structured constrained exponential loss	Lcstd exp = ∑y'∈y ℓ(y', y) max{0,1-h(x,y')}2
u max{0,1-v}2	Structured constrained squared-hinge loss	Lcstd hinge = ∑y'∈y ℓ(y', y) max{0,1-h(x,y')}
u max{0,1-v}	Structured constrained hinge loss	Lcstd hinge = ∑y'∈y ℓ(y', y) max{0,1-h(x,y')}
u min{max{0,1-v/ρ},1}	Structured constrained ρ-margin loss	Lcstd ρ = ∑y'∈y ℓ(y', y) min{max{0,1-h(x,y')} ,1}.
5 Structured constrained loss functions
In this section, we introduce another new family of surrogate losses for structured prediction that we prove to admit  $\mathcal{H}$ -consistency bounds. We will present a novel generalization of the constrained losses [Lee et al., 2004, Awasthi et al., 2022b] to structured prediction. Thus, we refer to them as structured constrained losses and define them as follows:
$$
\forall (x, y) \in \mathcal {X} \times \mathcal {Y}, \quad \mathrm {L} ^ {\text {c s t n d}} (h, x, y) = \sum_ {y ^ {\prime} \in \mathcal {Y}} \Phi_ {\ell \left(y ^ {\prime}, y\right)} (- h (x, y ^ {\prime})), \tag {10}
$$
with the constraint that  $\sum_{y\in \mathcal{Y}}h(x,y) = 0$  and  $\Phi_u\colon \mathbb{R}\to \mathbb{R}_+$  is an upper bound on  $v\mapsto u1_{v\leq 0}$  for any  $u\in \mathbb{R}_{+}$ . In standard constrained loss formulation, a single-variable function  $\Phi (v)$  that defines a margin-based loss is used. In (10), the single-variable function  $\Phi (v)$  is generalized to being a function of two variables  $\Phi_u(v)$ , which depends on both the target loss and the scores, to accommodate the structured prediction scenario. Specifically, we can choose  $\Phi_u(v) = ue^{-v}$ ,  $\Phi_u(v) = u\max \{0,1 - v\}^2$ ,  $\Phi_u(v) = u\max \{0,1 - v\}$ ,  $\Phi_u(v) = u\min \{\max \{0,1 - v / \rho \},1\}$ , which lead to new surrogate losses for structured prediction defined in Table 2. These surrogate losses are novel generalization of their corresponding counterparts [Lee et al., 2004, Awasthi et al., 2022b] in standard multi-class classification, where  $\ell = \ell_{0 - 1}$ . As with structured comp-sum losses, we will show that these structured constrained losses benefit from  $\mathcal{H}$ -consistency bounds in structured prediction as well, for any symmetric and complete hypothesis set  $\mathcal{H}$ .
Theorem 8 ( $\mathcal{H}$ -consistency bound of  $\mathsf{L}^{\mathrm{cstnd}}$ ). Assume that  $\mathcal{H}$  is symmetric and complete. Then, for any target loss  $\ell$ , hypothesis  $h \in \mathcal{H}$  and any distribution, we have
$$
\mathcal {R} _ {\mathrm {L}} (h) - \mathcal {R} _ {\mathrm {L}, \mathcal {H}} ^ {*} \leq \Gamma \left(\mathcal {R} _ {\mathrm {L} ^ {\text {c s t n d}}} (h) - \mathcal {R} _ {\mathrm {L} ^ {\text {c s t n d}}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathrm {L} ^ {\text {c s t n d}}, \mathcal {H}}\right) - \mathcal {M} _ {\mathrm {L}, \mathcal {H}}. \tag {11}
$$
where  $\Gamma(t) = 2\sqrt{\ell_{\mathrm{max}}t}$  when  $\mathsf{L}^{\mathrm{cstnd}} = \mathsf{L}_{\mathrm{exp}}^{\mathrm{cstnd}}$ ;  $\Gamma(t) = 2\sqrt{t}$  when  $\mathsf{L}^{\mathrm{cstnd}} = \mathsf{L}_{\mathrm{sq - hinge}}^{\mathrm{cstnd}}$ ; and  $\Gamma(t) = t$  when  $\mathsf{L}^{\mathrm{cstnd}} = \mathsf{L}_{\mathrm{hinge}}^{\mathrm{cstnd}}$  or  $\mathsf{L}_{\rho}^{\mathrm{cstnd}}$ .
The proof is included in Appendix F. As for Theorem 8, the key part of the proof is to upper bound the conditional regret of the target loss (Lemma 3) by that of a surrogate loss. Here too, we introduce a pseudo-conditional distribution  $q$ , which can be viewed as a weighted distribution of the original one,  $p(x)$ , with weights given by the target loss function. Then, we upper bound the best-in-class conditional error by the conditional error of a carefully selected hypothesis  $\overline{h}_{\mu} \in \overline{\mathcal{H}}$ .
As shown by Steinwart [2007, Theorem 3.2], for the family of all measurable functions, the minimizability gaps vanish:  $\mathcal{M}_{\mathrm{L}^{\mathrm{cstnd}},\mathcal{H}} = 0$  and  $\mathcal{M}_{\mathrm{L},\mathcal{H}} = 0$ . Therefore, when  $\mathcal{H} = \mathcal{H}_{\mathrm{all}}$ , the  $\mathcal{H}$ -consistency bounds provided in Theorem 6 imply the Bayes-consistency of these structured constrained losses.
Corollary 9. The structured constrained loss  $\mathsf{L}^{\mathrm{cstnd}}$  is Bayes-consistent for  $\mathsf{L}^{\mathrm{cstnd}} = \mathsf{L}_{\mathrm{exp}}^{\mathrm{cstnd}}, \mathsf{L}_{\mathrm{sq - hinge}}^{\mathrm{cstnd}}, \mathsf{L}_{\mathrm{hinge}}^{\mathrm{cstnd}},$  and  $\mathsf{L}_{\rho}^{\mathrm{cstnd}}$ .
As with the cases of structured comp-sum losses, Theorem 8 provides in fact stronger quantitative bounds than Bayes-consistency. They show that if the estimation error of the structured constrained loss  $\mathcal{R}_{\mathrm{L}^{\mathrm{comp}}}(h) - \mathcal{R}_{\mathrm{L}^{\mathrm{comp}},\mathcal{H}}^{*}$  is reduced to  $\epsilon$ , the estimation error of the target loss  $\mathcal{R}_{\mathrm{L}}(h) - \mathcal{R}_{\mathrm{L},\mathcal{H}}^{*}$  is upper bounded by  $2\sqrt{\ell_{\max}\epsilon}$  for  $\mathsf{L}_{\mathrm{exp}}^{\mathrm{cstnd}}$ ,  $2\sqrt{\epsilon}$  for  $\mathsf{L}_{\mathrm{sq - hinge}}^{\mathrm{cstnd}}$  and  $\epsilon$  for  $\mathsf{L}_{\mathrm{hinge}}^{\mathrm{cstnd}}$  and  $\mathsf{L}_{\rho}^{\mathrm{cstnd}}$ .
It is important to note that we can upper bound the minimizability gap by the approximation error, or finer terms depending on the magnitude of the parameter space as in [Mao et al., 2023h]. Furthermore, our  $\mathcal{H}$ -consistency bounds (Theorems 6 and 8) can be used to derive finite sample learning bounds for a hypothesis set  $\mathcal{H}$ . These bounds depend on the Rademacher complexity of the hypothesis set and the loss function, as well as an upper bound on the minimizability gap for the surrogate loss.
6 Optimization of  $\mathsf{L}_{\mathrm{log}}^{\mathrm{comp}}$  and  $\mathsf{L}_{\mathrm{exp}}^{\mathrm{comp}}$
In this section, we show that the gradient of the structured logistic loss  $\mathsf{L}_{\log}^{\mathrm{comp}}$  can be computed efficiently at any point  $(x_i,y_i)$  and therefore that this loss function is both supported by  $\mathcal{H}$ -consistency bounds and is of practical use. We similarly show that for  $\mathsf{L}_{\exp}^{\mathrm{comp}}$  in Appendix G.2.
Fix the labeled pair  $(x_{i},y_{i})$  and  $h\in \mathcal{H}$ . Observe that  $\mathrm{L}_{\log}^{\mathrm{comp}}(h,x_i,y_i)$  can be equivalently rewritten as follows:
$$
\begin{array}{l} \mathsf {L} _ {\log} ^ {\mathrm {c o m p}} (h, x _ {i}, y _ {i}) = \sum_ {y ^ {\prime} \in \mathcal {Y}} \overline {{\ell}} (y ^ {\prime}, y _ {i}) \log \left[ \sum_ {y ^ {\prime \prime} \in \mathcal {Y}} e ^ {h (x _ {i}, y ^ {\prime \prime}) - h (x _ {i}, y ^ {\prime})} \right] \\ = - \sum_ {y ^ {\prime} \in \mathcal {Y}} \bar {\ell} (y ^ {\prime}, y _ {i}) h (x _ {i}, y ^ {\prime}) + \left[ \sum_ {y ^ {\prime} \in \mathcal {Y}} \bar {\ell} (y ^ {\prime}, y _ {i}) \right] \log \left[ \sum_ {y ^ {\prime \prime} \in \mathcal {Y}} e ^ {h (x _ {i}, y ^ {\prime \prime})} \right] \\ = - \sum_ {y ^ {\prime} \in \mathcal {Y}} \bar {\ell} (y ^ {\prime}, y _ {i}) h (x _ {i}, y ^ {\prime}) + \bar {\ell} _ {i} \log Z _ {h, i}, \\ \end{array}
$$
where  $\overline{\ell}_i = \sum_{y' \in \mathcal{Y}} \overline{\ell}(y', y_i)$ , and  $Z_{h,i} = \sum_{y \in \mathcal{Y}} e^{h(x_i, y)}$ . Note that  $\overline{\ell}_i$  does not depend on  $h$  and can be pre-computed. Modulo normalization, this quantity is the average similarity of  $y_i$  to  $\mathcal{Y}$ , if we interpret  $\overline{\ell} = 1 - \ell$  as a similarity. While  $\mathcal{Y}$  may be very large, this can be often computed straightforwardly for most loss functions  $\ell$ . For example, for the Hamming loss, for sequences of length  $l$ , we have
$$
\frac {1}{| \mathcal {Y} |} \overline {{{\ell}}} _ {i} = \frac {1}{l} \mathbb {E} \bigg [ \sum_ {k = 1} ^ {l} \big (1 - \mathbb {1} _ {y _ {k} ^ {\prime} \neq y _ {i, k}} \big) \bigg ] = \frac {1}{l} \sum_ {k = 1} ^ {l} \mathbb {E} \big [ \mathbb {1} _ {y _ {k} ^ {\prime} = y _ {i, k}} \big ] = \frac {1}{l} \sum_ {k = 1} ^ {l} \frac {1}{2} = \frac {1}{2}.
$$
Thus, in this case,  $\overline{\ell}_i$  does not depend on  $i$  and is a universal constant. Similarly,  $\overline{\ell}_i$  can be shown to be a constant for many other losses.
Hypothesis set. For the remaining of this section, to simplify the presentation, we will consider the hypothesis set of linear functions  $\mathcal{H} = \left\{x \mapsto \mathbf{w} \cdot \Psi(x, y) : \mathbf{w} \in \mathbb{R}^d\right\}$ , where  $\Psi$  is a feature mapping from  $\mathcal{X} \times \mathcal{Y}$  to  $\mathbb{R}^d$ . Note that a number of classical structured prediction algorithms adopt the same linear hypothesis set: StructSVM [Tsochantaridis et al., 2005b], Max-Margin Markov Networks (M3N) [Taskar et al., 2003b], Conditional Random Field (CRF) [Lafferty et al., 2001b], Voted Conditional Random Field (VCRF) [Cortes et al., 2016]. Our algorithms can also be incorporated into standard procedures for training neural network architectures (see [Cortes et al., 2018], Appendix B).
Markovian features. We will also assume Markovian features, as is common in structured prediction. Features used in practice often admit this property. Furthermore, in the absence of any such assumption, it is known that learning and inference in general are intractable. We will largely adopt here the definitions and notation from [Cortes et al., 2016] and will consider the common case where  $\mathcal{Y}$  is a set of sequences of length  $l$  over a finite alphabet  $\Delta$  of size  $r$ . Other structured problems can be treated in similar ways. We will denote by  $\varepsilon$  the empty string and for any sequence  $y = (y_{1},\ldots ,y_{l})\in \mathcal{Y}$ , we will denote by  $y_{s}^{s^{\prime}} = (y_{s},\dots,y_{s^{\prime}})$  the substring of  $y$  starting at index  $s$  and ending at  $s^{\prime}$ . For convenience, for  $s\leq 0$ , we define  $y_{s}$  by  $y_{s} = \varepsilon$ .
We will assume that the feature vector  $\Psi$  admits a Markovian property of order  $q$ , that is it can be decomposed as follows for any  $(x,y) \in \mathcal{X} \times \mathcal{Y}$ :
$$
\Psi (x, y) = \sum_ {s = 1} ^ {l} \psi (x, y _ {s - q + 1} ^ {s}, s). \tag {12}
$$
for some position-dependent feature vector function  $\psi$  defined over  $\mathcal{X} \times \Delta^q \times [l]$ . We note that we can write  $\Psi = \sum_{k=1}^{p} \tilde{\Psi}_k$  with  $\tilde{\Psi}_k = (0, \ldots, \Psi_k, \ldots, 0)$ . In the following, abusing the notation, we will simply write  $\Psi_k$  instead of  $\tilde{\Psi}_k$ . Each  $\Psi_k$  corresponds to a Markovian feature vector based only on  $k$ -grams,  $p$  is the largest  $k$ . Thus, for any  $x \in \mathcal{X}$  and  $y \in \mathcal{Y}$ , we have
$$
\Psi (\mathbf {x}, y) = \sum_ {k = 1} ^ {p} \Psi_ {k} (x, y). \tag {13}
$$
For any  $k \in [1, p]$ , let  $\psi_k$  denote the position-dependent feature vector function corresponding to  $\Psi_k$ . Also, for any  $x \in \mathcal{X}$  and  $y \in \Delta^l$ , define  $\widetilde{\psi}$  by  $\widetilde{\psi}(x, y_{s-p+1}^s, s) = \sum_{k=1}^p \psi_k(x, y_{s-k+1}^s, s)$ . Observe then that we can write
$$
\Psi (x, y) = \sum_ {k = 1} ^ {p} \Psi_ {k} (x, y) = \sum_ {k = 1} ^ {p} \sum_ {s = 1} ^ {l} \psi_ {k} (x, y _ {s - k + 1} ^ {s}, s) = \sum_ {s = 1} ^ {l} \sum_ {k = 1} ^ {p} \psi_ {k} (x, y _ {s - k + 1} ^ {s}, s) = \sum_ {s = 1} ^ {l} \widetilde {\psi} (x, y _ {s - p + 1} ^ {s}, s).
$$
Gradient computation. Adopting the shorthand  $\mathbf{w}$  for  $h$ , we can rewrite the loss at  $(x_{i},y_{i})$  as:
$$
\mathsf {L} _ {\log} ^ {\mathrm {c o m p}} (\mathbf {w}, x _ {i}, y _ {i}) = - \mathbf {w} \cdot \left[ \sum_ {y ^ {\prime} \in \mathcal {Y}} \bar {\ell} (y ^ {\prime}, y _ {i}) \Psi (x _ {i}, y ^ {\prime}) \right] + \bar {\ell} _ {i} \log Z _ {\mathbf {w}, i}.
$$
Thus, the gradient of  $\mathsf{L}_{\log}^{\mathrm{comp}}$  at any  $\mathbf{w}\in \mathbb{R}^d$  is given by
$$
\begin{array}{l} \nabla L _ {\log} ^ {\operatorname {c o m p}} (\mathbf {w}) = - \sum_ {y ^ {\prime} \in \mathcal {Y}} \bar {\ell} (y ^ {\prime}, y _ {i}) \Psi (x _ {i}, y ^ {\prime}) + \bar {\ell} _ {i} \sum_ {y \in \mathcal {Y}} \frac {e ^ {\mathbf {w} \cdot \Psi (x _ {i} , y)}}{\sum_ {y ^ {\prime \prime} \in \mathcal {Y}} e ^ {\mathbf {w} \cdot \Psi (x _ {i} , y ^ {\prime \prime})}} \Psi (x _ {i}, y) \\ = -\sum_{y^{\prime}\in \mathcal{Y}}\overline{\ell} (y^{\prime},y_{i})\Psi \big(x_{i},y^{\prime}\big) + \overline{\ell}_{i}\underset {y\sim \mathrm{q}_{\mathbf{w}}}{\mathbb{E}}\left[\Psi \big(x_{i},y\big)\right], \\ \end{array}
$$
where  $q_{\mathbf{w}}$  is defined for all  $y \in \mathcal{Y}$  by  $q_{\mathbf{w}}(y) = \frac{e^{\mathbf{w} \cdot \Psi(x_i, y)}}{Z_{\mathbf{w}}}$  with  $Z_{\mathbf{w}} = \sum_{y \in \mathcal{Y}} e^{\mathbf{w} \cdot \Psi(x_i, y)}$ . Note that the sum defining these terms is over a number of sequences  $y$  that is exponential in  $r$  and that the computation appears to be therefore challenging. The following lemma gives the expression of the gradient of  $L_{\log}^{\mathrm{comp}}$  and helps identify the most computationally challenging terms.
Lemma 10. For any  $\mathbf{w} \in \mathbb{R}^d$ , the gradient of  $\mathsf{L}_{\log}^{\mathrm{comp}}$  can be expressed as follows:
$$
\nabla \mathsf {L} _ {\log} ^ {\mathrm {c o m p}} (\mathbf {w}) = \sum_ {s = 1} ^ {l} \sum_ {\mathbf {z} \in \Delta^ {P}} \left[ \bar {\ell} _ {i} \mathsf {Q} _ {\mathbf {w}} (\mathbf {z}, s) - \mathsf {L} (\mathbf {z}, s) \right] \widetilde {\psi} (x _ {i}, \mathbf {z}, s),
$$
where  $\mathsf{Q}_{\mathbf{w}}(\mathbf{z},s) = \sum_{y:y_{s - p + 1}^{s} = \mathbf{z}}\mathsf{q}_{\mathbf{w}}(y)$  and  $\mathsf{L}(\mathbf{z},s) = \sum_{y:y_{s - p + 1}^{s} = \mathbf{z}}\overline{\ell}(y,y_{i})$
Proof. Using the decomposition of the feature vector, we can write:
$$
\begin{array}{l} \sum_ {y \in \mathcal {Y}} \bar {\ell} (y, y _ {i}) \Psi (x _ {i}, y) = \sum_ {y \in \Delta^ {l}} \bar {\ell} (y, y _ {i}) \sum_ {s = 1} ^ {l} \widetilde {\psi} (x _ {i}, y _ {s - p + 1} ^ {s}, s) = \sum_ {s = 1} ^ {l} \sum_ {\mathbf {z} \in \Delta^ {p}} \left[ \sum_ {y: y _ {s - p + 1} ^ {s} = \mathbf {z}} \bar {\ell} (y, y _ {i}) \right] \widetilde {\psi} (x _ {i}, \mathbf {z}, s) \\ \underset {y \sim \mathbf {q} _ {\mathbf {w}}} {\mathbb {E}} \left[ \Psi (x _ {i}, y) \right] = \sum_ {y \in \Delta^ {l}} \mathsf {q} _ {\mathbf {w}} (y) \sum_ {s = 1} ^ {l} \widetilde {\psi} (x _ {i}, y _ {s - p + 1} ^ {s}, s) = \sum_ {s = 1} ^ {l} \sum_ {\mathbf {z} \in \Delta^ {p}} \left[ \sum_ {y: y _ {s - p + 1} ^ {s} = \mathbf {z}} \mathsf {q} _ {\mathbf {w}} (y) \right] \widetilde {\psi} (x _ {i}, \mathbf {z}, s). \\ \end{array}
$$
In light of this result, the bottleneck in the gradient computation is the evaluation of  $\mathsf{Q}_{\mathbf{w}}(\mathbf{z}, s)$  and  $\mathsf{L}(\mathbf{z}, s)$  for all  $s \in [l]$  and  $\mathbf{z} \in \Delta^p$ . In previous work [Cortes, Kuznetsov, Mohri, and Yang, 2016, Cortes, Kuznetsov, Mohri, Storchetus, and Yang, 2018], it was shown that the quantities  $\mathsf{Q}_{\mathbf{w}}(\mathbf{z}, s)$  can be determined efficiently, all together, by running two single-source shortest-distance algorithms over the  $(+, \times)$  semiring on an appropriate weighted finite automaton (WFA). The overall time complexity of the computation of all quantities  $\mathsf{Q}_{\mathbf{w}}(\mathbf{z}, s)$ ,  $\mathbf{z} \in \Delta^p$  and  $s \in [l]$ , is then in  $O(lr^p)$ , where  $r = |\Delta|$ .
We now analyze the computation of  $\mathsf{L}(\mathbf{z}, s)$  for a fixed  $\mathbf{z} \in \Delta^p$  and  $s \in [l]$ . Note that, unlike  $\mathsf{Q}_{\mathbf{w}}(\mathbf{z}, s)$ , this term does not depend on  $\mathbf{w}$  and can therefore be computed once and for all, before any gradient computation. The sum defining  $\mathsf{L}(\mathbf{z}, s)$  is over all sequences  $y$  that admit the substring  $\mathbf{z}$  at position  $s$ .
Rational losses. In Appendix G.1, we also give an efficient algorithm for the computation of the quantities  $\mathsf{L}(\mathbf{z}, s)$  in the case of Markovian losses. Here, we present an efficient algorithm for their computation in the important case of rational losses. This is a general family of loss functions based on rational kernels [Cortes, Haffner, and Mohri, 2004] that includes, in particular,  $n$ -gram losses, which can be defined for a pair of sequences  $(y, y')$  as the negative inner product of the vectors of  $n$ -gram counts of  $y$  and  $y'$ .
Our algorithm bears some similarity to that of Cortes et al. [2018] for the computation of the gradient of the VCRF loss function. It is however distinct because the structured prediction loss function we are considering and our definition of rational loss are both different. We will adopt a similar notation and terminology. Recall that for any sequence  $y$ , we denote by  $y_{i}$  the symbol in its  $i$ th position and by  $y_{i}^{j} = y_{i}y_{i + 1}\dots y_{j}$  the substring of  $y$  starting at position  $i$  and ending at  $j$ . We denote by  $\mathsf{E}_{\mathcal{A}}$  the set of transitions of a WFA  $\mathcal{A}$ . Let  $\mathcal{U}$  be a weighted finite-state transducer (WFST) over the  $(+, \times)$  semiring over the reals, with  $\Delta$  as both the input and output alphabet. Then, we define the rational loss associated to  $\mathcal{U}$  for all  $y, y' \in \Delta^*$  by  $\bar{\ell}(y, y') = \mathcal{U}(y, y')$ .
Let  $\overline{\mathcal{Y}}$  denote a WFA over the  $(+, \times)$  semiring accepting the set of all sequences of length  $l$  with weight one and let  $\mathcal{Y}_i$  denote the WFA accepting only  $y_i$  with weight one. Then, by definition, the weighted transducer  $\overline{\mathcal{Y}} \circ \mathcal{U} \circ \mathcal{Y}_i$  obtained by composition maps each sequence  $y$  in  $\Delta^l$  to  $y_i$  with weight  $\mathcal{U}(y, y_i)$ . The WFA  $\Pi_1(\overline{\mathcal{Y}} \circ \mathcal{U} \circ \mathcal{Y}_i)$  derived from that transducer by projection on the input (that is by removing output labels) is associating to each sequence  $y$  weight  $\mathcal{U}(y, y_i)$ . We use weighted determinization [Mohri, 1997] to compute an equivalent deterministic WFA denote  $\mathcal{M}$ . As shown by Cortes et al.