SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
Q _ {\mathcal {A}} = \left\{\left(y _ {t - p + 1} ^ {t}, t\right): y \in \Delta^ {l}, t = 0, \dots , l \right\},
$$
with  $\mathsf{l}_{\mathcal{A}} = (\varepsilon, 0)$  its single initial state,  $\mathcal{F}_{\mathcal{A}} = \{(y_{l - p + 1}^{l}, l): y \in \Delta^{l}\}$  its set of final states, and a transition from state  $(y_{t - p + 1}^{t - 1}, t - 1)$  to state  $(y_{t - p + 2}^{t - 1}b, t)$  with label  $b$  and weight  $\omega(y_{t - p + 1}^{t - 1}b, t) = \overline{\ell}_t(y_{t - p + 1}^{t - 1}b, y_i)$ , that is the following set of transitions:
$$
\mathsf {E} _ {\mathcal {A}} = \left\{\left(\left(y _ {t - p + 1} ^ {t - 1}, t - 1\right), b, \omega \left(y _ {t - p + 1} ^ {t - 1} b, t\right), \left(y _ {t - p + 2} ^ {t - 1} b, t\right)\right): y \in \Delta^ {l}, b \in \Delta , t \in [ l ] \right\}.
$$
By construction,  $\mathcal{A}$  is deterministic. The weight of a path in  $\mathcal{A}$  is obtained by multiplying the weights of its constituent transitions. In view of that,  $\mathsf{L}(\mathbf{z},s)$  can be seen as the sum of the weights of all paths in  $\mathcal{A}$  going through the transition from state  $(\mathbf{z}_1^{p - 1},s - 1)$  to  $(\mathbf{z}_2^p,s)$  with label  $z_{p}$ .
For any state  $(y_{t - p + 1}^t,t)\in Q_{\mathcal{A}}$ , we denote by  $\alpha ((y_{t - p + 1}^t,t))$  the sum of the weights of all paths in  $\mathcal{A}$  from the initial state  $\mathsf{l}_{\mathcal{A}}$  to  $(y_{t - p + 1}^t,t)$  and by  $\beta ((y_{t - p + 1}^t,t))$  the sum of the weights of all paths from  $(y_{t - p + 1}^t,t)$  to a final state. Then,  $\mathsf{L}(\mathbf{z},s)$  is given by
$$
\mathsf {L} (\mathbf {z}, s) = \alpha \bigl (\bigl (\mathbf {z} _ {1} ^ {p - 1}, s - 1 \bigr) \bigr) \times \omega (\mathbf {z}, s) \times \beta \bigl (\bigl (\mathbf {z} _ {2} ^ {p}, s \bigr) \bigr).
$$
Since  $\mathcal{A}$  is acyclic,  $\alpha$  and  $\beta$  can be computed for all states in linear time in the size of  $\mathcal{A}$  using a single-source shortest-distance algorithm over the  $(+, \times)$  semiring or the so-called forward-backward algorithm. Thus, since  $\mathcal{A}$  admits  $O(l|\Delta|^p)$  transitions, we can also compute all quantities  $\mathsf{L}(\mathbf{z}, s), s \in [l]$  and  $\mathbf{z} \in \Delta^p$ , in time  $O(lr^p)$ .
G.2 Efficient gradient computation for  $\mathsf{L}_{\mathrm{exp}}^{\mathrm{comp}}$
In this section, we provide a brief overview of the gradient computation for  $\mathsf{L}_{\mathrm{exp}}^{\mathrm{comp}}$ , which is similar to the approach used for  $\mathsf{L}_{\log}^{\mathrm{comp}}$ .
Note that the loss  $\mathsf{L}_{\mathrm{exp}}^{\mathrm{comp}}$  on a given point  $(x_{i},y_{i})$  can be expressed as follows:
$$
\begin{array}{l} \mathsf {L} _ {\exp} ^ {\mathrm {c o m p}} = \sum_ {y ^ {\prime} \in \Delta^ {l}} \bar {\ell} (y ^ {\prime}, y _ {i}) \sum_ {y ^ {\prime \prime} \neq y ^ {\prime}} e ^ {h (x _ {i}, y ^ {\prime \prime}) - h (x _ {i}, y ^ {\prime})} \\ = \sum_ {y ^ {\prime} \in \Delta^ {l}} \bar {\ell} \left(y ^ {\prime}, y _ {i}\right) \sum_ {y ^ {\prime \prime} \in \Delta^ {l}} e ^ {h \left(x _ {i}, y ^ {\prime \prime}\right) - h \left(x _ {i}, y ^ {\prime}\right)} - \sum_ {y ^ {\prime} \in \Delta^ {l}} \bar {\ell} \left(y ^ {\prime}, y _ {i}\right) \\ = \left[ \sum_ {y ^ {\prime \prime} \in \Delta^ {l}} e ^ {h \left(x _ {i}, y ^ {\prime \prime}\right)} \right] \sum_ {y ^ {\prime} \in \Delta^ {l}} \bar {\ell} \left(y ^ {\prime}, y _ {i}\right) e ^ {- h \left(x _ {i}, y ^ {\prime}\right)} - \sum_ {y ^ {\prime} \in \Delta^ {l}} \bar {\ell} \left(y ^ {\prime}, y _ {i}\right) \\ = \left[ \sum_ {y ^ {\prime \prime} \in \Delta^ {l}} e ^ {\mathbf {w} \cdot \Psi (x _ {i}, y ^ {\prime \prime})} \right] \sum_ {y ^ {\prime} \in \Delta^ {l}} \bar {\ell} (y ^ {\prime}, y _ {i}) e ^ {- \mathbf {w} \cdot \Psi (x _ {i}, y ^ {\prime})} - \sum_ {y ^ {\prime} \in \Delta^ {l}} \bar {\ell} (y ^ {\prime}, y _ {i}). \\ \end{array}
$$
The gradient of  $\mathsf{L}_{\mathrm{exp}}^{\mathrm{comp}}$  is therefore given by
$$
\begin{array}{l} \nabla L _ {\exp} ^ {\operatorname {c o m p}} (\mathbf {w}) = \left[ \sum_ {y ^ {\prime \prime} \in \Delta^ {l}} e ^ {\mathbf {w} \cdot \Psi \left(x _ {i}, y ^ {\prime \prime}\right)} \Psi \left(x _ {i}, y ^ {\prime \prime}\right) \right] \sum_ {y ^ {\prime} \in \Delta^ {l}} \bar {\ell} \left(y ^ {\prime}, y _ {i}\right) e ^ {- \mathbf {w} \cdot \Psi \left(x _ {i}, y ^ {\prime}\right)} \tag {36} \\ - \left[ \sum_ {y ^ {\prime \prime} \in \Delta^ {l}} e ^ {\mathbf {w} \cdot \Psi (x _ {i}, y ^ {\prime \prime})} \right] \sum_ {y ^ {\prime} \in \Delta^ {l}} \overline {{\ell}} (y ^ {\prime}, y _ {i}) e ^ {- \mathbf {w} \cdot \Psi (x _ {i}, y ^ {\prime})} \Psi (x _ {i}, y ^ {\prime}). \\ \end{array}
$$
An efficient computation of these terms is not straightforward since the summations run over an exponential number of sequences for  $y$ . However, we will leverage the Markovian property of the features to design an efficient computation. This approach is similar to what we demonstrated earlier for  $\mathsf{L}_{\log}^{\mathrm{comp}}$ . We start with identifying the most computationally challenging terms by rewriting the expression of the gradient of  $\mathsf{L}_{\exp}^{\mathrm{comp}}$  in the following lemma.
Lemma 19. For any  $\mathbf{w} \in \mathbb{R}^d$ , the gradient of  $\mathsf{L}_{\mathrm{exp}}^{\mathrm{comp}}$  can be expressed as follows:
$$
\nabla L _ {\mathrm {e x p}} ^ {\mathrm {c o m p}} (\mathbf {w}) = \sum_ {s = 1} ^ {l} \sum_ {\mathbf {z} \in \Delta^ {p}} \left[ N _ {\mathbf {w}} Q _ {\mathbf {w}} ^ {\prime} (\mathbf {z}, s) - Z _ {\mathbf {w}} C _ {\mathbf {w}} (\mathbf {z}, s) \right] \widetilde {\psi} (x _ {i}, \mathbf {z}, s),
$$
where  $\mathsf{Q}_{\mathbf{w}}^{\prime}(\mathbf{z},s) = \sum_{y:y_{s - p + 1}^{s} = \mathbf{z}}e^{\mathbf{w}\cdot \Psi (x_{i},y)}$ $C_{\mathbf{w}}(\mathbf{z},s) = \sum_{y:y_{s - p + 1}^{s} = \mathbf{z}}\overline{\ell} (y,y_{i})e^{-\mathbf{w}\cdot \Psi (x_{i},y)}$  and  $\mathsf{N}_{\mathbf{w}} = \sum_{y\in \Delta^l}\overline{\ell} (y,y_i)e^{-\mathbf{w}\cdot \Psi (x_i,y)}$
$$
\begin{array}{l} \sum_ {y \in \Delta^ {l}} e ^ {\mathbf {w} \cdot \Psi (x _ {i}, y)} \Psi (x _ {i}, y) = \sum_ {y \in \Delta^ {l}} e ^ {\mathbf {w} \cdot \Psi (x _ {i}, y)} \sum_ {s = 1} ^ {l} \widetilde {\psi} (x _ {i}, y _ {s - p + 1} ^ {s}, s) \\ = \sum_ {s = 1} ^ {l} \sum_ {\mathbf {z} \in \Delta^ {p}} \left[ \sum_ {y: y _ {s - p + 1} ^ {s} = \mathbf {z}} e ^ {\mathbf {w} \cdot \Psi (x _ {i}, y)} \right] \widetilde {\psi} (x _ {i}, \mathbf {z}, s) \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {y \in \Delta^ {l}} \bar {\ell} (y, y _ {i}) e ^ {- \mathbf {w} \cdot \Psi (x _ {i}, y)} \Psi (x _ {i}, y) = \sum_ {y \in \Delta^ {l}} \bar {\ell} (y, y _ {i}) e ^ {- \mathbf {w} \cdot \Psi (x _ {i}, y)} \sum_ {s = 1} ^ {l} \widetilde {\psi} (x _ {i}, y _ {s - p + 1} ^ {s}, s) \\ = \sum_ {s = 1} ^ {l} \sum_ {\mathbf {z} \in \Delta^ {p}} \left[ \sum_ {y: y _ {s - p + 1} ^ {s} = \mathbf {z}} \bar {\ell} (y, y _ {i}) e ^ {- \mathbf {w} \cdot \Psi (x _ {i}, y)} \right] \widetilde {\psi} (x _ {i}, \mathbf {z}, s). \\ \end{array}
$$
It was shown by Cortes et al. [2016, 2018] that all of the quantities  $\mathsf{Q}_{\mathbf{w}}^{\prime}(\mathbf{z}, s)$  for  $\mathbf{z} \in \Delta^{p}$  and  $s \in [l]$  and  $Z_{\mathbf{w}}$  can be computed efficiently in time  $O(lr^{p})$ , where  $r = |\Delta|$ . Thus, the remaining bottleneck in the gradient computation suggested by Lemma 19 is the evaluation of the quantities  $C_{\mathbf{w}}(\mathbf{z}, s)$  for  $\mathbf{z} \in \Delta^{p}$  and  $s \in [l]$  and  $N_{\mathbf{w}}$ . As with the loss function  $L_{\log}^{\mathrm{comp}}$  discussed in the previous section, we will analyze the computation of these quantities first in the case of rational losses, next in that of Markovian loss.
Rational losses. We will adopt the same notation as in the case of the  $\mathsf{L}_{\log}^{\mathrm{comp}}$  loss with the same definition of a rational loss:  $\ell$  is a rational loss if there exists a WFST over the  $(+, \times)$  semiring over the reals with  $\Delta$  as both the input and output alphabet such that for all  $y, y' \in \Delta^{*}$ , we have  $\overline{\ell}(y, y') = \mathcal{U}(y, y')$ .
Our algorithm is also somewhat similar to the one described for the  $\mathsf{L}_{\log}^{\mathrm{comp}}$  loss or that of Cortes et al. [2018] for the computation of the gradient of the VCRF loss function. There are, however, several differences here too because the quantities computed and thus the automata operations required are distinct.
Exactly as in the case of  $\mathsf{L}_{\log}^{\mathrm{comp}}$  loss, we first define a deterministic WFA  $\mathcal{M}$  over the  $(+, \times)$  semiring that can be computed in polynomial time and that admits a unique path labeled with any sequence
$y \in \Delta^l$ , whose weight is  $\mathcal{U}(y, y_i)$ . Next, as in [Cortes et al., 2018], we define a deterministic WFA  $\mathcal{A}$  such that
$$
\mathcal {A} (y) = e ^ {- \mathbf {w} \cdot \Psi (x _ {i}, y)} = \prod_ {s = 1} ^ {l} e ^ {- \mathbf {w} \cdot \widetilde {\psi} (x _ {i}, y _ {s - p + 1} ^ {s}, s)}.
$$
The set of states  $\mathsf{Q}_{\mathcal{A}}$  of  $\mathcal{A}$  are defined as  $\mathsf{Q}_{\mathcal{A}} = \left\{(y_{s - p + 1}^{s}, s): y \in \Delta^{l}, s = 0, \dots, l\right\}$ , with  $\mathsf{l}_{\mathcal{A}} = (\varepsilon, 0)$  its single initial state,  $\mathcal{F}_{\mathcal{A}} = \{(y_{l - p + 1}^{l}, l): y \in \Delta^{l}\}$  its set of final states, and with a transition from state  $(y_{s - p + 1}^{s - 1}, s - 1)$  to state  $(y_{s - p + 2}^{s - 1}a, s)$  with label  $a$  and weight, that is, the following set of transitions:
$$
\mathsf {E} _ {\mathcal {A}} = \Big \{\Big (\big (y _ {s - p + 1} ^ {s - 1}, s - 1 \big), a, e ^ {- \mathbf {w} \cdot \widetilde {\psi} \big (x _ {i}, y _ {s - p + 1} ^ {s - 1}   a, s \big)}, \big (y _ {s - p + 2} ^ {s - 1}   a, s \big) \Big): y \in \Delta^ {l}, a \in \Delta , s \in [ l ] \Big \}.
$$
Then, by definition of composition or intersection [Mohri, 2009], the WFA  $(\mathcal{M} \circ \mathcal{A})$  is deterministic and admits a unique path labeled with any given  $y \in \Delta^l$  whose weight is  $(\mathcal{M} \circ \mathcal{A})(y) = \mathcal{M}(y) \cdot \mathcal{A}(y) = \overline{\ell}(y, y_i)e^{-\mathbf{w} \cdot \Psi(x_i, y)}$ .
Now,  $\mathsf{N}_{\mathbf{w}}$  coincides with the sum of the weights of all accepted paths in this WFA. Thus, since  $(\mathcal{M}\circ \mathcal{A})$  is acyclic, it can be computed in time linear in the size of  $(\mathcal{M}\circ \mathcal{A})$ , that is its number of transitions. For any  $s\in [l]$  and  $\mathbf{z}\in \Delta^p$ ,  $\mathsf{C}_{\mathbf{w}}(\mathbf{z},s)$  is the sum of the weights of all paths in  $(\mathcal{M}\circ \mathcal{A})$  labeled with a sequence  $y$  admitting  $\mathbf{z}$  as a substring ending at position  $s$ . The states in the composition  $(\mathcal{M}\circ \mathcal{A})$  help us compute
Figure 5: Illustration of the WFA  $\mathcal{A}$  for  $\Delta = \{a, b\}$ ,  $p = 2$  and  $l = 2$ .
$\mathsf{C}_{\mathbf{w}}(\mathbf{z},s)$ . As in the case of the  $\mathsf{L}_{\log}^{\mathrm{comp}}$  loss, for any  $\mathbf{z}\in \Delta^{p}$  and  $s\in [l]$ , we define  $\mathsf{E}(\mathbf{z},s)$  as the set of transitions of  $(\mathcal{M}\circ \mathcal{A})$  constructed by pairing a transition  $(q_{\mathcal{M}},z_p,\omega_{\mathcal{M}},q_{\mathcal{M}}^{\prime})$  in  $\mathcal{M}$  with the transition  $((\mathbf{z}_1^{p - 1},s - 1),z_p,\omega (\mathbf{z},s),(\mathbf{z}_2^p,s))$  in  $\mathcal{A}$ . They admit the following form:
$$
\mathsf {E} (\mathbf {z}, s) = \left\{\left(\left(q _ {\mathcal {M}}, q _ {\mathcal {A}}\right), z _ {p}, \omega_ {\mathcal {M}} \cdot \omega (\mathbf {z}, s), \left(q _ {\mathcal {M}} ^ {\prime}, q _ {\mathcal {A}} ^ {\prime}\right)\right) \in \mathsf {E} _ {\mathcal {M} \circ \mathcal {A}}: q _ {\mathcal {A}} = \left(\mathbf {z} _ {1} ^ {p - 1}, s - 1\right) \right\}. \tag {37}
$$
The WFA  $(\mathcal{M} \circ \mathcal{A})$  is deterministic as a composition of two deterministic WFAs. Thus, there is a unique path labeled with a sequence  $y \in \Delta^l$  in  $(\mathcal{M} \circ \mathcal{A})$  and  $y$  admits the substring  $\mathbf{z}$  ending at position  $s$  iff that path goes through a transition in  $\mathsf{E}(\mathbf{z}, s)$  when reaching position  $s$ . Therefore, to compute  $\mathsf{C}_{\mathbf{w}}(\mathbf{z}, s)$ , it suffices for us to compute the sum of the weights of all paths in  $\mathsf{C}_{\mathbf{w}}(\mathbf{z}, s)$  going through a transition in  $\mathsf{E}(\mathbf{z}, s)$ . This can be done straightforwardly using the forward-backward algorithm or two single-source shortest-distance algorithm over the  $(+, \times)$  semiring [Mohri, 2002a], one from the initial state, the other one from the final states. Since  $(\mathcal{M} \circ \mathcal{A})$  is acyclic and admits  $O(l|\Delta|^p)$  transitions, we can compute all the quantities  $\mathsf{C}_{\mathbf{w}}(\mathbf{z}, s)$ ,  $s \in [l]$  and  $\mathbf{z} \in \Delta^p$ , in time  $O(l|\Delta|^p)$ .
Markovian loss. Here, we adopt the Markovian assumption and assume that  $\bar{\ell}$  can be decomposed as follows for all  $y,y^{\prime}\in \Delta^{l}$ :  $\bar{\ell} (y,y^{\prime}) = \prod_{t = 1}^{l}\bar{\ell}_{t}(y_{t - p + 1}^{t},y^{\prime})$ . Thus, the quantity  $\mathsf{C}_{\mathbf{w}}(\mathbf{z},s)$  can be written as:
$$
\begin{array}{l} C_{\mathbf{w}}(\mathbf{z},s) = \sum_{y:y_{s - p + 1}^{s}}\prod_{t = 1}^{l}\overline{\ell}_{t}(y_{t - p + 1}^{t},y_{i})e^{-\mathbf{w}\cdot \sum_{k = 1}^{l}\widetilde{\psi} (x_{i},y_{k - p + 1}^{k},k)} \\ = \sum_ {y: y _ {s - p + 1} ^ {s} = \mathbf {z}} \prod_ {t = 1} ^ {l} \overline {{\ell}} _ {t} (y _ {t - p + 1} ^ {t}, y _ {i}) \prod_ {k = 1} ^ {l} e ^ {- \mathbf {w} \cdot \widetilde {\psi} (x _ {i}, y _ {k - p + 1} ^ {k}, k)} \\ = \sum_ {y: y _ {s - p + 1} ^ {s} = \mathbf {z}} \prod_ {t = 1} ^ {l} \bar {\ell} _ {t} (y _ {t - p + 1} ^ {t}, y _ {i}) e ^ {- \mathbf {w} \cdot \widetilde {\psi} (x _ {i}, y _ {t - p + 1} ^ {t}, t)}. \\ \end{array}
$$
Then, we can proceed as in the Markovian loss case for the loss function  $\mathsf{L}_{\log}^{\mathrm{comp}}$  except that instead of the WFA  $\mathcal{A}$  used there, we define here a similar WFA  $\mathcal{A}'$ . The only difference is that the weight  $\omega(y_{t-p+1}^{t-1}b,t) = \overline{\ell}_t(y_{t-p+1}^{t-1}b,y_i)$  for the WFA  $\mathcal{A}$  is replaced with  $\omega'(y_{t-p+1}^{t-1}b,t) = \overline{\ell}_t(y_{t-p+1}^{t-1}b,y_i)e^{-\mathbf{w}\cdot\widetilde{\psi}(x_i,y_{t-p+1}^{t-1}b,t)}$ . With the same argument, we can compute all quantities  $C_{\mathbf{w}}(\mathbf{z},s)$ ,  $s \in [l]$  and  $\mathbf{z} \in \Delta^p$ , in time  $O(lr^p)$ . The quantity  $N_{\mathbf{w}}$  can also be efficiently computed in time  $O(lr^p)$  since it is the sum of the weights of all paths in  $\mathcal{A}'$ .
G.3 Efficient Inference
We focused on the problem of efficient computation of the gradient. Inference is also a key problem in structured prediction since the label with a highest score must be determined out of an exponentially large set of possible ones. However, for the linear hypotheses considered in the previous sections, this problem can be efficiently tackled since it can be cast as a shortest-distance problem in a directed acyclic graph, as in [Cortes, Kuznetsov, Mohri, and Yang, 2016].
More generally, an efficient gradient computation, efficient inference and other related algorithms can benefit from standard weighted automata and transducer optimization algorithms such as  $\epsilon$ -removal [Mohri, 2000, 2002b] and determinization [Mohri, 1997, Mohri and Riley, 1997, Allauzen and Mohri, 2003, 2004] (see also the survey chapter [Mohri, 2009]).