SlowGuess/NAACL_2021_OCR huggingface dataset:
1. If  $B$  is annotated with  $\bar{\mathbf{\Omega}}^-$ , then  $c'$  is replaced by  $(\varepsilon, u_1, \ldots, u_s)$ .
2. If  $B_{1}$  is annotated with  $\bar{\cdot}$ , then  $\mathbf{x}_0$  is inserted as the first symbol in the first component in  $c^{\prime}$ . After that, every occurrence of  $\mathbf{x}_i$  is replaced by  $\mathbf{x}_{i + 1}$ .
3. If the terminal was swapped during step (5), the terminal  $\sigma_{2}$  is removed from  $c^{\prime}$  and  $\sigma_{1}$  is added as the first symbol in the first component of  $c$ .
We remark that if  $B$  is annotated with  $\bar{\mathbf{\Gamma}}^-$ , then it must be the case that  $B_{1}$  is annotated with  $\bar{\mathbf{\Gamma}}^-$  as well or the terminal was swapped during step (5). Hence we do not add empty components here.
Terminating rule Let  $B \to (\sigma_2)$  be the rule at  $s$  and  $\sigma_1$  be the terminal received from the parent. We replace the rule by  $B \to (\sigma_1, \sigma_2)$  if  $B$  is annotated with  $-$  and by  $B \to (\sigma_1 \sigma_2)$  otherwise.
Passing the terminal to the child.
- If  $s$  is  $r$ , let  $\sigma$  be the terminal in  $c$ . We pass  $\sigma$  to the next node on the path to  $t$ .
- If  $s$  is neither  $r$  nor  $t$ , and there is a branching rule at  $s$ , we pass the terminal received from the parent to the next node on the path to  $t$ .
- If there is a monic rule of the form  $B \to c'(B_1)$  at  $s$ , we let  $\sigma_1$  be the terminal reveived from the parent and  $\sigma_2$  the terminal in  $c'$ . If the terminal in this rule was swapped during step (5), we pass  $\sigma_2$  to the next node on the path to  $t$ , otherwise we pass  $\sigma_1$ .
A.2 Reversal of step (4).
We recall that during step (4), certain nonterminals that occurred as the lhs of terminating rules were removed. For reverting this step, we assume that these nonterminals are predicted by an oracle. We replace every occurrence of a terminating rule of the form
-  $A \to (\sigma_1\sigma_2)$  by
$$
\begin{array}{l} A \to (\mathrm {x} _ {1} \mathrm {y} _ {1}) (A _ {1}, A _ {2}) \big (A _ {1} \to (\sigma_ {1}), A _ {2} \to (\sigma_ {2}) \big) \\ \text {a n d} \end{array}
$$
$A\to (\sigma_1,\sigma_2)$  by
$$
A \rightarrow (\mathrm {x} _ {1}, \mathrm {y} _ {1}) (A _ {1}, A _ {2}) (A _ {1} \rightarrow (\sigma_ {1}), A _ {2} \rightarrow (\sigma_ {2})),
$$
where  $A_{1}$  and  $A_{2}$  are the predicted nonterminals for  $\sigma_{1}$  and  $\sigma_{2}$ , respectively.
We replace every subderivation  $d'$  of the form  $A \to c(B)(d_1')$ , where  $\sigma$  is the terminal in  $c$  and  $A_1$  the predicted nonterminal for  $\sigma$ , as follows:
Table 4: Hyperparameters for the sequence tagger.
Parameter	supervised (small/large)	flair	bert
word embeddings	one-hot (32d/300d) + character-based (100d/100d)	fasttext + flair	top 4 bert layers
Bi-LSTM	2 layers, each 200/800 states	2 layers, each 800 states	no
linear layer	no. of supertags + no. of pos tags	same	same
dropout	0.5	0.5	0.1
loss	cross entropy (supertags + pos tags)	same	same
optimizer	Adam (β1 = 0.9, β2 = 0.999)	same	same
base learning rate (lr)	10-3	10-3	5·10-5
weight decay	0	0	0.1
lr scheduler	reduce on plateau (half lr if dev. F1 score decreases)	same as sup.	constant
batch size	32	32	32
training epochs	max. 100, or if lr < 10-4	same as sup.	5
k-best supertags	10	10	10
- if  $\sigma$  is the first symbol in  $c$ , then  $c'$  is obtained from  $c$  by replacing, for each  $i \in [\mathrm{fo}(B)]$ ,  $\mathbf{x}_i$  with  $\mathbf{y}_i$  and  $\sigma$  with  $\mathbf{x}_1$ ;  $d'$  is replaced by  $A \to c'(A_1, B)(A_1 \to (\sigma), d_1')$ ,
- otherwise,  $c'$  is obtained from  $c$  by replacing  $\sigma$  with  $y_1$ ;  $d'$  is replaced by  $A \to c'(B, A_1)(d'_1, A_1 \to (\sigma))$ .
The composition  $c'$  is constructed such that  $\llbracket c' \rrbracket_1(\sigma) = c$  in the first case and  $\llbracket c' \rrbracket_2(\sigma) = c$  in the second case.
A.3 Reversal of step (3).
We repeatedly replace every occurrence  $r$  of the form  $A + B \rightarrow c(\ldots)(\ldots)$  by
$$
A \to \operatorname {i d} _ {\mathrm {f o} (B)} (B) \left(B \to c (\dots) (\dots)\right),
$$
until there are no nonterminals of the form  $A + B$  left in the derivation. If  $r$  has a parent, then we replace the corresponding nonterminal  $A + B$  in the parent's rhs by  $A$ .
B Model parameters
Table 4 shows detailed parameters for our three reported models. The architecture of the supervised (small) models is the same as Stanojevic and Steedman (2020); Coavoux and Cohen (2019), and supervised (large) the same as Corro (2020), to allow fair comparisons.
Note, that the time needed to train the models varies heavily: As the BERT embeddings are only fine-tuned for a small amount of iterations, training the bert model took less than an hour using the Negra corpus. The bilstm model benefits from the fixed word embeddings as they are only computed once; training it took ca. an hour. Both supervised models train a lot slower, training for each of those took ca. 6 hours.