SlowGuess/ACL_2020_OCR huggingface dataset:
$$
\begin{array}{l} \left[ \begin{array}{c c c c c c c c c c c c c c c} \ddots & \backslash^ {\prime} & \backslash^ {\prime} & , & , & , & , - & ?, & ! & , \hat {r} - & \backslash \\ \hline \end{array} \right], \\ - 1 r b -, - r r b -, - l c b -, - r c b -, (), [ ], \\ \{\}, \{\}, <  , >, -, \dots , ^ {\prime} ]. \end{array}
$$
6. Delete all tokens with tag “-NONE-”.
7. Add a special token  $"
"$  to the end of each sentence.
8. Truncated the vocabulary at 9,999 according to the frequencies and assign all the out-of-vocabulary tokens a special token "<unk>".
9. After the above procedures, there are still minor differences to PTB-Concat. We then go through the whole Penn Treebank corpora to manually fix all the unmatched tokens.
These procedures ensure we have exactly the same training, validation and test sets as PTB-Concat, the only difference is that our datasets has
6622
additional grammar trees retained from the original PTB dataset. The resulting datasets then becomes PTB-Concat-Syn.
A.4 PRPN and PRPN-SYD
A.4.1 Parse-Read-Predict Network (PRPN)
The idea of PRPN builds upon an assumption that to predict a word  $x_{i}$ , we only need information for all precedent siblings in constituent tree. The model constitutes three components: (i) a parsing network that calculates the syntactic distance and parsing gates. (ii) a reading network to model the language, and (iii) a predict network to predict the next word.
PRPN first uses a two-layer convolutional network to calculate the syntactic distance  $d$  at timestep  $t$ :
$$
h _ {i} = \operatorname {R e L U} \left(W _ {c} \left[ \begin{array}{c} e _ {i - L} \\ e _ {i - L + 1} \\ \dots \\ e _ {i} \end{array} \right] + b _ {c}\right) \tag {20}
$$
$$
d _ {i} = \operatorname {R e L U} \left(W _ {d} h _ {i} + b _ {d}\right) \tag {21}
$$
Where  $e_{i - L},\ldots ,e_i$  are word embeddings,  $L$  is the lookback range.
Then the difference between distances is fed through hardtanh to model the degree  $\alpha_{j}^{t}$  that how much two words  $x_{t}$  and  $x_{j}$  are related:
$$
\alpha_ {j} ^ {t} = \frac {\operatorname {h a r d t a n h} \left(\left(d _ {t} - d _ {j}\right) \cdot \tau\right) + 1}{2} \tag {22}
$$
Where  $\mathrm{hardtanh}(x) = \max (-1,\min (1,x))$  and  $\tau$  is the temperature parameter.
For word  $x_{i}$ , the first precedent word  $x_{t}$  with a small value  $\alpha_{i}^{t}$  represents  $x_{t}$  and all its precedents are not likely to be siblings of  $x_{i}$ . The following parsing gate  $g_{i}^{t}$  models the probability of  $x_{t}$  and  $x_{i}$  being siblings:
$$
g _ {i} ^ {t} = \mathbf {P} \left(l _ {t} \leq i\right) = \prod_ {j = i + 1} ^ {t - 1} \alpha_ {j} ^ {t} \tag {23}
$$
The reading network is a variant of Long Short-Term Memory-Network (LSTMN) (Cheng et al., 2016) where the attention score is softly truncated by parsing gates:
$$
s _ {i} ^ {t} = \frac {g _ {i} ^ {t} \tilde {s} _ {i} ^ {t}}{\sum_ {i} g _ {i} ^ {t}} \tag {24}
$$
The predict network utilizes the structure-aware hidden states of reading network to predict the next word.
A.4.2 The PRPN-SYD model
We re-designed the parsing network. We use LSTM to encode each embedding sequence  $s = (e_0, e_1, \dots, e_n)$ . Because the task of language modeling prohibits seeing future words, we use unidirectional LSTM:
$$
h _ {0}, \dots , h _ {n} = \operatorname {L S T M} _ {w} \left(e _ {0}, \dots , e _ {n}\right) \tag {25}
$$
We stack a convolutional layer on top of the hidden states  $h_i$  of the LSTM, which helps gather local syntactic information:
$$
g _ {0}, \dots , g _ {n} = \operatorname {C O N V} \left(h _ {0}, \dots , h _ {n}\right) \tag {26}
$$
Next, syntactical information learned both locally and globally are integrated by using another unidirectional LSTM:
$$
\hat {h} _ {0}, \dots , \hat {h} _ {n} = \operatorname {L S T M} _ {d} \left(g _ {0}, \dots , g _ {n}\right) \tag {27}
$$
We pass the  $\hat{h}$  layer through two 2-layer fully-connected networks which output two respective sets of distance scalars:
$$
d _ {i} ^ {l m} = F F _ {l m} \left(\hat {h} _ {i}\right) \quad d _ {i} ^ {s y d} = F F _ {s y d} \left(\hat {h} _ {i}\right) \tag {28}
$$
Where  $d^{lm}$  is the distance for language modeling while  $d^{syd}$  is for syntactic distance prediction. For two sets of distances, we use the same objective functions as described in ONLSTM-SYD.
A.5 Trees
We visualize a set of sentences (14 sentences in total) and their corresponding trees in parallel to contrast the qualitative differences of the model induces trees and gold standard trees. Sentences are selected randomly from the dataset. In each of the following figures, we provide three trees for a same sentence, which corresponds to trees induced from the syntactic task (top) and language model task (middle) set of distances, as well as the gold-standard trees (bottom).
6623
Figure 5: Sentence 1. Trees induced from the syntactic task (top) and language model task (middle) set of distances, as well as the gold-standard trees (bottom).
Figure 6: Sentence 2. Trees induced from the syntactic task (top) and language model task (middle) set of distances, as well as the gold-standard trees (bottom).
Figure 7: Sentence 3. Trees induced from the syntactic task (top) and language model task (middle) set of distances, as well as the gold-standard trees (bottom).
6624
Figure 8: Sentence 4. Trees induced from the syntactic task (top) and language model task (middle) set of distances, as well as the gold-standard trees (bottom).
Figure 9: Sentence 5. Trees induced from the syntactic task (top) and language model task (middle) set of distances, as well as the gold-standard trees (bottom).
Figure 10: Sentence 6. Trees induced from the syntactic task (top) and language model task (middle) set of distances, as well as the gold-standard trees (bottom).
6625
Figure 11: Sentence 7. Trees induced from the syntactic task (top) and language model task (middle) set of distances, as well as the gold-standard trees (bottom).
Figure 12: Sentence 8. Trees induced from the syntactic task (top) and language model task (middle) set of distances, as well as the gold-standard trees (bottom).
Figure 13: Sentence 9. Trees induced from the syntactic task (top) and language model task (middle) set of distances, as well as the gold-standard trees (bottom).
6626
Figure 14: Sentence 10. Trees induced from the syntactic task (top) and language model task (middle) set of distances, as well as the gold-standard trees (bottom).
Figure 15: Sentence 11. Trees induced from the syntactic task (top) and language model task (middle) set of distances, as well as the gold-standard trees (bottom).
Figure 16: Sentence 12. Trees induced from the syntactic task (top) and language model task (middle) set of distances, as well as the gold-standard trees (bottom).
6627
Figure 17: Sentence 13. Trees induced from the syntactic task (top) and language model task (middle) set of distances, as well as the gold-standard trees (bottom).
Figure 18: Sentence 14. Trees induced from the syntactic task (top) and language model task (middle) set of distances, as well as the gold-standard trees (bottom).
6628