SlowGuess/ACL_2020_OCR huggingface dataset:
Wenyu Du $^{1,2}$ , Zhouhan Lin $^{3,4*}$ , Yikang Shen $^{3,4}$ , Timothy J. O'Donnell $^{3,5,6}$ , Yoshua Bengio $^{3,4}$  and Yue Zhang $^{1,2}$
$^{1}$ School of Engineering, Westlake University
$^{2}$ Institute of Advanced Technology, Westlake Institute for Advanced Study
<sup>3</sup>Mila <sup>4</sup>Université de Montréal
$^{5}$ Department of Linguistics, McGill University  $^{6}$ Canada CIFAR AI Chair
It is commonly believed that knowledge of syntactic structure should improve language modeling. However, effectively and computationally efficiently incorporating syntactic structure into neural language models has been a challenging topic. In this paper, we make use of a multi-task objective, i.e., the models simultaneously predict words as well as ground truth parse trees in a form called "syntactic distances", where information between these two separate objectives shares the same intermediate representation. Experimental results on the Penn Treebank and Chinese Treebank datasets show that when ground truth parse trees are provided as additional training signals, the model is able to achieve lower perplexity and induce trees with better quality.
It is widely believed in linguistics, cognitive science, and computational linguistics that the latent structure underlying how words combine to form sentences is best represented as a tree structure. The study of the computational mechanisms and systems of constraints that characterize such derivations or parse trees is a central question in these fields (Pollard and Sag, 1994; Steedman and Baldridge, 2011; Huddleston and Pullum, 2002; Adger, 2003; Bresnan, 2001; Chomsky, 1995; Sag et al., 2003).
Using syntactic information for the language modeling task has been a popular research topic since the 1990s. Early efforts included various approaches that attempted to incorporate shallow syntactic information such as POS tags (Heeman and Allen, 1997; Srinivas, 1996) as well as a more complete structures (Wright et al., 1994; Jurafsky et al., 1995). Most of such work falls under the topic of structured language modeling (Chelba and
Jelinek, 2000; Van Uytsel et al., 2001; Xu et al., 2002). With the resurgence of neural network approaches, sequential, large-scale neural language models have been shown to significantly outperform traditional language models (Merit et al., 2017; Yang et al., 2018) without using syntactic structural information. On another scenario, recent analysis also reveals that state-of-the-art sequential neural language models still fail to learn certain long-range syntactic dependencies (Kuncoro et al., 2018). Thus it is an interesting problem to explore the relation between language models and syntax and investigate whether syntax can be integrated to enhance neural language models.
To this end, two main lines of work have been investigated, namely transition-based and distance-based methods, respectively. The former strand of work has sought to jointly train a transition-based parser (Nivre, 2008; Zhang and Nivre, 2011; Andor et al., 2016) with a language model using a linearized structured sentence. For example, recurrent neural network grammars (RNNGs) model the joint probability of both words and trees by training a generative, top-down parser (Dyer et al., 2016; Cheng et al., 2017). Subsequent work (Kim et al., 2019b) has developed an unsupervised variant of RNNGs based on an expectation maximization algorithm, which enables the system to be used as a language model without access to parser data.
The second strand of work designs language models that are constrained using syntactic constituents induced using the notion of syntactic distance (Shen et al., 2017, 2018). The distances are a sequence of scalars between consecutive words, which are higher when there is a higher level of constituent boundary between the corresponding pair of words. While aligning nicely with the sequential nature of language models, syntactic distances can be transformed into syntactic tree structures with simple principles (Shen et al., 2017).
6611
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6611-6628 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
The major difference between the above two strands of work is that the former focuses more on parsing performance while the latter aligns better to language model settings. There are three main benefits of the syntactic distance approach. First, typical engineering tricks for language modeling such as batching and regularization (Merit et al., 2017) can be directly used. Second, unlike transition-based methods, which requires to model each sentence independently, distance-based models allow direct comparison with mainstream prior work on language modeling (Gal and Ghahramani, 2016; Merity et al., 2017; Yang et al., 2018) on the same datasets, which carry information across sentence boundaries. Third, there is no risk of compounding errors as compared to the transition-based approach. However, unlike for transition-based approaches (Kim et al., 2019b), for distance-based approaches there have been no studies examining the relationship between induced syntactic structure and human labeled syntactic structure, or whether human labeled syntactic trees can be used to improve language modeling (Dyer et al., 2016; Kim et al., 2019b).
To this end, we investigate distance-based language models with explicit supervision. In particular, we inject syntactic tree supervision into distance-based neural language models by breaking a syntactic tree into a label sequence, and extending a distance-based language model to include a multi-task objective that also learns to predict gold-standard labels. We choose the Ordered-Neuron LSTM (ON-LSTM) (Shen et al., 2018) as our baseline model, which gives the best results among distance-based models.
For making fair comparison with the dominant methods on language modeling, we also manually extend the most commonly-used dataset for evaluating language models, which we name PTB-Concat (Mikolov et al., 2010). It is a version of the Penn Treebank (PTB) (Marcus et al., 1993) dataset with syntactic trees removed, and with preprocessing of numbers, punctuation and singleton words. We add syntactic trees, thus directly compare distance-based methods with other language models.
Experimental results show that incorporating linguistically motivated structures could practically improve language modeling performance. To the best of our knowledge, this is the first work to successfully incorporate gold-standard syntactic trees into syntactic distance based language models. Ad
ditional experiments suggest that the level of improvement could also be achieved in other language models. Furthermore, analyses of the trees learned by the multi-task models demonstrate that they are different from both gold trees and unsupervisedly learned trees. $^{1}$
Using syntactic information for language modeling dates back to the last century. Srinivas (1996) proposed using shallow syntactic structures—so-called "super-tags"—which successfully reduced perplexity by  $38\%$  over a tri-gram based word-level language model. More complete parser integration is also explored under the heading of "structured language modeling" (Chelba and Jelinek, 2000). This research covers a wide range of different parsers, albeit mostly with N-gram models (Van Uytsel et al., 2001; Xu et al., 2002). Wright et al. (1994) and Jurafsky et al. (1995) extend bi-gram language models with a context-free grammar. Feed-forward neural language models were also explored (Xu et al., 2003). However, the performance does not approach that of the modern neural LMs.
Dyer et al. (2016) first proposed RNNG. Subsequent work extends the model with an encoder-decoder architecture (Cheng et al., 2017), unsupervised learning (Kim et al., 2019b), knowledge-distillation (Kuncoro et al., 2019) and computational psycholinguistics (Hale et al., 2018). Shen et al. (2017) first used syntactic distance to constrain language modeling. Its subsequent work (Shen et al., 2018) transfers the distance notion to LSTM cell. Our work extends distance-based methods in trying to introduce supervised syntax to these models. A very recent work makes use of attention over spans instead of syntactic distance to inject inductive bias to language models (Peng et al., 2019). However, the time complexity of injecting supervision is much higher than distance-based approach  $(\mathcal{O}(n^2)\mathrm{VS}\mathcal{O}(n))$
The overall structure of our model is shown in Figure 1. In particular, the ON-LSTM is taken as the base language model, and syntactic trees are added by conversion to distance metrics. The supervised distance values are taken as one additional output, resulting in a multi-view model.
<sup>1</sup>We release the code at https://github.com/wenyudu/SDLM.
6612
Figure 1: Split-head approach of constructing the two master forget gates in the multi-task setting.
3.1 Ordered Neurons LSTM
Ordered Neurons LSTM (ON-LSTM) (Shen et al., 2018) is built upon a vanilla LSTM model (Hochreiter and Schmidhuber, 1997) with two additional gates, namely a master input gate  $\tilde{i}_t$  and a master forget gate  $\tilde{f}_t$ , each being a vector of the same shape as the LSTM forget and input gates:
$$
f _ {t} = \sigma \left(W _ {f} \circ \left[ x _ {t}, h _ {t - 1} \right] + b _ {f}\right) \tag {1}
$$
$$
i _ {t} = \sigma \left(W _ {i} \circ \left[ x _ {t}, h _ {t - 1} \right] + b _ {i}\right) \tag {2}
$$
$$
o _ {t} = \sigma \left(W _ {o} \circ \left[ x _ {t}, h _ {t - 1} \right] + b _ {o}\right) \tag {3}
$$
$$
\hat {c} _ {t} = \tanh  \left(W _ {c} \circ \left[ x _ {t}, h _ {t - 1} \right] + b _ {c}\right) \tag {4}
$$
$$
\tilde {f} _ {t} = \operatorname {c u m a x} \left(W _ {\tilde {f}} \circ \left[ x _ {t}, h _ {t - 1} \right] + b _ {\tilde {f}}\right) \tag {5}
$$
$$
\tilde {i} _ {t} = 1 - \operatorname {c u m a x} \left(W _ {\tilde {i}} \circ [ x _ {t}, h _ {t - 1} ] + b _ {\tilde {i}}\right) \tag {6}
$$
where  $\text{cumax}$  is defined as the cumulative sum of softmax outputs, i.e.,  $\text{cumax}(\cdot) = \text{cumsum}(\text{softmax}(\cdot))$ . The cumax function provides an inductive bias to model hierarchical structures through enforcing units in the master forget gate  $\tilde{f}_t$  to increase monotonically from 0 to 1 and those in the master input gate  $\tilde{i}_t$  to decrease monotonically from 1 to 0. The two gates are applied on the original input and forget gates as follows:
$$
\omega_ {t} = \tilde {f _ {t}} \circ \tilde {i} _ {t} \tag {7}
$$
$$
\hat {f} _ {t} = f _ {t} \circ \omega_ {t} + \left(\tilde {f} _ {t} - \omega_ {t}\right) = \tilde {f} _ {t} \circ \left(f _ {t} \circ \tilde {i} _ {t} + 1 - \tilde {i} _ {t}\right) \tag {8}
$$
$$
\hat {i} _ {t} = i _ {t} \circ \omega_ {t} + (\tilde {i} _ {t} - \omega_ {t}) = \tilde {i} _ {t} \circ (i _ {t} \circ \tilde {f} _ {t} + 1 - \tilde {f} _ {t}) \tag {9}
$$
$$
c _ {t} = \hat {f} _ {t} \circ c _ {t - 1} + \hat {i} _ {t} \circ \hat {c} _ {t} \tag {10}
$$
$$
h _ {t} = o _ {t} \circ \tanh  \left(c _ {t}\right). \tag {11}
$$
ON-LSTM can learn the implicit structure of a language in the form of a binary tree in an unsupervised manner, through syntactic distances, which are calculated as:
$$
d _ {t} = D _ {m} - \sum_ {k = 1} ^ {D _ {m}} \tilde {f} _ {t} \tag {12}
$$
Figure 2: Binarized grammar tree and its corresponding syntactic distances. The heights of the bars stand for the values of the distances. To convert this tree to syntactic distances, we first assign all the words an initial value of 1, and then the non-leaf nodes are assigned distances in the order of  $d_{3} \to d_{2} \to d_{1} \to d_{4}$ , according to the procedures in the second part of Model section. On the other hand, given the distances, the tree can be recovered in a top-down process by setting up the split boundaries in descending order of distances (i.e.,  $d_{4} \to d_{1} \to d_{2} \to d_{3}$ ). Syntactically, a shorter distance between a pair of words indicates a closer relationship between the constituents on the two sides of the distance. Note that since only the relative order of the distances could affect the structure of the trees, valid values of these distances are not unique.
where  $D_{m}$  is the size of the hidden state. The syntactic distance  $d_{t}$  between two consecutive words is a scalar value, which can be interpreted as reflecting the syntactic relatedness between the constituents before and after time point  $t$ . In terms of trees, it can be thought of as the height the lowest tree node that encloses both words. In the case where we consider discrete trees, the height is given by the maximum path length from a leaf. In the more general case, it can be thought of as a scalar value measuring a continuous notion of node height. Figure 2 depicts a sample sentence with its syntactic distances and corresponding tree structures. More generally, the binary tree structure of a sequence with  $N$  tokens can be specified with a sequence of  $N - 1$  syntactic distances. This definition of distance makes the syntactic distance an ultrametric (Holly, 2001; Wu et al., 1999), a concept which is important in the theory of hierarchical agglomerative clustering (Johnson, 1967) and was first explored in a linguistic setting by Levelt (1974).
3.2 Converting Grammar Trees to Syntactic Distances
To integrate treebank trees into ON-LSTM, we need to first convert syntactic trees into a representation based on syntactic distances. Since the original grammar trees are not necessarily binary,
6613
we first split non-binary nodes by adding sentinel intermediate nodes to form a right-branched binary tree, following the steps in Stern et al. (2017). Now for a binary tree with  $N$  leaf nodes, we have  $N - 1$  non-leaf nodes that correspond to the  $N - 1$  slots between each of the adjacent word pairs, each of which are assigned a syntactic distance (Figure 2). The binary tree can thus be represented as a sequence of distances  $d_{1}, d_{2}, \ldots, d_{N - 1}$ .
The conversion from binary tree to syntactic distances thus translates to the assigning of a distance value for each of the  $N - 1$  non-leaf nodes in the tree. This is achieved in a bottom-up process. We first initialize a distance value of 1 at all of the leaf nodes, and then compute the syntactic distances of the parent nodes by recursively tracing back their parents. More specifically, for a certain parent node, its corresponding syntactic distance  $d_{P}$  is computed with respect to the syntactic distances of its children  $d_{L}$  and  $d_{R}$ , i.e.,
$$
d _ {P} = \max  \left\{d _ {L}, d _ {R} \right\} + 1. \tag {13}
$$
A more detailed algorithm flowchart of tree-to-distance conversion is given in Appendix A.1.
3.3 Auxiliary Syntactic Distance Outputs
In ON-LSTM the distances  $d_t$ 's in Equation 12 are used to infer the structure of grammar trees. Consequently, a straight-forward way to incorporate ground truth parse trees is to use the ground truth distances  $d_t^g$  to guide  $d_t$ , as depicted in Figure 1. Interestingly, directly forcing the structure inferred by language models to be coherent to linguist-tagged ground truth trees barely improves the language model performance (see Section 6). Instead, we introduce a "split-head" setting, which can practically improve LM performances by learning two sets of closely related syntactic distances.
In particular, we use another master forget gate  $\tilde{f}_t^w$  for inferring a set of distances that are trained to align with the gold-standard syntactic distances, while leaving the original distances  $d_t$  computed from  $\tilde{f}_t$  intact. To achieve this, we introduce an extra linear layer on top of the hidden states  $h_t^f$ , and from there infer a separate set of master forget gates. In this way, both of the master forget gates  $\tilde{f}_t$  and  $\tilde{f}_t^w$  share the same input  $h_t^f$ , but optimize two different sets of trees for the language modeling
and parsing task, respectively. i.e.,
$$
h _ {t} ^ {f} = W _ {\tilde {f}} \circ \left[ x _ {t}, h _ {t - 1} \right] + b _ {\tilde {f}} \tag {14}
$$
$$
\tilde {f} _ {t} = \operatorname {c u m a x} \left(h _ {t} ^ {f}\right) \tag {15}
$$
$$
\tilde {f} _ {t} ^ {w} = \operatorname {c u m a x} \left(W _ {s} \left(h _ {t} ^ {f}\right) + b _ {s}\right) \tag {16}
$$
The syntactic distances for the auxiliary supervised targets are then calculated as follows:
$$
d _ {t} ^ {w} = D _ {m} - \sum_ {k = 1} ^ {D _ {m}} \tilde {f} _ {t k} ^ {w} \tag {17}
$$
where  $\tilde{f}_{tk}^{w}$  is the  $k$ -th element in the vector  $\tilde{f}_t^w$
3.4 Grammar Trees as Auxiliary Supervised Targets for Language Modeling
With the additional master forget gate  $\tilde{f}_t^w$ , the model has two different sets of predictions. The first set is the language model outputs of ON-LSTM, predicting the next words. The second set is the distances calculated in Equation 17. The original language modeling structure of the ON-LSTM model is left intact after the modification, so we can continue to use the master forget gate  $\tilde{f}_t$  to update hidden states and calculate the softmax output in ON-LSTM for the language modeling part. We denote the negative log-likelihood loss in the language model part as  $L_{\mathrm{lm}}$ . For brevity, we do not discuss the details of the loss.
For aligning the syntactic distances, we perform a ranking loss between the learned syntactic distance  $d_t^w$  and ground truth distance  $d^g$ , which was first proposed by Burges et al. (2005). The goal is to encourage the model to produce the distances that have the same ranking order as the ground truth distances:
$$
L _ {\mathrm {s y d}} = \sum_ {i, j > i} \max  \left(0, \left(1 - \operatorname {s i g n} \left(d _ {i} ^ {g} - d _ {j} ^ {g}\right) \left(d _ {i} ^ {w} - d _ {j} ^ {w}\right)\right)\right). \tag {18}
$$
The joint objective function is thus to minimize the following loss:
$$
L = L _ {\mathrm {l m}} + \alpha L _ {\mathrm {s y d}} \tag {19}
$$
where  $\alpha$  is the scaling parameter.
We make test datasets in English and Chinese, respectively, both of which have parse trees and also language modeling benchmarks. For English, we use the Penn Treebank (PTB) dataset (Marcus
6614
et al., 1993). Mikolov et al. (2010) have provided a widely accepted version of PTB for language modeling. Several modifications are made to the original treebank. For example, all punctuation symbols are removed, all characters are lower-cased, the vocabulary size is truncated at 10,000 and all sentences are concatenated. However, this version of PTB discards the parse tree structures, which makes it unsuitable for comparing sequential language models with those utilizing tree structures. We refer to this version as PTB-Concat.
Dyer et al. (2016) proposed a different version of PTB, which retains the parse tree structures. Sentences are modeled separately, punctuation is retained, and singleton words are replaced with the Berkeley parser's mapping rules, resulting in much larger vocabulary size, 23,815-word types. Since it retains the parse trees, this dataset enables direct comparison between models that utilize parse trees with those who do not. But unfortunately, since the vocabulary is different from PTB-Concat, and the sentences are processed separately, the results are not directly comparable with those in PTB-Concat, on which most existing work on language modeling reports results. We refer to this version as PTB-Sepsent.
As mentioned above, a salient limitation of PTB-Sepsent is that it does not allow fair comparison with existing LM work on PTB-Concat. To address this issue, we propose a different variation of PTB dataset that both uses the same vocabulary size as PTB-Concat and at the same time retaining the ground-truth grammar trees. We pre-process the PTB dataset by following the same steps indicated by Mikolov et al. (2010) to obtain a modified treebank with the same vocabulary set as PTB-Concat. Sentences are concatenated, and we make sure that the sentences are the same to PTB-Concat, from token to token, in the training, validation, and test sets. This results in the same vocabulary as that of PTB-Concat, which allows us to directly compare models that utilize parse trees with the existing reports of performance on PTB-Concat. We refer to this version of PTB-Concat with syntax as PTB-Concat-Syn and we will cover preprocessing details in Appendix A.3.
For Chinese, we use the Chinese Treebank 5.1 (Xue et al., 2005), with the same settings as Kim et al. (2019b). Sentences are modeled separately and singleton words are replaced with a single <UNK> token. It will be referred to as CTB
Model	Param	Dev	Test
Gal and Ghahramani (2016) - Variational LSTM	66M	-	73.4
Kim et al. (2016) - CharCNN	19M	-	78.9
Merit et al. (2016) - Pointer Sentinel-LSTM	21M	72.4	70.9
Grave et al. (2016) - LSTM	-	-	82.3
Zoph and Le (2016) - NAS Cell	54M	-	62.4
Zilly et al. (2017) - Variational RHN	23M	67.9	65.4
Shen et al. (2017) - PRPN	-	-	62.0
Merit et al. (2017) - 3-layer AWD-LSTM	24M	60.0	57.3
Zolna et al. (2018) - Fraternal dropout	24M	58.9	56.8
Shen et al. (2018) - 3-layer ON-LSTM	25M	58.3	56.2
ONLSTM-SYD	25M	57.8	55.7
Yang et al. (2018) - AWD-LSTM-MoS	22M	56.5	54.4
Takase et al. (2018) - AWD-LSTM-DOC	23M	54.1	52.4
Table 1: Various language models evaluated on validation and test sets on PTB-Concat. Our model is denoted as ONLSTM-SYD, which incorporates tree structures during training. Yang et al. (2018) and Takase et al. (2018) focus on improving the softmax module of LSTM LM, which are orthogonal to ours.
Sepsent in the rest of the paper.
We evaluate the influence of syntactic supervision on distance-based language models, especially in terms of its language modeling performance. We are also going to analyze the induced syntax after introducing the structural supervision. In addition, extensive ablation tests are conducted to understand how syntactic supervision affects the language model.
5.1 Language Modeling
We first compare our models with existing sequential language models on PTB-Concat, and then we compare our model with transition-based language models on PTB-Sepsent and CTB-Sepsent, which have a larger vocabulary and also use additional grammatical structure.
Results on PTB-Concat We first validate the benefit of introducing structural signal to neural language models by training our proposed model on PTB-Concat-Syn with structural supervision, and then evaluate them on the plain validation/test set. We compare our model with the original ON-LSTM model, as well as various other strong LSTM language model baselines such as AWD-LSTM (Merit et al., 2017) and a mixture of softmax (Yang et al., 2018). We denote our syntactic-distance-augmented ON-LSTM model as ONLSTM-SYD.
For making fair comparison, we closely follow the hyperparameters and regularization of ONLSTM (Shen et al., 2018). The model is a three-layer ONLSTM-SYD language model with an embedding size of 400 and hidden layer units 1150. The dropout rates are 0.5, 0.45, 0.3, 0.45 for the
6615
Model	PTB-Sepsent	CTB-Sepsent
Kim et al. (2019b) - RNNLM	93.2	201.3
Kim et al. (2019b) - RNNG	88.7	193.1
Kim et al. (2019b) - URNNG	90.6	195.7
Kim et al. (2019b) - RNNG-URNNG	85.9	181.1
Kim et al. (2019b) - PRPN (default)	126.2	290.9
Kim et al. (2019b) - PRPN (finetuned)	96.7	216.0
ONLSTM-noAWD	69.0	167.7
ONLSTM	60.0	145.7
ONLSTM-SYD-noAWD	67.6	163.1
ONLSTM-SYD	59.6	140.5
Table 2: Language modeling perplexity on PTB-Sepsent and CTB-Sepsent. Kim et al. (2019b) report two results of PRPN, the default one using settings in Shen et al. (2017) and another one finetuned by themselves. Our models use the same hyperparameter settings as in Section 5.1.
word vectors, LSTM weight metrics, outputs between LSTM layers and the output of the last layer, respectively. The embedding dropout ratio is 0.125. The model is trained and finetuned for 1000 epochs in total and is switched to the fine-tuning phase at epoch 650. The ground truth syntactic structures are used to supervise the syntactic distances in the third layer of ONLSTM-SYD and the loss ratio  $\alpha$  is set to 0.75. We use this setting as the default setting for all the experiments.