SlowGuess/NAACL_2021_OCR huggingface dataset:
Mila/Université de Montréal
yi-kang.shen@umontreal.ca
tanjings@mila.quebec
Mila/McGill University
Syntax is fundamental to our thinking about language. Failing to capture the structure of input language could lead to generalization problems and over-parametrization. In the present work, we propose a new syntax-aware language model: Syntactic Ordered Memory (SOM). The model explicitly models the structure with an incremental parser and maintains the conditional probability setting of a standard language model (left-to-right). To train the incremental parser and avoid exposure bias, we also propose a novel dynamic oracle, so that SOM is more robust to wrong parsing decisions. Experiments show that SOM can achieve strong results in language modeling, incremental parsing and syntactic generalization tests, while using fewer parameters than other models.
Several recent works have systematically studied the linguistic abilities of modern language models, particularly syntax (Linzen et al., 2016; Marvin and Linzen, 2018; Gulordava et al., 2018). They find that most language models are good at capturing frequent syntactic structures but do not generalize well to those in the long tail. Moreover, although some excel at having low perplexity scores, this is less due to their syntactic ability but more due to capturing collocations (frequently co-occurring words). Recently, Hu et al. (2020) show that RNNs underperform on a syntactic generalization (SG) test set, whereas models that have an explicit notion of syntax, such as RNNG (Dyer et al., 2016), fare well on SG but at the cost of generally poorer language modeling (higher perplexity). Transformer-based models achieve strong performance when trained with large datasets, but are worse than random when trained on a small dataset.
These works showed that building language models with an explicit internal model of syntax
Figure 1: The mechanism of SOM. "Context" is a distributed representation of previous sentences. It could also represent the source sentence in a sequence to sequence task. It incrementally build subtrees given the input sentences. A RNN will take the context representation and the representations of subtrees in the current sentence to predict next token.
helps in achieving better performance in SG tasks and is also thought to help learn more efficiently in low data settings. However, building syntax-aware models that also obtain strong language modeling performance, when compared with recent transformer-based models, has until now seemed elusive. In this work, we propose a new syntax-aware language model dubbed Syntactic Ordered Memory (SOM; Fig. 1), which jointly acts as a language model and an incremental parser. SOM inherits the syntax representation used in Ordered Memory (OM; Shen et al. 2019) in which syntax trees are embedded in a grid-like memory representation. Whereas OM was trained as an unsupervised parser, SOM is explicitly trained to predict both ground-truth syntax trees incrementally and, using the predicted partial syntactic structure, to predict the next token. Fig.1 shows the mechanism of SOM.
SOM factorizes the next-token prediction process into two steps: first, we predict the attachment position for the next token with a zero-step lookahead parser, trained in a supervised fashion; then, we predict the next token distribution conditioned on the partially predicted structure. One way of training the incremental parser is to use teacher-forcing. However, this can lead to exposure bias, due to the fact that the model was never exposed to its own predictions during training. To avoid
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1660-1672 June 6-11, 2021. ©2021 Association for Computational Linguistics
this, we introduce a dynamic oracle (Goldberg and Nivre, 2012) for our model, so that our model can learn to recover from previous parsing mistakes during inference. We found this to be crucial to obtain good performance.
We compare SOM with existing methods that integrate syntax into language models. RNNGs (Dyer et al., 2016) and Ordered Neurons (Shen et al., 2018) are particularly related. RNNGs are generative models of language which define a joint distribution on syntactic structures and sequence of words. Ordered Neurons attempt to model the hierarchical structure of language by defining an ordering to the hidden states and the gates that impose that structure. We show that our proposed SOM model can achieve strong language modeling, parsing and SG performance even when trained on small amounts of data.
- We introduce SOM, a new syntax-augmented language model that learns an incremental parser and use its predictions to improve language modeling.
- We propose a novel dynamic oracle that allows to reduce the exposure bias and is instrumental to achieving good downstream performance.
- We report high SG score, language modeling and incremental parsing performance for various dataset sizes. We also find that jointly learning both language modelling and parsing improves both these capabilities in the model.
Syntax-aware models There has been work to integrate syntax into our current models of language. Socher et al. (2013) used parse trees for composing sentences in order to predict sentiment over movie reviews. However, having an external parser and restriction of batched computations in that early model made the method unwieldy. Bowman et al. (2016) introduced the SPINN model, which alleviated those issues, turning sentences into a sequence of actions to be executed by a shiftreduce parser. Our SOM model is based on shiftreduce as well, because of the incremental nature of the parsing we want to achieve. RNNG (Dyer et al., 2016; Kuncoro et al., 2016) was an example of integrating syntax information for language modelling.
There is also work that attempts to learn these syntactic structures without supervision. Kim et al. (2019) later devised an unsupervised version of the RNNG, a method which produced good parsing performance. DIORA (Drozdov et al., 2019, 2020) was a method that leveraged the Inside-Outside algorithm to construct sentence embeddings for downstream tasks, with the benefit of being able to read off parse trees in the encoding process.
Swayamdipta et al. (2019) finds that there are no improvements over using ELMo (Peters et al., 2018) embeddings when shallow syntactic information is included, concluding that ELMo-style pretraining has learned the syntactic information. However, Kuncoro et al. (2019) investigated the importance of the learnt syntactic knowledge RNNG in a large pre-trained model like BERT, they found that syntax information helps with downstream tasks. In our experiments, we find that explicitly training OM with syntax (with our dynamic oracle scheme) improves performance on syntactic generalization tasks.
Incremental Parsing & Language Modelling In SOM, we specifically focus on incremental parsing. Ghezzi and Mandrioli (1979) discusses incremental parsing in the context of programming languages, with shift-reduce parsers being a specific type of incremental parsing. OM, RNNG, and SPINN are models that were designed with shiftreduce in mind.
Incremental parsing lends itself well to the task of autoregressive language modelling. Since the parser only sees the prefix of a sentence, the model can use the partial parse to make a prediction about upcoming words. Demberg et al. (2013) summarises several empirical results that provide evidence for incremental and predictive parsing in humans, and makes several connections between incrementality (that comprehenders do not wait to the end of the sentence before building a representation) and prediction about future words coming in the sentence.
Given that an incremental parser processes a sentence from left to right, there are naturally some limitations. Hassan et al. (2009) show why either a beam or delay is necessary if performing incremental parsing with monotonic extensions: They experiment with a parser based on Combinatory Categorical Grammar (Steadman, 2000). They find that without the look-ahead, there is a  $30\%$  point reduction in the parsing results. One of our con
tributions in this paper is the one-step lookahead while performing parsing, but zero-step lookahead when performing next-word prediction, allowing the model to be trained jointly as a incremental parser and language model.
Despite the left-to-right nature of incremental parsing, this setting may aid language modelling too. Shieber (1983) suggests the biases may correspond to the way humans parse English, and use a modified shift-reduce parser to disambiguate between different parses of a sentence. There have been work that show that incremental parsing can improve language modelling. Kohn and Baumann (2016) demonstrate that combining an incremental dependency parser with a language model yields improvements in perplexity. Roark (2001) presents a top-down phrase structure parser that performs beam-search to generate connected intermediate structures for every sentence prefix. This model can be used for language modeling and beats triogram models on the Penn Treebank (Marcus et al., 1994)
Dynamic Oracles Since incremental parsing requires that we break down the problem of structure prediction into sequential decisions, we are prone to exposure bias. There are techniques to address this by allowing the model to make mistakes and supervising future actions based on the state arrived at (Daumé et al., 2009). Goldberg and Nivre (2012) introduces the concept of dynamic oracles for dependency parsing. Coavoux and Crabbé (2016) uses this technique for incremental constituency parsing, but uses morphological features, and does not perform language modelling. Fried and Klein (2018) cover in further detail the related work relating to dynamic oracles and parsing. We find that using dynamic oracles for training is crucial in seeing benefits in both language modelling and incremental parsing.
Evaluating Syntactic Generalization Recent tests have been developed that attempt to probe the linguistic abilities of language models. Gulordava et al. (2018) explores the extent to which RNNs are able to model grammar, independent of the semantics of the sentence. Marvin and Linzen (2018) evaluate language models on their ability to score sentences with and without the proper subject-verb agreements over a variety of different settings.
Hu et al. (2020) expands on these ideas, and propose a suite of syntactic generalization tests
Figure 2: The grid view of a tree structure. Blue arrows represent composing children into parent. Gray arrows represent copying from previous time step. Orange slots are memories generated at the current time step. Gray slots are memories copied from previous time step.
for language models over a series of different sized datasets. They find that while GPT-2 performs well, their performance is highly dependent on the scale of the language modeling training dataset, while other models remain more robust. In this paper, we use this test suite for the evaluation.
3 Ordered Memory
We first provide useful background on Ordered Memory. Ordered Memory (OM, Shen et al. 2019) is a recurrent neural network that explicitly models recursive structure through memory writing and erasing operations. OM maps the latent syntax into a  $T \times N$  memory grid  $\tilde{M}$ , where  $T$  is the length of input sequence and  $N$  is the maximum number of memory slots. Figure 2 gives an intuition of what the grid contains. Empty blocks in the figure represent memory slots that can be discarded during inference. Ideally, the memory network should generate the  $t$ -th column of the grid  $\tilde{M}_t$  at time step  $t$ . But generating  $\tilde{M}_t$  requires the model to have access about the tree structure which is usually latent. For this reason, OM induces the latent structure through inductive biases of its reading and writing operations.
As a recurrent model, OM performs one-step look-ahead incremental parsing through maintaining three states:
- Memory  $M_{t}$ : a matrix of dimension  $N \times D$ , where each occupied slot is a distributed representation for a node spanning an subsequence in  $x_{1},..,x_{t - 1}$  conditioned on  $x_{t}$ , i.e.  $M_{t}$  represents a one-step look-ahead parser stack. It's represented by gray blocks in Figure 3.
(a) The transition from time step 4 to 5.  $①$  The one-step look-ahead parser combines  $\hat{M}_{t - 1}$  and  $M_{t - 1}$  considering on the current input  $x_{t}$ , in this example, the split point of  $\hat{M}_{t - 1}$  and  $M_{t - 1}$  is  $i = 2$ .  $②$  Current input  $x_{t}$  is written into the lower slot of new candidate memory  $\hat{M}_t^{i - 1}$ .  $③$  The rest of new candidate memories  $\hat{M}_t^{\geq i}$  are generated with bottom-up recurrent composition.
(b) Predicting the next token at time step 4.  $①$  The zero-step look-ahead parser combines  $M_{t}$  and  $\hat{M}_t$  at time step t.  $②$  The recurrent network takes the combined memory  $M_t^{\mathrm{out}}$  as input and output a hidden state  $h_t = f(w_{\leq t})$ $③$ $h_t$  is then fed into an linear layer to compute  $p(x_{t + 1}|x_{\leq t})$
Figure 3: The recurrent transition (left) and prediction network (right) of SOM. In the recurrent transition, a one-step look-ahead parser predict the syntax once  $e$  is observed and can be seen as a posterior over the syntax given the current word. The prediction network uses a zero-step look-ahead parser to predict the location of the next phrase and acts as a prior on the syntactic structure.
- Candidate memory  $\hat{M}_t$ : a matrix of dimension  $N \times D$  contains representations for all possible new nodes at time step  $t$ . At next time step  $t + 1$ , the model will decide whether or not to write these candidates into memory  $M_{t + 1}$  conditioned on  $x_{t + 1}$ . They are represented by orange blocks in Figure 3. if the model is making correct parsing decisions, then  $M_t = \tilde{M}_{t - 1}$ .
- Memory mask  $\vec{\pi}_t$ :  $\vec{\pi}_t \in \{0,1\}^N$ , where each entry indicates whether the respective slot in  $\hat{M}_t$  is occupied by a candidate, e.g., if  $\vec{\pi}_t = (0,1,1)$ , then the occupied slots are  $\hat{M}_t^{\geq 2}$ . At next time step, the model can only choose a candidate from masked slots to write into the memory  $M_{t+1}$ .
At each time step, the model takes  $[M_{t-1}, \hat{M}_{t-1}, \vec{\pi}_{t-1}]$  and word embedding  $x_t$  as inputs, returning the outputs  $[M_t, \hat{M}_t, \vec{\pi}_t]$ .
To generate the new memory  $M_{t}$ , we combine  $M_{t-1}$  and  $\hat{M}_{t-1}$  to match  $\tilde{M}_{t-1}$ . The model uses  $x_{t}$  as its query to attend on previous candidates  $\hat{M}_{t-1}$ . The attention distribution is  $p_{t}$ , which models the split point of gray blocks and orange blocks in Figure 2. Suppose  $p_{t}$  is a one-hot distribution and  $p_{t}^{i} = 1$ . The candidates  $\hat{M}_{t-1}^{\leq i}$  are written into the respective memory slot  $M_{t}^{\leq i}$ , while  $M_{t-1}^{>i}$  are copied to  $M_{t}^{>i}$ :
$$
M _ {t} ^ {\le i} = \hat {M} _ {t - 1} ^ {\le i}, \quad M _ {t} ^ {> i} = M _ {t - 1} ^ {> i} \qquad (1)
$$
We will refer to the process of generating  $M_{t}$  as a one-step look-ahead parser, since the model is using the current input  $x_{t}$  as extra information to build the partial parse for time step  $t - 1$ . To generate new candidates  $\hat{M}_{t}$ , the input embedding  $x_{t}$  is written into  $\hat{M}_{t}^{i - 1}$ , and  $\hat{M}_{t}^{\geq i}$  are computed recurrently with eq.3:
$$
\hat {M} _ {t} ^ {<   i - 1} = \emptyset , \quad \hat {M} _ {t} ^ {i - 1} = x _ {t} \tag {2}
$$
$$
\hat {M} _ {t} ^ {j} = \operatorname {c e l l} \left(M _ {t} ^ {j}, \hat {M} _ {t} ^ {j - 1}\right), \quad \forall j \geq i \tag {3}
$$
where  $\mathrm{cell}()$  is the composition function that takes its children's representations as input and output the parent's representation. The non-empty slots in candidate memory are then  $\hat{M}_t^{\geq i - 1}$ , and they can be masked by:
$$
\overrightarrow {\pi} _ {t} ^ {<   i - 1} = 0, \quad \overrightarrow {\pi} _ {t} ^ {\geq i - 1} = 1 \tag {4}
$$
In other words,  $\overrightarrow{\pi}_t^i = \sum_{j\leq i + 1}p_t^j$  , and  $\overrightarrow{\pi}_t^i$  is monotonically increasing. More details of the OM can be found in Shen et al. (2019).
4 Syntactic Ordered Memory
We propose two augmentations to OM in order to better perform language modelling and incremental parsing: a prediction network and the dynamic oracle. a) Previous language models mostly focus on predicting the next token or a missing token. In our case, we are explicitly modeling the latent structure. By predicting the structure for the next token, we exploit this latent structure for word prediction.
This helps the model better organize information for predicting next word, allowing shortcuts to be created for long-term dependencies, as shown in Fig.1. b) If the model only observes states resulting from correct past decisions at training time, it will not be prepared to recover from its own mistakes during prediction, suffering from exposure bias (Schmidt, 2019; Fried and Klein, 2018). In the experiment section, we demonstrate how this phenomenon will significantly hurt the language model performance and, to a lesser extent, also hurt the parsing performance.
4.1 Prediction Network
At time step  $t$ , the prediction network takes  $[M_t, \hat{M}_t, \vec{\pi}_t]$  as input, and produces a probability distribution over the next token  $p(w_{t+1} | w_{\leq t})$ . To do this, we need to have a temporary estimate of the local structure. We therefore need to approximate  $p_{t+1}$  with a zero-step look-ahead prediction  $p'_t$ :
$$
\alpha_ {t} ^ {i} = \frac {\mathbf {w} _ {2} ^ {A t t} \operatorname {R e L U} \left(\mathbf {W} _ {1} ^ {A t t} \hat {M} _ {t} ^ {i} + b _ {1}\right) + b _ {2}}{\sqrt {N}} \tag {5}
$$
$$
p _ {t} ^ {\prime} = \operatorname {m a s k e d \_ s o f t m a x} \left(\alpha_ {t}, \operatorname {m a s k} = \overrightarrow {\pi} _ {t}\right) \tag {6}
$$
where  $\mathbf{W}_1^{Att}$  is  $N\times N$  weight matrix,  $\mathbf{w}_2^{Att}$  is a  $N$  dimension weight vector, and  $\alpha_{t}^{i}$  is a scalar. We then sample the slot at index  $i$  from the distribution  $p_t^\prime$ .  $i$  is the zero-step look-ahead parsing decision, which means that the next phrase will be a sibling of node  $\hat{M}_t^i$ . We therefore need to predict the next token conditioned on  $\hat{M}_t^i$  and its previous contexts. So we feed memory slots  $[M_t^N,M_t^{N - 1},\dots,M_t^{i + 1},\hat{M}_t^i ]$  into a recurrent neural network:
$$
h _ {t} = \operatorname {R N N} \left(M _ {t} ^ {N}, M _ {t} ^ {N - 1},..., M _ {t} ^ {i + 1}, \hat {M} _ {t} ^ {i}\right) \tag {7}
$$
where  $h_t$  is the final hidden state of the RNN. As shown in Figure 3b, the input sequence are representations of non-overlapping subtrees spanning from  $x_1$  to  $x_t$ .  $h_t$  can therefore be seen as a distributed representation of the sequence  $w_{\leq t}$ . In the RNN, we use the same architecture as the cell function in OM to model the recurrent transition
function:
$$
\left[ \begin{array}{l} f _ {j} \\ i _ {j} \\ c _ {j} \\ u _ {j} \end{array} \right] = \mathbf {W} _ {2} ^ {\text {C e l l}} \operatorname {R e L U} \left(\mathbf {W} _ {1} ^ {\text {C e l l}} \left[ \begin{array}{l} h _ {t} ^ {j + 1} \\ M _ {j} \end{array} \right] + b _ {1}\right) + b _ {2} \tag {8}
$$
$$
h _ {t} ^ {j} = \mathrm {L N} \left(\sigma \left(f _ {j}\right) \odot h _ {t} ^ {j + 1} + \sigma \left(i _ {j}\right) \odot M _ {j} + \sigma \left(c _ {j}\right) \odot u _ {j}\right) \tag {9}
$$
where  $\sigma$  is the sigmoid function, LN is layer normalization function,  $f_{j},i_{j},c_{j}$  are controlling gates,  $c_{j}$  is cell state, and  $h_t^{N + 1}$  is a zero vector. After obtaining  $h_t$ , we can compute the distribution over the next token and the language modelling loss:
$$
p \left(w _ {t + 1} \mid w _ {\leq t}\right) = \operatorname {s o f t m a x} \left(\mathbf {W} _ {e m b} h _ {t} + b\right) \tag {10}
$$
$$
L _ {\mathrm {L M}} = - \sum_ {t} \log \left(p \left(w _ {t + 1} \mid w _ {\leq t}\right)\right) \tag {11}
$$
4.2 Dynamic Oracle for SOM
Data:  $\theta_{1},\dots,\theta_{T},\Gamma$
Result:  $\xi_{1},\dots,\xi_{T}$
initialize  $\xi_{1} = N$
for  $i\gets 2$  to  $T$  do
$$
\begin{array}{l l} & j = \text {f i r s t \_ s i b l i n g} _ {\Gamma} (i); \\ & \mu_ {i} = \max  (\theta_ {j + 1},..., \theta_ {i - 1}); \\ & \xi_ {i} = \max  (\xi_ {j} - 1, \mu_ {i}); \end{array}
$$
Algorithm 1: The structure label generation algorithm, where  $\Gamma$  is the ground-truth tree and  $\theta_{i}$  is the structural decisions made by our model. This algorithm produces a parse close to the original given the errors already made, and that new gold parse is converted into grid decisions. Given  $\Gamma$ , the function first_sibling $_{\Gamma}(i)$  returns the index of the first token in the smallest clause that contains  $w_{i}$ , and where  $w_{i}$  is not the first token. Ideally,  $w_{i}$  should be written into the slot  $(\xi_{j} - 1)$ . For example, in Figure 2,  $c$  is written into the slot 2, then  $d, e$  should be written into the slot 1. However, the model could make a wrong decision between  $w_{j}$  and  $w_{i}$ . If the model has merged information from  $w_{j}$  into a higher slot  $\mu_{i}$ ,  $x_{i}$  should be written into slot  $\mu_{i}$  as well.
One way to provide a supervision signal for  $p_t$  and  $p'_t$  is to train the parser with static oracle: feed the gold tree to the model, and have the model predict future decisions. However, static oracle makes the language model overfit on the gold tree, resulting in bad perplexity scores (Table 2). Inspired
Type	Max	Median	Mean
Constituency	29	7	7.7
Dependency	16	4	4.2
Table 1: Statistics of tree depth for Penn Treebank. Dependency trees are converted from constituency tree with Stanford Corenlp toolkit.
Figure 4: The universal dependency tree is converted into a constituency tree  $\Gamma$  through merging the head and its children into one single constituent. Since the grid view only works with binary trees, we binarize n-ary nodes with a left branching bias.
by the dynamic oracles proposed in (Goldberg and Nivre, 2012; Coavoux and Crabbé, 2016), we propose a dynamic oracle for ordered memory, which dynamically changes the reference structure based on mistakes made by our model on previous steps. To do this, we build the structure label for each time step based on the gold tree and previous decisions made by the model. During training, we sample the model's decision from  $p_t$ :
$$
\theta_ {t} = \text {M u l t i n o m i a l} (p _ {t}) \tag {12}
$$
and we make greedy decisions during evaluation:
$$
\theta_ {t} = \operatorname {a r g m a x} \left(p _ {t}\right) \tag {13}
$$
The same operations are applied to  $p'_t$  as well.
We use the Algorithm.1 to convert the gold tree  $\Gamma$  into labels  $\xi_{t}$  for  $p_t$ . Since the zero-step look-ahead distribution  $p'_t$  should match the one-step look-ahead distribution  $p_{t+1}$  at next time step  $t+1$ , we use  $\xi_{t+1}$  as label for  $p'_t$ . The structure loss is the negative log-likelihood:
$$
L _ {\mathrm {S}} = - \sum_ {t} \left(\log \left(p _ {t} \left(\xi_ {t} \mid w _ {\leq t}\right)\right) + \log \left(p _ {t} ^ {\prime} \left(\xi_ {t + 1} \mid w _ {\leq t}\right)\right)\right)
$$
For our model, the depth of  $\Gamma$  has a linear relation to the computational complexity and GPU memory consumption. To maximize the model's efficiently, the gold tree  $\Gamma$  is constructed from universal dependency trees. There are two reasons