SlowGuess/NAACL_2021_OCR huggingface dataset:
we chose universal dependency trees instead of constituency trees: 1) In Table 1, the dependency trees are on average shallower than constituency trees; this means faster computation time and less memory consumption for our model. 2) Universal dependency trees can be applied to many more languages than Penn Treebank-style constituency grammar. Additionally, Penn Treebank-style trees can easily be converted to universal dependency trees. As shown in Figure 4, we convert the universal dependency tree into  $\Gamma$  by merging the head and its children into one single constituent.
We present the results of SOM on language modeling, syntactic generalization, and incremental parsing. Details of hyperparameters and experiment settings can be found in Appendix B.
Penn Treebank has one million words of 1989 Wall Street Journal corpus annotated with constituency trees. Since SOM primarily focuses on sentence-level structure and language modeling, we use the same preprocessing schema as  $\mathrm{RNNG}^2$  (Dyer et al., 2016). Sentences are modeled separately, punctuation is retained, and singleton words are replaced with the Berkeley parser's mapping rules $^3$ , resulting in 23,815-word types. Orthographic case distinction is preserved, and numbers (beyond singletons) are not normalized.
BLLIP is a large Penn Treebank-style parsed corpus of approximately 24 million sentences. We train and evaluate SOM on three splits of BLLIP: BLLIP-XS (40k sentences, 1M tokens), BLLIP-SM (200K sentences, 5M tokens), and BLLIP-MD (600K sentences, 14M tokens). They are obtained by randomly sampling sections from BLLIP 1987-89 Corpus Release 1. All models are tested on a shared held-out tested set.
Following the settings provided in (Hu et al., 2020), datasets are preprocessed into two different versions. The first setting is similar to the PTB dataset. Singleton words are mapped to UNK classes that preserve fine-grained information, such as orthographic case distinctions and morphological suffixes (e.g. UNK-ed, UNK-ly). The second setting use subword-level vocabulary extracted
2-21 for training, 24 for validation, 23 for evaluation.
<sup>3</sup>http://github.com/slavpetrov/
berkeleyparser
<https://universaldependencies.org/
Model	# parameters	ppl	p acc	UF1	p' acc
SOM	17.7M	77.68	0.927	87.96	0.870
SOM – Prediction network	13.0M	83.63	0.923	87.09	-
SOM – Prediction network – Language Modeling Loss	13.0M	-	0.925	86.26	-
SOM – Dynamic Oracle + Static Oracle	17.7M	129.27	0.913	86.58	0.849
SOM – Dynamic Oracle + Left-branching Oracle	17.7M	82.01	-	-	-
Inference with External Trees
SOM – Predicted tree + Gold tree	17.7M	60.87	0.947	100.00	0.884
Table 2: Ablation tests on the PTB dataset. “ $p$  acc” and “ $p'$  acc” are the prediction accuracies of the one-step look-ahead and zero-step look-ahead parsers respectively. “UF1” is the parsing performance with respect to the converted constituency tree  $\Gamma$ . “- Prediction network”: this model uses the last candidate memory slot  $\hat{M}_t^N$  to predict the next token, instead of using the  $h_t$  from the prediction network. “- Predicted tree + Gold tree”: the model's parsing decisions were replaced with ground truth decisions; these results can be considered as the performance upper bound of SOM.
Model	PTB
Without annotations
RNNLM	93.2
PRPN (Shen et al., 2017)	96.7
URNNG (Kim et al., 2019)	90.6
With annotations
RNNG (Dyer et al., 2016)	88.7
RNNG → URNNG (Kim et al., 2019)	85.9
SOM	77.7
Table 3: Perplexities on Penn Treebank datasets. With annotations are models that use the gold tree as supervision signal during training. Baseline results are from Kim et al. (2019)
Model	XS	SM	MD
n-gram	240.21	157.60	106.09
RNNG	122.46	86.72	69.57
LSTM	98.19	65.52	59.05
ON-LSTM	71.76	54.00	56.37
GPT-2	529.90*	183.10*	37.04*
SOM	70.41	51.47	31.95*
Table 4: Perplexities on BLLIP datasets achieved by different models. Perplexity scores across training dataset sizes are not strictly comparable for models that use word-level vocabulary. * results are using GPT-2's subword vocabulary.
from the GPT-2 pretrained model rather than the BLLIP training corpora.
Results of language modeling are given in Table 3 and Table 4. SOM consistently outperforms both the annotated model and non-annotated models. While GPT-2 seems to fail to learn on smaller datasets, SOM still outperforms GPT-2 on the BLLIP-MD dataset with far fewer parameters (34.8M vs 124.4M), and achieves comparable results with the GPT-2 that is trained on a 3 times larger dataset BLLIP-LG (Hu et al., 2020).
The ablation test results are shown in Table 2.
The biggest performance drop comes from replacing the dynamic oracle with static oracle. We believe that this is due to the model overfitting on the gold tree, and suffering from exposure bias as a result. Another big performance drop happens after removing the prediction network. This suggests that predicting the attaching nodes of the next phrase with the zero-step look-ahead parsers helps to predict the next token. Replacing the gold tree labels with trivial left-branching tree labels also hurts the perplexity. This suggests that learning syntactic structure helps language modeling.
5.2 Syntactic Generalization
Syntactic Generalization (SG) test suites evaluate the syntactic knowledge of neural language models. Hu et al. (2020) proposed a set of 34 test suites to evaluation 6 different aspects of syntax: 1) agreement, 2) licensing, 3) garden-path effects, 4) gross syntactic expectation, 5) center embedding, 6) long-distance dependencies.
Following their settings, we evaluate our language models trained on the BLLIP datasets. Language models are presented with a group of sentences with minor differences. To pass each test, the model needs to assign higher conditional probabilities to designated phrases in the sentence that are more grammatical.
Figure 6 shows the average accuracy over all model on the complete set of SG test suites. SOM achieves the best average accuracy, outperforms models with hierarchical structure bias (RNNG, ON-LSTM), and transformer-based model (GPT-2). However, according to Figure 8a in Appendix C.1, GPT-2 trained on BLLIP-LG and BLLIP-MD still outperform SOM. This could due to that the number of parameters in SOM is largely falling behind GPT-2.
Figure 6: Average SG accuracy by model class.
Figure 5 provides fine-grained results on six SG classes. SOM achieves strong performance on licensing, gross syntactic state, center embedding, and long-distance embeddings. These classes require the model to keep track of syntactic features across large syntactic chunks (e.g., relative or sub-ordination clauses). SOM can effectively keep this long-term information in higher-level memory slots, and revisit the information after the clause in the middle is ended. More detailed results can be found in Appendix C.1.
5.3 Incremental Parsing
Model	UF1
PRPN*	41.2
ONLSTM*	47.7
ONLSTM-SYD (Du et al., 2020)	61.3
Incremental Shift-reduce Parser	56.82
Shift-reduce + LM + Dynamic Oracle	58.04
SOM	67.27
Oracle Binary Trees	82.5
Table 5: Incremental parsing results on the standard PTB constituency trees. “*” means that the model is doing unsupervised grammar induction. Since we compare UF1 against the standard, nonbinarized trees (per convention), UF1 scores is upper bounded by the oracle binary trees score.
To evaluate SOM's performance on incremental parsing, we trained and evaluated our models on the standard PTB constituency trees. Baseline models include: a) a standard incremental shift-reduce parser with one-step look-ahead; b) a incremental shift-reduce parser that equipped with our predic
tion network and trained on same dynamic oracle and language model loss as our model; c) a recently proposed ONLSTM-SYD model (Du et al., 2020) that is also trained on both language model and parsing loss; d) unsupervised ONLSTM; e) unsupervised PRPN. As shown in Table 5, SOMs outperform all baseline models, including the shift-reduce parser that has the same extra components as SOMs. For language modelling performance, original constituency tree based models achieve similar perplexity as dependency tree based counterparts. But constituency tree based models require  $2 \times$  GPU time and memory to train and evaluate.
For ablation test, we also compare parsing results given by SOM with binary constituency trees  $\Gamma$  converted from universal dependency trees. These results are shown in Table 2. We observe that using static oracle instead of dynamic oracle results in the worst parsing performance. This suggests that our dynamic oracle helps the model to learn a better parser. After removing the language model loss, the UF1 drops 1.7 points. This suggests that the language model loss helps the model to learn better representations for syntax.
In this work, we propose a new language model with an integrated incremental parser. This was done by augmenting the Ordered Memory model with a prediction network, and by using a dynamic oracle for training it to perform incremental parsing. The resulting model models the joint distribution of syntactic structure and sequence words. We find that by using the dynamic oracle and explicitly modeling the syntax, we can achieve strong performance on language modelling and syntactic generalization and both these techniques are crucial in the model's performance.
4UF1 scores are computed by EVALB https://nlp.cs.nyu.edu/evalb/
Samuel R Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D Manning, and Christopher Potts. 2016. A fast unified model for parsing and sentence understanding. arXiv preprint arXiv:1603.06021.
Maximin Coavoux and Benoit Crabbé. 2016. Neural greedy constituent parsing with dynamic oracles.
Hal Daumé, John Langford, and Daniel Marcu. 2009. Search-based structured prediction. Machine learning, 75(3):297-325.
Vera Demberg, Frank Keller, and Alexander Koller. 2013. Incremental, predictive parsing with psycholinguistically motivated tree-adjoining grammar. Computational Linguistics, 39(4):1025-1066.
Andrew Drozdov, Subendhu Rongali, Yi-Pei Chen, Tim O'Gorman, Mohit Iyyer, and Andrew McCallum. 2020. Unsupervised parsing with s-diora: Single tree encoding for deep inside-outside recursive autoencoders. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4832-4845.
Andrew Drozdov, Pat Verga, Mohit Yadav, Mohit Iyyer, and Andrew McCallum. 2019. Unsupervised latent tree induction with deep inside-outside recursive autoencoders. arXiv preprint arXiv:1904.02142.
Wenyu Du, Zhouhan Lin, Yikang Shen, Timothy J O'Donnell, Yoshua Bengio, and Yue Zhang. 2020. Exploiting syntactic structure for better language modeling: A syntactic distance approach. arXiv preprint arXiv:2005.05864.
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. 2016. Recurrent neural network grammars. arXiv preprint arXiv:1602.07776.
Daniel Fried and Dan Klein. 2018. Policy gradient as a proxy for dynamic oracles in constituency parsing. arXiv preprint arXiv:1806.03290.
Carlo Ghezzi and Dino Mandrioli. 1979. Incremental parsing. ACM Transactions on Programming Languages and Systems (TOPLAS), 1(1):58-70.
Yoav Goldberg and Joakim Nivre. 2012. A dynamic oracle for arc-eager dependency parsing. In Proceedings of COLING 2012, pages 959-976.
Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of NAACL-HLT, pages 1195-1205.
Hany Hassan, Khalil Sima'an, and Andy Way. 2009. Lexicalized semi-incremental dependency parsing. In Proceedings of the International Conference RANLP-2009, pages 128-134.
Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger P Levy. 2020. A systematic assessment of syntactic generalization in neural language models. arXiv preprint arXiv:2005.03692.
Yoon Kim, Alexander M Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gabor Melis. 2019. Unsupervised recurrent neural network grammars. arXiv preprint arXiv:1904.03746.
Arne Kohn and Timo Baumann. 2016. Predictive incremental parsing helps language modeling.
Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A Smith. 2016. What do recurrent neural network grammars learn about syntax? arXiv preprint arXiv:1611.05774.
Adhiguna Kuncoro, Chris Dyer, Laura Rimell, Stephen Clark, and Phil Blunsom. 2019. Scalable syntax-aware language models using knowledge distillation. arXiv preprint arXiv:1906.06438.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535.
Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The penn treebank: annotating predicate argument structure. In Proceedings of the workshop on Human Language Technology, pages 114-119. Association for Computational Linguistics.
Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. arXiv preprint arXiv:1808.09031.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
Brian Roark. 2001. Probabilistic top-down parsing and language modeling. Computational linguistics, 27(2):249-276.
Florian Schmidt. 2019. Generalization in generation: A closer look at exposure bias. arXiv preprint arXiv:1910.00292.
Yikang Shen, Zhouhan Lin, Chin-Wei Huang, and Aaron Courville. 2017. Neural language modeling by jointly learning syntax and lexicon. arXiv preprint arXiv:1711.02013.
Yikang Shen, Shawn Tan, Arian Hosseini, Zhouhan Lin, Alessandro Sordoni, and Aaron C Courville. 2019. Ordered memory. In Advances in Neural Information Processing Systems, pages 5038-5049.
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2018. Ordered neurons: Integrating tree structures into recurrent neural networks. arXiv preprint arXiv:1810.09536.
Stuart M Shieber. 1983. Sentence disambiguation by a shift-reduce parsing technique. In Proceedings of the 21st annual meeting on Association for Computational Linguistics, pages 113-118. Association for Computational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642.
Mark Steedman. 2000. The syntactic process, volume 24. MIT press Cambridge, MA.
Swabha Swayamdipta, Matthew Peters, Brendan Roof, Chris Dyer, and Noah A Smith. 2019. Shallow syntax in deep water. arXiv preprint arXiv:1908.11047.
A Disentangling Semantic and Syntactic representations
Given the architecture of our model, we can easily disentangle the language model information flow and parsing information flow. Figure 7 illustrates the disentangled information and gradient flows in our model. The language model depends on both prior context and structural inputs, and derivatives are computed with respect to both of these inputs and backpropagated. However, while the structure also depends on both inputs, we limit backpropagation so that it can only update with respect to the syntactic input. This is because we want the parsing component to function independently of the language modelling component, but still leverage the semantic information to deal with syntactic ambiguity.
Figure 7: The schema of disentangling syntax from the language model. Solid lines represent dependency during inference, and gradients flow back during backpropagation. The dashed line represents the dependency during inference, but detached so that the gradients do not flow back during backpropagation.
It is possible that existing model architectures could implicitly learn to split these representations, even without the explicit disentanglement that we proposed here. Yet, Table 2 shows that entangled model can actually achieve stronger in-domain performance, thanks to the liberty to allocate capacity to the two different functionalities based on the training set.
To do so, we propose splitting word embeddings, memory slots, and intermediate hidden states into two segments: semantic segment and syntactic segment. We then replace linear layers in our cell functions with the following function:
$$
\left[ \begin{array}{c} y _ {s e m} \\ y _ {s y n} \end{array} \right] = \left[ \begin{array}{c c} {\bf W} _ {s e m 2 s e m} & {\bf W} _ {s y n 2 s e m} \\ 0 & {\bf W} _ {s y n 2 s y n} \end{array} \right] \left[ \begin{array}{c} x _ {s e m} \\ x _ {s y n} \end{array} \right]
$$
where  $y_{sem}$  and  $x_{sem}$  are the semantic segment which is optimized to minimize language modeling loss,  $y_{syn}$  and  $x_{syn}$  are the syntactic segment which is optimized to minimize both parsing and language modeling loss. This architecture results in the solid lines in Figure 7. Additionally, layer normalization functions are replaced with two separate functions for the two segments respectively.
Meanwhile,  $p_t$  still depends on both semantic and syntactic segment, but the structural loss does not backpropagate into the semantic segment:
$$
p _ {t} = f \left(x _ {t, s e m}, x _ {t, s y n}, \hat {M} _ {t, s e m}, \hat {M} _ {t, s y n}\right) \tag {14}
$$
$$
\frac {\partial p _ {t}}{\partial x _ {t , s e m}} = 0, \quad \frac {\partial p _ {t}}{\partial \hat {M} _ {t , s e m}} = 0 \tag {15}
$$
and the same for  $p'_t$ :
$$
p _ {t} ^ {\prime} = f \left(\hat {M} _ {t, s e m}, \hat {M} _ {t, s y n}\right) \tag {16}
$$
$$
\frac {\partial p _ {t} ^ {\prime}}{\partial \hat {M} _ {t , s e m}} = 0 \tag {17}
$$
This gradient detachment is represented by the dash line in Figure 7. In the experiment section, the disentangled models are denoted as dSOM, and entangled models are denoted as SOM. For dSOM, the dimension of semantic and syntactic segments for memory slots are denoted as  $D_{sem}$  and  $D_{syn}$  respectively. Among the proposed models, the eSOM has the best performance on the in-domain test sets. Appendix C.2 shows that the dSOM slightly outperforms eSOM in perplexity on out-of-domain test sets.
Model	XS	SM	MD
RNNG	22.8M	48.4M	81.1M
LSTM	13.4M	30.5M	52.2M
ONLSTM+AWD	30.8M	44.2M	61.2M
GPT-2	124.4M	124.4M	124.4M
dSOM	16.4M	39.5M	34.8M
eSOM	17.8M	41.4M	37.9M
Table 6: Parameter counts for different models
Dropout is applied before all linear layers in our model. They all share the same dropout rate, except the dropout before language model output layer has a different rate. We also applied embedding dropout which randomly set some embedding vectors to 0. Hyperparameters are chosen based on the perplexity on validation set.
C.1 Syntactic Generalization results
C.2 Out of Domain Evaluation
Out-of-domain Test set contains testsets from other English universal dependencies treebanks. It contains corpora of different genres, including academic, email, blog, fiction, legal, news, etc. We use these datasets to test the generalization ability of models that are trained on PTB.
Dataset	Dsem	Dsyn	#slots	embedding dropout	dropout	output dropout
PTB	300	100	15	0.1	0.3	0.5
BLLIP-XS	300	100	15	0.1	0.3	0.5
BLLIP-SM	400	100	15	0.1	0.2	0.2
BLLIP-MD-BPE	400	100	15	0	0.1	0.1
(a) Relationship between SG socre and perplexity on the held-out BLLIP test set.
(b) SG score on the pairs of test suites with and without intervening modifiers: Center Embedding, Cleft, MVRR, NPZ-Ambiguous, and NPZ-Object.
Figure 9: Left: Model class has a strong effect on SG scores. Right: Data scale has little effect on SG scores
Table 7: Hyperparameters. The hidden size of eSOM models are always the sum of  $D_{sem}$  and  $D_{syn}$
Dataset Metric	GUM	EWT	ParTUT	LinES	Pronouns	PUD	SG acc
ppl	UF1	ppl	UF1	ppl	UF1	ppl	UF1	ppl	UF1	ppl	UF1
eSOM	351.5	67.0	400.7	73.1	282.6	76.5	253.3	67.2	552.1	87.1	281.0	75.6	0.614
dSOM	350.3	66.4	403.0	72.3	269.8	74.3	252.1	66.6	565.8	87.3	280.3	75.1	0.581
LB	375.5	-	436.5	-	300.9	-	267.5	-	620.4	-	300.7	-	0.513
Table 8: Out of domain test results. Models are trained on PTB. The test sets are obtained from English universal dependencies treebank. "LB" stands for left-branching tree labels. Thanks to the structure information, our models generalize much better than the left-branching baseline.
Figure 10: Left: differences in model class induce significant differences in SG scores for several circuits. Right: differences in training data size do not reliably account for most of circuits.