SlowGuess/ACL_2020_OCR huggingface dataset:
The results are shown in Table 1. After adding structural signals into the model, our model ONLSTM-SYD significantly outperforms the original ON-LSTM model ( $p$ -value  $< 0.05$ ), indicating that incorporating linguist-tagged parse trees can contribute to language modeling positively.
Results on PTB-Sepsent and CTB-Sepsent PTB-Sepsent and CTB-Sepsent offer a comparable setting with other structure-aware supervised (Dyer et al., 2016) and unsupervised (Kim et al., 2019b) baselines. The results are listed in Table 2. ONLSTM-SYD performs better than ONLSTM, which indicates that supervised syntactic information can help improve language modeling.
The margin between our models and the baselines is rather large. We find that the set of regularization and optimization techniques proposed by Merity et al. (2017) contribute significantly to this margin. Because of the sequential and parallel nature of our model, it can directly inherit and benefit from this set of tricks. In contrast, it is non-trivial to use them for RNNG and URNNG. As a more rigorous analysis, we further conducted a set of experiments without those tricks (i.e. non
monotonically triggered ASGD, weight-dropped LSTM, finetuning). The performance (denoted as ONLSTM-SYD-noAWD) drops; however, the model still outperforms the other baselines by a significant margin.
5.2 Structure Analysis
In this subsection we analyze the model to see how the additional structural supervision affects the quality of inferred trees. Note that our goal here is to analyze the influence of ground truth syntactic information on the quality of the induced trees rather than to yield a better grammar induction performance, since our model is not strictly comparable to other models due to its extra structural supervision during training.
We follow the settings of Htut et al. (2018) to test our model on the WSJ10 and WSJ test sets, reporting the results in Table 3. The WSJ test set has 2416 sentences with arbitrary lengths, while WSJ10 consists of 7422 sentences of the whole WSJ corpora that contain no more than 10 words. We use both biased and unbiased distance-to-tree conversion algorithms for both ON-LSTM and our proposed model (c.f. Appendix A.1 and A.2 for a formal description of the biased and non-biased conversion algorithm). Since our model has two sets of trees learned simultaneously, we list all of them in Table 3.
Grammar Induction We can see that the trees learned by the joint loss show improved the F1 score and rely less on the branching bias of the tree constructing algorithm (see Dyer et al. (2019)). The big gap of F1 scores on WSJ between the biased and unbiased trees are altered after introducing the structural loss, and the LM unbiased trees significantly outperforms its baseline ON-LSTM. These indicate that the auxiliary supervised task not only lowers the perplexity, but also improves the qualities of the induced trees for the LM task.
Looking more into the trees, we find that compared to ON-LSTM, ONLSTM-SYD improves the label prediction accuracy for NP (noun phrases), VP (verb phrases) and PP (prepositional phrases) but fails to improve ADJP (adjective phrases). This suggests that different types of human-annotated constituents may have different influences on language modeling, or that human-annotated trees are themselves biased to differing degrees between different constituent types.
2We use the preprocessing script in URNNG's repository https://github.com/harvardnlp/urnng, which merges all UNK types.
6616
Model	Training
Objective	Induction
Algorithm	Parsing
F1	Depth
WSJ	Accuracy on
ADJP	WSJ by
NP	Tag
VP	R/L Ratio
on WSJ
WSJ10	WSJ
ON-LSTM	LM	Unbiased	63.2	39.0	4.9	37.9	42.8	49.6	54.2
ON-LSTM	LM	Biased	69.5	44.2	5.5	57.0	53.0	52.4	49.6
ONLSTM-SYD syd	LM+SYD	Unbiased	77.6	61.3	7.3	38.2	73.2	69.6	72.9
ONLSTM-SYD syd	LM+SYD	Biased	65.7	45.5	5.5	30.4	40.6	70.7	43.9
ONLSTM-SYDlm	LM+SYD	Unbiased	55.1	34.5	4.8	14.9	42.2	16.7	67.4
ONLSTM-SYDlm	LM+SYD	Biased	58.0	36.3	5.3	41.1	53.9	52.4	43.0
Binary Gold Standard Trees	-	-	88.1	85.6	6.4	100	100	100	100
Gold standard Trees	-	-	100	100	5.0	100	100	100	100
Random Trees (Htut et al., 2018)	-	-	32.2	18.6	5.3	17.4	22.3	-	16.0
Balanced Trees (Htut et al., 2018)	-	-	43.4	24.5	4.6	22.1	20.2	-	9.3
Left Branching Trees	-	-	19.6	9.0	12.4	-	-	-	-
Right Branching Trees	-	-	56.6	39.8	12.4	-	-	-	-
Table 3: Unlabeled parsing results evaluated on the WSJ10 and the full WSJ test set. Numbers in bold font indicate that they are the best compared to those computed from the other parts of the model (i.e., within the same section in the table). The Algorithm column represents whether bias or unbiased algorithm is performed. ONLSTM-SYD $^{syd}$  and ONLSTM-SYD $^{lm}$  represent two sets of trees induced from loss  $L_{\mathrm{syd}}$  and  $L_{\mathrm{lm}}$  respectively. The Accuracy columns represent the fraction of ground truth constituents of a given type that correspond to constituents in the model parses. The R/L Ratio column represents the ratio between the number of words that are left children of its parent, and those that are right children.
Branching Bias Syntactic trees of English naturally have a bias towards right branching structures. As shown in the last section of Table 3, right branching trees achieve a much higher F1 score than random, balanced or left branching trees. As pointed out by Dyer et al. (2019), PRPN and ONLSTM resort to a distance-to-tree algorithm with right-branching biases (See Appendix A.2).
For our model, a biased distance-to-tree algorithm yields worse results compared to its non-biased counterpart; but on unsupervised models such as ON-LSTM, biased algorithms yield better results than non-biased versions. This observation indicates that syntactic supervision leads to better tree structures as compared with fully unsupervised tree induction, which is intuitive.
Linguistic Analysis Our best parsing results are for trees decoded from the syntactic prediction objective using the unbiased algorithm. Interestingly, these trees tend to be deeper on average than the (binarized) gold standard trees (see Table 3). This appears to be driven by a failure of the model to identify constituents centered on deeply-embedded head words—instead, the model prefers right-branching structures. Some examples of trees are displayed in Figure 3. In the top part of the figure, we see the parse produced from the  $L_{\mathrm{syd}}$  distances of our model, in the middle the tree produced the  $L_{\mathrm{lm}}$  distances and, on the bottom, the gold standard tree. As can be seen in the figure, the  $L_{\mathrm{syd}}$ -based tree is largely right-branching and misses constituents centered on several deeply em
bedded heads, such as the verb said. By contrast, the  $L_{\mathrm{lm}}$ -based tree is considerably shallower than the gold-standard and consists of a sequence of smaller chunks that often mis-bracket words with respect to the gold-standard constituent boundaries.
Figure 4 illustrates these phenomenon in further detail. The plot at the top of the figure shows the proportion of constituents produced from  $L_{\mathrm{syd}}$  distances whose boundaries correspond to a gold constituent, broken down by height of nodes in the predicted tree. As the plot illustrates, the model fares better on relatively small constituents lower in trees, and makes more errors for constituents higher in the tree, reflecting mistakes on deeply-embedded heads. The bottom of the figure shows the same breakdown for  $L_{\mathrm{lm}}$ -based induced trees. Overall, the affect is similar, although  $L_{\mathrm{lm}}$ -based trees are shallower than the  $L_{\mathrm{syd}}$ -based trees. We believe the increased accuracy for the longest constituents is driven by the fact that, since the highest constituents cover long sentence spans and there are few possible long spans, these constituents have a higher baseline probability of being correct.
It appears that the  $L_{\mathrm{syd}}$  objective has learned a strong right-branching bias, leading to very deep trees (even with the unbiased decoder) whereas the  $L_{\mathrm{lm}}$  objective appears to be using a kind of predictive chunking of the sentence into small groups of words. It is tempting to speculate that these chunks may correspond to linguistic units used in prosodic planning or by the human sentence processor, while the deeper trees correspond more directly to the compositional structure underlying sentence meaning. We leave exploring this question to future
3Please refer to Appendix A.5 for visualizations of a more extensive set of sentences.
6617
Figure 3: Trees induced from the syntactic task distances in our model (top), the language modeling task distances (middle) as well as the gold-standard trees (bottom).
Figure 4: Accuracy breakdown w.r.t. constituent height in unbiased trees derived from the syntactic task distances in our model (top) and the language modeling distances (bottom). A constituent is considered as correct if its boundaries correspond to a true constituent. The constituents' heights are those in the predicted tree. Since constituents that represent the whole sentence always have correct boundaries, they are excluded from the calculation.
Parsing performance Our models give worse unlabeled parsing performance compared to transition-based methods. In particular, Kim et al. (2019a) report that unsupervised URNNG achieves
45.4 WSJ F1 in a similar setting, while another URNNG that finetunes a supervised RNNG model gives a much better F1 of 72.8, leading a 27.4 F1 improvement. In contrast, the F1 of our structure prediction trees is 61.3 in unbiased algorithm. This indicates that our model brings more benefits on the LM side rather than the parsing side.
Layer used for supervision Table 4 (Top) shows the performances where the supervised signal is injected into different layers. Although injecting syntax into the last layer gives the best syntactic distance for grammar induction, it fails to achieve a similar improvement on perplexity. This suggests that a better syntactic structure may not always lead to a better language model. The observation is consistent with prior research (Williams et al., 2018).
Tree structure We study the influence of the different types of supervised trees to the model. In addition to using the ground truth parse trees, we also tried to train the model with random trees instead, and without providing trees, in which case it degenerates to a vanilla ON-LSTM. From Table 4 (Middle) we can find that without supervision signals from gold standard parse trees the model performs worse than the full model. Random trees introduce noise to the model and downgrade both parsing and LM performance, indicating the importance of injecting meaningful syntax.
Multitask variants We also explored injecting the supervised syntactic information at different levels. One straight forward baseline is to add supervision signals directly on the syntactic distance
6618
Ablation Study	Experiment Detail	Validation PPL	Test PPL	WSJ F1
Layer for Supervision	1st layer	58.0	55.6	57.7
2nd layer	57.8	55.5	59.7
3rd layer	57.8	55.7	61.3
Tree Structure	No Parse Tree	58.3	55.9	39.0
Random Tree	60.2	57.5	32.4
Gold Parse Tree	57.8	55.7	61.3
Multitask Variants	Vanilla Multitasking	60.9	58.5	24.9
One set of trees	58.5	55.9	54.4
Two sets of trees	57.8	55.7	61.3
Table 4: Perplexity and unlabeled parsing F1 in ablation studies. We choose unbiased algorithm and the layer with supervision injected. For the unsupervised models, we report the layer with best F1 score. (Top) When supervising on different layers. (Middle) Using different tree structures for supervision. (Bottom) Different multitasking strategies.
in ON-LSTM, using one set of trees to guide both LM and parsing, as indicated in the Model section (Table 4 Bottom, one set of trees). Despite injecting stronger syntactic signals, this direct approach does not improve language model perplexity. This also reflects the fact that the most suitable syntactic structures for language modeling do not necessarily conform to human labeled syntax. In addition, we also use ON-LSTM hidden states for supervised syntactic distance prediction (Table 4 Bottom, vanilla multitasking). This approach fails to outperform its ON-LSTM baseline due to the same reason. In summary, there are mutual benefits between induced and supervised syntactic information, although they do not fully overlap.
Generalization to other LMs One practical question is whether the improvements found in our work can be generalized to other language models. To answer this question, we introduce the multitask scheme to PRPN (Shen et al., 2017), which is another model that is also able to learn unsupervised structures through language modeling. Similar to ON-LSTM, PRPN is also a syntactic distance method. We modify the PRPN model in the same spirit as in ON-LSTM. In addition, we change the encoding layer and use the output as syntactic distance embeddings  $l_{syd}$ . Then we map  $l_{syd}$  to two sets of syntactic distances  $d_{lm}$  and  $d_{syd}$  for language modeling and syntactic distance prediction, respectively. Syntactic supervision comes to  $d_{syd}$ . The model reaches a test perplexity of 60.5 in PTB-Concat ( $p$ -value  $< 0.05$ ), which also significantly outperforms the 62.0 from the original model. We refer readers to Appendix A.4 for the details of PRPN and our modified PRPN-SYD.
We investigated linguistic supervision for distance-based structure-aware language models, showing its strengths over transition-based counterparts in language modeling. Apart from the explicit observations in achieving strong perplexity scores, our model reveals several interesting aspects of the quality of the trees learned by the model. As a byproduct of our investigation, we release a version of PTB-Concat, which contains syntactic structures while at the same time the same pre-processing steps adopted by most previous work on neural language models.
We thank Zhiyang Teng, Qi He and all members at Text Intelligent Lab in Westlake University for insightful discussions. We also would like to thank all anonymous reviewers for their constructive comments. This work is supported by the National Natural Science Foundation of China (NSFC No. 61976180) and the Westlake University and Bright Dream Joint Institute for Intelligent Robotics. The corresponding author is Yue Zhang.
David Adger. 2003. Core Syntax: A Minimalist Perspective. Oxford University Press.
Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. arXiv preprint arXiv:1603.06042.
Joan Bresnan. 2001. Lexical functional syntax. Wiley-Blackwell, Oxford.
Christopher Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Gregory N Hullender. 2005. Learning to rank using gradient descent. In Proceedings of the 22nd International Conference on Machine learning (ICML-05), pages 89-96.
Ciprian Chelba and Frederick Jelinek. 2000. Structured language modeling. Computer Speech & Language, 14(4):283-332.
Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long short-term memory-networks for machine reading. arXiv preprint arXiv:1601.06733.
Jianpeng Cheng, Adam Lopez, and Mirella Lapata. 2017. A generative parser with a discriminative recognition algorithm. arXiv preprint arXiv:1708.00415.
6619
Noam Chomsky. 1995. The Minimalist Program. The MIT Press, Cambridge, Massachusetts and London, England.
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. 2016. Recurrent neural network grammars. arXiv preprint arXiv:1602.07776.
Chris Dyer, Gábor Melis, and Phil Blunsom. 2019. A critical analysis of biased parsers in unsupervised parsing. arXiv preprint arXiv:1909.09428.
Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In Advances in neural information processing systems, pages 1019-1027.
Edouard Grave, Armand Joulin, and Nicolas Usunier. 2016. Improving neural language models with a continuous cache. arXiv preprint arXiv:1612.04426.
John Hale, Chris Dyer, Adhiguna Kuncoro, and Jonathan R Brennan. 2018. Finding syntax in human encephalography with beam search. arXiv preprint arXiv:1806.04127.
Peter A Heeman and James F Allen. 1997. Incorporating pos tagging into language modeling. arXiv preprint cmp-lg/9705014.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Jan E Holly. 2001. Pictures of ultrametric spaces, the p-adic numbers, and valued fields. The American Mathematical Monthly, 108(8).
Phu Mon Htut, Kyunghyun Cho, and Samuel R Bowman. 2018. Grammar induction with neural language models: An unusual replication. arXiv preprint arXiv:1808.10000.
Rodney Huddleston and Geoffrey K. Pullum. 2002. The Cambridge Grammar of English Language. Cambridge University Press, Cambridge.
Stephen C Johnson. 1967. Hierarchical clustering schemes. Psychometrika, 32(3).
Daniel Jurafsky, Chuck Wooters, Jonathan Segal, Andreas Stolcke, Eric Fosler, Gary Tajchaman, and Nelson Morgan. 1995. Using a stochastic context-free grammar as a language model for speech recognition. In 1995 International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 189-192. IEEE.
Yoon Kim, Chris Dyer, and Alexander M Rush. 2019a. Compound probabilistic context-free grammars for grammar induction. arXiv preprint arXiv:1906.10225.
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In Thirtieth AAAI Conference on Artificial Intelligence.
Yoon Kim, Alexander M Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gabor Melis. 2019b. Unsupervised recurrent neural network grammars. arXiv preprint arXiv:1904.03746.
Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018. Lstms can learn syntax-sensitive dependencies well, but modeling structure makes them better. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1426-1436.
Adhiguna Kuncoro, Chris Dyer, Laura Rimell, Stephen Clark, and Phil Blunsom. 2019. Scalable syntax-aware language models using knowledge distillation. arXiv preprint arXiv:1906.06438.
Willem J. M. Levelt. 1974. Formal Grammars in Linguistics and Psycholinguistics, Volume 3: Psycholinguistic applications.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2017. Regularizing and optimizing LSTM language models. arXiv preprint arXiv:1708.02182.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843.
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association.
Joakim Nivre. 2008. Algorithms for deterministic incremental dependency parsing. Computational Linguistics, 34(4):513-553.
Hao Peng, Roy Schwartz, and Noah A Smith. 2019. Palm: A hybrid parser and language model. arXiv preprint arXiv:1909.02134.
Carl Pollard and Ivan A Sag. 1994. Head-driven phrase structure grammar. University of Chicago Press.
Ivan A. Sag, Thomas Wasow, and Emily M. Bender. 2003. Syntactic Theory: A Formal Introduction. CSLI, Stanford, CA.
Yikang Shen, Zhouhan Lin, Chin-Wei Huang, and Aaron Courville. 2017. Neural language modeling by jointly learning syntax and lexicon. arXiv preprint arXiv:1711.02013.
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2018. Ordered neurons: Integrating tree structures into recurrent neural networks. arXiv preprint arXiv:1810.09536.
B Srinivas. 1996. "almost parsing" technique for language modeling. In Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP'96, volume 2, pages 1173-1176. IEEE.
Mark Steedman and Jason Baldridge. 2011. Combinatory categorial grammar. Non-Transformational Syntax: Formal and explicit models of grammar, pages 181-224.
Mitchell Stern, Jacob Andreas, and Dan Klein. 2017. A minimal span-based neural constituency parser. arXiv preprint arXiv:1705.03919.
Sho Takase, Jun Suzuki, and Masaaki Nagata. 2018. Direct output connection for a high-rank language model. arXiv preprint arXiv:1808.10143.
Dong Hoon Van Uytsel, Filip Van Aelten, and Dirk Van Compernolle. 2001. A structured language model based on context-sensitive probabilistic left-corner parsing. In Proceedings of the second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies, pages 1-8. Association for Computational Linguistics.
Adina Williams, Andrew Drozdov*, and Samuel R Bowman. 2018. Do latent tree learning models identify meaningful structure in sentences? Transactions of the Association for Computational Linguistics, 6:253-267.
Jerry H Wright, Gareth JF Jones, and Harvey Lloyd-Thomas. 1994. A robust language model incorporating a substring parser and extended n-grams. In Proceedings of ICASSP'94. IEEE International Conference on Acoustics, Speech and Signal Processing, volume 1, pages I-361. IEEE.
Bang Ye Wu, Kun-Mao Chao, and Chuan Yi Tang. 1999. Approximation and exact algorithms for constructing minimum ultrametric trees from distance matrices. Journal of CO, 3(2).
Peng Xu, Ciprian Chelba, and Frederick Jelinek. 2002. A study on richer syntactic dependencies for structured language modeling. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 191-198. Association for Computational Linguistics.
Peng Xu, Ahmad Emami, and Frederick Jelinek. 2003. Training connectionist models for the structured language model. In Proceedings of the 2003 conference on Empirical methods in natural language processing, pages 160-167. Association for Computational Linguistics.
Naiwen Xue, Fei Xia, Fu-Dong Chiou, and Marta Palmer. 2005. The penn chinese treebank: Phrase structure annotation of a large corpus. Natural language engineering, 11(2):207-238.
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. 2018. Breaking the softmax bottleneck: A high-rank rnN language model. *ICLR*.
Yue Zhang and Joakim Nivre. 2011. Transition-based dependency parsing with rich non-local features. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 188-193. Association for Computational Linguistics.
Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutnik, and Jürgen Schmidhuber. 2017. Recurrent highway networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 4189-4198. JMLR.org.
Konrad Zolna, Devansh Arpit, Dendi Suhubdy, and Yoshua Bengio. 2018. Fraternal dropout. ICLR.
Barret Zoph and Quoc V Le. 2016. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578.
6621
A.1 Algorithms for transformation between parse trees and syntactic distances
The following tree-to-distance algorithm provides a set of distances given a tree. The node indicates the root node of the given tree.
The following distance-to-tree conversion algorithm provides an unbiased reconstruction of tree given a set of distances.
A.2 Distance-to-tree algorithm with right-branching bias
A.3 Details of generating our PTB-Concat-Syn version
Mikolov et al. (2010) briefly described the steps of converting from the original Penn Treebank dataset to his version of dataset, which later becomes the standard in language modeling task. We denote this version as PTB-Concat. In our paper, to get strictly the same PTB language modeling dataset, we follow his steps on the original Penn Treebank, while preserving the tree structure. Specifically, we took the following steps:
1. Convert all tokens to lowercase.
2. For tokens which are purely digits, or digits only with “.” or “-” are converted to token “N”.
3. Replace all “$” with “N”.
4. Delete tokens "\\" and "wa" if their POS tags are "POS" and "NNP", respectively.
5. Delete all tokens that fall into the following list: