SlowGuess/NAACL_2021_OCR huggingface dataset:
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In Proceedings of the 5th International Conference on Learning Representations, Toulon, France. OpenReview.net.
Andrew Drozdov, Subendhu Rongali, Yi-Pei Chen, Tim O'Gorman, Mohit Iyyer, and Andrew McCallum. 2020. Unsupervised parsing with S-DIORA: Single tree encoding for deep inside-outside recursive autoencoders. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4832-4845, Online. Association for Computational Linguistics.
Andrew Drozdov, Patrick Verga, Mohit Yadav, Mohit Iyyer, and Andrew McCallum. 2019. Unsupervised latent tree induction with deep inside-outside recursive auto-encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1129-1141, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Daniel Flannery, Yusuke Miayo, Graham Neubig, and Shinsuke Mori. 2011. Training dependency parsers from partially annotated corpora. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 776-784, Chiang Mai, Thailand. Asian Federation of Natural Language Processing.
Daniel Flannery and Shinsuke Mori. 2015. Combining active learning and partial annotation for domain adaptation of a Japanese dependency parser. In Proceedings of the 14th International Conference on Parsing Technologies, pages 11-19, Bilbao, Spain. Association for Computational Linguistics.
Merrill Garrett, Thomas Bever, and Jerry Fodor. 1966. The active use of grammar in speech perception. Perception & Psychophysics, 1(1):30-32.
James Paul Gee and François Grosjean. 1983. Performance structures: A psycholinguistic and linguistic appraisal. Cognitive Psychology, 15(4):411-458.
Jennifer Gillenwater, Kuzman Ganchev, João Graça, Fernando Pereira, and Ben Taskar. 2010. Sparsity in dependency grammar induction. In Proceedings of the ACL 2010 Conference Short Papers, pages 194-199, Uppsala, Sweden. Association for Computational Linguistics.
Kevin Gimpel and Noah A. Smith. 2012. Structured ramp loss minimization for machine translation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 221-231, Montréal, Canada. Association for Computational Linguistics.
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and
Statistics, pages 249-256, Chia Laguna Resort, Sardinia, Italy. PMLR.
John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics, Pittsburgh, PA, USA. Association for Computational Linguistics.
Wenjuan Han, Ge Wang, Yong Jiang, and Kewei Tu. 2019. Multilingual grammar induction with continuous language identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5728-5733, Hong Kong, China. Association for Computational Linguistics.
Serhii Havrylov, Germán Kruszewski, and Armand Joulin. 2019. Cooperative learning of disjoint syntax and semantics. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1118-1128, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015. Question-answer driven semantic role labeling: Using natural language to annotate natural language. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 643-653, Lisbon, Portugal. Association for Computational Linguistics.
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129-4138, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Rebecca Hwa. 1999. Supervised grammar induction using training data with limited constituent information. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 73-79, College Park, Maryland, USA. Association for Computational Linguistics.
Lifeng Jin, Finale Doshi-Velez, Timothy Miller, William Schuler, and Lane Schwartz. 2018. Unsupervised grammar induction with depth-bounded PCFG. Transactions of the Association for Computational Linguistics, 6:211-224.
Vidur Joshi, Matthew Peters, and Mark Hopkins. 2018. Extending a parser to distant domains using a few dozen partially annotated examples. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1190-1199, Melbourne, Australia. Association for Computational Linguistics.
Tadao Kasami. 1965. An efficient recognition and syntax-analysis algorithm for context-free language. Technical report, Hawaii University Honolulu Department of Electrical Engineering.
Yoon Kim, Chris Dyer, and Alexander Rush. 2019a. Compound probabilistic context-free grammars for grammar induction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2369-2385, Florence, Italy. Association for Computational Linguistics.
Yoon Kim, Alexander Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gabor Melis. 2019b. Unsupervised recurrent neural network grammars. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1105-1117, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, San Diego, California, USA.
Nikita Kitaev, Steven Cao, and Dan Klein. 2019. Multilingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3499-3505, Florence, Italy. Association for Computational Linguistics.
Dan Klein. 2005. The Unsupervised Learning of Natural Language Structure. Ph.D. thesis, Stanford University.
Zhenghua Li, Min Zhang, Yue Zhang, Zhanyi Liu, Wenliang Chen, Hua Wu, and Haifeng Wang. 2016. Active learning for dependency parsing with partial annotation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 344-354, Berlin, Germany. Association for Computational Linguistics.
Percy Liang, Slav Petrov, Michael Jordan, and Dan Klein. 2007. The infinite PCFG using hierarchical Dirichlet processes. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 688-697, Prague, Czech Republic. Association for Computational Linguistics.
Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In *In Proceedings of the ICML Workshop on Deep Learning for Audio, Speech and Language Processing*, Atlanta, Georgia, USA.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated
corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330.
Héctor Martínez Alonso, Željko Agić, Barbara Plank, and Anders Søgaard. 2017. Parsing Universal Dependencies without training. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 230-240, Valencia, Spain. Association for Computational Linguistics.
Seyed Abolghasem Mirroshandel and Alexis Nasr. 2011. Active learning for dependency parsing using partially annotated sentences. In Proceedings of the 12th International Conference on Parsing Technologies, pages 140-149, Dublin, Ireland. Association for Computational Linguistics.
Tahira Naseem and Regina Barzilay. 2011. Using semantic cues to learn syntax. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, pages 902-907, San Francisco, California, USA. The AAAI Press.
Tahira Naseem, Harr Chen, Regina Barzilay, and Mark Johnson. 2010. Using universal linguistic knowledge to guide grammar induction. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1234-1244, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
Hiroshi Noji, Yusuke Miyao, and Mark Johnson. 2016. Using left-corner parsing to encode universal structural constraints in grammar induction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 33-43, Austin, Texas, USA. Association for Computational Linguistics.
Fernando Pereira and Yves Schabes. 1992. Inside-outside reestimation from partially bracketed corpora. In Proceedings of the 30th Annual Meeting of the Association for Computational Linguistics, pages 128-135, Newark, Delaware, USA. Association for Computational Linguistics.
Barbara Plank. 2016. Keystroke dynamics as signal for shallow syntactic parsing. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 609-619, Osaka, Japan. The COLING 2016 Organizing Committee.
Stefan Riezler, Tracy H. King, Ronald M. Kaplan, Richard Crouch, John T. Maxwell III, and Mark Johnson. 2002. Parsing the Wall Street Journal using a lexical-functional grammar and discriminative estimation techniques. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 271-278, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Manabu Sassano. 2005. Using a partially annotated corpus to build a dependency parser for Japanese. In Proceedings of the Second International Joint Conference on Natural Language Processing: Full Papers, pages 82-92, Jeju Island, Korea. Springer-Verlag Berlin Heidelberg.
Manabu Sassano and Sadao Kurohashi. 2010. Using smaller constituents rather than sentences in active learning for Japanese dependency parsing. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 356-365, Uppsala, Sweden. Association for Computational Linguistics.
Yikang Shen, Zhouhan Lin, Chin-Wei Huang, and Aaron C. Courville. 2018. Neural language modeling by jointly learning syntax and lexicon. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, British Columbia, Canada. OpenReview.net.
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron C. Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. In Proceedings of the 7th International Conference on Learning Representations, New Orleans, LA, USA. OpenReview.net.
Haoyue Shi, Jiayuan Mao, Kevin Gimpel, and Karen Livescu. 2019. Visually grounded neural syntax acquisition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1842-1861, Florence, Italy. Association for Computational Linguistics.
Benjamin Snyder, Tahira Naseem, and Regina Barzilay. 2009. Unsupervised multilingual grammar induction. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 73-81, Suntec, Singapore. Association for Computational Linguistics.
Anders Søgaard. 2017. Using hyperlinks to improve multilingual partial parsers. In Proceedings of the 15th International Conference on Parsing Technologies, pages 67-71, Pisa, Italy. Association for Computational Linguistics.
Valentin I. Spitkovsky, Hiyan Alshawi, and Daniel Jurafsky. 2011. Punctuation: Making a point in unsupervised dependency parsing. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning, pages 19-28, Portland, Oregon, USA. Association for Computational Linguistics.
Valentin I. Spitkovsky, Daniel Jurafsky, and Hiyan Al-shawi. 2010. Profiting from mark-up: Hyper-text annotations for guided parsing. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1278-1287, Uppsala, Sweden. Association for Computational Linguistics.
Kathrin Spreyer and Jonas Kuhn. 2009. Data-driven dependency parsing of new languages using incomplete and noisy training data. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 12-20, Boulder, Colorado, USA. Association for Computational Linguistics.
Kathrin Spreyer, Lilja Øvrelid, and Jonas Kuhn. 2010. Training parsers on partial trees: A cross-language comparison. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10), pages 1978-1985, Valletta, Malta. European Language Resources Association.
Mitchell Stern, Jacob Andreas, and Dan Klein. 2017. A minimal span-based neural constituency parser. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 818-827, Vancouver, Canada. Association for Computational Linguistics.
Ben Taskar, Vassil Chatalbashev, Daphne Koller, and Carlos Guestrin. 2005. Learning structured prediction models: A large margin approach. In Proceedings of the 22nd International Conference on Machine Learning, pages 896-903, Bonn, Germany. Association for Computing Machinery.
Pengyu Wang and Phil Blunsom. 2013. Collapsed variational Bayesian inference for PCFGs. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 173-182, Sofia, Bulgaria. Association for Computational Linguistics.
Wenhui Wang and Baobao Chang. 2016. Graph-based dependency parsing with bidirectional LSTM. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2306-2315, Berlin, Germany. Association for Computational Linguistics.
Dani Yogatama, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Wang Ling. 2017. Learning to compose words into sentences with reinforcement learning. In Proceedings of the 5th International Conference on Learning Representations, Toulon, France. OpenReview.net.
Daniel H. Younger. 1967. Recognition and parsing of context-free languages in time  $n^3$ . Information and Control, 10(2):189-208.
Yue Zhang, Zhenghua Li, Jun Lang, Qingrong Xia, and Min Zhang. 2017. Dependency parsing with partial annotations: An empirical comparison. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 49-58, Taipei, Taiwan. Asian Federation of Natural Language Processing.
A.1 QA-SRL
He et al.'s (2015) question-answering-driven semantic role labeling dataset (QA-SRL) contains question-answer pairs for 1,241 sentences drawn originally from the training sections of the Penn Treebank (PTB; Marcus et al., 1993). The questions are generated by templates that ask about semantic arguments for all the predicates in a given sentence. Recorded human responses to the questions typically correspond to spans in the sentence. Each question can have multiple answers.
For all question-answer pairs, we first map the answers to consecutive spans in the corresponding sentences. We keep all exact matches when the answer text appears multiple times in the sentence, and we discard any answers that cannot be mapped to a consecutive span in the sentence.
A.2 Wikipedia
We randomly sample  $1\%$  of the articles from the 2020-05-01 snapshot of English Wikipedia<sup>8</sup>. We then split the documents into sentences and tokenize with spaCy.<sup>9</sup> This step leads to 926,077 sentences, as reported in Table 1. For ground-truth parse trees, we parse the sentences with Kitaev et al.'s (2019) state-of-the-art constituency parser trained on the PTB. For all the internal hyperlinks in the documents, where there is a hyperlink-tokenization mismatch, we retrieve the smallest span of tokens that covers the hyperlink. To construct the training set in our main experiments, we filter out sentences longer than 100 tokens and sentences without any multiple-token internal hyperlinks. These pre-processing procedures produce 332,079 training sentences.
Feature Extractor We use the pretrained  $\mathrm{BERT}_{\mathrm{base}}$  model as our feature extractor. For each word in the sentence, we tokenize it with BERT's WordPiece tokenizer, and we take the BERT vector of the last token at the final BERT hidden layer as representation for each word. The feature extractor is fine-tuned along with model training.
Span Scoring MLPleft and MLPright are single-layer MLPs: they both consist of a linear layer projecting BERT representations to 256-dimensional vectors, followed by a leaky ReLU activation function (Maas et al., 2013). The constituent scoring component has parameter  $W \in \mathbb{R}^{257 \times 257}$ . All the parameters are randomly initialized (Glorot and Bengio, 2010).
Training and Optimization We optimize the neural networks using the Adam optimizer (Kingma and Ba, 2015) with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$  and  $\epsilon = 1 \times 10^{-12}$ . For each batch, we sample 8 sentences from the training set and average the loss collected for each sentence. The gradients are clipped at 1.0 before back propagation. The learning rate linearly increases from zero to  $1 \times 10^{-5}$  in 2,000 training steps. After warmup, we keep training the model until we reach 20,000 training steps. We do not perform early stopping, since in the unsupervised parsing setting, we do not look at validation accuracies until we finish training. We leave it as future work to explore other model selection strategies.
Hyperparameter Selection We use the default recommended  $\beta_{1},\beta_{2}$  and  $\epsilon$  values for the Adam optimizer, and we use a typical fine-tuning learning rate for the pre-trained BERT model (Devlin et al., 2019). The number of training steps is based on our preliminary observation of the convergence of the training loss, and the batch size is limited by our computing hardware. We fix the initial values we set for the size of the biaffine matrix  $(257\times 257)$  and the number of warmup steps (2,000) throughout our experiments. A better hyperparameter selection strategy may lead to improved results.
Speed For a length- $n$  sentence, the time complexity for the CKY decoder is  $O(n^3)$ . On a RTX 2080 GPU, our model parses 409 sentences per second on average and the training process for each model finishes within 2 hours.
$^{8}$ https://dumps.wikimedia.org/enwiki/  $^{9}$ https://spacy.io  $^{10}$ Pytorch interface of the model is provided by https: //github.com/huggingface/transformers.