SlowGuess/ACL_2020_OCR huggingface dataset:
Table 1: Results for single models (no pre-training) on the PTB WSJ test set, Section 23.
and unary label classifiers ( $gc$  and  $uc$ ), the hidden dimension of the specific position-wise feedforward networks is 250, while those for pointing functions ( $gp$  and  $sp$ ) have hidden dimensions of 1024. Our model is trained using the Adam optimizer (Kingma and Ba, 2015) with a batch size of 100 sentences. Additionally, we use 100 warm-up steps, within which we linearly increase the learning rate from 0 to the base learning rate of 0.008. Model selection for testing is performed based on the labeled F1 score on the validation set.
Results for Single Models. The experimental results on PTB for the models without pre-training are shown in Table 1. As it can be seen, our model achieves an F1 of 92.78, the highest among the models using top-down inference strategies. Specifically, our method outperforms Stern et al. (2017a) and Shen et al. (2018) by about 1.0 point in F1-score. Notably, our model with LSTM encoder achieves an F1 of 92.26, which is still better than all the top-down parser methods.
On the other hand, while Kitaev and Klein (2018) and Zhou and Zhao (2019) achieve higher F1 score, their inference speed is significantly slower than ours because of the use of CKY based algorithms, which run at  $\mathcal{O}(n^3)$  time complexity for Kitaev and Klein (2018) and  $\mathcal{O}(n^5)$  for Zhou and Zhao (2019). Furthermore, their training objectives involve the use of structural hinge loss, which requires online CKY inference during training. This makes their training time considerably slower than that of our method, which is trained
http://nlp.cs.nyu.edu/evalb/
Model	F1
Our model BERTBASE-uncased	95.34
Our model BERTLARGE-cased	95.48
Kitaev and Klein (2018) ELMO	95.13
Kitaev et al. (2019) BERTLARGE-cased	95.59
directly with span-wise cross entropy loss. In addition, Zhou and Zhao (2019) uses external supervision (head information) from the dependency parsing task. Dependency parsing models, in fact, have a strong resemblance to the pointing mechanism that our model employs (Ma et al., 2018). As such, integrating dependency parsing information into our model may also be beneficial. We leave this for future work.
Results with Pre-training Similar to Kitaev and Klein (2018) and Kitaev et al. (2019), we also evaluate our models with BERT (Devlin et al., 2019) embeddings. Following them in the inclusion of contextualized token representations, we adjust the number of self-attentive layers to 2 and the base learning rate to 0.00005.
As shown in Table 2, our model achieves an F1 score of 95.48, which is on par with the state-of-the-art models. However, the advantage of our method is that it is faster than those methods. Specifically, our model runs at  $\mathcal{O}(n^2)$  worst-case time complexity, while that of Kitaev et al. (2019) is  $\mathcal{O}(n^3)$ . Comparison on parsing speed is discussed in the following section.
Parsing Speed Comparison. In addition to parsing performance in F1 scores, we also compare our parser against the previous neural approaches in terms of parsing speed. We record the parsing timing over 2416 sentences of the PTB test set with batch size of 1, on a machine with NVIDIA GeForce GTX 1080Ti GPU and Intel(R) Xeon(R) Gold 6152 CPU. This setup is comparable to the setup of Shen et al. (2018).
As shown in Table 3, our parser outperforms Shen et al. (2018) by 19 more sentences per second, despite the fact that our parsing algorithm runs at  $\mathcal{O}(n^2)$  worse-case time complexity while the one used by Shen et al. (2018) can theoretically run at  $\mathcal{O}(n\log n)$  time complexity. To elaborate further, the algorithm presented in Shen et al.
Table 2: Restuls on PTB WSJ test set with pretraining.
Model	# sents/sec
Petrov and Klein (2007)	6.2
Zhu et al. (2013)	89.5
Liu and Zhang (2017)	79.2
Stern et al. (2017a)	75.5
Kitaev and Klein (2018)	94.40
Shen et al. (2018)	111.1
Our model	130.2
Table 3: Parsing speed for different models computed on the PTB WSJ test set.
(2018) can only run at  $\mathcal{O}(n^2)$  complexity. To achieve  $\mathcal{O}(n\log n)$  complexity, it needs to sort the list of syntactic distances, which the provided code<sup>2</sup> does not implement. In addition, the speed up for our method can be attributed to the fact that our algorithm (see Algorithm 2) uses a while loop, while the algorithm of Shen et al. (2018) has many recursive function calls. Recursive algorithms tend to be less empirically efficient than their equivalent while/for loops in handling low-level memory allocations and function call stacks.
3.2 SPMRL Multilingual Experiments
Setup. Similar to the English PTB experiments, we use the predicted POS tags from external taggers (provided in the SPMRL datasets). The train-valid/test split is reported in Table 6. For single model evaluation, we use the identical hyperparameters and optimizer setups as in English PTB. For experiments with pre-trained models, we use the multilingual BERT (Devlin et al., 2019), which was trained jointly on 104 languages.
Results. The results for the single models are reported in Table 4. We see that our model achieves the highest F1 score in Basque and Swedish, which are higher than the baselines by 0.52 and 1.37 respective in F1. Our method also performs competitively with the previous state-of-the-art methods on other languages.
Table 5 reports the performance of the models using pre-trained BERT. Evidently, our method achieves state-of-the-art results in Basque and Swedish, and performs on par with the previous best method by Kitaev et al. (2019) in the other five languages. Again, note that our method is considerably faster and easier to train than the
$^{2}$ https://github.com/hantek/distance-parser
Model	Basque	French	German	Hebrew	Hungarian	Korean	Polish	Swedish
(Anders Bjorkelund and Szanto, 2014)	88.24	82.53	81.66	89.80	91.72	83.81	90.50	85.50
(Coavoux and Crabbé, 2017)	88.81	82.49	85.34	89.87	92.34	86.04	93.64	84.0
(Kitaev and Klein, 2018)	89.71	84.06	87.69	90.35	92.69	86.59	93.69	84.45
Our Model	90.23	82.20	84.91	90.63	91.07	85.36	93.99	86.87
Table 4: SPMRL experiment single model test.
Model	Basque	French	German	Hebrew	Hungarian	Korean	Polish	Swedish
(Kitaev et al., 2019)	91.63	87.43	90.20	92.99	94.90	88.80	96.36	88.86
Our model	92.02	86.69	90.28	93.67	94.24	88.71	96.14	89.10
Table 5: SPMRL experiment pre-trained model test (with pretraining).
Language	Train	Valid	Test
Basque	7,577	948	946
French	14,759	1,235	2,541
German	40,472	5,000	5,000
Hebrew	5,000	500	716
Hungarian	8,146	1,051	1,009
Korean	23,010	2,066	2,287
Polish	6,578	821	822
Swedish	5,000	494	666
Table 6: SPMRL Multilingual dataset split.
method of Kitaev et al. (2019).
Prior to the neural tsunami in NLP, parsing methods typically model correlations in the output space through probabilistic context-free grammars (PCFGs) on top of sparse (and discrete) input representations either in a generative regime (Klein and Manning, 2003) or a discriminative regime (Finkel et al., 2008) or a combination of both (Charniak and Johnson, 2005). Beside the chart parser approach, there is also a long tradition of transition-based parsers (Sagae and Lavie, 2005)
Recently, however, with the advent of powerful neural encoders such as LSTMs (Hochreiter and Schmidhuber, 1997), the focus has been switched more towards effective modeling of correlations in the input's latent space, as the output structures are nothing but a function of the input (Gaddy et al., 2018). Various neural network models have been proposed to effectively encode the dense input representations and correlations, and have achieved state-of-the-art parsing results. To enforce the structural consistency, existing neural
Parsing methods either employ a transition-based algorithm (Dyer et al., 2016; Liu and Zhang, 2017; Kitaev and Klein, 2019) or a globally optimized chart-parsing algorithm (Gaddy et al., 2018; Kitaev and Klein, 2018).
Meanwhile, researchers also attempt to convert the constituency parsing problem into tasks that can be solved in alternative ways. For instance, Fernández-González and Martins (2015) transform the phrase structure into a special form of dependency structure. Such a dependency structure, however, requires certain corrections while converting back to the corresponding constituency tree. Gómez and Vilares (2018) and Shen et al. (2018) propose to map the constituency tree for a sentence of  $n$  tokens into a sequence of  $n - 1$  labels or scalars based on the depth or height of the lowest common ancestors between pairs of consecutive tokens. In addition, methods like (Vinyals et al., 2015b; Vaswani et al., 2017) apply the sequence-to-sequence framework to "translate" a sentence into the linearized form of its constituency tree. While being trivial and simple, parsers of this type do not guarantee structural correctness, because the syntax of the linearized form is not constrained during tree decoding.
Our approach differs from previous work in that it represents the constituency structure as a series of pointing representations and has a relatively simpler cross entropy based learning objective. The pointing representations can be computed in parallel, and can be efficiently converted into a full constituency tree using a top-down algorithm. Our pointing mechanism shares certain similarities with the Pointer Network (Vinyals et al., 2015a), but is distinct from it in that our method points a word to another word within the same encoded sequence.
We have presented a novel constituency parsing method that is based on a pointing mechanism. Our method utilizes an efficient top-down decoding algorithm that uses pointing functions for scoring possible spans. The pointing formulation inherently captures global structural properties and allows efficient training with cross entropy loss. With experiments we have shown that our method outperforms all existing top-down methods on the English Penn Treebank parsing task. Our method with pre-training rivals the state-of-the-art method, while being faster than it. On multilingual constituency parsing, it also establishes new state-of-the-art in Basque and Swedish.
We would like to express our gratitude to the anonymous reviewers for their insightful feedback on our paper. Shafiq Joty would like to thank the funding support from his Start-up Grant (M4082038.020).
Agnieszka Falenska Richard Farkas Thomas Mueller Wolfgang Seeker Anders Bjorkelund, Ozlem Cetinoglu and Zsolt Szanto. 2014. The imswroclaw-szeged-cis entry at the spmrl 2014 shared task: Reranking and morphosyntax meet unlabeled data. In Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of NonCanonical Languages, pages 97-102.
Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine n-best parsing and MaxEnt discriminative reranking. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), pages 173-180, Ann Arbor, Michigan. Association for Computational Linguistics.
Maximin Coavoux and Benoit Crabbé. 2017. Multilingual lexicalized constituency parsing with word-level auxiliary tasks. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 331-336, Valencia, Spain. Association for Computational Linguistics.
James Cross and Liang Huang. 2016. Span-based constituency parsing with a structure-label system and provably optimal dynamic oracles. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1-11, Austin, Texas. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Greg Durrett and Dan Klein. 2015. Neural CRF parsing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 302-312, Beijing, China. Association for Computational Linguistics.
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 199-209, San Diego, California. Association for Computational Linguistics.
Daniel Fernández-González and André F. T. Martins. 2015. Parsing as reduction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1523-1533, Beijing, China. Association for Computational Linguistics.
Jenny Rose Finkel, Alex Kleeman, and Christopher D. Manning. 2008. Efficient, feature-based, conditional random field parsing. In Proceedings of ACL08: HLT, pages 959-967, Columbus, Ohio. Association for Computational Linguistics.
David Gaddy, Mitchell Stern, and Dan Klein. 2018. What's going on in neural constituency parsers? an analysis. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 999-1010. Association for Computational Linguistics.
Carlos Gómez, Rodríguez and David Vilares. 2018. Constituent parsing as sequence labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1314-1324. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Nikita Kitaev, Steven Cao, and Dan Klein. 2019. Multilingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3499–3505, Florence, Italy. Association for Computational Linguistics.
Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676–2686, Melbourne, Australia. Association for Computational Linguistics.
Nikita Kitaev and Dan Klein. 2019. Tetra-tagging: Word-synchronous parsing with linear-time inference. CoRR, abs/1904.09745.
Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 423-430, Sapporo, Japan. Association for Computational Linguistics.
Jiangming Liu and Yue Zhang. 2017. Shift-reduce constituent parsing with neural lookahead features. Transactions of the Association for Computational Linguistics, 5:45-58.
Xuezhe Ma, Zeong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018. Stackpointer networks for dependency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1403-1414, Melbourne, Australia. Association for Computational Linguistics.
Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Comput. Linguist., 19(2):313-330.
Slav Petrov and Dan Klein. 2007. Improved inference for unlexicalized parsing. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 404-411, Rochester, New York. Association for Computational Linguistics.
Kenji Sagae and Alon Lavie. 2005. A classifier-based parser with linear run-time complexity. In Proceedings of the Ninth International Workshop on Parsing Technology, pages 125-132. Association for Computational Linguistics.
Djame Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito,inho D. Choi, Richard Farkas, Jennifer Foster, Iakes Goenaga, Koldo Gojenola Galletebeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepiórkowski, Ryan Roth, Wolfgang Seeker, Yannick Versley, Veronika Vincze, Marcin Wolinski, Alina Wróblewska, and Eric Villemonte de la Clergerie. 2013. Overview of the SPMRL 2013
shared task: A cross-framework evaluation of parsing morphologically rich languages. In Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 146-182, Seattle, Washington, USA. Association for Computational Linguistics.
Yikang Shen, Zhouhan Lin, Athul Paul Jacob, Alessandro Sordoni, Aaron Courville, and Yoshua Bengio. 2018. Straight to the tree: Constituency parsing with neural syntactic distance. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1171-1180, Melbourne, Australia. Association for Computational Linguistics.
Mitchell Stern, Jacob Andreas, and Dan Klein. 2017a. A minimal span-based neural constituency parser. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 818-827.
Mitchell Stern, Daniel Fried, and Dan Klein. 2017b. Effective inference for generative neural parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1695-1700, Copenhagen, Denmark. Association for Computational Linguistics.
Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, NAACL '03, pages 173-180. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015a. Pointer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2692-2700. Curran Associates, Inc.
Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015b. Grammar as a foreign language. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2773-2781. Curran Associates, Inc.
Junru Zhou and Hai Zhao. 2019. Head-driven phrase structure grammar parsing on penn treebank. In
Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 2396-2408.
Muhua Zhu, Yue Zhang, Wenliang Chen, Min Zhang, and Jingbo Zhu. 2013. Fast and accurate shift-reduce constituent parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 434-443. Association for Computational Linguistics.
Proof of Proposition 1 Given  $\mathcal{P}(T) = \{(i \to p_i, l_i) : i = 1, \ldots, n-1; i \neq p_i\}$ , generated from tree  $T$  (here we omit the unary leaves and POS-tags), we at first define the inverse  $\mathcal{H}'$  as follows:
$$
\begin{array}{l} \mathcal {H} ^ {\prime} (\mathcal {P} (T)) = \left\{\left(\left(m i n \left(i, p _ {i}\right), m a x \left(i, p _ {i}\right)\right), l _ {i}\right): \right. \\ i = 1, \dots , n - 1 \} \\ \end{array}
$$
We would prove  $\mathcal{H}'(\mathcal{P}(T)) = T$
A binary tree  $T$  has exactly  $n - 1$  internal nodes (or spans). It is noteworthy to mention that for each pointing  $(i\to p_i,l_i)$ $\left(\left(\min (i,p_i),\max (i,p_i)\right),l_i\right)$  is a span in  $T$ . As we consider  $i$  from 1 to  $n - 1$ , there are totally at most n-1 such spans in  $\mathcal{H}'(\mathcal{P}(T))$  (we do not know whether these spans are not be distinct). Therefore, if we can prove that all  $\left(\left(\min (i,p_i),\max (i,p_i)\right),l_i\right)$  spans are distinct for  $i = 1,\ldots ,n - 1$ ,  $\mathcal{H}'(\mathcal{P}(T))$  will cover all the span in  $T$ , therefore,  $\mathcal{H}'(\mathcal{P}(T)) = T$ . We prove this by contradiction.
Assume that there exist  $i,j\in \{1,\ldots ,n - 1\}$  such that  $(\min (i,p_i),\max (i,p_i)) =$ $(\min (j,p_j),\max (j,p_j))$  for  $j\neq i$  .First, if  $p_i = n$  , then according to the above condition,  $(\min (j,p_j),\max (j,p_j)) =$ $(\min (i,n),\max (i,n)) = (i,n)$  . This means, either  $j = n$  or  $j = i$  , which contradicts with our initial assumption that  $j\neq i$  and  $j\in \{1,\dots ,n - 1\}$  . So,  $p_i$  cannot be equal to  $n$  . Similarly, we can prove that  $p_j$  also cannot be equal to  $n$  . Thus, we can conclude that  $p_i,p_j\in \{1,\ldots ,n - 1\}$  . Now, without loss of generality, let us assume that  $j > i$  .With this assumption, the two spans will be identical if and only if  $p_i = j$  and  $p_j = i$  . In this case, the span  $(i,j)$  would be the largest span that starts with  $i$  and ends at  $j$  . However, since  $1\leq i <   j\leq n - 1$  the span  $(i,j)$  must be a left or right child of another (parent) span. If  $(i,j)$  is the left child, then the parent span needs to start with  $i$  , making it
larger than  $(i,j)$ . This contradicts to the property that  $(i,j) = (i,p_i)$  is the largest span that starts or ends at  $i$ . Similarly, if  $(i,j)$  is the right child, then the parent span needs to end at  $j$ , making it larger than  $(i,j)$ . This again contradicts to the property that  $(j,i) = (j,p_j)$  is the largest span that starts or ends at  $j$ .
In conclusion, we have  $\mathcal{H}'(\mathcal{P}(T)) = T$ . This would guarantee that  $\mathcal{H}$  and  $\mathcal{H}'$  are one-to-one: If there exist  $T_1, T_2$  such that  $\mathcal{P}(T_1) = \mathcal{P}(T_2)$ , we would have  $\mathcal{H}'(\mathcal{P}(T_1)) = \mathcal{H}'(\mathcal{P}(T_2))$  or  $T_1 = T_2$ . If there exist  $T_1, T_2$  such that  $\mathcal{H}'(\mathcal{P}(T_1)) = \mathcal{H}'(\mathcal{P}(T_2))$ , we would have  $T_1 = T_2$ .