SlowGuess/ACL_2020_OCR huggingface dataset:
spans, PSN model only needs to consider few spans, which is more local and it is enough for the perspan-normalization to handle this situation. For long spans, our model needs to do normalization on more spans and the state space becomes large linearly. So the accuracy decreases fast, and there is no advantage compared with PSN model which uses CKY algorithm for inference. For spans of other lengths, our locally normalized method can take all spans with the same right boundary into consideration and add sum-to-one constraints on their scores. As a result, our model outperforms PSN model even without the help of accurate inference.
Influence of Constituent Type Then we compare the accuracy of different constituent types. Table 1 shows the results of nine types which occur most frequently. Our model all performs better
Table 2: Final results on the PTB test set.  $\spadesuit$  means the models use Transformer as their encoder.  $\heartsuit$  means generative models.
Model	LR	LP	F1
Global Model
Kitaev and Klein (2018a)	86.8	88.1	87.4
Zhou and Zhao (2019)	89.4	90.1	89.7
Local Model
Dyer et al. (2016)	-	-	84.6
Liu et al. (2018)	-	-	85.4
Liu and Zhang (2017b)	85.2	85.9	85.5
Vilares et al. (2019)	-	-	85.6
Liu and Zhang (2017a)	-	-	86.1
Shen et al. (2018)	86.4	86.6	86.5
Fried and Klein (2018)	-	-	87.0
Teng and Zhang (2018)	87.1	87.5	87.3
Our Model	87.9	89.3	88.6
Our Model	87.4	89.9	88.7
Pre-training/Ensemble/Re-ranking
Kitaev and Klein (2018b)	91.6	92.0	91.8
Our Model (+BERT)	91.7	92.4	92.0
Our Model (+BERT)	91.9	92.3	92.1
Table 3: Final results on the CTB test set.  $\spadesuit$  means the models use Transformer as their encoder. Note that Zhou and Zhao (2019) uses gold POS tags in their code, so we rerun their code using predicted POS tags for fair comparison.
Model	LR	LP	F1
Full model	92.31	92.87	92.59
- MLPl and MLPr	92.15	92.72	92.43
- normalization	91.25	92.93	92.08
+ label linearization	90.79	91.56	91.17
Table 4: Ablation test on the PTB test set. Here we use the same settings as in Section 4.3.
than PSN model, especially in types SBAR, ADJP and QP. When optimizing the representation of one split point, our model can consider all of the words before it, which can be helpful to predict some types. For example, when we predict an adjective phrase (ADJP), its representation has fused the words' information before it (e.g. linking verb like "is"), which can narrow the scope of prediction.
We perform several ablation experiments by modifying the structure of the decoder layer. The results are shown in Table 4.
First, we delete the two different split point representations described in Equation (2) and directly use the output of LSTM as the final representation. Final performance slightly decreases, which indi
Inference Algorithm	LR	LP	F1
G(i, j)	92.31	92.87	92.59
k = max{k' | dk' ≤ i}	92.39	92.75	92.57
k = arg min{k' | dk'}	91.93	93.21	92.57
Table 5: Results of different inference algorithms described in Section 3.4.
Model	sents/sec
Global Model
Stern et al. (2017a)	20
Kitaev and Klein (2018a)♦(w. Cython)	150
Zhou and Zhao (2019)♣(w. Cython)	159
Local Model
Teng and Zhang (2018)	22
Stern et al. (2017a)	76
Liu and Zhang (2017b)	79
Shen et al. (2018)	111
Shen et al. (2018) (w/o tree inference)	351
Vilares et al. (2019)	942
Our Model	220
Our Model♦	155
Table 6: Parsing speeds on the PTB test set.  $\spadesuit$  means the models use Transformer as their encoders. "w. Cython" stands for using Cython to optimize the python code. "w/o tree inference" stands for evaluating without tree inference. The model in Kitaev and Klein (2018a) is ran by ourselves, and other speeds are extracted from their original papers.
cates that distinguishing the representations of left and right boundaries of a span is really helpful.
Then we delete the local normalization on partial spans and only calculate the probability of each span to be a left child. The inference algorithm is the same as our full model. Final result decreases by 0.5 F1, despite improvement on precision. This might be because our normalization method can add constraints on all the spans with the same right boundary, which makes it effective when only one span is correct.
Finally, we try to predict the labels sequentially, which means assigning each split  $i$  a tuple  $(d_i, \ell_i^{\text{left}}, \ell_i^{\text{right}})$ , where  $\ell_i^{\text{left}}$  and  $\ell_i^{\text{right}}$  represent the labels of the longest spans ending and starting with  $i$  in the tree, respectively. This may make our model become a sequence labeling model similar to Gómez-Rodríguez and Vilares (2018). However, the performance is very poor, and this is largely due to the loss of structural information in the label prediction. Therefore, how to balance efficiency and label prediction accuracy might be a research
problem in the future.
4.5 Inference Algorithms
We compare three inference algorithms described in Section 3.4. The results are shown in Table 5. We find that different inference algorithms have no obvious effect on the performance, mainly due to the powerful learning ability of our model. Thus we use the third method which is the most convenient to implement.
4.6 Parsing Speed
The parsing speeds of our parser and other parsers are shown in Table 6. Although our inference complexity is  $\mathcal{O}(n\log n)$ , our speed is faster than other local models, except Shen et al. (2018) which evaluates without tree inference and Vilares et al. (2019) which utilizes a pure sequence tagging framework. This is mainly due to the simplicity of our model and the parallelism of matrix operations for structure prediction. Compared with globally normalized parsers like Zhou and Zhao (2019) and Kitaev and Klein (2018a), our model is also faster even if they use optimization for python code (e.g. Cython 6). Other global model like Stern et al. (2017a) which infers in  $O(n^3)$  complexity is much slower than ours, and this shows the superiority of our linearization in speed.
Globally normalized parsers often have high performance on constituent parsing due to their search on the global state space (Stern et al., 2017a; Kitaev and Klein, 2018a; Zhou and Zhao, 2019). However, they suffer from high time complexity and are difficult to parallelize. Thus many efforts have been made to optimize their efficiency (Vieira and Eisner, 2017).
Recently, the rapid development of encoders (Hochreiter and Schmidhuber, 1997; Vaswani et al., 2017) and pre-trained language models (Devlin et al., 2018) have enabled local models to achieve similar performance as global models. Teng and Zhang (2018) propose two local models, one does normalization on each candidate span and one on each grammar rule. Their models even outperform the global model in Stern et al. (2017a) thanks to the better representation of spans. However, they still need an  $\mathcal{O}(n^3)$  complexity inference algorithm to reconstruct the final parsing tree.
$^{6}$ https://cython.org/
Meanwhile, many work do research on faster sequential models. Transition-based models predict a sequence of actions and achieve an  $\mathcal{O}(n)$  complexity (Watanabe and Sumita, 2015; Cross and Huang, 2016; Liu and Zhang, 2017a). However, they suffer from the issue of error propagation and cannot be parallel. Sequence labeling models regard tree prediction as sequence prediction problem (Gomez-Rodriguez and Vilares, 2018; Shen et al., 2018). These models have high efficiency, but their linearizations have no direct relation to the spans, so the performance is much worse than span-based models.
We propose a novel linearization method closely related to the spans and decode the tree in  $\mathcal{O}(n\log n)$  complexity. Compared with Teng and Zhang (2018), we do normalization on more spans, thus achieve a better performance.
In future work, we will apply graph neural network (Velickovic et al., 2018; Ji et al., 2019; Sun et al., 2019) to enhance the span representation. Due to the excellent properties of our linearization, we can jointly learn constituent parsing and dependency parsing in one graph-based model. In addition, there is also a right linearization defined on the set of right child spans. We can study how to combine the two linear representations to further improve the performance of the model.
In this work, we propose a novel linearization of constituent trees tied on the spans tightly. In addition, we build a new normalization method, which can add constraints on all the spans with the same right boundary. Compared with previous local normalization methods, our method is more accurate for considering more span information, and reserves the fast running speed due to the parallelizable linearization model. The experiments show that our model significantly outperforms existing local models and achieves competitive results with global models.
The authors would like to thank the reviewers for their helpful comments and suggestions. The authors would also like to thank Tao Ji and Changzhi Sun for their advices on models and experiments. The corresponding author is Yuanbin Wu. This research is (partially) supported by STCSM (18ZR1411500), the Foundation of
State Key Laboratory of Cognitive Intelligence, iFLYTEK(COGOS-20190003), and an open research fund of KLATASDS-MOE.
Do Kook Choe and Eugene Charniak. 2016. Parsing as language modeling. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2331-2336. The Association for Computational Linguistics.
James Cross and Liang Huang. 2016. Span-based constituency parsing with a structure-label system and provably optimal dynamic oracles. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 1-11.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. Open-Review.net.
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In *NAACL HLT* 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 199-209.
Daniel Fried and Dan Klein. 2018. Policy gradient as a proxy for dynamic oracles in constituency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 469-476.
Daniel Fried, Mitchell Stern, and Dan Klein. 2017. Improving neural parsing by disentangling model combination and reranking effects. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 2: Short Papers,
pages 161-166. Association for Computational Linguistics.
David Gaddy, Mitchell Stern, and Dan Klein. 2018. What's going on in neural constituency parsers? an analysis. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 999-1010.
Carlos Gómez-Rodríguez and David Vilares. 2018. Constituent parsing as sequence labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 1314-1324.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Juneki Hong and Liang Huang. 2018. Linear-time constituency parsing with rnns and dynamic programming. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 477-483.
Tao Ji, Yuanbin Wu, and Man Lan. 2019. Graph-based dependency parsing with graph neural networks. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 2475-2485. Association for Computational Linguistics.
Nikita Kitaev and Dan Klein. 2018a. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 2675-2685.
Nikita Kitaev and Dan Klein. 2018b. Multilingual constituency parsing with self-attention and pre-training. CoRR, abs/1812.11760.
Ying Li, Zhenghua Li, Min Zhang, Rui Wang, Sheng Li, and Luo Si. 2019. Self-attentive biaffine dependency parsing. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 5067-5073. ijcai.org.
Wang Ling, Chris Dyer, Alan W. Black, and Isabel Trancoso. 2015. Two/too simple adaptations of word2vec for syntax problems. In *NAACL HLT* 2015, The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Colorado, USA, May 31 - June 5, 2015, pages 1299-1304. The Association for Computational Linguistics.
Jiangming Liu and Yue Zhang. 2017a. In-order transition-based constituent parsing. Transactions of the Association for Computational Linguistics, 5:413-424.
Jiangming Liu and Yue Zhang. 2017b. Shift-reduce constituent parsing with neural lookahead features. Transactions of the Association for Computational Linguistics, 5:45-58.
Lemao Liu, Muhua Zhu, and Shuming Shi. 2018. Improving sequence-to-sequence constituency parsing. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 4873-4880. AAAI Press.
Chunpeng Ma, Akihiro Tamura, Lemao Liu, Tiejun Zhao, and Eiichiro Sumita. 2017. Improving feature-rich transition-based constituent parsing using recurrent neural networks. IEICE Transactions, 100-D(9):2205-2214.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. Computational Linguistics, 19(2):313-330.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar; A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532-1543. ACL.
Yikang Shen, Zhouhan Lin, Athul Paul Jacob, Alessandro Sordoni, Aaron C. Courville, and Yoshua Bengio. 2018. Straight to the tree: Constituency parsing with neural syntactic distance. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 1171-1180.
Mitchell Stern, Jacob Andreas, and Dan Klein. 2017a. A minimal span-based neural constituency parser. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 818-827.
Mitchell Stern, Daniel Fried, and Dan Klein. 2017b. Effective inference for generative neural parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 1695-1700.
Changzhi Sun, Yeyun Gong, Yuanbin Wu, Ming Gong, Daxin Jiang, Man Lan, Shiliang Sun, and Nan Duan. 2019. Joint type inference on entities and relations
via graph convolutional networks. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 1361-1370. Association for Computational Linguistics.
Zhiyang Teng and Yue Zhang. 2018. Two local models for neural constituent parsing. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pages 119-132.
Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2003, Edmonton, Canada, May 27 - June 1, 2003. The Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 6000-6010.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Tim Vieira and Jason Eisner. 2017. Learning to prune: Exploring the frontier of fast and accurate parsing. TACL, 5:263-278.
David Vilares, Mostafa Abdou, and Anders Søgaard. 2019. Better, faster, stronger sequence tagging constituent parsers. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 3372-3383.
Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey E. Hinton. 2015. Grammar as a foreign language. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2773-2781.
Taro Watanabe and Eiichiro Sumita. 2015. Transition-based neural constituent parsing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing
of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers, pages 1169-1179.
Junru Zhou and Hai Zhao. 2019. Head-driven phrase structure grammar parsing on penn treebank. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 2396-2408.