SlowGuess/ACL_2020_OCR huggingface dataset:
Dev	Test
UAS	LAS	UAS	LAS
PTB
Biaffine17	-	-	95.74	94.08
F&K19	-	-	-	91.59
Li19	95.76	93.97	95.93	94.19
Ji19	95.88	93.94	95.97	94.31
Zhang19	-	-	-	93.96
LOC	95.82	93.99	96.08	94.47
CRF w/o MBR	95.74	93.96	96.04	94.34
CRF	95.76	93.99	96.02	94.33
CRF2O w/o MBR	95.92	94.16	96.14	94.49
CRF2O	95.90	94.12	96.11	94.46
CoNLL09
Biaffine17	-	-	88.90	85.38
Li19	88.68	85.47	88.77	85.58
LOC	89.07	86.10	89.15	85.98
CRF w/o MBR	89.04	86.04	89.14	86.06
CRF	89.12	86.12	89.28	86.18†
CRF2O w/o MBR	89.29	86.24	89.49	86.39
CRF2O	89.44	86.37	89.63‡	86.52‡
NLPCC19
LOC	77.01	71.14	76.92	71.04
CRF w/o MBR	77.40	71.65	77.17	71.58
CRF	77.34	71.62	77.53‡	71.89‡
CRF2O w/o MBR	77.58	71.92	77.89	72.25
CRF2O	78.08	72.32	78.02‡	72.33‡
Table 1: Main results. We perform significance test against LOC on the test data, where “†” means p < 0.05 and “‡” means p < 0.005. Biaffine17: Dozat and Manning (2017); F&K19: Falenska and Kuhn (2019); Li19: Li et al. (2019); Ji19: Ji et al. (2019); Zhang19: Zhang et al. (2019).
sentences per second, which is able to meet the requirements of a real-time system. More discussions on efficiency are presented in Appendix A.
Table 1 lists the main results on the dev and test data. The trends on dev and test are mostly consistent. For a fair comparison with previous works, we only consider those without using extra resources such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019). We can see that our baseline LOC achieves the best performance on both PTB and CoNLL09.
On PTB, both CRF and CRF2O fail to improve
Figure 5: Convergence curves (LAS vs. training epochs) on dev data of PTB, CoNLL09, and NLPCC19.
P	SIB	UCM	LCM
R	F
PTB
LOC	91.16	90.80	90.98	61.59	50.66
CRF	91.24	90.92	91.08	61.92	50.33
CRF2O	91.56	91.11	91.33	63.08	50.99
CoNLL09
LOC	79.20	79.02	79.11	40.10	28.91
CRF	79.17	79.55	79.36	40.61	29.38
CRF2O	81.00	80.63	80.82	42.53	30.09
Table 2: Sub- and full-tree performance on test data.
the parsing accuracy further, probably because the performance is already very high. However, as shown by further analysis in Section 4.3, the positive effect is actually introduced by structural learning and high-order modeling.
On CoNLL09, CRF significantly outperforms LOC, and CRF2O can further improve the performance.
On the partially annotated NLPCC19 data, CRF outperforms LOC by a very large margin, indicating the usefulness of structural learning in the scenario of partial annotation. CRF20 further improves the parsing performance by explicitly modeling second-order subtree features. These results confirm our intuitions discussed in Section 3.4. Please note that the parsing accuracy looks very low because the partially annotated tokens are usually difficult for models.
Impact of MBR decoding. For CRF and CRF20, we by default to perform MBR decoding, which employs the Eisner algorithm over marginal probabilities (Smith and Smith, 2007) to find the best tree.
$$
\boldsymbol {y} ^ {*} = \arg \max  _ {\boldsymbol {y}} \left[ \sum_ {i \rightarrow j \in \boldsymbol {y}} p (i \rightarrow j | \boldsymbol {x}) \right] \tag {13}
$$
Figure 6: LAS on PTB (left) and CoNLL09-test (right) regarding the amount of training data (dependencies vs. sentences).
Table 1 reports the results of directly finding 1-best trees according to dependency scores. Except for PTB, probably due to the high accuracy already, MBR decoding brings small yet consistent improvements for both CRF and CRF2O.
Convergence behavior. Figure 5 compares the convergence curves. For clarity, we plot one data point corresponding to the peak LAS every 20 epochs. We can clearly see that both structural learning and high-order modeling consistently improve the model. CRF2O achieves steadily higher accuracy and converges much faster than the basic LOC.
Performance at sub- and full-tree levels. Beyond the dependency-wise accuracy (UAS/LAS), we would like to evaluate the models regarding performance at sub-tree and full-tree levels. Table 2 shows the results. We skip the partially labeled NLPCC19 data. UCM means unlabeled complete matching rate, i.e., the percent of sentences obtaining whole correct skeletal trees, while LCM further requires that all labels are also correct.
For SIB, we evaluate the model regarding unlabeled adjacent-sibling subtrees (system outputs vs. gold-standard references). According to Equation 6,  $(i,k,j)$  is an adjacent-sibling subtree, if and only if  $w_{k}$  and  $w_{j}$  are both children of  $w_{i}$  at the same side, and there are no other children of  $w_{i}$  between them. Given two trees, we can col
bg	ca	cs	de	en	es	fr	it	nl	no	ro	ru	Avg.
UD2.2
LOC_MST	90.44	91.11	91.04	80.21	86.86	90.67	87.99	91.19	88.24	90.35	86.24	93.01	88.95
LOC	90.45	91.14	90.97	80.02	86.83	90.56	87.76	91.14	87.72	90.74	86.20	93.01	88.88
CRF	90.73	91.25	91.01	80.56†	86.92	90.81†	88.16	91.64†	88.10	90.85	86.50	93.17†	89.14‡
CRF2O	90.77	91.29	91.54†	80.46	87.32†	90.86†	87.96	91.91‡	88.62‡	91.02†	86.90‡	93.33‡	89.33‡
using raw text
Ji19	88.28	89.90	89.85	77.09	81.16	88.93	83.73	88.91	84.82	86.33	84.44	86.62	85.83
CRF2O	89.72	91.27	90.94	78.26	82.88	90.79	86.33	91.02	87.92	90.17	85.71	92.49	88.13
UD2.3
LOC_MST	90.56	91.03	91.98	81.59	86.83	90.64	88.23	91.67	88.20	90.63	86.51	93.03	89.23
LOC	90.57	91.10	91.85	81.68	86.54	90.47	88.40	91.53	88.18	90.65	86.31	92.91	89.19
CRF	90.52	91.19	92.02	81.43	86.88†	90.76†	88.75	91.76	88.08	90.79	86.54	93.16‡	89.32‡
CRF2O	90.76	91.12	92.15‡	81.94	86.93†	90.81‡	88.83†	92.34‡	88.21†	90.78	86.62	93.22‡	89.48‡
using gold POS tags
Zhang19	90.15	91.39	91.10	83.39	88.52	90.84	88.59	92.49	88.37	92.82	84.89	93.11	89.85
CRF2O	91.32	92.57	92.66	84.56	88.98	91.88	89.83	92.94	89.85	93.26	87.39	93.86	90.76
Table 3: LAS on UD2.2 and UD2.3 test datasets. Again,  $\dagger$  and  $\ddagger$  means significance level at  $p < {0.05}$  and  $p < {0.005}$  respectively against the LOC parser.
lect all adjacent-sibling subtrees and compose two sets of triples. Then we evaluate the P/R/F values. Please note that it is impossible to evaluate SIB for partially annotated references.
We can clearly see that by modeling adjacent-sibling subtree scores, the SIB performance obtains larger improvement than both CRF and LOC, and this further contributes to the large improvement on full-tree matching rates (UCM/LCM).
Capability to learn from partial trees. To better understand why CRF20 performs very well on partially annotated NLPCC19, we design more comparative experiments by retaining either a proportion of random training sentences (full trees) or a proportion of random dependencies for each sentence (partial trees). Figure 6 shows the results.
We can see that the performance gap is quite steady when we gradually reduce the number of training sentences. In contrast, the gap clearly becomes larger when each training sentence has less annotated dependencies. This shows that CRF20 is superior to the basic LOC in utilizing partial annotated data for model training.
4.4 Results on Universal Dependencies
Table 3 compares different models on UD datasets, which contain a lot of non-projective trees. We adopt the pseudo-projective approach (Nivre and
Nilsson, 2005) for handling the ubiquitous non-projective trees of most languages. Basically, the idea is to transform non-projective trees into projective ones using more complex labels for post-processing recovery.
We can see that for the basic local parsers, the direct non-projective  $\mathrm{LOC}_{\mathrm{MST}}$  and the pseudoprojective LOC achieve very similar performance.
More importantly, both CRF and CRF2O produce consistent improvements over the baseline in many languages. On both UD2.2 and UD2.3, Our proposed CRF2O model achieves the highest accuracy for 10 languages among 12, and obtains significant improvement in more than 7 languages. Overall, the averaged improvement is 0.45 and 0.29 on UD2.2 and UD2.3 respectively, which is also significant at  $p < 0.005$ .
On average, our CRF2O parser outperforms Ji et al. (2019) by 2.30 on UD2.2 raw texts following CoNLL-2018 shared task setting, and Zhang et al. (2019) by 0.91 on UD2.3 data with gold POS tags. It is noteworthy that the German (de) result is kindly provided by Tao Ji after rerunning their parser with predicted XPOS tags, since their reported result in Ji et al. (2019) accidentally used gold-standard sentence segmentation, tokenization, and XPOS tags. Our CRF2O parser achieves an average LAS of 87.64 using their XPOS tags.
Batchification has been widely used in linear-chain CRF, but is rather complicated for tree structures. Eisner (2016) presents a theoretical proof on the equivalence of outside and back-propagation for constituent tree parsing, and also briefly discusses other formalisms such as dependency grammar. Unfortunately, we were unaware of Eisner's great work until we were surveying the literature for paper writing. As an empirical study, we believe this work is valuable and makes it practical to deploy TreeCRF models in real-life systems.
Falenska and Kuhn (2019) present a nice analytical work on dependency parsing, similar to Gaddy et al. (2018) on constituency parsing. By extending the first-order graph-based parser of Kiperwasser and Goldberg (2016) into second-order, they try to find out how much structural context is implicitly captured by the BiLSTM encoder. They concatenate three BiLSTM output vectors  $(i,k,j)$  for scoring adjacent-sibling subtrees, and adopt max-margin loss and the second-order Eisner decoding algorithm (McDonald and Pereira, 2006). Based on their negative results and analysis, they draw the conclusion that high-order modeling is redundant because BiLSTM can implicitly and effectively encode enough structural context. They also present a nice survey on the relationship between RNNs and syntax. In this work, we use a much stronger basic parser and observe more significant UAS/LAS improvement than theirs. Particularly, we present an in-depth analysis showing that explicitly high-order modeling certainly helps the parsing model and thus is complementary to the BiLSTM encoder.
Ji et al. (2019) employ graph neural networks to incorporate high-order structural information into the biaffine parser implicitly. They add a three-layer graph attention network (GAT) component (Velicković et al., 2018) between the MLP and Biaffine layers. The first GAT layer takes  $\mathbf{r}_i^h$  and  $\mathbf{r}_i^m$  from MLPs as inputs and produces new representation  $\mathbf{r}_i^{h1}$  and  $\mathbf{r}_i^{m1}$  by aggregating neighboring nodes. Similarly, the second GAT layer operates on  $\mathbf{r}_i^{h1}$  and  $\mathbf{r}_i^{m1}$ , and produces  $\mathbf{r}_i^{h2}$  and  $\mathbf{r}_i^{m2}$ . In this way, a node gradually collects multi-hop high-order information as global evidence for scoring single dependencies. They follow the original local head-selection training loss. In contrast, this work adopts global TreeCRF loss and explicitly incorporates high-order scores into the biaffine parser.
Zhang et al. (2019) investigate the usefulness of structural training for the first-order biaffine parser. They compare the performance of local head-selection loss, global max-margin loss, and TreeCRF loss on multilingual datasets. They show that TreeCRF loss is overall slightly superior to max-margin loss, and LAS improvement from structural learning is modest but significant for some languages. They also show that structural learning (especially TreeCRF) substantially improves sentence-level complete matching rate, which is consistent with our findings. Moreover, they explicitly compute the inside and outside algorithms on CPUs via Cython programming. In contrast, this work proposes an efficient second-order TreeCRF extension to the biaffine parser, and presents much more in-depth analysis to show the effect of both structural learning and high-order modeling.
This paper for the first time presents second-order TreeCRF for neural dependency parsing using tri-affine for explicitly scoring second-order subtrees. We propose to batchify the inside algorithm to accommodate GPUs. We also empirically verify that the complex outside algorithm can be implicitly performed via efficient back-propagation, which naturally produces gradients and marginal probabilities. We conduct experiments and detailed analysis on 27 datasets from 13 languages, and find that structural learning and high-order modeling can further enhance the state-of-the-art biaffine parser in various aspects: 1) better convergence behavior; 2) higher performance on sub- and full-tree levels; 3) better utilization of partially annotated data.
The authors would like to thank: 1) the anonymous reviewers for the helpful comments, 2) Wenliang Chen for helpful discussions on high-order neural dependency parsing, 3) Tao Ji for kindly sharing the data and giving beneficial suggestions for the experiments on CoNLL18 datasets, 4) Wei Jiang, Yahui Liu, Haoping Yang, Houquan Zhou and Mingyue Zhou for their help in paper writing and polishing. This work was supported by National Natural Science Foundation of China (Grant No. 61876116, 61525205, 61936010) and a Project Funded by the Priority Academic Program Development (PAPD) of Jiangsu Higher Education Institutions.
Jiong Cai, Yong Jiang, and Kewei Tu. 2017. CRF autoencoder for unsupervised dependency parsing. In Proceedings of EMNLP, pages 1638-1643.
Xavier Carreras. 2007. Experiments with a higher-order projective dependency parser. In Proceedings of EMNLP, pages 957-961.
Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of EMNLP, pages 740-750.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, pages 4171-4186.
Timothy Dozat and Christopher D. Manning. 2017. Deep bioaffine attention for neural dependency parsing. In Proceedings of ICLR.
Timothy Dozat, Peng Qi, and Christopher D. Manning. 2017. Stanford's graph-based neural dependency parser at the CoNLL 2017 shared task. In Proceedings of CoNLL, pages 20-30.
Andrew Drozdov, Patrick Verga, Mohit Yadav, Mohit Iyyer, and Andrew McCallum. 2019. Unsupervised latent tree induction with deep inside-outside recursive auto-encoders. In Proceedings of NAACL, pages 1129-1141.
Jason Eisner. 2000. Bilexical grammars and their cubic-time parsing algorithms. In Advances in Probabilistic and Other Parsing Technologies.
Jason Eisner. 2016. Inside-outside and forward-backward algorithms are just backprop (tutorial paper). In Proceedings of WS, pages 1-17.
Agnieszka Falenska and Jonas Kuhn. 2019. The (non-)utility of structural features in BiLSTM-based dependency parsers. In Proceedings of ACL, pages 117-128.
Jenny Rose Finkel, Alex Kleeman, and Christopher D. Manning. 2008. Efficient, feature-based, conditional random field parsing. In Proceedings of ACL, pages 959-967.
David Gaddy, Mitchell Stern, and Dan Klein. 2018. What's going on in neural constituency parsers? an analysis. In Proceedings of NAACL, pages 999-1010.
Jan Hajic, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antonia Martí, Lluis Marquez, Adam Meyers, Joakim Nivre, Sebastian Padó, Jan Štepanek, Pavel Stránek, Mihai Surdeanu, Nianwen Xue, and Yi Zhang. 2009. The CoNLL-2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proceedings of CoNLL, pages 1-18.
Rebecca Hwa. 1999. Supervised grammar induction using training data with limited constituent information. In Proceedings of ACL, pages 73-79.
Tao Ji, Yuanbin Wu, and Man Lan. 2019. Graph-based dependency parsing with graph neural networks. In Proceedings of ACL, pages 2475-2485.
Xinzhou Jiang, Zhenghua Li, Bo Zhang, Min Zhang, Sheng Li, and Luo Si. 2018. Supervised treebank conversion: Data and approaches. In Proceedings of ACL, pages 2706-2716.
Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of ACL, pages 313-327.
Terry Koo and Michael Collins. 2010. Efficient third-order dependency parsers. In Proceedings of ACL, pages 1-11.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of NAACL, pages 2475-2485.
Phong Le and Willem Zuidema. 2014. The inside-outside recursive neural network model for dependency parsing. In Proceedings of EMNLP, pages 729-739.
Ying Li, Zhenghua Li, Min Zhang, Rui Wang, Sheng Li, and Luo Si. 2019. Self-attentive biaffine dependency parsing. In Proceedings of IJCAI, pages 5067-5073.
Zhenghua Li, Min Zhang, Yue Zhang, Zhanyi Liu, Wenliang Chen, Hua Wu, and Haifeng Wang. 2016. Active learning for dependency parsing with partial annotation. In Proceedings of ACL, pages 344-354.
Xuezhe Ma and Hai Zhao. 2012. Fourth-order dependency parsing. In Proceedings of COLING, pages 785-796.
Ryan McDonald, Koby Crammer, and Fernando Pereira. 2005a. Online large-margin training of dependency parsers. In Proceedings of ACL, pages 91-98.
Ryan McDonald and Fernando Pereira. 2006. Online learning of approximate dependency parsing algorithms. In Proceedings of EACL, pages 81-88.
Ryan McDonald, Fernando Pereira, Kiril Rbarov, and Jan Hajic. 2005b. Non-projective dependency parsing using spanning tree algorithms. In Proceedings of EMNLP, pages 523-530.
Joakim Nivre, Yoav Goldberg, and Ryan McDonald. 2014. Squibs: Constrained arc-eager dependency parsing. CL, pages 249-257.
3304
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of LREC, pages 1659-1666.
Joakim Nivre and Jens Nilsson. 2005. Pseudoprojective dependency parsing. In Proceedings of ACL, pages 99-106.
Wenzhe Pei, Tao Ge, and Baobao Chang. 2015. An effective neural network model for graph-based dependency parsing. In Proceedings of ACL-IJCNLP, pages 313-322.
Xue Peng, Zhenghua Li, Min Zhang, Rui Wang, Yue Zhang, and Luo Si. 2019. Overview of the nlpcc 2019 shared task: cross-domain dependency parsing. In Proceedings of NLPCC, pages 760-771.
Fernando Pereira and Yves Schabes. 1992. Inside-outside reestimation from partially bracketed corpora. In Proceedings of ACL, pages 128-135.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL, pages 2227-2237.
David A. Smith and Noah A. Smith. 2007. Probabilistic models of nonprojective dependency trees. In Proceedings of EMNLP, pages 132-140.
Robert Endre Tarjan. 1977. Finding optimum branchings. Networks, pages 25-35.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In Proceedings of ICLR.
Wenhui Wang and Baobao Chang. 2016. Graph-based dependency parsing with bidirectional LSTM. In Proceedings of ACL, pages 2475-2485.
Xinyu Wang, Jingxian Huang, and Kewei Tu. 2019. Second-order semantic dependency parsing with end-to-end neural networks. In Proceedings of ACL, pages 4609-4618.
Daniel Zeman, Jan Hajic, Martin Popel, Martin Pothast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018. CoNLL 2018 shared task: Multilingual parsing from raw text to universal dependencies. In Proceedings of CoNLL, pages 1-21.
Xingxing Zhang, Jianpeng Cheng, and Mirella Lapata. 2017. Dependency parsing as head selection. In Proceedings of EACL, pages 665-676.
Zhisong Zhang, Xuezhe Ma, and Eduard Hovy. 2019. An empirical investigation of structured output modeling for graph-based neural dependency parsing. In Proceedings of ACL, pages 5592-5598.
A More on Efficiency
Training speed. During training, we greedily find the 1-best head for each word without tree constraints. Therefore, the processing speed is faster than the evaluation phase. Specifically, for LOC, CRF and CRF2O, the average one-iteration training time is about 1min, 2.5min and 3.5min on PTB. In other words, the parser consumes about 700/300/200 sentences per second.
MST decoding. As Dozat et al. (2017) pointed out, they adopted an ad-hoc approximate algorithm which does not guarantee to produce the highest-scoring tree, rather than the ChuLiu/Edmonds algorithm for MST decoding. The time complexity of the ChuLiu/Edmonds algorithm is  $O(n^{2})$  under the optimized implementation of Tarjan (1977). Please see the discussion of McDonald et al. (2005b) for details.
For  $\mathrm{LOC}_{\mathrm{MST}}$ , we directly borrow the MST decoding approach in the original parser of Dozat and Manning (2017).  $\mathrm{LOC}_{\mathrm{MST}}$  achieves 94.43 LAS on PTB-test (inferior to 94.47 of LOC, see Table 1), and its parsing speed is over 1000 sentences per second.
Faster decoding strategy. Inspired by the idea of ChuLiu/Edmonds algorithm, we can further improve the efficiency of the CRF parsing models by avoiding the Eisner decoding for some sentences. The idea is that if by greedily assigning a local max-scoring head word to each word, we can already obtain a legal projective tree, then we omit the decoding process for the sentence. We can judge whether an output is a legal tree (single root and no cycles) using the Tarjan algorithm in  $O(n)$  time complexity. Further, we can judge whether a tree is a projective tree also in a straightforward way very efficiently. In fact, we find that more than  $99\%$  sentences directly obtain legal projective trees on PTB-test by such greedy assignment on marginal probabilities first. We only need to run the decoding algorithm for the left sentences.