SlowGuess/ACL_2020_OCR huggingface dataset:
The best results for the different reranking models on the PTB test set are summarized in table 5. We include in the table the results for reranking the top parse trees of different sizes ( $k = 10,64$ ).  $R e r a n k e r$  is the ranked list produced by the reranking model only.  $M i x t u r e$  is the result for combining the output score given by the rerankers and the score of the base parser as described in §3.3. Following Zhu et al. (2015), we do not use the exact linear equation (6), but do logarithmic scaling of the base parser's score. The parameter  $\alpha$  is optimized based on the results on the development set, which has the same  $k$  as the test set. Since the correct tree is not always in the  $k$ -best list, we also show an upper bound performance for our
<sup>9</sup>We did not do a hyperparameter optimization, but increased the number of parameters in the best RCNN-shared models and observed no significant improvement.
In practice, we do not train on the whole  $k$ -best trees when  $k$  is large, but down-sample  $k$  in each batch to keep the training time efficient. See the appendix for details.
Model	UAS w/o punct.	Δ
Zhu et al. (2015)
Baseline	92.35
Mixture reranker	93.83	+1.48
With oracle	94.16
Ours (Mixture GCNs)
Baseline	92.05
Mixture reranker	93.06	+1.01
With oracle	94.99
rerankers where we manually add the gold trees to the input list (with oracle). Note that with oracle is the result from the reranker, not from the mixture reranking model because the correct tree does not have a score from the base parser if it is not included in the  $k$ -best list.
Combining the score from both the reranker and the base parser consistently improves over the reranking score alone (except for the GCN reranker  $k_{test} = 10$ ), which confirms our hypothesis that the parser and the reranker complement each other by looking at different scoring criteria. Although the accuracy drops when reranking longer lists, the mixture scores are always higher. Compared to the RCNN-shared models, the GCN models benefit less from the mixture models, maybe because the
Table 6: Reproduction of reranking results on the PTB test set for the GCN reranker  $(k = 64)$ .
Model	UAS	LAS
Baseline	90.19	87.90
Top tree	88.36	86.28
IORNN (ktrain = 10)
Reranker (ktest = 10)	89.32	87.16
Mixture (α = 0.91)	89.47	87.41
RCNN-shared (ktrain = 50)
Reranker (ktest = 50)	89.50	86.12
Mixture (α = 0.1)	90.12	87.87
With oracle	92.76	90.06
GCN (ktrain = 50)
Reranker (ktest = 50)	89.96	87.50
Mixture (α = 0.11)	90.33	88.21
With oracle	94.29	92.85
GCNs rank trees more similar to the base parser.
The upper bound performance (with oracle) shows that we can still improve results with a better  $k$ -best list. Interestingly, although we achieve modest improvements compared to Zhu et al. (2015), our upper bound is higher than theirs. A comparison of results with the original RCNN paper on their data is given in table 6.
4.4 Neural Reranking for MRLs
We now evaluate the reranking models that have proved to be effective for English (IORNN, RCNN-shared (+BiLSTMs) and GCNs) on German and Czech data. Note that the RCNN model only ranks unlabeled trees while the other two models also consider the dependency labels, which is particularly important for non-configurational languages. All models are trained with the same hyperparameter settings as for English. The mixture scores are combined using equation 6 except that we optimize the IORNN mixture model using the original tool provided by the authors.
The results for the different reranking models are presented in table 7 and 8. Neither the IORNN nor the RCNN-shared reranker can surpass the baseline. The GCN mixture model is the only model that shows significant improvements over the other models  $(p < .001)$  including the baseline, although small ( $\sim 0.15 - 0.3\%$  LAS).
Taking a closer look at different grammatical functions in the output, we can see a clear differ
Table 7: Performance of different rerankers on the German SPMRL test set.
Model	UAS	LAS
Baseline	91.87	88.85
Top tree	91.02	88.28
IORNN (ktrain = 10)
Reranker (ktest = 10)	91.07	87.97
Mixture (α = 0.94)	91.42	88.54
RCNN-shared (ktrain = 50)
Reranker (ktest = 50)	90.68	86.63
Mixture (α = 0.07)	91.79	88.80
With oracle	93.28	89.99
GCN (ktrain = 50)
Reranker (ktest = 50)	91.12	87.84
Mixture (α = 0.09)	91.89	89.01
With oracle	94.47	92.42
Table 8: Performance of different rerankers on the Czech UD test set.
Label	German	Czech
Baseline	ΔGCNs	Baseline	ΔGCNs
nsubj	89.20	1.48	91.50	0.46
obj	82.84	1.91	90.10	0.54
iobj	67.25	1.15	60.92	2.22
conj	81.78	0.72	74.15	1.23
Table 9: Labeled F1 differences between the baseline and the GCN mixture model for selected dependency types from the German and Czech test sets.
ence between the reranking results and the baseline (table 9). Although the overall accuracy is similar, our reranking results show a better performance for core arguments (nsubj: subject, obj: direct object, iobj: indirect object) and conjunctions (conj).
Through our experiments, we have shown that neural reranking models, which have demonstrated their effectiveness on English data, fail to improve baseline parsing results when applied to German and Czech. This brings us to the question whether this failure is due to the differences between the languages or simply due to the lower quality in the German and Czech  $k$ -best lists that are input to the rerankers. It is conceivable that language-specific properties such as the freer word order and richer morphology in German and Czech might make it harder for our models to learn a good representation capturing the quality of a specific parse tree. However, when we add the correct parse tree to the  $k$ -best list (with oracle results in table 5, 7 and 8), the accuracy goes up to  $94\%$  for English, German and Czech, which effectively eliminates the first reason.
This points to the method used to obtain the  $k$ -
Figure 1: UAS for the GCN reranking mixture model with respect to the gold tree ratio in the  $k$ -best lists.
best list as the main factor responsible for the low results for German and Czech. Beam search, although being straightforward to implement, fails to create high quality  $k$ -best lists for the base parsers used for both languages (§4.1). While several projective parsers support  $k$ -best parsing (Huang and Sagae, 2010; McDonald and Pereira, 2006), there is, to the best of our knowledge, no out-of-the-box parsing system that implements an effective non-projective  $k$ -best parsing algorithm (as, for example, Hall (2007)'s algorithm).
Gold tree ratio Clearly, the (upper bound) tree accuracy in the  $k$ -best list determines the reranking performance. In all datasets, we observe that the accuracy decreases when sentence length increases. Overall, the (unlabeled) tree accuracy in the English  $k$ -best list is  $\sim 5\%$  higher than in the German data, but is behind that in the Czech data. This, however, is not caused by a larger amount of long sentences in the German data. For sentences of same length, the top  $k$  trees from the PTB contain more gold trees than those from the German SPMRL and Czech UD datasets.
We further study the effect of the gold tree ratio for reranking by removing the gold trees from the  $k$ -best list to reduce the ratio to a certain level. Figure 1 shows that the gold tree ratio strongly correlates with the reranking results.
$k$ -best list variation We measure the variation between the trees in the  $k$ -best lists by calculating the standard deviation of their UAS. Figure 2 illustrates the UAS standard deviation distribution in the data for the three languages for  $k = 10$ . In each dataset, the tree UAS variation in the English data is the highest, followed by German and then Czech, which shows that the re-arranging method used to generate German and Czech  $k$ -best trees tends to return more similar trees. We hypothesize
Figure 2: Tree UAS standard deviation of 10-best lists. From left to right: English, German, Czech.
that reranking benefits from diversity, especially if the data contains hard negative examples (incorrect trees that are very similar to the correct one). The gap between reranker performance and with oracle results shows that the reranker is able to detect the correct tree among the incorrect ones because they are very different from each other.
Reranking models Among the neural rerankers, the RCNNs are prone to error propagation from the lower levels, and the IORNNs are sensitive to the order of the child nodes. Both models did not work very well when moving to German and Czech compared to the GCNs, which disregard the top-down or left-to-right order.
In practice, parser output reranking is not a very cost effective way to improve parsing performance, unless we have a fast way to generate high quality output trees. However, the small improvement in core arguments might be useful for downstream applications that require high quality prediction of core arguments.
We have evaluated recent neural techniques for reranking dependency parser output for English, German and Czech and presented a novel reranking model, based on graph convolutional networks (GCNs). We were able to reproduce results for English, using existing rerankers, and showed that our novel GCN-based reranker even outperformed them. However, none of the rerankers works well on the two morphologically rich(er) languages.
Our analysis gave some insights into this issue. We showed that the failure of the rerankers to improve results for German and Czech over the baseline is due to the lower quality of the  $k$ -best lists. Here the gold tree ratio in the  $k$ -best list plays an important role, as the discriminative rerankers are very well able to distinguish the gold trees from other trees in the list, but their performance drops notably when we remove the gold trees from the list. In addition, we observe a higher diversity in the English  $k$ -best list, as compared to German and Czech, which helps the rerankers to learn the differences between high- and low-quality trees.
We conclude that the prerequisite for improving dependency parsing with neural reranking is a diverse  $k$ -best list with a high gold-tree ratio. The latter is much harder to achieve for MRLs where the freer word order and high amount of
non-projectivity result in a larger number of tree candidates, reflected by a lower gold tree ratio.
This work was supported by the Leibniz Science Campus "Empirical Linguistics and Computational Modeling", funded by the Leibniz Association under grant no. SAS-2015-IDS-LWC and by the Ministry of Science, Research, and Art (MWK) of the state of Baden-Württemberg.
Eduard Bejček, Eva Hajičová, Jan Hajič, Pavlina Jínová, Václava Kettnerová, Veronika Kolářová, Marie Mikulová, Jiří Mirovský, Anna Nedoluzhko, Jarmila Panevová, Lucie Poláková, Magda Ševčíková, Jan Štepánek, and Šárka Zikánová. 2013. Prague dependency treebank 3.0. LIN-DAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
Bernd Bohnet. 2010. Top accuracy and fast dependency parsing is not a contradiction. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 89-97, Beijing, China. Coling 2010 Organizing Committee.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine n-best parsing and MaxEnt discriminative reranking. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), pages 173-180, Ann Arbor, Michigan. Association for Computational Linguistics.
Michael Collins and Terry Koo. 2005. Discriminative reranking for natural language parsing. Computational Linguistics, 31(1):25-70.
Timothy Dozat and Christopher D. Manning. 2017. Deep bioaffine attention for neural dependency parsing. In The 5th International Conference on Learning Representations, Toulon, France.
Gertrud Faaß and Kerstin Eckart. 2013. SdeWaC - A corpus of parsable sentences from the web. In Language Processing and Knowledge in the Web: 25th International Conference, GSCL 2013, Darmstadt, Germany, September 25-27, 2013. Proceedings, pages 61-68, Berlin, Heidelberg. Springer.
Keith Hall. 2007. K-best spanning tree parsing. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 392-399, Prague, Czech Republic. Association for Computational Linguistics.
Katsuhiko Hayashi, Shuhei Kondo, and Yuji Matsumoto. 2013. Efficient stacked dependency parsing by forest reranking. Transactions of the Association for Computational Linguistics, 1:139-150.
Liang Huang. 2008. Forest reranking: Discriminative parsing with non-local features. In Proceedings of ACL-08: HLT, pages 586-594, Columbus, Ohio. Association for Computational Linguistics.
Liang Huang and David Chiang. 2005. Better k-best parsing. In Proceedings of the Ninth International Workshop on Parsing Technology, pages 53-64, Vancouver, British Columbia. Association for Computational Linguistics.
Liang Huang and Kenji Sagae. 2010. Dynamic programming for linear-time incremental parsing. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1077-1086, Uppsala, Sweden. Association for Computational Linguistics.
Phong Le and Willem Zuidema. 2014. The inside-outside recursive neural network model for dependency parsing. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 729-739, Doha, Qatar. Association for Computational Linguistics.
Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 302-308, Baltimore, Maryland. Association for Computational Linguistics.
Diego Marcheggiani and Ivan Titov. 2017. Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1506-1515, Copenhagen, Denmark. Association for Computational Linguistics.
Ryan McDonald and Fernando Pereira. 2006. Online learning of approximate dependency parsing algorithms. In 11th Conference of the European Chapter of the Association for Computational Linguistics.
Thomas Mueller, Helmut Schmid, and Hinrich Schütze. 2013. Efficient higher-order CRFs for morphological tagging. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 322-332, Seattle, Washington, USA. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Vikas Raunak, Vivek Gupta, and Florian Metze. 2019. Effective dimensionality reduction for word embeddings. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 235–243, Florence, Italy. Association for Computational Linguistics.
Djame Seddah, Sandra Kübler, and Reut Tsarfaty. 2014. Introducing the SPMRL 2014 shared task on parsing morphologically-rich languages. In Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of Non-Canonical Languages, pages 103–109, Dublin, Ireland. Dublin City University.
Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013. Parsing with compositional vector grammars. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 455-465, Sofia, Bulgaria. Association for Computational Linguistics.
Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2010. Learning continuous phrase representations and syntactic parsing with recursive neural networks. In Proceedings of the NIPS 2010 Deep Learning and Unsupervised Feature Learning Workshop.
Hao Zhou, Yue Zhang, Shujian Huang, Junsheng Zhou, Xin-Yu Dai, and Jiajun Chen. 2016. A search-based dynamic reranking model for dependency parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1393-1402, Berlin, Germany. Association for Computational Linguistics.
Chenxi Zhu, Xipeng Qiu, Xinchi Chen, and Xuanjing Huang. 2015. A re-ranking model for dependency parser with recursive convolutional neural network. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1159-1168, Beijing, China. Association for Computational Linguistics.
A Appendix: Training Details for Neural Network Rerankers
A.1 Down-sampling the  $k$ -best list
In order to maintain an efficient run-time for our discriminative rerankers without scarifying the diversity we get from a longer  $k$ -best list, we apply down-sampling for each training instance. Namely, in each step, we use only 10 randomly selected trees (when  $k > 10$ ) in addition to the gold tree for each sentence to back-propagate.
A.2 IORNN reranker
We use the code provided by Le and Zuidema (2014) $^{10}$  to train all IORNN rerankers with default hyperparameters for English, German and Czech. The default number of training epochs is set to 50. Due to time limit, we could only train the model for Czech which is the largest of our datasets up to 27 epochs, which took 15 days on a CPU. The program processes a single sentence at a time rather than batching or multithreading. For computing the mixture score, we use the tool provided in the repository instead of ours. The authors do logarithmic scaling for the score of the reranker in the mixture model combination:
$$
s (x, y) = \alpha \log s _ {r} (x, y, \Theta) + (1 - \alpha) s _ {b} (x, y)
$$
A.3 RCNN, RCNN-shared and GCN rerankers
For all discriminative rerankers, in the experiment with the English data, we do logarithmic scaling for the score of the base parser in the mixture model combination:
$$
s (x, y) = \alpha s _ {r} (x, y, \Theta) + (1 - \alpha) \log s _ {b} (x, y)
$$
In the experiments with German and Czech data, we do not scale the score of the base parser and use equation 6.
10https://github.com/lephong/ iornn-depparse