SlowGuess/ACL_2020_OCR huggingface dataset:
Even more drastically, Ko routinely expresses this semantics by using an existential construction with the literal meaning ' (for) X there was a possibility of doing Y' (instead of X could do Y), which converts the En aux into nsubj+acl. In this case, a tree transformation seems to be sufficient to bridge this divergence.
5.4 nmod  $\rightarrow$  acl+X in Korean
Ko also differs from other languages in the extent that it uses relative clauses for nominal modification. Table 2 shows that nmod has a high percentage for "other" mappings (48%). Investigation of this long tail shows that to a large extent it consists of acl-based constructions: acl+advmod, acl+nsubj, acl+obj. Added to acl, the cumulative share of acl-based constructions is on par with compound, the main correspondent of this relation for non-possessive nmod (possessive nmod are the only ones that map to nmod in Ko). This discrepancy is due to the fact that Ko nearly
obligatorily adds contextually-predictable predicates to oblique relations such as actions [taken] in Crimea or people [being] without children. The Korean PUD does not demote these verbs to functional-word status (such an approach is advocated for in [Gerdes and Kahane, 2016]) but turns them into clause-leading verbs, thus yielding an  $\mathsf{acl} + \mathsf{X}$  divergence. $^8$
Language	ru	fr	zh	ko	jp
Thematic	Full	25	4	8	5	5
nsubj to obj/obl	78	57	43	25	53
Promotional	0	0	0	0	0
Demotional	10	2	4	19	1
Structural	83	67	17	0	35
Conflational	10	5	5	6	2
Categorical	nsubj+obj	8	12	23	11	4
nsubj+(i) obj/obl	51	34	25	11	13
#Sentences	995	999	999	884	999
Table 1: Prevalence in absolute counts of translation divergences as defined in Dorr (1994). Row headings are explained in § 6.
6 Revisiting Dorr's Divergences
We turn to show that the types of the seminal classification of divergences from Dorr (1994) can be, with a single exception, recast in terms of UD divergences. We quote the original formulations of the divergences illustrated through English-Spanish or English-German examples.
Thematic divergence: E: I like Mary  $\Leftrightarrow$  S: María me gusta a mi 'Mary pleases me.' In UD, this corresponds to the situation when the original obj or obl becomes the nsubj and vice versa. The divergence will correspond to a CSR of type (nsubj, obj) or (nsubj, obl). A "full" thematic divergence will also involve the inverse divergence (obj, subj) or (obl, subj).
The list of examples we can discuss goes on. For example, while investigating the cross-linguistic patterning of English advcl, we noticed that it often gets mapped to ccomp in French and acl in Russian. Both divergent annotations seem to be erroneous as the sentences they appear in are covered by the definition of advcl provided in the UD manual. However, the French case is interesting in that the source advcl in question can be characterized semantically: instead of denoting a secondary action, they reflect a sequence of events or parallel scenes (e.g., Columbus sailed across the Atlantic... sparking a period of European exploration of the Americas). Another problem is presented by multi-word expressions analyzed as proper nouns where all tokens have the same POS tag. The UD manual advises to retain the original parts of speech in proper nouns consisting of phrases (e.g., Cat on a Hot Tin Roof) but allows to treat "words that are etymologically adjectives" as PROPN in names such as the Yellow Pages. When such names are translated, PROPN get reanalyzed as ADJ, NOUN, etc., producing spurious CLMD.
$^{7}$ Cf. also 23 En nsubj edges mapped to Ru nsubj+obl. Inspection of these sentences reveals that the CLMD can be ascribed to metaphorical usage (e.g., the sense of read employed in the post reads has no direct correspondent in Ru). Some such cases can be disambiguated using existing annotation schemes.
Promotional divergence: E: John usually goes home  $\Leftrightarrow$  S: Juan suele ir a casa 'John tends to go home.' This corresponds to the situation where the original root predicate becomes an xcomp, and the original advmod takes its place as the root. Corresponding CSR type: (advmod, xcomp).
Demotional divergence: E: I like eating  $\Leftrightarrow$  G: Ichesse gern 'I eat likingly.' The original xcomp becomes the root predicate, and the original root predicate is demoted to the position of an advmod. The relevant CSR type is (xcomp, advmod).
Structural divergence: E: John entered the house  $\Leftrightarrow$  S: Juan entro en la casa 'John entered in the house.' The original obj becomes an obl. CSR type: (obj, obl).
Conflational divergence: E: I stabbed John  $\Leftrightarrow$  S: Yo le di punaladas a Juan 'I gave knife-wounds to John.' The original root predicate is in a one-to-many alignment with a combination of a root predicate and its obj.
Categorical divergence: E: I am hungry  $\Leftrightarrow$  G: Ich habe Hunger 'I have hunger.' The original root predicate becomes an obj. CSR type: (nsubj, nsubj + obj).
Lexical divergence: E: John broke into the room  $\Leftrightarrow$  S: Juan forzo la entrada al cuarto 'John forced (the) entry to the room.' Divergences of this type arise whenever aligned words have at best partially overlapping semantic content and never appear on their own but always with other divergences. The information necessary to ascertain the degree of word-meaning overlap is not embedded into UD or any other cross-lingual annotation scheme; therefore we were unable to provide a formal interpretation of this type of divergence.
Frequencies of Dorr's Divergences in PUD are presented in Table 1 (except for Lexical divergences, which are hard to formalize). It is evident that these types only account for a small portion of the encountered divergences, the point already made for En-Zh in DX17. It seems then that "hand-crafted" translation divergences, however insightful they may be, receive attention disproportionate to their empirical frequency.
7 Perspectives for Automation
One of the strengths of our approach is that it only relies on UD parses and alignments, for which automatic tools exist in many languages. To demonstrate the feasibility of an automated protocol, we conducted an analysis of the WMT14 En-Zh News Commentary corpus.<sup>9</sup> We used TsinghuaAligner (Liu and Sun, 2015) and pretrained English and Chinese UD parsers from the StanfordNLP toolkit (Qi et al., 2018). To verify that the aligner we are using is adequate for the task, we aligned the En-Zh PUD corpus pair and checked the resulting precision and recall of the edges corresponding to content words.<sup>10</sup> The results  $(\mathrm{P} = 0.86, \mathrm{R} = 0.32)$  indicate that the automated approach is able to recover around a third of the information obtained through manual alignment with reasonable precision. Importantly, we find recall to be nearly uniform for all source edge types, which suggests that the low recall can be mitigated by using a larger corpus without biasing the results.
The POS and edge-type confusion matrices built from this experiment are very similar to the ones reported in this paper (save for compound, which is not produced by the Stanford Zh parser), and are not reproduced here (they can be found in the Supplementary Materials).
8 Applicability for Zero-Shot Parsing
We come to demonstrate the applicability of our method for analyzing the performance of a downstream cross-lingual transfer task. We consider zero-shot cross-lingual parsing (Ammar et al., 2016; Schuster et al., 2019) as a test case and investigate to what extent the performance of a zero-shot parser on a given dependency label can be predicted from its stability in translation. As test sets we use the test sets of GSD UD corpora for the five languages (Ru, Fr, Zh, Ko, and Jp), as well as the corresponding PUD corpora. We train a parser following the setup of Mulcaire et al. (2019) and use a pretrained multilingual BERT (Devlin et al., 2019), feeding its output embeddings into a biaffine-attention neural UD parser (Dozat and Manning, 2017) trained on the English EWT corpus. We evaluate the parser's ability to predict relation types by computing F-scores for each de
9http://www.statmt.org/wmt14/ 10These were defined here as edges with the following labels: root, nsubj, amod, nmod, advmod, nummod, acl, advcl, xcomp, compound, flat, obj, obl.
pendency label (save for labels corresponding to function words that were generally not aligned). Appendix E gives full implementation details.
We start by computing Spearman correlations between F-scores and the PRESERVATION indices, defined as the proportion of identity mappings in the confusion matrices for each corpus (e.g., PRESERVATION for acl in Ru is 0.48, while in Jp it is 0.37). The correlations are very strong for some languages and noticeable for others ( $\rho = 0.62, 0.75, 0.31, 0.42, 0.77$  for Ru, Fr, Zh, Ko, and Jp respectively on GSD test sets, and  $\rho = 0.7, 0.82, 0.72, 0.84, 0.68$  on PUD).
We hypothesize that the preservation of a relation in translation is related to the ability of a zero-shot parser to predict it. In order to control for obvious covariates, we introduce two control variables: (1) SOURCE-SIDE HARDNESS (test-set F-scores attained by the parser on English dependency relations) and (2) TARGET-SIDE HARDNESS (F-scores attained by a parser trained on the target-language UD GSD corpus on the target-language test set). We use a mixed-effects model with PRESERVATION, SOURCE-SIDE HARDNESS, and TARGET-SIDE HARDNESS as fixed effects, random intercepts for language, and F-scores for dependency relations as the dependent variable. We then used likelihood-ratio test to compute  $p$ -values for the difference in predictive power between the model without PRESERVATION and one with it. The  $p$ -value (using Holm correction) is highly significant ( $< 0.001$ ) for the PUD corpora, and for GSD it is significant with  $p = 0.02$ .
These results suggest that morphosyntactic differences between languages, as uncovered by our method, play a role in the transferability of parsers across languages. This also underscores the potential utility of bridging CLMD for improving syntactic transfer across languages.
The presented methodology gives easy access to different levels of analysis. On one hand, by focusing on content words, the approach abstracts away from much local-syntactic detail (such as reordering or adding/removing function words). At the same time, the methodology and datasets provide means to investigate essentially any kind of well-defined CLMD. Indeed, since function words in UD tend to be dependents of content words, we may analyze the former by considering the distri
bution of function word types that each type of content word has. Moreover, sub-typing dependency paths based on their linear direction can allow investigating word-order differences.[11]
Other than informing the development of crosslingual transfer learning, our analysis directly supports the validation of UD annotation. For example, we reveal inconsistencies in the treatment of multi-word expressions across languages. Thus, the translation of many NPs with adjectival modifiers, such as Celtic sea or episcopal church, are analyzed as compounds. Languages such as Ru, lacking a truly productive nominal-compound relation, carve this class up based mostly on the POS of the dependent element (e.g., episcopal corresponds to a Ru amod), its semantic class (e.g., compounds with cardinal directions are Ru amods), and whether the dependent element itself has dependents (these mostly correspond to Ru nmods). Our method can be used to detect and bridge such inconsistencies.
In conclusion we note that our analysis suggests that considerable entropy in the mapping between the syntactic relations of the source and target sides can be reduced by removing inconsistencies in the application of UD, and perhaps more importantly by refining UD with semantic distinctions that will normalize corresponding constructions across languages to have a similar annotation. This will simultaneously advance UD's stated goal of "bringing out cross-linguistic parallelism across languages" and, as our results on zero-shot parsing suggest, make it more useful for cross-linguistic transfer.
We thank Nathan Schneider for helpful comments and anonymous reviewers for useful feedback. This work was supported by the Israel Science Foundation (grant no. 929/17).
<sup>11</sup>Surface-Syntactic Universal Dependencies may be a better fit for such type of analysis, see (Gerdes et al., 2018).
Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2016. Many languages, one parser. Transactions of the Association for Computational Linguistics, 4:431-444.
Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Simaan. 2017. Graph convolutional encoders for syntax-aware neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1957-1967, Copenhagen, Denmark. Association for Computational Linguistics.
Alexandra Birch, Miles Osborne, and Philipp Koehn. 2008. Predicting success in machine translation. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 745-754.
Hans Boas, editor. 2010. Contrastive studies in construction grammar. John Benjamins.
Marine Carpuat, Yogarshi Vyas, and Xing Niu. 2017. Detecting cross-lingual semantic divergence for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 69-79.
Marco Damonte and Shay B. Cohen. 2018. Cross-lingual abstract meaning representation parsing. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1146–1155. Association for Computational Linguistics.
Dun Deng and Nianwen Xue. 2017. Translation divergences in chinese-english machine translation: An empirical investigation. Computational Linguistics, 43(3):521-565.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Bonnie J. Dorr. 1994. Machine translation divergences: a formal description and proposed solution. Computational linguistics, 20(4):597-635.
Bonnie J Dorr, Rebecca J Passonneau, David Farwell, Rebecca Green, Nizar Habash, Stephen Helmreich, Eduard Hovy, Lori Levin, Keith J Miller, Teruko Mitamura, et al. 2010. Interlingual annotation of parallel text corpora: a new framework for annotation and evaluation. Natural Language Engineering, 16(3):197-243.
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open-Review.net.
Jacek Fisiak, editor. 1984. Contrastive linguistics: Prospects and problems. Mouton.
Kim Gerdes, Bruno Guillaume, Sylvain Kahane, and Guy Perrier. 2018. SUD or Surface-Syntactic Universal Dependencies: An annotation scheme near-isomorphic to UD. In Universal Dependencies Workshop 2018, Brussels, Belgium.
Kim Gerdes and Sylvain Kahane. 2016. Dependency annotation choices: Assessing theoretical and practical issues of Universal Dependencies. In Proceedings of the 10th Linguistic Annotation Workshop.
Rebecca Hwa, Philip Resnik, Amy Weinberg, Clara Cabezas, and Okan Kolak. 2005. Bootstrapping parsers via syntactic projection across parallel text. Natural Language Engineering, 11:311-325.
Sylvain Kahane, Marine Courtin, and Kim Gerdes. 2017. Multi-word annotation in syntactic treebanks-propositions for universal dependencies. In Proceedings of the 16th International Workshop on Treebanks and Linguistic Theories, pages 181-189.
Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1318-1326, Portland, Oregon, USA. Association for Computational Linguistics.
Mikhail Kozhevnikov and Ivan Titov. 2013. Cross-lingual transfer of semantic role labeling models. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1190–1200, Sofia, Bulgaria. Association for Computational Linguistics.
Yang Liu and Maosong Sun. 2015. Contrastive unsupervised word alignment with non-local features. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAIåž15, page 2295åžS2301. AAAI Press.
Ryan McDonald, Slav Petrov, and Keith Hall. 2011. Multi-source transfer of delexicalized dependency parsers. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 62-72, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith. 2019. Low-resource parsing with crosslingual contextualized representations. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 304-315, Hong Kong, China. Association for Computational Linguistics.
Ad Neeleman and Kriszta Szendroi. 2007. Radical pro drop and the morphology of pronouns. Linguistic Inquiry, 38(4):671-714.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1659-1666, Portoroz, Slovenia. European Language Resources Association (ELRA).
Edoardo Maria Ponti, Helen O'Horan, Yevgeni Berzak, Ivan Vulic, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2019. Modeling language variation and universals: A survey on typological linguistics for natural language processing. Computational Linguistics, 45(3):559-601.
Edoardo Maria Ponti, Roi Reichart, Anna Korhonen, and Ivan Vulic. 2018. Isomorphic transfer of syntactic structures in cross-lingual nlp. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1531–1542.
Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D. Manning. 2018. Universal dependency parsing from scratch. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 160-170, Brussels, Belgium. Association for Computational Linguistics.
Mohammad Sadegh Rasooli and Michael Collins. 2017. Cross-lingual syntactic transfer with limited resources. Transactions of the Association for Computational Linguistics, 5:279-293.
Tanja Samardžic, Lonneke Van Der Plas, Golijhan Kashaeva, and Paola Merlo. 2010. The scope and the sources of variation in verbal predicates in English and French. In Proceedings of the Ninth International Workshop on Treebanks and Linguistic Theories, pages 199-211, Tartu. Northern European Association for Language Technology (NEALT).
Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1599–1613, Minneapolis, Minnesota. Association for Computational Linguistics.
Jana Šindlerova, Zdeńska Urešová, and Eva Fučíková. 2013. Verb valency and argument noncorrespondence in a bilingual treebank. In Proceedings of the Seventh International Conference Slovko, pages 100-108.
Anders Søgaard and Dekai Wu. 2009. Empirical lower bounds on translation unit error rate for the full class of inversion transduction grammars. In Proceedings of the 11th International Conference on Parsing Technologies (IWPT'09), pages 33-36, Paris, France. Association for Computational Linguistics.
Margita Šoštaric, Christian Hardmeier, and Sara Stymne. 2018. Discourse-related language contrasts in English-Croatian human and machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 36-48, Brussels, Belgium. Association for Computational Linguistics.
Elior Sulem, Omri Abend, and Ari Rappoport. 2015. Conceptual annotations preserve structure across translations: A French-English case study. In Proceedings of the 1st Workshop on Semantics-Driven Statistical Machine Translation (S2MT 2015), pages 11-22, Beijing, China. Association for Computational Linguistics.
Vered Volansky, Noam Ordan, and Shuly Wintner. 2015. On the features of translationese. Digital Scholarship in the Humanities, 30(1):98-118.
Yogarshi Vyas, Xing Niu, and Marine Carpuat. 2018. Identifying semantic divergences in parallel text without annotations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1503-1515, New Orleans, Louisiana. Association for Computational Linguistics.
Philip Williams, Rico Sennrich, Matt Post, and Philipp Koehn. 2016. Syntax-based statistical machine translation. Synthesis Lectures on Human Language Technologies, 9(4):1-208.
Tak-sum Wong, Kim Gerdes, Herman Leung, and John Lee. 2017. Quantitative comparative syntax on the Cantonese-Mandarin parallel dependency treebank. In Proceedings of the Fourth International Conference on Dependency Linguistics (Depling 2017), pages 266-275, Pisa, Italy. Linköping University Electronic Press.
Nianwen Xue, Ondrej Bojar, Jan Hajic, Martha Palmer, Zdenka Uresová, and Xiuhong Zhang. 2014. Not an interlingua, but close: Comparison of English AMRs to Chinese and Czech. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 1765-1772, Reykjavik, Iceland. European Language Resources Association (ELRA).
Daniel Zeman, Martin Popel, Milan Straka, Jan Hajic, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gokirmak, Anna Nedoluzhko, Silvie Cinkova, Jan Hajic jr., Jaroslava Hlavacova, Václava Kettnerová, Zdenka
Uresova, Jenna Kanerva, Stina Ojala, Anna Missilä, Christopher D. Manning, Sebastian Schuster, Siva Reddy, Dima Taji, Nizar Habash, Herman Leung, Marie-Catherine de Marneffe, Manuela Sanguinetti, Maria Simi, Hiroshi Kanayama, Valeria dePaiva, Kira Droganova, Héctor Martínez Alonso, CagrÁs Çöltekin, Umut Sulubacak, Hans Uszkoreit, Vivien Macketanz, Aljoscha Burchardt, Kim Harris, Katrin Marheinecke, Georg Rehm, Tolga Kayadelen, Mohammed Attia, Ali Elkahky, Zhuoran Yu, Emily Pitler, Saran Lertpradit, Michael Mandl, Jesse Kirchner, Hector Fernandez Alcalde, Jana Strnadová, Esha Banerjee, Ruli Manurung, Antonio Stella, Atsuko Shimada, Sookyoung Kwak, Gustavo Mendonca, Tatiana Lando, Rattima Nitisaroj, and Josie Li. 2017. Conll 2017 shared task: Multilingual parsing from raw text to universal dependencies. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1-19, Vancouver, Canada. Association for Computational Linguistics.
Chunting Zhou, Xuezhe Ma, Junjie Hu, and Graham Neubig. 2019. Handling syntactic divergence in low-resource machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1388-1394, Hong Kong, China. Association for Computational Linguistics.