SlowGuess/NAACL_2022_OCR huggingface dataset:
Exact translation accuracy has limitations, as there are potential translations that could be acceptable given a specific context (e.g. a synonym). For that reason, we performed a human evaluation of a sample of sentences on  $(10\%)$  of each evaluation
Figure 2: Accuracy (exact translation) for Verbs in the English  $\rightarrow$  Spanish translations. Results are grouped by the training frequency and whether the word belongs to the vocabulary of the model (In V) or not (Not in V).
set, focusing on two scores<sup>11</sup>:
1. Semantic score: evaluates the meaning of the word used in the automatic translation (system output) and how it compares with the gold standard translation. Scale goes from 1 (no relationship at all) to 4 (it is the same lemma).
2. Grammar score: evaluates the grammatical form and how it compares with the gold standard translation. Scale goes from 1 (different inflection) to 3 (same inflection).
Synthesis In Figure 3, we show the annotation scores for the semantic and grammar metrics, for both nouns (top) and verbs (bottom). We also divide the analysis w.r.t. the frequency of the word in the training data. For nouns, we observe similar patterns as in the automatic analysis, where in the one-morpheme column, the proportion of the highest score is slightly larger than in the other columns, suggesting they are easier to generate for the model.
<sup>11</sup>Details of the annotation protocol are in the Appendix
Figure 3: Semantic score (left) and Grammar score (right) annotation for Turkish, for different frequency ranges of Nouns (top) and Verbs (bottom). Bubbles represent the proportion of the amount of scored annotations (1-4 or 1-3) divided by the subtotal elements of their respective columns (or number of morphemes). The orange inner bubble represents the amount of samples with 'zero' accuracy (in the automatic analysis) in each category.
The pattern is even more explicit in the highest frequency block (rightmost one). The verbs tend to have more distributed grammar scores, suggesting the difficulty of generating inflected forms may remain equally high even when the words are more frequent. Single morpheme verbs are very rare in Turkish and generally contain exceptional forms which reflects in the low translation accuracy in the highest frequency block. We also observe that a good proportion of translated words with ‘zero’ accuracy (not the exact translation, see the orange inner bubbles) has been annotated with highest semantic (same lemma) or grammar (same inflection) score, suggesting in some cases that the model is successful in generalization.
Fusion Figure 4 shows the semantic and grammar annotation scores for Spanish verbs. For the semantic scores (top), in all levels, the gap between the non-fusional and fusional verbs is reduced, for all the frequency groups. This means that the model is indeed able to generalise and offer alternative translations (not the exact verb), which is more complex to measure with automatic metrics. In the grammar scale (bottom), however, we still note a slight advantage in the maximum score (3) of the non-fusional verbs against the fusional ones for the two highest frequency subplots (middle and right). This indicates that, with highly frequent verbs, it is still more difficult to translate correct forms with a fusion degree higher than zero. Similarly as for synthesis, we observe that there is a significant proportion of 'zero' accuracy cases (orange inner bubbles) for the highest scores in most cases. This indicates that the model could generalise and trans
Figure 4: Semantic (top) and Grammar (bottom) annotation for Spanish.
late verbs with similar meanings and not the exact, but close, forms.
5 Segment-level Analysis of Synthesis and Fusion in Machine Translation
Following up the word level analysis, we study the relationship between machine translation difficulty and the degree of synthesis or fusion at the segment level. For this purpose, we process a set of translation systems for the language pair we want to evaluate. The general steps are as follow:
1. For each system output, we compute automatic evaluation metrics (BLEU (Papineni et al., 2002), chrF (Popovic, 2015) and/or COMET (Rei et al., 2020)) with respect to
Figure 5: Overview of significant predictors for degree of synthesis across our TR-EN and EN-TR models.
Figure 6: Overview of significant predictors for degree of fusion across our ES-EN and EN-ES models.
the reference set, per sentence.12
2. For each sentence of the evaluation set, we compute potential predictor variables for the automatic metric, such as the degree of synthesis or fusion. We complement the predictor variable list with other heuristics, such as the length of the sentence in characters (char.count) or words (word.count). The full list of all the predictors per language pair is in the Appendix.
3. With the previous inputs, we generate generalized linear models per system output and evaluation metric, in which each model's output is set to the predictor variables. The goal is to identify which predictors affect each method's performance.
4. Following model creation, we extract the significant predictors of each model. This provides an indication of which variables can be used to predict the outcome of the model's dependent variable – in our case the degree of synthesis or fusion, or any heuristic.[13]
Synthesis on En-Tr and Tr-En We first start evaluating the English-Turkish and Turkish-English language pairs. The evaluated models are EnTr1, EnTr2, and TrEn2 (details in the Appendix). Also, as we are studying synthesis in Turkish, all predictors are computed on the Turkish side, regardless of the translation direction.
Figure 5 presents an overview of the significant predictors on En-Tr and Tr-En systems, where we observe a large impact of the synthesis variable on
the chrF scores of two different systems (EnTr1 and TrEn2). The only other heuristic that achieves a notable impact on a system output is morph.count, or the length of Turkish sentence in morphemes, split by a morphological analyser. Other predictors have a minor effect.
Fusion on En-Es and Es-En In a similar way, we evaluate the impact of fusion in English-Spanish (EnEs1, EnEs2) and Spanish-English (EsEn1, EsEn2) models (see the Appendix for details). Again, as we are studying fusion in Spanish, all predictors are computed on the Spanish side, regardless of the translation direction.
Figure 6 presents an overview of the significant predictors, where we observe that  $R$ .fusion verb, or the ratio of the degree of fusion over the number of verbs in the sentence, is the predictor that has the highest impact in most system outputs (EnEs1, EnEs2 and EsEn2). Additionally,  $R$ .fusion.swEsEn2 (or the ratio of the degree of fusion over the number of subwords input in the EsEn2 model) also has a high impact in one system output (EnEs2, which uses the same segmentation model as EsEn2).
Analysis on En-De and De-En Finally, we extend the analysis to English-German and German-English language pairs, using the respective evaluation sets of the WMT2018 campaign (Bojar et al., 2018), and the system outputs provided for all the participants (measured in BLEU). For computing synthesis, we use the different segmentation methods we compared in §3.1. However, for fusion, we only use a shallow proxy with the number of morphological features that are tagged using a morphological analyser. In this case, the predictors are computed for both the source and target side.
$^{12}$ Based on the analysis of Kocmi et al. (2021), we prefer to report COMET and chrF over BLEU.  ${ }^{13}$ For simplification purposes, in the following analysis and plots, we only show the predictors that show a significant effect on the system outputs.
Figure 7: Overview of significant predictors across DE-EN models.
We present an overview of these significant predictors for German-English in Figure 7 (and the Appendix contains the results for English-German in Figure 8). We can observe that ref.SYN.uniLM and ref.SYN.PtrNet are the predictors that impact most of the different system outputs. These variables refer to the synthesis computed on the reference side (English) using uniLM or PtrNet as the morpheme segmentation method, respectively. Furthermore, we observe that src-ref.Rfeat_token has also some effect over one system output, which is a shallow proxy for the fusion degree in the source w.r.t. to reference segment (using the ratio of number of features per number of tokens).
It is important to note the limitations of this study. Overall results do not suggest that translating into more analytic languages (e.g. Chinese) or more agglutinative ones (e.g. Turkish) is easier than their counterparts. Highly analytic ones present the significant issue of word coverage and vocabulary size of the model. Besides, we cannot isolate the fusional degree from synthesis entirely. For instance, Turkish is a highly agglutinative language, but also highly synthetic, and there are languages that present both agglutinative and fusional traits, like Navajo. Moreover, the language scope is another limitation: is it possible to extend it to further languages in a practical way? Synthesis can be calculated directly only if the morphological analyser splits the word into morphemes, and fusion poses several issues as mentioned before. Furthermore, Payne (2017) also indicated that the discourse can impact the computed degrees due to the diversity of the vocabulary. This study focuses on news data only, and it will be relevant to extend it to different domains.
To address the limitations, we consider that our word level analysis, that targets specific POS, has been fundamental to enable the study of the indexes, and to partially isolate them from each other. The selection of our study cases was also relevant. Spanish verbs do not present more than three morphemes, keeping a low synthesis value across all the analysis, whereas Turkish is more agglutinative than fusional. Moreover, to rapidly extend the evaluation for new languages and domains, we could follow a less fine-grained analysis in each index. For instance, we can compare synthesis=1 vs. synthesis>1 instead of granulating per number of morphemes, or fusion=0 vs. fusion>0, as we did in this work.
In conclusion, we proposed methods to quantify the indices of synthesis and fusion in automatic and semi-automatic ways, respectively. Besides, for the chosen language pairs, we observed that the studied degrees have an impact in machine translation performance at both word and segment level, where we included a human evaluation of the former case.
Our analysis opens the possibility for further fine-grain evaluation approaches for MT and other NLP generation tasks. For instance, as future work, we can ask: are we improving the automatic translation of highly fusional words or segments? Following our methodology, we could stratify evaluation sets to measure how our models performs in different parts of the spectrum. Besides, the indices could also be helpful for evaluation approaches in morphological segmentation. Furthermore, another potential research avenue is to aid model training in MT: e.g. knowing which segments are more or less synthetic and/or fusional could be beneficial for sampling strategies.
The annotations in this paper were compensated accordingly (see Appendix). Also, for all the datasets used in the research, we stick to the ethical standards giving credit to the original author. We encourage future work that take advantage of these resources, to cite also the original sources of the data.
This work was supported by funding from the European Union's Horizon 2020 research and innovation programme under grant agreements No 825299 (GoURMET) and the EP-SRC fellowship grant EP/S001271/1 (MTStretch). Also, we acknowledge the support from eBay. Besides, the study was performed using resources provided by the Cambridge Service for Data Driven Discovery (CSD3) operated by the University of Cambridge Research Computing Service (http://www.csd3.cam.ac.uk/), provided by Dell EMC and Intel using Tier-2 funding from the Engineering and Physical Sciences Research Council (capital grant EP/P020259/1), and DiRAC funding from the Science and Technology Facilities Council (www.dirac.ac.uk). We express our thanks to Kenneth Heafield and Rico Sennrich, who provided us with access to the computing resources.
Moreover, the first author was granted financial support from the European Association for Machine Translation (EAMT), under its programme "2020 Sponsorship of Activities", and from the European Cooperation in Science and Technology (COST) under the programme CA18231 - Multi3Generation: Multi-task, Multilingual, Multimodal Language Generation.
Chantal Amrhein and Rico Sennrich. 2021. How suitable are subword segmentation strategies for translating non-concatenative morphology? In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 689-705, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Muller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019.
Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1-61, Florence, Italy. Association for Computational Linguistics.
Johannes Bjerva and Isabelle Augenstein. 2018. From phonology to syntax: Unsupervised linguistic typology at different levels with language embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 907-916, New Orleans, Louisiana. Association for Computational Linguistics.
Johannes Bjerva and Isabelle Augenstein. 2021. Does typological blinding impede cross-lingual sharing? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 480–486, Online. Association for Computational Linguistics.
Johannes Bjerva, Yova Kementchedjhieva, Ryan Cotterell, and Isabelle Augenstein. 2019a. A probabilistic generative model of linguistic typology. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1529-1540, Minneapolis, Minnesota. Association for Computational Linguistics.
Johannes Bjerva, Yova Kementchedjhieva, Ryan Coterell, and Isabelle Augenstein. 2019b. Uncovering probabilistic implications in typological knowledge bases. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3924-3930, Florence, Italy. Association for Computational Linguistics.
Johannes Bjerva, Elizabeth Salesky, Sabrina J. Mielke, Aditi Chaudhary, Giuseppe G. A. Celano, Edoardo Maria Ponti, Ekaterina Vylomova, Ryan Cotterell, and Isabelle Augenstein. 2020. SIGTYP 2020 shared task: Prediction of typological features. In Proceedings of the Second Workshop on Computational Research in Linguistic Typology, pages 1-11, Online. Association for Computational Linguistics.
Ondrej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. Findings of the 2013 Workshop on Statistical Machine Translation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 1-44, Sofia, Bulgaria. Association for Computational Linguistics.
Ondrej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 272-303, Bel
gium, Brussels. Association for Computational Linguistics.
Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pretraining. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 4617-4624, Online. Association for Computational Linguistics.
Bernard Comrie. 1989. Language universals and linguistic typology: Syntax and morphology. University of Chicago press.
Mathias Creutz and Krista Lagus. 2002. Unsupervised discovery of morphemes. In Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning, pages 21-30. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2112-2128, Online. Association for Computational Linguistics.
Ramy Eskander, Judith Klavans, and Smaranda Muresan. 2019. Unsupervised morphological segmentation for low-resource polysynthetic languages. In Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 189-195, Florence, Italy. Association for Computational Linguistics.
Joseph H Greenberg. 1960. A quantitative approach to the morphological typology of language. International journal of American linguistics, 26(3):178-194.
Zellig S Harris. 1951. Methods in structural linguistics.
Marcin Junczys-Dowmunt, Kenneth Heafield, Hieu Hoang, Roman Grundkiewicz, and Anthony Aue. 2018. Marian: Cost-effective high-quality neural machine translation in  $\mathbf{C} + +$ . In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 129-135, Melbourne, Australia. Association for Computational Linguistics.
Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, pages 478-494, Online. Association for Computational Linguistics.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79-86, Phuket, Thailand.
Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Manuel Mager, Özlem Çetinoglu, and Katharina Kann. 2020. Tackling the low-resource challenge for canonical segmentation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5237-5250, Online. Association for Computational Linguistics.
Arya D. McCarthy, Christo Kirov, Matteo Grella, Amrit Nidhi, Patrick Xia, Kyle Gorman, Ekaterina Vylomova, Sabrina J. Mielke, Garrett Nicolai, Miikka Silfverberg, Timofey Arkhangelskiy, Nataly Krizhanovsky, Andrew Krizhanovsky, Elena Klyachko, Alexey Sorokin, John Mansfield, Valts Ernstreits, Yuval Pinter, Cassandra L. Jacobs, Ryan Cotterell, Mans Hulden, and David Yarowsky. 2020. UniMorph 3.0: Universal Morphology. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 3922-3931, Marseille, France. European Language Resources Association.
Jamshidbek Mirzakhalov, Anoop Babu, Duygu Ataman, Sherzod Kariev, Francis Tyers, Otabek Abdurafov, Mammad Hajili, Sardana Ivanova, Abror Khaytbaev, Antonio Laverghetta Jr., Bekhzodbek Moydinboyev, Esra Onal, Shaxnoza Pulatova, Ahsan Wahab, Orhan Firat, and Sriram Chellappan. 2021. A large-scale study of machine translation in Turkic languages. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5876-5890, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tahir Naveed, Imitaz Saeed Siddiqui, and Shaftab Ahmed. 2005. Parallel needleman-wunsch algorithm for grid. In Proceedings of the PAK-US International Symposium on High Capacity Optical Networks and Enabling Technologies (HONET 2005), Islamabad, Pakistan.
Arturo Oncevay, Barry Haddow, and Alexandra Birch. 2020. Bridging linguistic typology and multilingual machine translation with multi-view language representations. In Proceedings of the 2020 Conference
on Empirical Methods in Natural Language Processing (EMNLP), pages 2391-2406, Online. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Thomas E Payne. 2017. Morphological typology. In The Cambridge Handbook of Linguistic Typology, pages 78-94. Cambridge University Press.
Edoardo Maria Ponti, Helen O'Horan, Yevgeni Berzak, Ivan Vulic, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2019. Modeling language variation and universals: A survey on typological linguistics for natural language processing. Computational Linguistics, 45(3):559-601.
Hoifung Poon and Pedro Domingos. 2009. Unsupervised semantic parsing. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1-10, Singapore. Association for Computational Linguistics.
Maja Popovic. 2015. *chrF: character n-gram F-score* for automatic MT evaluation. In *Proceedings of the Tenth Workshop on Statistical Machine Translation*, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685-2702, Online. Association for Computational Linguistics.
Hasim Sak, Tunga Güngör, and Murat Saraçlar. 2008. Turkish language resources: Morphological parser, morphological disambiguator and web corpus. In International Conference on Natural Language Processing, pages 417-427. Springer.
Edward Sapir. 1921. Types of linguistic structure.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Petra Steiner. 2016. Refurbishing a morphological database for German. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1103-1108, Portoorž, Slovenia. European Language Resources Association (ELRA).