SlowGuess/NAACL_2022_OCR huggingface dataset:
Arturo Oncevay $^{\epsilon*}$  Duygu Ataman $^{\eta}$  Niels van Berkel $^{\alpha,\mu}$  Barry Haddow $^{\epsilon}$  Alexandra Birch $^{\epsilon}$  Johannes Bjerva $^{\alpha,\sigma}$
<sup>e</sup>University of Edinburgh <sup>η</sup>New York University <sup>a</sup>Aalborg University (Aalborg<sup>μ</sup> | Copenhagen<sup>σ</sup>) a. oncevay@ed.ac.uk
Theoretical work in morphological typology offers the possibility of measuring morphological diversity on a continuous scale. However, literature in Natural Language Processing (NLP) typically labels a whole language with a strict type of morphology, e.g. fusional or agglutinative. In this work, we propose to reduce the rigidity of such claims, by quantifying morphological typology at the word and segment level. We consider Payne (2017)'s approach to classify morphology using two indices: synthesis (e.g. analytic to polysynthetic) and fusion (agglutinative to fusional). For computing synthesis, we test unsupervised and supervised morphological segmentation methods for English, German and Turkish, whereas for fusion, we propose a semi-automatic method using Spanish as a case study. Then, we analyse the relationship between machine translation quality and the degree of synthesis and fusion at word (nouns and verbs for English-Turkish, and verbs in English-Spanish) and segment level (previous language pairs plus English-German in both directions). We complement the word-level analysis with human evaluation, and overall, we observe a consistent impact of both indexes on machine translation quality.
One of the first barriers to develop language technologies is morphology, i.e., how systematically diverse their word formation processes are. For instance, agglutination and fusion are two morphological kind of processes that concatenate morphemes to a root with explicit or non-explicit boundaries, respectively. Processing morphologically-diverse languages and evaluating morphological competence in NLP models is relevant for language generation and understanding tasks, such as machine translation (MT). It is unfeasible to develop models
with capacity large enough to encode the full vocabulary of every language, and it is a must to rely on subword segmentation approaches that help to constrain the capacity when generating rare, or even new words (Sennrich et al., 2016). Hence, understanding morphology is essential to develop robust subword-based models and evaluate the quality of their outputs (Vania and Lopez, 2017). Nevertheless, there is a potential gap between the probing of whether an NLP model can handle "morphological richness", and what is a proper measure of "morphological richness" from linguistic typology.
In most of the recent NLP literature, different types of languages (e.g. agglutinative, polysynthetic) are chosen to test a more diverse handling of morphological richness (Ponti et al., 2019). There is, however, a debate as to whether languages can indeed be classified into discrete morphological categories. Payne (2017) provided a morphological typology measurement in a continuous spectrum using the indices of synthesis and fusion. Synthesis measures if a segment is highly analytic or synthetic (from 1 to more), whereas fusion measures whether it is highly agglutinative or fusional (from 0 to 1). And surprisingly, with respect to NLP publications, it is possible to identify English segments with a very low fusion index, meaning that they are highly agglutinative<sup>1</sup>.
From a more applied perspective, if the references of an evaluation set (in any language generation task) are labelled with the indices, we could perform a stratified analysis (e.g. low fusion and high fusion) to determine how well an NLP model handles morphology for multiple languages. For example, we could assess whether a machine translation model is failing in generating more fusional
For instance, in the following fragment (Payne, 2017), the index of fusion is 1/8 or 0.125 (fusional morpheme joints are marked with a dot and the rest with a hyphen): "The company-'s great break-through came.PAST when they decided to buy trike-s to sell their ice cream around the street-s in the nine-teen twenty-s".
*Work started when the first author was doing a research internship with JB at Aalborg University, Campus Copenhagen
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1308 - 1321 July 10-15, 2022 ©2022 Association for Computational Linguistics
than agglutinative segments for a specific target language. Knowing and quantifying that problem concerning morphology is the first step towards proposing a solution. Our contributions then are listed as follows:
- We present the first computational quantification of synthesis and fusion using standard NLP evaluation sets.
- We analyse the relationship between the two indices and machine translation quality at word-level, and observe that a higher degree of synthesis or fusion usually corresponds to less accurate translations in specific word types (studying nouns and verbs in English-Turkish, and verbs in English-Spanish).
- We complement this evaluation with manual annotation of synthesis and fusion.
- We extend the analysis at segment-level, using the aforementioned language pairs plus English-German in both directions, and identify that some synthesis and fusion-based predictors are significant for MT system outputs.
Furthermore, we release all the annotated data and evaluation results<sup>2</sup>.
2.1 Morphological typology
The field of morphological typology characterises languages in terms of their word and sentence building strategies (Payne, 2017), such as agglutination or fusion. In current NLP literature, Turkish is labelled as a highly agglutinative language for the explicit boundaries between their morphemes, whereas Spanish is labelled as fusional for the opposite reason.
However, early typological studies started to quantify these strategies with parameters, and avoided to characterise languages with a single type in a holistic way (e.g. Sapir (1921); Greenberg (1960); Comrie (1989)). In this context, Payne (2017) recently highlighted the indices of synthesis and fusion, which are defined as follows.
2.1.1 Synthesis
The index of synthesis offers a scale to contrast highly analytic or synthetic languages. This implies whether a word is composed by one (analytic) or several (synthetic) morphemes (Payne, 2017). Synthesis can be computed as the ratio of number
of morphemes per words, it is closer to 1 when the language is more analytic (e.g. Mandarin, or English to a less degree), and gets higher the more synthetic the language is (e.g. Turkish, Inuktitut). Polysynthesis can be present when the synthesis degree is higher than 3, although the boundary is arguable. Besides, as we claim in this study, any language can present different levels of synthesis if we evaluate them at a more fine-grained level.
2.1.2 Fusion
Fusion is the ratio of the fusional morphemes joints<sup>3</sup> per the total number of joints. This index goes from 0 to 1, or from highly agglutinative (e.g. Turkish) to highly fusional (e.g. Spanish) cases. However, we noticed that the computation of fusion is complex to automatise. For instance, Payne (2017) indicates potential cases to identify fusional joints, such as in prefixes, suffixes, infixes, circumfixes, compounding, non-concatenative processes (reduplication, apophony, subtractive morphology) or autosegmental morphemes. Current automatic tools are not designed to identify these cases for most languages.
2.2 Morphological typology on NLP
A survey by Ponti et al. (2019), on computational typology for NLP, pointed out that morphological knowledge is potentially helpful for analysing the difficulty in generation tasks such as language modelling and neural MT for both unsupervised and supervised settings. More specifically, they suggested that the degree of fusion (related to the index of fusion proposed by Payne (2017)) impacts in the rate of less frequent words, which is a relevant parameter for generation tasks.
Besides, the studies that address morphological typology are related to either the development of morphological analysis systems or the evaluation of typologically diverse languages in terms of morphology (e.g. Vania and Lopez (2017); Xu et al. (2020)). However, the typology used to distinguish languages varies across different studies. For instance, Vania and Lopez (2017) considers four phenomena to label languages: fusionality, agglutination, reduplication and root-pattern; whereas Xu et al. (2020) considers more fine-grained elements such as affixation (prefixation, infixation and suffixation) or partial reduplication. Similarly,
<sup>3</sup>Or how many grammatical, syntactic and semantic features are joint. More than one feature can be fused in a single morpheme.
2https://github.com/aoncevay/ quantifying-synthesis-and-fusion
a fine-grained analysis on non-concatenative morphology for MT was performed by Amrhein and Sennrich (2021). It is important to note that none of the previous studies have addressed the phenomena as a continuous index but as discrete features.
Furthermore, other studies refer only to morphological typological features as part of the task of typological feature prediction from linguistic databases (Bjerva and Augenstein, 2018; Bjerva et al., 2019a,b, 2020; Bjerva and Augenstein, 2021), and further applications of general typological concepts on MT are scarce and do not focus on morphology (Oncevay et al., 2020).
2.3 Morphological segmentation and analysis
Morphological segmentation (Harris, 1951) aims to split a word into morphemes. There are both supervised (e.g. pointer generator networks (Mager et al., 2020)) and unsupervised approaches (e.g. the Morfessor family of methods (Creutz and Lagus, 2002; Poon and Domingos, 2009) or Adaptor Grammars (Eskander et al., 2019)), where the former ones have outperformed the latter ones.
Besides, the most widespread unsupervised segmentation methods (Byte-Pair-Encoding (BPE; Sennrich et al., 2016) and a method based on unigram language modelling (Kudo, 2018)) are not linked at all to morphological segmentation, but they are used to constrain the vocabulary size for neural generation tasks.
Finally, it is important to note that the index of synthesis can be computed with a robust morphological analyser or segmentation model (to count the number of morphemes), but neither of them are built to compute the index of fusion directly.
3 How to compute Synthesis and Fusion?
3.1 Synthesis: automatic computation
To automatically compute the index of synthesis, we require to perform a robust morphological segmentation. A rule-based morphological analyser and disambiguator might be the best option if available (which we use later for Turkish in §4.2), but for the purpose of the study, we compare well-known supervised and unsupervised methods:
- Byte-Pair-Encoding (BPE) and Unigram Language Model (uniLM) $^4$  from SentencePiece (Kudo and Richardson, 2018).
- Morfessor (Poon and Domingos, 2009).
English	German
#morphs.	1	2	3	4	1	2	3	4
16,914	28,900	1,798	73	13,061	32,007	5,808	360
Accuracy Count
uniLM64k	0.54	0.52	0.49	0.59	0.35	0.27	0.21	0.18
BPE64k	0.5	0.53	0.5	0.52	0.29	0.33	0.28	0.26
Morfessor	0.22	0.47	0.55	0.48	0.17	0.26	0.28	0.25
PtrNet	0.82	0.84	0.56	0.81	0.74	0.86	0.7	0.42
Exact Segmentation Precision
uniLM64k	0.54	0.52	0.6	0.8	0.29	0.38	0.32	0.22
BPE64k	0.5	0.44	0.56	0.76	0.24	0.33	0.23	0.08
Morfessor	0.21	0.58	0.7	0.78	0.17	0.45	0.44	0.36
PtrNet	0.76	0.67	0.81	0.8	0.67	0.73	0.72	0.62
Table 1: Accuracy count and segmentation precision for English and German using unsupervised and supervised segmentation methods. Results are grouped by the expected number of morphemes (e.g. "1" means that the word should not be split).
- Pointer Generator Network (PtrNet) from the implementation of Mager et al. (2020).
3.1.1 Datasets and evaluation
We used the CELEX dataset of segmented words for English and German (Steiner, 2016, 2017), where we randomly split training and evaluation data (80-10-10). Besides, for the unsupervised methods, we use the newscommentary-v15 (Barrault et al., 2019) and EuroParl-v10 (Koehn, 2005) corpora<sup>5</sup>. Furthermore, we define two metrics to assess the performance on computing synthesis:
- Accuracy count: Evaluates if the number of obtained morphemes in the hypothesis segmentation is the same as in the reference.
- Exact segmentation precision: Analyses if the split morphemes are the same. We first perform an automatic alignment between the hypothesis and reference segments with the parallel Needleman-Wunsch algorithm for sequences (Naveed et al., 2005), and then compute the exact match at morpheme level.
3.1.2 Results and discussion
Table 1 shows the scores on morphological segmentation for both English and German. We observe that both BPE and uniLM under-perform when it is not expected to split the word (column "1"). This is a pattern observed by Bostrom and Durrett (2020), where they noted that unsupervised segmentation methods tend to over-split the roots of words. They both improve their accuracy and precision when
5Other languages like Danish are also available and was tested, but we did not report the results here as there are not complementary MT evaluation sets.
4We analysed several vocabulary sizes (4k, 8k, 16k, 32k, 64k) but report only the best one, which is 64k for all cases.
Example (es): Hablaremos de la propuesta con la que se condeno a la ex primer ministra y fue apoyada por 147 diputados en la votación.
Verbs	Features (spaCy)	Features (UniMorph)	Segmentation	feats. per morph	fusional morph. joints	total joints	Fusion index
hablaremos (we will talk)	Mood=Ind, Number=Plur, Person=1, Tense=Fut, VerbForm=Fin	V;IND;FUT;1;PL	habl - are - mos	0 - 2(IND:FUT) - 2(1;PL)	0+(2-1)+(2-1) = 2	2+2 = 4	0.5
condenó (condemned)	Mood=Ind, Number=Sing, Person=3, Tense=Past, VerbForm=Fin	V;IND;PST;3;SG;PFV	conden - σ	0 - 5(IND:PST;3; SG;PFV)	0+(5-1) = 4	4+1 = 5	0.8
apoyada (supported)	Gender=Fem, Number=Sing, VerbForm=Part	V.PTCP;PST;FEM;SG	apoy - ada	0 - 3(PST;FEM;SG)	0+(3-1) = 2	2+1 = 3	0.66
Table 2: Annotation example in Spanish. We first identify the verbs (in bold) and obtain their morphological features (using spaCy and the UniMorph schema). Then, we split the verb into its morphemes (segmentation), and identify which features are fused in each morpheme (feats. per morph). Finally, we compute the index of fusion by dividing the fusional morpheme joints by the total joints (which includes the agglutinative or explicit boundaries). On a side note, examples of verbs with zero fusion are in the infinitive (e.g. haber (to talk)) and gerund (e.g. hablando (talking)) forms.
the number of expected morphemes is larger. Unexpectedly, Morfessor also under-performs in the "1" case for both languages, and only surpasses the other unsupervised methods when we measure precision for many morphemes. Furthermore, The PtrNet supervised method outperforms the rest in almost all scenarios.
We conclude that, to compute synthesis, we should prioritise, besides a rule-based morphological analyser, a supervised segmentation method like PtrNet if data is available. We take advantage of this for the segment-level analysis in §5.
3.2 Fusion: Semi-automatic computation
Calculating fusion should be approached in a case by case scenario, as there are different considerations provided by Payne (2017). Therefore, there is not an automatic tool that can obtain the fusion score directly. We decided to focus on Spanish $^{6}$  as a case study, where verbs and auxiliary verbs contains the highest degree of fusion of all the parts-of-speech (POS).
Procedure We observed that we could perform an annotation per paradigm and the termination of the verb (-ar, -er, -ir), as the fusion degree will remain the same regardless of the lemma<sup>7</sup>. Then, on a chosen Spanish corpus:
1. Perform an automatic annotation of POS and morphological features<sup>8</sup>.
2. Review the automatic annotation of special cases. For instance, there are specific verb forms that are missed as adjectives. We corrected the POS and morphological annotation of those cases in a manual step.
3. Obtain a set of all unique verb paradigms and morphological features in the corpus, considering the three different types of verb terminations in Spanish as different elements<sup>9</sup>.
Now there is a list of unique verb paradigms and terminations that can be annotated both in synthesis and fusion. The steps are as follows:
1. For each unique verb paradigm and termination, segment a verb sample into its morphemes. E.g. the verb habló ('talked'), is split in habl-ó, and habláramos ('we were to speak') in habl-ára-mos.
2. Analyse how many morphological features are fused in each morpheme: if you change a value of a feature, will the surface form or morpheme will change? E.g. in habl-ó, -ó participates in 5 features (mode (indicative), subject person (third person), subject number (singular), tense (past) and aspect (perfective)). For habl-ára-mos, -ára includes the past and subjunctive, whereas -mos denotes the person and number. If any of aforementioned feature changes its value, the surface will change too.
3. Count and aggregate the results per mor
<sup>6</sup>We chose this language because of the ease of finding annotators and MT training and evaluation data.
Except for irregular ones, which presents a limitation and potential noise. To reduce the risk of a biased assessment, we also performed a human evaluation.
<sup>8</sup>We use the spaCy model es_dep_news_trf, available at https://spacy.io/models/es#es_dep_
news_trf. It has an accuracy of 0.99 in POS and morphological tagging in the UD Spanish AnCora dataset (Taulé et al., 2008), which contains news texts mostly.
<sup>9</sup>Using the Unimorph database (McCarthy et al., 2020) is another alternative for extracting all the possible unique inflections. We aligned and considered both tag sets for the annotation, as shown in Table 2.
phemes and obtain the fusion for each verb paradigm. E.g. the fusion for  $habl-ó$  is  $4/5 = 0.8$ , and for  $habl-ára-mos$  is  $2/4 = 0.5$ .
Finally, with the annotation in the unique list of verb inflections and terminations, we can extend the degree of fusion to all the verbs in the original Spanish corpus. An example of the annotation process is shown in Table 2.
4 Word-level analysis of Synthesis and Fusion in Machine Translation
In this analysis, we ask the following question: how difficult is translating a word concerning its index of synthesis or fusion? For evaluating synthesis, we work with Turkish<sup>10</sup> nouns and verbs, and for fusion, we keep working on Spanish verbs. For both cases, English is the source language in the translation task.
4.1 Experimental design
The experiment consists of comparing a gold standard reference with machine translation system outputs at the word level:
1. For both the reference and system output, we automatically tag all the words with a morphological analyser (the Boun morphological analyser and disambiguator (Sak et al., 2008) for Turkish and an spaCy model trained on the Ancora Universal Dependency parser (Taulé et al., 2008) for Spanish). The POS is needed to filter the target words. For synthesis in Turkish, the number of morphemes works as a proxy, as we are working at the word level. For fusion in Spanish, we need the inflection to obtain the degree of fusion from the annotated unique list (see 3.2).
2. Align the words between the reference and system output. We use the awesome-align (Dou and Neubig, 2021) tool by fine-tuning the multilingual BERT (Devlin et al., 2019) model for word-alignment, using the reference and system output as parallel corpora.
3. Calculate the translation accuracy (exact match of the word, 0 or 1) for the target POS. We then fine-grain the results concerning the degree of synthesis (number of morphemes) or fusion.
Total	#1	#2	#3	#4	#5+
Verbs	3,834	133	2,265	1,036	308	92
Nouns	10,680	5,899	2,974	1,556	244	7
Table 3: Number of nouns and verbs in the Turkish reference set, and their respective number of morphemes.
Additionally, we control different confounds: frequency of the word in the training set, and whether the full word is part of the vocabulary input of the model or not. Finally, we complement the analysis with a human evaluation (see §4.4).
4.2 Synthesis analysis: English  $\rightarrow$  Turkish
Data We use the NEWSTEST2018.EN-TR evaluation set from WMT (Bojar et al., 2018), with 3,000 samples. In the Turkish side there are 45,944 tokens, and Table 3 shows the distribution of the number of morphemes obtained with Sak et al. (2008).
Model We use an English-Turkish system trained with the TIL corpus of 39.9M parallel sentences (Mirzakhalov et al., 2021). On the NEWTEST2018.EN-TR set, the performance is 13.06 and 49.54 in BLEU and chrF, respectively.
Results and discussion Figure 1 shows the average accuracy (exact translation, 0 or 1) of nouns and verbs in NEWSTEST2018.EN-TR, where the number of morphemes is a proxy for the index of synthesis. In most cases, especially with a higher training frequency, we observe that the average accuracy drops as the number of morphemes increases from 1 to more. This is clearer in nouns than in verbs, which have fewer cases to analyse overall. Between 2, 3 or more than 4 morphemes the differences are not significant, and sometimes is not consistent (e.g. verbs with the highest frequency). However, we can argue that analytic nouns (synthesis=1) are easier to translate than synthetic nouns (synthesis>1) for the English→Turkish direction. The pattern holds for whether the word is part of the vocabulary of the model or not, although rare words (frequency in  $[0,10^{3}]$  have generally lower translation accuracy than more frequent words (frequency > 100).
4.3 Fusion analysis: English→Spanish
Data We use the NEWSTEST2013.EN-ES evaluation set from WMT (Bojar et al., 2013) with 3,000 samples. In the Spanish side there are 62,055 tokens, with 6,317 verbs, and where 1,411 of them are more agglutinative (fusion=0) and 4,822 more fusional (fusion>0).
$^{10}$ Turkish presents high synthesis and agglutination (Zingler, 2018), meaning that there are words composed with several morphemes and the morpheme boundaries are explicit, respectively. We focus on verbs and nouns, which usually contain more morphemes than other parts-of-speech. We chose this language due to the availability of an open-source rule-based morphological analyser and an expert annotator.
Figure 1: Accuracy (exact translation) for Nouns (top) and Verbs (bottom) in the English  $\rightarrow$  Turkish translations. Results are grouped by the training frequency of the words (less to more frequent from left to right), and each subplot presents the scores for all the words, and whether they belong or not to the vocabulary input of the model. The number of samples are stacked in each bar, and we do not show entries with less than 30 samples.
Model For training, we use the MarianNMT toolkit (Junczys-Dowmunt et al., 2018), a Transformer-base model (Vaswani et al., 2017) with default parameters, and four NVIDIA V100 GPUs. We obtained different English-Spanish models using the newscommentary-v8 (Bojar et al., 2013) and EuroParl (Koehn, 2005) datasets with joint vocabulary sizes of 8k, 16k and 32k (using unigram-LM from SentencePiece (Kudo and Richardson, 2018)). For this analysis, we chose the best performing system: combining both datasets (2.2M sentences) with 16k pieces. On NEWSTEST2013.EN-ES, the performance is 31.6 BLEU points.
Results and discussion Figure 2 shows the average accuracy of verbs in NEWSTEST2013.EN-ES for verbs without and with some degree of fusion. In the two higher frequency subplots (middle and right), we can observe that the average accuracy of the non-fusional verbs is higher than the fusional ones, and the pattern holds whether the verb is present in the vocabulary input of the model or not. The exception is for the least frequent verbs, although this is explained as the model do not have enough information to learn from, regardless of their degree of fusion.