Venue: NAACL 2022

# Quantifying Synthesis and Fusion and their

Impact on Machine Translation

Arturo Oncevay\({}^{\ast\epsilon}\)   Duygu Ataman\({}^{\eta}\)    Niels van Berkel\({}^{\alpha,\mu}\)

Barry Haddow\({}^{\epsilon}\)   Alexandra Birch\({}^{\epsilon}\)   Johannes Bjerva\({}^{\alpha,\sigma}\)

\({}^{\epsilon}\)University of Edinburgh \(~{}^{78}\)New York University

\({}^{\alpha}\)Aalborg University (Aalborg\({}^{\mu}\)\(|\) Copenhagen\({}^{\sigma}\)) a.oncevay@ed.ac.uk

Work started when the first author was doing a research internship with JB at Aalborg University, Campus Copenhagen

###### Abstract

Theoretical work in morphological typology offers the possibility of measuring morphological diversity on a continuous scale. However, literature in Natural Language Processing (NLP) typically labels a whole language with a strict type of morphology, e.g. fusional or agglutinative. In this work, we propose to reduce the rigidity of such claims, by quantifying morphological typology at the word and segment level. We consider Payne (2017)'s approach to classify morphology using two indices: synthesis (e.g. analytic to polysynthetic) and fusion (agglutinate to fusional). For computing synthesis, we test unsupervised and supervised morphological segmentation methods for English, German and Turkish, whereas for fusion, we propose a semi-automatic method using Spanish as a case study. Then, we analyse the relationship between machine translation quality and the degree of synthesis and fusion at word (nouns and verbs for English-Turkish, and verbs in English-Spanish) and segment level (previous language pairs plus English-German in both directions). We complement the word-level analysis with human evaluation, and overall, we observe a consistent impact of both indexes on machine translation quality.

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1308 - 1321

July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

One of the first barriers to develop language technologies is morphology, i.e., how systematically diverse their word formation processes are. For instance, agglutination and fusion are two morphological kind of processes that concatenate morphemes to a root with explicit or non-explicit boundaries, respectively. Processing morphologically-diverse languages and evaluating morphological competence in NLP models is relevant for language generation and understanding tasks, such as machine translation (MT). It is unfeasible to develop models with capacity large enough to encode the full vocabulary of every language, and it is a must to rely on subword segmentation approaches that help to constrain the capacity when generating rare, or even new words Sennrich et al. (2016). Hence, understanding morphology is essential to develop robust subword-based models and evaluate the quality of their outputs Vania and Lopez (2017). Nevertheless, there is a potential gap between the probing of whether an NLP model can handle "morphological richness", and what is a proper measure of "morphological richness" from linguistic topology.

In most of the recent NLP literature, different types of languages (e.g. agglutinative, polysynthetic) are chosen to test a more diverse handling of morphological richness Ponti et al. (2019). There is, however, a debate as to whether languages can indeed be classified into discrete morphological categories. Payne (2017) provided a morphological typology measurement in a continuous spectrum using the indices of synthesis and fusion. Synthesis measures if a segment is highly analytic or synthetic (from 1 to more), whereas fusion measures whether it is highly agglutinative or frisonal (from 0 to 1). And surprisingly, with respect to NLP publications, it is possible to identify English segments with a very low fusion index, meaning that they are highly agglutinative1.

Footnote 1: For instance, in the following fragment Payne (2017), the index of fusion is 1/8 or 0.125 (fusional morpheme joints are marked with a dot and the rest with a hyphen): “The company’s great break-through came.PAST when they decided to buy trike-sto to sell their ice cream around the street-s in the nine-teen twenty-s”.

From a more applied perspective, if the references of an evaluation set (in any language generation task) are labelled with the indices, we could perform a stratified analysis (e.g. low fusion and high fusion) to determine how well an NLP model handles morphology for multiple languages. For example, we could assess whether a machine translation model is failing in generat

than agglutinative segments for a specific target language. Knowing and quantifying that problem concerning morphology is the first step towards proposing a solution. Our contributions then are listed as follows:

* We present the first computational quantification of synthesis and fusion using standard NLP evaluation sets.
* We analyse the relationship between the two indices and machine translation quality at word-level, and observe that a higher degree of synthesis or fusion usually corresponds to less accurate translations in specific word types (studying nouns and verbs in English-Turkish, and verbs in English-Spanish).
* We complement this evaluation with manual annotation of synthesis and fusion.
* We extend the analysis at segment-level, using the aforementioned language pairs plus English-German in both directions, and identify that some synthesis and fusion-based predictors are significant for MT system outputs. Furthermore, we release all the annotated data and evaluation results2. Footnote 2: [https://github.com/aoncevay/quantifying-synthesis-and-fusion](https://github.com/aoncevay/quantifying-synthesis-and-fusion)

## 2 Background and related work

### Morphological typology

The field of morphological typology characterises languages in terms of their word and sentence building strategies Payne (2017), such as agglutination or fusion. In current NLP literature, Turkish is labelled as a highly agglutinative language for the explicit boundaries between their morphemes, whereas Spanish is labelled as frusional for the opposite reason.

However, early typological studies started to quantify these strategies with parameters, and avoided to characterise languages with a single type in a holistic way (e.g. Sapir (1921); Greenberg (1960); Comrie (1989)). In this context, Payne (2017) recently highlighted the indices of synthesis and fusion, which are defined as follows.

#### Synthesis

The index of synthesis offers a scale to contrast highly analytic or synthetic languages. This implies whether a word is composed by one (analytic) or several (synthetic) morphemes Payne (2017). Synthesis can be computed as the ratio of number of morphemes per words, it is closer to 1 when the language is more analytic (e.g. Mandarin, or English to a less degree), and gets higher the more synthetic the language is (e.g. Turkish, Inuktitut). Polysynthesis can be present when the synthesis degree is higher than 3, although the boundary is arguable. Besides, as we claim in this study, any language can present different levels of synthesis if we evaluate them at a more fine-grained level.

#### Fusion

Fusion is the ratio of the fusional morphemes joints3 per the total number of joints. This index goes from 0 to 1, or from highly agglutinative (e.g. Turkish) to highly fusional (e.g. Spanish) cases. However, we noticed that the computation of fusion is complex to automatise. For instance, Payne (2017) indicates potential cases to identify fusional joints, such as in prefixes, suffixes, infixes, circumfixes, compounding, non-concatenative processes (reduplication, apophony, substructure morphology) or autosegmental morphemes. Current automatic tools are not designed to identify these cases for most languages.

Footnote 3: Or how many grammatical, syntactic and semantic features are joint. More than one feature can be fused in a single morpheme.

### Morphological typology on NLP

A survey by Ponti et al. (2019), on computational typology for NLP, pointed out that morphological knowledge is potentially helpful for analysing the difficulty in generation tasks such as language modelling and neural MT for both unsupervised and supervised settings. More specifically, they suggested that the degree of fusion (related to the index of fusion proposed by Payne (2017)) impacts in the rate of less frequent words, which is a relevant parameter for generation tasks.

Besides, the studies that address morphological typology are related to either the development of morphological analysis systems or the evaluation of topologically diverse languages in terms of morphology (e.g. Vania and Lopez (2017); Xu et al. (2020)). However, the typology used to distinguish languages varies across different studies. For instance, Vania and Lopez (2017) considers four phenomena to label languages: fusionality, agglutination, reduplication and root-pattern; whereas Xu et al. (2020) considers more fine-grained elements such as affixation (prefixation, infixation and suffixation) or partial reduplication. Similarly, 

a fine-grained analysis on non-concatenative morphology for MT was performed by Amrhein and Sennrich (2021). It is important to note that none of the previous studies have addressed the phenomena as a continuous index but as discrete features.

Furthermore, other studies refer only to morphological typological features as part of the task of topological feature prediction from linguistic databases Bjerva and Augenstein (2018); Bjerva et al. (2019, 2020); Bjerva and Augenstein (2021), and further applications of general typological concepts on MT are scarce and do not focus on morphology Oncevay et al. (2020).

### Morphological segmentation and analysis

Morphological segmentation Harris (1951) aims to split a word into morphemes. There are both supervised (e.g. pointer generator networks Mager et al. (2020)) and unsupervised approaches (e.g. the Morfessor family of methods Creutz and Lagus (2002); Poon and Domingos (2009) or Adaptor Grammars Eskander et al. (2019)), where the former ones have outperformed the latter ones.

Besides, the most widespread unsupervised segmentation methods Byte-Pair-Encoding BPE; Sennrich et al. (2016) and a method based on unigram language modelling Kudo (2018)) are not linked at all to morphological segmentation, but they are used to constrain the vocabulary size for neural generation tasks.

Finally, it is important to note that the index of synthesis can be computed with a robust morphological analyser or segmentation model (to count the number of morphemes), but neither of them are built to compute the index of fusion directly.

## 3 How to compute Synthesis and Fusion?

### Synthesis: automatic computation

To automatically compute the index of synthesis, we require to perform a robust morphological segmentation. A rule-based morphological analyser and disambiguator might be the best option if available (which we use later for Turkish in SS4.2), but for the purpose of the study, we compare well-known supervised and unsupervised methods:

* Byte-Pair-Encoding (BPE) and Unigram Language Model (uniLM)4 from SentencePiece Kudo and Richardson (2018). Footnote 4: We analysed several vocabulary sizes (4k, 8k, 16k, 32k, 64k) but report only the best one, which is 64k for all cases.
* Morfessor Poon and Domingos (2009).
* Pointer Generator Network (PtrNet) from the implementation of Mager et al. (2020).

#### 3.1.1 Datasets and evaluation

We used the CELEX dataset of segmented words for English and German Steiner (2016, 2017), where we randomly split training and evaluation data (80-10-10). Besides, for the unsupervised methods, we use the newscommentary-v15 Barrault et al. (2019) and EuroParl-v10 Koehn (2005) corpora5. Furthermore, we define two metrics to assess the performance on computing synthesis:

Footnote 5: Other languages like Danish are also available and was tested, but we did not report the results here as there are not complementary MT evaluation sets.

* Accuracy count: Evaluates if the number of obtained morphemes in the hypothesis segmentation is the same as in the reference.
* Exact segmentation precision: Analyses if the split morphemes are the same. We first perform an automatic alignment between the hypothesis and reference segments with the parallel Needleman-Wunsch algorithm for sequences Naveed et al. (2005), and then compute the exact match at morpheme level.

#### 3.1.2 Results and discussion

Table 1 shows the scores on morphological segmentation for both English and German. We observe that both BPE and uniLM under-perform when it is not expected to split the word (column "1"). This is a pattern observed by Bostrom and Durrett (2020), where they noted that unsupervised segmentation methods tend to over-split the roots of words. They both improve their accuracy and precision when

\begin{table}
\begin{tabular}{c|c c c||c c c c}  & \multicolumn{4}{c||}{**English**} & \multicolumn{4}{c}{**German**} \\ \hline \#morphs. & 1 & 2 & 3 & 4 & 1 & 2 & 3 & 4 \\  & 16,914 & 28,900 & 1,798 & 73 & 13,061 & 32,007 & 5,808 & 360 \\ \hline \hline \end{tabular} \multicolumn{8}{c}{**Accuracy Count**} \\ uniLM\({}_{64k}\) & 0.54 & 0.52 & 0.49 & 0.59 & 0.35 & 0.27 & 0.21 & 0.18 \\ BPE\({}_{64k}\) & 0.5 & 0.53 & 0.5 & 0.52 & 0.29 & 0.33 & 0.28 & 0.26 \\ Morfessor & 0.22 & 0.47 & 0.55 & 0.48 & 0.17 & 0.26 & 0.28 & 0.25 \\ PtrNet & **0.82** & **0.84** & **0.56** & **0.81** & **0.74** & **0.86** & **0.7** & **0.42** \\ \hline \multicolumn{8}{c}{**Exact Segmentation Precision**} \\ uniLM\({}_{64k}\) & 0.54 & 0.52 & 0.6 & 0.8 & 0.29 & 0.38 & 0.32 & 0.22 \\ BPE\({}_{64k}\) & 0.5 & 0.44 & 0.56 & 0.76 & 0.24 & 0.33 & 0.23 & 0.08 \\ Morfessor & 0.21 & 0.58 & **0.7** & 0.78 & 0.17 & 0.45 & 0.44 & 0.36 \\ PtrNet & **0.76** & 0.67 & **0.81** & **0.8** & **0.67** & **0.73** & **0.72** & **0.62** \\ \end{tabular}
\end{table}
Table 1: Accuracy count and segmentation precision for English and German using unsupervised and supervised segmentation methods. Results are grouped by the expected number of morphemes (e.g. ”1” means that the word should not be split).

 

the number of expected morphemes is larger. Unexpectedly, Morfessor also under-performs in the "1" case for both languages, and only surpasses the other unsupervised methods when we measure precision for many morphemes. Furthermore, The PtrNet supervised method outperforms the rest in almost all scenarios.

We conclude that, to compute synthesis, we should prioritise, besides a rule-based morphological analyser, a supervised segmentation method like PtrNet if data is available. We take advantage of this for the segment-level analysis in SS5.

### Fusion: Semi-automatic computation

Calculating fusion should be approached in a case by case scenario, as there are different considerations provided by Payne2017. Therefore, there is not an automatic tool that can obtain the fusion score directly. We decided to focus on Spanish6 as a case study, where verbs and auxiliary verbs contains the highest degree of fusion of all the parts-of-speech (POS).

Footnote 6: We chose this language because of the ease of finding annotators and MT training and evaluation data.

ProcedureWe observed that we could perform an annotation per paradigm and the termination of the verb (-ar, -er, -ir), as the fusion degree will remain the same regardless of the lemma7. Then, on a chosen Spanish corpus:

Footnote 7: Except for irregular ones, which presents a limitation and potential noise. To reduce the risk of a biased assessment, we also performed a human evaluation.

1. Perform an automatic annotation of POS and morphological features8. Footnote 8: We use the spaCy model es_dep_news_trf, available at [https://spacy.io/models/es#es_dep_usage](https://spacy.io/models/es#es_dep_usage)
2. Review the automatic annotation of special cases. For instance, there are specific verb forms that are missed as adjectives. We corrected the POS and morphological annotation of those cases in a manual step.
3. Obtain a set of all unique verb paradigms and morphological features in the corpus, considering the three different types of verb terminations in Spanish as different elements9. Footnote 9: Except for irregular ones, which presents a limitation and potential noise. To reduce the risk of a biased assessment, we also performed a human evaluation.

Now there is a list of unique verb paradigms and terminations that can be annotated both in synthesis and fusion. The steps are as follows:

1. For each unique verb paradigm and termination, segment a verb sample into its morphemes. E.g. the verb _hablo_ ('talked'), is split in _habil-\(\delta\)_, and _hablaramos_ ('we were to speak') in _habl-ara-mos_.
2. Analyse how many morphological features are fused in each morpheme: if you change a value of a feature, will the surface form or morpheme will change? E.g. in _habil-\(\delta\)_, -\(\delta\)_ participates in 5 features (mode (indicative), subject person (third person), subject number (singular), tense (past) and aspect (perfective)). For _halbl-ara-mos_, -_ara_ includes the past and subjunctive, whereas _-mos_ denotes the person and number. If any of aforementioned feature changes its value, the surface will change too.
3. Count and aggregate the results per morphemes_trf. It has an accuracy of 0.99 in POS and morphological tagging in the UD Spanish AnCora dataset [4], which contains news texts mostly.

Footnote 9: Using the Unimorph database [11] is another alternative for extracting all the possible unique inflections. We aligned and considered both tag sets for the annotation, as shown in Table 2.

\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|l|} \hline \multicolumn{2}{|l|}{Example (es):} & **HABLareons de la propuesta con la que se & **condenda** a la ex primer ministerà y fuc & **noyada** por 147 dipattods en la votación. \\ \hline Verbs & Features (spaCy) & Features (UniMorph) & Segmentation & featas. per morph & frisonal morph & total joints & Fusion index \\ \hline hbalzemos & Mood=Ind, Number=Pfur, & V:IND;FUT;1;PL & hald - rare - mos & 0 & -0+(2-1)+(2-1) & 2+2 = 4 & 0.5 \\ _(we will talk)_ & Person=1, Tense=Fut, & & & 2\({}_{\text{IND}}\)J(Ur) & -2 & & & \\  & VerbForm=Fin & & & & & & \\ \hline condenó & Mood=Ind, Number=Sing, & V:IND;PST;3;SG;PFV & conden – \(\delta\) & 0 & -0+(5-1) = 4 & 4+1 = 5 & 0.8 \\ _(confeuned)_ & Persona=1, Tense=Fast, & & & & 5\({}_{\text{ND}}\)J(PST), & & & \\  & VerbForm=Fin & & & & 5\({}_{\text{OP}}\)J(PV) & & & \\ \hline apoyada & Gender=Fern, Number=Sing, VerbForm=Part & & V:PTCP;STFAM;SG & apoys - ada & 0 & -0+(3-1) = 2 & 2+1 = 3 & 0.66 \\ _(supported)_ & & here=Sing, VerbForm=Part & & & 3\({}_{\text{SPSTEMAGSO}}\) & -0+(3-1) = 2 & 2+1 = 3 & 0.66 \\ \end{tabular}
\end{table}
Table 2: Annotation example in Spanish. We first identify the verbs (in bold) and obtain their morphological features (using spaCy and the UniMorph schema). Then, we split the verb into its morphemes (segmentation), and identify which features are fused in each morpheme (feats. per morph). Finally, we compute the index of fusion by dividing the fusional morpheme joints by the total joints (which includes the agglutinative or explicit boundaries). On a side note, examples of verbs with zero fusion are in the infinitive (e.g. hablar _(to talk)_) and gerund (e.g. hablando _(talking)_) forms.

 

phemes and obtain the fusion for each verb paradigm. E.g. the fusion for _habl-\(\diamond\)_ is 4/5 = 0.8, and for _habl-ara-mos_ is 2/4 = 0.5.

Finally, with the annotation in the unique list of verb inflections and terminations, we can extend the degree of fusion to all the verbs in the original Spanish corpus. An example of the annotation process is shown in Table 2.

## 4 Word-level analysis of Synthesis and Fusion in Machine Translation

In this analysis, we ask the following question: how difficult is translating a word concerning its index of synthesis or fusion? For evaluating synthesis, we work with Turkish10 nouns and verbs, and for fusion, we keep working on Spanish verbs. For both cases, English is the source language in the translation task.

Footnote 10: Turkish presents high synthesis and agglutination Zingler, 2018), meaning that there are words composed with several morphemes and the morpheme boundaries are explicit, respectively. We focus on verbs and nouns, which usually contain more morphemes than other parts-of-speech. We chose this language due to the availability of an open-source rule-based morphological analyser and an expert annotator.

### Experimental design

The experiment consists of comparing a gold standard reference with machine translation system outputs at the word level:

1. For both the reference and system output, we automatically **tag all the words with a morphological analyser** (the Boun morphological analyser and disambiguator Sak et al. (2008) for Turkish and an spaCy model trained on the Ancora Universal Dependency parser Taule et al. (2008) for Spanish). The POS is needed to filter the target words. For synthesis in Turkish, the number of morphemes works as a proxy, as we are working at the word level. For fusion in Spanish, we need the inflection to obtain the degree of fusion from the annotated unique list (see 3.2).
2. **Align the words** between the reference and system output. We use the awseome-align Dou and Neubig (2021) tool by fine-tuning the multilingual BERT Devlin et al. (2019) model for word-alignment, using the reference and system output as parallel corpora.
3. Calculate the translation accuracy (exact match of the word, 0 or 1) for the target POS.

We then fine-grain the results concerning the degree of synthesis (number of morphemes) or fusion. Additionally, we control different confounds: frequency of the word in the training set, and whether the full word is part of the vocabulary input of the model or not. Finally, we complement the analysis with a human evaluation (see SS4.4).

### Synthesis analysis: English\(\rightarrow\)Turkish Data

We use the newstest2018.en-tr evaluation set from WMT Bojar et al. (2018), with 3,000 samples. In the Turkish side there are 45,944 tokens, and Table 3 shows the distribution of the number of morphemes obtained with Sak et al. Sak et al. (2008).

ModelWe use an English-Turkish system trained with the TIL corpus of 39.9M parallel sentences Mirzakhalov et al. (2021). On the newstest2018.en-tr set, the performance is 13.06 and 49.54 in BLEU and chrF, respectively.

Results and discussionFigure 1 shows the average accuracy (exact translation, 0 or 1) of nouns and verbs in newstest2018.en-tr, where the number of morphemes is a proxy for the index of synthesis. In most cases, especially with a higher training frequency, we observe that the average accuracy drops as the number of morphemes increases from 1 to more. This is clearer in nouns than in verbs, which have fewer cases to analyse overall. Between 2, 3 or more than 4 morphemes the differences are not significant, and sometimes is not consistent (e.g. verbs with the highest frequency). However, we can argue that analytic nouns (synthesis=1) are easier to translate than synthetic nouns (synthesis>1) for the English\(\rightarrow\)Turkish direction. The pattern holds for whether the word is part of the vocabulary of the model or not, although rare words (frequency in \([0,10^{3}]\) have generally lower translation accuracy than more frequent words (frequency \(>100\)).

### Fusion analysis: English\(\rightarrow\)Spanish

DataWe use the newstest2013.en-es evaluation set from WMT Bojar et al. (2013) with 3,000 samples. In the Spanish side there are 62,055 tokens, with 6,317 verbs, and where 1,411 of them are more agglutinative (fusion=0) and 4,822 more fusional (fusion>0).

\begin{table}
\begin{tabular}{c|c c c c c}  & Total & \#1 & \#2 & \#3 & \#4 & \#5+ \\ \hline Verbs & 3,834 & 133 & 2,265 & 1,036 & 308 & 92 \\ Nouns & 10,680 & 5,899 & 2,974 & 1,556 & 244 & 7 \\ \end{tabular}
\end{table}
Table 3: Number of nouns and verbs in the Turkish reference set, and their respective number of morphemes.

 

ModelFor training, we use the MarianNMT toolkit Junczys-Dowmunt et al. (2018), a Transfomer-base model Vaswani et al. (2017) with default parameters, and four NVIDIA V100 GPUs. We obtained different English-Spanish models using the newscommentary-v8 Bojar et al. (2013) and EuroParl Koehn (2005) datasets with joint vocabulary sizes of 8k, 16k and 32k (using unigram-LM from SentencePiece Kudo and Richardson (2018)). For this analysis, we chose the best performing system: combining both datasets (2.2M sentences) with 16k pieces. On newstest2013.en-es, the performance is 31.6 BLEU points.

Results and discussionFigure 2 shows the average accuracy of verbs in newstest2013.en-es for verbs without and with some degree of fusion. In the two higher frequency subplots (middle and right), we can observe that the average accuracy of the non-fusional verbs is higher than the fusional ones, and the pattern holds whether the verb is present in the vocabulary input of the model or not. The exception is for the least frequent verbs, although this is explained as the model do not have enough information to learn from, regardless of their degree of fusion.

### Human evaluation

Exact translation accuracy has limitations, as there are potential translations that could be acceptable given a specific context (e.g. a synonym). For that reason, we performed a human evaluation of a sample of sentences on (10%) of each evaluation set, focusing on two scores11: Semantic score: evaluates the meaning of the word used in the automatic translation (system output) and how it compares with the gold standard translation. Scale goes from 1 (no relationship at all) to 4 (it is the same lemma). Flammar score: evaluates the grammatical form and how it compares with the gold standard translation. Scale goes from 1 (different inflection) to 3 (same inflection).

Footnote 11: Details of the annotation protocol are in the Appendix

SynthesisIn Figure 3, we show the annotation scores for the semantic and grammar metrics, for both nouns (top) and verbs (bottom). We also divide the analysis w.r.t. the frequency of the word in the training data. For nouns, we observe similar patterns as in the automatic analysis, where in the one-morpheme column, the proportion of the highest score is slightly larger than in the other columns, suggesting they are easier t

Figure 1: Accuracy (exact translation) for Nouns (top) and Verbs (bottom) in the English\(\rightarrow\)Turkish translations. Results are grouped by the training frequency of the words (less to more frequent from left to right), and each subplot presents the scores for all the words, and whether they belong or not to the vocabulary input of the model. The number of samples are stacked in each bar, and we do not show entries with less than 30 samples.

 

The pattern is even more explicit in the highest frequency block (rightmost one). The verbs tend to have more distributed grammar scores, suggesting the difficulty of generating inflected forms may remain equally high even when the words are more frequent. Single morpheme verbs are very rare in Turkish and generally contain exceptional forms which reflects in the low translation accuracy in the highest frequency block. We also observe that a good proportion of translated words with 'zero' accuracy (not the exact translation, see the orange inner bubbles) has been annotated with highest semantic (same lemma) or grammar (same inflection) score, suggesting in some cases that the model is successful in generalization.

FusionFigure 4 shows the semantic and grammar annotation scores for Spanish verbs. For the semantic scores (top), in all levels, the gap between the non-fusional and fusional verbs is reduced, for all the frequency groups. This means that the model is indeed able to generalise and offer alternative translations (not the exact verb), which is more complex to measure with automatic metrics. In the grammar scale (bottom), however, we still note a slight advantage in the maximum score (3) of the non-fusional verbs against the fusional ones for the two highest frequency subplots (middle and right). This indicates that, with highly frequent verbs, it is still more difficult to translate correct forms with a fusion degree higher than zero. Similarly as for synthesis, we observe that there is a significant proportion of 'zero' accuracy cases (orange inner bubbles) for the highest scores in most cases. This indicates that the model could generalise and translate verbs with similar meanings and not the exact, but close, forms.

## 5 Segment-level Analysis of Synthesis and Fusion in Machine Translation

Following up the word level analysis, we study the relationship between machine translation difficulty and the degree of synthesis or fusion at the segment level. For this purpose, we process a set of translation systems for the language pair we want to evaluate. The general steps are as follow:

1. For each system output, we compute **automatic evaluation metrics** (BLEU Papineni et al., 2002), chrF Popovic (2015) and/or COMET Rei et al. (2020)) with respect to

Figure 4: Semantic (top) and Grammar (bottom) annotation for Spanish.

Figure 3: Semantic score (left) and Grammar score (right) annotation for Turkish, for different frequency ranges of Nouns (top) and Verbs (bottom). Bubbles represent the proportion of the amount of scored annotations (1-4 or 1-3) divided by the subtotal elements of their respective columns (or number of morphemes). The orange inner bubble represents the amount of samples with ‘zero’ accuracy (in the automatic analysis) in each category.

 

the reference set, per sentence.12 Footnote 12: Based on the analysis of Kocmi et al. (2021), we prefer to report COMET and chrF over BLEU.
2. For each sentence of the evaluation set, we compute potential **predictor variables** for the automatic metric, such as the degree of synthesis or fusion. We complement the predictor variable list with other heuristics, such as the length of the sentence in characters (_char.count_) or words (_word.count_). The full list of all the predictors per language pair is in the Appendix.
3. With the previous inputs, we generate **generalized linear models** per system output and evaluation metric, in which each model's output is set to the predictor variables. The goal is to identify which predictors affect each method's performance.
4. Following model creation, we extract the significant predictors of each model. This provides an indication of which variables can be used to predict the outcome of the model's dependent variable - in our case the degree of synthesis or fusion, or any heuristic.13 Footnote 13: For simplification purposes, in the following analysis and plots, we only show the predictors that show a significant effect on the system outputs.

Synthesis on En-Tr and Tr-EnWe first start evaluating the English-Turkish and Turkish-English language pairs. The evaluated models are EnTr1, EnTr2, and TrEn2 (details in the Appendix). Also, as we are studying synthesis in Turkish, all predictors are computed on the Turkish side, regardless of the translation direction.

Figure 5 presents an overview of the significant predictors on En-Tr and Tr-En systems, where we observe a large impact of the _synthesis_ variable on the chrF scores of two different systems (EnTr1 and TrEn2). The only other heuristic that achieves a notable impact on a system output is _morph.count_, or the length of Turkish sentence in morphemes, split by a morphological analyser. Other predictors have a minor effect.

Fusion on En-Es and Es-EnIn a similar way, we evaluate the impact of fusion in English-Spanish (EnEs1, Ens2) and Spanish-English (EsEn1, EsEn2) models (see the Appendix for details). Again, as we are studying fusion in Spanish, all predictors are computed on the Spanish side, regardless of the translation direction.

Figure 6 presents an overview of the significant predictors, where we observe that _R.fusion.verb_, or the ratio of the degree of fusion over the number of verbs in the sentence, is the predictor that has the highest impact in most system outputs (EnEs1, EnEs2 and EsEn2). Additionally, _R.fusion.swEEsn2_ (or the ratio of the degree of fusion over the number of subwords input in the EsEn2 model) also has a high impact in one system output (EnEs2, which uses the same segmentation model as EsEn2).

Analysis on En-De and De-EnFinally, we extend the analysis to English-German and German-English language pairs, using the respective evaluation sets of the WMT2018 campaign Bojar et al. (2018), and the system outputs provided for all the participants (measured in BLEU). For computing synthesis, we use the different segmentation methods we compared in SS3.1. However, for fusion, we only use a shallow proxy with the number of morphological features that are tagged using a morphological analyser. In this case, the predictors are computed for both the source and target side.

Figure 5: Overview of significant predictors for degree of synthesis across our TR-EN and EN-TR models.

Figure 6: Overview of significant predictors for degree of fusion across our ES-EN and EN-ES models.

 

We present an overview of these significant predictors for German-English in Figure 7 (and the Appendix contains the results for English-German in Figure 8). We can observe that _ref.SYN.uniLM_ and _ref.SYN.PtrNet_ are the predictors that impact most of the different system outputs. These variables refer to the synthesis computed on the reference side (English) using uniLM or PtrNet as the morpheme segmentation method, respectively. Furthermore, we observe that _src-ref.R.feat.token_ has also some effect over one system output, which is a shallow proxy for the fusion degree in the source w.r.t. to reference segment (using the ratio of number of features per number of tokens).

## 6 Discussion

It is important to note the limitations of this study. Overall results do not suggest that translating into more analytic languages (e.g. Chinese) or more agglutinative ones (e.g. Turkish) is easier than their counterparts. Highly analytic ones present the significant issue of word coverage and vocabulary size of the model. Besides, we cannot isolate the fusional degree from synthesis entirely. For instance, Turkish is a highly agglutinative language, but also highly synthetic, and there are languages that present both agglutinative and fusional traits, like Navajo. Moreover, the language scope is another limitation: is it possible to extend it to further languages in a practical way? Synthesis can be calculated directly only if the morphological analyser splits the word into morphemes, and fusion poses several issues as mentioned before. Furthermore, Payne (2017) also indicated that the discourse can impact the computed degrees due to the diversity of the vocabulary. This study focuses on news data only, and it will be relevant to extend it to different domains.

To address the limitations, we consider that our word level analysis, that targets specific POS, has been fundamental to enable the study of the indexes, and to partially isolate them from each other. The selection of our study cases was also relevant. Spanish verbs do not present more than three morphemes, keeping a low synthesis value across all the analysis, whereas Turkish is more agglutinative than fusional. Moreover, to rapidly extend the evaluation for new languages and domains, we could follow a less fine-grained analysis in each index. For instance, we can compare synthesis=1 vs. synthesis>1 instead of granulating per number of morphemes, or fusion=0 vs. fusion>0, as we did in this work.

## 7 Conclusion and future work

In conclusion, we proposed methods to quantify the indices of synthesis and fusion in automatic and semi-automatic ways, respectively. Besides, for the chosen language pairs, we observed that the studied degrees have an impact in machine translation performance at both word and segment level, where we included a human evaluation of the former case.

Our analysis opens the possibility for further fine-grain evaluation approaches for MT and other NLP generation tasks. For instance, as future work, we can ask: are we improving the automatic translation of highly frisonal words or segments? Following our methodology, we could stratify evaluation sets to measure how our models performs in different parts of the spectrum. Besides, the indices could also be helpful for evaluation approaches in morphological segmentation. Furthermore, another potential research avenue is to aid model training in MT: e.g. knowing which segments are more or less synthetic and/or fusional could be beneficial for sampling strategies.

Figure 7: Overview of significant predictors across DE-EN models.

 

Ethical Considerations

The annotations in this paper were compensated accordingly (see Appendix). Also, for all the datasets used in the research, we stick to the ethical standards giving credit to the original author. We encourage future work that take advantage of these resources, to cite also the original sources of the data.

## Acknowledgements

This work was supported by funding from the European Union's Horizon 2020 research and innovation programme under grant agreements No 825299 (GoURMET) and the EPSRC fellowship grant EP/S001271/1 (MTStretch). Also, we acknowledge the support from eBay. Besides, the study was performed using resources provided by the Cambridge Service for Data Driven Discovery (CSD3) operated by the University of Cambridge Research Computing Service ([http://www.csd3.cam.ac.uk/](http://www.csd3.cam.ac.uk/)), provided by Dell EMC and Intel using Tier-2 funding from the Engineering and Physical Sciences Research Council (capital grant EP/P020259/1), and DiRAC funding from the Science and Technology Facilities Council (www.dirac.ac.uk). We express our thanks to Kenneth Heafield and Rico Sennrich, who provided us with access to the computing resources.

Moreover, the first author was granted financial support from the European Association for Machine Translation (EAMT), under its programme "2020 Sponsorship of Activities", and from the European Cooperation in Science and Technology (COST) under the programme CA18231 - Multi3Generation: Multi-task, Multilingual, Multimodal Language Generation.

## References

* Amrhein and Sennrich [2021] Chantal Amrhein and Rico Sennrich. 2021. How suitable are subword segmentation strategies for translating non-concatenative morphology? In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 689-705, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Barrault et al. [2019] Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Muller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_, pages 1-61, Florence, Italy. Association for Computational Linguistics.
* Bjerva and Augenstein [2018] Johannes Bjerva and Isabelle Augenstein. 2018. From phonology to syntax: Unsupervised linguistic typology at different levels with language embeddings. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 907-916, New Orleans, Louisiana. Association for Computational Linguistics.
* Bjerva and Augenstein [2021] Johannes Bjerva and Isabelle Augenstein. 2021. Does typological blinding impede cross-lingual sharing? In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 480-486, Online. Association for Computational Linguistics.
* Bjerva et al. [2019a] Johannes Bjerva, Yova Kenentchedijhieva, Ryan Cotterell, and Isabelle Augenstein. 2019a. A probabilistic generative model of linguistic typology. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 1529-1540, Minneapolis, Minnesota. Association for Computational Linguistics.
* Bjerva et al. [2019b] Johannes Bjerva, Yova Kenentchedjiheva, Ryan Cotterell, and Isabelle Augenstein. 2019b. Uncovering probabilistic implications in typological knowledge bases. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3924-3930, Florence, Italy. Association for Computational Linguistics.
* Bjerva et al. [2020] Johannes Bjerva, Elizabeth Salesky, Sabrina J. Mielke, Aditi Chaudhry, Giuseppe G. A. Celano, Edoardo Maria Ponti, Ekaterina Vylomova, Ryan Cotterell, and Isabelle Augenstein. 2020. SIGTYP 2020 shared task: Prediction of typological features. In _Proceedings of the Second Workshop on Computational Research in Linguistic Typology_, pages 1-11, Online. Association for Computational Linguistics.
* Bojar et al. [2013] Ondrej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. Findings of the 2013 Workshop on Statistical Machine Translation. In _Proceedings of the Eighth Workshop on Statistical Machine Translation_, pages 1-44, Sofia, Bulgaria. Association for Computational Linguistics.
* Bojar et al. [2018] Ondrej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_, pages 272-303, Bel 

gium, Brussels. Association for Computational Linguistics.
* Bostrom and Durrett (2020) Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pretraining. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 4617-4624, Online. Association for Computational Linguistics.
* Comrie (1989) Bernard Comrie. 1989. _Language universals and linguistic typology: Syntax and morphology_. University of Chicago press.
* Creutz and Lagus (2002) Mathias Creutz and Krista Lagus. 2002. Unsupervised discovery of morphemes. In _Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning_, pages 21-30. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Dou and Neubig (2021) Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 2112-2128, Online. Association for Computational Linguistics.
* Eskander et al. (2019) Ramy Eskander, Judith Klavans, and Smaranda Muresan. 2019. Unsupervised morphological segmentation for low-resource polysynthetic languages. In _Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonology, and Morphology_, pages 189-195, Florence, Italy. Association for Computational Linguistics.
* Greenberg (1960) Joseph H Greenberg. 1960. A quantitative approach to the morphological typology of language. _International journal of American linguistics_, 26(3):178-194.
* Harris (1951) Zellig S Harris. 1951. Methods in structural linguistics.
* Junczys-Dowmunt et al. (2018) Marcin Junczys-Dowmunt, Kenneth Heafield, Hieu Hoang, Roman Grundkiewicz, and Anthony Aue. 2018. Marian: Cost-effective high-quality neural machine translation in C++. In _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation_, pages 129-135, Melbourne, Australia. Association for Computational Linguistics.
* Kocmi et al. (2021) Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In _Proceedings of the Sixth Conference on Machine Translation_, pages 478-494, Online. Association for Computational Linguistics.
* Moehn (2005) Philipp Moehn. 2005. Europarl: A parallel corpus for statistical machine translation. In _Proceedings of Machine Translation Summit X: Papers_, pages 79-86, Phuket, Thailand.
* Kudo (2018) Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
* Kato and Richardson (2018) Taku Kato and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
* Mager et al. (2020) Manuel Mager, Ozlem Cetinoglu, and Katharina Kann. 2020. Tackling the low-resource challenge for canonical segmentation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5237-5250, Online. Association for Computational Linguistics.
* McCarthy et al. (2020) Arya D. McCarthy, Christo Kirov, Matteo Grella, Amrit Nidhi, Patrick Xia, Kyle Gorman, Ekaterina Vylomova, Sabrina J. Mielke, Garrett Nicolai, Miikka Silfverberg, Timofey Arkhangelskiy, Nataly Krizhanovsky, Andrew Krizhanovsky, Elena Klyachko, Alexey Sorodin, John Mansfield, Valts Ernestris, Yuval Pinter, Cassandra L. Jacobs, Ryan Coterell, Manas Hudlen, and David Yarowsky. 2020. UniMorph 3.0: Universal Morphology. In _Proceedings of the 12th Language Resources and Evaluation Conference_, pages 3922-3931, Marseille, France. European Language Resources Association.
* Mirzakhalov et al. (2021) Jamshidek Mirzakhalov, Anoop Babu, Duygun Ataman, Sherzed Kariev, Francis Tyers, Otabek Abduraufov, Mammad Hajili, Sardana Ivanova, Abror Khaytbaev, Antonio Laverytheta Jr., Bekhzdobbek Moydhinboyev, Esra Onal, Shaxnoza Pulatova, Ahsan Wahah, Orhan Firat, and Sriram Chellappan. 2021. A large-scale study of machine translation in Turkic languages. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 5876-5890, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Naveed et al. (2005) Tahir Naveed, Imitaz Saeed Siddiqui, and Shaftab Ahmed. 2005. Parallel needleman-wunsch algorithm for grid. In _Proceedings of the PAK-US International Symposium on High Capacity Optical Networks and Enabling Technologies (HONET 2005), Islamabad, Pakistan_.
* Oncevag et al. (2020) Arturo Oncevag, Barry Hadlow, and Alexandra Birch. 2020. Bridging linguistic typology and multilingual machine translation with multi-view language representations. In _Proceedings of the 2020 Conference 

on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2391-2406, Online. Association for Computational Linguistics.
* Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Payne [2017] Thomas E Payne. 2017. Morphological typology. In _The Cambridge Handbook of Linguistic Typology_, pages 78-94. Cambridge University Press.
* Ponti et al. [2019] Edoardo Maria Ponti, Helen O'Horan, Yevgeni Berzak, Ivan Vulic, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2019. Modeling language variation and universals: A survey on typological linguistics for natural language processing. _Computational Linguistics_, 45(3):559-601.
* Poon and Domingos [2009] Hoifung Poon and Pedro Domingos. 2009. Unsupervised semantic parsing. In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing_, pages 1-10, Singapore. Association for Computational Linguistics.
* Popovic [2015] Maja Popovic. 2015. chrF: character n-gram F-score for automatic MT evaluation. In _Proceedings of the Tenth Workshop on Statistical Machine Translation_, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
* Rei et al. [2020] Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2685-2702, Online. Association for Computational Linguistics.
* Sak et al. [2008] Hasim Sak, Tunga Gungor, and Murat Saraclar. 2008. Turkish language resources: Morphological parser, morphological disambiguator and web corpus. In _International Conference on Natural Language Processing_, pages 417-427. Springer.
* Sapir [1921] Edward Sapir. 1921. Types of linguistic structure.
* Sennrich et al. [2016] Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
* Steiner [2016] Petra Steiner. 2016. Refurbishing a morphological database for German. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)_, pages 1103-1108, Portoroz, Slovenia. European Language Resources Association (ELRA).
* building a morphological treebank for German from two resources. In _Proceedings of the 16th International Workshop on Treebanks and Linguistic Theories_, pages 146-160, Prague, Czech Republic.
* Taule et al. [2008] Mariona Taule, M. Antonia Martri, and Marta Recasens. 2008. AnCora: Multilevel annotated corpora for Catalan and Spanish. In _Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)_, Marrakech, Morocco. European Language Resources Association (ELRA).
* Vania and Lopez [2017] Clara Vania and Adam Lopez. 2017. From characters to words to in between: Do we capture morphology? In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2016-2027, Vancouver, Canada. Association for Computational Linguistics.
* Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_, pages 5998-6008. Curran Associates, Inc.
* Xu et al. [2020] Hongzih Xu, Jordan Kodner, Mitchell Marcus, and Charles Yang. 2020. Modeling morphological typology for unsupervised learning of language morphology. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6672-6681, Online. Association for Computational Linguistics.
* Zingler [2018] Tim Zingler. 2018. Reduction without fusion: Grammaticalization and wordhood in turkish. _Folia Linguistica_, 52(2):415-447.

## Appendix A Human Evaluation

### Annotation Protocol

This study measures the translation quality of translations generated by a translation system. You are given a list of sentences where one column lists each word in the gold standard (correct) translation and the corresponding column the system-generated translations. The evaluation of the translations will rely on the two scores described below. The scores to use in the evaluation are:

Semantic scoreevaluates the meaning of the word used in the automatic translation (system output) and how it compares with the gold standard translation.

Please assign each word in the output one of the scores you find most appropriate:

1. There is no relationship between the two lemmas 

2. The lemmas are different but the translation does not fit well in the context
3. The lemmas are different but it is still an acceptable translation (e.g. synonym)
4. It is the same lemma

Grammar scoreevaluates the grammatical form and how it compares with the gold standard translation.

Please assign each word in the output one of the scores you find most appropriate:

1. The word is inflected in a different way and it is not necessarily correct
2. The word has different inflection but it is still grammatically correct
3. The words have the same inflection, and it is correct

Please annotate all words in the translations in the file shared with you. In your evaluation try assigning the two scores to each word independently. The inflection of the word measures the morphological feature and should also be evaluated independently from the analyzer output which is automated and may contain errors.

The file contains example annotations for your reference, please ask any questions related to unresolved annotation examples by contacting the project coordinators.

### Annotators

For both Turkish and Spanish, the annotators were contacted directly due to their expertise in morphology (both of them are PhD students in Linguistics and Computational Linguistics, respectively), besides requiring that they are native speakers of the target languages. Also, they were paid more than the minimum wage per hour of annotation of their country of residence, and were told that the annotated data will be released upon acceptance of the study.

## Appendix B Segment-level Analysis of Synthesis and Fusion

### List of machine translation systems

* EnTr1: the same system used in SS4.2
* EnTr2: Transformer-base model Vaswani et al. (2017) with joint vocabulary size of 8k pieces (unigram language modelling from SentencePiece Kudo and Richardson (2018), and trained with a sample (10%) of the corpus of EnTr1.
* EnEs1: the same system used in SS4.3
* EsEn1: similar configuration than Ens1 but in the opposite direction
* EnEs2: same configuration as EnEs1 (model and vocabulary) but with smaller training data. It uses only newsfcommentary-v8 data, with around 300k sentences).
* EsEn2: similar configuration than EnEs2 but in the opposite direction.

### List of predictors

Tables 4, 5 and 6 describes all the predictors used at the segment level analysis of English-Turkish, English-Spanish and English-German (both directions), respectively.

### Results on English-German

Figure 8 shows the analogous results for English to German, where the synthesis-based variables presents a high impact w.r.t. the other predictors.

\begin{table}
\begin{tabular}{l|p{199.2pt}} Predictor & Description \\ charcount & number of characters \\ word_count & number of words (no punct. or numbers) \\ morphn_count & number of morphemes. \\ synthesies & ratio of morphn_count / word.count \\ N+Vword_count & number of Nonus and Verbs \\ N+Vmorph.count & number of morphemes of the Nouns and Verbs \\ N+Vsynthesis & ratio of N+Vmorph.count / word.count \\ swEfl2.Count & number of subwords processed by the EsFnT1 model \\ swEInTe2.count & number of subswords processed by the EnTr2 model \\ swThFe2a.count & number of subswords processed by the TrEn2 model \\ synsyn.swEntTi & ratio of swEnTf.count / word.count (synthesis proxy) \\ syn.swEntTi & ratio of swEnTf.count / word.count (synthesis proxy) \\ sym.svFinTi@2 & ratio of swEnTf.count / word.count (synthesis proxy) \\ \end{tabular}
\end{table}
Table 4: List of predictors for En-Tr and Tr-En. All variables are computed on the Turkish segment of the evaluation set.

\begin{table}
\begin{tabular}{l|p{199.2pt}} Predictor & Description \\ charcount & number of characters \\ word\_count & number of words (no punct. or numbers) \\ verb\_count & number of verbs \\ fusion & sum of the degree of fusion of all the verbs in the segment \\ R.fusion\_verb & ratio of fusion / verb\_count \\ R.fusion\_word & ratio of fusion / word\_count \\ swEu 

* [26] M. C. Bader, A. K. Goyal, and R. J. Sorensen. The role of the social network in a large-scale social network. In _Proceedings of the 10th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 357-384, 2009.
* [27] D. E. Kingma and P. Ba. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6120_, 2014.
* [28] H. Li, Y. Zhang, X. Liu, and Z. Sun. Learning to improve the learning rate by reducing the number of hidden layers. In _Advances in Neural Information Processing Systems_, volume 1, pages 1-10, 2015.
* [29] L. Ma, T. Wang, and W. Chen. Learning to improve the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [30] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [31] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [32] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [33] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [34] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [35] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [36] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [37] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [38] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [39] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [40] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [41] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [42] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [43] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [44] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [45] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [46] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [47] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [48] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [49] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [50] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [51] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [52] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [53] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [54] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [55] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [56] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [57] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [58] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [59] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [60] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [61] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [62] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [63] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [64] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [65] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [66] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [67] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [68] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [69] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [70] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [71] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [72] N. I. Ng, M. C. Bader, and R. J. Sorensen. The impact of the learning rate by reducing the number of hidden layers. In _Proceedings of the 1st International Conference on Machine Learning_, pages 1-10, 2015.
* [73]