GATITOS: Using a New Multilingual Lexicon for Low-resource Machine Translation
Authors: Alexander Jones
Isaac Rayburn Caswell
Orhan Firat
Ishank Saxena
Venue: EMNLP 2023 Main (2023)
Keywords: machine translation
low-resource
lexicons
dictionaries
unsupervised
NMT
MT
data augmentation
TL;DR: Data augmentation using multilingual lexicons improves the performance of massively multilingual NMT models on low-resource languages

Abstract: Modern machine translation models and language models are able to translate without having been trained on parallel data, greatly expanding the set of languages that they can serve.  However, these models still struggle in a variety of predictable ways, a problem that cannot be overcome without at least some trusted bilingual data. This work expands on a cheap and abundant resource to combat this problem: bilingual lexica. We test the efficacy of bilingual lexica in a real-world set-up, on 200-language translation models trained on web-crawled text. We present several findings:  (1) using lexical data augmentation, we demonstrate sizable performance gains for unsupervised translation; (2) we compare several families of data augmentation, demonstrating that they yield similar improvements, and can be combined for even greater improvements; (3) we demonstrate the importance of carefully curated lexica over larger, noisier ones, especially with larger models; and (4) we compare the efficacy of multilingual lexicon data versus human-translated parallel data. Based on results from (3), we develop and open-source GATITOS, a high-quality, curated dataset in 168 tail languages, one of the first human-translated resources to cover many of these languages.

GATITOS:
Using New Multilingual Lexicon for Low-resouree Machine Translation
Alex Jones".
Isaac Caswell
Ishank Saxena
Orhan Firat
Google Research
Abstract
(MMMT) refers to the concept of training single machine translation model on many languages and language pairs using a shared set of parameters, and has also seen success in recent years (Firat et al., 2016; Wu et al., 2016; Johnson et al., 2017; Aharoni et al., 2019; Fan et al., 2022; NLLBTeem et al., 2022; Bapna et al., 2022: Siddhant et al.. 2022). Training these models typically relies large-scale parallel corpora mined from the web (Resnik and Smith, 2003; Uszkoreit et al., 2010; Esplà-Gomis. 2009; Bañón al., 2020).
Modern machine translation models and lan- guage models able to translate without hav- ing been trained on parallel data, greatly ex- panding the set of languages that they can serve. However, these models still struggle in vari- ety of predictable ways, problem that can- not overcome without least some trusted bilingual data. This work expands cheap and abundant resource to combat this prob- lem: bilingual lexica (BILEXs). We test the efficacy of bilingual lexica in real-world set- up. on Longuang translation models trained web-crawled text. We present several find- ings: (1) using lexical data augmentation, we demonstrate sizable performance gains for un- supervised translation; (2) compare several families of data augmentation, demonstrating that they yield similar improvements, and can be combined for even greater improvements; (3) we demonstrate the importance of carefully curated lexica over larger, noisier ones, espe- cially with larger models; and (4) we compare the efficacy of multilingual lexicon data ver- human-translated parailel data. Based results from (3), we develop and open-source GATITOS, high-quality, curated dataset cover- ing 170 mostly low-resource languages the time of this submission, one of the first human- translated resources to support many of these languages'.
However, beyond the traditional technique of training NMT models with human-translated paral- lel texts, a number of other strategies have shown success recently, especially on lower-resource lan- guages. One of these techniques is self-superviser training using monolingual corpora (Siddhant et al.. 2020; Cheng et al., 2021). With this approach, NMT models are pretrained or jointly trained on a self-supervised task with monolingual data, such as the MASS (Song et al., 2019b) or BART (Lewis et al., 2020; Liu et al., 2020) tasks, as well as the usual neural machine translation task. This training regime can aid the model in performing zero-shot translation (Bapna et al., 2022; Siddhant et al., 2022), in cases where language has mono- lingual data but parallel data. Moreover, both the self-supervised task and the supervised MT task can be modeled as neural (Seq2Seq) problems, meaning a single Seq2Seq model can be used for training on both tasks.
1 Introduction
Neural machine translation (NMT) has emerged the dominant way of training machine transla- tion models (Bahdanau al., 2015), where trans- lation is modeled as a sequence-tor sequence-to-sequenn -esuueecc task to be learned by neural networks (Sutskever et al., 2014). Massively multilingual machine translation
Other techniques that have proven useful for low-resource MT include back-translation (Sen- nrich et al., 2016; Caswell et al., 2019; Feld- man and Coto-Solano, 2020) and the incorporation of language models into MT training (Gulcchre et al., 2017; Baziotis et al., 2020; Freitag al., 2022b). There has also been extensive work on training completely unsupervised MT systems us- ing monolingual corpora only (Artetxe et al., 2017, 2019). For example, Artetxe et al. (2017) uses a
"Equal contribution. icaswell@google com, 1 com. Work done while interning on the Translate team Google.
Correspondence
hhttps://github..e htt.ggithuu..c.omeeeee www..//aw/oooooeeoooooooooooooee research/ur1-n1pp tree/main/gatitos

---

combination of denoising autoencoding with pre- trained cross-lingual embeddings and on-the-fly back-translation to achieve reasonable MT perfor- mance with zero parallel data.
retain their efficacy, but the quality of the translated bilingual lexica becomes more important than the sheer quantity of lexical data points used for data augmentation. For instance, small, high-quality lexica like GATITOS show about 5x larger CHRF improvement than larger, noisier lexica like Panlex. Throughout this paper, experiments are done on only 24 GATITOS languages; based on the success of these, we expand the dataset to 170 languages. all low-resource, and open-source
In our work, we supplement the approach that combines supervised and self-supervised training with multilingual lexica. The motivation for us- ing this resource is as follows. Despite the suc- cesses of the approach combining supervised and self-supervised training, cross-lingual vocabulary alignment is still highly imperfect in these mod- els, especially for low-resource and unsupervised languages (see Bapna et al. (2022) for examples of some common failure modes). That is, training on all languages using a shared set of parameters is insufficient to induce perfect cross-lingee -onsull vo- cabulary alignment. Of course, we are not the first experiment with multilingual lexica to improve NMT performance, or multilingual NLP applica- tions more generally; Section gives more details.
2 Related Work
A number of works have looked at using multi- lingual lexicon data augmentation for NMT and other NLP tasks. The first class of augmenta- tions that we experiment with is "codeswitchingg" where words in the source sentence are swapped out for their dictionary translations to create mixed- language sentences. This approach has been used for a range of multilingual NLP tasks, including MT (Reid and Artetxe, 2022: Yang et al., 2020; Liu et al., 2021; Lin et al., 2020, 2021; Pan et al., 2021; Yang al., 2021; Kumar al., 2022: Khatri et al., 2021; Kuwanto et al., 2021; Xia al., 2019). Many of these, however, only look at codeswitch- ing between the source and target languages, e.g. substituting source words with dictionary transla- tions into the target language, or word-for-word BiLex translations of the target to make synthetic back-translated data (Nag et al., 2020). Qin et al. (2020) experiment with codeswitching on NLI, sen- timent classification, document classification, dia- logue state tracking, and spoken language under- standing, Malon (2021) looks at codeswitching em- beddings for language modeling, and Wang et al. (2022) experiment on NER, POS tagging, and de- pendency parsing, Another similar work is Chaud- hary et al. (2020), in which the MLM task is modi- fied such that instead of predicting masked source tokens in the source language, the authors provided language embeddings to cue the model predict the masked tokens different language instead. Codeswitching augmentations go by a variety of different names, e.g. "dictionary denoising" (Reid and Artetxe, 2022), "Random Aligned Substitution" (Lin et al., 2020), or "code-mixing". In our paper, we will stick to the term "codeswitching"" though will try to point out where identical or similar approach has been tried under a different name.
Using the publicly available massively multilin- gual lexicon Panlex (Kamholz et al., 2014), we demonstrate that this added lexical data leads to small but significant gains over a baseline model on average, even for high-resource languages; and with smaller but carefully curated bilingual lexica, the gains are substantially larger. In both cases, the gains most significant for unsupervised and low-resource languages. Our contributions are as follows:
1. We provide a thorough comparison of several lexicon-based data augmentation variants for MT, all of which are simple, generalizable, and easy to implement:
2. We test these approaches "in the wild", i.e. in highly multilingual, web-mined data regime such as production systems tend to use, with hundreds of languages and billions of monolingual and parallel sentenees;
3. We explore the effects of lexical data quality and quantity;
4. We demonstrate the efficacy of bilingual lexicon-based approaches as models scale;
5. We open-source the high-quality multilingual GATITOS lexicon for low-resource languages.
The tl;dr of this paper is that bilingual lexica help low-resource and zero-shot NMT in almost all cases, and that most training-tinm time augmentation methods have similar efficacy, and can be com- bined to be more effective. When scaling up to larger and more expressive models, these methods
The second class of augmentations we experi- ment with involves prepending lexicon translations

---

source sentences as additional cross-lingual sig- nal, as instead of swapping out words in the source sentence. This approach has been tried as well for enhancing MT performance, e.g. in Song et al. (2019a); Maheshwari et al. (2022); Niehues (2021); Michon et al. (2020); Zhong and Chiang (2020); Susanto et al. (2020), and for similar tasks like lan- guage modeling (Yu et al., 2021). One potential advantage this approach has over the codeswitching method that it can be applied inference time as well: multilingual lexicon entries can be prepended to sentence queries to steer the model toward more accurate word translations. Outside NMT mod- els, this lexical prompting approach has also been applied to translation with LLMs: Ghazvininejad et al. (2023) provide LLMs with dictionary transla- tions of some of the source sentence words, which the model can use cover gaps in its vocabulary coverage (although the authors do not experiment with truly low-resource languages). With the rise in popularity of LLMs for MT and other tasks, this is an exciting area for further research.
ments translated into 170 very low-resource lan- guages; however, the time of the experiments in the paper, it covered only 24. The English source text is a mixture of words from a variety of sources, including frequent tokens in the En- glish language, words for numbers, months, days of the week, Swadesh words, names of the lan- guages themselves (including the endonym), and few short sentences. The tokens were manually reviewed by the authors of this paper ensure they looked reasonable. As the name implies, this dataset is mostly very short, consisting of 93% sin- gle tokens. There are also some short sentences, though only 23 entries have over 5 tokens. We hope this dataset will complement existing publicly available multilingual lexicons like MUSE (Con- neau et al., 2017; Lample al., 2017)
3.5 Panlex
Panlex (Kamholz al., 2014) is a free, open-access massive online database consisting of word and phrase translations for 5000+ languages, sourced from 2500 individual dictionaries. Panlex contains 1.3B translations across all language pairs. For our experiments, we use a subset of the Panlex database covering 177 languages and containing 66M word pairs. Languages were chosen largely by availability of eval sets; details in Appendix J.1.
3 Training data
Models are trained on sentence-level web text.
3.1 Monolingual data
The monolingual training data is from clean, sentence-level web-mine following the approach set forth in Caswell et al. (2020). To make rapid training and development possible, we subsampled the monolingual data for the 100 highest-resource languages to 10% of its original size. In sum, this totaled about 4B sentences, or about 80B tokens. For the large model experiments in section 8.1, we used the full data, totaling 27B sentences (540B tokens).
Evaluation
We use two translation evaluation sets: FLORES- 200 (NLLBTeam al., 2022; Goyal al., 2021; Guzmán et al., 2019). an open-sourced evalua- tion set consisting of 2009 English Wikipedia sen- tences translated by humans into 200 languages, and GATONES, an in-house evaluation set of 1,200 English sentences translated into various languages. We use the SacreBLEU (Post, 2018) 8) implementa- tion of CHRF for our evaluation metric. Higher- quality, embedding- based metrics like BL.ERR (Sellam et al., 2020) are not available for these languages, and CHRF seems to be the one of best of the surface-level metrics for low-resource lan- guages (Bapna et al., 2022; Koemi et al., 2021; Freitag et al., 2022a).
3.2 Parallel data
We use an in-house mine of parallel data. It tends to be much noisier than the monolingual data. All par- allel data are sampled to 10% of their original size, resulting in 9B parallel sentences (162M tokens) into English and the same number out of English, as well as 700K non- Son---nglihh-enteee sentence-pain The large models in Section 8.1 use the full dataset.
For this study, we only evaluate on English- centric directions. The reason is that, although both evaluation sets multi-way -ay parallel, they both also English-orige original, and thus lose the same infor-
3.3
Multilingual lexica
3.4
GATITOS
The GATITOS dataset is a new dataset open-sourced in this paper. It consists of 4000 short English seg-
2 "signature nrefs: 1 case: mixed|eff: press/nce In: n |space: no version:2.3.1

---

mation that pivot translations do. Furthermore, this study places more weight on the en-txx direction than the xx ex direction. The main reason for this is that the xx -x- direction is generally an easier di- rection for models to learn (since they see so much more English text), so the en-*xx direction is usu- ally the limiting reagent when it comes to model quality; result we care more about improving this direction.
data. While variety of papers (e.g. Reid and Artetxe (2022); Yang et al. (2020)) have explored specific augmentations on particular language pairs, we believe our paper is the first to undertake a rig- orous comparison of different augmentation strate- gies across hundreds of languages in real-world setting.
6.1 Codeswitching
In our "codeswitching" augmentation strategy, words in the source sentence are swapped out for their dictionary translations to create mixed- language sentences. We experiment with this aug- mentation both monolingual and parallel data. The details of this method are described below.
5
Model
For our experiments we use Transformer Big encoder-decoder model (Vaswani al., 2017) with approximately 475M parameters. We train each model for 400K steps on 64 TPU v2 chips. Our models assign a 40% weight to the translation task, and a 60% weight to the MASS task. For mod- els augmented with monolingual data augmenta- tion, we split the 60% weight on the MASS task into 30% weight on the augmented data and 30% weight on the non-augmented data. Parallel data augmentations were done in an analogous way. For the raw-token- pair augmentation, we add in this task with a 5% weight and shrink the other weights accordingly. We use a task-specific token for each of the tasks the models may see, namely translation, MASS, GlowupMono, GlowupParallel, CodeswitchMong and CodeswitchParalle..
6.1.1 Codeswitched Autoencoding
Our multilingual codeswitching autoencoding (MCA) approach is similar the "dictionary de- noising" objective in Reid and Artetxe (2022). Let D represent multilingual lexicon containing word or phrase translation pairs for many languages. Given a source sentence x = (I1 ( ..., x,,) from monolingual corpus Xmono. we substitute each to- ken in x for its dictionary translation with proba- bility Pt, = 0.4. (More implementation details in Appendix Section H). Note that Reid and Artetxe (2022) also apply additional noise on top of codeswitching, along the lines of (m)BART (Lewis et al., 2020; Liu al., 2020). For simplicity and so we can better examine the effects of lexicon information in isolation, we do not do this.
6 Methods
In this paper, we divide our augmentation ap- proaches into two classes: "codeswitchingg" ap- proaches, which involve substituting source sen- tence words for their dictionary translations, and "GLOWUP," " which entails prepending dictionary translations of source words to source sentences. The main difference between these approaches is whether dictionary translations are substituted for source text (in the case of codeswitching) added the sentence (in the case of GL GLOWUP). As third augmentation, we experiment with training on raw lexicon token pairs directly, treating them like any other parallel data.
6.1.2 Codeswitching MT
Our codeswitching MT task is essentially the same approach as described in Section 6.1.1, except it applies to parallel rather than monolingual data. Given a source sentence from parallel corpus X paraltet, we perform the identical procedure de- scribed in section Section 6.1.1 to obtain multilin- gual codeswitched sentence x'. We then train the model the translation task using sentence pairs (a', y), where (æ,, y) is a sentence pair in X parallel This method is effectively identical to the Random Aligned Substitution method proposed Lin et al. (2020). As with MCA, we use ptr = 0.4 and apply the augmentation on half the available parallel data.
The novelty of our contribution lies not so much in any one of our methods, but rather in (1) the ap- plication of these methods to unsupervised machine translation; (2) the number of methods we compare in controlled experiments; (3) the scale of our ex- periments, in terms of number of languages, data quantity, and model capacity; and (4) the applica- tion of these methods to "in the wild" web-crawled
6.2 Lexical prompting (GLOWUP)
The second class of lexical augmentations we ex- periment with is lexical prompting, which we

---

call GLOWUP (Guiding Lexical Outputs With Un- derstandable Prompts). This method prepends (sre, transl) pairs to the beginning of source sentence for some uniform random fraction of the words in that sentence that are in out bilex. These hints can then be used to help the model guess the translation or the denoised sentence. The GLOWUP task has the advantage that it can be used infer- ence time, without retraining model, and may be simpler implement. However, does result in longer and less balanced sequence lengths, which can pose problems for decoding.
7.1.2 Token-pair-only model
In addition the baseline model, we also experi- ment with the extremely simple approach of pro- viding raw word pairs from multilingual lexica to the model as additional parallel data. That is, given dictionary entry and its translation t, we pro- vide the model with a "sentence" pair of the form (<2translation> <2lang> <2script> s. t). We call this token-pair baseline GatiPanlexTokenPPan
7.1.3 Single augmentation models
We also train models on each of the augmenta- tions described in Section As noted in that sec- tion, we only augment half the relevant data (mono- lingual, parallel, or both) before training each of these models, leaving the other half to be trained identically to the baseline (i.e. joint training on the MT task and MASS). The models with sin- gle augmentation are named after their augmenta- tion, viz. Modeswitch.Mnno, CodeswitchParal- lel, GlowupMono, and GlowupParallel; those with two or more augmentations include them all in the name, viz. CodeswitchhMonoParaalll GlovupMonoParalle, Codeswitch lelGatiPanlex, and CoooupMonoParrllelGaan Panlex. We leave experimentation with hybrid codeswitch-GLowUP approach (e.g. Codeswitch- MonoGiowupMono) for future work.
6.2.1 MASS with monolingual GLOWUP
To apply GLOWUP to monolingual text, we simply first sample the lexical prompts, and after those translations are prepended to the source, we ap- ply MASS in the standard way mask random subspans. This may mean that the GLOWUP prompt itself is masked.
6.2.2 GLOWUP-MT
The extension of GLOWUP to parallel data (GLOWUP- MT), is effectively the same as the monolingual variant of the task, but without the MASS element, For given sentence pair (x, 3) in the training cor- pus, the prompting is performed on the source sen- tence v, essentially to give it hints about how pro- duce y. The model is then trained on the translation task using (x', y), with the task token <2glowup> instead of <2translation>. Like GlowupMono, this can be applied in an inference-only way.
8 Results
We evaluate all our models on the FLORES-200 dataset (NLLBTeam et al., 2022; Goyal et al., 2021; Guzmán et al., 2019), which contains English- aligned parallel sentences for 200 languages. In Appendix section we also report scores on the in-house GATONES eval set.
7 Experiments
7.1 Training regimes
In our experiments, we train models with various combinations of the augmentations outlined above, well as baseline (with no data augmentation of any kind) and a model where we simply provide word pairs from the lexicon as additional paral- lel data. The details of our training regimes are discussed below.
We use the following resourcedness classifica- tions for our analysis:
1. High Resource Languages (HRLs): > 2B to-
tal training tokens in parallel data
2. Medium-Resouree
Languages (MRLs):
360M to 2B training tokens in parallel data
3. Low-Resource Languages (LRLs): 1 to 360M
7.1.1 Baseline
training tokens in parallel data
We first train a baseline model with no data augmen- tation, using the monolingual and parallel data de- scribed in Sections 3.1 and 3.2, respectively. This model is comparable to the model trained in Bapna al. (2022), but smaller, and without iterative back- translation; larger models with more data, are ex- plored in Section 1.
4. Unsupervised Languages (URLs): no parallel data
The results relative to the baseline, in ACHRF, are summarized in Tables (en-*xx) and 2 (xx -en). few trends jump out. Firstly, all models trained with only monolingual data aug- mentations see consistent performance gains over

---

Model
μ
HRL
MRI.
LRI.
URL
LRI
URL
Baseline
39.7
50.5
46.6
34.4
28.7
35.4
26.6
GatiPanlex GatiPanlexTokenPaiinn
TokenPairs
+0.4
-0.3
-0.1
+0.5
+1.4
+1.8
+5.1
ModoswitchMMno
+0.8
+0.8
+0.9
+0.8
+0.4
+1.9
+4.8
CodeswitchParallel
-1.9
-1.7
-0.9
+0.2
+2.1
CodeswiichMonnParallel
-0.2
-1.0
-0.7
+0.0
+1.1
+1.5
45.2
CodeswiccMMonGGüiPnnee
+1.0
+0.4
+0.6
+1.3
+1.5
+2.8
+7.0
GlowupMono
+1.1
+1.5
+1.4
+1.0
+0.5
+2.2
+5.6
GlowupParallel
-2.0
-1.8
+0.4
+0.8
GlowapMonoParallell
+0.3
+0.1
+0.2
+0.1
+1.0
+1.3
+3.4
GiowupponoGatiPanles
+1.2
+1.3
+1.2
+1.4
+0.8
+2.5
+5.6
Table I: performance the FLORES-200 200 measured in ACHRF over the baseline. Gains particularly strong the languages with GATITOS (last columns), reaching +7 CHRP for unsupervised language pairs.
Model
μ
HRL
MRL
LRL
URL
LRLear
URLear
Baseline
47.2
57.2
52.1
43.6
37.0
49.1
36.1
GatiPanlex TokenPairs
-0.0
-0.3
-0.3
+0.1
+0.5
+ +1.0
+2.4
Mone
+0.2
+0.5
+0.4
+0.1
-0.1
+0.8
+1.5
CodeswitchParallel
-1.2
-1.6
-1.5
-1.2
-0.3
+0.0
+1.2
CodoswitchMonnPaaallll
-0.8
-0.7
-0.9
-0.8
-0.7
-0.1
CadsswicchMonoGGaiPPnnexxx
+0.1
+0.3
+0.1
+0.3
+0.9
+2,9
GlowupMono
+0.7
+1,2
+1,1
+0.6
+0.0
+1.5
+2.0
GlowupParallel
-1.3
-1.7
-1.6
-1.2
-0.6
-0.1
+1.2
PlaraplonnPParallc
+0.1
+0.0
-0.0
-0.0
+0.5
+0.6
+1.7
GiovuppMonoanniPenlee
+0.6
+1.0
+0.8
+0.6
-0.1
+1.3
+1.8
Table 2: xx-ee performance on the FLORES-200 200 test set, measured in ACHRF over the baseline, showing weaker version of the same trends from en en-xxx.
the baseline. Conversely, models with only paral- lel data augmentations show performance degra- dations. Models mixing monolingual and parallel data augmentations fare in-between those poles.
ning augmentations for URLs remain CodeSwitch- MonoGatiPanlex and GlowapMonoGatiPalle; the former having average gain of +7.0CHRI on FLORES 200and +8.0CHRF on GATONES, and the latter having +5.6CHRF on FLORES-200 and + +9.5CHRF on GATONES.
These general trends are the same between en-*xx and xx-ten directions, though the gains are generally lower in the xx-ten direction. As noted Section 4, this is expected, and this direction is less of a priority for translation improvements. Results on GATONES, in Appendix B, show the same trends, though the performance gains tend to be larger for all augmentations, with CodeSwitch MonoGatiPanless gaining +2.3 CHRF for URLs.
Finally, although it was not the first place model in any category, the GatiPanlexTTkenPain has large gains in all directions over baseline, and only falls short of the more complex augmentations by small margin. Furthermore, when used in conjunc- tion with either Glowup or Codeswitch, it further improves performance across all categories. This may be one of the most useful long-term findings of this study: raw token pairs perform roughly on par with all the other fancier augmentations!
Despite these trends seeming robust across mod- els, the effect sizes are relatively small, maxing out at about +1.5 average ACHRF gain. How- ever, the picture changes dramatically when only looking at the subset of languages that has the higher-quality GATITOS training lexica. For these 26 languages, every augmentation, even the paral- lel ones, have large performance gains. The win-
8.1 Scaling up: bigger models, more data
Sometimes, results on smaller models not trans- fer to larger models. To this end, we train larger transformer models with 1.6B parameters using

---

10x the parallel data and 10x the high-resource monolingual data.
decoding errors we were seeing, but we leave an investigation of this hypothesis for future work.
We trained three large models: baseline, a token-pair model, and token-panne pairs Codeswitch- Mono model. The CHRF scores for these models can be seen in Table 3. There are a number of obvious differences between these results and the results on the smaller (175M parameter) models trained with = toth th the data. First, the positive impact of the data augmentations is smaller in all categories, indicating that the gains previously seen from augmentations are partially washed out in the larger-data, larger model regime. On FLOORES-200 en en->xx, both models very close baseline- within the realm of noise. On GATONES en-*xx, they see consistent small gains of around +0.5 CHRF, much smaller than previously. For both eval sets in the -x direction, there are consis- tent small losses.
8.3 Oracles: trusted parallel text
Parallel text is much more costly to produce than bilingual lexica, but also contains many more use- ful signals, including examples of word usage in context. But how much more helpful is it, really, than bilingual lexica? The answer seems to be "much more helpful".
To measure this, we trained model with mix- ture of thirteen public parallel datasources (Ap- pendix A) covering our lowest-resource languages. These are high-quality, trusted datasets, prepared by community members- - a very different resource than the web-mined parallel data that the model is otherwise trained on.
Table 4 reports on the 24 GATITOS languages, comparing four models: the standard baseline and GatiPanlexTokenn
model, well the "Paral- lel" model (which adds the external parallel trans- lation task with a 5% weight) and the "Parallel CatiPanlexTTokenPain model, which uses both the token-pain and the parallel data, with a combined weight of 5%. On both eval sets, we see that using Bilex yields gain of around +3.5 CHRF, but using true parallel yields much larger gain of about +10 CHRF. Using the bilingual lexica top of the par- allel data yields further gain of about +0.5 CHRFF demonstrating that, though many of the gains have been washed out by the true parallel data, there are still modest gains to be had from bilex training. Full results are in Appendix Table 9.
Is this the Bitter Lesson (Sutton, 2019) getting us again? Perhaps-but the picture is less bleak than first appears. When we look at the subset of the lan- guages where we have a higher assurance that the bilingual lexica are higher-quality quality-namely, those that use GATITOS bilingual lexica-we still see consistent For these 26 languages, all models see consistent gains, and the gains are biggest on unsupervised languages.
Overall, the takeaway from these experiments is that one has ensure that the data is of high quality when applying lexical data augmentation such large scale. While saw substantial improvements for many languages, these were bal- anced out by losses for other languages (especially those with only Panlex, but not GATITOS, data).
8.4 How many token pairs do I really need?
We examine the relationship between the num- ber of lexical token pairs provided during train- ing and CHRF. First, we perform regressions us- ing ACHRF over baseline the outcome variable and three predictor variables: (I) number of Pan- lex entries, (2) number of GATITOS entries, and (3) number of monolingual sentences. We include monolingual sentences in the regression to control for it as a confound. To eliminate parallel data quantity as confound, we only perform this anal- ysis on URLs.
8.2 Glowup Decoding
In principle, one of the advantages of the Glowup- Parallel approach is that lexica can be used at in- ference time. Therefore, we experimented with decoding the eval sets not with the translation task ID, but with the Glowup task ID, along with the relevant lookups from the lexica. Unfortu- nately, these decodes failed impressively, with per- formance degrading the more prompts that were included. Model decodes often had long sequences of control tokens. Further work should not disre- gard this direction; indeed, a variant of this likely has particular promise in the world of foundation models. The current approach likely just needs some tweaks to eliminate this sort of oft-of oft-of-domain domain
As expected, both the number of Panlex word pairs for a given language and the number of GATI- TOS word pairs have a positive 3 coefficient. How- ever, note that the B for GATITOS is R 3x larger than 3 for Panlex on FLORES- 200, and v 39x larger for GATONES. We conclude that GATITOS is

---

HRL
MRL
LRL
URL
LRLGAT
URLGAT
FL.ORES- 200 en +XX
BaselineRig
56.1
51.2
38.5
35.5
24.4
36.0
GuiPaallexTokeePuunnBBi
-0.5
-0.3
+0.1
-0.3
+1.2
+2.2
CotewiicchMMmoGGaPPeleeeiiig
+0.1
+0.1
-0.3
+0.9
+3.4
GATONES en >XX
BaselineBig
34.1
26.9
25.5
23.9
27.1
GatiPanlexTokeePPaissBB
+0.2
+0.7
+1.6
+3.5
CodewwichaMomGGaiinniiiiiig
+0.4
+0.7
+0.2
+1.4
+4.2
Table 3: Average CHRF scores by resource category for the larger models, reported in delta relative baseline. These models trained with 10x the data and 3x the parameters. The gains whole are washed out somewhat, hut for those low-resouree and unsupervised language pairs with the more trusted GATITOS data (last two columns), there is still noticeable gain.
Baseline
MadiPanlexTokenPaae
Parallel
Parallel + GatiPanlex
FLORES-200
21.1
24.4
33.6
34.2
GATONES
20.2
23.9
29.5
30.1
Table 4:
Comparing en-*x improvements from token-pair data to oracle: training on trusted parallel data. Parallel data is much more effective than GATITOS alone, but the combination of the two is the most effective overall.
more efficient for improving MT than Panlex, prob- ably due to higher quality. Full regression results are given in Appendix Table 6.
prompting, and raw token-pair training. The results show that applying any of these augmentations to monolingual data yields substantial improvements, and that they be combined for greater effect. The leader (by small margin) the combination of CodeswitchMono and raw token-pair training. These results hold when scaling up model and data size, but in the settings with more data and larger models, the quality of the bilingual lexica plays a relatively bigger role, and augmentation with the noisier Panlex begins to lag in quality behind the much smaller, yet higher-quality, GATITOS dataset.
The most practical qucstion we can seek to an- swer is, If can spend $X on translating tokens, how much quality increase can expect? To in- vestigate this "bang for buck" question in a more controlled way, we observe the effects of the GATI- TOS dataset in isolation, without Panlex. We train "GatiiokenPParr" model, which is identical to the model, except the token- pair task has only GATITOS data. Thus, this tells us specifically what gains we can expect if we are to get 4,500 tokens' worth per language.
TratiPanlexTooeePiiie
Future work will likely want to focus on prompt- ing foundation models with bilingual lexica. Large Language Models show promise on machine trans- lation for high-resouree languages (Jiao et al.. 2023), but their capabilities on low-resource lan- guages have yet to be thoroughly explored. Ad- ditionally, more thorough investigation of the trade-off between cost and quality for tiny datasets can be explored: with a fixed budget time or money, should one spend their time translating text, making monolingual text, or making bilingual lex- ica?
The result of this experiment is gain of +4.9 CHRF on FLORES-200 and +5.2 CHRF on GATONES, respectively, for en - URLs; the im- provement for languages with some parallel data is reduced but significant, at +1.6/2.3 CHRF resp. Full results reported in Appendix Table 10.
9 Conclusions
In this paper we explore the ways that that aug- menting training data with bilingual lexicon infor- mation can improve the performance of machine translation models on low-resource and unsuper- vised languages, and open-source the GATITOS dataset, which leads average gains of about +7 CHRF on unsupervised languages. We perform extensive experimentation with three main types lexical augmentation: codeswitching, lexical
Limitations
There are several limitations with the present work. For one, we rely on the automated metric CHRFF which is less reliable than human ratings. Similarly, though we perform some qualitative evaluation error types (Appendix G), detailed human eval-

---

uation might reveal the precise ways in which our models are failing.
Twelfth Language Resources and Evaluation Confer- ence, pages 2218-4222 Marseille, France. European Language Resources Association.
A second limitation is that, while we open- source GATITOS, the base training data from our models is not opensourced (to protect copyright), and thus our experiments are not replicable.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2019. An effective approach to unsupervised machine trans- lation. In Proceedings of the 57th Annual Meeting of the A Association for Computational Linguistics, pages 194-203, Florence, Italy. Association for Computa- tional Linguistics.
Finally, we leave open two important questions: 1) how much data do I need translated before can reach X quality? and 2) if can only translate X tokens' worth of data, what is the best way to select those X tokens? The present work only partially answers the first question, and does not address the second at all.
Mikel Artetxe, Gorka Labaka, Eneko Agire, and Kyunghyun Cho. 2017. Unsupervised neural ma- chine translation. CoRR, abs/1710. 1041.
Dzmitry Bahdanau. Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Represenations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
11 Ethics Statement
Improving the state of technology for under-served communities is usually considered a positive contri- bution. There are various nuances this, however, including questions of consent of and involvement of the affected community, "helicopter NLP," and data sovereignty. By open-sourcing GATITOS to be used by all affected communitics, we hope to respect their data sovereignty and not keep this resource from them. Throughout this project we have also sought advice from community members, which we hope will alleviate the other concerns.
Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hicu Hoang, Miquel Esplà-Gomis, Mikel L. Foreada, Amir Kamran, Faheem Kirefu, Philipp Koehn, Sergio Ortiz. Rojas, Leopoldo Pla Sempere, Gema Ramirez-Sánchez, Elsa Sarrias, Marek Strelec, Brian Thompson, William Waites, Dion Wiggins, and Jaume Zaragoza. 2020. ParaCrawl: Web-scale acqui- sition of parallel corpora. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4555 4567, Online. Association for Computational Linguistics.
Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Fi- rat. Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey. Theresa Breiner, Vera Axelrod, Jason Riesa, Yuan Cao, Mia Xu Chen, Klaus Macherey, Maxim Krikun, Pidong Wang. Alexander Gutkin, Apurva Shah, Yan- ping Huang, Zhifeng Chen, Yonghui Wu, and Mac- duff Hughes. 2022. Building Machine Translation Systems for the Next Thousand Languages. arXiv e-prints, page N.Xi::2205.03988
References
Antonov, Alexander. 2022. Chuvash-Russian par- allel corpus. https://github cotollccoo/aaaa com/ALAntonov/ chv_corpus.
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- fional Linguislics: Human Language Technologies, Volume (Long and Short Papers). pages 3874-3884, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Christos Baziotis, Barry Haddow, and Alexandra Birch. 2020. Language model prior for low lowrresource neural machine translation. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 7622-7634, Online. As- sociation for Computational Linguistics
Benjamin Akera, Jonathan Mukiibi, Lydia Sanyu Nag- gayi, Claire Babirye, Isaac Owomugisha, Solomon Nsumba, Joyce Bakatumba-Nabende Engineer Bain- omugisha, Ernest Mwebaze, and John Quinn. 2022. Machine translation for african languages: Commu- nity creation of datasets and models in uganda. In 3rd Workshop on African Natural Language Processing.
Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. 2020. Language id in the wild: Unex- pected challenges on the path to a thousand- language web text corpus.
Isaac Caswell, Ciprian Chelba, and David Grangier. 2019. Tagged back-translation. In Proceedings of the Fourth Conference on Machine Translation (Volume
Andersen, Jógvan. 2021. Sprotin Translations. https:
/ ///gtthub.com//protttttto.o.e wtt.://ow///tettteeeeee wwt.rottinntraslaatioss.
1: Research Papers), pages 33-63, Florence, Italy. Association for Computational Linguisties.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gre- gor Weber. 2020. Common voice: A massively- multilingual speech corpus. In Proceedings of the
Aditi Chaudhary, Karthik Raman, Krishna Srinivasan, and Jiecao Chen. 2020. DICT-MLM: Improved Mul- tilingual Pre-Training using Bilingual Dictionaries. arXiv e-prints, page arXiv:2010.1 12566.

---

Yong Cheng, Wei Wang, Lu Jiang, and Wolfgang Macherey. 2021. Self-supervised and supervised joint training for resource-rich machine translation. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1825-1835. PMI.R
Markus Freitag, David Vilar, David Grangier. Colin Cherry, and George Foster. 2022b. A natural diet: Towards improving naturalness of machine transla- tion output. In Findings of the Association for Com- putational Linguistics: ACL 2022, pages 3340-3353, Dublin, Ireland. Association for Computational Lin- guistics.
Luis Chiruzzo. Santiago Góngora, Aldo Alvarez, Gus- Torales, and Yliana Rodríguez. 2022. Jojajovai: A parallel Suarrni-Spaanise corpus for MT benchmarking, In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2098-2107, Mar- scille, France. European Language Resources Asso- ciation.
Marjan Ghazvininejad, Hila Gonen, and Luke Zettle- moyer. 2023. Dictionary- based phrase-level prompt- ing of large language models for machine translation.
tavo Giménez-Lugo, Marvin Agüero-Torales.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Kr- ishnan, Mare' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2021. The flores- 101 evaluation benchmark for low-resource and multilingual ma- chine translation.
Alexis Conneau, Guillaume Lample, Marc' Aurelio Ran- zato, Ludovic Denoyer, and Hervé Jégou, 2017. Word translation without parallel data, arXiv preprint arXiv:7710.04087.
Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, and Yoshua Bengio. 2017. On integrating a lan- guage model into neural machine translation. Com- puter Speech Language, 45:137-148.
Chris Chinenye Emezue and Femi Pancrace Bonaven- ture Dossou. 2020. FFR v1.1: Fon-French neural ma- chine translation. In Proceedings of the The Fourth Wideming Natural Language Processing Workshop, pages 83-87, Seaule, USA. Association for Compu- tational Linguistics.
Francisco Guzmán, Peng-Jen Chen, Myle Oit, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Mare Aurelio Ranzato. 2019. Two new evaluation datasets for low-resource machine translation: Nepali-english and sinhala-english.
Miquel Esplà-Gomis. 2009. Bitextor: a free/open- source software to harvest translation memories from multilingual websites. In Beyond Translation Mem- ories: New Tools for Translators Workshop, Ottawa, Canada.
Asmelash Hadgu, Gebrekirstos Gebremeskel, and Abel Aregawi. 2022. НommMT. https://gt thub. com/ asmelashteka//HormM.. Accessed: 2023-03-24.
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? preliminary study.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma. Ahmed El-Kishky, Siddharth Goyal. Man- deep Baines, Onur Celebi. Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vi- taliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2022. Beyond english-centric multilingual machine translation. J. Mach. Learn, Res., 22(1).
Melvin Johnson, Mike Schuster, Quoc V. Lee Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system: En- abling zero-shot translation. Transactions of the As- sociation for Computational Linguistics, 5:339-351
Isaac Feldman and Rolando Coto-Solano. 2020. Neu- ral machine translation models with transtranslttion for the extremely low-resouree indigenous language Bribri. Proceedings of the 28th International Con- ference on Computational Linguistics, pages 3965-
David Kamholz, Jonathan Pool, and Susan Colowick. 2014. PanLex: Building resource for panlingual lexical translation. In Proceedings of the Ninth In- ternational Conference on Language Resources and Evaluation (LREC'14) 14), pages 3145-3151 Reykjavik, Iceland. European Language Resources Association (ELRA).
3976, Barcelona, Spain (Online). International Com- mittee on Computational Linguistics.
Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 866-875, San Diego, California. Association for Computational Linguistics.
Jyotsana Khatri, Rudra Murthy, Tamali Banerjee, and Pushpak Bhattacharyya. 2021. Simple measures of bridging lexical divergence help unsupervised neural machine translation for low-resource languages.
Tom Koemi, Christian Federmann, Roman Grund- kiewicz, Marcin Junczys-Dowmunt, Dowmunt, Hitokazu Mat- sushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, pages 478 494, Online. Association for Computational Linguistics.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi kiu Lo, Craig Stewart, Eleftherios Avramidis, George Foster, Alon Lavie, and Andr' e F. T. Martins. 2022a. Results of wmt22 metries shared task: Stop using bleu - neural metrics are better and more robust.

---

Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allah- sera Tapo, Nishant Subramani, Artem Sokolov, Clay- tone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, An- nette Rios, Isabel Papadimitrion, Salomey Osei, Pe- dro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, An- dre Niyongabo Rubungo. Toan Q. Nguyen, Math- ias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyak- eni, Jamshidbek Mirzakhalov, Tapiwanashe Matan- gira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaven- ture F. P. Dossou, Sakhile Diamini, Nisansa de Silva, Sakine Çabuk Balli, Stella Biderman, Alessia Bat- tisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, aa Duygu Ata- man, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. 2022. Quality at a glance: An audit of web-crawled multilingual datasets. Trunsactions of the Association for Compu- tational Linguistics, 10:50-72.
Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- rics, 8:726-742.
Zihan Liu, Genta Indra Winata, and Pascale Fung. 2021. Continual mixed-language pre-training for extremely low low-resource neural machine translation. In Find- ings of the Association for Computational Linguis- rics: ACL-IICNLP 2021, pages 2706-2718, Online. Association for Computational Linguisties.
Manuel Mager, Arturo Oncevay, Ableen Ebrahimi, John Ortega, Annette Rios, Angela Fan, Ximena Gutierrez- Vasques, Luis Chiruzzo, Gustavo Giménez-Lugo, Ri- cardo Ramos, Ivan Vladimir Meza Ruiz, Rolando Coto-Solano, Alexis Palmer, FElisabeth Mager-Hois, Vishrav Chaudhary, Graham Neubig, Ngoc' Thang Vu, and Katharina Kann. 2021. Findings of the Ameri- casNLP 2021 shared task on open machine transla- tion for indigenous languages of the Americas. In Proceedings of the First Workshop on Natural Lan- guage Processing for Indigenous Languages of the Americas, pages 202-217, Online. Association for Computational Linguistics.
Nalin Kumar, Deepak Kumar, and Subhankar Mishra. 2022. Dict-nmt: Bilingual dictionary based nmt for extremely low resource languages.
Ayush Maheshwari, Piyush Sharma, Preethi Jyothi, and
Ganesh Ramakrishnan. 2022. Dictdis: Dictionary constrained disambiguation for improved nmt.
Garry Kuwanto, Afra Feyza Akyüreek, Isidora Chara Tourni, Siyang Li, and Derry Wijaya. 2021. Low- resource machine translation for low-resource lan- guages: Leveraging comparable data, code-switching and compute resources. CoRR, abs/2103..1221
Christopher Malon. 2021. embeddings with word definitions. CORR, abs/2103.03842.
Overcoming poor word
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. 2017. Unsupervised ma- chine translation using monolingual corpora only. arXiv preprint arXiv: 1at::17110000000
Elise Michon, Josep Crego, and Jean Senellart. 2020. Integrating domain terminology into neural machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3925-3937, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Chazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequee to-seueecce pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880 Online. Association for Computa- tional Linguisties.
Jonathan Mukiiii, Babirye Claire, and Nakatumba- Nabende Joyce. 2021. The Makerere MT Corpus: English to Luganda parallel corpus.
Sreyashi Nag, Mihir Kale, Varun Lakshminarasimhan, and Swapnil Singhavi. 2020. Incorporating bilingual dictionaries for low resource semi-supervised neural machine translation. CoRR, abs/2004.02071
Yusen Lin, Jiayong Lin, Shuaicheng Zhang, and Haoy- ing Dai. 2021. Bilingual dietionary-based language model pretraining for neural machine translation. CoRR, abs/2103.07044
Jan Nichues. 2021. Continuous learning in neural ma- chine translation using bilingual dictionaries. CoRR, abs/2 102.558.
Zehui Lin, Xiao Pan, Mingxuan Wang. Xipeng Qiu, Jiangtao Feng, Hao Zhou, and Lei Li. 2020. Pre- training multilingual neural machine translation by leveraging alignment information. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2649- 2663, Online. Association for Computational Lin- guistics.
NLLBTeam, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Hef- fernan, Elahe Kalbassi, Janice Lam, Daniel Licht. Jean Maillard, Anna Sun, Skyler Wang. Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez. Prangthip Hansanti, John Hoffman. Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and

---

Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human- centered machine translation.
Aditya Siddhant, Ankur Bapna, Orhan Firat, Yuan Cao, Mia Xu Chen, Isaac Caswell, and Xavier Garcia. 2022. Towards the next 1000 languages in mul- tilingual machine translation: Exploring the syn- ergy between supervised and self-supervised learning. CoRR, abs/2201 .031 10.
Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. 2021. Contrastive learning for many-to-many multilingual neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Compu- lational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages ciation for Computational Linguistics.
Kai Song, Yue Zhang, Heng Yu, Weihua Luo, Kun Wang, and Min Zhang. 2019a. Code-switching for enhancing NMT with pre- specified translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguisties: Humun Language Technologies, Volume (Long and Short Papers), pages 449-459, Minneapolis, Minnesota. Association for Computa- tional Linguisties.
Online. Asso-
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191. Brussels, Belgium. Association for Computa- tional Linguistics.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2019b. MASS: Masked sequence to se- quence pre-training for language generation. In Pro- ceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Ma- PMLR.
Libo Qin, Minheng Ni, Yue Zhang, and Wanxiang Che. 2020. Cosda-ml: ml: Multi-lingual sode-switching data augmentation for zero-shot cross-lingual nlp. Pro- ceedings of the Twenty-Ninth International Joint Con- ference on Artificial Intelligence, IJCAI-20, pages 3853-3860. International Joint Conferences on Arti- ficial Intelligence Organization. Main track.
chine Learning Research, pages
Raymond Hendy Susanto, Shamil Chollampatt, and Lil- ing Tan. 2020. Lexically constrained neural machine translation with levenshtein transformer. CoRR, abs/2004.12681.
Machel Reid and Mikel Artetxe. 2022. PARADISE:
Ilya Sutskever, Oriol Vinyals, and Quoe Le. 2014. Se- quence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.
Exploiting parallel data for multilingual sequence- to-sequence pretraining. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 800-810, Seattle, United States. Association for Computational Lin- guistics.
Richard Sutton. 2019. The bitter lesson.
Jörg Tiedemann. 2020. The tatoeba translation chal- lenge- realistic data sets for low resource and multi- lingual MT. CoRR, abs/2010.06354.
Philip Resnik and Noah A. Smith. 2003. The web as a parallel corpus. Computational Linguistics, 29(3):349-30..
Tlisha, Nart. 2022. pus. https: //gi thub. / Latalinngul-PPrallll---o- 2023-03-24.
Multilingual Parallel Cor-
Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020. BLEURT: learning robust metrics for text generation. CoRR, abs/2004.04666.
Accessed:
Jakob Uszkoreit, Jay Ponte, Ashok Popat, and Moshe Dubiner. 2010. Large scale parallel document mining for machine translation. In Proceedings of the 23rd International Conference on Computational Linguis- tics (Coling 2010), pages 1101-1109, Beijing, China. Coling 2010 Organizing Committee.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany. Association for Computational Lin- guistics.
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Iskander Shakirov and Aigiz Kunafin. 2023. Bashkir- russian parallel corpora.
Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, Mia Chen, Sneha Kudugunta, Naveen Arivazhagan, and Yonghui Wu. 2020. Leveraging monolingual data with suef-suuervi for multilingual neural ma- chine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 2827-2835, Online. Association for Computational Linguistics.
Xinyi Wang, Sebastian Ruder, and Graham Neubig. 2022. Expanding pretrained models to thousands more languages via exicon-based adaptation. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), pages 863-877, Dublin, Ireland. Association for Computational Linguistics.

---

Barack Wanjawa, Lilian Wanzare, Florence Indede. Owen McOnyango, Edward Ombui, and Lawrence Muchemi. 2022. Kencorpus: A kenyan language cor- pus swahili, dholuo and luhya for natural language processing tasks.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao. Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine trans- lation. CORR, abs/1609.00 Ne//16699084444 08144.
Mengzhou Xia, Xiang Kong, Antonios Anastasopou- los, and Graham Neubig. 2019. Generalized data augmentation for low-resource translation. CoRR, abs/1906.03785.
Jian Yang. Yuwei Yin, Shuming Ma, Haoyang Huang, Dongdong Zhang, Zhoujun Li, and Furu Wei. 2021. Multilingual agreement for multilingual neural ma- chine translation.
Proceedings of the 59th nual Meeting of the Association for Computational Linguistics and the I 11th International Joint Confer- ence on Natural Language Processing (Volume Short Papers), pages 233-239, Online. Association for Computational Linguistics.
Zhen Yang. Bojie Hu, Ambyera Han, Shen Huang, and Qi Ju. 2020. CSP:code-switching pre-training for neural machine translation. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 2624 2636. Online. Association for Computational Linguistics.
Wenhao Yu, Chenguang Zhu, Yuwei Fang, Donghan Yu, Shuohang Wang. Yichong Xu, Michael Zeng, and Meng Jiang. 2021. Dict-bert: Enhaneing lan- guage model pre- pre-training with dictionary. CoRR, abs/2110.06490.
Xing Jie Zhong and David Chiang. 2020. Look it up: Bilingual and monolingual dictionaries improve neu- ral machine translation. CoRR, abs/2010.05997.

---

GlowupMono w/ masking (multilingual lexically MASS)
Source
<2glowup> <2en> <2Latn> <src> we <tgt> nous <src> see <tgt> ka
ye <2en> <MASK> Saturday, drove <MASK> <MASK> <MASK> countryside.
Target Last Saturday, drove out to the countryside.
GlowupParallel
texically-prompped translation)
Source <2glowup> <2fr> <2Latn> <src> came <tgt> venu <src> dramatic <tgt> dramatique <src> fashion <tgt> façon <2fr> The performance came to a close in dramatic fashion.
Target
performance s'est conclue façon dramatique.
Figure I: Examples of the monolingual (top) and parallel (bottom) GLOMUP augmentation strategics. In both cases, random
tokens in the source sentence are prepended to the source sentence, along with their translations in random language from a multilingual lexicon. As in Figure 2, differential color coding is done to draw attention to dictionary translations in different languages.
CodeswitchMono (multilingual codeswitching autoencoding)
Source
<ccoeeswitch> <2en> <2Latn> Dernier Saturday, noi yatwaye out to कहहीर the panpa.
Target
Last Saturday, we drove out see the countryside.
CodeswitchPPralles (codessiihngg translation)
Source
tch <2fr> <2Latn> The (sJosmoo came til Близкий in dr armaatisk fashion.
Target
La performance s'est conclue de façon dramatique.
Figure 2: Examples the monolingual (top) parallel (bottom) codeswitching augmentation strategies. both cases, random tokens the source sentence are replaced their translations in random language from a multilingual lexicon. Color coding is used to indicate which source words have been swapped their dictionary translation. The different are used simply to point out the fact that the words are from codeswitched words in each sentence come different languages.
Public parallel data used
2022). We were not able to use AmericasNL.P (Mager et al., 2021) because of the license.
The datasets we used were HornMT (Hadgu et al.. 2022), SALT (Sunbird AT Language Translation) dataset (Akera et al., 2022), FFR: Fon-French Neural Machine Translation (Emezue and Dos- sou, 2020), Tatoeba (Tiedemann, 2020), The Mak- erere MT Corpus: English to Luganda parallel cor- pus (Mukiibi et al., 2021), Commonvoice (Ardila et al., 2020), Kencorpus: Kenyan Languages Cor- pus (Wanjawa et al., 2022), Chuvash-Russin parallel corpus( Antonov, Alexander, 2022),
Abkhaz Corpus (Tlisha, Nart, 2022), Bashkir Corpus(Shakirov and Kunafin, 2023), Sprotin
Faroese Corpus(Andere)
Jógvan, 2021), Jojajovai
Guarani- Guarani-Spanish Parallel Corpus (Chiruzzo et al., 2022), and the NLLB Seed data (NLLB et al.,

---

Source
Khawwelah hian tawng chi hrang 5000 chuang a awma, heng zinga tawng ai tam chuan tawng hmangtu
maktaduai aia tam a nci a
Reference
The world has over 5,000 different languages, including more twenty with million or more speakers. There are 5000 houses the city, 5000 houses in the city, and 5000 houses the city.
Baseline
Augment
There are 5000 languages in the world, 20 of which are million languages.
Source
Maani isumagarpunga inuunerma sinnerani pil ailnaarussanngorungaa.
Reference
Here think I'll remain happy throughout my life.
Baseline
believe am the only one who has ever died.
Augment mean, I'm blessed with lifetime of joy.
Source haraxb Bbt Xbanrr хsлар xlaaaxap Kвамелаn Ma расшириеемости, дasp e нами xу.a uservoice Reference If you are interested in continuing the conversation around extensibility, connect with us via uservoice
Baseline If you want use the extension, see uservoice
Augment If you want learn more about extensibility, contact uservoice
Table 5:
Examples of translations where our lexical data augmentation methods appear to have helped the model choose the correct vocabulary, in Mizo (lus), Kalaallisut (kI), and Chechen (ce). Words appearing in the lexical data are colored green.
No. PanLex
No. GAT.
No. mono
Intercept
Adjusted R
AFLORES-200 AGATONES
6.55e-5
1.96e-4
-6.71e-7
1.93
0.40
6.59e-6
2.58c-1
-7.78c-8
2.91
0.23
Table 6: Linear regression results in the en-txx direction. Here, the dependent variable ACHRF over the baseline, and the independent variables are: number of PanLex word pairs for given language (No. PanLex), number of GATITOS word pairs (No. GATITOS), and number of monolingual sentences (No, mono).
B Results on GATONES
nificantly smaller than the improvement from using IiiiPanleTTokenPaae alone, and the GlowupMono augmentation by itself actually results in losses on URLs. So taking the GATONES and FLORES-200 results together, it seems that adding raw token pairs as additional parallel data is the best way, out of the techniques we tried, to improve performance in the xx->en direction for very low-resource lan- guages.
The main paper reports the scores on the more widely-used FLORES 200 eval set; this section re- ports on the other dataset that we evaluate our mod- els on. This is an in-house eval set, which we call GATONES (Google AuTOmatic NTL Eval Set). The dataset contains 63 languages, most of which are unsupervised or low-resource. Here some notes, by translaiton direction:
en-*x The evaluation results on GATONES for the en -xx direction are summarized Table The trends are mostly the same as what we saw for the FLLORES en en->xx evaluation. Codeswitch- MonoGatiPanle emerges, even more definitively than on FLORES-200, as the best model, winning all categories except MRLs.
C Bilexica vs true parallel data
In the main paper in Section 8.3, we evaluate on the 24 GATITOS languages, comparing four models with different combinations of trusted parallel data and GATITOS bilingual lexica. The full results are given here in Table 9, on both FLORES-200 and GATONES, sorted from highest-resource to lowest- resource.
xx- The evaluation results for the xx->en di- rection are given in Table 8. The single most im- portant takeaway from this part of the analysis is the same as it was for the FLORES- 200 evaluation: Pairs Parn model helps URLs the most in this direction, with a ACHRF of +1.1 over the baseline. Yet again, the improvements are smaller in this direction than for en >xx The only other thing that stands out about this part of the evaluation that the GlowupMono augmen- tation doesn't It seem be helpful according to this test set as for the FI.ORFS set. Although GlowapMonoParaalee and GlowapMonoGatiPan lex do reasonably well, their improvements are sig-
the plain GatiPanlex

---

Model
MRI.
LRI.
URL
LRL
URLGAT
Baseline
21.4
28.7
22.9
19.1
17.1
20.0
GaiiPanlexTokenPairs
+1.7
+0.6
+1.6
+1.8
+3.6
+8.4
Codo
+1.4
+1.6
+0.8
+1.9
+3.2
+8.3
CodeswitchParallel
-1.8
-0.9
-0.0
+0.9
+3.4
CodeswiichhMoooPrrlllll
+1.4
-0.1
+0.7
+2.2
+2.5
+6,1
CodeswitchMoomGGaalaanee
+2.2
+1.1
+2.1
+2.3
+3.9
+8.0
GlowupMono
+0.3
+1.0
+0.6
-0.2
+2.0
+7.6
GlowupPurallel
-1.6
-0.7
+0.6
+1.0
+3.0
GlowupMonoParallel
+0.2
-0.3
-0.7
+1.1
+2.1
+7.6
GlowupMonnGatiPanlex
+1.3
+1.4
+1.9
+0.7
+3.5
+9.5
Table 7: en->xx performance the GATONES measured ACHRF the baseline.
Model
MRL
LRL
URL
LRLaar
URLGAT
Baseline
29.4
43.2
30.2
27.0
24.7
22.1
GatiPanlexTokenPaiis
+0.8
+0.0
+0.5
+1.1
+1.1
+3.2
CodeswitchMono
+0.1
+0.1
+0.0
+0.2
+0.8
+2.2
CodeswitchParallel
-0.4
-1.2
-0.9
+0.2
+0.3
+1.8
CodeswicchMonmaaalltt
-0.5
-0.8
-0.7
+0.2
+1.6
Codeswic
+0.5
+0.0
+0.5
+0.6
+1.1
+3.3
GlowupMono
-0.0
+0.9
+0.2
-0.4
+0.5
+1.7
GlowupParallel
-0.6
-1.5
-0.9
-0.3
+0.0
+1.3
GlowupMonoParallel
+0.0
-0.2
-0.2
+0.2
+0.2
+1.6
GlowapMonoGartPanlee
+0.2
+0.8
+0.3
+0.1
+0.8
+3.3
Table 8: xx-ten performance on the GATONES test set, measured in ACHRF over the baseline.

---

mean
BSD
1g
cc
bho
hm
gn
ti
om
Purallel toks
375169
2296670
606853
315905
275417
154023
148549
130953
112958
43511
42018
en- Bilex toks
16324
8015
4500
8298
6755
24665
24665
70854
4500
6484
4500
FLORES-200 en-+AX
Baseline
21.1
30.1
26.9
19.1
15.5
TouPannexTokoaaaaa
24.4
37.3
31.9
32.7
30.5
25.5
22.6
19.6
19.4
8.1
16.1
Parailel
33.6
45.6
39.4
28
40.6
30.7
25
39.2
15.7
24.1
Parallel GatiPantex
34.2
45.3
45.3
38.2
31.1
40.8
30.1
24.9
39.9
18.1
28.8
GATONTS
Raseline
20.2
32.1
28.3
27.5
23.5
15.8
13.5
23.9
15.6
6.9
14.6
GatiPanlex GatiPankeTTkeenaiiii TokenPairs
23,9
37.3
28.7
31.3
28.1
26.2
22.5
23.5
16.1
15.6
Parallel
29.5
42.9
39.9
35.5
26.9
38.5
30.3
23.6
28.2
11.1
18
Parallel CodiPanleeTookenauus
30.1
42.6
37,9
35
29.6
38.7
30.4
22.6
30.1
13.1
20.8
Table 9: Comparing improvements from token-pair data to the oracle: training on trusted parallel data.
D Training only on GATITOS
The main paper (Section 8.4) describes the results training models only on GATITOS and not on PanLex. Full results are reported here, Table 10, reported in delta versus the baseline model for both FLORES-20 200 and GATONES in the en-+xx direction. The improvement for unsupervised lan- guages is around +5.0 CHRF for both eval sets; the improvement for languages with some parallel data is less but still noticeable, hovering around +2 CHRF. The largest improvement in Goan Konkani (+11.0 1.0 CHRF), with Mizo, Ilocano, and Bambara close on its heels with gains of around +8 CHRF. Only Maithili, which has interesting prop- erties a close dialect of Hindustani, sees loss on both eval sets. The gains are not obviously related to the total number of tokens per language.
As aside, it is heartening that F1.ORES-200 and GATONES seem to agree very nicely, despite their different domains (Wikipedia versus web + ansser)..answers..

---

TOTAL TOKENS
1.8M
2.1M
2.6M
3.6M
3.7M
5.4M
6.2M
14,7M
14.8M
16M
16.9M
26.1M
Pirl
Huri
ff
mni Mtei
kri
doi
bm
ny
gom
bho
qu
À FLORES--200 200
+1.6
+4,9
+0.1
+5.0
+0.8
+4.5
+2.9
+4.2
A GATONES
+2.3
+5.2
+0.0
+0.7
+5.1
+1.5
+7.6
0.1
+11.0
+5.1
+4.8
+3.9
+1.7
+5.3
TOTAL TOKENS
26.4M
27M
28M
40M
41M
52M
52M
80M
115M
124M
157M
167M
204M
505M
mai
In
1g
gn
nso
dv
ckb
A FLORES-200
-1.2
+1.3
+1.6
-0.4
-0.5
+7.0
+0.9
-0.2
+0.2
+8.6
+3.0
+3.3
A GATONES
+2.0
-1.2
1.2
+3.0
+0.5
+0.2
+1.7
+0.1
+0.4
+9.3
+3.4
+2.7
Table 10: Improvements on languages with GATITOS data when training ONLY on GATITOS data, in the en-xx direction. Sorted by total training tokens (mono, parallel, and bilex), in millions.
E Effects distributionally similar noun mistranslation
tend make fewer mistakes on them, though such UNMT-style mistakes do occur.
As expected, the aaugneee model performs better on these tokens. Two things are worth noting, First, the model improves noticeably on the complementary distribution that do not appear in the lexicon training data-bu un- surprisingly improves more on the words that are present in GATITOS. Second, the improvements are not as large as expected: why 100% accuracy? Digging into the outputs, it seems that this is mainly due to the high baseline of (a) undertranslation: (b) hallucination; and (c) copying, as we expect from a model trained without various other tricks like back-translation (see Section G for an analysis of common error types). This point is underscored by the models' imperfect performance on the "easy" class of numbers.
Part of the motivation for using bilingual lexicons for unsupervised translation was to see whether we could repair the common error mode of mis- translating distributionally similar nouns. Bapna et al. (2022) note that this error mode is particularly common for two categories of nouns: animals and colors.
not now getting
To measure improvement on this phenomenon, we define the token hit-rale as the following: for some set of desired tokens D, let RD be the sub- set of the eval set such that each reference con- tains least one token in D. The hit-rate is then the percentage of times in Rp that the model cor- rectly generated one of the desired tokens in D. For instance, if the desired tokens are "kitten" and "puma". RD is the set of references containing one of these words, e.g. "The kitten lies" and "A Puma eats hot chip". If the model produces "kitten lie floor" and "Crocodile charge they phone" from the corresponding sources, it has hit-rate of 50%. since it correctly got "kitten"" but not "puma".
F Biggest winners
We also look at the top 5 languages that were the biggest gainers over the baseline for each model. In some cases these may represent remarkable suc- cesses of particular approach-thouge in other cases they may represent noisy outliers, as is to be expected when evaluating 200 languages.
Table 11 looks at the token hit-rate for the mod- els BaselineBig and OodeswwithhMonoGatiPPe lexBig, for the categories of animals occurring in GATITOS (bear, bee, bird, butterfly, cat, chicken, deer, dog, elephant, fish, frog, goat, horse, in- sect, lion, monkey, parrot, pig, rabbit, sheep, snail, snake, tiger, turkey, turile), animals NOT appear- ing in GATITOS (ant, antelope, buffalo, cheetah, crocodile, dolphin, dormouse, gorilla, jellyfish, koala, leopard, moose, mosquito, newt, ocelot, ot- ter, reindeer, robin, scorpion, shark, sloth, spider, springbok, tortoise, velociraptor), colors (black, white, red, blue, yellow, green, purple, orange, grey), and numbers (one, two, three, four, five, six, seven, eight, nine, ten, hundred, million). All num- bers and colors appear in GATITOS. Numbers are included as a weak control, since the model will
F.1 FLORES-200
en-*xx The biggest winners for each model in the en-*xx direction for the FLORES 200 evalua- tion are given in Table 12.
There are seven languages that gained at least 5 CHRF over the baseline on at least one model trained with data augmentation. These languages are:
Bhojpuri (bho): up +11.5 CHRF
2. Ilocano (ilo): up to +9.1 CHRI
Serbian (sr): up to +8.3 CHRF
4. Bambara (bm): up to +8.1 CHRF
5. Tibetan (bo): up to +8.0 CHRF

---

FLORES-200
GATONES
cat.
A E GAT.
A & GAT.
colors
#5
A E GAT.
A A GAT.
colors
#s
BasclineBig
36.6
36.6
55.3
63.3
33.4
25.8
35.4
53.5
CostewwichMomoGGaPPaiiccckk
45.5
40.7
66.5
66.8
44,9
32.5
47.8
58.0
+8.9
+4.0
+11.1
+3.6
+11.5
+6.7
+12.4
4.5
Table 11: Comparing token hit-rate on classes of nouns known have issues for UNMT models, along with weak control of numbers. There are large improvements for animals in the GATITOS training lexicon (AA & CAT), as well as their complementary distribution, animals not in GATITOS (A & GAT)), and colors. Number hit-rate has minor bump.
107
10
10a
101
10a
Figure 3: Number of word pairs available in Panlex for each 4750 BCP-47 languages (log scale).
6. Nuer (nus): up to +6688 CHRF
again that the magnitude of improvement in this direction is smaller, likely because the baseline per-
7. Mizo (lus): up to +6.2 CHRF Unsurprisingly, most of these languages are un- supervised low-resource, except for Serbian which is medium-resource in our dataset. Of the seven languages listed above, we use Panlex data for Ilocano, Serbian, Bambara, Tibetan, Nuer, and Mizo, and there is GATITOS data for Bhojpuri, Σlo cano, Bambara, and Mizo. As will be discussed in Section 8.4, the GATITOS bilingual lexica are clearly very useful resource for MT, although evi- dently Panlex alone can help as well. Another inter- esting finding is that Nuer, which has English- aligned entries in Panlex but 20K -ngllishhh- aligned entries, still sees large improvements when translating from English. This is evidence that lex- icon data can improve performance even in the zero-shot case, where e.g. the model learns better vocabulary alignment between English and Nuer despite not receiving explicit alignment informa- tion during training. In Section 8.4, look at the relationship between the number of lexical data points for language and the CHRF improvement, which provides some insight (albeit not perfect clar- ity) into why these particular languages did well.
formance is higher and there is less improvement to be made simply by better aligning vocabulary tross-iigguisticall... Some of the biggest winners in this direction that weren't 't already discussed for the en->x direction are: 1. Tsonga (ts): up 1-3.1 CHRF
2. Guarani (gn): up +2.8 CHRF
Bashkir (ba): up to +2.5 CHRF
Minangkabau (min): up to +2.5 CHRF
F.2 GATONES
en-*x The biggest winners on GATONES in the en >xx direction are given in Table 14. Though there is some overlap with the biggest winners on the FL.ORES-200 dataset (e.g. Πlocano, Bambara, Mizo, Bhojpuri), number of different languages perform well too, some of which simply aren't 't included the FLORES-200 set. The languages which gain > 5.0 CHRF on this part of the evalua- tion are:
1. Adyghe (ady): up to +14.1 CHRF
2. Kedah Malay (meo): up to +12.6 CHRF
3. Goan Konkani (gom): up to +11.6 CHRF
4. Bhojpuri (bho): up +10.1 CHRF
xx-+en Table 13 shows the top 5 biggest win-
5. 1loсanо (ilo): up to +9.8 CHRI
ners per model for the xx >en direction. Clearly
Avar (av): up to +9.5 CHRF
there is lot of overlap with the en-xxx direction, although there are some differences. Also, note
Bambara (bm): up to +9.0 CHRF
8. Mizo (lus): up to +8.9 CHRF

---

Model
GatiPanlleXTokenPairs
ilo (+7.5)
nus (+6.8)
lus (+6.2)
bm (+5.5)
(+4.4)
CodeswitchMono
bho (+8.6)
bo (+5.6)
lus (+5.2)
ilo (+5.0)
quy (+4.9)
CodeswitchParallel
ilo (+3.7)
bho (+3.5)
as (+2.4)
lus (+2.3)
min (+1.9)
CodeswicchhMnnParrllll
bho (+9.4)
ilo (+6.4)
bo (+6.2)
lus (+6.0)
In (+4.9)
CodesvinchMonnoaiiPPaex
ilo (+9.1)
(+8.3)
bm (+8.1)
bho (+7.9)
bo (+6.8)
GlowupMono
(+6.2)
acq (+4.5)
bo (+3.8)
aeb (+3.1)
bho (+3.1)
GlowupParallel
bho (+4.4)
shn (+3.8)
sg (+3.7)
kac (+3.6)
kpb (+3.5)
PlrrullonoParallell
bho (+12.3)
sr (+7.1)
(+4.5)
bo (+4.1)
nus (+3.3)
GlowupMonoGaiiPalex
bho (+14.5)
bo (+8.0)
(+7.3)
bm (+5.7)
acq (4.2)
Table 12: Top 5 biggest winners per model (en-+xx) on the FLORES-200 test set, measured in ACHRF over the baseline.
Model
CauiPanlexTokenPaee
bm (43.5)
ts (+3.1)
lus (+2.7)
(+2.6)
tum (+2.1)
Mode
bo (+2.5)
lus (+2.4)
ti (+1.4)
ko (+1.3)
ay (+1.3)
CodeswitchParallel
mni (+2.2)
min (+1.9)
kg (+1.7)
lus (+1.4)
kbp (+1.4)
Podes
bo (+2.2)
lus (+1.9)
ay (+1.4)
bm (+1.1)
(+1.0)
CodeswichhMomooGaiPPnneess
lus (+5.0)
bo (+3.5)
bm (+3.2)
(+2.8)
ba (+2.5)
GlowupMono
bo (+2.4)
ti (+2.2)
ks (+2.2)
(+2.0)
mai (+2.0)
GlowupParallel
min (+2.1)
ee (+1.9)
kg (+1.6)
kac (+1.5)
kbp (+ (+1.5)
PaaalplonoParalllll
lus (+2.9)
min (+2.5)
bo (+1.8)
ace (+ 1.8)
bug (+1.8)
GiovapMonoGatiiaaleee
lus (+2.2)
(+2.2)
ko (+1.9)
sa (+1.8)
ckb (+1.7)
Table 13: Top 5 biggest winners per model (xx-+en) on the FL.RRS-2000 test set, measured in ACHRF over the bascline.

---

9. Madurese (mad): up to +6.6 CHRF
guages between the GatiPanlexTTkenPPal model and the GatiTokenPairs model. The second of these models is trained on a strict subset the data that the first is. If language performs better with this subset of the data, we can presume that the Panlex data was on average lower quality; if a language performs better on the superset, the Panlex data might still be lower quality, but its quantity at least makes up for performance some degree. The languages that do over +0.3 CHRF better on the subset data are ts, dv, bm, lus, ff, and ckb, suggesting that those may have poorer-quality Pan- lex data, with the largest difference being lus at +2.7 CHRF; those that do better on the superset are gom, mni-Mtei, kri, 1n, doi, ay, sa, ti, mai, and as, suggesting that Panlex still adds useful signal there.
10. Assamese (as): up to +6.3 CHRF
11. Pattani Malay (mfa): up to +5.7 CHRF
12. Kalaallisut (kl): up to +5.4 CHRF
13. Tibetan (bo): up to +5.2 CHRF
xx >en The biggest winners in the xx >en direc- tion are given Table 13. Many of the biggest win- ners overlap with the en -xx direction, but some of the languages that haven't yet been mentioned are:
1. Manipuri (mni ( (-Mtei)): up to +7.7 CHRF
2. Dogri (doi): up to +5.44 CHRF
3. Dhivehi (dv): up to +3.4 CHRF
4. Tigrinya (ti): up to +3.00 CHRF
F.3 Big model winners
The biggest winners for the 1.6B parameter models are given in Tables 16, 17, 18, and 19.
The picture that begins to come together is that Panlex often has some useful signal, but also con- tains considerable amounts of noise. For a less expressive model that is already not able to reach very high quality, some noise in the lexicons does not hurt, and Panlex can help the model get off the ground for the rewest-resource languages. But for a stronger baseline model that produces higher- quality translations on average, this noise can tively harm performance. Therefore, more care- fully curated bilingual lexica, like GATITOS, will tend to will yield higher quality results when used for model training with bigger models, as evinced in Table 21.
F.4 Empirical study of the quality of Panlex
One way to judge the quality of dataset is to review it manually, as in (Kreutzer et al., 2022); another is to the empirical effects on model quality of training on it. As byproduct of using Panlex for this project, can we judge the quality of Panlex for different languages?
To reduce noise, we average the scores on the three main uses of the bilexes, namely the TokenPairs model, the Glowup model, and the Codeswitch model. We average the FLORES-200 and GATONES scores. We then compare those scores to the baseline model for both en->xx and xx-ten. For the purposes of this analysis, we treat any absolute delta of under 0.3 CHRF to be noise. The result is displayed in Tables 20 and 21.
One would like to say that the upper left-hand corner represents languages with unequivocally high-quality lexical data, and the lower right-hand cornrr represents languages with poor quality lexi- cal data. Alas, however, this picture rather mud- died when we scale up to larger models, as we see that many languages jump from one bucket to another. Nonetheless, we do see the trend that GATITOS languages tend to cluster to the upper left- hand corner in both cases, and that Shan ('shn') and Latin ('la') do poorly in all cases, and should likely be avoided by practitioners.
Teasing out the confound of the mixed GATI- TOS and Panlex data: For the 26 GATITOS lan- guages, it is harder to trust the previous analysis. However, we can compare the scores of these lan-

---

Model
TatiPanllxTokcePaaiis
gom (+11.6)
ilo (+8.1)
meo (+7.7)
bm (+6.8)
lus (+6.6)
CodeswitchMono
ady (+14.1)
av (+9.5)
meo (+6.8)
mad (+6.3)
gom (+5.3)
CodeswitchParallel
tiv (+3.6)
min (+3.4)
as (+3.3)
iso (+3.2)
mfa (+2.9)
CodeswiithhonnPParalle
lus (+7.7)
ady (+6.8)
mad (+6.6)
bm (+6.0)
mfa (+5.7)
CoteswiicaMonnGGaiPaneee
bho (+10.4)
gom (+9.9)
ilo (+9.8)
bm (+9.0)
lus (+8.9)
GlowupMono
meo (+10.4)
bho (+6.6)
bo (+5.2)
gom (+5.0)
mfa (+4.1)
GlowupParallel
bal (+4.1) 1)
yua (+3.1)
meo (+3.0)
(+2.9)
mni (+2.6)
GlowupMonnPParallel
meo (+7.1)
gom (+5.9)
mad (+3.9)
za (+3.7)
mni (+3.5)
GlowapMonoGariPanlee
meo (+12.6)
gom (+11.6)
as (+6.3)
(+6.2)
(+5.4)
Table 14: Top 5 biggest winners per model (en-*xx) on the GATONES test measured in ACHRF over the baseline.
Model
PatiPPnnexTookenPee
bm (+4.7)
mni-Mtei (+4.7)
gn (+3.8)
lus (+3.6)
ilo (+3.5)
ModeswitchMono
(+3.5)
mni-Mtei (+2.9)
yua (+2.5)
dy (+2.3)
lus (+2.2)
CodeswitchParallel
mni (+3.0)
cv (+2.5)
av (+2.1)
lus (+1.9)
ee (+1.8)
CodeswitchMomoPrrllle
mni-Mtei (+7.7)
av (43.8)
lus (+2.4)
bm (+1.5)
chr (+1.5)
CadeswitchMonnoGaiPaalle
mni-Mtei (+5.4)
lus (+5.3)
gn (+3.5)
bm (+3.5)
dv (+3.4)
GlowupMono
ti (+3.0)
bo (+1.8)
or (+1.6)
que (+1.4)
dv (+1.4)
GlowupParallel
ee (+2.2)
mad (+1.6)
yua (+1.6)
av (+1.6)
bm (+1.5)
GlowupMonoParallel
mad (+2.8)
min (+2.2)
lus (+2.0)
que (+1.6)
gom (+1.3)
GlowapMonoGatiPanlex
mni-Mtei (+6.0)
doi (+5.4)
gom (+3.5)
dy (+3.3)
bm (+2.7)
Table 15: Top biggest winners per model (xx-ten) on the GATONES test set, measured in ACHRF over the baseline.
Model
MatiPanlexTTokePPiiise
is (+7.5)
din (+6.0)
In (+ 5.8)
ilo (+5.3)
ay (+4.1)
CodeswiiccMommGGGiPaneee.aa ts (+6.9)
bm (+5.6)
ilo (+5.2)
In (+4.8)
mag (+3.8)
Table 16: Top 5 biggest winners per model (en->xx) the FLORES-200 set for the 1.6B parameter models, measured in
ACHRF over the baseline.
Model
GatiPaaleeTTokeenaiiiBiigg
tpi (+9.1)
mni (+6.2)
bm (+5.5)
Is (+3.5)
ay (+3.1)
CodtswiiclMonnGaaPPaleeee
tpi (+5.1)
(+3.0)
mni (+2.2)
bm (+1.9)
Itg (+1.2)
Table 17: Top 5 biggest winners per model (xx-ten) on the FLORES-200 test set for 1.6B parameter models, measured
ACHRF over the baseline.
Model
GatiPanleeTokeenniirBBB ts (+7.7)
gom (+6.1)
ilo (+6.0)
dv (+5.8)
bm (+5.1)
CoteswicchMomGGuPPallliii ts (+7.3)
bm (+7.0)
ilo (+6.8)
mni-Mtci (+5.5)
gom (+5.3)
Table 18: Top 5 biggest winners model (en-+xx) on the GATONES test set for the 1.6B parameter models, measured in
ACHRF over the baseline.

---

Model
CatiPanlleTTokePPaiisiii
(+8.7)
mni (+8.4)
bm (+6.6)
kl (+5.8)
(+5.0)
CodeswiichMoonoGGaiPaalaeiigB
cv (+6.9)
kl (+6.2)
ce (+3.4)
av (+2.8)
chr (+2.7)
Table 19: Top biggest winners per model (xx-+en) GATONES
1.6B parameter models, measured
A ACHRF over the baseline.
Win xx-
Neut. xX-+en cjk ckb ny tn
Loss xX xX-ten
Win cn-+X
aa ace av ay hei ev doi dy dyu ee gn gom khp kri lg lus mad min mni-Mtei nso nus om que quy ts yua
acm acq acb af am ape ar ar-MA arz as awa az ha bbe be bg bho es ey da de el en et eu fa-AF fi fj to ga gl gu hu hy kk km kn ko ks ku lb to Itg lv mag mfa mn mk ml mn mni mr ms mt nl no pa m ro ru rw sen si sk sl sn so sq sr su sv sw ta te tg th tr tt uk ur uz vi war xh yue zh zu
ca ce ceh
iso it iw ja jv ka
Neut. en -XX
ak bug pag tpi
bew kmb
ban brx- Beng es få gd ha hi ht ig ky In pap pl ps sd skr ik ug vec yi yo zza
pem sm ady her kac
sat- sat-Beng
mai mi my
T Loss
din ff fon kg
ahr ber-l Lat- hnc la shn
Table 20: Languages sorted whether it helps hurts include PanLex and GATITOS, for the smaller models (Transformer
Big, 475M). GATITOS languages bolded.
Win xx- ay ev ee
Neut.
xx-ten
Win en en- Neut.
awa bei din doi mag quy
aeb ba ckb dyu kg Ig mfa pag pap m sat- at-Beng skr su Is yua
mni-Mici Itg ce km mg tpi zza
gomn
af am ar az ban be bg bho bn bs ca es cy de eo es et eu fa fa-AF If fi fr gd gl gu ha hi hnc hr ht hu hy id is it iw ja ka kk kn ko ky lb lo It ly mad mi mk ml mn mr ms mt ne no ny pem pl pt ro ru si sk sl so sq sr sv sw ta to th tk tr uk ur uz vi yi yo zh zu bbc cjk kmb
ace aem acq apc ar-MA arz bew brx-Beng bug ceb dz fj ig jv ks nso oc om or son sd sm sn st ug vec xh yue
en-XXX
Loss +X-
ady ber-Latn min mni
tum
as ber bo fon iso kac kbp ku la mai nus que shn ti war wo
Table 21: Languages sorted by whether helps hurts include PanLex and GATITOS, for the bigger models (Transformer
1.6 1,6B). GATITOS languages bolded.

---

12
Lexicon
10
STT
Neither
AchrF baseline (enxx)
word (PanLex+STT)
Figure Number of lexicon word pairs augmented vs. ACHRF over baseline for unsupervised languages en- direction, Results for FLORES-220 and GATONES are combined here.
DchiF over baseline (xxen)
2
Lexicon
Paniex
Both
STT Nether
100000
150000
200000
Number lexicon word pairs (PanLex+STT)
Figure Number of lexicon word pairs sugmented data vs. ACHRF over baseline for unsupervised languages in xx xx-ten direction, Results 200 and GATONES are combined here.
Does lexical augmentation fix common MT mistakes?
are given below.
The results of this analysis given in Table 22. For each error type, we computed the percentage of sentences that were affected by dividing the num- ber affected sentences by the total number sentences in the eval set. FLORES- has 806248 sentence pairs across all languages and GATONES has 309887.
In evaluating the "big" models with 1.6B parame- ters, wished to whether preferred lexical data augmentation methods GGatiPanlexToken2 ColeswichhMonoGaiPalleee reduced several common types MT errors. The errors we looked at were (1) null output, or the "question mark phe- nomenon," where the model simply outputs some unrelated symbol (such as question marks) instead of actual text; (2) copying, where the model copies some all of the source sentence in its prediction; and repetition. where the model erroneously re- peats the same word phrase many times. There are other error types we could look at, like halluci- nation, but we stick with these three basic types for this paper. More precise definitions of these errors
Null output (question mark phenomenon) The first error type occurs when the model outputs only "??', or some other arbitary character, as its predic- tion. Instances of this likely indicate catastrophic effects of cut-of-dommin phenomena for O-shot translation.
Copying Another common enror copying, where the model's prediction is close or identi-

---

% question marks
% near copy
% repetition
FLORES-200
BaselineBig
0.7
4.5
3.4
GatiPanleeTookeePaiisSSt
0.9
3.6
2.7
CodewiichhMomoGGlPaleeeBeB
0.4
4.7
3.2
GATONES
BaselineBig
0.6
2.2
3.1
Big
0.6
1.8
2.2
CodeswiichMonnoaaiPaaaaaata
0.4
2.3
2.6
Table 22: The frequencies of three common error types in MT in each of the eval sets, as percentage of the total sentences in the set that had each issue (lower is better). Exact descriptions of the error types are given in Section
cal to the source sentence. In measuring this phe- nomenon, we said that any prediction with > 85% character-level similarity to the source sentence considered copy. To measure character- level similarity, we took the multiset intersection of the character frequencies in the souree and the charac- ter frequencies in the prediction, and then divided the size the intersection by the number of char- acters in the source.
with probability Per. to obtain the codeswitched sentence x'. When substituting a source word for its translation, choose translation uniformly at random from all available translations in all lan- guages. Because of this, it is usually the case that is codeswitched into many languages. Finally, we train the model reconstruct the monolingual sentence x from using the same sequence to- sequence model and loss function as for the MT task.
Repetition The last common enror type exam- ined was repetition. To count these mistakes, divided the total number of tokens in a sentence by the number of unique tokens. If the ratio was > 3, we counted the prediction as an instance of erroneous repetition.
In our experiments we use use Par = 0.4. We apply MCA to all 208 languages in our corpus, but augment only half the available monolingual data and train the remaining half with MASS (Song et al., 2019b), done in the baseline training regime (Bapna et al., 2022). We prepend task token, <2codeswi tch>, to the codeswitched sen- tences to cue the model to perform the MCA task, as well as language (<2lang>) and script (<2script>) tokens. The <2lang> and script <2script> tags are used in all models, including the baseline.
H Comparing sampling strategies for translating tokens
As one recalls from Section 6.1.1, the Codeswitch augmentation works as follows: Let D represent a multilingual lexicon containing word or phrase translation pairs for many languages. Given a source sentence x = = (21, from mono- lingual corpus Xmono- we substitute each token in x for its dictionary translation with probability Ptr- However, there is an issue with this formulation. Because the lexica we use do not have exhaus- tive coverage across languages, is often the case that simply looping over x and attempting to trans- late each token with probability Ptr would result in translating a fraction of æ that is significantly less than Ptr- So in order to approximate this de- sired fraction ptr as closely possible, we first count how many tokens in have dictionary trans- lations. Let this number be k. We then compute the adjusted probability p., = max( nPt.. 1), and sam- A ple from amongst the words in x with translations
Since this augmentation samples each token with some probability. the number of tokens translated in given sentence follows a binomial distribu- tion. The Glowup augmentation, however, samples a number of tokens to translate uniformly at ran- dom from all possible translatable tokens. So one has binomial distribution over N tokens sampled, and the other has uniform distribution does this make difference?
To test this we trained version of the CodeswitchMono model using uniform sampling. The average CHRF of the CodeswitchMono form model was 0.1 0.2 higher on all four of the (en >xx xx >en) x (Fr.ORFS-20 GATONES) directions. We conclude that this may have a slight benefit, but the difference is within the realm of

---

noise, and does affect the conclusions else- where in this paper.
I Relationship between number of tokens and MT performance
We also graph the relationship between number of lexical word pairs and ACHRF in Figures (en >xx) and (xx-ven) for URLs only. The re- sults for FLORES-200 and GATONES are combined in these plots. In both directions, we observe a moderate positive relationship between the number of lexical word pairs for a given language in the augmented data and the ACHRF over the baseline.
J Languages
J.1 Rationale for Language Choice
Although this project is aligned with the 1000- language initiative from Bapna et al. (2022), we wanted use smaller models for more rapid itera- tion, and as result, commensurately smaller data and number of languages to comfortably in the model. Therefore, we chose to work with about 200 languages.
With this in mind, we also wanted to choose specifically those languages whose performance we could measure. Therefore, our approach was as follows:
Include all languages with supervised (paral- lel) data, for maximal cross-lingual transfer
Include all languages that have non-zero data and a FLORES-22 200 eval set Include all languages that have non-zero data and a GATONES eval set
JL2 Complete Language data
The following table gives a list of the languages used in our experiments, along with some linguistic and resauree-related statisties. The numbers for data resources (i.e. Mono, Parallel, Panlex, and GaTItOS) refer the amount of data actually used in our experiments, not necessarily the total amount data available. For example, we subsampled the parallel and high-resource monolingual data we had available by a factor of 10.

---

BCP-47
Language
Mono
Purilel
PanLex
GATITOS
Speak:
Script
Cont.
Family
English
HRL
738.M
472044
3.0M
0
964M
Latn
Euroge
Indo European
Spunish
175.M
3850M
2.4M
Euroge
aude-uuooean
do
German
HRL
18
389.3M
2.8M
4K
130M
Latn
Europe
Indo European
id
Tralenssion
HRI.
95M
97
LIM
4K
198M
Lain
Asia
Austrmesiam
yuc
MRL
Asia
hu
Hungarian
HRL
724M
50.9M
1.3M
O
13M
Lain
Europe
Indo- European
pl
HRL
Europe
Indo- huroeen
zta
Mandazin
HRL
67.6M
215.7M
6K
4K
1092M
Hans
Asia
Sino Tibecan
Maori
MRL
674M
1.MM
474K
0
50K
Lain
Oreania
Austrmnsiam
ko
Korean
HRL
67.2M
128
1.3M
SK
77M
Kore
Asia
Koreanic
Jegunese
HRL
65.3M
2:2M
4K
126M
Jpan
Asin
Japonic
MRL
59AM
ru
Russion
HRI.
57M
294. 7M
2.5M
4K
268
Cyrl
Fungu
Indo-Furgee Furopean
gd
Scotlish Gaelic
MKL
56.9M
Euroge
Indo European
tr
Turkish
HRL
56.5M
159.2
1.2M
4K
7IM
Latn
Asia
Turkic
Somali
MRI.
56.4
13M
112K
16M
Latn
Africa
Afro-Asiatic
Thoi
53.1M
69 IM
1.6
Asia
ha
Hausa
MRL
50AM
LSM
202K
0
SOM
Latn
Africa
Afro-Asiate
Indo- Europenn
pr
Portuguese
HRI.
48.5M
1.M
4K
230M
Latn
Furge
Indo- Mulo-Prroeann
vi
Vietnamese
HRL
47.7M
825X
Asia
Austrossiatic
fr
French
HRL
4SM
481 AM
2.5M
4K
230M
Europe
Indu- European
ccb
Cetuano
MRL
45.9M
WEM
KIK
20M
Latn
Asia
Austranesian
yo
Yoruba
MRL.
43.4
244K
Africa
Niger-Cungo
sd
Sindhi
MRL
42.3M
LXM
43K
26M
Arab
Asia
Indo Europezn
41.2M
Inde-Fun mule-Fureeean
ng
Malagasy
MRL
40:2
3AM
116K
25M
Latn
Africa
Austranesian
Malay
HRL
53AM
Asia
Austruncsian
Mar.
35:5M
0
SBM
Africa
Afre Asiati
hew
Berawi
URL
W3:3M
0
SK
O
5M
Latn
Asia
Malay Creole
ny
Nyunja
MRL
32,9
Africa
Niger-Cungo
nl
Dusch
HRL
32M
MSS2
16M
KK
22M
Lain
Europe
Buro-Eur
uk
Ukrainian
Furgee
mude-Puropean
sv
Swedish
HRL
WTTE
122.0M
1.M
12M
Latn
Europe
Indo European
how
Howaiian
MRL
30AM
695K
156K
24K
Lain
Americas
Austrmnsiam
ro
Romasian
HRL
45M
Europe
Europenn
Czech
HRL
30M
10S
166
13M
Latn
Europe
Indo European
hmn
Hmong
27.7M
4M
Europe
Himong
yi
Yiddish
MRL.
27AM
760K
2M
Hobe
Funey
Indu- Furopean
Persian
Arah
Asia
Indo mnde-buropenn
iz
Labo
MRL
27AM
61TK
49K
27M
Lain
Africa
Niger-Congo
Labvinn
MRI.
26M
22AM
2M
Tain
Funge
Indo- Funopenn
ac
Arabic
25.6M
310M
Asia
olb
Sorani
MRL
25.1M
155X
S3K
4K
7M
Arab
Asia
Indo- European
Tatar
Europe
Turkic
on
Sameon
MRI.
23.9M
50ZK
SKK
S10K
Lain
Cheania
Austranesian
Zala
MRL
23.M
144K
Affica
Niger-Congo
no
Norweginn
HRL
23.1M
85.M
4K
SM
Lain
Eurcpe
Indu- European
Sesocho
MRL
Z2AM
IEM
35K
6M
Latn
Africa
Niger-Cango
WLTI
Asia
Dravidian
or
Dilla (Criya)
MRI.
21.3M
169K
16K
35M
Orya
Asin
Indu- Fano-unnee
Shona
MRL
Afria
Niger-Cango
bo
Tibetas
LXL
17.M
282K
56K
IM
Thht
Asia
Mine-Tihetan
el
Greek
HRL
17.3M
5IM
1.M
13M
Grek
Europe
Indo European
Finnish
HRI.
17:1
48AM
2.M
6M
Fanqee
hi
Hindi
HRL
16.5M
75.M
449K
4K
381M
Deva
Asia
Indo European
Africa
Niger-Cungo
me
Marathi
MRI.
15AM
81M
154K
75M
Dosa
Asin
sk
Slovak
HRL
15.6
639M
7M
Euroge
Indo European
hy
Armenian
MRL
15.4M
6.M
35LK
5M
Armn
Asia
Indo European
Kazakh
MRL
1533
66M
241K
13M
Cyrl
Asia
Turkic
Danish
HRL
15.2M
789M
Europe
buropen
mk
Macedonian
MRL
15M
66M
363K
O
2M
Cyrl
Europe
Indo European
bg.
Rulgssia
MRI.
Funope
Iuhe-Funogean
sr
Sertiun
MRL
14.1M
30AM
200K
SM
Cyst
Euroge
Indo European
ml
Malayalam
MRL
13:9M
64M
163K
3M
Mism
Asia
Dravidian
uz
Azerbaijani
MRL
13.5M
19.6
23M
Lain
Asia
Turkic
is
feclandic
MRI.
134M
15
6OBK
310K
Tain
Funge
Indo- Funopenn
Telnga
MRL
12.7M
Asia
Dravidian
nc
Nepali
MRL
11.6
9.7M
16M
Deva
Asia
Bulo-Enrepean
Mazandernni
URI
11AM
Asin
Indo- Europen
meo
Kedah Malay
URL
11.M
O
2M
Lain
Asia
Austronesian
Fetunian
MRL.
11.2M
30
IM
Eungee
Uralic
tw
Hebrew
HRL
10.8
SM
707K
SM
Hobe
Asia
Afro- Asiatic
Afre-Asiatic
K Kinyarwanda
LXI.
BLIM
BISK
56K
1OM
Latn
Africa
Niger-Cango
Mongolian
Mongelic
ur
Undu
MRI.
9.9M
15:2M
400K
163M
Arab
Asia
Indo- Funogean
Lev, Anbii
URL
Asia
Atto-Asiatic
hr
Creatian
MRL
9.7M
17AM
738K
7M
Lain
Europe
Indo European
Gi
Filipino
MRL
9.5M
25.3M
61K
45M
Lain
Asin
Austroesiaa
Ascumese
Beng
Asia
Indu- Funopenn
Egyptian Arabic
URL
9.2M
0
10LK
0
58M
Arah
Afric
Afto-Asiat
251K
Indu Buropean
Pipiamento
TIRI.
9.1M
0
99X
341K
Lain
Amercaa
Partggeee Croole
Dari
LRL
8.7M
1OK
Arah
Asia
Indo Europein
am
Mcsop. Arahic
URL
8.7M
JIK
15M
Arah
Asia
Afro-vsiatio

---

BCP-47
Language
Cat.
Mono
Purilel
PanLex
GATITOS
Speak:
Script
Cont.
Family
Lithanian
MKL
8.MM
30:M
611K
O
2M
Latn
Euroge
Indo European
Mixo
Asia
Sino-Tiheran
be
Belarusion
LRL
8.3M
6.5M
508K
0
2M
Cyrl
Europe
Indo European
km
Kannada
LXI
8M
5.M
116K
0
47M
Kida
Asia
Davvidian
Asia
European
Burmese
LRL
7.8M
5.5M
121K
0
43M
Mymr
Asia
Tine-Tibctan
Occitan
Indo-Purgeen
bn
Bengali
MRL
7.3M
21.7M
259K
262M
Beng
Asia
Indo European
af
Afrikmans
MRL
7.1M
12.7M
258K
18M
Affica
Indo-Buneeen
cu
Basque
7M
6AM
792K
540K
Lain
Europe
Langrage iooltt
Gujarati
LXL
6.9M
5.M
198
47M
Guje
Asin
Indo- Furogeun
gl
Calician
13.1
Europe
Indo European
Sanekrit
LRI.
6.2M
11K
168K
4K
100K
Dosa
Asia
Indo- Furopean
Siovenian
27M
Indo kuropean
up
Uyghur
LRL
5.7M
526X
116K
JOM
Arab
Asia
Turkic
ba
Rodikkr
LXII
3.6M
SI3K
112K
IM
Cydl
Funge
Turkie
Sinhula
5.6M
Indo- Furopenn
Oromo
LKL
5.AM
203K
IOK
4K
24M
Latn
Africa
Afto-Asiatic
Zaza
Indo- Buropenn
us
Urbhk
MRI.
5.3M
11:2M
34M
Latn
Asin
Turkie
Swahili
MRL
5.2M
10.5M
150M
Afria
Niger
Khmnr
MRL
5.IM
8M
188K
17M
Khonr
Asia
Austroniatic
ay
Kyrgyz
LXL
4.9
29M
174K
SM
Cyrl
Asin
Turkic
4.M
vec
Venetian
URL
4.4
0
202K
O
4M
Latn
Europe
Indo European
Cstalan
Fungee
Inde- Purooenn
ts
Turkmen
LXL
JM
410K
177K
7M
Lain
Asia
Turkic
Tirrinya
LRL
3.9M
67K
SM
Ethi
Afric
Afro-Asiale
ps
Panjasi
3.M
Furopenn
Albanian
MRL
3.2M
10AM
209K
13M
Latn
Euroge
Indo Europezn
ka.
Georgian
3M
11.M
Chusash
URL
2.8M
0
121K
0
IM
Cyrl
Europe
Turkic
ilo
Bocano
Asia
Balachi
URL
2.5M
IGK
SM
Arab
Asia
Indo European
Experanto
LRL
2AM
7.5M
1.3M
2M
Lain
Eurce
Consrucicd
cy
2.2M
Europe
buropenn
la
Latin
LRL
2.2M
22M
740K
0
0
Latn
Europe
Indo European
Dzongkha
2.IM
200K
Maleese
LRI
2.1M
73M
247K
470K
Lain
Funey
Afho-siatic
Tswuna
Niger-Congo
l
Luganda
LRL
2M
3IK
4K
4M
Lain
Africa
Niger-Congo
la
Hailian
1.II.
2M
34M
177K
SM
Tain
Americas
French Creole
Sepedi
Niger-Cungo
ps
Pashio
LRL
LRM
2.1M
50M
Arab
Asia
Balo-Euroeenn
Zhuang
ga
brihh
1.I1
1.7M
AM
428K
1M
Lain
Funge
Indo- Puroeen
Tok Pisin
LRL
1.7M
120K
Oceania
English Creole
pem
Nigerian Pidgin
1.6
24K
4K
40M
Africa
English Croole
ib
Lunembourgish
LXL.
1.5M
4AM
183K
420K
Latn
Europe
Inde-Burogeur
Kurmanja
LSM
Indo European
n
Tajik
LRI.
LAM
1.5M
194K
SM
Cysl
Asin
Lingala
Niger-Congo
ce
Chechen
URL
LAM
113K
IM
Cysl
Euroge
Cancasian
Maichili
URL
1.M
65M
Deva
Asia
Indo European
i-
Javancec
1.5M
6.M
128K
Asia
Austraesimm
t.
Tsonga
LKL
1.M
2K
9K
4K
13M
Latn
Africa
Niger-Cango
ak
Twi
LXI.
1.5M
38K
115K
4K
11M
Lain
Africa
Niger-Cango
Lacn
Tamazight
URL
1.MM
Affic
Afro Asiatc
su
Sundanese
LRL
1.2M
2.7M
SOX
34M
Loin
Asia
Austronesian
fs
Westem Frisian
LRL
I.IM
4.6M
850K
Europe
Indo-Eursee mud-Euueeean
Samiki
974K
Asia
European
bbe
Barak Toba
URL
932K
0
23K
0
2M
Latn
Asia
Austrunesian
(PHs)
gn
Guarani
LXL
861K
1.M
6K
4K
SM
Lain
Americas
Tupian
Quechua
LRL
B-2K
46X
9M
Americas
Quechuan
bus
Buginese
URL
797K
0
J9K
0
6M
Lain
Asia
Austroncsian
Fac
1.I1.
796K
4K
OOX
4K
4M
Tain
Africa
Nigo-CCmgs
ing
Latgalian
796K
Euroge
European
kl
Kalaulisut
LRL
741K
500
4SK
4K
56K
Lain
Americas
Eskimo- Alout
Bhojpuri
Asin
Indo- muno-enooeann
ar
Lain
Arabic
URL
634K
4K
2M
Lain
Asia
Afro Asiatic
Pangawinan
URL
594K
19K
IM
Lain
Asia
Austronssia
Shun
URL
S66K
0
16K
3M
Mymr
Asia
Kra-Dai
min
Miningkabsm
URL
533K
0
TEK
6M
Latn
Asin
Austranesian
Chokwe
694K
Niger Cango
Yisnteus
URI.
419K
0
6TK
7663
Lain
Americas
Mayan
eg
Nghundi Crecle
iso
Looko
URL
109K
0
3K
420X
Lain
Africa
Niger-Congo
Kachin
URII
4IZK
0
10K
940K
Lain
Asin
Simn: Tibetan
kg
Konga
376K
Nigor-Cango
gom
Goan Konkani
URL
311K
0
37K
4K
2M
Deva
Asin
Indo Europein
Indo
nw
Avaric
TIRI.
RITK
0
216K
0
760K
Cysl
Funge
Northrss Caucasion
thr
URL
297K
Affei
Canoo
ady
Adbghe
296K
25K
575K
Cyrl
Eureee
Northwss Cancasion

---

BCP-47
Mono
Purilel
PanLex
GATITOS
Speak:
Script
Cont.
Family
wo
Wolof
LX1.
289K
240K
94K
0
4M
Lain
Areaa
Niger-Congo
Chhurtigahh
Asia
Indo-Euroeein
ny
Aymara
LRL
267K
600
91K
4K
2M
Latn
Americas
Asmaran
KTiche"
URI
250K
0
54K
0
2M
Lain
Amercca
Mayan
Asia
Mosop, Arabic
URL
216K
0
3K
O
7M
Arah
Asia
Asiatic
Nigor-Cango
Balinese
LRL
185K
3OX
O
2M
Latn
Asia
Austronesian
bon
Baméura
URL
187K
73K
4K
14M
Affica
Mand
Dogri
URL
179K
Deva
Asia
Indo- European
Tumbuka
LXI
171K
4K
TK
2M
Latn
Africa
Niger-Cango
Baculé
Niger-Cong Siger-Congo
Ayunacho Quachun
URI
140K
S4K
SOOK
Tain
Americas
Quachuan
Madurese
135K
Asia
Austronesian
awa
Awadhi
URL
136K
0
35M
Deva
Asia
Indo European
Dyula
URI
130K
SK
4K
3M
Latn
Africa
Mand
Ahp
Kabiya
129K
Nige-Cango
kni
Kito
URL
129K
0
IOK
4K
496K
Latn
Africa
English Creole
Niger-Cungo
mni
Maniguri
[RRI
TOSK
2K
1M
Beng
Asin
Minn-Tihctan
Misi
Manipuri
URL
103K
IM
Asia
Sino Tibetan
Tamnaxight
URL
96K
30M
Ting
Africa
Afre-Asaatic Asiatic
kmb
Kimbundu
UKL
94K
0
TK
AM
Latn
Africa
Niger-Cango
Sicilian
Europe
buropenn
tf
Fulah
LRL
95K
4K
30K
4K
30M
Latn
Afáic
Niger Congo
ffr
Afro-siatic
ks
Kashmiri
LXL
71K
24K
6M
Arah
Asia
Indo- Europein
Magahi
URL
6AK
HM
Deva
Asia
Indo Europenn
76K
Iragarian
din
Dinka
URL
62K
0
3K
IM
Latn
Africa
Nilo Saharan
abr
Ahirani
URL
24K
0
2M
Deva
Asia
Indo European
Naer
Saluran
mia
Partani Malay
URL
7K
1000
Arah
Asia
Austronesian
sa-BBng
Santali
URL
7K
6M
Beng
Asia
Austrniatic
brx-Heng
Beng

---

K
Full results
The full results on FLORES-200 for the various models we trained are available in Tables 24 and 25. Model abbreviations are clarified in Table 23. Scores from the NLLB model are included as refer- ence, though keep in mind that the smaller research models in this paper will naturally have lower qual- ity; even the "Big" models are 30x smaller, and not optimized with
and so on.
Model Name
Abbr. T
Description
Baseline
Trained on MASS + translation
GatiPanleXTokemPairs
Baseline + 5% token pairs
GuuPPanlexToocePPPttSSmmpp1
T75
token-pairs sampled 75%
CaiiPalkxrTokeePaarsSaappo
Tso
token-pairs sampled 50%
GuiPaalexTTokenPitrsSSmpp
T25
token-pairs sampled 25%
GatiTokenPairs
Taat
token pairs with only GATITOS
Mono
CM
See $6
Codeswitchh
Cp
See $6
CodeswiichhMomoParalld
CмP
See $6
CodeswicchhonnoGriPPnnees
CиT
See $6
GlowupMono
Gm
See $6
GlowupParallel
Gp
See $6
PlowupMonoParallll
GMP
See $6
GlovupMonoGatiPanlex
G
See $6
BaselineBig
BBGG
Big version of the Baseline
($8.1)
CariPanlexTokeePaairBBig CodeswicchMommGuiPPneeeiiii NLLB
TBIG
Big version of the TP ($8.1) Big version of C ($8.1)
BT
NL
NLLB 54B model
Table 23: Names, abbreviations, and descriptions of the full model results in Tables 25 and 24.

---

lang
car.
B
T
T7G
Tsa
Tas
Teatitus
Car
Cp
Cur
Csr T GM
Gp
Gup
Gy
Bare
Tare
(CM
TlarG
NLLB
89.1
41d
40.0
89.6
198
39.7
40.4
38.3
39.5
40.7
40.8
38.6
40.0
4017
44.5
44,4
44,1
50.7
TRRL
mecm
50.5
50.1
50.5
50.1
50.1
50.1
51.3
48.5
49.4
50.9
52.0
48.4
50.6
51.6
56.0
55.5
56.1
58.5
MRL
46.6
46.4
46.5
46.4
46.4
46.4
47.5
44,9
45.8
47.2
48.0
44.8
46.8
47.6
51.2
50.9
51.3
53.8
LRL
mean
34.4
34.9
34,9
34,6
34.6
34.6
35.2
33.3
34,4
35.7
35.4
33.4
34.5
35.6
38.5
38.6
38,8
50.3
URL.
mean
28.7
30.2
29.5
28,8
29.5
29.1
29.2
27.8
29.9
30.2
29.2
29.2
29.7
29.5
34.8
34.5
34,5
40.1
00M
mean
12.6
10.9
13.2
12.3
12.5
13.7
9.7
12.2
12.2
11.0
12.9
AUR
mean
30.4
31.6
31.1
90.6
31.0
30.8
31.0
29.4
31.3
31.9
31.1
30.4
31.2
31.3
35.9
35.7
35.8
43.2
ace
URL
27.4
29.9
25.8
25,9
29.8
28.4
31.3
28.4
30.3
31.0
28.5
28.2
29.6
29.6
33.2
33.2
32.7
31.5
URL
42.0
41.5
42.5
41.3
41.3
41.7
43.4
39.8
41.3
42.9
44.4
39.3
42.5
43.4
49.0
48.2
48.8
42.1
bog
URI.
39.5
89.9
40.4
39.6
40.3
40.8
44.
38.5
43.5
45.3
42.7
42.5
49.0
46.7
49.2
26.8
aeb
URL
36.4
16.4
35.9
35.1
36.1
37.6
39.4
35,4
37.2
39.1
40.5
34.0
38.4
38.5
42.4
41.2
43.6
61.2
apc
40.1
39.8
39.5
50.0
47.8
ar-MA
URI.
36.0
35.7
36.2
35.6
35.7
35.5
36.8
34.5
35.2
363
37.3
34.0
36.0
36.5
39.9
39.8
40.2
28.3
URL
36.4
33.5
328
33.3
35.3
32.5
38.4
27.5
35.7
37.5
38.6
30.7
33,9
37.2
43.7
42.5
43.5
50.5
awa
URL
43.0
43.5
42.9
42.2
42.1
41.4
43.5
57.0
42.4
44.0
44.8
41.0
43.1
44.4
46.9
43.4
46.7
27.1
ber
URL
9.2
9.0
5.7
8.8
8.1
6.7
9.7
8.9
9.6
9.8
10.2
9.7
13.6
13.5
12.1
34,3
hm
URI.
20.5
21.0
20,4
18.1
20.0
144
16.6
22.6
13.8
16.9
14.3
18.4
23.1
26.5
28.7
bug
URL
29.3
29.1
29.4
29.2
29.2
29.4
29.6
29.0
29.3
29.5
29.7
29.1
29.4
29.6
20.1
29.8
30.1
30.4
ejk
14.7
16.1
din
URL
12.8
9.9
11.2
9.8
9.4
9.7
11.6
8.1
13.3
11.5
8.3
11.0
11.2
8.3
12.6
18.6
15.5
19.8
dyu
URL
12.8
15.6
14.4
14.3
15.0
13.8
11.7
14,5
15.9
16.2
12.3
14.8
14.4
15.1
17.7
19.6
20.6
50.7
fon
URI.
12.3
13.7
12.3
11.1
13.0
11.8
10.7
12.7
12.5
10.6
11.5
14.3
13.4
10.0
18.7
17.0
15.8
31.3
hne
URL
45.8
46.1
46.1
45.6
46.7
44.0
46.2
40.2
44.6
45.4
43.1
45.0
45.3
47.6
49.1
50.0
49.1
38.5
35.5
43.0
41.5
42.4
40,9
39.5
43.5
36.3
37.7
37.9
42.7
48.0
47.9
kac
URL
16.5
17.2
16.3
13.5
17.6
13.7
12.9
15.1
16.1
16.4
15.3
20.1
18.0
12.0
26.5
23.8
22.5
46.8
khp
URI.
12.6
13.4
13.6
10.2
11.
14.1
13.0
12.4
14.2
12.4
13.0
16.2
14.6
13.7
21.2
18.3
48.4
kmb
URL
19.2
23.4
19.4
19.2
20.6
17.6
15.7
19.8
22.2
19.0
19.1
22.4
21.8
17.5
25.0
24.7
20.5
41.4
URL
29.1
29.2
29.1
29.8
30.9
30.4
31.7
30.1
30.6
31.7
30.5
3L.
30.6
31.4
38.1
38.2
38.7
34,5
24.2
24.2
18.1
21.8
26.6
21.6
20.3
26.2
25.7
19.5
19.0
20.0
29.3
30.5
32.8
57.5
URL
47.3
48.0
46.4
46.0
47.1
45.0
48.2
38.5
45.1
48.4
47.9
45.8
47.5
49.1
47.5
50.2
51.3
23.5
mai
44.0
43,9
43.8
41.3
47.7
min
URL
38.2
494
39.5
39.8
39.7
39.4
40.0
40.0
40.0
40.1
39.8
39.7
39.5
39.4
45.7
43.5
43.6
39
mni
URL
13.2
12.
15.7
14.2
14.6
14.1
12.8
15.1
14.5
12.8
15.7
16.2
17.2
14.6
22.9
20.3
15.3
38.0
mus
URL
9.6
16.4
12.0
12,0
11.1
10.9
10.8
10.1
13.3
11.3
11.4
12.1
11.6
9.4
17.1
17.6
16.3
28.3
pag
URI.
35,9
36.9
36.3
35,7
36.1
36.3
35.1
38.8
37.3
36.4
34.9
36.8
37.0
35.6
39.9
40.2
40.9
55.6
pup
41.0
39.6
40.9
41.1
40.5
42.1
38.5
39.4
quy
URL
26.6
30.5
27.5
29.6
28.5
31.1
28.8
28.3
30.7
31.2
27.6
28
28.6
26.5
31,5
31.6
34,5
49.6
scn
39.3
39.0
38.9
39.5
43.7
g
URI.
21.9
25.1
23.1
21.9
22.4
21.4
19.0
22.3
25.9
21.5
23.4
25.6
25.9
19.8
31.2
29.8
28.7
56.8
shn
URL
14.2
15.6
14.7
13,5
15.6
10.6
11.9
12.2
8.5
12.6
8.1
18.1
12.2
13.9
25,1
16.2
21.3
38.3
vee
URL
42.3
42.2
42.2
42,1
42.0
42.1
42.8
42.0
42.3
43.0
42.5
41.5
42.6
42.8
44.4
44.1
44,3
41.9
war
URI.
51.5
53.2
$3.0
52.9
53.1
52.3
51.7
53.2
$1.2
51.2
52.3
52.0
52.3
$1.9
57.1
56.1
S3.1
31.8
am
30.6
30.5
31.6
30,3
30,9
32.5
32.0
33.2
31.3
38.4
37.7
59.3
LRL
13.1
15.3
17.0
12,9
160
16.0
11.1
13.5
14.5
18.6
13.9
12.8
15.1
15.9
20.7
18.2
17.2
48.9
LRL
22.9
22.9
23.5
ha
LRI.
14.6
16.8
13.0
14.3
15.5
15.9
13.6
15.2
13.2
17.3
16.3
17.2
12.9
15.8
16.9
15.6
18.3
50.8
bun
LRL
41.0
40.6
40.6
40,3
40.8
40.7
41.
39.8
40.2
41.3
41.0
39.0
40.6
41.3
42.4
43.1
42.2
53.2
IRI.
37.7
37.5
380
37.
37.5
37.2
38.4
36.2
38.0
38.9
37.8
3X.4
42.0
41,8
42.3
44.7
bho
LRL
14.7
17.9
20.0
21,9
24.7
19.2
23.2
18.2
23.1
25.5
24.4
19.1
16.0
21.6
21.0
15.4
22.1
$1.0
18.0
17.9
23.3
LRI.
64.4
63.9
64.7
64.1
63.8
64.0
65.8
61.4
63.1
65.55
66.8
61.2
64.3
66.1
72.2
71.9
72.6
45.4
T.I.
5.5
6.0
4.9
4.9
6.7
5.1
4.8
53.3
ee
LRL
26.9
29.5
28.9
28.3
27.2
29.8
28.4
25.2
28.5
30.5
26.8
26.3
26.8
25.6
28.6
32.0
31.3
40.4
LRL
57.6
57.5
57.7
S7.4
57.3
57.4
58.1
56.7
$7.0
58.0
58.2
56.5
57.6
58.0
59.0
58.8
59,1
27.9
47.6
47.1
47.8
47.0
46.8
47.0
48.1
$2.6
$2.4
A=-FF
LRL
47.4
47.1
47.5
46.7
47.0
46.6
48.0
45.9
46.1
47.0
49.4
45.0
48.3
48.6
53.3
53.1
$2.9
54.0
19.6
19.0
f
LRI.
33.7
348
35.6
33.7
35.6
34.0
34.0
33.7
35.5
34.8
34.0
34.2
35.7
35.2
39.7
39.6
39.7
42.3
LRI.
38.5
38.2
38.4
38,0
38.3
38.2
40.1
37.7
39.0
39.6
38.7
37.5
37.0
38.5
44,4
44.7
45.1
43.3
gu
LRL
47.9
47.6
48.6
47.9
47.6
47.9
49.5
46.1
47.3
49.1
50.2
45.7
48.0
49.7
55.0
54,8
55.7
40.8
I.I.
19.1
19.6
19.2
18.5
19.4
18.7
20.3
18.4
19.2
19.4
19.0
19.3
19.3
18.0
20.8
20.7
20.1
31.6
gu
LRI.
49.5
49.9
49.1
49.4
49.4
50.7
50.3
51.2
54.1
54.5
LRL
49.6
49.1 49.6
49.3
49.4
49.4
49.9
48.6
48.9
49.7
50.2
48.4
49.6
49.9
$1.2
51.1
$1.0
48.3
T.I.
51.7
81.8
52.7
52,1
51.5
51.7
52.5
50.7
$2.3
53.1
50.7
51.6
$2.8
54.5
54.3
54,4
50.6
kg
LRL
33.1
31.
33.0
32
31.0
32.9
32.3
31.9
32.9
32.8
32.9
32.9
33.1
32.1
32.8
35,2
35.1
6L.1
LRL.
50.5
50.4
51.
50,0
49.9
49.9
51.4
48.4
49.2
51.2
52.2
48.4
50.6
$1.77
55.5
55.0
55.7
42.8
LRL
08
08
0.8
0.8
088
0.7
0.8
0.8
0.8
0.8
0.8
1,0
40.9
ku
TRI.
33.2
33.1
328
32,5
32.9
32.4
33.8
31.5
31.7
33.4
34.1
30.9
33.4
33.9
37.9
37.1
37.0
35.4
LRL
41.1
41.5
40.9
41.
41.8
39.7
45.7
45.7
53.9
Ib
TRL
50,0
49.8
49.9
49
49.9
49.6
50.7
48.4
49.4
50.5
51.8
48
50.1
50.4
53.9
$3.5
$4.3
55.4
LRI.
30.1
31.9
31.4
31,5
30.5
31.7
32.0
29.4 32.3
32.7
26.6
29.0
29.9
31.3
30.9
34.4
32.0
61.5
LRL
31,9
34.8
35.2
34,9
32.3
33.1
28.9
30.9
36.0
38.6
32.3
29.0
30.7
33.0
33.8
39.6
38.5
51.8
mn
TRI.
42.4
41.9
41.5
41.3
41.9
41.8
43.0
39.9
40.7
42.7
43.7
39.9
42.4
43.2
47.8
47.5
48.D
50.9
mE
I.I.
62.6
62.7
62.4
62.5
62.0
62.7
63.5 60.7
63.1
64.0
60.3
62.4
63.8
68.2
68.0
68.2
60.4
my
LRL
35.3
34.0
34.9
34,4
34.6
34.0
36.0
52.7 34.0
37.0
36.8
22.2
35.1
36.5
40.8
39,2
40.0
$4.9
RSO
32
31,5
31.5
31.7
31.9
31.7
33.4
33.3
oC
I.I.
47.6
47.6
47.6
49.5
46.5
49.2
47.0
48.7
$1.6
$1.1
47.9
49.3
48.6
49.3
$2.6
$2.6
52.6
$1.0
LRL
15.5
15.5
15.5
15.7
15.5
15.3
16.0
15.3
16.0
16.1
15.4
15.3
15.2
15.6
17.1
16.8
16.8
42.2
pa
LRL
44.7
44.6
44.3
44,0
44.3
44.3
45.3
42.5
43.4
45.0
45.8
42.7
44.6
45.5
49.7
49,2
49,2
40.3
LRI.
33.7
33.3
33.2
32,8
33.5
32.9
33.9
32.0
32.9
33.9
34.7
32.5
33.8
34.2
36.5
36.3
36.3
59.1
I.R.
31.1
31.8
31.5
31.6
3
32.0
32.5
30.4
32.5
322
30.9
32.2
35.8
53.8
LRL
34,4
35.5
34.7
34,8
3399
34,5
35.7
33.2
34.0
34,9
35.1
33.1
34.9
35.4
44.1
42.5
43.5
57.2
23.9
24.3
22.4
34.9
72.6
LRI.
42.1
42.2
4
41.9
4
42.0
43.8
39.5
40.7
43.5
44.7
39.2
42.9
44.4
49.8
49.4
50.2
70.8
LRL
46.5
46.6
47.1
46.8
46.6
46.8
47.2
45.9
46.8
47.4
47.5
45.8
46.8
47.3
46.2
46.2
47.4
$4.7

---

lang
car.
B
T
T7G
Tsa
Tas
Teatitos
Car
Cp
Cur
Car T Gy
Gp
Gup
Gy
Bare
Tare
(CM
TiarG
NLLB
T.R..
43.3
42.
425
42.
43.1
42.1
44.4
40.6
41.4
44.0
45.0
41.0
41,4
44.7
48.1
47.9
48.1
6400
LRL
5.9
8.0
8.1
6.6
52
6.8
1.7
5.5
7.0
8.1
6.8
4.0
4.5
6.3
9.5
9.0
7.6
47.7
LR1
42.0
42.7
42.1
40.3
416
41.9
43.7
39.2
42.4
42.9
43.9
40.1
41.7
43.7
50.8
50.5
51,1
38.6
LRL
35.6
35.5
35.1
36,5
35.3
36.1
36
34.7
34,8
36.2
36.7
35.0
36.2
36.7
41.1
40.7
40.5
48.0
ppi
L.R.
25.2
25.2
25.2
25,2
25.2
25.0
25.2
25.2
25.2
25.1
25.0
24.6
25.0
25.0
25,3
25.3
25,2
56.6
33.2
37.6
35.9
36.1
37.4
35.3
31.
37.3
31.
32.2
32.9
33,9
41.3
40.7
Tum
LRL
52.1
32
51.6
32.0
92.3
32.1
31.4
51.0
31.0
30.1
31.3
90.8
30.8
31.9
32.4
$2.5
31.3
45.7
ng
LRI.
36.2
37.0
36.9
35,9
36,9
35,5
38.6
32,
35.3
39.2
38.1
32.8
37.3
38.6
47.9
47.7
47.9
58.3
wO
LRL
13.9
15.2
14.5
14.8
14.4
13.0
11.6 12.1
14.8
14.2
14.0
14.7
14.3
14.8
21.4
20.4
20.9
54.9
sh
LRL
47.6
47.1
47.2
47.3
47.6
47.3
48.1
46.3
46.4
47.9
48.8
46.0
47.8
48.1
52.2
51.6
52.1
48.8
64.5
64.1
64.5
64.2
64.2
64.2
6499
63.0
63.5
64.3
65.5
62.9
64.5
65.1
67.9
67.5
67.8
45.0
MRL
42.1
42.0
42.1
41.7
41.9
418
42.6
41.1
41.3
42.2
43.2
40.8
42.1
42.6
45.2
45.2
45.1
50.4
59.6
61.2
65.6
65.8
hn
MRI.
47.1
46.7
47.2
46.5
47.0
46.5
48.0
45.1
46.1
47.8
48.8
44.7
47.1
48.2
$1.9
$1.3
51.8
50.9
MRL
52.9
52.7
53.3
52.8
52.6
52.7
$3.9 51.3 52.4
53.4
54.8
51.4
53.1
54.4
58.4
57.8
58.5
49.2
MRL
00.7
60.4
60.8
60.4
604
60.5
61.4
59.4
60.3
61.2
62.0
59.1
60.8
61.6
64.8
64.7
65.3
52.5
ocb
MRL
58.6
58.6
58.7
58.6
98.4
58.1
59.5
57.6
57.9
58.7
59.4
57.5
58.9
59.0
61.2
60.5
60.9
50.6
ckh
31,9
35.0
34,9
33,5
81.8
38.9
28.9
37.1
31.9
30.3
31.2
43.2
44.7
44.6
61.2
MRL
52.3
51.7
52.4
518
$1.9
51.7
$2.9 50.0
$1.0
$2.5
53.9
49.8
52.2
53.2
58.2
57.8
58.3
$2.6
58.4
6018
60.7
61.D
gd
MRL
48.9
48.4
48.8
48.6
48.6
47.9
49.1
47.4
48.1
48.9
49.8
47.0
48.6
49.7
53.0
$2.6
52.9
57.3
MRL
56.8
56.6
57.0
56,9
56.6
56.5
57.3
56.0
56.3
57.0
57.7
55.8
57.0
57.5
60.0
59.7
60.1
63.2
43.6
42.7
43.5
42.8
43.2
42.4
44.3
44.0
44.3
41.7
44.4
43.5
49.1
48.3
49.0
56.0
hr
MRL
51.9
51.7
52.1
51.5
51.6
51.5
52.8
50.3
51.3
52.6
53.5
50.3
52.1
53.1
57.0
56.6
57.0
58.8
hy
47.7
47.9
47.9
47.3
47.
47.7
49.3
48.5
49.8
53.3
53.3
MRL
38.2
37.7
38.2
37.6
37.9
37,5
38.5
36.8
37.1
38.88
39.2
36.7
38.0
38.88
41.3
41.0
41.4
60.1
MRI.
43.8
43.6
44.2
43.4
43.6
43.7
44.7
41.8
43.0
44.5
45.7
41.7
43.8
45.3
51.0
50.4
51,1
41.8
ka
MRL
45.2
45.3
45.4
45.0
44,9
45.0
46.3
43.5
44.3
45.6
46.5
43.0
45.2
46.0
49.5
49.3
49.6
57.8
KK
MRL
49.6
49.5
49.7
49.5
494
49.4
51.0
46.9
49.1
50.3
51.6
47.0
49.9
51.1
55.6
55.2
55.5
49.7
km
38.3
38.0
M8.7
38,
38.2
37.6
39.5
36.6
39.0
40.2
36.8
38.8
43.0
43.0
43.0
lo
MRL
44.0
44.1
44.3
44,0
44.0
43.7
44.8
42.4
43.1
44.5
45.2
41.8
44.0
44.6
47.3
47.8
47.7
61.9
49.4
48.9
49.1
50.5
49.8
51.0
56.2
56.4
Iv
MRL
51.8
51.4
51.
51.6
51.
51.6
52.9
49.6
50.4
52.3
53.7
49.7
52.2
$3.0
58.77
$7.9
58.7
48.7
mg
MRL
44.1
44.3
44.3
44.3
44.8
43.9
45.2
42.7
43.3
45.2
45.9
42.5
44.9
45.5
49.3
48.8
49.0
43.5
m
MRL
40.6
40.5
40.3
40.5
40.7
40.2
41.0
40.2
40.6
40.9
41.1
39.8
40.9
40.6
41.8
41.7
41.8
60.1
mk
MRI.
57.4
57.2
57.1
57.1
57.1
57.2
58.4
58.5
56.7
57.9
58.8
55.4
57.5
58.5
61.8
61.5
61.7
51.1
48.3
47.6
47.8
47.7
47.9
47.6
49.5
49.0
50.3
55.2
55.4
mr
MRL
45.3
44.8
44.7
44,8
44,9
44.9
46
43.3
44.2
45.5
46.5
43.2
45.6
46.2
50.2
49.8
50.5
52.8
49.4
50.7
53.5
53.1
27.4
ny
MRI.
44.6
44.1
44.7
44.5
44.3
44.5
45.3
43.1
43.8
44.8
45.8
43.2
44.6
45.3
48.0
47.7
48.1
50.1
MRL
40.6
39.5
39.2
39.2
39,3
39,5
41.
37.4
38.5
40.5
41.6
37.4
40.1
41.4
47.3
46.1
47.1
61.4
sd
MRL
44.0
43.5
43,3
43,2
43.7
43.1
43,9
42.8
42.6
43.6
44.4
42.5
43.6
44.0
46.1
46.1
46.2
54.2
MRL
50.7
49.9
50.6
50,1
50.1
50.3
51.5
48.8
49.7
51.0
52.
48.4
50.7
51.7
56.1
56.1
56.6
47.1
sm
49.2
48.7
48.9
48.8
49.4
49.7
47.5
$2.0
51.6
51.2
MRL
43.4
43.3
43.
45.3
43.
43.2
440
42,3
42.9
43.5
44.2
42.0
43.5
43.8
46.4
46.1
46.
56.0
42.5
b
MRL
54.1
54.1
53.9
54.0
53.8
53.9
54.9
52.8
53.5
54.8
55.4
52.6
54.3
$5.0
58.1
57.9
58.2
47.4
MRL
31.2
34.8
31.7
34,8
34,4
36.8
35.8
33.1
38.9
39.4
31.5
36.6
38.2
41,9
43.7
42.5
62.3
44.6
44.5
44.3
44.6
44.4
44.3
44.9
43.5
44.7
45.5
43.4
44.8
45.2
47.2
47.1
47.1
58.8
sw
MRL
58.3
58.1
58.2
58.1
58.0
58.1
59.2
56.6
57.4
58.6
59.8
56.3
58.4
59.5
62.4
62.2
62.2
$3.5
52.5
52.4
53.5
58.5
5H.5
te
MRI. 52.6
52.5
52.2
52.2
52.5
51.9
53.8
49.9
$1.6
$3.5
54.4
49.9
52.8
$3.8
58.4
58.1
58.1
53.6
MRI. 34,7
34.9
34.4
34,4
34.2
38.8
38.
33.1
36.0
37.6
37.0
33.3 35.9
37.6
39,9
40.2
42.0
59.9
ur
MRL
46.2
46.0
45.9
45.5
45.5
45.6
46.5
44.4
45.3
46.2
47.0
44.5
46.4
46.9
49.2
48.9
49.3
62.1
MRL
51.7
51.4
51.3
$1,2
51.
51.3
52.4
49.8
50.5
51.8
53.5
49.8
51.8
$2.6
56.2
55.5
55.9
45.9
yi
36.6
3600
36.4
36,5
36.6
366
36.2
36.2
36.5
36.5
36.2
36.4
37.2
36,9
37.1
58.1
MRL
20.9
20.9
20.7
21.1
20.9
20.7
21.0
20.8
20.9
21.0
21.0
20.5
20.9
20,9
21.3
21.1
21.4
57.9
yue
12.5
13.5
17.88
zu
MRL
49.7
49.1
49.4
49.6
49.3
49.4
50.7
47.7
48.5
50.1
51.3
47.5
49.7
50.9
55.2
54.4
$4.9
67.1
HRI.
47.7
47.4
48.0
47,4
47.4
47.1
49.0
45.1
46.2
48.4
50.2
44.6
47.9
49.1
56.1
55.2
56.0
59.4
2s
HRL
50.6
50.1
50.6
50.1
50.
49.9
51.4
48.3
49.5
$1.0
52.1
48.2
50.6
$1.7
56.4
55.9
56.7
53.5
da
HRI.
62.8
62.2
62.7
62.4
62.4
62.3
63.2
61.2
61.7
62.8
64.00
60.8
62.7
63.4
67.3
66.8
67.3
63.6
56.2
55,5
56.2
57.5
61.4
60.9
el
HRL
45.1
44.8
45.5
45.0
44.9
44.8
46.4 43.1 44.1
45.9
46.9
43.0
45.5
46.3
50.9
50.4
$1.2
54.3
HRI.
51.6
$1.4
$1.7
51.5
514
51.4
51.9 50.7
$1.8
52.3
50.5
51.6
$2.1
54.5
54.2
54,6
61.6
HRRL
46.0
45.2
46.1
45.0
45.3
45.0
46.7 44.0
44.6
46.0
47.0
43.5
45.9
47.1
50.7
49.9
50.7
59.1
HRL
48.5
48.2
48.6
480
48.1
48.0
49.6 46.4
47.5
48.9
50.3
46.4
48.5
50.0
55.3
$4.6
55.5
68.5
HRL
63.3
62.7
63.4
63,0
62.8
62.7
64.0 61.7 62.3
63.5
64.7
61.5
63.2
64.1
68.6
68.3
68.7
59.3
hi
HRI.
54.4
53.9
54.3
53,8
53.8
53.9
54.8 52.4 $3.0
54.5
55.3
52.5
54.4
$5.1
58.2
$7.8
58.4
68.4
47.5
47.2
47.9
47.2
47.2
47.2
48.4
49.8
54,9
58.7
HRI.
63.8
63.7
64.2
658
63
64.0
64.7 62.5
63.3
64.4
65.3
62.6
64.2
65.0
68.4
68.0
68.2
51.4
HRL
52.8
52.7
$3.1
$2.6
52.5
52.6
53.4 51.7
52.2
53.2
53.8
51.5
52.8
53.7
57.0
56.6
57.2
61.2
iw
HRL
46.3
45.4
46.7
45.9
45.7
46.0
48.0
43.6
44.9
47.1
48.9
43.3
46.8
47.9
55.2
54.2
55.4
59.6
HRI.
29.3
29.0
29.4
28.7
28.9
29.0
30.
26.9
28.2
29.6
31.0
27.1
29.6
30.8
36.4
35.7
36.9
59.8
HRI.
26.6
264
26.9
26.5
26.5
26.3
27.6
24.3
25.7
27.6
24.4
27.2
28.3
33.4
32.3
33.4
60.7
ms
HRRL
64.1
63.9
64.1
63.8
63.9
63.9
64.7 62.4
63.2
64.2
65.4
62.5
64.1
64.9
67.7
67.4
67.6
70.6
52.5
52.4
$2.8
53.5
52.6
55.8
56.2
no
HRI.
57.6
57.2
$7.5
57.5
576
57.5
57.9 56.3
56.6
$7.7
58.7
56.1
57.6
58.2
60.8
60.6
60.8
38.0
HIRL
43.8
43.3
43.4
4
43.5
43.5
44. 42,0
42.7
44,1
44.8
42.0
43.5
44.7
48.4
48.3
49.0
61.
Pt
HRL
64.0
63.9
64.0
64.0
63.6
63.7
64.7 62.6
63.4
64.2
65.
62.6
64.4
65.1
68.4
67.9
68.4
52.1
HRI.
57.0
57.2
56.9
56.9
56.7
56.8
58.0
55.2 56.4
$7.6
58.8
55.3
57.5
58.1
62.6
62.4
62.7
29.3
HRI.
49.5
49.2
49.3
49.3
49.0
48.9
50.0 47.7 48.4
49.8
51.1
47.5
49.7
55.3
$4.9
69.6
sk
IRRL
51.6
51.3
51
51.2
51.
51.0
52.5 49.7 50.4
$2.1
53.6
49.3
51.7
52.9
58.6
58.2
58.7
35.0
HRL.
62.0
618
62.4
63.4
67.2
66.7
67.0
HRI.
44.2
44.0
43.4
43.5
43.4
44.0
44.7
41.6
42.8
45.0
45.6
41.
44.1
45.6
50.1
50.0
50.2
$7.8
HRL
53,4
52.8
52.7
52.9
52.8
53.0
54.3
51.0
52.0
53.6
55.1
50.9
53.2
54.5
59.6
59.2
59.8
59.0

---

lang
car.
B
T
T7G
Tsa
Tas
Teatitus
Car
Cp
Cиr
Car
Gy
Gp
Gup
Gu T Bare
Tare
(Cy
TlarG
NLLB
HRI.
49.5
49.2
49.3
49.1
49.4
49.0
50.5
41.4
48.6
49.9
51.2
47.5
49.9
50.6
55A4
55.0
55.1
65.2
vi
IIRL
49.9
50.0
50.0
49.7
49.9
49.8
50.7
47.9
48.8
51.0
$1.8
47.9
50.4
51.2
55,9
55.1
56.1
71.4
IRRL
22.3
22.1
22.1
22.2
22.0
22.0
23.0
19,9
21.2
22.7
24.1
20.2
22.6
23.4
29.6
29.3
29.9
56.0
ar-Latn
00M
3.0
3.6
3.5
2.9
2.3
2.2
2.9
3.8
2.9
3.0
2.8
3.6
1.6
2.1
kr
00M
11.1
6.3
5.3
4.3
4.8
17.5
17.4
4.2
15.9
6.7
17.0
12.8
17.9
3.9
15.3
6.0
6.7
31.5
13.1
15,5
14.8
13.6
10.5
12.3
taq
00M
9.8
16.4
15.1
18.0
18.2
18.0
18.9
14.8
13.6
16.0
17.6
12.6
18.5
14.9
16.0
18.0
16.3
25.9
00M
11.7
20.4
19,8
22.3
21.8
16.0
21.9
18.8
20.7
20.2
21.7
16.5
23.4
18.5
19.4
22.4
21,1
56.1
luo
00M
2
15.2
6.5
14,8
6.8
7.9
10.3
3.9
7.2
10.9
13.7
3.6
14.8
6.6
11.4
16.2
19.8
41.7
Imo
0OM
11.6
19.6
18.4
21.6
21.1
21.0
22.5
17.9
19.6
20.1
21.5
15.8
22.4
18.1
18.8
21.4
20.5
38.7
0OM
11,3
20.7
19,2
23,0
22.7
20.9
22.4
19.3
20.5
20.6
22.4
16.3
23.9
18.7
17.5
22,9
19.2
51.6
fur
OOM
11.8 20.3
19.3
22.5
22.1
20.7
22.0
18.2
20.5
20.1
22.5
16.0
23.8
18.8
19.5
22.2
18.7
58.7
1699
16.4
13.9
16.7
56.7
lij
0OM
11.1 20.7
19.1
22.5
22.1
22.7
22.3
18.6
22.6
20.1
22.6
162
23.7
18.9
20.3
22.3
18.6
56.1
bjn
OOM
15.1
14.2
16.7
16.4
16.4
16.9
13.6
14.3
14.9
16.9
11.9
17.2
13.9
14.7
16.7
14.7
518
00M
11.1
19.8
18.5
22.0
2
21.7
21,2
17.6
20.9
19.3
22.0
16.4
22.7
18.8
19.3
21.7
19.4
58.1
00M 888
15.4
13.7
16.5
16.2
16.6
15.6
14,2
16.0
15.3
16.8
11.1
17.8
13.5
15.5
17.1
10.5
49.2
130
15.8
15.5
14.2
149
13.5
16.2
10.7
16.9
10.3
16.5
42.3
lua
0OM 9.1
15.1
14.2
16.6
16.8
17.3
16.2
13.6
15.3
14.9
16.5
12.9
17.1
13.7
15.2
16.3
14.4
39.4
15.1
13.2
4.2
6.5
10.0
10.6
min Arah
00M 0.7
1.0
0.3
12
0.8
4.4
1.4
0.6
0.9
3.5
4.8
0.4
6.3
L.0
0.3
0.3
ace-Arab
00M
0.3
0.9
1.3
1.2
3.0
1.0
3.7
2,0
0.8
6,5
1,9
0.4
21.1
mas
00M
14.7
12.3
10.2
9.5
11.5
4.1
69
100
10.7
138
6.7
11.1
15.6
17.7
26.4
ast
0OM
11.9
21.2
19.8
23.1
23.7
24.0
22.4
18.5
21.3
20.5
24.6
16.7
24.2
19.6
20.2
23.0
20.9
$9.6
ans
13.3
13.7
12.9
17.5
2.1
19.6
11.7
kea
00M
3.3
18.8
7.1
15,5
8.6
10.6
13.5
4.6
888
13.1
12.4
4.1
17.4
9.3
14.1
18.2
21.2
44.9
toq-
00M
5.5
5.2
3,9
5.3
3.7
5.2
5.9
4.7
6.6
6.3
19.3
bjn-Arab
00M
5.5
2.1
4.7
4.7
12.4
9.6
11.9
9.5
9.8
12.6
5.4
12.7
9.6
0.4
4.1
8.0
20.4
lang
cat.
B
T
Tro
T
Tap
Tetarstos
Car
Cp
CUP
CAr T
GM
Gp
GUr
GM
BayG
Tazo
(CM Thane
NLLB
13.7
12.0
7.6
11.0
8.9
109
12.5
11.0
9.6
16.3
31.2
Latn
00M
8.8
14.8
14.0
16.1
15.8
16.1
16.3
13.3
15.0
14.4
16.0
11.6
16.7
13.6
14.0
16.1
13.5
$1.1
kr-Arab
00M
244
0.5
1.0
0.9
8.7
6.9
8.3
7.1
8.9
4.4
8.9
2.6
1.0
1.7
12.7
zh-Hont
1.3
9.6
9.1
1.3
7.9
6.9
8.0
13.9
13.9
10.4
17.6
sip
0OM 0.4
13.1
14.2
14.3
13.9
6.2
19.1
3.4
17.5
12.9
17.6
8.6
13.3
9.8
1.1
5.6
19.5
56.0
ks-Dova
35
3.7
kab
00M
1.5
4.4
2.7
8.0
1.6
4.7
2.3
2.1
1.7
2.3
1.8
1.5
3.7
0.9
4.7
3.4
4.5
37.9
aoth
00M
146
233
5.4
0.4
0.5
0.8
04
0.2
0.4
27.8
Table 24: Full Results of CHRF on FLORES for the models that we trained, in the en-*xX direction. See Table 23 to demystify the model abbreviations.
lang
CoL.
B
T
T75
Tao
T25
TeAT
Csr
Cy
Carr
Car
Gar
Gp
GMr
Gy T
Baze
Taze
(C2z
TaiG NLLB
mean
47.2
47.
47.1
468
46.4
47.4
46.0
414
47.3
47.9
45.9
47.3
52.9
59.9
IRL
meam
57.2
56.9
56.9
56.8
56.8
56.5
57.7
55.6
56.5
57.5
58.4
55.5
57.2
58.1
62.6
62.0
62.6
65.7
518
50.6
50.5
52.1
5%.1
63.3
LRL.
mean
43.6
43.
43.5
43.0
43.5
42.8
43.8
42.4
42.8
43.9
44.3
42.4
43.6
44.3
50.2
50.0
49.7
60.4
URL
37.0
37,5
37.2
36.7
36.6
36.4
36,9
36.7
36.2
36.9
37.0
3644
37,5
37.1
43.4
43.3
42.3
49.2
0OM
mean
32.5
52.5
32.5
81.8
32.1
31.5
33.0
31,5
31.4
32.7
330
31.4
52.8
32.8
37.9
37.5
37.3
URL
39.0
39.4
39.1
38.6
38.6
38.3
38.9
38.4
38.2
39.0
39.2
38.2
39.3
39.3
45.4
45.3
44.5
52.6
000
URL
29.6
30.0
30.2
29.0
29.0
29.0
29.1
30.8
29.3
28.8
27.8
302
31.4
29.4
36.2
35.1
34,1
41.2
URI.
$1.2
50.7
50.9
50.5
50.5
$1.5
49.0
50.0
51.5
52.2
48.9
50.8
$2.2
58.1
$7.7
57.4
61.4
acg
URL
52.5
51.8
52.2
51.9
52.1
51.7
$2.9
50.5
51.2
52.8
$4.0
50.3
52.5
$3.6
59.7
58.8
58.9
32,9
acb
URL
46.1
45.9
46.0
45.7
45.8
45.4
46.6
44,4
45.1
46.4
47.2
44.4
46.1
47.2
53.2
$2.3
$2.5
73.7
ape
50.8
50.6
50.9
50.4
50.5
50.4
$1.4
49.2
50.0
49.0
51.0
59.5
62.2
ar-MA
URI.
41.6
41.6
41.6
41.2
41.5
41.0
41.9
40.1
41.3
42.1
42.6
40.1
42.1
42.7
49.0
48.2
47.7
35.8
48.7
48.5
48.8
49.2
48.2
48.9
49,9
47.0
64.5
owa
URL
54.3
538
55.8
53.
53.8
53.5
53.7
52.6
53.2
53.9
55.2
52.7
$4.1
$4.9
641.5
60.5
60.3
35.4
URL
23.7
23.1
22.3
21.9
22.5
17.5
21.4
22.3
21.1
20.5
22.0
23.4
22.5
20.9
34.3
32.4
28.3
43.0
bu
URL
22.1
25.7
24.9
23.8
22.8
23.9
23.4
22.2
23.3
25.4
21.6
22.6
22.6
23.4
24.2
29.7
26.2
53.5
bug
URI
24.8
25.5
25.5
24.4
25.0
24.5
26.1
25.4
24.5
25.1
26.2
24.4
26.6
25.2
29.3
29.0
27.8
35.7
gjk
20.2
20.7
20.6
19.6
19.8
20.5
19.9
20.3
20.9
20.1
19.8
20.6
51.3
din
URL
21.8
21.3
21.7
20.2
20.1
20.3
21.1
20.9
20.1
20.7
20.5
21.2
20.8
20.5
22.3
22.3
22.1
30.0
dyu
18.2
19.4
17.7
19.2
17.4
17.6
18.5
67.4
fon
URI. 19.7
21.5
20.3
19.4
19.0
19.4
19.8
20.9
19.3
19.5
19.8
20.6
21.2
19.8
22..3
23.1
2016
36.6
hne
URL
55.4
54.4
55.0
54.6
55.2
54.8
$4.3
54.5
53.7
54.7
56.0
51.
56.0
56.4
64.0
63.9
63.9
44.9
URL
44.7
46.6
46.0
45.8
44.5
45.2
44.7
45.1
44.1
45.7
43.5
44.4
44,4
45.8
55.2
55.1
55.6
42.2
kac
URI
18.7
19.1
17.5
17.7
17.3
18.4
17.8
18.6
17.4
18.9
18.3
20.2
19.4
18.0
21.8
20.7
18.9
66.7
Abp
23.5
25.0
23,4
22.8
22.9
22.9
24,9
23.4
25.0
23.6
62.3
kmb
URL
20.5
20.5
20.2
18.8
19.7
19.0
20.4
20.0
20.1
20.4
20.3
20.4
20.7
20.3
23.6
23.4
23.2
53.9
33.7
lus
URL
20.3
22.9
22,9
19.9
20.9
23.4
22.7
21.7
22.2
25.2
21.5
19.8
23.2
21.4
23.5
23.7
22.6
73.8
mag
URL
56.6
55.2
55.5
56.0
55.7
55.3
$6.0
55.2
54.7
55.9
$7.5
54.9
57.0
$7.6
63.8
63.3
63.5
35.4
mai
54.8
54.4
54.7
54.7
54.5
54.5
54.6
54.0
53.7
54.9
56.8
53.8
55.7
56.1
63.2
62.0
44.
min
URL
39.8
40.8
41.0
40.8
40.9
40.2
39.6
41.8
40.3
38.9
38.2
42.0
42.4
39.3
48.2
49,2
47.5
43.7
mni
32.8
32
31.8
28.6
27.7
28.4
29.7
29.8
36.7
mus
URI.
18.0
19.0
18.2
17.8
17.3
17.4
17.9
18.2
17.2
16.4
17.3
18.0
18.1
16.5
20.1
20.4
17.1
32.7
pag
URI.
38.2 39.8
39.0
39.5
37.3
37.2
37.7
37.5
37.5
38.5
38.5
39.7
38.2
64.0
pap
URL
57.3
56.9
56.9
56.0
56.2
55.4
57.3
54.7
55.9
56.2
56.3
55.3
56.8
$6.5
68.0
66.1
66.3
51.5
quy
URL
27.8
29,3
28.3
28.
27.7
28.8
28.6
28.5
78.7
29.6
2688
28.2
27.9
28.7
33.3
33.2
33.5
52.3
scn
51.2
51.6
514
50.9
51.2
50.5
51.6
50.3
50.2
51.
$1.8
50.3
51.3
59.2
60.7
URL
21.4 23.3
22.0
21.4
21.6
21.0
21.6
22,4
21.8
22.3
22.3
23.6
22,8
21.8
27.3
25.6
24.8
65.3

---

lang
car.
B
T7a
Tao
TaG
TOAT
Car
Cp
Cure
Car
Gay
Gp
GMF
Gy
Baze
Tare
(Cy TiaiG
NLLB
shn
URI
13.1
14.4
324
12.
31.
29.7
10.6
52.7
32.7
30.2
31.6
121
33.6
12.8
42.7
40.7
33.0
48.4
vec
URL
53.4
54.6
54,0
53.9
54.5
53.9
54.1
52.9
$2.9
53.5
54.1
52.5
53.8
51.8
63.0
62.7
61.4
42.0
URL
58.1
58.9
58.5
58.6
58.6
58.7
57.3
57.6
56.8
57.0
58.1
57.
58.6
57.3
66.0
65.8
65.2
42.9
am
LRL
46,8
46.5
46.9
462
47.1
46.1
47.4
45.2
460
47.0
48.8
44.7
47.3
48.5
54.7
5388
54.2
71.1
LRI.
41.1
41.0
38.5
37.
39.7
37.0
38.8
36.1
36.2
41.9
41.8
37.4
39.3
41.2
49.5
47.3
46.3
45.4
22.2
23.8
24.3
23.0
22-4
23.5
22,9
24.6
21.8
22.5
23.1
70.2
ba
LRI.
37.7
56.0
$2.9
35.
55.9
31.0
$7.0
$4.1
33.5
40.2
39.4
55.6
55.2
41A
39.8
42.1
52.1
ban
LRL
44.9
44.8
45.1
43.6
44,1
44.0
44.3
45.1
43.5
43.6
44.3
449
45,2
45.7
48.9
49.9
48.5
54.9
be
LRL
47.9
47.6
47.6
47.3
47.
47.1
48.2
46.5
47.2
48.0
48.8
46.
47.7
48.5
52.2
51.8
52.3
48.2
bio
LRL
47.1
46.7
46.9
46.6
47.1
46.8
47.1
46.5
46.4
47.1
48.2
46.3
47.8
48.2
54.5
54,1
56,1
53.4
LRI.
13.7
15.2
12
144
13.7
14.0
16.2
13.0
160
17.2
162
14.5
15.6
22.2
20.7
19.3
63.6
LRL
68.5
68.2
68.5
67.7
68.1
67.5
69.2
66.2
67.3
69.0
70.0
65.8
68.5
19.6
75.3
74.9
75.6
44.0
21.4
22.2
21.3
20,3
21.6
2000
20.5
29.9
63.0
DO
LRI.
27.4
28.9
28,8
27.6
26.8
28.5
27.4
28.4
27.7
28.7
26.7
29.3
27.6
28.8
28.0
30.0
26.7
$2.0
LRL
61.7
61.3
614
612
61.3
61.1
62.0
60.4
61.1
61.9
62.4
60.3
61.6
62.4
65.8
65.2
66.0
52.4
eu
LRL
50.7
50.2
50,2
50.1
50.1
49.7
51.1
48.6
49.9
50.7
51.8
48.8
50.5
51.7
56.9
56.3
56.9
68.0
O-AF
LRL
54.6
54.1
54.3
54.0
54.4
53.8
55.0
52,9
53.4
54.7
55.9
53.1 54.6
55.7
61.2
60.2
6019
61.7
LRI.
20.5
21,9
21.6
20.9
20.4
20.7
21.3
20.5
21.5
22.0
20.4
20,9
21.0
20.9
22.8
23.8
61.2
5
LRL
32.5
316
31.3
29.7
312
29.4
29.7
32.8
28.9
28.8
31.7
32.4
32.1
29.3
36.5
36.7
32.5
54.6
LRI.
50.6
$1.3
$1.4
ga
LRI.
57.0
56.6
56.3
56.3
56.2
56.2
57.5
55.0
56.1
57.5
58.5
54.8 57.1
58.0
65.2
64.4
65.4
61.8
LRL
316
34.3
33.8
32.6
32,5
32.8
32.4
31.2 31.4
34.4
30.6
31.6 31.9
33.6
42.2
42.4
42.3
$2,9
LRI.
566
560
56.1
58.9
56.3
55.8
57.1
54.7
55.6
56.8
57.8
54.7
56.6
57.3
62.3
60.1
h
LRL
55.3
54.8
55.0
55.1
55.0
54.4
55.7
54.0
54.5
55.5
56.5
54.0
55.2
$6.1
61.3
60.7
61.2
63.6
22
51.5
52.0
41.4
51.6
52.5
51.1
51.1
50.8
52.3
54.6
LRL
30.9
318
31.6
31.4
31.
31.0
30.4
32.6
30.9
30.5
30.5
32.5 31,3
31.3
34.2
35.1
31.1
77.2
LRI.
53.1
52.7
52.7
52.7
52.9
52.4
53.6
51.4
52.0
53.1
54.4
51.2
53.2
$3.9
59.0
58.4
59.2
54.6
ks
LRL
316
31.5
32,6
30.5
31.6
30.0
32.2
31.1
30.9
32.4
33.7
32.0
33.2
34.1
43.0
43.2
40.1
43.7
LRL
43.5
43.1
434
42.8
42.9
42.3
43.4
42.2
42.4
43.6
44.3
42.1
43,5
44.1
51.2
49.8
49.9
65.0
LRI.
4
45.0
44.9
44.9
45.2
44.3
45.8
43.6
44.6
45.7
46.5
43.6
45.5
50.2
64.0
lb
LRL
59.8
59.2
59.3
59.0
59.1
59.0
60.3
58.2 58.6
60.1
61.1
58.1
59,9
60.6
66.5
66.2
66.6
65.4
lg
29.6
310
30,5
29.5
29.2
29.9
29,9
30.2 30.0
29,9
29.3
73.4
In
LRI
36.5
57.5
37.3
56
56.3
36.7
36.6 55.6
36.4
37.7
35.7
35.5
$6.0
37.9
44.9
44.4
44.3
59.5
LRI.
478
47.5
47.4
47.1
47.2
46.9
48.2 46.0
46.8
47.8
49.2
45.9
48.0
48.9
54.3
54,1
54.3
62.7
ml
LRL
69.7
69.1
69.2
69.2
69.3
68.9
70.2 68.0 68.6
69.8
70.9
67.8 69.5
70.6
75.5
75.0
75.4
68.1
LRI.
44.7
43.9
44.0
43.7
44,5
43.0
45.5
42.4 44.0
45.0
45.6
42.6
44.3
45.8
50LS
49.9
5114
66.7
nSO
47.8
47.9
47.9
47.2
47.1
47.6 46.2 46.9
48.4
46.5 47.8
$2.5
LRL
85
65.
65.6
65.
65.3
65.2
66.6
64.6
66.0
66.5
66.3
64.6 66.0
66.4
71.2
70.4
70.3
67.9
om
30.3
30.5
30,3
29.7
29.4
28.4
29.8
61.8
pa
LRI.
55.6
55.1
55.5
5
55.4
54,9
56.2
54.0
54.9
56.3
57.2
53.8
56.1
56.9
63.6
62.5
GR.2
59.8
LRL
47.8
46.8
47.2
46.8
46.9
46.7
47.8 46.3
46.5
47.8
48.5
45.8
47.7
48.1
54.0
53.2
52.8
71.2
m
LRL
36.0
35.5
3
35.3
35.7
34.9
34.6 34.7
34.6
34.7
36.1
35.1 35.5
35.8
44.4
43.1
43.1
57.4
LRI.
41.1
40.4
40.1
39.4
40.2
39.3
39.5
59.5
39.2
39.3
40.8
39.6
40.6
41.0
49.9
48.7
49.0
65.2
39.3
40.0
39.8
39.8
39.8
40.6
39.5
40.1
40,2
39.1 39.7
77.1
LRI.
50.7
50.4
50.0
50.3
50.2
49.9
51.6 48.6
49.8
51.1
52.7
48.4
50.7
52.2
58.5
57.2
58.3
77.6
51.8
61.1
g
LRI.
50.
50.1
49.6
49.8
49.9
49.2
50.6 48.2
49.4
50.5
$1.7
48.2
50.3
51.3
57.8
56.8
57.4
70.2
LRL
302
31,3
30,7
27.5
33.2
28.5
31,5
26.0
29.4
31.1
32.3
24.5
30.4
31.3
42.1
40.5
40.1
56.3
LRI.
506
50.0
50,1
50.
49.8
49.3
$1.0 48.0
49.2
52.2
48.4
50.6
51.5
59.0
58.6
40.9
LRL
43.9
43.8
43.6
43.5
43.9
43.1
43.6
42.4
42.6
43.4
44.6
42.4
43.8
44.2
50.0
49.7
49.6
63.7
43.0
41.6
43.9
43.4
68.4
t
LRI.
35.7
38.8
38.4
37.6
37.8
38.7
35.7
36.3
36.4
37.2
360
37.0
36.4
37.2
41.1
44.6
41.2
58.4
LRI.
32.1
34.1
33.1
31.9
32.7
32.7
31.8 33.2 32.5
32.3
32.7
33.2
32.6
32.8
34.7
34.4
34.3
55.4
ng
LRL
43.2
43.2
43.2
42.8
43.0
42.3
43.9
41.7 42.6
43.6
44.5
41.4
43.2
44.5
49.9
49.1
49.3
63.1
LRL
31.3
31.6
31.0
31.0
31.7
31.1
29.6
28.8
29.7
29.0
29.5
29.6
30,3
29.3
40.8
403.3
57.7
61.2
49.4
48.9
48.9
48.
49.0
48.6
49.5 47.8
48.2
48.9
50.1
47.7
49.4
49.4
55.8
54.9
$1.9
af
MRL
70.4
69.9
70.2
69.9
69.9
69.7
70.7
69.2 69.6
70.4
71.7
68.8
70.4
71.2
75,1
74.5
74.9
60.6
480
47.9
46.5
48.2
29.2
46.4
66.2
bg
MRL
61.4
60.8
61.0
61.
61.2
60.8
61.8 59.8
60.5
61.6
62.3
59.8
61.3
62.1
66.3
65.9
66.5
68.4
MRL 53.2
52.7
53,1
52.6
52.7
52.2
54.2
51.3 52.5
53.3
54.5
$1.1 53.3
54.3
59.8
59.4
60.0
58.3
bs
MRL 60.8
60.5
60.6
60.5
60.3
60.4
61.5 59.6
60.2
61.2
62.0
59.3
60.9
61.7
66.0
65.6
65.8
62.9
MRL 64.3
63.9
64.2
64.0
639
63.8
64.7
63.2 63.8
64.6
65.2
63.1
64.4
65.1
688.6
68.1
68.5
61.6
ceb
MRL 58.5
58.2
58.2
58.0
58.3
18.1 57.8
59.3
57.2 58.4
58.3
ckb
MRL 42.8
44.0
43.8
43.2
42,3
43.3
43.4 41.9 42.0
44.2
43.2
41.3
42.2
44.5
55.1
$4.0
54.2
67.4
MRI. 56.7
56.0
56.0
560
55.6
$7.0 54.9
55.3
56.6
$7.8
54.6
56.5
$7.3
63.1
62.4
63.0
59.5
GI MRL 60.9
60.8
60.8
50.6
60.5
60.3
61.6 59.6
59.9
61.1
62.4
59.5
61.0
61.9
67.5
66.8
67.3
61.6
MRL 518
51.2
50,9
50.6
5 30.7
52.0 50.0
51.1
51.8
52.8
49.7
51,3
52.2
57.8
57.5
58.1
64.9
gl MRL 61.5
61.4
61.3
2
61.3
61.2
61.8
60.5
61.1
61.8
62.2
60.5
61.6
62.3
65.1
64.7
65.0
68.7
ha
MRL 41.2
40.7
40.
89.8
41.6
39.9
41.4
40.7
39.8
40.5
41.5
40.4
41.9
41.5
45.8
45.6
45.6
61.9
MRL 57.8
57.4
57.7
57.4
57.6
58.2
56.7
57.2
58.1
58.9
56.5
58.0
62.7
by
MRL 54.6
54.6
54.7
54.5
54.3
53.8
55.7
55.1
54.3
55.7
55.9
52.8 54.7
55.8
61.2
60.7
61.7
55.9
MRL 38.6
38.4
38.6
37.2
38.4
37.6
38.6
37.8
37.8
38.6
39.7
37.6
38.9
38.8
43.9
43.3
43.1
67.3
IS
MRL
51.4
51.0
50.9
50.9
50.8
50.7
$2.0
49.5 50.3
51.7
$2.7
49.4 51.2
52.1
58.2
57.5
58.4
72.4
MRI.
49.6
49.6
49.6
49.5
49.5
49.0
50.6
48.0
49.4
50.5
$1.0
48.1
49.6
50.7
$5.1
54.8
55.6
62.0
MRL 52.5
52.1
52.1
52.4
52.0
51.5
$3.2
50.7
51.6
52.8
$3.8
50.6 52.6
59.1
59.1
68.8
km
MRL
47.7
47.3
47.5
46.5
48.1
45,5
48.3
45.9
47.3
47.9
49.0
462
47.0
48.3
52.2
52.1
52.9
77.2
52.1
51.6
52.1
81.8
52.5
50.7
51.6
53.2
51.0 $2.0
59.2
5H.4
68.9
Ir
MRI.
$2.9
52.8
52.6
52.7
52.5
52.2
$3.5
51.3
52.1
53.1
$4.5
51.2
53.0
53.8
58.6
58.3
58.7
65.8
Iv
MRL
56.6
56.2
56.2
56.2
56.2
55.9
57.1
55,1
55.6
56.6
57.7
55.0
56.6
57.5
62.5
62.1
62.4
$2.9
mg
MRL 42.3
41.5
41.8
40.4
41.3
40.9
41.8
41.0
409
41.2
429
40.6
42.0
42.4
46.8
46.9
46.7
57.8
MRL
41.7
41.7
41.7
41.4
41.3
41.0
42.5
40.2
41.8
42.6
42.7
40.3
41.8
43.0
48.2
47.5
48.3
64.4
mk
MRL 61.3
60.9
60.8
60.7
60.9
60.6
61.5
59.6
60.5
62.1
59.6
61.3
66.1
66.1
6488
ml
MRL 53.6
53.2 53.0
53.0
53.3
52.3
$4.1
51.4
52.8
54.1
$49
51.3
54,0
$1.5
60.6
59.5
60.7
62.8
53.5
54,3
52,0
52.7
549
59.3
ne
MRL 57.2
56.6
56.6
56.8
56.9
56.5
57.4
55.4
56.1
57.3
58.8
55.3
57.4
58.2
63.7
62.9
63.6
47.9
MRL
41.1 41.1
40.6
40.5
41.1
40.2
41.1
40.5
40.2
40.7
41.8
404
41.5
41.3
45.2
45.4
44.9
67.9

---

lang
car-
B
T75
Tao
TaG
TOAT
Car
Cp
Care
Car
Guy
Gp
GMPE
Gy
Bare
Tere
(CM TlarG
NLLB
MRI.
50.3
49.2
49.3
48.7
50.5
48.2
49.5
5000
520
47.9
50.2
51.2
(4..
59.1
68.8
MRL
53.1
52.3
52.7
52.3
52.3
52.1
53.3
51.7
52.1
53.2
54.1
51.2
53.3
60.0
59.1
59.3
59.1
MRL
56.0
55.8
55.7
55.6
58.6
55.2
56.2
54.5
55.1
56.0
57.0
54.5
55.9
56.6
61.1
60.7
61.1
56.5
SI
MRL
46.0
46.5
46.6
45.5
46.5
454
46.1
45.3
45.7
45.8
47.3
45.3
46.3
51.6
51.6
49.6
64.2
MRL
42.3
42.0
42.0
41.7
42.0
41.2
42.1
41.5
41.4
42.0
43.1
41.
42.2
42.4
47.3
46.7
46.4
58.7
MRL
999
59.6
59.5
4944
59.7
99.2
60.3
58.5
99.0
59.8
61.0
580
59.9
50.5
65.3
64.8
65.2
34.4
MRL
60.2
59.9
99.7
60.1
99,8
60.7
58.6
99.1
60.5
61.6
58.3
60.3
66.3
66.1
66.3
71.3
M
MRL
50.3
49.7
49.9
49.6
49.7
49.3
50.2
48.7
48.9
49.9
51.2
48.7
49.9
50.8
58.1
57.1
57.4
63.0
MRL
56.3
55.8
55,9
55.6
55.9
55.5
56.6
54.4
54.8
56.0
57.7
54.3
56.1
63.2
62.2
62.6
63.4
51.
50.6
50.9
50.4
50.7
49.0
50.2
52.3
49.2
66.8
MRL
56.6
55.1
55.7
55.6
55.6
55.2
56.7
53.6
55.0
56.1
57.6
$1.9
56.1
$7.3
62.8
61.9
62.7
60.9
OF
MRI
53.0
52.5
52.7
52.4
52.8
52.3
53.7
31.5 52.2
53.4
54.6
51.4
53.2
54.2
59.5
58.7
59.5
69.4
MRL
53.3
52.7
52.9
52.6
52.8
53.9
51.4
$2.3
53.2
54.7
51.3
53.4
54.2
39.6
59.4
59.6
54.1
yI
MRL
48.2
47.5
49.0
45.9
46.7
47.9
48.7
45.3
45.9
48.7
48.0
45.2
46.3
48.4
52.1
$3.9
51.6
61.2
MRL
35.4
35.5
35.4
35.0
35.2
34,8
35.8
34.0
35.2
35.8
36.5
34.1
35.2
36.0
40.6
40.1
401.4
62.9
yue
46.1
466
47.3
MRL
49.9
49.2
49.3
48.9
49.5
48.7
49.7
48.3
48.2
50.2
50.7
48.1
49,9
$1.1
56.3
$6.4
56.3
68.3
CS
HRRL
59.0
58.7
58,9
58.7
58.6
58.5
59.5
57.6
58.5
59.4
60.2
57.4
59.0
60.0
64.0
63.4
63.9
65.7
HRL
66.2
65.6
65.8
65.7
65.5
65.5
66.6
64.8
65.3
66.0
66.8
64.7
66.0
66.5
70.1
69.7
70.3
70.4
61.4
60.9
61.4
66.1
el
TIRL
$4.9
54.6
54.6
54.8
54,8 54.5
55.5
53.5 54.6
55.4
56.3
$1.4 55.0
$6.0
60.0
59.5
60.5
64.9
Ca
HRL
54.6
54.2
54.3
54.3
54,5
53.9
$5.0
52.8
53.6
$4.9
56.1
52.6
$4,8
$3.7
60.7
59.8
60.7
66.7
$3.0
52.7
$2.6
$2.5
52.2
S3.7
50.9
52.2
53.5
546
50.9
53.0
69.7
TIRL
63.5
63.2
63.2
63.2
63.3
63.1
63.8
62.3
62.8
63.6
64.5
620
63.7
64.1
67.7
67.2
67.6
68.3
HRL
59.2
58.5
588
586
588
58.5
59.5
57.8 58.1
59.1
60.3
57.3 59.3
60.0
64.8
64.0
64.6
71.3
id
HRRL
618
61.4
61.7
614
61.
61.2
62.2
60.6
61.0
61.8
62.7
60.3 61.9
62.6
66.4
66.0
66.3
60.0
57.3
HRL
56.4
55.9
55.9
55.
55.8
55.5
37.6
5388
55.6
57.0
58.4
53.7 56.3
57.7
64.3
63.6
64.8
64.2
HRIL
47.9
47.7
47.6
47.4
47.6
47.2
48.6
46.1
47.3
48.5
49.3
45.9
48.1
49.1
54.5
53.7
54.5
66.4
HRL
48.2
48.3
48.1
47.9
48.1
47.6
49.6
46.5
48.1
49.2
49.9
46.1
48.7
49.8
55.2
$4.8
55.5
68.6
HRI.
62.1
61.6
618
61.7
61.8
61.3
62.4
60.7
61.2
62.1
63.3
60.5
62.2
63.0
67.22
66.4
66.9
69.0
HRL
62.6
62.4
62.2
62.3
62.3
62.1
63.0
61.3 61.7
62.8
63.4
61.1 62.5
63.2
66.6
65.8
66.5
59.0
pl
p
HRI.
66.3
66.0
66.0
6600
66.0
65.8
66.6
05.2 (3.9
66.5
67.4
65.2
66.5
67.0
70.3
69.7
70.1
$9.5
TIRL
628
62.7
62.5
62.4
62.4
62.4
63.5
61.5
62.2
64.1
61.3
62.8
67.6
67.3
67.4
59.5
HRL
56.1
56.0
56.0
8
55.9
55.5
56.6
54.8
55.4
56.3
57.2
54.5
56.1
60.1
6118
72.7
sk
HRI.
58.5
58.5
58.1
58.5
58.5
58.2
59.3
57.3 58.0
58.9
60.2
57.2 58.9
$9.7
64.4
63.8
64,2
58.2
th
HRI.
49.2
48.5
48.3
48
48.
47.3
49.4
4688
48.2
49
50.4
47.2 488
49.7
56.1
53.4
56.3
61.2
HRI.
57.9
57.4
57.3
57.3
57.4
57.3
58.2
56.4
57.0
58.0
59.1
56.1
57.8
58.5
63.3
62.6
63.3
69.2
HRL
54.1
53.5
53.2
53.4
53.3
53.3
54.5
52.2
53.4
54.3
55.5
52.
54.3
54.7
39.7
59.5
70,1
48.5
48.7
48.4
49.3
46.7
48.0
49.1
50.2
468
49.0
61.4
Table 25: Full Results of CHRF FLORES for the models that trained, in the direction. See Table 23 to demystify the model abbreviations.


Reviews:
[{"review_id": "quf3ErmaG9", "reviewer": "Reviewer_zAXT", "summary": "", "questions": "- While I appreciate that the lexicon was tested “in the wild”, the results are hard to reproduce. Even for reproducing the results based on Appendix A, a few details are missing (vocab. size, used word piece model, learning rate etc.).\n- What is the intuition of introducing GLOWUP as a new task? What would happen if you do GLOWUP for the regular translation task (<2translation>)? It might work better during inference (Section 8.2) then.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and constructive criticism. \n\nWe acknowledge that one of the weaknesses of our paper is the use of in-house training data, and that the research community would be benefited by open-sourcing this data. However, due to the policies of our organization, we are not able to release the training data at this time. Though not ideal, we defend our use of in-house data for the following reasons:\n\n1. **This project aimed to examine the effect of multilingual lexicon data on large, massively multilingual models trained on lots of data. This is the setting in which SOTA production MT models are trained, and we wanted to replicate that.** Other works have already examined similar data augmentation approaches in settings with smaller models and less data, but **we wanted to see how these methods scale.** Our in-house dataset presented the best opportunity for doing these experiments.\n2. **The majority of the low-resource languages in our paper do not have publicly available data, or the data available for them is of dubious quality.** Since we were mainly focused on improvements in very low-resource languages, including this data in our training was crucial. Again, we would like to open-source our training data for these languages, but unfortunately this is out of our control.\n\nIn regard to your criticism of the GLOWUP strategy: we considered the method you proposed as well, where the lexicon translations are added as prefixes to the decoder. **However, due to the large number of data augmentation variants we were already experimenting with, we opted to try just the simpler approach that involved only modification of the source text.** Future work should certainly explore the decoder-prefix strategy, as well as the use of the <2translation> tag rather than <2glowup>. **The latter experiment could be added to the paper during the revision period.** To answer your question, the intuition behind using a different tag was that it might help the model distinguish between the different tasks and avoid confusion.\n\nOther things we can add during the revision period to address your concerns include significance tests (p-values) for some of the tables, plus the training details you requested like learning rate, vocab size, etc.\n\n**Finally, we would like to stress that although our training data is not open-source, one of the main contributions of this paper is the release of GATITOS, a multilingual lexicon that now covers 184 low-resource languages, many of which do not have any other multilingual resources available on the web.** We believe this open-source dataset will prove highly useful in building systems for extremely low-resource languages, and we ask that you give this fact ample weight in considering the merits of our paper. **The experimental results should be considered secondary to the open-sourced dataset.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Curating a high-quality, multilingual lexicon and open-sourcing it\n- Experimental results beyond the usual, lab-conditioned translation tasks", "reasons_to_reject": "- Experimental results are hard to reproduce since data are mostly in-house, details on public dataset experiments are missing \n- GLOWUP approach is not convincing (why not adding the corresponding translation as prefix to the decoder, see Missing References and Questions)\n- No significance test, e.g. for Table 3: Seems simply adding the lexicon to the training data is the most efficient and effective approach; are differences to CodeSwitchMonoGatiPanlex significant? (Same for Table 4: Parallel vs. Parallel+GatiPanlex)", "questions_to_authors": "- While I appreciate that the lexicon was tested “in the wild”, the results are hard to reproduce. Even for reproducing the results based on Appendix A, a few details are missing (vocab. size, used word piece model, learning rate etc.).\n- What is the intuition of introducing GLOWUP as a new task? What would happen if you do GLOWUP for the regular translation task (<2translation>)? It might work better during inference (Section 8.2) then.", "paper_topic_and_main_contributions": "This paper introduces a new multilingual lexicon covering 184 languages.\nThe dataset will be open-sourced and targets low-resource languages while providing high quality.\nTo demonstrate the high quality of the lexicon, the authors provide experimental results on different translation tasks for different scenarios showing that leveraging these additional data yields higher translation quality.\nThe author use both established and newly developed methods for integrating lexicon data into the model training.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "A similar approach to GLOWUP is presented in “Prompting Neural Machine Translation with Translation Memories” (Rehemann et al., AAAI 2023). \nHowever, the corresponding translation is the prefix of the decoded sentence which might yield better results also in your case.", "Typos_Grammar_Style_and_Presentation_Improvements": "- Section 5: A short introduction of MASS would be great\n- Section 5/6: Please introduce abbreviations first before using them (e.g. Glowup or MASS)\n- Provide some examples in the methods section (not in the Appendix), in particular for newly introduced methods such as GLOWUP\n- The naming in Section 7.1.3 is not very intuitive. I took me a while to understand the “GatiPanlex” suffix.\n- Personally I prefer to avoid using casual formulations such as “tl;dr” or “Is this the Bitter Lessen […] getting us again? Perhaps— […]”"}}, {"review_id": "jMeVH8OrnJ", "reviewer": "Reviewer_htr1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We are happy you found our work valuable. Of course, we would greatly appreciate it if you elaborate on your reasons for supporting our paper during discussion with other reviewers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strength of the paper is its experimentation on different augmentation strategies across hundreds of languages in a real-world setting.", "reasons_to_reject": "none", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work demonstrates that adding publicly available multilingual lexicon data and carefully curated bilingual data to low-resource languages and high-resource languages respectively can yield some significant gains over baseline models.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kAVtAwi4jr", "reviewer": "Reviewer_GrpG", "summary": "", "questions": "A. The writing is not clear, and the work of the paper seems to be unfinished. Have you done the rest of the work, as mentioned in Line 227? \n\nB. The paper is titled with the dataset GATITOS, could you provide more details about this dataset?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and constructive criticism. We will address your concerns in a pointwise fashion.\n\n- We apologize for any confusion caused by typos or other writing errors, and will make sure to correct these errors during revision\n\n- We believe the motivation of our paper is clearly stated in the Introduction:\n1. We provide a thorough comparison of several lexicon-based data augmentation variants for MT, all of which are simple, generalizable, and easy to implement \n2. We test these approaches “in the wild”, i.e. in a highly multilingual, web-mined data regime such as production systems tend to use, with hundreds of languages and billions of monolingual and parallel sentences \n3. We explore the effects of lexical data quality and quantity\n4. We demonstrate the efficacy of bilingual lexicon-based approaches as models scale\n5. We open-source the high-quality multilingual GATITOS lexicon for 184, mostly low-resource languages\n\n- At the time of writing the paper, GATITOS only covered 24 languages. Currently, it covers 184 languages and is publicly available. GATITOS is an ongoing project, and we hope to add more languages in the future.\n\n- During revision, we can certainly add more details about the GATITOS dataset to our paper. Of course, it would be helpful during the discussion period if you clarified what sort of dataset metadata you are interested in seeing so we can make sure it appears in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors conduct experiments for many existing multilingual lexica methods on low-resource machine translation tasks.\n\n2. A publicly available dataset named GATITOS is proposed.", "reasons_to_reject": "1. The paper is difficult to follow, there are some writing issues, such as Line 122.\n\n2. The motivation is not clear, which refers to other papers in Line 89. It should be directly and logically illustrated in the paper.\n\n3. The work is not completed, as indicated in Line 227. The GATITOS dataset only covers 24 languages, which may be the main contribution to this paper.", "questions_to_authors": "A. The writing is not clear, and the work of the paper seems to be unfinished. Have you done the rest of the work, as mentioned in Line 227? \n\nB. The paper is titled with the dataset GATITOS, could you provide more details about this dataset?", "paper_topic_and_main_contributions": "The author claims that cross-lingual vocabulary alignment is still highly imperfect in the approach combining supervised and self-supervised methods, especially for low-resource and unsupervised languages. To address these issues, the authors propose to use multilingual lexica for the translation tasks and design a dataset name GATITOS.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4xGReo0nNT", "reviewer": "Reviewer_TqkJ", "summary": "", "questions": "-) What is the reason for not using or combining large publicly available datasets? Even in a smaller preliminary study then validated on the in-house data?\n\n-) Section 6.1.1, can you have a situation where two entries in the bilingual lexica overlap in the source but with very different translations? If you have this situation, how do you manage it?\nFor instance,\nMonolingual sentence: A B C D G\nBilingual entries: B -> X and B C -> Y Z", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and constructive criticism. We will address your concerns in a pointwise fashion.\n\n- **During revision, we can certainly elaborate on how the combination of supervised and self-supervised training impacted our experiments.** The brief explanation is this: we experimented with two data augmentation approaches (Codeswitch and GLOWUP) on both monolingual data (used for self-supervised training) and parallel data (used for supervised training). We found that augmenting monolingual data was much more successful than augmenting parallel data. That is, **our methods worked much better when incorporated in the self-supervised part of training than in the supervised part.** We can add a discussion of why this might be the case, but our hypothesis is that the data-augmented supervised task is simply much more difficult for the model than the data-augmented self-supervised task, so much so that the model doesn’t learn as effectively from the former.\n\n- Another reviewer also commented on our use of in-house data, so we will give the same response here:\n\nWe acknowledge that one of the weaknesses of our paper is the use of in-house training data, and \nthat the research community would be benefited by open-sourcing this data. However, due to the \npolicies of our organization, we are not able to release the training data at this time. Though not ideal, we defend our use of in-house data for the following reasons:\n\n1. **This project aimed to examine the effect of multilingual lexicon data on large, massively multilingual models trained on lots of data. This is the setting in which SOTA production MT models are trained, and we wanted to replicate that.** Other works have already examined similar data augmentation approaches in settings with smaller models and less data (which is why the “initial study” you suggested would be pretty redundant), but we wanted to see how these methods scale to production-level scenarios. Our in-house dataset presented the best opportunity for doing these experiments.\n2. **Many of the low-resource languages in our paper do not have publicly available data, or the data available for them is of dubious quality.** Since we were mainly focused on improvements in very low-resource languages, including this data in our training was crucial. Again, we would like to open-source our training data for these languages, but unfortunately this is out of our control.\n\n- We can add clarification about the contributions of our paper. Essentially, **there are two broad contributions of our work to the research community: the open-sourcing of GATITOS, and the experiments we did to demonstrate how multilingual lexicons could be applied to multilingual MT at scale.** We acknowledge that the first of these contributions is applicable to a much broader range of researchers and labs, since anyone can use our publicly available lexicon but only a small number of industry labs can train massively multilingual NMT models from scratch. And while it is true that we could have done additional experiments at a much smaller scale using publicly available data, we believe the existing literature already covers this area very well already and that repeating these efforts would be mostly redundant.\n\n- **It is true that our methods show the greatest improvement on unsupervised languages, followed by low-resource ones. These languages were the explicit focus of our paper**,  thus the title “Using a New Multilingual Lexicon for Low-resource Machine Translation”. This does reduce the scope of the paper, but we strongly believe that these languages deserve close attention and that our work is useful even if it only significantly improves the performance of these languages.\n\n- You make a good point about the length of our paper and its suitability for a journal rather than a conference. We performed many experiments and analyses, and not all of them fit within 8-9 pages. That being said, **we believe our work would still be useful when presented in the EMNLP format, since one needn't look at every experiment we did to appreciate the contribution of our paper and the open-source dataset associated with it.**\n\n- We gave the problem of polysemy a lot of consideration when deciding how to implement the data augmentation. Ultimately, we ruled out using something like Word Sense Disambiguation for the sake of simplicity. \nAs stated in the Introduction, we wanted our data augmentation techniques to be “simple, generalizable, and easy to implement.” When working with hundreds of languages, we couldn’t guarantee that we could do WSD accurately for the entire dataset (especially the low-resource languages we were focused on). \nFurthermore, WSD is something MT models learn implicitly during training, so **even though some of our lexical substitutions may have been incorrect in context, this is not fundamentally a problem during training: it simply makes the task a denoising task, in which the model can learn implicitly when a word in not translated in the correct sense.** As has been demonstrated by works like BART, source-side noise is often *helpful* in training MT models, as long as the target side is clean.\nIn light of this, we simply sample lexicon translations at random during the data augmentation process.\n\n- During revision, we will add the citation you mentioned. Thank you for bringing this work to our attention.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strenghtnesses:\n\n-) This paper addresses an interesting problem and it is supported by a large set of analyses and experiments covering several language pairs. The evaluation tries to address different aspects of the problem with ad hoc experiments. I really appreciate the analysis of Section 8.4.\n\n-) The findings are interesting, in particular for the unsupervised MT. This aspect can be more emphasized in the paper. \n\n-) The discussion is grounded and I really appreciate the paragraph on the takeaway message at the end of each section in section 8.\n\n-) The release of the dataset is an important contribution to fostering research activities in low-resourced languages.", "reasons_to_reject": "Major weaknesses:\n-) The introduction gives some emphasis on the impact of the proposed data augmentation methods on the combination of supervised and self-supervised training, but this aspect has been barely discussed in the paper. \n\n\n\n-) Probably the main issue of this paper is the use of in-house data. This aspect has also been commented on in the limitation of the paper by the authors. The use of these data does not allow any other researcher to replicate the results of this paper. I believe that the main contributions of scientific papers should be replicable to foster innovation and improvement. However, if there is a need of using in-house data, please motivate it. It is not clear from the paper the reason because it is not possible to use huge quantities of available data (e.g. from the OPUS repository or the Paracrawl project). \nThe lack of data in some language directions can be a good point, but an initial study could have been done and presented in fewer language directions using public data. The findings could be then validated using the in-house data.\n\n-) There is a high value in this work, but the overall message of the paper is a bit confusing. The paper starts by emphasizing the impact of bilingual lexica on the combination of supervised and self-supervised methods, but this aspect has not been further discussed in the paper. The paper presents some approaches for data augmentation with bilingual lexica, but it is stated (beginning of section 6) that these approaches are not the main contributions, but the conclusion mainly highlights the impact of different approaches. The final message is that the proposed techniques have an impact mainly on unsupervised NMT, reducing the scope of the paper. The appendix contains a lot of material that can be easily a new paper by itself. Overall, some work is needed to improve the paper's clarity and facilitate the reader's appreciation of the paper. \nAll these factors make me wonder if a clear and more direct message could be obtained by focusing the paper only on unsupervised NMT and then having a longer version of the paper (probably for a journal) with all the results.", "questions_to_authors": "-) What is the reason for not using or combining large publicly available datasets? Even in a smaller preliminary study then validated on the in-house data?\n\n-) Section 6.1.1, can you have a situation where two entries in the bilingual lexica overlap in the source but with very different translations? If you have this situation, how do you manage it?\nFor instance,\nMonolingual sentence: A B C D G\nBilingual entries: B -> X and B C -> Y Z", "paper_topic_and_main_contributions": "This paper presents an investigation of the importance and impact of using bilingual lexica in real-world setting machine translation (MT), which means 200 translation models trained on a huge quantity of web-crawl (parallel and monolingual) data. For this purpose, several approaches to using bilingual lexica for data augmentation are tested and evaluated in different conditions. In particular, emphasis has been placed on multilingual machine translation, the combination of supervised and self-supervised training, the quality and quantity of the lexical data, and the impact on the performance of bilingual lexica when increasing the quantity of training data. As the results of this analysis, a small and high-quality bilingual lexicon covering 184 low-resource languages is made available with this paper. The results show that: 1) the main impact on performance when using lexical data augmentation is visible in unsupervised MT, 2) the data augmentation approaches show to be effective and have high complementarity, 3) the gains given by using augmented data significantly reduces when scaling up the model, and 4) the quality of bilingual lexica is fundamental to have an impact on MT more than the size.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "There is an important strand of research in (statistical and neural) MT on the use of bilingual terminology lists. This is not mentioned in the related work. A seminal paper for neural MT on this topic is\nDinu, Georgiana, Prashant Mathur, Marcello Federico, and Yaser Al-Onaizan. \"Training Neural Machine Translation to Apply Terminology Constraints.\" In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3063-3068. 2019.\n\nThe use of bilingual terminology lists is clearly connected to this work and some of the approaches proposed in the past overlap with the methods mentioned in this paper.", "Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\n\n\n-) Structure of Section 3.3 seems to be wrong. Sections 3.4 and 3.5 should be subsubsections.\n\n-) Page 2, line 122: “The tl;dr”\n\n-) Related Work: NLI and MLM are not previously introduced.\n\n-) Page 6, line 488: missing space between FLORES-200 and “and”\n\n-) Page 7, line 530: missing space “wins.” and “For”\n\nPresentation Improvements:\n\n-) The structure of the paper can be improved by moving Sections 3, 4, and 5 after Section 6. In case, there is an interest in emphasizing the data conditions (monolingual vs bilingual), only Sections 4 and 5 can be moved.\n\n\n-) To facilitate the reader, please move Table 4 to the previous page."}}]

Author rebuttal:

Meta-review: In this paper the authors present a curated bilingual lexicon covering 168 language pairs, and
make an experimental comparison of different ways of using this lexicon in data augmentation
for low-resource MT.

For soundness, reviewers zAXT, GrpG and TqkJ all agree on a score of 3. The main criticism being
that the experiments are impossible to reproduce because of the use of in-house data, and the lack
of detail on the experimental settings. To me, this is a valid criticism, the authors could have
used the Opus data set for example, and aspects such as the quality and domain of the dataset used
in these experiments is not available to the community. Given this, I think a score of 3 is reasonable.

There is a disagreement between the reviewers about excitement, ranging from 2 (GrpG) to 4 (TqkJ).
The latter reviewer emphasises the wide coverage of the experimental work, the interesting implications
of the results on unsupervised MT, and the usefulness of the data set (which is released as open-source).
To me, these all seem like worthy contributions.

Reviewer GrpG raised some concerns about the clarity of the paper, although did not back up their
concerns with any valid specific points. I note that reviewer zAXT also had several questions and
suggestions, and reviewer TqkJ raised issues about the structure of the paper. These comments
from reviewers suggest that there are clarity issues with the paper, and that the
the authors should take into account the                                                    
reviewers that provided constructive comments.

Lastly, reviewer GrpG raised a question about the completeness of the data set. In the discussion, it
was revealed that the data set was incomplete at submission time, but has now been completed with
184 languages. The small version (with 24 languages) was used in the experiments. The authors need to
clarify this is their update. I would also ask them to clarify the licence of the data set.

Overall, I expect that the data set is a worthwhile contribution, and the experiments shed light
on data augmentation for low-resource languages.
Decision: Accept-Main