Venue: NAACL 2025

# Alligators All Around:

Mitigating Lexical Confusion in Low-resource Machine Translation

Elizabeth Nielsen Isaac Caswell Jiaming Luo Colin Cherry Google {eknielsen, icaswell, jmluo, colincherry}@google.com

###### Abstract

Current machine translation (MT) systems for low-resource languages have a particular failure mode: When translating words in a given domain, they tend to confuse words within that domain. So, for example, _lion_ might be translated as _alligator_, and _orange_ might be rendered as _purple_. We propose a recall-based metric for measuring this problem and show that the problem exists in a dataset comprising 122 low-resource languages. We then show that this problem can be mitigated by using a large language model (LLM) to post-edit the MT output, specifically by including the entire Gattos lexicon for the relevant language as a very long context prompt. We show gains in average ChrF score over the set of 122 languages, and we show that the recall score for relevant lexical items also improves. Finally, we demonstrate that a small dedicated MT system with a general-purpose LLM as a posterior outperforms a generalisation LLM translator with access to the same lexicon data, suggesting a new paradigm for LLM use.

## 1 Introduction

Machine translation systems have recently expanded to cover many previously unsupported languages Bapna et al. (2022b); NLLB et al. (2022). However, MT systems for low-resource languages (LRLs) still face many challenges. One particular difficulty is learning the correct mapping of words between two languages. This paper is motivated by the observation that some LRL MT models tend to confuse certain lexical items belonging to similar domains. This problem is first reported in Bapna et al. (2022b), who report this issue with unsupervised, sentence-level NMT, giving the following examples from their models. Examples from their paper are reproduced in Table 1.

These examples show that the model consistently errs by confusing lexical items that share similar distributions, such as using _crocodile_ to translate other animal terms. This pattern is observed in the "next thousand languages" (NTL) MT models of Bapna et al. (2022b) over many language pairs and within relatively high-frequency lexical domains, including numbers, colors, animals, days of the week, and months. In this paper, we refer to the tendency to confuse words within a domain as the "alligator problem."1 As we show in this paper, this pattern isn't only found in MT-specific models, but in translations produced by large language models (LLMs) as well.

Footnote 1: Not the “crocodile problem”, because somewhere between encountering the crocodile-filled examples from Bapna et al. (2022b) and starting this work, we confused alligators and crocodiles. We kept the name, though, since our mistake is itself a nice illustration of the problem.

Using a development set consisting of data from 122 URLs, we show that this problem is widespread in translations of the NTL models, which are described in Bapna et al. (2022b). We then propose a method for prompting an LLM with lexical information to post-edit these translations, both translating into and out of English, leading to better performance on these frequently confused lexical items, as well as higher machine translation quality overall. The lexical information is provided by incorporating the Gatitos lexicon Jones et al. (2023) into the LLM prompt. We further show that the LLM is able to improve its performance on these lexical items even when the lexicon entries presented in the prompt don't exactly match the source string because of morphological inflections.

This method combines the in-depth knowledge of the specialist NTL MT systems with the generalist abilities of the LLM. We show that the LLM is incapable of matching the MT system's performance on its own, even when given access to the lexicon, despite the fact that the MT system is much smaller, at only 850M parameters. However, given the specialist MT model's best hypotheses, the LLM can fix the MT model's persistent lexical confusions as a post-editor, making use of the infor

mation in the lexicon. Our primary contributions are:

* Demonstrating that the "alligator problem" (lexical confusion on distributionally similar words) is a failure mode not only in traditional MT, but also in LLMs.
* Developing a targeted evaluation for the alligator problem, and demonstrating a method for fixing the problem by using an LLM as post-editor with a lexicon as context.
* Revealing that specialist MT models still far outperform generalist LLMs on LRL translation, and introducing a new paradigm of generalist-LLM-as-post-editor.

## 2 Related work

MT for low-resource languagesBefore LLMs, for Very Low-Resource Language MT -- i.e. anything beyond the most frequent hundred languages or so -- there existed no parallel text at all outside of religious domains. In these cases, the only option was Unsupervised Machine Translation (UNMT), which uses only monolingual text to translate. This was pioneered in Lample et al. (2017); Artetxe et al. (2017); Song et al. (2019), and eventually Bapna et al. (2022a) scaled up to 1000 languages in the NTL models. However, the unsupervised paradigm led to tell-tale mistakes, such as the "alligator problem" discussed here.

LLMs then barged in and changed all these paradigms, although they still perform poorly out of the box on LRLs Kocmi et al. (2023). A common approach is in-context learning, or ICL Brown et al. (2020); Agarwal et al. (2024) which gives examples in the prompt. ICL examples for LRLs have included diverse context like sentence pairs Zhang et al. (2024); Tanzer et al. (2024), dictionaries Elsner and Needle (2023), the full GatiTos lexicon Reid et al. (2024), and a full grammar of the Kalamang language Tanzer et al. (2024). A popular variant of ICL is RAG, or Retrieval-augmented generation Rubin et al. (2022), which draws only on examples for ICL that are relevant to the current sentence being translated. Despite its popularity, Vilar et al. (2023); Zhu et al. (2024); Zhang et al. (2023) find exemplar quality is more important than relevance.

LLMs as post-editors.Another less common approach for LRL MT has focused on automatic post-editing (APE) translations with LLMs, which is an approach often used in high-resource MT Bhattacharyya et al. (2023); Zerva et al. (2024). Chen et al. (2024) let an LLM iteratively self-correct its translation, Lim et al. (2024) have a model post-edit its own translations from related higher-resource languages into the target language, and Xu et al. (2024) iteratively apply fine-grained error correction from an LLM. However, these efforts have focused on a base model and a post-editor that are the same size, and both large.

Rare word translationMany MT models struggle specifically with translating rare words, including MT models for high-resource languages. In our case, we study the inverse problem of difficulties with _common_ words, but the approaches necessary to fix may be the same. Prior work includes placing soft constraints on the output terminology Bergmanis and Pinnis (2021) and augmenting parametric models with non-parametric datastores such as parallel corpora Khandelwal et al. (2021) or lexica Zhang et al. (2021). The latter is more similar to our approach, though we present a lexicon to the LLM as a part of a prompt, rather than using it during the training phase.

## 3 Methods

The approach we take to solving this problem is to (1) generate output for a set of LRLs using a specialist MT system; (2) create prompts for post-editing each segment that include the entire GatiTos lexicon, and (3) use these prompts to generate post-edited output using a generalist LLM. The example in Table 2 illustrates how a single Udmurt example passes through the pipeline of specialist MT system and LLM-posteditor:

### Data

Evaluation data.To measure the magnitude of this problem, we evaluate the performance of the

\begin{table}
\begin{tabular}{l|p{85.4pt} p{162.3pt}} \hline \hline Language & reference & translation \\ \hline Meitéilon (mini) Twi (ak) & I believe a **lion** is **stronger** than a **tiger**. I would want to be a **dog** for a day. & I believe a snake is **stronger** than a crocodile. \\  & I would want to be a **dog** for a day. & I want to be a crocodile just one day. \\ \end{tabular}
\end{table}
Table 1: Examples from Bapna et al. (2022a) of the “alligator problem” 

models on 122 LRLs, translating into and out of English (complete list in Appendix C). The evaluation data comprises segments from Flores-200 (NLLB et al., 2022), ntrexBarrault et al. (2019); Federmann et al. (2022) and GatonesJones et al. (2023). For each language pair, there are 600-1000 segments.

Prompting data.This lexical information comes from the Gatito's lexicon Jones et al. (2023). This is a 4000-entry multilingual lexicon with English segments, which have been translated by human translators into 170 very low resource languages. These lexical segments include frequent English tokens (including words for numbers, months, and days of the week), Swadesh wordlistsSwadesh (1952), and some short English sentences.

### Metrics

General MT quality:To measure general quality we report ChrF score Popovic (2015).

Alligator recall:ChrF will not necessarily reflect wins or losses in the alligator problem. To directly measure this problem, we propose a recall-based metric over a set of predetermined lexical items with similar distributions, which we call _alligator recall_. The selected lexical items are shown in Appendix A, and are grouped into the domains of animals, colors, weekdays, months, common numbers, and rare numbers. They are restricted to terms that are in the Gatito's lexicon. For a given evaluation set, we find all references that have one of these words, and score the model hypotheses on whether they 1) produced the exact correct word (correct); 2) produced a _different_ in-domain word (confusion, i.e., the alligator problem); or 3) produced neither a correct nor incorrect word (unknown). If a total of \(N\) alligator words appear in the set of all reference strings, and the model's hypotheses produce the corresponding correct alligator word \(R\) times and a different in-domain word \(W\) times, then we report the corresponding alligator scores as follows:

\[\text{correct}=\frac{R}{N} \tag{1}\]

\[\text{confusion}=\frac{W}{N} \tag{2}\]

\[\text{unknown}=\frac{N-R-W}{N} \tag{3}\]

We only report alligator recall for the into-English direction. Measuring the presence or absence of a word in the model output via simple string matching is problematic for more morphologically complex languages. For example, the Udmurt word for _April_ is listed in citation form as _oshtolez'_. However, in one phrase in our evaluation data, "in April 2020," it is inflected to _osstholeze_ -- with the final character of the citation form (transliterated as ') removed, and the suffix -_e_ added. If we calculated alligator recall on Udmurt target data, we would count inflections like these as non-matches, unless we accounted for morphological inflection. However, accommodating the diverse morphologies of 122 languages is outside the scope of this paper. Therefore, for the out-of-English translation direction, we report only ChrF.

### Models

We use the NTL MT models as our baseline Bapna et al. (2022b). These are sentence-level, unsupervised transformer translation models, that are trained as follows: First, for each language in their training data (a set which includes our 122 evaluation languages), an encoder-decoder Transformer model with 6B parameters is trained. Because data is limited, this first phase uses a MASS de-noising task on monolingual data Song et al. (2019). The second phase of training consists of iterative backtranslation, where the models are used to generate parallel data via online translation, and then trained on this synthetic data. Finally, these models are distilled into multilingual 850M parameter encoder-decoder models, and cover either the en > xx or xx > en direction.

\begin{table}
\begin{tabular}{l|c} \hline
**Source** & 5:30 chasysen 2:30 chasoz’ vordis‘konsven könsynaloz’ \\
**Reference** & between 5:30 am to 2:30 am from Mondays to **Saturdays** \\
**MT output** & from 5:30 a.m. to 2:30 a.m. Monday through **Friday** \\
**Post-edit** & from 5:30 a.m. to 2:30 a.m. Monday through **Saturday** \\ \hline \end{tabular}
\end{table}
Table 2: An example of how the MT model and postediting step render a single example from Udmurt. The alligator problem is shown by the error highlighted in red, which is corrected by the postedting step.

 

For post-editing, we use the LLM Gemini 1.5 Pro Reid et al. (2024), whose long context (up to 10M tokens) is ideal for our purposes. We perform greedy decoding to generate outputs.

## 4 Results and discussion

Tables 3 and 4 show that the best performance comes from using the LLM as a post-editor, and including the entire Gatitos lexicon in the prompt. The models we compare are (1), the MT models alone (our baseline), (2) the LLM model alone, and (3) the LLM as post-editor of the MT model output. The exact prompt templates are in Appendix B. The prompts given to the LLM include all 4000 entries from Gaitos for the given language, except when noted otherwise.

As shown in Table 3, lexical confusion is present in the initial MT system output, but when averaged over all evaluation languages, its severity is limited. When we subsample the 20% of languages with the highest level of lexical confusion, it becomes clear that this issue is much more severe for some languages than for others.2 The highest quality output is consistently produced by prompting the LLM to postedit the MT system output. The lexical recall gains are particularly concentrated in the languages that had the highest rates of lexical confusion.

Footnote 2: For the list of languages constituting the high-confusion group, see the table in Appendix C.

Other attempted methods fall short of the performance of LLM post-editing with access to the whole lexicon. The LLM on its own is a relatively poor translator, even given the entire Gatitos lexicon. On these high-confusion languages, we also experiment with presenting the LLM with a few different levels of lexical information: no lexical information, prompts with only the words in the given segment, and prompts with the whole lexicon. No lexical information is, as expected, a worse condition, but even limiting the prompt to include only the lexical items that are present in the source is unhelpful -- this condition under-performs even the baseline.

As expected, the prevalence of lexical confusion correlates with the overall performance of the MT systems on a language, as shown in Table 3, where languages with higher confusion have lower ChrF score. For per-language scores, see Appendix C.

### Morphology and the shortcomings of string-match RAG

One reason why prompts with targeted lexical information fail may be that retrieving words from the lexicon for the prompt is difficult in languages with complex morphology: string matching can't retrieve words that don't appear in the _citation form_ (the uninflected root form) in MT input. To measure how often a retrieval from the lexicon would fail, we identify times when an English word from our evaluation list (see Appendix A) appears in the gold reference in the xx\(\rightarrow\)en direction. We then count how often the word is missing in the initial MT system output, but appears in the post-editing output of the LLM prompted with the whole lexicon. Of the cases where post-editing recovers the correct word, we measure how often the corresponding source language token (from Gaittos) appears in the source in citation form.

The citation form occurs in the source side only 56.1% of the total times that the post-editing procedure correctly recovered a lexical item. This suggests that the LLM was able to use information in the lexicon even when retrieval of the correct item from the lexicon would have required going beyond an exact match. A significant source of these retrieval failures is likely the morphological inflections in the source string that complicate retrieval. Recall the example given in Section 3.2: the Udmurt word for _April_ is _oshtolez_', but it appears in the evaluation data in an inflected form, _oshtoleze_, as part of a phrase meaning, "in April 2020." In this inflected form, the final character of the citation form (transliterated as ') is removed, and the suffix _-e_ added. This makes direct retrieval of this item from the lexicon difficult. Additionally, the substitution of synonyms in the source string would affect this. Whether these retrieval failures are due to morphological inflection or synonymy, the LLM is able to recover the correct target word in many of these cases when simply given the entire lexicon and handles lexical variations itself.

## 5 Conclusion

This work is the first to document and quantify the _alligator problem_ in Large Language Models for low resource languages, a systemic translation error mode that is not well captured in metrics like Chrf. This problem is much reduced, though not fully eliminated, by our proposed approach of lexicon-augmented post-editing. This also suggests 

a new paradigm for generalist models like LLMs, exploiting their better general-purpose reasoning and tool use to use them as post-editors. The small, specialized MT model provides a strong baseline for translation performance, one that the LLM cannot meet on its own, even when given access to a lexicon. However, the LLM can better extract and use information from a resource like Gatitos, and therefore improve upon its superior's work. The LLM is also able to overcome challenges such as complex morphology that would make it prohibitively difficult to use the lexicon directly to post-edit the MT output.

### Limitations

One limitation of this work is the fact that exact string matching is used in the alligator recall evaluation, which doesn't account for morphological inflection or synonymy. So for example, if the word _twelve_ appeared in the reference and the model output _a dozen_, this would fall into the unknown category of the metric rather than the correct category, where it likely belongs. Likewise, if the reference word is morphologically infected in such a way that the citation form doesn't appear in the output (e.g., _geese_ instead of _goose_), it would fall into the unknown category. This is mitigated by the fact that the set of evaluation words we use have relatively few synonyms (weekdays, months, and common numbers, for example). All of them are also nouns with regular plurals, so even when they appear in an inflected form (plural being the only option for English nouns), the citation form should appear as a substring in the target output.

Other limitations include using a hand-picked set of words over which to evaluate the alligator problem. Finally, it would be preferable to be able to perform the alligator recall metric on non-English output. Addressing the English-only nature of this evaluation would require handling the morphology of 122 very low-resource languages, which would almost certainly require producing more resources for them, which lies outside the scope of this work.

## References

* Agarwal et al. [2024] Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, and Hugo Larochelle. 2024. Many-shot context learning.
* Artetxe et al. [2017] Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2017. Unsupervised neural machine translation. _arXiv preprint arXiv:1710.11041_.
* Bapna et al. [2016] Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firata, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, Theresa Breiner, Vera Axelrod,

\begin{table}
\begin{tabular}{l l c} \hline  & **ChrF** (\(\uparrow\)) \\ \hline \multirow{3}{*}{**All langas.**} & **Baseline** & 43.5 \\  & **Direct translation** & 40.9 \\  & **Post-edit, whole lex.** & **44.0** \\ \hline \multirow{3}{*}{**High-conf.**} & **Baseline** & 37.6 \\  & **Direct translation** & 35.9 \\ \cline{1-1}  & **Post-edit, whole lex.** & **38.2** \\ \cline{1-1}  & **Post-edit, target lex.** & 34.4 \\ \cline{1-1}  & **Post-edit, no lex.** & 36.5 \\ \hline \end{tabular}
\end{table}
Table 4: ChRF scores for the en\(\rightarrow\)xx direction. High-confusion languages are the top 20% of languages by confusion in the xx\(\rightarrow\)en direction. Alligator scores are not reported in this direction, since it can’t be reliably calculated on non-English output.

\begin{table}
\begin{tabular}{c l c c|c|c} \hline  & & \multicolumn{3}{c|}{**Alligator recall scores**} & \\  & & **Correct** (\(\uparrow\)) & **Confusion** (\(\downarrow\)) & **Unknown** (\(\downarrow\)) & **ChrF** (\(\uparrow\)) \\ \hline \multirow{3}{*}{**All languages**} & **Baseline** & 59.4 & 3.8 & 36.9 & 52.4 \\  & **Direct translation** & 2.9 & 4.1 & 93.0 & 48.5 \\  & **Post-edit, whole lex.** & **62.4** & **2.8** & **34.8** & **53.2** \\ \hline \multirow{3}{*}{**High-confusion**} & **Baseline** & 49.6 & 8.3 & 42.2 & 45.3 \\  & **Direct translation** & 2.4 & 3.6 & 94.0 & 39.8 \\ \cline{1-1}  & **Post-edit, whole lex.** & **57.0** & **4.8** & **38.2** & **46.3** \\ \cline{1-1}  & **Post-edit, targeted lex.** & 53.0 & 6.0 & 40.9 & 43.7 \\ \cline{1-1}  & **Post-edit, no lex.** & 51.1 & 7.2 & 41.7 & 44.9 \\ \hline \end{tabular}
\end{table}
Table 3: ChrF and lexical recall scores for the ex\(\rightarrow\)ren translation direction. High-confusion languages are the top quintiple of languages by confusion score. “Post-edited” scores represent the output of the LLM that has been prompted to postedtit the MT output.

 

Jason Riesa, Yuan Cao, Mia Xu Chen, Klaus Macherey, Maxim Krikun, Pidong Wang, Alexander Gutkin, Apurva Shah, Yanping Huang, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2022a. Building machine translation systems for the next thousand languages.
* Bapna et al. (2022b) Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Nikhil Baljekar, Xavier Garcia, Wolfgang Macherey, Theresa Breiner, Vera Saldinger Axelrod, Jason Riesa, Yuan Cao, Mia Chen, Klaus Macherey, Maxim Krikun, Pidong Wang, Alexander Gutkin, Apu Shah, Yanping Huang, Zhifeng Chen, Yonghui Wu, and Macduff Richard Hughes. 2022b. Building machine translation systems for the next thousand languages. Technical report, Google Research.
* Barrault et al. (2019) Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Muller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT'19). In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_, pages 1-61, Florence, Italy. Association for Computational Linguistics.
* Bergmanis and Pinnis (2021) Toms Bergmanis and Marcis Pinnis. 2021. Facilitating terminology translation with target lemma annotations. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 3105-3111, Online. Association for Computational Linguistics.
* Bhattacharyya et al. (2023) Pushpak Bhattacharya, Rajen Chatterjee, Markus Freitag, Diptesh Kanojia, Matteo Negri, and Marco Turchi. 2023. Findings of the WMT 2023 shared task on automatic post-editing. In _Proceedings of the Eighth Conference on Machine Translation_, pages 672-681, Singapore. Association for Computational Linguistics.
* Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbaiah, Jared Kaplan, Praffulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Ma teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. _ArXiv_, abs/2005.14165.
* Chen et al. (2024) Pinzhen Chen, Zhicheng Guo, Barry Haddow, and Kenneth Heafield. 2024. Iterative translation refinement with large language models.
* Elsner and Needle (2023) Micha Elsner and Jordan Needle. 2023. Translating a low-resource language using GPT-3 and a human-readable dictionary. In _Proceedings of the 20th SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology_, pages 1-13, Toronto, Canada. Association for Computational Linguistics.
* news test references for MT evaluation of 128 languages. In _Proceedings of the First Workshop on Scaling Up Multilingual Evaluation_, pages 21-24, Online. Association for Computational Linguistics.
* Jones et al. (2023) Alexander Jones, Isaac Caswell, Orhan Firat, and Ishank Saxena. 2023. GATITOS: Using a new multilingual lexicon for low-resource machine translation. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 371-405, Singapore. Association for Computational Linguistics.
* Khandelwal et al. (2021) Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2021. Nearest neighbor machine translation. _ICLR 2021_.
* Kocmi et al. (2023) Tom Kocmi, Eleftheios Avramidis, Rachel Bawden, Ondrej Bojar, Anton Dworkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Makoto Nagata, Toshiaki Nakazawa, Martin Popel, Maja Popovic, and Mariya Shmatova. 2023. Findings of the 2023 conference on machine translation (WMT23): LLMs are here but not quite there yet. In _Proceedings of the Eighth Conference on Machine Translation_, pages 1-42, Singapore. Association for Computational Linguistics.
* Lample et al. (2017) Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ronzato. 2017. Unsupervised machine translation using monolingual corpora only. _arXiv preprint arXiv:1711.00043_.
* Lim et al. (2024) Zheng Wei Lim, Nitish Gupta, Honglin Yu, and Trevor Cohn. 2024. Mufu: Multilingual fused learning for low-resource translation with llm.
* NLLB et al. (2022) Team NLLB, Marta Costa-jussa, James Cross, Onur Celebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Daniel Licht, Jean Mallard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Gonzalez, Prangthip Hansanti, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. Technical report.
* Popovic (2015) Maja Popovic. 2015. chrF: character n-gram F-score for automatic MT evaluation. In _Proceedings of the Tenth Workshop on Statistical Machine Translation_, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
* Reid et al. (2021) Michel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis 

Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mira Glaese, Thibault Sotitaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross McIlroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Eric Moreira, Kareem W. Ayoub, Megha Goel, Clemens Meyer, Gregory Thornton, Zhen Yang, Henry Michalewski, Zaheer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, Luke Vilnis, Oscar Chang, Nobuyuki Morioka, George Tucker, Ce Zheng, Oliver Woodman, Nithya Attaluri, Tomas Kociskky, Evgenii Eltyshev, Xi Chen, Timothy Chung, Vittorio Selo, Siddhartha Brahma, Petko Georgiev, Ambrose Slone, Zhenkai Zhu, James Lottes, Siyuan Qiao, Ben Caine, Sebastian Riedel, Alex Tomala, Martin Chadwick, J Christopher Love, Peter Choy, Sid Mittal, Neil Houlsby, Yunhao Tang, Matthew Lamm, Libin Bai, Qiao Zhang, Luheng He, Yong Cheng, Peter Humphreys, Yujia Li, Sergey Brin, Albin Cassiver, Ying-Qi Miao, Lukas Zilka, Taylor Tobin, Kelvin Xu, Lev Proleev, Daniel Sohn, Alberto Magni, Lisa Anne Hendricks, Isabel Gao, Santiago Ontan'on, Oskar Bunyan, Nathan Byrd, Abhanshu Sharma, Biao Zhang, Mario Pinto, Rishika Sinha, Harsh Mehta, Dawei Jia, Sergi Caelles, Albert Webson, Alex Morris, Becca Roelofs, Yifam Ding, Robin Strudel, Xuehan Xiong, Marvin Ritter, Mostafa Dehghani, Rahma Chaabouni, Abhijit Karmarkar, Guangda Lai, Fabian Mentzer, Bibo Xu, YaGuang Li, Yujing Zhang, Tom Le Panne, Alex Goldin, Behnam Neyshabur, Kate Baumli, Anshem Levskaya, Michael Laskin, Wenhao Jia, Jack W. Bae, Kefan Xiao, Antoine He, Skye Giordano, Lakshman Yagati, Jean-Baptiste Lespiau, Paul Natsev, Sanjay Ganapathy, Fangyu Liu, Danilo Martins, Nanxin Chen, Yunnan Xu, Megan Barnes, Rhys May, Arpi Vezer, Junhyuk Oh, Ken Franko, Sophie Bridgers, Ruizhe Zhao, Boxu Wu, Basil Mustafa, Sean Sechrist, Emilio Parisotto, Thanumalyayan Sankaranarayan Piliali, Chris Larlari, Chenje Gu, Christina Sorokin, Maxim Kriku, Alexey Gusenov, Jessica Landon, Romina Datta, Alexander Pritzel, Phoebe Thacker, Fan Yang, Kevin Hui, A.E. Hauth, Chih-Kuan Yeh, David Barker, Justin Mao-Jones, Sophia Austin, Hannah Sheahan, Parker Schwuh, James Svensson, Rohan Jain, Vinay Venkatesh Ramasesh, Anton Briukhov, Da-Woon Chung, Tamara von Glehn, Christina Butterfield, Priya Jhakra, Matt Wiethoff, Justin Frye, Jordan Grimstad, Beer Changpinyo, Charline Le Lan, Anna Borisova, Yonghui Wu, Paul Voigtlaender, Tara N. Sainath, Charlotte Smith, Will Hawkins, Kris Cao, James Besley, Srivastan Srinivasan, Mark Ommerick, Colin Gaffney, Gabriela de Castro Surita, Ryan Burnell, Bogdan Damoc, Junwhan Ahn, Andrew Brock, Mantas Papiraskas, Anastasia Petrushkina, Seb Noury, Lorenzo Blanco, Kevin Swersky, Arun Ahuja, Thi Avraham, Vedant Misra, Raoul de Liedekerekre, Mariiko Inuma, Alex Polozor, Sarah York, George van den Driessche, Paul Michel, Justian Chiu, Roy Bleurs, Zach Gleicher, Adria Rescenes, Alban Rustemi, Elena Gribovskaya, Aurko Roy, Wiktor Goworek, Sebastien M. R. Arnold, Lisa Lee, James LeeThorp, Marcello Maggioni, Enrique Piqueras, Kartikey Badola, Sharad Vikram, Lucas Gonzalez, Anirudh Baddepudi, Evan Senter, Jacob Devlin, James Qin, Michael Azzam, Maja Trebacz, Martin Polacek, Kashyap Krishnakumar, Shuq yin Chang, Matthew Tung, Ivo Penchev, Rishabb Joshi, Kate Olszewska, Carrie Muir, Mateo Wirth, Ale Jakse Hartman, Joshua Newman, Sheleeman Kashem, Vijay Bolina, Elahe Dabir, Joost R. van Amersfoort, Zarafali Ahmed, James Cobon-Kerr, Aishwaya B Kamatharn, Arnar Mar Hrafnkelsson, Leo Hou, Ian Mackminnon, Alexandre Fredette, Eric Noland, Xiance Si, Emanuel Taropa, Dong Li, Phil Crone, Amol Gulati, \(\grave{S}\)ebastin Cevey, Jonas Adler, Ada Ma, David Silver, Simon Tokumine, Richard Powell, Stephan Lee, Michael B. Chang, Samer Hassan, Dinana Mincu, Antoine Yang, Nir Levine, Jenny Brennan, Mingqiu Wang, Sarah Hodkinson, Jeffrey Zhao, Josh Lipschultz, Andrea Pope, Michael B. Chang, Cheng Li, Laurent El Shafey, Michela Paganiini, Sholto Douglas, Bernd Bohnet, Fabio Pardo, Seth OdOOom, Mihaela Rosca, Cicero Nogueira dos Santos, Cedar Sapparkar, Arthur Guez, Tom Hudson, Steven Hansen, Chulaynuth Asawaroengchai, Ravikachandra Addanal, Tianhe Yu, Wojciech Stokowice, Mina Khan, Justin Gilmer, Jaenoon Lee, Carrie Grimes Bostock, Keran Rong, Jonathan Caton, Pedram Pejman, Filip Pavelic, Geoff Brown, Vivek Sharma, Mario Lucvi'c, Rajkumar Samuel, Josip Djolnaga, Amel Mandhane, Lars Lowe Sjosund, Elena Buchatskaya, Elspeth White, Natalic Clay, Jiepu Jiang, Hyeontaek Lim, Ross Hemsley, Jane Labanowski, Nicola De Cao, David Steiner, Sayed Hadi Hashemi, Jacob Austin, Anita Gergely, Tim Blyth, Joe Stanton, Kaushik Shivakumar, Aditya Siddhant, Anders Andreassen, Carlos L. Araya, Nikhil Sethi, Rakesh Shivanna, Steven Hand, Ankurb Bapna, Ali Khodaei, Antoine Miech, Garrett Tanzer, Andy Swing, Shnatanu Thakoor, Zhufeng Pan, Zachary Nado, Stephanie Winkler, Dian Yu, Mohammad Saleh, Lorenzo Maggiore, Iain Barr, Minh Giang, Thais Kagohara, Ivo Danihelka, Amit Marathe, Vladimir Feinberg, Mohamed Elhawaty, Nimesh Ghelani, Dan Morgan, Helen Miller, Lexi Walker, Richard Tanburn, Mukaram Tariq, Disha Shrivastava, Fei Xia, ChungCheng Chiu, Zoe C. Ashwood, Khuslen Baatauskh, Sina Samangooei, Fred Alcober, Axael Sigerngren, Paul Komarek, Katerina Tsihlas, Anudhyan Boral, Ramona Comanescu, Jeremy Chen, Rubio Liu, Dawn Bloxwich, Charlie Chen, Yanhua Sun, Fangxiaoyu Feng, Matthew Mauger, Xerexes Douwalla, Vincent Hellendorn, Michael Sharman, Ivy Zheng, Krishna Hardisan, Gabriel Barton-Maron, Craig Swanson, Dominika Rogozoi'nska, Aleks Andreev, Paul Kishan Rubenstein, Ruoxin Sang, Dan Hurt, Gamaleddin Elsayed, Ren shen Wang, Dave 

Lacey, Anastasija Ii'ic, Yao Zhao, Woohyun Han, Lora Aroyo, Chimezie Iwuanyanwu, Vitaly Nikolaev, Balaji Lakshminarayanan, Sadegh Jazayeri, Raphael Lopez Kaufman, Manir Vardarajan, Chetan Tekur, Doug Fritz, Misha Khalman, David Reitter, Kingschuk Dasgupta, Shourya Sarcar, T. Ormduff, Javier Snaider, Fantine Huot, Johnson Jia, Rupert Kemp, Nejc Trdin, Anitha Vijayakumar, Lucy Kim, Christof Angermueller, Li Lao, Tianqi Liu, Haibin Zhang, David Engel, Somer Greene, Anais White, Jessica Austin, Lilly Taylor, Shereen Ashraf, Daniyi Liu, Maria Georgaki, Irene Cai, Yana Kulizhsaya, Sonam Goenka, Brennan Saeta, Kiran Vodrahalli, Christian Frank, Dario de Cesare, Brona Robenek, Harry Richardson, Mahmoud Alnahlawi, Christopher Yew, Priya Ponnamgalli, Marco Tagliascchi, Alex Korchemmy, Yelin Kim, Dinghua Li, Bill Rosgen, Kyle Levin, Jeremy Wiesner, Praseem Banzal, Praveren Srinivasan, Hongkin Yu, cCauglar Unlu, David Reid, Zora Tang, Daniel F. Finchelstein, Ravin Kumar, Andre Eliseeff, Jin Huang, Ming Zhang, Rui Zhu, Ricardo Aguilar, Mai Gim'enez, Jiawei Xia, Olivier Dousse, Willi Gierke, Soheil Hassas Yeganeh, Damion Yates, Komal Jalan, Lu Li, Eri Latorre-Chimonto, Duc Dunge Nguyen, Ken Durden, Praven Kallakuri, Yaxin Liu, Matthew Johnson, Tony Tsai, Alice Talbert, Jagamine Liu, Alexander Neitz, Chen Elkind, Marco Selvi, Mimi Jasarevic, Livio Baldini Soares, Albert Cui, Pidong Wang, Alek Wenijao Wang, Xinyu Ye, Krystal Kallarackal, Lucia Loher, Hoi Lam, Josef Broder, Daniel Niels Holtmann-Rice, Nina Martin, Bramandia Ramadhana, Daniel Toyama, Mrinal Shukla, Suojoy Basu, Abhi Mohan, Nicholas Fernando, Noah Friedel, Kim Paterson, Hui Li, Ankush Garg, Jane Park, Donghyun Choi, Diane Wu, Sankalg Singh, Zhishuai Zhang, Amir Globerson, Lily Yu, John Carpenter, Felix de Chaumont Quitry, Carey Radebough, Chu-Cheng Lin, Alex Tudor, Prakash Shroff, Drew Garmon, Dayou Du, Neera Vats, Han Lu, Shariq Iqbal, Alexey Yakubovich, Nilesh Tripuraneni, James Manyika, Haroon Qureshi, Nan Hua, Christel Ngani, Maria Abi Raad, Hannah Forbes, Anna Bulanova, Jeff Stanway, Mukund Sundaranjan, Victor Ungureanu, Colton Bishop, Yunje Li, Balaji Venkatramann, Bo Li, Chloe Thornton, Salvatore Scellato, Nikhesh Gupta, Yicheng Wang, Ian Tenney, Xihui Wu, Ashish Shenoy, Gabriel Carvajal, Diana Gage Wright, Ben Bariach, Zhuyun Xiao, Peter Hawkins, Sid Dalmia, Cl'ement Farabet, Pedro Valenzuela, Quan Yuan, Christopher A Welty, Ananth Agarwal, Mianna Chen, Wooycol Kim, Brrice Hulse, Nandita Dukikipati, Adam Paszke, Andrew Bolt, Elnaz Davoodi, Kiam Choo, Jennifer Beattie, Jennifer Prendki, Harsha Vashishst, Reca Santamaria-Fernandez, Luis C. Cobo, Jarek Wilkiewicz, David Madras, Ali Elqursh, Grant Uy, Kevin Ramirez, Matt Harvey, Tyler Lichetly, Heiga Zen, Jeff Seibert, Clara Huiyi Hu, A. Ya. Khordinia, Maigo Le, Asat Aharoni Megan Li, Lily Wang, Sandepe Kumar, Alejandro Lincce, Norman Casagrande, Jay Hoover, Dallia El Badaway, David Soergel, Denis Vuukov, Matt Miecnikowski, Jiri Simsa, Anna Koop, Praveen Kumar, Thibault Seldlam, Daniel Vlasic, Samira Daruki, Sir Shabat, John Zhang, Guolong Su, Kalpesch Krishna, Jaigeng Zhang, Jeremiahi Liu, Yi Sun, Evan Palmer, Alireza Ghaffarkhahah, Xi Xiong, Victor Cotutta, Michael Fink, Lucas Dixon, Ashwin Sreevarus, Adrian Goedeckemeyer, Alek Dimitiriev, Mohsen Jafari, Remi Crocker, Nicholas Fitzgerald, Aviral Kumar, Sanjay Ghemawat, Ivan Philips, Frederick Liu, Yannie Liang, Rachel Sterneck, Alena Repina, Marcus Wu, Laura Knight, Marin Georgeiev, Hyo Lee, Harry Askham, Abihishe Chakalam, Annie Louis, Carl Crous, Hardite Cate, Desste Petrova, Michael Quinn, Denese Owusu-Afriyvie, Achintya Singhal, Nan Wei, Solomon Kim, Damien Vincent, Milad Nasr, Christopher A. Choquette-Choo, Reiko Tojo, Shawn Lu, Diego de Las Casas, Yuchung Cheng, Tolga Bolukbaasi, Katherine Lee, Saaber Fateh, Rajagopal Anantharnayanan, Mityerap Patel, Charbel El Kaed, Jing Li, Jakub Synowski, Shreyas Rammohan Belle, Zhe Chen, Jaclyn Konzelmann, Siim Poder, Roopal Garg, Vinod Koverkathu, Adam Brown, Chris Dyer, Rosanne Liu, Azade Nova, Jun Xu, Junwen Bai, Slav Petrov, Demis Hassabis, Korrav Kavukcuoglu, Jeffrey Dean, Orirol Vinyals, and Alexandx Chronopoulou. 2024. Gemini 1.5: Undocking multimodal understanding across millions of tokens of context. ArXivabs/2403.05530.
* Song et al. (2019) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tiet-Yan Liu. 2019a. MASS: masked sequence to sequence pre-training for language generation. CoRRabs/1905.02450.
* Song et al. (2019) Kaito Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tiet-Yan Liu. 2019b. MASS: Masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of _Proceedings of Machine Learning Research_, pages 5926-5936. PMLR.
* Swadesh (1952) Morris Swadesh. 1952. Lexico-statistic dating of prehistoric ethnic contacts: With special reference to north american indians and esklmos. _Proceedings of the American Philosophical Society_, 96(4):452-463.
* Tanzer et al. (2024) Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazzi. 2024. A benchmark for learning to translate a new language from one grammar book.
* Vilar et al. (2023) David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2023. Prompting palm for translation: Assessing strategies and performance.
* Xu et al. (2024)Wenda Xu, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, Biao Zhang, Zhongtao Liu, William Yang Wang, Lei Li, and Markus Freitag. 2024. Fine-grained llm agent: Pinpointing and refining large language models via fine-grained actionable feedback.
* Zerva et al. (2024) Chrysoula Zerva, Frederic Blain, Jose G. C. De Souza, Diptesh Kanojia, Sourabh Deoghare, Nuno M. Guerreiro, Giuseppe Attanasio, Riccardo Rei, Constantin 

Orasan, Matteo Negri, Marco Turchi, Rajen Chatterjee, Pushpak Bhattacharyya, Markus Freitag, and Andre Martins. 2024: Findings of the quality estimation shared task at WMT 2024: Are LLMs closing the gap in QE? In _Proceedings of the Ninth Conference on Machine Translation_, pages 82-109, Miami, Florida, USA. Association for Computational Linguistics.
* Zhang et al. (2023) Biao Zhang, Barry Haddow, and Alexandra Birch. 2023. Prompting large language model for machine translation: A case study.
* Zhang et al. (2024) Chen Zhang, Xiao Liu, Jiuheng Lin, and Yansong Feng. 2024. Teaching large language models an unseen language on the fly.
* Zhang et al. (2021) Tong Zhang, Long Zhang, Wei Ye, Bo Li, Jinan Sun, Xiaoyu Zhu, Wen Zhao, and Shikun Zhang. 2021. Point, disambiguate and copy: Incorporating bilingual dictionaries for neural machine translation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3970-3979, Online. Association for Computational Linguistics.
* Zhu et al. (2024) Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024. Multilingual machine translation with large language models: Empirical results and analysis.

## Appendix A Lexical items for recall metric

### LLM prompts

#### Direct translation prompt with entire lexicon

You are asked to translate the text below into {target_language_name}.

Note the following translations:

\[\text{source\_word}_{1}\] means \[\text{target\_word}_{1}\] \[\text{source\_word}_{2}\] means \[\text{target\_word}_{2}\] ... \[\text{source\_word}_{n}\] means \[\text{target\_word}_{n}\]

Please output only the translation of the text without any other explanation.

\[\text{source\_language\_name}\]: \[\text{source\_text}\] \[\text{target\_language\_name}\]:

#### Post-editing prompt with no lexical information

You are asked to edit the following translation from {source_language_name} into {target_language_name}. The proposed translation is high-quality, but may have some incorrect words.

Please output only the translation of the text without any other explanation.

\[\text{source\_language\_name}\]: \[\text{source\_text}\] \[\text{target\_language\_name}\]: \[\text{MT\_output}\]

#### Post-editing prompt with lexical information (whole lexicon or subset)

You are asked to edit the following translation from {source_language_name} into {target_language_name}. The proposed translation is high-quality, but may have some incorrect words.

Note the following translations:

\[\text{source\_word}_{1}\] means \[\text{target\_word}_{1}\] \[\text{source\_word}_{2}\] means \[\text{target\_word}_{2}\] ... \[\text{source\_word}_{n}\] means \[\text{target\_word}_{n}\] Please output only the translation of the text without any other explanation.

#### Source_language_name

: {source_text}\] \[\text{target\_language\_name}\]: \[\text{target\_language\_name}\]: \[\text{MT\_output}\]
 



\begin{table}
\begin{tabular}{c c c c c c} \hline
**Animals** & **Common numbers** & **Colors** & **Rarer numbers** & **Weekdays** & **Months** \\ \hline cat & two & black & eighteen & Monday & January \\ chicken & three & white & eighty & Tuesday & February \\ frog & four & red & fifteen & Wednesday & March \\ bird & five & blue & fifty & Thursday & April \\ bee & six & yellow & forty & Friday & May \\ fish & seven & green & forty-two & Saturday & June \\ horse & eight & purple & fourteen & Sunday & July \\ goat & nine & orange & nineteen & August \\ elephant & ten & grey & ninety & September \\ butterfly & hundred & seventeen & October \\ dog & million & seventh & November \\ deer & & sixteen & December \\ bear & & sixth & & \\  & & ten & & \\  & & ten thousand & & \\  & & thirteen & & \\  & & twenty-one & & \\  & & zero & & \\  & eleven & & & \\ twelve & & & & \\ \end{tabular}
\end{table}
Table 5: Words used for our recall metric for evaluating the prevalence of in-domain lexical confusion.





\begin{table}
\begin{tabular}{l c c c c|c c c c} \hline \hline  & \multicolumn{6}{c}{**xx\(\rightarrow\)en**} \\ \hline  & \multicolumn{4}{c|}{**MT output**} & \multicolumn{5}{c}{**Post-edited**} \\  & **Correct** & **Confusion** & **Unknown** & **ChrF** & **Correct** & **Confusion** & **Unknown** & **ChrF** \\ \hline

[MISSING_PAGE_POST]

k** & 75.7 & 1.9 & 22.4 & 66.1 & 80.5 & 1.9 & 17.6 & 65.0 \\ \hline \end{tabular}
\end{table}
Table 6: Lexical recall and ChrF scores before and after post-editing for translation into English. The languages whose codes are highlighted in blue constitute the top 20% with the highest confusion scores, before editing. These are reported on as “high-confusion languages” elsewhere.





\begin{tabular}{l c c c c|c c c c} \hline  & \multicolumn{4}{c|}{**MT output**} & \multicolumn{5}{c}{**Post-edited**} \\  & **Correct** & **Confusion** & **Unknown** & **ChrF** & **Correct** & **Confusion** & **Unknown** & **ChrF** \\ \hline
**bjn** & 75.4 & 0.4 & 24.2 & 65.3 & 76.6 & 0.4 & 23.0 & 66.5 \\

[MISSING_PAGE_POST]

j** & 79.7 & 1.1 & 19.3 & 71.9 & 81.8 & 0.9 & 17.3 & 73.5 \\ \hline \end{tabular}





\begin{tabular}{l c c c|c c c c c} \hline  & \multicolumn{4}{c|}{**MT output**} & \multicolumn{5}{c}{**Post-edited**} \\  & **Correct** & **Confusion** & **Unknown** & **ChrF** & **Correct** & **Confusion** & **Unknown** & **ChrF** \\ \hline

[MISSING_PAGE_POST]

e** & 67.8 & 6.9 & 25.3 & 60.3 & 72.4 & 2.9 & 24.6 & 61.7 \\ \hline \end{tabular}





\begin{table}
\begin{tabular}{l c c c|c c c c c} \hline  & \multicolumn{4}{c|}{**MT output**} & \multicolumn{5}{c}{**Post-edited**} \\  & **Correct** & **Confusion** & **Unknown** & **ChrF** & **Correct** & **Confusion** & **Unknown** & **ChrF** \\ \hline
**vec** & 79.3 & 1.2 & 19.4 & 69.9 & 79.1 & 1.2 & 19.6 & 71.4 \\
**war** & 71.7 & 0.4 & 28.0 & 75.5 & 72.4 & 0.2 & 27.5 & 75.5 \\
**wo** & 48.8 & 2.0 & 49.1 & 41.7 & 48.1 & 1.4 & 50.5 & 42.1 \\
**yua** & 52.1 & 2.1 & 45.8 & 42.7 & 52.9 & 2.9 & 44.1 & 44.4 \\
**zap** & 19.5 & 3.3 & 77.1 & 22.3 & 21.4 & 3.8 & 74.8 & 22.9 \\ \hline
**Average** & **59.3** & **3.8** & **36.9** & **52.4** & **62.4** & **2.8** & **34.8** & **53.2** \\ \hline \end{tabular}
\end{table}
Table 7: ChrF scores before and after post-editing for translation out of English. The languages whose codes are highlighted in blue constitute the top 20% with the highest confusion scores before editing, in the into-English direction. These are reported on as “high-confusion languages” elsewhere.





\begin{tabular}{l c r} \hline  & **Pre-edit ChrF** & **Post-edit ChrF** \\ \hline
**crs** & 69.2 & 69.8 \\
**ctg** & 33.0 & 34.1 \\
**cv** & 49.6 & 48.5 \\
**din** & 25.6 & 26.4 \\
**dov** & 41.0 & 41.5 \\
**dyu** & 22.3 & 22.4 \\
**dz** & 43.0 & 43.8 \\
**fa-AF** & 48.2 & 46.7 \\
**ff** & 32.4 & 31.3 \\
**fj** & 60.5 & 60.2 \\
**fo** & 56.5 & 57.6 \\
**ton** & 26.1 & 25.9 \\
**fur** & 60.4 & 61.7 \\
**gaa** & 48.8 & 48.5 \\
**gv** & 22.9 & 24.0 \\
**hil** & 63.7 & 63.6 \\
**hne** & 57.2 & 56.2 \\
**hrx** & 47.5 & 51.3 \\
**iba** & 45.2 & 44.6 \\
**jam** & 60.7 & 55.2 \\
**kac** & 43.5 & 44.2 \\ kbd & 36.8 & 40.4 \\ kek & 31.9 & 35.1 \\
**kg** & 50.2 & 51.0 \\ kha & 54.3 & 57.0 \\
**kl** & 42.4 & 43.6 \\
**kr** & 32.8 & 33.3 \\
**ks-Deva** & 33.8 & 25.0 \\
**ks** & 24.0 & 34.7 \\
**ktu** & 63.2 & 64.7 \\
**kv** & 39.9 & 42.0 \\
**li** & 55.0 & 54.1 \\
**ljj** & 57.4 & 58.0 \\
**lmo** & 39.2 & 40.2 \\
**ltg** & 64.0 & 63.8 \\
**lu** & 24.7 & 24.5 \\
**luo** & 41.2 & 41.5 \\
**mad** & 40.7 & 40.6 \\
**mak** & 44.9 & 46.3 \\
**mm** & 28.8 & 25.9 \\
**mpe** & 66.5 & 66.3 \\
**mn** & 42.1 & 41.4 \\
**min** & 58.6 & 59.4 \\
**ms-Arab** & 66.2 & 59.9 \\
**mrr** & 36.8 & 36.4 \\
**nd** & 41.8 & 43.2 \\
**ndc-ZW** & 27.9 & 29.6 \\
**new** & 37.4 & 36.9 \\
**nhe** & 38.6 & 41.2 \\
**nr** & 58.8 & 57.2 \\ \end{tabular}





\begin{tabular}{l c r} \hline  & **Pre-edit ChrF** & **Post-edit ChrF** \\ \hline
**nus** & 32.5 & 30.6 \\
**oc** & 68.3 & 69.5 \\
**os** & 45.9 & 46.2 \\
**pa-Arab** & 43.3 & 45.1 \\
**pag** & 53.0 & 53.0 \\
**pam** & 47.7 & 47.3 \\
**pap** & 66.1 & 68.1 \\
**quc** & 24.7 & 25.3 \\
**rhg-Latin** & 20.6 & 24.0 \\
**rn** & 44.9 & 45.5 \\
**rom** & 37.0 & 36.4 \\
**sah** & 46.9 & 48.7 \\
**sat-Latin** & 22.8 & 24.4 \\
**scn** & 51.9 & 53.0 \\
**se** & 46.8 & 48.8 \\
**sg** & 30.5 & 31.1 \\
**shn** & 40.7 & 39.6 \\
**ss** & 56.2 & 55.9 \\
**sus** & 34.9 & 28.6 \\
**szl** & 59.2 & 59.5 \\
**tcy** & 39.1 & 40.9 \\
**tet** & 60.0 & 59.8 \\
**tiv** & 26.3 & 27.1 \\
**tn** & 55.8 & 55.7 \\
**to** & 52.0 & 54.6 \\
**tipi** & 51.9 & 52.3 \\
**trp** & 36.5 & 40.6 \\
**tum** & 44.7 & 45.0 \\
**ty** & 56.6 & 54.8 \\
**tyv** & 43.1 & 44.7 \\
**udm** & 45.9 & 46.2 \\
**ve** & 55.6 & 52.1 \\
**vec** & 55.4 & 54.7 \\
**war** & 61.8 & 63.0 \\
**wo** & 29.8 & 29.3 \\
**yua** & 38.5 & 39.5 \\
**zap** & 17.8 & 18.3 \\ \hline
**Average** & **43.4** & **43.8** \\ \end{tabular}