Venue: NAACL 2024

Fixing Rogue Memorization in Many-to-One Multilingual Translators of Extremely-Low-Resource Languages by Rephrasing Training Samples

Paulo Cavalin\({}^{1}\), Pedro H. Domingues\({}^{2}\), Claudio Pinhanez\({}^{1}\), and Julio Nogima\({}^{1}\)

\({}^{1}\)IBM Research, Brazil

\({}^{2}\)Pontificia Universidade Catolica do Rio, PUC-Rio, Brazil

###### Abstract

In this paper we study the fine-tuning of pre-trained large high-resource language models (LLMs) into many-to-one multilingual machine translators for extremely-low-resource languages such as endangered Indigenous languages. We explore those issues using datasets created from pseudo-parallel translations to English of _The Bible_ written in 39 Brazilian Indigenous languages using mBART50 and WMT19 as pre-trained models and multiple translation metrics. We examine bilingual and multilingual models and show that, according to machine translation metrics, same-linguistic family models tend to perform best. However, we also found that many-to-one multilingual systems have a tendency to learn a "rogue" strategy of storing output strings from the training data in the LLM structure and retrieving them instead of performing actual translations. We show that rephrasing the output of the training samples seems to solve the problem.

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 4503–4514

June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

With the explosion in popularity of _Large Language Models (LLMs)_, there is currently a _de facto standard_ to create _machine translators (MTs)_ for low-resource languages: take one LLM pre-trained on a large corpus with self-supervised techniques and fine-tune it using a much smaller parallel downstream corpus in that language Lee et al. (2022). That usually results in better translation accuracy than with a model trained from scratch but with limited data Adelani et al. (2022). For under-resourced languages, it is also common to improve translation quality even further by the use of data from multiple languages and large multilingual models Saleh et al. (2021).

In this work we show experimentally that the reality of creating translators for extremely low-resource languages is, unfortunately, much more difficult. Notably, we show that when performing such "standard" fine-tuning of multilingual LLMs, we have to be very careful on interpreting results which rely only on average scores of reference-based translation quality metrics. We present a case where improved averaged measurements, instead of signifying better machine translators, are in fact the result of a "rogue" model which stores outputs from the training data and retrieves them as canned outputs. In other words, the fine-tuned model hallucinates translations by generating texts which are disconnected to the input: they are perfectly-memorized reproductions of training samples. Because the MT sometimes succeeds in producing a "perfect", high-score with a memorized answer, the average score of the model is inflated, higher than other approaches, although the model is, in fact, totally inappropriate for the actual task.

We present strong evidence that the memorization side-effect is a result of the inclusion of repetitions of target texts in training set, which is an artifact of many-to-one translators, since the memorization effect directly increases with the number of languages in the multilingual model. We then show that rephrasing the target text helps to avoid memorization without affecting translation quality and that is a viable solution for many other applications since now is feasible to create rephrasings of texts with LLMs.

Our findings come in a context where multilingual fine-tuning of LLMs is often used when extremely-low-resource languages are involved. In our experiments we fine-tuned pre-trained LLMs to make MTs from 39 _Brazil Indigenous languages (BILs)_ to English1. Those languages are typical of scenarios where there is almost no data available for training but we can find translations of official documents or religious texts to fine-tune 

existing models, that is, often with many-to-one or one-to-many mappings of the same texts to a higher-resourced language such as English.

For our experimental study, we compiled a parallel corpus with pseudo-alignments of verses from _The Bible_, and fine-tuned two commonly-used LLMs, i.e. _mBART50_ and _WMT19_, relying on one bilingual and two multilingual fine-tuning strategies. The results are evaluated with three standard reference-based metrics for machine translation, i.e. the \(n\)_-gram_ based BLEU score and the trained neural metrics named BLEURT and BERTScore.

To quantify the memorization behavior and to provide evidence that our many-to-one multilingual MTs became content retrievers, we implement a memorization metric based on finding the smallest distance of the generated text to target sentences in the training set, in this case the verses from The Bible in English. Our metric suggests that the problem worsens as more languages are added to the training set. We then show that, by using rephrases of the output verses from 10 different versions of Ghe Bible in English, we can build a multilingual translator for 10 BILs of the same linguistic family in which the memorization issues virtually disappear.

The main contribution of this paper is to show that automatic evaluation metrics can be deceiving in extremely-low-resource scenarios and how important it is to perform different forms of evaluation of the outputs, including qualitative inspections. This paper also shows that strictly following well-known approaches from the NLP literature, such as multilingual training, can result in models with undesired memorization effects. We also propose and demonstrate that such undesired behaviour can be fixed with rephrasings which can be easily generated with current days' LLMs.

## 2 Related Work

Most of the recent impressive results of NLP have been observed with English and a few other languages. Despite efforts in expanding such research for other languages [3, 4, 6], results have been limited, covering at best about 200 languages (NLLB Team et al., 2022), leaving behind almost 7,000 other languages. In Brazil, none of the about 200 Indigenous languages spoken in the country are covered by LLMs. Many of the Brazilian Indigenous Languages (BILs), similarly to other extremely-low-resource languages in the world, are endangered, facing the challenge of disappearing in just a few decades from today. Pinhanez:2023:EMNLP provide a good discussion on why and how AI can contribute to the survival of such languages, and at same, the benefits to AI of working with endangered languages.

One big challenge in scaling up the current AI progress to more languages is the data-hunger needs of current NLP models. Since the vast majority of the languages in the world are low-resourced or extremely-low-resourced, one hope to overcoming this data-scarcity issue is the use of self-supervised pre-trained models and _fine-tune_ such models to downstream tasks such as MT. In that case, the general-purpose language knowledge of the pre-trained model is transferred and reused in a context of a much smaller training set through fine-tuning of its parameters [8].

Another way to circumvent the lack-of-data problem is to use _multilingual models_, where data from several languages are combined [3, 1]. The main assumption is that multilingual models can leverage the shared linguistic characteristics of related languages, expanding the utility of very small datasets. But the non-deliberated use of extra languages may worsen the performance, a phenomenon usually called _negative transfer_[15].

A more particular approach, common in some extremely-low-resource languages such as Indigenous languages, is to rely on multi-way multilingual corpora, in either many-to-one or one-to-many directions [1, 2]. The same monolingual text, translated to many languages, is used in the training set multiple times as either source or target text [16]. Typically those texts are official documents and religious texts, such as The Bible [16, 17, 18, 19]. One effect of such approach is that samples appear repeatedly in the training set, either as source or target texts, what can trigger a known problem in LLMs: memorization [10].

Memorization in LLMs consists of generating identical or nearly-identical reproduction of training samples during inference time. Notice that memorization can be beneficial for some applications, such as closed-book Q&A, and it is not necessarily a type of overfitting [14]. But in terms of machine translation, given that it 

has been shown that removing duplications is beneficial in terms of improving translation quality of MT systems Lee et al. (2022); Ramirez-Sanchez et al. (2020), we explore here whether the negative impact of duplications on translation quality is due to the memorization of duplicated training samples. Together with other factors such as model size and vocabulary size Kharitonov et al. (2021), duplication of examples in the training set is known to sometimes produce memorization Zhang et al. (2021); Carlini et al. (2023).

## 3 Translating Indigenous Languages

Creating machine translators and other NLP tools with extremely-low data resources is of key importance for Indigenous languages and in particular for the about 2,800 languages which are in danger of disappearing in the world Moseley (2010). Modern NLP technology tools may not only be a way to contribute to the vitalization of those languages but also paths to propagate the culture and support the political rights of their communities Mager et al. (2018); Zhang et al. (2022); Liu et al. (2022); Pinhanez et al. (2023).

In this paper we explore particular issues and difficulties faced by the construction of _machine translators_ for languages which are spoken by hundreds or at most thousands of people, often in semi-isolated conditions and with extremely low presence in the web, making web crawling and crowdsourcing unfeasible ways to gather data.

Our main source of data are the many translations to Indigenous languages of the The Bible. Since it is structured in numbered verses, it is relatively easy to create quasi-parallel datasets (that is, the translations may differ in style), as done in other works Mayer and Cysouw (2014); Adelani et al. (2022). It is possible to use The Bible as data for many of the BILs, since it has been translated into many of those languages by Christian churches Franchetto (2008). We understand that there are important ethical, historical, and cultural issues around the use of The Bible as a source of data for Indigenous languages, which we discuss in detail in Section 9 at the end of the paper.

### Brazilian Indigenous Languages

Brazil was home to about 270 Indigenous languages according to the Census of 2010, the last comprehensive assessment of linguistic diversity in Brazil IBGE (2010)2. Those languages were spoken by approximately 800 thousand people (IBGE, 2010), half of them living in Indigenous lands. Storto (2019) provides a good overview of the history, structure, and characteristics of BILs. Almost all of those languages are considered endangered Moseley (2010). We adopted here the Indigenous language classification, nomenclature, and data from the 2010 Brazilian Census by IBGE IBGE (IBGE, 2010) and language acronyms according to ISO 639-3.

Footnote 2: There is some discussion about the accuracy of those numbers, see Franchetto (2020); Storto (2019).

### The _The Bibles_ Dataset

We consider in this work 39 Indigenous languages spoken in Brazil of which we were able to find translations for the _New Testament_ of The Bible, a book which comprises about 7,000 verses in its English versions. Table 1 lists the 39 Indigenous languages used in this work, comprising 36 spoken primarily in Brazil and 3 other Guarani languages used mostly in Paraguay and Bolivia but also spoken in some areas in Brazil.

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c|c}  & \multicolumn{5}{c|}{**Agapsed sentences**} \\ \hline
**Name** & **Local Bass** & **Rank** & **Density** & **Mean** & **Total** & **Total** \\ \hline
**Boros** & 100 & 10000 & 10000 & 1000 & 1000 & 200 & 200 \\ \cline{1-1} \cline{3-8}  & against & Mean & 10000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & starting & Mean & 10000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & starting & Mean & 10000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & starting & mean & 10000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & starting & mean & 10000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 10000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 10000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 10000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 10000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 10000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 10000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 10000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 100000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 100000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 100000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 100000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 100000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 100000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 100000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 100000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 100000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 100000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 100000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 100000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 100000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 100000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 100000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}  & descending & mean & 100000 & 10000 & 10000 & 10000 & 10000 & 10000 \\ \cline{1-1} \cline{3-8}

 

This dataset, henceforth called _The Bibles_, was obtained in its majority from the _ebible_ website3. A few other languages of our dataset were obtained from the _YouVersion_ online platform4. The Bibles dataset consists of 188,033 parallel verses from the New Testament in English and in the 39 Indigenous languages listed in table 1. The parallelism among translations of the same verse, performed by the authors, has a reasonable quality although we are aware that the source of the translations comes from different versions of The Bible in several languages and has diverse narrative styles.

Footnote 2: [https://developers.com/en-GB/](https://developers.com/en-GB/)

To avoid cross-contamination in the decoder and allow us to study memorization issues without data leakage between the training and test sets, we used the _Matthew_ chapter from the New Testament as the source of test set and the remainder as the training set. We are aware that there are some similarity among verses could happen between the book of Matthew and the other synoptic gospels, but we see the existence of some similarity as positive, since in most practical multi-language training setup there will be some level of similar sentences.

The resulting training sets for the fine-tuning strategies vary in size (see Table 1). The bilingual models were fine-tuned, on average, with 4,160 pairs of sentences. To fine-tune Tupi-Guarani family models, the training set comprised 43,869 sentence pairs, and for the all languages models, 162,225 training pairs were used for fine-tuning.

## 4 Experimental Study

In this section we describe the methodology and results of fine-tuning LLMs into machine translators for the 39 BILs presented in Table 1. The translation direction is always from a BIL to English.

### Methodology

We employed two distinct LLMs in the fine-tuning process. The first LLM is m**BART50**Tang et al. (2020), which is an extended version of _mBART_Liu et al. (2020). With 680M parameters, mBART50 is pre-trained with masked language modeling on 203M sentences. This model is a common choice for training multilingual MTs for low-resource languages Lee et al. (2022a); Chen and Abdul-Mageed (2022).

The second LLM is **WMT19**Ng et al. (2019), which is a 315M-parameter German-to-English machine translator pre-trained on about 28M pairs of translated sentences and more than 500M back-translated sentences. This model is not as popular as mBART in the MT literature but was chosen because of its smaller size and good performance in decoding English texts.

We downloaded the models from _HuggingFace56_. Both were fine-tuned with a learning rate of \(2^{-5}\) and a batch size of 16. mBART50 was trained for 4 epochs and WMT19 for 5.

Footnote 5: [https://huggingface.co/facebook/mbart-large-50](https://huggingface.co/facebook/mbart-large-50)

Our evaluation considers three different fine-tuning strategies, that result in three different types of models, and two groups of test sets. First, we consider **bilingual (BL)** models, created by fine-tuning each of the LLMs on source-to-target pairs exclusively from each of the BILs on Table 1, yielding 39 different bilingual models. Second, we consider the extreme multilingual case where we finetune each of the LLMs with **all languages (AL)** at once. Third, we created in-between multilingual solutions, the **Tupi-family (TF)** models, where the training set comprises only the 10 languages belonging to the Tupi-Guarani family: Guarani of Paraguay, Bolivia (2), Kayowa, and Mbya; Guajajara, Ka'apor, Kaiabi, Nheengatu, and Tenharim.

With the goal of measuring the impact of the previously-mentioned models, we defined two distinct sets of experiments. The first one, **BL39 vs AL39**, considering all 39 BILs in a single test set, so that we can compare the impact of a bilingual model against a multilingual one trained with all the languages, i.e. BL vs AL. The second set is **BL10 vs TF10 vs AL10**, where we focus in comparing the BL models not only against AL but also against TF which is more targeted at Tupi-family languages. For the latter, the test set contains only the 10 languages used to train the TF models.

We used three metrics to evaluate the results, combining the traditional **BLEU** score Papineni et al. (2002) with more recent neural-based metrics which are considered to be more robust and better correlated with human scores Freitag et al. (2022). For BLEU, we compute the average of sentence-level BLEU scores7 computed with the SacreBLEU Python package Post (2018). For the 

neural-based metrics, we consider **BLEURT**(Sellam et al., 2020) and **BERTScore**(Zhang et al., 2020).

### Study Results

Table 2 shows the average and standard deviation scores in the different test sets of the languages for the fine-tuning of mBART50 and WMT19 to different fine-tuning strategies, for all three metrics.

Overall, mBART50 seems to perform slightly better than WMT19 but with higher standard deviation, and the three metrics afford similar results comparatively. The three models yield similar results but the bilingual ones present smaller standard deviation when all languages are considered. The TF10 model seems to perform slightly better than the bilingual models but with a higher standard deviation. As we show in the next section, high standard deviations are, in fact, a symptom that the translation model has started to perform what we call _rogue memorization_, that is, it has become a retriever of the contents of the training set.

A complementary view on the effects of the high standard deviations can be observed with the distribution of BLEU scores computed with samples from the training set, as show in Figure 1. From these distributions we can clearly observe that the BL39 and BL10 models (both for mBART50 and WMT19) usually result in right-skewed normal distributions, while for AL39 and AL10 the shapes of the distributions resemble more binomial distributions. We can also observe some shift to a binomial distribution in the TF10 model in the case of mBART50.

## 5 Memorization Issues

By examining the actual output of the mBART50-AL39 model, we saw cases where the output trans

\begin{table}
\begin{tabular}{l|r r r r r r r r r r} \hline  & \multicolumn{4}{c}{mBART} & \multicolumn{4}{c}{WMT19} \\ \cline{2-17}  & BLEURT & std & BERTScore & std & BLEU & std & BLEU & std & BERTScore & std & BLEU & std \\ \hline BL10 & 0.368 & 0.079 & 0.876 & 0.025 & 9.520 & 8.554 & 0.346 & 0.068 & 0.858 & 0.031 & 6.265 & 6.273 \\ \hline TF10 & **0.385** & 0.116 & **0.879** & 0.036 & **12.325** & 16.218 & **0.346** & 0.077 & **0.861** & 0.030 & **6.626** & 8.188 \\ \hline AL10 & 0.323 & 0.095 & 0.859 & 0.029 & 6.393 & 11.345 & 0.314 & 0.083 & 0.854 & 0.027 & 5.312 & 10.061 \\ \hline BL39 & **0.343** & 0.070 & **0.867** & 0.025 & **7.152** & 6.653 & **0.327** & 0.063 & 0.845 & 0.032 & **4.618** & 4.901 \\ \hline AL39 & 0.316 & 0.087 & 0.856 & 0.027 & 5.525 & 9.963 & 0.307 & 0.073 & **0.851** & 0.024 & 4.422 & 8.171 \\ \hline TF10 to BL10 & **5%** & 47\% & **0\%** & 43\% & 29\% & 90\% & **0\%** & 13\% & **0\%** & -2\% & **6\%** & 31\% \\ \hline AL to BL10 & -12\% & 20\% & -2\% & 16\% & -33\% & 33\% & -9\% & 21\% & 0\% & -14\% & -15\% & 60\% \\ \hline Al39 to BL39 & -8\% & 24\% & -1\% & 9\% & -23\% & 50\% & -6\% & 15\% & **13\%** & -24\% & -4\% & 67\% \\ \hline \end{tabular}
\end{table}
Table 2: Summary of the average and standard deviation scores in different test sets of languages for the fine-tuning of mBART50 and WMT19 to different models for all metrics.

\begin{table}
\begin{tabular}{l l|l l} \hline
**MODEL** & **LANGAGE** & **ORBRIMAL** & **COMPEDED** & **GENRATED** & **GEMRATED VERS** \\ \cline{2-5}  & & day is then please ask you... each smartphone's & has said too far, what do your use applied, guess said to him, friends have done and the kids \\  & & xavier distance lying video like, tans, hairs taken using kototto human, taking hair, tans or jogging head and one at your left, in your kingdom, place to fly his head. & Uake 95.8 \\ \hline nBART50-AL39 & Aguil & worktone vehicle sensors, torso behaviour & wrtg head and one at your left, in your kingdom, & 1 tell you he truth, this connection will not pass \\  & & write top player swim, men, Muse angle & wrtg head and vice versa & prevolential the whole world, what she has & many surf at these things take pilots. \\ \hline nBART50-AL53 & Mawel & Mawel & be distributed here, a neural versus & toure stillobe fed in memory of her. & dweird silo the fold in memory of her. \\ \hline nBART50-AL6 & Nhomegrist & and antirowisthows ikages grow up the environment & his head was brought on a printer and given to the gif, and give brought it to her mother, nobody out of the vineyard. & no they retired him, Allied him, and there’s \\  & & turning space else how he knowhands, cost away ee & the gif, and give brought it to her mother, & nobody out of the vineyard. & Maui 12.8 \\ \hline nBART50-AL8 & Mountain & warstrace positive minimumaries, (dis Ins 

lation was not only an incorrect translation, but a literal verse of The Bible, easily found through a search procedure in the Internet. We saw similar cases in the mBART50-TF10 model and in the WMT19-AL39. Examples of some mBART50 cases are shown in Table 3. As shown in the table, instead of attempting to translate the input string, the fine-tuned systems produce literal verses totally unrelated to the expected output.

Another issue we could see in this qualitative analysis was that there were many exact copies of output sentences. Translation from different languages of sentences with the same meaning and structure is expected to generate very diverse outputs. We saw the occurrence of repetitions as evidence of rogue memorization. The extent of the problem is evidenced by the results of the analysis presented in Table 4, which shows for each model the number of distinct outputs. For instance, in the case of the experiments with the 39 BILs (AL39 and BL39), which comprised 25,808 different inputs from the samples, we would expect about the same number of distinct outputs, as discussed above. However, in the mBART50-AL39 model, only 3,951 distinct outputs were produced, about 15% of what would be reasonable to expect, configuring a ratio of about 7 inputs mapping to exactly the same output (\(7:1\)). The WMT19-AL39 model yielded a similar ratio (\(6:1\)). Notice that the issue is not present in the bilingual models but slightly visible in the mBART50-TF10 model.

To more precisely quantify the extent of the memorization by the models, we implemented a metric to quantify memorization, based on computing the smallest distance of a generated output to a sample from the training. The assumption is that, when memorization occurs, the decoder reproduces a training sample, which in this case is one verse of The Bible, even when the translation-quality metric presents a low value. For that, we compute the Euclidean distance between the output and the references of all training samples, and return the smallest distance as the measured value. The distance is computed on a semantic representation of the texts, using the _Sentence Transformers_ Python library8, using the _all-MiniLM-L6-v2_ model9. With that we are computing even soft memorization effects since we are not relying on a exact match as usually employed in previous works Tirumala et al. (2022); Carlini et al. (2023).

Footnote 8: [https://huggingface.co/sentence-transformers](https://huggingface.co/sentence-transformers)

Footnote 9: [https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)

In Figure 2 we present the distribution of our memorization metric considering a training examples as inputs. In those plots, the further the distribution is shifted to the left, the stronger is the production of memorized literal verses. The results with *-AL39 (mBART50-AL39 and WMT19-AL39) and *-AL10 evaluations, compared with *-BL39 and *-BL10, clearly show that the multilingual model strongly memoriz

Figure 1: Distribution of BLEU scores of samples from the training set for mBART50 (top) and WMT19 (bottom).

\begin{table}
\begin{tabular}{|l|c|c|c|c|c|c|} \hline  & \multicolumn{3}{c|}{**mBART50**} & \multicolumn{3}{c|}{**WMT19**} \\ \cline{2-7}
**MODEL** & distinct & distinct & inputs to & distinct & distinct & inputs to & distinct \\  & outputs & expected & output & outputs & expected & output \\ \hline AL39 & 3951 & 15\% & 7:1 & 4494 & 17\% & 6:1 \\ \hline BL39 & 24351 & 94\% & 1:1 & 24001 & 93\% & 1:1 \\ \hline expected & 25808 & 100\% & 1:1 & 25808 & 100\% & 1:1 \\ \hline AL10 & 2452 & 33\% & 3:1 & 2506 & 33\% & 3:1 \\ \hline TF10 & 6405 & 85\% & 1:1 & 7219 & 96\% & 1:1 \\ \hline BL10 & 7057 & 94\% & 1:1 & 7297 & 97\% & 1:1 \\ \hline expected & 7498 & 100\% & 1:1 & 7498 & 100\% & 1:1 \\ \hline \end{tabular}
\end{table}
Table 4: Number of distinct outputs and proportions of input to distinct outputs for the different models.

 

With the bilingual models, the mean minimum distance ranges from 0.62 with mBART50-BL10 to 0.66 with WMT19-BL39, while with the all language models, the mean distance falls to a range between 0.14 with mBART50-AL10 to 0.16 with WMT19-AL10. With TF10 models, the memorization effect gradually increases with the number of languages in the training set, since this model presents mean values of 0.58 and 0.36 for WMT19-TF10 and mBART-TF10, respectively. Interestingly, with the latter we also observe a slightly bimodal distribution, indicating that the peak of the distribution is in the middle of a shifting process between the mBART50-BL10 and mBART50-AL10.

We conducted a similar evaluation considering the outputs generated from inputs of the test set. The results are displayed in Figure 3. Surprisingly, we observe quite similar results to what we observed with the training set. That is, the results with the all languages models present strong evidence of producing memorized verses, with mean minimum distances to an output from the training set ranging from 0.14 with mBART50-AL39 to 0.17 with WMT19-AL39, while the bilingual models presents a much higher range of mean values, from 0.62 with mBART50-BL39 to 0.64 with WMT19-BL39. Once again, both mBART50-TF10 and WMT19-TF10 present some in-between tomemorization transition, with mean values of 0.59 and 0.47, respectively. What is impressive, from these results on the test set, is that it seems that the memorization occurs in the decoder side of the Transformer architecture. And when the models are doing heavy memorization, there is some disconnection to the encoders. That is, _no matter the input sentence, the model simply generates one of the sentences it had memorized_.

## 6 Using Rephrasings to Overcome Memorization Issues

In this section we investigate whether the use of rephrasings instead of the repeated outputs can al

Figure 3: Histograms with the frequency of the minimum distance of a generated translations of test samples to examples in the training set.

Figure 2: Histograms with the frequency of the minimum distance of a generated translations of training samples to examples in the training set.

 

leviate the memorization issue. As we saw, the inclusion of repeated outputs results in models that memorize those outputs as the number of repetitions increase, i.e. as the number of languages in the multilingual training set increase. Thus, by using outputs that are similar in semantics but which are not exact duplicates, we evaluate the impact on MT quality metrics and on memorization.

For this investigation we rely on rephrasings of The Bible in English written by different authors. Although we could have used an LLM such as _ChatGPT_ and the like to generate as much rebrnsings as possible, we relied on different versions of the Bible translated only by humans to avoid errors which could be introduced by LLMs. Given that translations of The Bible are not so abundant, we were able to collect 10 rephrasings which were then used to evaluate the impact on the mBART50-TF10 model, which is based on 10 different languages.

The results were very positive. The scores (and respective standard deviations) achieved by this model are of 0.408 (\(\pm\)0.085), 0.887 (\(\pm\)0.029), and 10.432 (\(\pm\)10.123), for BLEUR, BERTScore, and BLEU, respectively. That represent gains of 6% and 1% with BLEURT and BERTScore (compared with 0.385 and 0.879, respectively, without rephrasing), and a loss of 15% with BLEU (12.325 without rephrasing). Notice that, despite the loss in BLEU, we see a decrease of 38% in the standard deviation with that metric (from 16.218 to 10.432), and of 27% and 20% with BLEURT and BERTScore (from 0.116 and 0.036, respectively, to 0.085 and 0.029). We strongly believe that those decreases in standard deviations are directly related to the decrease of memorization, as we show next.

To show the reduction in memorization, we present in Figure 4 the plots of the memorization-effect metric comparing the outputs generated by the mBART50-TF10r model and the previous mBART50-TF10, considering both training and test samples as input sentences. Considering the test samples, as shown in Figure 4d, this evaluation resulted in a mean distance of 0.61 from a generated output to a training sample. The same evaluation considering the non-rephrased related model, i.e. mBART50-TF10, presented in Figure 4c, resulted in a mean distance of 0.47. With the outputs generated from training inputs, the difference is even more drastic. The mean distance for the non-rephrased model is of 0.36 (fig. 4a), while the model trained on rephrased sentences presents a mean distance of 0.90 (fig. 4b). We can see in the latter a very clear peak towards the right side of the plot, evidence that memorization is not happening.

Another evidence of decrease in memorization was obtained by computing the number of distinct outputs using the same methodology used to produce table 4. The mBART50-TF10 model produced 6,405 distinct outputs, about 85% of the ideal number of 7,498, while the model using rephrasings mBART50-TF10r generated 7464 distinct outputs, about 99.5% of maximum possible, and better than any other model we tested.

## 7 Final Discussion

The first main contribution of this work is to present a clear case of fine-tuning LLMs to many-to-one translators where high scores in metrics such as BLEU can be misleading, since high values come, in fact, from a rogue strategy of retrieving verses stored from the training set. We also show that common symptoms of this problem are high standard deviations of the metrics (which, therefore, should be always reported), bimodal distributions of results, and low numbers of distinct outputs. We also present a method to compute how close the output is to translations identical to the output texts of the training sets which can also be used to identify the undesirable fine-tuning of translators into rogue retrievers of stored training data.

The second contribution is to present a case where an LLM, solely due to fine-tuning, was able to memorize complex parts of the outputs of the

Figure 4: Comparison of the histograms of frequency of the minimum distance of a translation generated with mBART TF10 of test samples to examples in the training set without (left) and with (right) rephrasing.

 

training set and reconstruct them with high accuracy. Although memorization has been seen in the training of LLMs, we show here a case where there is reasonable evidence that it happened during the fine-tuning process with very low amounts of data.

The third contribution is a method to reduce the memorization effect relying on rephasings of training samples. Using 10 different versions of The Bible in English, instead of identical output verses, we show that the model presents very low memorization levels together with improved translation quality and more diversity of outputs. Although we did this study with real rephrasings, we notice that LLM-generated rephrasings is a viable alternative when rephrasings are not available. Evaluating how successful the use of commercial-grade, readily available LLMs is to avoid memorization issues in many-to-one scenarios is an important part of our future work.

We also wonder if similar memorization issues may be present in some of previous research on multilingual translators for low-resource languages and, if so, whether they have compromised their findings. The belief that multilingual models are usually better than bilingual ones seems to be based on a reliance on averages of sample-level translation quality metrics which, as shown here, may hide the use of incorrect strategies. It seems essential not only that research in this area reports the standard deviation of all results but also that efforts are made to detect potential disguise of memorization strategies when fine-tuning LLMs, possibly by using some of the methods suggested in this paper.

Lastly, we believe the issues described in this paper are not exclusive to Bibles and multilingual translations. In tasks such as fine-tuning conversation LLMs using question answering data from traditional chatbots, when several questions are mapped to the same answer, we might face similar issues. Investigating that issue is outside the scope of this work but it is an interesting future direction.

## 8 Limitations

The main limitation of this work is the use of The Bible as the main source of data. Not only it is limiting in terms of domain, it is also likely that this type of data was used in the pre-training of LLMs. Also, The Bible itself contains intrinsic repetitions of texts which can contribute to memorization.

Another limitation is the lack of a more in-depth analysis of the loss curves in the fine-tuning of the LLMs and not using the memorization-effect metric as a way to mitigate the memorization effects. That is, our proposed metric could be used as an alternative to model selection or early stopping. With that, we believe we could reduce the negative effects of memorization but the resulting impact on translation accuracy is unknown.

## 9 Ethical Considerations

It is essential to consider the ethical aspects of the goals and methods of any work with Indigenous languages. First, we abide to the belief that the decision of whether to create or not a MT for an Indigenous language has to be done by the people who speak the language, fully informed and, whenever possible, as a participant of the process Mihesuah (1993); Sahota (2007); Straits et al. (2012).

Moreover, we understand the complex political and ideological choices involved in the process of language vitalization McCarty (2008); McCarty (2009); McClarty (2011); Shulist and Granadillo (2022) and the use technologies to support it Harding et al. (2012); Liu et al. (2022). This was made clear by the Indigenous communities in the _Los Pinos Declaration10_, which enshrines that language-related efforts have be done by and with Indigenous peoples: _"Nothing for us without us."_

Footnote 10: [https://en.unesco.org/sites/default/files/los_pinos_declaration_170720_en.pdf](https://en.unesco.org/sites/default/files/los_pinos_declaration_170720_en.pdf).

However, the main goal of the research described in this paper is to determine technically feasible paths to construct MTs given the restrictions imposed by extremely-low-resource languages to current technologies. Therefore, even if the results of our work were extremely positive, we would refrain to make the MTs available without express consent of the corresponding community.

We are also aware that one of the unfortunate aspects of past and present colonial history of Indigenous peoples is related to different forms of Christianism. As a consequence, the Bible is one of the most commonly found document translated to several of those languages, by Jesuits in the early days of colonization and in the last 100 years often by Evangelical churches Franchetto (2008). As such, the translations of The Bible are often associated to different forms of cultural abuse and violence and to the establishment of orthographies of domination Franchetto (2008).

However, the reality is that such texts are one of the few available sources of parallel multilingual 

datasets for many Indigenous languages. We thus take the use of The Bible in this work as an "exceptional" first step, as a sort of "toxic" data which should not be used, in principle, for any actually deployed system unless with explicit agreement of the Indigenous community. Nevertheless we believe such kind of data can be used carefully for in-laboratory technical experiments in well-contained contexts such as the study described in this paper. To mitigate some of those risks, we implemented in this study some of the protocols suggested in Pinchanez et al. (2023), including the adoption of containment procedures.

## References

* Adelani et al. [2021] David Adelani, Jesujoba Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Michel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajuddeen Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong, Michael Beukman, Shamsudeon Muhammad, Guyo Jarsco, Oreen Yousuf, Andre Niyongabo Rubongo, Gilles Hacheme, Eric Peter Wairagala, Muhammad Umair Nasir, Benjamin Ajibade, Tunde Ajayi, Yvonne Gitau, Jade Abbott, Mohamed Ahmed, Millicent Ochieng, Anouluwapo Aremu, Perez Ogayo, Jonathan Mukiibi, Fatoumata Ouoba Kabore, Godson Kalipe, Degruene Mbaye, Allahsera Augustte Tapo, Vicotre Mendjokam Kogagne, Edwin Munkoh-Baubeng, Valencia Wagner, Idris Abdulmumin, Ayodele Awokoya, Happy Buzaqa, Blessing Sibanda, Andiswa Bhkula, and Sam Manthalu. 2022. A few thousand translations go a long way! leveraging pre-trained models for African news translation. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL'24)_, pages 3053-3070, Seattle, United States. Association for Computational Linguistics.
* Aharoni et al. [2019] Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL'19)_, pages 3874-3884, Minneapolis, Minnesota. Association for Computational Linguistics.
* Ajii et al. [2022] Ahham Fixri Aji, Genta Indra Winata, Fabiri Koto, Samuel Cahyawijaya, Ade Romadhony, Rahmad Mahendra, Kemal Kurniawan, David Moeljadi, Ratityo Eko Prasjo, Timothy Baldwin, Jay Han Lau, and Sebastian Ruder. 2022. One country, 700+ languages: NLP challenges for underrepresented languages and dialects in Indonesia. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (NAACL'22)_, pages 7226-7249, Dublin, Ireland. Association for Computational Linguistics.
* Bollmann et al. [2021] Marcel Bollmann, Rahul Aralikate, Hector Murrieta Bello, Daniel Herschovich, Miryam de Ihoneux, and Anders Sogaard. 2021. Moses and the character-based random babbling baseline: CoAStaL at AmericasNLP 2021 shared task. In _Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas_, pages 248-254, Online. Association for Computational Linguistics.
* Carlini et al. [2023] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. _arXiv preprint: 2202.07646_.
* Chen and Abdul-Mageed [2022] Wei-Rui Chen and Muhammad Abdul-Mageed. 2022. Improving neural machine translation of indigenous languages with multilingual transfer learning. _arXiv preprint: 2205.06993_.
* Dabre et al. [2020] Raj Dabre, Chenhui Chu, and Anoop Kunchukattan. 2020. A survey of multilingual neural machine translation. _ACM Comput. Surv._, 53(5).
* Dabre et al. [2019] Raj Dabre, Atusushi Fujita, and Chenhui Chu. 2019. Exploiting multilingualN through multistage fine-tuning for low-resource neural machine translation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP'19)_, pages 1410-1416, Hong Kong, China. Association for Computational Linguistics.
* Franchetto [2008] Bruna Franchetto. 2008. The war of the alphabets: indigenous peoples between the oral and the written. _Mana_, 4:31-59.
* Franchetto [2020]Bruna Franchetto. 2020. Lingua (s): cosmopoloticas, micropolticas, macropolitcas. _Campos-Revista de Antropologia_, 21(1):21-36.
* neural metrics are better and more robust. In _Proceedings of the Seventh Conference on Machine Translation (WMT22)_, pages 46-68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
* Goyal et al. [2022] Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjiana Krishnan, Mare'Aureluio Razmato, Francisco Guzman, and Angela Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. _Transactions of the Association for Computational Linguistics_, 10:522-538.
* Harding et al. [2012] Anna Harding, Barbara Harper, Dave Stone, Catherine O'Neill, Patricia Berger, Stuart Harris, and Jamie Donatto. 2012. Conducting research with tribal communities: Sovereignty, ethics, and data-sharing issues. _Environmental health perspectives_, 120(1):6 

* [13]C. A. Berman, M. J. C. Krizhevsky, and R. L. Krause (2016) The impact of deep learning on the classification of natural language models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 457-489. Cited by: SS2.
* [14]J. Dosovitskiy, S. Ioffe, and T. Ertas (2016) Visualizing neural networks with convolutional layers. arXiv preprint arXiv:1602.01665. Cited by: SS2.
* [15]K. Lee, Q. Li, Y. Zhang, W. Liu, X. Wang, Z. Chen, H. Yang, and G. Xu (2020) Deep residual learning for image recognition in context-oriented visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 1036-1040. Cited by: SS2.
* [16]L. McCarty (2011) Ethnography and language policy. Vol. 39, Routledge New York. External Links: ISBN 978-1-4503-3213-1, Link, Document Cited by: SS2.
* [17]A. Mueller, G. Nicolai, A. D. McCarthy, D. Lewis, W. Wu, and D. Yarowsky (2020) An analysis of massively multilingual neural machine translation for low-resource languages. In Proceedings of the 12th Language Resources and Evaluation Conference (LREC'20), Marseille, France, pp. 3710-3718. External Links: Link, Document Cited by: SS2.
* [18]M. Mayer, X. Gutierrez-Vasques, G. Sierra, and I. Meza-Ruiz (2018) Challenges of language technologies for the indigenous languages of the Americas. In Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, New Mexico, USA, pp. 55-69. External Links: ISBN 978-1-4503-3213-1, Link, Document Cited by: SS2.
* [19]T. Meyer and M. Cysouw (2014) Creating a massively parallel Bible corpus. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), Reykjavik, Iceland, pp. 3158-3163. External Links: Link, Document Cited by: SS2.
* [20]E. Kharitonov, M. Baroni, and D. Hepkes (2021) How bpe affects memorization in transformers. arXiv preprint: 2110.02782. Cited by: SS2.
* Findings (ACL'22), Dublin, Ireland, pp. 58-67. External Links: Link, Document Cited by: SS2.
* [22]R. Larochelle, P. Perronneau, F. Pereira, and J. Leskovec (2016) Learning to be efficient for large scale conversational dialog tasks. In Proceedings of the European Conference on Empirical Methods in Natural Language Processing (EMNLP), Florence, Italy, pp. 1-10. External Links: Link, Document Cited by: SS2.
* [23]D. Ma, M. R. Costa-jussa, J. Cross, O. Celebi, M. Elbayad, K. Heiferran, E. Kalbassi, J. Lam, D. Licht, J. Manillard, A. Sun, S. Wang, G. Wenzek, A. Youngblood, B. Akula, L. Barrault, G. Mojia Gonzalez, P. Hansanti, J. Hoffman, S. Jarrett, K. Ram Sadagopan, D. Rowe, S. Spruit, C. Tran, P. Andrews, N. Fazil Ayan, S. Bhosale, S. Edunov, A. Fan, C. Gao, V. Goswami, F. Guzman, P. Kopohn, A. Mourachko, C. Ropers, S. Saleem, H. Schwenk, and J. Wang (2022) No language left behind: scaling human-centered machine translation. arXiv preprint: 2207.04672.

 

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL'02), Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Pinhanez et al. (2023) C. Pinhanez, P. Cavalin, M. Vasconcelos, and J. Nogima. 2023. Balancing social impact, opportunities, and ethical constraints of using ai in the documentation and vitalization of indigenous languages. In Proc. of IJCAI'23, Macao, China.
* Post (2018) Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, Brussels, Belgium. Association for Computational Linguistics.
* Ramirez-Sanchez et al. (2020) Gema Ramirez-Sanchez, Jaume Zaragoza-Bernabeu, Marta Banon, and Sergio Ortiz Rojas. 2020. Bifixer and bicleaner: two open-source tools to clean your parallel data. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, Lisboa, Portugal. European Association for Machine Translation.
* Sahota (2007) Puneet Chawla Sahota. 2007. Research regulation in american indian/alaska native communities: Policy and practice considerations. Technical report, National Congress of American Indians Policy Research Center.
* Saleh et al. (2021) Fahimeh Saleh, Wray Buntine, Gholamreza Haffari, and Lan Du. 2021. Multilingual neural machine translation: Can linguistic hierarchies help? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021), ppas 1313-1330, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proceedings of the 48th Annual Conference of the Association of Computational Linguistics (ACL'20). Association for Computational Linguistics.
* Shulist and Granadillo (2022) Sarah Shulist and Tania Granadillo. 2022. Language ideology planning as central to successful revitalization projects. Language Documentation & Conservation, 16.
* Storto (2019) Luciana Raccanello Storto. 2019. _Linguas indigenas: tradicato, universais e diversidade_. Mercado de Letras.
* Straits et al. (2012) K.J.E. Straits, D.M. Bird, E. Tsinajinnie, J. Espinoza, J. Goodkind, O. Spencer, and T.G.P. Workgroup. 2012. Guiding principles for engaging in research with native american communities. UNM Center for Rural and Community Behavioral Health & Albuquerque Area Southwest Tribal Epidemiology Center.
* Tang et al. (2020) Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint: 2008.00401.
* Tirumala et al. (2022) Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022. Memorization without overfitting: Analyzing the training dynamics of large language models. arXiv preprint: 2205.10770.
* Vazquez et al. (2021) Raul Vazquez, Yves Scherrer, Sami Virpioja, and Jorg Tiedemann. 2021. The Helsinki submission to the AmericasNLP shared task. In Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas, pages 255-264, Online. Association for Computational Linguistics.
* Zhang et al. (2021) Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramer, and Nicholas Carlini. 2021. Counterfactual memorization in neural language models. arXiv preprint: 2112.12938.
* Zhang et al. (2022) Shiyue Zhang, Ben Frey, and Mohit Bansal. 2022. How can NLP help revitalize endangered languages? a case study and roadmap for the Cherokee language. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL'22), Dublin, Ireland. Association for Computational Linguistics.
* Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In Proc. of the 2020 International Conference on Learning Representations (ICLR'20).