Small Data, Big Impact: Leveraging Minimal Data for Effective Machine Translation
Year: 2023

Abstract:
For many languages, machine translation progress is hindered by the lack of reliable training data. Models are trained on whatever preexisting datasets may be available and then augmented with synthetic data, because it is often not economical to pay for the creation of largescale datasets. But for the case of low-resource languages, would the creation of a few thousand professionally translated sentence pairs give any benefit? In this paper, we show that it does. We describe a broad data collection effort involving around 6k professionally translated sentence pairs for each of 39 low-resource languages, which we make publicly available. We analyse the gains of models trained on this small but high-quality data, showing that it has significant impact even when larger but lower quality pre-existing corpora are used, or when data is augmented with millions of sentences through backtranslation.

heading: 1 Introduction
text: State of the art machine translation models are able to cover hundreds of languages (Ma et al., 2021; Wang et al., 2022; Siddhant et al., 2022; NLLB Team et al., 2022) by relying on large amounts of annotated (Skadin, š et al., 2014; Lison and Tiedemann, 2016; Agić and Vulić, 2019) and unannotated web crawled data (Schwenk et al., 2021; Heffernan et al., 2022). Translation for low-resource languages still faces significant challenges related to data availability, since many of these languages have neither large-scale parallel corpora nor a big presence on the web (Adelani et al., 2022b).
Techniques such as self-supervised learning (Ma et al., 2021; Liu et al., 2021) and backtranslation (Sennrich et al., 2016; Edunov et al., 2018; Fan et al., 2020) can be effective tools to reduce the
∗jeanm@meta.com †vedanuj@meta.com
reliance on annotation for translation models. In some cases, these techniques can be combined or even be applied iteratively (Hoang et al., 2018), leading to a feedback loop that can generate increasingly better translations. In order to be effective, however, such methods still require a certain amount of seed parallel data, which can be used to kickstart the process.
As a result, researchers and communities looking to train translation systems for low-resource languages may find themselves wondering how much parallel data is required to achieve a given performance target level.
In this paper, we describe a data collection effort for 39 low-resource languages, involving the creation of over 6k seed sentence pairs per language by professional translators, which we make publicly available with an open license. We analyse the behaviour of bilingual translation systems trained on varying amounts of this data, with and without the addition of pre-existing publicly available parallel datasets, and find that even comparatively small amounts of professionally produced parallel sentences can have an outsized impact. We find that gains coming from high quality data are further enhanced when training multilingual models of closely related high- and low-resource languages, and even more so when augmenting the dataset via backtranslation.
Overall, our results show that employing relatively small but high-quality, professionally translated datasets constitutes a promising and viable way towards achieving performant machine translation for low-resource languages, especially for those with high-resource relatives. This holds true even for languages for which some pre-existing data might already be publicly available, further highlighting the importance of high-quality training datasets.
Notably, parallel datasets of the scale discussed here are compact enough that coverage for a new
2740
language could plausibly be collected by a relatively small group of volunteers in a week, making these results relevant for the usage of machine translation technologies in crisis situations (Lewis et al., 2011).
Our main contributions are:
1. The creation and public release of a professionally translated seed dataset for 39 lowresource languages.1
2. An analysis of the impact of this high-quality data, both in isolation and also when combined with pre-existing datasets, based on hundreds of trained models.
3. A study of how gains from high-quality parallel data compound when using multilingual training and backtranslation, showing that benefits from high-quality data do not get washed away when using stronger models or data augmentation.

heading: 2 Background
text: Low-resource language translation Despite very successful recent advances in neural machine translation, most of the gains have only benefited a handful of so called high-resource languages, which have enough textual resources to satisfy the substantial data requirements of state-of-theart techniques. The vast majority of the world’s languages are low-resource, and researchers have increasingly been focusing on evaluating performance in this challenging setting (Wenzek et al., 2021).
Benchmarks Traditionally, one of the biggest challenges to the development of low-resource translation systems has been the lack of high quality evaluation data. Several benchmarks focus on specific sets of languages, such as the MADAR dataset for Arabic dialects (Bouamor et al., 2018), the Autshumato benchmark covering 11 South African languages (McKellar, 2017), or the TICO-19 benchmark covering 35 languages for the domain of medical information related to the COVID-19 pandemic (Anastasopoulos et al., 2020). More recently, the FLORES-101 dataset (Goyal et al., 2022) and its expansion to over 200 languages (NLLB Team et al., 2022) has enabled multilingual evaluation across tens of thousands of directions, including many
1https://github.com/facebookresearch/ flores/tree/main/nllb_seed.
low-resource languages. Its domain is composed of an even mixture of travel guides (Wikitravel), children’s literature (Wikijunior), and news content (Wikinews).
Training corpora Much important work has gone towards the development of parallel corpora for low-resource languages, most of which focusing on individual language pairs (Tapo et al., 2021; Ali et al., 2021; Adelani et al., 2021; Azunre et al., 2021, inter alia). Adelani et al. (2022a) study the case of 15 low-resource African languages, most of which already have tens or hundreds of thousands of parallel sentences in the religious domain, and investigate how combining pre-trained models and a newly created corpus can lead to effective domain transfer.
Low-resource training Amongst the techniques that can be used to decrease the reliance on manually annotated data, bitext mining (Schwenk et al., 2021; Ramesh et al., 2022) enables finding pairs of translations among large collections of unannotated monolingual text. Heffernan et al. (2022) show its effectiveness for low-resource languages, but point out that it can be limited for the most data scarce languages. Backtranslation (Sennrich et al., 2016; Edunov et al., 2018) can be used to create pseudoparallel data from monolingual data in a target language. It relies on an initial, potentially low-quality translation model – thereby having some requirements on annotated data – and can also be applied iteratively for improved performance (Hoang et al., 2018). Self-supervision (Siddhant et al., 2020) is a method employing monolingual text denoising as a joint training objective, and its use has been suggested as a way of kick-starting an iterative backtranslation pipeline. Finally, multilingual translation, which is often combined with one or more of the above techniques, has been shown to improve low-resource translation performance via cross-lingual transfer (Firat et al., 2016; Fan et al., 2020; Ma et al., 2021; Wang et al., 2022; Siddhant et al., 2022; NLLB Team et al., 2022)
Training without parallel data Within the area of low-resource translation, Bapna et al. (2022) describe the development of translation systems for low-resource languages without using any parallel data at all, relying instead on crawled monolingual data and language transfer. Methods which don’t require parallel data are likely complementary to the seed data approach proposed in this
paper. However, the over-reliance on cross-lingual transfer from a high-resource language opens up the risk of a translation system flattening the differences between related languages, as observed by NLLB Team et al. (2022) for Arabic dialects. This is a particularly thorny issue for communities of speakers of endangered languages which are at risk of being displaced by a related higher-resource language – as is the case for several of the languages covered in this paper. In such cases, we recommend the seed data approach, which opens the door for the communities to take ownership in preserving their languages, and aligns well with their desire to preserve the distinctiveness of their language in technological applications.
Crisis MT Low-resource machine translation has been studied in the context of crisis events, and has been proposed as a component of a rapid response infrastructure (Lewis et al., 2011). In particular, Lewis (2010) describe the creation of a system for Haitian Creole after the devastating 2010 earthquake, and Anastasopoulos et al. (2020) built a dataset to facilitate access to information related to the COVID-19 pandemic.

heading: 3 Data collection
text: Regardless of the many modelling improvements aimed at reducing the amount of required supervision, it is likely impossible for translation models to reach acceptable levels of quality without even small amounts of parallel data. This is especially true for approaches that explicitly rely on the preexistence of parallel corpora, such as backtranslation. As a result, low-resource languages with corpora that are too small to enable the use of these techniques are cut off from the improvements they bring. With this in mind, we set up a data collection effort for a number of low-resource languages which fit this criterion, resulting in a dataset of around six thousand English sentences translated into each of 39 low-resource languages.
Language selection In order to choose which languages to collect data for, we took several factors into account. First, we looked at the list of languages supported by Wikipedia. The usergenerated encyclopedia is one of the most visited websites in the world, and constitutes an important means of knowledge dissemination for many low-resource language communities. Crucially, Wikipedia has an open process towards support-
ing new languages,2 which has led to the platform supporting over 300 languages in 2022.3 This list of languages was cross-referenced with those currently supported by machine translation benchmarks, including the large FLORES-200 dataset.4 We then focussed our attention to those languages for which not enough high quality data was currently publicly available for large-scale training, looking in particular at those languages with fewer than 100,000 parallel training sentences and prioritising those with the least amount of high quality data (as determined by automatic metrics such as language identification). Finally, we partnered with linguists and identified those languages for which professional translators would be available.
Source sentence selection The dataset consists of English sentences translated into a number of low-resource languages. The source data was sampled from Wikimedia’s List of articles every Wikipedia should have,5 a collection of 10,000 Wikidata IDs corresponding to notable topics in different fields of knowledge and human activity. These are split into 11 categories such as People, History, Philosophy and Religion, Geography. We uniformly sampled a subset of IDs from which we would draw data, and mapped these to the corresponding English Wikipedia articles. From each of these articles we then sampled triplets of contiguous sentences, such that some amount of context would be provided, and ensured a maximum of one triplet would be sampled per article to guarantee a relatively uniform coverage of topics.
Finding translators The parallel dataset was created through human translation. We identified translators through various specialised language service providers. Through a vetting process, we selected translators that were native speakers in the target language, with a minimum of two years of professional experience and a degree in a relevant field of studies, such as translation or linguistics. All translators were additionally required to have a high level of English fluency, and had to pass an initial test to assess their translation proficiency.
2https://meta.wikimedia.org/wiki/ Language_proposal_policy
3https://meta.wikimedia.org/wiki/List_ of_Wikipedias
4https://github.com/facebookresearch/ flores
5https://meta.wikimedia.org/wiki/List_ of_articles_every_Wikipedia_should_have/ Expanded
Translation workflow Translators were provided with a clear set of instructions for the project, which can be seen in Appendix B. In addition to these general instructions, in order to avoid issues of mismatching script, spelling system or dialect with the available evaluation benchmarks, we established a set of linguistic guidelines to match the data that was collected for the FLORES-200 dataset. Translators referenced these guidelines while working on the creation of the dataset. The source sentences were translated directly from English for most languages. The only exceptions were Acehnese and Banjar in the Arabic script and Tamasheq in the Tifinagh script, which were transliterated from their respective Latin script datasets, that had in turn first been translated from English. Following this process we conducted a linguistic quality assessment phase in which all translations were checked for conformance with the linguistic guidelines, and automatic quality control checks were performed.
Compensation range The hourly compensation for translators averaged 25.80 US dollars, with a median of 25.60. The productivity rate generally ranged between 200-250 words per hour, with the exception of the Acehnese and Banjar transcriptions into Arabic which required less effort. Transcription of Tamashek into Tifinagh proved to be more difficult, and had a productivity rate close to that of translation. The full costs for the project also included quality assurance as well as other various expenses incurred by the language providers we partnered with.
Final dataset The final dataset size was chosen in order to obtain at least 6,000 parallel sentences per direction, while simultaneously maximising language coverage. Given the available budget, this resulted in a final dataset of 6,193 sentences translated into 39 languages, including three transcribed directions. The dataset is released under the open CC-BY-SA 4.0 license. A full list of the languages can be found in Appendix A.

heading: 4 Experimental Setup

heading: 4.1 Data
text: Bilingual models Our first set of experiment focuses on bilingual machine translation, both into and out of English. Beyond our newly developed seed corpus described in Section 3, we sourced additional pre-existing parallel sentences with En-
glish through the OpenSubtitles corpus (Lison and Tiedemann, 2016), the QCRI educational domain corpus (Abdelali et al., 2014), the PMIndia corpus (Haddow and Kirefu, 2020), the MultiIndicMT corpus (Nakazawa et al., 2021) as well as the GlobalVoices, Gnome, KDE, Sardware, Tatoeba, Ubuntu and Wikimedia corpora available through the OPUS repository (Tiedemann, 2012). The parallel sentences were obtained through the mtdata tool (Gowda et al., 2021).
Multilingual models Our second set of experiments involves training multilingual machine translation models for two clusters of related languages: an Italic model, trained on six low-resource languages (fur_Latn, lij_Latn, lmo_Latn, scn_Latn, srd_Latn, vec_Latn) and three related high-resource languages (cat_Latn, ita_Latn, spa_Latn) along with English; and an Indo-Aryan model, with four lowresource (bho_Deva, hne_Deva, kas_Deva, mag_Deva) and two related high-resource languages (hin_Deva, ben_Beng), together with English. For these experiments, we collected additional parallel sentences between any two of the languages within each group. On top of the corpora mentioned in the previous paragraph, we also used the EU Bookshop (Tiedemann, 2012) and Europarl (Koehn, 2005) corpora for certain high-resource directions.
Backtranslation For the backtranslation experiments of Section 4.4, we sourced monolingual data from the Common Crawl project,6 and filtered it with the LID model provided by NLLB Team et al. (2022) in order to obtain a maximum of 2M sentences per language.
All models are evaluated on the devtest split of the FLORES-200 benchmark.

heading: 4.2 Bilingual experiments
text: For the bilingual experiments, we divide our 39 focus languages into two broad groups. The larger group, which we call unresourced languages, consists of the 27 languages for which we could find little (< 1 k) or no pre-existing parallel data available through public sources. The second group, which we call barely-resourced languages, consists of those languages that had at least one thousand pre-existing publicly available parallel sentences – these are listed in Table 1.
6https://commoncrawl.org/
In order to study the data scaling properties, we randomly partition each seed dataset into three chunks: one consisting of 1k seed parallel sentences, one consisting of 2k, and the final one consisting of the remaining 3k sentences.
For each unresourced language, we consider two directions, into and out of English. For each direction, we train three models: on the first, the first two, and all three chunks of the seed data (training corpus sizes of 1k, 3k and 6k sentences respectively). This results in 162 models overall.
For the barely-resourced languages, we take the same basic approach, but always include the preexisting publicly available data. In addition, we also train models using the whole seed dataset only, and the publicly available data only. This results in 120 models.
All bilingual models use a transformer architecture (Vaswani et al., 2017) with 6 encoder layers and 6 decoder layers, 8 attention heads, 512- dimensional embeddings, 0.3 dropout, an effective batch size of 130k tokens, and are trained with an inverse square root learning rate schedule with warmup. Data for each model is tokenised with a language pair specific sentencepiece model (Kudo and Richardson, 2018). Training is conducted with fairseq (Ott et al., 2019), with each model being trained on a machine with 8 NVIDIA Tesla V100 Volta 32GB GPUs for at most 12 hours.

heading: 4.3 Multilingual Experiments
text: Low-resource languages have been shown to significantly benefit from multilingual transfer (Arivazhagan et al., 2019; Bapna et al., 2022; NLLB Team et al., 2022), so it is reasonable to expect that any attempts at boosting low-resource transla-
tion performance would also involve multilingual training. In order to evaluate the data scaling and language transfer properties in this useful setting, we design an additional set of experiments focusing on two groups of languages.
• We train an Italic model on the low-resource Friulian, Ligurian, Lombard, Sicilian, Sardinian and Venetian, combined with the related high-resource Catalan, Italian and Spanish, plus English.
• We train an Indo-Aryan model on the lowresource Bhojpuri, Chhattisgarhi, Kashmiri (Devanagari script) and Magahi, combined with the related high-resource Hindi and Bengali, plus English.
Each model is trained on all available parallel data between any of its languages. We further conduct an ablation experiment for each model, by removing all seed data and training on the publicly available data only. The training setup is analogous to that of the bilingual experiments, but the architecture is scaled up to 12 layers and 8 attention heads for both encoder and decoder, 1024- dimensional embeddings, 0.1 dropout, and an effective batch size of 524k tokens. Multilingual models are trained on four machines, each with 8 NVIDIA Tesla V100 Volta 32GB GPUs, for a maximum of 48 hours.

heading: 4.4 Backtranslation
text: Our final set of experiments involves generating backtranslation data with the multilingual models, and training new multilingual models with this additional data. As discussed in Section 2, this technique can be particularly effective for improving low-resource translation performance. The unlabelled monolingual data it relies upon is more easily obtainable than parallel sentences (Heffernan et al., 2022), making this technique particularly important to boost performance for particularly data scarce settings. We run this experiment both using pre-existing data only, as well as with the addition of all seed data.
Despite monolingual data taking centre stage in backtranslation, the technique still depends on the existence of a seed translation model to augment the unannotated sentences with synthetic translations. We experiment with generating backtranslation data for the two multilingual models of Section 4.3, using both the full and ablated models.
For the Italic model, we provide backtranslations from the six low-resource languages into both eng_Latn and ita_Latn, and vice versa. For the Indo-Aryan model, we provide backtranslations from the four low-resource languages into both eng_Latn and hin_Deva, and vice versa.

heading: 5 Results and Analysis
text: We report all results using automatic evaluation metrics against the FLORES-200 benchmark. We rely on the chrF++ score (Popović, 2017), which is based on character-level n-gram overlap, and is complemented by unigram and bigram features. This score overcomes the limitations inherent to the more commonly used BLEU metric (Papineni et al., 2002), which relies on the availability of tokenisation tools for all languages and fails to accurately account for highly agglutinative languages.

heading: 5.1 Bilingual Experiments
text: A summary of bilingual translation performance on the unresourced languages in reported in Figures 1a and 1b. At the lowest training data level, consisting of 1k sentences, we obtain an average chrF++ score of 12.6 eng-xxx and 13.9 xxx-eng. Moving to the 3k-sized corpus, the average increases to 19.9 eng-xxx and 20.6 xxx-eng. Training on the full seed corpus, this further increases to 22.9 eng-xxx and 23.7 xxx-eng. On the whole, models perform at a similar level on the two translation directions, with a slightly larger spread on the eng-xxx direction.
Results on languages that already had some amount of parallel data publicly available – which we call barely-resourced – are reported separately, in Figures 1c and 1d. We find that, even though these languages already have pre-existing training data (accounting for 12k sentences per language, on average) the addition of a mere 1k parallel sentences from our high-quality dataset brings the average performance up from 12.9 to 19.0 chrF++ in the eng-xxx direction, and from 16.0 to 20.9 chrF++ in the xxx-eng direction. Notably, we see that training without the publicly available data has little effect. Indeed, the removal of all public data accounts for a mere average chrF++ drop of 0.7 eng-xxx and 1.1 xxx-eng , underlining the fundamental role that high quality annotated data can play in improving performance for data-scarce languages.

heading: 5.2 Multilingual Experiments
text: Results for the multilingual experiments on the Italic and Indo-Aryan language clusters are reported in Table 2.
For the xxx-eng directions, which target highresource English, we see that gains from multilingual training are substantial, averaging 25.6 chrF++ for the Italic model and 20.2 chrF++ for the IndoAryan model when compared to their respective bilingual versions (Appendix D). The multilingual model sees a lot more English data as target, and performs better on it. Gains are still sizable but relatively smaller for the eng-xxx directions, into low-resource languages. In this case, the average performance difference is of 13.6 and 16.2 chrF++ for the Italic and Indo-Aryan models, respectively.
For a comparison of the effects of seed data collection, column ∆ in Table 2 measures the performance difference of the P+6k and P multilingual models. For the eng-xxx direction the average difference is 14.0 and 12.9 chrF++ for the Italic and Indo-Aryan models respectively; in the reverse directions, the difference is 14.6 and 9.8. This confirms that the beneficial effects of cross-lingual transfer do not compensate for the gains achieved by higher quality data.

heading: 5.3 Backtranslation
text: Performance for the two multilingual models keeps steadily improving when adding backtranslation. By looking at column ∆noBT of Table 3, which compares multilingual models with and without backtranslated data, we see that all models trained with backtranslated data outperform their base counterparts for every single direction. Gains from backtranslation are generally more pronounced for the P models, which are trained without seed data. Overall, the same trend as in previous experiments holds true: as revealed by column ∆, which compares the P+6k and P backtranslation-augmented models, the models trained with seed data achieve the best performance for every direction.

heading: 6 Analysis
text: Figure 2 brings together the average performance of all models trained on the Italic and Indo-Aryan language clusters – bilingual, multilingual, and multilingual with backtranslation – both when trained only on pre-existing data alone (first set of bars), and when trained with the addition of high-quality seed data (hatched bars).
The same trends hold throughout our experiments: even with modelling improvements that aim to reduce the amount of required supervision, such as multilingual training and backtranslation, we observe that models trained on as little as 6k high-quality seed parallel sentences always come out ahead. This is true even for languages such as mag_Deva and hne_Deva, for which tens of thousands of pre-existing parallel sentences are
publicly available. Crucially, we see that the multilingual model with seed data (“Multilingual, P+6k” in the graph) outperforms in all but one case the version without seed data but with backtranslation (“Multilingual+BT, P”). In other words, even adding vast amounts of monolingual data (as much as 2M sentences for xxx-eng) cannot make up the difference that 6k high-quality parallel sentences make.

heading: 7 Conclusions
text: In this paper, we have described a parallel data collection effort involving 6k seed parallel sentences for 39 languages, and investigated the effects of this relatively small but high-quality dataset on machine translation performance. By training hundreds of bilingual translation models, we have looked at the data scaling properties, and found that even when several thousand pre-existing sentences are already available, adding as little as a thousand high-quality parallel sentences can significantly boost performance.
To answer the question of whether stronger models can compensate for the lack of high-quality data, we moved beyond simple bilingual models
and introduced two modelling improvements: multilingual training of closely related low- and highresource languages, and backtranslation. We found that models trained with the additional high-quality data performed consistently better. Even when augmenting the models with vast amounts of monolingual data via backtranslation, the beneficial effects of seed data were still present.
Overall, the results show that collecting highquality parallel data, produced by native speakers and manually aligned, is a fundamentally important investment for training machine translation models.

heading: 8 Limitations
text: Other ways of reducing the amount of required supervision could be attempted, but we do not expect that these would change the outcomes significantly. Self-supervised learning via masking / denoising objectives, either in the form of an auxiliary task or via the use of pretrained models, is one such approach. This however generally underperforms backtranslation, which can utilise the same monolingual data to more effect (NLLB Team et al., 2022), as we see in the experiments of Appendix E. Iterative backtranslation might offer an additional boost for data-scarce settings, but is very computationally intensive, complex, and any gains would almost certainly apply to models trained with the addition of seed data too.
The seed datasets that we release bring about large translation performance gains for a number of low-resource languages. We note that, due to budgetary and complexity constraints, the source data we used was sourced from English Wikipedia only. This is likely to have two effects. First, translating English-original data leads to so-called translationese effects one the low-resource side (Volansky et al., 2015), leading to decreased effectiveness for directions that target low-resource languages. Second, the data is unlikely to adequately cover diverse content from multiple cultures. An interesting avenue for future research would therefore involve studying the effects of seed parallel data that is originally translated from low-resource languages.

heading: A Full language list
text: The full list of languages covered by the seed dataset is shown in Table 4.

heading: B Translation instructions
text: We include below the instructions that were shared with translators participating in this project.
Important note
Your translations will be used to help train a Machine Translation engine. For this reason, this project requires Human Translation. The use of Machine Translation is strictly prohibited. Please read the section on Machine Translation for more details.
General instructions
1. You will be translating different contents from Wikipedia pages. The source URL is available for more context. Please refer to it.
2. Do not convert any units of measurement. Translate them exactly as noted in the source content.
3. As the source material is Wikipedia pages, translations should use a formal tone.
4. Provide fluent translations without deviating too much from the source structure. Only allow necessary changes.
5. Do not expand or replace information compared to what is present in the source documents. Do not add any explanatory or parenthetical information, definitions, etc.
6. Do not ignore any meaningful text that was present in the source.
7. In case of multiple possible translations, please pick the one that makes the most sense (e.g., for gender concordance, cultural fit in the target language, level of formality, etc.).
8. Translations must be faithful to the source in terms of pragmatics such as (if applicable) level of hedging/modality, sentiment and its intensity, negation, speech effects (disfluencies), etc.
9. For proper nouns and common abbreviations, please see the guidelines on Named Entities below.
10. Idiomatic expressions should not be translated word for word. Use an equivalent idiom, if one exists. If no equivalent idiom exists, use an idiom of similar meaning. If no similar expressions exist in the target language, paraphrase the idiom such that the meaning is retained in the target language.
11. When a pronoun to be translated is ambiguous (for instance, when it could be interpreted as either him/her or he/she), opt for gender neutral pronouns (such as them/they) if those exist in the target language. However, when a pronoun to be translated is clearly marked for gender, you should follow the source material and continue to mark for gender.
Machine translation
The translations you will provide are going to be used to train new Machine Translation engines. For this reason, the translations you provide should not be biased by existing Machine Translation providers. Therefore:
1. Translators should not reference any Machine Translation engine at all when translating, to avoid being biased by it.
2. All translations will be inspected, and those that are found to be too close to Machine Translation output will be returned to the translator. These will need to be revised, or the translator will be required to provide a quick explanation as to why the translation cannot
be modified further without affecting its meaning.
Named entities Named Entities are people, places, organisations, etc., that are commonly referred to using a proper noun. This section provides guidance on how to
handle Named Entities. Please review the following guidelines carefully:
1. If there is a commonly used term in the target language for the Named Entity:
(a) If the most commonly used term is the same as in the source language, then keep it as it is.
(b) If the most commonly used term is a translation or a transliteration, then use that.
2. If there is no commonly used term:
(a) If possible, a transliteration of the original term should be used.
(b) If a transliteration would not be commonly understood in the context, and the source term would be more acceptable, you may retain it.

heading: C Experimental details
text: We compute ChrF++ scores using the sacrebleu implementation,7 with the following signature: chrF2++|nrefs:1|case: mixed|eff:yes|nc:6|nw:2|space:no| version:2.1.0.
Training is conducted via the fairseq framework; example training configurations for both bilingual and multilingual models are made available.8

heading: D Performance of bilingual models
text: The full results of bilingual translation experiments for unresourced and barely-resourced languages is reported in Tables 5 and 6 respectively.

heading: E Self-supervised learning
text: In order to evaluate the effectiveness of selfsupervised learning on monolingual data (SSL), we conduct a series of experiments with our two multilingual models of Sections 4.3 and 4.4.
The setup of these experiments follows the denoising autoencoder technique of Liu et al. (2021). One possible approach would be to pre-train on a denoising task, and subsequently fine-tune on
7https://github.com/mjpost/sacrebleu 8https://github.com/fairinternal/
fairseq-py; training configurations are at https: //github.com/facebookresearch/fairseq/ tree/nllb/examples/nllb/modeling/train/ conf/cfg
machine translation. As this was shown to hurt performance by NLLB Team et al. (2022), we instead follow their recommended multi-tasking approach. Along with the regular machine translation training, target sentences in noised form are fed to the encoder, with the objective of maximising the likelihood of predicting the unnoised sentence. Noising is performed by randomly masking spans of a sentence with a mixture of special <mask> tokens or randomly sampled tokens from the model’s vocabulary. The experiments are conducted in the P+6k setting, including all pre-existing publicly available corpora as well as the full seed data. To be able to directly compare the SSL and BT approaches, for these experiments we reuse the monolingual corpora of Section 4.4.
As can be seen in Table 7, we find that backtranslation outperforms self-supervised learning with the denoising objective on every single direction evaluated. Comparing these models to the ones
trained without SSL in Table 2, we see that selfsupervision is generally beneficial when translating into the xxx-eng direction, but noticeably hurts performance when translating into one of the lowresource languages.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Left blank.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: section 4

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: section 3
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. All datasets used were intended for machine translation
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Data collected from Wikipedia

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? section 3 – demographic information not available due to privacy regulations
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. For the created dataset: section 3 and Appendix A .
C 3 Did you run computational experiments? sections 4, 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? section 4 (we report the size of the model in terms of layers, embedding size, etc.)
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
7 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? hyperparameter details in suppl. material

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? we report both descriptive statistics and exact per-model performance

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 3 and appendix
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? section 3
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? this information is proprietary to the language service providers we relied upon

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? section 3

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? data comes from wikipedia
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? such information was not available to us due to privacy regulations

heading: Future Work
text: An interesting avenue for future research would therefore involve studying the effects of seed parallel data that is originally translated from low-resource languages