Venue: NAACL 2022

# A Few Thousand Translations Go A Long Way!

 Leveraging Pre-trained Models for African News Translation

David Heoluwa Adelani\({}^{1*}\), Jesujoba Oluwadara Alabi\({}^{2*}\), Angela Fan\({}^{3*}\), Julia Kreutzer\({}^{4*}\), Xiaoyu Shen\({}^{5}\), Mached Reid\({}^{6**}\), Dana Ruiter\({}^{1}\), Dietrich Klakow\({}^{1}\), Peter Nabende\({}^{7*}\), Ernie Chang\({}^{1*}\), Tajuddeen R. Gwalabe\({}^{8**}\), Freshia Sackey\({}^{9*}\), Bonaventure F. P. Dossou\({}^{10*}\), Chris Chinenye Emezue\({}^{1*}\), Colin Leong\({}^{12*}\), Michael Beukman\({}^{13*}\), Shamsudden H. Muhammad\({}^{14*}\), Guyo D. Jarso\({}^{*}\), Oreen Yousuf\({}^{15*}\), Andre N. Rubungo\({}^{16*}\), Gilles Hacheme\({}^{17*}\), Eric P. Wairagala\({}^{7*}\), Muhammad U. Nasir\({}^{18*}\), Benjamin A. Ajibade\({}^{e}\), Tunde Oluwaseyi Ajayi\({}^{1}\), Yvonne Wambui Gitau\({}^{*}\), Jade Abbott\({}^{7}\), Mohamed Ahmed\({}^{19*}\), Millicent Ochigneg\({}^{19*}\), Anouluwapo Aremu\({}^{*}\), Perez Ogayo\({}^{20*}\), Jonathan Mukibi\({}^{7*}\), Fatounata Ouoba Kabore\({}^{*}\), Godson Koffin Kalipe\({}^{*}\), Derguene Mbaye\({}^{21*}\) Allahsera Augustse Tapo\({}^{22*}\), Victoire M. Koagne\({}^{*}\), Edwin Munkoh-Buabeng\({}^{*}\), Valencia Wagner\({}^{23*}\), Idris Abdulumin\({}^{24*}\), Ayodele Awokoya\({}^{25*}\), Bappy Buzaba\({}^{*}\), Blessing Sibanda\({}^{26*}\), Andiswa Bukula\({}^{27*}\), Sam Manthalu\({}^{28}\)

###### Abstract

Recent advances in the pre-training of language models leverage large-scale datasets to create multilingual models. However, low-resource languages are mostly left out in these datasets. This is primarily because many widely spoken languages are not well represented on the web and therefore excluded from the large-scale crawls used to create datasets. Furthermore, downstream users of these models are restricted to the selection of languages originally chosen for pre-training. This work investigates how to optimally leverage existing pre-trained models to create low-resource translation systems for 16 African languages. We focus on two questions: 1) _How can pre-trained models be used for languages not included in the initial pre-training?_ and 2) _How can the resulting translation models effectively transfer to new domains?_ To answer these questions, we create a _new_ African news corpus covering 16 languages, of which eight languages are not part of any existing evaluation dataset. We demonstrate that the most effective strategy for transferring both to additional languages and to additional domains is to fine-tune large pre-trained models on small quantities of high-quality translation data.

+
Footnote †: Masakhane NLP, \({}^{1}\)Saardal University, Germany, \({}^{2}\)Inria, France, \({}^{3}\)Meta AI, \({}^{4}\)Google Research, \({}^{5}\)Amazon Alexa AI, \({}^{6}\)The University of Tokyo, Japan, \({}^{7}\)Makerere University, Kampala, Uganda, \({}^{8}\)UCAS, China, \({}^{9}\)JKUAT, Kenya, \({}^{10}\)Jacobs University, Germany, \({}^{11}\)TUM, Germany, \({}^{12}\)University of Dayton, USA, \({}^{13}\)University of the Witwatersrand, South Africa, \({}^{14}\)LIAAD-INESC TEC, Porto, Portugal, \({}^{15}\)Uppsala University, Sweden, \({}^{16}\)UPc, Spain, \({}^{17}\)Ai4Imnov \({}^{18}\)Ominor AI \({}^{19}\)Microsoft Africa Research Institute, Kenya \({}^{20}\)CMU, USA, \({}^{21}\)Baamtu, \({}^{22}\)RIT, USA, \({}^{23}\)SPU, South Africa, \({}^{24}\)ABU, Nigeria, \({}^{25}\)UI Ibadan, Nigeria, \({}^{26}\)NUST, Namibia \({}^{27}\)SADiLaR, South Africa, \({}^{28}\)University of Malawi, Malawi

## 1 Introduction

Enormous efforts have been invested in making language and translation models more multilingual while leveraging the maximal amount of data for training, most prominently large crawls of monolingual and parallel data from the web El-Kishky et al. (2020); Schwenk et al. (2021b, a); Xue et al. (2021). The resulting models are now capable of translating between hundreds of languages, including language pairs that in isolation do not have large collections of parallel data Tang et al. (2020); Xue et al. (2021); Fan et al. (2021). For example, M2M-100 Goyal et al. (2021) can translate (with low accuracy) between Hausa and Yoruba, two of the most widely spoken languages in Nigeria, even though there is barely any parallel data available for training. For languages that are not included in the set of training languages, the model would have no knowledge on how to generate translations. Does this mean there is no hope for languages that do not have large presence on the web and are therefore not included in these pre-trained models?

We investigate _how large-scale pre-trained models can be leveraged for the translation of unseen low-resource languages and domains_. We address this question by studying 16 African languages that are largely underrepresented in NLP research Joshi et al. (2020); \(\forall\) et al. (2020) and further have little to no training data available (SS3). These languages provide an ideal testbed for two challenging knowledge transfer tasks: **(1)** How can pre-trained models create translations for languages unseen at train

ing time? and **(2)** Since training data may only exist in single domain (i.e. religious texts), how can a model be trained in one domain and translate another effectively at test time?

These questions are extremely relevant for our chosen languages because they all have millions of native speakers and a massive need for translation technologies. For example, news concerning the African continent are almost exclusively published in English, French, or Arabic, and thereby inaccessible for speakers of only native African languages. This creates a bottleneck for information transmission, which becomes even more critical in times of crises Oktem et al. (2020); Anastasopoulos et al. (2020); Oktem et al. (2021). Furthermore, the task of translating news has historically played a central role in translation research, e.g. in shared tasks since 2008 Callison-Burch et al. (2008) and as a test for determining human parity Hassan et al. (2018); Laubli et al. (2018); Toral et al. (2018). To spur the development of dedicated news translation models for Africa, we construct a benchmark of news translation for translating between 16 native African languages and English or French (SS4).

This allows us to compare three approaches to leveraging large-scale multilingual models for the translation of previously unseen languages: **(1)** zero-shot transfer, **(2)** continual pre-training on monolingual data, and **(3)** multi-domain fine-tuning on parallel data (SS5). We find that fine-tuning pretrained models on a few thousand sentences of high quality bitext is remarkably effective, and can be further augmented with continual pre-training on African languages and fine-tuning on news domain data (SS6). Our contributions are the following:1

Footnote 1: All data, models and code are publicly available on [https://github.com/masakhane-io/lafand-mt](https://github.com/masakhane-io/lafand-mt) under academic license.

1. We create a **new African news corpus** for machine translation (following principles of participatory research \(\forall\) et al. (2020)) covering 16 African languages.
2. We **adapt several multilingual pre-trained models** (MT5, ByT5, mBART, M2M-100) to these largely unseen languages, and evaluate their quality on news translation.
3. We quantify the **effectiveness of small in-domain translation sets** by measuring domain transfer effects and comparing fine-tuning strategies.

We find that having a targeted collection of translations is surprisingly effective, showcasing the power of local knowledge in so-called "zero-resource" scenarios Bird (2020). This paints a promising picture for the development of NLP technology for understudied languages: being able to customize these models for new language of interest with as little as 2k sentences and a few fine-tuning steps, MT developers and users from any language community are less dependent on choices and monetary interest of industry powerhouses from the Global North Paullada (2020).

## 2 Related Work

African MT Datasets.One of the major challenges of developing MT models for African languages is lack of data. There are many attempts to automatically crawl and align sentences from the web Schwenk et al. (2021, 2020). Nevertheless, the resulting corpora for many African languages are typically small and of poor quality Kreutzer et al. (2021). Other cleaner parallel sources are mostly from religious sources, like the Bible covering over 1600 languages McCarthy et al. (2020) and JW300 Agic and Vulic (2019) from JW.org with over 343 languages, including over 100 African languages. Apart from the training dataset, evaluation datasets are needed to test the performance of multilingual MT models. The FLORES-101 Goyal et al. (2021) evaluation set, sourced from Wikipedia and manually translated, covers the largest number of languages, including 20 African languages. Finally, while other evaluation datasets for translating into or from African languages have been developed Siminyu et al. (2021); Emezue and Dossou (2020); Azunre et al. (2021); Nyoni and Bassett (2021); Gezmu et al. (2021); Ali et al. (2021), unfortunately there are only a few African languages with evaluation datasets in the news domain Adelani et al. (2021); Mabuya et al. (2021); Ezeani et al. (2020) but ours covers 11 African languages (SS4).

Low-resource MT.Interest in low-resource MT has been increasing both within the MT research community Haddow et al. (2021), as well as in native speaker communities (\(\forall\) et al., 2020; Azunre et al., 2021; Mager et al., 2021). On the modeling side, many techniques have been developed: unsupervised MT Lample et al. (2018) leverages monolingual data, single multilingual models capable of translating between many languages Firat et al. (2016); Johnson et al. (2017); Aharoni et al. ( 

2019; Fan et al., 2021), multilingual unsupervised models leverage a related language (with parallel data) to assist translating the low-resource language that might not even have any monolingual data Ko et al. (2021). Unfortunately, unsupervised MT typically performs poorly on low-resource languages Marchisio et al. (2020).

Transfer learning from high-resource languages has achieved more promising results: Transfer from multilingual pre-trained language models (PLM), like mBART50 Tang et al. (2020) and MT5 Xue et al. (2021), and large-scale multilingual MT often outperforms bilingual MT Tran et al. (2021); Yang et al. (2021). For low-resource languages this strategy outperforms the baseline Transformer models Birch et al. (2021); Adelani et al. (2021); Lee et al. (2022). The performance can be further improved by large scale pre-training Reid et al. (2021); Emezue and Dossou (2021).

## 3 Focus Languages and Their Data

Focus Languages.We focus on 16 African languages with varying quantities of available data Joshi et al. (2020), including moderately low-resource languages such as Swahili and Hausa, and very low-resource languages such as Ghomala'2 with the Bible being its largest available corpus. Table 1 provides an overview of the focus languages, including the language families, location and number of speakers, and the source and original language for our corpus. The languages are from four language families: Afro-Asiatic (e.g. Hausa), Nilo-Saharan (e.g. Luo), English Creole (e.g. NigerianPidgin/Naija) and Niger-Congo. Most of the languages (13 out of 16) are from the Niger-Congo family, which is the largest language family in Africa. Six of the languages are predominantly spoken in Francophone countries of Africa, while the remainder are predominantly spoken in Anglobone countries of Africa. In contrast to previous work V et al. (2020); Gowda et al. (2021), we do not focus exclusively on translation to/from English since this is not the primary language of the Francophone Africa community. All languages are spoken by at least one million speakers.

Language Characteristics.All languages are written in Latin script, using letters of the basic Latin alphabet with a few omissions (e.g "c", "q", "x", "z") and additions (e.g. "e", "\({}_{>}\)", "j", "Q", including digraphs like "gb", "kp", "gh", and sometimes more than two-character letters). 13 of the languages are tonal, and about nine make use of diacritics. Many African languages are morphologically rich. For example, all Bantu languages are agglutinative. Fon, Mossi, and Yoruba are highly isolating. All languages follow the Subject-Verb-Object sentence structure like English and French. Table C provides more details.

Existing Parallel Corpora.We curate publicly available parallel data for our focus languages, which consists primarily of text in the religious domain. For most African languages, the largest available parallel corpora is JW300 Agic and Vulic (2019), sourced from jw.org, which publishes biblical texts as well as lifestyle and opinion columns. Varying quantities of data are available for 11 of the 16 focus languages. Ewe, Igbo, Swahili, Setswana, Twi, Yoruba, and isiZulu have over 400K parallel sentences. Hausa and Mossi have slightly more than 200K parallel sentences, while Fon and Naija have around 30K sentences. For the remaining

\begin{table}
\begin{tabular}{p{78.3pt} p{14.2pt} p{14.2pt}|p{14.2pt}|p{14.2pt}|p{14.2pt} p{14.2pt} p{14.2pt}} \hline
**Target Language** & **Family** & \multicolumn{1}{p{14.2pt}}{**African Region**} & \multicolumn{1}{p{14.2pt}}{**No. of Source**} & \multicolumn{1}{p{14.2pt}}{**News**} \\ \cline{1-1}  & & & & & & & & \\ \hline Bunking (ban) & NC/ Mading & West & 14M Peruch & Malrewheat & 3002/ 1484/ 1600 & Balote & 28K \\ GhomalE® (ob.) & NC/ Gasfields & Central & 1M French & Cameron Web & 2232/ 1133/ 1400 & Babo & 8K \\ Flow (two) & NC/ Kwa & West & 2M French & Benin Web TV & 2067/ 144/ 1563 & WJ00 & 618X \\ Frost (ron) & NC/ Volta-Neger & West & 2M French & OGR, Nation, Heart, Main Luce, LBG & 267/ 122/ 1579 & WJ00 & 32k \\ Huang (hau) & Arto-Axiute / Chade & West & 6M English & Whitmart, NE Driess, Vispees & 1308/ 1300/ 1500 & WJ00 & 236K \\ lgu (ho) & NC/ Volta-Neger & West & 2M English & [Eczenii et al., 2020] & 6998/ 1500/ 1500 & WJ00 & 415K \\ Lang (laug) & NC/ Bamts & East & 2M English & Independent Uganda & 4075/ 1500/ 1500 & Babo & 31K \\ Lang (laug) & No-Sharma & East & 4M English & Lightve, Standard Media & 4265/ 1500/ 1500 & Babo & 31K \\ Moses (ros) & CM/ Car & West & North & Brushbaalan, Lefrano & 2287/ 1487/ 1524 & WJ00 & 216K \\ Nijaja (pmn) & English-Creole & West & 2M English & DailyTus Niepens & 4290/ 1484/ 1564 & WJ00 & 238K \\ Swedish (vava) & NC/ Bamts & East \& Cenual & 93M English & Global Vacc, OPUS & 3072/ 1791/ 1513 & WJ00 & 872K \\ Skewman (tsm) & NYE Barty & South & 14M English & SABNE News & 2100/ 1340/ 1500 & WJ00 & 870K \\ Alaa/Wit (ku1) & NC/ Kwa & West & 94M English & StartEM Pilot News & 3307/ 1284/ 1500 & WJ00 & 601K \\ Word (wooi) & NC/ Scengerphia & West & 5M French & Sweene, Rolling, YouthPos,Socailailleink & 3060/ 1500/ 1500 & Babo & 22K \\ Turkish(yor) & NC/ UMa-Neger & West & 42M English & [Adelman et al., 2021) & 6644/ 1544/ 1558 & WJ00 & 460K \\ JiaZhu (zu1) & NC/ Branta & South & 2M English & Mathuyu et al. (2021) & 3500/ 1287/ 998 & WJ00 & 667K \\ \end{tabular}
\end{table}
Table 1: **Languages and Data Details for MAFDAN-MT Corpus.** Language, family (NC: Niger-Congo), number of speakers, news source, news (NEWS), and religious domain (REL) data split. The languages highlighted in gray did not previously have news-domain data before MAFAND-MT.

 

five languages that are not in the JW300 corpus,3 we make use of the Bible.4 We aligned the sentences automatically by the verses (around 31k in total). Ghomala' only has the New Testament with 8k verses. Bambara and Wolof are missing some verses and books, leading to a total size of 28K and 22K. Table 1 summarizes this information about the religious (REL) corpora.

Footnote 3: Some languages like Luo and Luganda are covered by JW300 but are no longer available at the time of paper writing.

Footnote 4: Crawled/downloaded from [https://ebible.org/](https://ebible.org/), except for Bambara that we obtained from [https://live.bible.is/](https://live.bible.is/) and Ghomala’ from www.beblia.com

## 4 MAFAND-MT African News Corpus

### Data Collection Process

We introduce our newly translated news corpus; MAFAND-MT -- **M**asakhane **A**nglo & **E**ranco **A**frica **N**ews **D**ataset for **M**achine **T**ranslation. Table 1 gives the news source and data splits for 11 African languages which includes six languages (bm, bbj, ewe, fon, mos, wol) spoken predominantly in Francophone Africa and five languages (lug, luo, pcm, tsn, twi) spoken predominantly in Anglophone Africa. The MAFAND-MT corpus was created in three steps:

1. **Crawling and preprocessing** of news websites from local newspapers that are publishing in English and French. Raw texts from the web were segmented into sentences. Most languages were crawled from one or two sites, except for Wolof and Fon that were crawled from four and seven news websites respectively due to local French language newspapers having very few articles. We also ensured that the articles came from a variety of topics e.g. politics, sports, culture, technology, society, religion, and education. This was carried out by native speakers of the target language with source language proficiency.
2. **Translation** of 5k-8k sentences by professional translators.The translation process took one to four months depending on the availability of the translators.
3. **Quality control** was provided by native speakers, who discussed and, if possible, fixed problematic translations and ran automatic checks to detect misspellings, duplicated sentences, and alignment problems. Following the recommendations of \(\forall\) et al. (2020), we design the process to be _participatory_: Everyone involved in the corpus creation is a native speaker of the respective target languages and has societal knowledge about the communities that speak those languages. This is particularly important for curation and quality control to ensure that the resulting material is appropriate and relevant for stakeholders of the final MT models (\(\forall\) et al., 2020; Kreutzer et al., 2021). Furthermore, everyone received appropriate remuneration. To enable cross-disciplinary knowledge transfer between participants in the individual steps, every language was assigned a coordinator. The coordinator conducted the initial curation in the first step, and communicated with translators and quality checkers throughout the following steps.

Other Available Parallel Corpora.We found five African languages with available parallel texts in the news domain: Hausa5, Igbo (Ezeani et al., 2020), Swahili6, Yorbua (Adelani et al., 2021), and isiZulu (Mabuya et al., 2021). Table 1 provides news source, the TRAIN, DEV and TEST splits. Appendix B provides details on the pre-processing of the available news corpora.

Footnote 5: [https://www.statmt.org/wmt21/translation-task.html](https://www.statmt.org/wmt21/translation-task.html)

Footnote 6: [https://sw.globalvoices.org/](https://sw.globalvoices.org/)

### Monolingual News Corpus

To adapt available multilingual pre-trained models via continued pre-training to African languages, we curated texts from the 17 highest-resourced African languages and three non-native African languages that are widely spoken on the continent (Arabic, English, and French). The selection of African languages is based on their coverage in mC4 (Xue et al., 2021), AfriBERTa corpora (Ogueji et al., 2021), and other publicly available news websites like VOA and BBC. We limited the size of the corpus extracted from mC4 to the first 30 million sentences (roughly 1GB of data) for Afrikaans, Amharic, Arabic, English, French, and Swahili. In total, we collected about 12.3 GB of data. Appendix C provides more details about the pre-training corpus.

## 5 Models and Methods

### Baseline Models

We experiment with pre-trained multilingual models and our own bilingual MT baselines. We focus 

on pre-trained models that are approximately 500M parameters, both for computational feasibility and comparability across various different models.

Transformer Baseline.We train Transformer Vaswani et al. (2017) sequence-to-sequence models from scratch for each language pair using JoeyNMT Kreutzer et al. (2019). We tokenize the bitext using a joint SentencePiece7 unigram model Kudo (2018), with a character coverage of 1.0 and a maximum sentence length of 4096 tokens and create a vocabulary of 10K subwords. Models are trained on the concatenation of REL and NEWS corpora for each language.

Pre-trained Models.We consider three language models, MT5 Xue et al. (2021b), ByT5 (a tokenizer T5) Xue et al. (2021a), mBART50 Tang et al. (2020), and the multilingual translation model M2M-100 Fan et al. (2021b) for our experiments. We use MT5-base and ByT5-base, and M2M-100 with 418M parameters. Table 2 gives the pretrained model size, number of African languages covered, and the focus languages supported.

### Transfer Learning Across Languages

We describe two methods for adding new languages to existing models: continual pre-training and many-to-many multilingual translation.

Continual Pre-training.The effectiveness of PLMs is limited on extremely low-resource languages because they rarely, if ever, occur in the pre-training corpus Wang et al. (2020); Liu et al. (2021). As shown in Table 2, even for MT5 and M2M-100, which cover 100 languages, less than half of the African languages under study are included. To adapt the existing PLMs to our languages corpora and domains, we apply continual pre-training Gururangan et al. (2020); Liu et al. (2021) using our collected monolingual corpus. Specifically, before fine-tuning on the parallel MT data, models are pre-trained with their original training objective and vocabulary8 on the monolingual corpus. Pre-training parameters can be found in the appendix. We refer to the models adapted to African languages as AfriMT5, AfriByT5, and AfriMBART.

Footnote 8: Changing the vocabulary Gururangan et al. (2020) to fit the languages, or adding MT-focused training objectives for word alignment Liu et al. (2021) can potentially improve the performance further, which we leave for future work.

Many-to-Many Translation.We fine-tuned M2M-100 for African multilingual translation to create English- and French-centric models. For the English-centric model, the M2M-100 model was fine-tuned on the news data for en={hau, ibo, lug, luo, pcm, swa, tsn, twi, yor, zul} while the French-centric model is trained on fr-{ham, bbj, ewe, fon, mos, wol}. Languages not included in the pre-trained M2M-100 model were assigned the language code of a language included in M2M-100 but excluded from our study.

### Transfer Learning Across Domains

As there is very limited MT data on the news domain, we compare different methods that combine the _large_ data from the religious domain (REL) and the _small_ data from the NEWS domain (NEWS) to fine-tune M2M-100:

1. REL+NEWS: Fine-tuning on the aggregation of REL and NEWS.
2. REL\(\rightarrow\)NEWS: Training on REL, followed by fine-tuning on NEWS.
3. REL+NEWS\(\rightarrow\)NEWS: REL+NEWS, followed by additional fine-tuning on NEWS.

Each fine-tuning stage lasts for three epochs. We evaluate translation quality with BLEU Papineni et al. (2002) using SacreBLEU Post (2018)9 and ChrF Popovic (2015).

Footnote 9: “intl” tokenizer, all data comes undokenized.

## 6 Results and Discussion

We successfully adapt several multilingual pre-trained models to previously unseen African languages and quantify the effectiveness of small in-domain translation datasets. We discuss the effects of domain shift and analyze mitigation strategies.

### Adaptation to the Focus Languages

We demonstrate that fine-tuning with a few thousand high-quality bitext is effective for adding new languages to pre-trained models. Further, continuing to pre-train to specialize models to African languages further improves performance.

 

Zero-Shot Translation.Table 3 and Table 4 gives the result of zero-shot evaluation on NEWS. We evaluate only on the M2M-100 dataset because it has been pre-trained on parallel texts with a few of our focus languages. We observe very poor performance (\(<5\) BLEU) on the languages except for zul (\(>13\) BLEU) and swa (\(>20\) BLEU) in both translation directions. For swa, its likely that the performance is reasonable because M2M-100 has seen more bitext during pre-training (\(2.4\)M sentences in CCAligned El-Kishky et al. (2020)). Other African languages except for Afrikaans have less than 600K sentences in CCAligned, and are also of a lower quality Kreutzer et al. (2021) which affect overall zero-shot performance.

Performance after Fine-tuning.We found impressive performance after fine-tuning PLMs and M2M-100 on few thousand sentences (mostly 2K-7K sentences, except for swa with 30K sentences), including languages not seen during pre-training. For _eu/fr-xx_, MT5 has a poor transfer performance with average BLEU of \(7.2\), despite being pretrained on 101 languages. ByT5 outperforms MT5 by over \(3\) BLEU on average, even though their performances were reported to be similar in previous work Xue et al. (2021). This indicates that ByT5 might be preferable over MT5 when translating low-resource languages. Surprisingly, mBART50 that was only pre-trained on 50 languages and 2 African languages outperformed MT5 and ByT5 which are pre-trained on 101 languages. Overall, we found M2M-100 to be the best model, most likely because it was pre-trained on a translation task. In general, BLEU scores are relatively low (\(<15\) BLEU for 9 out of 16 languages for en/fr-xx and 7 in xx-en/fr) even when fine-tuning M2M-100 on in-domain data, which suggests that developing more effective methods for fine-tuning might be a promising future direction. The languages with the best quality according to BLEU on the target side are pcm, swa and tsn, and pcm, zul, and swa on the source side.

BLEU scores are higher when translating from an African language, which is expected due to the more frequent exposure to English and French on the target side during pre-training, and BLEU being penalized more for morphologically rich languages like bbj, lug, swa, tsn, and zul). The ChrF metric works better for them. For example, fine-tuning M2M-100 on NEWS and evaluating on zul has a BLEU of \(21.0\) in _eu/fr-xx_, and BLEU of \(37.8\) in the _xx-en/fr_ showing a large gap in performance in both directions. However, with the ChrF, we find a smaller performance gap (\(51.2\) in _eu/fr-xx_ and \(55.5\) in the _xx-en/fr_.

Continual Pre-training.We observe an improvement in BLEU when we utilize AfriMT5 and AfriByT5, for languages included in our continual pre-training corpus (Appendix C). Other languages also benefit despite not being seen during continual pre-training, possibly due to language similarity. For example, AfriByT5 on _fr-bam_ improved by \(1.9\) BLEU over ByT5 and AfriMT5 on _en-tsn_ improved by \(3.6\) BLEU over MT5. On average, AfriMT5 im

\begin{table}
\begin{tabular}{l c c c c c c|c c c c c c c c c c c c} \hline \hline  & & & \multicolumn{2}{c}{_e/gr-xx_} & \\
**Model** & **bm** & **bbj** & **ewe** & **fon** & **mos** & **woul** & **han** & **lbg** & **lag** & **locp** & **sem** & **swa** & **tsu** & **vqr** & **all** & **AVG** & **MED** \\ \hline \multicolumn{11}{c}{**BLEU**} \\ \hline MEM-100 0-hot & – & – & – & – & – & 1.3 & 0.4 & 2.8 & – & – & – & 20.1 & 1.1 & – & 2.1 & 5.6 & – & \\ \hline MTS & \(1.5\) & \(0.4\) & \(2.2\) & \(1.6\) & \(0.1\) & \(0.9\) & \(2.8\) & \(18.0\) & \(3.0\) & \(3.1\) & \(34.1\) & \(25.1\) & \(3.4\) & \(1.7\) & \(4.8\) & \(11.7\) & \(7.2\) & \(2.9\) \\ AfriMITS & \(2.1\) & \(0.8\) & \(3.7\) & \(2.5\) & \(0.1\) & \(1.8\) & \(5.1\) & \(19.6\) & \(5.2\) & \(4.6\) & **35.0** & **26.7** & \(7.0\) & \(2.7\) & \(6.2\) & \(13.2\) & \(8.5\) & \(4.8\) \\ \hline BYTS & \(9.5\) & \(1.8\) & \(5.5\) & \(3.8\) & \(0.1\) & \(0.0\) & \(8.3\) & \(21.8\) & \(12.1\) & \(8.4\) & \(30.1\) & \(24.4\) & \(14.7\) & \(6.0\) & \(7.5\) & \(14.0\) & \(10.9\) & \(8.4\) \\ AfriHyT5 & \(11.4\) & \(2.2\) & \(5.2\) & \(3.7\) & \(0.2\) & \(6.4\) & \(9.3\) & \(22.7\) & \(13.1\) & \(8.9\) & \(30.0\) & \(24.7\) & \(17.0\) & \(6.1\) & \(7.6\) & \(15.3\) & \(11.5\) & \(9.1\) \\ \hline mBART50 & \(18.6\) & \(2.4\) & \(5.3\) & \(6.2\) & \(0.8\) & \(9.7\) & \(8.9\) & \(21.1\) & \(12.0\) & \(10.0\) & \(34.1\) & \(25.8\) & \(16.8\) & \(7.5\) & \(10.0\) & **21.2** & \(13.2\) & \(10.0\) \\ ArifMBART & \(15.3\) & \(2.4\) & \(5.4\) & \(7.4\) & \(0.6\) & \(8.6\) & \(10.4\) & \(22.4\) & \(10.0\) & \(9.8\) & \(30.0\) & \(22.7\) & \(12.8\) & \(6.3\) & \(9.6\) & \(20.1\) & \(11.9\) & \(9.9\) \\ \hline MEM-100 & **22.7** & **2.9** & **6.4** & **7.1** & **1.0** & **12.4** & **16.0** & **24.7** & **14.3** & **11.5** & **33.9** & **26.7** & **24.7** & **8.8** & **12.8** & \(21.0\) & **15.4** & **18.6** \\ MEM-100-ENFR & \(18.5\) & \(2.2\) & \(6.2\) & \(4.3\) & \(0.8\) & \(10.6\) & \(7.0\) & \(22.4\) & \(8.9\) & \(9.5\) & \(34.9\) & \(26.4\) & \(19.7\) & \(7.0\) & \(5.6\) & \(15.6\) & \(12.5\) & \(9.2\) \\ \hline \hline \multicolumn{11}{c}{**CIRNF**} \\ \hline MEM-100 0-hot & – & – & – & – & \(-4.3\) & \(12.4\) & \(19.0\) & – & – & – & \(47.7\) & \(8.7\) & – & \(10.4\) & \(20.1\) & – \\ \hline MTS & \(10.0\) & \(7.4\) & \(9.7\) & \(11.5\) & \(7.9\) & \(9.1\) & \(23.6\) & \(41.1\) & \(24.9\) & \(21.6\) & \(6.1\) & \(53.7\) & \(22.8\) & \(17.8\) & \(20.8\) & \(36.0\) & \(23.9\) & \(21.2\) \\ AfriMS & \(14.0\) & \(12.7\) & \(16.6\) & \(14.8\) & \(8.2\) & \(13.8\) & \(29.7\) & \(43.1\) & \(30.4\) & \(25.7\) & **64.7** & \(55.1\) & \(31.5\) & \(24.5\) & \(24.3\) & \(40.3\) & \(27.9\) & \(25.0\) \\ \hline DyT5 & \(27.8\) & \(17.7\) & \(23.8\) & \(16.1\) & \(8.8\) & \(22.9\) & \(31.3\) & \(46.5\) & \(40.0\) & \(32.2\) & \(58.1\) & \(52.5\) & \(38.6\) & \(27.9\) & \(25.5\) & \(40.3\) & \(31.9\) & \(29.6\) \\ AfrihyT5 & \(31.4\) 

proved over MT5 by \(1.3\) BLEU in en/fr-xx and \(2.4\) BLEU in the _xx-en/fr_. The improvement for AfriByT5 was much smaller: \(0.6\) and \(0.9\) BLEU in _en/fr-xx_ and _xx-en/fr_ translation directions. For AfriMBART, we did not see any improvement on average, only the performance of hau (\(1.5\) BLEU) and ibo (\(0.7\) BLEU) improved in _en/fr-xx_ direction. However, in the _xx-en/fr_ direction, fon, tsn, twi, and zul improved by 2.7-6.0 BLEU.

Many-to-Many Multilingual MT.Training on the combined news corpus from all languages that use French or English separately does not appear to help much. We see slight improvements for most languages only in the _xx-en/fr_ direction.

### Adaptation to the News Domain

To improve over the baseline performance on NEWS, we train bilingual Transformer models (as a baseline) and M2M-100 on a combination of REL and NEWS. We chose M2M-100 because it was the best performing model. Table 5 gives the BLEU on three settings: REL+NEWS, REL\(\rightarrow\)NEWS, and REL+NEWS\(\rightarrow\)NEWS. In general, the improvement depends on the size of REL corpus. For languages trained on the Bible such as bbj, bam, lug, luo, and wol, the improvement is minimal. For M2M-100, the REL+NEWS performance does not improve over NEWS despite the larger quantity of training data. This demonstrates that increasing the size in the target domain is the most helpful strategy (see Figure 2). Similarly, combining REL+NEWS is not very helpful for xx-en/fr.An alternative approach is REL\(\rightarrow\)NEWS, which allows the model to develop a good understanding of the desired language before adapting to the news domain. We observe an increase on \(1.1\) BLEU over REL+NEWS in the en/fr-xx direction. However, the best strategy is REL+NEWS\(\rightarrow\)NEWS, especially for xx-en/fr where it yields an improvement over NEWS and REL+NEWS by \(2.0\) and \(1.5\) BLEU, respectively.

### Analysis of Domain Shift

Is a small in-domain set essential for finetuning?If we train models _only_ on previously available religious data, they are not capable of translating news well due to the strong _domain bias_. This is illustrated in Figure 1: All models perform much worse on NEW

Figure 1: **Domain shift of M2M-100 Transformer models trained on en/fr-xx (top) or xx-en/fr (bottom) REL domain and tested on the NEWS vs. REL domains.**



main. When the quantity of religious training data is small, the loss in translation performance on the news test set is largest, c.f. bbj (8k of REL data) with a drop of -95.5% BLEU or bam (-93.5%, 28k) and luo (-93.5%, 31k). This indicates that when the REL training data is sparse, it is insufficient to teach the M2M-100 model a more general understanding required for translating NEWS. However, when the religious training data is larger, this loss is reduced, c.f. when translating to zul (667k, -67%), swa (-69.3%, 872k), and tsn (-71%, 870k). While this is the general trend, pcm, whose religious training data is small (23k), has the lowest drop in performance (-59.3%), which may be due to the strong similarity to its source language.

How many sentences in the target domain are required?Figure 2 shows how for three selected language pairs with a large (fr-bam), medium (eng-ibo) and relatively small (eng-swa) domain gap, the quality of target domain translations improves as we increase the size of the target domain corpus. For all three pairs, fine-tuning M2M

\begin{table}
\begin{tabular}{l l} \hline _bmm-fp_ & \\ SRC & Ani k’a fou ye ka corancez tangao be sigii rutoloma saba kan. \\ TGT & Et leur dire que la transitione se propose sur trois piliers. \\ REL & Et qu quo ter dise que la puissance du milieu est sur trois sauttemelles. \\ R+N-N & Et de leur dire que la forcee de la transitione repose sur trois piliers. \\ \hline _lag-en_ \\ SRC & Murasaki Shikhbu yawandnika dkitabo ekikiyuvu akassookera \\ ddata mu sus jonas. \\ RT & Murasaki Shishuba wrote the world’s first full novel. \\ REL & And Monrshach Shikub writes a full scroll of the first in all the earth. \\ R+N-N & Murasaki Shikibu wrote a complete book first in the word. \\ \end{tabular}
\end{table}
Table 6: **Results adapting to Domain Shift, xx-en/fr**. We calculate BLEU and ChrF on the news domain when training on different combinations of REL and NEWS.

\begin{table}
\begin{tabular}{l c c c c c|c c c c c c c c c c c c c c} \hline \hline  & & \multicolumn{4}{c}{_x-fp_} & \multicolumn{4}{c}{_xx-cn_} \\
**Model** & **bam** & **bbj** & **ewe** & **fan** & **mos** & **wolf** & **han** & **libo** & **lnug** & **pem** & **swa** & **tsn** & **twil** & **yer** & **mtl** & **AVG** & **MED** \\ \hline Transformer & & & & & & & & & & & & & & & & & & & \\ REL+NEWS & 4.9 & 0.6 & 6.3 & 2.2 & 3.7 & 2.2 & 11.2 & 17.4 & 5.6 & 3.1 & 19.5 & 28.0 & 29.9 & 9.8 & 12.0 & 27.3 & 11.1 & 8.0 \\ REL+NEWS & 4.7 & 0.8 & 6.5 & 2.4 & 3.1 & 2.5 & 11.0 & 17.4 & 6.3 & 18.9 & 27.9 & 24.6 & 10.1 & 11.0 & 28.5 & 11.1 & 8.3 \\ REL+NEWS\(=\)NIRES & 5.8 & 1.0 & 7.1 & 2.4 & **4.1** & 2.6 & 13.2 & 18.2 & 6.8 & 3.7 & 21.4 & 28.7 & 24.5 & 10.4 & 12.6 & 30.1 & 12.0 & 8.8 \\ \hline MEM-100 & & & & & & & & & & & & & & & & & & & & \\ REL+NEWS & 24.0 & 5.8 & 10.9 & 9.7 & 2.3 & 10.1 & 15.3 & 21.1 & 21.1 & 13.3 & **4.6** & 29.4 & 27.0 & 12.5 & 17.4 & 30.6 & 18.4 & 16.4 \\ REL+NEWS & 20.3 & 5.9 & 11.4 & 9.6 & 2.3 & 10.5 & 17.4 & **21.9** & 20.6 & 13.7 & 44.3 & **30.6** & 27.7 & **13.2** & **18.0** & 36.0 & 19.0 & 17.7 \\ REL+NEWS\(=\)NIRES & **25.8** & **6.3** & **11.6** & **9.9** & 2.6 & **11.5** & **18.2** & 21.5 & **22.4** & **14.3** & 44.0 & 30.5 & **27.8** & **13.2** & **18.0** & **38.1** & **19.7** & **18.1** \\ \hline \hline \multicolumn{11}{c}{**CIRF**} \\ \hline REL+NHCS & 24.7 & 12.6 & 29.4 & 16.1 & **17.6** & 19.9 & 31.7 & 43.1 & 26.9 & 23.0 & 47.8 & 53.5 & 49.8 & 34.4 & 33.4 & 49.6 & 32.1 & 30.6 \\ REL+NEWS & 23.0 & 12.7 & 29.8 & 16.6 & 17.2 & 13.3 & 30.6 & 42.8 & 28.7 & 20.0 & 47.3 & 53.3 & 50.8 & 34.4 & 32.2 & 50.4 & 31.8 & 30.2 \\ REL+NEWS\(=\)NIRES & 26.5 & 14.7 & 30.7 & 17.6 & 18.8 & 21.8 & 33.8 & 44.0 & 29.5 & 24.7 & 50.8 & 54.1 & 50.6 & 35.1 & 34.4 & 51.4 & 33.7 & 32.2 \\ \hline MEM-100 & & & & & & & & & & & & & & & & & & & & & \\ REL+NEWS & 47.1 & 27.5 & 36.4 & 27.9 & 16.6 & 34.0 & 36.8 & 47.5 & 47.2 & 37.3 & **68.9** & 54.7 & 53.0 & 38.4 & 40.2 & 53.3 & 41.7 & 39.3 \\ REL+NEWS & 44.5 & 27.7 & 37.0 & 28.2 & 16.8 & 34.4 & 39.6 & **48.0** & 47.0 & 38.0 & 68.7 & **55.8** & 53.6 & **38.7** & 40.7 & 56.4 & 42.2 & 40.2 \\ REL+NEWS\(=\)NIRES & **49.0** & **28.5** & **37.2** & **28.9** & 17.2 & **35.3** & **40.2** & 47.9 & **48.5** & **38.3** & 68.6 & 55.7 & **54.0** & **38.7** & **41.0** & **57.7** & **42.9** & **40.6** \\ \hline \hline \end{tabular}
\end{table}
Table 7: **Example translations for M2M-100 finetuned on REL or REL+NEWS\(\rightarrow\)NEWS (R+N\(\rightarrow\)N). Terms in red are typical for bbilical texts, while the terms in blue are more neutral expressions.**

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c c c c c} \hline \hline  & & \multicolumn{4}{c}{_x-fp_} & \multicolumn{4}{c}{_xx-cn_} \\
**Model** & **bam** & **bbj** & **ewe** & **fan** & **moes** & **wolf** & **ham** & **libo** & **lnug** & **lmug** & **perm** & **swa** & **tsn** & **twil** & **yer** & **mtl** & **AVG** & **MED** \\ \hline Transformer & & & & & & & & & & & & & & & & & & & & & & & & & & & & & \\ REL+NEWS & 46.8 & 22.1 & 36.7 & 26.5 & 30.7 & 28.3 & 11.0 & 17.1 & 26.9 & 23.0 & 47.8 & 53.5 & 49.8 & 34.4 & 33.4 & 49.6 & 32.1 & 30.6 \\ REL+NEWS & 23.0 & 12.7 & 29.8 & 16.6 & 17.2 & 13.3 & 30.6 & 42.8 & 28.7 & 20.0 & 47.3 & 53.3 & 50.8 & 34.4 & 32.2 & 50.4 & 31.8 & 30.2 \\ REL+NEWS & 26.5 & 14.7 & 30.7 & 17.6 & 18.8 & 21.8 & 33.8 & 44.0 & 29.5 & 24.7 & 50.8 & 51.4 & 50.6 & 35.1 & 34.4 & 51.4 & 33.7 & 32.2 \\ \hline
**MEM-100** & & & & & & & & & & & & & & & & & & & & & & & \\ REL+ 

100 or ByT5 on 2.5\(k\) sentence pairs of in-domain data (NEWS) is sufficient to outperform the bilingual Transformer baselines that were additionally trained on larger amounts of out-of-domain data (REL). Surprisingly, this procedure not only works for languages included during pre-training (swa), but also for previously unseen languages (ibo, bam). M2M-100 tends to adapt to the new data more quickly than ByT5, but in all cases, models continue to learn with additional in-domain data. This shows how much more effectively a small number of in-domain translations can be used when they serve for fine-tuning multilingual pre-trained models rather than training bilingual MT models from scratch.

Examples of Domain Bias.To illustrate the challenge of overcoming domain bias, we show examples translating from bam and lug in Table 7. The M2M-100 model fine-tuned only on REL succeeds in roughly capturing the meaning of the sources, but using bibical terms, such as "scroll" instead of "novel". Adding our news corpus to fine-tuning resolves these issues (e.g. "book").

How general is our news corpus?Table 8 shows the zero-shot evaluation of M2M-100 fine-tuned on our small NEWS corpora on other domains: religious (REL) and Wikipedia (FLORES). We evaluated the Wikipedia domain on the FLORES _devtest_ and the REL domain on either JW300 or Bible (lug, luo, wol). As a baseline, we evaluated the zero-shot performance of M2M-100 (not fine-tuned, \({\mathbf{\mathcal{K}}}\)) on FLORES10 using spBLEU (i.e. sentencepiece BLEU [6]). We noticed very poor performance except for Swahili -- as discussed in SS6.1. After fine-tuning on our new data (\({\mathbf{\mathcal{missingmissing}}}\)), transfer is largely improved across the bench (up to +17 BLEU for en-ibo). The same trend holds for the religious domain. This shows that even though our data comes from the news domain, it helped the model generalize to other domains. Hence, expanding African news corpora and developing better MT models for news pays off even for other domains of interest.

Footnote 10: except for Luo which is not supported

## 7 Conclusion

We have created MAFAND-MT, a corpus of 16 African languages to study translation systems for low-resource languages in the news domain. We investigate how to most effectively adapt large-scale pre-trained models to incorporate new languages and new domains. Our findings suggest that as little as 2k sentences are sufficient for fine-tuning, with an improved performance, paving the way for others to create new translation systems without relying on large collections of web-sourced text. This has strong implications for languages that are spoken by millions but lack presence on the web.

In the future, we hope to expand our coverage to additional under-resourced languages, and to develop even more effective fine-tuning objectives. Currently, we are extending our corpus to Chichewa, Kinyarwanda, Shona, and isiXhosa, including an expansion of the Hausa corpus, they will be released under MAFAND-MT dataset name.

Figure 2: **Number of fine-tuning sentences needed to exceed the performance of a bilingual Transformer model.**



Acknowledgment

This work was carried out with support from Lacuna Fund, an initiative co-founded by The Rockefeller Foundation, Google.org, and Canada's International Development Research Centre. David Adelani acknowledges the EU-funded Horizon 2020 projects: COMPPRISE ([http://www.compriseh2020.eu/](http://www.compriseh2020.eu/)) under grant agreement No. 3081705 and ROXANNE under grant number 833635. We thank Chester Chester Palen-Michel and Constantine Lignos for providing the VOA corpus for this research, and Google for providing GCP credits to run some of the experiments. Finally, we thank Davor Orlic and Knowledge4All for their administrative support throughout the project.

## References

* D. Adelani, D. Ruiter, J. Alabi, D. Adebonojo, A. Ayeni, M. Adeyemi, A. Ester, C. Espana-Bonet, and C. Espana-Bonet (2021a)The effect of domain and diacritics in Yvonba-English neural machine translation. In Proceedings of Machine Translation Summit XVIII: Research Track, Virtual, pp. 61-75. External Links: Link Cited by: SS1.
* D. Ifeoluwa Adelani, J. Abbott, G. Neubig, D. D'souza, J. Kreutzer, C. Lignos, C. Talenchel, H. Buzzaba, S. Rijhwani, S. Ruder, S. Mayhew, I. Abbe Azime, S. H. Muhammad, C. Emzeue, J. Nakatumba-Nabende, P. Ogayo, A. Anouluwapo, C. Gitau, D. Mbaye, J. Alabi, S. Muhie Yimam, T. Raibl Gowadge, I. Ezenai, R. Niyongabo, J. Mukibi, V. Oriende, I. Orfex, D. Davigd, S. Ngom, T. Adewumi, P. Ryason, M. Adeyemi, G. Muriuki, E. Anebi, C. Chukwuneke, N. Odu, E. Peter Wairagala, S. Oyerinde, C. Siro, T. Saul Bateesan, T. Oloyede, Y. Wambui, V. Afindoe, D. Nabagerreka, M. Katsutime, A. Awokoya, M. MAGUPUPD, D. Gerbreydannes, H. Tilaye, K. Niwaike, D. Wolde, A. Faye, B. Sibanda, O. Ahia, B. F. P. Dossou, K. Ogueji, T. Ibrahim DIOP, A. Diallo, A. Akinfaderin, T. Marengerek, and S. Osei (2021b)MasakhaNER: named entity recognition for African languages. Transactions of the Association for Computational Linguistics9, pp. 1116-1131. External Links: ISSN 0018-93X5, Document, Link Cited by: SS1.
* Z. Agic and I. Vulic (2019)JW300: a wide-coverage parallel corpus for low-resource languages.. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 3204-3210. External Links: ISBN 978-3-540-32133-1, Document, Link Cited by: SS1.
* R. Aharoni, M. Johnson, and O. Firat (2019)Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 3874-3884. External Links: ISBN 978-3-540-32133-1, Document, Link Cited by: SS1.
* F. M. D. A. Ali, A. Gaines, and J. L. A. Malavi (2021)Towards a parallel corpus of portuguese and the bantu language emakhnuva of mozamabine. ArXivabs/2104.05753. External Links: Link Cited by: SS1.
* A. Anastasopoulos, A. Cattelan, Y. Dou, M. Federico, C. Federmann, D. Genzel, F. Guzzman, J. Hu, M. Hughes, P. Koehn, R. Lazar, W. Lewis, G. Neubig, M. Niu, A. Oktem, E. Paquin, G. Tang, and S. Tur (2020)PICO-19: the translation initiative for COvid-19. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Online, pp. . External Links: Link Cited by: SS1.
* P. Azunure, S. Osei, S. Addo, L. Asamoah Adu-Gyamfi, S. Moore, B. Adabankal, B. Opoku, C. Asare-Nyarko, S. Nyrakov, C. Ampoash, E. Dansoa Appiah, F. Akwerth, R. Nii Lante Lawson, J. Budu, E. Debrah, N. Boateng, W. Ofori, E. Buabeng-Munkoh, F. Adjjei, I. Kojo Essel Ampomah, J. Otoo, R. Borkor, M. Birago Mensah, L. Mesnah, M. Engla-Gomi, and J. Ben Hayfron-Acquah (2021)NIp for ghanian languages. AfricaNLP Workshop, pp. . External Links: Link Cited by: SS1.
* P. Azunure, S. Osei, S. Addo, L. Asamoah Adu-Gyamfi, S. Moore, B. Adabankal, B. Opoku, C. Asare-Nyarko, S. Nyrakov, C. Amoaba, E. Dansoa Appiah, F. Akwerth, R. N. Lane Lawson, J. Budu, E. Debrah, N. Adowaa Boateng, W. Ofori, E. Bugheng-Munkoh, F. Adjei, I. Kojo Essel Ampomah, J. Otoo, R. Norteby Broder, S. Apposah, J. Bosch, A. Morell, A. Moncari, A. Chepanog, A. Capnah, A. Chapina, A. Depnia, F. Akwerth, R. Nii Lante Lawson, J. Budu, E. Debrah, N. Adowaa Boateng, W. Ofori, E. Baubeng-Munkoh, F. Adjei, I. Kojo Essel Ampomah, J. Otoo, R. Borkov, S. Abdobalavo, A. Chepanog, A. Capnah, A. Depnia, F. Akwerth, R. Nii Lante Lawson, J. Budu, E. Debrah, N. Adowaa Boateng, W. Ofori, E. Bubeng-Munkoh, F. Adjei, I. Kojo Essel Ampomah, J. Otoo, R. Norteby Broder, S. Apposah, J. Bosch, A. Dragoi, A. Capnah, A. Depnia, F. Akwerth, R. Nii Lante Lawson, J. Budu, E. Debrah, N. Adowaa Boateng, W. Ofori, E. Bouabeng-Munkoh, F. Adjei, I. Kojo Essel Ampomah, J. Otoo, R. Norteby Broder, S. Apposah, J. Bosch, A. Dragoi, A. Capnah, A. Depnia, F. Akwerth, R. Nii Lante Lawson, J. Budu, E. Debrah, N. Adowaa Boateng, W. Ofori, E. Busheng-Munkoh, F. Adjei, I. Kojo Essel Ampomah, J. Otoo, R. Norteby Broder, S. Apposah, J. Bosch, A. Dragoi, A. Capnah, A. Depnia, F. Akwerth, R. Nii Lante Lawson, J. Budu, E. Debrah, N. Adowaa Boateng, W. Ofori, E. Busheng-Munkoh, F. Adjei, I. Kojo Essel Ampomah, J. Bosch, A. Dragoi, A. Capnah, A.

 

Sevi Sarisik, Peggy van der Kreft, and Kay Macquarrie. 2021. Surprise language challenge: Developing a neural machine translation system between Pashto and English in two months. In _Proceedings of Machine Translation Summit XVIII: Research Track_, pages 92-102, Virtual. Association for Machine Translation in the Americas.
* Bird (2020) Steven Bird. 2020. Decolonising speech and language technology. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 3504-3519, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Cahyawijaya et al. (2021) Samuel Cahyawijaya, Genta Indra Winata, Bryan Wilie, Karissa Vincentio, Xiaohong Li, Adhiguna Kuncoro, Sebastian Ruder, Zhi Yuan Lim, Syafri Bahar, Masayu Khodra, Ayu Purwarianti, and Pascale Fung. 2021. IndoNLG: Benchmark and resources for evaluating Indonesian natural language generation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 8875-8898, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Callison-Burch et al. (2008) Chris Callison-Burch, Philipp Koehn, Christof Monz, Josh Schroeder, and Cameron Shaw Fordye, editors. _Proceedings of the Third Workshop on Statistical Machine Translation_. Association for Computational Linguistics, Columbus, Ohio.
* El-Kishky et al. (2020) Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzman, and Philipp Koehn. 2020. CCAligned: A massive collection of cross-lingual web-document pairs. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020)_, pages 5960-5969, Online. Association for Computational Linguistics.
* Emezue and Dossou (2021) Chris Chinenye Emezue and Bonaventure F. P. Dossou. 2021. MMTAfrica: Multilingual machine translation for African languages. In _Proceedings of the Sixth Conference on Machine Translation_, pages 398-411, Online. Association for Computational Linguistics.
* Emzeue and Bonaventure Dossou (2020) Chris Chinenye Emezue and Femi Pancrace Bonaventure Dossou. 2020. FFR v1.1: Fon-French neural machine translation. In _Proceedings of the The Fourth Widening Natural Language Processing Workshop_, pages 83-87, Seattle, USA. Association for Computational Linguistics.
* Ezeani et al. (2020) Ignatius Ezeani, Paul Rayson, I. Onyenwe, C. Uchechukwu, and M. Hepple. 2020. Igboenglish machine translation: An evaluation benchmark. _ArXiv_, abs/2004.00648.
* Fan et al. (2021a) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin. 2021a. Beyond english-centric multilingual machine translation. _Journal of Machine Learning Research_, 22(107):1-48.
* Fan et al. (2021b) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiya Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chandhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin. 2021b. Beyond english-centric multilingual machine translation. _Journal of Machine Learning Research_, 22(107):1-48.
* Fan et al. (2021c) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiya Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chandhary, et al. 2021c. Beyond english-centric multilingual machine translation. _Journal of Machine Learning Research_, 22(107):1-48.
* Firat et al. (2016) Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016. Multi-way, multilingual neural machine translation with a shared attention mechanism. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 866-875, San Diego, California. Association for Computational Linguistics.
* Vik et al. (2020) Vilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Taiwo Fagbonbugbe, Solomon Oluwole Akinola, Shamsuddeen Muhammad, Salomon Kabongo Kobenamualu, Saolomey Osei, Freshia Sackey, Rubango Andre Niyongabo, Ricky Macharm, Perez Ogayo, Oreayoghene Ahia, Musseres Berne, Medotlova Adegemi, Masabata Mokgesi-Selinga, Lawrence Okgebemi, Laura Martinus, Kolawole Tajudeen, Kevin Degila, Kelechi Ogueji, Kathleen Siminyu, Julia Kreutzer, Jason Webster, Jamil Toure Ali, Jade Abbott, Iroro Orife, Ignatius Ezeani, Idris Abdulkadir Damgana, Herman Kamper, Hady Elshazr, Goodness Duru, Ghollah Kioko, Murhabazi Espoiri, Elan van Biljon, Daniel Whitenack, Christopher Onyefuluchi, Chris Chinenye Emezue, Bonaventure F. P. Dossou, Blessing Sibanda, Blessing Bassey, Ayodele Olabiyi, Arshath Ramkilowan, Alp Oktem, Adewale Akinfaderin, and Abdallah Bashir. 2020. Participatory research for low-resourced machine translation: A case study in African languages. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, Online.
* Gezmu et al. (2021) Andargachew Mekonnen Gezmu, A. Nurnberger, and Tesfaye Bayu Batti. 2021. Extended parallel corpus for amharic-english machine translation. _ArXiv_, abs/2104.03543.
* Gowda et al. (2021) Thamme Gowda, Zhao Zhang, Chris Mattmann, and Jonathan May. 2021. Many-to-English machine translation tools, data, and pretrained models. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_, pages 306-316, Online. Association for Computational Linguistics.
* Goyal et al. (2021) Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjan Krishnan, Marc'Aurelio Ranzato, Francisco Guzman, and Angela Fan. 2021. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. _ArXiv_, abs/2106.03193.
* Gururangan et al. (2020) Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In 

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Vol. 109, pp. 342-360. Cited by: SS1.
* B. Haddow, R. Bowden, A. Valerio Miceli Barone, J. Helcl, and A. Birch (2021)Survey of low-resource machine translation. ArXivabs/2109.00486. External Links: Link, 2109.00486 Cited by: SS1.
* H. Hassan, A. Aue, C. Chen, V. Chowdhary, J. Clark, C. Federmann, X. Huang, M. Junczys-Dowmunt, W. Lewis, M. Li, S. Liu, T. Liu, R. Luo, A. Menezes, T. Qin, F. Seide, X. Tan, F. Tian, L. Wu, S. Wu, Y. Xia, D. Zhang, Z. Zhang, and M. Zhou (2018)A achieving human parity on automatic chinese to english news translation. CoRRabs/1803.05567. External Links: Link, 1803.05567 Cited by: SS1.
* M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu, Z. Chen, N. Thorat, F. Viegas, M. Wattenberg, G. Corrado, M. Hughes, and J. Dean (2017)Google's multilingual neural machine translation system: enabling zero-shot translation. Transactions of the Association for Computational Linguistics5, pp. 339-351. External Links: Document, ISSN 1574-469X, Link Cited by: SS1.
* P. Joshi, S. Santy, A. Budhiraja, K. Bali, and M. Choudhury (2020)The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 6282-6293. External Links: Link, 2001.00266 Cited by: SS1.
* W. Ko, A. El-Kishky, A. Renduchintala, V. Chaudhary, N. Goyal, F. Guzzan, P. Fung, P. Koehn, and M. Diab (2021)Adapting high-resource NMT models to translate low-resource related languages without parallel data. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 802-812. External Links: Link, 2109.00486 Cited by: SS1.
* J. Kreutzer, I. Caswell, L. Wang, A. Wahab, D. van Esch, N. Ulizi-Orshikh, A. Augustse Tapo, N. Subramanian, A. Sokolov, C. Sikisoste, M. Setyawan, S. Sarin, S. Samb, E. Sagot, C. Rivera, A. Rios Gonzales, I. Papadimitriou, S. Osei, P. Oritz Suarez, I. Orife, K. Ogueji, R. Andre Niyongabo, Toan Q. Nguyen, M. Muller, A. Muller, S. M. Mohammad, S. Mnagia, J. Usairu, A. Graves, A. Mohamed, A. Mnichlau, A. Mnih, J. Mirza, A. Mnichlau, A. Mnih, J. Mirza, A. Mnih, J.

 

Coto-Solano, Alexis Palmer, Elisaabeth Mager-Hois, Vishrav Chaudhary, Graham Neubig, Ngoc Thang Vu, and Katharina Kann. 2021. Findings of the AmericansNLP 2021 shared task on open machine translation for indigenous languages of the Americas. In _Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas_, pages 202-217, Online. Association for Computational Linguistics.
* Marchisio et al. (2020) Kelly Marchisio, Kevin Duh, and Philipp Koehn. 2020. When does unsupervised machine translation work? In _Proceedings of the Fifth Conference on Machine Translation_, pages 571-583, Online. Association for Computational Linguistics.
* McCarthy et al. (2020) Arya D. McCarthy, Rachel Wicks, Dylan Lewis, Aaron Mueller, Winston Wu, Oliver Adams, Garrett Nicolai, Matt Post, and David Yarowsky. 2020. The Johns Hopkins University Bible corpus: 1600+ tongues for typological exploration. In _Proceedings of the 12th Language Resources and Evaluation Conference_, pages 2884-2892, Marseille, France. European Language Resources Association.
* Niyongabo et al. (2020) Rubungo Andre Niyongabo, Qu Hong, Julia Kreutzer, and Li Huang. 2020. KINNEWS and KIRNEWS: Benchmarking cross-lingual text classification for Kinyardana and Kirundi. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 5507-5521, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Nyoni and Bassett (2021) Evander Nyoni and Bruce A. Bassett. 2021. Low-resource neural machine translation for southern african languages. _ArXiv_, abs/2104.00366.
* Ogueji et al. (2021) Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. 2021. Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages. In _Proceedings of the 1st Workshop on Multilingual Representation Learning_, pages 116-126, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Oktem et al. (2021) Alp Oktem, Eric DeLuca, Rodriguez Bashizi, Eric Paquin, and Grace Tang. 2021. Congolese swahili machine translation for humanitarian response. _AfricaNLP Workshop_.
* Oktem et al. (2020) Alp Oktem, Mirko Plitt, and Grace Tang. 2020. Tigrinya neural machine translation with transfer learning for humanitarian response. _AfricaNLP Workshop_.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Paullada (2020) Amandalynne Paullada. 2020. How does machine translation shift power? _Resistance in AI Workshop_.
* Popovic (2015) Maja Popovic. 2015. chrF: character n-gram F-score for automatic MT evaluation. In _Proceedings of the Tenth Workshop on Statistical Machine Translation_, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
* Post (2018) Matt Post. 2018. A call for clarity in reporting BLEU scores. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
* Reid et al. (2021) Machel Reid, Junjie Hu, Graham Neubig, and Yutaka Matsuo. 2021. AfroMT: Pretraining strategies and reproducible benchmarks for translation of \(8\) African languages. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 1306-1320, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Schwenk et al. (2021a) Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzman. 2021a. WikitMatrix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 1351-1361, Online. Association for Computational Linguistics.
* Schwenk et al. (2021b) Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021b. CCMatrix: Mining billions of high-quality parallel sentences on the web. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6490-6500, Online. Association for Computational Linguistics.
* african language program. _ArXiv_, abs/2104.02516.
* Tang et al. (2020) Y. Tang, C. Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. _ArXiv_, abs/2008.00401 

Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the untainable? reassessing claims of human parity in neural machine translation. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 113-123, Brussels, Belgium. Association for Computational Linguistics.
* Tran et al. (2021) Chau Tran, Shruti Bhosale, James Cross, Philipp Koehn, Sergey Edunov, and Angela Fan. 2021. Facebook ai wmt21 news translation task submission. _arXiv preprint arXiv:2108.03265_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 5998-6008.
* Wang et al. (2020) Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020. Extending multilingual BERT to low-resource languages. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 2649-2656, Online. Association for Computational Linguistics.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remii Louf, Morgan Funtowicz, Joe Davison, Sam Shelfeir, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Phu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Xue et al. (2021a) Linting Xue, Aditya Barua, Noah Constant, Rami Alfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2021a. Byt5: Towards a token-free future with pre-trained byte-to-byte models. _ArXiv_, abs/2105.13626.
* Xue et al. (2021b) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhart, Aditya Barua, and Colin Raffel. 2021b. mT5: A massively multilingual pre-trained text-to-text transformer. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 483-498, Online. Association for Computational Linguistics.
* Yang et al. (2021) Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, and Furu Wei. 2021. Multilingual machine translation systems from microsoft for wmt21 shared task. _ArXiv_, abs/2111.02086.

## Appendix A Language Characteristics

Table 9 provides the details about the language characteristics.

## Appendix B Available Parallel Corpora

We found Five African languages with publicly available parallel texts in the news domain: Hausa, Igbo, Swahili, Yoruba, and isiZulu. Table 1 provides news source, the TRAIN, DEV and TEST splits.

**Hausa** The Hausa Khamehei11 corpus contains 5,898 sentences, we split them into TRAIN (3,098), DEV (1,300), and TEST split (1,500).

Footnote 111: [https://www.statmt.org/wmt21/translation-task.html](https://www.statmt.org/wmt21/translation-task.html)

IgooThe Igbo corpus (Ezeani et al., 2020) has 9,998 sentences, we extract 6,998 sentences for TRAIN, and the remaining for DEV and TEST splits.

SwahiliThe Global Voices12 corpus contains 30,782 sentences, which we use for the TRAIN split. We additionally crawled newer (2019-2021) publications of Swahili articles from the Global Voices website, this gives a total of 3,626 sentences, they were aligned and manually verified by Swahili speakers. They are split into the DEV and TEST splits.

Footnote 12: [https://sw.globalvoices.org/](https://sw.globalvoices.org/)

YorubaThe DENYO-20k (Adelani et al., 2021a) corpus contains sentences from different domains (TED talks, books, software localization, proverbs, and news), from which we select the news domain sentences for the TRAIN, DEV and TEST splits.

isiZuluThe Umsuka corpus (Mabuya et al., 2021) contains 9,703 training sentences and 1,984 evaluation sentences. 4,739 training sentences were translated from English-isziZulu, and the remaining from isiZulu-English. We only keep the training sentences translated into isiZulu, and split them into 3,500 for TRAIN and 1,239 sentences for DEV. From the existing evaluation set we select only the 998 English-isiZulu translations for TEST. Umsuka provides two translations for each English sentence, but we use only the first.

## Appendix C Monolingual Corpus PLMs adaptation

Table 10 provides the details about the Monolingual corpus used to adapt the pre-trained language models (PLMs), their size and source of corpora. The African languages pre-trained are: Afrikaans, Amharic, Hausa, Igbo, Malagasy, Chichewa, Oromo, Naija, Kinyardanda, Kirundi,



Shona, Somali, Sesotho, Swahili, isiXhosa, Yoruba, and isiZulu.

## Appendix D Model Hyper-parameters and Reproducibility of Results

For the pre-trained models, we fine-tune the models using HuggingFace transformer tool (Wolf et al., 2020) with the default learning rate (\(5e-5\)), batch size of \(10\), maximum source length & maximum target length of \(200\), beam size of \(10\), and number of epochs is \(3\) except for models trained on only NEWS which we set to \(10\). We make All the experiments were performed on a single GPU (Nvidia V100).

For fine-tuning pre-trained models, especially for mBART50 that only supports two African languages, the target language is required to be specified during decoding from among those that the model has seen during pre-training, we follow past works (Madaan et al., 2020; Cahyawijaya et al., 2021; Lee et al., 2022) in selecting another closely-related language that is represented in the pre-trained model. For convenience, we make use of Swahili (sw) as the target language when an African language is not represented since Swahili is represented in all the pre-trained models. The only exception is Nigerian-Pidgin, where we make use of French (fr) since it is closely related to English. When a language is represented in the pre-trained model like M2M-100 has seen Yorba (yo), we make use of the correct language code.

To train AfriMT5 and ByT5, we start with MT5 and ByT5. We pre-train with the learning rate \(1e-4\), \(10,000\) warm up steps and a batch size of \(2048\) for one epoch. For mBART50, we pre-train with learning rate of \(5e-5\) for \(50,000\) steps

\begin{table}
\begin{tabular}{l l c r} \hline \hline
**Language** & **Source** & **Size (MB)** & **No. of sentences** \\ \hline Affikans (aft) & mcC4 (subset) (Xue et al., 2021b) & 752.2MB & 3.697,430 \\ Ambaric (amb) & mcC4 (subset), and VOA & 1,300MB & 2.913,801 \\ Arabic (ran) & mcC4 (subset) & 1,300MB & 3.393,75 \\ English (eng) & mcC4 (subset), and VOA & 2,200MB & 8.626,571 \\ French (fra) & mcC4 (subset), and VOA & 960MB & 4.731,196 \\ Hausau (bau) & mcC4 (all), and VOA & 594.1MB & 3,290,382 \\ Igbo (ibo) & mcC4 (all), and ArtifBERn Corpus (Ogueji et al., 2021) & 287.5MB & 1.534,825 \\ Malaguys & mcC4 (all) & 639.68MB & 3.044,59 \\ Chichewa (nya) & mcC4 (all), Chickew News Corpus (Siminyuk et al., 2021) & 373.8MS & 2.203,040 \\ Moreno (orm) & afBiERTA Corpus, and VOA & 67.3MNg & 490,399 \\ Najja (cyan) & afBiERTA Corpus, and VOA & 54.8MB & 166.842 \\ Rwanda-Rundti (kir/kin) & afBiERTA Corpus, KINNEWS \& KIRNEWS (Niyongabo et al., 2020), and VOA & 84MB & 303,838 \\ Showa (ana) & mcC4 (all), and VOA & 545.2MB & 2.693,028 \\ Somali (son) & mcC4 (all), and VOA & 1,000MB & 3.480,960 \\ Sessofto (aco) & mcC4 (all) & 234MBs & 1.107,565 \\ Swahlii (syn) & mcC4 (all) & 823.5MB & 4.220,346 \\ iaiXhosa (sho) & mcC4 (all), and Isolerwe Newspaper & 178.4MB & 832,954 \\ Yorkio (yet) & mcC4 (all), Alzueye News, Asajcre News, Awikonko News, BBC, and VON (Adelani et al., 2021) & 179.3MB & 897.299 \\ siZulu (zul) & mcC4 (all), and Isolerwe Newspaper & 700.7MB & 3.252,035 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Monolingual Corpora (after pre-processing – we followed AfriBERTa (Ogueji et al., 2021) approach) , their sources and size (MB), and number of sentences.

\begin{table}
\begin{tabular}{l l l l l l l} \hline \hline  & **No. of** & **Latin Letters** & **Letters** & **Latters** & \multicolumn{1}{l}{**Sentence**} \\
**Language** & **Letters** & **Omitted** & **added** & **Tanality** & **diacrites** & **morphology** & **structure** \\ \hline Bambara (hum) & 27 & q,v,x & _r, z, y, p, n_ & yes, 2 tones & yes & isolating & SVO \& SOV \\ Ghondil® (bbj) & 40 & q, w, x, y & by, dz, o, u, e, gh, my, nt, y, sk, o, pf, & yes, 5 tones & yes & agglutinative & SVO \\ Enw! (ewey) & 35 & c, j, q & d, dr, _z, f, y_ & b, x, h, y, z, y, a, ts, v & yes, 3 tones & yes & isolating & SVO \\ Fon (ron) & 33 & q & q, e, g, hpk, br, y, x, sw & yes, 3 tones & yes & isolating & SVO \\ Hausua (hau) & 4 & p,q,v,x & f, d, fc, y, kw, gw, ky, f, gy, sh, ts & yes, 2 tones & no & agglutinative & SVO \\ Iglbo (ibo) & 34 & c, q, x, t & ch, gh, gn, sp, kp, tw, nv, ug, o, sh, u & yes, 2 tones & yes & agglutinative & SVO \\ Laguna (bag) & 25 & b, x, q & n, y & yes, 3 tones & no & agglutinative & SVO \\ Lao (luo) & 31 & c, q, x, v, z & ch, dh, mb, nd, ng’, mg, np, th, sh & yes, 4 tones & no & agglutinative & SVO \\ Mossi (mos) & 26 & c, j, q & x & yes, 2 tones & yes & isolating & SVO \\ Naija (pm) & 26 & – & – & – & no & no & mostly analytic & SVO \\ Swahili (swa) & 33 & x, q & dn, ch, gh, kn, neg’, sny, sh, tn, ts & no & yes & agglutinative & SVO \\ Setswana (nsa) & 36 & c, q, v, x, z & kg, kh, ng, avg, op, ph, s, tl, th, ts, ts, ti, st, th & yes, 2 tones & no & agglutinative & SVO \\ Akam-Tvi (wi) & 22 & c,j,4v,x,x & r, \(z\) & yes, 5 tones & no & isolating & SVO \\ Wolof (woul) & 29 & b, v,x & p, a, _b, c, \(d\) & 0, h, cg, \(e\) & 0, h, cg, \(e\) & yes & agglutinative & SVO \\ Yorbità (gor) & 25 & c, q, v, x, z & q, sb, g, s & \(o\) & yes & isolating & SVO \\ issiZulu (zul) & 55 & – & ns, rs, na, ph, th, hy, sq, nl, bn, ni, ch, & yes, 3 tones & no & agglutinative & SVO \\  & & neg, neq, ng, ta, ks, nl, nth, sh, hs, th, sg, nk, ss, hi, th, sg, nk, gs, nx, sh, gc, nn, dl, nc, qh & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 9: Linguistic Characteristics of the Languages 

using Fairseq Ott et al. (2019) without modifying the mBART50 vocabulary. Table 11 has the names of all the models that are publicly available on HuggingFace Model Hub 13. In total, we have 357 models from 22 x 16 bilingual models, two English/French-centric models, and three adapted models to African languages (i.e AfriMT5, AfriByT5, and AfriMBART).

Footnote 13: [https://huggingface.co/masakhane](https://huggingface.co/masakhane)

## Appendix E BLEU vs spBLEU

Table 12 and Table 13 compares BLEU and spBLEU metric for the domain transfer experiments. We observe that spBLEU gives higher scores than BLEU especially in the direction of _en/fr-xx_, which shows that it may be better for evaluating African languages. Although, further analysis and human evaluation are still needed to show that spBLEU is generally better. On the other hand, in the _xx-en/fr_, there is no much difference in the scores between BLEU and spBLEU.

## Appendix F Qualitative Analysis

The following examples from the Fon-to-French translations of the test set illustrate the advantage of multilingual modeling and its limitations:

* **Source** (fon): Louis Guy Alimanyiqkpo kpodlssa Etchlekoun kpc o, sin azan mkpan dye a, ye do wuve se we tawun d ob agbaza me, co ye ka tuun ffi e azan ne lez gosin e a.
* **Reference** (fr): Les faits Louis Guy Alimagnidokopo et Issa Etchlekoun se plaignent depuis quelques jours de multiples douleurs, ignorant l'origine reelle de leurs maux.
* **Bilingual Transformer** (REL+NEWS, fon\(\rightarrow\)fr): on ne peut pas avoir une trendaine d'annees ni un jeune homme ni un jeune homme d'age pour un jeune homme qui soit 12 ans.
* **MZM-100** (REL+NEWS\(\rightarrow\)NEWS, fon\(\rightarrow\)fr): Louis Guy Alimanyion et Issa Etchlekoun ont depuis plusieurs jours souffert d'une maladie grave malgre les consequences de cette maladie qu'ils ne connaissent pas.
* **MZM-100** (REL+NEWS\(\rightarrow\)NEWS, fr\(\rightarrow\)fron): Sin azan yyyweyre de dye dokpoo we en e kan Louis Guy Alimangidokop kpodo Issa Etchlek&n kpan de 3 {ba xo q [y wuv e gege w wc, ye ka tuun n n u vuv teynq de 3 a.

The translation of the bilingual Transformer model

\begin{table}
\begin{tabular}{l l} \hline \hline
**Model Name** & **HuggingFace Model name** & **Remark** \\ \hline AHMPS & masakhanes/afrirt=t-base & mT8-base adaptation to 17 African languages, English, French and Arabic. \\ AEHSYTS & masakhanes/afr=tiypt-base & BYTs-based adaptation to 17 African languages, English, French and Arabic. \\ AEMSHART & masakhana/afri=mabxt50 & mBAT50 adaptation to 17 African languages, English, French and Arabic. \\ NEWS (MFS) & masakhanes/afru=ts.(src\(\leftarrow\)ct)\_news & MTSF time-tuned on [xst-] detection using parallel NEWS corpus. \\ MEWS (EMATTS) & masakhanes/afru=s.(src\(\leftarrow\)ct)\_twalks & MATHTS times-tuned on [xst-] detection using parallel NEWS corpora. \\ NEWS (BFTS) & masakhanes/barrs0/bfets.(src\(\leftarrow\)ct)\_twalks & BYTs-tuned on [xst-] detection using parallel NEWS corpus. \\ NEWS (ABTAPS) & masakhanes/charr=type.(src\(\leftarrow\)ct)\_twalks & ABTAPS Fine-tuned on [xst-] detection using parallel NEWS corpus. \\ NEWS (mBZANTC9) & masakhanes/marr=top.(src\(\leftarrow\)ct)\_twalks & MZM-100 fine-tuned on [xst-] detection using parallel NEWS corpus. \\ NEWS (MEM-100) & masakhanes/mm100-tuned on [xst-] detection using parallel NEWS corpus. \\ NEWS (MEM-100) & masakhanes/mm100-tuned on [xst-] detection using parallel NEWS corpus. \\ NEWS (MEM-100) & masakhanes/mm100-tuned on [xst-] detection using parallel NEWS corpus. \\ NEWS (MEM-100-tuned) & masakhanes/mm100-tuned on [xst-] detection using parallel NEWS corpus. \\ NEWS (MEM-100-tuned) & masakhanes/mm100-tuned on [xst-] detection using parallel NEWS corpus. \\  & & \\ \end{tabular}
\end{table}
Table 11: Model names on HuggingFace Model Hub. For bilingual models, supply the correct src or tgt language. English/French make use of a 2-letter language code i.e en or rf, while all the African languages make us of 3-letter language codes e.g. yor.

\begin{table}
\begin{tabular}{l l r r r r r r r r r} \hline \hline  & \multicolumn{4}{c}{**Tuned**} & \multicolumn{4}{c}{**fon**} \\
**Evaluation** & \multicolumn{4}{c}{**on**} & & & & & & & & & & \\
**Domain** & **NEWS** & **hau** & **libo** & **liug** & **liug** & **swua** & **wal** & **yer** & **rail** \\ \hline _cn/gr-xx_ & & 2.4 & 2.0 & 0.9 & – & 19.6 & 0.4 & 1.0 & 1.9 \\ FLORES & ✓ & 2.9 & 12.3 & 4.9 & 8.8 & 22.5 & 4.2 & 5.1 & 8.4 \\ REL & ✗ & 2.5 & 1.8 & 0.0 & – & 14.6 & 0.0 & 1.4 & 2.1 \\ REL & ✓ & 6.7 & 9.4 & 1.1 & 2.4 & 17.4 & 2.7 & 8.2 & 8.3 \\ NEWS & ✗ & 0.4 & 2.4 & 1.8 & – & 20.1 & 1.3 & 2.1 & 5.6 \\ NEWS & ✓ & 14.4 & 2.0 & 13.0 & 10.8 & 27.9 & 11.1 & 12.8 & 16.5 \\ \hline _cx-en/rf_ & & & & & & & & & & \\ FLORES & ✗ & 6.6 & 6.0 & 2.6 & – & 26.2 & 1.1 & 2.7 & 10.5 \\ FLOES & ✓ & 5.4 & 11.6 & 6.9 & 10.3 & 25.3 & 4.6 & 7.9 & 18.1 \\ REL & ✗ & 9.7 & 5.9 & 0.5 & – & 22.3 & 0.3 & 1.8 & 7.8 \\ REL & ✓ & 7.7 & 10.7 & 1.8 & 2.6 & 20.5 & 1.7 & 8.8 & 12.9 \\ NEWS & ✗ & 2.2 & 6.4 & 4.8 & – & 25.2 & 0.8 & 3.0 & 13.8 \\ NEWS & ✓ & 17.2 & 18.5 & 19.4 & 12.8 & 29.9 & 9.5 & 16.0 & 36.6 \\ \hline \hline \end{tabular}
\end{table}
Table 12: **BLEU on Wikipedia domain** (FLORES), REL, and NEWS for M2M-100 before (✗) and after (✓) fine-tuning on NEWS.

 

is very poor and far from the Fon source, highlighting how poorly the model generalized from the few thousand training sentences. The M2M-100 model gives a more meaningful and adequate translation. M2M-100 makes a surprising but beautiful move, switching _se plaigment depuis quelques jours de multiples douleurs (sin azan mxbpan dye z, ye d o wave se we tauun do agabaza me)_ to _ont depuis plusieurs jours souffert d'une maladie grave_. The BLEU score here might be low but the meaning is conserved and even more detailed than the French reference. In fact, in this source context, _wave_ means _souffrir, software (suffer, suffering)_: the French reference made use of _se plaigment (complaining)_ which makes less sense than _souffert_ used in the M2M-100 prediction. M2M-100 also learned the style of the sentence: _cof ve katu fun if e azon ne lee gosin (but they do know the origin of their sufferings) e ok (**NOT**)_ - this last part is crucial for the meaning of the entire sentence. Given the structural and morphological differences between Fon and French, we expected it to be more complicated to predict. However, this translation is structurally wrong even though any French native speaker would understand the conveyed message quickly and easily. In the M2M-100 translation, the word _malgre_ is at the wrong place, corrupting syntax and logic of the second clause. A perfect translation (in the idea to be expressed) would be: "Louis Guy Alimanyion et Issa Etchlekoun ont depuis plusieurs jours souffert d'une maladie grave malgre (dont) _ils ne connaissent pas les consequenees (causes/raisons) de-eette-maladie quilis ne connaissent pas."

In the opposite translation direction, fr\(\rightarrow\)fon, M2M-100 (REL+NEWS\(\rightarrow\)NEWS) still preserved some sense of logical reasoning and predicted the last part right _ye ka tuun nu e wu wive yeton (they do know why they are suffering) de 3 a (**NOT**)_. However, the model had some limitations: the names which are part of the translation are not spelled correctly. Some expressions are incomplete: For instance _sin azan + number_ means _since xxx days_ but _yeywe_ is not a number, and do not have any meaning in this context.

## Appendix G Limitations and Risks

Despite the promising results, our work has the following limitations:

1. **Translation quality**: Even the best model scores low BLEU on some of the reported languages (bbj,mos,zul), in particular when translating into them.
2. **Evaluation**: Our evaluation is focused on BLEU. We report ChrF results as well, but without a deeper human evaluation, we cannot make claims about the absolute quality of the translations. Manual inspections of translations like the example discussed in Section F gave us the impression that translations are surprisingly fluent and make good use of language-specific expressions when translating into English or French, but that errors in grammar and logic can be easily overlooked. Automatic reference-based metrics like BLEU and ChrF might not be able to capture the semantic relatedness to the reference sufficiently, as well potentially being tricked by word matches in incoherent phrases.
3. **Language bias**: We have shown that even when not included in pre-training, and without large out-of-domain data, significant gains in translation quality can be achieved. However, language-specific biases, in terms of resourcedness, morphology, standardization, inclusion in pre-trained models and available corpora, or relatedness to other languages, still affect the relative quality of translations, and require more efforts to be overcome.
4. **Domain limitations**: While we showed a rapid adaptation to the news domain and the auxiliary benefit of the religious domain, our study also revealed how automatically estimated translation quality drops when the test domain is narrow. Therefore, future work should aim to expand the study to multiple test domains and develop systematic methods

\begin{table}
\begin{tabular}{l l r r r r r r r r} \hline  & \multicolumn{6}{|c}{**Tuned**} \\
**Evaluation** & **on** & & & & & & & & \\
**Domain** & NEWS & **hu** & **libo** & **liq** & **liq** & **swa** & **wel** & **yer** & **zul** \\ \#/\%: & & & & & & & & & \\ HORES & ✗ & 2.6 & 2.8 & 0.8 & – & 29.9 & 0.6 & 1.5 & 3.3 \\ FLORES & ✓ & 4.0 & 19.9 & 7.6 & 13.7 & 27.1 & 8.2 & 13.4 & 19.2 \\ REL & ✗ & 1.2 & 1.0 & 0.0 & – & 11.0 & 0.0 & 0.4 & 1.6 \\ REL & ✓ & 1.7 & 10.3 & 3.3 & 5.4 & 14.6 & 6.7 & 16.0 & 13.0 \\ NEWS & ✗ & 0.6 & 4.1 & 2.3 & – & 21.4 & 1.2 & 2.4 & 5.6 \\ NEWS & ✓ & 20.2 & 31.6 & 22.6 & 16.4 & 31.4 & 19.9 & 25.5 & 27.6 \\ \hline _xz-mxbp_ & & & & & & & & & \\ \#/\%: & & & & & & & & & \\ HORES & ✗ & 8.0 & 7.2 & 3.7 & – & 26.9 & 0.0 & 3.8 & 11.9 \\ HOLES & ✓ & 16.3 & 12.0 & 7.7 & 11.8 & 25.8 & 7.5 & 9.3 & 19.2 \\ REL & ✗ & 6.4 & 3.7 & 0.5 & – & 15.4 & 0.4 & 0.9 & 8.5 \\ REL & ✓ & 3.8 & 6.0 & 1.7 & 2.5 & 13.9 & 1.7 & 5.7 & 12.5 \\ NEWS & ✗ & 2.6 & 9.1 & 7.2 & – & 27.8 & 1.0 & 3.9 & 15.7 \\ NEWS & ✓ & 17.6 & 22.8 & 24.4 & 15.8 & 23.0 & 12.3 & 17.5 & 39.0 \\ \hline \end{tabular}
\end{table}
Table 13: **spBLEU on Wikipedia domain** (FLORES), REL, and NEWS for M2M-100 before (✗) and after (✓) fine-tuning on NEWS.

 

for distilling knowledge from multiple narrow domains.
5. **Language coverage**: Africa has thousands of other languages that are not covered in our study but deserve the same attention. We hope that our work is encouraging enough to inspire native speakers of those languages not covered here to collect translations, run our code, and report their findings to the NLP research community, so that we can make joint progress in developing language technology for more people.

We believe that our translation models carry similar risks of causing harm by inaccurate and biased translations as the underlying large pre-trained models. M2M-100 is trained on large collections of texts crawled from the web, and the quality for most of the languages studied here is questionable Kreutzer et al. (2021). Our fine-tuning successes show that some obvious biases can be overcome when the quality of the fine-tuning set is controlled (see the examples in Section 6.3), but we cannot guarantee that biases prevailing in the pre-training corpus or more subtle biases will not occur with other inputs. Together with a careful human evaluation, this should be the main concern for future work on the produced models. The methodology of rapid fine-tuning might also be misused to tune the models towards harmful content or purposes that harm the speakers of the languages presented here.