SlowGuess/NAACL_2022_OCR huggingface dataset:
A Few Thousand Translations Go A Long Way! Leveraging Pre-trained Models for African News Translation
David Ifeoluwa Adelani $^{1*}$ , Jesujoba Oluwadara Alabi $^{2*}$ , Angela Fan $^{3*}$ , Julia Kreutzer $^{4*}$ , Xiaoyu Shen $^{5}$ , Machel Reid $^{6*}$ , Dana Ruiter $^{1}$ , Dietrich Klakow $^{1}$ , Peter Nabende $^{7*}$ , Ernie Chang $^{1*}$ , Tajuddeen R. Gwadabe $^{8*}$ , Freshia Sackey $^{9*}$ , Bonaventure F. P. Dossou $^{10*}$ , Chris Chinenye Emezue $^{11*}$ , Colin Leong $^{12*}$ , Michael Beukman $^{13*}$ , Shamsuddeen H. Muhammad $^{14*}$ , Guyo D. Jarso $^{*}$ , Oreen Yousuf $^{15*}$ , Andre N. Rubungo $^{16*}$ , Gilles Hacheme $^{17*}$ , Eric P. Wairagala $^{7*}$ , Muhammad U. Nasir $^{18*}$ , Benjamin A. Ajibade $^{*}$ , Tunde Oluwaseyi Ajayi $^{*}$ , Yvonne Wambui Gitau $^{*}$ , Jade Abbott $^{*}$ , Mohamed Ahmed $^{19*}$ , Millicent Ochieng $^{19*}$ , Anuoluwapo Aremu $^{*}$ , Perez Ogayo $^{20*}$ , Jonathan Mukiibi $^{7*}$ , Fatoumata Ouoba Kabore $^{*}$ , Godson Koffi Kalipe $^{*}$ , Derguene Mbaye $^{21*}$  Allahsera Auguste Tapo $^{22*}$ , Victoire M. Koagne $^{*}$ , Edwin Munkoh-Buabeng $^{*}$ , Valencia Wagner $^{23*}$ , Idris Abdulmumin $^{24*}$ , Ayodele Awokoya $^{25*}$ , Happy Buzaaba $^{*}$ , Blessing Sibanda $^{26*}$ , Andiswa Bukula $^{27*}$ , Sam Manthalu $^{28}$
$^{*}$ Masakhane NLP,  $^{1}$ Saarland University, Germany,  $^{2}$ Inria, France,  $^{3}$ Meta AI,  $^{4}$ Google Research,  $^{5}$ Amazon Alexa AI,  $^{6}$ The University of Tokyo, Japan,  $^{7}$ Makerere University, Kampala, Uganda,  $^{8}$ UCAS, China,  $^{9}$ JKUAT, Kenya,  $^{10}$ Jacobs University, Germany,  $^{11}$ TUM, Germany,  $^{12}$ University of Dayton, USA,  $^{13}$ University of the Witwatersrand, South Africa,  $^{14}$ LIAAD-INESC TEC, Porto, Portugal,  $^{15}$ Uppsala University, Sweden,  $^{16}$ UPC, Spain,  $^{17}$ Ai4Innov  $^{18}$ Ominor AI  $^{19}$ Microsoft Africa Research Institute, Kenya  $^{20}$ CMU, USA,  $^{21}$ Baamtu,  $^{22}$ RIT, USA,  $^{23}$ SPU, South Africa,  $^{24}$ ABU, Nigeria,  $^{25}$ UI Ibadan, Nigeria,  $^{26}$ NUST, Namibia  $^{27}$ SADIaR, South Africa,  $^{28}$ University of Malawi, Malawi
Recent advances in the pre-training of language models leverage large-scale datasets to create multilingual models. However, low-resource languages are mostly left out in these datasets. This is primarily because many widely spoken languages are not well represented on the web and therefore excluded from the large-scale crawls used to create datasets. Furthermore, downstream users of these models are restricted to the selection of languages originally chosen for pre-training. This work investigates how to optimally leverage existing pre-trained models to create low-resource translation systems for 16 African languages. We focus on two questions: 1) How can pre-trained models be used for languages not included in the initial pre-training? and 2) How can the resulting translation models effectively transfer to new domains? To answer these questions, we create a new African news corpus covering 16 languages, of which eight languages are not part of any existing evaluation dataset. We demonstrate that the most effective strategy for transferring both to additional languages and to additional domains is to fine-tune large pre-trained models on small quantities of high-quality translation data.
Enormous efforts have been invested in making language and translation models more multilingual
while leveraging the maximal amount of data for training, most prominently large crawls of monolingual and parallel data from the web (El-Kishky et al., 2020; Schwenk et al., 2021b,a; Xue et al., 2021b). The resulting models are now capable of translating between hundreds of languages, including language pairs that in isolation do not have large collections of parallel data (Tang et al., 2020; Xue et al., 2021a; Fan et al., 2021b). For example, M2M-100 (Goyal et al., 2021) can translate (with low accuracy) between Hausa and Yorubá, two of the most widely spoken languages in Nigeria, even though there is barely any parallel data available for training. For languages that are not included in the set of training languages, the model would have no knowledge on how to generate translations. Does this mean there is no hope for languages that do not have large presence on the web and are therefore not included in these pre-trained models?
We investigate how large-scale pre-trained models can be leveraged for the translation of unseen low-resource languages and domains. We address this question by studying 16 African languages that are largely underrepresented in NLP research (Joshi et al., 2020;  $\forall$  et al., 2020) and further have little to no training data available ( $\S 3$ ). These languages provide an ideal testbed for two challenging knowledge transfer tasks: (1) How can pre-trained models create translations for languages unseen at train-
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3053 - 3070 July 10-15, 2022 ©2022 Association for Computational Linguistics
ing time? and (2) Since training data may only exist in single domain (i.e. religious texts), how can a model be trained in one domain and translate another effectively at test time?
These questions are extremely relevant for our chosen languages because they all have millions of native speakers and a massive need for translation technologies. For example, news concerning the African continent are almost exclusively published in English, French, or Arabic, and thereby inaccessible for speakers of only native African languages. This creates a bottleneck for information transmission, which becomes even more critical in times of crises (Öktem et al., 2020; Anastasopoulos et al., 2020; Öktem et al., 2021). Furthermore, the task of translating news has historically played a central role in translation research, e.g. in shared tasks since 2008 (Callison-Burch et al., 2008) and as a test for determining human parity (Hassan et al., 2018; Läubli et al., 2018; Toral et al., 2018). To spur the development of dedicated news translation models for Africa, we construct a benchmark of news translation for translating between 16 native African languages and English or French (§4).
This allows us to compare three approaches to leveraging large-scale multilingual models for the translation of previously unseen languages: (1) zero-shot transfer, (2) continual pre-training on monolingual data, and (3) multi-domain fine-tuning on parallel data ( $\S 5$ ). We find that fine-tuning pretrained models on a few thousand sentences of high quality bitext is remarkably effective, and can be further augmented with continual pre-training on African languages and fine-tuning on news domain data ( $\S 6$ ). Our contributions are the following: $^{1}$
1. We create a new African news corpus for machine translation (following principles of participatory research  $\forall$  et al. (2020)) covering 16 African languages.
2. We adapt several multilingual pre-trained models (MT5, ByT5, mBART, M2M-100) to these largely unseen languages, and evaluate their quality on news translation.
3. We quantify the effectiveness of small in-domain translation sets by measuring domain transfer effects and comparing finetuning strategies.
We find that having a targeted collection of translations is surprisingly effective, showcasing the power of local knowledge in so-called "zero-resource" scenarios (Bird, 2020). This paints a promising picture for the development of NLP technology for understudied languages: being able to customize these models for new language of interest with as little as  $2\mathrm{k}$  sentences and a few fine-tuning steps, MT developers and users from any language community are less dependent on choices and monetary interest of industry powerhouses from the Global North (Paullada, 2020).
African MT Datasets. One of the major challenges of developing MT models for African languages is lack of data. There are many attempts to automatically crawl and align sentences from the web (Schwenk et al., 2021a,b). Nevertheless, the resulting corpora for many African languages are typically small and of poor quality (Kreutzer et al., 2021). Other cleaner parallel sources are mostly from religious sources, like the Bible covering over 1600 languages (McCarthy et al., 2020) and JW300 (Agić and Vulić, 2019) from JW.org with over 343 languages, including over 100 African languages. Apart from the training dataset, evaluation datasets are needed to test the performance of multilingual MT models. The FLORES-101 (Goyal et al., 2021) evaluation set, sourced from Wikipedia and manually translated, covers the largest number of languages, including 20 African languages. Finally, while other evaluation datasets for translating into or from African languages have been developed (Siminyu et al., 2021; Emezue and Dossou, 2020; Azunre et al., 2021b; Nyoni and Bassett, 2021; Gezmu et al., 2021; Ali et al., 2021), unfortunately there are only a few African languages with evaluation datasets in the news domain (Adelani et al., 2021a; Mabuya et al., 2021; Ezeani et al., 2020) but ours covers 11 African languages ( $\S 4$ ).
Low-resource MT. Interest in low-resource MT has been increasing both within the MT research community (Haddow et al., 2021), as well as in native speaker communities  $(\forall$  et al., 2020; Azunre et al., 2021a; Mager et al., 2021). On the modeling side, many techniques have been developed: unsupervised MT (Lample et al., 2018) leverages monolingual data, single multilingual models capable of translating between many languages (Firat et al., 2016; Johnson et al., 2017; Aharoni et al.,
1All data, models and code are publicly available on https://github.com/masakhane-io/ lafand-mt under academic license.
Target Language	Family	African Region	No. of Speakers	Source Lang.	NEWS	REL
Source	Split Sizes	Source	Total Size
Bambara (bam)	NC / Manding	West	14M	French	Maliweb.net	3302/ 1484/ 1600	Bible	28K
Ghomálá' (bbj)	NC / Grassfields	Central	1M	French	Cameroun Web	2232/ 1133/ 1430	Bible	8K
Éwé (ewe)	NC / Kwa	West	7M	French	Benin Web TV	2026/ 1414/ 1563	JW300	618K
Fon (fon)	NC / Volta-Niger	West	2M	French	ORTB, Nation, Héraut, Matin Libre, LBLibéré, LE Précis, Visages.	2637/ 1227/ 1579	JW300	32K
Hausa (hau)	Afro-Asiatic / Chadic	West	63M	English	WMT2021: Khamenei.v1(Ezeani et al., 2020)	3098/ 1300/ 1500	JW300	236K
Igbo (ibo)	NC / Volta-Niger	West	27M	English	6998/ 1500/ 1500	JW300	415K
Luganda (lug)	NC / Bantu	East	7M	English	Independent Uganda	4075/ 1500/ 1500	Bible	31K
Luo (luo)	Nilo-Saharan	East	4M	English	Lolwe, Standard Media	4262/ 1500/ 1500	Bible	31K
Mossi (mos)	NC / Gur	West	8M	French	Burkina24, Lefaso	2287/ 1478/ 1574	JW300	216K
Naija (pcm)	English-Creole	West	75M	English	Daily Trust Nigeria	4790/ 1484/ 1564	JW300	23K
Swahili (swa)	NC / Bantu	East & Central	98M	English	Global Voices, OPUS	30782/ 1791/ 1835	JW300	872K
Setswana (tsn)	NC / Bantu	South	14M	English	SABC News	2100/ 1340/ 1500	JW300	870K
Akan/Twi (twi)	NC / Kwa	West	9M	English	StarrFM, Citi News	3337/ 1284/ 1500	JW300	601K
Wolof (wol)	NC / Senegambia	West	5M	French	Seneweb, Jotna, Yerim Post, Socialnetlink	3360/ 1506/ 1500	Bible	22K
Yorubá (yor)	NC / Volta-Niger	West	42M	English	(Adelani et al., 2021a)	6644/ 1544/ 1558	JW300	460K
isiZulu (zul)	NC / Bantu	South	27M	English	(Mabuya et al., 2021)	3500/ 1239/ 998	JW300	667K
Table 1: Languages and Data Details for MAFAND-MT Corpus. Language, family (NC: Niger-Congo), number of speakers, news source, news (NEWS), and religious domain (REL) data split. The languages highlighted in gray did not previously have news-domain data before MAFAND-MT.
2019; Fan et al., 2021a), multilingual unsupervised models leverage a related language (with parallel data) to assist translating the low-resource language that might not even have any monolingual data (Ko et al., 2021). Unfortunately, unsupervised MT typically performs poorly on low-resource languages (Marchisio et al., 2020).
Transfer learning from high-resource languages has achieved more promising results: Transfer from multilingual pre-trained language models (PLM), like mBART50 (Tang et al., 2020) and MT5 (Xue et al., 2021b), and large-scale multilingual MT often outperforms bilingual MT (Tran et al., 2021; Yang et al., 2021). For low-resource languages this strategy outperforms the baseline (Transformer) models (Birch et al., 2021; Adelani et al., 2021a; Lee et al., 2022). The performance can be further improved by large scale pre-training (Reid et al., 2021; Emezue and Dossou, 2021).
3 Focus Languages and Their Data
Focus Languages. We focus on 16 African languages with varying quantities of available data (Joshi et al., 2020), including moderately low-resource languages such as Swahili and Hausa, and very low-resource languages such as Ghomálá'² with the Bible being its largest available corpus. Table 1 provides an overview of the focus languages, including the language families, location and number of speakers, and the source and original language for our corpus. The languages are from four language families: Afro-Asiatic (e.g. Hausa), Nilos-Saharan (e.g. Luo), English Creole (e.g. Nigerian-Pidgin/Naija) and Niger-Congo. Most of the languages (13 out of 16) are from the Niger-Congo
family, which is the largest language family in Africa. Six of the languages are predominantly spoken in Francophone countries of Africa, while the remainder are predominantly spoken in Anglophone countries of Africa. In contrast to previous work ( $\forall$  et al., 2020; Gowda et al., 2021), we do not focus exclusively on translation to/from English since this is not the primary language of the Francophone Africa community. All languages are spoken by at least one million speakers.
Language Characteristics. All languages are written in Latin script, using letters of the basic Latin alphabet with a few omissions (e.g. "c", "q", "x", "z") and additions (e.g. "ε", "ɔ", "ŋ", "ɔ", including digraphs like "gb", "kp", "gh", and sometimes more than two-character letters). 13 of the languages are tonal, and about nine make use of diacritics. Many African languages are morphologically rich. For example, all Bantu languages are agglutinative. Fon, Mossi, and Yorubá are highly isolating. All languages follow the Subject-Verb-Object sentence structure like English and French. Table C provides more details.
Existing Parallel Corpora. We curate publicly available parallel data for our focus languages, which consists primarily of text in the religious domain. For most African languages, the largest available parallel corpora is JW300 (Agić and Vulić, 2019), sourced from JW.org, which publishes biblical texts as well as lifestyle and opinion columns. Varying quantities of data are available for 11 of the 16 focus languages. Éwé, Igbo, Swahili, Setswana, Twi, Yorubá, and isizulu have over 400K parallel sentences. Hausa and Mossi have slightly more than 200K parallel sentences, while Fon and Naija have around 30K sentences. For the remaining
2Spoken by an estimated 1.1M people in Cameroon
five languages that are not in the JW300 corpus, $^{3}$  we make use of the Bible. $^{4}$  We aligned the sentences automatically by the verses (around 31k in total). Ghomálá' only has the New Testament with 8k verses. Bambara and Wolof are missing some verses and books, leading to a total size of 28K and 22K. Table 1 summarizes this information about the religious (REL) corpora.
4 MAFAND-MT African News Corpus
4.1 Data Collection Process
We introduce our newly translated news corpus; MAFAND-MT — Masakhane Anglo & Franco Africa News Dataset for Machine Translation. Table 1 gives the news source and data splits for 11 African languages which includes six languages (bam, bbj, ewe, fon, mos, wol) spoken predominantly in Francophone Africa and five languages (lug, luo, pcm, tsn, twi) spoken predominantly in Anglophone Africa. The MAFAND-MT corpus was created in three steps:
1. Crawling and preprocessing of news websites from local newspapers that are publishing in English and French. Raw texts from the web were segmented into sentences. Most languages were crawled from one or two sites, except for Wolof and Fon that were crawled from four and seven news websites respectively due to local French language newspapers having very few articles. We also ensured that the articles came from a variety of topics e.g. politics, sports, culture, technology, society, religion, and education. This was carried out by native speakers of the target language with source language proficiency.
2. Translation of 5k-8k sentences by professional translators. The translation process took one to four months depending on the availability of the translators.
3. Quality control was provided by native speakers, who discussed and, if possible, fixed problematic translations and ran automatic checks to detect misspellings, duplicated sentences, and alignment problems.
Following the recommendations of  $\forall$  et al. (2020), we design the process to be participatory: Everyone involved in the corpus creation is a native speaker of the respective target languages and has societal knowledge about the communities that speak those languages. This is particularly important for curation and quality control to ensure that the resulting material is appropriate and relevant for stakeholders of the final MT models ( $\forall$  et al., 2020; Kreutzer et al., 2021). Furthermore, everyone received appropriate remuneration. To enable cross-disciplinary knowledge transfer between participants in the individual steps, every language was assigned a coordinator. The coordinator conducted the initial curation in the first step, and communicated with translators and quality checkers throughout the following steps.
Other Available Parallel Corpora. We found five African languages with available parallel texts in the news domain: Hausa<sup>5</sup>, Igbo (Ezeani et al., 2020), Swahili<sup>6</sup>, Yorubá (Adelani et al., 2021a), and isizulu (Mabuya et al., 2021). Table 1 provides news source, the TRAIN, DEV and TEST splits. Appendix B provides details on the pre-processing of the available news corpora.
4.2 Monolingual News Corpus
To adapt available multilingual pre-trained models via continued pre-training to African languages, we curated texts from the 17 highest-resourced African languages and three non-native African languages that are widely spoken on the continent (Arabic, English, and French). The selection of African languages is based on their coverage in mC4 (Xue et al., 2021b), AfriBERTa corpora (Ogueji et al., 2021), and other publicly available news websites like VOA and BBC. We limited the size of the corpus extracted from mC4 to the first 30 million sentences (roughly 1GB of data) for Afrikaans, Amharic, Arabic, English, French, and Swahili. In total, we collected about 12.3 GB of data. Appendix C provides more details about the pre-training corpus.
5 Models and Methods
We experiment with pre-trained multilingual models and our own bilingual MT baselines. We focus
<sup>5</sup>https://www.statmt.org/wmt21/translation-task.html
<sup>6</sup>https://sw.globalvoices.org/
<sup>3</sup>Some languages like Luo and Luganda are covered by JW300 but are no longer available at the time of paper writing. <sup>4</sup>Crawled/downloaded from https://ebible.org/, except for Bambara that we obtained from https://live.bible.is/ and Ghomála' from www.beblia.com
Pre-trained
Model (PM)	PM
Size	# African
Lang.	Focus languages covered
MT5/ByT5	580M	13	hau, ibo, swa, yor, zul
Afri[*T5]	580M	17	hau, ibo, pcm, swa, yor, zul
mBART50	610M	2	swa
AfriMBART	610M	17	hau, ibo, pcm, swa, yor, zul
M2M-100	418M	17	hau, ibo, lug, swa, tsn, wol,
yor, zul
Table 2: Language coverage and size for pre-trained models. Afri\*T5] refers to AfriMT5/ByT5.
on pre-trained models that are approximately 500M parameters, both for computational feasibility and comparability across various different models.
Transformer Baseline. We train Transformer (Vaswani et al., 2017) sequence-to-sequence models from scratch for each language pair using JoeyNMT (Kreutzer et al., 2019). We tokenize the bitext using a joint SentencePiece unigram model (Kudo, 2018), with a character coverage of 1.0 and a maximum sentence length of 4096 tokens and create a vocabulary of 10K subwords. Models are trained on the concatenation of REL and NEWS corpora for each language.
Pre-trained Models. We consider three language models, MT5 (Xue et al., 2021b), ByT5 (a token-free T5) (Xue et al., 2021a), mBART50 (Tang et al., 2020), and the multilingual translation model M2M-100 (Fan et al., 2021b) for our experiments. We use MT5-base and ByT5-base, and M2M-100 with 418M parameters. Table 2 gives the pretrained model size, number of African languages covered, and the focus languages supported.
5.2 Transfer Learning Across Languages
We describe two methods for adding new languages to existing models: continual pre-training and many-to-many multilingual translation.
Continual Pre-training. The effectiveness of PLMs is limited on extremely low-resource languages because they rarely, if ever, occur in the pretraining corpus (Wang et al., 2020; Liu et al., 2021). As shown in Table 2, even for MT5 and M2M-100, which cover 100 languages, less than half of the African languages under study are included. To adapt the existing PLMs to our languages corpora and domains, we apply continual pre-training (Gururangan et al., 2020; Liu et al., 2021) using our collected monolingual corpus. Specifically, before fine-tuning on the parallel MT data, models are pretrained with their original training objective and vo
cabulary<sup>8</sup> on the monolingual corpus. Pre-training parameters can be found in the appendix. We refer to the models adapted to African languages as AfriMT5, AfriByT5, and AfriMBART.
Many-to-Many Translation. We fine-tuned M2M-100 for African multilingual translation to create English- and French-centric models. For the English-centric model, the M2M-100 model was fine-tuned on the news data for en-{hau, ibo, lug, luo, pcm, swa, tsn, twi, yor, zul} while the French-centric model is trained on fr-{bam, bbj, ewe, fon, mos, wol}. Languages not included in the pre-trained M2M-100 model were assigned the language code of a language included in M2M-100 but excluded from our study.
5.3 Transfer Learning Across Domains
As there is very limited MT data on the news domain, we compare different methods that combine the large data from the religious domain (REL) and the small data from the NEWS domain (NEWS) to fine-tune M2M-100:
1. REL+NEWS: Fine-tuning on the aggregation of REL and NEWS.
2. REL  $\rightarrow$  NEWS: Training on REL, followed by fine-tuning on NEWS.
3. REL+NEWS  $\rightarrow$  NEWS: REL+NEWS, followed by additional fine-tuning on NEWS.
Each fine-tuning stage lasts for three epochs. We evaluate translation quality with BLEU (Papineni et al., 2002) using SacreBLEU (Post, 2018)<sup>9</sup> and ChrF (Popovic, 2015).
We successfully adapt several multilingual pretrained models to previously unseen African languages and quantify the effectiveness of small indomain translation datasets. We discuss the effects of domain shift and analyze mitigation strategies.
6.1 Adaptation to the Focus Languages
We demonstrate that fine-tuning with a few thousand high-quality bitext is effective for adding new languages to pre-trained models. Further, continuing to pre-train to specialize models to African languages further improves performance.
Changing the vocabulary (Gururangan et al., 2020) to fit the languages, or adding MT-focused training objectives for word alignment (Liu et al., 2021) can potentially improve the performance further, which we leave for future work.
$9^{th}$  "intl" tokenizer, all data comes untokenized.
<sup>7</sup>https://github.com/google/sentencepiece