Better Quality Pre-training Data and T5 Models for African Languages
Authors: Akintunde Oladipo
Mofetoluwa Adeyemi
Orevaoghene Ahia
Abraham Toluwase Owodunni
Odunayo Ogundepo
David Ifeoluwa Adelani
Jimmy Lin
Venue: EMNLP 2023 Main (2023)
Keywords: multilingual
low-resource languages
african languages
TL;DR: We release the largest quality pre-training data across 16 African languages and pretrain better T5 models

Abstract: In this study, we highlight the importance of enhancing the quality of pretraining data in multilingual language models. 
Existing web crawls have demonstrated quality issues, particularly in the context of low-resource languages.
Consequently, we introduce a new multilingual pretraining corpus for $16$ African languages, designed by carefully auditing existing pretraining corpora to understand and rectify prevalent quality issues. To compile this dataset, we undertake a rigorous examination of current data sources for thirteen languages within one of the most extensive multilingual web crawls, mC4, and extract cleaner data through meticulous auditing and improved web crawling strategies. Subsequently, we pretrain a new T5-based model on this dataset and evaluate its performance on multiple downstream tasks. 
Our model demonstrates better downstream effectiveness over existing pretrained models across four NLP tasks, underscoring the critical role data quality plays in pretraining language models in low-resource scenarios. Specifically, on cross-lingual QA evaluation, our new model is more than twice as effective as multilingual T5. All code, data and models are publicly available at https://github.com/castorini/AfriTeVa-keji.

Better Quality Pretraining Data and T5 Models for African Languages
Akintunde Oladipo', Mofetoluwa Adeyemi', Orevaoghene Ahia², Odunayo Ogundepo',
Abraham Toluwalase Owodunni", David Ifeoluwa Adelani ,Jimmy Lin'
University of Waterloo University of Washington
Masakhane
University College London
aooladip@uwaterloo.co
Abstract
age gap. However, previous work (Kreutzer al., 2022) has shown that mC4 and other existing large- scale pretraining corpora have numerous quality issues, particularly for the low-resource African languages they contain.
In this study, we highlight the importance of enhancing the quality of pretraining data in multilingual language models. Existing web crawls have demonstrated quality issues, partic- ularly in the context of low-resource languages. Consequently, introduce pretraining corpus for 16 African languages, de- signed by carefully auditing existing pretrain- ing corpora to understand and rectify preva- lent quality issues. To compile this dataset, we undertake a rigorous examination of cur- rent data sources for thirteen languages within one of the most extensive multilingual web crawls, mC4, and extract cleaner data through meticulous auditing and improved web crawl- ing strategies. Subsequently, we pretrain new T5-based model this dataset and eval- uate its performance on multiple downstream tasks. Our model demonstrates better down- stream effectiveness over existing pretrained models across four NLP tasks, underscoring the critical role data quality plays in pretrain- ing language models in low-resource scenar- ios. Specifically, on cross-lingual QA evalu- ation,
Against this backdrop, indigenous efforts build language resources for African languages have converged to two approaches: (1) Small high- quality data (e.g.. 1GB) pretraining where most data are from the clean or verified sources like news domain (Ogueji et al., 2021). (2) Large aggrega- tion of all available data (e.g.. 15 - 42 GB) from noisy or unverified sources like CC-100 (Conneau et al., 2020), and mC4, combined with high-quality sources like news corpora (Adelani et al., 2022: Alabi et al., 2022: Adebara al., 2022).
multilingual
This tradeoff between quantity and quality is forced by the unavailability of large, quality pre- training data for African languages. Motivated by this need, introduce new multilingual pretrain- ing corpus in 20 African languages. We draw from Kreutzer al. (2022)'s 'ss audit of existing pretrain- ing corpora to understand prevailing quality issues. For mC4, they cite a high ratio both of sentences in incorrect languages (15.98% average) and nonlin- guistic content (1 ..40% average). We trace these issues to the quality of data sources used in mC4 for the languages in our study and design heuristics to effectively extract clean monolingual text.
model is more than twice as effective as multilingual T5. All code, data and model are publicly available https: //gi thib..u..co
1 Introduction
As language models have scaled up in size and mul- tilingual capability in recent years, commensurate effort has followed to curate pretraining data (Raf- fel et al., 2020) to support this growth and improve the alignment of language models.
More notably, we demonstrate how large-scale web crawls and document-level datasets, such as mC4, can be enhanced through meticulous audit- ing of their document sources i.e., base URLs (e.g.. voahausa. com). Interestingly, for numerous credible sources, mC4 encompasses fewer docu- ments than what is actually available. We conduct our own web crawl of these sources, collecting more documents than what is present in mC4 for
Earlier multilingual models such mBERT (De- vlin et al., 2019) and XLM-R (Conncau et al., 2019) were trained on monolingual data from Wikipedia and/or other large-scale web crawls which included only few African languages. The introduction of mC4 (Xue et al., 2021), a document-level dataset spanning 101 languages helped alleviate this cover-
While OSCAR (Suarez et al., 2019: Abadji et al., 2022) includes African languages, three of them have roughly 1000 documents. All 6 languages amount to less than 200MB

---

We present WURA,² a multilingual dataset com- prising 16 African languages and 4 high-resource languages popularly spoken on the African conti- nent Arabic, English, French, and Portuguese. the respective languages. We consolidate the result of our efforts (cleaning and crawling) with data from other sources, notably Wikipedia, and include four high-resource languages - Arabic, English, French & Portuguese.
of in-language and noisy or non-linguistic text, and may best be filtered at sentence-level. Noting all of these issues and findings, we filter at three levels:
Corpus-level. We first rank unique websites descending order of the number of documents they contribute to the mC4 corpus for each language. Then, we select the top 20% of websites for each language and collect documents sourced from web- sites in this list. This preserves high potential sources for further document level filtering.
To evaluate the quality of our new corpus, we pretrain a new T5-based LM on the collected dataset and benchmark its performance multiple downstream tasks. Our model demonstrates im- proved effectiveness over existing pretrained LMs further highlighting the importance of carefully curated datasets for pretraining language models in low-resource scenarios. Our model was sig- nificantly better than the baseline mT5 models across four different downstream tasks. Specif- ically, cross-lingual QA evaluation, new model achieves more than double the performance of multilingual T5.
Document-level. At document level, we filter out documents that do not contain at least 5 stop- words in them (Caswell et al., 2020) using stop- words from Stopword Lists for African Languages dataset.
Passage-level. After document-level filtering, we chunk the dataset into passages of roughly 512 to- kens. Finally, we filter out passages that contain fewer than 4 unique words or contain repetition for more than 20% of its word length; have more than 10% of its characters are numeric or contain markers of possibly offensive content such as in- cluded in the Toxicity-200 dataset (NLLB Team et al., 2022) for the relevant language.
2
WURA Dataset
The curation of WURA was carried out in a three- part process: - (i) Auditing and cleaning mC4 (ii) Crawling indigenous websites and (iii) Combina- tion with existing language resources.
While Kreutzer et al. (2022)'s audit of mC4 did not yield significant amount of offensive content (0.06% of sentences they audited) and our web crawls mainly focused on verified news publica- tions, these filters ensure that non-linguistic and offensive contents are removed at the passage level.
2.1 Auditing and Cleaning mC4
2.1.1 Language Contamination
Kreutzer et al. (2022) reports mC4's high ratio of non-linguistic content and sentences in incorrect languages, with African languages being of particu- lar concern. The authors report significant loss (up to 50%) in recall of correct in-language sentences as they increased precision of their automatic lan- guage classification.
2.1.2 mC4 is a Great Source!
Xue et al. (2021)'s inclusion of the URL each document is sourced from makes the mC4 corpus even more useful as a data source. Commonly, multiple articles are collected from the same base website, e.g., news publications. For many news publications that provide sitemap, we find that there are fewer articles in mC4 than is actually available the websites. Further, mC4 only covers up to August, 2020 so updating the crawls up the current day yields more data.
Our manual audit of mC4 corroborates the doc- umented issues. We highlight three important findings: (1) The distribution of mC4 document sources has a long tail. Many individual news publications yield thousands of documents in the mC4. (2) Documents from news publications are more likely to be of higher quality i.e., both in- language and grammatical compared to documents from other web sources. (3) Some documents are from websites which translate content using online translation tools. Such documents are often mix
We initiate focused crawls for such websites and this leads to significant increase (> 100% for Hausa and Somali) in the amount of articles avail- able per language. For all languages we consider except Chichewa, Sesotho, Xhosa and Zulu, we collect 1.39M articles (see Table from credible sources found in mC4.
Wura means Gold in Yoruha - with more refining, the quality of our data and model improves.
hhttps://ww.ker comaa stopword-lists-foe cirss--oo--arrircn -anguag-ss

---

Model
Size
anh
eng
fra
hau
ibo
lin
Iug
OmT
run
sna
5on
SND
tir
xhe
yor
AYG
AVG
Va-hhe-
229M
87.0
80.3
71.9
85.8
79.9
82.8
60..2
82.9
95.2
80.0
84.4
58.0
80.7
55.2
09.4
86.4
77.5
78.4
580M
78.2
89.8
75.0
79.2
90.4
76.1
71.8
86.2
FlanTS-hace
580M
54.5
92.4
88.9
84.5
86.6
90.6
84,1
85.8
97.8
87.3
90.6
76.0
79.0
41.5
90.8
88.9
82.5
83.2
AliMMTS-use
580M
90.2
90.3
87.4
87.9
88.0
88.6
84.8
83.9
96.6
91.0
91.5
77.8
84.4
80.8
91.6
88.8
87.7
87.8
AttTeVa V2
428M
92.8
90.6
88.0
89.4
86.1
86.0
91.1
90.8
96.8
92.3
93.3
75.7
87.0
86.4
93.6
92.3
89.5
88.9
Table I: MasakhaNews classification results: Evaluation is done using the weighted FI score and the scores presented are averuged across seeds. VfriTeVa V2 surpasses m mTS-base by up to 10 points. The average scores excluding languages not in the mC4 corpus are also provided in AVGSL.
2.2 Combination with Existing Language Resources and Non-African Languages
cross-lingual QA settings using in- in-lang queries and the provided gold passages English.
Following previous works (Alabi et al., 2022; Ade- bara et al., 2022), we include certain non-African languages in our pretraining data. Specifically, we include over 240, 000 articles newly crawled from 10 African news websites reporting in En- glish, French and Portuguese. We also include sample 1.5M Wikipedia articles for English and French, as well as Wikipedia articles written in Egyptian Arabic. For the African languages, we include all Wikipedia articles. Finally, we dedupli- cate using the document URLs. In doing this, we prioritize news articles in our focused crawls over their existing counterparts in mC4
3.2.2 Machine Translation
We evaluated using MAFAND-MT (Adelani al., 2022) - a machine translation benchmark the news domain. MAFAND-MT contains few thou- sand parallel training sentences (2, 500-30, 000 sen- tences) for 16 African languages, ideal for evaluat- ing the effective adaptation of pretrained LMs to new languages and domains.
3.2.3 Summarization
For summarization, we use XL-Sum (Hasan et al., 2021), abstractive summarization dataset which covers 44 languages, including African lan- guages. The authors establish strong baselines on both low and high-resource languages in the dataset through multilingual finetuning of mT5.
Final Dataset Statistics Table 6 presents a sta- tistical summary of our dataset. The combined dataset from crawling, combining with existing sources and deduplication amounts to 33GG of data across all languages and ~99BB for African languages.
3.2.4 Text Classification
We use the news topic classification dataset recently introduced by Adelani et al. (2023) for 16 African languages, MasakhaNews. The authors establish multiple baselines on the dataset using both clas- sical machine learning models and finetuning prompting language models.
3 Experimental Setup
3.1 Model
Using t5x and seqio (Roberts al., 2022), pretrain TS (Shazeer, 2020; Raffel al., 2020) model with subword-tokenizer of vocabulary size 150, 000. We pretrain for 521, 288 steps on the span-corruption objective using the Adafactor op- timizer. Each training batch consists of 512 ex- amples, each with an input of 512 tokens and an output of 114 tokens. Our new model is known as AfriTe Va V2, 428M parameter model.
3.3 Baseline Models
We compare
model, AfriTe V2, with the base variants of existing multilingual T5 models: mT5 (Xue al., 2021), ByT5 (Xue et al., 2022) and FlanT55 (Chung et al., 2022), as well as Afri- centric models: AfriTeVa (Ogundepo et al., 2022), AfriMTS & AfriByT5 (Adelani et al., 2022).
mT5 was pretrained on the mC4 corpus which is the starter point for this work while ByT5 is the byte-level adaptation of the m model. FlanT5 is '5 cnttruction-inetuneee for improved performance. AfriTeVa. AfriMT5 and AfriByT5 models provide closer comparison given the nature and focus of our research. While AfriTeVa a T5 model pretrained on a small corpus (~BG)) AfriMT5 &
3.2
Downstream Tasks
3.2.1
Cross-lingual Question Answering
We evaluated models on the set of AfriQA Ogundepo et al. (2023), a cross-lingual question answering dataset with questions 10 African languages and gold passages English or French. We evaluated zero-shot generative

---

Model
hau
ibo
pem
swa
yor
zul
AVG
hau
ibo
yor
zul
AVG
m'sS-busee
2.8
18.0
34.1
25.1
4.8
11.7
16.1
5.8
18.9
42.2
29.5
12.3
22.4
21.9
AfriMT3-base
5.1
19.6
35.0
26.7
6.2
13.2
17.5
10.4
19.5
44.6
30.6
13.8
24.0
23.8
ByT5-base
8.3
21.8
30.1
24.4
14.0
17.7
12.9
21.0
39.4
27.1
11.5
22.8
22.5
AfriByT5-hase
9.3
22.7
30.0
24.7
7.6
15.3
18.3
13.5
20.7
39.5
27.0
11.9
24.0
22.8
AfriTeVa V2
13.4
20.7
31.1
28.0
12.1
15.6
20.3
16.2
16.7
40.5
31.0
17.6
28.4
25.1
Table 2: MAFAND-MT results:
Evaluation is done using the BLEU score and we obtain significantly better performance on average across all languages in both the en-xx and xx-en directions, except for ibo and pcm.
AfriByTS are adapted from mT5 and ByT5 models using continual pretraining. Apart from AfriTe Va, AfriTe Va V2 has ~26% less parameters than the other bascline models.
4.1.3 Summarization
We perform multilingual training for 35, 000 steps and sample each batch from a single language. Ta- ble 3 shows we match the performance of mT5 on orm & pcm and gain improvements over baseline Rouge scores for the other languages consider, with yor benefiting the most.
4 Result and Discussion
4.1 Downstream Performance
4.1.4
Text Classification
this section, we compare Va V2 to base- line models on selected tasks. For each down- stream task, we evaluate under the same conditions. We performed per-language finetuning for machine translation & text classification, multilingual finc- tuning over 35K steps for summarization.
Our results for the news classification task are pre- sented in Table 1. We finetune AfriTe V2 on MasakhaNews for each language, framing it as a text-to-text task by predicting the class of each ticle in the decoding sequence and report results of random seeds. On average, AfriTe Va V2 yields better Fl scores across all languages and has the best Fl score on 10 out of 16 languages.
4.1.1 Cross-lingual Question Answering:
AlriTeVa V2 achieves very impressive results in the cross-lingual question-answering task, espe- cially for languages our pretraining data. We finetune on the train set of Squad 2.0 (Rajpurkar et al., 2016) dataset and evaluate the models per- formance on the test set AfriQA. We compare per- formance generative gold passage answer pre- diction, with in-language queries and English pas- sages. Table 4 shows that AfriTe Va V2 achieves much better Fl scores and Exact Match accuracies (~2x) across out languages compared to using mT5-Base Base as the back- bone model.
4.2 Discussion
4.2.1 Results for Nigerian Pidgin
AfriTeVa V2 does not outperform baselines for text classification, machine translation and sum- marization on Nigerian Pidgin (pcm). We note that AfriTeVa V2 was not pretrained on Nigerian Pid- creole,
gin. As Nigerian Pidgin an English- models pretrained on large amounts of English text are expected to be performant for the language. However, AfriTe Va V2 was pretrained on far less English text than the baselines compare to, save for AfriTe Va. Still, obtains results for Nigerian Pidgin that are competitive with the best basclines across the evaluation tasks.
4.1.2 Machine Translation
We observe higher BLEU scores when translating from African languages into English than in the re- verse direction. According to Table 2, we achieve better score average, topping mT5 and AfriMT5 base models by 1-13 points. While both ByT5- style models show greater effectiveness over the mT5 models, AfriTeVa V2 consistently improves over both results for all languages except ibo and pcm, an English- based creole language.
4.2.2 Impact of Data Quality on LMs
Previous works have shown the correlation be- tween the quality of the data used in pretraining model and the performance of the trained model (Rae al., 2021; Kreutzer et al., 2022; Hernan- dez et al., 2022). AfriTe Va V2's improvement over baselines in downstream tasks suggests that this is true. We note that AfriTe Va V2 outperforms the

---

Model
orm
pem
swa
yor
AVG
19.7777// 41.8188.288
18.7/6.//1.2 31.6/10.22445 33.4/127//266
31.7/11 2/2511 37.7/17.9/3099 316621662222 18.0/1..1/29.9 1/16.
92.7/12.9/26..
AfrTEVa V2
37.7/14.6/29.1
33.112266.11
38.1/17.833.99
38.9/16.7/29.99
34.4/14.222.22
Table XL-SUM results: Performance based on Rouge-1, Rouge-2 and Rouge-L. Va V2 is generally more effective than mT5.
Metric
Model
lau
yor
m15
2.9
25.8
41.7
25.5
5.3
11.9
24.7
noisier common crawl data. Obtaining such high- quality data is tedious since it involved curating several verified sources manually. Thus, previ- ous works have prioritized filtering of the com- 13.9
17.5
FI
AfrillVaaBare
AfriMTS-Bass
6.4
39.7
40.7
30,3
5.3
21.8
31.9
25.2
m13
1.1
22.3
34.7
20.2
3.5
7.8
20.9
EM
AfiTTeVa-Reee
2.7
4.
AfrMTSS Base
4.2
33.0
33.0
23.1
2.9
15.7
25.5
19.6
mon crawl data to produce better quality dataset for pretraining (Conneau et al., 2020; Ortiz Suárez et al., 2019; Xue et al., 2021; Bapna al., 2022). However, quality issues still persist in those filtered corpora. An alternative to this is basically aggregat- ing high quality data for African languages mostly from verified sources (Ogueji et al., 2021; Leong et al., 2022; Palen-Michel et al., 2022). However, this often results in smaller sized corpus.
AfrieVaa V2
36.7
47.7
29,5
27.4
Table 4: Cross-lingual Question Answering Results: Fl and Exact Match (EM) Accuracy scores on the test set AfriQA (Ogundepo et al., 2023). For both metrics, Va V2 outperforms mT5 except for twi.
larger AfriMT5 & AfriByT5 (Alabi et al., 2022) which were trained on unfiltered mC4 corpus.
However, our pretraining dataset, WURA, con- tains 15.x more data than mC4 contains across 16 African languages. Thus, more experiments are needed separate the effects of scale from that of data quality.
The current models with impressive performance on African languages simply aggregate both low- quality data and high-quality data for pretrain- ing (Alabi et al., 2022; Adebara et al., 2022). The quality of these models implies that there must be significant portions of the data that are of good qual- ity. To this end, we systematically and rigorously filtered these low-quality data from mC4 corpus for African languages, similar to the OSCAR dataset approach. 5 Too the best of our knowledge, no pre- vious work has done this. OSCAR dataset only has few documents for African languages e.g., 37. 37.2MB for Afrikaans dataset while our filtered corpus has more than 4.5 GB.
5 AfriTeVa Va V2 Large Model
We also pre-train large variant of AftiTeVa V2 using the same configuration of the Ta-large model except for the vocabulary size which we set to be 150, 000, similar to the configuration of AfriTe Va V2 (base) as detailed in subsection 3.1. We present the effectiveness of scaling to a large model size on summarization and news topic clas- sification tasks in Appendix C.
7 Conclusion
6 Related Work
In this work, we look to address the lack of large, quality pretraining dataset for African languages. While previous works have highlighted quality is- sues in existing pretraining dataset such as mC4, we demonstrate how these datasets can be enhanced by auditing their document sources and incorpo- rating rigorous data filtering methods. To high- light the effectiveness of our approach and the relevance of this new dataset, we train a new T5 model, AfriTe V2, ments show significant improvements across exist- ing NLP benchmarks for African languages under- scoring the impact of qualitative pretraining data training language models.
Absence of large monolingual corpus has always been the major challenge of leveraging the bene- fits of self-supervised pretraining for building rep- resentation and language models for African lan- guages. The most available corpus are mostly from religious corpus like Bible (Resnik et al., 1999) or JW300 (Agié and Vulić, 2019), Wikipedia and Common Crawl archive. The latter often has sig- nificant quality issues (Kreutzer et al., 2022).
dataset. Our experi-
Earlier works building word representation models for African languages showed the impor- tance of developing FastText embeddings with small high-quality data (Alabi et al., 2020) over pretrained FastText embeddings developed from
"Due to space constraint, we include results in appendix.
hntps:///osar-pproeet.orr

---

Limitations
Perez Ogayo, Jonathan Mukiibi, Fatoumata Quoba Kabore, Godson Kalipe, Derguene Mbaye, Allah- sera Auguste Tapo, Victoire Memdjokam Koagne, Edwin Munkoh-Buabeng, Valencia Wagner, Idris Abdulmumin, Ayodele Awokoya, Happy Buzaaba, Blessing K. Sihanda, Andiswa Bukula, and Sam Manthalu. 2022. A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation. In North American Chap- ter of the Association for Computational Linguistics.
The representativeness of our dataset poses a po- tential limitation. Despite our efforts to collect data from multiple African news websites, it is possible that our dataset does not fully capture the breadth and diversity of African news articles. The reliance on specific websites and the utilization of the mC4 dataset, along with existing corpora, may introduce inherent bias that our work does not address. Furthermore, our implementation of several-level filtering techniques, including the removal of non- linguistic content in the target language, does not guarantee the complete removal of all text in differ- ent languages or other toxic contents that may be present in the existing corpus.
David Ifeoluwa Adelani, Marek Masiak, Israel Abebe Azime, Jesujoba Oluwadara Alabi, Atnafu Lam- bebo Tonja, Christine Mwase, Odunayo Ogun- depo, Bonaventure F. P. Dossou, Akintunde Oladipo. Doreen Nixdorf, Chris C. Emezue, Sana Sabah Al-Azzawi, Blessing K. Sibanda, Davis David, Lolwethu Ndolela, Jonathan Mukiibi. Tunde Oluwaseyi Ajayi. Tatiana Moteu Ngoli, Brian Odhiambo, Abraham Toluwase Owodunni, Nnae- meka C. Obiefuna. Shamsuddeen Hassan Muham- mad, Saheed Salahudeen Abdullahi, Mesay Gemeda Yigezu, Tajuddeen Rabiu Gwadabe, Idris Abdulmu- min, Mahlet Taye Bame, Oluwabusayo Olufunke Awoyomi, Iyanuoluwa Shode, Tolulope Anu Ade- lani, Habiba Abdulganiy Kailani, Abdul-Hakeem Omotayo, Adetola Adeeko, Afolabi Abeeb, An- uoluwapo Aremu. Olanrewaju Samuel, Clemen- cia Siro, Wangari Kimotho, Onyekachi Raphael Ogbu, Chinedu E. Mbonu, Chiamaka Ijeoma Chuk- wuneke, Samuel Fanijo, Jessica Ojo. Oyinkan- sola F. Awosan, Tadesse Kehede Guge, Sakayo Toad- oum Sari, Pamela Nyatsine, Freedmore Sidume. Oreen Yousuf, Mardiyyah Oduwole, Ussen Ki- manuka, Kanda Patrick Tshinu, Thina Diko, Siyanda Nxakama, Abdulmejid Tuni Johar, Sino- dos Gebre, Muhidin A. Mohamed, Shafie Abdi Mohamed, Fuad Mire Hassan, Moges Ahmed Mehamed, Evrard Ngabire, and Pontus Stenetorp. 2023. MasakhaNEWS: News Topic Classification for African languages. ArXiv, abs/2304.09972.
Lastly, we acknowledge the need for future work to include more African languages. Our dataset only covers 16 languages, limiting the generaliz- ability of our findings across the wide range of languages spoken in Africa.
Acknowledgements
This research was supported in part by the Nat- ural Sciences and Engineering Research Council (NSERC) of Canada and an AI for Social Good grant from the Waterloo Al Institute. Computa- tional resources were provided by Compute On- tario and Compute Canada. We also thank the Google TRC program for providing us free cloud TPU access.
References
Željko Agié and Ivan Vulić. 2019. JW300: A Wide- Coverage Parallel Corpus for Low-Resource Lan- guages. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics. pages 220--3210, Florence, Italy. Association for Computational Linguistics.
Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022. Towards a Cleaner Document- Oriented Multilingual Crawled Corpus. ArXiv, abs/2201.0664/
Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Alcides Alcoba Inciarte. 2022. SERENGETI: Massively Multilingual Language Models for Africa. ArXiv, abs/2212.
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Yulia Tsvetkov. 2023. Do All Languages Cost the Same? Tokenization in the Era of Commercial Lan- guage Models. ArXiv, abs/2305. 137077
David Ifcoluwa Adelani, Jesujoba Oluwadara AI- abi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang. Tajuddeen Rabiu Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong, Michael Beukman, Shamsud- deen Hassan Muhammad, Guyo Dub Jarso, Oreen Yousuf, Andre Niyongabo Rubungo, Gilles Hacheme, Fric Peter Wairagala, Muhammad Umair Nasir, Ben- jamin Ayoade Ajibade, Tunde Oluwaseyi Ajayi, Yvonne Wambui Gitau, Jade Z. Abbott, Mohamed Ahmed, Millicent A. Ochieng, Anuoluwapo Aremu,
Jesujoba Alabi, Kwabena Amponsah-Kaake
David Adelani, and Cristina España- Bonet. 2020. Massive vs. Curated Embeddings for Low-Resourced Lan- guages: The Case of Yoruba and Twi. In Proceedings of the Tweifth Language Resources and Evaluation Conference, pages 2754-2762, Marseille, France. Tu- ropean Language Resources Association.
Jesujoba Oluwadara Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022. Adapt-

---

ing Pre-trained Language Models to African Lan- guages via Multilingual Adaptive Fine- Fine-Tuning. In International Conference on Computational Linguis- rics.
Mukiibi, Blessing K. Sibanda, Bouaventure F. Dos- sou, Andiswa Bukula, Rooweither Mabuya, Allah- sera Auguste Tapo, Edwin Munkoh- Buabeng. Vic- toire Memdjokam Koagne, Fatoumata Ouoba Kabore, Amelia Taylor, Godson Kalipe, Tebogo Macuowa, Vukosi Marivate, Tajuddeen Rabiu Gwadabe, Mbon- ing Tchiaze Elvis, Tkechukwu E. Onyenwe, Gra- tien Gualbert Atindogbé, Tolulope Anu Adelani, Idris Akinade. Olanrewaju Samuel, Marie-Rosette Nahimana. Th' Th'eogene Musabeyezu. Emile Niy- omutabazi. Ester Chimhenga, Kudzai Gotosa, Patrick Mizha, Apelete Agbolo, Seydou T. Traoré, Chinedu Uchechukwu, Aliyu Yusuf, Muhammad Abubakar Abdullahi, and Dietrich Klakow. 2023. Masakha- POS: Part-of-Speech Tagging for Typologically Di- verse African Languages. In Annual Meeting of the Association for Computational Linguistics.
Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Fi- rat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi N. Baljekar, Xavier Gareía. Wolfgang Macherey. Theresa Breiner. Vera Axelrod, Jason Riesa, Yuan Cao, Mia Xu Chen, Klaus Macherey, Maxim Krikun, Pidong Wang. Alexander Gutkin, Apurva Shah, Yanping Huang. Z. Chen, Yonghui Wu, and Macduff' Hughes. 2022. Building Machine Trans- lation Systems for the Next Thousand Languages. ArXiv, abs/2205.03983.
Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna, 2020. Language ID in the Wild: Unexpected Challenges on the Path a Thousand- Language Web Text Corpus. ArXiv, 2as/2200.14571
Guillaume Lample and Alexis Conneau. 2019. Cross- lingual Language Model Pretraining. In Neural In- formation Processing Systems.
Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang. Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai. Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scal- ing citructiionFinetunee Language Models. ArXiv, abs/2210.11410
Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Samin, Yuan-Fang Li, Yong-Bin Kang. M. So- hel Rahman, and Rifat Shahriyar. 2021. XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages. In Findings.
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfiel-DDddss Tom Henighan, Tris- tan Hume, et al. 2022. Scaling Laws and Inter- pretability of Learning from Repeated Data. ArXiv, abs/2205.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2019. Unsupervised Cross-lingual Representation Learning at Scale. In Annual Meeting of the Association f Computational Linguistics.
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orhikkh, Allah- sera Tapo, Nishant Subramani, Artem Sokolov, Clay- tone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, An- nette Rios, Isabel Papadimitriou. Salomey Osei, Pe- dro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, An- dre Niyongabo Rubungo, Toan Q. Nguyen, Math- ias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyak- eni, Jamshidbek Mirzakhalov, Tapiwanashe Matan- gira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaven- ture F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Balh, Stella Biderman, Alessia Bat- tisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ata- man, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. 2022. Quality at Glance: An Audit of Web-Crave Web-Crawled Multilingual Datasets. Transactions of the Association for Com- putational Linguistics, (2:
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Out, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised Cross- lings-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational L Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In North American Chapter of the Asso- ciation for Computational Linguistics.
Cheikh M. Bamba Dione, David Ifeoluwa Adelani, Pe- ter Nabende, Jesujoba Oluwadara Alabi, Thapelo Sin- dane, Happy Buzaaba, Shamsuddeen Hassan Muham- mad, Chris C. Emezue, Perez Ogayo, Anuoluwapo Aremu, Catherine Gitau, Derguene Mbaye, Jonathan
Taku Kudo and John Richardson. 2018 SentencePiece: A Simple and I Language Independent Subword Tok- enizer and Detokenizer for Neural Text Processing. In Conference on Empirical Methods in Natural Lan- guage Processing.

---

Colin Leong, Joshua Nemecek, Jacob Mansdorfer, Anna Filighera, Abraham Owodunni, and Daniel White- nack. 2022. Bloom Library: Multimodal Datasets in 300+ Languages for a Variety of Downstream Tasks, In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 8608-8621, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Pedro Javier Ortiz Suárez, Benoit Sagot, and Laurent Romary. 2019. Asynchronous Pipelines For Pro- cessing Huge Corpora on Medium to Low-Resource Infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CML.C-7) 2019. Cardiff, 22nd July 2019, pages 16, Mannheim. Leibniz-Institut für Deutsche Sprache.
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Her- fernan, Elahe Kalbassi, Janice Lam. Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia-Gonzalez Prangthip Hansamti. John Hoffman, Semarley Jarret, Kaushik Ram Sadagopan, Dirk Rowe, Shanmon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn. Alexandre Mourachko, Christophe Rop- ers. Saftyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scal- ing human-centered machine translation. ArXiv, 0e//2207.04672..
Chester Palen-Michel, June Kim, and Constantine Lig- nos. 2022. Multilingual Open Text Release 1: Public Domain News in 44 Languages. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2080-2089, Marseille, France. Eu- ropean Language Resources Association.
Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. 2023. Language Model Tokeniz- Introduce Unfairness Between Languages. ArXiv, abs/2305. 15425.
Jack W Rae. S Borgeaud, T Cai, K. Millican, J Hoff- mann, HF Song. Aslanides, S Henderson, R Ring. S Young, et al. 2021. Scaling Language Models: Methods, Analysis & Insights from Training Gopher (2021). ArXiv, abs/21 12. 1446.
Kelechi Ogueji, Yuxin Zhu, and Jimmy J. Lin. 2021. Small Data? No Problem! Exploring the Viabil- ity of Pretrained Multilingual Language Models for Low-resourced Languages. Proceedings of the 1st Workshop Multilingual Representation Learning.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the Lim- its of Transfer Learning With Unified Text-to-Text Transformer. The Journal of Machine Learning Re- search,
Odunayo Ogundepo, Tajuddeen R. Gwadabe, Clara E. Rivera, Jonathan H. Clark, Sebastian Ruder, David Ifeoluwa Adelani, Bonaventure F. P. Dos- sou, Abdou Aziz DIOP, Claytone Sikasote, Gilles Hacheme, Happy Buzaaba, Ignatius Ezeani, Roowei- ther Mabuya, Salomey Osei, Chris Emezue, Al- bert Njoroge Kahira. Shamsuddeen H. Muham- mad, Akintunde Oladipo, Abraham Toluwase Owodunni, Atnafu Lambebo Tonja, Tyanuoluwa Shode, Akari Asai, Tunde Oluwaseyi Ajayi, Clemen- cia Siro, Steven Arthur, Mofetoluwa Adeyemi. Orevaoghene Ahia, Aremu Anuoluwapo, Oyinkan- sola Awosan, Chiamaka Chukwuneke, Bernard Opoku, Awokoya Ayodele, Verrah Otiende, Chris- tine Mwase, Boyd Sinkala, Andre Niyongabo Rubungo, Daniel A. Ajisafe, Emeka Felix Onwueg- buzia. Habib Mbow, Emile Niyomutabazi, Eunice Mukonde, Falalu Ibrahim Lawan, Ibrahim Said Ah- mad, Jesujoba O. Alabi, Martin Namukombo, Mhonu Chinedu, Mofya Phiri, Neo Putini, Ndumiso Mn- goma, Priscilla A. Amuok, Ruqayya Nasir Iro, and Sonia Adhiambo. 2023, AfriQA: Cross-linges lingual Open-Retrieval Question Answering for African Lan- guages.
55511.
Pranav Rajpurkar. Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Conference Empirical Methods in Natural Language Processing.
Philip Resnik, Mari Broman Olsen, and Mona T. Diab. 1999. The Bible as a Parallel Corpus: Annotating the 'Book of 2000 Tongues'. Computers and the Humanities, 33:129-153.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury. Daniel Andor, Sha- ran Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowyez, Alexandru Salcianu, Mare van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdh- cry, Jasmijn Bastings, Jannis Bulian, Xavier Gar- cía, Jianmo Ni, Andrew Chen. Kathleen Kenealy, Clark, Stephan Lee, Daniel H Garrette, James Lee- Thorp, Colin Raffel, Noam M. Shazeer, Marvin Rit- ter. Maarten Bosma, Alexandre Passos, Jeremy B. Maitin- Sheeard,, Noah Fiedel, Mark Omernick, Bren- nan Sacta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. 2022. Scal- ing Up Models and Data with 15x and seqio. ArXiv, abs/2203.1 17189.
Odunayo Jude Ogundepo, Akintunde Oladipo, Mofe- toluwa Adeyemi. Kelechi Ogueji, and Jimmy Lin. 2022. VAr Extending Small Data Pretraining Approaches to Sequence-to-Sequeee Proceedings of the Third Workshop on Deep Learn- for for Low-Resource Natural Language Processing, pages 126-135.
Models. In
Noam M. Shazeer. 2020. GLU Variants Improve Trans- former. ArXiv, abs/2002.05202.

---

Pedro Ortiz Suarez, Benoît Sagot, and Laurent Romary, 2019. Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures.
Sampling
Yocab Size
hum
ibo
kin
VWS
sho
yur
aul
100,000
1.29
1.62
1.80
1.90
1.76
1.24
2.37
2.05
2.22
Conts (
150,000
1.23
1:25
1.56
1:64
1.19
2.30
1.92
Linting Xue, Aditya Barua, Noah Constant, Rami Al- Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards Token- Free Future with Pre-trained Byte-to- Syte-to-Byte Models. Transactions of the Association for Computational Linguistics, 10:291-306.
250,000
1.22
1.47
1.54
1.61
1.53
1.19
2.08
1.90
1.91
100,000
1.25
1.43
1.22
1,65
1.54
1.29
2.07
1.67
1.90
Config
145
1.25
1.54
20000000
1.20
1.38
1.23
1.36
1.55
Table 5: Tokenizer Fertilities: We measure the fertil- ities of our tokenizers with varying vocabulary sizes using the MasakhanePOOS dataset. The 150k tokenizer gives the best trade-off in size and fertility scores across all languages, especially in the second sampling config- uration.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami A1-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. m A Massively Multilingual Pre-trained Text-to-Text Transformer. In Proceed- ings of the 202] Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies. pages 483- 498, Online. Association for Computational Linguis- tics.
include the category under which each article was published. This information may be useful for iden- tification of the domains in our dataset. We also release list of the top document URLs for each language and invite native speakers to audit these sources to help us improve the quality of WURA.
Judit Ács. 2019. Exploring BERT's Vocabulary.
Data
A.1 mC4 Audit
B Tokenization
We aim to tease out heuristics that are guaranteed to help quickly and reliably extract high-quality monolingual text across the African languages in mC4. First, we reduce the source URL of each doc- ument to its h and keep a list of unique hostnames that exist for each language. For each language, we first sample hostname then sample 20 documents sourced from the sampled hostname. This sampling strategy not only allows to audit more documents and sources faster, it allows us trace existing quality issues the source URLs that produced the documents. We follow non-expert au- diting strategies proposed by Kreutzer et al. (2022). Additionally, we also visit the hostname URL" to ascertain its purpose for speakers of the language and translate paragraphs in the document using Google Translate.
In multilingual settings, the design of tokenizers has great impact on the downstream utility and cost of inference of language models across lan- guages (Petrov et al., 2023; Ahia et al., 2023). We characterize the performance our tokenizers us- ing fertility (Ács.. 2019), defined as the number of subwords created per word (or per dataset) by the tokenizer. We compute fertility on the langauges covered by MasakhanePOS (Dione et al., 2023). We train multiple unigram language models on our dataset using Sentencepiece (Kudo and Richardson, 2018) with vocabulary sizes ranging from 100, 000 to 250, 000. As shown in Table 6 above, our dataset sizes varies over orders of mag- nitude between languages. To alleviate unfair treat- ment of the lowest-resoureed of the languages we consider, we follow Guillaume Lample and Alexis Conneau (2019) to learn the unigram language models on sentences sampled according to a multi- nomial distribution with probabilities 4ii=1...N cal- culated as follows:
A.2 Web Crawling
We open-source Otelemuye, 8 an extensible frame- work for large scale web-crawls. In our work, we crawl at a safe pace that does not degrade the web- site's performance and respect the rules websites publish in their robots.txt." Where possible,
Σ= where ΣN and 0.3. N denotes the number of languages and n,, the number of sentences in language i. We denote this as sampling configuration D.. We also investigate a sampling configuration 2 in which we further up- sample languages which still do not have adequate
4: P."
'The hostname property of the URL interface is string con- taining the domain name of the URL
"Some hostnames may have moved to new addresses or shut down permanently. In such cases, check the Internet Archive.
"https://githe thutthhubcc.... con/ theyorubayesiin//otelemme "https: h http///deelllo/e opers. opers.google ctw htam///aacc///aa /occ/ crawling- hew.ling-indeiin//eeeee indexing/robots/ingg
htttt:////iiuaaacmcc/stttrrrrrrr dataset

---

African Languages mC4
Language
# Crawled Articles
Wikipedia Articles
# mC4 Articles
# Combined Articles
# De-duped Articles
Size (GB)
Articles
Afrikaans (afr)
139, 977
107, 860 860
978, 740
226, 577
158, 680
4.8
Amharic (amh)
22, 831
15, 713
112, 843
151, 387
150, 958
1.2
Chichewa (nya)
135
42,917
44, 052
44, 052
0.4
Hausa (hau)
247, 507
25, 957
147, 028
420, 492
399,866
0.9
Igbo (ibo)
6, 196
16, 158
34, 802
57, 156
57, 095
0.2
Malagasy (mlg)
35, 839
95, 612
110, 841
242, 292
240, 233
0.5
Sesotho (sot)
1,076
41,
42, 623
42,623
0.2
Shona (sna)
10, 637
10,847
48, 337
69, 821
67, 762
0.5
Somali (som)
585, 928
11,241
513, 028
1, 110, 197
1, 084, 982
2.3
Swahili (swa)
265, 733
01,
831,
173, 912
11151,995
3.5
Xhosa (Xho)
1,554
24, 992
26,546
26, 546
0.1
Yoruba (yor)
28, 463
915
20,463 463
81,841
81, 632
0.1
Zulu (zul)
11,331
387
72, 718
72, 718
0.7
African Languages not in mC4
Afaan Oromoo (orm)
18, 675
1,535
22, 410
22, 410
0.06
Kinyarwanda (kin)
17, 218
7,423
32, 437
32, 437
0.10
Tigrinya (tir)
8, 728
427
9, 155
9, 155
0.03
Total
1, 193,,, 097
422,536
2, 968, 087
4, 793, 623
652,, 549
18.9
Other Languages
Arabic (arz)
1, 1,617,44 402
1, 617, 402
1, 6,.77, 402
0.72
English (eng)
31, 727
1, 500, 000
1, 531, 727
1,531, 727
4.0
French (fra)
103, 529
1,500, 000
1, 603, 529
1, 603, 529
3.6
Portuguese (por)
107, 670
1, 1,102, 551
1, 210, 221
1,210, 221
2.3
Total
1, 6366,,60 023
6, 1422 489
2, 968, 087
10, 756, 502
10, 615, 428
29.5
Table 6: WURA Dataset Statisties: We provide the count of crawled articles, Wikipedia articles, original mC4 articles, and final size before passage-level filtering for each language. In total, we have ~4.7M articles, more than 1.5 times what mC4 contains across 16 African languages.
Model
hau
ibo
AVG
AirTeNaa V2 (Base)
22.6/8. 1/17.7 3/29. 38.1/16.
16. 1/57/14.11
29 29.3/10.1/23.2 5/2991
24.2/15.5//2..9 16.2/15.1/26.9 29.8/10.0/2..1
30.9/12.6/24.6 34.2/13.9/267
AirTeVa V2 (Large)
34.9/12./2.9
16.85 2/144
38.8144.90000
38.3/18.1/314
38.2/16.0027.6
Table 7: XL-SUM results: Performance based on Rouge- 1, Rouge 2 and Rouge-L. L. VariTeVa V2 Large outperforms VariTeVa V2 Base across all languages considered.
Model
anth
fra
hau
ibo
lin
Iug
tir
xho
AVG
AVGSL
AliiTVas V2 (Base)
92.8
90.6
88.0
89.4
86.1
86.0
91.1
90.8
96.8
92.3
93.3
75.7
87.0
86,4
93.6
92.3
89.5
88.9
AfiioVaa V2 (Large)
92.4
91.1
88.2
89.8
88.4
90.2
92.1
88.2
96.9
92.6
93.2
77.9
86.0
86.0
94,6
91.8
90.0
89.3
Table 8: MasakhaNews Classification Results: Evaluation is done using the weighted F1 score and the scores
presented are averaged across seeds. AfriTeVa V2 Large marginally improves overs Base results.
representation after sampling sentences with the calculated probabilities. Simply, after calculating probabilities using (, upsample by factor of 10 for ibo, kin, nya, sna, sot, tir, xho, and factor of for amh, arz, mlg, som. We make this choice of upsampling factor taking into consider- ation the maximum amount data we can train with given our CPU resources. The fertility tok- enizers trained on the sentences obtained by both
sampling configurations are presented in Table 5.
Across both configurations 1 & 2, we ob-
tain the best tradeoff between fertility distribu- tions across the languages and vocabulary size 150, 000. Tokenizers obtained from ② perform better across board, improving fertility markedly for ibo, kin, nya, sna, xho, yor and zul without affecting fertility for hau and swa negatively.

---

C AfriTeVa Va V2 Large
We also pretrain a large variant of AfriTe V2 and present its effectiveness on summarization (Table 7) and classification (Table 8). For summarization, we finetune both models for 10 epochs and make inference using beam search with width of 4. We gain improvements over the base model across both tasks, particularly for summarization where ibo benefits the most.


Reviews:
[{"review_id": "jdwNmmyQ4y", "reviewer": "Reviewer_Rqmg", "summary": "", "questions": "A. In the abstract, you write that the improvement was achieved for 17 languages. Based on which task do you make this claim? Which language not included in the set of 16 African languages benefited?\n\n\nB. Ad passage-level processing: how did you identify the potentially offensive chunks?\n\n\nC. Why did you not compare to other baselines for QA and summarization tasks?\n\nD. Suggestion: It would be interesting to compare the effectiveness of ArawaT5 in cross-lingual transfer.\n\n", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer Rqmg for engaging with our work and providing thoughtful feedback. You found our work well-motivated, and our curation detailed and well-documented. You also noted that the end result of our endeavour is a large, clean corpus beneficial for the wider NLP community and improved T5 models for African languages. \n\n**Addressing your concerns**\n\n**ArawaT5 underperforms T5 in Pidgin:** Indeed, ArawaT5 does not outperform baselines for text classification and summarization on Nigerian Pidgin but we remain competitive, especially on the summarization task! We note that ArawaT5 was not pretrained on Nigerian Pidgin. As Nigerian Pidgin is an English-based creole, models pretrained on large amounts of English text are expected to be performant for the language. However, ArawaT5 was pretrained on far less English text than the baselines we compare to, save for AfriTeVa. Still, our results for Nigerian Pidgin remain competitive with the best baselines across the evaluation tasks. \n\n**Comparison of QA and summarization results with other baselines:** We acknowledge that comparing our models with other models beyond mT5 is important. We have indeed run these experiments for QA and find that our model outperforms AfriTeVa and AfriMT5 by at least 9 F1 points on average. \n| Metric               | Model            | bem | hau  | bo  | kin  | twi | yor  | zul  | AVG           |\n|----------------------|------------------|--------------|---------------|---------------|---------------|--------------|---------------|---------------|---------------|\n| F1  | mT5              | 2.9          | 25.8          | 41.7          | 25.5          | 5.3 | 11.9          | 24.7          | 17.6          |\n|                      | AfriTeVa-Base    | 3.5          | 4.6           | 5.5           | 4.8           | **5.4**          | 6.1           | 4.4           | 4.9           |\n|                      | AfriMT5-Base     | **6.4** | 39.7          | 40.7          | 30.3          | 5.3          | 21.8          | 31.9          | 25.2          |\n|                      | ArawaT5 | 5.7          | **45.4**   | **57.1** | **45.4** | 2.1          | **37.6** |**45.9** | **34.2** |\n| EM  | mT5              | 1.1          | 22.3          | 34.7          | 20.2          | **3.5** | 7.8           | 20.9          | 13.9          |\n|                      | AfriTeVa-Base    | 2.0          | 2.7           | 4.2           | 3.2           | 3.1          | 3.9           | 3.1           | 3.2           |\n|                      | AfriMT5-Base     | 4.2          | 33.0          | 33.0          | 23.1          | 2.9          | 15.7          | 25.5          | 19.6          |\n|                      | ArawaT5 | **5.2** | **36.7** | **47.7** | **33.7** | 1.4          | **29.5** | **37.8** | **27.4** |\n\nOur summarization results using AfriMT5 and AfriTeVa will also be included in the final version to ensure that researchers can properly contextualize the results we obtain using ArawaT5. \n\n**Questions from reviewer Rqmg:**\n- **In the abstract, you write that the improvement was achieved for 17 languages. Based on which task do you make this claim? Which language not included in the set of 16 African languages benefited?:** \nAs previously mentioned, we did not include Nigerian-Pidgin and this gives a total of 16. This mistake will be corrected in the camera-ready version of our paper.\n- **At passage-level processing: How did you identify the potentially offensive chunks?**\nFor this work, our cleaning pipeline filters out any passages that contain at least one toxic word included in Facebook’s [Toxicity-200](https://github.com/facebookresearch/flores/tree/main/toxicity) dataset for the relevant language. While Kreutzer et al’s audit of mC4 found that mC4 did not contain a significant amount of offensive content (0.06% of sentences they audited) and our web crawls mainly focused on verified news publications, we feel strongly that this toxicity filtering is a necessary component of our data cleaning pipeline since we plan to expand this dataset over time. \n\n**Suggestion: It would be interesting to compare the effectiveness of ArawaT5 in cross-lingual transfer.**\n\nThank you for the suggestion. Cross-lingual transfer is one of the reasons we included Arabic, English, French and Portuguese in our pretraining data. We believe this will allow zero-shot evaluation of ArawaT5 on tasks like information retrieval where African languages have little or no data. Also, there are a number of languages (such as Fon, Fufulde etc.) not included in our corpus because they do not have sufficiently large data. We plan to explore how well ArawaT5 generalizes to these languages and tease out the limits and tradeoffs of multilingualism as we increase the number of languages supported by our models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Well-motivated study and highly needed in the community work on improving language resources for low and mid-resourced African languages.\n\nThe curation method is ingenious and includes multiple well-documented steps aimed at getting improved data representation for the languages in question.\n\nThe authors not only introduce a new dataset but also trains a model outperforming a set of strong baseline models comprising both general multilingual models and the models centered on African languages. The gains are manifested for diverse tasks (QA, text classification, translation, summarization) and are especially impressive on AfricaQA (+100% relative improvement in F1).\n", "reasons_to_reject": "QA and summarization results are only compared against the mT5 model, ignoring the other baselines. The reason for that is not given in the paper.", "questions_to_authors": "A. In the abstract, you write that the improvement was achieved for 17 languages. Based on which task do you make this claim? Which language not included in the set of 16 African languages benefited?\n\n\nB. Ad passage-level processing: how did you identify the potentially offensive chunks?\n\n\nC. Why did you not compare to other baselines for QA and summarization tasks?\n\nD. Suggestion: It would be interesting to compare the effectiveness of ArawaT5 in cross-lingual transfer.\n\n", "paper_topic_and_main_contributions": "The work introduces a new unannotated corpus for 16 African languages called WURA. For languages in question, WURA is better curated and contains 1.5 times more articles than its predecessors (mC4). \n\nThe dataset is used for a pre-training language model based on T5 architecture called AwawaT5. The proposed model outperforms previous general and African-centric language models for most settings with four diverse downstream tasks.\n\nI’m impressed by the richness of contributions of this short paper: both in data curation and language modeling. \n", "excitement": 5, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 1: I suggest grouping the columns with results for African and other languages.\n\nL179: Indicate that Table 5 is in the appendix. Also, here the style is inconsistent only the number is part of the link. I personally prefer this style of referring tables and figures to the one used in other places.\n\nL281: ArawaT5 underperforms T5 also in Pidgin. The results for Swahili are on par."}}, {"review_id": "QeGEJuTcTL", "reviewer": "Reviewer_A7Gh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer A7gh for engaging with our work and acknowledging that we introduced a useful dataset and improved upon strong baselines across a diverse set of evaluation tasks. \n\n**Addressing your concerns**\n\n**Sufficient evidence of improvement in performance:** We acknowledge your concern regarding evidence that our better quality data actually translates to improved model performance across the chosen evaluation tasks. While this concern may be laid to rest by ablation experiments in which we pretrain a language model on the unfiltered mC4 corpus, we were constrained by available resources at the time of submission. Fortunately, we have secured additional TPU Research Cloud Credits to support our work and will begin these ablation experiments. We also note that our model outperforms AfriMT5 which was trained on the unfiltered mC4 corpus by Adelani et al. Beyond this, as generative models take centre stage, the benefits of our cleaned corpora extend beyond model performance to solving the problem of model toxicity at the data level. \n\n**For fair comparisons, results should include mC4 averages without unseen languages** This is a fair criticism of our work. We agree that including averages that exclude language unseen by mT5 and other baseline models during pretraining is worthwhile. Note that since our corpus builds upon mC4 which mT5 & AfriMT5 were trained on, Afaan Oromoo, Kirundi and Tigrinya are the only languages present in our pretraining data but not theirs. Thus, averages without unseen languages do not change the conclusions of our work. For text classification on seen languages alone, ArawaT5 averages 88.9 F1 points compared to mT5’s  78.6. When translating from English into languages mT5 was pretrained on, ArawaT5 averages 18.0 BLEU compared to mT5’s 12.5, AfriMT5’s 14.0 and AfriByT5’s 15.9.  We observe a similar trend when translating into English, as well as for our summarization rouge scores. \n\n**Comparisons of corpus statistics (AfriTeVa vs. Wura):** A major aspect of work is our meticulous audit and extension of the mC4 corpora and this is why we compare the statistics of mC4 versus the dataset we introduce, Wura. AfriTeVa was pretrained on the 1 GB AfriBERTa dataset introduced by Ogueji et al. While we are willing to include an additional table that makes this comparison, we note that the AfriBERTa corpus is a paragraph-level dataset not a document-level dataset level like mC4 and Wura. As such, we can only compare on the basis of the disk size of data available per language.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. a new corpus with a higher proportion of African data.\n2. Superior performance of the new T5 model pre-trained on this corpus, compared to mT5 (pre-trained on the original mC4) and AfriTeVa (pre-trained on small high-quality 1GB data). ", "reasons_to_reject": "1. My main concern is: though the model achieves better performance, I cannot find sufficient evidence to support if the improvement truly comes from, well, \"better quality\".\n2. I suspect that the improvement might be caused by the larger scale. For instance, because the mC4 takes the majority of the corpus according to Table 5, the authors find more available data than the original mC4 and they even update the crawls up to the current day (line 152~156).  However, I cannot find any ablation test to support that the filtering process actually works.\n3. More comparisons of corpus statistics of African languages among the new corpus, the original m4c and 1GB data (AfriTeVa) are needed.\n4.  The comparison is not fair. At least, the author should provide additional AVG of excluding those languages not in mC4.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a new multilingual corpus for 16 African languages and four pivot languages by combining (1) audited and cleaned mC4 (2) crawled indigenous websites and (3) Wikipedia. The empirical evaluation on four cross-lingual tasks suggests that the model pre-trained on the new corpus significantly outperforms mT5 (pre-trained on the original mC4) and AfriTeVa (pre-trained on small high-quality 1GB data). ", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "72BxhDBG2D", "reviewer": "Reviewer_ckvB", "summary": "", "questions": "1. Are you planning on making the model and the dataset publicly available?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer ckvB for engaging with our work and providing thoughtful feedback. You found our curation techniques detailed, well-documented and thoughtful, and touted our work very relevant to the broader NLP community given our consistent improvement upon baselines across a diverse set of evaluation tasks. \n\n**Addressing your concerns**\n\n**Inconsistent claims and naming. Incorrect use of terms:** We thank you for pointing these out and take full responsibility. All the instances pointed out will be corrected and we will ensure consistency across the paper in our final submission.\n\n**Reproducibility questions:** We plan to release both the data and model artefacts to support further research. Our data cleaning pipeline as well as other code artifacts will be open-sourced. For downstream tasks, we evaluated our T5 model under the same conditions as the baselines to which we compared. Our camera-ready submission will also include an appropriate description of hyperparameters used in downstream tasks. All of this will ensure the reproducibility of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a new pretraining corpus for 16 low-resourced African languages. This is beneficial to the wider NLP community to help accelerate the development of newer models. The data creation section was thorough and detailed, highlighting the careful thinking that went into the data.\n2. The authors demonstrate the effectiveness and the high quality of their newly created dataset by using it to pretrain a new T5 model that achieves better results than current existing mT5 models on a variety of tasks. The experiments are convincing and well-explained.", "reasons_to_reject": "1. Some parts of the paper were confusing and hard to read:\n\n   a) Section 1: In the intro, the authors mention that they introduce a dataset for 20 African languages. This is not precise as the dataset contains 16 African languages and four other languages that are spoken in Africa.\n\n   b) Section 1 (L091): the authors mention that their model achieves better performance than mT5 models on 19 languages, but the abstract (L021) mentions 17 languages.\n\n2. Reproducing the results when it comes to the downstream tasks is going to be hard, given that no discussion of hyperparameters is mentioned.", "questions_to_authors": "1. Are you planning on making the model and the dataset publicly available?", "paper_topic_and_main_contributions": "In this paper, the authors introduce a new multilingual dataset comprising 16 African languages and high-resourced languages popularly spoken in Africa - Egyptian Arabic, English, French, and Portuguese. The dataset was created by: 1) extensively auditing and cleaning mC4, a very popular and commonly used multilingual dataset for pretrained models; 2) crawling websites across multiple domains such as news and Wikipedia. Further, the authors use their newly created dataset to pretrained a multilingual T5 model, and they evaluate its performance on multiple downstream tasks. The newly introduced model improves over existing multilingual models across 17 languages and four downstream tasks.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\n* L025: our new model more than doubles the performance --> Our new model achieves more than double the performance\n* L054: news corpora\n* L077: for the respective language --> for the respective languages \n * L094: our new model more than double the performance --> our new model achieves more than double the performance\n* L112: We highlight three important findings of note of our own --> We highlight three important findings\n\nOther comments:\n* Table 4: F1 for hau should be bold.\n* Why Table 5 calls the four non-African languages pivot languages? This is a specific language-transfer lingo, I would avoid it \n* The use of pre-training vs pertaining is not consistent across the paper, I would stick to either one\n\n"}}]

Author rebuttal:

Meta-review: This paper creates a high-quality pretraining corpus in African languages from mC4, pretrains a Transformer-based language model, and further evaluated on multiple downstream tasks. The paper has a clear beneficial contribution to the community, not only in the trained model but also in cleaned corpus and preprocessing strategies. However, some straightforward improvement points of the current submission are identified by the reviewers such as fair comparison (e.g., "results should include mC4 averages without unseen languages"), and presentation clarity (e.g., the language code "pcm" should be explicitly mentioned as Nigerian Pidgin either in the main text or the appendix). During the rebuttal, further constructive comments on the analysis of the "quality" of the pretraining corpus have been brought up, especially since the paper is focused on auditing and cleaning mC4, but less ablation study is conducted. Nevertheless, the paper contributes to low-resource and underrepresented languages, especially on the creation of a new and better quality corpus along with empirical comparison against commonly-used multilingual models. We believe that this contribution is beneficial for the NLP community. Based on the reviews and rebuttal, the mmends accepting this paper, but the authors are highly encouraged to address the comments by Reviewer ckvB and Reviewer A7Gh.
Decision: Accept-Main