SERENGETI: Massively Multilingual Language Models for Africa
Year: 2023

Abstract:
Multilingual pretrained language models (mPLMs) acquire valuable, generalizable linguistic information during pretraining and have advanced the state of the art on task-specific finetuning. To date, only ∼ 31 out of ∼ 2, 000 African languages are covered in existing language models. We ameliorate this limitation by developing SERENGETI, a massively multilingual language model that covers 517 African languages and language varieties. We evaluate our novel models on eight natural language understanding tasks across 20 datasets, comparing to 4 mPLMs that cover 4 − 23 African languages. SERENGETI outperforms other models on 11 datasets across the eights tasks, achieving 82.27 average F1. We also perform analyses of errors from our models, which allows us to investigate the influence of language genealogy and linguistic similarity when the models are applied under zero-shot settings. We will publicly release our models for research.1

heading: 1 Introduction
text: Pretraining NLP models with a language modeling objective has gained popularity as a precursor to task-specific finetuning (Ettinger, 2020). Pretrained models like BERT (Devlin et al., 2019), ELMo (Peters et al., 2018), Roberta (Liu et al., 2019), GPT (Radford et al., 2018, 2019; Brown et al., 2020a), and BART (Lewis et al., 2020) have advanced the state of the art in a wide variety of tasks, demonstrating how these models acquire valuable, generalizable linguistic information during the pretraining process. However, training language-specific models is possible for only a few languages which have large amounts of data. A popular alternative has been pretrained multilingual language models (mPLM) such as mBERT (Devlin
1https://github.com/UBC-NLP/serengeti ⋆ Authors contributed equally.
et al., 2019) and XML-R (Conneau et al., 2020). mPLMs are trained on large amounts of unlabelled data from multiple languages so that low resource languages may benefit from shared vocabulary and other linguistic information from high-resource and similar languages in the model. The vast majority of the world’s ∼ 7, 000 languages today remain uncovered by mPLMs, however.
African languages are no exception. Although there are few mPLMs that support a small number of African languages (Devlin et al., 2019; Ogueji et al., 2021; Nzeyimana and Niyongabo Rubungo, 2022; Alabi et al., 2022a; Jude Ogundepo et al., 2022; Conneau et al., 2020), these cover only a total of 31 languages. This is grossly inadequate considering that Africa is believed to be home to ∼ 2, 000 languages (Eberhard et al., 2021). Each of these languages encapsulates unique features that are essential in preserving linguistic diversity. The same way every species embodies essential value to the natural ecosystem, each language plays a
1498
crucial role in the linguistic ecosystem. That is, each language encodes knowledge about people, their traditions, wisdom, and environment, as well as how it is that they interact with the sum of the concepts in their own culture (Adebara and AbdulMageed, 2022). This in turn allows people and communities to preserve and transmit their knowledge, values, unique modes of thinking, meaning and expression, history, culture, traditions, and memory to next generations, while participating in society and constructing their future (UNESCO 66260, 2022).
Language technology plays an important role in building inclusive knowledge societies, providing access to education and information, supporting freedom of expression, cultural and linguistic diversity, and further stimulating innovation. This technology thus has great impact on multiple domains, including education, government, health, recreation, among others. This motivates adequate representation of African languages in the ongoing technological revolution. This is also likely to connect Africa to the rest of the world. Building technologies for African languages may also aid languages that may be at risk of falling into a state of disuse at an alarming rate, thus hopefully preventing subsequent language death that may become inevitable (Adebara and Abdul-Mageed, 2022).
Developing LMs that represent a large number of African languages is therefore very crucial for achieving progress in Afrocentric NLP (Adebara and Abdul-Mageed, 2022) and indeed in addressing issues related to representation bias in artificial intelligence and linguistic diversity - two research themes of international relevance (Bender et al., 2021). Motivated by this call for Afrocentric NLP, we introduce SERENGETI. SERENGETI is a massively multilingual language model exploiting a large manually-curated dataset for 517 African languages and language varieties. These languages belong to 14 language families and are written in 5 different scripts. In addition to these African languages, SERENGETI is also pretrained on the top 10 most spoken languages globally.
We also introduce AfroNLU, an extensive benchmark exploiting 20 different datasets across 28 different languages and language varieties for various NLP tasks. For even richer evaluation, we also apply our models to an African language identification task covering all the 517 languages in our pretraining. To the best of our knowledge,
AfroNLU is the most extensive and inclusive evaluation benchmark proposed to date for African NLP.
Our contributions in this work are as follows: (1) we collect a large dataset of 517 African languages and language varieties and exploit it to develop SERENGETI. (2) we propose AfroNLU, a new extensive benchmark for African NLU that has the widest and most inclusive coverage for African NLP today. (3) we benchmark SERENGETI on AfroNLU and show through meaningful comparisons how our model excels and acquire new SOTA. (4) we offer a linguistically motivated analysis of model performance substantiated in language genealogy, allowing us for the first time to derive insights across the widest range of African languages in the African NLP literature to date.
The rest of the paper is organized as follows: In Section 2 we discuss related work. We describe genealogical information in Section 3. Next, we give a detailed description of SERENGETI in Section 4. In Section 5 we describe AfroNLU, the benchmark we create. We present performance of SERENGETI in Section 6 and compare it to other mPLMs. We conclude in Section 7, and outline a number of limitations and use cases for our work in Section 8 and Section 9.

heading: 2 Related Work
text: Afrocentric NLP. An Afrocentric approach to technology development is crucial for African languages. An afrocentric approach will mean that what technologies to build and how to build, evaluate, and deploy them arises from the needs of local African communities (Adebara and Abdul-Mageed, 2022). We provide more details in Section B in the Appendix. African Language Models. Here, we briefly describe language models covering any number of African languages. Since we develop encoder-only models in this work, we will restrict our discussion to this category of models. We provide information about the African languages covered by these models in Table 1. AfriBERTa (Ogueji et al., 2021) is trained using a Transformer with the standard masked language modelling objective and covers 11 African languages. The pretraining corpus for this model is small (only 108.8 million tokens), when compared to many other models. AfroLM (Dossou et al., 2022) supports 23 African languages, the largest number of African languages before SERENGETI. It is trained on a multi-domain dataset
2
from various sources (Adelani et al., 2022a; Alabi et al., 2022b; Jude Ogundepo et al., 2022; Niyongabo et al., 2020). It uses a self-active learning framework and achieves SOTA on NER, sentiment analysis, and text classification. Afro-XLMR (Alabi et al., 2022a) uses language adaptation on the 17 most-resourced African languages and three other high-resource foreign languages widely used in Africa (i.e., English, French, and Arabic) simultaneously to provide a single model for crosslingual transfer learning. Authors show that AfroXLM-R has competitive results with AfriBERTa and XLM-R on NER, topic classification, and news classification. KINYaBERT (Nzeyimana and Niyongabo Rubungo, 2022) uses a two-tier BERT architecture that involves a morphological analyzer and explicitly represents morphological information for Kinyawanda–a morphologically rich African language. Authors show that KINYaBERT achieves good convergence and accuracy, and is robust on multiple downstream tasks. mBERT (Devlin et al., 2019) is a multilingual variant of BERT trained on 104 languages including four African languages. XLM-R (Conneau et al., 2020) uses a Transformer based architecture and obtains SOTA on cross-lingual classification, sequence labeling, and question answering on 100 languages including eight African languages.

heading: 3 Genealogy of African Languages
text: Genealogical or genetic classification groups languages based on their historical and evolutionary relationships. Genetically related languages are often classified into similar families in a hierarchical tree like structure that shows the level of similarity between the languages. Languages with a higher degree of similarity belong to the same class while languages with a lower degree of similarity are further subdivided into different classes
and subclasses. Two closely related languages can therefore be viewed as sisters of the same parent language/ancestor–they are languages that evolved over time and/or space from an older parent language (Gerhardt, 2020). Typological classification differs from geneological classification in that the former is based on grammatical features or types (Vossen, 2020). For instance, a typological classification would group tone languages together, or split languages based on their morphological structure into, for instance, isolating or agglutinating languages. Despite this difference, languages that belong to the same family often share similar typological information (Gerhardt, 2020). For example, most Benue-Congo languages are tone languages (Williamson, 2006). In the case of African languages, where typological information is scarcely available (Adebara and Abdul-Mageed, 2022; Güldemann, 2018), utilizing genetic classes may be a useful way to determine typological information. If the typological information of one language in a group is known, we may make a sensible assumption that other languages in that group perhaps share similar features with minor variations. We use geneological classification information in evaluating SERENGETI’s behaviour. Specifically, we investigate the relationship between language similarity and model performance in zero-shot scenarios for South African languages in some datasets in our benchmark. We use classification information from Ethnologue (Eberhard et al., 2021) in all our analyses. We provide a broad overview of the families in our models under six broad ancestors in Section D in the Appendix.

heading: 4 SERENGETI

heading: 4.1 Pretraining Data
text: SERENGETI is pretrained using 42GB of data comprising a multi-domain, multi-script collection
3
of texts that we manually curate. The pretraining data covers 517 African languages and the 10 most spoken languages globally (i.e., Arabic, English, French, German, Greek, Italian, Portuguese, Russian, Spanish, and Turkish). The multi-domain dataset comprises texts from religious, news, government documents, health documents, and existing corpora written in five scripts from the set {Arabic, Coptic, Ethiopic, Latin, and Vai}. For the top ten foreign languages, we randomly select 1M paragraphs from Wikipedia for each language to use in our overall pretraining data. We provide further details of the pretraining data in Section C in the Appendix. We also show all languages in our pretraining data in Tables F.1, F.2, and F.3.

heading: 4.2 Preprocessing
text: To prepare the raw data for pretraining, we perform light preprocessing to retain a faithful representation of the naturally occurring text. Specifically, we ensure that images and non-text materials are not in our dataset by using regular expression and manual curation techniques. We do not perform any further preprocessing of the data before splitting the text off into tokens. For tokenization, we use a WordPiece tokenizer (Song et al., 2021). We experiment with two vocabulary sizes, 110K and 250K.

heading: 4.3 SERENGETI Models
text: We pretrain both Electra style (Clark et al., 2020b; Chi et al., 2021) as well as XLM-R style (Conneau et al., 2020) models, as follows. SERENGETI-E110 and SERENGETI-E250. We first pretrain Electra (Chi et al., 2021) style models. Electra uses a multilingual replaced token detection (MRTD) objective for training. Unlike other training objectives, the goal of MRTD is to
distinguish real input tokens from corrupted tokens. Models built with this objective are pretrained as discriminators rather than generators. We train the models with two vocabulary sizes, 110K and 250K, and hence refer to them as SERENGETIE110 and SERENGETI-E250. Each of these models has 12 layers and 12 attention heads. We pretrain each model for 40 epochs with a sequence length of 512, a learning rate of 2e− 4 and a batch size of 216 and 104 for the SERENGETI-E110 and SERENGETI-E250, respectively. We pre-train the models on 1 Google Cloud TPU with 8 cores (v3.8) from TensorFlow Research Cloud (TFRC). 2
SERENGETI Model. Apart form the Electra models, we also experiment with an XLM-R base architecture. We train the model with a 250K vocabulary size for 20 epochs. This model has 12 layers and 12 attention heads, a sequence length of 512 and a batch size of 8. We pre-train this model on 80 M50 AMD Pod GPUs with 16G ram. Our XLMR model has better performance compared to the Electra models as we will show. We provide information about each model we build and compare with in Table 2.

heading: 5 AfroNLU Benchmark
text: Our goal is to evaluate our models extensively, and so we combine all available datasets we could acquire to create an evaluation benchmark that we refer to as AfroNLU. AfroNLU is composed of seven different tasks, covering both token and sentence level tasks, across 18 different datasets. The benchmark covers a total of 32 different languages and language varieties. In addition we evaluate our best model (SERENGETI) on an African lan-
2https://www.tensorflow.org/tfrc.
4
guage identification (LID) task covering all the 517 languages in our pretraining collection. For LID, we use two datasets to test SERENGETI. This puts AfroNLU at a total of 20 different datasets and eight different tasks. To the best of our knowledge, our evaluation benchmark is the most extensive compared to previous published research. We provide detailed statistics of the datasets comprising AfroNLU in Table 3. We also provide a detailed comparison of our AfroNLU benchmark with evaluation data from other models in Table 4.
We now describe each of the downstream tasks in AfroNLU.

heading: 5.1 Named Entity Recognition (NER)
text: We evaluate our models on NER datasets across multiple languages. We use MasakhaNER data (Ifeoluwa Adelani et al., 2021), WikiAnn (Pan et al., 2017; Rahimi et al., 2019), Yoruba-Twi NER data (Alabi et al., 2020), Distance Supervision NER (DS NER) Data (Hedderich et al., 2020) and multiple NER data from SADiLaR. For our experiments, we use the region aggregates on MasakhaNER. Specifically, we use MasakhaNER-east, MasakhaNERwest, and MasakhaNER-eastwest. MasakhaNEReast includes NER data for Amharic, Kinyawanda, Luganda, Luo, and Swahili. MasakhaNER-west includes NER data for Hausa, Igbo, Nigerian-Pidgin, Wolof, and Yoruba. MasakhaNER-eastwest, on the other hand, includes a combination of MasakhaneNER-east and MasakhaneNER-west. Data from SADiLaR cover ten indigenous South African languages and is annotated for person, organisation, location, and miscellaneous named entities. Miscellaneous named entities refer to all rigid designators that do not fall into one of the other categories, including temporal expressions (dates
5
and times), URLs, numerical expressions, publications, names of languages, nationalities, among others. More details about the datasets are in Table 3.

heading: 5.2 Part of Speech Tagging
text: We test our models on POS tagging datasets for Igbo taken from IgboNLP (Onyenwe et al., 2018, 2019). In Table 3, we provide the statistical details for the dataset.

heading: 5.3 Phrase Chunks
text: We evaluate our models on phrase chunks datasets for ten Indigenous languages of South Africa (see Table 3). The data has annotations for noun, verb, adjective, adverbial, and prepositional phrase chunks. Words not belonging to these phrase types are labelled with the tag O.

heading: 5.4 Sentiment Analysis
text: We finetune our model on three sentiment analysis datasets, including Bambara Sentiment dataset (Diallo et al., 2021), YOSM–a new Yorùbá Sentiment Corpus for Movie Reviews (Shode et al., 2022), and the Nigerian Pidgin sentiment dataset (Oyewusi et al., 2020), respectively. Some details of these datasets is in Table 3.

heading: 5.5 News classification
text: We use news classification datasets for Amharic (Azime and Mohammed, 2021), Kinyarwanda (Niyongabo et al., 2020), Kirundi (Niyongabo et al., 2020), and Swahili (David, 2020a,b). The Amharic dataset contains six classes–news, sport, politics, international news, business, and entertainment. The Swahili dataset also has six categories including local news, international, finance, health, sports, and entertainment. The datasets for Kinyarwanda and Kirundi have 14 and 12 categories each, respectively. Again, data statistics are in Table 3.

heading: 5.6 Topic classification
text: We include topic classification datasets for Yorùbá and Hausa (Hedderich et al., 2020). The Yorùbá and Hausa datasets contain news titles collected from VOA Hausa and BBC Yorùbá news sites. The Yorùbá dataset has seven topics–Nigeria, Africa, world, entertainment, health, sports, and politics, while the Hausa dataset is categorized into five topics - Nigeria, Africa, world, health, and politics. In Table 3, we provide details about the data split sizes.

heading: 5.7 Question Answering
text: We use TYDIA question answering dataset (Clark et al., 2020a). The dataset has a primary task and a gold passage task. The primary task has two subtasks, one for passage selection and another that is a minimal answer span. For the passage selection subtask, a list of passages is given and the required response is either the index of the passage where the answer to the question is or null (if no answer exists in the passage). The minimal answer span subtask on the other hand gives a full article and the expected answer is either the start and end byte indices of the minimal span that answers the question, yes or no response, or null (if no minimal answer exists). For the gold passage task, a correct answer is predicted from a passage containing one answer. This is similar to existing reading comprehension. We use the Kiswahili dataset alone, since it is the only African language in the dataset. Details about the data splits can be found in Table 3.

heading: 5.8 Language Identification
text: We also evaluate SERENGETI on the task of language identification (LID). LID focuses on identifying the human language a piece of text or speech segment belongs to, making automatic LID an important first step in processing human language appropriately (Tjandra et al., 2021; Thara and Poornachandran, 2021). We use datasets from AfroLID (Adebara et al., 2022b) for this task. AfroLID data is a multi-genre, multi-script dataset for 517 African languages. We compare the performance of AfroLID data on our models with performance on AfroLID tool. To ensure a fair comparison, the data used for AfroLID is completely different from the data used for SERENGETI. We also evaluate our LID model on AfriSenti dataset (Muhammad et al., 2022; Yimam et al., 2020).

heading: 6 Experimental Setup and Evaluation
text: We evaluate SERENGETI on eight task clusters in the benchmark, and report results on our Test set in Table 5. We also report performance on our Dev set in Table E.1 (Appendix). For each task cluster, we finetune for a maximum of 25 epochs with a patience value of five. We compare results from SERENGETI, SERENGETI-E110, and SERENGETI-E250 to encoder-only models covering any number of African languages. Specifically, we compare with XLMR, mBERT, Afro-XLMR, and AfriBERTa. We report the results of each experiment as an average of three runs, showing
6
the standard deviation. We also evaluate SERENGETI on language identification and show results on Afrolid in Table 6 and on Afrisenti in Table 7. For multilingual datasets in each task, we show evaluation results per language, comparing the performance of various models in Table E.4 in the Appendix.

heading: 6.1 Performance Analysis
text: We report the results for seven of our eight tasks in Table 5. Named Entity Recognition (NER). SERENGETI sets a new SOTA on six out of eight datasets
on the NER cluster. The lowest F1 across all models are on NCHLT and Yoruba-Twi datasets (on both Dev and Test). SERENGETI achieves best performance on both of these datasets on Test (with 73.18 F1 on the first and 71.25 on the second). Phrase Chunking. SERENGETI outperforms all models on the phrase chunking task on both Dev and Test data, reaching 90.51 F1 on Test. Part of Speech (POS) Tagging. In the POS tagging task, SERENGETI outperformed all other models in the Dev. and Test sets. News Classification. Our SERENGETI outperforms other models on three out of four datasets on Test data (and on two datasets on Dev).3. We do not report SOTA results for Amharic, Kirnews, and Kinnews datasets because their authors report performance in accuracy (and so are not comparable to our results). We show performance of SERENGETI on each category in the news classification cluster in Figure E.1 in the Appendix. Sentiment Analysis. SERENGETI-E250 outperforms other models on one out of three tasks in our sentiment analysis task cluster. Afro-XMLR and AfriBERTa outperform other models on one each. To further investigate performance, we conduct an error analysis on the three sentiment datasets (see Figure E.2 in the Appendix). Topic Classification. AfriBERTa outperforms
3Our SERENGETI-E110 outperforms SERENGETI on one dataset in Dev and Test sets
7
other models on both tasks in our topic classification cluster, followed by SERENGETI. We show confusion matrices for Hausa and Yoruba topic classification in Figure E.3 in the Appendix. Language Identification. SERENGETI outperforms AfroLID on AfroLID and AfriSenti data (see Table 6 and 7 for details). We also compare the performance of SERENGETI to AfroLID, and Franc4, on the 88 African languages represented in Franc in Table E.3 (Appendix). SERENGETI outperforms AfroLID and Franc with an average F1 score of 96.29. SERENGETI outperforms both models on 59 languages and has similar results with AfroLID on 19 languages. Next, we evaluate the performance of SERENGETI on Creole languages. Again, we record improvement in results for Creole languages when compared with AfroLID. SERENGETI outperforms AfroLID in 7 out of 9 languages and acquires similar scores on 2 languages. We assume that the addition of the ten most spoken languages to the pretraining data for SERENGETI may have helped the model learn the Creoles better. This is because Creoles share some features including vocabularies and syntax with some of those top ten languages.

heading: 6.2 Error Analysis
text: In the sentiment analysis cluster, best performance is recorded for positive categories while negative categories have the worst performance. A finegrained analysis of the Yoruba sentiment dataset found that SERENGETI failed to correctly categorize sentiment if the polarity item(s) were not seen in training, can be associated with both positive and negative sentiments, the polarity item(s) is a negation, or if ambivalent markers are present in the sentence. We provide a table showing examples of each type of error we found in Table E.2 in the Appendix. For the news classification task, politics and tourism are the best performing classes while education and relationships have the worst performance on kirnews and kinnews respectively. It is important to mention that the worst performing categories do not have the smallest data sizes. For the topic classification, the best performance is on the world class for Hausa topic modelling while entertainment and sport have best performance for Yoruba. The worst performance is on Nigeria and health for Hausa and Yoruba topic datasets respectively.
4A publicly available LID tool covering 88 African languages.

heading: 6.3 Imbalanced Distribution
text: We find imbalances in the class distributions for all datasets except YOSM. We find a positive correlation between the size of each category in a dataset and the model accuracy. We also find a positive correlation with the number of examples in a specific class and the accuracy we acquire. We provide confusion matrices that represents the sizes of each category and the performance of SERENGETI in Figures E.4, E.5, and E.6 in the Appendix.

heading: 6.4 Genealogy & Language Contact
text: Our preliminary analyses show that language similarity may improve model performance in zeroshot settings. This we believe is due to high crosslingual transfer information (Conneau et al., 2020) from similar languages. Similar languages often share many features (e.g., vocabulary, syntax, and script) sometimes up to a point of mutual intelligibility (Nassenstein, 2019; Arndt, 2015; RoyCampbell, 2006). Languages in contact may also have such similarities. By language in contact, we mean all languages that speakers of a specific language interact with and influence. A language can be in contact with another due to trade, geographic proximity, migration, or even colonization. Languages in contact can influence each other in multiple ways, such as borrowing words, grammatical structures, phonology, or orthographic conventions (Matras, 2009). To illustrate our hypothesis, we select two datasets with South African (SA) languages in AfroNLU - NCHLT-ner and phrasechunk. We select SA languages because they are contact languages (see Figure D.5 in Appendix for a genealogical classification tree that highlights the SA languages.) (Nassenstein, 2019; Arndt, 2015; Roy-Campbell, 2006).
To determine the significance of language similarity and language contact in our own zero-shot settings, we measure the Jaccard similarity between the pretraining data for the SA languages (see Table 8). We find strong similarities between some of these languages (see bolded examples in Table 8). We also finetune a BERT model and compare the performance of BERT with MBERT. We do this because BERT does not include any similar language in its representation.
XLM-R, mBERT, and AfriBERTa are not trained on most SA languages but have high scores in zero-shot settings see Table 9 and Table E.4 in Appendix. We argue that XLM-R in addition to
8
for the genealogy analysis. (F1) score is the evaluation metric. We use Red highlights to indicate languages in zero-shot setting. We evaluate BERT, a monolingual model as a sanity check for our evaluation.
cross-lingual transfers from other languages acquires representation from afr and xho where xho alone shares more than 0.4 similarity with afr, nbl, nso, and zul. mBERT also learns representation from afr while AfriBERTa learns representations from Gahuza which is a code-mixed variety of KIN and RUN. BERT on the other hand significantly performs lower than MBERT in all languages except on ssw, and ven (Phrase chunk). SERENGETI, however, outperforms other models on these languages which demonstrates the impact of pretraining on each of these languages.
These analyses are in no way conclusive, but do provide insights on how linguistic information may impact model performance in zero-shot set-
tings. Future work can further probe the influence of similar languages in a more in-depth fashion. (See Appendix F for detailed analysis).

heading: 7 Conclusion
text: We reported our efforts to develop SERENGETI, a suite of three massively multilingual language models for African NLP. SERENGETI outperforms 4 mPLMs on 11 datasets across 8 tasks. We provide extensive evaluations of model outputs, including zero-shot performance of the mPLMs. We also offer broad linguistically-motivated analyses of model performance.
9

heading: 8 Limitations
text: We identify the following limitations for our work:
1. Due to limited access to a wide network of native speakers from the majority of languages, we were able to manually inspect only a subset of languages present in our pretraining data. Specifically, we could only manually evaluate Afrikaans, Yorùbá, Igbo, Hausa, Luganda, Kinyarwanda, Chichewa, Shona, Somali, Swahili, Xhosa, Bemba, and Zulu. Future work should focus on increasing the subset of languages evaluated manually in order to ensure quality. We believe automatic analyses are not sufficient before development of models that get deployed in particular applications.
2. Another limitation is related to our inability to perform extensive analysis of biases and hateful speech present in our pretraining data. Again, this is due to relatively restricted access to native speakers (and even automated tools) to perform this analysis. As a result, we cannot fully ensure that our models is free from biases and socially undesirable effects. Therefore, it is important that these models be used with care and caution, and be analyzed for biases and socially undesirable effects before use.
3. Additionally, due to unavailability of sufficient computing resources, we were unable to evaluate large language models such as BLOOM, even though it covers 22 African languages.
4. Finally, even though AfroNLU has diverse tasks at the word and sentence level, these tasks only cover few African languages. We therefore encourage the creation of more datasets for downstream NLU tasks in more (and more diverse) African languages. We believe broader benchmarks will continue to be important for future progress in African NLP.

heading: 9 Ethics Statement and Wider Impacts
text: SERENGETI aligns with Afrocentric NLP where the needs of African people is put into consideration when developing technology. We believe SERENGETI will not only be useful to speakers of the languages supported, but also researchers of
African languages such as anthropologists and linguists. We discuss below some use cases for SERENGETI and offer a number of broad impacts.
1. SERENGETI aims to address the lack of access to technology in about 90% of the world’s languages, which automatically discriminates against native speakers of those languages. More precisely, it does so by focusing on Africa. To the best of our knowledge, SERENGETI is the first massively multilingual PLM developed for African languages and language varieties. A model with knowledge of 517 African languages, is by far the largest to date for African NLP.
2. SERENGETI enables improved access of important information to the African community in Indigenous African languages. This is especially beneficial for people who may not be fluent in other languages. This will potentially connect more people globally.
3. SERENGETI affords opportunities for language preservation for many African languages. To the best of our knowledge, SERENGETI consists of languages that have not been used for any NLP task until now. We believe that it can help encourage continued use of these languages in several domains, as well as trigger future development of language technologies for many of these languages.
4. To mitigate discrimination and bias, we adopt a manual curation of our datasets. Native speakers of Afrikaans, Yorùbá, Igbo, Hausa, Luganda, Kinyarwanda, Chichewa, Shona, Somali, Swahili, Xhosa, Bemba, and Zulu also manually evaluated a subset of the data to ensure its quality. The data collected for this work is taken from various domains to further ensure a better representation of the language usage of native speakers.
5. Although LMs are useful for a wide range of applications, they can also be misused. SERENGETI is developed using publicly available datasets that may carry biases. Although we strive to perform analyses and diagnostic case studies to probe performance of our models, our investigations are by no means comprehensive nor guarantee absence of bias in the data. In particular, we do not have access
10
to native speakers of most of the languages covered. This hinders our ability to investigate samples from each (or at least the majority) of the languages.

heading: Acknowledgements
text: MAM gratefully acknowledges support from Canada Research Chairs (CRC), the Natural Sciences and Engineering Research Council of Canada (NSERC; RGPIN-2018-04267), the Social Sciences and Humanities Research Council of Canada (SSHRC; 435-2018-0576; 895-2020-1004; 895- 2021-1008), Canadian Foundation for Innovation (CFI; 37771), Digital Research Alliance of Canada,5 UBC ARC-Sockeye,6 Advanced Micro Devices, Inc. (AMD), and Google. Any opinions, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of CRC, NSERC, SSHRC, CFI, the Alliance, AMD, Google, or UBC ARCSockeye.

heading: B Literature Review
text: Representation learning is an integral part of modern NLP systems. It has significantly improved the state of the art in natural language understanding (NLU) and natural language generation (NLG). We now discuss Afrocentric NLP, Multilingualism in NLP, Diversity and Inclusion in NLP, MLMs, and LMs for African languages.
19
B.1 Afrocentric NLP
More than 2, 000 Indigenous languages are spoken in Africa, which is about a third of all languages spoken in the world (Eberhard et al., 2021). Unfortunately, the majority of these languages have not received any NLP attention to date. Rather, most NLP research has focused on higher resource languages. Most of these resourceful languages are typologically very different from Indigenous African languages. Methods used to develop technologies for these languages remain Western-centric, and may not be directly extensible to Indigenous African languages (Adebara and Abdul-Mageed, 2022). Existing NLP technologies also mostly function within the contexts of values and beliefs that reflect western societies and pose unique challenges if the technologies are applied within African com-
munities. Afrocentric NLP adopts a holistic approach to NLP throughout the life cycle of NLP policy making to model development and deployment. It discourages the current language data gas flaring policies that have led to the low resource status of many Indigenous African languages. Afrocentric NLP entails an understanding of the need for multidimensional policies that influence the language policy in education, media, government, and other domains to create ever-increasing, multi-domain, big data sources for NLP. During the archival and collection of language data, Afrocentric NLP necessitates respect of user consent, data sovereignty, wishes of local communities, and privacy (Sutherland, 2018; Daigle, 2021; Makulilo, 2012). For model development, approaches tailored to the unique typological features of African languages
20
are of utmost priority. This also means development of models that understand simple to complex tones–a common feature in about 80% of African languages–serial verb constructions, and many other features (Hyman, 2003; Creissels et al., 2008). Afrocentric NLP also prioritizes deploying models in formats that people without programming experience can easily use. Furthermore, from an Afrocentric approach, development of certain NLP applications such as language models, language identification tools, spelling checkers, language specific keyboards, and machine translation systems is crucial to advance NLP for African languages.
B.2 Multilingualism in NLP
Multilingualism, the ability to handle multiple languages within a single system or model, is becoming increasingly important as the amount of text and speech data in many languages increase. NLP systems capable of handling multiple languages can provide greater access to information and communication for people who speak languages other than those most commonly used or supported by NLP.
Multilingualism in NLP (Ruder, 2022) is mainly achieved through building (1) a single model trained on several languages (Devlin et al., 2019; Conneau et al., 2020) and (2) transfer learning (Raffel et al., 2020; He et al., 2022; Ruder et al., 2019). In the former, large transformer models have achieved state-of-the-art on many tasks while the latter has enabled the use of low-resource languages through finetuned on various NLP tasks. Due to lack of adequate (or good quality) pretraining data (Kreutzer et al., 2021), transfer learning is often the most accessible method for a few low resource languages. Unfortunately, about 94% of the world’s languages are either left-behinds, in that it is probably impossible to build NLP resources for them, or scraping-bys with no labelled datasets (Joshi et al., 2020). For the left-behinds, labelled and unlabelled data is unavailable and even transfer learning approaches are beyond reach. So far, to the best of our knowledge, the largest multilingual model for African languages is pretrained on only 28 African languages (Dossou et al., 2022).
Most multilingual models are often trained with no more than 100 languages because increasing the number of language would mean decreasing its capacity to learn representations of each language (Conneau et al., 2020). Nevertheless, in-
creasing model size was shown to ameliorate this problem (Goyal et al., 2021). In some cases, these benchmarks are translations from English (Artetxe et al., 2020; Nzeyimana and Niyongabo Rubungo, 2022; Ponti et al., 2020) and may not necessarily be a good evaluation for the languages. This is because translating from a source language may mask concept gaps and differences in linguistic constituents (Segerer, 2008) in the target language. That is, translations are at best approximations of the target language (Adebara and Abdul-Mageed, 2022; Joshi et al., 2020). For example, when translating into English (which marks (in)definiteness morphologically) from Yorùbá (which uses bare nouns but marks these features contextually), ambiguities arise (Adebara et al., 2022a).
For evaluation of multilingual models, several benchmarks have been created(Artetxe et al., 2020) with most of these supporting English and other high-resource languages. More recently, a few evaluation sets were introduced for African languages (Ifeoluwa Adelani et al., 2021; Shode et al., 2022; Niyongabo et al., 2020).We include these evaluation sets in our benchmark, which we henceforth refer to as AfroNLU.
When evaluating multilingual models, reporting model performance for each language in the benchmark is preferred because reporting the results as a single value on all languages may mask the model’s performance on individual languages (Ruder, 2022). Large pre-training data, finetuning data, and evaluation benchmarks remain open challenging questions for achieving progress in multilingual NLP. For SERENGETI, we report results for each language in each benchmark across the 9 tasks we evaluate on.
B.3 Diversity and Inclusion in NLP
Diversity relates to the level of variety within a system. It is the measure of distinctiveness between the various individuals within a group. Inclusion on the other hand relates to the level of representation or alignment of an individual within a group and the ability for that individual to function to its fullest ability (Fosch-Villaronga and Poulsen, 2022; Mitchell et al., 2020). Diversity and inclusion in NLP has gained increasing attention in recent years. In general, there is an acknowledgement that overrepresentation (and under-representation) of certain groups in the data used to train models (Mitchell et al., 2020) can be amplified by resulting technologies. This raises concerns about the technology and
21
how it is that it can further existing biases and societal inequalities. But these biases can be exhibited in various ways beyond training data, including the algorithms implemented, the diversity of researchers and engineers developing the models, and the societal and cultural context in which they are used.
Although this is starting to change, often times most of the data exploited in NLP models come from closely related Western languages. Most of these languages are Indo-European (Aji et al., 2022; Joshi et al., 2020), and many of them share close geographic proximity and typology. In addition, the people who speak these languages have similar cultures. The implication is that several linguistic phenomena and typologies are underrepresented in NLP data while those prevalent in Indo-European languages are over-represented (Chakravarthi and Muralidaran, 2021). About 88.38% of the 2, 679 languages whose typology is described in WALS (Dryer, 2013) have not been used in NLP (Joshi et al., 2020). Many ideas and topics, alien to Western cultures have also never been seen (Adebara and Abdul-Mageed, 2022; Bender, 2011) in NLP data. African languages–and indeed many low resource languages–have rich linguistic typology, probably not seen in any other language in the world (Bender, 2011). An obvious problem with the current lack of diversity in NLP data is that the methods and models developed have overfit to these Indo-European typologies and cannot generalize to other typologies. Similarly, machine translation systems have been found to exhibit gender, racial (Bolukbasi et al., 2016; Caliskan et al., 2017; Chakravarthi and Muralidaran, 2021) and stylistic biases (Hovy et al., 2020) in their outputs perpetuated through the data used for training.
A number of studies have also found that algorithms could exhibit biases (Hooker, 2021; Buolamwini and Gebru, 2018; Dwork et al., 2011). For example, a recent study that investigated performance of Amazon Transcribe and Google Speech-To-Text on British English reported notably higher error rates for second language speakers of different varieties of British English (Markl, 2022). In another study, an evaluation of automatic speech recognition systems show substantial performance differences between ’standard’ US English and African American English (AAE) varieties (Koenecke et al., 2020). In this study, commercial ASR systems developed by Amazon, Apple,
Google, IBM, and Microsoft were evaluated and higher rates of errors were recorded for speakers of AAE than speakers of standard US varieties. Similar studies have also recorded higher errors in nonwhite users of English (Wassink et al., 2022; Martin and Tang, 2020). Other studies also reported differences in the performance of Youtube’s automatic caption in different settings. One study reported higher accuracy in the transcriptions of US English compared with Indian English (Meyer et al., 2020). Another reported lower accuracy scores for women and speakers of Scottish English (Tatman, 2017) and non-white speakers of English (Tatman and Kasten, 2017).
Apart from data and algorithmic biases, the diversity crises in AI research is also argued to perpetuate historical biases (Freire et al., 2021). A more inclusive and diverse workforce could promote the exploration of questions and solutions beyond currently investigated research questions (Fosch-Villaronga and Poulsen, 2022). Several initiatives have been adopted to increase diversity in AI, including providing travel grants to marginalized communities to attend conferences, creating mentoring opportunities, special workshops, and community diversity chairs. A number of organizations have also been developed to promote diversity and inclusion in AI and NLP, such as Masakhane, Black in AI, LatinX in AI.
The impact of using biased systems in decision making have been extensively studied. Algorithmic decision-making using biased systems have been shown to have significant discriminatory effects in health (Obermeyer et al., 2019; Eubanks, 2018), employment (Barocas and Selbst, 2016), housing (Buolamwini and Gebru, 2018; Barocas and Selbst, 2016), government benefit allocation (Eubanks, 2018), policing (Buolamwini and Gebru, 2018; Barocas and Selbst, 2016; Angwin et al., 2018), and freedom (Angwin et al., 2018). Lack of diversity also has implication on access to technology. Currently, due to the use of a few high resource languages in NLP, there is limited global access to important applications such as machine translation, speech processing, information retrieval, and sentiment analysis. These technologies play an important role in ensuring a language thrives and offer major contributions to ongoing communication, literacy, education, and translation efforts in communities worldwide. These languages which have barely been used for NLP,
22
usually referred to as low-resource languages, represent more than 90% of the world’s 7, 000 languages (Joshi et al., 2020). The current focus of NLP on resource-rich languages does also have aggravating effects on the language endangerment problem which has been of serious concern for linguistics and language policy around the world. An alarming 50− 90% of languages have been envisaged to go extinct by the end of the century due to the domination by some of these resource-rich languages (Besacier et al., 2014).
Overall, diversity and inclusion in NLP remain active areas of research and comprise pressing issues of international significance. SERENGETI contributes to diversity and inclusion in NLP as follows: (1) We develop SERENGETI, a suite of massively, multilingual language models that support 517 African languages and language varieties. To the best of our knowledge, more than 400 of these languages have never been represented in any language model to date. (2) The languages we support belong to 14 language families. (3) We provide a massive benchmark covering 28 languages across eight different tasks.
B.4 Multilingual Language Models
MLMs have proven effective for cross-lingual NLU and NLG, often outperforming monolingual language models (Conneau et al., 2020). Different objectives have been adopted for training (Doddapaneni et al., 2021), using Transformer architectures. These LMs use one of the three different variants of Transformer architectures–encoder-decoder, encoder-only and decoder-only (Cai et al., 2022).
In the encoder-decoder models, input is encoded by the encoder side and the decoder conducts the operation to predict the sequence one token at a time or just reconstruct it by denoising. MBART (Liu et al., 2020), AfriTeva (Jude Ogundepo et al., 2022), M2M100 (Fan et al., 2020), and MT5 (Xue et al., 2021) are representatives for this architecture. Encoder-only models use only the encoder part of the transformer architecture, while decoderonly models use its decoder only. Some examples of encoder-only models are BERT (Devlin et al., 2019), XLMR (Conneau et al., 2020), and Electra (Chi et al., 2021), while BLOOM (Scao et al., 2022), GPT (Radford et al., 2018, 2019; Brown et al., 2020b), OPT (Zhang et al., 2022) are examples of decoder-only models. Most LMs developed for African languages use an encoder-only architecture, except AfriTEVA and AfroT5 which
use encoder-decoder architectures. These models are further finetuned on specific tasks. Finetuning has demonstrated its effectiveness on various NLU and NLG downstream tasks including part of speech tagging (Conneau et al., 2020), named entity recognition (Ushio and Camacho-Collados, 2021; Conneau et al., 2020), and question answering (Conneau et al., 2020). Finetuning follows a transfer learning approach which attempts to transfer knowledge from other sources to benefit a current task. This is based on the premise that previous knowledge may improve solutions for a current task (Pan and Yang, 2010; Raffel et al., 2020; He et al., 2022; Ruder et al., 2019). Transfer learning allows the domains, tasks, and distributions used in training and testing to be different thereby enabling a new task to leverage previously acquired domain knowledge. Potential benefits include faster learning, better generalization, and a more robust system. In the real world, we find many examples of transfer learning where humans transfer previous knowledge while learning or performing a task. For instance, knowing how to play the piano may facilitate learning to play the guitar and knowing how to ride a bicycle may facilitate learning to ride a motorbike. Finetuning is thus done by reusing the LM’s parameters as a starting point, while adding one task-specific layer trained from scratch. Finetuning can be done on an individual or joint basis (Kitaev et al., 2019). In the former, a model is finetuned on single language for a specific downstream task. In the later, training data from a combination of multiple languages can be jointly finetuned in a single model.

heading: C Pretraining Data
text: We provide details of our pretraining data below: Religious Domain. Our religious data is taken from online Bibles, Qurans, and data crawled from the Jehovah’s witness website. We also include religious texts from the book of Mormon. News Domain. We collect data from online newspapers (Adebara and Abdul-Mageed, 2022) and news sites such as Voice of America, Voice of Nigeria, BBC, Global voices, and DW news sites. We collect local newspapers from 27 languages from across Africa. Government Documents. We collect government documents South African Centre for Digital Language Resources (SADiLaR), and the Universal Declaration of human rights (UDHR) in multiple languages.
23
Health Documents. We collect multiple health documents from the Department of Health, State Government of Victoria, Australia. We collect documents in Amharic, Dinka, Harari, Oromo, Somali, Swahili, and Tigrinya. Existing Corpora. We collect corpora available on the web for different African languages, including from Project Gutenberg for Afrikaans, South African News data. for Sepedi and Setswana, OSCAR (Abadji et al., 2021) for Afrikaans, Amharic, Somali, Swahili, Oromo, Malagasy, and Yoruba. We also used Tatoeba for Afrikaans, Amharic, Bemba, Igbo, Kanuri, Kongo, Luganda, Malagasy, Sepedi, Ndebele, Kinyarwanda, Somali, Swahili, Tsonga, Xhosa, Yoruba, and Zulu; Swahili Language Modelling Data for Swahili; Ijdutse corpus for Hausa; Data4Good corpora for Luganda, CC-100 for Amharic, Fulah, Igbo, Yoruba, Hausa, Tswana, Lingala, Luganada, Afrikaans, Somali, Swahili, Swati, North Sotho, Oromo, Wolof, Xhosa, and Zulu; Afriberta-Corpus for Afaan / Oromo, Amharic, Gahuza, Hausa, Igbo, Pidgin, Somali, Swahili, Tigrinya and Yoruba; mC4 for Afrikaans, Amharic, Hausa, Igbo, Malagasy, Chichewa, Shona, Somali, Sepedi, Swahili, Xhosa, Yoruba and Zulu.

heading: D Typology Information for AfroNLU
text: SERENGETI consists of languages from 14 families including: Afro-Asiatic, Austronesean, CreoleEnglish, Creole-French, Creole-Kongo, CreoleNgbandi, Creole-Portuguese, khoe-kwadi-Hainum, khoe-kwadi-Nama khoe-kwadi-Southwest, IndoEuropean, Niger-Congo, and Nilo Saharan. We discuss the classes from AfroNLU which includes Afro-Asiatic, Austronesian, Creole-English, NigerCongo, and Nilo-Saharan.
D.1 Afro-Asiatic
Afro-Asiatic (aka Hamito-Semitic) is one of the language families of Africa. It consists of five or six branches: Berber, Chadic, Cushitic, Egyptian, Omotic (or a single Cush-Omotic), and Semitic(Porkhomovsky, 2020; Comrie, 2017). Many Afro-Asiatic languages are spoken in Central, East, North, and West Africa. They are also spoken in the Middle East and in scattered communities in Europe, the United States, and the Caucasus (Frajzyngier, 2018). In Figure D.1, we show relationship between the Afro-asiatic languages in AfroNLU.
D.2 Austronesian
Austronesian languages are found along Mainland Southeast Asia, through Indonesia, Western New Guniea, and the Madagascar area in Africa (Eberhard et al., 2021). Many of them have been shown to exhibit an isolating word structure. This means that the words in these languages are of minimal morphological complexity (Gil and Schapper, 2020). In Figure D.2, we show the geneology for Malagasy, the only Austronesian language in our benchmark.
D.3 Creole
A creole language is one spoken initially only in situations of contact between speakers of two or more mutually unintelligible languages, and not as a language within an ethnic group (Sommer, 2020). Historically, creoles have evolved along trade routes or in colonized communities particularly when several groups of people without a common lingua franca are forced to communicate in the presence of a dominant language. Creole languages therefore often include lexical items and grammatical features from multiple contact languages. Usually, one dominant language that is also referred to as the lexifier language contributes a majority of the vocabulary. Creole languages are classified based on their geographical location and are further grouped according to their main lexifier languages, their presumed origins, and the major languages with which they are in contact (i.e., contact languages). Figure D.3 shows the geneology for Nigerian Pidgin, the only Creole in our pretraining collection.
D.4 Indo-European
Afrikaans is the only "Indigenous" Indo-European language spoken in Africa. Although it may also be viewed as not being truly Indigenous to Africa (Kirsten, 2018). Indo-European languages were originally domiciled in Europe, Iran, Turkey, Western Asia and India (Clackson, 2007; Eberhard et al., 2021; Comrie, 2017; Kirsten, 2018). However, due to migration, Indo-European languages are spoken around the world. In 2003, over 2.5 billion people spoke an Indo-European language (Clackson, 2007). In Figure D.4, we show the geneology for Afrikaans.
D.5 Niger-Congo
Niger-Congo, also referred to as NigerKordofanian, is the largest language family
24
in Africa (Good, 2020; Comrie, 2017). It consists of the highest number of languages and speakers in Africa. Niger-Congo languages spread across sub-Saharan Africa, with Benue-Congo, including Bantu languages dominating the southern part of the continent. Figure D.5 shows the Niger-congo languages in our collection. Although we use similar colours for languages which are sisters of the same parent, only some of those languages are mutually intelligible. That is speakers of each individual language understand each other’s language without learning it. Specifically, Kinyawanda (kin) and Kirundi (run) are mutually intelligible (Nassenstein, 2019). Ndebele, Siswati, Xhosa, and Zulu also share various levels of intelligibility mutually intelligible (Arndt, 2015; Roy-Campbell, 2006). Sepedi, Sotho, and Tswana also share some levels of mutual intelligibility (Roy-Campbell, 2006).
D.6 Nilo-Saharan
Nilo-Saharan is subdivided into four branches that include North Eastern, Central Sudanic and two disputed branches–Songhay and Koman (Dimmendaal
et al., 2019; Dimmendaal, 2020; Comrie, 2017). These branches are further divided into other subgroups, languages, and dialects. Nilo-Saharan languages are spoken predominantly by eastern and central African pastoralists, and includes in its main Chari-Nile branch the Central Sudanic and Eastern Sudanic (also called Nilotic) languages. Figure D.6 shows the Nilo-saharan languages in our pretraining data.

heading: E Evaluation
text: E.1 Performance Analysis
In this section, we provide more information about our evaluation procedure and results using visualizations and tables. Figure E.1 shows the confusion matrix for the news classification cluster. Figure E.2 shows the performance of SERENGETI on the sentiment analysis cluster. Each confusion matrix represents each dataset in the sentiment analysis cluster. In Figure E.3, we show SERENGETI performance on each category in the topic classification datasets.
E.2 Error Analysis
In the sentiment analysis cluster, best performance is recorded for positive categories while negative categories have the worst performance. A finegrained analysis of the Yoruba sentiment dataset found that SERENGETI failed to correctly categor-
25
ize sentiment if the polarity item(s) were not seen in training, can be associated with both positive and negative sentiments, the polarity item(s) is a negation, or if ambivalent markers are present in the sentence. We provide a table showing examples of each type of error we found in Table E.2 in the Appendix. For the news classification task, politics and tourism are the best performing classes while education and relationships have the worst performance on kirnews and kinnews respectively. It is important to mention that the worst performing categories do not have the smallest data sizes. For the topic classification, the best performance is on the world class for Hausa topic modelling while entertainment and sport have best performance for Yoruba. The worst performance is on Nigeria and health for Hausa and Yoruba topic datasets respectively. E.3 Imbalanced Distribution
We find imbalances in the class distributions for all datasets except YOSM. We find a positive correlation between the size of each category in a dataset and the model accuracy. The larger the number of examples in a specific class, the better the accuracy,
26
Cluster Task SOTA XLMR mBERT Afro-XLMR AfriBERTa Serengeti-E110 Serengeti-E250 Serengeti
NER
masakaner-v1 84.8±0.3 85.59 ±0.20 82.82 ±0.10 87.79 ±0.33 85.19 ±0.08 86.11±0.27 86.42 ±0.26 88.82 ±0.18 masakaner-v2 85.7±0.1⋆ 87.00 ±0.12 85.07±0.83 87.46 ±0.06 86.19 ±0.11 86.51 ±0.22 86.81 ±0.24 88.98 ±0.20 masakaner-east — 83.52 ±1.03 82.85 ±0.42 87.28 ±0.68 83.33 ±0.56 85.64 ±0.50 87.12 ±0.62 88.09 ±0.57 masakaner-eastwest — 87.70 ±0.30 87.29 ±0.33 89.34 ±0.07 87.77 ±0.34 88.14 ±0.26 88.96±0.15 90.38 ±0.17 masakaner-west — 89.77 ±0.53 90.28 ±0.46 89.97 ±0.23 89.36±0.46 88.24 ±0.52 89.44 ±0.56 91.58 ±0.08 nchlt-ner — 72.19 ±0.13 71.44 ±0.07 73.22 ±0.2 69.25 ±0.25 65.67 ±0.07 65.86 ±0.16 73.81 ±0.18 yoruba-twi-ner — 57.40 ±2.51 75.35 ±0.78 68.02 ±2.01 82.40 ±0.04 65.6 ±2.87 62.45 ±1.04 79.68 ±1.42 wikiann — 84.82 ±0.24 84.68 ±0.85 87.00 ±1.12 84.58 ±0.46 84.21 ±0.12 85.64 ±0.36 86.91 ±0.31
Phrase Chunking phrase-chunk — 90.41 ±0.10 89.62 ±0.24 91.54 ±0.24 89.47 ±0.22 91.99 ±0.02 91.70 ±0.27 92.01 ±0.18 POS igbo-pos — 85.40 ±0.04 85.31 ±0.16 85.23 ±0.26 85.35 ±0.07 85.39 ±0.14 85.54 ±0.12 85.36 ±0.18
News
amharic-news — 85.83 ±0.56 60.83 ±0.91 85.97 ±0.34 87.03 ±0.35 86.37 ±0.42 86.13 ±0.20 86.84 ±0.32 kinnews — 76.5 ±0.91 77.98 ±0.41 79.15 ±0.57 78.21 ±0.41 80.09 ±0.68 79.54 ±1.00 79.32 ±1.49 kirnews — 53.77 ±2.54 66.87 ±1.48 86.77 ±1.49 86.72 ±0.21 73.63 ±6.66 83.18 ±1.3 85.39 ±2.73 swahili-news-v0.2 — 88.43 ±0.31 85.28 ±0.21 88.89 ±0.58 88.76 ±0.82 88.09 ±1.02 86.97 ±1.31 89.29 ±0.74
Sentiment Analysis bambara-v2 — 46.22 ±1.94 65.00 ±2.00 62.81 ±1.35 60.19 ±1.61 60.50 ±0.94 63.90 ±3.5 63.17 ±0.51 pidgin-tweet — 69.99 ±0.41 69.00 ±0.44 71.41 ±0.16 69.47 ±0.84 69.98 ±0.35 69.64 ±0.23 68.27 ±1.11 yosm — 81.18 ±1.63 83.99 ±0.49 85.50 ±0.87 87.47 ±0.53 85.33 ±0.76 83.00 ±1.32 84.83 ±2.93 Topic hausa-topic — 84.75 ±1.88 83.48 ±1.52 87.83 ±0.53 88.41 ±0.49 87.50 ±0.11 88.21 ±0.61 89.52 ±1.11 yoruba-topic — 64.37 ±3.15 82.81 ±1.56 86.60 ±1.21 85.74 ±2.23 78.11 ±4.55 73.07 ±3.38 83.58 ±1.68
AfroNLU Score 77.77 79.54 82.96 80.92 80.03 80.43 83.04
Table E.1: Performance of models on seven AfroNLU benchmark DEV datasets. (F1) score is the evaluation metric. In QA task, we train the models on English squad TRAIN and DEV datasets. We exclude the QA from AfroNLU DEV datasets. We use a dash (-) for tasks without a known SOTA.
Category Sentence Gold Prediction
Ambivalence Markers Kò burú s ˙ ùgbò ˙ n ó ti pé ˙ jù positive negative
Sinimá tì a lè pè nì ìràwó ˙ sinimá tì ò ǹ ko
˙ mó ˙ nà mó ˙ nà s ˙ ùgbò ˙ n n tì kò nì ohun ámúye ˙ ni. negative positive
Negation Eré síse naa ko dára to, ìtàn naa kò yeni, ní èrò tèmi òs
˙ èré tó daa jù ni ìyá náà negative positive
S ˙ e oun tó o fé ˙ . negative positive
Not seen in training Wo ˙ n rí sinima yìí s ˙ e, àgbó ˙ dò ˙ wò ni positive negative
Irú yádi fíímù. Mo kórìrá gbogbo dídágbé mi nìkan kejì tì o. Ìdo
˙ tí ńlá! negative positive
Polarity item can be either Ìkìlò ˙ . O ní láti wo ìparí eré yìí nítorí wípé ńkan positive or negative s ˙ e ˙ lè ˙
ní ìparí eré náà. positive negative Nìkan ní ìpò àwàdà Nollywood gbòòrò. S
˙ é ó ní
ìdánílójú nítòótó ˙ . negative positive
Table E.2: Error analysis of Yoruba Sentiment analysis dataset. The polarity items are highlighted in red.
27
a) Kinnews b) Kirnews
Figure E.1: Confusion matrices showing the performance of SERENGETI for each categories in Kirnews and Kinnews classification datasets. The categories are (1) politics, (2) sports, (3) economy, (4) health, (5) entertainment, (6) history, (7) technology, (8) tourism, (9) culture, (10) fashion (11) religion, (12) environment, (13) education, and (14) relationship. Kirnews does not have Class 8 and 10.
a) Bambara Sentiment Analysis b) Pidgin Tweets c) YOSM Sentiment
Figure E.2: Confusion matrices showing the performance of SERENGETI for each category in Bambara, Pidgin tweets, and YOSM datasets.
a) Hausa Topic Classification. b) Yoruba Topic Classification.
Figure E.3: Confusion matrices showing the performance of SERENGETI for each categories in Hausa and Yoruba topic classification datasets. A="Africa", E="Entertainment", H="Health", N="Nigeria", P="Politics", S="Sport", W="World"
28
although we find a few exceptions. We provide confusion matrices that represents the sizes of each category and the performance of SERENGETI in Figures E.4, E.5, and E.6.
29
a) Kinnews b) Kirnews
Figure E.4: Confusion matrices showing the performance of SERENGETI for each categories in Kirnews and Kinnews classification datasets.
a) Bambara Sentiment Analysis b) Pidgin Tweets c) YOSM Sentiment
Figure E.5: Confusion matrices showing the performance of SERENGETI for each category in Bambara, Pidgin tweets, and YOSM datasets.
a) Hausa Topic Classification. b) Yoruba Topic Classification.
Figure E.6: Confusion matrices showing the performance of SERENGETI for each categories in Hausa and Yoruba topic classification datasets. A="Africa", E="Entertainment", H="Health", N="Nigeria", P="Politics", S="Sport", W="World"
30
ISO-639-3 SERENGETI AfroLID Franc ISO-639-3 SERENGETI AfroLID Franc ISO-639-3 SERENGETI AfroLID Franc aar 100.00 96.00 74.00 kde 99.00 95.00 60.00 pov 98.00 93.00 82.00 ada 100.00 100.00 98.00 kdh 100.00 99.00 95.00 run 97.00 91.00 68.00 afr 100.00 97.00 81.00 kea 98.00 96.07 0.00 sag 100.00 100.00 30.00
amh 99.00 97.00 36.00 kin 94.00 89.00 47.00 shk 100.00 100.00 93.00 bam 92.00 70.00 30.00 kmb 98.00 94.00 71.00 sna 98.00 97.00 91.00 bba 100.00 100.00 83.00 kng 99.00 98.00 58.00 som 98.00 95.00 89.00 bci 97.00 98.00 92.00 koo 96.00 96.00 96.00 sot 92.00 88.00 93.00 bem 98.00 94.00 90.00 kqn 99.00 98.00 84.00 ssw 92.00 86.00 68.00 bfa 100.00 99.00 91.00 kqs 99.00 95.00 73.00 suk 100.00 99.00 34.00 bin 100.00 99.00 97.00 ktu 98.00 93.00 19.00 sus 99.00 99.00 96.00 bum 98.00 97.00 72.00 lia 98.00 97.00 100.00 swh 95.00 77.00 70.00 cjk 98.00 96.00 56.00 lin 98.00 99.00 98.00 tem 99.00 99.00 88.00 crs 97.00 96.00 83.00 lot 100.00 99.00 93.00 tir 100.00 99.00 97.00 dag 100.00 100.00 100.00 loz 100.00 95.00 92.00 tiv 100.00 100.00 99.00 dga 98.00 100.00 78.00 lua 98.00 99.00 87.00 toi 98.00 98.00 80.00 dip 98.00 93.00 86.00 lue 98.00 95.00 68.00 tsn 81.00 76.00 33.00 dyu 95.00 96.00 0.00 lug 96.00 87.00 64.00 tso 97.00 99.00 94.00 ewe 93.00 97.00 97.00 lun 97.00 97.00 86.00 twi 100.00 100.00 87.00 fat 98.00 98.00 94.00 men 98.00 98.00 99.00 umb 100.00 99.00 76.00 fon 98.00 97.00 92.00 mfq 92.00 95.00 88.00 vai 100.00 100.00 100.00 fuf 96.00 93.00 52.00 mos 99.00 97.00 90.00 ven 98.00 95.00 85.00 fuv 95.00 94.00 61.00 nba 100.00 99.00 61.00 vmw 98.00 97.00 95.00 gaa 98.00 95.00 97.00 nbl 79.00 74.00 47.00 wol 87.00 81.00 21.00 gaz 94.00 94.00 96.00 ndo 97.00 96.00 76.00 xho 75.00 67.00 30.00 gjn 100.00 98.00 99.00 nso 89.00 83.00 59.00 xsm 99.00 99.00 53.00 gkp 68.00 63.00 69.00 nya 99.00 92.00 75.00 yor 99.00 98.00 66.00 hau 95.00 88.00 77.00 nym 98.00 99.00 54.00 zdj 98.00 96.00 63.00 ibb 99.00 98.00 84.00 nyn 95.00 92.00 92.00 zul 68.00 50.00 40.00 ibo 97.00 97.00 88.00 nzi 100.00 97.00 98.00 kbp 100.00 100.00 98.00 pcm 96.00 96.00 82.00
SERENGETI Average f1_score: 96.29 AfroLID Average f1_score: 91.63 Franc Average: f1_score 74.81
Table E.3: F1-scores for SERENGETI, AfroLID, and Franc on AfroLID’s dataset for 88 languages.
31

heading: F Detailed Geneaology and Language Contact Analysis
text: In this Section, we use Figures and Tables to provide evidence for the influence of similar languages in zero-shot settings. First, we highlight in purple the similar languages that we perform genealogy analysis on in Figure E.7. In the figure, the languages with mutual intelligibility are presented in similar coloured circles. To determine the significance of language similarity and language contact in our own zero-shot settings, we measure the Jaccard similarity between the pretraining data for the South African languages in AfroNLU (see Table 8). To calculate the Jaccard similarities, we removed digits, emojis, and punctuation marks. We do this to ensure that we reduce interference with the similarity scores. We find strong similarities between some of these languages as in the bolded examples in Table 8.
We find that although XLM-R, mBERT, and AfriBERTa are not trained on most most of these languages, we record high scores in zero-shot settings see Table E.4). We argue that XLM-R in addition to cross-lingual transfers from other languages acquires representation from afr and xho where xho alone shares more than 0.4 similarity with afr, nbl, nso, and zul. mBERT also learns representation from afr while AfriBERTa learns representations from Gahuza which is a code-mixed variety of kin and run. SERENGETI however, outperforms other models on these datasets indicating that learning the representation of each language improves performance.
Next, we finetune a BERT model and compare the performance of BERT with MBERT. We do this because BERT is a monolingual model and does not include any similar language in its representation. In Table 9, BERT significantly performs lower than MBERT in all languages in NCHLTNER. BERT also has lower performance on the phrase-chunk dataset in all languages except on ssw, and ven.
This analysis is far from being conclusive and future work can further probe the influence of similar languages in more detail. This is necessary to evaluate to what extent similar languages have an influence on performance in zero-shot settings and why in zero shot settings, some monolingual models outperform multilingual ones. For example, in the case of ssw and ven.
32
Figure E.7: A genetic classification of Niger-Congo languages in AfroNLU. We highlight in purple the list of languages relevant to our geneaology and language contact analysis. Languages which share stronger mutual intelligibility is represented in similar colours.
33
Cluster Dataset Lang. XLMR mBERT Afro-XLMR AfriBERTa SERENGETI N am ed E nt ity R ec og ni tio n (N E R ) masakaner-v1 amh 73.98±0.64 0.0 ±0.0 77.38±0.47 69.61±0.76 74.26±0.54 hau 91.39±0.24 88.25 ±0.42 91.92±0.86 91.12±0.37 92.03±0.59 ibo 84.55 ±0.15 84.44 ±0.97 87.51±0.92 87.95±0.54 87.82±0.63 kin 73.54 ±0.35 71.02 ±1.34 78.46±0.34 75.07 ±0.51 78.56±0.34 lug 78.65 ±1.25 79.07 ±2.01 82.11 ±0.99 77.84 ±0.4 84.61±0.4 luo 74.28 ±1.87 74.47 ±0.08 75.20 ±1.23 70.76 ±1.57 77.28±1.61 pcm 88.89 ±0.56 88.88 ±0.91 90.07±0.18 87.65±0.43 89.65±0.63 swa 87.68±0.98 86.12±0.5 87.77±0.1 87.72±0.13 88.08±0.13 wol 63.4 ±0.68 64.25 ±1.66 68.09 ±1.65 60.9 ±1.69 66.26±1.47 yor 78.97 ±0.93 79.45±0.36 83.76±0.34 79.89±0.89 83.08±1.18 masakaner-v2 bam 80.66 ±0.99 79.2 ±1.43 81.04 ±0.31 78.55 ±0.42 82.11±0.53 bbj 72.82 ±1.07 62.44 ±0.59 73.31 ±0.74 71.97 ±1.61 73.66±0.87 ewe 88.54 ±0.23 84.19 ±1.12 89.58 ±0.54 86.97 ±0.4 89.75±0.14 fon 82.34 ±0.09 77.87 ±0.47 82.62 ±0.73 78.66 ±0.39 82.86±0.53 hau 86.09±0.61 82.66 ±1.46 87.29±0.67 86.14±0.38 87.33±0.62 ibo 89.67 ±0.28 84.04 ±1.09 91.99±0.11 91.56±0.36 92.28±0.21 kin 84.04 ±0.48 83.53 ±0.81 86.51±0.3 83.22 ±0.25 86.38±0.35 lug 86.18 ±0.22 85.78 ±1.41 88.17 ±0.56 85.32 ±0.49 89.24±0.37 mos 74.55 ±0.65 67.75 ±1.84 75.25 ±0.71 69.95±0.89 73.74±1.62 nya 90.23 ±0.14 88.6 ±0.65 91.84±0.23 88.83 ±0.11 91.29±0.19 pcm 89.11 ±0.1 87.90 ±1.0 89.27±0.4 87.81±0.45 88.77±0.37 sna 94.15 ±0.19 93.06 ±0.75 95.35±0.16 93.51 ±0.32 95.92±0.2 swa 92.37±0.05 91.09±0.33 93.06±0.14 92.43±0.11 92.87±0.33 tsn 85.69 ±0.89 85.02 ±0.85 88.24 ±0.26 83.58 ±0.79 88.43±0.1 twi 79.60 ±1.45 78.05 ±2.3 79.94 ±1.6 75.35 ±0.81 80.25±1.1 wol 85.14 ±0.34 83.65 ±1.11 84.60 ±0.4 81.68 ±0.38 85.97±0.43
xho 87.6±0.15 86.24 ±1.2 89.59±0.37 86.18 ±0.17 88.76±0.76 yor 86.56 ±0.36 83.45±1.63 88.91±0.27 87.45±0.17 87.99±0.61 zul 86.32 ±0.6 84.16 ±1.75 89.75±0.16 84.9 ±0.27 90.41±0.24
nchlt-ner
afr 80.68±0.75 80.08±0.29 80.55±0.11 74.5 ±0.64 81.57±0.59 nbl 74.64 ±0.66 73.48 ±0.18 75.26±0.28 72.28 ±0.67 77.13±0.67 nso 77.0 ±1.23 78.75 ±0.45 80.13±0.51 75.45 ±1.09 80.69±0.64 sot 54.71 ±1.51 54.68 ±0.49 55.57±0.2 54.09 ±0.98 56.26±1.52 ssw 71.75 ±0.65 71.24 ±0.75 72.35±1.02 69.38 ±0.58 73.37±0.82 tsn 77.02 ±0.22 76.35 ±0.47 77.68±0.96 73.89 ±1.41 79.05±0.75 tso 74.24 ±0.08 72.95 ±0.67 74.85 ±0.43 71.05 ±0.9 75.13±0.31 ven 64.06 ±0.31 63.11 ±1.27 64.39 ±0.36 63.24 ±1.26 65.42±0.76 xho 70.77±2.45 68.54 ±1.44 72.37±0.39 67.00 ±1.27 72.92±0.29 zul 69.44 ±0.62 67.74 ±1.46 70.28±0.49 67.17 ±0.15 71.20±0.44
Wikiann
amh 57.76±0.45 33.96 ±1.83 64.27±1.91 60.16±2.83 68.11±1.75 ibo 73.6 ±1.32 70.83 ±1.86 73.93±1.12 76.14±1.42 75.73±2.78 kin 69.67 ±2.07 77.35 ±4.47 82.24±2.17 79.8 ±1.06 79.78±1.78 swh 88.09±0.32 88.00±0.28 88.83±0.47 86.13±0.2 89.16±0.35 yor 83.8 ±2.06 81.96±0.88 87.96±1.24 82.77±0.23 85.00±2.42
Ph ra
se C
hu nk
in g
phrase-chunk
afr 95.34±0.16 95.68±0.30 95.13±0.06 90.22 ±0.81 96.01±0.14 nso 96.57 ±0.61 96.85 ±0.55 98.36±0.2 96.47 ±0.14 98.28±0.1 sot 82.93 ±0.38 83.08 ±0.78 85.28±0.61 82.18 ±0.93 85.69±0.76 ssw 82.9 ±1.03 81.91 ±0.47 84.73±0.18 83.24 ±0.11 83.45±0.12 tsn 92.77 ±0.16 92.64 ±0.66 94.11±0.49 92.71 ±0.42 94.03±0.19 tso 86.42 ±0.46 86.90 ±0.31 87.39 ±0.18 86.73 ±0.95 89.32±0.43 ven 92.31 ±0.45 90.47 ±0.32 92.42 ±0.68 92.02 ±0.33 92.54±0.21 zul 87.30 ±0.26 87.29 ±1.04 88.67±0.66 85.74 ±0.55 90.05±0.81
Table E.4: Performance of mPLMs on each language in each task. (F1) score is the evaluation metric. We use Red highlights to indicate languages in zero-shot setting.
34
ISO-639-3 Language ISO-639-3 Language ISO-639-3 Language ISO-639-3 Language
aar Afar / Qafar bky Bokyi dow Doyayo gol Gola aba Abe / Abbey bmo Bambalang dsh Daasanach gqr Gor abn Abua bmv Bum dua Douala gso Gbaya, Southwest acd Gikyode bom Berom dug Chiduruma gud Dida, Yocoboue ach Acholi bov Tuwuli dwr Dawro gur Farefare ada Dangme box Bwamu / Buamu dyi Sénoufo, Djimini guw Gun adh Jopadhola / Adhola bqc Boko dyu Jula gux Gourmanchema adj Adjukru / Adioukrou bqj Bandial ebr Ebrie guz Ekegusii afr Afrikaans bsc Oniyan ebu Kiembu / Embu gvl Gulay agq Aghem bsp Baga Sitemu efi Efik gwr Gwere aha Ahanta bss Akoose ego Eggon gya Gbaya, Northwest ajg Aja bst Basketo eka Ekajuk hag Hanga akp Siwu bud Ntcham eko Koti har Harari alz Alur bum Bulu eto Eton hau Hausa amh Amharic bun Sherbro etu Ejagham hay Haya ann Obolo bus Bokobaru etx Iten / Eten hbb Nya huba anu Anyuak / Anuak buy Bullom So ewe Ewe heh Hehe anv Denya bwr Bura Pabir ewo Ewondo her Herero asa Asu bwu Buli fak Fang hgm Haillom asg Cishingini bxk Bukusu fat Fante hna Mina atg Ivbie North-Okpela-Arhe byf Bete ffm Fulfulde, Maasina ibb Ibibio ati Attie byv Medumba fia Nobiin ibo Igbo avn Avatime bza Bandi fip Fipa idu Idoma avu Avokaya bzw Basa flr Fuliiru igb Ebira azo Awing cce Chopi fon Fon ige Igede bam Bambara chw Chuabo fub Fulfulde, Adamawa igl Igala bav Vengo cjk Chokwe fue Fulfulde, Borgu ijn Kalabari bba Baatonum cko Anufo fuf Pular ikk Ika bbj Ghomala cme Cerma fuh Fulfulde, Western Niger ikw Ikwere bbk Babanki cop Coptic ful Fulah iqw Ikwo bci Baoule cou Wamey fuq Fulfulde Central Eastern Niger iri Rigwe bcn Bali crs Seychelles Creole fuv Fulfude Nigeria ish Esan bcw Bana csk Jola Kasa gaa Ga iso Isoko bcy Bacama cwe Kwere gax Oromo, Borana-Arsi-Guji iyx yaka bdh Baka daa Dangaleat gaz Oromo, West Central izr Izere bds Burunge dag Dagbani gbo Grebo, Northern izz Izii bem Bemba / Chibemba dav Dawida / Taita gbr Gbagyi jgo Ngomba beq Beembe dga Dagaare gde Gude jib Jibu ber Berber dgd Dagaari Dioula gid Gidar jit Jita bex Jur Modo dgi Dagara, Northern giz South Giziga jmc Machame bez Bena dhm Dhimba gjn Gonja kab Kabyle bfa Bari dib Dinka, South Central gkn Gokana kam Kikamba bfd Bafut did Didinga gkp Kpelle, Guinea kbn Kare bfo Birifor, Malba dig Chidigo gmv Gamo kbo Keliko bib Bisa dik Dinka, Southwestern gna Kaansa kbp Kabiye bim Bimoba dip Dinka, Northeastern gnd Zulgo-gemzek kby Kanuri, Manga bin Edo diu Gciriku gng Ngangam kcg Tyap biv Birifor, Southern dks Dinka, Southeastern gof Goofa kck Kalanga bjv Bedjond dnj Dan gog Gogo kdc Kutu
Table F.1: Languages covered in SERENGETI - Part I.
35
ISO-639-3 Language ISO-639-3 Language ISO-639-3 Language ISO-639-3 Language kde Makonde laj Lango mfh Matal ngb Ngbandi, Northern kdh Tem lam Lamba mfi Wandala ngc Ngombe kdi Kumam lap Laka mfk Mofu, North ngl Lomwe kdj Ng’akarimojong lee Lyélé mfq Moba ngn Bassa kdl Tsikimba lef Lelemi mfz Mabaan ngo Ngoni kdn Kunda lem Nomaande mgc Morokodo ngp Ngulu kea Kabuverdianu lgg Lugbara mgh Makhuwa-Meetto nhr Naro ken Kenyang lgm Lega-mwenga mgo Meta’ nhu Noone khy Kele / Lokele lia Limba, West-Central mgq Malila nih Nyiha kia Kim lik Lika mgr Mambwe-Lungu nim Nilamba / kinilyamba kik Gikuyu / Kikuyu lin Lingala mgw Matumbi nin Ninzo kin Kinyarwanda lip Sekpele mif Mofu-Gudur niy Ngiti kiz Kisi lmd Lumun mkl Mokole nka Nkoya / ShiNkoya kki Kagulu lmp Limbum mlg Malagasy nko Nkonya kkj Kako lnl Banda, South Central mlr Vame nla Ngombale kln Kalenjin log Logo mmy Migaama nnb Nande / Ndandi klu Klao lom Loma mnf Mundani nnh Ngiemboon kma Konni loq Lobala mnk Mandinka nnq Ngindo kmb Kimbundu lot Latuka moa Mwan nse Chinsenga kmy Koma loz Silozi mos Moore nnw Nuni, Southern knf Mankanya lro Laro moy Shekkacho nso Sepedi kng Kongo lsm Saamya-Gwe / Saamia moz Mukulu ntr Delo knk Kuranko lth Thur / Acholi-Labwor mpe Majang nuj Nyole kno Kono lto Tsotso mpg Marba nus Nuer koo Konzo lua Tshiluba mqb Mbuko nwb Nyabwa koq Kota luc Aringa msc Maninka, Sankaran nxd Ngando kqn Kikaonde lue Luvale mur Murle nya Chichewa kqp Kimré lug Luganda muy Muyang nyb Nyangbo kqs Kisi lun Lunda mwe Mwera nyd Olunyole / Nyore kqy Koorete luo Dholuo / Luo mwm Sar nyf Giryama kri Krio lwg Wanga mwn Cinamwanga nyk Nyaneka krs Gbaya lwo Luwo mws Mwimbi-Muthambi nym Nyamwezi krw Krahn, Western maf Mafa myb Mbay nyn Nyankore / Nyankole krx Karon mas Maasai myk Sénoufo, Mamara nyo Nyoro ksb Shambala / Kishambala maw Mampruli myx Masaaba nyu Nyungwe ksf Bafia mbu Mbula-Bwazza mzm Mumuye nyy Nyakyusa-Ngonde / Kyangonde ksp Kabba mck Mbunda mzw Deg nza Mbembe, Tigon ktj Krumen, Plapo mcn Masana / Massana naq Khoekhoe nzi Nzema ktu Kikongo mcp Makaa naw Nawuri odu Odual kua Oshiwambo mcu Mambila, Cameroon nba Nyemba ogo Khana kub Kutep mda Mada nbl IsiNdebele oke Okpe kuj Kuria mdm Mayogo ncu Chunburung okr Kirike kus Kusaal mdy Maale ndc Ndau oku Oku kvj Psikye men Mende nde IsiNdebele orm Oromo kwn Kwangali meq Merey ndh Ndali ozm Koonzime kyf Kouya mer Kimiiru ndj Ndamba pcm Nigerian Pidgin kyq Kenga mev Maan / Mann ndo Ndonga pem Kipende kzr Karang mfe Morisyen / Mauritian Creole ndv Ndut pkb Kipfokomo / Pokomo lai Lambya mfg Mogofin ndz Ndogo
Table F.2: Languages covered in SERENGETI - Part II
36
ISO-639-3 Language ISO-639-3 Language ISO-639-3 Language
pov Guinea-Bissau Creole tcd Tafi won Wongo poy Pogolo / Shipogoro-Pogolo ted Krumen, Tepo xan Xamtanga rag Lulogooli tem Timne xed Hdi rel Rendille teo Teso xho Isixhosa rif Tarifit tex Tennet xnz Mattokki rim Nyaturu tgw Senoufo, Tagwana xog Soga rnd Uruund thk Tharaka xon Konkomba rng Ronga / ShiRonga thv Tamahaq, Tahaggart xpe Kpelle rub Gungu tir Tigrinya xrb Karaboro, Eastern run Rundi / Kirundi tiv Tiv xsm Kasem rwk Rwa tke Takwane xtc Katcha-Kadugli-Miri sag Sango tlj Talinga-Bwisi xuo Kuo saq Samburu tll Otetela yal Yalunka sba Ngambay tog Tonga yam Yamba sbd Samo, Southern toh Gitonga yao Yao / Chiyao sbp Sangu toi Chitonga yat Yambeta sbs Kuhane tpm Tampulma yba Yala sby Soli tsc Tshwa ybb Yemba sef Sénoufo, Cebaara tsn Setswana yom Ibinda ses Songhay, Koyraboro Senni tso Tsonga yor Yoruba sev Sénoufo, Nyarafolo tsw Tsishingini yre Yaoure sfw Sehwi ttj Toro / Rutoro zaj Zaramo sgw Sebat Bet Gurage ttq Tawallammat zdj Comorian, Ngazidja shi Tachelhit ttr Nyimatli zga Kinga shj Shatt tui Toupouri ziw Zigula shk Shilluk tul Kutule zne Zande / paZande sid Sidama tum Chitumbuka zul Isizulu sig Paasaal tuv Turkana sil Sisaala, Tumulung tvu Tunen sna Shona twi Twi snf Noon umb Umbundu sng Sanga / Kiluba urh Urhobo snw Selee uth ut-Hun som Somali vag Vagla sop Kisonge vai Vai sor Somrai ven Tshivenda sot Sesotho vid Chividunda soy Miyobe vif Vili spp Senoufo, Supyire vmk Makhuwa-Shirima ssw Siswati vmw Macua suk Sukuma vun Kivunjo sus Sosoxui vut Vute swa Swahili wal Wolaytta swc Swahili Congo wbi Vwanji swh Swahili wec Guere swk Sena, Malawi wes Pidgin, Cameroon sxb Suba wib Toussian, Southern taq Tamasheq wmw Mwani tcc Datooga wol Wolof
Table F.3: Languages covered in SERENGETI - Part III.
37
ISO-639-3 #Tokens swh 2,912,488,735 afr 1,264,478,436 som 587,549,878 swa 499,792,448 hau 286,806,539 amh 241,700,000 mlg 137,852,716 zne 89,981,183 sna 75,413,519 ... ... bam 3,262 har 3,066 dyo 1,797 fvr 1,680 tbz 1,578 ddn 1,372 fuc 1,336 knc 1,097 eot 1,041 cgg 845
Table F.4: The sizes of the top 10 and bottom 10 languages in SERENGETI pretraining.
38
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
8
3 A2. Did you discuss any potential risks of your work? 8

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: 1 and 7
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: 4 and 5

heading: 3 B1. Did you cite the creators of artifacts you used?
text: 2, 4 and 5

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: 2, 4 and 5

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 9
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We use only publicly available data to develop our models. Our data comes from 517 languages and language varieties and hence it is challenging to carry out manual investigation on it. However, since the data belong to the public domain, we do not have serious concerns about privacy or anti-social language beyond what already exists online and is accessible to anyone.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? 2, 3, 4, 5, 6
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 2, 3, 4, 5, 6
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments?

heading: 5 and 6

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 5

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4, 5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 2, 3, 4, 5, 6, Appendix

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 2, 4, 5, 6, Appendix
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 7
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: This analysis is far from being conclusive and future work can further probe the influence of similar languages in more detail. This is necessary to evaluate to what extent similar languages have an influence on performance in zero-shot settings and why in zero shot settings, some monolingual models outperform multilingual ones. For example, in the case of ssw and ven. 32
Figure E. 7: A genetic classification of Niger-Congo languages in AfroNLU. We highlight in purple the list of languages relevant to our geneaology and language contact analysis. Languages which share stronger mutual intelligibility is represented in similar colours. 33
Cluster Dataset Lang. XLMR mBERT Afro-XLMR AfriBERTa SERENGETI N am ed E nt ity R ec og ni tio n (N E R ) masakaner-v1 amh 73. 98±0. 64 0. 0 ±0. 0 77. 38±0. 47 69. 61±0. 76 74. 26±0. 54 hau 91. 39±0. 24 88. 25 ±0. 42 91. 92±0. 86 91. 12±0. 37 92. 03±0. 59 ibo 84. 55 ±0. 15 84. 44 ±0. 97 87. 51±0. 92 87. 95±0. 54 87. 82±0. 63 kin 73. 54 ±0. 35 71. 02 ±1. 34 78. 46±0. 34 75. 07 ±0. 51 78. 56±0. 34 lug 78. 65 ±1. 25 79. 07 ±2. 01 82. 11 ±0. 99 77. 84 ±0. 4 84. 61±0. 4 luo 74. 28 ±1. 87 74. 47 ±0. 08 75. 20 ±1. 23 70. 76 ±1. 57 77. 28±1. 61 pcm 88. 89 ±0. 56 88. 88 ±0. 91 90. 07±0. 18 87. 65±0. 43 89. 65±0. 63 swa 87. 68±0. 98 86. 12±0. 5 87. 77±0. 1 87. 72±0. 13 88. 08±0. 13 wol 63. 4 ±0. 68 64. 25 ±1. 66 68. 09 ±1. 65 60. 9 ±1. 69 66. 26±1. 47 yor 78. 97 ±0. 93 79. 45±0. 36 83. 76±0. 34 79. 89±0. 89 83. 08±1. 18 masakaner-v2 bam 80. 66 ±0. 99 79. 2 ±1. 43 81. 04 ±0. 31 78. 55 ±0. 42 82. 11±0. 53 bbj 72. 82 ±1. 07 62. 44 ±0. 59 73. 31 ±0. 74 71. 97 ±1. 61 73. 66±0. 87 ewe 88. 54 ±0. 23 84. 19 ±1. 12 89. 58 ±0. 54 86. 97 ±0. 4 89. 75±0. 14 fon 82. 34 ±0. 09 77. 87 ±0. 47 82. 62 ±0. 73 78. 66 ±0. 39 82. 86±0. 53 hau 86. 09±0. 61 82. 66 ±1. 46 87. 29±0. 67 86. 14±0. 38 87. 33±0. 62 ibo 89. 67 ±0. 28 84. 04 ±1. 09 91. 99±0. 11 91. 56±0. 36 92. 28±0. 21 kin 84. 04 ±0. 48 83. 53 ±0. 81 86. 51±0. 3 83. 22 ±0. 25 86. 38±0. 35 lug 86. 18 ±0. 22 85. 78 ±1. 41 88. 17 ±0. 56 85. 32 ±0. 49 89. 24±0. 37 mos 74. 55 ±0. 65 67. 75 ±1. 84 75. 25 ±0. 71 69. 95±0. 89 73. 74±1. 62 nya 90. 23 ±0. 14 88. 6 ±0. 65 91. 84±0. 23 88. 83 ±0. 11 91. 29±0. 19 pcm 89. 11 ±0. 1 87. 90 ±1. 0 89. 27±0. 4 87. 81±0. 45 88. 77±0. 37 sna 94. 15 ±0. 19 93. 06 ±0. 75 95. 35±0. 16 93. 51 ±0. 32 95. 92±0. 2 swa 92. 37±0. 05 91. 09±0. 33 93. 06±0. 14 92. 43±0. 11 92. 87±0. 33 tsn 85. 69 ±0. 89 85. 02 ±0. 85 88. 24 ±0. 26 83. 58 ±0. 79 88. 43±0. 1 twi 79. 60 ±1. 45 78. 05 ±2. 3 79. 94 ±1. 6 75. 35 ±0. 81 80. 25±1. 1 wol 85. 14 ±0. 34 83. 65 ±1. 11 84. 60 ±0. 4 81. 68 ±0. 38 85. 97±0. 43
xho 87. 6±0. 15 86. 24 ±1. 2 89. 59±0. 37 86. 18 ±0. 17 88. 76±0. 76 yor 86. 56 ±0. 36 83. 45±1. 63 88. 91±0. 27 87. 45±0. 17 87. 99±0. 61 zul 86. 32 ±0. 6 84. 16 ±1. 75 89. 75±0. 16 84. 9 ±0. 27 90. 41±0. 24
nchlt-ner
afr 80. 68±0. 75 80. 08±0. 29 80. 55±0. 11 74. 5 ±0. 64 81. 57±0. 59 nbl 74. 64 ±0. 66 73. 48 ±0. 18 75. 26±0. 28 72. 28 ±0. 67 77. 13±0. 67 nso 77. 0 ±1. 23 78. 75 ±0. 45 80. 13±0. 51 75. 45 ±1. 09 80. 69±0. 64 sot 54. 71 ±1. 51 54. 68 ±0. 49 55. 57±0. 2 54. 09 ±0. 98 56. 26±1. 52 ssw 71. 75 ±0. 65 71. 24 ±0. 75 72. 35±1. 02 69. 38 ±0. 58 73. 37±0. 82 tsn 77. 02 ±0. 22 76. 35 ±0. 47 77. 68±0. 96 73. 89 ±1. 41 79. 05±0. 75 tso 74. 24 ±0. 08 72. 95 ±0. 67 74. 85 ±0. 43 71. 05 ±0. 9 75. 13±0. 31 ven 64. 06 ±0. 31 63. 11 ±1. 27 64. 39 ±0. 36 63. 24 ±1. 26 65. 42±0. 76 xho 70. 77±2. 45 68. 54 ±1. 44 72. 37±0. 39 67. 00 ±1. 27 72. 92±0. 29 zul 69. 44 ±0. 62 67. 74 ±1. 46 70. 28±0. 49 67. 17 ±0. 15 71. 20±0. 44
Wikiann
amh 57. 76±0. 45 33. 96 ±1. 83 64. 27±1. 91 60. 16±2. 83 68. 11±1. 75 ibo 73. 6 ±1. 32 70. 83 ±1. 86 73. 93±1. 12 76. 14±1. 42 75. 73±2. 78 kin 69. 67 ±2. 07 77. 35 ±4. 47 82. 24±2. 17 79. 8 ±1. 06 79. 78±1. 78 swh 88. 09±0. 32 88. 00±0. 28 88. 83±0. 47 86. 13±0. 2 89. 16±0. 35 yor 83. 8 ±2. 06 81. 96±0. 88 87. 96±1. 24 82. 77±0. 23 85. 00±2. 42
Ph ra
se C
hu nk
in g
phrase-chunk
afr 95. 34±0. 16 95. 68±0. 30 95. 13±0. 06 90. 22 ±0. 81 96. 01±0. 14 nso 96. 57 ±0. 61 96. 85 ±0. 55 98. 36±0. 2 96. 47 ±0. 14 98. 28±0. 1 sot 82. 93 ±0. 38 83. 08 ±0. 78 85. 28±0. 61 82. 18 ±0. 93 85. 69±0. 76 ssw 82. 9 ±1. 03 81. 91 ±0. 47 84. 73±0. 18 83. 24 ±0. 11 83. 45±0. 12 tsn 92. 77 ±0. 16 92. 64 ±0. 66 94. 11±0. 49 92. 71 ±0. 42 94. 03±0. 19 tso 86. 42 ±0. 46 86. 90 ±0. 31 87. 39 ±0. 18 86. 73 ±0. 95 89. 32±0. 43 ven 92. 31 ±0. 45 90. 47 ±0. 32 92. 42 ±0. 68 92. 02 ±0. 33 92. 54±0. 21 zul 87. 30 ±0. 26 87. 29 ±1. 04 88. 67±0. 66 85. 74 ±0. 55 90. 05±0. 81
Table E. 4: Performance of mPLMs on each language in each task. (F1) score is the evaluation metric. We use Red highlights to indicate languages in zero-shot setting. 34
ISO-639-3 Language ISO-639-3 Language ISO-639-3 Language ISO-639-3 Language
aar Afar / Qafar bky Bokyi dow Doyayo gol Gola aba Abe / Abbey bmo Bambalang dsh Daasanach gqr Gor abn Abua bmv Bum dua Douala gso Gbaya, Southwest acd Gikyode bom Berom dug Chiduruma gud Dida, Yocoboue ach Acholi bov Tuwuli dwr Dawro gur Farefare ada Dangme box Bwamu / Buamu dyi Sénoufo, Djimini guw Gun adh Jopadhola / Adhola bqc Boko dyu Jula gux Gourmanchema adj Adjukru / Adioukrou bqj Bandial ebr Ebrie guz Ekegusii afr Afrikaans bsc Oniyan ebu Kiembu / Embu gvl Gulay agq Aghem bsp Baga Sitemu efi Efik gwr Gwere aha Ahanta bss Akoose ego Eggon gya Gbaya, Northwest ajg Aja bst Basketo eka Ekajuk hag Hanga akp Siwu bud Ntcham eko Koti har Harari alz Alur bum Bulu eto Eton hau Hausa amh Amharic bun Sherbro etu Ejagham hay Haya ann Obolo bus Bokobaru etx Iten / Eten hbb Nya huba anu Anyuak / Anuak buy Bullom So ewe Ewe heh Hehe anv Denya bwr Bura Pabir ewo Ewondo her Herero asa Asu bwu Buli fak Fang hgm Haillom asg Cishingini bxk Bukusu fat Fante hna Mina atg Ivbie North-Okpela-Arhe byf Bete ffm Fulfulde, Maasina ibb Ibibio ati Attie byv Medumba fia Nobiin ibo Igbo avn Avatime bza Bandi fip Fipa idu Idoma avu Avokaya bzw Basa flr Fuliiru igb Ebira azo Awing cce Chopi fon Fon ige Igede bam Bambara chw Chuabo fub Fulfulde, Adamawa igl Igala bav Vengo cjk Chokwe fue Fulfulde, Borgu ijn Kalabari bba Baatonum cko Anufo fuf Pular ikk Ika bbj Ghomala cme Cerma fuh Fulfulde, Western Niger ikw Ikwere bbk Babanki cop Coptic ful Fulah iqw Ikwo bci Baoule cou Wamey fuq Fulfulde Central Eastern Niger iri Rigwe bcn Bali crs Seychelles Creole fuv Fulfude Nigeria ish Esan bcw Bana csk Jola Kasa gaa Ga iso Isoko bcy Bacama cwe Kwere gax Oromo, Borana-Arsi-Guji iyx yaka bdh Baka daa Dangaleat gaz Oromo, West Central izr Izere bds Burunge dag Dagbani gbo Grebo, Northern izz Izii bem Bemba / Chibemba dav Dawida / Taita gbr Gbagyi jgo Ngomba beq Beembe dga Dagaare gde Gude jib Jibu ber Berber dgd Dagaari Dioula gid Gidar jit Jita bex Jur Modo dgi Dagara, Northern giz South Giziga jmc Machame bez Bena dhm Dhimba gjn Gonja kab Kabyle bfa Bari dib Dinka, South Central gkn Gokana kam Kikamba bfd Bafut did Didinga gkp Kpelle, Guinea kbn Kare bfo Birifor, Malba dig Chidigo gmv Gamo kbo Keliko bib Bisa dik Dinka, Southwestern gna Kaansa kbp Kabiye bim Bimoba dip Dinka, Northeastern gnd Zulgo-gemzek kby Kanuri, Manga bin Edo diu Gciriku gng Ngangam kcg Tyap biv Birifor, Southern dks Dinka, Southeastern gof Goofa kck Kalanga bjv Bedjond dnj Dan gog Gogo kdc Kutu
Table F. 1: Languages covered in SERENGETI - Part I. 35
ISO-639-3 Language ISO-639-3 Language ISO-639-3 Language ISO-639-3 Language kde Makonde laj Lango mfh Matal ngb Ngbandi, Northern kdh Tem lam Lamba mfi Wandala ngc Ngombe kdi Kumam lap Laka mfk Mofu, North ngl Lomwe kdj Ng’akarimojong lee Lyélé mfq Moba ngn Bassa kdl Tsikimba lef Lelemi mfz Mabaan ngo Ngoni kdn Kunda lem Nomaande mgc Morokodo ngp Ngulu kea Kabuverdianu lgg Lugbara mgh Makhuwa-Meetto nhr Naro ken Kenyang lgm Lega-mwenga mgo Meta’ nhu Noone khy Kele / Lokele lia Limba, West-Central mgq Malila nih Nyiha kia Kim lik Lika mgr Mambwe-Lungu nim Nilamba / kinilyamba kik Gikuyu / Kikuyu lin Lingala mgw Matumbi nin Ninzo kin Kinyarwanda lip Sekpele mif Mofu-Gudur niy Ngiti kiz Kisi lmd Lumun mkl Mokole nka Nkoya / ShiNkoya kki Kagulu lmp Limbum mlg Malagasy nko Nkonya kkj Kako lnl Banda, South Central mlr Vame nla Ngombale kln Kalenjin log Logo mmy Migaama nnb Nande / Ndandi klu Klao lom Loma mnf Mundani nnh Ngiemboon kma Konni loq Lobala mnk Mandinka nnq Ngindo kmb Kimbundu lot Latuka moa Mwan nse Chinsenga kmy Koma loz Silozi mos Moore nnw Nuni, Southern knf Mankanya lro Laro moy Shekkacho nso Sepedi kng Kongo lsm Saamya-Gwe / Saamia moz Mukulu ntr Delo knk Kuranko lth Thur / Acholi-Labwor mpe Majang nuj Nyole kno Kono lto Tsotso mpg Marba nus Nuer koo Konzo lua Tshiluba mqb Mbuko nwb Nyabwa koq Kota luc Aringa msc Maninka, Sankaran nxd Ngando kqn Kikaonde lue Luvale mur Murle nya Chichewa kqp Kimré lug Luganda muy Muyang nyb Nyangbo kqs Kisi lun Lunda mwe Mwera nyd Olunyole / Nyore kqy Koorete luo Dholuo / Luo mwm Sar nyf Giryama kri Krio lwg Wanga mwn Cinamwanga nyk Nyaneka krs Gbaya lwo Luwo mws Mwimbi-Muthambi nym Nyamwezi krw Krahn, Western maf Mafa myb Mbay nyn Nyankore / Nyankole krx Karon mas Maasai myk Sénoufo, Mamara nyo Nyoro ksb Shambala / Kishambala maw Mampruli myx Masaaba nyu Nyungwe ksf Bafia mbu Mbula-Bwazza mzm Mumuye nyy Nyakyusa-Ngonde / Kyangonde ksp Kabba mck Mbunda mzw Deg nza Mbembe, Tigon ktj Krumen, Plapo mcn Masana / Massana naq Khoekhoe nzi Nzema ktu Kikongo mcp Makaa naw Nawuri odu Odual kua Oshiwambo mcu Mambila, Cameroon nba Nyemba ogo Khana kub Kutep mda Mada nbl IsiNdebele oke Okpe kuj Kuria mdm Mayogo ncu Chunburung okr Kirike kus Kusaal mdy Maale ndc Ndau oku Oku kvj Psikye men Mende nde IsiNdebele orm Oromo kwn Kwangali meq Merey ndh Ndali ozm Koonzime kyf Kouya mer Kimiiru ndj Ndamba pcm Nigerian Pidgin kyq Kenga mev Maan / Mann ndo Ndonga pem Kipende kzr Karang mfe Morisyen / Mauritian Creole ndv Ndut pkb Kipfokomo / Pokomo lai Lambya mfg Mogofin ndz Ndogo
Table F. 2: Languages covered in SERENGETI - Part II
36
ISO-639-3 Language ISO-639-3 Language ISO-639-3 Language
pov Guinea-Bissau Creole tcd Tafi won Wongo poy Pogolo / Shipogoro-Pogolo ted Krumen, Tepo xan Xamtanga rag Lulogooli tem Timne xed Hdi rel Rendille teo Teso xho Isixhosa rif Tarifit tex Tennet xnz Mattokki rim Nyaturu tgw Senoufo, Tagwana xog Soga rnd Uruund thk Tharaka xon Konkomba rng Ronga / ShiRonga thv Tamahaq, Tahaggart xpe Kpelle rub Gungu tir Tigrinya xrb Karaboro, Eastern run Rundi / Kirundi tiv Tiv xsm Kasem rwk Rwa tke Takwane xtc Katcha-Kadugli-Miri sag Sango tlj Talinga-Bwisi xuo Kuo saq Samburu tll Otetela yal Yalunka sba Ngambay tog Tonga yam Yamba sbd Samo, Southern toh Gitonga yao Yao / Chiyao sbp Sangu toi Chitonga yat Yambeta sbs Kuhane tpm Tampulma yba Yala sby Soli tsc Tshwa ybb Yemba sef Sénoufo, Cebaara tsn Setswana yom Ibinda ses Songhay, Koyraboro Senni tso Tsonga yor Yoruba sev Sénoufo, Nyarafolo tsw Tsishingini yre Yaoure sfw Sehwi ttj Toro / Rutoro zaj Zaramo sgw Sebat Bet Gurage ttq Tawallammat zdj Comorian, Ngazidja shi Tachelhit ttr Nyimatli zga Kinga shj Shatt tui Toupouri ziw Zigula shk Shilluk tul Kutule zne Zande / paZande sid Sidama tum Chitumbuka zul Isizulu sig Paasaal tuv Turkana sil Sisaala, Tumulung tvu Tunen sna Shona twi Twi snf Noon umb Umbundu sng Sanga / Kiluba urh Urhobo snw Selee uth ut-Hun som Somali vag Vagla sop Kisonge vai Vai sor Somrai ven Tshivenda sot Sesotho vid Chividunda soy Miyobe vif Vili spp Senoufo, Supyire vmk Makhuwa-Shirima ssw Siswati vmw Macua suk Sukuma vun Kivunjo sus Sosoxui vut Vute swa Swahili wal Wolaytta swc Swahili Congo wbi Vwanji swh Swahili wec Guere swk Sena, Malawi wes Pidgin, Cameroon sxb Suba wib Toussian, Southern taq Tamasheq wmw Mwani tcc Datooga wol Wolof
Table F. 3: Languages covered in SERENGETI - Part III. 37
ISO-639-3 #Tokens swh 2,912,488,735 afr 1,264,478,436 som 587,549,878 swa 499,792,448 hau 286,806,539 amh 241,700,000 mlg 137,852,716 zne 89,981,183 sna 75,413,519. bam 3,262 har 3,066 dyo 1,797 fvr 1,680 tbz 1,578 ddn 1,372 fuc 1,336 knc 1,097 eot 1,041 cgg 845
Table F. 4: The sizes of the top 10 and bottom 10 languages in SERENGETI pretraining. 38
ACL 2023 Responsible NLP Checklist