Cheetah : Natural Language Generation for 517 African Languages (2024)

Abstract: Low-resource African languages pose unique challenges for natural language processing (NLP) tasks, including natural language generation (NLG). In this paper, we develop Cheetah, a massively multilingual NLG language model for African languages. Cheetah supports 517 African languages and language varieties, allowing us to address the scarcity of NLG resources and provide a solution to foster linguistic diversity. We demonstrate the effectiveness of Cheetah through comprehensive evaluations across six generation downstream tasks. In five of the six tasks, Cheetah significantly outperforms other models, showcasing its remarkable performance for generating coherent and contextually appropriate text in a wide range of African languages. We additionally conduct a detailed human evaluation to delve deeper into the linguistic capabilities of Cheetah. The findings of this study contribute to advancing NLP research in low-resource settings, enabling greater accessibility and inclusion for African languages in a rapidly expanding digital landscape. The GitHub repository for the Cheetah project is available at https://github.com/UBC-NLP/Cheetah.

heading: 1 Introduction
text: The linguistic diversity present in African languages poses unique challenges for NLG systems. With over 2, 000 languages spoken across the African continent (Eberhard et al., 2021), the need for effective NLG solutions that can accommodate this rich linguistic ecosystem cannot be over-emphasized. This is especially important because traditional NLG approaches have primarily focused on high-resource languages, such as English and French due to the availability of large-scale datasets and resources. Consequently, low-resource
⋆ Authors contributed equally.
languages, including numerous African languages, have been marginalized in NLG research and development. Developing robust NLG systems for the diverse needs of African communities is challenging due to the scarcity of extensive language datasets, limited linguistic research, and variations across these languages. To address these challenges, recent advancements in language modeling and transfer learning techniques have shown promise in supporting NLG in low-resource languages. Pretrained language models, such as GPT-3 (Radford et al., 2018, 2019; Brown et al., 2020), mT5 (Xue et al., 2021), and mT0 (Muennighoff et al., 2022), have demonstrated remarkable capabilities in understanding and generating human-like text. These models capture the statistical regularities and syntactic structures of the languages they are trained on, making them valuable starting points for supporting NLG in low-resource settings.
In this paper, we present a pioneering work on NLG in African languages by introducing Cheetah: a novel language model (LM) specifically designed to support 517 African languages and language varieties. To the best of our knowledge, Cheetah supports the largest number of African languages and language varieties. Leveraging a vast
12798
corpus of text data collected from diverse sources, Cheetah learns intricate linguistic information that characterizes each African language. The contributions of this work are three fold. First, we address the scarcity of NLG resources for African languages by providing a comprehensive language model that covers a wide range of languages spoken on the continent. Second, we demonstrate the efficacy of our approach through extensive evaluations across six downstream task clusters. Each cluster includes multiple languages, showcasing the model’s ability to generate coherent and contextually appropriate text in different African languages. Third, we perform fine grained human analysis of Cheetah using a controlled machine translation (MT) test set. This uncovers model behaviour that is not visible with automatic metrics. By supporting NLG in African languages, we foster linguistic diversity, empower African communities to express themselves in their native languages, and bridge the digital divide. This paper serves as a foundational step towards promoting Afrocentric NLP (Adebara and Abdul-Mageed, 2022) that prioritizes inclusivity and cultural preservation in language technology, emphasizing the importance of catering to the unique linguistic needs of diverse populations.
The rest of the paper is organized as follows: In Section 2, we discuss related work. In Section, 4 we describe AfroNLG, the benchmark we create for evaluation. We provide details of Cheetah in Section 3. We present performance of Cheetah in Section 5 and compare it to other multilingual models. We present controlled test sets in Section 5.1. We conclude in Section 6, and outline a number of limitations and use cases for our work in Section 7 and Section 8.

heading: 2 Literature Review
text: One of the challenges in NLG is to generate coherent and semantically meaningful text. Various approaches have been proposed, including templatebased (Becker, 2002; Van Deemter et al., 2005), rule-based (Dušek and Jurčíček, 2015; van Miltenburg et al., 2020), and statistical approaches (Li et al., 2016). More recently, deep learning approaches (Sutskever et al., 2014) including the transformer model (Vaswani et al., 2017) have achieved SoTA results in various NLG tasks such as text summarization (Shi et al., 2021) and machine translation (Vaswani et al., 2017).
While these models have shown impressive results, they often require a large amount of training data and computing resources. However, only a few African languages have benefited from these advancements due to inadequate data. To address this issue, researchers have proposed transfer learningbased approaches, where a pretrained model is finetuned for a specific NLG task. Transfer learning (Raffel et al., 2020; He et al., 2022; Ruder et al., 2019) has enabled the use of low-resource languages on various NLP tasks. Due to lack of adequate (or good quality) pretraining data (Kreutzer et al., 2021), transfer learning is often the most accessible method for only a few low-resource languages leaving behind a vast majority of extremely low-resource languages. This is because about 90% of the world’s languages is claimed to be either leftbehinds, in that it is probably impossible to build NLP resources for them, or scraping-bys with no labelled datasets (Joshi et al., 2020). For the leftbehinds, labelled and unlabelled data are unavailable and even transfer learning approaches are beyond reach while the scraping-by languages have no labelled data with which to evaluate model performance.

heading: 2.1 Language Models
text: Only a few African languages have benefited from the recent advancement of language models (LM) due to inadequate data sizes. We now describe encoder-decoder LMs that support NLP tasks in African languages. We describe these under two broad headings: massively multilingual models and African models. We summarize the models and African languages they cover in Table 1. Multilingual Models: The massively multilingual models such as mBART (Liu et al., 2020), MT0 (Muennighoff et al., 2022), and mT5 (Xue et al., 2021) are trained on several languages. However, in most cases, only a few African languages are represented. Among the mentioned models, mT0 is pretrained on the highest number of African languages (n=13). African Models. Adelani et al. (2022) use pretrained T5, mT5, and mBART models and develop AfriByT5, AfriMT5, AfriMBART respectively to investigate machine translation in zero-shot and out-of-domain settings. The authors experiment on 17 African languages and demonstrate that further pretraining is effective for adding new languages to pretrained models. Jude Ogundepo et al.
(2022) train AfriTeVa, an encoder-decoder language model from scratch on ten African languages and English using similar training objectives like T5 model. AfriTeVa-V2 (Oladipo et al., 2023) has enhanced support for 16 African languages with improved quality pretraining data. 1 African Natural Language Understanding. Several works attempt to improve the performance on African NLU tasks by proposing multilingual and African-dedicated models such as mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020), AfriBERTa (Ogueji et al., 2021), AfroLM (Dossou et al., 2022), Afro-XLM-R (Alabi et al., 2022), KINYaBERT (Nzeyimana and Niyongabo Rubungo, 2022), and SERENGETI (Adebara et al., 2023).

heading: 2.2 Benchmarks
text: Multiple benchmarks have been developed for NLG. However, only a few of Africa’s 2, 000 languages have been supported to date. In most cases, the benchmarks support only the machine translation task. We provide a brief overview under two headings: African and multilingual. We summarize key information about each benchmark in Table C.1. African Benchmarks. AfroMT (Reid et al., 2021a) is a multilingual machine translation benchmark. It consists of translation tasks between English and eight African languages — Afrikaans, Xhosa, Zulu, Rundi, Sesotho, Swahili, Bemba, and Lingala. Menyo-20k (Adelani et al., 2021) is an MT evaluation benchmark for English-Yorùbá. Multilingual with African Languages. FLoRES-200 (Costa-jussà et al., 2022; Guzmán et al., 2019) is an evaluation benchmark that provides MT evaluation support in 200 languages including 52 African languages. GEM (Gehrmann et al., 2021, 2022) referenced as “living" benchmark, comprises of 40 tasks and supports 52 languages including 10 African languages. NLLB Seed Data (Costa-jussà et al., 2022) is a set of professionally-translated sentences sampled from Wikipedia. It consists of around six thousand sentences in 39 languages which include 8 African language. Similarly, NLLB Multi Domain (Costa-jussà et al., 2022) is an MT evaluation benchmark made from a set of
1After we finished our work, we became aware of a new version of AfriTeva, AfriTevaV2 (Oladipo et al., 2023) that was released only recently (December, 2023). We plan to evaluate AfriTevaV2 in our camera-ready version of this work.
professionally-translated sentences in the news and health domains. It consists of approximately 3, 000 sentences in each domain and supports 8 languages including 2 African languages. Toxicity-200 (Costa-jussà et al., 2022) is an evaluation benchmark to evaluate the presence of toxic items in the MT text. It provides support for 50 African languages. XGLUE (Liang et al., 2020) is a cross-lingual, multi-task benchmark created with multilingual and bilingual corpora. It supports 19 languages and one African language, i.e., Swahili.

heading: 3 Cheetah

heading: 3.1 Pretraining Data
text: We are guided by three main principles in developing this data: quality, linguistic diversity, and coverage. Quality. Developing NLP technologies for low resource languages poses a significant challenge due to the limited availability of high-quality training data. To address this issue, we undertook the task of manually curating a diverse corpus spanning multiple domains, including news articles, health documents, religious texts, legal documents, and social media feeds. This manual curation approach was necessary because there were no existing datasets available for the majority of the languages we aimed to support, and we wanted to ensure the utilization of reliable and high-quality data. Coverage. In all, we train Cheetah using a 42G multi-domain corpus across 517 African languages and language varieties. The languages are spoken in 50 of 54 African countries and they are written with five scripts. This provides support to at least 500M Africans. Linguistic Diversity. The inclusion of languages from various domains, geographical regions, and linguistic typologies, along with the utilization of reliable data sources, contributes to enhancing the robustness and quality of Cheetah. Our data consists of languages from 14 language families in Africa written in five different orthographies. Furthermore, our data spans languages with a vast array of exotic linguistic features including tone, vowel and consonant harmony, reduplication, word orders, and word classes.
More details about pretraining is in Appendix A.

heading: 3.2 Implementation Details
text: Vocabulary. We use SentencePiece (Kudo and Richardson, 2018) to encode text as WordPiece
tokens (Sennrich et al., 2016) with 250K WordPieces. We also include data covering the ten top spoken languages globally: Arabic, English, French, German, Greek, Italian, Portuguese, Russian, Spanish, and Turkish. We use Wikipedia dumps for these ten languages. We use 1M sentences for each language. However, we only include it in the vocabulary. Models Architecture. We pretrain Cheetah using the encoder-decoder architecture (Xue et al., 2021). Each of the encoder and decoder components is similar in size and configuration to T5, with 12 layers each with 12 attention heads, and 768 hidden units for the base model. In total, this results in a model with ∼ 580 million parameters. We provide further details in Table B.1. Objective. We use an unsupervised (denoising) objective. The main idea is to feed the model with masked (corrupted) versions of the original sentence, and train it to reconstruct the original sequence. The denoising objective (Xue et al., 2021) works by randomly sampling and dropping out 15% of tokens in the input sequence. All consecutive spans of dropped-out tokens are then replaced by a single sentinel token. Pretraining Procedure For pretraining Cheetah, we use a learning rate of 0.01, a batch size of 1, 024 sequences, and a maximum sequence length of 1, 024. We pretrain each model for 1M steps. We train our models on Google Cloud TPU with 128 cores (v3− 128) from TensorFlow Research Cloud (TFRC).2

heading: 4 AfroNLG Benchmark
text: We create AfroNLG, a multi-lingual, multi-task benchmark comprising 67 test sets across six task
2https://sites.research.google/trc/about/
clusters. AfroNLG includes cloze tasks, machine translation, paraphrase, question answering, summarization, and title generation. It supports 527 languages, including 517 African languages and language varieties and the top 10 world languages. To the best of our knowledge, this is the most extensive benchmark till date for African languages. Table C.1 shows, how our benchmark compares to others. We provide the details of each task cluster and datasets in what follows. For detailed statistics about the task clusters, refer to Appendix C.
Cloze Test. In order to comprehensively evaluate Cheetah across all the languages it was pretrained on, we employ cloze-tasks as our evaluation approach and perform two cloze tasks experiments. These tasks assess the model’s ability to fill in missing information. In the first cloze task, which we henceforth call mask-one, we randomly mask only one token in each sentence. In the second clozetask, which we call mask-at-least-one, we randomly mask at least one token and not more than 10% of the tokens in each sentence. For each of the 517 languages, we construct a cloze-task dataset comprising 200 data points for each language in the Train set, 100 examples for each language in the Test set, and 50 data points for each language in the Dev set. We ensure that there is no overlap between the data used for the cloze tasks and the pretraining data. We show an example of our cloze task in Figure C.1.
Machine Translation. We include only datasets pertaining African languages in our benchmark. In selecting the languages for our MT benchmark, we strive to keep datasets that have been used in any published machine translation task. This allows us to cover a diverse set of languages and compare our models to existing SoTA across a large number of
language pairs. Our benchmark thus contains data from Afro-MT3 (Reid et al., 2021b), Lafand-MT4 (Adelani et al., 2022), PidginUNMT5 (Ogueji and Ahia, 2019), and SALT6 (Akera et al., 2022). The datasets we consider make up 35 language pairs.
Paraphrase. A paraphrase task aims to create semantically similar and fluent paraphrases given an input text (Chen et al., 2023; Palivela, 2021). We use the TaPaCo dataset (Scherrer, 2020) for our paraphrase generation benchmark. TaPaCo is a freely available paraphrase corpus for 73 languages extracted from the Tatoeba database. The dataset has four African languages: Afrikaans, Berber (a macro-language), Amazigh, and Kirundi.
Question Answering. The QA task aims to provide answers to questions based on a knowledge base also referred to as contexts. We use TYDIA7 QA dataset (Clark et al., 2020). The dataset has a primary task and a gold passage task. In our benchmark, we only include the gold passage task, where a correct answer is predicted from a passage containing one answer, similar to the existing reading comprehension task.
Summarization. Summarization is the task of generating an abridged version of a text, while capturing the salient ideas and the intended information from the original text (Nallapati et al., 2016; King et al., 2022). We use the subset of XLSum (Hasan et al., 2021), an abstractive summarization dataset, that consists of African languages including Amharic, Hausa, Igbo, Kirundi, Oromo, Pidgin, Somali, Swahili, Tigrinya, and Yorùbá. We also develop new test sets using data we crawled from the web, which are non-overlapping with XLSum. Specifically, we crawl data from BBC and Voice of Africa (webpages) for Hausa, Ndebele, and Swahili.
Title Generation. The title generation task returns a single sentence title for a given article. Similar to the summarization task, we use XL-SUM to create a news title generation dataset. We also collect a new test set for title generation across 15 languages. The dataset comprises ∼ 6, 000 BBC and Voice of Africa articles, non-overlapping with XL-Sum, and is particularly useful for zero-shot title generation.
3https://github.com/machelreid/afromt 4https://github.com/masakhane-io/lafand-mt 5https://github.com/keleog/PidginUNMT 6https://github.com/SunbirdAI/salt 7https://github.com/google-research-datasets/tydiqa

heading: 5 Evaluation and Results
text: We evaluate Cheetah on six task clusters of AfroNLG benchmark and compare to performance on mT0, mT5, Afri-MT5, and AfriTeVa. We report results in Table 2. For all models, we finetune on the training data split (Train) for 20 epochs with an early stopping of 5 epochs, learning-rate of 5e− 5, batch size of 16, and sequence length of 512. All experiments were performed on 4 GPUs (Nvidia V100). We report the results of each experiment as an average of three runs, each with a different seed.8 We show evaluation results per language and provide information of model performance next. Cloze Test. Cheetah outperforms all other models on both cloze tasks as in Table 2. We show the results for each language that is supported by the models compared in Table D.1 and Table D.2. The performance of all models on mask-one is better than the performance on mask-at-least-one, reflecting how increasing the number of masked tokens makes the task more challenging. It is also important to mention that since evaluation is based on BLEU it does not reflect correct synonyms that each model may have generated to replace the masked tokens. Machine Translation. Cheetah sets a new SOTA on 23 tasks surpassing previous models. The mT0 and AfriTEVA models also demonstrate strong performance on six languages. Notably, pairs with French as the source language tend to yield the lowest BLEU scores, indicating relatively lower translation quality. On the other hand, the language pair involving English to Nigerian Pidgin, specifically on LafandMT and PidginUNMT, showcases the highest BLEU scores. We assume that the similarity between the Nigerian Pidgin and English contributes favourably to translation quality in these scenarios. We also report CHRF and CHRF++ results in Table C.4 and Table C.5 in the Appendix. Paraphrase. In the three paraphrase tasks, Cheetah demonstrates remarkable superiority over all other models. Specifically, we achieve an impressive ROUGE score of 46.0 on the Berber paraphrase task, surpassing the second-best model by a margin of approximately two points. Question Answering. In the task of question answering, mT0 exhibits superior performance compared to other models. While mT5 achieves the second-highest performance, Cheetah attains the third-highest performance in this task.
8Specifically, we use seed values 41, 1512, and 20235.
Summarization. Cheetah sets a new SOTA on 11 languages, outperforming other models by an average margin of at least three points. Detailed results can be found in Table 2. Title Generation. On the Title generation task, Cheetah sets a new SOTA on 11 languages. We report results in Table 2.

heading: 5.1 Investigating linguistic capabilities
text: In order to further test the utility of our models, we use grammar templates to construct test data in English. We use nine linguistic rules and 19 lexical items to generate 152 sentences. Next, we use our model to translate from source to target and manually evaluate the quality of the generated data. We design new evaluation metrics, faithfulness and fluency, for the manual evaluation (see Section 5.2). A detailed description follows. Grammar templates. We use grammar templates (McCoy et al., 2019) developed with context-free grammars (CFG) on the source side to construct controlled test sets in English. We use CFG on the source side alone because constituents and constituent order differs across languages. We adopt this method for two reasons. First, utilizing grammar templates provides a standardized framework that ensures that the same grammatical phenomena are tested consistently. By employing a uniform approach, we can effectively isolate and evaluate specific linguistic features, facilitating a more rigorous and meaningful comparison of language model performance. Second, grammar templates exhibit a high degree of flexibility, allowing for easy modification and extension to encompass a wide range of linguistic phenomena. This adaptability not only facilitates the incorporation of new linguistic features but also enables the evolution of our test sets to match the dynamic landscape of natural language processing research.
Other alternatives to templates include using parsed corpora (Bender et al., 2011) or naturally occurring sentences. For the languages we explore, there are no good quality parsers, making automatic parsing inaccessible for this analysis. Furthermore, when a corpus is parsed automatically, the likelihood of encountering parsing errors escalates with the intricacy of the sentence structure (Bender et al., 2011; Marvin and Linzen, 2018). Conversely, if the test set exclusively comprises sentences with accurate gold parses, sourcing an ample quantity of instances showcasing syntactic complexit-
ies becomes an arduous task (Marvin and Linzen, 2018). Furthermore, the utilization of naturally occurring sentences introduces potential complications that might confound the interpretation of experiments (Ettinger et al., 2018). The templates include transitive and intransitive structures, negative and affirmative structures, and structures with gender and number. Table E.1 provides examples of generated sentences using the templates.The entire generated grammar is available at our GitHub:9. Inference. We test three of our finetuned machine translation models with the generated dataset. This allows us to evaluate how much linguistic information the models have acquired during pretraining and finetuning. Specifically, we use the English→Hausa, English→Swahili, and English→Yorùbá based on MT0, MT5, AfriTEVA, and Cheetah models that were finetuned on the LafandMT dataset. We do not include Afri-MT5 in this analysis because it has very low scores across several tasks as shown in Table 2. Notably, Hausa, Swahili, and Yorùbá have distinct typologies and the performance of each model on each language gives further insight of performance across varying typological features. Table E.2 shows a number of linguistic differences between the three languages. This method can be generalized to any African language.

heading: 5.1.1 Linguistic Details
text: Morphology Morphologically, both Hausa and Swahili are classified as agglutinative languages (Jaggar, 2017; Dryer and Haspelmath, 2013), characterized by the systematic addition of prefixes, suffixes, and affixes to root words or stems. This process imparts precise grammatical meanings, encompassing tense, case, mood, person, number, and more. Conversely, Yorùbá exhibits an analytic structure, relying on word order and discrete function words to denote grammatical relationships, with minimal use of inflections or affixes. The following are examples from the generated (1) Hausa, (2) Swahili, and (3) Yorùbá, respectively.
(1) a. Bai neg.masculine barshi leave ba at-all
‘he did not leave him’
9generatedsentences
b. Bata Neg.feminine barshi leave ba at-all
‘she did not leave him’
(2) a. Ha-ku-mu-a-cha 3pl.sg.sub-neg-3pl.sg.obj-leave
‘He did not leave him’
b. Ha-ku-mu-a-cha 3pl.sg.sub-neg-3pl.sg.obj-leave
‘She did not leave him’
(3) a. Òhun 3pl.sg.sub ò neg kúrò leave lÓ ˙ dÒ ˙from è 3̇pl.sg.obj
‘He did not leave him’
b. Òhun 3pl.sg.sub ò neg kúrò leave lÓ ˙ dÒ ˙ from è ˙ 3pl.sg.obj
‘She did not leave him’
Phonology In terms of phonology, Yorùbá and Hausa are tonal languages, where pitch distinctions contribute to word differentiation. However, Hausa features a relatively simpler tone system compared to Yorùbá and in most cases tone is not marked in Hausa orthography. Only dictionaries and pedagogical materials indicate tone in text. Yorùbá on the other hand has three tones and indicating tones in orthography significantly reduces ambiguity (Adebara and Abdul-Mageed, 2022). Swahili, in contrast, is devoid of tones altogether.

heading: 5.2 Human evaluation
text: To evaluate the effectiveness of each model across different languages, we assess the generated output’s faithfulness and fluency using a five-point Likert scale. Faithfulness measures how accurately a model’s output corresponds to the input sentence, while fluency assesses the grammatical coherence and plausibility of the generated output. We use both metrics because a model can produce coherent output that may not be faithful to the input sentence. This way, if faithfulness penalizes a model for outputs that are not true to the input or that include additional unnecessary information, fluency complements our evaluation of the quality of the same
model if the output is fluent. For each grammar category, we return the average Likert point for each language and across the different models model.

heading: 5.3 Annotation
text: We annotated each model’s output for faithfulness and fluency. For Hausa and Yorùbá, two expert annotators evaluated the model’s output for faithfulness and fluency. We ensured that each annotator has native speaker competency in reading and writing (while some had a linguistic background). We gave specific annotation instructions (see Section E in the Appendix) to ensure the values are not assigned arbitrarily. We also ensured that the annotators do not know who created which models to prevent any biases. We report the Cohen’s Kappa agreement scores in Table E.3 (Appendix). For Swahili, only one annotator made it to the final annotation task since we could not acquire high quality annotations from other annotators. The Swahili annotator who did the final annotation is a university lecturer with a Ph.D. in linguistics.

heading: 5.4 Fluency and Faithfulness Performance
text: We report the distribution of faithfulness and fluency scores across all models and languages in Figure 2. Overall, Cheetah produces more faithful and more fluent outputs than other models on all languages. We now go on to provide a brief analysis of model performance. More details are in in Appendix F. Intransitives In the case of Hausa examples, all three models can generate intransitive sentences with varying levels of fluency and faithfulness with Cheetah outperforming other models. In the context of Swahili, errors primarily relate to tense, possibly because Swahili has an agglutinative structure, and the models may lack exposure to a comprehensive range of grammatical features during training. In the case of Yorùbá, all models consistently incorporate at least one object in each intransitive case. This could be because intransitive sentences in Yorùbá lack a clear direct object, making it challenging for machine translation models to select the accurate translation. Additionally, some intransitive phrases in Yorùbá can be polysemous, further complicating the translation process. We report the distribution of scores in Figure G.1. Transitives Cheetah demonstrates the capability to provide three distinct semantic senses for the polysemous transitive verb treated whereas the other models typically produce only a single semantic in-
terpretation. For Swahili, certain instances exhibit the deletion or simplification of object markers in an ungrammatical manner. Figure G.2 shows the distribution of model performance on transitives.
Negative mT0, MT5, and AfriTeVa have a tendency to output the negation of the antonym of the verb in each sentence rather than the negation of the verb. Cheetah also makes similar mistakes about 5% of the time. Affirmative. The models generally perform better in the context of the affirmative examples than on the negated examples. However, in the context of Hausa, mT5, mT0, and AfriTeVa consistently output the antonym of negated verb. For instance, the models return “Sara left" rather than “Sara did not leave". In the Swahili examples, we also find cases of double negation (which is not grammatically correct in Swahili). We show the distribution of results in Figure G.5 and Figure G.4.
Gender/Agreement Yorùbá does not distinguish gender, yet Cheetah uses Arábìrin (female) before every occurrence of the name “Sara" to indicate that the it has a high probability of being feminine (see Figure G.3). However, “Fred" is not annotated this way. For Hausa, which requires agreement between the gender of the noun and the verb, we find Cheetah outperforming both mt0 and mt5 significantly. AfriTeVa, however, has very low accuracy in the context of gender. Furthermore, mt0,
mt5, and Cheetah return connotations for love and relationships for each examples where a male and female pronoun co-occur cross-lingually.

heading: 6 Conclusion
text: We introduced Cheetah, a massively multilingual language model designed for African natural language generation. We also propose a new African language generation benchmark, dubbed AfroNLG, that is both sizeable and diverse. We evaluate Cheetah on AfroNLG comparing it to three other models, two multilingual and one dedicated to African languages. The performance of Cheetah surpasses that of all other models we evaluate. This is demonstrated by its superior AfroNLG score, which is approximately three times better than the combined performance of other models. Furthermore, Cheetah outperforms all other models across 48 out of 65 test sets spanning six task clusters. We further analyze our model’s robustness to lexical complexity and carry out human evaluation to inspect the model’s perform on a controlled test set. Again, our results underscore superiority of our model.

heading: 7 Limitations
text: We identify the following limitations for our work:
1. The limitations of our language model include
the limited scope of our evaluation. Future work should focus on increasing the subset of languages evaluated manually in order to ensure quality. We believe automatic analyses are not sufficient for development of models that get deployed in particular applications.
2. Another limitation is related to our inability to perform extensive analysis of biases and hateful speech present in our pretraining data. Again, this is due to relatively restricted access to native speakers (and even automated tools) to perform this analysis. As a result, we cannot fully ensure that our models are free from biases and socially undesirable effects. Therefore, it is important that these models be used with care and caution, and be analyzed for biases and socially undesirable effects before use.
3. Additionally, due to unavailability of sufficient computing resources, we were unable to evaluate larger multilingual language models.

heading: 8 Ethics Statement and Wider Impacts
text: Cheetah aligns with Afrocentric NLP where the needs of African people is put into consideration when developing technology. We believe Cheetah will not only be useful to speakers of the languages supported, but also researchers of African languages such as anthropologists and linguists. We discuss below some use cases for Cheetah and offer a number of broad impacts.
1. Cheetah aims to address the lack of access to technology in about 90% of the world’s languages, which automatically discriminates against native speakers of those languages. More precisely, it does so by focusing on Africa. To the best of our knowledge, Cheetah is the first massively multilingual PLM developed for African languages and language varieties. A model with knowledge of 517 African languages, is by far the largest to date for African NLP.
2. Cheetah enables improved access of important information to the African community in Indigenous African languages. This is especially beneficial for people who may not be fluent in other languages. This will potentially connect more people globally.
3. Cheetah affords opportunities for language preservation for many African languages. To the best of our knowledge, Cheetah consists of languages that have not been used for any NLP task until now. We believe that it can help encourage continued use of these languages in several domains, as well as trigger future development of language technologies for many of these languages.
4. Although LMs are useful for a wide range of applications, they can also be misused. Cheetah is developed using publicly available datasets that may carry biases. Although we strive to perform analyses and diagnostic case studies to probe performance of our models, our investigations are by no means comprehensive nor guarantee absence of bias in the data. In particular, we do not have access to native speakers of most of the languages covered. This hinders our ability to investigate samples from each (or at least the majority) of the languages.
5. We emphasize the ethical use of these models.

heading: Acknowledgments
text: We acknowledge support from Canada Research Chairs (CRC), the Natural Sciences and Engineering Research Council of Canada (NSERC; RGPIN2018-04267), the Social Sciences and Humanities Research Council of Canada (SSHRC; 435- 2018-0576; 895-2020-1004; 895-2021-1008), Canadian Foundation for Innovation (CFI; 37771), Digital Research Alliance of Canada,10 and UBC Advanced Research Computing-Sockeye.11

heading: A Pretraining Data
text: We provide details of our pretraining data below: Religious Domain. Our religious data is taken from online Bibles, Qurans, and data crawled from the Jehovah’s witness website. We also include religious texts from the book of Mormon. News Domain. We collect data from online newspapers (Adebara and Abdul-Mageed, 2022) and news sites such as Voice of America, Voice of Nigeria, BBC, Global voices, and DW news sites. We collect local newspapers from 27 languages from across Africa. Government Documents. We collect government documents South African Centre for Digital Language Resources (SADiLaR), and the Universal Declaration of human rights (UDHR) in multiple languages. Health Documents. We collect multiple health documents from the Department of Health, State Government of Victoria, Australia. We collect documents in Amharic, Dinka, Harari, Oromo, Somali, Swahili, and Tigrinya. Existing Corpora. We collect corpora available on the web for different African languages, including from Project Gutenberg for Afrikaans, South African News data. for Sepedi and Setswana, OSCAR (Abadji et al., 2021) for Afrikaans, Amharic, Somali, Swahili, Oromo, Malagasy, and Yoruba. We also used Tatoeba for Afrikaans, Amharic, Bemba, Igbo, Kanuri, Kongo, Luganda, Malagasy, Sepedi, Ndebele, Kinyarwanda, Somali, Swahili, Tsonga, Xhosa, Yoruba, and Zulu; Swahili Language Modelling Data for Swahili; Ijdutse corpus for Hausa; Data4Good corpora for Luganda, CC-100 for Amharic, Fulah, Igbo, Yoruba, Hausa, Tswana, Lingala, Luganada, Afrikaans, Somali, Swahili, Swati, North Sotho, Oromo, Wolof, Xhosa, and Zulu; Afriberta-Corpus for Afaan / Oromo, Amharic, Gahuza, Hausa, Igbo, Pidgin, Somali, Swahili, Tigrinya and Yoruba; mC4 for Afrikaans, Amharic, Hausa, Igbo, Malagasy, Chichewa, Shona, Somali, Sepedi, Swahili, Xhosa, Yoruba and Zulu.

heading: B Models

heading: C AfroNLG Benchmark
text: We report statistics of AfroNLG benchmark in Table C.2 and C.1 respectively.
C.1 CHRF and CHRF++ Results

heading: D Cloze Task Results
text: We provide results on the performance of each model on individual languages. We use a dash ’-’ to indicate that a specific model does not support a language.

heading: E Annotation
text: We gave the following annotation rules to our annotators: Faithfulness refers to how close to the English sentence the model output is. It should be annotated with values between 1 and 5. Faithfulness should be evaluated independently of the fluency of the model output. Below are some detailed explanations for the scale for faithfulness:
• Give a value 1 if model output is not related to the source sentence.
• Give a value 2 if the model output is the opposite of the source sentence.
• Give a value 3 if the model output is somewhat related to the source sentence. It should have some words or phrases that make it related to the source.
• Give a value 4 if the model output is closely related but changes the meaning slightly (e.g difference in gender, number etc)
• Give a value 5 if the model output is an exact translation
Fluency is how grammatically correct the model is. Faithfulness and fluency should be judged independently. That is, even if the output is not faithful, don’t use it to determine the fluency score and vice versa. Here are some detailed explanations on how to assign the values:
• Give a value 1 if model output is completely ungrammatical and nonsensical.
• Give a value 2 if the model output is reasonable but includes some foreign words or gibberish.
• Give a value 3 if the model output contains some grammatical phrases but also contains some ungrammatical phrases.
• Give a value 4 if the model output is almost grammatical (but may have a few errors like spelling mistakes)
• Give a value 5 if the model output is very fluent and sounds looks like what a native speaker will say.
To prepare for the official annotation process, each evaluator annotated 10 random samples for the language pair they were assigned to. Following the individual evaluations, we reviewed any annotation errors and inconsistencies in assessments and assigned another random pair of 10 samples. In the second phase, we removed evaluators for which the quality of their annotations were deemed of a poor quality.

heading: F Fluency and Faithfulness Performance
text: We report the distribution of faithfulness and fluency scores across all models and languages in Figure 2. Overall, Cheetah produces more faithful and more fluent outputs than other models on all languages. We now go on to provide detailed analysis of model performance. Intransitives In the case of Hausa examples, all three models manage to produce intransitive examples. However, Cheetah consistently appends objects to these intransitive examples. This inclination to add objects might stem from biases within the data used for pretraining or finetuning Cheetah. Nevertheless, it is worth noting that Cheetah outperforms other models by generating more fluent and more faithful Hausa outputs. In the Swahili context, all models successfully generate intransitive translations, with model errors primarily related to tense. This performance discrepancy in Swahili can be attributed to its agglutinative structure, with models potentially lacking exposure to a comprehensive range of grammatical features during pretraining
or finetuning. In the context of Yorùbá, all models consistently incorporate at least one object in each intransitive case. Notably, mT0 generates an output without an object approximately 5.88% of the time. This may be because intransitive sentences inherently lack a clear direct object, making it more challenging for machine translation models to grasp context and select the accurate translation. In certain instances, some intransitive phrases can be polysemous, further complicating the translation process. Intransitive English verbs do not always retain their intransitive nature in Yorùbá. Furthermore, transitives with optional/truncated objects tend to have a compulsory object in Yorùbá. This phenomenon potentially contributes to the models’ tendency to append objects to intransitive Yorùbá phrases. For instance, whereas the intransitive "slept" in "John slept" maps to the intransitive form "John sùn" in Yorùbá, the intransitive verb "prayed", in "John prayed" becomes "John gbàdúrà", a transitive verb in Yorùbá. On the other hand, the transitive verb "ate" in "John ate", has an optional/truncated object in English but becomes "John je
˙ un", a transitive with an obligatory object.
In Yorùbá, both "ate" and "prayed" are transitive verbs that require an object. They are derived from "je
˙ " (eat) and "oúnje ˙ " (food), which give rise to "je ˙ un" and "gbà" (collect) and "àdúrà" (prayer),
resulting in "gbàdúrà" respectively.
Transitives In the context of transitives, Cheetah stands out as the top-performing model across all three languages, as illustrated in Figure 2. Cheetah demonstrates the capability to provide three distinct semantic senses for the polysemous transitive verb treated whereas the other models typically produce only a single semantic interpretation. In Swahili examples, certain instances exhibit the deletion or simplification of object markers in an ungrammatical manner. For a visual representation of the annotation of intransitive sentences in Yorùbá, please refer to Figure G.3. Figure G.2 shows the distribution of model performance on transitives.
Negative In the context of Yorùbá, all models are able to produce the correct negation marker including the correct tone marks. The tone patterns on negation markers may vary based on the context of words before and after the negation marker and it was interesting to see these variations in the models outputs. Despite this, mT0, MT5, and AfriTeVa have a tendency to output the negation of the antonym of the verb in each sentence rather than the negation of the verb. Cheetah also makes similar mistakes about 5% of the time.
Gender/Agreement We find interesting cases of gender in the model’s output. For example, whereas Yorùbá grammar does not distinguish gender, Cheetah uses Arábìrin (female) before every occurrence of the name “Sara" to indicate that the it has a high probability of being feminine (see Figure G.3). It is important to mention that “Fred" is not annotated this way. For Hausa, which requires agreement between the gender of the noun and the verb, we find Cheetah outperforming both mt0 and mt5 significantly. AfriTeVa, however, has very low accuracy in the context of gender. Furthermore, mt0, mt5, and Cheetah return connotations for love and relationships for each examples where a male and female pronoun co-occur cross-lingually.
Number Cheetah significantly outperforms all three models in accurately assigning appropriate number markers. We also find that when translating the word "you" into Hausa, Swahili, or Yorùbá, all four models use either singular or plural forms. We assume that this is due to the fact that the second person in English (i.e., “you") can be either singular or plural while each of these languages have a different word for the singular and plural forms.
compare Cheetah with. Red describes zero-shot performance greater than 0.
Category Example
Intransitive He left Intransitive + Negation We did not leave Transitive You left Lagos Transitive + Negation She did not leave them
Table E.1: Some examples of sentences generated with the templates
Lang. Family # Tone Gender Morphology
Hausa Afro-Asiatic Two Two Isolating Swahili N.C. Bantu None Five Agglutinative Yourba N.C. Non-Bantu Three None Isolating
Table E.2: Some linguistic differences between Hausa, Swahili, and Yoruba. N.C. refers to Niger-Congo
hau yor
Model Faith. Flu. Faith. Flu.
mT0 90.54 97.62 96.57 93.92 mT5 93.51 96.48 82.23 81.10 AfriTeVa 87.27 96.94 88.56 84.73 Cheetah 96.61 97.26 87.11 92.64
Table E.3: Kappa scores for Faithfulness (i.e., Faith.) and Fluency (i.e., Flu.) across the four models and three languages we evaluate.
G Results on Quality Evaluation
Figure G.1: Faithfulness and fluency for Intransitives in Hausa, Swahili, and Yorùbá
Figure G.2: Faithfulness and fluency for Transitives in Hausa, Swahili, and Yorùbá
Figure G.3: Performance on some intransitive examples in the Yorùbá test set. The correct words have no highlights, plausible words or phrases are highlighted with yellow ink while wrong words and phrases are highlighted with grey highlights. We use plausible to refer to words or phrases that can be used in place of the gold or which give additional information.
Figure G.4: Faithfulness and fluency for Intransitives + Negation in Hausa, Swahili, and Yorùbá
Figure G.5: Faithfulness and fluency for Transitives + Negation in Hausa, Swahili, and Yorùbá

heading: Future Work
text: We believe that it can help encourage continued use of these languages in several domains, as well as trigger future development of language technologies for many of these languages. 4. Although LMs are useful for a wide range of applications, they can also be misused. Cheetah is developed using publicly available datasets that may carry biases. Although we strive to perform analyses and diagnostic case studies to probe performance of our models, our investigations are by no means comprehensive nor guarantee absence of bias in the data. In particular, we do not have access to native speakers of most of the languages covered. This hinders our ability to investigate samples from each (or at least the majority) of the languages. 5. We emphasize the ethical use of these models