Venue: NAACL 2025

# IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models

David Ifeoluwa Adelani\({}^{1,2*}\), Jessica Ojo\({}^{1,3*}\), Israel Abebe Azime\({}^{4*}\), Jian Yun Zhuang\({}^{5}\), Jesujoba O. Alabi\({}^{4*}\), Xuanli He\({}^{6}\), Millicent Ochieng\({}^{7}\), Sara Hooker\({}^{8}\), Andiswa Bukula\({}^{9}\), En-Shiun Annie Lee\({}^{10}\), Chiamaka Chukwuneke\({}^{11}\), Happy Buzaaba\({}^{12}\), Blessing Sibanda\({}^{*}\), Godson Kalipe\({}^{*}\), Jonathan Mukibi\({}^{13*}\), Salomon Kabongoo\({}^{14*}\), Foutse Yuehgoh\({}^{15*}\), Mmaishidi Setakia\({}^{9}\), Lolfwethu Ndolela\({}^{*}\), Nkiruka Odu\({}^{*}\), Rooweither Mabuya\({}^{9}\), Shamsudeen Hassan Muhammad\({}^{16}\), Salomey Osei\({}^{17*}\), Sokhar Samb\({}^{18*}\), Tadesse Kebede Guge\({}^{19*}\), Tombekai Vangoni Sherman\({}^{20}\), Pontus Stenetorp\({}^{6}\)

###### Abstract

Despite the widespread adoption of Large language models (LLMs), their remarkable capabilities remain limited to a few high-resource languages. Additionally, many low-resource languages _(e.g.,_ African languages) are often evaluated only on basic text classification tasks due to the lack of appropriate or comprehensive benchmarks outside of high-resource languages. In this paper, we introduce IrokoBench--a human-translated benchmark dataset for 17 typologically-diverse low-resource African languages covering three tasks: natural language inference (AfriXNLI), mathematical reasoning (AfriMGMSM), and multi-choice knowledge-based question answering (AfriMMLU). We use IrokoBench to evaluate zero-shot, few-shot, and translate-test settings (where test sets are translated into English) across 10 open and six proprietary LLMs. Our evaluation reveals a significant performance gap between high-resource languages (such as English and French) and low-resource African languages. We observe a significant performance gap between open and proprietary models, with the highest performing open model, Gemma 2 27B only at 63% of the best-performing proprietary model GPT-4o performance. In addition, machine translating the test set to English before evaluation helped to close the gap for larger models that are English-centric, such as Gemma 2 27B and LLMa 3.1 70B. These findings suggest that more efforts are needed to develop and adapt LLMs for African languages.

+
Footnote †: April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

+
Footnote †: April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

+
Footnote †: April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

In recent years, the capabilities of large language models (LLMs) have greatly improved, from coherent chat experiences to solving complex and knowledge-intensive tasks like mathematical reasoning, coding, and question answering (QA) [1]; Jiang et al. (2024); Gemini-Team et al. (2024). These models have also demonstrated the ability to quickly learn new and challenging tasks with few in-context learning examples and through chain-of-thought reasoning Brown et al. (2020); Shi et al. (2022); Wei et al. (2022). However, most state-of-the-art (SoTA) LLMs are primarily trained on high-resource languages (HRLs), resulting in sub-optimal performance for languages unseen during pre-training Touvron et al. (2023); Ojo et al. (2023). Furthermore, this language coverage bias is reflected in the evaluation stage, predominantly conducted in English and a few other HRLs.

There has been considerable effort to create benchmarks for African languages, but they typically cover simpler tasks, or are specific to narrow tasks such as machine translation, and, more recently, reading comprehension Bandarkar et al. (2023); Aremu et al. (2023). Some diverse reasoning benchmarks have included Swahili--the most spoken native African language--for tasks like commonsense reasoning Ponti et al. (2020) and natural language inference Conneau et al. (2018). Consequently, current multilingual evaluations of LLMs do not accurately reflect their capabilities in reasoning and knowledge-intensive tasks across the majority of African languages.

Furthermore, the few comprehensive evaluations that exist across languages often rely on machine translation of English benchmarks Singh et al. (2024). While automatic translation from English benchmarks is a popular approach given the cost

and time investment required for human translation, it often suffers from noise and biases Vanmassenhove et al. (2021); Lee et al. (2022); Khiu et al. (2024); Hartung et al. (2023); Savoldi et al. (2021) or fail to reflect cultural context Wang et al. (2022); Ji et al. (2023); Pudjiati et al. (2022). Automatic curation may also amplify any of the ubiquitous issues with the quality of broad pretraining sets Luccioni and Viviano (2021); Kreutzer et al. (2022); Ferrara (2023).

**In this paper, we seek to address both the diversity and breadth of evaluation coverage.** We introduce IrokoBench, a human curated benchmark dataset for 17 typologically diverse African languages which encompasses three complex tasks: natural language inference (NLI), mathematical reasoning, and multi-choice knowledge-based QA. The datasets were created by human translating a subset of English cross-lingual NLI (XNLI) Conneau et al. (2018), English Multilingual Grade School Math (MGSM) Shi et al. (2023), and Massive Multitask Language Understanding (MMLU) Hendrycks et al. (2021), evaluation datasets into each of the 16 languages using professional translators.

We conduct a large-scale evaluation of IrokoBench to assess zero-shot, few-shot, and translate-test settings (where test sets are translated into English) performance across 10 open and six proprietary LLMs. Our main contributions can be enumerated as follows:

1. [leftmargin=*]
2. **We introduce and release IrokoBench**, a human-translated benchmark that includes 16 languages from various geographical regions in Africa, all with varying degrees of "low-resourcedness" Joshi et al. (2020).
3. **Sharp cliff in performance across all models on low-resource languages** Our evaluation shows a large gap (\({\sim}45\%\) on average) between the performance of high-resource languages (_e.g.,_ English) and African languages on all LLMs evaluated. Notably, Swahili performs better than other African languages, likely due to its large corpus on the web.
4. **Models generally perform poorly in language evaluation** This can be attributed to the inability of current SoTA LLMs to respond in the native languages of the users. Machine translating the test set to English before evaluation helped to close the gap for English-centric models; however, requiring users to always translate their prompts to English may not be a desirable behavior.
5. **IrokoBench highlights the performance divide between open and proprietary models on low-resource languages.** We find that proprietary closed models generally outperform open models for African languages. However, even these proprietary models exhibit substantial performance drops, due to the limited monolingual web data for African languages. The lowest performance is observed in languages such as _Ewe_, _Lingala_, _Luganda_, _Twi_ and _Wolof_, which each have less than 50 million characters of available data Kudugunta et al. (2023). Among the tasks evaluated, AfriMGSM proves most challenging for LLMs, followed by AfriMMLU and AfriXML.

We release IrokoBench on GitHub 1 and HuggingFace under the CC BY-SA 4.0 license 2 to further multilingual evaluation and research.

Footnote 1: [https://github.com/masakhane-io/masakhane-nlu](https://github.com/masakhane-io/masakhane-nlu)

Footnote 2: [https://huggingface.co/collections/maskhane/irokobench-665a2l9b6d471ted3f81af3b1](https://huggingface.co/collections/maskhane/irokobench-665a2l9b6d471ted3f81af3b1)

## 2 Related Work

Multilingual Evaluation of LLMs:The evaluation of multilingual capabilities of LLMs has garnered significant attention. This has led to an increase in research that explores their performance across diverse linguistic landscapes Ahuja et al. (2023); Dai et al. (2023); Hendy et al. (2023); Bang et al. (2023); Ustun et al. (2024); Singh et al. (2024). Despite this growing interest, there remains a notable lack of representation of African low-resource languages in these studies. Ojo et al. (2023); Azime et al. (2024) address a broader spectrum of African languages, aligning more closely with our research. However, their study focuses on conventional NLP tasks, such as text classification, question answering, and text generation tasks. To address the lack of difficult benchmarks, a few works automatically translated MMLU benchmarks Lai et al. (2023), but they do propagate errors of machine translation (MT) engines. Moreover, this is not applicable to low-resource languages with low-quality MT systems Adelani et al. (2022); Costa-jussa et al. (2024). Our research advances this by evaluating LLMs on more complex tasks using newly developed, human-annotated benchmarks specifically for African languages 

African Benchmark Datasets:Due to the limited representation of African languages in the field of NLP, there has been a growing effort to create benchmark datasets for African languages to enable research on these languages. Initiatives such as Masakhane have been instrumental in the creation of standard benchmark for tasks such as machine translation (Adelani et al., 2022), named entity recognition (Adelani et al., 2021, 2022b), part of speech tagging (Dione et al., 2023), news topic classification (Adelani et al., 2023), and sentiment analysis (Muhammad et al., 2023). There are also several multilingual benchmark datasets that cover a few African languages, such as SIB-200 (Adelani et al., 2024), Flores (Goyal et al., 2022; Costa-jussa et al., 2024), Aya dataset and Collection (Singh et al., 2024) and Taxi1500 (Ma et al., 2023). However, despite all these efforts, African languages still lack quality and more difficult datasets, our work fills this gap.

## 3 IrokoBench

### Languages covered by IrokoBench

IrokoBench3 cover 19 languages, 17 native African languages, and two European languages (English and French) which are widely spoken and official in many African countries. English and French are also the source languages we translated from. The 17 diverse and widely spoken African languages are from four regions of Africa: seven from West Africa (Ewe, Hausa, Igbo, Twi, Vai, Wolof, Yoruba), five from East Africa (Amharic, Kinyarwanda, Luganda, Swahili, and Oromo), four from Southern Africa (chiShona, isiXhosa, isiZulu, and Sesotho), and Central Africa (Lingala). These languages are from three language families in Africa: one from Mande, three from Afro-Asiatic and 13 from the Niger-Congo family--where we cover eight Bantu languages. Table 9 provides an overview of the languages covered, including their family, regions, and the number of native speakers.

Footnote 3: Our benchmark name comes from lrökö— a large hardwood in West Africa that is very durable for making bench.

### Tasks covered by IrokoBench

The selection of these tasks is primarily driven by their coverage across various domains and downstream tasks for diverse use cases. Additionally, they enable the evaluation of logical, abstract, and reasoning capabilities in LLMs, which is the hallmark of human intelligence (Bowman et al., 2015; Hendrycks et al., 2021). Figure 1 provides examples of the different tasks covered in our datasets. We provide their descriptions below:

AfriXNLIThe task of NLI involves the classification of a pair of sentences--a premise and a hypothesis as _entailment_, _neutral_, or _contradiction_ semantic relation. For example, the sentence "so I'm not really sure why" _contradicts_ "I am certain as to the reason why" but has a _neutral_ relation to "I don't know why he transferred schools". Here, we human translate the English portion of XNLI (a multilingual dataset comprising 15 languages, including Swahili) into the 15 African languages (excluding Swahili). While the original XNLI dataset has over 2,500/5,000 as DEV/TEST split, Each language in AfriXNLI has only 450 DEV instances and 600 test instances. We selected an equal num

Figure 1: **Task Description for IrokoBench datasets. Both AfriMGSM and AfriXML focus on QA, while AfriXNLI focuses on natural language inference between two pairs of sentences. For clarity, this figure provides examples in English.**



ber of instances from the 10 domains of XNLI. The task is evaluated using the _accuracy metric_ since the dataset has balanced classes.

AfriMMLUThis is a multi-choice knowledge QA curated from freely available online sources by undergraduate and graduate students in the USA. The subjects cover simple general knowledge questions like "global fact" to highly-technical questions like "professional law" and "professional medicine". MMLU are often grouped into _STEM_, _Humanities_, _Social Sciences_, and _Others_ category. We focus on five subjects that we believe are culturally unbiased (or international) and that are simpler to translate since many of the subjects covered are only taught in African countries using English or French, making it extremely difficult to translate highly technical subjects, especially the STEM subjects. Table 1 shows the five subjects covered: two social science subjects (high-school geography and high-school microeconomics), one STEM subject (elementary mathematics), one humanities subject (international law), and one OTHER category (global facts). In total, we translated 608 question-answer pairs, with 500 instances in the test split, 100 questions per subject. The task is evaluated using the _option prediction accuracy_.

AfriMGSMThis is a QA task with questions obtained from grade school mathematical word problems created by human problem writers. AfriMGSM expands the original MGSM dataset Shi et al. (2023), which contains 250 QA pairs and 11 languages (including Swahili), to 15 more languages. The dataset consists of 8 training examples for few-shot and chain-of-thought prompting and 250 as a test set. We evaluated this task using the _Exact Match metric_, which is popularly used for QA tasks.

### Data collection process

TranslationWe recruited language coordinators for each of the 16 African languages and French, and asked them to recruit professional translators to translate the sentences. The translation process took about two months, they started with XNLI, then MGSM and MMLU. Each translator received an appropriate remuneration for their work. 4 Most of the translators translated from English except for _Ewe_, _Lingala_ and _Wolof_ translators that translated from French since they are from the Francophone region of Africa. Additionally, we translated the MMLU dataset to French by professional translators and from French to these three languages. Many of the Francophone translators understand French and English but are more fluent in French, so they could cross-check from English if the French sentences were not clear enough.

Footnote 4: We recruited a logistic company in Kenya that managed all recruitment and payments—each country has different rates. For example, we paid $549.78 for the translation of 1020 XXNI samples in South Africa, $355.86 in Nigeria.

Quality controlRegarding quality control, language coordinators reviewed and corrected any poorly translated sentences. Translators received payment only after this phase to ensure the quality of translations. For additional checks, we computed COMET Rei et al. (2020) quality estimation (QE) scores between the human translation and the original sentences based on AfriCOMET QE metric Wang et al. (2024). In general, the distribution of the scores (range between 0 and 1) reflect that most translated sentences are between \(0.7\) and \(1.0\) for about 13 language pairs except for _Lingala_, _Twi_, and _Wolof_ where the average is around \(0.5\). Further analysis shows that we cannot rely on these scores for those three languages since they are not covered in the pre-training of the original AfroX-LMR encoder Alabi et al. (2022) used to build the AfriCOMET QE metric. Similar findings were reported in the original AfriCOMET QE paper that Twi had worse correlation with human judgement (_i.e.,_ 0.279 for Pearson, and 0.060 for Spearman) Wang et al. (2024). We provide further analysis of the COMET scores in Appendix A.3.

### LLMs used for evaluation

Open LLMsWe evaluate on two _encoder-decoder open LLM_: mTO-XXL-MT Muennighoff et al. (2023), and Aya-101 Ustun et al. (2024) that have been instruction fine-tuned and multilingual T5 pre-trained on 101 languages (mT0 and Aya-101) models. Furthermore, these models are also all designed to be _massively multilingual_ and explicitly optimized to work outside of English. The languages covered during instruction tuning differ for different models, mT0 and Aya covered 46 and 101 languages respectively.

Additionally we evaluate on eight _decoder-only open LLM_ models: BLOOMZ 7B BigScienceWorkshop et al. (2023), Gemma 2 (9B & 2FB) Rivierre et al. (2024), LLaMa 3 8B Meta (2024), LLaMa 3.1 (8B & 7OB) Dubey et al. (2024), Command-R August version Cohere (2024), and 

LLaMaX 3 8B Lu et al. (2024). These models' weights are openly available under various licenses, ranging from fully permissive to non-commercial, research-only licenses. We evaluate the instruction-tuned variant of these models. All models were pre-trained from scratch except LLaMaX that undergo continue pre-training on 100 languages including 13 languages covered in IrokoBench, except Ewe, Twi, Kinyarwanda and Vai. We used LLaMAX3-8B-Alpaca, instruction-tuned on English Alpaca Taori et al. (2023).

Closed LLMWe limit our evaluation to only OpenAI GPT (3.5-0125, 4-Turbo-0125, 4o-mini-07-18, 4o-08-06) OpenAI (2024), Gemini-1.5-Pro Reid et al. (2024), and Claude OPUS Anthropic (2024) models. Recent work has shown that proprietary models tend to exhibit better multilingual capabilities Ahuja et al. (2023b), although specifics regarding their pre-training and instruction fine-tuning processes are not disclosed.

### Evaluation Settings

Evaluation Set-upWe conduct two types of evaluations: _in-language_ and _translate-test_ evaluation, where test instances are automatically translated into English using a machine translation (MT) engine. For MT, we use NLLB-200 (3.3B) Costajussa et al. (2024). In both in-language and translate-test setups, we perform cross-lingual transfer experiments from English and zero-shot evaluations by prompting LLMs. Few-shot evaluations are performed only for the three best models (two open and one closed) in the in-language setting. For AfriMGSM in both settings, we use **Chain-of-Thought (COT)** reasoning.

We use the EleutherAI LM Evaluation Harness (lm-eval) tool Biderman et al. (2024)--a popular evaluation tool that is helping to standardize LLM evaluation, especially for open models on HuggingFace Model Hub. For closed models, we employ a **verbalizer**Gao et al. (2021); Schick and Schutze (2021) for prediction and evaluation. All models are prompted with **five different templates**. We provide more details in Appendix A.2. 5

Footnote 5: We make use of Cohere API for Command-R inference.

Cross-lingual transfer experimentsWe first conduct a study on cross-lingual transfer in a supervised learning setting by fine-tuning the English training data (\(400K\) instances) from Conneau et al. (2018) and evaluating on the remaining languages. This experiment focuses solely on the NLI task due to the availability of training data for supervised learning. The evaluation employs several masked language models, including XLM-R Conneau et al. (2020), Serengeti Adebara et al. (2023), AfroXLMR-{base, large} Alabi et al. (2022), AfroXLMR-76L Adelani et al. (2024). We report the result of the best model in the paper and others in Appendix A.6.

Zero- and few-shot evaluationIn a zero-shot setting, we use the prompts detailed in subsection A.2. For few-shot evaluations, we conduct a 5-shot assessment for both AfriMMLU and AfriXML, and an 8-shot assessment for AfriMGSM.

## 4 Results

### Overall Results

Large performance gaps between high-resource languages and African languagesTable 2 shows the result of zero-shot evaluation for various LLMs. On average, there is a significant performance gap between African languages and English (up to \(28\%\)) and French (up to \(19\%\)) on the best LLM. The best LLM for African languages is GPT-4o, with an average performance of 59.0 across the evaluated tasks. Finally, as shown in Table 9, the languages with the lowest performance have the least monolingual data on the web.

Large performance gaps between closed and open weights modelsOur results, as presented in Table 2, indicate that the closed models Claude

\begin{table}
\begin{tabular}{l l c p{113.8pt} p{113.8pt}} \hline  & No. of & No. Instances & & Average Length \\
* / 46 \\ AfriMLUL & 17 (incl. French) & 25 / 83 / 500 & elementary mathematics, high-school geography, International law, global facts, high school microeconomics & 18 / 17 / 17 \\ AfriXXNLI & 15 (excl. Swahili) &
- / 450 / 600 & face-to-face, telephone, oxford university press (ow), fiction, travel, government, nineteleven, letters, slate, verbatim &
- / 10 / 10 (hyp.)
- / 18 / 17 (pre.) \\ \hline \end{tabular}
\end{table}
Table 1: **The IrokoBench datasets: dataset name, number of African languages covered, data split, and the subjects or domains covered. We included English, French, and Swahili in all benchmarks.**
 

Opus, Gemini-1.5-Pro, GPT-4-Turbo, and GPT-4o consistently outperform the open models on IrokoBench. The top-2 closed models achieve average performance scores ranging from \(52.9\) to \(59.0\) across all tasks. The performance gap between the best closed model (GPT-4o) and the best open model (Gamma 2 27B) is \(21.9\). Notably, the largest performance differences for _in-language_ setting are observed in the AfriMMLU and AfriMGSM tasks, where GPT-4o outperforms Gamma 2 27B by \(20.1\) and \(24.1\), respectively. For the AfriXNLI task, mT0-XXL-MT and Aya-101 perform better than bigger models like Section 2 27B and LLaMa 3.1 70B with 27B and 70B parameters respectively.

Majority of models perform worse for language promptingMost users would prefer to prompt in their native language; however, we find that almost all models we benchmark perform better with prompts translated into English. Only a few exceptions, such as GPT-4-Turbo, and GPT-4o, perform better in the in-language evaluation. Specifically, Command R and LLaMa 3.1 8B benefit the most from the translate-test approach, showing average improvements of \(+19.8\) and \(+13.3\), respectively. Notably, Section 2 27B achieves the best overall results for the AfriMGSM task with the translate-test, outperforming GPT-4o by \(+3.5\). We attribute this boost to the fact that these models are heavily English-centric.

### Task-specific results

Here, we examine the individual language performance per task, comparing which task prefers _in-language v.s. translate-test_ evaluation. We compare the performance on a subset of LLMs: Aya 

101, Gemma 2 27B LLAMa 3.1 7OB, and GPT-4o. Other LLMs are in Appendix A.4.

AfriMMLU evaluation is better when promptting in-language for closed LLMsTable 3 shows the result of different LLMs on AfriMMLU using _in-language_ vs. _translate-test_. For GPT-4o, we found _in-language_ prompting to be generally better. However, for open models like LLaMa 3.1 7OB and Gamma 2 27B, we find a large improvement in the performance of the _translate-test_ for 15 out of 16 African languages. The only language that did not improve is wol, probably due to poor MT performance on NLLB-200, as reported in Costa-jussa et al. (2024). This shows the benefit of translate-test for prompting English-centric LLMs when evaluating low-resource languages.

AfriMMLU performance by subjectsTable 4 shows the result of GPT-4o by subjects. Interestingly, _elementary math_ achieved the best overall accuracy, where 13 out of 16 African languages achieved at least 70% despite struggling with AfriMGSM. The performance difference to AfriMGM may be due to the multi-choice options of MMLU, which may be slightly easier than freeform answer. _International law_ also achieved impressive performance with 10 out of 16 languages achieving 70%. All languages struggle the most with _Global facts_ including English and French. Similarly, many African languages find it difficult to answer questions in _geography_ and _microeconomics_ subjects. This presents an opportunity for improving LLMs for the education domain.

AfriMGGN performance receives significant boost with translate-test for open modelsTable 5 shows that we can achieve a significant boost in performance with the translate-test on all lan 

guages we evaluated. We hypothesize that the current LLMs are better at reasoning in English than other languages. Gemma 2 27B and LLaMa 3.1 70B improved by \(+17.6\) and \(+21.0\) respectively when questions are asked in English, while closed models like GPT-4o dropped in performance.

### Few-shot results, Cross-Lingual Transfer, Sensitivity to Prompt Templates

Cross-lingual transfer in-language achieves better resultsWhen there is large enough labeled data in English, we could leverage this cross-lingual signal for zero-shot evaluation. We trained on 400k English NLI examples, and we performed zero-shot transfer in in-language and translate-test setting. Table 6 shows that cross-lingual transfer using an Africa-centric smaller language model (AfroXLMR-76L) gave better results than prompting LLMs on average. AfroXLMR-76L has been pre-trained on all languages in AfriXNLI, which explains the impressive performance. However, for multilingual encoders that have not seen some of the languages, prompting GPT-4o seems to be better, as shown in Appendix A.6.

Impact of few-shot vs zero-shotFigure 2 shows the few-shot results for the IroxoBench datasets leveraging Gemma 2 27B, LLaMa 3.1 70B and GPT-4o when we provide few examples in in-language setting. We found out that Gemma 2 27B and LLaMa 3.1 70B LLM consistently benefited the least from additional few shots examples for classification tasks (AfriXNLI and AfriMMLU) where Gemma 2 27B improved by \(+13.2\) and \(+4.9\) respectively. Similarly, LLaMa 3.1 70B improved on both AfriXNLI and AfriMMLU tasks with \(+10.7\) and \(+7.0\). However, for reasoning tasks, only GPT-4o improved in performance by \(6.3\), other LLMs dropped in performance, probably due to their inability to reason in non-English languages. Surprisingly, GPT-4o did not benefit from additional examples for the classification tasks.

Sensitivity to prompt templatesTo understand whether sensitivity to prompts impacts results, we perform an ablation for all the IrokoBench tasks where we evaluate the performance of five different prompts (see subsection A.2). Table 7 shows the results of five prompts we tested for (three of the prompts most preferred by different models are

\begin{table}
\begin{tabular}{l c c c c c|c c c c c c|c c c c c c c} \hline  & \multicolumn{5}{c}{AfrixNLI} & \multicolumn{5}{c}{AfriMMLU} \\
**Model** & **t1** & **t2** & **t3** & **t4** & **t5** & **Ave.** & **t1** & **t2** & **t3** & **t4** & **t5** & **Ave.** & **t1** & **t2** & **t3** & **t4** & **t5** & **Ave.** \\ \hline Aya-101 & 47.79 & 49.49 & 45.7 & **51.5** & 48.3 & 46.82\({}_{2.2}\) & 29.6 & **29.7** & **29.7** & **29.7** & 29.6 & 29.7\({}_{1.0}\) & **1.4** & 4.1 & 4.2 & **4.4** & 4.3 & 43.0\({}_{1.0}\) \\ Gemma 2 27B & 38.3 & **40.3** & 33.4 & 34.1 & 35.4 & 36.32\({}_{3.2}\) & **39.9** & 39.2 & 38.7 & 39.1 & 39.2\({}_{2.0}\) & 22.3 & 26.3 & 25.0 & **27.0** & **26.2** & 25.4\({}_{1.19}\) \\ GPT-4o & 55.3 & 49.6 & **64.3** & 62.9 & 59.7 & 58.44\({}_{4.6}\) & **60.0** & 55.8 & 59.0 & 39.6 & 59.6 & 54.8\({}_{4.8r.8r.4}\) & 46.7 & **49.8** & 48.0 & 49.5 & 49.6 & 48.7\({}_{41.3}\) \\ \hline \end{tabular}
\end{table}
Table 7: **Ablation results: Effect of using different prompts (t1, t2, t3, t4, and t5) for IroxokoBench datasets. Best prompt results are in bold. Average computed on only African languages. Second best prompt are underlined.**

\begin{table}
\begin{tabular}{l l} \hline \hline
**Prompt 12** \\ ((premise)) \\ Question: (Hypothesis) True, False, or Neither? Answer: \\ \hline
**Prompt 13** \\ Given the following premise and hypothesis in English, identify if the premise entails, contradicts, or is neutral towards the hypothesis. Please respond with exact ‘entailment’, ’contradiction’, or ’neutral’. \\ Prenise: ('Ormseis’) \\ Hypothesis: (Hypothesis) \\ \hline
**Prompt 14** \\ You are an expert in Natural Language Inference (NLI) specializing in the (Language) language. Analyze the premise and hypothesis given in (Language), and determine the relationship between them. Respond with one of the following options: ”entailment”, ‘ contradiction’, or ’neutral’. \\ Premise: ('Ormseis’) \\ Hypothesis: ('Dynthesis) \\ \hline \hline \end{tabular}
\end{table}
Table 8: Three different prompts preferred by different models for AfriXNLI 

shown in Table 8). On AfriXNLI, we find that simpler prompt Nie et al. (2020) i.e. {(premise}) Question: {(hypothesis)} True, False, or Neither?, have better results for the open models like Aya-101 and Gemma 2 27B, while GPT-4o prefers t3 where a detailed task description is provided. The best prompt for Aya-101 is t4 where the {(language)} name is mentioned, which shows additional language information may be useful in improving performance.

On AfriMMLU, we find GPT-4o perform worse for t4.6. However, other models are not very sensitive to the use of different prompts. In general, we do not find AfriMGSM to be sensitive to different prompts. In subsection A.4, we provide the results of five prompt templates for all LLMs evaluated.

Footnote 6: Analyze each question critically and determine the most correct option based on your understanding of the subject matter ” Question: {question}. Choices A: {choice1}, B:{choice2}, C: {choice3}, D: {choice4}

## 5 Conclusion

In this paper, we introduced IrokoBench, a new benchmark for evaluating large language models (LLMs) on African languages. IrokoBench comprises three datasets focused on different tasks: natural language inference (AfriXNLI), multi-choice knowledge QA (AfriMMLU), and mathematical reasoning (AfriMGSM). Unlike previous benchmarks, which primarily involve simple text classification tasks, these datasets assessed the LLMs' abilities in complex and knowledge-intensive areas. Our evaluation revealed a significant performance gap between high-resource languages (_e.g.,_ English and French) and African languages. Additionally, we observed a substantial disparity in performance between open models and proprietary models, with the latter generally outperforming the former, particularly in mathematical reasoning tasks. We hope that IrokoBench will serve as a valuable benchmark for evaluating future LLMs developed or adapted for African languages.

LimitationsOur benchmark has a few limitations: (1) **The benchmark is human-translated** which may include some translationese effects, it would have been better if they are all generated in the native African languages. However, this parallel translation allows us to evaluate and compare the same sentences in all these languages. (2) **We only cover three language families in Africa**, Nilo-Saharan, Austronesian, and Khoisan language groups are missing, one of the reason we excluded them is either lack of contact with professional translators or limited translation budget, we hope to extend to more languages in the future.

## Acknowledgment

This work was supported by Lacuna Fund, an initiative co-founded by The Rockefeller Foundation, Google.org, and Canada's International Development Research Centre. Additional support was provided through compute credits from Oracle and the Cohere For AI Research Grant. We extend our gratitude to Charles Riley for facilitating our connection with the Vai translator for the MGSM data translation.

We are also deeply thankful to OpenAI for granting API credits through their Researcher Access API program to Masakhane, enabling the evaluation of GPT-3.5 and GPT-4 LLMs. Similarly, we appreciate Google for providing GCP credits via the Gemma 2 Academic Program, which supported the Gemini-1.5-Pro inference. Lastly, we would like to thank Hailey Schoelkopf and Lintang Sutawika for their invaluable assistance with the EleutherAI _lm-eval_ tool.

## References

* Adebara et al. [2023] Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Alcides ALcoba Inciare. 2023. SERENGETI: Massively multilingual language models for Africa. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 1498-1537, Toronto, Canada. Association for Computational Linguistics.
* Addelani et al. [2023] David Addelani, Jesujoba Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajuddeen Gowadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong, Michael Beukman, Shamsudeben Muhammad, Guyo Jarso, Oreen Yousuf, Andre Niyongabo Rubung, Gilles Hacheme, Eric Peter Wairagala, Muhammad Umair Nasir, Benjamin Ajibade, Trunde Ajayi, Yvonne Gitua, Jade Abbott, Mohamed Ahmed, Millicent Ochieng, Anouluwapo Aremu, Perez Ogayo, Jonathan Mukibi, Fatouramata Ouoba Kabore, Godson Kalipe, Derguene Mbaye, Allahsera Auguste Tapo, Victoire Memdjokam Koagne, Edwin Munkoh-Buabeng, Valencia Wagner, Idris Abdulmumin, Ayodele Awokoya, Happy Buzabaaba, Blessing Sibanda, Andiswa Bucula, and Sam Matthaila. 2022A. A few thousand translations go a long way! leveraging pre-trained models for African news translation. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: 

Human Language Technologies_, pages 3053-3070, Seattle, United States. Association for Computational Linguistics.
* Adelani et al. (2024) David Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassijyev, Jesujoba Alabi, Yanke Mao, Haonan Gao, and En-Shiun Lee. 2024. SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 226-245, St. Julian's, Malta. Association for Computational Linguistics.
* Adelani et al. (2021) David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijnwani, Michael Beukman, Chester PalenMichel, Constantine Lignos, Jesusjoba Alabi, Shamasuddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukhula, Rooweither Mabuya, Bonaventure F. P. Dossou, Bessing Sibanda, Happy Buzabaq, Jonathan Mukibii, Godson Kalipe, Berguene Mbye, Amelia Taylor, Fatounata Kabore, Chris Chinenye Emezue, Anoluwapo Aremu, Perez Ogayo, Catherine Gitau, Edwin Munkoh-Baugeng, Victoire Memdjokam Koagne, Allahsera Augustte Tapo, Tebogo Macucwa, Vukosi Marivate, Mohoning Tchiaze Elvis, Tajuddeen Gowadabe, Tosin Adewumi, Orevaoghene Ahia, and Joyee Nakatumba-Naberbe. 2021B. MasakhanNER 2.0: Africa-centric transfer learning for named entity recognition. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 4488-4508, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Adelani et al. (2021) David Ifoulawa Adelani, Jade Abbott, Graham Neubig, Daniel D'souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijnwani, Sebastian Ruder, Stephen Mayhew, Israel Abehr Azimee, Shamsulddeem H. Muhammad, Chris Chinney Emezue, Joyce Nakatumba-Naberbe, Perez Ogayo, Aremn Anouhuvapo, Catherine Gitau, Derguene Mbaye, Jesuioba Alabi, Seid Muhie Yimam, Tajuddeen Rabiu Gowadabe, Ignatius Ezeani, Rubung Andre Niyongabo, Jonathan Mukibi, Verrah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Ryson, Mofetoluwa Adeyemi, Gerald Muritvik, Emmanuel Anebi, Chiamak Chukwankee, Nikurka Odu, Eric Peter Wariagala, Samuel Overinde, Clemencia Siro, Tobius Saul Batesea, Temilola Oloyede, Yvonne Wambui, Victor Akindoe, Deborah Nabagerka, Maurice Katussime, Ayodele Awokoya, Mohammadane MBOUP, Dibora Grebreydannes, Henko Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Oregagne Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahim DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Salomey Osei. 2021. MasakhaNER: Named entity recognition for African languages. _Transactions of the Association for Computational Linguistics_, p:1116-1131.
* Adelani et al. (2021) David Ifeoluwa Adelani, Marek Misiak, Israel Abebe Azime, Jesujoba Alabi, Atnafu Lambebo Tonja, Christine Mwase, OdunayO Ogundepo, Bonaventure F. P. Dossou, Akintunde Doladipo, Lorecn Nixdorf, Chris Chinenye Emezue, Sana Al-zzawi, Blessing Sibanda, Disuda Daviy Laroloveth Mololela, Jonathan Mukibii, Trude Ajayi, Tainana Motcu, Brian Odhambo, Abraham Owodunni, Nnaemeka Obiefuna, Muhidin Mohamed, Shamsudden Hassan Muhammad, Teshome Mulugeta Abbau, Saehed Abdulali Salahudeen, Mesay Gemeda Yigezu, Tajuddenen Gwalobe, Idris Abdulummun, Mahlet Taye, Ohluawbasuayo Avyopomi, Ivanquinwa Shode, Tolloope Adelani, Habiba Abdulganiyu, Abdul-Hakeem Omotayo, Adetola Adeeko, Abeeb Afolabi, Anouluwapo Aremu, Olannewaji Samuel, Clemencia Siro, Wangari Kimotho, Onyekachi Ogbu, Clinedu Mbonu, Chiamaka Chukwuweke, Samuel Fanija, Jessica Ojo, Oynkinasalo Awousa, Tadessee Rebede, Toadoum Sarl Sakaya, Pamela Nystaine, Freedmore Sidume, Oreen Yousuf, Maryiyyah Odwole, Kanda Tshinu, Ussen Kirmanuka, Thina Diko, Siyanda Naxakama, Sinodos Nigusse, Abdulmejjid Johar, Shafie Mohamed, Fuad Mirre Hassan, Moges Ahmed Mehamed, Evrard Nagphire, Jules Jules, Ivan Seskenugu, and Pontuts Stetentorp. 2023. MasakhaNEWS: News topic classification for African languages. In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 144-159, Nusa Dua, Bali. Association for Computational Linguistics.
* Ahuja et al. (2023a) Kabir Ahuja, Harshitz Diddeee, Rishav Hada, Milliccent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023a. MEGA: Multilingual evaluation of generative AI. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 4232-4267, Singapore. Association for Computational Linguistics.
* Ahuja et al. (20223b) Sanchit Ahuja, Divyanshu Aggarwal, Varun Gunma, Ishaan Watts, Ashutosh Sathe, Milliccent Ochieng, Rishaw Hada, Prachi Jain, Maxamed Axmed, Kalika Bali, and Sunayana sitaram. 2023b. Megexverse: Benchmarking large language models across langaugases, modalities, models and tasks. _ArXiv_, abs/2311.07463.
* Alabi et al. (2022) Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022. Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 4336-4349, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
* Anthropic (2024) Anthropic. 2024. Claude -- anthropic.com. [https://www.anthropic.com/claude](https://www.anthropic.com/claude). [Accessed 01-06-2024].

 

* Aremu et al. (2023) Anuoluwapo Aremu, Jesujoba O. Alabi, and David Ifeolouwa Adelani. 2023. Yorc: Yoruba reading comprehension dataset. _Preprint_, arXiv:2308.09768.
* Azime et al. (2024) Israel Abebe Azime, Mitiku Johannes Fuge, Atanfu Lambebo Tonja, Tadesse Destaw Belay, Aman Kassahun Wassie, Eyasu Shiferaw Jada, Yonas Chanie, Walalign Tewabe Sewneutie, and Seid Muhie Yimam. 2024. Enhancing amharic-llama: Integrating task specific and generative datasets. _arXiv preprint arXiv:2402.08015_.
* Bandarkar et al. (2023) Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023. The belebebe benchmark: a parallel reading comprehension dataset in 122 language variants. _Preprint_, arXiv:2308.16884.
* Bang et al. (2023) Yejin Bang, Samuel Cahyaviijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tizeheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 675-718, Nusai Dua, Bali. Association for Computational Linguistics.
* Biderman et al. (2024) Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Albam Flixi Aji, Pawan Sasank Ammannanchio, Sidney Black, Jordan Clive, Anthony DiPoft, Julen Etxraniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lowering, Niklas Muennighoff, Ellile Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang, Genta Indra Winata, Francois Yvon, and Andy Zou. 2024. Lessons from the trenches on reproducible evaluation of language models. _Preprint_, arXiv:2405.14782.
* Boyszkiewicz et al. (2023) BigScience-Workshop, ; Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagne, Alexandra Sasa Luccioni, Francois Yvon, Matthias Galde, Jonathan Tow, Alexander M. Rush, and et al. 2023. Bloom: A 176b-parameter open-access multilingual language model. _Preprint_, arXiv:2211.05100.
* Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
* Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. _Preprint_, arXiv:2005.14165.
* Cohere (2024)Cohere. 2024. Command R -- cohere.com. [https://cohere.com/command](https://cohere.com/command). [Accessed 01-06-2024].
* Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzzhan, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451, Online. Association for Computational Linguistics.
* Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
* Costa-jussa et al. (2024) Marta R Costa-jussa, James Cross, Onur Celebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, AI Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Gosumi, Francisco Guzman, Philipp Koehn, Alexandre Mourachko, Christophe Rogers, Safiyyah Saleem, Holger Schwenk, Jeff Wang, and NLLB Team. 2024. Scaling neural machine translation to 200 languages. _Nature_, 630(8018):841-846.
* Bolone et al. (2020) Cheikh M. Bamba Dione, David freoluwa Adelani, Peter Nabende, Jesusijoba Alaibi, Thapelo Sindane, Happy Buzaaba, Shamsuddeeen Hassan Muhammad, Chris Chinenye Emezue, Perez Ogayo, Anouluvapog Aremu, Catherine Gitau, Derguene Mbaye, Jonathan Mukibih, Blessing Sibanda, Bonaventure F. P. Dossou, Andiswa Bukuka, Rooweiher Mabuya, Allahsera Auguste Tapo, Edwin Munkoh-Baubeng, Vicotiore MemdjobAM Koagne, Fatounata Ouoba Kabbore, Amelia Taylor, Godson Kalipe, Tehogo Macucwa, Vukosi Marivate, Tajudden Gowadabe, Mobning Tch Maize Elvis, Ikechukwu Onyenwe, Gratien Aidnodgebe, Tolulope Adelani, Idris Akinade, Olannewwaji Samuel, Marien Nahiman, Theogene Musabeyezu, Emile Niyomutabazi, Ester Chinnhenga, Kudzi Gotosa, Patrick Mizha, Apelete Agbolo, Seydou Traore, Chinedu Uchechukwu, Aliyu Yusuf, Muhammad Abdullahi, and Dietrich Klakov. 2023 

MasakhaPOS: Part-of-speech tagging for typologically diverse African languages. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers)_, pages 10883-10900, Toronto, Canada. Association for Computational Linguistics.
* Dubey et al. [2024] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, and et al. 2024. The llama 3 herd of models. _ArXiv_, abs/2407.21783.
* Ferrara [2023] Emilio Ferrara. 2023. Should chatgpt be biased? challenges and risks of bias in large language models. _arXiv preprint arXiv:2304.03738_.
* Gao et al. [2021] Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3816-3830, Online. Association for Computational Linguistics.
* Gezini-Team et al. [2024] Gemini-Team, Michel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry, Lepikhin, Timothy Lillicrap, Jean Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, and et al. 2024. Gemini 1:5. Unlocking multimodal understanding across millions of tokens of context. _Preprint_, arXiv:2403.05530.
* Goyal et al. [2022] Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzato, Francisco Guzzan, and Angelia Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. _Transactions of the Association for Computational Linguistics_, 10:522-538.
* Hartung et al. [2023] Kai Hartung, Aatrici Herygers, Shubham Kurlekar, Khabbab Zakaria, Taylan Volkan, Soren Grottrup, and Munir Georges. 2023. Measuring sentiment bias in machine translation. _Preprint_, arXiv:2306.07152.
* Hendrycks et al. [2021a] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring massive multitask language understanding. _Proceedings of the International Conference on Learning Representations (ICLR)_.
* Hendrycks et al. [2021b] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the MATH dataset. In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
* Hendy et al. [2023] Amr Hendy, Mohamed Gomaa Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hans Hassan Awadalla. 2023. How good are gpt models at machine translation? a comprehensive evaluation. _ArXiv_, abs/2302.09210.
* Ji et al. [2023] Meng Ji, Meng Ji, Pierrette Bouillon, and Mark Seligman. 2023. _Cultural and Linguistic Bias of Neural Machine Translation Technology_, page 100-128. Studies in Natural Language Processing. Cambridge University Press.
* Jiang et al. [2024] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lello Renard Lavaud, Lucile Sauliner, Mariene Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Syzrom Antoniak, Teven Le Scao, Theophile Gervet, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2024. Mixtral of experts. _Preprint_, arXiv:2401.04088.
* Joshi et al. [2020] Pratik Joshi, Sebastian Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6282-6293, Online. Association for Computational Linguistics.
* Kihu et al. [2024] Eric Kihu, Hasti Toossi, David Anugraha, Jinyu Liu, Jiaxu Li, Juan Flores, Leandro Roman, A. Seza Dogruoz, and En-Shiun Lee. 2024. Predicting machine translation performance on low-resource languages: The role of domain similarity. In _Findings of the Association for Computational Linguistics: EACL 2024_, pages 1474-1486, St. Julian's, Malta. Association for Computational Linguistics.
* Kreutzer et al. [2023] Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahshan Wahab, Daan van Esch, Nasanbayar Ulzi-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyuan, Supheakhmugkol Sarin, Sokhar Samb, Benoit Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubupso, Toan Q. Nguyen, Mathias Muller, Andre Muller, Shamusddeeen Hassan Muhammad, Nanda Muhmammad, Ayanda Mnyekeni, Jamshidbek Mirzakhalov, Taipwanase Matangira, Colin Leong, Nze Lawson, Sneha Kudogunta, Yacine Jenrite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Skahile Damini, hisansa de Silva, Sakine Cabuk Ball, Stella Biderman, Alessia Battisti, Ahmed Barwa, Ankur Bapna, Pallavi Bajkedar, Israel Abebe Azime, Ayodee Awokoya, Duygu Ataman, Orevaoghene Alais, Oghenefego Ahia, Sweta Agrawal, and Moretoluwaya Adegemii. 2022. Quality at a glance: An audit of web-crawled multilingual datasets. _Transactions of the Association for Computational Linguistics_, 10:50-72.
* Kutaguutta et al. [2023] Sneha Kuduguuta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A. Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2023. Madlad-400: 

A multilingual and document-level large audited dataset. _Preprint_, arXiv:2309.04662.
* Lai et al. (2023a) Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023a. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 318-327, Singapore. Association for Computational Linguistics.
* Lai et al. (2023b) Viet Mac Lai, Nghia Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023b. ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 13171-13189, Singapore. Association for Computational Linguistics.
* Lee et al. (2022) En-Shiun Lee, Sarubi Thillainathan, Shravan Nayak, Surangika Ranathunga, David Adelani, Ruisi Su, and Arya McCarthy. 2022. Pre-trained multilingual sequence-to-sequence models: A hope for low-resource language translation? In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 58-67, Dublin, Ireland. Association for Computational Linguistics.
* Lu et al. (2024) Yinquan Lu, Wenhao Zhu, Lei Li, Yu Qiao, and Fei Yuan. 2024. LIamax: Scaling linguistic horizons of llm by enhancing translation capabilities beyond 100 languages. _arXiv preprint arXiv:2407.05975_.
* Luccioni and Viviano (2021) Alexandra Luccioni and Joseph Viviano. 2021. What's in the box? an analysis of undesirable content in the Common Crawl corpus. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, pages 182-189, Online. Association for Computational Linguistics.
* Ma et al. (2023) Chunlan Ma, Ayyoob ImaniGooghari, Haotian Ye, Ehsaneddin Asgari, and Hinrich Schutze. 2023. Taxi1500: A multilingual dataset for text classification in 1500 languages. _Preprint_, arXiv:2305.08487.
* Meta (2024)Meta. 2024. Introducing Meta LIama 3: The most capable openly available LLM to date -- ai.meta.com. [https://ai.meta.com/blog/meta-llama-3/](https://ai.meta.com/blog/meta-llama-3/). [Accessed 01-06-2024].
* Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almburak, Samuel Albaniez, Zaid Ajfaretai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15991-16111, Toronto, Canada. Association for Computational Linguistics.
* Muhammad et al. (2023) Shamsudeen Muhammad, Idris Abdulumin, Abinew Apele, Nedjma Ousidinou, David Adelani, Seid Yimam, Ibrahim Ahmad, Meriem Beloucif, Saif Mohammad, Sebastian Ruder, Oumaima Hourrane, Alipio Jorge, Pavel Brazzdil, Felermino Ali, Davis David, Salomey Osei, Bello Shehu-Bello, Falalu Lawn, Tajuddeen Gwaude, Samuel Rutunda, Tadesce Destaur Belay, Wendimin Messelle, Hailu Balcha, Sisya Chala, Hagos Gebremichael, Bernard Opoku, and Stephen Arthur. 2023. AfriSenti: A Twitter sentiment analysis benchmark for African languages. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 13968-13981, Singapore. Association for Computational Linguistics.
* Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4885-4901, Online. Association for Computational Linguistics.
* Ojo et al. (2023) Jessica Ojo, Kelecchi Ogueiji, Pontus Stenetorp, and David I Adelani. 2023. How good are large language models on african languages? _arXiv preprint arXiv:2311.07978_.
* OpenAI (2024) OpenAI. 2024. Introducing ChatGPT. [https://openai.com/index/chatgpt/](https://openai.com/index/chatgpt/). [Accessed 01-06-2024].
* OpenAI et al. (2024) OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lamas Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Atlenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, and et al. 2024. GPT-4 technical report. _Preprint_, arXiv:2303.08774.
* Ponti et al. (2020) Edorado Maria Ponti, Goran Glavas, Olga Majewska, Qianchu Liu, Ivan Vulic, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2362-2376, Online. Association for Computational Linguistics.
* Pudijati et al. (2022) Danti Pudijati, Ninuk Lustyantie, Ifan Iskandar, and Tira Nur Fitria. 2022. Post-editing of machine translation: Creating a better translation of cultural specific terms. _Language Circle: Journal of Language and Literature_, 17(1):61-73.
* Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2685-2702, Online. Association for Computational Linguistics.

 

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean-Baptiste Alayrac, Radt Soricu, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sotitaux, and et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. _ArXiv_, abs/2403.05530.
* Riviere et al. (2024) Gemma Team Borgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatriraju, Le'onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram'e, Johan Ferret, Peter Liu, Pouya Dehghani Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravi Kumar, Charline Le Lan, Sammy Jerome, and et al. 2024. Gemma 2: Improving open language models at a practical size. _ArXiv_, abs/2408.00118.
* Savoldi et al. (2021) Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2021. Gender bias in machine translation. _Preprint_, arXiv:2104.06001.
* Schick and Schulze (2021) Timo Schick and Hinrich Schutze. 2021. It's not just size that matters: Small language models are also few-shot learners. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2339-2352.
* Shi et al. (2023) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivat s, Sorush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. Language models are multilingual chain-of-thought reasoners. In _The Eleventh International Conference on Learning Representations_.
* Singh et al. (2022) Shivalika Singh, Fredie Vargus, Daniel Dsouza, Borje F. Karlsson, Abinaya Mahendrian, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deviidas Maciunas, Laura OMahoary, Mike Zhang, Ramith Hetitarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzeminski, Hakimeh Fadaei, Irem Ergun, Ifeoma Okoh, Aisha Alaagib, Oshan Mudanmayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surva Guthkonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Ustun, Marzieh Fadaee, and Sara Hooker. 2024. AyA dataset: An open-access collection for multilingual instruction tuning. _Preprint_, arXiv:2402.06619.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca).
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahaiiri, Yasmine Babaei, Nikolay Bashbykov, Souumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, and et al. 2023. LIAA 2: Open foundation and fine-tuned chat models. _Preprint_, arXiv:2307.09288.
* Ustun et al. (2024) A. Ustun, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bharadi, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction finetuned open-access multilingual language model. _ArXiv_, abs/2402.07827.
* Ustun et al. (2021) Ahmet Ustun, Viraat Aryabumi, Zheng-Xin Yong, Weij- Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bahandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Numeaninghoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction finetuned open-access multilingual language model. _Preprint_, arXiv:2402.07827.
* Vanmassenhove et al. (2021) Eva Vanmassenhove, Dimitar Shterionov, and Matthew Grilliam. 2021. Machine translationese: Effects of algorithmic bias on linguistic complexity in machine translation. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 2203-2213, Online. Association for Computational Linguistics.
* Wang et al. (2021) Jiayi Wang, David Adelani, Sweta Agrawal, Marek Misiak, Ricardo Rei, Elfehieria Briakou, Marine Carpuat, Xuanfi He, Sofia Bourhim, Andiswa Bakula, Muhidin Mohamed, Temitayo Olatoye, Tosin Adewumi, Hamam Mokaped, Christine Musse, Wangui Kimotho, Foutes Yuebogh, Anouluwapo Aremu, Jessica Ojo, Shamysudade Muhammad, Salomey Osei, Abdul-Hakcem Omotoya, Chiamak Chukwunekee, Perez Ogayo, Ouraima Hourrane, Salma El Anigri, Lohvethu Ndolela, Thabiso Mangwana, Shafie Mohamed, Hassan Ayinde, Ohuvabusayo Awoyomi, Lama Alkhaled, Sana Al-azzawi, Naome Etori, Millican Obeling, Clemencia Siro, Njoroge Kiragu, Eric Muchir, Wangari Kimtoho, Toadowsar Sarif Sakaya, Lyse Naomi Warba, Daoud Apbolate, Simbiat Ajao, Ivanquoliuwa Shode, Ricky Macharm, Ruqayya Ito, Saheed Abdullahi, Stephen Moore, Bernard Opoku, Zainab Akinjobi, Abeeb Afollabi, Nnameka Obiefuna, Onyekachi Ogbu, Sam Ochiang', Verrah Otietone, Chinedu Mboun, Yao Lu, and Pontus Stenetorp. 2024. AfriMTE and AfricOMET: Enhancing COMET to embrace under-resourced African languages. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 5997-6023, Mexico City, Mexico. Association for Computational Linguistics.
* Wang et al. (2022) Jun Wang, Benjamin Rubinstein, and Trevor Cohn. 2022. Measuring and mitigating name biases in 

neural machine translation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2576-2590, Dublin, Ireland. Association for Computational Linguistics.
* Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in neural information processing systems_, 35:24824-24837.

## Appendix A Appendix

### Language covered

Table 9 provide the languages covered in the IrokoBench, their language family, regions located in Africa, number of speakers, and size of monolingual data available on the web based on the MADLAD cleaned corpus Kudugunta et al. [2023]--we only report number of characters in mega bytes. Additionally, we added an indication whether this language is covered in the pre-training of Aya-101 and BLOOMZ 7B LLMs.

### Prompts Template and Evaluation Tool

We use the EleutherAI LM Evaluation Harness (lm-eval) tool Biderman et al. [2024]--a popular evaluation tool that is helping to standardize LLM evaluation. The tool allows for three types of evaluation: _log-likelihood_, _perplexity_, and _generation_. The log-likelihood is more suitable for multiple-choice tasks since it helps to restrict the model's option to fewer choices--more appropriate for weaker models. However, the log-likelihood approach cannot evaluate the generative capabilities of LLMs to generate coherent and relevant answers. Moreover, closed models are only accessible via API and do not provide access to the log probabilities, making it impossible to use the log-likelihood approach. To extract the correct answers for the task, we employed a **verbalizer**Gao et al. [2021]; Schick and Schutze [2021]. For AfriMGSM, we used the default verbalizer provided by the tool. However, for AfriXNLI and AfriMMLU, we manually created a verbalizer for the closed models and used the _log-likelihood_ request type for the open models.

The prompt templates used for evaluation of different tasks are in Table 11, Table 12 and Table 10.

### AfriCOMET metric scores for XNLI translation

We employ AfriCOMET evaluation metrics, as developed by Wang et al. [2024], to automatically assess the quality of translations for our newly created benchmarks. Figure 3 depicts the histogram of scores obtained from AfriCOMET for AfriXNLI, illustrating promising results and offering compelling evidence for the effectiveness of our translations Amharic, Yoruba, isizulu). However, the performance of this metric depends on if the language we are evaluating is covered in the pre-training of the base model of the metric i.e. AfroXLMR-large. In the case of Lingala, Twi and Wolf, the performance of the metric does not correlate with the human translation since they are not covered in AfroXLMR. Similar findings were reported in the original AfriCOMET QE paper that Twi had worse correlation with human judgement (_i.e.,_ 0.279 for Pearson, and 0.060 for Spearman) Wang et al. [2024].

### Task-specific results for all models

We provide the entire results of all LLMs and all prompts on AfriMMLU, AfriXNLI and AfriMGSM tasks are shown in Table 14, Table 13 and Table 15. We performed evaluation on 5 prompts for all models except Claude Opus which we limit to one prompt due to API inference cost.

### Comparison between in-language and translate-test results

We provide the results comparing the in-language and translate-test results on all LLMs in Table 16, Table 17, and Table 18.

### Cross-lingual transfer results for XNLI

In Table 19, we compare different multilingual masked language model (MLM) performance on African languages. XLM-R-large has 559M parameters and is trained on 100 languages, but only a few African languages are covered (amh, hau, orm, swa, and xho). Serengeti, on the other hand, has been pre-trained on all languages in IrokoBench, but it only has 240M parameters. AfroXLMR was adapted from XLM-R through continual pretraining on 17 African languages including 11 in IrokoBench (amh, nau, ibo, kin, orm, sna, sot, swa, xho, yor, and zul). AfroXLMR-7GL follows the same technique by performing continual pre-training on XLM-R-large on 76 languages (72African), all languages covered in IrokoBench are part of its pre-training.

We found Africa-centric MLM to perform better on average than massively multilingual models like XLM-R-large. Serengeti and AfroXLMR 

base improved over larger-sized XLM-R-large by \(+3.5\) and \(+5.3\) points, respectively. Similarly, fine-tuning AfroXLMR-large, a larger version of AfroXLMR-base, results in an improved boost in performance with \(11.8\) points. The best overall results were achieved by AfroXLMR-76L with a \(16.1\) boost in performance over XLM-R-large. This is probably because all the languages are used in pretraining. We make use of AfroXLMR-76L has the baseline for all LLMs. Interestingly, we find GPT-4o to be competitive or better than other MLMs except the AfroXLMR-76L on average.

Figure 3: Evaluation of **AfriXNLI** translations using AfriCOMET metric scores.

 



\begin{table}
\begin{tabular}{l l



**Prompt 1**

Please identify whether the premise entails or contradicts the hypothesis in the following premise and hypothesis. The answer should be exact entailment, contradiction, or neutral.

Premise: (premise)

Hypothesis: {hypothesis} Is it entailment, contradiction, or neutral?

**Prompt 2**

{(premise)}

Question: {hypothesis} True, False, or Neither? Answer:

**Prompt 3**

Given the following premise and hypothesis in English, identify if the premise entails, contradicts, or is neutral towards the hypothesis. Please respond with exact 'entailment', 'contradiction', or 'neutral'.

Premise: {premise} Hypothesis: {hypothesis}

**Prompt 4**

You are an expert in Natural Language Inference (NLI) specializing in the {Language} language. Analyze the premise and hypothesis given in {language}, and determine the relationship between them. Respond with one of the following options: ''entailment', 'contradiction', or 'neutral'.

Premise: (premise)

Hypothesis: {hypothesis}

**Prompt 5**

Based on the given statement, is the following claim 'true', 'false', or 'inconclusive'.

Premise: (premise)

Hypothesis: {hypothesis}





**Prompt 1**

You are a highly knowledgeable and intelligent artificial intelligence model answers multiple-choice questions about (subject)

Question: {question} A: {choice1} B: {choice2} C: {choice3} D: {choice4} Answer:

**Prompt 2**

As an expert in (subject), choose the most accurate answer to the question below. Your goal is to select the correct option 'A', 'B', 'C', or 'D' by understanding the nuances of the topic.

Question: {question} A: {choice1} B: {choice2} C: {choice3} D: {choice4} Answer:

**Prompt 3**

You are a subject matter expert in (subject). Utilizing your expertise in (subject), answer the following multiple-choice question by picking 'A', 'B', 'C', or 'D'.

Question: {question} A: {choice1} B: {choice2} C: {choice3} D: {choice4} Answer:

**Prompt 4**

Analyze each question critically and determine the most correct option based on your understanding of the subject matter

A: {choice1} B: {choice2} C: {choice3} D: {choice4} Answer:

**Prompt 5**

Given your proficiency in (subject), please answer the subsequent multiple-choice question with 'A', 'B', 'C', or 'D'.

Question: {question} A: {choice1} B: {choice2} C: {choice3} D: {choice4} Answer:

\begin{table} \end{table}
Table 12: Five different prompt used for prompt sensitivity experiments in AfriMMLU


\begin{tabular}{l|c c c c c c c c c c c c c c c c c c c c c c c c c} \hline




\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c c c c c c c c c c c|c} \hline \hline





\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c c c c c c c c c c c c c} \hline
**Model** & **eng** & **fra** & **amh** & **ewe** & **hau** & **ibo** & **kin** & **lin** & **ling** & **orm** & **sma** & **sot** & **swa** & **twi** & **vael** & **wol** & **xho** & **yor** & **zul** & **ave** \\ \hline mT0-XXL-MT & & & & & & & & & & & & & & & & & & & & & & & & & & & \\ iH & 3.6 & 4.8 & 4.4 & 1.6 & 4.8 & 1.2 & 3.6 & 2.9 & 3.0 & 3.5 & 4.2 & 4.8 & 5.6 & 10.8 & 3.2 & 4.4 & 3.6 & 1.2 & 0.8 & 2 & 4.8 & 2 & 3.2 & 2.8 \\ t2 & 4.0 & 4.4 & 3.2 & 3.2 & 3.2 & 0.4 & 3.2 & 1.2 & 3.2 & 1.2 & 2.4 & 3.6 & 4.4 & 1.6 & 1.6 & 2.0 & 2.8 & 0.8 & 2.8 & 2.4 \\ t3 & 3.2 & 4.0 & 3.6 & 2.0 & 3.2 & 0.4 & 3.2 & 2.0 & 3.2 & 0.8 & 4.0 & 2.8 & 3.2 & 1.6 & 1.6 & 2.4 & 3.2 & 1.6 & 2.8 & 2.4 \\ t4 & 3.6 & 4.4 & 3.6 & 2.8 & 4.0 & 0.8 & 3.2 & 0.8 & 3.6 & 1.2 & 2.0 & 3.2 & 4.0 & 2.0 & 1.6 & 2.0 & 3.2 & 2.8 & 2.4 & 2.5 \\ t5 & 3.6 & 4.4 & 3.6 & 2.8 & 3.2 & 0.4 & 2.8 & 2.0 & 3.2 & 0.8 & 2.4 & 3.6 & 4.0 & 1.6 & 0.8 & 0.8 & 4.0 & 2.0 & 2.4 & 2.4 \\ wee & 3.6 & 4.4 & 3.7 & 2.5 & 3.7 & 0.6 & 3.2 & 1.6 & 3.4 & 1.0 & 2.8 & 3.4 & 3.8 & 1.6 & 1.3 & 1.8 & 3.6 & 1.8 & 2.7 & 2.5 \\ \hline Agui-101 & & & & & & & & & & & & & & & & & & & & & & \\ iH & 10.8 & 9.6 & 6.8 & 3.2 & 7.2 & 3.2 & 3.6 & 4.4 & 2.4 & 4.8 & 6.8 & 10.8 & 6.6 & 1.6 & 0.8 & 1.2 & 4.4 & 3.2 & 3.6 & 4.4 \\ t2 & 11.6 & 10.4 & 7.6 & 2.8 & 4.4 & 2.8 & 5.2 & 4.4 & 2.4 & 3.2 & 6.2 & 7.2 & 8.4 & 1.6 & 1.6 & 0.8 & 4.4 & 4.4 & 3.6 & 4.1 \\ t3 & 10.0 & 9.2 & 7.2 & 3.6 & 7.2 & 3.2 & 6.4 & 5.2 & 1.6 & 3.2 & 4.8 & 8.0 & 6.8 & 1.6 & 0.8 & 0.8 & 4.0 & 3.6 & 3.6 & 4.2 \\ \hline \end{tabular}
\end{table}
Table 14: **AfriMMLU results for all prompt templates and their averages. Best prompt in Gray**

41
* [id=0.8]
* [id=0.9]
* [id=0.7]
* [id=0.

 



\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c c c c c c c c c c} \hline \hline
**Model** & **eng** & _fra_ & **amh** & **ewe** & **han** & **ibo** & **kin** & **lin** & **ling** & **orma** & **sma** & **sat** & **swn** & **trui** & **wd** & **sho** & **yer** & **znd** & **ave** \\ \hline
**In-language** & & & & & & & & & & & & & & & & & & & & & & \\ afro-limr-large76L. & 882.3 & 78.5 & 58.3 & 73.3 & 70.0 & 65.8 & 33.3 & 68.0 & 69.3 & 70.8 & 70.8 & 70.3 & 59.5 & 51.8 & 73.0 & 63.2 & 72.5 & 65.7 \\ mTQ-XLMT (i2) & 63.5 & 61.5 & 58.3 & 38.5 & 57.5 & 56.5 & 51.3 & 33.5 & 54.2 & 47.2 & 54.8 & 56.0 & 55.5 & 48.8 & 39.7 & 56.3 & 52.3 & 55.0 & 51.0 \\ Avg-ill (q1) & 67.0 & 59.7 & 64.2 & 43.2 & 57.0 & 58.5 & 54.3 & 33.5 & 51.7 & 51.5 & 55.7 & 52.2 & 56.5 & 47.0 & 36.7 & 55.2 & 54.5 & 55.3 & 51.5 \\ BLOCOMZ T9 (G2) & 60.3 & 56.0 & 36.8 & 35.7 & 36.5 & 44.7 & 38.5 & 38.3 & 41.5 & 35.2 & 43.3 & 40.5 & 45.8 & 37.5 & 36.0 & 39.8 & 45.2 & 40.3 & 39.4 \\ LaMa 3 sB (k2) & 43.9 & 47.4 & 40.2 & 34.3 & 35.6 & 39.1 & 32.9 & 37.3 & 33.5 & 35.4 & 35.9 & 34.1 & 34.3 & 33.3 & 36.8 & 31.7 & 38.5 & 35.4 \\ LLMa 3 l 8B (l2) & 53.0 & 50.2 & 38.8 & 37.7 & 36.5 & 38.5 & 38.3 & 32.3 & 37.5 & 35.0 & 35.8 & 34.8 & 43.8 & 37.0 & 32.3 & 33.8 & 38.5 & 34.0 & 36.6 \\ LLAM X 38 h (l1) & 53.0 & 49.3 & 40.2 & 35.8 & 44.8 & 42.5 & 37.7 & 29.8 & 42.7 & 45.7 & 40.7 & 39.7 & 45.7 & 36.3 & 37.3 & 46.2 & 44.0 & 42.0 & 40.8 \\ Gemma 2 mg (k2) & 53.5 & 50.7 & 43.2 & 35.3 & 47.0 & 40.7 & 40.2 & 32.8 & 38.8 & 37.8 & 42.3 & 40.2 & 46.3 & 37.2 & 35.2 & 42.5 & 41.3 & 44.5 & 40.3 \\ German 2 mg (k2) & 67.8 & 63.3 & 47.0 & 36.8 & 49.7 & 46.2 & 40.5 & 30.2 & 41.7 & 35.6 & 46.0 & 43.5 & 57.0 & 36.0 & 36.7 & 45.2 & 45.0 & 42.8 & 42.8 \\ LLMa 3 l 70b (k2) & 57.3 & 50.7 & 43.2 & 34.3 & 42.2 & 48.3 & 36.5 & 32.8 & 37.5 & 34.7 & 34.5 & 38.3 & 44.0 & 36.0 & 34.7 & 39.3 & 39.0 & 37.0 & 38.0 \\ CommandR (Aug) (l3) & 79.9 & 75.7 & 45.7 & 38.9 & 45.1 & 42.4 & 45.6 & 34.5 & 39.0 & 44.8 & 48.9 & 45.1 & 45.3 & 41.8 & 34.5 & 45.9 & 45.9 & 40.8 & 43.4 \\ gpt-3-5-utube (l25 ufa) & 70.0 & 65.2 & 38.7 & 38.2 & 39.0 & 42.3 & 42.2 & 20.0 & 43.0 & 42.5 & 45.3 & 48.3 & 59.3 & 49.5 & 44.7 & 40.5 & 43.8 & 42.1 \\ glot-at-min-2024-07-18 (t3) & 86.0 & 82.0 & 52.7 & 64.8 & 57.2 & 68.3 & 51.5 & 51.7 & 61.3 & 63.0 & 56.2 & 67.5 & 47.5 & 43.2 & 64.5 & 54.2 & 50.2 & 54.2 \\ gpt+4urbo-2024-04-09 (t3) & 87.3 & 81.0 & 62.0 & 38.2 & 70.3 & 66.0 & 65.8 & 33.7 & 67.5 & 61.0 & 71.7 & 67.8 & 67.8 & 41.7 & 38.0 & 71.5 & 62.0 & 66.7 & 59.5 \\ glot-at-02024-08-06 (t3) & 87.2 & 82.3 & 71.8 & 45.0 & 75.2 & 68.2 & 68.0 & 32.7 & 69.8 & 71.2 & 71.3 & 71.8 & 71.5 & 55.8 & 52.7 & 72.0 & 64.5 & 67.5 & 64.3 \\ Clado OPUS (l1) & 85.7 & 74.7 & 61.3 & 54.5 & 41.5 & 45.0 & 64.5 & 30.5 & 63.7 & 50.2 & 50.7 & 68.3 & 70.5 & 56.0 & 50.0 & 68.8 & 63.7 & 63.7 & 65.1 \\ Gemini-1.5-prot(l3) & 88.5 & 82.0 & 75.3 & 47.5 & 73.3 & 68.2 & 64.8 & 33.2 & 64.8 & 61.3 & 73.7 & 68.5 & 68.3 & 54.5 & 42.3 & 69.3 & 61.8 & 65.0 & 62.0 \\ \hline
**Translate-test** & & & & & & & & & & & & & & & & & & & & & & & & & & & \\ afro-limr-large76L. & 83 & 73.7 & 54.3 & 67.2 & 66 & 63 & 32.8 & 65.7 & 65.8 & 71.2 & 70.2 & 73 & 56.8 & 47.5 & 74.2 & 63.7 & 72 & 63.6 \\ mtQ-XLMMT (i2) & 59.8 & 54.5 & 45.3 & 52.7 & 50.0 & 49.8 & 34.5 & 48.2 & 50.2 & 55.0 & 53.0 & 56.8 & 46.2 & 42.7 & 54.8 & 50.7 & 54.3 & 49.9 \\ Aya-101 dj & 61.2 & 60.7 & 40.8 & 53.8 & 52.3 & 50.3 & 33.0 & 48.7 & 51.3 & 54.0 & 56.0 & 54.0 & 44.5 & 39.3 & 56.8 & 51.0 & 57.0 & 50.2 \\ BLOOOTZ T9 (I2) & 56.8 & 52.8 & 42.7 & 51.3 & 45.7 & 43.3 & 42.6 & 45.8 & 52.0 & 51.3 & 51.2 & 44.7 & 40.2 & 52.8 & 47.5 & 51.3 & 47.6 \\ LLMa 3 sB (k2) & 42.7 & 38.7 & 37.9 & 38.0 & 35.0 & 33.2 & 40.7 & 40.2 & 40.0 & 40.8 & 39.5 & 36.2 & 35.3 & 40.3 & 30.6 & 39.8 & 38.2 \\ LLMa 3 l 8B (l2) & 49.8 & 49.2 & 42.1 & 45.3 &



\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c c c c c c c c c c} \hline
**Model** & **eng** & _fra_ & **anh** & **ewe** & **hau** & **ibo** & **kin** & **lin** & **ling** & **orm** & **sma** & **set** & **swa** & **twist** & **wod** & **sho** & **yor** & **rail** & **avv** \\ \hline
**in-language** & & & & & & & & & & & & & & & & & & & & & & \\ mT0-XXL-MT (1) & 36.4 & 33.0 & 27.4 & 24.8 & 29.2 & 28.4 & 27.2 & 28.6 & 26.0 & 27.4 & 26.6 & 30.4 & 32.2 & 30.2 & 24.8 & 26.8 & 28.8 & 28.2 & 27.9 \\ Av=101 (2) & 40.0 & 36.6 & 31.6 & 25.4 & 33.4 & 36.8 & 30.8 & 27.8 & 28.0 & 26.2 & 28.2 & 31.8 & 32.2 & 26.8 & 25.2 & 32.0 & 28.4 & 29.8 & 29.7 \\ BLOOMZ T9 (2) & 33.8 & 32.2 & 19.0 & 23.6 & 24.0 & 24.2 & 27.0 & 23.0 & 21.2 & 23.6 & 24.8 & 24.2 & 27.2 & 25.2 & 26.0 & 24.4 & 26.8 & 22.8 & 24.1 \\ LLMaS 38 (8t) & 55.4 & 46.0 & 29.4 & 26.4 & 27.8 & 30.6 & 30.4 & 28.4 & 25.0 & 21.6 & 28.6 & 28.6 & 32.0 & 26.6 & 26.4 & 23.4 & 29.7 & 28.1 \\ LLMaM 31 8B (51) & 62.6 & 56.0 & 33.2 & 30.6 & 33.6 & 31.4 & 29.2 & 30.3 & 31.8 & 32.4 & 26.8 & 31.3 & 39.2 & 27.0 & 26.8 & 27.2 & 34.0 & 30.1 \\ LLMaX 38 (8b) & 47.2 & 38.0 & 30.2 & 28.0 & 33.4 & 32.0 & 27.8 & 28.4 & 25.2 & 30.0 & 31.6 & 29.4 & 35.6 & 25.8 & 27.0 & 28.0 & 29.8 & 26.6 & 29.3 \\ German 29 8B (1) & 69.8 & 62.6 & 41.3 & 32.9 & 37.9 & 39.2 & 31.8 & 37.0 & 30.5 & 32.8 & 39.7 & 34.7 & 48.9 & 32.1 & 28.7 & 33.2 & 33.2 & 36.1 & 35.4 \\ German 22 1B (1) & 76.6 & 66.4 & 49.6 & 32.4 & 42.4 & 40.2 & 38.0 & 32.6 & 33.6 & 44.6 & 34.6 & 34.0 & 42.0 & 41.0 & 42.4 & 39.9 \\ LLaMa 3.1 7OB (1) & 76.4 & 69.4 & 41.6 & 32.2 & 47.6 & 47.2 & 38.6 & 40.0 & 34.4 & 35.6 & 41.6 & 39.0 & 55.8 & 28.4 & 31.6 & 34.2 & 41.4 & 40.4 & 39.4 \\ Communlink (Aug) (49) (40) & 48.0 & 49.0 & 28.4 & 27.8 & 25.8 & 27.2 & 28.0 & 28.6 & 29.8 & 28.0 & 28.0 & 28.0 & 26.6 & 29.0 & 25.8 & 28.2 & 28.4 & 27.2 & 27.4 & 27.8 \\ gpt-3-ventho-25 (1) & 72.0 & 66.6 & 31.2 & 34.8 & 30.9 & 39.8 & 34.2 & 34.0 & 39.0 & 38.0 & 39.8 & 39.3 & 52.4 & 37.3 & 32.8 & 36.7 & 32.6 & 38.1 \\ grat-do mini-2024-07-18 (1) & 82.6 & 78.2 & 37.2 & 28.0 & 51.4 & 46.2 & 52.8 & 44.0 & 39.4 & 26.3 & 46.8 & 64.0 & 38.0 & 31.6 & 50.6 & 48.4 & 50.0 & 45.5 \\ gpt-4-tuple-2024-09-01 (9) & 80.4 & 79.4 & 44.4 & 32.4 & 63.2 & 61.8 & 60.4 & 53.6 & 49.6 & 49.4 & 63.4 & 63.4 & 74.4 & 35.2 & 35.0 & 54.4 & 63.6 & 54.2 \\ gpt-4+0224-08-06 (1) & 87.4 & 83.2 & 59.8 & 33.6 & 67.2 & 67.2 & 64.2 & 61.0 & 52.8 & 61.0 & 67.6 & 67.4 & 77.4 & 43.2 & 37.8 & 70.2 & 61.2 & 68.2 & 60.0 \\ Claude OPUS (1) & 74.6 & 64.4 & 57.6 & 33.6 & 39.4 & 43.6 & 42.2 & 43.6 & 41.0 & 40.4 & 43.4 & 47.0 & 55.0 & 38.2 & 33.2 & 42.6 & 43.6 & 43.8 & 43.0 \\ Gemini-1-5-pro (1) & 82.0 & 81.8 & 68.0 & 39.0 & 71.2 & 70.4 & 65.0 & 55.2 & 53.0 & 55.8 & 66.8 & 67.6 & 78.4 & 48.2 & 32.0 & 69.2 & 57.4 & 66.0 & 60.2 \\ \hline
**Translate-test** & & & & & & & & & & & & & & & & & & & & & & & & & \\ mT0-XXL-MT (1) & 32.2 & 30.0 & 27.2 & 28.8 & 27.4 & 29.8 & 28.0 & 26.0 & 27.2 & 29.8 & 30.4 & 30.4 & 25.4 & 27.0 & 30.2 & 29.8 & 26.4 & 28.4 \\ Ava=101 (2) & 37.8 & 32.4 & 28.6 & 31.0 & 31.6 & 31.8 & 33.6 & 27.2 & 28.6 & 32.4 & 34.2 & 31.2 & 31.8 & 28.6 & 30.8 & 34.2 & 32.2 & 31.1 \\ BLOOMZ T9 (2) & 31.0 & 28.6 & 24.4 & 31.0 & 24.6 & 29.0 & 26.4 & 27.2 & 28.6 & 29.2 & 28.0 & 30.6 & 30.8 & 24.6 & 25.2 & 30.2 & 27.4 & 27.9 \\ LLMaS 38H (4) & 39.8 & 34.6 & 30.6 & 31.8 & 31.2 & 30.8 & 33.0 & 29.0 & 29.2 & 30.8 & 34.5 & 34.2 & 31.3 & 28.4 & 33.0 & 32.6 & 33.2 & 31.8 \\ LLMaM 3.1 8B (1) & 51.0 & 46.0 & 36.2 & 40.0 & 43.6 & 43.2 & 42.8 & 33.4 & 39.8 & 40.8 & 45.6 & 48.2 & 33.8 & 31.2 & 44.2 & 43.6 & 44.4 & 41.1 \\ LLMaX 38B (5) & 39.4 & 37.6 & 32.4 & 35.2 & 37.6 & 35.0 & 33.8 & 33.2 & 33.2 & 36.4 & 35.8 & 41.0 & 36.2 & 34.2 & 36.2 & 34.6 & 38.4 & 38.5 \\ Gemma 29 8B (1) & 59.4 & 47.2 & 41.2 & 45.2 & 47.8 & 46



\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c c c c c c c|c} \hline
**Model** & **eng** & **fra** & **amh** & **ewe** & **hau** & **ibo** & **kin** & **lin** & **lung** & **orm** & **sna** & **sot** & **swa** & **twi** & **wol** & **skho** & **yor** & **zul** & **ave** \\ \hline XLM-R-large & **90.5** & **84.3** & 75.8 & 37.2 & 68.2 & 39.3 & 40.5 & 32.8 & 37.7 & 59.2 & 41.2 & 39.7 & 74.2 & 39.0 & 43.8 & 63.5 & 38.5 & 62.2 & 49.6 \\ Serengeti & 77.3 & 60.7 & 54.8 & 51.3 & 61.0 & 56.0 & 55.2 & 36.2 & 55.3 & 46.3 & 58.3 & 55.0 & 66.2 & 42.7 & 43.8 & 56.3 & 53.5 & 57.8 & 53.1 \\ Afro-XLMR-base & 81.5 & 78.5 & 71.3 & 36.3 & 68.8 & 59.8 & 57.7 & **37.8** & 44.2 & 56.7 & 59.8 & 61.5 & 67.0 & 40.8 & 41.0 & 61.0 & 52.2 & 62.7 & 54.9 \\ Afro-XLMR-large & 86.5 & 82.3 & 77.2 & 39.7 & 75.2 & 69.8 & 64.5 & 35.3 & 57.8 & 69.7 & 68.0 & 69.2 & **74.3** & 39.5 & 39.8 & 69.0 & 61.0 & 72.2 & 61.4 \\ Afro-XLMR-76L-large & 88.2 & 83.3 & **78.5** & **58.3** & **73.3** & **70.0** & 65.8 & 33.3 & **68.0** & **69.3** & **70.8** & **70.8** & 73.3 & **59.5** & **51.8** & **73.0** & **63.2** & **72.5** & **65.7** \\ \hline GPT-do & 86.2 & 78.7 & 66.7 & 48.3 & 69.2 & 68.2 & **66.8** & 31.2 & 67.2 & 66.2 & 69.8 & 68.3 & 72.5 & 53.2 & 49.5 & 72.5 & 63.5 & 70.0 & 62.7 \\ \hline \end{tabular}
\end{table}
Table 19: **Cross-lingual transfer results on AfriXNLI: We fine-tuned various multilingual encoders on English training data, and evaluated on other languages. Best result per language in bold**