Venue: NAACL 2025

AfriHate: A Multilingual Collection of Hate Speech and Abusive Language Datasets for African Languages

Shamsuddeen Hassan Muhammad\({}^{1,2,*}\), Idris Abdulumini\({}^{3}\), Abinew Ali Ayele\({}^{4,5}\), David Ifeoluwa Adelani\({}^{6}\), Ibrahim Said Ahmad\({}^{2,7}\), Saminu Mohammad Aliyu\({}^{2}\), Nelson Odihambo Onyango\({}^{8}\), Lilian D. A. Wanzare\({}^{5}\), Samuel Rutunda\({}^{9}\), Lukman Jibril Atiyu\({}^{10}\), Esubalew Alemneh\({}^{4}\), Oumaima Hourrane\({}^{12}\), Hagos Tesfahun Gebremichael\({}^{4}\), Elyas Abdi Ismail\({}^{20}\), Meriem Beloucif\({}^{13}\), Ebrahim Chekolj fibril\({}^{14}\), Andiswa Bukula\({}^{15}\), Rooweither Mabuya\({}^{15}\), Salomey Osei\({}^{16}\), Abigail Oppong\({}^{17}\), Tadesse Destaw Belay\({}^{18,19}\), Tadesse Kebede Guge\({}^{11}\), Tesla Tegegne Asfaw\({}^{4}\), Chiamaka Ijeoma Chukwueneke\({}^{21}\), Paul Rottger\({}^{22}\), Seid Muhie Yimam\({}^{5}\), Nedjima Ousidhoum\({}^{23}\)

###### Abstract

Hate speech and abusive language are global phenomena that need socio-cultural background knowledge to be understood, identified, and moderated. However, in many regions of the Global South, there have been several documented occurrences of (1) absence of moderation and (2) censorship due to the reliance on keyword spotting out of context. Further, high-profile individuals have frequently been at the center of the moderation process, while large and targeted hate speech campaigns against minorities have been overlooked. These limitations are mainly due to the lack of high-quality data in the local languages and the failure to include local communities in the collection, annotation, and moderation processes. To address this issue, we present **AfriHate**: a multilingual collection of hate speech and abusive language datasets in 15 African languages, annotated by native speakers. We report the challenges related to the construction of the datasets and present various classification baseline results with and without using LLMs. We find that model performance highly depends on the language and that multilingual models can help boost the performance in low-resource settings.1

Footnote 1: The datasets, individual annotations, and hate speech and offensive language lexicons are available on [https://github.com/AfriHate/AfriHate](https://github.com/AfriHate/AfriHate).

**Content Warning: This paper contains examples of hate speech and offensive language.**

**[Volume 1: Long Papers]**** 

especially for low-resource languages. To bridge this gap, we present **AfriHate** a collection of new hate speech and abusive language Twitter datasets in 15 languages spoken in various African regions: Algerian Arabic, Amharic, Igbo, Kinyarwanda, Hausa, Moroccan Arabic, Nigerian Pidgin, Oromo, Somali, Swahili, Tigrinya, Twi, isiXhosa, Yorbaba, and isiZulu. The datasets are annotated by native speakers and include three classes: **hate**, **abusive/offensive**, or **neutral**-neither hateful nor abusive. The targets of the hateful tweets were further labeled based on six common attributes used to discriminate against people: **ethnicity**, **politics**, **gender**, **disability**, **religion**, or **other**. Table 1 shows a sample of the datasets in various languages.

We report the data collection and annotation strategies and challenges when building **AfriHate**, present various classification baselines with and without using LLMs, and discuss the results. We find that model performance highly depends on the language and that multilingual models can help boost the performance in low-resource settings. We publicly release the datasets and individual labels, in addition to manually-curated hate speech and offensive language lexicons. These provide a valuable foundation for the research community interested in hate speech and abusive language, African languages, and researchers interested in studying disagreements.

## 2 Related Work

The fast-spreading nature of hate speech and abusive language have been at the center of a significant amount of NLP work in recent years Talat and Hovy (2016); Vigna et al. (2017); Basile et al. (2019); Mansur et al. (2023). However, as there is no unanimous definition of hate speech, researchers have adopted different ones when building resources. For instance, some studies define hate speech as any speech that can cause danger or harm to disadvantaged groups Davidson et al. (2017), others focus on whether the speech is intended to promote hatred Gitari et al. (2015), or whether it dehumanises protected groups Vidgen et al. (2021), which leads to various challenges such as the lack of generalisability Yin and Zubiaga (2021).

Despite Africa being home to more than 2,000 languages and the increasing interest in building hate speech and offensive languages resources for non-English languages Ousidhoum et al. (2019); Rottger et al. (2022); Madeddu et al. (2023), few datasets focus on African languages, such as Ambaric Ayele et al. (2024, 2023, 2022), Afaan Oromo Ababu and Woldebyohannis (2022), Yoruba Ilevbare et al. (2024), Hausa Vargas et al. (2024); Adam et al. (2023), and Nigerian Pidgin Ndabula et al. (2023); Ilevbare et al. (2024); Aliyu et al. (2022);

\begin{table} \end{table}
Table 1: Examples of hateful and abusive instances in AffriHate. All hateful posts are assigned targets.

 

Tonneau et al., 2024). Moreover, most resources adopt a binary labeling scheme (hate/offensive) (e.g., Aliyu et al. (2022)), or do not add the target attributes (e.g., Illevbare et al. (2024)). Other work Tonneau et al. (2024) relies on active learning for annotating some data instances, which is not ideal when labeling hate speech for under-resourced African languages. That is, when focusing on underrepresented cultures ML models that deal with hate speech tend to be culturally insensitive Lee et al. (2023, 2024).

Furthermore, a limited number of studies involve African communities in the dataset creation process Adelani et al. (2021); Maroniokolakis et al. (2022); Abduluminin et al. (2024). We take a step towards addressing this problem by developing 15 new datasets for hate and offensive speech in various languages spoken across the African continent.

## 3 Creating AfriHate

AfriHate covers 15 languages from various African regions. In Table 2, we report the scripts of these languages and the main regions where they are spoken. The collection includes tweets from 2012 to 2023 collected using the Academic API before the suspension of free academic access. The API allowed us to collect up to 10 million tweets per month, and Twitter/X is a commonly used platform in African countries with documented cases of hate speech propagation Adjai and Lazaridis (2013); Egbunike et al. (2015); Oriola and Kotze (2020); Ridwanullah et al. (2024); Raborife et al. (2024).

### Data Collection

Except for Amharic and Tigrinya, the Twitter API does not support African languages, which makes the data collection challenging. We, therefore, follow strategies adopted by previous work such as Muhammad et al. (2022, 2023) and use various heuristics based on hate speech and abusive language keywords, user handles, stopwords, hashtags, and locations. Table 3 shows the number of keywords used for data collection in each language.

Since interpreting hate and abusive content heavily depends on understanding political and socio-cultural contexts, we have adopted language-dependent collection and annotation strategies. As previous work that relied on specific keywords to collect data (e.g., Talat and Hovy, 2016; Basile et al., 2019), we follow an analogous strategy. Similarly to Ousidhoum et al. (2019), we use a larger set of keywords and include culture-specific controversial topics in the lists. Despite the diversity of topics and the large sizes of the lists (see Table 3), an initial pre-annotation phase revealed a limited number of hateful tweets for some languages such as Nigerian-Pidgin and Hausa. Therefore, we used additional heuristics to collect more tweets: 1) keyword crowd-sourcing, 2) manual data collection, and 3) using existing datasets, as we explain in the following.

Crowd-sourcing KeywordsTo crowd-source additional keywords, we first asked native speakers to provide us with a list of hateful, abusive, or controversial keywords. Then, we contacted social media influencers, who asked their followers to

\begin{table}
\begin{tabular}{l l l l c} \hline \hline
**Language** & **Code** & **Subregion** & **Spoken in** & **Script** \\ \hline Algerian Arabic/Darija & arq & North Africa & Algeria & Arabic \\ Annaric & ann & East Africa & Ethiopia, Eritrea & Ethiopic \\ Hausa & nau & West Africa & Northern Nigeria, Niger, Ghana, and Cameroon, & Latin \\ Igbo & ibo & West Africa & Southeast Negria & Latin \\ Kinarywanda & kin & East Africa & Rwanda & Latin \\ Moroccan Arabic/Darija & ary & North Africa & Morocco & Arabic/Latin \\ Nigerian Pidgin & pcn & West Africa & Nigeria, Ghana, Cameroon, & Latin \\ Orrono & orn & East Africa & Ethiopia, Kenya, Somalia & Latin \\ Somali & som & East Africa & Somalia, Ethiopia, Djibouti, Kenya & Latin \\ Swahili & swa & East Africa & Kenya, Tanzania, Uganda, DR Congo, Rwanda, Burundi, Mozambique & Latin \\ Tigrinya & tri & East Africa & Ethiopia, Eritrea & Ethologic \\ Twi & tvi & West Africa & Ghana & Latin \\ Xhousa & stso & Southern Africa & Mozambique, South Africa, Zimbabwe, Eswatini & Latin \\ Yorubá & yor & West Africa & Southwestern and Central Nigeria, Benin, and Togo & Latin \\ Zulu & zul & Southern Afric & Southern Africa & Latin \\ \hline \hline \end{tabular}
\end{table}
Table 2: **Information about the AfriHate languages**: their ISO codes, subregions, countries in which they are mainly spoken, and the writing scripts included in AfriHATE.

 

share abusive or hateful keywords in their local languages by filling in a form created for anonymous collection. This helped us diversify the lists as the followers come from various backgrounds.

We also used off-the-shelf curated hate speech lexicons from PeaceTech Lab2 and Hatebase.3 The lists were post-processed by native speakers prior to the collection of the tweets.

Footnote 2: [https://www.peacetechlab.org/the-peacetech-toolbox](https://www.peacetechlab.org/the-peacetech-toolbox)

Footnote 3: [https://hatebase.org](https://hatebase.org)

Manual Data CollectionFor Kinyarwanda and Twi, native speakers manually collected all the tweets using a combination of keywords and user handles. We curated a list of user handles of public figures who frequently post hateful or abusive content, and collected tweets from their profiles.

Using Existing DatasetsFor Nigerian Pidgin, we used a subset of an additional existing hate speech dataset proposed by Tonneau et al. (2024). Since some instances in this dataset were annotated using active learning, we re-annotated those labeled as hateful or offensive. Similarly, for Swahili, we re-annotated the negative instances from the sentiment analysis dataset introduced by Muhammad et al. (2023), the misinformation dataset by Amol et al. (2023), and the hate speech one by Ombui et al. (2019) into our predefined classes (hate, offensive and normal).

We further labeled the targets of the re-annotated tweets into our target attribute categories, i.e., disability, ethnicity, gender, politics, religion, and others.

### Data Processing

We further cleaned the collected tweets and removed retweets, tweets containing less than three words, duplicates, URLs, invisible characters, and redundant white spaces. We converted the tweets written in Latin script to lowercase and anonymised the tweets by replacing @mentions with a placeholder @_user_.

### Language Identification

We collected the tweets using location and keywords. This is particularly challenging for African languages since people within one location can speak different languages. That is, keywords and hashtags may appear in more than one language, which makes the data selection more difficult.

Open-source and closed-source language identification (LID) tools used in previous studies Muhammad et al. (2022, 2023) often show low accuracy in African languages, especially when used in social media posts. This is largely due to the unique linguistic characteristics of these languages, such as the common usage of code-mixing and digraphia, i.e., a language can be written in more than one script.

To address these limitations, we built a LID model that improves the identification performance for social media text data in our target languages. It achieves this through continued pretraining of AfroXLMR Alabi et al. (2022) on Glot500-c corpus which covers 511 predominantly low-resource languages Imani et al. (2023). We further fine-tuned the model on AfriSenti-LID dataset, which focuses on social media data and covers most of the languages included in AfriHATE and a similar Twitter-sphere. Our LID tool and its documentation can be found on our project page.4

Footnote 4: [https://github.com/hausanlp/AfriLID](https://github.com/hausanlp/AfriLID)

### Data Annotation

#### 3.4.1 Pre-Annotation and Data Selection

We randomly sampled tweets in each language and conducted a pilot annotation, which showed a large class imbalance despite collecting tweets using abusive and hateful keywords. For instance, most tweets in Hausa were neutral (neither hateful nor abusive) due to keywords carrying multiple meanings depending on the region where the word is used, i.e., it can have a neutral connotation in some parts of Nigeria. For example, the word _Aboki_ means "friend" in Northern Nigeria, while it can be an insult in the Southern part of the country.

\begin{table}
\begin{tabular}{l r r r r r r r r r r r r r r} \hline \hline  & **amb** & **aruq** & **ary****huu** & **ibo** & **kin** & **oro** & **pcm** & **son** & **swa** & **tir** & **twi** & **xho** & **yor** & **zul** \\ \hline
**Hate** & 88 & 62 & 41 & 36 & 46 & 264 & 126 & 23 & 45 & 24 & 58 & 20 & 42 & 68 & 31 \\
**Abusive** & 74 & 40 & 0 & 149 & 118 & 362 & 159 & 26 & 67 & 12 & 66 & 86 & 177 & 109 & 118 \\ \hline \hline \end{tabular}
\end{table}
Table 3: **Number of keywords used for data collection. For Algerian Arabic (arya), the list includes controversial topics that are neither hateful nor offensive.** 

As this would have led to an insufficient number of instances in each target class, we included a **pre-annotation** phase to ensure each class covered a reasonable percentage of the data.

During the pre-annotation, we provided the annotators with a distinct pool of tweets and asked them to select those likely to be hateful or abusive. We then aggregated the pre-selected tweets, and multiple annotators labeled them. Table 4 indicates the languages for which we conducted a pre-annotation step, i.e., only those for which we observed a significantly high imbalance during the pilot annotation.

#### 3.4.2 Recruiting Annotators

The unavailability of annotators for African languages on common platforms like Amazon Mechanical Turk and Prolific makes traditional crowd-sourcing methods impractical. As Kirk et al. (2023) demonstrated that trained annotators achieve higher quality results, we trained native speakers and recruited them as annotators. For each language, we also recruited a native speaker as a language lead who would control for the quality of the annotations. We used Label Studio as an annotation platform5 and an adapted version of the Potato annotation tool.6

Footnote 5: [https://labelstud.io/](https://labelstud.io/)

Footnote 6: [https://github.com/davidjurgens/potato](https://github.com/davidjurgens/potato)

#### 3.4.3 Annotation Task

LabelsWe provided the annotators with thorough guidelines (see Figure 1 in the Appendix). We asked the annotators to choose one of three categories: **hate**, **abusive/offensive**, or **neutral**. The latter means that the tweet is neither hateful nor abusive. Tweets spotted in a language different from the target one were labeled **Indeterminate** and were later excluded from the final dataset.

Similarly to Ayele et al. (2024); Ousidhoum et al. (2019); Fortuna et al. (2019), annotators had to select the target(s) of the hateful tweets. That is, the common attribute(s) based on which the tweet is discriminating against people: **Ethnicity**, **Politics**, **Gender**, **Disability**, **Religion**, or **Other**. We do not include targets for offensive and abusive tweets as these can often be generic and directed towards an individual as previously reported by Zampieri et al. (2019) (e.g., see the Algerian Arabic example in Table 1). Details about the targets can be found in Figure 1 in the Appendix. For some languages such as Hausa, we asked the annotators to spot which words made them label the tweet hateful or abusive.

Final label selectionAs shown in Table 4, for languages where we conducted a pre-annotation step, each tweet was annotated by 3 annotators, leading to a total of 4 labels per tweet with the pre-annotation label counting as one. On the other hand, for instances in datasets for which we did not carry out a pre-annotation step, 3 to 4 annotators

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c} \hline \hline
**Language** & **amh** & **argq** & **ary** & **hau** & **ibo** & **kin** & **oro** & **pcm** & **son** & **swa** & **tir** & **twi** & **xho** & **yor** & **zul** \\ \hline
**Manually Collected** & ✗ & ✗ & ✓ & ✗ & ✓ & ✗ & ✗ & ✗ & ✗ & ✗ & ✓ & ✗ & ✗ \\
**Pre-Annotation** & ✗ & ✓ & ✓ & ✓ & ✓ & ✗ & ✓ & ✗ & ✗ & ✗ & ✓ & ✓ & ✗ & ✗ \\ \hline
**Total Annotators** & 11 & 7 & 3 & 3 & 6 & 3 & 9 & 3 & 7 & 5 & 8 & 3 & 3 & 4 & 3 \\
**Annotators per Instance** & 4 & 3 & 3 & 3 & 3 & 4 & 3 & 4 & 3 & 4 & 3 & 3 & 4 & 3 \\ \hline
**Free-Marginal Multirater Kappa †** & 0.63 & 0.68 & 0.61 & 0.75 & 0.80 & 0.81 & 0.63 & 0.65 & 0.46 & 0.55 & 0.46 & 0.75 & 0.62 & 0.68 & 0.81 \\ \hline \hline \end{tabular}
\end{table}
Table 4: **Collection and annotation details for AfriHatte**. The table shows if the data was manually collected, whether a pre-annotation step was conducted, the total number of annotators, the number of annotators per instance, and the inter-annotator agreement (Free-Marginal Multirater Kappa).

\begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c} \hline \hline
**Class** & **amb** & **argv** & **aroq** & **huu** & **ibo** & **kin** & **oro** & **pcm** & **son** & **suva** & **tir** & **tvi** & **xho** & **yo****** & **zul** \\ \hline
**Hate** & 2,246 & 162 & 674 & 343 & 251 & 1,268 & 2,293 & 1,177 & 388 & 3,974 & 2,940 & 443 & 210 & 150 & 147 \\  & (45,759) & (31,093) & (41,953) & (52,953) & (59,016) & (26,953) & (48,963) & (11,719) & (3,953) & (18,953) & (58,905) & (11,493) & (5,793) & (11,916) & (3,475) \\ \hline
**Abusive** & 1,353 & 778 & 2,270 & 2,336 & 3,510 & 1,146 & 667 & 5,238 & 1,404 & 7,708 & 1,070 & 3,278 & 1,550 & 2,655 & 1,839 \\  & (27,393) & (62,279) & (48,093) & (52,563) & (70,256) & (24,393) & (13,278) & (94,653) & (30,175) & (36,693) & (21,193) & (40,978) & (42,179) & (54,643) & (42,733) \\ \hline
**Neutral** & 1,359 & 310 & 1,690 & 3,965 & 1,242 & 2,308 & 2,072 & 4,184 & 2,868 & 9,410 & 1,062 & 180 & 1,923 & 2,074 & 2,322 \\  & (27,493) & (24,873) & (36,953) & (39,653) & (24,873) & (48,995) & (41,278) & (70,953) & (61,616) & (44,693) & (23,953) & (4,653) & (52,736) & (42,573) & (53,93) \\ \hline \hline \end{tabular}
\end{table}
Table 5: **Number of instances per class in each dataset. Percentages are shown below each absolute count. In total, AfriHatte contains 90,437 instances across 15 languages.** 

were assigned to each tweet, and the final gold label was determined by majority voting, i.e., two out of three labels or three out of four labels.

Table 5 and Table 6 show the final number of instances per class and the target distributions for all the languages.

#### 3.4.4 Inter-Annotator Agreement

To assess inter-annotator agreement (IAA), we computed the free marginal Randolph's Kappa score [10] for each dataset. Table 4 shows the IAA scores for all the datasets, the total number of annotators in each, and the number of annotators per instance. The IAA scores range from \(0.46\) to \(0.81\), indicating medium to high agreement levels. The highest agreement scores are reported for kin, and twi, which can be due to the manual collection of only potentially abusive and hateful tweets. For other languages such as nau and zul, the high agreement can be attributed to the pre-annotation step, which helped us filter tweets that were later annotated.

### Dataset Statistics

As shown in Table 5, most datasets are imbalanced, and the _hate_ class includes fewer instances in 9 out of 15 languages. The variations in the class distributions are due to the differences between the languages and the data collection techniques. Further, the target distributions also differ because of socio-cultural characteristics related to local politics, social dynamics, and an unavoidable degree of selection bias [27].

Data SplitsWe split the AfriHate datasets based on the various label distributions. As reported in Table 7, each test set includes a minimum of 100 instances in each class (i.e., hate, abusive, and normal). This guarantees a more robust evaluation of the different models.

## 4 Experiments

### Setup

We compare the performance of three main sets of approaches on the AfriHate datasets:

1. Fine-tuning a BERT-based pre-trained language model (PLM),
2. Few-shot learning with SetFit with BERT-based PLM [20]: a few-shot approach using BERT-like PLMs,
3. Prompting large language models (LLMs) in zero and few-shot settings.

Fine-tuning PLMsWe use four widely adopted Africa-centric PLMs that have been shown to consistently perform better on African languages than massively multilingual PLMs such as XLM-R [10]. The models are AfriBERTa-large [11], AfriTeVa V2 base [11], AfroXLMR [1] and AfroXLMR-76L [1]. Each model was trained for 20 epochs over 5 runs with a batch size of 32, maximum sequence length of 128, and a learning rate of \(5e-5\) except for AfroXLMR-76L where we used a learning rate of \(3e-5\). The rest of the hyperparameters were the default values set in the HuggingFace fine-tuning

\begin{table}
\begin{tabular}{l r r r r r r r r r r r r r r} \hline \hline
**Hate Target** & **amh** & **arq** & **ary** & **hau** & **ibo** & **kin** & **oro** & **pcm** & **son** & **swa** & **tir** & **twi** & **xho** & **yor** & **zul** \\ \hline
**Disability** & 0 & 0 & 26 & 0 & 0 & 6 & 0 & 1 & 0 & 112 & 0 & 32 & 3 & 2 & 0 \\
**Ethnicity** & 902 & 269 & 150 & 32 & 383 & 94 & 462 & 537 & 74 & 2,799 & 573 & 245 & 94 & 103 & 241 \\
**Gender** & 15 & 2 & 6 & 2 & 3 & 24 & 12 & 50 & 17 & 117 & 0 & 12 & 111 & 13 & 0 \\
**Politics** & 1,247 & 118 & 63 & 0 & 6 & 1,096 & 1,550 & 87 & 703 & 233 & 2,251 & 32 & 0 & 33 & 0 \\
**Religion** & 130 & 6 & 207 & 38 & 0 & 10 & 31 & 105 & 19 & 336 & 16 & 14 & 0 & 22 & 0 \\
**Others** & 199 & 0 & 0 & 0 & 0 & 0 & 767 & 15 & 208 & 501 & 85 & 14 & 0 & 76 & 12 \\ \hline \hline \end{tabular}
\end{table}
Table 6: **Data distribution of hate speech targets in AfriHate.**

\begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c} \hline \hline
**Split** & **anh** & **ary** & **aro** & **hau** & **ibo** & **kin** & **oro** & **pcm** & **son** & **swa** & **tir** & **twi** & **xho** & **yor** & **zul** \\ \hline
**Train** & 3,467 & 3,240 & 716 & 4,566 & 3,419 & 3,302 & 3,517 & 7,416 & 3,174 & 14,760 & 3,547 & 2,564 & 2,502 & 3,336 & 2,940 \\  & (69.9%) & (69.9%) & (57.3%) & (68.7%) & (68.2%) & (69.9%) & (68.8%) & (70.9%) & (68.1%) & (70.6%) & (68.9%) & (68.7%) & (67.9%) & (68.4%) & (68.2%) \\ \hline
**Dev** & 744 & 695 & 211 & 1,029 & 774 & 706 & 763 & 1,590 & 741 & 3,164 & 760 & 639 & 559 & 724 & 640 \\  & (15.6%) & (15.0%) & (16.9%) & (15.5%) & (15.4%) & (15.0%) & (15.1%) & (15.9%) & (15.9%) & (15.9%) & (15.0%) & (15.0%) & (16.4%) & (13.5%) & (14.8%) & (14.9%) \\ \hline
**Test** & 747 & 699 & 323 & 1,049 & 821 & 714 & 759 & 1,593 & 745 & 3,168 & 765 & 698 & 622 & 819 & 728 \\  & (35.7%) & (15.1%) & (25.8%) & (35.8%) & (16.8%) & (15.1%) & (13.1%) & (13.8%) & (13.6%) & (16.0%) & (13.0%) & (13.9%) & (13.9%) & (16.9%) & (16.8%) & (16.9%) \\ \hline \hline \end{tabular}
\end{table}
Table 7: **Number of instances included in the training (train), development (dev), and test splits of the different datasets with percentages shown below each absolute count.** 

pipeline for text classification.

SetFit Few-shot LearningSetFit is a few-shot learning approach based on sentence-transformer models like LabSE Feng et al. (2022). It works by, first, fine-tuning a pre-trained sentence transformer model on a few examples in a contrastive manner. Then, the resulting model is used to generate rich text embeddings, which are used to train a classification head. We used LaBSE to train classifiers with the following configurations:

1. for **zero-shot learning**, we trained the transformers for one epoch using the dummy dataset generated by the framework (2 x [This sentence is {Label}, ...]), where {Label} can be **neutral**, **abusive/offensive** or **hate**;
2. for **few-shot learning**, we trained each model for three epochs using 5, 10, and 20 shots. All the classifiers were trained using a batch size of 32.

Prompting LLMsWe prompt one closed model (GPT-4o) and nine open models of various model sizes (0.4B to 70B). The open models are: InkubaLM-0.4B Tonja et al. (2024), mT0-small Muennighoff et al. (2023), BLOOMZ 7B Scao et al. (2022), Mistral 7B Jiang et al. (2023), Aya-23-35B Aryabumi et al. (2024), LLaMa 3.1 {8B & 70B} Dubey et al. (2024), and Gemma 2 (9B & 27B) Team et al. (2024). All the models were prompted using five prompt templates with clear definitions of the **abusive/offensive**, **hate**, and **neutral** categories. For hate speech, we used the definition adopted by the United Nations and another one from the Merriam Webster's dictionary. We report the average scores across all the templates in Section 4.2. The full prompts can be found in Appendix A.1.

### Experimental Results

Monolingual vs. Multilingual Fine-tuningWe compare monolingual fine-tuning where we train on a language and evaluate on the same language to multilingual fine-tuning--where we combine the training data of all the languages and evaluate the results for each.

Table 8 shows the results of the fine-tuning experiments. We find that encoder-only models perform better than the T5-style models, i.e., AfriTeVa V2. On average, AfroXLMR-76L achieved the best performance, most likely due to the fact that it was pre-trained on all the languages included in **AfriHate**. While AfroXLMR was not pretrained on some languages such as Tigrinya (tir) and Twi (twi), it still achieves performance that is comparable to AfroXLMR-76L. AfriBERTa generally struggles with Arabic dialects such as Algerian Daria (aro) and Moroccan Darija (ary) as the Arabic script was not included in its pre-training.

Overall, multilingual fine-tuning of the multilingual AfroXLMR model leads to the best results on 11 out of 15 languages, and comparable results on the remaining languages except for Yoruba (gor), where AfriBERTa led to the best result likely because of its Africa-centric tokenizer.

Table 9 shows the per-class accuracy across different languages. Overall, multilingual models perform better for languages with a low percentage of the "hate" category in the training data (e.g., \(<200\)) such as ary, xho, yor, and zul with an F-score improvement of \(+1.7\), \(+5.7\), \(+4.9\), and \(+10.8\), respectively.

Zero-shot vs. Few-shot SettingsTable 10 shows the results of both zero-shot and few-shot experiments. SetFit performs slightly better than all open LLMs in zero-shot settings (\(36.9\)), and GPT-4o leads to the best overall performance with \(61.9\) F1 points.

When considering the few-shot settings, 5-shot models show the biggest boost in performance, where Gemma-2-9B and other bigger models (Gamma 2 27B and LLaMa 3.1 70B) improve by

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c c} \hline \hline
**Model** & **ath** & **ary** & **arq** & **hau** & **ibo** & **kin** & **oro** & **pcm** & **son** & **swa** & **tifr** & **twi** & **xbo** & **yo** & **zul** & **avg** \\ \hline  & \multicolumn{10}{c}{**Monolingual Fine-Tuning**} \\ \hline
**ArtiBERTa** & 69.54 & 67.93 & 30.48 & 82.28 & 89.53 & 79.43 & 73.43 & 66.90 & 65.52 & 91.36 & 73.07 & 74.54 & 81.07 & 72.37 & 83.75 & 72.33 \\
**AfriFeVA** v2 & 73.91 & 76.71 & 25.25 & 79.06 & 83.95 & 77.60 & 71.61 & 68.69 & 69.65 & 90.68 & 72.36 & 64.96 & 54.67 & **79.88** & 69.05 & 68.73 \\
**AfroXLMR** & 70.65 & 80.16 & 61.18 & 81.93 & 89.30 & **80.72** & 72.11 & 67.98 & 66.84 & 91.44 & 74.52 & 77.17 & 82.49 & 72.15 & 83.44 & 76.15 \\
**AfroXLMR**-76L** & 74.36 & 80.05 & 53.52 & **82.78** & 89.59 & 79.58 & 76.63 & 68.38 & 71.09 & **91.72** & 76.27 & 76.65 & 84.40 & 72.35 & 84.65 & 76.45 \\ \hline \multicolumn{10}{c}{**Multilingual Fine-Tuning**} \\ \hline
**AfroXLMR-76L** & **75.25** & **80.76** & **63.31** & 82.20 & **89.85** & 79.56 & **77.62** & **69.20** & **72.26** & 91.22 & **77.55** & **78.68** & **86.83** & 74.32 & **86.81** & **78.16** \\ \hline \end{tabular}
\end{table}
Table 8: Model performances after fine-tuning BERT-based LLMs. The best performance for each language is highlighted in **bold**.

 

about \(+20\) points. The performance boost with additional 10 and 20 shots is more limited for LLMs (\(+3.0\) improvement), whereas SetFit consistently benefits from additional examples. The best results reached for closed models are at 20-shots, where GPT-4o achieved an overall F1-score of \(70.8\) while Lemma 2 27B achieved the best overall results for any open model with an F1-score of \(57.2\).

In our performance analysis per different classes shown in Table 9, GPT-4o generally performs worse than full fine-tuning in monolingual or multilingual settings. However, we find that it achieves significantly better performance for hate detection in a few languages such as arq (\(+44.0\)), som (\(+14.9\)), and yor (\(+46.1\)) compared to monolingual fine-tuning. For languages without enough training data for the hate category such as Yorba, prompting LLMs might provide a better detection

\begin{table}
\begin{tabular}{l c r r r r r r r r r r r r r|r} \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**\# Shots**} & \multicolumn{3}{c}{**amh**} & \multicolumn{3}{c}{**ary**} & \multicolumn{3}{c}{**arq**} & \multicolumn{3}{c}{**hau**} & \multicolumn{3}{c}{**ibo**} & \multicolumn{3}{c}{**kin**} & \multicolumn{3}{c}{**gr/o**} & \multicolumn{3}{c}{**pctn**} & \multicolumn{3}{c}{**gen**} & \multicolumn{3}{c}{**son**} \\  & 0 & 33.79 & 46.54 & 27.4 & 23.57 & 36.90 & 25.29 & 26.26 & 43.46 & 32.18 & 44.38 & 49.30 & 51.13 & 37.38 & 48.46 & 35.08 & 37.41 \\  & 5 & 44.89 & 33.82 & 36.07 & 49.93 & 51.52 & 55.35 & 36.56 & 53.28 & 48.09 & 54.90 & 35.36 & 33.23 & 47.49 & 34.36 & 33.18 & 43.20 \\  & 10 & 49.42 & 35.81 & 46.55 & 49.72 & 39.31 & 53.10 & 39.10 & 57.00 & 42.29 & 62.64 & 39.16 & 45.76 & 43.52 & 49.73 & 40.33 & 46.23 \\  & 20 & 50.13 & 40.49 & 54.24 & 55.40 & 65.15 & 57.53 & 40.91 & 59.09 & 48.59 & 47.93 & 44.00 & 53.72 & 46.92 & 55.64 & 50.62 & 52.97 \\ \hline
**Mistral-7B=v1** & 0 & 21.18 & 17.46 & 11.98 & 6.84 & 8.04 & 15.18 & 20.82 & 6.76 & 8.16 & 18.88 & 25.62 & 8.38 & 9.72 & 8.92 & 13.08 \\  & 5 & 37.68 & 31.60 & 44.54 & 34.90 & 36.84 & 46.98 & 39.46 & 50.00 & 34.12 & 58.58 & 35.64 & 32.04 & 34.24 & 41.04 & 32.54 & 39.35 \\  & 10 & 39.68 & 36.50 & 50.56 & 37.82 & 36.94 & 45.72 & 39.02 & 51.20 & 31.82 & 6.318 & 35.84 & 30.82 & 36.24 & 49.09 & 32.46 & 40.53 \\  & 20 & 36.64 & 35.46 & 52.94 & 38.44 & 38.98 & 52.28 & 39.24 & 54.60 & 33.28 & 66.28 & 37.42 & 30.44 & 34.12 & 46.06 & 35.00 & 42.10 \\ \hline
**avg-23-35Eb** & 0 & 17.04 & 19.20 & 17.20 & 23.04 & 14.01 & 23.34 & 23.80 & 20.02 & 18.56 & 34.04 & 17.10 & 10.68 & 20.32 & 17.48 & 20.24 & 19.20 \\  & 5 & 37.78 & 56.40 & 57.00 & 39.70 & 42.70 & 48.78 & 39.24 & 57.90 & 38.72 & 69.84 & 35.08 & 40.24 & 36.76 & 48.88 & 44.58 & 46.28 \\  & 10 & 38.50 & 53.96 & 61.82 & 44.42 & 45.80 & 52.82 & 42.16 & 59.26 & 37.60 & 74.10 & 55.88 & 39.92 & 36.18 & 48.84 & 45.00 & 47.72 \\  & 20 & 38.10 & 52.90 & 63.76 & 46.14 & 45.52 & 57.26 & 42.26 & 61.68 & 38.66 & 75.68 & 38.12 & 38.54 & 37.74 & 51.52 & 45.25 & 48.92 \\ \hline
**Gamma-29B** & 0 & 27.42 & 28.36 & 25.40 & 14.48 & 15.00 & 24.08 & 23.74 & 24.74 & 10.92 & 37.12 & 52.52 & 19.50 & 13.40 & 21.66 & 12.52 & 21.59 \\  & 5 & 56.60 & 57.68 & 61.14 & 49.98 & 48.12 & 60.26 & 45.10 & 60.30 & 46.42 & 74.78 & 48.40 & 46.50 & 35.82 & 55.96 & 46.60 & 52.91 \\  & 10 & 57.84 & 61.08 & 61.72 & 56.62 & 53.78 & 63.54 & 45.78 & 60.54 & 51.72 & 78.16 & 45.48 & 48.16 & 38.82 & 60.22 & 52.58 & 55.74 \\  & 20 & 69.96 & 59.94 & 64.38 & 55.90 & 54.56 & 61.41 & 44.70 & 62.54 & 52.22 & 79.24 & 49.44 & 48.92 & 39.02 & 60.50 & 53.68 & 56.38 \\ \hline
**German-2-27B** & 0 & 47.88 & 41.24 & 41.12 & 28.96 & 31.36 & 42.08 & 33.46 & 49.78 & 31.10 & 59.88 & 30.84 & 27.74 & 25.64 & 41.46 & 28.02 & 36.96 \\  & 5 & 59.62 & 64.14 & 65.62 & 54.62 & 56.41 & 61.08 & 46.76 & 61.78 & 54.12 & 81.18 & 52.12 & 47.26 & 40.48 & 61.72 & 54.28 & 57.39 \\  & 10 & 60.70 & 61.36 & 68.38 & 58.90 & 56.84 & 64.86 & 69.66 & 61.60 & 52.86 & 81.94 & 49.82 & 50.86 & 41.90 & 59.66 & 56.94 & 58.01 \\  & 20 & 62.82 & 59.86 & 56.80 & 59.60 & 58.76 & 66.06 & 46.90 & 52.40 & 52.90 & 28.98 & 33.53 & 50.34 & 43.08 & 59.10 & 58.58 & 58.88 \\ \hline
**Lima-31-70B** & 0 & 36.34 & 43.34 & 42.64 & 35.64 & 32.52 & 38.52 & 35.14 & 48.66 & 31.14 & 60.14 & 27.08 & 25.54 & 28.98 & 40.64 & 35.50 & 37.21 \\  & 5 & 58.52 & 62.44 & 28.58 & 52.88 & 58.14 & 55.40 & 63.48 & 52.72 & 57.76 & 44.42 & 43.90 & 

of this class compared to fine-tuning BERT-like PLMs.

Overall ResultsOur results show that fine-tuning multilingual models leads to a better performance for the majority of the **AfriHate** languages. That is, AfroXLMR-76L achieves an average macro F1 score of **78.16**. As for zero-shot and few-shot settings, **GPT-40** outperformed other models, with average F1 scores of **61.89** and **70.79** in zero-shot and 20-shot settings, respectively.

As a whole, these results highlight the advantages of multilingual and context-specific models in hate and abusive language detection for African languages.

## 5 Conclusion

We introduced **AfriHate**, the first large-scale collection of hate and abusive language datasets in 15 African languages: Algerian Arabic, Amharic, Igbo, Kinyarwanda, Hausa, Moroccan Arabic, Nigerian Pidgin, Oromo, Somali, Swahili, Tigrinya, Twi, Xhoza, Yoruba, and Zulu. The datasets were annotated by native speakers as hate speech, abusive, or neutral. We discussed our data collection strategies and highlighted the challenges faced during the data collection and annotation. We then reported baseline experiments using Africacentric pre-trained language models as well as prompted open and closed LLMs showing a large gap in the performance across languages.

**AfriHate** is a first step towards building high-quality hate speech resources for African languages. We publicly release all the datasets, scripts, models, and lexicons to the research community.

### Limitations

While we collected **AfriHate** using large sets of keywords, we acknowledge the unavoidable presence of selection bias (Ousidhoum et al., 2020) as no dataset can capture the full range of hate speech contexts across various languages and cultures. In addition, although we recruited annotators who come from different socio-cultural backgrounds, opinions on hate speech remain subjective and one cannot include all possible perspectives of what constitutes hate speech or abuse. We mitigate the problem by sharing the individual annotations with the research community studying the problem.

Further, when using language identification to collect data, challenges due to code-mixing and digraphia, make the task non-trivial given the common usage of multilinguality in African languages. We address the problem by asking the annotators to flag any tweet that is not in the target language. We acknowledge, nevertheless, instances that may have been missed by our annotators.

Finally, we report on the various dataset statistics and model features. However, given the fact that we use some closed models in our experiments, and the class imbalance problem which is inherent to hate speech datasets, we do not claim that our results are fully replicable or generalisable.

### Ethical Considerations

AnnotatorsThe annotators involved in this study were compensated for their work by more than the minimum wage and any demographic information about them was shared with consent. We acknowledge the difficulty of annotating hate speech and abusive language on people's well-being. Therefore, the annotators could reach out to us and were allowed to quit at any time.

Language UseOur datasets focus on hate speech and abusive languages in 15 African languages. However, we do not claim that they represent the full usage of these languages. We further acknowledge the socio-cultural biases that can come with the data as views on hate highly differ from one person to another and those shared by our annotators cannot include all possible perspectives.

#### Intended uses and potential misuses

Our datasets focus on hate speech and abusive language. They present a first step towards studying the phenomenon in some low-resource African languages. However, as malicious data actors can misuse our resources, we follow the suggestions made by Schlichtkrull et al. (2023) for automated fact-checking researchers and clearly state the following:

* Models built using our datasets **should not be used for automated removal**.
* Our _data subjects_ are social media users.
* Our _data actors_ and _model owners_ should be users, moderators, experts, and researchers with background knowledge in the field, especially on the limitations of automated hate speech and abusive language detection models.
* Given the sensitivity of the task and the high risk of false positives, any constructed or 

deployed model using our data should be **human-in-the-loop** with the humans being native or near-native speakers.

## Acknowledgments

We are grateful for the work and insights shared by our annotators. This work was carried out with support from Lacuna Fund, an initiative co-founded by The Rockefeller Foundation, Google.org, and Canada's International Development Research Centre. The views expressed herein do not necessarily represent those of Lacuna Fund, its Steering Committee, its funders, or Meridian Institute. We also thank Nigerian Artificial Intelligence Research Scheme (NAIRS) for supporting this project. We thank Label Studio for providing the academic version annotation tool used in this project. We also thank OpenAI for providing API credits to Masakhane. Shamsuddeen acknowledges the support of Google DeepMind. Paul was supported by a MUR FARE 2020 initiative under grant agreement Prot. R20YSMBZ8S (INDOMITA) and the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (No. 949944, INTEGRATOR). Abinew acknowledges the support of LT group, University of Hamburg.

## References

* Aababu et al. [2022] Teshome Mulugeta Ababu and Michael Melese Woldeyohannis. 2022. Afan oromo hate speech detection and classification on social media. In _Proceedings of the thirteenth language resources and evaluation conference_, pages 6612-6619.
* Abdulmunin et al. [2024] Idris Abdulmumin, Sthembiso Mikhwanazi, Mahlatse Mbooi, Shamsuddeen Hassan Muhammad, Ibrahim Said Ahmad, Neo Putini, Mieheleketo Mathebula, Matimba Shingange, Tajudden Gwadabe, and Vukosi Marivate. 2024. Correcting FLORES evaluation dataset for four African languages. In _Proceedings of the Ninth Conference on Machine Translation_, pages 570-578, Miami, Florida, USA. Association for Computational Linguistics.
* Adam et al. [2023] Fatima Muhammad Adam, Abubakar Yakubu Zandam, and Isa Innwa-Dutse. 2023. Detection of offensive and threatening online content in a low resource language. _arXiv preprint arXiv:2311.10541_.
* Adelani et al. [2024] David Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba Alabi, Yanke Mao, Haonan Gao, and En-Shium Lee. 2024. SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 226-245, St. Julian's, Malta. Association for Computational Linguistics.
* Adelani et al. [2020] David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D'souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaba, Shruti Rijhwanti, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumbasubende, Perez Ogayo, Aremu Anouluwapo, Catherine Gitau, Deguene Mbaye, Jesusjoba Alabi, Seid Muhie Yimamon, Tajuddeen Rabiu Gradabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukibii, Verrah Oientedo, Iroro Orife, Davis David, Samba Ngom, Tosin Adevumi, Paul Rayson, Mofetlouya Adegemi, Gerald Muruiaki, Emmanuel Anebi, Chiamakchukwuneke, Nkiurka Odu, Eric Peter Wairagala, Samuel Overinde, Clemencia Siro, Tobius Saul Batesea, Temilola Oloyede, Yvonne Nambu, Victor Aknode, Deborah Nabagereka, Maurice Katustime, Ayodele Awokoya, Mouthamadane MBOUP, Dibora Gebrebyohanghes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulay Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahim DIOP, Abdoulay Diallo, Adevalae Akinfaderin, Tendai Marengeree, and Salomey Osei. 2021. MasakhaNER: Named Entity Recognition for African Languages. _Transactions of the Association for Computational Linguistics_, 9:1116-1131.
* Adjaji and Lazaridis [2013] Carol Adjaji and Gabriella Lazaridis. 2013. Migration, xenophobia and new racism in post-apartheid south africa. _International journal of social science studies_, 1:192-205.
* Alaibi et al. [2022] Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022. Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning. In _Proceedings of the 29th International Conference on Computational Linguistics_, pages 4336-4349, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
* Aliyu et al. [2022]Saminu Mohammad Aliyu, Gregory Maksha Waijga, Muhammad Murtala, Shamsuddeen Hassan Muhammad, Idris Abdulummin, and Ibrahim Said Ahmad. 2022. Herdbobia: a dataset for hate speech against fulani in nigeria. _arXiv preprint arXiv:2211.15262_.
* Amol et al. [2023] Cynthia Amol, Lilian Wanzare, and James Obuhuma. 2023. Politikweli: A swahili-english code-switched twitter political misinformation classification dataset. In _International Conference on Speech and Language Technologies for Low-resource Languages_, pages 3-17. Springer.
* Arora et al. [2023] Arnav Arora, Preslav Nakov, Momchil Hardalov, Sheikh Muhammad Sarwar, Vibha Nayak, Yoan Dhikov, Dimitrina Platkova, Kyle Dent, Ameya Bhatawdekar, Guillaume Bouchard, et al. 2023. Detecting harmful content on online platforms: what 

platforms need vs. where research efforts go. _ACM Computing Surveys_, 56(3):1-17.
* Aryabumi et al. [2024] Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Kelly Marchisio, Sebastian Ruder, Acyr F. Locatelli, Julia Kreutzer, Nick Frosst, Phil Blunsom, Marzieh Fadaee, A. Ustun, and Sara Hooker. 2024. Aya 23: Open weight releases to further multilingual progress. _ArXiv_, abs/2405.15032.
* Ayele et al. [2022] Abinew Ali Ayele, Skadi Dinter, Tadesse Destaw Belay, Tesfa Tegegne Asfaw, Seid Muhie Yimam, and Chris Biemann. 2022. The 5js in Ethiopia: Amharic hate speech data annotation using toblox crowdsourcing platform. In _2022 International Conference on Information and Communication Technology for Development for Africa (ICT4DA)_, pages 114-120. IEEE.
* Ayele et al. [2024] Abinew Ali Ayele, Esubalev Alemmehel Jalew, Adem Chanie Ali, Seid Muhie Yimam, and Chris Biemann. 2024. Exploring boundaries and intensities in offensive and hate speech: Unveiling the complex spectrum of social media discourse. In _Proceedings of the Fourth Workshop on Threat, Aggression & Cyberbullying @ LREC-COLING-2024_, pages 167-178, Torino, Italia. ELRA and ICCL.
* Aylele et al. [2023] Abinew Ali Ayele, Seid Muhie Yimam, Tadesse Destaw Belay, Tesfa Asfaw, and Chris Biemann. 2023. Exploring Amharic hate speech data collection and classification approaches. In _Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing_, pages 49-59, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria.
* Basile et al. [2019] Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. In _Proceedings of the 13th international workshop on semantic evaluation_, pages 54-63.
* Conneau et al. [2020] Alexis Conneau, Kartikav Khandelwal, Naman Goyal, Vishra Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451, Online. Association for Computational Linguistics.
* Davidson et al. [2017] Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In _Proceedings of the 11th International AAAI Conference on Web and Social Media_, ICWSM '17, pages 512-515.
* Dubey et al. [2021] Abhimanyu Dubey, Abhinav Jauhi, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenaev, Arthur Hinsvark, Arnu Rao, Aston Zhang, Aurelien Rodriguez, and et al. 2024. The llama 3 herd of models. _ArXiv_, abs/2407.21783.
* Egbunike et al. [2015] Nwaschukwu Andrew Egbunike, Noel A. Ihebuor, and Ngozi Joy Onyechi. 2015. Nature of tweets in the 2015 nigerian presidential elections.
* Feng et al. [2022] Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 878-891, Dublin, Ireland. Association for Computational Linguistics.
* Fortuna et al. [2019] Paula Fortuna, Joao Rocha da Silva, Leo Wanner, Sergio Nunes, et al. 2019. A hierarchically-labeled portuguese hate speech dataset. In _Proceedings of the third workshop on abusive language online_, pages 94-104.
* Gitari et al. [2015] Nijagi Dennis Gitari, Zhang Zuping, Hanytrwimfurra Damien, and Jun Long. 2015. A lexicon-based approach for hate speech detection. _International Journal of Multimedia and Ubiquitous Engineering_, 10(4):215-230.
* Ilkebare et al. [2024] Comfort Ilkebare, Jesujoba Alabiql, David Ifeoluwa Adelani, Firdous Bakare, Oluwatogvin Aiola, and Oluwaseyi Adeymo. 2024. EkoHate: Abusive language and hate speech detection for code-switched political discussions on Nigerian Twitter. In _Proceedings of the 8th Workshop on Online Abuse and Harms (WOAH 2024)_, pages 28-37, Mexico City, Mexico. Association for Computational Linguistics.
* Imani et al. [2023] Ayyoob Imani, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, Andre Martins, Francois Yuon, and Hinrich Schutze. 2023. Glot500: Scaling multilingual corpora and language models to 500 languages. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1082-1117, Toronto, Canada. Association for Computational Linguistics.
* Jiang et al. [2023] Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elia Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2023. Msternal Tb. _ArXiv_, abs/2310.06825.
* Kirk et al. [2023] Hannah Rose Kirk, Wenjie Yin, Bertie Vidgen, and Paul Rottger. 2023. Semeval-2023 task 10: Explainable detection of online sexism. _arXiv preprint arXiv:2303.04222_.
* Lee et al. [2024] Nayeon Lee, Chani Jung, Junho Myung, Jiho Jin, Jose Camacho-Collados, Juho Kim, and Alice Oh. 2024.

 

Exploring cross-cultural differences in English hate speech annotations: From dataset construction to analysis. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 4205-4224, Mexico City, Mexico. Association for Computational Linguistics.
* Lee et al. (2023) Nayeon Lee, Chani Jung, and Alice Oh. 2023. Hate speech classifiers are culturally insensitive. In _Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP)_, pages 35-46, Dubrovnik, Croatia. Association for Computational Linguistics.
* Madeddu et al. (2023) Marco Madeddu, Simona Frenda, Mirko Lai, Viviana Patti, and Valerio Basile. 2023. Disaggregate it corpus: A disaggregated italian dataset of hate speech. In _Italian Conference on Computational Linguistics_.
* Mandela (1994) Nelson Mandela. 1994. _Long Walk to Freedom: The Autobiography of Nelson Mandela_. Little, Brown and Company.
* Mansur et al. (2023) Zainab Mansur, Nazlia Omar, and Sabrina Tiun. 2023. Twitter hate speech detection: A systematic review of methods, taxonomy analysis, challenges, and opportunities. _IEEE Access_, 11:16226-16249.
* Maroniokakis et al. (2022) Antonis Maroniokakis, Axel Wisiorek, Leah Nann, Haris Jabbar, Sahana Udupa, and Hinrich Schutze. 2022. Listening to affected communities to define extreme speech: Dataset and experiments. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 1089-1104.
* Mathew et al. (2019) Binny Mathew, Ritam Dutt, Pawan Goyal, and Animesh Mukherjee. 2019. Spread of hate speech in online social media. In _Proceedings of the 10th ACM conference on web science_, pages 173-182.
* Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Agji, Khalid Almubarak, Samuel Albanie, Zaid Aylafeai, Albert Webson, Edward Raff, and Colin Raffell. 2023. Crosslingual generalization through multitask finetuning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15991-16111, Toronto, Canada. Association for Computational Linguistics.
* Muhammad et al. (2023) Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Abinew Ali Apele, Nedjma Ostidhoum, David Ifeoluwa Adelani, Seid Muhie Yimam, Ibrahim Said Ahmad, Meriem Beloucif, Saif M Mohammad, Sebastian Ruder, et al. 2023. Afirsenti: A twitter sentiment analysis benchmark for african languages. In _Proceedings of EMNLP_.
* Muhammad et al. (2016) Shamsuddeen Hassan Muhammad, David Ifeoluwa Adelani, Sebastian Ruder, Ibrahim S'id Ahmad, Idris Abdulmunin, Bello Shehu Bello, Monojit Chaudhury, Chris Chinney Emezue, Saehed Salathouede Abdullahi, Anouluwapo Aremu, Alipip Jorge, and Pavel Brazdil. 2022. NajiaSenti: A Nigerian Twitter sentiment corpus for multilingual sentiment analysis. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_, pages 590-602, Marseille, France. European Language Resources Association.
* Ndadbula et al. (2023) Joseph Nda Ndadbula, Oyenike Mary Olanrewaju, and Faith O Echobu. 2023. Detection of hate speech code mix involving english and other nigerian languages. _Journal of Information Systems and Informatics_, 5(4):1416-1431.
* Oguezji et al. (2021) Kelecchi Oguezji, Yuxin Zhu, and Jimmy Lin. 2021. Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages. In _Proceedings of the 1st Workshop on Multilingual Representation Learning_, pages 116-126, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Oladipo et al. (2023) Akintunde Oladipo, Mofetoluwa Adeyemi, Orevaoghene Ahia, Abraham Owodunni, Odunayo Ogundepo, David Adelani, and Jimmy Lin. 2023. Better quality pre-training data and t5 models for African languages. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 158-168, Singapore. Association for Computational Linguistics.
* Ombui et al. (2019) Edward Ombui, Lawrence Muchemi, and Peter Wagacha. 2019. Hate speech detection in code-switched text messages. In _2019 3rd International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT)_, pages 1-6. IEEE.
* Oriola and Kotze (2020) Oluwafemi Oriola and Eduan Kotze. 2020. Evaluating machine learning techniques for detecting offensive and hate speech in south african tweets. _IEEE Access_, 8:21496-21509.
* Ousdihoun et al. (2019) Nedjma Ousdihoun, Zizheng Lin, Hongming Zhang, Yangqiu Song, and Dit-Yan Yeung. 2019. Multilingual and multi-aspect hate speech analysis. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 4675-4684, Hong Kong, China. Association for Computational Linguistics.
* Ousdihoun et al. (2020) Nedjma Ousdihoun, Yangqiu Song, and Dit-Yan Yeung. 2020. Comparative evaluation of label-agnostic selection bias in multilingual hate speech datasets. In _Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP)_, pages 2532-2542.
* Raborife et al. (2024) Mpho Raborife, Blessing Oguchi Oguokiri, and Keithhele D. Aruleba. 2024. The role of social media in xenophobic attack in south africa. _Journal of the Digital Humanities Association of Southern Africa (DHASA)_ 

Justus J Randolph. 2005. Free-marginal multirater kappa (multirater k [free]): An alternative to fleiss' fixed-marginal multirater kappa. _Online submission_.
* Ridwanullah et al. (2024) Abdulhameed Olaitan Ridwanullah, Sulaiman Ya'u Sule, Bashiru Usman, and Lauratu Umar Abdulsalam. 2024. Politicization of hate and weaponization of twitter/x in a polarized digital space in nigeria. _Journal of Asian and African Studies_, page 00219096241230500.
* Rottger et al. (2022) Paul Rottger, Haitham Seelawi, Debora Nozza, Zeerak Talat, and Bertie Vidgen. 2022. Multilingual HatedCheck: Functional tests for multilingual hate speech detection models. In _Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH)_, pages 154-169, Seattle, Washington (Hybrid). Association for Computational Linguistics.
* Saha et al. (2019) Koustuv Saha, Eshwar Chandrasekharan, and Munmun De Choudhury. 2019. Prevalence and psychological effects of hateful speech in online college communities. _Proc ACM Web Sci Conf_, 2019:255-264.
* Scao et al. (2021) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili'c, Daniel Hesslow, Roman Castagn'e, Alexandra Sasha Luccioni, Francois Yvon, Matthias Galle, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammannanchoni, Thomas Wang, Benoit Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurenccon, Yacine Jerenite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa Etache, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris C. Emezue, Christopher Klamm, Colin Leong, Daniel Alexander van Strien, David Ifeoluwa Adellani, Dragomir R. Radev, Eduardo Gonz'alez Pomerferada, and et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. _ArXiv_, abs/2211.05100.
* Schlichtkrull et al. (2023) Michael Schlichtkrull, Nedjma Osidhoum, and Andreas Vlachos. 2023. The intended uses of automated fact-checking artefacts: Why, how and who. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 8618-8642.
* Shahid and Vashisha (2023) Farhana Shahid and Aditya Vashisha. 2023. Decolonizing content moderation: Does uniform global community standard resemble utopian equality or western power hegemony? In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_, pages 1-18.
* Talat and Hovy (2016) Zeerak Talat and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In _North American Chapter of the Association for Computational Linguistics_.
* Team et al. (2024) Gemma Team, Morgan Riviere, Shreya Pathak, Pier Giuseppe Sensca, Cassidy Hardin, Surya Bhupatiraju, Leonard Hussent, Thomas Mesward, Bobak Shahriari, Alexandre Ram'e, Johan Ferret, Peter Liu, Pouyo Dehghan Tafti, Abe Friesen, Michelle Casabon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammry Jerome, and et al. 2024. Gamma 2: Improving open language models at a practical size. _ArXiv_, abs/2408.00118.
* Tonja et al. (2024) Atnafu Lambebo Tonja, Bonaventre F. P. Dossou, Jessica Ojo, Jenalea Rajab, Fadel Thior, Eric Peter Wairagala, Aremu Anouluwapo, Pelonomi Moilooa, Jade Abbott, Vukosi Marivate, and Benjamin Rosman. 2024. Inkbulam: A small language model for low-resource african languages. _ArXiv_, abs/2408.17024.
* Tonneau et al. (2024) Manuel Tonneau, Pedro Quinta De Castro, Karim Lasri, Ibrahim Faroud, Lakshmi Subramanian, Victor Ozcozo-Olvera, and Samuel Fraiberger. 2024. Najja-Hate: Evaluating hate speech detection on Nigerian Twitter using representative data. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 9020-9040, Banggkk, Thailand. Association for Computational Linguistics.
* Tunstall et al. (2022) Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg. 2022. Efficient few-shot learning without prompts. _ArXiv_, abs/2209.11055.
* Vargas et al. (2024) Francielle Alves Vargas, Samuel Guimaraes, Shamsudden Hassan Muhammad, Diego Alves, Ibrahim Said Ahmad, Idris Abdulmunin, Diallo Mohamed, Thiago Alexandre Salgueiro Pardo, and Fabricio Beneventuto. 2024. Hausahate: an expert annotated corpus for haus haate speech detection. In _Proceedings_.
* Vidgen et al. (2021) Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021. Learning from the worst: Dynamically generated datasets to improve online hate detection. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1667-1682, Online. Association for Computational Linguistics.
* Vigna et al. (2017) Fabio Del Vigna, Andrea Cimino, Felice Dell'Orletta, Marinella Petrocchi, and Maurizio Tesconi. 2017. Hate me, hate me not: Hate speech detection on facebook. In _Italian Conference on Cybersecurity_.
* Yin and Zubiaga (2021) Wenjie Yin and Arkaitz Zubiaga. 2021. Towards generalisable hate speech detection: a review on obstacles and solutions. _PeerJ Computer Science_, 7:e598.
* Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the type and target of offensive posts in social media. In _Annual Conference of the North American Chapter of the Association for Computational Linguistics_. Association for Computational Linguistics (ACL) 

Appendix

### Prompt templates

```
\begin{tabular}{|p{341.7pt}|} \hline Fromp \\ \hline Read the following label definitions and provide a label without any explanations. \\ Hate: Hate speech is public speech that expresses hate or encourages violence towards a person or group based on something such as race, religion, gender, ethnicity, sexual orientation or other characteristics. \\ Abusive: Abusive and offensive language means verbal messages that use words in an inappropriate way and may include but is not limited to swearing, name-calling, or profanity. Offensive language may upset or embarrass people because it is rude or insulting. \\ Neutral: Neutral language is neither hateful nor abusive or offensive. It does not contain any bad language. \\ Text: tweet Label: & \\ \hline \end{tabular}
``` 
\begin{tabular}{|p{341.7pt}|} \hline Fromp \\ \hline Read the following text and definitions: \\ Text: tweet. \\ Definitions: Hate: Hate speech is public speech that expresses hate or encourages violence towards a person or group based on something such as race, religion, gender, ethnicity, sexual orientation or other than extremities. \\ Abuse: Abusive and offensive language are verbal messages that use words in an inappropriate way and may include but is not limited to swearing, name-calling, or profanity. Offensive language may upset or embarrass people because it is rude or insulting \\ Neutral: Neutral language is neither hateful nor abusive or offensive. It does not contain any bad language. \\ Text: tweet. Which of these definitions (hate, abuse, neutral) apply to this tweet? \\ \hline \end{tabular}

``` 
\begin{tabular}{|p{341.7pt}|} \hline Fromp \\ \hline Read the following label definitions and provide a label without any explanations. \\ Hate: Hate speech is public speech that expresses hate or encourages violence towards a person or group based on something such as race, religion, gender, ethnicity, sexual orientation or other characteristics. \\ Abusive: Abusive and offensive language means verbal messages that use words in an inappropriate way and may include but is not limited to swearing, name-calling, or profanity. Offensive language may upset or embarrass people because it is rude or insulting \\ Neutral: Neutral language is neither hateful nor abusive or offensive. It does not contain any bad language. \\ Text: tweet. Which of these definitions (hate, abuse, neutral) apply to this tweet? \\ \hline \end{tabular}

 



**Proposition 5.10**: _The number



\begin{table}
\begin{tabular}{l c r r r r r r r r r r r r r r r|r} \hline
**Model** & **\# Shots** & **anh** & **arry** & **aroq** & **hau** & **ibo** & **kin** & **oro** & **pcn** & **son** & **sws** & **tir** & **twi** & **xho** & **yor** & **zwl** & **Avg.** \\ \hline
**SetFit** & 0 & 33.79 & 46.54 & 27.40 & 23.57 & 36.90 & 25.29 & 26.26 & 43.46 & 32.18 & 44.38 & 49.30 & 51.13 & 37.38 & 48.46 & 35.08 & 37.41 \\  & 5 & 44.89 & 33.82 & 36.07 & 49.93 & 51.52 & 55.35 & 36.56 & 53.28 & 48.09 & 54.90 & 35.36 & 33.23 & 47.49 & 34.36 & 33.18 & 43.20 \\  & 10 & 49.42 & 35.81 & 46.55 & 49.72 & 39.31 & 53.10 & 39.10 & 57.00 & 42.29 & 62.64 & 39.16 & 45.76 & 43.52 & 49.73 & 40.33 & 46.23 \\  & 20 & 50.13 & 40.49 & 54.24 & 55.40 & 65.15 & 57.35 & 40.91 & 59.09 & 48.95 & 74.93 & 40.04 & 53.72 & 46.92 & 56.54 & 50.67 & 52.97 \\ \hline
**InkubaLM-0.4B** & 0 & 22.96 & 19.38 & 18.14 & 20.02 & 18.42 & 22.44 & 26.16 & 16.86 & 20.76 & 20.74 & 25.10 & 17.44 & 20.58 & 19.26 & 20.38 & 20.58 \\  & 5 & 28.04 & 24.70 & 20.06 & 21.78 & 21.98 & 25.78 & 26.86 & 19.08 & 25.06 & 20.48 & 29.26 & 24.50 & 20.34 & 20.98 & 21.68 & 23.55 \\  & 10 & 27.00 & 26.46 & 23.92 & 24.48 & 23.52 & 27.22 & 29.14 & 23.68 & 25.99 & 24.85 & 27.92 & 24.02 & 24.68 & 23.68 & 25.06 & 25.42 \\  & 20 & 27.06 & 26.42 & 24.60 & 24.30 & 23.36 & 28.66 & 29.85 & 28.58 & 23.76 & 25.88 & 24.64 & 30.04 & 23.32 & 23.56 & 23.14 & 25.20 & 25.42 \\ \hline
**mt0-small** & 0 & 18.86 & 16.98 & 13.42 & 12.22 & 9.72 & 17.40 & 20.68 & 12.40 & 15.16 & 14.07 & 19.30 & 8.56 & 14.20 & 12.04 & 13.62 & 14.58 \\  & 5 & 26.38 & 25.76 & 21.66 & 19.94 & 21.28 & 23.04 & 27.78 & 25.98 & 21.40 & 20.30 & 26.20 & 12.66 & 19.96 & 17.72 & 19.50 & 21.97 \\  & 10 & 27.38 & 23.72 & 20.40 & 20.94 & 22.36 & 21.08 & 27.94 & 25.58 & 19.88 & 16.39 & 26.62 & 12.08 & 18.56 & 19.90 & 18.88 & 21.48 \\  & 20 & 26.28 & 23.38 & 19.86 & 21.60 & 22.14 & 21.18 & 28.16 & 25.06 & 19.96 & 20.05 & 28.04 & 13.44 & 19.12 & 19.76 & 18.50 & 21.77 \\ \hline
**bloomaz-7b1-mt** & 0 & 17.18 & 21.86 & 21.86 & 18.60 & 18.42 & 21.98 & 19.38 & 22.60 & 16.74 & 25.10 & 16.92 & 17.70 & 19.24 & 22.36 & 19.80 & 19.92 \\  & 5 & 22.74 & 21.40 & 25.88 & 23.10 & 27.88 & 25.36 & 26.22 & 26.66 & 23.54 & 27.03 & 25.64 & 28.34 & 23.26 & 24.80 & 22.62 & 24.96 \\  & 10 & 27.86 & 21.06 & 25.88 & 24.40 & 27.90 & 22.54 & 27.86 & 26.24 & 25.48 & 19.83 & 28.58 & 25.36 & 24.16 & 23.92 & 25.18 \\  & 20 & 25.46 & 19.92 & 24.64 & 29.76 & 27.22 & 24.18 & 26.64 & 26.82 & 25.76 & 24.50 & 28.48 & 28.84 & 23.24 & 25.28 & 23.88 & 25.35 \\ \hline
**Mistral-7B+v0.1** & 0 & 21.18 & 17.46 & 11.98 & 6.84 & 8.04 & 15.18 & 20.82 & 6.76 & 8.16 & 10.64 & 25.62 & 8.38 & 9.72 & 8.22 & 8.92 & 12.53 \\  & 5 & 37.68 & 31.60 & 44.54 & 34.90 & 36.84 & 46.98 & 39.46 & 50.00 & 34.12 & 49.58 & 35.64 & 32.04 & 34.24 & 41.04 & 32.54 & 38.75 \\  & 10 & 39.68 & 36.50 & 50.56 & 37.82 & 36.94 & 45.72 & 39.02 & 51.20 & 31.82 & 54.34 & 35.84 & 30.82 & 36.44 & 24.90 & 32.46 & 39.94 \\  & 20 & 36.64 & 35.46 & 52.94 & 38.44 & 38.98 & 25.28 & 39.24 & 54.60 & 33.28 & 55.98 & 37.42 & 30.44 & 34.12 & 46.40 & 35.00 & 44.14 \\ \hline
**avg-23-35B** & 0 & 17.04 & 19.20 & 17.20 & 20.34 & 14.04 & 21.34 & 20.38 & 20.02 & 18.56 & 12.20 & 17.10 & 16.08 & 20.32 & 17.48 & 20.24 & 18.34 \\  & 5 & 37.78 & 56.40 & 57.00 & 39.70 & 42.00 & 48.78 & 39.24 & 57.90 & 38.72 & 56.32 & 35.68 & 40.24 & 36.76 & 48.88 & 44.58 & 45.38 \\  & 10 & 38.50 & 53.96 & 61.82 & 44.42 & 45.80 & 52.82 & 42.16 & 59.26 & 37.60 & 61.08 & 35.88 & 39.92 & 36.18 & 48.44 & 45.00 & 46.86 \\  & 20 & 38.10 & 52.90 & 63.76 & 46.14 & 46.52 & 57.62 & 42.26 & 61.68 & 38.68 & 37.68 & 38.12 & 38.54 & 37.74 & 51.52 & 45.42 & 45.78 \\ \hline
**Gamma-2-9B** & 0 & 27.42 & 28.36 & 25.46 &



\begin{table}
\begin{tabular}{l c r r r r r r r r r r r r r r r r r r}
**model** & **\#shots** & **anh** & **ary** & **arq** & **hau** & **ibo** & **kin** & **oro** & **pcn** & **son** & **sws** & **tir** & **twi** & **xho** & **yor** & **zul** & **Avg.** \\ \hline
**SetFit** & 0 & 33.20 & 42.92 & 26.63 & 24.02 & 38.61 & 21.29 & 28.72 & 39.23 & 29.26 & 35.26 & 47.32 & 52.15 & 37.62 & 41.27 & 35.03 & 35.50 \\  & 5 & 45.11 & 35.91 & 31.89 & 50.81 & 50.30 & 51.26 & 36.50 & 52.54 & 49.93 & 54.83 & 31.63 & 38.40 & 46.14 & 31.62 & 35.71 & 42.84 \\  & 10 & 49.26 & 37.05 & 44.27 & 50.43 & 40.80 & 48.60 & 42.56 & 56.87 & 40.27 & 62.88 & 39.74 & 49.28 & 42.60 & 49.21 & 42.03 & 46.39 \\  & 20 & 50.33 & 40.92 & 53.25 & 54.34 & 65.65 & 57.28 & 43.08 & 58.44 & 48.86 & 74.68 & 38.95 & 55.87 & 46.78 & 53.85 & 51.10 & 52.89 \\ \hline
**InkubaLM-0.4B** & 0 & 39.74 & 23.62 & 30.26 & 22.66 & 23.04 & 30.96 & 41.14 & 19.84 & 28.30 & 27.02 & 47.24 & 23.16 & 27.30 & 23.70 & 27.38 & 29.02 \\  & 5 & 39.12 & 23.72 & 31.88 & 25.28 & 34.36 & 32.60 & 44.88 & 23.76 & 30.86 & 25.62 & 48.12 & 43.96 & 28.14 & 26.88 & 27.52 & 32.45 \\  & 10 & 37.20 & 29.52 & 32.64 & 38.38 & 27.74 & 37.58 & 43.26 & 28.82 & 41.26 & 31.22 & 45.84 & 32.90 & 34.22 & 31.54 & 35.76 & 35.19 \\  & 20 & 37.64 & 29.18 & 32.68 & 38.64 & 28.70 & 37.84 & 43.06 & 28.98 & 41.34 & 31.37 & 48.88 & 32.30 & 33.82 & 30.84 & 36.12 & 35.43 \\ \hline
**mt0-small** & 0 & 38.14 & 23.36 & 31.76 & 28.70 & 17.02 & 35.56 & 43.74 & 22.70 & 31.34 & 27.47 & 42.80 & 14.66 & 28.32 & 23.16 & 27.62 & 29.09 \\  & 5 & 36.70 & 24.90 & 34.12 & 29.18 & 29.48 & 31.36 & 41.10 & 32.06 & 25.26 & 27.90 & 42.56 & 16.20 & 33.04 & 28.38 & 32.86 & 31.01 \\  & 10 & 37.40 & 25.74 & 35.06 & 26.36 & 33.44 & 28.30 & 40.46 & 31.98 & 24.00 & 23.37 & 43.04 & 15.88 & 31.32 & 29.72 & 32.00 & 30.54 \\  & 20 & 36.50 & 25.98 & 35.08 & 26.20 & 34.50 & 28.44 & 40.30 & 31.54 & 23.86 & 23.40 & 44.76 & 17.08 & 31.68 & 28.92 & 32.30 & 30.70 \\ \hline
**bloomix-7b1-mt** & 0 & 34.62 & 34.38 & 33.08 & 32.66 & 29.18 & 36.20 & 37.30 & 32.76 & 33.90 & 38.53 & 55.82 & 28.88 & 31.94 & 33.70 & 32.34 & 33.68 \\  & 5 & 32.36 & 37.34 & 46.74 & 37.52 & 62.56 & 30.88 & 31.96 & 48.14 & 33.90 & 40.97 & 36.68 & 68.50 & 39.36 & 48.86 & 39.64 & 42.36 \\  & 10 & 35.16 & 36.96 & 45.92 & 37.72 & 62.40 & 27.74 & 33.06 & 47.10 & 35.04 & 36.97 & 38.70 & 67.94 & 38.56 & 47.66 & 39.44 & 44.02 \\  & 20 & 33.16 & 35.60 & 44.18 & 37.88 & 63.12 & 29.12 & 31.82 & 47.04 & 34.78 & 37.93 & 38.72 & 68.84 & 38.14 & 49.08 & 39.08 & 41.90 \\ \hline
**Mistral-7B-v0.1** & 0 & 42.26 & 30.62 & 17.28 & 10.52 & 13.00 & 27.62 & 45.42 & 11.14 & 13.86 & 18.88 & 55.02 & 14.28 & 16.36 & 13.44 & 14.54 & 22.95 \\  & 5 & 37.92 & 43.46 & 57.14 & 51.96 & 58.42 & 50.40 & 45.70 & 59.32 & 52.64 & 58.58 & 41.36 & 69.52 & 47.50 & 57.72 & 45.96 & 51.84 \\  & 10 & 40.42 & 46.68 & 60.66 & 56.38 & 61.74 & 50.00 & 46.70 & 60.42 & 51.50 & 63.18 & 44.88 & 70.26 & 46.40 & 60.50 & 46.42 & 53.74 \\  & 20 & 39.48 & 45.08 & 61.70 & 56.98 & 65.10 & 55.64 & 48.82 & 61.22 & 54.02 & 64.28 & 51.38 & 70.04 & 48.64 & 64.94 & 50.16 & 55.83 \\ \hline
**aya-23-35SB** & 0 & 34.54 & 32.76 & 28.52 & 37.76 & 18.66 & 41.04 & 42.48 & 30.34 & 40.76 & 34.04 & 35.84 & 14.40 & 35.42 & 28.22 & 35.96 & 32.72 \\  & 5 & 42.20 & 61.86 & 64.38 & 55.32 & 56.42 & 56.56 & 49.96 & 61.12 & 57.46 & 69.84 & 45.70 & 63.86 & 48.96 & 63.48 & 51.14 & 56.55 \\  & 10 & 45.74 & 61.22 & 69.00 & 59.78 & 63.98 & 58.86 & 52.30 & 62.08 & 56.98 & 74.10 & 51.08 & 69.58 & 49.06 & 66.50 & 52.44 & 59.50 \\  & 20 & 46.76 & 59.70 & 71.40 & 59.84 & 68.30 & 62.06 & 54.14 & 65.44 & 57.78 & 75.68 & 56.90 & 71.20 & 51.72 & 70.54 & 56.12 & 61.84 \\ \hline
**Llama-3.1-8B** & 0 & 29.14 & 34.10



## Chapter 1 Introduction

Figure 1: Annotation Guidelines and Definitions