AfriSenti: A Twitter Sentiment  Analysis Benchmark for African Languages
Authors: Shamsuddeen Hassan Muhammad
Idris Abdulmumin
Abinew Ali Ayele
Nedjma OUSIDHOUM
David Ifeoluwa Adelani
Seid Muhie Yimam
Ibrahim Said Ahmad
Meriem Beloucif
Saif M. Mohammad
Sebastian Ruder
Oumaima Hourrane
Alipio Jorge
Pavel Brazdil
Felermino D. M. A. Ali
Davis David
Salomey Osei
Bello Shehu-Bello
Falalu Ibrahim Lawan
Tajuddeen Gwadabe
Samuel Rutunda
Tadesse Destaw Belay
Wendimu Baye Messelle
Hailu Beshada Balcha
Sisay Adugna Chala
Hagos Tesfahun Gebremichael
Bernard Opoku
Stephen Arthur
Venue: EMNLP 2023 Main (2023)
Keywords: Africa
Sentiment
Dataset
NLP

Abstract: Africa is home to over 2,000 languages from over six language families and has the highest linguistic diversity among all continents. This includes 75 languages with at least one million speakers each. Yet, there is little NLP research conducted on African languages. Crucial in enabling such research is the availability of high-quality annotated datasets. In this paper, we introduce AfriSenti, a sentiment analysis benchmark that contains a total of >110,000 tweets in 14 African languages (Amharic, Algerian Arabic, Hausa, Igbo, Kinyarwanda, Moroccan Arabic, Mozambican Portuguese, Nigerian Pidgin, Oromo, Swahili, Tigrinya, Twi, Xitsonga, and Yoruba) from four language families. The tweets were annotated by native speakers and used in the AfriSenti-SemEval shared task (with over 200 participants, see website: https://afrisenti-semeval.github.io). We describe the data collection methodology, annotation process, and the challenges we dealt with when curating each dataset. We further report baseline experiments conducted on the AfriSenti datasets and discuss their usefulness.

AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages
Shamsuddeen Hassan Muhammad Idris Abdulmumin Abinew Ali Ayele', Nedjma Ousidhoum David Ifeoluwa Adelani Seid Muhie Yimam,, Ibrahim Sa'id Ahmad Meriem Beloucif , Saaf M. Mohammad"),
Sebastian Ruder"1, Oumaima Hourrane Pavel Brazdil's 3, Alípio Jorge 1,13 Felermino Dário Mário António Ali', Davis D Salomey Osei Bello Shehu Bello", Falalu Ibrahim 16 Tajuddeen Gwadabe Samuel Rutunda", Tadesse Belay Wendimu Baye Messelle', Hailu Beshada Balcha Sisay Adugna Chala Hagos Tesfahun Gebremichael Bernard Opoku"', Steven Arthur
University Porto, Portugal Baayero University Kano, "Ahmadu Bello University. Zaria, 'Bahir Dar University,
" "University Cambridge, 'Cardiff University. UUniversity College London, Universitää Hamburg, 'Uppsala University. National Research Council Canada, "Google Research, Hassan II University of Cusablanca, LLIAAD INESC TEC.
dLah, 15 University of Deusto, "Kaduna State University, DDigital Umuganda, "Wolo University. "Jimma University, Fraunhofer FIT, Accra Institute of Technology. 'Masakhane NLP, "HausaNLP
shmuhammad.c cscēbuk. eeriouukkedu.ng.. edu.
Abstract
Morocco
Africa is home to over 2,000 languages from more than six language families and has highest linguistic diversity among all conti- nents. These include 75 languages with at least one million speakers each. Yet, there is little NL.P research conducted African lan- guages. Crucial to enabling such research is the availability of high-quality annotated datasets. In this paper, we introduce AfriSenti, a senti- ment analysis benchmark that contains total of >> 110,000 tweets in 14 African languages (Amharic, Algerian Arabic, Hausa, Igbo, Kin- yarwanda, Moroccan Arabic, Mozambican Por- tuguese, Nigerian Pidgin, Oromo, Swahili, Tigrinya, Twi, Xitsonga, and Yorùbá) from four language families. The tweets were annotated by native speakers and used in the AfriSenti- SemEval shared task !
Gianaa
Languages
Rwando
Algurian
TATONTE
Arrhuri: Oroma, Tigrinys
Iyho, Nigerann Pagn.
Kmyanwands
Portuguaso, Xisongg
Twe
Figure 1: Countries and languages represented in AfriSenti: Amharic, Algerian Arabic, Hausa, Igbo, Kin- yarwanda, Moroccan Arabic, Mozambican Portuguese, Nigerian Pidgin, Oromo, Swahili, Tigrinya, Twi, Xit- songa, and Yorùbá.
We describe the data collection methodology, annotation process, and the challenges we dealt with when curating each dataset. We further report baseline experiments conducted on the different datasets and discuss their usefulness.
2007) and in recent years, it has seen a prolifera- tion of communication in digital and social media. Code-switching common these new forms of communication where speakers alternate between two or more languages in the context of a single conversation (Angel et al., 2020; Thara and Poor- nachandran, 2018; Santy et al., 2021). However, despite this linguistic richness, African languages have been comparatively under-represented in nat- ural language processing (NLP) research.
1 Introduction
Africa has a long and rich linguistic history, experi- encing language contact, language expansion, de- velopment of trade languages, language shift, and language death on several occasions. The continent is incredibly diverse linguistically and home to over 2,000 languages. These include 75 languages with at least one million speakers each. Africa has rich tradition of storytelling, poems, songs, and literature Wallace, 2002; Hurter-Blcck
An influential sub-area of NLP deals with sen- timent, valence, emotions, and affect language (Liu, 2020). Computational analysis of emotion
'The AfriSenti Shared Task had over 200 participants. See website: https: / wwwarffiiiiiseeee seneval. .ol..githh gitbub.ic io

---

Lang-
Tweet
Sentiment
anh
8 aah' hcom the 8ithmm hebt "HiH-, RA-AA di Afz.m- FROG 0r4 Mr20i yar NOR?
negalive
arq
Guser ... وابوو lasis مررة "JA4 2
الشرين
negative
ary
Ebble داار Je 1s غقا ale واا4يي gs:w) J برشر واار ملرمم
negalive
ary
rabi ykhali alhbiba makayn ghir nachat chi machat
positive
hau
#USER. Aunt rahma i luv wallsh irin totally dinnan mmadu
positive
akowaro ya ofuma kai daalu
positive
kin
Guser Ariko akokamu ngo inyebebe unyujijemo sisawa wangu
negative
orm
Ruser Jawaar Kenya OMN haala akkamin argachuu dandeenya
neutral
por
Honestidade é algo que não compra. Infelizmente a humanidade esqueceu disso por causs das suas ambições.
positive
pcm
E. don lay wey don dey crush this fine
positive
SWa
Asaute sana watu wa Sirari jimbo la Tarime vijijini Huu ni Upendo
positive
usio na Mashaka kwa Mbunge wenu John Heehe
tir
Tuser how Tietes" 23 117 ardbd bRC An 7P44 8all h-90!
negative
t80
Quser Kuser Yu tindzavs ? Tsiks mbangui mpfann e nita ku despro- gramara
negative
saf den check bp na wo kwame danso di twe da kor negalive wawu
yor
onírèógbé aláádūgbö ati olójūkóköró
negative
Table 1: Examples of tweets and their sentiments in the different AfriSenti Languages. Note that the collected tweets Moroccan Arabic/Darija (ary) are written both Arabic and Latin scripts. The translations can found in the Appendix (Table 10).
states language and the creation of systems that predict these states from utterances have applica- tions in literary analysis and culturomics (Hamilton et al., 2016; Reagan et al., 2016), e-commerce (e.g., tracking feelings towards products), and rescarch in psychology and social science (Dodds et al., 2015; Hamilton al., 2016). Despite the tremendous amount of work in this important space over the last two decades, there is little work on African languages, partially due to a lack of high-quality annotated data.
and provide examples annotated tweets Table 1. The datasets were used in the first Afrocentric SemLival shared task SemEval 2023 Task 12: Sen- timent analysis for African languages (AfriSenti- SemEval) (Muhammad et al., 2023). We publicly release the data, which provides further opportuni- ties investigate the difficulty of sentiment anal- ysis for African languages by e.g., building sen- timent analysis systems for various African lan- guages, and studying of sentiment and contempo- rary language use in these languages.
To enable sentiment analysis research in African languages, present AfriSenti, the largest sen- timent analysis benchmark for under-teeeeeented African languages- unguages--overing 1100000+ tweets an- notated as positive, negative or neutral, in 14 languages² (Amharic, Algerian Arabic, Hausa, lgbo, Kinyarwanda, Moroccan Arabic, Mozambi- can Portuguese, Nigerian Pidgin, Oromo, Swahili, Tigrinya, Twi, Xitsonga, and Yorùbá) from four language families AAfro-Asiatic English Creole, Indo-European and Niger-Congo)*, We show the represented countries and languages Figure
Our contributions are: (1) the creation of the largest Twitter dataset for sentiment analysis African languages by annotating ten new datasets and curating four existing ones (Muhammad et al., 2022), (2) the discussion of the data collection and annotation process in 14 low-resource African lan- guages, (3) the release of sentiment lexicons for these languages, (4) the presentation of classifica- tion baseline results using our datasets.
2 Related Work
Research in sentiment analysis developed since the early days of lexicon-based sentiment analy- sis approaches (Mohammad et al., 2013; Tabouda et al., 2011 I; Turney, 2002) to more advanced ML
FFor simplicity, we use the term language to refer to lan- guage varieties including dialects.
'The datasets are publicly availahle on sattps://gthhhe www./moraiiio /ataafrriietiii. senetii-eeeevllll waw.sserrr-ttaeaaaereeeraaarssee sestteeeneval..... semeval-

---

Language ISO Code
Subregion
Spoken
Script
Amharic
anh arq hau
East Africa
Ethiopia
Ethiopic
Algerian Arahic/Darja
North Africa
Algeria
Arabic
Hausa
West Africa
Northern Nigeria, Ghana, Cameroon,
Latinn
Tgbo
bb
West Africa
Southeastern Nigeria
Latin
Kinyarwanda
kin
East Africa
Rwanda
Latin
Moroccan Arajic/Darija ary Mozumbican Portuguese pt-MZ
North Africa
Morocco
Arabic/Latin
Southeastern Africa
Mozambique
Latin
Nigerian Pidgin
West Africa
Nigeria, Ghana, Cameroon,
Latin
Oromo
East Africa
Ethiopia
Latin
Swahili
swa
East Africa
Tanzania, Kenya, Mozambique
Latin
Tigrinya
tir
East Africa
Ethiopia
Fthiopic
Twi
twi
West Africa
Ghana
Latin
Xitsonga
Southern Africa
Mozambique, South Africa, Zimbahwe, Eswatini Southwestem and Central Nigeria
Latin
Yorihá
yor West Africa
L.tiin
Table 2: African languages included in our study (Lewis, 2009). For each language, report its ISO code, the African sub-regions is mainly spoken in, the writing scripts included its dataset collection.
approaches (Agarwal and Mittal, 2016; Le and Nguyen, 2020), deep learning- beasenggbbased based methods (Ya- dav and Vishwakarma, 2020; Zhang et al., 2018), and hybrid approaches (Gupta and Joshi, 2020; Kaur al., 2022). Nowadays, Pretrained Language Models (PLMs), e.g., XLM-R (Conneau et al., 2020), mDeBERTaV3 (He et al., 2021), AfriB- ERTa (Ogueji et al., 2021b), AfroXLMR (Alabi et al., 2022) and XLM-T (Barbieri et al., 20226), help achieve state-of-the-art performance for this task.
AfriSenti Language Families
Afro-Asiatic
Niger-
Hausa
Twi
Amharic
Igbo
Oromo
Yoruba
Tigrinya
Swahiii
Algerian Arabic
Xitsonga
Kinyarwanda
English Creole
Indo-European
Nigerian Pidgin
Portuguese
Recent work in sentiment analysis focused on subtasks that tackle new challenges, includ- ing aspect-based (Chen et al., 2022), multimodal (Liang et al., 2022), explainable ) (Cambria et al., 2022), and multilingual sentiment analysis (Muhammad et al., 2022). On the other hand, standard sentiment analysis subtasks such polarity classification (positive, negative, neu- tral) are widely considered saturated and solved (Poria al., 2020), with an accuracy of 97.5% in certain domains (Jiang al., 2020; Raffel al., 2020). However, while this may be true for high- resource languages in relatively clean, long-form text domains such as movie reviews, noisy user- generated data in under-represented languages still presents a challenge (Yimam et al., 2020). Addi- tionally, African languages present other difficul- ties for sentiment analysis such dealing with tone, code-switching, and digraphia (Adebara and Abdul-Mageed, 2022). Existing work sentiment analysis for African languages has therefore mainly focused on polarity classification (El Abdouli et al.,
Figure 2: The language Family (in green) of each lan- guage (in yellow) included in AfriSenti.
2017; Martin et al., 2021; Mataoui et al., 2016; Moudjari et al., 2020; Muhammad et al., 2022; Yi- mam et al., 2020). Our benchmark, AfriSenti, is the largest multilingual dataset for sentiment analysis in African languages.
Overview of the AfriSenti Datasets
AfriSenti covers 14 African languages (see Table 2), each with unique linguistic characteristics and writing systems. As shown in Figure 2, the bench- mark includes six languages of the Afro-Asiatic family, six languages of the Niger-Congo family, one from the English Creole family, and one from the Indo- auropeu family.
Writing Systems Scripts serve not only as a means of transcribing spoken language, but also as powerful cultural symbols that reflect people's

---

identity (Sterponi and Lai, 2014). For instance. the Bamun script is deeply connected to the iden- tity of Bamun speakers in Cameroon, while the
Lang.
New
Existing
Source
ama test
train, dev
Yimam et al. (2020)
arq
all
X
Geez/Ethiopic script (for Amharic and Tigrinya) ary
all
X
evokes the strength and significance of Ethiopian culture (Sterponi and Lai, 2014). Similarly, the Ajami script, a variant of the Arabic script used in various African languages such as Hausa, serves as reminder of the rich African cultural heritage of
hau
all X
Muhammad et al. (2022)
ibo
all
Muhammad et al. (2022)
kin
all
X
orm
all
X
pcm
X
all
Muhammad et al. (2022)
pt-MZ
all
X
all
X
the Hausa community (Gee, 2005).
all
X
African languages, with a few exceptions, use the Latin script, written from left to right, or the Arabic script, written from right to left (Gee, 2005; Meshesha and Jawahar, 2008), with the Latin script being the most widely used in Africa (Eberhard et al., 2020). Ten languages out of fourteen in AfriSenti are written in Latin script, two in Arabic script, and two in Ethiopic (or Geez) script. On social media, people may write Moroccan Arabic (Moroccan Darija) and Algerian Arabic (Algerian Darja) in both Latin and Arabic characters due to various reasons including access to technology. i.e., the fact that Arabic keyboards were not easily accessible on commonly used devices for many years, code-switching, and other phenomena. This makes Algerian and Moroccan Arabic digraphic, i.e. their texts can be written in multiple scripts on social media 4. Similarly, Amharic is digraphic and is written in both Latin and Geez script (Belay et al., 2021).
tso
all
X
twi
all
X
yor
all
Muhammad et al. (2022)
Table 3: The AfriSenti datasets. We show the new and previously available datasets (with their sources).
task.
4 Data Collection and Processing
Twitter's Limited Support for African Lan- guages Since many people share their opinions on Twitter, the platform is widely used to study sentiment analysis (Muhammad et al., 2022). How- ever, the Twitter API's support for African lan- guages is limited $, which makes it difficult for researchers collect data. Specifically, the Twitter language API supports only Amharic out of more than 2,000 African languages This disparity language coverage highlights the need for further research and development in NLP for low-resource languages.
Geographic Representation AfriSenti covers the majority of African sub-regions. Many African languages are spoken in neighbouring countries within the same sub-regions. For instance, varia- tions of Hausa are spoken in Nigeria, Ghana, and Cameroon, while Swahili variants are widely spo- ken in East African countries, including Kenya, Tanzania, and Uganda. AfriSenti also includes datasets in the top three languages with the highest numbers of speakers in Africa (Swahili, Amharic, and Hausa). Figure shows the geographic distri- bution of the languages represented in AfriSenti.
4.1 Tweet Collection
We used the Twitter Academic API to collect tweets. However, the API does not provide language identification for tweets in African lan- guages, used location-based and vocabulary- based heuristics to collect the datasets.
4.1.1 Location-based data collection
For all languages except Algerian Arabic and Afaan Oromo, we used a location-based collec- tion approach to filter out results. Hence, tweets were collected based on the names of the countries where the majority of the target language speakers
New and Existing Datasets AfriSenti includes existing and newly created datasets shown in Table 3. For the existing datasets whose test sets are public, we created new test sets further evaluate their performance in the
shared
'The data collection process was conducted before Decem- 20th 2022. Le.. before change of policy that took place in 2023.
'Table shows an example of Moroccan Arabic/Darija tweets written in Latin and Arabic script. For Algerian Ara- bic/Darja and Amharic, AfriSenti includes data in only Arabic and Geez, scripts.
"ht htap:://bble /bblg. wwwtter. woom///nnneerinn/enn uu// 72015/eaaluating-aaa
idet ricction-aai petratiioo-prrrrrrm
ance

---

5,00
Lang.
Manually
Translated
Source
iipostiinc Rincutral
7,000
ama
K
Yimam et al. (2020)
arq
hau
Muhammad et al. (2022)
2.000
ibo
Muhammad et al. (2022)
ary
4,000
Yimam al. (2020)
2,000
Muhammad et al. (2022)
pt-HZ V
kin X
1,000
swa
tir
Vimam et al. (2020)
16
tso
Languages
yor
Muhammad al. (2022)
Figure 3: Label distributions for the different AfriSenti datasets (i.e., number positive, negative, neutral tweets).
Table 4: Manually collected and translated lexicons in AfriSenti.
are located. For Afaan Oromo, tweets were col- lected globally due to the small size of the initial data collected from Ethiopia.
tive speakers verified the generated lists before use. This approach worked the best for Xistonga.
Data collection using sentiment lexicons As data collection based on stopwords sometimes re- sults in tweets that are inadequate for sentiment analysis (c.g.. too many neutral tweets), we used a sentiment lexicon-a dictionary of positive and negative words-for for tweet collection. This allows for a balanced collection across sentiment classes (positivehnegaaiiveeeuutee
4.1.2 Vocabulary-based Data Collection
As different languages are spoken within the same region in Africa (Amfo and Anderson, 2019), the location-based approach did not help in all cases. For instance, searching for tweets from "Lagos" (Nigeria) returned tweets in multiple languages, such Yorùbú, lgbo, Hausa, Pidgin, English, etc.
For Moroccan Arabic, used emotion word list curated by Outchakoucht and Es-Samaali (2021).
To address this, combined the location-based approach with vocabulary- based collection strate- gies. These included the use of stopwords, senti- ment lexicons, and a language detection tool. For languages that used the Geez script, we used the Ethiopic Twitter Dataset for Amharic (ETD-AM), which includes tweets that were collected since 2014 (Yimam al., 2019).
Table provides details on the sentiment lexi- cons in AfriSenti and indicates whether they were manually created or translated.
Data collection using mixed lists of words Be- sides stopwords and sentiment lexicons, native speakers provided lists of language-specti terms including generic words. For instance, this strategy helped us collect Algerian Arabic tweets, and the generic terms included equivalents of words such as " الللي " (the crowd) and names of Algerian cities. Figure 3 shows the distribution of the three sen- timent labels (i.e., positive, negative, and neutral) for each language.
Data collection using stopwords Most African languages do not have curated stopword lists (Emezue et al., 2022). Therefore, we created stop- word lists for some AfriSenti languages and used them to collect data. We used corpora from differ- ent domains, i.e., rank words based on their frequency (Adelani et al., 2021). We filtered out the top 100 words by deleting domain specific words (e.g.. the word God in religious texts) and created lists based on the top 50 words that appeared across domains.
data and religious texts,
4.2 Language Detection
As we mainly used heuristics for data collection, the collected tweets included some in different lan- guages. For instance, when collecting tweets ing lists of Amharic words, some returned tweets were in Tigrinya, due Amharic-Tigrinya code- mixing. Similarly, when searching for Algerian Arabic tweets, Tunisian, Moroccan, and Modern Standard Arabie tweets were found due to overlap-
We also used word co-occurrence-bae based ap- proach extract stopwords (Liang et al., 2009) using text sources from different domains. We lower-cased and removed punctuation marks and numbers, constructed co-occurrence graph, and filtered out the words that occurred most often. Na-

---

ping terms. Hence, we used different techniques for language detection as a post-processing step.
Prabhakaran et al. (2021) showed that the ma- jority vote conceals systematic disagreements be- tween annotators resulting from their sociocultural backgrounds and experiences. Therefore, we re- lease all the individual labels to the research com- munity. We report the free marginal multi-rater kappa scores (Randolph, 2005) in Table 5 since chance-adjusted scores such Fleiss-k can be low despite high agreement due to the imbalanced la- bel distributions (Falotico and Quatto, 2015; Mathe- son, 2019; Randolph, 2005). We obtained interme- diate to good levels of agreement (0.40- - 0.75) across all languages, except for Afaan Oromo where we obtained a relatively low agreement score due to the annotation challenges that we discuss in Section 5.
Language detection using existing tools Few African languages have pre-existing language de- tection tools (Keet, 2021). We used Google CLD3? and the Pycld2 library for the supported AfriSenti languages (Amharic, Oromo and Tigrinya).
Manual language detection For languages that do not have a pre-existing tool, the detection was conducted by native speakers. For instance, anno- tators who are native speakers of Twi and Xitsonga manually labeled 2,000 tweets in these languages. In addition, native speakers collected the Alge- rian Arabic tweets, they deleted all possible tweets expressed in another language or a different Arabic variation.
Table 6 shows the number of tweets in each of the 14 datasets. The Hausa collection of tweets is the largest among all the datasets and the Xitsonga dataset is the smallest one. Figure 3 shows the dis- tribution of the labeled classes in the datasets. We observe that the distribution for some languages such as ha is fairly equitable while in others such as pem, the proportion of tweets in each class varies widely. Sentiment annotation for African languages presents some challenges (Muhammad et al., 2022) that we highlight in the following.
Language detection using pre-trained language models To reduce the effort spent on language de- tection, we also used a pre-trained language model fine-tuned on 2,000 manually annotated tweets (Caswell al., 2020) to identify Twi and Xitsonga. Despite our efforts detect the languages, we note that as multilingualism is common in African soci- eties, the final dataset contains some codc-mixed tweets.
4.3 Tweet Anonymization and Pre-processing We anonymized the tweets by replacing all @ tions by @user and removed all URLs. For the Nigerian language test sets, further lower-cased the tweets (Muhammad et al., 2022).
Hausa, Igbo, and Yorùbá Hausa, Igbo, and Yorùbá are tonal languages, which contributes the difficulty of the task as the tone is rarely fully rendered in written form. E.g., in Hausa, Bàaba means dad and Baabà means mum (typically writ- ten Baba on social media), and in Yorùbá, èdè means language, and edé means crayfish (typically written ede on social media).
5 Data Annotation Challenges
Tweet samples were randomly selected based on the different collection strategies. Then, with the exception of the Ethiopian languages, each tweet was annotated by three native speakers.
Further, the intonation, which is crucial to the understanding of the tweet may not be conveyed in the text. E.g., ò nwèkwärå mgbe i naenwe sense ? (will you ever be able to talk sensibly? - You' 're a fool.) and ò nwěkwårå mgbe naenwe sense some- times you with great maturity. -I'm impressed.) are almost identical but carry different sentiments (i.e., negative and positive, respectively). this case, the difference the intonation may not be clear either due to the (non)use of punctuation or the lack of context.
We used the Simple Sentiment Questionnaire annotation guide by Mohammad (2016) and used majority voting (Davani et al., 2021) to determine the final sentiment label for each tweet (Muham- mad et al., 2022). We discarded the cases where all annotators disagreed. The datasets of the three Ethiopian languages (Amharic, Tigriniya, and Oromo) were annotated using two independent an- notators, and then curated by third more expe- rienced individual who decided the final gold labels.
Twi significant portion of tweets in Twi were ambiguous, making it difficult to categorize senti- ment accurately. Some tweets contained symbols not in the Twi alphabet, which is a frequent oc-
hhttps://github com//oogle/cld3 "https: hrtppppi..orppooeee/ppc.....o

---

3-way
2-way
Lang.
arq
ary
hau
ibo
kin
pcm
pt-MZ
tso
twi
yor
tir
K
0.41
0.62
0.66
0.61
0.43
0.60
0.50
0.50
0.51
0.65
0.47
0.20
0.51
Table 5: Inter-annotator agreement scores using the free marginal multi-rater kappa (Randolph, 2005) for the different languages.
ama
arq
hau
ibo
ary
orn
pcm
pt-MZ
kin
swa
tir
tso
twi
yor
train
5,985
1,652
14,173
10,193
5,584
5,122
3,064
3,303
1,811
805
3,482
8,523
dev 1,498
415
2,678
1,842
1,216
397
1,282
768
828
454
399
204
389
2,091
test 2,000
959
5,304
3,683
2,962
2,097
4,155
3,663
1,027
749
2,001
255
950
4,516
Total
9,483
3,062
22.155
15.718
9.762
2,494
10,559
7.495
5.158
3,014
2,400
1,264
4,821
15,130
Table 6: Splits and sizes of the AfriSenti datasets. We do not allocate training splits for Afaan Oromo (orm) and Tigrinya (tir) due the limited size of the data and only evaluate on them in zero-shot transfer setting in $6.
currence due to the lack of support for certain Twi letters on keyboards (Scannell, 2011). I). For example, "o" is replaced by the English letter "c", and "c" is replaced by "3".
Algerian Arabic For Algerian Arabic, the main challenge was the use of sarcasm. When this caused disagreement among the annotators, the tweet was further labeled by two other annotators. If the annotators did not agree on final label, the tweet was discarded. As Twitter is also commonly used to discuss controversial topics in the region, we found a large number of offensive tweets shared among the users. We removed the offensive tweets to protect the annotators and avoid including such instances in sentiment analysis dataset.
Additionally, tweets were often annotated as neg- ative (cf. Figure 3) due to some common expres- sions that could be seen as offensive depending on the context. E.g., "Tweaa" was once considered an insult but has become a playful expression through trolling, and "gyae gyimii" is commonly used by young people say "stop" while its literal meaning is "stop fooling".
Experiments
Mozambican Portuguese and Xitsonga One of the significant challenges for the Mozambican Por- tuguese and Xitsonga data annotators was the pres- ence of code-mixed and sarcastic tweets. Code- mixing in tweets made challenging for the an- notators to determine the intended meaning of the tweet as it involved multiple languages spoken in Mozambique that some annotators were unfamiliar with. Similarly, the presence of two variants of Xitsonga spoken in Mozambique (Changana and Ronga) added to the complexity of the annotation task. Additionally. we excluded many tweets from the final dataset as sarcasm present tweets was another source of disagreement among the annota- tors.
6.1 Setup
For our baseline experiments, we considered three settings: (1) monolingual baseline models based on multilingual pre-trained language models for 12 AfriSenti languages with training data, (2) multi- lingual training of all 12 languages and their eval- uation on a combined test of all 12 languages, (3) zero-shot transfer to Oromo (orm) and Tigrinya (tir) from any of the 12 languages with available training data. We used standard configuration for text classification fine-tuning on HuggingFaa Face with learning rate of 2c 5 for smaller PLMs and le 5 for larger PLMs, a batch size of 128, and 10 epochs.
Ethiopian languages For Afaan Oromo and Tigrinya, challenges included finding annotators and the lack of reliable Internet connection and access personal computers. Although we trained the Oromo annotators, we observed severe prob- lems in the quality of the annotated data, which led to a low agreement score.
Monolingual baseline models We fine-tune mas- sively multilingual PLMs trained on 100 languages from around the world well Africa-centric PLMs trained exclusively on languages spoken in Africa. For the massively multilingual PLMs, we selected two representative PLMs: XLM-R- base & large) (Conneau et al., 2020) and m mDBBER-

---

Lang
In XLM-R
InDeERRT?
ATOXLMM ANTRERT
In
XLM-T
ATHiERTa
XLM-R
base
AfroXLMR
base
mDDBERTT
base
XLM-T
XLM-R large
AfroXLMR
large
anh
36.9
661.2
54.9
57.6
GUL
61.8
61.6
arg
47.7
65.9
65.5
65.7
69.5
63.9
68.3
ary
524
58.3
57.7
hau
78.7
77.2
75.7
75.7
ibo
X
78.6
75.6
76.3
77.5
76.1
76.5
79.5
kin
62.7
$6.7
67.2
65.5
590
55.7
70,6
pen
Rt-HZ
70.1
66.6
63.6
71.3
71.6
71.6
SNB
V
X
61.5
57.8
608
59.5
58.4
614
63.4
51.6
47.4
45.9
47.4
538
43.7
47.3
yor
72.9
62.7
70.0
63.4
61.2
62.4
74.1
AVG
61.7
61.9
63.9
64.2
64.7
63.1
67.2
Table 7: Accuracy scores of monolingual baselines for AfriSenti on the 12 languages with training splits. Results are averaged over 5 runs.
TaV3 (He al., 2021). For the Africa-centric models, we made use of AfriERTaa-arre (Ogueji et al., 2021a) and AfroXLMR- large] (AI- abi et al., 2022) - an XLM-R model adapted to African languages. AfriBERTa was pre-trained from scratch on 11 African languages including nine of the AfriSenti languages while AfroXLMR supports 10 of the AfriSenti languages. Addition- ally, fine-tune XLM-T (Barbieri et al., 2022a), XLM-R model adapted the multilingual Twit- ter domain, supporting over 30 languages but fewer African languages due to a lack of coverage by Twitter's language API (cf. $4).
Model
AfriBERTa-large
64.7
XLMMR L.M-R-base
64.3
AfroXLMR MR-base
68.4
mDDBERRTa V3-FRaa
66.1
XLM-T-bass
65.9
XLM-R-large
66.9
AfroXLMR-
71.2
Table Multilingual training evaluation on com- bined test sets of all languages. We show the average scores over 5 runs.
benefit of scaling for large PLMs. Nevertheless, scaling is of limited use for XLM-R-large it has not been pre-trained on many African languages.
6.2 Experimental Results
Our results show the importance of both lan- guage and domain specific pre-training and high- light the benefits of scale for appropriately pre- trained models.
Table 7 shows the results of the monolingual base- line models on AfriSenti. AfriBERTa obtained the worst performance average (61.7), especially for languages it was not pre-trained on (e.g.. < 50 for the Arabic dialects) in contrast the languages it was pre-trained on, such as hau, ibo, swa, yor. XLM-R-baad base led to a performance comparable to AfriBERTa on average, performed worse for most African languages except for the Arabic dialects and pt-MZ. On the other hand, AfroXLMR-han base and mDeBERTaV3 achieve similar performances, although AfroXLMR- base performs slightly better for kin and pcm compared to other models.
Table 8 shows the performance of multilin- gual models fine-tuned on the combined training data and evaluated on the combined test data all languages. Similarly to earlier, AfroXLMMR large achieves the best performance, outperforming Afro LMR-base, XLM-R-large, and XLM--Tbase by more than 2.5 F1 points.
Finally, Table 9 shows the zero-shot cross- lingual transfer performance from models trained on different source languages with available train- ing data in the test-only languages orm and tir. The best source languages are Hausa Amharic for orm and Hausa or Yorubá for tir.
Overall, considering models with up to 270M parameters, XLM-T achieves the best performance which highlights the importance of domain -emmai--ppecffc pre-training. XLM-T performs particularly well on Arabic and Portuguese dialects, i.c., arq, ary and pt-MZ, where it outperforms AfriBERTa by 21.8. 14.2, and 13.0 and AfroXLMR-baa by 4.0, 5.9, and 4.7 F1 points respectively. AfroXI.MR- achieves the best overall performance and improves over X by 2.5 Fl points, which shows the
Interestingly, Hausa outperforms multilin- gually trained model. The impressive performance for transfer between Hausa and Oromo may be due to the fact that both are from the same lan- guage family and share a similar Latin script. Fur- thermore, Hausa has the largest training dataset

---

Target Language tir
explicitly forbid the use of the datasets for commer- cial purposes or by state actors, unless explicitly approved by the dataset creators. Automatic sen- timent systems are also not reliable at individual instance-level and are impacted by domain shifts. Therefore, systems trained on our datasets should not be used to make high-stakes decisions for indi- viduals, such in health applications. See Moham- mad (2022, 2023) for comprehensive discussion of ethical considerations relevant sentiment and emotion analysis.
Source Language
orm
AVG
amh
46.5
62.6
54.6
27.5
56.0
41.8
ary
42.5
58,6
50.6
hau
47.1
68.6
57.9
ibo
41.7
39.8
40.8
kin
43.6
64.8
54.2
pcm
26.7
58,2
42.5
por 28.7
21.5
25.1
36.8
31.8
21.5
15.8
18.7
9.8
15.6
12.7
yor
53.2
mul multilingual
42.0
54.2
Limitations
Table 9:
Zero-shot evaluation on orm and tir. All source languages trained MR-ale..
When collecting the data, we deleted offensive tweets and controlled for the most conflicting ones by adding an annotation round or removing some tweets as explained in Section 5, We acknowledge that sentiment analysis is subjective task and, therefore, our data can still suffer from the label bias that most datasets suffer from. However, we share all the attributed labels to mitigate this prob- lem and help the research community interested in studying the disagreements.
AfriSenti. Both linguistic similarity and size of source language data have been shown to correlate with successful cross-lingual transfer (Lin al., 2019).
However, is unclear why Yoràbá performs par- ticularly well for tir despite the difference in the script. One hypothesis is that Yorùbá may be good source language in general, as claimed in Adelani et al. (2022) where Yorùbá was the second best source language for named entity recognition in African languages.
Given the scarcity of data in African languages, we had to rely on keywords and geographic loca- tions for data collection. Hence, our datasets are sometimes imbalanced, as shown in Figure 3. Finally, although we focused on 14 languages. we intend to collect data for more languages. We invite the community to extend our datasets and improve on them.
7 Conclusion and Future Work
We presented AfriSenti, a collection of sentiment Twitter datasets annotated by native speakers in 14 African languages: Amharic, Algerian Ara- bic, Hausa, Igbo, Kinyarwanda, Moroccan Arabic, Mozambican Portuguese, Nigerian Pidgin, Oromo, Swahili, Tigrinya, Twi, Xitsonga, and Yorūbá, used the first Afro-centric SemEval shared task- SemEval 2023 Task 12: Sentiment analysis for African languages (Afriieeni-SSmmEvaa)) We re- ported the challenges faced during data collection and annotation, in addition to experimental results in different settings.
Acknowledgements
We thank all the volunteer annotators involved this project. Without their support and valuable contributions, this project would not have been pos- sible. This research was partly funded by the La- cuna Fund, initiative co-founded by The Rocke- feller Foundation, Google.org, and Canada's Inter- national Development Research Centre.
The views expressed herein do not necessarily represent those of Lacuna Fund, its Steering Com- mittee, its funders, or Meridian Institute. We are grateful to Adnan Ozturel for helpful comments on draft of this paper. We thank Tal Perry for pro- viding the LightTag (Perry, 2021) annotation tool. We also thank the Language Technology Group, University of Hamburg. for allowing us use the WebAnno (Yimam et al., 2013) annotation tool for all the Ethiopian languages annotation tasks. David Adelani acknowledges the support of DeepMind
We publicly release the datasets and other re- sources, such as the collection lexicons for the re- search community intetrested sentiment analysis and under-represented languages. In the future, we plan to extend AfriSenti to additional African languages and other sentiment analysis sub-tasks.
8 Ethics Statement
Automatic sentiment analysis can be abused by those with the power to suppress dissent. Thus, we

---

Academic Fellowship Programme. Finally, we are grateful for the support of Masakhane.
4488 4500,, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Basant Agarwal and Namita Mittal. 2016. Machine
learning approach for sentiment analysis. In Promi- nent feature extraction for sentiment analysis, pages 21-45. Springer.
References
Ife Adebara and Muhammad Abdul-Mageed. 2022. To- wards afrocentric NLP for African languages: Where we are and where we can go. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 3814-3841, Dublin, Ireland. Association for Compu- tational Linguistics.
Jesujoba O. Alabi, David Ifeoluwa Adelami, Marius Mosbach, and Dietrich Klakow, 2022. Adapting pre- trained language models to African languages via multilingual adaptive fine-tuning. In Proceedings of the 29th International Conference on Computational Linguistics, pages -3364439, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
David Ifeoluwa Adelani, Jade Abbott, Graham Neu- big. Daniel D' souza, Julia Kreutzer, Constantine Lig- nos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew. Is- rael Abebe Azime, Shamsuddeen Muhammad, Chris Chinenye Emezue, Joyce nakatumba-Nabendee Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye. Jesujoba Alabi, Seid Muhie Yi- mam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Ver- rah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chuk- wuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akin- ode, Deborah Nabagereka, Maurice Katusiime, Ayo- dele Awokoya, Mouhamadane MBOUP, Dibora Ge- breyohannes, Henok Tilaye, Kelechi Nwaike, De- gaga Wolde, Abdoulaye Faye, Blessing Sibanda, Ore- vaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji. Thiemo Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Sa- lomey Osei. 2021. MasakhaNER: Named entity recognition for African languages. Transactions of the Association for Computational Linguistics, 9:1116-1131.
Nana Aba Appiah Amfo and Jemima Anderson. 2019. Multilingualism and language policies in the african context: lessons from ghana.
Jason Angel, Segun Taofeek Aroyehun, Antonio Tamayo, and Alexander Gelbukh. 2020. NLP-CIC at SemEval- task Analysing sentiment in code- switching language using a simple deep-learning clas- sifier. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 957-962, Barcelona (online). International Committee for Computational Linguistics.
JoAnne Wallaee 2002. Talk that talk: Story- telling and analysis rooted in african american oral tradition. Qualitative Health Research, 12(3):400- 426. PMID: 11918105.
Francesco Barbieri, Luis Espinosa Anke, and Jose Comacho-Collados. 2022a. Xlm-t: Multilingual lan- guage models in twitter for sentiment analysis and beyond. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 258- 266.
Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collane 2022b. XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 258-266, Marseille, France. European I. Language Re- sources Association.
David Ifcoluwa Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoha O. Al- abi, Shamsuddeen H. Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Roowei- ther Mabuya, Bonaventure F. P. Dossou, Bless- ing Sibanda. Happy Buzaaba, Jonathan Mukibi,, Godson Kalipe, Derguene Mbaye, Amelia Taylor, Fatoumata Kabore, Chris Chinenye Emezue, An- uoluwapo Aremu, Perez Ogayo, Catherine Gitau, Edwin Munkoh-Buabenn
Tadesse Destaw Belay, Abinew Ali Ayele, Getie Gelaye. Seid Muhie Yimam, and Chris Biemann. 2021. Im- pacts of homophone normalization on semantic mod- els for amharic. In 2021 International Conference Information and Communication Technology for Development for Africa (ICT4DA), pages 101-106.
Victoire M. Koagne, Al- lahsera Auguste Tapo, Tehogo Macuewa, Vukosi Marivate, Elvis Mhoning, Tajuddeen Gwadabe, Tosin Adewumi, Orevaoghene Ahia, Joyce Nakatumba- Nabende, Neo L. Mokono, Ignatius Ezeani, Chia- maka Chukwancke, Mofetoluwa Adeyemi, Gilles Q. Hacheme, Idris Abdulmumin, Odunayo Ogundepo. Oreen Yousuf, Tatiana Moteu Ngoli, and Dietrich Klakow. 2022. MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition. In Proceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing, pages
Erik Cambria. Qian Liu, Sergio Decherchi, Frank Xing, and Kenneth Kwok. 2022. Sentienet 7: commonsense-baan based neurosymbolic ai framework for explainable sentiment analysis. Proceedings of LREC 2022.
Jan Carter-Black. 2007. Teaching cultural competence: An innovative strategy grounded in the universality

---

of storytelling as depicted in african and african amer- ican storytelling traditions. Journal of Social Work Education, 43(1) (1)):311500)
Quintin Gee. 2005, Review of script displays of african languages by current software. New Review of Hy- permedia and Multimedia, 1 11:247 - 255.
Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. 2020. Language ID in the wild: Unex- pected challenges on the path to aoousand-language web text corpus. In Proceedings of the 28th Inter- national Conference on Computational Linguisties, pages 6588-6608, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics.
Itisha Gupta and Nisheeth Joshi. 2020. Enhanced twit- ter sentiment analysis using hybrid approach and by accounting local contextual semantic. Journal of intelligent systems, 29( 191)):611-1625.
William I. Hamilton, Jure Leskovec, and Dan Juraf- sky. 2016. Diachronic word embeddings reveal sta- tistical laws of semantic change. arXiv preprint urXiv:1605. 1605.09096.
Chenhua Chen, Zhiyang Teng, Zhongqing Wang, and Yue Zhang. 2022. Discrete opinion tree induction for aspect- based sentiment analysis. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2051-2064
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre- training with gradient- srediient-dientangled embedding shar- ing.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Compulational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Haoming Jiang, Pengcheng He. Weizhu Chen, Xi- aodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. SMART: Robust and efficient fine-tuning for pre- trained natural language models through principled regularized optimization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 2177-2190, Online. Association for Computational Linguistics.
Dr Kaur et 2022. Incorporating sentimental analysis into development of a hybrid classification model: A comprehensive study. International Journal of Health Sciences, 6: 6:1709-1720.
Aida Mostafazadeh Davani, Mark D'iaz, and Vinodku- mar Prabhakaran. 2021. Dealing with disagreements: Looking beyond the majority vote in subjective an- notations. Transactions of the Association for Com- putational Linguistics, 10:92-110.
C. Maria Keet. 2021. Natural language generation re- quirements for social robots in sub-saharan africa. In 2021 IST-Africa Conference (IST-Africa), pages 1-8. IEEE.
Peter Sheridan Dodds, Eric M Clark, Suma Desu, Morgan R Frank, Andrew J Reagan, Jake Ryland Williams, Lewis Mitchell, Kameron Decker Harris, Isabel M Kloumann, James P Bagrow, et al. 2015. Human language reveals universal positivity bias. Proceedings of the national academy of sciences, 112(8):2389-2399
Hoai Bac Le and Huy Nguyen. 2020. Twitter sentiment analysis using machine learning techniques. In Inter- national Conference on Computer Science, Applied Mathematics and Applications.
David M. Eberhard, Gary F. Simons, and Charles D. Fennig. 2020. Ethnologue: Languages of the World. Twenty-third edition. Dallas, Texas: SII. Interna- tional. Url: http:://wwwt www.:eewnooogge.comm com.
Paul M. A. Lewis. 2009. Ethnologue: languages of the world.
Wei Liang, Yuming Shi, Chi K. Tsc, Jing Liu, Yanli Wang, and Xunqiang Cui. 2009. Comparison of co- occurrence networks the chinese and english lan- guages. Physica A: Statistical Mechanics and its Applications, 388(23):4901-4999
Abdeljalil El Abdouli, Larbi Hassouni, and Houda Anoun. 2017. Sentiment analysis of moroccan tweets using naive bayes algorithm. International Journal of Computer Science and Information Security (LJCSIS), 15(12).
Yunlong Liang, Fandong Meng. Jinan Xu, Yufeng Chen, and Zhou. 2022. Msetd: multimodal sentiment chat translation dataset. arXiv preprint arXiv:2202. 13645.
Chris Chinenye Emezue, Hellina Hailu Nigatu, Cyn- thia Thinwa, Helper Zhou, Shamsuddeen Has- san Muhammad, Lerato Louis, Idris Abdulmu- min, Samuel Gbenga Oyerinde, Benjamin Ayoade Ajibade, Olanrewaju Samuel, et al. 2022. The african stopwords project: Curating stopwords for african languages. In 3rd Workshop on African Natural Lan- guage Processing.
Yu Hsiang Lin, Chian Yu Chen, Jean Lee, Zirui Li. Yuyan Zhang. Mengzhou Xia, Shruti Rijhwani, Junx- ian He, Zhisong Zhang, Xuezhe Ma, Antonios Anas- tasopoulos. Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for cross- pross-lingual learn- ing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3125-3135, Florence, Italy. Association for Compu- tational Linguistics.
Rosa Falotico and Piero Quatto. 2015. Fleiss' kappa statistic without paradoxes. Quality & Quantity, 49(2):463-470.

---

Bing Liu. 2020. Sentiment analysis: Mining opinions, sentiments, and emotions. Cambridge university press.
Shamsuddeen Hassan Muhammad, David Ifcoluwa Ade- lani, Sebastian Ruder, Ibrahim Sa'id Ahmad, Idris Abdulmumin, Bello Shehu Bello, Monojit Choud- hury, Chris Chinenye Emezue, Saheed Salahudeen Abdullahi, Anuoluwapo Aremu, Alípio Jorge, and Pavel Brazdil. 2022. NaijaSenti: A Nigerian Twitter sentiment corpus for multilingual sentiment analy- sis. In Proceedings of the Thirteenth Language Re- sources and Evaluation Conference, pages 590-602. Marseille, France. European Language Resources Association.
Gati L Martin, Medard E Mswahili, and Young-Secr -oung-Seob Jeong. 2021. Sentiment classification in swahili language using multilingual bert. arXiv preprint arXiv:2104.0000
M'hamed Mataoui, Omar Zelmati, and Madiha Boumechache. 2016. A proposed lexicon-based sen- timent analysis approach for the vemacuuar algerian arabic. Research in Computing Science, 1 110(1) 1-555 70.
Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. 2021a. Small data? problem! exploring the viability of pretrained multilingual language models for low- resourced languages. In Proceedings of the 1st Work- shop on Multilingud Representation Learning, pages 116-126, Punta Cana, Dominican Republic. Associa- tion for Computational Linguistics.
Granville Matheson. 2019. We need talk about reliability: making better use of test-retest studies for study design and interpretation. PeerJ, 7:c6918, 18.
Million Meshesha and C. V. Jawahar. 2008. Indigenous scripts of african languages. Indilinga: African Jour- nal of Indigenous Knowledge Systems, 6:132-142.
Kelechi Ogueji, Yuxin Zhu, and Jimmy J. Lin. 2021b. Small data? no problem! exploring the viability of pretrained multilingual language models for low- resourced languages. Proceedings of the 1st Work- shop on Mulrilingual Representation Learning.
Saif Mohammad, 2016. A practical guide senti- ment annotation: Challenges and solutions. In Pro- ceedings of the 7th Workshop Computational Ap- proaches to Subjectivity, Sentiment and Social Media Analysis, pages 174-179, San Diego, California. As- sociation for Computational Linguistics.
Aissam Outchakoucht and Hamza Ss-Samaali. 2021. Moroccan dialect -darija- open dataset.
Tal Perry. 2021. LightTag: Text annotation platform. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing: Sys- tem Demonstrations, pages 20-27, Online and Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Saif M. Mohammad. 2022. Ethics sheet for automatic emotion recognition and sentiment analysis. Compu- tational Linguistics, 48(2):239-22 278.
Saif M. Mohammad. 2023. Best practices in the cre- ation and use of emotion lexicons. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Dubrovnik, Croatia. Association for Computational Linguistics.
Soujanya Poria. Devamanyu Hazarika, Navonil Ma- jumder, and Rada Mihalcea. 2020. Beneath the tip of the iceberg: Current challenges and new directions in sentiment analysis research. JEEE Transactions on Affective Computing.
Saif M. Mohammad, Svetlana Kiritchenko, and Xiao- Dan Zhu. 2013. Nre-canada: Building the state-of- the-art in sentiment analysis of tweets. In Interna- tional Workshop Semantic Evaluation.
Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark D'iaz, 2021. On releasing annotator- level labels and information in datasets. ArXiv, abs/2110.05693
Leila Moudjari, Karima Akli-Astouati, and Farah Be- namara. 2020. An Algerian corpus and an annota- tion platform for opinion and emotion analysis. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1202- 1210, Marseille, France. European Language Resources Association.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, el al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1166,
Shamsuddeen Hassan Muhammad, Idris Ahdulmu- min. Seid Muhie Yimam, David Ifeoluwa Ade- lani, Ibrahim Said Ahmad, Nedjma Ousidhoum, Abinew Ali Ayele, Saif Mohammad, Meriem Be- loucif, and Sebastian Ruder. 2023, SemEval-2023 task 12: Sentiment analysis for African languages (AfriSenti-SemTEva)), In Proceedings of the 17th International Workshop Semantic Evaluation (SemEval-2022) pages 2319-2337, Toronto, Canada. Association for Computational Linguistics.
Justus J Randolph. 2005. Free-marginal multirater kappa (multirater k [free]]: An alternative to fleiss' fixed- marginal multirater kappa. Online submission.
Andrew J. Reagan, Lewis Mitchell, Dilan Kiley, Christo- pher M. Danforth, and Peter Dodds. 2016. The emotional ares of stories are dominated by six basic shapes. EPJ Data Science, 5(1):1-12. Copyright- EPJ Data Science is copyright of Springer, 2016; Last updated- 2017-02-06.

---

Sebastin Santy, Anirudh Srinivasan, and Monojit Choud- hury. 2021. BERTologiCoMix: How does code- mixing interact with multilingual BERT? In Proceed- ings of the Second Workshop on Domain Adaptation for NLP, pages 11-121,, Kyiv, Ukraine. Association for Computational Linguistics.
Kevin P Scannell. 2011. Statistical unicodification of african languages. Language resources and evalua- tion, 45(3):375-386.
Laura Sterponi and Paul F. Lai. 2014. Culture and language development. In Farzad Sharifian, editor, The Routledge Handbook of Language and Culture, pages 339-356. Routledge, London, UK.
Maite Taboada, Julian Brooke, Milan Tofiloski, Kim- berly D. Voll, and Manfred Stede. 2011. Lexicon- based methods for sentiment analysis. Computa- tional Linguistics, 37:267-307.
S Thara and Prabaharan Poornachandran. 2018. Code- mixing: A brief survey. In 2018 International Con- ference on Advances Computing, Communications and Informatics (ICACCI), pages 2382-2388.
Peter D. Turney. 2002. Thumbs up or thumbs down? se- mantic orientation applied unsupervised classifica- tion of reviews. In Annual Meeting of the Association for Computational Linguistics.
Ashima Yadav and Dinesh Kumar Vishwakarma. 2020. Sentiment analysis using deep learning architectures: a review, Artificial Intelligence Review, 43(6) 4385.
Seid Muhie Yimam, Hizkiel Mitiku Alemayehu, Abinew Ali Ayele, and Chris Biemann. 2020. Ex- ploring amharic sentiment analysis from social me- dia texts: Building annotation tools and classification models. In International Conference on Computa- tional Linguistics.
Seid Muhie Yimam, Abinew Ali Ayele, and Chris Bie- mann. 2019. Analysis of the Ethiopic Twitter Dataset for Abusive Speech in Amharic. In In Proceedings of International Conference On Language Technologies For All: Enubling Linguistic Diversity And Multilin- gualism Worldwide (LT4ALL 2019), pages 1-5, Paris, France.
Seid Muhie Yimam, Iryna Gurevych, Richard Eckart de Castilho, and Chris Biemann. 2013. WebAnno: A flexible, web-based and visually supported system for distributed annotations. In Proceedings of the 51st Annual Meeting of the Association for Compu- tational Linguistics: System Demonstrations, pages 1-6, Sofia, Bulgaria. Association for Computational Linguistics.
Lei Zhang, Shuai Wang, and Bing Liu. 2018. Deep learning for sentiment analysis: survey. Wiley Interdisciplinary Reviews: Data Mining and Knowl- edge Discovery, 1253.

---

A Appendix
Lang.
Tweet
Sentiment
amh
aah% Klook thc &hh hty TARAAA CA-AA" & Khd PENG negative Arriii ya- A2&?
They are telling us that the cruel barbarian behind the bar and got chains. Wait! Are they chaining him and servicing him coffee?
Gloss:
arq
@user ... ونووي lemin مرق و شاع ون شرم & ai الشروق
negative
Gloss: Since it was founded, Echourouk [newspaper /TV channel] has always been shameful, low level and populist.
ary
Fibli جططر Je wi بغارر ale االددي المادي j بكشي يا مدوم رال
negative
Gloss: Do you want them to start being harsh on everyone to be relieved
ary
rabi ykhali alhbiba makayn ghir nachat o chi machat Gloss: God bless you, my dear let the fun begin
positive
hau
@USER Aunt rahma luv wallah irin totally dinnan
positive
Gloss: @USER Aunty rahina swear love you very much.
ibo
akowaro ya ofuma kai daalu nwanne mmaad Gloss: they told it well my fellow sister well done at the end we will be all right
positive
kin
@user Ariko akokanu ngo inyebebe unyujjemoo sisawa wangu Gloss: @user but this thing of miscreant you just mentioned not good dear
negative
orm
@user Jawaar Kenya OMN haala akkamiin argachuu dandeenya Gloss: @USER Our Jewar how can accoss /ccoss/reach out OMN.
neutral
por
Honestidade é algo que não se compra. Infelizmente humanidade esqueceu disso por causa das suas ambições.
positive
Gloss: Honesty something you can't buy. Unfortunately, humanity has forgotten this because of its ambitions.
pcm
E don tay wey I don dey crush this fine woman ...
positive
Gloss: have had crush on the beautiful woman for a while
Asante sana watu Sirari jimbo la Tarime vijijini Huu ni Upendo usio na Mashaka kwa Mbunge wenu John Heche
positive
Gloss: Thank you very much people of Sirari, rural Tarime province
This is Undoubted Love for your Member of Parliament John Heche
tir
@user haw Tizh-g" 1944Thhh::hhadd '1120 a. 48.1290 MO TT44 SAM h-20!
negative
Gloss: If were to advise youyoou better get rid these few
tao
@user &user Yu, tindzava 2 Tsika mbangui mpfann nita ku despro- gramara
negative
Gloss:
Ah! gossiping? Quit drugs dude, they'll 'll mess you up...
twi
messi saf den check bp kwame danso di twe da kor aaa negative wawu
yor
onírèégbè uláàdúgbö ati olójúkökörò
negative
Gloss: mischievous and coveteous neighbour
Table 10: Annotated tweets with their English translations.


Reviews:
[{"review_id": "xSHspiDcCx", "reviewer": "Reviewer_DhE5", "summary": "", "questions": "1. Sentiment categories are not explicitly mentioned in the text. I can guess from Table 1 that there are three categories (positive, negative and neutral). The authors should provide the categories in the text. \n\n2. More details on the models will be helpful. How the fine tunning was performed? What are the tunning parameters? The architecture of the system will be helpful. \n\n3. As the F1-score for the system are no great, what can be done to improve the system performances? \n\n4. What was the reason to test the cross-lingual system only on orm and tir languages? \n\n5. Is there any statistics avaialble on percentage of code-switching and code-mixing? It will be great to have these information in the paper. \n\n6. Are these trained models will also be avialble for the researchers?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's feedback, and we would like to address some concerns about the lack of innovation in our implementation. \n\nWe agree that our implementation is not novel, but that was not our goal. The main contribution of this work is to introduce new data resources for 14 African languages (from 3 language families), which is aligned with the EMNLP's main contribution: \"new data resources, particularly for low-resource languages\" (see https://2023.emnlp.org/calls/main_conference_papers/#contributions)). \n\nWe believe that our data resource is valuable for the NLP community, as it can enable new research directions and applications for these languages. Most importantly, it will encourage a new generation of African researchers who can work on languages that matter deeply to them and their communities. Responses to questions are below:\n\n1. The sentiment categories used are indeed: positive, negative, and neutral. It was an oversight and we will explicitly mention these categories at multiple appropriate places in the text. \n\n2. We will add the batch size, sequence length and other details in the camera ready. We used the standard configuration for text classification fine-tuning on HuggingFace sample codes, the only change is lowering the learning rate for the larger pre-trained language models  (1e-5).\n\n3. Expanding our dataset with more data could improve. We believe that trying different techniques such as transfer learning and continuous training could also help increase the F1 scores. \n\n4. We chose zero-shot setting for orm and tir because we were only able to collect a small amount of data. We decided to use the collected data for development and test split. \n\n5. We will add statistics about the code-switching. \n\n6. All models will be made public and shared on HuggingFace.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This dataset with 14 African languages benefits the NLP community to develop sentiment classification tools. \n2. The data collection challenges were described in detail which is useful for NLP researchers working for low-resource languages.", "reasons_to_reject": "1. There is no innovation in implemetation for NLP development.", "questions_to_authors": "1. Sentiment categories are not explicitly mentioned in the text. I can guess from Table 1 that there are three categories (positive, negative and neutral). The authors should provide the categories in the text. \n\n2. More details on the models will be helpful. How the fine tunning was performed? What are the tunning parameters? The architecture of the system will be helpful. \n\n3. As the F1-score for the system are no great, what can be done to improve the system performances? \n\n4. What was the reason to test the cross-lingual system only on orm and tir languages? \n\n5. Is there any statistics avaialble on percentage of code-switching and code-mixing? It will be great to have these information in the paper. \n\n6. Are these trained models will also be avialble for the researchers?", "paper_topic_and_main_contributions": "This paper describes a sentiment dataset creation for 14 African languages and performed several experiments to classify the tweets into sentiment categories.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pKSBS3XZlG", "reviewer": "Reviewer_TvNi", "summary": "", "questions": "I'm going to ask my questions in the order they pop up in the text, but I'll try to give a sense of which ones I think are more important to address:\n\n1) You have picked sentiment and in lines 47-56, you mention a number of reasons that sentiment may be useful (literary analysis, culturonomics, commerce, psychology, social science). One that I don't think those quite capture would be \"linguistic\". Here, I'm thinking about how, for example, there's a malefactive particle in at least one Ethiopian language that lets a speaker/writer give the sense that someone did something AGAINST someone else, in Dutch there's \"tet\" which marks surprise/irritation, in Czech diphthongs can mean affection (or pejoration), in Cantonese, throwing a -k at the end of particles can intensify the emotion being expressed, in Tongan there are determiners that express sympathy, in Manambu there is a \"frustrative\" case marking for when things are done in vain, etc. \n\nIn my heart of hearts, my hope is that your dataset could help people find affective markers/word-order/morphemes but...is that plausible, do you think? Or is your expectation that what your annotators picked up on were louder words like adjectives (\"awful\", \"wonderful\", etc). I don't think this particularly needs to be added to your paper, but I am curious about how you'd look at the dataset being used for more linguistic/sociolinguistic/pragmatics research questions.\n\n2) You mentioned using Saif Mohammad's sentiment guidelines, I think you mean the \"base\" questions he has (not the ones that try to get at what the sentiment is direct towards). It probably would be good to share the explicit instructions for annotators, probably in an appendix. This I do think is probably a minor weakness and worth addressing, btw. It also connects to some of your comments on code-switching texts—you have some annotators who don't know parts of those mixed language tweets and it feels like they should have a \"I don't understand this\" option (which would also be relevant for everyone else since I am certain as a native English speaker you could give me completely English tweets that I could not understand to give a sentiment to). \n\n2b) Part of this question comes from looking at your examples in Table 1. The Amharic example kicks off with the very powerful bigram \"ጨካኝ አረመኔ\" (~brutal barbarian, pitiless pagan) but the rest of it has a much lighter tone as far as I can tell. I don't know the referent but it strikes me that this is fairly sarcastic and has an intent to be humorous which is \"negative\" in a way that is less clear than \"I hate X\". You mention sarcasm a number of times—as does Mohammad 2016—I'm wondering what you might say about that topic more generally. \n\nThis also feels related to the annotation problem you describe in Twi with “Tweaa” being potentially offensive depending on context—many annotators have a hard time distinguishing a speaker/writer’s sentiment when they use taboo terms (the terms that one community of speakers might think is unexceptional or just expressive may be no-go’s for other speakers who ignore the speaker intention). Do you have any thoughts about your annotators’ predilections on these things? \n\nIf I gave you a magic button to press that would replace positive/negative/neutral with any categories of your choosing with high accuracy, which categories would you pick? That is, is valence the real thing you most want or is it a stand-in/approximation/first-step for something you actually would rather collect but is harder?\n\n3) You briefly mention some of the ways African languages present challenges for sentiment analysis that English and other languages may not. You mention tone, code-switching, and digraphia in lines 127-128 but I don't think you actually talk about tone. Here, I assume you mean phonemic tone since lots of African languages are tonal in nature. You discuss code-switching and digraphia but do you have anything to say about phonemic tone and your dataset/annotations? About half your languages use tone but I think only Yorùbá marks them in writing...but maybe the lack of marking them in tweets causes some extra homographs but...I'm not sure how that would be different than other languages which are replete with homographs (English lead=metal, lead=guide). \n\n4) You have to do a bunch of things to get a meaningful (not-all-neutral) set of tweets. I think you do a pretty good job but I wonder if you could address what you see as the consequences of these choices for possible uses of the dataset. This is a suggestion I think would help the paper. \n\n4b) A minor thing but why are the Nigerian language test sets lower-cased (you reference Muhammad et al 2022, but it’d be nice if you give the reason so readers don’t have to track that reference down). \n\n4c) You discard “full disagreement” tweets—are those going to be in the dataset? I think that may help with the use of your dataset, provided it’s easy for people to see that these are more for agreement/annotator kinds of questions. I think your mention of Prabhakaran et al (2021) means that you probably do release these but it’s not completely clear to me. \n\n4d) What do you think are the consequences of removing offensive tweets in Algerian Arabic? And was your intention more to produce a non-offensive dataset or to not disturb your annotators? I think it is relevant to share your intentions for this. \n\n5) The idea that Yorùbá is a generally good source language strikes me as very strange. I mean, you are empirically showing it to be the case and mention Adelani et al’s similar finding in NER but...this strikes me as “surely a coincidence” or perhaps because there are some proper names with similar sentiment in Ethiopia and Nigeria…what do you think the most reasonable explanation of this is?\n\n6) Your plan is to keep releasing more languages (and enable others to follow your lead), which is great. I wonder which languages are at the top of your list. Do you want coverage in terms of speaker populations? Coverage over countries that yet covered? Languages that are outside the language families you’ve already covered? I’m interested in your actual “Top Five” but even moreso for what leads you to that Top Five instead of another Top Five. I think being explicit about this helps readers understand where you're headed/why and what they should keep an eye out for or help with. \n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for providing feedback, and insightful suggestions. Below, we respond to the questions asked.\n\n1. \n\nWe appreciate the reviewer's insights on how the datasets can be useful to better understand how language is used in the African context and hope that researchers with expertise in linguistics find our dataset to be valuable.\n\n2. \n\n(a): Yes, we are referring to the base questions. The one called, “A Simple Sentiment Questionnaire”. We will add this specificity in the paper. We will also add the annotation guide in the  appendix section. In some languages (e.g., Algerian Arabic), we added language-specific examples and slightly edited the guidelines, we will make these guidelines public as well.\n\n(b): Sarcasm: Since sarcasm was not a direct focus of this work, we have not explored it deeply yet. However, that ist interesting future work and we mentioned in lines 378-379 that sarcasm is generally one of the main challenges in our annotation.\n\n3. \n\nIgbo is also a tonal language and marks them in writing. The tone in this language changes the sentiment. For example, below are identical tweets, but when used with different tones, they convey different sentiments. \n\nò nwèkwàrà mgbe i naenwe sense ? ( Will you ever be able to talk sensibly? – You’re a fool.)\n\nò nwèkwàrà mgbe i naenwe sense ( Sometimes you act with great maturity – I’m impressed.)\n\nWe will add this in the revised paper.\n\n4.  \n\n(a)  We acknowledge that our dataset is not representative of the general distribution of sentiments in each language, as we applied various filtering and sampling methods to obtain a balanced and diverse set of tweets (we will add this point in the camera-ready). Therefore, someone looking at the dataset should not assume that random tweets for that language have the same distribution of pos/neg/neu as in the dataset for that language. Moreover, the performance of models trained on our dataset may vary depending on the target data they are applied to, especially if the target data has a very different distribution of classes.  \n\n(b) The test set is already lower-cased in Muhammad et al. 2022. We will update this information in the paper.\n\n(c) The full disagreement data points are not part of our final dataset. However, we will release the individual annotator's label separately, including the full disagreement. This can be helpful for researchers interested in learning from disagreement.\n\n(d)  For offensive language instance, we did this for both reasons as offensive language detection was not the focus of this work and as in many languages these are the only labeled datasets (and thus may be used for many different tasks), we believe it is better to not include large amounts of offensive instances.\n\n5. We intend to run more experiments in order to answer this question. \n\n6. We do not have a top 5 but we intend to collaborate with speakers of other African languages and release more datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A very good reason to accept this paper is the dataset itself and the ability to talk about collection and annotation outside the parts of the world NLP researchers typically focus on. The authors share specific problems and their solutions that should also be good for discussion and help researchers who are interested in working on other lower resource languages. \n\nThey are also releasing the annotations, not just the majority-vote label, which should be useful for researchers interested in methodologies and inter-annotator agreement topics. ", "reasons_to_reject": "There are no risks for having this paper presented at the conference. In the next section, you'll see me ask a bunch of questions of the authors but I don't really consider these weaknesses, certainly not enough to jump up to a reason to reject. ", "questions_to_authors": "I'm going to ask my questions in the order they pop up in the text, but I'll try to give a sense of which ones I think are more important to address:\n\n1) You have picked sentiment and in lines 47-56, you mention a number of reasons that sentiment may be useful (literary analysis, culturonomics, commerce, psychology, social science). One that I don't think those quite capture would be \"linguistic\". Here, I'm thinking about how, for example, there's a malefactive particle in at least one Ethiopian language that lets a speaker/writer give the sense that someone did something AGAINST someone else, in Dutch there's \"tet\" which marks surprise/irritation, in Czech diphthongs can mean affection (or pejoration), in Cantonese, throwing a -k at the end of particles can intensify the emotion being expressed, in Tongan there are determiners that express sympathy, in Manambu there is a \"frustrative\" case marking for when things are done in vain, etc. \n\nIn my heart of hearts, my hope is that your dataset could help people find affective markers/word-order/morphemes but...is that plausible, do you think? Or is your expectation that what your annotators picked up on were louder words like adjectives (\"awful\", \"wonderful\", etc). I don't think this particularly needs to be added to your paper, but I am curious about how you'd look at the dataset being used for more linguistic/sociolinguistic/pragmatics research questions.\n\n2) You mentioned using Saif Mohammad's sentiment guidelines, I think you mean the \"base\" questions he has (not the ones that try to get at what the sentiment is direct towards). It probably would be good to share the explicit instructions for annotators, probably in an appendix. This I do think is probably a minor weakness and worth addressing, btw. It also connects to some of your comments on code-switching texts—you have some annotators who don't know parts of those mixed language tweets and it feels like they should have a \"I don't understand this\" option (which would also be relevant for everyone else since I am certain as a native English speaker you could give me completely English tweets that I could not understand to give a sentiment to). \n\n2b) Part of this question comes from looking at your examples in Table 1. The Amharic example kicks off with the very powerful bigram \"ጨካኝ አረመኔ\" (~brutal barbarian, pitiless pagan) but the rest of it has a much lighter tone as far as I can tell. I don't know the referent but it strikes me that this is fairly sarcastic and has an intent to be humorous which is \"negative\" in a way that is less clear than \"I hate X\". You mention sarcasm a number of times—as does Mohammad 2016—I'm wondering what you might say about that topic more generally. \n\nThis also feels related to the annotation problem you describe in Twi with “Tweaa” being potentially offensive depending on context—many annotators have a hard time distinguishing a speaker/writer’s sentiment when they use taboo terms (the terms that one community of speakers might think is unexceptional or just expressive may be no-go’s for other speakers who ignore the speaker intention). Do you have any thoughts about your annotators’ predilections on these things? \n\nIf I gave you a magic button to press that would replace positive/negative/neutral with any categories of your choosing with high accuracy, which categories would you pick? That is, is valence the real thing you most want or is it a stand-in/approximation/first-step for something you actually would rather collect but is harder?\n\n3) You briefly mention some of the ways African languages present challenges for sentiment analysis that English and other languages may not. You mention tone, code-switching, and digraphia in lines 127-128 but I don't think you actually talk about tone. Here, I assume you mean phonemic tone since lots of African languages are tonal in nature. You discuss code-switching and digraphia but do you have anything to say about phonemic tone and your dataset/annotations? About half your languages use tone but I think only Yorùbá marks them in writing...but maybe the lack of marking them in tweets causes some extra homographs but...I'm not sure how that would be different than other languages which are replete with homographs (English lead=metal, lead=guide). \n\n4) You have to do a bunch of things to get a meaningful (not-all-neutral) set of tweets. I think you do a pretty good job but I wonder if you could address what you see as the consequences of these choices for possible uses of the dataset. This is a suggestion I think would help the paper. \n\n4b) A minor thing but why are the Nigerian language test sets lower-cased (you reference Muhammad et al 2022, but it’d be nice if you give the reason so readers don’t have to track that reference down). \n\n4c) You discard “full disagreement” tweets—are those going to be in the dataset? I think that may help with the use of your dataset, provided it’s easy for people to see that these are more for agreement/annotator kinds of questions. I think your mention of Prabhakaran et al (2021) means that you probably do release these but it’s not completely clear to me. \n\n4d) What do you think are the consequences of removing offensive tweets in Algerian Arabic? And was your intention more to produce a non-offensive dataset or to not disturb your annotators? I think it is relevant to share your intentions for this. \n\n5) The idea that Yorùbá is a generally good source language strikes me as very strange. I mean, you are empirically showing it to be the case and mention Adelani et al’s similar finding in NER but...this strikes me as “surely a coincidence” or perhaps because there are some proper names with similar sentiment in Ethiopia and Nigeria…what do you think the most reasonable explanation of this is?\n\n6) Your plan is to keep releasing more languages (and enable others to follow your lead), which is great. I wonder which languages are at the top of your list. Do you want coverage in terms of speaker populations? Coverage over countries that yet covered? Languages that are outside the language families you’ve already covered? I’m interested in your actual “Top Five” but even moreso for what leads you to that Top Five instead of another Top Five. I think being explicit about this helps readers understand where you're headed/why and what they should keep an eye out for or help with. \n", "paper_topic_and_main_contributions": "The authors present a dataset of 12 African languages, annotated for sentiment by native (and usually local) speakers. They describe challenges in collection and annotation. They also present some results from pre-trained language models, discussing which models do best with which languages and which source languages do the best in zero-shot learning for Oromo and Tigrinya. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nDCeASmceL", "reviewer": "Reviewer_j4yL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for providing feedback. The main contribution of this work is to introduce new data resources for 14 African languages (from 3 language families), which is aligned with the EMNLP's main contribution: \"new data resources, particularly for low-resource languages\". We believe that our data resource is valuable for the NLP community, as it can enable new research directions and applications for these languages. Most importantly, it will encourage a new generation of African researchers who can work on languages that matter deeply to them and their communities. \n\nBelow, we respond to the reviewer’s concerns..\n\n1. With the additional page, if accepted, we will add a discussion of class imbalance for different languages, possible reasons for this, and their implications.  \n\n\n2. For orm and tig languages, we don't have enough training data. Therefore, we chose to conduct a zero-shot experiments for these languages.  In fact, Twitter is not the most commonly used platform in some African languages including Ethiopia, where most orm and tig speakers live. Hence, this was  an expected challenge in low-resource settings (less availability of suitable public data, difficulty in obtaining qualified annotators, etc.). Still, we are happy we could create larger datasets for 12 other languages, and at least some data for ‘orm’ and ‘tig’. \n\n3. We will add the suggested per-label precision and recall in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A new useful dataset that can be used for sentiment analysis in Afriacan languages.\n- Extensive experimental results on the benchmark.", "reasons_to_reject": "- Limited explanation of imbalanced data distribution in Table 6. Why there is no training samples in 'orm' and 'tir'? I see that there was a difficulty in collecting data from annotators, but it does not explain why we don't have training samples. For example, we can have 1500 training samples and 600 test samples in 'orm' language.\n- More detailed experimental results are required. In Table 8, only F1 score is reported, but it is better to give per-label precision and recall as well.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a new Twitter benchmark for sentiment analysis in african languages, and provide experimental results with few baseline models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces 14 sentiment analysis datasets, comprised of Tweets from 14 African languages. It is thorough and detailed as indicated by its Soundness scores (two 4s and one 3), describing not only the data collection and annotation processes and experimental benchmarks but also the specific challenges associated with curating each dataset. Reviewers acknowledge the positive and far-reaching impact these discussions can have on future research in low-resource NLP, as indicated by the Excitement scores (two 4s, one 3). In addition to its contributions to the field of sentiment analysis, this paper may be of interest to those working on inter-annotator agreement topics, as full annotations are released alongside the majority vote label (Reviewer TvNi). Likewise, discussion of language-specific challenges and probable solutions may be of consequence to those interested in data collection methodologies. Minor individual concerns were raised by each reviewer (e.g. missing details regarding fine tuning, precision and recall, reasons for removing offensive Tweets), but all were adequately resolved during the author rebuttal period.

In light of this, only minor revisions, addressing reviewers’ comments and questions, need to be made to ensure this paper is camera ready.
Decision: Accept-Main