NLP-LISAC at SemEval-2023 Task 12: Sentiment Analysis for Tweets expressed in African languages via Transformer-based Models
Year: 2023

Abstract:
This paper presents our systems and findings for SemEval-2023 Task 12: AfriSenti-SemEval: Sentiment Analysis for Low-resource African Languages. The main objective of this task was to determine the polarity of a tweet (positive, negative, or neutral). Our submitted models (highest rank is 1 and lowest rank is 21 depending on the target Track) consist of various Transformer-based approaches.

heading: 1 Introduction
text: Sentiment analysis is a branch of natural language processing that involves the use of computational techniques to identify and extract subjective information from textual data. It involves the classification of text based on the writer’s attitude, emotions, and opinions toward a particular subject or topic.
In recent years, there has been a growing interest in sentiment analysis for African languages. This is because African languages are some of the most widely spoken languages in the world, yet they are often underrepresented in natural language processing research. However, sentiment analysis for African languages presents unique challenges due to the linguistic and cultural diversity of the continent.
One of the main challenges of sentiment analysis for African languages is the lack of annotated data. Annotated data is essential for training machine learning models for sentiment analysis, but there is a limited amount of annotated data available for many African languages. Additionally, African languages often have complex grammatical structures and a wide range of dialects, making it difficult to develop effective sentiment analysis models.
Despite these challenges, there have been some recent advances in sentiment analysis for African languages. Researchers have used transfer learning and cross-lingual models to develop sentiment analysis models for African languages, which can
leverage existing data from other languages to improve performance. There have also been efforts to create annotated datasets specifically for African languages.
This paper presents our findings on SemEval2023 Task 12: AfriSenti-SemEval: Sentiment Analysis for Low-resource African Languages (Muhammad et al., 2023b). Our method consists of finetuning various transformer-based (Vaswani et al., 2017) models.
The rest of the paper is structured in the following manner: Section 2 provides the main objective of each sub-task. Section 3 describes the employed models. Section 4 details the experiments. And finally, Section 5 concludes this paper.

heading: 2 Task Description
text: The AfriSenti-SemEval Shared Task 12 is based on a collection of Twitter datasets in 14 African languages for sentiment classification. It consists of three sub-tasks:
• Task A: Monolingual Sentiment Classification: The objective of this task is to determine the polarity (positive, negative, or neutral) of a tweet in a specific language, using training data in that language. If a tweet expresses both positive and negative sentiments, the stronger sentiment should be selected. There are 12 tracks in this sub-task, each for a different language: Track 1 for Hausa, Track 2 for Yoruba, Track 3 for Igbo, Track 4 for Nigerian Pidgin, Track 5 for Amharic, Track 6 for Algerian Arabic, Track 7 for Moroccan Arabic/Darija, Track 8 for Swahili, Track 9 for Kinyarwanda, Track 10 for Twi, Track 11 for Mozambican Portuguese, and Track 12 for Xitsonga (Mozambique Dialect).
• Task B: Multilingual Sentiment Classification: Given combined training data from
199
Task-A (Track 1 to 12), determine the polarity of a tweet in the target language (positive, negative, or neutral). This sub-task has only one track with 12 languages (Hausa, Yoruba, Igbo, Nigerian Pidgin, Amharic, Algerian Arabic, Moroccan Arabic/Darija, Swahili, Kinyarwanda, Twi, Mozambican Portuguese, and Xitsonga (Mozambique Dialect)), Track 16: 12 languages in Task A.
• Task C: Zero-Shot Sentiment Classification: Given unlabelled tweets in two African languages (Tigrinya and Oromo), leverage any or all of the available training datasets (in Task:A ) to determine the sentiment of a tweet in the two target languages. This task has two (2) tracks, Track 17: Zero-Shot on Tigrinya, and Track 18: Zero-Shot on Oromo.

heading: 3 System Description
text: To tackle AfriSenti-SemEval Shared Task 12, several transformer-based models have been used depending on each Track:
• Hausa: we have fine-tuned xlm-robertabase-finetuned-hausa1 which is: "a Hausa RoBERTa (Liu et al., 2019) model obtained by fine-tuning xlm-roberta-base (Conneau et al., 2020) model on Hausa language texts. It provides better performance than the XLMRoBERTa on text classification and named entity recognition datasets".
• Yoruba: we have fine-tuned xlm-robertabase-finetuned-yoruba2 which is a: "Yoruba RoBERTa model obtained by fine-tuning xlmroberta-base model on Yorùbá language texts. It provides better performance than the XLMRoBERTa on text classification and named entity recognition datasets".
• Igbo: we have fine-tuned xlm-robertabase-finetuned-igbo3 which is: "an Igbo RoBERTa model obtained by fine-tuning xlm-roberta-base model on Hausa language texts. It provides better performance than the XLM-RoBERTa on named entity recognition datasets".
1https://huggingface.co/Davlan/ xlm-roberta-base-finetuned-hausa
2https://huggingface.co/Davlan/ xlm-roberta-base-finetuned-yoruba
3https://huggingface.co/Davlan/ xlm-roberta-base-finetuned-igbo
• Nigerian Pidgin, Amharic, Morrocan Darija, Mozambican Portuguese, Xitsonga, and Multilingual: we have fine-tuned mdebertav3-base4 (He et al., 2021a,b) which is: "a multilingual version of DeBERTa that use the same structure as DeBERTa and was trained with CC100 multilingual data. The mDeBERTa V3 base model comes with 12 layers and a hidden size of 768. It has 86M backbone parameters with a vocabulary containing 250K tokens which introduces 190M parameters in the Embedding layer".
• Algerian Arabic: we have fine-tuned DziriBERT5 (Abdaoui et al., 2021) which is: "the first Transformer-based Language Model that has been pre-trained specifically for the Algerian Dialect. It handles Algerian text contents written using both Arabic and Latin characters. It sets new state-of-the-art results on Algerian text classification datasets, even if it has been pre-trained on much less data ( 1 million tweets)".
• Swahili: we have fine-tuned xlm-robertabase-finetuned-swahili6 which is: "a Swahili RoBERTa model obtained by fine-tuning xlmroberta-base model on Swahili language texts. It provides better performance than the XLMRoBERTa on text classification and named entity recognition datasets".
• Kinyarwanda: we have fine-tuned xlmroberta-base-finetuned-kinyarwanda7 which is: "a Kinyarwanda RoBERTa model obtained by fine-tuning xlm-roberta-base model on Kinyarwanda language texts. It provides better performance than the XLM-RoBERTa on named entity recognition datasets".
• Twi, Zero-Shot Tigrinya, and Zero-Shot Oromo: we have fine-tuned AfriBERTa large8 (Ogueji et al., 2021) which is: "a pre-trained multilingual language model with around 126 million parameters. The model
4https://huggingface.co/microsoft/ mdeberta-v3-base
5https://huggingface.co/alger-ia/ dziribert
6https://huggingface.co/Davlan/ xlm-roberta-base-finetuned-swahili
7https://huggingface.co/Davlan/ xlm-roberta-base-finetuned-kinyarwanda
8https://huggingface.co/castorini/ afriberta_large
has 10 layers, 6 attention heads, and 768 hidden units. The model was pretrained on 11 African languages namely - Afaan Oromoo (also called Oromo), Amharic, Gahuza (a mixed language containing Kinyarwanda and Kirundi), Hausa, Igbo, Nigerian Pidgin, Somali, Swahili, Tigrinya, and Yorùbá. The model has been shown to obtain competitive downstream performances on text classification and Named Entity Recognition on several African languages, including those it was not pretrained on".
xlm-roberta-base, DziriBERT, and AfriBERTa were fine-tuned on the merge of training and dev sets related to each Track (for example, DziriBERT was fine-tuned on the merge of Algerian Arabic training and dev sets provided by the organizers to handle Track 6: Algerian Arabic, xlm-robertabase-finetuned-hausa was fine-tuned on the merge of Hausa training and dev sets provided by the organizers to handle Track 1: Hausa, and xlm-robertabase-finetuned-kinyarwanda was fine-tuned on the merge of Kinyarwanda training and dev sets provided by the organizers to handle Track 9: Kinyarwanda), with the exception of Zero-Shot Tracks where no training set is provided, therefore, AfriBERTa large was fine-tuned on the dev set only for Tigrinya and Oromo.
mDeBERTa V3 base was fine-tuned on the merge of training and dev sets of all the first 12 tracks (Task A) along with the dev set of Track 16 (Task B: Multilingual Sentiment Classification).

heading: 4 Experimental Results
text: The experiments have been conducted in Google Colab environment9, The following libraries: Transformers - Hugging Face10 (Wolf et al., 2020), and Keras11 were used to fine-tune and to assess the performance of our models.

heading: 4.1 Datasets
text: The dataset (Muhammad et al., 2023a) involves tweets labeled with three sentiment classes (positive, negative, neutral) in 14 African languages. Each tweet is annotated by three annotators following the annotation guidelines in (Mohammad, 2016). Figure 1 represents a sample of the Moroccan Arabic/Darija training set.
9https://colab.research.google.com/ 10https://huggingface.co/docs/transformers/index 11https://keras.io/

heading: 4.3 Experimental Settings

heading: 4.2 Evaluation Metric
text: • Algerian Arabic: we have fine-tuned DziriBERT using a batch size of 8 and 4 epochs.
• Swahili: we have fine-tuned xlm-robertabase-finetuned-swahili using a batch size of 8 and 6 epochs.
• Kinyarwanda: we have fine-tuned xlmroberta-base-finetuned-kinyarwanda using a batch size of 16 and 7 epochs.
• Twi: we have fine-tuned AfriBERTa large using a batch size of 8 and 5 epochs.
• Zero-Shot Tigrinya: we have fine-tuned AfriBERTa large using a batch size of 8 and 6 epochs.
• Zero-Shot Oromo: we have fine-tuned AfriBERTa large using a batch size of 8 and 8 epochs.
Table 1 summarizes the hyperparameters settings used during the fine-tuning.

heading: 4.4 System Performance
text: Table 2 depicts the results of our proposed approaches on SemEval-2023 Task 12: AfriSentiSemEval: Sentiment Analysis for Low-resource African Languages. We can see that DziriBERT performs well on the Algerian Arabic test set by achieving first place. For Morrocan Darija, mDeBERTa V3 base secures third place and outperforms DarijaBERT13 (around 0.51 average F1 score) even if it was pre-trained on a total of ∼3 Million sequences of Darija dialect representing 691MB of text or a total of 100M tokens. For Zero-Shot Tracks, AfriBERTa large achieves poor results on Tigrinya, and average results on Oromo despite being pre-trained on both languages. XLMRoBERTa (base-sized model) yields average results for Hausa, Yoruba, Igbo, and poor results for Swahili and Kinyarwanda. mDeBERTa V3 base
13https://github.com/AIOXLABS/DBert
secures 17th place out of 33 in Track 16: Multilingual by achieving an average F1 score of 67.30% with a margin of 7.76% from the top score (75.06% F1 score).

heading: 5 Conclusion
text: In this paper, we described our approach for tackling SemEval 2023 Task 12: AfriSenti-SemEval: Sentiment Analysis for Low-resource African Languages (Muhammad et al., 2023b). Our proposed approach consisted of various Transformer-based models. We secured 1st position in Track 6: Algerian Arabic, and 3rd position in Track 7: Moroccan Arabic/Darija.
Future studies will focus on improving the obtained results by (1) incorporating external resources including sentiment corpus (Muhammad et al., 2022; Yimam et al., 2020; Mabokela and Schlippe, 2022) and lexicons for African Languages, and (2) fine-tuning other Transformerbased models (Devlin et al., 2019) pretrained on African Languages 14 (Martin et al., 2022).

heading: Future Work
text: Future studies will focus on improving the obtained results by (1) incorporating external resources including sentiment corpus (Muhammad et al. , 2022; Yimam et al. , 2020; Mabokela and Schlippe, 2022) and lexicons for African Languages, and (2) fine-tuning other Transformerbased models (Devlin et al. , 2019) pretrained on African Languages 14 (Martin et al. , 2022)