ABCD Team at SemEval-2023 Task 12: An Ensemble Transformer-based System for African Sentiment Analysis
Year: 2023

Abstract:
This paper describes the system of the ABCD team for three main tasks in the SemEval-2023 Task 12: AfriSenti-SemEval for Low-resource African Languages using Twitter Dataset. We focus on exploring the performance of ensemble architectures based on the soft voting technique and different pre-trained transformerbased language models. The experimental results show that our system has achieved competitive performance in some Tracks in Task A: Monolingual Sentiment Analysis, where we rank the Top 3, Top 2, and Top 4 for the Hause, Igbo and Moroccan languages. Besides, our model achieved competitive results and ranked 14 place in Task B (multilingual) setting and 14 and 8 place in Track 17 and Track 18 of Task C (zero-shot) setting.

heading: 1 Introduction
text: The AfriSenti-SemEval Shared Task 12 (Muhammad et al., 2023b) aims at building Sentiment Analysis (SA) systems for 17 African languages, including Hausa, Yoruba, Igbo, Nigerian Pidgin from Nigeria, Amharic, Tigrinya, and Oromo from Ethiopia, Swahili from Kenya and Tanzania, Algerian Arabic dialect from Algeria, Kinyarwanda from Rwanda, Twi from Ghana, Mozambique Portuguese from Mozambique and Moroccan Arabic/Darija from Morocco. This shared task has three main tasks, including two zero-shot tracks, one multilingual track and 12 monolingual tracks. The zero-shot tracks require training two zero-shot models where each model works for only one language, and each model has training from the 12 languages in the monolingual tracks. The multilingual track requires training multilingual SA models that are able to handle all languages. Twelve monolingual tracks need training individual monolingual models where each model works for only one language. The dataset involves tweets labelled with three sentiment classes (positive, negative, neutral)
in 14 African languages. Each tweet is annotated by three annotators following the annotation guidelines in (Mohammad, 2016). They used a form of a majority vote to determine the sentiment of the tweet (Mohammad, 2022; Yimam et al., 2020). All the tweet of the dataset is code-mixed, which can increase the performance of the model.
In this paper, we propose an ensemble architecture for the AfriSenti-Semeval shared task. Our ensemble architecture is based on the pre-trained transformer-based language models and soft voting technique. In our case, the AfroXLMR, AfriBERTa, and LaBSE are employed as the base classifier because these models support African languages in the shared task. The final prediction is combined from the base classifier using the soft voting technique.
The rest of the paper is organized as follows. Section 2 provides the related work. The system description is presented in Section 3, followed by evaluation results in Section 5. The experimental setup and conclusion is discussed in Section 4 and Section 6, respectively.

heading: 2 Related Work
text: Sentiment Analysis has been studied in the NLP field for two past decades for low-resource languages. However, in the case of low-resourced languages such as African languages, the studies have yet to progress well compared to those of high-resourced languages due to the reasons such as the unavailability of annotated corpora.
In the Arabic language, a set of different machine learning and deep learning approaches has been applied to the sentiment analysis task of the language (Abdulla et al., 2013; Duwairi et al., 2014; Heikal et al., 2018; Shoukry and Rafea, 2012). In (Abdulla et al., 2013), the authors have constructed an Arabic sentiment analysis dataset extracted from Twitter and classified it as a negative and positive sentiment. The methods they used were both lexicon-based
324
in which they constructed a set of words labelled with some polarity indicators, and using those constructed lexicons; they implemented an algorithm that tells if the whole tweet is positive or negative depending on the number of positive or negative lexicon it contains. At the same time, they have also used supervised methods of machine learning models such as support vector machine (SVM) (Zhang, 2001), and K-nearest neighbours (KNN) (Bishop and Nasrabadi, 2006) trained on the constructed dataset. They have achieved the best result by using SVM classifiers when compared to other methods they have used.
In other Arabic sentiment analysis studies by (Heikal et al., 2018), the authors have used deep learning models to outperform the state-of-the-art result. Their approach was an ensemble of convolutional neural network (CNN) (Heaton, 2018) and long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) models through which they have achieved an F1-Score of 64.46% beating the previous state-of-the-art by more than 10%. However, Arabic has also benefited from the recent breakthrough in NLP studies related to transformer models and transfer learning techniques. A pretrained Arabic version of BERT has achieved stateof-the-art results in many NLP downstream tasks of Arabic, including sentiment analysis, named entity recognition and question answering (Antoun et al., 2020).
In the case of Amram et al. (2018), the authors studied the effect of the input as tokens or morphemes. They concluded that for linear neural network architectures such as multi-layer perceptron (MLP) (Heaton, 2018) and (CNN), the token level granularity had achieved higher accuracy, while for LSTM architecture, the level of the morpheme achieved better results. Furthermore, they achieved state-of-the-art results for the Hebrew sentiment analysis task, using the CNN network of token level input with more than the accuracy of 89%.
With the emergence of deep learning models, Amharic sentiment analysis has also benefited from the breakthrough in the field of NLP using deep learning. In another study for Amharic sentiment classification, Yeshiwas Getachew and Abebe Alemu (Alemu, 2018) used a deep learning approach to model sentiment analysis tasks of the language. They have constructed a new dataset collected from social media platforms, Facebook, and labelled by Amrahic linguist experts. Using a fully
connected neural network with different parameter tuning, including the number of neurons, they achieved an accuracy of 96.61% in the validation set.

heading: 3 System Description

heading: 3.1 Approach
text: The diagram in Figure 1 illustrates our ensemble approach for Task A. The framework consists of three main layers: a pre-processing layer, a layer of contextual language-based models, and a voting ensemble layer. Firstly, the input text is subject to several processing steps in the pre-processing layer. Following this, we fine-tune different pre-trained contextual language models in order to obtain probability outputs of labels. Finally, the probability outputs from individual models are combined using a soft voting technique to get the final prediction. The detailed structure of the framework is described in the following.
Pre-processing Layer: Pre-processing is one of the essential components in the classification system in the NLP field. However, the languages in the shared task are unfamiliar; therefore, we design a standard list of pre-processing steps for all languages, including:
• Word Normalization: We used the regular expression technique to normalize some words or phrases which are the same meanings in the sentence. For example, we replace the “URL” with the “website” word.
• Noise Removal: We observed that there are many noises, such as punctuation and special characters, in the dataset. We found that these noises are not necessary for the sentence-level dataset. Therefore, we remove it from the samples.
• Emoji Replacement: Because of the large amount of emoji in the dataset, up to 28.85% (based on our statistics), the emoji is very important for the model which can increase the performance of the model. We replace the emoji with three labels such as negative, neutral and positive.
Fine-tuning Language Model: As can be seen in Figure 1, we utilize the power of three pre-trained contextual language models, including AfroXLMR (Alabi et al., 2022), AfiBERTa (Ogueji
et al., 2021), and LaBSE (Feng et al., 2022) model as the base classifiers. To fine-tune the language models, we followed the approach of (Devlin et al., 2019a), which is presented in detail below:
Given a pre-processed Twitter with N words: X “ tx1, x2, ...., xNu, we first use a corresponding tokenizer to prepare the inputs. Then, we employ a pre-trained language model with L transformer layers to calculate the contextualized representations HL “ thLcls, hL1 , ...., hLNu P RNˆdimh , where dimh is the dimension of the representation vector. Finally, we extract the contextualized representation hLcls of [CLS] token in the last L transformer layer as the feature of the input. The obtained representations is directly fed to the linear layer with the Softmax activation function to calculate the score vector ŷpaq for each class.
ŷ “ softmaxpW ¨ hLcls ` bq (1)
where W and b are the learnable parameters of the output layer. We use the Category Cross-Entropy loss to optimize the model.
Soft voting Scheme: Our motivation for applying an ensemble approach is to take advantage of the performances of various models. Given predictions tŷθ1 , ŷθ2 , .., ŷθnu the n base classifiers. We applied the simple soft voting technique to merge the predictions of the base models. In our case, the individual classifiers are treated equally. We sum up the probability output of n classifiers and choose the sentiment class with the highest probability as the final prediction.

heading: 3.2 Pre-trained Contextual Language Models
text: We briefly explain the three pre-trained language models used in this paper.
• AfroXLMR: AfroXLMR is a large language model developed by (Alabi et al., 2022) and is released for the community of researchers in African languages. It is based on the XLMRoBERTa architecture and applies the multilingual adaptive fine-tuning technique to 17 African languages and three high-resource languages (English, French, and Arabic) simultaneously.
• AfriBERTa: AfriBERTa is a transformerbased multilingual language model trained on 11 African languages, all of which are low-resource. It had developed by (Ogueji et al., 2021). The author had trained a transformer (Vaswani et al., 2017) with the standard masked language modelling objective of (Devlin et al., 2019b) without next sentence prediction. This is also the same approach used in XLM-R (Conneau et al., 2020).
• LaBSE: LaBSE is a language-agnostic BERT sentence embedding model supporting up to 109 languages. This language model was developed by (Feng et al., 2022). To have the best method for learning multilingual sentence embeddings, they combined the best methods for learning monolingual and cross-lingual representations, including: masked language modelling (MLM), translation language modelling (TLM) (Lample and Conneau, 2019), dual encoder translation ranking (Guo et al.,
2018), and additive margin softmax (Yang et al., 2019).

heading: 4 Experimental Setup
text: Data and Preprocessing: We utilized the official training set for training models. As the competition rules stipulated, no additional data was used during the training process (Muhammad et al., 2023a). The development set was used to optimize the hyper-parameters for each track and task.
Evaluation Metrics: The evaluation metric for three Tasks (A, B, C) is a Weighted F1-score between submission and test gold set.
Configuration Settings: We implemented our models using Trainer API from Hugging Face library (Wolf et al., 2020). The maximum input length is set as 128 tokens, and the number of epochs is set as 10 with a batch size of 32 for all languages. We used an AdamW optimizer with a linear schedule warmup technique.
Submitted Systems: We submitted the different models based on the task to the evaluation phase. For task A - Monolingual SA, we submit the performance of the ensemble soft voting model for all languages. For task B, because of the limitation of computation resource languages, we divide the training into 5 folds and train the LaBSE model on each fold to predict the test set. Then, we combine the results of 5 folds using the soft
voting technique. For Task C, our strategy is to utilize the Google Translation1 to translate the source language to a target language. In our case, we translate the test set on Track 17 and Track 18 into the Hausa language. We choose the Hause language as the source language for the following reasons: (1) the number of samples in the training set and distribution between labels.

heading: 5 Results and Discussion
text: In this section, we present the official results of our final submission model for three main Tasks in the AfriSenti-SemEval Shared Task competition. For task A, we only compare results with the results from the two top teams for each track. In comparison, we report the top 5 systems for task B and task C, respectively.
Task A: Monolingual Sentiment Classification Table 1 presents the performances of our ensemble model compared with two top teams for 12 tracks. Our system gives competitive results on the three tracks such as Track 1 (Hausa), Track 2 (Yoruba), Track 3 (Igbo) , Track 7 (Moroccan), and Track 8 (Swahili). Unfortunately, our submission system is not effective for the remainder Tracks. To explore the reason, we report the results of base models and ensemble systems on the development set. As seen in Table 2, we observe that the performance of the ensemble model decreases significantly on some Tracks (e.g. Track 4, Track
1https://translate.google.com/
11) due to the poor performance of base models. Moreover, we noticed that our approach is effective for Tracks with a lot of samples and balance in the training data. In this study, we intend to investigate the effectiveness of soft voting techniques based on different transformer-based models for African languages. Therefore, we used the ensemble model as the final submission system instead of the best model on the development set.
Task A: Multilingual Sentiment Classification Table 3 present the results of our submission on Task B. Officially, we achieved the F1-score of 69.22% on the test set (Top 14). As mentioned in Section 4, due to the limitation of computational resources, we must split the training set as the 5 folds and use the LaBSE model as the main classifiers. Then we use the soft-voting technique to merge the prediction of 5 folds on the test set.
Task C: Zero-Shot Sentiment Classification Table 4 shows our submission results compared with the top 5 systems in Task C - Zero-shot sentiment analysis. We ranked 14th and 8th among all participating systems for Track 17 and Track 18, respectively. One of the reasons for the poor performance of our submission system is the error in the translation process to translate the
source language to the target language.

heading: 6 Conclusion
text: In this paper, we presented a simple and efficient ensemble architecture for sentiment analysis tasks in the SemEval-2023 Task 12: AfriSenti-SemEval. Our system is based on fine-tuning the pre-trained transformer-based language model as the base classifiers and the soft voting technique to combine the prediction of different base classifiers. Our experiments demonstrated that it achieves competitive results on some languages in Task A: Monolingual Sentiment Analysis without relying on any additional resources. For future work, we plan to improve our system to handle the imbalanced problem in some languages. Besides, data augmentation is
also a promising direction to enhance the overall system’s performance.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their valuable comments and the AfriSentiSemEval organization for organizing the shared task competition. This research was supported by The VNUHCM-University of Information Technology’s Scientific Research Support Fund.

heading: Future Work
text: For future work, we plan to improve our system to handle the imbalanced problem in some languages. Besides, data augmentation is
also a promising direction to enhance the overall system’s performance