GMNLP at SemEval-2023 Task 12: Sentiment Analysis with Phylogeny-Based Adapters
Year: 2023

Abstract:
This report describes GMU’s sentiment analysis system for the SemEval-2023 shared task AfriSenti-SemEval. We participated in all three sub-tasks: Monolingual, Multilingual, and Zero-Shot. Our approach uses models initialized with AfroXLMR-large, a pre-trained multilingual language model trained on African languages and fine-tuned correspondingly. We also introduce augmented training data along with original training data. Alongside finetuning, we perform phylogeny-based adaptertuning to create several models and ensemble the best models for the final submission. Our system achieves the best F1-score on track 5: Amharic, with 6.2 points higher F1-score than the second-best performing system on this track. Overall, our system ranks 5th among the 10 systems participating in all 15 tracks.

heading: 1 Introduction
text: With the increasing use of the internet and social media, the digital availability of various languages is rapidly expanding. This expansion opens avenues for Natural Language Processing (NLP) applications such as Sentiment Analysis (SA) and Machine Translation (MT). Nevertheless, despite African languages comprising 30% of around 6,000 living languages (Skutnabb-Kangas et al., 2003), most of them are not supported by modern language technologies, leading to an ever-widening gap in language technology access (Joshi et al., 2020; Blasi et al., 2022).
Recently, SA has gained increasing attention, with its applications in various domains, such as public health, literature, and social sciences (Mohammad, 2022). Despite the growth in this area, most previous works do not include African languages. This shared task focuses on SA on a
*Joint contributions: MA performed model training, RX worked on data processing and paper writing, FF constructed the model.
Twitter dataset in 14 African languages, including Hausa (ha), Yoruba (yo), Igbo (ig), Nigerian Pidgin (pcm), Amharic (am), Tigrinya (tg), Oromo (or), Swahili (sw), Xitsonga (ts), Algerian Arabic (dz), Kinyarwanda (kr), Twi (twi), Mozambican Portuguese (pt), and Moroccan Darija (ma).
This paper presents a novel SA system that effectively addresses the challenge of low-resource and multilingual sentiment classification for multiple African languages. We leverage multilingual language models and propose data augmentation methods to increase the training data size. In addition, we perform phylogeny-based adapter-tuning to create several models. These models are then ensembled to create the final model.

heading: 2 Related Work
text: While SA is a popular research area in NLP, its application in African languages is relatively rare. This is mainly due to the lack of resources, making it challenging to obtain the data needed to train and evaluate the models (Mabokela and Schlippe, 2022). One solution is creating resources, such as annotated datasets. However, this requires a significant amount of manual annotation (Shode et al., 2022). Using augmented data to improve per-
1172
formance in low-resource languages is another approach that has been explored for various tasks (Xia et al., 2019; Muhammad et al., 2022; Alam and Anastasopoulos, 2022; Xie and Anastasopoulos, 2023, inter alia), where synthetic data is generated from existing data to increase the size of the training set.
Leveraging pre-trained multilingual language models is a popular choice for SA in African languages (Dossou et al., 2022; Martin et al., 2021; Alabi et al., 2022; Muhammad et al., 2022; Martin et al., 2022). These language models are trained on a large amount of data from different languages, including African languages, which enables them to capture a wide range of linguistic features and patterns. While these pre-trained models have shown promising results, they are imperfect in handling low-resource languages.
Adapters are designed to adapt a large pretrained language model to a downstream task, enabling efficient transfer learning (Alabi et al., 2022; Ansell et al., 2021). Phylogeny-based adapters (Faisal and Anastasopoulos, 2022), similar to the hierarchical ones of Chronopoulou et al. (2022) enable knowledge sharing across related languages from the phylogeny-informed tree hierarchy. Our work builds on this approach to address the challenge of SA in low-resource scenarios, demonstrating that it can effectively adapt a pre-trained multilingual language model to the target language with limited training data.

heading: 3 Task Description
text: The AfriSenti-SemEval Shared Task 12 (Muhammad et al., 2023b) consists of three sub-tasks: Monolingual (Task A), Multilingual (Task B), and Zero-Shot (Task C). The primary objective of this shared task is to determine the sentiment of a tweet in a target language, which could be positive, negative, or neutral. A stronger emotion should be chosen when a tweet exhibits positive and negative sentiments.
Task A: Monolingual Task A aims to determine the sentiment of tweets for each language in a monolingual setting. Table 2 shows the detail of each language. Every language is a track for this sub-task, creating Tracks 1 through 12.1
1Track 13 through Track 15 were not included in the final competition as the respective datasets for these tracks were not released.
Task B: Multilingual Task B aims to determine the sentiment for tweets using a combination of all training data from the 12 languages in Task A. This sub-task includes only one track, Track 16.
Task C: Zero-Shot Task C aims to identify the sentiment of tweets from either Tigrinya or Oromo under a zero-shot setting, i.e., no training data are available for these languages). This sub-task is divided into Track 17 (Tigrinya) and 18 (Oromo).

heading: 4 System Overview

heading: 4.1 Data Source
text: The dataset used in the system is mainly sourced from AfriSenti (Muhammad et al., 2023a), which is already split into train, dev, and test. In addition, we use PanLex (Kamholz et al., 2014) and Stanford Sentiment Tree Bank (SST) (Socher et al., 2013) for data augmentation.
Panlex The goal of Panlex is to facilitate the translation of lexemes between all human languages. A broader lexical and linguistic coverage can be achieved when lexemic translations are used instead of grammatical or corpus data. A total of 20 million lexemes have been recorded in PanLex in 9,000 languages, and 1.1 billion pairwise translations have been recorded.
Stanford Sentiment Tree Bank (SST) The Stanford Sentiment Tree Bank has a sentiment score from 0 to 1 for each sentence. We labeled the sentence as negative if the score was less than or equal to 0.35. We labeled it neutral if the score was between 0.35 and 0.65, and the rest of the sentences were positive.

heading: 4.2 Data Pre-Processing
text: The AfriSenti dataset underwent some preprocessing to remove noise, including @user and RT handlers, URLs, extra while spaces, multiple consecutive punctuations, and characters.2 Emojis were intentionally retained since they are an important part of sentiment analysis, as they often convey emotional content.

heading: 4.3 Data Augmentation
text: To improve the robustness of our language systems to variations in language, we utilize data augmentation techniques to increase the amount of available training data. Specifically, we create three datasets using a dictionary-based augmentation approach, an MT approach, and a combination of the first two approaches. This is just a concatenation of the datasets obtained by the two approaches above.
Dictionary Based To create more data and handle code-mixed sentences, we employ a naive wordto-word translation augmentation method:
• First, we obtain a Panlex bilingual dictionary from English to a corresponding language.
• Second, we have obtained the (English) sentences from the English Stanford Sentiment Tree Bank.
• Third, we replace any word from the sentence of the Tree Bank that matches an entry from the dictionary with its translation in the corresponding language.
The intuition behind this is that not all English sentence words will be replaced, so it will imitate code-mixing. Also, we anticipate that the translated sentences will largely be ungrammatical, as they are just word-to-word translations with no morphological information, which may lead to word order and morphosyntax errors.
Machine Translation Based We introduce an augmentation technique based on MT. We leveraged the best-performing MT model of Alam and Anastasopoulos (2022), which handles almost all of the task’s languages, to translate sentences from English Stanford Sentiment Treebank to the corresponding language.
2For example, "hellooooo" to "helloo".

heading: 4.4 Model Overview
text: Our system uses a transformer-based multilingual model, AfroXLMR-large (Alabi et al., 2022). AfroXLMR-large is developed by performing language adaptation of the XLM-R-large model (Conneau et al., 2019) on 17 distinct African languages, including Afrikaans, Amharic, Hausa, Igbo, Malagasy, Chichewa, Oromo, NigerianPidgin, Kinyarwanda, Kirundi, Shona, Somali, Sesotho, Swahili, isiXhosa, Yoruba, and isiZulu, which collectively represent the major African language families. AfroXLMR-large also incorporates three high-resource languages: Arabic, French, and English. When we fine-tune the model for our task, we fine-tune only a task-adapter instead of finetuning the whole model.
In addition to the base model, we utilize phylogeny-based adapter-tuning (Faisal and Anastasopoulos, 2022) to generate multiple models for different language families. As described by Houlsby et al. (2019), adapters are small neural components designed to adapt a large pretrained language model to a downstream task using lightweight layers inserted between each layer of the pre-trained model, enabling efficient adaptation. Here we use phylogeny-based adapters, similar to Faisal and Anastasopoulos (2022), which enables knowledge sharing across similar or related languages from the same language family or genus. Table 2 shows the language family hierarchy for the 12 languages in Task A, and figure 1 presents a visualization of the architecture. The language families are chosen based on the phylogenetic relationships among languages, which reflects their evolutionary history and linguistic similarities. For example, the Arabic languages, such as Algerian Arabic and Moroccan Darija, are all derived from the Afroasiatic language family and share many linguistic features. The adapter creates several models we can ensemble to hopefully obtain better results.

heading: 5 Experiments
text: Experimental Setup We trained our model using the Adam optimizer (Kingma and Ba, 2014), with a learning rate of 1e−4. The number of epochs was set to 5, and the batch size was 32, with a maximum sequence length of 128. All experiments were conducted on one A100 GPU. We report the weighted F1-score for the model’s performance.

heading: 5.1 Model Selection
text: We experiment with multiple pre-trained models to decide which pre-trained model works best with the given languages. Here we focus on the 12 languages from Task A. Note that for this analysis, we use the unprocessed data with a slightly different data split, as we did not have access to the gold labels for the dev set. In this experiment, we split the train set into train, dev, and test sets, with proportions of 80%, 10%, and 10%, respectively.
We experiment with the following models • mBERT (Devlin et al., 2018): a multilingual ver-
sion of BERT, which is pre-trained on a large amount of multilingual data from Wikipedia. • XLM-RoBERTa (Conneau et al., 2019): a multilingual version of RoBERTa (Liu et al., 2019), which is pre-trained on 2.5TB of CommonCrawl data over 100 languages. • AfriBERTa (Ogueji et al., 2021): a model based on mBERT and (continued) pre-trained on 11 African languages. • AfroXLMR-large (Alabi et al., 2022): a model that is based on XLM-R-large and pre-trained
on 17 African language. We report the performance in Table 3 for unprocessed data. Except for Algerian Arabic, Mozambican Portuguese, and Twi, all the other languages have the highest F1-score using the AfroXLMRlarge. AfroXLMR-large has the average best score, so we decided to conduct all our further experiments focusing on AfroXLMR-large.

heading: 5.2 Dataset Selection
text: To decide which dataset works best with the given languages, we create monolingual models with only that language’s training data. We test them on the dev set provided by the shared task. • Clean: All the pre-processed training data from
the shared task, without any data augmentation. • Clean + Dictionary-based: All the pre-
processed training data from the shared task are mixed and shuffled with all the Dictionary-based augmented translation datasets. We did not find any bilingual lexicon on Panlex for Mozambican Portuguese. • Clean + MT-based: All the pre-processed training data from the shared task are mixed and shuffled with all the MT-based augmented translation datasets. The model we used for translation does not support Twi, Mozambican Portuguese, Nigerian Pidgin, Moroccan Arabic, and Algerian Arabic. Therefore, we do not use the MT data for these languages. • Clean + Both: All the pre-processed training data from the shared task are mixed and shuffled with all the MT-based augmented translation and Dict-based augmented translation if both datasets are available for a language. Table 4 shows the result on the dev set when
under each of our 4 dataset settings. The first takeaway is that for almost all languages, data augmentation helps. The only exception is Hausa (and Portuguese) which is comparably high-resource. We observe improvements in languages well-supported by the MT system, like Igbo, Swahili, Xitsonga, or Yoruba. Table 4 also shows which dataset type works best for a certain language.
We use that information to compile a combined dataset termed "Best" throughout the rest of the paper. The Best dataset combines the different datasets based on the highest score from different languages. We present the combination of the Best dataset in Table 9 and use it to finetune multilingual models as above.

heading: 5.3 Language ID Information
text: In a multilingual setting, all sentences of all the languages from Task A are combined to create the training set. To assess the impact of language ID information on the model’s performance, we train the model using tagged and untagged versions based on the five datasets described in Section 5.2. Language ID information is not provided in the untagged datasets, which is the same as in the original multilingual dataset. The language ID information is included for the tagged datasets by adding a token denoting the language id at the beginning of each sentence. We evaluate the model on all languages from Task A and their combination. Table 5 presents the performance of our model with tagged and untagged datasets.
We observe that models trained with the tagged dataset generally performed better in the multilingual setting. This indicates that language ID information is helpful for the model to make accurate
predictions when dealing with examples from different languages. This observation further supports our decision to use the phylogeny-based adapter, which incorporates language family and genus information into the training.
In addition, comparing the results in Table 6, we find that the multilingual model outperforms the dedicated monolingual models, suggesting that a multilingual model can more effectively capture sentiment across multiple African languages than one single model. For all future experiments, hence, we only use multilingual models.

heading: 5.4 Phylogeny-based Adaptation
text: In all the experiments up until now, we have finetuned only the task-adapter. We will now adaptertune the AfroXLMR-large by inserting phylogenybased adapter stacks (see Figure 1) inside the Language Model (LM). The intuition behind this is that leveraging the phylogenetic relationships between languages can transfer knowledge and alleviate low-resource scenarios. We will call these adapters family-adapter, genus-adapter, and language-adapter. When adapter-tuning, all the other parameters of the LM will be kept frozen, and only the adapter parameters will be updated through a joint training scheme. Here, we will train the adapters with the Masked Language Modeling task. After the fine-tuning, we will have four adapter components to use in four stack combinations to get our final model. We do not train the task-adapter at this stage but use the previously trained task-adapters.
We benchmark the different combinations as: 1. Task-Adapter 2. Lang-Adapter + Task Adapter
3. Genus-Adapter + Lang-Adapter + TaskAdapter 4. Family-Adapter + Genus-Adapter + LangAdapter + Task-Adapter
Table 7 shows the results with the best out of the above four configurations. We adapter-tune the MLM using the Clean dataset. For task-adapter we use the ones that got the highest scores in Table 5 for different languages. We also adapter-tune the MLM using the Clean + Dictionary-based, Clean + Both, and Best datasets. See Appendix B for all the results.

heading: 6 Results and Discussion
text: Monolingual Performance For Task A, we ensemble the best five models we found for each language. We use majority voting to obtain the final output, resolving ties randomly.
Our system’s test performance in 12 languages is presented in Table 8. Notably, our system demonstrates impressive results on Track 5: Amharic, ranking first on the leaderboard and outperforming the second-place system by a significant margin of 6.2 points. In addition, our system falls only 0.8 points short of the top-performing system in Track 9: Kinyarwanda. In both Track 5 and Track 9, the best data are from Clean and Dictionarybased. While our system does not achieve the top performance for other tracks, it still achieves highly competitive results across all tracks, with a top-10
ranking in 4 out of the 12 tracks. Our success in fine-tuning AfroXLMR-large can be attributed to several factors. First, AfroXLMRlarge is a powerful pre-trained language model specifically designed for African languages. This provides an excellent starting point for fine-tuning specific tasks. Second, we carefully selected the best dataset for each language, which ensures that our system is trained on the most appropriate data for the particular language.
Multilingual Performance For Task B, we can not follow the same procedure as in Task A, as the language ID information is absent; we cannot utilize phylogeny-based adapter-tuning to enable information sharing between similar languages during inference time. We could have used a Lang ID model but chose not to because langID for African languages is untrustworthy. So, we use the best model from Table 5 for this task. Table 8 presents the performance of our system, which falls only 3.82 points short of the top-performing system, achieving a respectable 7th place ranking. While it is not ideal, it still demonstrates the potential of our approach in real-world scenarios where language ID information is often missing.
One possible reason for the suboptimal performance could be the limitations of the available data. Although we carefully select the best dataset from each language, the amount and quality of the data are still limited. In addition, the absence of lan-
guage ID information makes it more challenging to distinguish between closely related languages, which can lead to a higher degree of semantic ambiguity and reduce the accuracy of our system.
Zero-Shot Performance For Task C, we use the same model as Task B. Table 8 also shows the zero-shot performance of our system on Track 17: Tigrinya and Track 18: Oromo. Our system ranks 11th and 13th on the leaderboards, falling short of the top-performing models by 9.34 and 4.36 points, respectively. It highlights the potential of our system for zero-shot cross-lingual transfer learning, although further improvements may be needed to achieve state-of-the-art performance in these languages. For example, we could have used additional unlabeled data in the two languages to continue training the base model as (Muller et al., 2021). We leave these explorations for future work.

heading: 7 Conclusion
text: This paper describes GMU’s SA systems for the AfriSenti SemEval-2023 shared task. We participated in all three sub-tasks: Monolingual, Multilingual, and Zero-Shot. As a starting point for our system, we leverage AfroXLMR-large, a pre-trained multilingual language model specifically trained on African languages and then fine-tuned with original and augmented training data. To further enhance our system, we utilize phylogeny-based adaptertuning, which involves adapting to the target languages using knowledge from related languages in the phylogenetic tree. Multiple models are created and ensembled to obtain the best results. Our system outperforms all other systems in track 5:
Amharic, achieving the highest F1-score with a remarkable 6.2-point higher than the second-best performing system.

heading: Acknowledgements
text: This work was generously supported by NSF awards IIS-2125466 and 2040926 and a Sponsored Research Award from Meta. We thank the GMU Office of Research Computing for granting us access to large computing resources.

heading: B Adapter-tuning
text: Tables 10 and 11 present adapter-tuning results on Dict-based datasets, Best, and Both datasets, respectively.

heading: Future Work
text: We leave these explorations for future work