GunadarmaXBRIN at SemEval-2023 Task 12: Utilization of SVM and AfriBERTa for Monolingual, Multilingual, and Zero-shot Sentiment Analysis in African Languages
Year: 2023

Abstract:
This paper describes our participation in Task 12: AfriSenti-SemEval 2023, i.e., Track 12 of subtask A, Track 16 of subtask B, and Track 18 of subtask C. To deal with these three tracks, we utilize Support Vector Machine (SVM) + One vs Rest, SVM + One vs Rest with SMOTE, and AfriBERTa-large models. In particular, our SVM + One vs Rest with SMOTE model could obtain the highest weighted F1-Score for Tracks 16 and 18 in the evaluation phase, that is, 65.14% and 33.49%, respectively. Meanwhile, our SVM + One vs Rest model could perform better than other models for Track 12 in the evaluation phase.

heading: 1 Introduction
text: Sentiment Analysis (SA) is one of the Natural Language Processing (NLP) tasks. SA is used to recognize and classify the author’s opinion or experience of an object, such as a specific topic, product, service, organization, social events, politics, and economic (Nahili et al., 2021; Yang, 2021; Mehmood et al., 2019). The sentiment of the opinion could be classified as positive, neutral, or negative. Implementation of SA is employed in various sectors, such as business, healthcare, and education (Nandwani and Verma, 2021). However, SA research mostly focused on high-resource language, i.e., English and Chinese (Nasim and Ghani, 2020). In other words, low-resource language is still not yet addressed well (Muhammad et al., 2022), such as African language (Muhammad et al., 2023b), which is the focus of task 12.
Task 12 AfriSenti-SemEval1 (Muhammad et al., 2023b) aims to do sentiment analysis on African
1https://codalab.lisn.upsaclay.fr/ competitions/7320
datasets collected from Twitter. This task 12 consists of three subtasks, i.e., monolingual sentiment classification (subtask A), multilingual sentiment classification (subtask B), and zero-shot sentiment classification (subtask C). In total, there are 14 datasets provided in the task 12, namely, Hausa, Yoruba, Igbo, Nigerian Pidgin, Amharic, Algerian Arabic, Moroccan Arabic/Darija, Swahili, Kinyarwanda, Twi, Mozambican Portuguese, Xitsonga (Mozambique Dialect), Tigrinya, and Oromo.
In this AfriSenti-SemEval task, we participate in Track 12: Xitsonga (Mozambique Dialect) of subtask A, Track 16: 12 languages in subtask A of subtask B, and Track 18: Zero-Shot on Oromo of subtask C. To deal with each subtask, we utilize three models, namely, Support Vector Machine (SVM) + One vs Rest (Model 1), SVM + One Vs Rest with SMOTE (Model 2), and AfriBERTalarge (Ogueji et al., 2021) (Model 3). The highestobtained F1-Scores of our participation are 50.56%, 65.14%, and 33.49% for Track 12, Track 16, and Track 18, respectively. The code used in this work is available on GitHub2.

heading: 2 Background

heading: 2.1 Dataset Description
text: The organizer of the AfriSenti-SemEval task provides training, development, and evaluation datasets in subtasks A and B. Note that the training dataset in subtask B is the combination of all training datasets in subtask A. On the other hand, the organizer prepares only development and evaluation datasets in subtask C. Thus, participants can utilize any or all training datasets available in
2https://github.com/aquemos/ GunadarmaXBRIN-at-SemEval-2023-task-12
869
subtask A as their training dataset for subtask C. The provided training datasets consist of ID, tweet, and label. Table 1 shows the data distribution of each class for each track in subtask A. Meanwhile, Table 2 exhibits the number of development and evaluation datasets of each subtask that we follow.

heading: 2.2 Related Work
text: Utilization of fine-tuning method on several pretrained models obtained a good result for SA in African languages. For example, Fourati et al. (2021) used mBERT and mBERT with Convolutional Neural Network (CNN) models to perform SA on Tunizi dataset. They obtained a macro F1-score of 0.558 and 0.580 when using the mBERT model and mBERT with CNN one, respectively. The mBERT model was also employed by Martin et al. (2021) to do SA on the Swahili dataset, in which they obtained an F1-score of 0.91 and 0.81 on negative and positive classes, respectively. Another fine-tuning method is AfriBERTa used in Muhammad et al. (2022) for monolingual, multilingual, and zero-shot SA on Hausa, Igbo, Yoruba, and Nigerian Pigdin datasets. In particular, the work of Muhammad et al. (2022) obtained a weighted F1-score around 16-45% for monolingual SA, weighted F1-score of 78.3% for multilingual SA, and an average F1-score of 49.9% for zeroshot SA. Since the focus of the AfriSenti-SemEval task is similar to that of Muhammad et al. (2022), i.e., monolingual, multilingual, and zero-shot SA, we consider employing the AfriBERTa-large in this work. However, the number of African languages in our work is much more than that of in the Muhammad et al.’s (2022) one, viz., Hausa, Yoruba, Igbo, Nigerian Pidgin, Amharic, Algerian Arabic, Moroccan Arabic/Darija, Swahili, Kinyarwanda, Twi, Mozambican Portuguese, Xitsonga, and Oromo.
SVM is one of the supervised machine learning algorithms mostly used in SA (Taj and Girisha, 2021). For example, Demircan et al. (2021) used SVM with undersampling for SA in an imbalanced Turkey dataset, i.e., monolingual. They obtained F1-scores of 0.8, 0.75, and 0.92 in negative, neutral, and positive classes, respectively. On the other hand, using SVM also worked well in the multilingual imbalanced dataset consisting of German, French, and English (Pustulka-Hunt et al., 2018). In particular, Pustulka-Hunt et al. (2018) obtained an F1-score of 0.913 using 10-fold cross-validation
in their work. Based on the good result of Demircan et al. (2021) and Pustulka-Hunt et al. (2018), we are motivated to employ SVM for monolingual and multilingual SA in this work. Since SVM is a binary linear classifier (Khan et al., 2022) and the dataset in this AfriSenti-SemEval task is multiclass, we employ One Vs Rest strategy on SVM. Moreover, training datasets in subtask A are imbalanced. Thus, classifiers could not work well in such unequal distributions (Flores et al., 2018). Therefore, we utilize an oversampling technique, i.e., SMOTE, on the minority class to the training datasets in subtask A.

heading: 3 Method
text: Details of preprocessing, models, and evaluation employed in this work are as follows:

heading: 3.1 Dataset Preprocessing
text: Diacritic plays an important role in the African language because the diacritic’s position can affect the meaning of words (Nwankwo, 2021). For example, Àwon omó fo abó (The children washed the dishes) and Àwon omó fó abó (The children broke the dishes) have different meanings in Yoruba (Lanfrica, 2022). For this reason, we do not remove punctuation marks in our preprocessing step. Specifically, the preprocessing step is deleting emojis, mentions, URLs, hashtags, and extra whitespaces. It also includes converting tweets into lowercase. This preprocessing step is applied to the training, development, and evaluation datasets.

heading: 3.2 Models

heading: 3.2.1 Baseline
text: Fine-tuning on pre-trained models is conducted as baseline models by the organizer of the AfriSenti-SemEval task. The pre-trained models used in Muhammad et al. (2023a) are AfriBERTa-large, XLM-R-base, AfroXLMR-base, mDeBERTaV3-base, XLM-T-base, XLM-R-large, and AfroXLMR-large. The baseline models are divided into three: (1) the pre-trained model is trained with training data from a target language for the monolingual baseline, (2) a multilingual dataset of all 12 languages is used as the training dataset for the multilingual baseline, and (3) training dataset from any of the 12 languages or multilingual dataset is used as a training dataset for zero-shot baseline (Muhammad et al., 2023a).

heading: 3.2.2 AfriBERTa-large
text: AfriBERTa-large3 is one of the pre-trained multilingual models which covers 11 African languages, viz., Oromo, Amharic, Gahuza, Hausa, Igbo, Nigerian Pidgin, Somali, Swahili, Tigrinya, and Yoruba (Ogueji et al., 2021). Thus, we use Hugging Face Transformers to train the AfriBERTa-large tokenizer and fine-tune the AfriBERTa-large model to deal with this AfriSenti-Semeval task.
Since the AfriBERTa-large tokenizer still does not support Algerian Arabic, Moroccan Arabic, Twi, Mozambican Portuguese, Xitsonga, and Oromo languages, we train the AfriBERTa-large tokenizer with a corpus collected from several sources, i.e., Leipzig4, Sadilar5, and GitHub6 (see Table 3). We also train the AfriBERTa-large tokenizer with Kinyarwanda corpus because it just supports a code-mixed language of Kinyarwanda and Kirundi (Gahuza) (Ogueji et al., 2021). Note that we can not find a corpus for Twi and Mozambican Portuguese languages. However, according to University of Cambridge Language Centre Resources; Rutgers, the Akan language belongs to the Twi language. Therefore, we use the Akan corpus to train the Twi language. Meanwhile, Portuguese corpus is used to train Mozambican Portuguese since Portuguese is the national language in Mozambique
3https://huggingface.co/castorini/afriberta_ large
4https://wortschatz.uni-leipzig.de/en/ /download
5https://repo.sadilar.org/handle/20.500.12185/ 364
6https://github.com/asmelashteka/HornMT/blob/ main/data/orm.txt
(UNICEF). In particular, we fine-tune the preprocessed training dataset of Xitsonga to participate in Track 12: Xitsonga (Mozambique Dialect) of subtask A, which is monolingual SA. On the other hand, we use all preprocessed training data of subtask A to fine-tune a model to deal with Track 16 of subtask B and Track 18 of subtask C, which are multilingual and zero-shot SA, respectively. Note that 90% of the training dataset is employed to train the model, while 10% of the training dataset is used for evaluation and optimization during the training process. The detail of training model hyperparameters is in Appendix A.1.

heading: 3.2.3 SVM + One Vs Rest
text: According to Murphy (2018), SVM can deal with multiclass classification problems, that is, by utilizing the One Vs Rest (One Vs All) approach. Thus, we use SVM7 with One Vs Rest8 from Scikit Learn in this work.
To deal with Track 12 of subtask A, we train the SVM + One Vs Rest model with text vectorization of preprocessed Xitsonga training dataset. Meanwhile, SVM + One Vs Rest model is trained with text vectorization of preprocessed all training datasets in subtask A to deal with Track 16 of subtask B and Track 18 of subtask C. Text vectoriza-
7https://scikit-learn.org/stable/modules/ generated/sklearn.svm.SVC.html
8https://scikit-learn.org/stable/modules/ generated/sklearn.multiclass.OneVsRestClassifier. html
tion process uses Term Frequency - Inverse Document Frequency (TF-IDF)9 from Scikit Learn with default parameters. TF-IDF is used based on the successful work of Ahuja et al. (2019) and Dharma and Saragih (2022) that employing TF-IDF with SVM generated good results for sentiment analysis.

heading: 3.2.4 SVM + One Vs Rest with SMOTE
text: Synthetic Minority Oversampling (SMOTE) is one of the oversampling techniques that synthesizes a new example with the existing one (Mutanov et al., 2021). In this work, we utilize SMOTE from Imbalanced Library10. This oversampling is applied to all training datasets vectorized with TF-IDF as in 3.2.3. In particular, the oversampled training dataset of Xitsonga is used to train the SVM + One Vs Rest model to deal with Track 12 of subtask A. On the other hand, SVM + One Vs Rest model is trained with all oversampled training datasets in subtask A for Track 16 of subtask B and Track 18 of subtask C.

heading: 3.3 Evaluation
text: The submission of each subtask is evaluated with weighted precision, weighted recall, weighted F1Score, macro precision, macro recall, and macro F1-Score. The team ranking is ordered by the weighted F1-Score of each team’s last submission.

heading: 4 Results
text: Table 4 shows the obtained results of our submissions in the AfriSenti-SemEval task. Specifically, the results are the obtained weighted F1Score of our participation in Tracks 12, 16, and 18. The results detail can be found in https: //afrisenti-semeval.github.io/results/.
Based on the results in Table 4, we note that the best model in the development phase could not be the best model in the evaluation one. For example, our AfriBERTa-large could outperform our SVM + One Vs Rest (with or without SMOTE) in the development phase of Track 18. However, the AfriBERTa-large could not perform better than the SVM + One Vs Rest (with or without SMOTE) in the evaluation phase of Track 18, even worst. We try analyzing this distinction by observing the con-
9https://scikit-learn.org/stable/modules/ generated/sklearn.feature_extraction.text. TfidfVectorizer.html
10https://imbalanced-learn.org/stable/ references/generated/imblearn.over_sampling. SMOTE.html
fusion matrix11 of our models using Scikit Learn12. We found that the AfriBERT-large could predict positive instances better than SVM + One Vs Rest (with or without SMOTE) in the development phase of Track 18. On the other hand, the SVM + One Vs Rest (with or without SMOTE) could classify negative and neutral instances better than the AfriBERTlarge in the development phase of Track 18. Note that the number of positive instances in the development phase is larger than that of negative and neutral ones, while the number of positive instances in the evaluation phase is smaller than that of negative and neutral ones. This circumstance might be the reason that the AfriBERTa-large could not perform well in the evaluation phase as in the development one. Moreover, other different characteristics between the development dataset and the evaluation one might affect this inconsistency performance. However, such a presumption should be investigated further in the future.
Table 4 also shows that our AfriBERTa-large could not perform as well as the baseline’s AfriBERTa-large in the evaluation phase, even worse. The underperforming of our AfriBERTalarge might be due to the use of additional corpora and the choice of fine-tuning hyperparameters when training the AfriBERTa-large. Nevertheless, it would be better to analyze further this conjecture in the next work.
Overall, our SVM + One Vs Rest (with or without SMOTE) could perform better than our AfriBERTa-large in the evaluation phase of tracks 12, 16, and 18. Sadly, our best-obtained results are still unsatisfactory, even far below the results obtained by the highest-ranking team in tracks 12, 16, and 18. This shortcoming might be due to the dataset characteristic. For example, we found that several Hausa tweets with positive sentiments in the evaluation dataset of Track 16 were misclassified by our models as negative ones. To analyze this misclassification, we extracted the positive and negative words in the tweets using the lexicon sentiment provided by AfriSenti13. The extraction result shows that several positive tweets are containing negative words, but positive words do not exist (see Table 5). Moreover, code-mixed between Hausa
11The confusion matrix is available in Appendix A.2 12https://scikit-learn.org/stable/modules/
generated/sklearn.metrics.confusion_matrix.html 13https://github.com/afrisenti-semeval/ afrisent-semeval-2023/tree/main/sentiment_ lexicon
and English appears in the tweets. Thus, such circumstances might affect the performance of our models. Nonetheless, further investigation should be performed on positive and negative phrases to analyze the classification error.
However, our obtained results might be comparable to the baseline ones since we believe the difference is not significant. This condition might indicate that this task is complicated and challenging. Although each language in the multilingual dataset has different language families, a particular
language with particular language families might have a similarity to another one from different language families. For example, the Hausa language is classified as an Afro-Asiatic language family, while the Yoruba language is classified as a Niger-Kongo language family. Although these two languages have different language families, we found in the dataset that 177 words from the two languages are similar. In fact, these similar words could have different sentiments (see Table 6).

heading: 5 Conclusion
text: We utilized SVM + One Vs Rest, SVM + One Vs Rest with SMOTE, and AfriBERTa-large to deal with Tracks 12, 16, and 18 of the AfriSentiSemEval task. Our highest-obtained weighted F1Scores were 50.56%, 65.14%, and 33.49% for Tracks 12, 16, and 18, respectively. Although our obtained results indicated that our models could not perform as well as or better than the baseline models, the SVM + One Vs Rest (with or without
SMOTE) results could be compared with those of the baseline ones. For further work, it will be better to explore hyperparameter tuning on the pre-trained model to improve the model performance. In addition, there are several tweets written in Latin script. Thus, it may be beneficial to translate Arabic into Latin and vice versa when dealing with this task in the future.

heading: A Appendix
text: A.1 Training Hyperparameters
A.2 Confusion Matrix

heading: Future Work
text: Thus, it may be beneficial to translate Arabic into Latin and vice versa when dealing with this task in the future