PICT at StanceEval2024: Stance Detection in Arabic using Ensemble of Large Language Models (2024)

Abstract: This paper outlines our approach to the StanceEval 2024 Arabic Stance Evaluation shared task. The goal of the task was to identify the stance, one out of three (Favor, Against or None) towards tweets based on three topics, namely COVID-19 Vaccine, Digital Transformation and Women Empowerment. Our approach consists of fine-tuning BERT-based models efficiently for both, Single-Task Learning as well as Multi-Task Learning, the details of which are discussed. Finally, an ensemble was implemented on the best-performing models to maximize overall performance. We achieved a macro F1 score of 78.02% in this shared task. Our codebase is available publicly1.

heading: 1 Introduction
text: In recent years, the exponential growth of social media platforms, online news outlets and digital communication has also led to a surge of usergenerated content. Effectively analyzing the opinions and attitudes expressed within such content to understand the perspective of the user regarding the topic is called Stance Detection, which is a critical task in the field of Natural Language Processing (NLP). Previous works like (Küçük and Can, 2020) have focused on multiple definitions of the task. The most widely used one (topic-phrase stance) is the stance of a text - favor, against, neutral is detected concerning a topic like ’Women Empowerment’. Arabic is the fifth most spoken language in the world with over 420 million speakers worldwide. Stance detection has been extensively studied for English and other languages, but the lack of research on this topic for Arabic language exposes the gap in research. Reasons for this are challenges due to unique scripting, morphology and
∗first author, equal contribution 1http://github.com/ishaan-shukla10/
StanceEval2024
dialects, which increase the complexity of texts to be processed by machine learning models. One other reason is the limited data available in Arabic for stance detection. There are extremely limited datasets available like (Baly et al., 2018a) which covers topics like the war in Syria and has very few data points and (Khouja, 2020) which deals with news titles. The StanceEval Shared Task (Alturayeif et al., 2024) at the ArabicNLP conference aims to detect writers’ stances towards three selected topics like COVID-19 Vaccine, Digital Transformation and Women Empowerment on the MAWQIF dataset (Alturayeif et al., 2022). Earlier approaches like (Khouja, 2020) use character-level LSTMs and a multilingual variant of BERT (mBERT) (Devlin et al., 2019). Other approaches include the usage of BERT-based models for zero shot stance detection (Allaway and McKeown, 2020), using gradient boosting classifiers (Baly et al., 2018b) or using end-to-end memory networks (Mohtarami et al., 2018). We built upon the approach used by (Alhindi et al., 2021) using language specific BERT based models like CamelBERT (Inoue et al., 2021), MARBERT (Abdul-Mageed et al., 2021) and AraBERT (Antoun et al., 2021) and some of their variants. We ranked 7th in this shared task, with a net macro-F1 of 78.02%. Individual macro-F1 scores for categories of ’Women empowerment’, ’Covid Vaccine’ and ’Digital Transformation’ were 75.52%, 79.85% and 78.69% respectively. The subsequent sections describe our approach used for the task. We also analyze the performance of our pipeline and present ablations after rigorous experiments.

heading: 2 Data
text: The dataset contains 3502 examples belonging to 3 distinct topics - COVID-19 Vaccine, Women Empowerment and Digital Transformation. The distribution of examples across these topics is evenly
837
distributed. The possible values of stance are - Favor, Against and None. There is a slight class imbalance in this category. The class ’Favor’ has the highest number of examples (2148) whereas the class ’None’ has the least (333). The data distribution is illustrated in Figure 1. The dataset was a multi-label dataset with each data point having an annotation for stance, sarcasm and sentiment. We used a train-validation split of 80:20 on the given dataset. The training set had 2805 examples and the validation set had 697 examples. The test dataset contained 619 examples on which the system was finally evaluated. We observed a significant imbalance in the distribution of class labels. To mitigate that we used methods like weighted random sampling, however, there was no performance improvement even after using such methods. Hence, we did not use any data sampling methods while training the final system.

heading: 3 System Description
text: The problem of Stance Detection was a multi-class classification problem, in which the stance of given tweets was to be evaluated from a total of 3 classes. Both, Single-Task Learning (STL) and Multi-Task Learning (MTL) models were implemented in the system. We decided to use Multi-Task Learning because it can improve performance on the primary task while aiding in generalization due to the inductive bias introduced by the additional tasks (Caruana, 1997). To aid in MTL we used a focal loss in order to provide the right gradient signals to aid the primary task. The equation of the loss used is shown in Equation 1. The sum of the individual task-wise losses was divided by 6 for normalization. The primary task was stance detection and the auxiliary tasks were sarcasm detection and sentiment detection. It can be clearly observed from Table 2 and 3 that the MTL models have outperformed the STL models.
Loss = (sent+ 4 ∗ stance+ sarc)
6 (1)
Then, depending on their performance on testing data, the best models were selected without any discrimination between STL and MTL models. We used models like MARBERT, MARBERTv2, AraBERT, CamelBERT, QARiB (Abdelali et al., 2021) their variants as the backbones in our system in the various experiments conducted. The pretrained checkpoints for all the models were used through the HuggingFace (Wolf et al., 2020). We used basic pre-processing like the removal of links, mentions and English words. Also, the tweets contained emojis. To effectively deal with them we decided to replace each emoji with its description in Arabic. The models were then fine-tuned by connecting a linear layer to the pooler output was to the pooler output. Hyperparameter tuning was carried out to obtain the best possible model performance, by considering learning rate, weight decay, number of epochs, batch size and context length. For the top 3 models, the best set of hyperparameters we found are mentioned in Table 1. CrossEntropy loss was used for the individual tasks along with the Adam optimizer. The scoring metric of the task was macro F1 score for the ’Favor’ and ’Against’ stances only. The top models were selected based on the criterion of epoch-wise validation F1 score. We observed that an ensemble performed better than individual models. To account for all possible
combinations from the top 5 MTL and STL models we tested all possible combinations of the models and accordingly chose the best system. Hard voting was used in the ensemble system. If there was no consensus in the system then the label predicted by the model with the highest validation F1-score was used as the final prediction.

heading: 4 Results
text: A total of 13 models were used for experimentation. The models were the variants of the models mentioned above post-trained or fine-tuned on domain-specific data like tweets or dialect identification. The results of the top 5 models are depicted in Table 2 and Table 3 when trained as Single-Task Learning and Multi-Task learning models respectively. We observe clearly that models using the MTL paradigm show better performance as compared to STL models. However, in some cases, we observed that the MTL models overfit the training data as compared to their STL counterparts like CamelBERT. The bold entries from the tables show the models that were in the final system. The final ensemble system consisted of an MTL MARBERT, STL AraBERT and STL CamelBERT. The final results of our system can be seen in Table 4. Out of the 3 topics the system performed the best on the topic - COVID-19 Vaccine.

heading: 5 Discussion
text: In this paper, we designed a system to perform stance detection on tweets belonging to various topics. To develop this system we performed rigorous experiments. The results of our system on
the test dataset can be seen in Table 4. To further analyze our system we have plotted the confusion matrices of the system according to the various topics. One of the main points we observed is that the label ’None’ consistently gave the worst performance across all the 3 topics. We believe this is due to the fact there is a lack of any distinct words or semantic structure in the tweets belonging to this category unlike the categories of ’Favor’ and ’Against’. Another observation was that the models overfit when trained in an MTL setup as compared to the STL setup in some cases. We believe this can be mitigated by deciding the proper task order and giving the right importance to each task. The disparity in performance across topics despite of equal distribution of examples can be attributed to the fact that during pre-training the model encountered some topics more frequently and hence they have robust representations as compared to some other topics. One of the major drawbacks of our system is that the system tends to overfit on the data provided. To mitigate this we propose to use methods like finding the proper ordering of tasks so model can build robust representations and generalise better instead of overfitting and also trying to use more data. We can try to find additional data or try to use synthetic data. We can also try to improve ensembling approaches. One other drawback that needs to be addressed is that the methods we use require large amounts of data. In low-resource settings alternatives like synthetic data or efficient architectures need to be researched more so the task can be performed across multiple languages.

heading: 6 Conclusion
text: This paper aims to describe our approach for the StanceEval 2024 Shared Task which involved classification of tweets towards 3 topics. We conducted experiments with multiple BERT-based models like AraBERT, MarBERT, CamelBERT and QARiB. We explored both single-task learning (STL) and multi-task learning (MTL) setups, finding that the MTL approach, which incorporated auxiliary tasks of sarcasm detection and sentiment analysis, outperformed the STL models. We also show that an ensemble of the best performing STL and MTL models achieved a strong macro F1-score of 78.02% on the evaluation dataset, with the models performing best on the COVID-19 Vaccine topic, followed by Digital Transformation and Women Empowerment. We also show that Multi-task Learning benefits the models and its drawbacks. We foresee several future directions for the work done. One direction
can be to develop and use more sophisticated methods for ensembling and multi-task learning. Another direction is the development of systems that can achieve similar performance in low-resource settings either using synthetic data or by using efficient architectures.

heading: Future Work
text: We foresee several future directions for the work done. One direction
can be to develop and use more sophisticated methods for ensembling and multi-task learning. Another direction is the development of systems that can achieve similar performance in low-resource settings either using synthetic data or by using efficient architectures