StanceCrafters at StanceEval2024: Multi-task Stance Detection using BERT Ensemble with Attention Based Aggregation (2024)

Abstract: Stance detection is a key NLP problem that classifies a writer’s viewpoint on a topic based on their writing. This paper outlines our approach for Stance Detection in Arabic Language Shared Task (StanceEval2024), focusing on attitudes towards the COVID-19 vaccine, digital transformation, and women’s empowerment. The proposed model uses parallel multi-task learning with two fine-tuned BERT-based models combined via an attention module. Results indicate this ensemble outperforms a single BERT model, demonstrating the benefits of using BERT architectures trained on diverse datasets. Specifically, Arabert-Twitterv2, trained on tweets, and Camel-Lab, trained on Modern Standard Arabic (MSA), Dialectal Arabic (DA), and Classical Arabic (CA), allowed us to leverage diverse Arabic dialects and styles. The code has been made open-source on the link https://github.com/gufranSabri/StanceDetectionMultiTaskLearning.

heading: 1 Introduction
text: Stance detection or identification is a growing research topic in the fields of Natural Language Processing (NLP), social media analysis, and information retrieval (IR). It seeks to classify a writer’s viewpoint on a certain topic based on their writing towards a target (e.g., notion, idea, event) (AlDayel and Magdy, 2021). It is known to have several applications, including predicting election/referendum results, retrieving information, and classifying rumors (Küçük and Can, 2021). Stance detection is known to be closely related to sentiment analysis, with the target classes Favor, Against, and None (Feldman, 2013). It is also related to other tasks, such as sarcasm detection and irony detection. Despite its relevance, it has gotten little attention in the Arabic language.
Data-driven neural networks have demonstrated outstanding performance for a wide variety of tasks.
Traditionally, neural networks are trained separately for each task. However, neural networks frequently require extensive labeled training samples to achieve superior results (Ahmed et al., 2023). Given the limited data for Arabic stance identification and the availability of related tasks, it is necessary to employ novel techniques to construct robust models.
Multi-task learning (MTL) is an approach that handles numerous tasks concurrently and can leverage knowledge from related tasks to address data shortages (Thung and Wee, 2018). It assumes that all learning processes, or at least a portion of them, are related to one another. Learning from multiple common tasks simultaneously allows models to collect generic information beyond task-specific characteristics (Zhang and Yang, 2021). MTL has been used in the field of Arabic stance identification with various weighting strategies, yielding better results than training a model using only the stance detection task (Alturayeif et al., 2023).
This paper details our work on the Stance Detection in Arabic Language Shared Task (StanceEval2024) (Alturayeif et al., 2024). The StanceEval2024 shared task identifies writers’ attitudes (Favour, Against, or None) on COVID-19 vaccination, digital transformation, and women empowerment. The key challenge in this task is imbalance of importance between the tasks. Therefore, we apply parallel MTL combining sentiment, sarcasm, and stance tasks to share knowledge across multiple tasks. Moreover, we utilize diverse weighting loss techniques to give more importance to the primary task. The remaining sections of the paper are organized as follows: Section 2 describes the StanceEval2024 dataset, Section 3 describes the methodology, Section 4 demonstrates the experimental results, Section 5 discusses the key findings, and Section 6 concludes the work. The code has been made open-source and available on GitHub on the link
811
https://github.com/gufranSabri/StanceDetectionMultiTaskLearning.

heading: 2 Data
text: The MAWQIF dataset is a multi-label Arabic dataset designed for stance identification (Alturayeif et al., 2022). It includes 4121 tweets around three topics: COVID-19 vaccine, digital transformation, and women’s empowerment. Each tweet is annotated with stance, sentiment, and sarcasm. The dataset is separated into two sets: training and testing. Training accounts for 85% of the data, while testing accounts for 15%. Table 1 provides a summary of the number of tweets belonging to each category.
In this study, we only considered the stance labels: Against and Favour. The dataset is preprocessed using the ArabertPreprocessor using the bert-base-arabertv02-twitter model. The dataset was further pre-processed by removing all the hashtags, URLs, mentions, punctuation, and repeating characters, as well as applying normalization.

heading: 3 System
text: This study explored using multiple BERT models for stance detection in Arabic text. We began with a single BERT model in the shared layers and then, inspired by the work in (Dang et al., 2020) and (Ganaie et al., 2022), we experimented with combining two models (Arabert, Arabert-twitter, Camel-bert) to leverage their diverse training data. We evaluated different weighting and aggregation techniques (averaging, attention) to find the optimal combination (Figure 1). Finally, we explored various loss weighting methods (static, relative) to further improve performance.
During training, the following configurations were used: 20 epochs, learning rate of 2e-5, weight decay of 1e-5, dropout of 0.1, batch size of 4, seed value of 42, and max token length of 128. The
models were validated based on the validation F1score.

heading: 3.1 MTL
text: In MTL, parameter sharing techniques determine how models leverage information across tasks. This study employs hard parameter sharing, where the initial layers of a model are shared by all tasks, promoting the learning of generalizable features. Each task then has its own output layer to capture task-specific information. Additionally, we utilize joint training, a technique where all tasks are trained simultaneously. This allows the model to learn shared representations that benefit all tasks while also enabling each task to adapt to its specific requirements. This combination of hard parameter sharing and joint training fosters knowledge transfer and improves the model’s overall performance (Zhang and Yang, 2021).

heading: 3.2 Aggregation techniques
text: Ensemble aggregation techniques are crucial in aggregating predictions from several models to improve a system’s overall performance and robustness. In this study, we experimented with mean aggregation and attention-based aggregation.

heading: 3.2.1 Mean Aggregation
text: The mean aggregation technique is the simplest ensemble aggregation technique. It operates by averaging multiple models’ predictions to produce the final prediction. Although it is simple and straightforward, this technique assumes that all models are equally useful, which is not always the case (Briskilal and Subalalitha, 2022).

heading: 3.2.2 Attention based aggregation
text: The attention aggregation technique overcomes the limitation of the averaging technique by assigning different weights to each model’s predictions based on their relevance and considering the context of an input. It can dynamically choose the most suitable
models for each input, resulting in more accurate predictions (Mnassri et al., 2022).

heading: 3.3 Task weighting
text: In the MTL paradigm, the total loss for all tasks can be defined in various ways, including combining all loss terms with manual weights per loss (static) or using a weighted sum with learnable weights (dynamic) (Gong et al., 2019).

heading: 3.3.1 Static weighted sum
text: The static weighted sum calculates the total loss for each sentiment, sarcasm, and stance task while multiplying each loss by a given weight. The static weighted sum is formulated as shown in Equation 1.
L = n∑
i=1
wixi (1)
Where n refers to the number of tasks (stance, sentiment, and sarcasm), xi refers to a specific loss value, and wi refers to the weight multiplied by the corresponding loss. In this study, w1 was set to 0.6, w2 was set to 0.3, and w3 was set to 0.1 (Alturayeif et al., 2023).

heading: 3.3.2 Relative weighted sum
text: The relative weighted sum is a dynamic weighting based on the intuition that tasks with higher losses should receive more attention. This method dynamically adjusts weights during optimization to allocate a larger weight to the stance loss, as shown in Equation ??
L = ωxstance + ω
2 xsentiment +
ω 3 xsarcasm (2)
Where ω is the learnable weight.

heading: 4 Results
text: In this section, the results of the developed models are reported during the development and test phases.

heading: 4.1 Development phase results
text: Table 2 summarizes the validation F1-score of fine-tuning Arabert-Twitter, Arabert, and CamelBert as single models and ensemble combinations. AraBERT-twitter has the highest F1-score when used alone, with a score of 89.8937, while CAMeLLab has the lowest F1-score among the single models, with a score of 84.9208. Interestingly, combining the best performing and weakest performing model resulted in the highest F1-score of 90.2293. Accordingly, we can conclude that combined models yield higher validation F1-score than solo models, likely because the combination of models can capture more diverse features and generalize better.
Table 3 indicates that using attention instead of averaging slightly improves the Arabert-twitter and Camel-bert ensemble. The attention ensemble method, when paired with a static weighting mechanism, has the best validation F1-score (91.3668). It demonstrates that employing a static weighting technique can be very effective in such a scenario.

heading: 4.2 Test phase results
text: Table 4 shows the Macro F1 score of the submitted model (Arabert-Twitter + CamelBert aggregated using attention mechanism with static weighting). The suggested model excelled at the Woman Empowerment and Digital Transformation tasks, but struggled with the COVID vaccination. Our model significantly surpassed the results obtained in the Digital Transformation task when compared to the Mawqif dataset results. Nonetheless, a minor decline is observed in the results related to COVID Vaccination and Woman Empowerment.

heading: 5 Discussion
text: Our suggested model exceeded the findings from the Mawqif dataset by 2.79%. We discovered that pairing Arabert-Twitter with CamelBert yielded
the greatest results. Arabert’s success can be attributed to the Mawqif dataset, which was acquired via Twitter. Arabert-twitter was trained on tweets that were not Modern Standard Arabic (MSA), whereas CamelBert was trained on MSA, dialectal Arabic (DA), and classical Arabic (CA), consequently combining them enhanced the results. We also observed that aggregating the models using the attention mechanism performed better than the mean mechanism. The attention mechanism can outperform the mean mechanism in MTL since it dynamically weights model contributions based on their relevance. Moreover, we found that using static weighting leads to attaining the best performance. We assume that the reason behind this is the low confidence present in the dataset. Static weighting is less sensitive to noise and outliers in the loss values. the proposed framework performed relatively better on the Women’s Empowerment target and relatively worse on the COVID Vaccine target. We believe the cause is that we fine-tuned the models using Against and Favour labels. Woman Empowerment has the fewest None labels, whereas COVID Vaccine has most None labels.
Encouraged by the success of MTL with attention-based combination, a future direction can be in regards to exploring Graph Neural Networks (GNNs). This would involve constructing a graph where each node represents a BERT model’s output, and then combining them using techniques like graph convolution, similar to work in (Liu et al., 2018), (Du and Wang, 2022). Moreover, incorpo-
rating more weighting techniques, such as Hierarchical and Uncertainty weighting could be a direction to improve the results. Employing uncertainty sampling is another way to improve generalization with less data as suggested in (Pilault et al., 2020).

heading: 6 Conclusion
text: This study explored an MTL approach for stance detection in Arabic text, focusing on attitudes towards COVID-19 vaccines, digital transformation, and women’s empowerment. The proposed model architecture included a shared task layer comprising Arabert-Twitter and CamelBert, which were combined via an attention mechanism, followed by a task-specific layer for each task. This ensemble approach, combined with static loss weighting, achieved a strong validation F1-score of 91.3668, outperforming architectures with a single BERT model in the shared layers. The results suggest that leveraging diverse training data through multitask learning with attention-based combinations can improve the model’s ability to capture various features and generalize better. On the test set, while the model excelled at the Women’s Empowerment task (85.06 F1 score), it highlights the need for further development to improve performance on topics like COVID-19 vaccination (79.84 F1 score). Overall, the model achieved a promising macro F1 score of 81.68 on the blind test set.

heading: Acknowledgments
text: We are deeply grateful to Dr. Hamzah Luqman for his support and guidance. We also thank KFUPM for providing essential resources, including GPUs.

heading: Future Work
text: Encouraged by the success of MTL with attention-based combination, a future direction can be in regards to exploring Graph Neural Networks (GNNs). This would involve constructing a graph where each node represents a BERT model’s output, and then combining them using techniques like graph convolution, similar to work in (Liu et al. , 2018), (Du and Wang, 2022). Moreover, incorpo-
rating more weighting techniques, such as Hierarchical and Uncertainty weighting could be a direction to improve the results. Employing uncertainty sampling is another way to improve generalization with less data as suggested in (Pilault et al. , 2020)