StanceEval 2024: The First Arabic Stance Detection Shared Task (2024)

Abstract: Recently, there has been a growing interest in analyzing user-generated text to understand opinions expressed on social media. In NLP, this task is known as stance detection, where the goal is to predict whether the writer is in favor, against, or has no opinion on a given topic. Stance detection is crucial for applications such as sentiment analysis, opinion mining, and social media monitoring, as it helps in capturing the nuanced perspectives of users on various subjects. As part of the ArabicNLP 2024 program, we organized the first shared task on Arabic Stance Detection, StanceEval 2024. This initiative aimed to foster advancements in stance detection for the Arabic language, a relatively underrepresented area in Arabic NLP research. This overview paper provides a detailed description of the shared task, covering the dataset, the methodologies used by various teams, and a summary of the results from all participants. We received 28 unique team registrations, and during the testing phase, 16 teams submitted valid entries. The highest classification F-score obtained was 84.38.

heading: 1 Introduction
text: The rapid expansion of social media platforms, online news sources, and digital communication has significantly increased user-generated content in recent years. This surge in online interactions has created a growing need for automated tools and techniques to analyze the opinions and attitudes expressed in these vast text streams effectively. Stance detection, a crucial task in Natural Language Processing (NLP), aims to identify a writer’s position or perspective on a specific topic or entity by analyzing their written text and social media activity, including preferences and connections. It has various applications in marketing, politics, and social media analysis.
Stance detection is closely related to sentiment analysis. While sentiment analysis focuses on
identifying the explicit sentiment polarity of a text—categorized as Positive, Negative, or Neutral—stance detection classifies the viewpoint of a text towards a specific target as Favor, Against, or None. The target in stance detection is often abstract, like ideological topics, and may not be explicitly mentioned in the text, whereas sentiment analysis usually deals with non-ideological subjects. Additionally, the alignment between sentiment and stance in a text can vary; a text may have a positive sentiment while opposing the target, or vice versa.
In this paper, we summarize the results of the StanceEval 20241 shared task where participants were asked to develop models that detect writers’ stances (Favor, Against, or None) towards three topics: COVID-19 vaccine, digital transformation, and women empowerment. The MAWQIF dataset, comprising Arabic tweets collected from Twitter, was used for training and evaluating the proposed models. Each tweet in this dataset has been manually annotated with three labels: stance, sentiment, and sarcasm.

heading: 2 Literature Review
text: Stance detection involves classifying a writer’s stance on a given subject (target) based on written text (input). The output is typically categorized into one of the following: Favor, Against, None (Alturayeif et al., 2023b). The majority of stance detection research has concentrated on supervised machine learning models developed using manual human annotations (Alturayeif et al., 2023b). This includes approaches using Support Vector Machines (SVM) (Lai et al., 2019; Gómez-Suta et al., 2023), Convolutional Neural Networks (Zhou et al., 2019; Alkhalifa et al., 2021), and Recurrent Neural Networks (Yang et al., 2020). Unsupervised and weakly supervised learning approaches have also
1https://sites.google.com/view/stanceeval/home
774
been used for stance detection, including the use of Graph Neural Networks, inferring user’s stance based on their past tweets and the tweets of their interaction graph neighbors (Zhang et al., 2023) and label propagation on user interaction networks (Weber et al., 2013). While unsupervised and weakly supervised techniques may not achieve the same level of accuracy as supervised learning methods, they address the challenge of obtaining a sufficient amount of labeled data (Alturayeif et al., 2023b).
Transfer learning is another powerful paradigm utilized for stance detection. It leverages a pretrained model, initially trained on a broad, general task, and adapts it for the specific target task of stance detection. This approach significantly reduces the labeled data required for the target task, enhancing efficiency and effectiveness in model training (Alturayeif et al., 2023b). Transductive transfer learning was mainly used for domain adaptation (Sun et al., 2022) and cross-lingual learning (Mohtarami et al., 2019). As for inductive transfer learning approaches, sequential transfer learning is the most commonly used type in stance detection, which uses a source task model to improve the target model’s performance in a sequential manner (Liu et al., 2022; Ye et al., 2021). Multitask transfer learning is another type of inductive transfer learning in which several input attributes are learned sequentially or in parallel. In the context of stance detection, these attributes include rumor veracity, sarcasm, and sentiment (Khandelwal, 2021; Ye et al., 2021; Alturayeif et al., 2023a).
With the advent of Large Language Models (LLMs), researchers have started exploring their potential in stance detection (Lan et al., 2024). A recent benchmarking study by Cruickshank et al. (Cruickshank and Ng, 2024) on multiple stance detection datasets (e.g., SemEval 2016 (Mohammad et al., 2016a), COVID-LIES (Hossain et al., 2020), and phemerumors (Kochkina et al., 2018)) demonstrated that few-shot LLM prompting results can be competitive with supervised models. However, these results often show inconsistent performance (Cruickshank and Ng, 2024). Due to the inconsistency, marginal performance differences, and the high inference cost, it remains an open research question whether LLMs should be employed for stance detection (Cruickshank and Ng, 2024).
In this shared task, we invite participants to develop and evaluate stance detection models specifically tailored to the Arabic language. Alternatively, participants can explore various prompting styles of
LLMs to perform stance detection on the MAWQIF dataset (Alturayeif et al., 2022), the first dataset designed for Arabic stance detection.

heading: 3 Task Description
text: For the StanceEval-2024 task, participants are invited to showcase their approaches to stance detection. Solutions may utilize machine learning techniques tailored specifically for stance detection or explore alternative methodologies. Participants have the freedom to choose between single-task or Multi-Task Learning (MTL) paradigms. In singletask learning, the focus lies solely on stance data for model development and training. Conversely, MTL-based models offer the flexibility to incorporate additional information, such as sentiment and sarcasm cues from each tweet, to enhance the performance of the stance detection system. The provided dataset encompasses annotations for stance, sentiment, and sarcasm for each tweet.
Specifically, participants are tasked with developing models capable of detecting writers’ stances towards three specific topics: COVID-19 vaccine, digital transformation, and women empowerment. The possible stance labels are as follows:
• Favor: Indicates that the author supports the target. This could be explicit support or alignment with the target, or the presence of information such as news, quotes, or stories that reveal support for the target.
• Against: Indicates that the author opposes the target. This could be explicit opposition or alignment with the target, or the presence of information such as news, quotes, or stories that reveal opposition to the target.
• None: Indicates that the tweet provides no hint as to the author’s stance toward the target. This could include inquiries or neutral news that does not express any positive or negative position.
Examining the data annotations unveils a disparity between stance and sentiment in texts. It is evident that a tweet can convey a negative sentiment despite holding a favorable stance, and vice versa. Table 1 presents some examples showcasing the annotation of stance, sentiment, and sarcasm dimensions.

heading: 3.1 Dataset
text: The MAWQIF dataset (Alturayeif et al., 2022), a comprehensive resource for Arabic stance detection, is utilized for the StanceEval-2024 shared task. This dataset is meticulously crafted to aid in the analysis and development of models for stance detection, incorporating additional dimensions of sentiment and sarcasm to enhance the performance of stance detection systems.
The annotation process for the MAWQIF dataset involved multiple iterations to ensure high-quality annotations. Initially, annotators encountered challenges when asked to annotate stance, sentiment, and sarcasm simultaneously. To address this, annotation tasks were separated for each dimension, improving the consistency and reliability of the annotations. Each tweet was reviewed by a panel of three to seven annotators. Annotations were considered final when confidence scores exceeded 0.7, or after seven annotations if the required confidence was not met. Test questions were incorporated to evaluate annotator reliability, and annotations from annotators with performance below 80% were excluded from the final dataset.
The dataset comprises a total of 4,121 tweets, covering three distinct topics: "COVID-19 vaccine," "digital transformation," and "women empowerment." Here’s the breakdown of tweets for
each theme: COVID-19 vaccine: 1,373 tweets, Digital transformation: 1,348 tweets, Women empowerment: 1,400 tweets. This dataset is structured as a multi-label dataset, enabling models to utilize information from various dimensions simultaneously. For the purpose of training and evaluation, we partitioned the dataset into training and testing subsets, with an 85% to 15% split, respectively. Detailed statistics regarding this partitioning can be found in Table 2. A blind test set was provided to ensure fair comparison among participants, and it will be made publicly available after the evaluation period ends.
The dataset, an interactive visualization of the data, and the annotation guidelines can be accessed via the task repository 2.

heading: 3.2 Evaluation Metrics
text: In our evaluation, we utilized the macro F1-score (Fmacro) as the primary metric. This metric is computed as the average of the F1-scores for the "Favor" and "Against" categories. Specifically, the Fmacro is calculated separately for each target, and then the overall Fmacro is computed across all targets. Fmacro is computed using the following formula:
2https://github.com/NoraAlt/ Mawqif-Arabic-Stance
Fmacro = Ffavor + Fagainst
2 (1)
where Ffavor and Fagainst are calculated as follows:
Ffavor = 2PrecisionfavorRecallfavor Precisionfavor +Recallfavor (2)
Fagainst = 2PrecisionagainstRecallagainst Precisionagainst +Recallagainst
(3) The Fmacro evaluation metric offers a comprehensive evaluation of overall performance while addressing imbalanced data. It ensures equal contribution from both majority and minority classes, providing objective results even with imbalanced datasets.
We selected the Fmacro metric to maintain consistency with other stance detection datasets that report their results using this metric (Mohammad et al., 2016b). It is important to note that the "none" class, although sparse in the data, was still included during training. However, it was not considered in the evaluation because our focus was solely on the "Favor" and "Against" labels for this task. In practice, we view the "None" class as non-interesting or negative in Information Retrieval terms. Misclassifying "None" instances can impact metric scores negatively, and accurately predicting "None" is crucial to avoid labeling penalties. This approach aligns with other stance detection studies, where reporting results specifically for the "favor" and "against" stance labels using Fmacro is a common practice (Mohammad et al., 2016b; Alturayeif et al., 2023b).

heading: 4 Shared Task Teams & Results

heading: 4.1 Baselines
text: Four variants of the BERT model are provided to compare the submitted systems (Alturayeif et al., 2022). We fine-tuned the AraBERT-twitter (Antoun et al., 2020), MARBERT (Abdul-Mageed et al., 2020), and CAMeLBERT-da (Inoue et al., 2021) models. We will refer to these models as Baseline I, II, and III, respectively.
The baseline models have been fine-tuned using the training data of this shared task. We utilized only the stance label of each training sample during the fine-tuning process. Several pre-processing
stages were performed to prepare the training data for fine-tuning pre-trained models. These stages involved removing non-Arabic letters, repeated characters, diacritics, and tatweel. We used a WordPiece tokenizer (Wu et al., 2016) to segment the input text into tokens, and a sequence of up to 128 tokens was fed into BERT-based models. These models were fine-tuned for 20 epochs using the AdamW optimizer with a learning rate of 2e-5.

heading: 4.2 Results
text: In total, we received 28 unique team registrations. During the testing phase, 16 of these teams submitted valid entries. Out of 16 teams, we accepted 13 description papers for publication. Table 3 lists the 16 teams along with the citation of accepted papers.
Table 4 presents the results of the participating teams, ordered based on the Fmacro metric. We report the results of each topic and the average Fmacro across all topics. Additionally, we compare these results against our baseline models. As shown in the table, the AlexUNLP-BH (Badran et al., 2024) achieved the highest overall Fmacro score with 84.38% followed by the MGKM (Alghaslan and Almutairy, 2024) team with 82.06% Fmacro, wwhile the StanceCrafters (Hasanaath and Alansari, 2024) team secured the third place with 81.68%. Notably, four teams outperformed Baseline-I, while twelve teams surpassed Baseline-II. The gap between Baseline-I and the best-performing model (AlexUNLP-BH) is significant at 5.49%, whereas it is 1.52% with the SMASH (Hariri and Farha, 2024) model. For Baseline-II, twelve models surpassed it, and even the worst-performing model in this group outperformed it by 0.32 Fmacro.

heading: 4.3 Overview of Participating Systems
text: AlexUNLP-BH (Badran et al., 2024) implemented target-specific models using AraBERT variants: AraBERTv0.2-Twitter-base for women empowerment and digital transformation topics, and AraBERT COVID-19 for the vaccine topic. To enhance model generalization, they employed data augmentation techniques like random word removal and synonym replacement. Multi-task learning, incorporating sarcasm and sentiment analysis, was utilized to improve stance detection performance. Class imbalances were addressed using weighted cross-entropy loss. Additionally, they
Rank Team WomenEmpowerment Covid Vaccine Digital Transformation Overall Fmacro
applied contrastive loss to optimize the distance between similar and dissimilar sentence pairs, further enhancing model performance. The team adopted an ensemble method, combining models trained with different contrastive loss functions, resulting in a comprehensive system. Their approach achieved a macro F1 score of 84.38 and ranked first in the StanceEval 2024 shared task.
ANLP RG (Amal et al., 2024) The CAMeLBERTda, MARBERT, and AraBERT-twitter models were fine-tuned for sentiment, sarcasm, and stance detection tasks. Preprocessing steps, including HTML
tag removal and the replacement of URLs, email addresses, and attributions, were applied to prepare the input text. Among these models, AraBERTtwitter demonstrated the most effective performance in stance detection, achieving the best overall results.
CUFE (Ibrahim, 2024) utilized the newly-released Llama 3-8B model for Arabic stance detection. Each sample in the dataset was prompted using triplets consisting of the topic, text, and stance. The prompt employed the topic and text to predict the stance as a stance detection task. The Llama model
was fine-tuned using LoRA. The team achieved a ranking of ninth in the task.
dzStance (Lichouri et al., 2024) employed a combination of Term Frequency-Inverse Document Frequency (TF-IDF) features and Sentence Transformers for stance detection. TF-IDF captures the significance of terms in the document and weights them based on their frequency and rarity across documents. Sentence Transformers encode the contextual and semantic nuances of sentences. These two feature sets are integrated using a weighted fusion strategy and fed into a neural network architecture specifically designed for stance detection.
ISHFMG_TUN (Jaballah, 2024) proposed an ensemble approach for stance detection using a voting technique that incorporated multiple classifiers. During training, features were extracted from the input text and weighted using TF-IDF. Language modeling with different n-grams was employed to extract features at both the character and word levels. These features were then concatenated and fed into a set of classifiers, including SGDClassifier, LinearSVC, Multinomial Naive Bayes, Ridge Classifier, and Random Forest Classifier. Stance prediction was performed using a majority voting technique.
MGKM (Alghaslan and Almutairy, 2024) finetuned three large language models (LLMs)—GPT3.5-Turbo, Meta-Llama-3-8B-Instruct, and Falcon7B-Instruct—for stance detection using the MAWQIF dataset. Preprocessing involved using the AraBERT pre-processor to handle Arabic tweets with special characters. Models were fine-tuned using a Low-Rank Adaptation (LoRA) approach to manage model sizes locally, with training conducted over three epochs. GPT-3.5-Turbo-0125 emerged as the top-performing model, achieving an impressive F1 score of 82.06 and securing second place in the competition. This study highlights the effectiveness of fine-tuning LLMs for languagespecific tasks while acknowledging the computational challenges involved.
PICT (Shukla et al., 2024) employed an MTL approach to create a stance detection system, with stance detection as the primary task and sarcasm and sentiment detection as auxiliary tasks. The model was trained using a focal loss function, and various BERT model variants—including MARBERT, AraBERT, CAMeLBERT, and QARiB—were evaluated as potential
backbones for the system. Among these variants, the MARBERT model achieved the highest reported F1-score when used in the MTL framework.
Rasid (AlShenaifi et al., 2024) employed BERT fine-tuning with a 3-way classification layer as a head, focusing mainly on the MARBERT model due to its inclusion of both dialectal and modern standard Arabic. Additionally, they trained two models based on AraBERT and constructed an ensemble classifier. The Ensemble Classifier comprised Logistic Regression, Support Vector Machine, and Multinomial Naive Bayes, augmented with a TF-IDF Vectorizer. Their experiments demonstrated that MARBERT achieved the best results across all topics, followed by the AraBERT model.
SMASH (Hariri and Farha, 2024) evaluated six BERT-based models (AraBART, AraBERT, mBERT, CAMeLBERT-DA, MARBERT, and QARiB) and seven large language models (LLMs) (AceGPT, Gemma, Llama, Command R, WizardLM, Mistral, Mixtral). BERT and BART-based models were fine-tuned for stance detection using the provided training data, while LLMs were evaluated using a zero-shot setup. The team experimented with different Arabic and English prompts, finding that the best performance was achieved using English prompts and labels. They also reported that using the Arabic translation of the target names improved performance. Notably, Command R and LLAMA 3(70b) models demonstrated higher performance in identifying the stance of Arabic sentences compared to other models, according to the reported results.
StanceCrafters (Hasanaath and Alansari, 2024) proposed an MTL model by leveraging shared knowledge across sentiment, sarcasm, and stance tasks. The model architecture consists of a shared task layer comprising Arabert-Twitter and CamelBert BERT-based models combined via an attention mechanism, followed by task-specific layers for each task. Various weighting and aggregation techniques (averaging and attention) were evaluated to find the optimal combination. Additionally, static and relative loss weighting techniques were assessed to give more importance to the primary task. The highest F1-score was obtained by combining two modules via an attention mechanism and assigning a static weight to each task.
TAO (Melhem et al., 2024) fine-tuned ARABERT
for stance detection. The preprocessing of the data involved removing non-Arabic letters, numbers, and extra spaces. Additionally, diacritics, Tatweel, and words containing special symbols were eliminated. The cleaned data was then used to fine-tune ARABERT for the stance detection task.
Team_Zero (Galal and Kaseb, 2024) utilized pre-trained language models as feature extractors to avoid the cost of fine-tuning. They employed three Arabic BERT models: MARBERTAV, AraBERTv0.2-Twitter, and Parallel-Sum on Top of BERT (P-SUM-MTL). For MARBERTAV and AraBERTv0.2-Twitter, the team used their pretrained representations without additional tuning. Features were extracted directly from the models and fed into a logistic regression classifier to determine the stance of each tweet. P-SUM-MTL is a sophisticated architecture that considers embeddings from each of the last four layers of the BERT model. Each layer’s output is processed through a subsequent BERT layer, followed by a task-specific classifier in a multitask learning setup. The architecture calculates the overall loss by summing the losses from all individual classifiers, thereby effectively integrating insights from multiple tasks. The team applied majority voting for the outputs of the three models to obtain the final results.
StanceAlert (Alofi and Mnasri, 2024) employed comparative analysis to correlate dates with topics. For example, they observed that discussions on women empowerment were prevalent from 2020 until September 2021, followed by a decline. They then utilized a histogram-based approach to identify the most frequent words for each topic and stance pair. For fine-tuning, they used the bert-basearabertv02-twitter model with logistic regression to predict stance. Through this approach, the authors achieved 12th place in the task.

heading: 5 Conclusion
text: In conclusion, the StanceEval shared task, organized as part of the ArabicNLP 2024 program, has made significant strides in advancing Arabic stance detection. With the participation of 28 teams, 16 of which submitted valid entries and 13 provided detailed description papers, it reflects the growing interest and engagement in this research area. The comprehensive analysis presented in this paper underscores the diverse approaches and methodologies employed by the teams, offering valuable insights into the current state of stance detection
in Arabic texts. The results and comparative analysis not only highlight the strengths and limitations of various models but also lay the groundwork for future improvements and innovations. This shared task represents a crucial step towards the development of more effective and robust tools for analyzing user-generated text, with broad applications across politics, marketing, and social media analysis.

heading: Acknowledgments
text: The authors would like to acknowledge the support received from the Saudi Data and AI Authority (SDAIA) and King Fahd University of Petroleum and Minerals (KFUPM) under the SDAIA-KFUPM Joint Research Center for Artificial Intelligence Grant no. JRC-AI-RFP-14.

heading: Future Work
text: The results and comparative analysis not only highlight the strengths and limitations of various models but also lay the groundwork for future improvements and innovations. This shared task represents a crucial step towards the development of more effective and robust tools for analyzing user-generated text, with broad applications across politics, marketing, and social media analysis