MGKM at StanceEval2024: Fine-Tuning Large Language Models for Arabic Stance Detection (2024)

Abstract: Social media platforms have become essential in daily life, enabling users to express their opinions and stances on various topics. Stance detection is a task that identifies the viewpoint expressed in text toward a target subject. Despite the growing importance of Arabic tweets in shaping public opinion, there is a lack of research on stance detection in this domain. In this work, we evaluate the effectiveness of fine-tuning three Large Language Models (LLMs) in detecting target-specific stances in the MAWQIF dataset (Alturayeif et al., 2022). The LLMs assessed are ChatGPT-3.5turbo, Meta-Llama-3-8B-Instruct, and Falcon7B-Instruct. Our findings demonstrate that finetuning substantially enhances the stance detection capabilities of LLMs in Arabic tweets. GPT-3.5-Turbo exhibits the highest performance among the evaluated models, achieving a macro-F1 score of 82.93. Our work ranked second in the StanceEval2024 leaderboard, on a blind test set (Alturayeif et al., 2024).

heading: 1 Introduction
text: In the age of digital communication, social media platforms have become the norm in people’s daily lives, not just as a medium for social interaction but also as a platform for expressing opinions and stances towards a wide array of topics, from politics and events to services and controversial issues. This amplified the need for advanced stance detection technologies, aimed at detecting whether the author of a text is in favor, against, or neutral towards a specific subject, which is a task that is becoming increasingly important for decision-making processes across various sectors such as businesses and public authorities (Alturayeif et al., 2022).
Stance detection operates primarily on analyzing textual input to predict the author’s viewpoint, which may be explicitly or implicitly conveyed. Other author’s social activities such as retweets and likes can be used to enhance model performance.
The task is further categorized into target-specific, cross-target, and target-independent detection, each with its unique challenges and requirements, emphasizing the complexity of this field.
Initially, identifying viewpoints relied mostly on rules and traditional machine-learning methods. For instance, support vector machines (SVM) were highly regarded in the early stages (Mohammad et al., 2016; Walker et al., 2012; Anand et al., 2011). However, the emergence of deep learning models transformed this landscape significantly. These models excelled in processing vast amounts of data and uncovering intricate patterns within it (Zhang et al., 2019; Huang et al., 2018; Dey et al., 2018; Zarrella and Marsh, 2016; Wei et al., 2016). Subsequently, pre-trained language models, such as BERT emerged, aiding in a richer comprehension of text through contextual analysis (Kawintiranon and Singh, 2021; Li et al., 2021; Devlin et al., 2018).
Nowadays, LLMs such as OpenAI’s ChatGPT and Meta AI’s LLaMa-2 are revolutionizing natural language processing (NLP) (Touvron et al., 2023; Qin et al., 2023). These models, trained on extensive datasets, demonstrate a remarkable ability to mimic human language nuances with high accuracy (Yin et al., 2023; Zhao et al., 2023). Unlike older models, they can tackle different types of questions and grasp language nuances better, making them valuable for NLP tasks, including stance detection.
Refining LLMs such as GPT, LLaMa3, and Falcon for particular tasks through fine-tuning improves their accuracy and applicability for contextually sensitive stance detection on social media channels (Zhang et al., 2023d). By tailoring to the unique linguistic patterns and expressions prevalent in social media discourse, these models are empowered to exceed traditional methods, demonstrating remarkable proficiency in understanding sentiments and viewpoints.
This research examines the improved perfor-
816
mance of fine-tuned LLMs with the Mawqaf dataset to analyze diverse user opinions. We demonstrate that fine-tuning greatly enhances the models’ comprehension of user perspectives, providing a more profound understanding of online discussions. Our study emphasizes the benefits of fine-tuning in natural language processing (NLP), particularly for stance detection, and showcases its advantages over conventional, less customized approaches.

heading: 2 Related Work

heading: 2.1 Stance Detection
text: Stance detection is a well-studied task in natural language processing that involves identifying an entity’s opinion about a specific target (Ng and Carley, 2022). Unlike sentiment analysis, which can operate independently of context, stance classification requires understanding the context and target. This task is significant across various fields, prompting the development of numerous benchmark datasets and methodologies. Historically, the focus has been on supervised machine learning models, such as Support Vector Machines, which performed well in the SemEval-2016 stance detection competition (Lai et al., 2018; Elfardy and Diab, 2016; Mohammad et al., 2016). Neural networkbased models, including convolutional neural networks (Wei et al., 2016), recurrent neural networks (Zarrella and Marsh, 2016), and advanced architectures using textual entailment (Zhao and Yang, 2020) and data augmentation (Kawintiranon and Singh, 2021), are also widely used. Recent approaches have explored multi-task learning and transfer learning with transformer-based neural networks (Alturayeif et al., 2023; Yang et al., 2019; Zhao and Yang, 2020). Despite strong in-domain performance, these models often struggle to generalize to new data or targets (Ng and Carley, 2022; Alturayeif et al., 2023).
While supervised learning with human annotations dominates the field, unsupervised techniques are also explored. Unsupervised stance labeling uses language homogeneity for classification (Zhang et al., 2023c). For instance, graph neural networks analyze information from Twitter users to infer stances based on their past tweets and interactions (Zhang et al., 2023c). Another method involves label propagation within user interaction networks, mapping user relationships, and deriving stances from aggregated data within these networks
(Pick et al., 2022; Weber et al., 2013). These methods do not require predefined stance labels but often rely on specific assumptions about user behavior and language.
Zero-shot methods, which allow models to classify items without prior examples, are also used in stance classification. (Allaway and McKeown, 2023) discuss zero-shot stance detection techniques and introduce adaptations of the SemEval-2016 dataset and their VAST dataset for zero-shot classification. They outline three main paradigms: topic, language, and genre. In these paradigms, the model is trained on all data except for one element reserved for testing. Although zero-shot models typically underperform compared to fully supervised models, they provide valuable insights into stance detection without prior exposure to specific cases (Allaway and McKeown, 2023).

heading: 2.2 Large Language Models for Stance Detection
text: LLMs excel in tasks such as reading comprehension and solving math problems. They are trained on extensive datasets and can evaluate sentences and generate responses based on given prompts. Recent research has explored their application in stance detection, which will be discussed in the section.
Despite the development of various LLMs, much of the research has concentrated on the GPT family (e.g., ChatGPT, GPT-3) (Achiam et al., 2023), yielding mixed results. For instance, (Zhang et al., 2022) found that ChatGPT, using an instructionbased prompt, outperformed supervised models on the SemEval2016 benchmark dataset. Conversely, (Aiyappa et al., 2023) noted that while ChatGPT shows performance improvements in stance detection, the reliability of these results might be compromised by potential data contamination from its extensive training data. (Mets et al., 2024) evaluated ChatGPT for zero-shot stance detection on a custom dataset concerning immigration topics in news articles across multiple languages. Their findings indicated that although ChatGPT’s performance approached that of the best supervised models, it was still inferior for stance classification. Recent studies have introduced various Chainof-Thought prompting techniques, demonstrating improved performance by leveraging LLMs’ reasoning abilities (Zhang et al., 2023a,b). (Lan et al., 2023) achieved state-of-the-art results on benchmark stance datasets using a multistage Tree-of-
Thought-like prompt. Given the overlap between stance and sentiment classification, (Kheiri and Karimi, 2023) examined several OpenAI models for sentiment analysis, concluding that GPT models, especially when fine-tuned, surpass other models. Despite these advances, the effectiveness of LLMs in stance classification, particularly with prompt engineering and without fine-tuning, remains uncertain.
Fine-tuning LLMs poses significant challenges due to their immense size, often requiring multiple high-end GPUs and extensive memory capacity. These models, containing billions of parameters, require vast amounts of data and extensive training time, leading to high costs that can be prohibit individual researchers or small organizations. In response, techniques like Low-Rank Adaptation (LoRA) have been developed to mitigate the resource intensity of fine-tuning LLMs. LoRA introduces trainable low-rank matrices that modify the pre-trained weights during the adaptation phase, rather than retraining all parameters. This significantly reduces the number of trainable parameters, lowering memory usage and decreasing computational demands, allowing these models to be adapted with fewer resources and making it feasible using consumer-grade GPUs. (Hu et al., 2021)
LLMs operate based on prompts—free-text inputs that instruct the model on the desired output. The field of prompt engineering has emerged to optimize these inputs for better outputs (Schmidt et al., 2023; White et al., 2023; Ramlochan, 2023). Prominent prompt engineering techniques include zero-shot prompting, where the LLM receives only the task description, and few-shot prompting, which includes a few examples within the prompt (White et al., 2023; Brown et al., 2020; Wei et al., 2023). Unlike fine-tuning, these examples do not involve adjusting model weights but provide context to aid understanding. Although effective, fewshot prompting can be unstable due to factors like the order of examples (Zhao et al., 2021; Lu et al., 2021). While these techniques have shown promise, the optimal approach for using LLMs in stance detection remains an open question in research.

heading: 3 Methods

heading: 3.1 Models
text: We fine-tuned different LLMs and assessed their performance in detecting the author’s stance. We selected three models for evaluation: GPT-
3.5-Turbo-0125, Meta-Llama-3-8B-Instruct, and Falcon-7B-Instruct. We choose the Instruct variants given their pre-trained nature towards instructions. Since the dataset includes Arabic tweets with special characters such as emojis, we preprocessed them using the AraBERT pre-processor before passing them to the models.

heading: 3.2 Dataset
text: The experiment is based on the MAWQIF dataset, which consists of 4,121 tweets in multi-dialectal Arabic. Each tweet is annotated with a stance (Favor, Against, None) toward one of three targets: COVID-19 vaccine, Digital Transformation, and Women Empowerment. Additionally, it includes annotations for Sentiment (Positive, Negative, Neutral) and Sarcasm (Yes, No) polarities (Alturayeif et al., 2022).

heading: 3.3 System Prompt
text: The following example applies to Meta-Llama-38B-Instruct and Falcon-7B-Instruct training. For GPT-3.5-Turbo-0125, we followed the guideline provided by OpenAI. However, the system prompt part is shared between all models. An example of the prompt with input and output used for training follows:

heading: 3.4 Evaluation Metrics
text: We followed the standard evaluation protocols in (Alturayeif et al., 2024). The primary evaluation metric used is the macro F1-score. The macro F1score is calculated as the average of the F1-scores for the "Favor" and "Against" categories. The score for the "None" stance is ignored since it is undersampled in the dataset. This metric is computed for each target separately, and then the overall macro
F1-score is computed across all targets, which is given as follows:
Fmacro = Ffavor + Fagainst
2

heading: 3.5 Experimentation
text: We fine-tuned GPT-3.5-Turbo-0125 through OpenAI’s web services, and the two open-source models locally. To reduce the model sizes to a manageable scale for the local training, we applied LowRank Adaptation (LoRA) (Hu et al., 2021) using the LitGPT (AI, 2023) framework. The dataset was divided into an 85-15% train-validation split with stratification based on stance.
All models were fine-tuned over 3 epochs using a single NVIDIA A100 PCIE with 40GB RAM. The hyperparameters for fine-tuning include LoRA rank of 32, α of 16, dropout rate of 0.05, batch size of 8, 10 warm-up steps, and a learning rate of 2× 10−4. Meta-Llama-3-8B-Instruct and Falcon7B-Instruct required around 20 and 15 minutes to train respectively, whereas GPT-3.5-Turbo-0125 took around 30 minutes.

heading: 4 Results
text: Our results, shown in Table 1, GPT-3.5-Turbo demonstrated the highest overall performance, achieving a macro F1-score of 82.93%. It performed consistently well across all target categories: Women Empowerment (90.42%), COVID19 Vaccine (90.12%), and Digital Transformation (82.86%). This indicates GPT-3.5-Turbo-0125’s robust generalization capabilities and superior ability to handle multi-dialectal Arabic tweets.
Meta-Llama-3-8B-Instruct showed moderate performance with a macro F1-score of 74.33%. It performed well in detecting ’favor’ stances but struggled with ’against’ stances, particularly in the
Digital Transformation category. This suggests limitations in the model’s comprehension and classification capabilities.
Falcon-7B-Instruct had the lowest performance, with a macro F1-score of 40.73%. The model’s performance varied significantly across different categories: Women Empowerment (42.03%), COVID19 Vaccine (37.04%), and Digital Transformation (43.12%). The model struggled particularly with the ’against’ class, achieving scores of 3.92% for Women Empowerment, 15.69% for COVID-19 Vaccine, and 0% for Digital Transformation. The exact reasons for Falcon-7B-Instruct’s poor performance remain uncertain, but it seems the model struggles significantly with accurately classifying ’against’ stances, which heavily impacted its overall effectiveness.
fine-tuning LLMs significantly enhances their ability to detect stances in Arabic tweets. GPT-3.5Turbo-0125 emerged as the most effective model, while Meta-Llama-3-8B-Instruct and Falcon-7BInstruct showed potential but need further optimization. Future research can focus on refining hyperparameters, and instruction sets, or integrating Retrieval-Augmented Generation (RAG) to ingest a few examples similar to the input query as context to the model to improve LLM performance.

heading: 5 Conclusion
text: In this paper, we conducted a detailed analysis of stance detection on the MAWQIF dataset, an Arabic-language corpus annotated for multiple opinion dimensions across various dialects. Our experiments demonstrate the effectiveness of LLMs in enhancing the accuracy of stance detection. We evaluated the performance of three different LLMs, GPT-3.5-Turbo-0125, Meta-Llama-3-8B-Instruct, and Falcon-7B-Instruct and observed notable dif-
ferences in their ability to handle the complexities of multi-dialectal Arabic in social media texts.
The results indicate that fine-tuning LLMs significantly improves their ability to understand and detect stances in Arabic tweets. Notably, GPT-3.5Turbo-0125 emerged as the top performer, achieving remarkable precision in identifying both ’favor’ and ’against’ stances, underscoring the potential of fine-tuned LLMs for language-specific applications. The effectiveness of fine-tuning is further validated by the significant improvement over baseline models. Additionally, the research highlights the challenges associated with fine-tuning LLMs, such as the substantial computational resources required and the complexities of adapting these models to specialized tasks. However, techniques like LoRA have proven effective in mitigating these challenges, facilitating more accessible and efficient fine-tuning processes.
As we move forward, the insights gained from this study can guide future research towards enhancing model robustness, exploring more diverse datasets, and refining computational techniques to better meet the evolving needs of natural language processing applications. The integration of stance detection models into practical applications promises to improve decision-making processes, social media monitoring, and public sentiment analysis, making significant strides towards more informed and responsive digital communication platforms.

heading: Future Work
text: As we move forward, the insights gained from this study can guide future research towards enhancing model robustness, exploring more diverse datasets, and refining computational techniques to better meet the evolving needs of natural language processing applications. The integration of stance detection models into practical applications promises to improve decision-making processes, social media monitoring, and public sentiment analysis, making significant strides towards more informed and responsive digital communication platforms