SMASH at StanceEval 2024: Prompt Engineering LLMs for Arabic Stance Detection (2024)

Abstract: This paper presents our submission for the Stance Detection in Arabic Language (StanceEval) 2024 shared task conducted by Team SMASH of the University of Edinburgh. We evaluated the performance of various BERTbased and large language models (LLMs). MARBERT demonstrates superior performance among the BERT-based models, achieving F1 and macro-F1 scores of 0.570 and 0.770, respectively. In contrast, the Command-R model outperforms all models with the highest overall F1 score of 0.661 and macro F1 score of 0.820.

heading: 1 Introduction
text: Stance detection, also known as stance classification, prediction, or identification, is an NLP task that aims to determine the viewpoint of a text’s author (Favor, Against, or Neutral) towards a specific target topic (Küçük and Can, 2020). Furthermore, it has gained significance both as a standalone problem and in conjunction with applications such as sentiment analysis, argument mining, sarcasm detection, rumor detection, fact-checking, and fake news detection (Alhindi et al., 2021; Küçük and Can, 2020). Arabic NLP presents unique processing challenges due to its diverse dialects and complex orthographic features (Darwish et al., 2021; Abu Farha and Magdy, 2020). Furthermore, the limited resources, such as annotated corpora for Arabic compared to English, intensify these challenges (Abu Farha and Magdy, 2021; Darwish et al., 2021; Abdul-Mageed et al., 2011). Hence, the task of stance detection is gaining increasing attention in underrepresented languages such as Arabic.
StanceEval 2024 shared task (Alturayeif et al., 2024), co-organized with the ArabicNLP 2024 conference, is motivated by the aforementioned factors, aiming to adapt and overcome these challenges by confronting the distinct aspects of Arabic language processing.
This paper describes our participation in the StanceEval shared task, where we achieved 4th place. It also provides a detailed overview of the models we used to tackle the task. We compared the performance of BERT-based models and prompt engineering on large language models (LLMs). Our official submission was based on a prompt engineering approach utilizing CommandR model, which achieved an F1 score of 0.670 and macro F1 score of 0.821. For BERT-based models, MARBERT was the best, with an F1 of 0.568 and macro F1 score of 0.761
The rest of this paper is organized as follows: Section 2 provides a detailed description of the dataset used for this work. Section 3 outlines the experimental setup and briefly describes the models we utilized. Subsequently, the results, discussions, and findings are presented in Section 4, and the paper concludes with a summary and implications of the research in Section 5.

heading: 2 Data
text: The dataset used in the StanceEval 2024 shared task is Mawqif, a multi-label Arabic dataset for targetspecific stance detection (Alturayeif et al., 2022, 2024). Mawqif dataset contains 4,121 sentences covering three topics, ‘COVID-19 vaccine’, ‘digital transformation’, and ‘women empowerment’ as shown in Table 1. Mawqif dataset is a multilabel dataset with labels for stance (Favor, Against, None), sentiment (Positive, Negative, Neutral), and sarcasm (Sarcastic and Non-sarcastic). For evaluation, the shared task organizers split Mawqif dataset
800
into two subsets: a training and a blind test set. However, we split the training dataset into training and validation sets for our experiments. Table 1 shows the distribution of the dataset subsets we used.

heading: 3 Methodology

heading: 3.1 Models
text: This section provides an overview of the models used in the experiments. The experiments span two types: fine-tuned models and zero-shot models.

heading: 3.1.1 Fine-tuned Models
text: This section delves into the specifics of the models we used in the experiments, which were fine-tuned on the training data for the task. The models include the following:
• AraBART (Kamal Eddine et al., 2022): an Arabic model based on BART (Lewis et al., 2020) in which the encoder and the decoder are pretrained end-to-end. It is pre-trained on a corpus of 73GB. • AraBERT: an Arabic-specific BERT model provided by (Antoun et al., 2020). We utilized the two AraBERT models, v0.2-base and v2-base. Both models are pre-trained on 24GB of data from Wikipedia, news articles, and the Open Source International dataset (OSIAN). • mBERT (Devlin et al., 2018): a multilingual BERT model developed by Google AI and trained on 104 languages from Wikipedia’s data. • CAMeLBERT (Inoue et al., 2021): we utilized two models, the dialectal Arabic (DA) and the mixed model (Mix), which trained on mixed data of Modern Standard Arabic, dialectal Arabic, and classical Arabic. • MARBERT (Abdul-Mageed et al., 2021): a model trained on a set of 128GB dataset, consisting of 1B tweets. • MARBERTv2 (Abdul-Mageed et al., 2021): a version of MARBERT model but trained further on 61GB of MSA data in addition to an 8.6GB of Arabic news dataset. • QARiB (Chowdhury et al., 2020; Abdelali et al., 2021): is a dialectal Arabic BERT model. It was trained on data from tweets and a combination of Arabic GigaWord, Abulkhair Arabic Corpus, and OPUS.

heading: 3.1.2 Zero-shot Models
text: We tested the models below in a zero-shot setup. For the experiments, we utilize the quantized versions through Ollama1. The following are the models’ details and the specific variants used for the experiments:
• AceGPT (Huang et al., 2023): An opensource, culturally aware LLM was developed to align with the values of Arabic-speaking communities. We utilized the quantized model of (13b2). • Gemma (Gemma Team et al., 2024): an open model built by Google DeepMind (Google AI, 2024). We utilized the 7b instruct model (7b-instruct-v1.1-fp16). • LLAMA 3: an LLM developed by Meta (Meta, 2024) . In the experiments we employed the versions: (70b-instruct), and (8b-instruct-fp16). • Command-R3: an LLM from Cohere, optimized for conversational interaction and long context tasks. It was trained on a massive corpus in multiple languages with main focus on critical business use-cases. We utilized the quantized version (35b-v0.1-q8_0). • Command-R+4: an updated LLM from Cohere in the Command-R family, which incorporates complex RAG functionality and multistep tool use (agents). We used the model (104b-q2_K) in our experiments. • WizardLM-25 (7B): an LLM developed by Microsoft AI. We used the model (wizardlm2:7b-fp16). • Mistral 7b (Jiang et al., 2023): an LLM model with open weights in 8x7b and 8x22b parameter sizes. We utilized (instruct-v0.2-fp16). • Mixtral (8x7b): a high-quality sparse mixture of experts model (SMoE) with open weights provided by (Jiang et al., 2023).

heading: 3.2 Experimental Setup
text: In our experiments, we tackled the task as a classification problem and used fine-tuned models (i.e., BERT and BART-based models) and zero-shot models (i.e., LLMs).
1https://ollama.com/ 2Ollama’s model tag: salmatrafi/acegpt:13b 3https://docs.cohere.com/docs/command-r 4https://docs.cohere.com/docs/command-r-plus 5https://wizardlm.github.io/WizardLM2/
For the BERT and BART-based models, we finetuned each model for 7 epochs using the AdamW optimizer, with an initial learning rate of 2e-05. The batch size was set to 128, with the weight decay parameter of 0.01.
Furthermore, for the zero-shot setup, we experimented with various prompts to achieve the best performance. We systematically evaluated a diverse set of language models for their ability to detect stance in Arabic text. Each model was prompted to classify the stance of each sentence as ‘favor’, ‘against’, or ‘neutral’ towards the target topic. The output of the models was then processed to extract the assigned label. This includes matching both American and British versions of the word favour/favor, and the rare cases of mixed Arabic answers such as (
¬avor). After multiple experiments and iterations, we noticed that prompting the model to produce answers in Arabic significantly impairs the performance, exhibiting ‘hallucination’ behavior, as the output would usually be random Arabic sentences without a clear answer. Furthermore, integrating labels in Arabic, as opposed to English, resulted in a deterioration of performance. Thus, we decided to go with an English prompt that relies on asking the model to provide the stance of a given text. We also noticed that using the Arabic translation of the target names would improve the performance. In this setup, we believe that we minimized any issues that might arise when asking the models to generate Arabic text, and we limit our reliance on their capabilities in Arabic to comprehension only. Table2 shows the Arabic translation of the target topics. Appendix A includes some of the prompts we tested. The final prompt is as follows:
You are an expert in analyzing people’s opinions. You are an expert in Arabic. You will be given an Arabic sentence as input. Your task is to identify the stance towards the topic or subject discussed in the sentence. Your task is to identify whether the sentence is in favor of the topic, against it, or
neutral. Your output should be one of the following: favor, against, or neutral. You should not provide any further information. Your answer should be in English. éÊÒm.Ì'@: input_sentence ¨ñ ñÖÏ @: target_topic_in_Arabic What is the stance in the given sentence? [favor, against, neutral].
All models were evaluated using the official metric for the task, which is the average macro F1 score over the favor and against classes:
F1 = 1
2 (FF1 + F A 1 ) (1)
Where FF1 and F A 1 are the F1 scores over the
favor and against classes respectively.

heading: 4 Results and Discussion

heading: 4.1 Results on the evaluation set
text: Table 3 shows the F1 and macro F1 scores for the models on the validation and the test sets. It is worth noting that we relied on the results on the validation set to select the model to be used for the official submission. The results on the tests were calculated after the gold labels were made public and are included for comparative purposes. From the table, it is clear that MARBERT performs best among the BERT/BART-based models with a macro F1 score of 0.770 and an F1 score of 0.570. For the LLMs in a zero-shot setup, their performance is comparable with the BERT/BART-based models, and some of them are on par with MARBERT. However, Command-R model achieves the best performance with an F1 score of 0.820, which shows the strong performance of the new larger LLMs.
Generally, Command-R and LLAMA 3(70b) models demonstrated notably higher performance in identifying the stance of Arabic sentences compared to smaller models. This finding indicates the advantage of larger model architectures in handling complex linguistic tasks. Secondly, variability in performance across models highlights the significance of model-specific tuning and context adaptation for improving stance detection outcomes. In fact, the outcome of the post-evaluation testing, i.e., testing the performance of the models on the released labeled test set, confirms the findings.

heading: 4.2 Official submission
text: Based on the results on the validation set, the best model is Command-R. Thus, it was used for the official submission. Our team, SMASH, was ranked 4th with a macro F1 score of 0.8041.

heading: 4.3 Discussion
text: From the results in Table 3, it is clear that fine-tuned models are achieving relatively good performance, with MARBERT being the best with a macro F1 of 0.770. This would probably be due to the fact that it was trained on tweets, which matches the nature of the task at hand. It is worth noting that MARBERT showed a stable good performance on multiple social media-related NLP tasks (Abdul-Mageed et al., 2021; Abu Farha and Magdy, 2021). For zero-shot models, their performance was variable, depending on the amount of Arabic text present in their training data. The best model was Command-R, with a macro F1 of 0.820.
The confusion matrix in Figure 1, shows the detailed performance of Command-R and MARBERT, the top two models. From the figure, it is noticeable that Command-R is better at identifying the ‘against’ and ‘neutral’ classes, while MARBERT is better at identifying the ‘favor’ class. Generally, it seems that MARBERT has a tendency to classify ‘neutral’ cases as ‘favor’, which might be due to the imbalance in the dataset. To address this, we balanced the training data by upsampling the ’against’ class. Following this adjustment, the model achieved an F1 score of 0.543 and a macro F1 score of 0.757. These results are slightly lower
than the results without data balancing. This gap in performance shows the promising capabilities of LLMs as they achieve excellent performance despite being multilingual.

heading: 5 Conclusion
text: This study explores various models for the Stance Detection in the Arabic Language (StanceEval) 2024 shared task, focusing on both BERT-based and new large language models (LLMs). Our experiments demonstrated that while MARBERT excelled among the BERT-based models, new larger models such as Command-R achieved the highest overall performance. These findings significantly highlight the potential of LLMs in capturing the nuanced semantic and contextual elements necessary for stance detection in Arabic. However, it is worth noting that all of these models are multilingual, which explains why some of them achieved relatively low performance. This signifies the importance of developing Arabic-specific LLMs, which would help improve the performance on Arabic NLP tasks.

heading: 6 Limitations
text: A major limitation of our work is that all of the models we used are either English models or multilingual models. Using Arabic-specific models would provide better performance and ability to handle Arabic, its dialects, and cultural context. We tried using JAIS, but we weren’t able to load the model using the hardware available to us. Another limitation of this study is the application of a zero-shot learning approach. We would implement few-shot learning techniques in future works, as it has been shown that providing in-context examples enhances the performance of large language models.

heading: A Prompt Experiments

heading: Future Work
text: We would implement few-shot learning techniques in future works, as it has been shown that providing in-context examples enhances the performance of large language models