Are Dialects Better Prompters? A Case Study on Arabic Subjective Text Classification
Authors: Leila MOUDJARI, Farah Benamara
Accepted at: acl25

Are Dialects Better Prompters? A Case Study on Arabic Subjective Text Classification

Leila Moudjari¹ and Farah Benamara¹²
¹ IRIT, Université de Toulouse, ANITI, Toulouse, France
² IPAL, CNRS-NUS-A*STAR, Singapore

Abstract
This paper investigates the effect of dialectal prompting, variations in prompting script and model fine-tuning on subjective classification in Arabic dialects. To this end, we evaluate the performances of 12 widely used open source LLMs across four tasks and eight benchmark datasets. Our results reveal that specialized fine-tuned models with Arabic and Arabizi scripts dialectal prompts achieve the best results, which provides new findings on the fine-tuning of LLMs for low-resource languages.

1 Motivations
Large Language Models (LLMs) have gained significant attention for their ability to perform various NLP tasks, including text classification. Studies have demonstrated that these models achieve high accuracy on a wide range of classification tasks by leveraging prompt engineering and customized prompts (Brown et al., 2020; Sun et al., 2023; Bareiß et al., 2024). However, the results for subjective classification remain a challenge. For example, Zhang et al. (2024) have shown that LLMs perform poorly in classifying highly subjective content, such as irony and emotions, where contextual subtleties play a crucial role. Fine-tuning these models on large affective datasets has been shown to improve performance for tasks involving nuanced emotional understanding (Liu et al., 2024).

Additionally, studies indicate that LLMs tend to perform better when prompts are either in English or translated to English compared to when they are in the target language of the testset (Perak et al., 2024; Ondrejová and Šuppa, 2024; Nguyen et al., 2024; Cahyawijaya et al., 2024; Bareiß et al., 2024). This is particularly salient for low-resource languages such as dialectal Arabic and its regional variants, where similar declines were observed (Abdelali et al., 2024; Ahuja et al., 2023; Koto et al., 2024; Zhang et al., 2023; Abdelaziz et al., 2024).

Improvements have, however, been observed with code-mixed prompts in few-shot settings, i.e., instructions in English and examples in MSA (Ahuja et al., 2023).

When it comes to subjective text classification in Arabic, three main challenges remain: (1) Task-specific transformer fine-tuned models consistently outperform LLMs when prompted in zero or few-shot settings (Zhang et al., 2023), (2) The results when using native language prompts in MSA vs. English were not conclusive (e.g., increase in subjectivity detection but decrease in sentiment analysis (Abdelali et al., 2024)), and (3) LLMs struggle with more nuanced subjective tasks like detecting sarcasm and emotions (Abdelali et al., 2024).

In this paper, we aim to address these challenges by investigating, for the first time as far as we know, the effectiveness of dialect-specific prompting in improving subjective text classification in Arabic. Our work makes contributions in both advancing prompt engineering methodologies and studying the effects of linguistic variation in prompt design, focusing on the following dimensions:

1. Dialectal prompting. We explore how LLMs respond to prompts written in Maghrebi (Algerian, Tunisian) and Levantine dialects, comparing performances with prompts in English and MSA.

2. Variations in prompting script. Code-switching may impact LLMs performances. We therefore vary prompts’ scripts, exploring Latin, Arabic, and newly Arabizi (Yaghan, 2008), the informal Arabic chat alphabet in which words are written in their transliterated form using Latin characters and numerals (e.g., "a3tini" (give me)).

3. LLMs fine-tuning. We compare fine-tuned vs. non fine-tuned versions of 12 open source LLMs while varying their sizes and the training process
in terms of training strategy (instruction vs. chat-tuned), language (Arabic vs. multilingual) and specialization (affective vs. general-purpose).

4. Robustness and generalization. We design experiments to assess how well fine-tuned models generalize across dialects, tasks, and scripts, shedding light on the impact of linguistic variation on model adaptability.

Our results provides new findings on fine-tuning LLMs for low-resource languages: (a) Fine-tuned models with dialect prompts written in Arabic script and Arabizi achieve the best, (b) Arabic-centric models demonstrate stronger efficiency, (c) LLMs achieve strong performances outperforming fine-tuned Bert-like smaller models, (d) Transfer learning of fine-tuned models across datasets/tasks as well as switching to a different dialect at inference time, significantly degrade performance. Our models are available to the research community.1

2 Methodology

2.1 Classification tasks

We explore 4 subjective tasks of various complexity, selecting publicly available datasets for each task where instances from various length are written either in MSA only, a specific dialect, or mixture of MSA and/or different dialects. We consider a wide range of Arabic dialects from different regions: North Africa (Algerian (Dz), Tunisian (Tn)), Levantine (Lv), and Gulf (Gl). Our datasets are as follows (see Appendix A for a detailed description, including the train/test split):

(1) Sentiment analysis (SA): OCA_{MSA} (Rushdi-Saleh et al., 2011), Sem17_{MSA+Mixed} (Rosenthal et al., 2017), Twifil-sent_{Dz} (Moudjari et al., 2020), and TSAC_{Tn} (Medhaffar et al., 2017).

(2) Emotion recognition (EM): Twifil-emo_{Dz} (Moudjari et al., 2020), Sem18_{MSA+Mixed} (Mohammad et al., 2018).

(3) Irony detection (ID): IDAT_{MSA+Mixed} (Ghanem et al., 2019).

(4) Crisis management (CM): Flood_{Lv+Gl} (Alharbi and Lee, 2019).

In order to study the impact of dialect prompting in LLMs performances, we further analyzed the distribution of script in our datasets (cf. Appendix A.2). Arabic scripts are a majority on all datasets, except for OCA_{MSA} where data is mixed with Latin scripts coming from English movie titles. Maghrebi dialect datasets contain instances exclusively written in Latin characters. In Table 2 from the same Appendix, we also analyzed the frequency of MSA samples relying on the ALDi score (Keleg et al., 2024) where \( 0 \leq ALDi < 0.1 \) indicating that an instance is expected to be in MSA, otherwise it is likely in dialect. All ALDi scores are > 0.1 with Flood_{Lv+Gl} having the smallest scores. After a manual check, we observed that these scores may not reflect the real percentage of dialect, in particular when the inputs are in Arabizi. We provide examples and discussion on this issue in the Limitation Section and Appendix A.2.

2.2 Experimental Settings

Models. See Appendix B.1:

• Transformers. We employ three models that have been pre-trained on MSA and social media content from various dialects: AraBERTv2 (Antoun et al., 2020), CAMeLBERT (Inoue et al., 2021), and DarijaBERT (Gaanoun et al., 2024).

• Non fine-tuned LLMs. We use 12 LLMs from three families: (a) General-purpose multilingual LLMs: SOLAR10.7B (Kim et al., 2023), Mistral7b, Mistral-Instruct7b(Jiang et al., 2023), LLaMA3sb (Dubey et al., 2024) and LLaMA3-Instructsb, (b) Arabic-centric LLMs: Jais-family13b (Sengupta et al., 2023), AceGPT8b (Huang et al., 2024) and AceGPT-chat8b, and (c) Specialized affective models: EmoLlama7b (Liu et al., 2024), EmoLlama-chat7b and EmoLlama-chat13b.

• Fine-tuned LLMs. These 12 models have also been fine-tuned with QLoRA (Quantized Low-Rank Adaptation) (Dettmers et al., 2024).

Prompts. Given a task and to avoid bias, all LLMs share the same zero-shot prompts2 and parameters (see Appendix C and B.2, resp.). We utilize role-playing by assigning roles to LLMs (Liu et al., 2023) and direct the models to generate a single response based on the provided labels only to limit hallucination. As LLMs may have different output formats, we consider a class correct if it exactly corresponds to task categories. When the model is unable to confidently assign a label,

1https://huggingface.co/DialectPrompting/Dialectal_LLMs

2Experiments with few-shot and in-context mixing (Shankar et al., 2024) were not conclusive.
we implemented two strategies: assigning a "neutral" label when it is included in the task annotation scheme, and resorting to a random false label, as suggested by Zhang et al. (2023), otherwise.

We also vary the languages and scripts:

• English Prompts: Instructions are written in English, with English labels, and examples are randomly sampled from the task dataset following (Abdelali et al., 2024; Koto et al., 2024).

• Arabic Prompts: English instructions are translated into MSA following (Ahuja et al., 2023; Zhang et al., 2023). We use ChatGPT then manually checked the translations by a native speaker, while examples are kept unchanged (i.e., same language as in the target task).

• Dialectal Prompts: Arabic instructions as well as labels are manually translated into a dialect \( d = \{Lv, Dz, Tn\} \) written either in Arabic (\( ar \)) or Arabizi (\( abz \)) scripts. For Lv, we experiment with \( ar \) and \( abz \), however and given that Dz and Tn are two Maghrebi dialects, we explore Dz prompting with \( abz \) script to capture its distinct code-mixed nature while Tn with \( ar \). This is also motivated by the very similar preliminary results obtained when prompting LLMs in Dz and Tn using Arabic script. In the following, the notation \( d_s \) indicates a prompt written in a dialect \( d \) using script \( s \in \{ar, abz\} \).

Impact of fine-tuning and prompting variations.
We design four experimental settings to evaluate the impact of dialectal and script variation on model generalization:

• In-dataset: All inputs are from a distribution present during training/testing. This is the standard within-dataset baseline.

• In-dataset\(_{DA}\): A variant of the above where dialectal instances are isolated based on their ALDi scores, teasing apart MSA samples to better assess the effectiveness of our prompting strategies on dialect-specific data.

• Cross-dataset transfer: It focuses on evaluating task-level generalization across datasets with different dialects/scripts, without any exposure to the test distribution. Therefore, a model trained on one dataset is tested on another—within the same task but differing in dialect and/or script—without additional fine-tuning on the target data.

• Fine-tuning transfer: This setting examines the ability of LLMs to generalize when the prompting conditions at inference time differ from those seen during fine-tuning. It helps answer whether switching the prompting language (e.g., from Tunisian to MSA or Arabizi) degrades performance, and whether adaptation to one variant enhances or constrains generalization.

3 Results

To reduce randomness and enhance generalizability, all models are evaluated on the same test set and run three times with different seed of examples, if any; we report the averaged macro-F1 across runs.

In-dataset results. The main findings are as follows (cf. Table 1, together with Figure 3 as well as Tables 11 and 7 in Appendix D.1):

(a) Dialect: Prompting with the target dataset dialect is not always effective, likely due to varying sensitivities of LLMs to specific dialects, as acknowledged in a recent study (Mousi et al., 2025). For instance, LLaMA and its derivative as well as AceGPT excel in Gulf Arabic, while IAIS in Levantine. Additionally, multilingual models often confuse dialects (e.g., Gulf with Levantine or MSA) and tend to switch to MSA in uncertain cases.

(b) Script: We observe a correlation between the dominant script in the dataset and the prompt used (see Figure 1 in Appendix D.1). For instance, in Twifil-sent\(_{Dz}\), where 50% of instances use a mix of scripts, the best results were achieved after fine-tuning using Algerian prompting. Similarly, in Sem18\(_{MSA+Mixed}\), where 83% of instances are written in Latin script, the best results were achieved using Levantine with Arabizi. There are, however, two exceptions: (i) OCA\(_{MSA}\), where 73% of the data is mixed between Arabic and Latin script. Best scores have been achieved with MSA which is consistent as sentiment tags are mostly derived from the review (and not from the movie titles). (ii) Twifil-emo\(_{Dz}\), where English prompts LLaMA3\(_{ab}\) were the best. This is likely due to these LLMs being trained on significantly larger English datasets compared to Arabic. After fine-tuning, Levantine with Arabizi achieved the best results (35.81), with MSA closely behind.
<table>
  <tr>
    <th></th>
    <th>Sem17<sub>MSA+Mixed</sub></th>
    <th>TwiFil-sent<sub>Dz</sub></th>
    <th>OCA<sub>MSA</sub></th>
    <th>TSAC<sub>Tn</sub></th>
    <th>Sem18<sub>MSA+Mixed</sub></th>
    <th>TwiFil-emo<sub>Dz</sub></th>
    <th>IDAT<sub>MSA+Mixed</sub></th>
    <th>Flood<sub>Lv+Gf</sub></th>
  </tr>
  <tr>
    <th colspan="9">Best Baseline</th>
  </tr>
  <tr>
    <td>Reported SOTA*</td>
    <td>61.00</td>
    <td>79.52</td>
    <td>90.60</td>
    <td>63.35</td>
    <td>49.53 (6 classes)</td>
    <td>59.19 (6 classes)</td>
    <td>84.40</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Transformer</td>
    <td>71.22</td>
    <td>74.46</td>
    <td>91.00</td>
    <td>93.90</td>
    <td>24.23</td>
    <td><span style="background-color:#b3e5ff">38.06</span></td>
    <td>83.75</td>
    <td><span style="background-color:#b3e5ff">70.87</span></td>
  </tr>
  <tr>
    <th colspan="9">Non Fine-tuned LLMs</th>
  </tr>
  <tr>
    <td>Best Eng</td>
    <td>42.18</td>
    <td>65.25</td>
    <td>87.92</td>
    <td><span style="background-color:#b3e5ff">92.86</span></td>
    <td>24.77</td>
    <td>27.44</td>
    <td><span style="background-color:#b3e5ff">58.45</span></td>
    <td>38.05</td>
  </tr>
  <tr>
    <td>Best MSA</td>
    <td><span style="background-color:#b3e5ff">50.94</span></td>
    <td>68.28</td>
    <td><span style="background-color:#b3e5ff">90.99</span></td>
    <td>67.30</td>
    <td>24.77</td>
    <td>21.97</td>
    <td>39.54</td>
    <td>31.63</td>
  </tr>
  <tr>
    <td>Best Dialect</td>
    <td>41.31</td>
    <td><span style="background-color:#b3e5ff">75.92</span></td>
    <td>89.00</td>
    <td>70.80</td>
    <td>22.86</td>
    <td>23.98</td>
    <td>56.80</td>
    <td>33.98</td>
  </tr>
  <tr>
    <td>→ dialect<sub>Levantine</sub></td>
    <td>DZ<sub>zh</sub></td>
    <td>LV<sub>ar</sub></td>
    <td>LV<sub>zh</sub></td>
    <td>DZ<sub>zh</sub></td>
    <td>LV<sub>ar</sub></td>
    <td>LV<sub>ar</sub></td>
    <td>LV<sub>ar</sub></td>
    <td>LV<sub>ar</sub></td>
  </tr>
  <tr>
    <th colspan="9">Fine-tuned LLMs</th>
  </tr>
  <tr>
    <td>Best Eng</td>
    <td>80.36</td>
    <td><span style="background-color:#b3e5ff">89.59</span></td>
    <td>98.00</td>
    <td>97.98</td>
    <td>30.66</td>
    <td>27.78</td>
    <td>84.54</td>
    <td>67.04</td>
  </tr>
  <tr>
    <td>Best MSA</td>
    <td>81.39</td>
    <td>86.77</td>
    <td><span style="background-color:#b3e5ff">99.98</span></td>
    <td>98.89</td>
    <td>30.20</td>
    <td>33.38</td>
    <td>87.34</td>
    <td>67.57</td>
  </tr>
  <tr>
    <td>Best Dialect</td>
    <td><span style="background-color:#b3e5ff">85.23*</span></td>
    <td>87.13</td>
    <td><span style="background-color:#b3e5ff">99.98</span></td>
    <td><span style="background-color:#b3e5ff">99.34*</span></td>
    <td><span style="background-color:#b3e5ff">32.41*</span></td>
    <td><span style="background-color:#b3e5ff">35.81*</span></td>
    <td><span style="background-color:#b3e5ff">94.89*</span></td>
    <td><span style="background-color:#b3e5ff">67.06*</span></td>
  </tr>
  <tr>
    <td>→ dialect<sub>Levantine</sub></td>
    <td>Th<sub>ar</sub></td>
    <td>DZ<sub>zh</sub></td>
    <td>LV<sub>ar</sub></td>
    <td>DZ<sub>zh</sub></td>
    <td>LV<sub>ar</sub></td>
    <td>LV<sub>ar</sub></td>
    <td>LV<sub>ar</sub></td>
    <td>LV<sub>ar</sub></td>
  </tr>
</table>

Table 1: Best baseline and best LLMs results before and after fine-tuning in terms of macro F1-score. Best score per language/dialect are in bold while overall best performance for each dataset is highlighted in green. *: The difference between dialect vs. MSA prompting is statistically significant at \( p < 0.01 \). Δ: Reported state of the art results using different experimental settings (see Section 4 for a discussion).

(c) Fine-tuning (FT) vs. non fine-tuning (N-FT): FT LLMs outperform transformers in all tasks, except CM and ED in Twifil-emo<sub>Dz</sub> with a small decrease of around 3%. N-FT results are in-line with state of the art findings where English prompts performed the best (Koto et al., 2024). FT results on the other hand, show consistent improvement, with dialectal prompting yielding the best performance. Also, dialects with shared linguistic features, such as Algerian and Tunisian, led to comparable outcomes across models, indicating the models’ ability to leverage these similarities.

(d) Models size, language and specialization. We observe that specialized LLMs fine-tuned on affective data (i.e. EmoLlama family) do not exhibit specific improvement over general-purpose LLMs. When comparing pre-training strategies, instruction-tuned models, particularly Llama3-inst<sub>zh</sub> outperformed other non Arabic-centric models across nearly all datasets, with Emomalla-chat<sub>13b</sub> closely following in two datasets. When it comes to Arabic vs. multilingual LLMs, language-specific AceGPT achieves best results in almost all datasets beating JAIS, the other Arabic LLM. Even without fine-tuning, prompting in MSA exhibits better performances than English, while dialects are the more productive prompts after fine-tuning.

In-dataset<sub>DA</sub> results. Table 8 in Appendix D.2 shows that when a dataset contains a high percentage of MSA, as seen in TSAC<sub>Tn</sub> (ALDi > 0.1), performance drops significantly with a lost 62.94% in F-score, while Flood<sub>Lv+Gf</sub> experienced a 19.75% decline. Conversely, when the MSA proportion is lower, the impact of ALDi-based filtering is minimal, and in some cases, even beneficial. For instance, Twifil-sent<sub>Dz</sub> saw a slight improvement of 0.2% after removing texts with ALDi=0. Additionally, the presence of dialectal texts can also enhance classification performance on MSA instances, as observed in most cases, whereas for OCA<sub>MSA</sub>, the inclusion of dialect samples did not impact the results.

Cross-dataset results. We measured transfer learning of our best models across SA and ED tasks (the two others containing one dataset each). For SA (see Table 9 Appendix D.2), models perform best when prompted in the same script. For instance, in Sem17<sub>MSA+Mixed</sub>, the top-performing model used Tunisian, while the best transfer learning results were achieved with Levantine. Levantine prompts generally yield stronger results, as Arabic-centric models tend to perform better on non-Maghrebi Arabic.

For ED (see Table 10 in Appendix D.2), we observe the same tendency. For example, in Sem18<sub>MSA+Mixed</sub>, the best model was prompted in Levantine Arabizi, while the strongest transfer learning occurred with Algerian Arabizi. Algerian and Tunisian dialects exhibit strong mutual transferability. In TwiFil, the best-performing Algerian model showed optimal transfer when prompted in Tunisian Arabic, and vice versa.

Fine-tuning transfer results. Models fine-tuned on a specific dialect exhibit varying levels of robustness when tested on different dialects (see yellow cells in Tables 9 and 10 in Appendix D.2). In particular, models initially trained on larger data from MSA and Levantine/Gulf exhibit greater adaptability when fine-tuned on other regionally specific dialects (e.g., Tunisian, Algerian
Arabizi), see for e.g., the Sem17 (Tn_{ar}) model that achieved the highest performance in SA, followed by the Twifil-sent (Dz_{abz}) model — both built on AceGPT_{sb}.

A similar trend is observed in ED, demonstrating superior generalization, therefore maintaining a competitive score comparable to the best in-dataset fine-tuning.

4 Discussions

Comparison with state of the art. We acknowledge that while dialectal prompting improves performance over MSA prompting, the magnitude of these improvements varies across dialects. Overall, the improvements range from 1% (Twifil-sent_{Dz}) to 7% (IDAT_{MSA+Mixed}) as shown in the first line of Table 1. Our results beat SOTA models for Sem17_{MSA+Mixed}, Twifil-sent_{Dz}, OCA_{MSA}, TSAC_{Tn} and IDAT_{MSA+Mixed}.

For the other datasets, it is important to mention that direct comparison with reported results is unfair as we are using different experimental settings due to differences in the main task (e.g., Sem18 emotion shared task is a multi-label classification and not a multiclass classification), number of classes (e.g., the latest reported results (Moudjari et al., 2021) on Twifil-emo_{Dz} (resp. Sem18_{MSA+Mixed}) multiclass task use a set of 6 emotion classes and not the whole set composed of 10 (resp. 11) classes as we do here), or evaluation setting (e.g., in the Flood corpus, the train/test split is done by crisis events vs. random in our case). To allow a better comparison, we therefore designed similar baseline transformer models that we fine-tuned on each dataset following the same train/test split ratio across datasets.

Additionally, we conducted statistical significance testing to assess the impact of dialectal prompting compared to MSA prompting. Results reveal that dialect-specific prompting significantly outperforms MSA prompting (at \( p < 0.05 \)) on six out of eight datasets, providing strong empirical support for incorporating dialectal variation in prompt design.

Mixed dialectal data. Our findings reinforce the advantage of using dialectal Arabic for prompting, especially when fine-tuned to align with the dataset’s dominant script. We have however to consider cases where datasets are composed of mixed dialectal data. In this case, selecting a single dominant dialect for prompting can be challenging. However, most dialects share a substantial overlap with MSA, which serves as a linguistic backbone for many Arabic dialects.

Maghrebi dialects. They are among the least represented, as they incorporate more Amazigh and French words. Despite this, they are quite similar which benefits fine-tuning. For example, in TSAC_{Tn}, using Algerian prompts yielded the best results. Similarly, in Sem17_{MSA+Mixed}, MSA performed best before fine-tuning, but post-fine-tuning, Tunisian showed greater consistency, followed by Algerian, especially using AceGPT. This suggests that fine-tuning amplifies dialect-specific nuances, enabling the model to better adapt to unique characteristics of dialectal data. Notably, Maghrebi data within the dataset, which might have been misclassified when using MSA. This is illustrated in the following example from Twifil-emo_{Dz}:

ربی نجيب المجر الهيم أستينا غبنا مغينا غير ضار عاجلنا غير اجلا (May God bring goodness. O Allah, grant us abundant, life-giving, beneficial, and non-harmful rain—soon, not delayed.).

where the model prompted in ENG/MSA predicted "Sadness", possibly influenced by the expressions of supplication. However, the gold label is "Trust", reflecting a hopeful, faith-driven tone common in religious invocations. When prompted in dialect, the model correctly predicted "Trust", indicating that dialectal input facilitated better alignment with the culturally embedded emotional cues of the text.

5 Conclusion

This study examined the ability of 12 open source LLMs to handle Arabic dialects in both Arabic and Arabizi scripts to improve subjective text classification. We highlighted the impact of Arabic-centric LLMs fine-tuning and prompt design on models performances, showing that dialectal prompts generally outperformed MSA and English prompts after fine-tuning. Transfer learning, on the other hand, demonstrated limited effectiveness. While models pre-trained on MSA and Levantine/Gulf showed some ability to adapt to other regionally specific varieties (e.g. Maghrebi), their performances remained significantly lower compared to in-dataset configuration.

We believe our study will open the door to future directions in prompting LLMs for low-resource languages, beyond English or translated prompts.
Acknowledgment

We would like to gratefully thank the anonymous reviewers and the ACs for providing useful suggestions for improving the paper. This work has been supported by the French government as part of France 2030. Funded by the European Union - Next Generation EU as part of the France Relance. It has also been partially supported by DesCartes; the National Research Foundation, Prime Minister’s Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) program. Our work has also benefited from the AI Interdisciplinary Institute ANITI under the Grant agreement n°ANR-23-IACL-0002.

Limitations

This work advances the development of more inclusive NLP models by addressing the challenges posed by under-represented Arabic dialects. We however believe that its implications extend beyond a single language. Indeed, the challenges we addressed (dialectal variation, script differences, and cross-linguistic adaptability) are prevalent in other morphologically rich and diglossic languages, such as Hindi-Urdu. Future research will focus on evaluating language-centric LLMs, expanding the scope to include a broader range of dialects, beyond Arabic. This will provide deeper insights into the impact of script variations and code-switching on LLM performance, further enhancing their adaptability.

We utilized various open-source large language models in our experiments. It is important to acknowledge that these LLMs can exhibit biases and may encounter issues concerning token limit. Therefore, a critical approach should be adopted when interpreting the experimental outcomes.

Given the very good scores achieved by OCA and TSAC datasets, we conducted a contamination check using the Contamination Database.3 Even if we do not find any evidence of overlap with our evaluation datasets, potential data contamination remains and should be acknowledged. Indeed, limited transparency in pre-training datasets and the lack of studies on Arabic dataset contamination leave room for uncertainty which encourages further research in Arabic datasets contamination.

A final limitation is related to the blurred distinction between MSA and dialects and across dialects, as MSA linguistic expressions often appear in daily speech. Our study relied on publicly available datasets that do not explicitly separate MSA, dialects, or non-Arabic samples, which may have caused some misrepresentations. Moreover, the ALDi metric proved unreliable for highly dialectal texts. Developing better metrics for Arabic dialects identification is a timely direction to explore in the future.

Ethics Statement

The data used for conducting the experiments are composed of texts taken from datasets publicly available to the research community.

Very few instances from the datasets we consider (in particular those for emotion and irony detection) may contain offensive or abusive language. In these cases (less than 0.5%), and to comply with the policies, LLMs generally refrain from generating any response prioritizing safe and respectful interactions, therefore the neutral class is assigned by default.

References

Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdallahil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, et al. 2024. Larabench: Benchmarking arabic ai with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 487–520.

AhmedElmogtaba Abdelmoniem Ali Abdelaziz, Ashraf Hatim Elneima, and Kareem Darwish. 2024. LLM-based MT data creation: Dialectal to MSA translation shared task. In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024, pages 112–116, Torino, Italia. ELRA and ICCL.

Kabir Ahuja, Harshita Diddee, Rishav Hada, Milli cent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. MEGA: Multilingual evaluation of generative AI. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4232–4267, Singapore. Association for Computational Linguistics.

AI@Meta. 2024. Llama 3 model card.

Alaa Alharbi and Mark Lee. 2019. Crisis detection from arabic tweets. In Proceedings of the 3rd Workshop on Arabic Corpus Linguistics, pages 72–79.

3https://huggingface.co/spaces/CONDA-Workshop/Data-Contamination-Database
Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9.

Patrick Bareiß, Roman Klinger, and Jeremy Barnes. 2024. English prompts are better for nli-based zero-shot emotion classification than target-language prompts. In Companion Proceedings of the ACM Web Conference 2024, WWW ’24, page 1318–1326, New York, NY, USA. Association for Computing Machinery.

Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonnell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. GPT-NeoX-20B: An open-source autoregressive language model. In Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models, pages 95–136, virtual+Dublin. Association for Computational Linguistics.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Samuel Cahyawijaya, Holy Lovenia, and Pascale Fung. 2024. LLMs are few-shot in-context low-resource language learners. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 405–433, Mexico City, Mexico. Association for Computational Linguistics.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Kamel Gaanoun, Abdou Mohamed Naira, Anass Al-lak, and Imade Benelallam. 2024. Darijabert: a step forward in nlp for the written moroccan dialect. International Journal of Data Science and Analytics, pages 1–13.

Bilal Ghanem, Jihen Karoui, Farah Benamara, Véronique Moriceau, and Paolo Rosso. 2019. Idat@fire2019: Overview of the track on irony detection in arabic tweets. In Mehta P., Rosso P., Majumder P., Mitra M. (Eds.) Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2019). CEUR Workshop Proceedings. In: CEUR-WS.org, Kolkata, India, December 12-15, pages 10–13.

Haoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023a. Not all languages are created equal in LLMs: Improving multilingual capability by cross-lingual-thought prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12365–12394, Singapore. Association for Computational Linguistics.

Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2024. AceGPT, localizing large language models in Arabic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8139–8163, Mexico City, Mexico. Association for Computational Linguistics.

Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Ziche Liu, et al. 2023b. Acegpt, localizing large language models in arabic. arXiv preprint arXiv:2309.12053.

Go Inoue, Bashar Alhafni, Nurpeis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Amr Keleg, Walid Magdy, and Sharon Goldwater. 2024. Estimating the level of dialectness predicts inter-annotator agreement in multi-dialect Arabic datasets. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 778–789, Bangkok, Thailand. Association for Computational Linguistics.

Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoon Kim, Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee, and Sunghun Kim. 2023. Solar 10.7b: Scaling large language models with simple yet effective depth upscaling. Preprint, arXiv:2312.15166.

Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Al-
mubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics ACL 2024, pages 5622–5640, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Pingsheng Liu, Zhengjie Huang, Xiechi Zhang, Linlin Wang, Gerard de Melo, Xin Lin, Liang Pang, and Liang He. 2023. A disentangled-attention based framework with persona-aware prompt learning for dialogue generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13255–13263.

Zhiwei Liu, Kailai Yang, Qianqian Xie, Tianlin Zhang, and Sophia Ananiadou. 2024. Emolms: A series of emotional large language models and annotation tools for comprehensive affective analysis. KDD ’24, page 5487–5496, New York, NY, USA. Association for Computing Machinery.

Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, and Iryna Gurevych. 2024. Are emergent abilities in large language models just context learning? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5098–5139, Bangkok, Thailand. Association for Computational Linguistics.

Salima Medhaffar, Fethi Bougares, Yannick Estève, and Lamia Hadrich-Belguith. 2017. Sentiment analysis of tunisian dialects: Linguistic resources and experiments. In Proceedings of the third Arabic natural language processing workshop, pages 55–61.

Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. SemEval-2018 task 1: Affect in tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 1–17, New Orleans, Louisiana. Association for Computational Linguistics.

Leila Moudjari, Karima Akli-Astouati, and Farah Benamara. 2020. An Algerian corpus and an annotation platform for opinion and emotion analysis. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1202–1210, Marseille, France. European Language Resources Association.

Leila Moudjari, Farah Benamara, and Karima Akli-Astouati. 2021. Multi-level embeddings for processing arabic social media contents. Computer Speech & Language, 70:101240.

Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shamnur Absar Chowdhury, and Firoj Alam. 2025. AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, Abu Dhabi, UAE. Association for Computational Linguistics.

Xuan-Phi Nguyen, Mahani Aljunied, Shafiq Joty, and Lidong Bing. 2024. Democratizing LLMs for low-resource languages by leveraging their English dominant abilities with linguistically-diverse prompts. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3501–3516, Bangkok, Thailand. Association for Computational Linguistics.

Viktória Ondrejová and Marek Šuppa. 2024. Can LLMs handle low-resource dialects? a case study on translation and common sense reasoning in šariš. In Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024), pages 130–139, Mexico City, Mexico. Association for Computational Linguistics.

Benedikt Perak, Slobodan Beliga, and Ana Meštrović. 2024. Incorporating dialect understanding into LLM using RAG and prompt engineering techniques for causal commonsense reasoning. In Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024), pages 220–229, Mexico City, Mexico. Association for Computational Linguistics.

Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in twitter. In Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017), pages 502–518, Vancouver, Canada. Association for Computational Linguistics.

Mohammed Rushdi-Saleh, M Teresa Martín-Valdivia, L Alfonso Ureña-López, and José M Pereira-Ortega. 2011. Oca: Opinion corpus for arabic. Journal of the American Society for Information Science and Technology, 62(10):2045–2054.

Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Alham Fikri Aji, Zhengzhong Liu, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Preslav Nakov, Timothy Baldwin, and Eric Xing. 2023. Jais and jaischat: Arabic-centric foundation and instruction-tuned open generative large language models. Preprint, arXiv:2308.16149.

Bhavani Shankar, Preethi Jyothi, and Pushpak Bhattacharyya. 2024. In-context mixing (ICM): Code-mixed prompts for multilingual LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4162–4176, Bangkok, Thailand. Association for Computational Linguistics.

Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023. Text classification via large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8990–9005, Singapore. Association for Computational Linguistics.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. Transactions on Machine Learning Research. Survey Certification.

Mohammad Ali Yaghan. 2008. Arabizi: A contemporary style of arabic slang. Design issues, 24:39–52.

Chiyu Zhang, Khai Doan, Qisheng Liao, and Muhammad Abdul-Mageed. 2023. The skipped beat: A study of sociopragmatic understanding in LLMs for 64 languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2630–2662, Singapore. Association for Computational Linguistics.

Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Pan, and Lidong Bing. 2024. Sentiment analysis in the era of large language models: A reality check. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3881–3906, Mexico City, Mexico. Association for Computational Linguistics.

A Subjective Datasets

A.1 Detailed Description

We detail below the datasets used in this study.

OCA_{MSA} (Rushdi-Saleh et al., 2011): An MSA movie reviews dataset containing 500 reviews split into 250 positive and 250 negative instances collected from different Arabic web pages and blogs and automatically annotated using author’s rating.

Sem17_{MSA+Mixed} (Rosenthal et al., 2017): The dataset was used in the SemEval2017 Task 4, sub-task A on identifying the overall sentiment of a tweet. The corpus is split into train and test sets containing respectively 3, 355 and 6, 100 tweets. The corpus does not target a given dialect.

Twifil-sent_{DZ} and Twifil-emo_{DZ} (Moudjari et al., 2020): This is the largest Algerian dataset of about 11,000 tweets where each tweet has been manually annotated by crowd-sourcing for sentiment analysis (Twifil-sent_{DZ}) (positive, negative and neutral using a majority vote), as well as emotion (Twifil-emo_{DZ}) relying on a taxonomy of 10 emotions (Happiness, Anger, Disgust, Fear, Sadness, Surprise, Trust, Love, Anticipation, and Neutral).

TSAC_{Tn} (Medhaffar et al., 2017): A Tunisian Sentiment Analysis Corpus of about 17k Facebook comments manually annotated into 8, 215 positive and 8, 845 negative.

Sem18_{MSA+Mixed} (Mohammad et al., 2018): The SemEval-2018 Task 1 Affect in Tweets about predicting emotion intensity. We make use of the Emotion Classification (E-C) task dataset4 which contains tweets collected in 2017 and manually annotated into 11 emotions categories (Anger, Anticipation, Disgust, Fear, Joy, Love, Optimism, Pessimism, Sadness, Surprise, and Trust).

IDAT_{MSA+Mixed} (Ghanem et al., 2019): It contains tweets about different political issues and events related to the Middle East that was held during the years 2011 to 2018. Tweets are written with formal Arabic (MSA) and different Arabic language varieties: Egypt, Gulf, Levantine, and Maghrebi dialects. Each tweet has been manually annotated into Ironic or Not-Ironic.

Flood_{Lv+Gl} (Alharbi and Lee, 2019), an Arabic-Twitter-Corpus-for-Flood-Detection The corpus includes 4,037 human-labeled Arabic Twitter messages for four high-risk flood events that occurred in 2018: Jordan floods, Kuwait floods, Qurayyat floods and Al-Lith floods. The selected events took place in different areas of the Arab world: Jordan, Kuwait, northern Saudi Arabia and western Saudi Arabia, respectively. The tweets were labeled based on relatedness to the crisis and information type (Other useful information, Affected individuals, Caution and advice, Donations and volunteering, Infrastructure and utilities damage, Not applicable, Sympathy and emotional support).

A.2 Dataset Statistics

Table 3 shows statistics about the datasets we consider here. Classification tasks are either binary or multiclass with a maximum of 11 classes for emotion detection. Four datasets target only dialects: Twifil-sent_{DZ}, TSAC_{Tn}, Twifil-emo_{DZ}, and Flood_{Lv+Gl}, while three are a mix between various dialects and MSA. We also consider one dataset in MSA only: OCA_{MSA}.

Figure 1 further analyzes the characteristics of our datasets in terms of the used script: Arabic, or Latin/Arabizi and mixed context in the training sets. We observe the presence of Latin and mixed script in all datasets, Flood_{Lv+Gl} having the highest number of instances in Arabic script.

4https://huggingface.co/datasets/SemEvalWorkshop/sem_eval_2018_task_1
<table>
  <tr>
    <th>Dataset</th>
    <th>Sem17<sub>MSA+Mixed</sub></th>
    <th>Twifil-sent<sub>Dz</sub></th>
    <th>OCA<sub>MSA</sub></th>
    <th>TSAC<sub>Tn</sub></th>
    <th>Sem18<sub>MSA+Mixed</sub></th>
    <th>Twifil-emo<sub>Dz</sub></th>
    <th>IDAT<sub>MSA+Mixed</sub></th>
    <th>Flood<sub>Lv+Gl</sub></th>
  </tr>
  <tr>
    <td>Train</td>
    <td>21.20</td>
    <td>19.28</td>
    <td>32.40</td>
    <td>27.67</td>
    <td>51.24</td>
    <td>19.15</td>
    <td>38.17</td>
    <td>16.66</td>
  </tr>
  <tr>
    <td>Test</td>
    <td>18.32</td>
    <td>20.40</td>
    <td>54.40</td>
    <td>29.03</td>
    <td>52.87</td>
    <td>19.81</td>
    <td>37.04</td>
    <td>16.59</td>
  </tr>
</table>

Table 2: ALDi scores in our datasets.

<table>
  <tr>
    <th rowspan="2">Task</th>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Source</th>
    <th colspan="2">Classes</th>
    <th rowspan="2">Train</th>
    <th rowspan="2">Test</th>
  </tr>
  <tr>
    <th>SA</th>
    <th>ER</th>
  </tr>
  <tr>
    <td rowspan="4">SA</td>
    <td>OCA<sub>MSA</sub></td>
    <td>IMDB</td>
    <td>2 (positive, negative)</td>
    <td></td>
    <td>400</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Sem17<sub>MSA+Mixed</sub></td>
    <td>tweets</td>
    <td>3 (positive, negative, neutral)</td>
    <td></td>
    <td>3,355</td>
    <td>6,100</td>
  </tr>
  <tr>
    <td>Twifil-sent<sub>Dz</sub></td>
    <td>tweets</td>
    <td>3 (positive, negative, neutral)</td>
    <td></td>
    <td>7,144</td>
    <td>2,069</td>
  </tr>
  <tr>
    <td>TSAC<sub>Tn</sub></td>
    <td>facebook</td>
    <td>2 (positive, negative)</td>
    <td></td>
    <td>13,665</td>
    <td>1,981</td>
  </tr>
  <tr>
    <td rowspan="2">ER</td>
    <td>Sem18<sub>MSA+Mixed</sub></td>
    <td>tweets</td>
    <td>11 (pessimism, fear, optimism, trust, anticipation, joy, anger, sadness, disgust, love, surprise)</td>
    <td></td>
    <td>4,034</td>
    <td>1,009</td>
  </tr>
  <tr>
    <td>Twifil-emo<sub>Dz</sub></td>
    <td>tweets</td>
    <td>10 (happiness, anger, disgust, fear, sadness, Surprise, Trust, Love, Anticipation, and Neutral)</td>
    <td></td>
    <td>3,885</td>
    <td>1,148</td>
  </tr>
  <tr>
    <td>ID</td>
    <td>IDAT<sub>MSA+Mixed</sub></td>
    <td>tweets</td>
    <td>2 (ironic, non-ironic)</td>
    <td></td>
    <td>4,024</td>
    <td>1,006</td>
  </tr>
  <tr>
    <td>CM</td>
    <td>Flood<sub>Lv+Gl</sub></td>
    <td>tweets</td>
    <td>7 (Affected individuals, Caution and advice, Donations and volunteering, Infrastructure and utilities damage. Not applicable. Other useful information, Sympathy and emotional support)</td>
    <td></td>
    <td>3,223</td>
    <td>806</td>
  </tr>
</table>

Table 3: Datasets used in this study.

We also analyzed datasets in term of dialect/MSA distribution. Table 2 presents the average ALDi score for each train and test set. Manual inspection revealed inconsistencies, particularly with non-Arabic scripts (e.g., Arabizi), where ALDi fails to reflect dialectal content accurately. For instance:

• MSA example written in arabizi: "hatha al-makan jamil jidan" (This place is very beautiful) → ALDi=0.116.

• Tunsian example in Arabizi: "chbih yet-haz wyet7at akeka 5o zina esghir hhhhhhhh mas5fo" (What’s with Zina’s little brother acting all hyped up? Hahaha, poor thing.") → ALDi=0.0522, which incorrectly suggests MSA.

These inconsistencies indicate that ALDi scores have to be improved: If MSA was very high in the datasets, then MSA prompts should have given the best results, which is not the case. However, the ALDi scores reinforce our hypothesis that fine-tuning elevates the model’s ability to better adapt to the unique characteristics of dialectal data.

A.3 Train/Test Split
We used the canonical train/test splits provided by the original papers for most datasets to ensure consistency with prior research (see Table 3). To preserve split integrity, no additional shuffling or resampling was performed. For datasets without predefined splits (OCA and Flood), we applied train/test split with stratification and a fixed seed (125) to ensure balanced classes and reproducibility.

B Models

B.1 Models Description
We make use of the following open source models:

– AraBERTv2 (Antoun et al., 2020): is a family of pre-trained transformer-based language models specifically designed for Arabic text based on BERT architecture and is optimized for various Arabic language processing tasks, such as text classification, named entity recognition, and sentiment analysis. We tested different version, and we report the best results with bert-base-arabert-v02 and bert-base-arabert-v02-twitter(fine-tuned on 60M Arabic tweets).

– CAMeLBERT (Inoue et al., 2021). Another collection of pre-trained BERT-based models tailored for Arabic NLP tasks. The results reported here are based on bert-base-arabic-camelbert-mix, which was pre-trained on a diverse corpus comprising Modern Standard Arabic (MSA), dialectal Arabic, and Classical Arabic.

– DarijaBERT (Gaanoun et al., 2024) is a BERT-based model designed for the Moroccan Arabic dialect, “Darija.” It follows the BERT-base architecture without the Next Sentence Prediction
(NSP) objective. This Arabizi-specific version of DarijaBERT was trained on 4.6 million sequences of Darija written in Latin letters, using a dataset sourced from YouTube comments. Notably, the Arabizi model outperformed the other two variants from the same family.

– AceGPT (Huang et al., 2023b) is a collection of Arabic-centric LLMs specifically designed to advance Arabic language understanding and generation. The models are developed using three key strategies: (1) localized pre-training on extensive Arabic corpora, (2) localized fine-tuning with Arabic-specific questions paired with GPT-4-generated Arabic responses, and (3) reinforcement learning with AI feedback (RLAIF) to enhance performance. In this work, we utilize AceGPT_8b and AceGPT-chat_8b, which are Arabic-focused LLMs built upon the LLaMA-3 architecture. These models are optimized to handle both MSA and Arabic dialects, making them well-suited for various Arabic natural language processing tasks, such as text classification, generation, and understanding.

– Jais-Family (Sengupta et al., 2023). It consists of a series of bilingual English-Arabic LLMs developed to excel in Arabic while maintaining strong English capabilities. These models are designed to handle various tasks, including text generation, comprehension, and summarization.

– LLaMA3_8b and LLaMA3-Instruct_8b (Touvron et al., 2023; AI@Meta, 2024) are part of the Meta LLaMA 3 family, a collection of pretrained and instruction-tuned generative LLMs. Pretrained on over 15 trillion tokens from publicly available sources. Instruction-tuned versions are optimized for dialogue tasks.

– Mistral-7b and Mistral-Instruct7b (Jiang et al., 2023). Mistral-7b is a 7-billion parameter dense transformer trained on a diverse corpus. Instruction-tuned variants of Mistral-7b are further optimized for chat-based and instruction-following tasks.

– SOLAR-inst_{10.7b} (Kim et al., 2023) The model was trained on publicly available datasets. It is instruction-tuned using supervised fine-tuning on a mixture of task-oriented prompts to perform instruction-following tasks such as question answering and dialogue.

– EmoLlama_{7b}, EmoLlama-Instruct_{7b} and EmoLlama-chat_{7b} are part of the EmoLLMs project (Liu et al., 2024), a collection of models designed for comprehensive affective analysis. This project focuses on the emotional understanding of text, offering models that can classify sentiments, detect emotional intensity, and predict sentiment strength. These variants are fine-tuned on the Meta LLaMA2 models with 234K data based on various classification and regression tasks.

B.2 Hyper-parameters

All transformer models have been trained using the Adam optimizer with an epsilon value of \(1e-8\), a fixed batch size of 16 for training and 128 for validation, a learning rate of \(2e-5\), and a sequence length of 128 tokens. This configuration is designed to balance efficient optimization with stable performance across various tasks, as the Adam optimizer’s epsilon parameter helps maintain stability by preventing division by very small numbers in the moment estimates.

The number of epochs varies across different datasets (see Table 4): for AraBERTv2,
OCA_{MSA} and Sem18_{MSA+Mixed}, the models were trained for 7 epochs, while IDAT_{MSA+Mixed}, Sem17_{MSA+Mixed}, and Twifil-sent_{D_2} were trained for 5 epochs each and TSAC_{Tn} for 6. For CAMeL-BERT, Flood_{Lv+Gt} was trained for 4 epochs, and Twifil-emo_{D_2} for 7 epochs. This configuration ensures stable performance and efficient optimization across a wide range of tasks.

LLMs non fine-tuned hyper-parameters are shown in Table 5. To reduce the verbosity and keep the models focused on our scheme, we set the temperature to 0 to encourage deterministic responses. However, for the Llama models, which do not accept a temperature of 0, we adjusted it to 0.001.

For fine-tuning, we applied Quantized Low-Rank Adaptation (QLoRA) (Dettmers et al., 2024) to each of the previously mentioned open-source models (the hyper-parameters are summarized in Table 6). The choice of QLoRA is primarily due to its efficiency in fine-tuning large models on resource-constrained hardware while maintaining strong performance (we used a single GPU with 80GB of VRAM).

Each model was fine-tuned on the full training set of each dataset, and performance was assessed on a development set. The results are reported on the test set.

C Prompts

Figure 2 presents the prompting formats employed throughout our experiments, illustrating variations across English, MSA, and several Arabic dialects in both Arabic script and Arabizi. The first frame outlines the general prompt template structure applied consistently across instruction-tuning, inference, and evaluation stages.

We adopt a one-class-shot In-Context Learning prompting to control hallucination, using a concise prompting template with a single example of a single class and a 0 temperature, as extended prompts did not yield better results which was pointed out in other studies (Abdelaziz et al., 2024). This approach provided competitive performance on par with transformers across various tasks and datasets.

We provide the best performing prompts, namely fine-tuned one-class-shot ICL. We only illustrate them for sentiment analysis per language/dialect as they are the same for the emotion and irony detection, the only difference being the role and task

Prompting Template Used for Fine-Tuning


System: A) {prompt_task}




User: B) {user_text}




Assistant: C) {call_for_action}: {answer}



English Prompt
A) You are an Arabic sentiment text detector. Classify the text as positive, negative, or neutral. Return only the label.
B) Give only the label for this text: [text]
C) This text is mostly: [label]

MSA Prompt
(A) انت كاشف المشاعر في النصوص العربية. حدد التصنيف: محايد، إيجابي، أو سلبي.
(B) أعط فقط الشعور بهذا النص: [text]
(C) هذا النص في الغالب: [label]

Levantine (Arabic Script)
(A) إنت مُصنِّف مشاعر للنصوص، بتسلسل نص عربى وتعطيه تصنيف: محايد، إيجابي، أو سلبي.
(B) أعطيني بس التصنيف لهالنص: [text]
(C) هالنص علاقته: [label]

Levantine (Arabizi Script)
A) Enta mosannif masha3er lilnosooos, btistalim nas 3arabi w bta3ti tasnif: neutral, positive aw negative.
B) A3tini bas eltasnif la halnas: [text]
C) Hal nass 3al aghlab: [label]

Algerian (Arabizi Script)
A) Ntaya déetecteur ta3 les sentiments fi les texts ta3 l3rbiya w tgeneri l'étiquette: neutre, positive wla negative.
B) A3tini ghir le label ta3 had le texte: [text]
C) Had le texte ghalibane: [label]

Tunisian (Arabic Script)
(A) انت بديكتور تاء المشاعر في النصوص، وتعطي تصنيف: محايد، إيجابي، أو سلبي.
(B) أعطيني غير التصنيف تاء هاد التكسيت: [text]
(C) هذا التكسيت في الغالب: [label]

Figure 2: Prompt Design Across Dialects.
<table>
  <tr>
    <th>Dataset</th>
    <th>Sem17<sub>MSA+Mixed</sub></th>
    <th>Twifil-sent<sub>Dz</sub></th>
    <th>OCA<sub>MSA</sub></th>
    <th>TSAC<sub>Tn</sub></th>
    <th>Sem18<sub>MSA+Mixed</sub></th>
    <th>Twifil-emo<sub>Dz</sub></th>
    <th>IDAT<sub>MSA+Mixed</sub></th>
    <th>Flood<sub>Lv+Gl</sub></th>
  </tr>
  <tr>
    <td>Epoch</td>
    <td>5</td>
    <td>5</td>
    <td>7</td>
    <td>6</td>
    <td>7</td>
    <td>7</td>
    <td>5</td>
    <td>4</td>
  </tr>
</table>

Table 4: Epoch number for each dataset.

<table>
  <tr>
    <th>Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>num_return_sequences</td>
    <td>1</td>
  </tr>
  <tr>
    <td>top_p</td>
    <td>1</td>
  </tr>
  <tr>
    <td>max_new_tokens</td>
    <td>10</td>
  </tr>
  <tr>
    <td>temperature</td>
    <td>0</td>
  </tr>
  <tr>
    <td>do_sample</td>
    <td>False</td>
  </tr>
</table>

Table 5: Parameters used for testing LLM models.

<table>
  <tr>
    <th>Parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>per_device_train_batch_size</td>
    <td>8</td>
  </tr>
  <tr>
    <td>per_device_eval_batch_size</td>
    <td>8</td>
  </tr>
  <tr>
    <td>optimizer</td>
    <td>paged_adamw_32bit</td>
  </tr>
  <tr>
    <td>learning rate</td>
    <td>2e-4</td>
  </tr>
  <tr>
    <td>logging_steps</td>
    <td>1</td>
  </tr>
  <tr>
    <td>gradient_accumulation_steps</td>
    <td>1</td>
  </tr>
  <tr>
    <td>epochs</td>
    <td>1</td>
  </tr>
  <tr>
    <td>fp16</td>
    <td>True</td>
  </tr>
  <tr>
    <td>Lora attention dimension (rank)</td>
    <td>64</td>
  </tr>
  <tr>
    <td>lora_alpha</td>
    <td>16</td>
  </tr>
  <tr>
    <td>lora_dropout</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>load_in_4bit</td>
    <td>True</td>
  </tr>
  <tr>
    <td>bnb_4bit_quant_type</td>
    <td>nf4</td>
  </tr>
  <tr>
    <td>torch_dtype</td>
    <td>bfloat16</td>
  </tr>
</table>

Table 6: Parameters used for fine-tunning the models with QLoRA.

description. For crisis management, we additionally enhance the prompts with the definition of each label.

D Detailed Results

D.1 Results per LLM and Tasks

Table 11 details all our results per LLMs for each dataset, langue/dialect and its associated scripts. A more synthetic view of the results is given in Table 7 and its corresponding plot in Figure 3 to visually compare performance drops before and after fine-tuning, across different models and datasets. Table 8 on the other hand gives a comparison of the best fine-tuned model across all datasets before and after removing MSA samples following their ALDi scores.

It is interesting to note that fine-tuned LLMs outperform transformers in all tasks, except crisis management and emotion detection in Twifil-emo<sub>Dz</sub> with a small decrease of around 3%. For 6 datasets out of 8, the best performance was obtained with AraBERTv2, CAMeLBERT being more productive for Flood<sub>Lv+Gl</sub>, and Twifil-emo<sub>Dz</sub> datasets. DarijaBert was far behind the first two models. For example, for Twifil-emo<sub>Dz</sub> F1-score=13.74 (vs. 38.06) and Twifil-sent<sub>Dz</sub> F1-score=65.87 (vs. 74.46).

The results of OCA<sub>MSA</sub> and TSAC<sub>Tn</sub> are quite intriguing. We believe this can be attributed to the fact that both datasets (IMDB movie reviews and Facebook comments respectively) contain longer texts compared to the others, which are primarily sourced from Twitter. The increased text length likely provided the models with more context to infer the class. Additionally, both datasets focus on binary classification, which may have further simplified the task by reducing the complexity of decision boundaries.

Another interesting trend is observed: datasets composed primarily of texts written in Arabic script tend to achieve the best results when prompted in Arabizi (e.g., TSACTn, Twifil-emoDz). We hypothesize that this may be due to the fact that LLMs are predominantly trained on English data, and multilingual prompting helps bridge the gap and enhance performance (Huang et al., 2023a). This hypothesis is further supported by the observation that Lvabz contains English words, whereas DZabz incorporates French, highlighting the impact of linguistic overlap in model adaptation.

Finally, small instruction and chat-tuned LLMs fine-tuned on specific tasks can outperform larger models, confirming recent findings (Zhang et al., 2023; Lu et al., 2024). This challenges the assumption that performance scales smoothly with model size, as earlier studies suggested (Black et al., 2022; Wei et al., 2022).

D.2 Transfer Learning Results

Tables 9 and 10 present the best results for both Cross-dataset transfer learning and Fine-tuning transfer, with the latter highlighted in yellow.

We observe that when a dataset performs best with a specific script, transfer learning from a model fine-tuned with a different script tends to yield lower results. For example, Twifil-sent<sub>Dz</sub> achieves its highest performance with AceGPT8b
<table>
  <tr>
    <th></th>
    <th>Sem17<sub>MSA+ Mixed</sub></th>
    <th>Twili-sent<sub>Dz</sub></th>
    <th>OCA<sub>MSA</sub></th>
    <th>TSAC<sub>Tn</sub></th>
    <th>Sem18<sub>MSA+ Mixed</sub></th>
    <th>Twilli-emo<sub>Dz</sub></th>
    <th>IDAT<sub>MSA+ Mixed</sub></th>
    <th>Flood<sub>_v4+GI</sub></th>
  </tr>
  <tr>
    <th>Fine-tuned ?</th>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>x</td>
    <td>✓</td>
    <td>x</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Emollama7</td>
    <td>Eng</td>
    <td>Eng</td>
    <td>Eng</td>
    <td>LV<sub>var</sub></td>
    <td>MSA</td>
    <td>MSA</td>
    <td>Eng</td>
    <td>LV<sub>msa</sub></td>
    <td>Eng</td>
    <td>LV<sub>msa</sub></td>
    <td>TD</td>
  </tr>
  <tr>
    <td>Emollama-chat7</td>
    <td>Eng</td>
    <td>Eng</td>
    <td>Eng</td>
    <td>LV<sub>en</sub></td>
    <td>DZA<sup>*</sup></td>
    <td>Eng</td>
    <td>DZA<sup>*</sup></td>
    <td>Eng</td>
    <td>Eng</td>
    <td>LV<sub>dza</sub></td>
    <td>Eng</td>
  </tr>
  <tr>
    <td>Emollama-chat16</td>
    <td>Eng</td>
    <td>Eng</td>
    <td>Eng</td>
    <td>DZA</td>
    <td>MSA</td>
    <td>Eng</td>
    <td>LV<sub>msa</sub></td>
    <td>Eng</td>
    <td>DZA</td>
    <td>LV<sub>dza</sub></td>
    <td>MSA*</td>
  </tr>
  <tr>
    <td>Llama3<sub>MPA</sub></td>
    <td>Eng</td>
    <td>Eng</td>
    <td>Eng</td>
    <td>MSA</td>
    <td>ENG</td>
    <td>ENG</td>
    <td>ENG</td>
    <td>MSA</td>
    <td>ENG</td>
    <td>Eng</td>
    <td>ENG</td>
  </tr>
  <tr>
    <td>Llama3-inst50</td>
    <td>MSA</td>
    <td>LV<sub>msa</sub></td>
    <td>LV<sub>var</sub></td>
    <td>MSA*</td>
    <td>DZ<sup>*</sup></td>
    <td>ENG</td>
    <td>ENG</td>
    <td>LiV<sub>msa</sub></td>
    <td>LV<sub>msa</sub></td>
    <td>LVdza</td>
    <td>MSA</td>
  </tr>
  <tr>
    <td>Mistral-B</td>
    <td>DZA<sub>ari</sub></td>
    <td>DZA<sub>ari</sub></td>
    <td>ZDA<sup>*</sup></td>
    <td>LV<p_{var}}</td>
    <td>MSA</td>
    <td>DZA<sup>*</sup></td>
    <td>DZA<sup>*</sup></td>
    <td>LV<p_{var}}(DZA<sup>*</sup>)</td>
    <td>DZA<sup>*</sup></td>
    <td>Thar</td>
    <td>LV</td>
  </tr>
  <tr>
    <td>Mistral Inst516</td>
    <td>DZA<sub>ari</sub></td>
    <td>DZA<sub>ari</sub></td>
    <td>DZA<sup>*</sup></td>
    <td>ENG</td>
    <td>DZA<sup>*</sup></td>
    <td>DZA<sup>*</sup></td>
    <td>ENG</td>
    <td>DZA<sup>*</sup></td>
    <td>ENG</td>
    <td>ENG</td>
    <td>LV<p_{var}</td>
  </tr>
  <tr>
    <td>SOLAR-inst107</td>
    <td>ENG</td>
    <td>LV<p_{var}}</td>
    <td>LV<p_{var}}</td>
    <td>MSA</td>
    <td>DZA<sup>*</sup></td>
    <td>ENG</td>
    <td>LV<p_{var}}</td>
    <td>DZA<sup>*</sup></td>
    <td>ENG</td>
    <td>Thar</td>
    <td>ENG</td>
    <td>LV<p_{var}}</td>
  </tr>
  <tr>
    <td>AceGPT-10</td>
    <td>DZA<sup>*</sup></td>
    <td>LV<p_{var}}</td>
    <td>MSA</td>
    <td>ENG</td>
    <td>DZA<sup>*</sup></td>
    <td>LV<p_{var}}</td>
    <td>ENG</td>
    <td>REP</td>
    <td>ENG</td>
    <td>REV</td>
    <td>ENG</td>
    <td>DZA<sup>*</sup></td>
  </tr>
  <tr>
    <td>AceGPT-chal650</td>
    <td>MSA</td>
    <td>Thar</td>
    <td>DZA<sup>*</sup></td>
    <td>DZA<sup>*</sup></td>
    <td>ENG</td>
    <td>Thar</td>
    <td>ENG</td>
    <td>MSA</td>
    <td>ENG</td>
    <td>DZA<sup>*</sup></td>
    <td>MSA</td>
    <td>MSA</td>
  </tr>
  <tr>
    <td>Jais-family10</td>
    <td>ENG</td>
    <td>MSA</td>
    <td>ENG</td>
    <td>LV<p_{var}}</td>
    <td>MSA</td>
    <td>LV<p_{var}}</td>
    <td>MSA</td>
    <td>LV<p_{var}}</td>
    <td>MSA</td>
    <td>Thar</td>
    <td>MSA</td>
    <td>Eng</td>
  </tr>
</table>

Table 7: Results per models showing outperforming language/script before and after fine-tuning in different colors: Grey for English, pink for MSA, green for Algerian Arabizi, orange for Tunisian Arabic, light and dark blue for Levantine Arabic and Arabizi respectively. Each entry represents the best performance across different prompting methods for each model. Stars in the cells indicate best configuration per dataset.

<table>
  <tr>
    <th>TASK</th>
    <th>SA</th>
    <th>ER</th>
    <th>ID</th>
    <th>CM</th>
  </tr>
  <tr>
    <th>ModelDatasets</th>
    <th>Sem17<sub>MSA+ Mixed</sub></th>
    <th>Twili-sent<sub>Dz</sub></th>
    <th>OCA<sub>MSA</sub></th>
    <th>TSAC<sub>Tn</sub></th>
    <th>Sem18<sub>MSA+ Mixed</sub></th>
    <th>Twilli-emo<sub>Dz</sub></th>
    <th>IDAT<sub>MSA+ Mixed</sub></th>
    <th>Flood<sub>_v4+GI</sub></th>
  </tr>
  <tr>
    <td>Best macro-F1 scores (all instances)</td>
    <td>95.24</td>
    <td>89.32</td>
    <td>83.94</td>
    <td>97.34</td>
    <td>90.64</td>
    <td>55.51</td>
    <td>90.67</td>
    <td>62.5</td>
  </tr>
  <tr>
    <td>% of instances with ALD≥0</td>
    <td>83.51</td>
    <td>82.22</td>
    <td>87.00</td>
    <td>71.07</td>
    <td>94.00</td>
    <td>94.41</td>
    <td>119.78</td>
    <td>118.61</td>
  </tr>
  <tr>
    <td>Best macro-F1 scores for instances with ALD≥0</td>
    <td>84.49</td>
    <td>89.79</td>
    <td>100.00</td>
    <td>99.33</td>
    <td>31.21</td>
    <td>35.17</td>
    <td>93.28</td>
    <td>64.35</td>
  </tr>
  <tr>
    <td>% of instances with ALD<0.1</td>
    <td>67.24</td>
    <td>76.62</td>
    <td>91</td>
    <td>99.36</td>
    <td>31.71</td>
    <td>42.42</td>
    <td>43.54</td>
    <td>65.14</td>
  </tr>
  <tr>
    <td>Best macro-F1 scores on instances with ALD<0.1</td>
    <td>80.68</td>
    <td>89.17</td>
    <td>100.00</td>
    <td>36.4</td>
    <td>24.83</td>
    <td>32.31</td>
    <td>90.74</td>
    <td>44.83</td>
  </tr>
</table>

Table 8: Comparison of the best fine-tuned model across all datasets before and after removing MSA samples following their ALDi scores set to two thresholds: \( ALDi > 0 \) and \( ALDi > 0.1 \) indicating that a sample is expected to be in dialect.

![Performance comparison of LLMs across datasets: Baseline vs. Pre- and Post-Fine-Tuning.](page_1092_1130_567_312.png)

Figure 3: Performance comparison of LLMs across datasets: Baseline vs. Pre- and Post-Fine-Tuning.
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Best in-dataset FT</th>
    <th colspan="3">Best dialect transfer</th>
  </tr>
  <tr>
    <th>Sem17</th>
    <th>TwiIh-emo</th>
    <th>TSAC</th>
  </tr>
  <tr>
    <td>Sem17<sub>MSA+Mixed</sub></td>
    <td>85.23 (Tn<sub>ar</sub>)</td>
    <td><b>43.58 (Lv<sub>abz</sub>)</b></td>
    <td>18.61 (MSA)</td>
    <td>18.61 (Lv<sub>abz</sub>)</td>
  </tr>
  <tr>
    <td>TwiIh-Sent<sub>en</sub></td>
    <td>87.13 (DZ<sub>abz</sub>)</td>
    <td>35.13 (Lv<sub>abz</sub>)</td>
    <td>28.62 (Lv<sub>abz</sub>)</td>
    <td>17.82</td>
  </tr>
  <tr>
    <td>TSAC<sub>Tn</sub></td>
    <td>99.34 (DZ<sub>abz</sub>)</td>
    <td>54.48 (DZ<sub>abz</sub>)</td>
    <td>55.61 (DZ<sub>abz</sub>)</td>
    <td>45.93 (MSA)</td>
  </tr>
</table>

Table 9: Sentiment Analysis transfer learning results. Cross-dataset: Best transfer learning vs. in-dataset performances in terms of macro F1-scores. Fine-tuning transfer: highlighted in yellow.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Best in-dataset FT</th>
    <th colspan="2">Best dialect transfer</th>
  </tr>
  <tr>
    <th>Sem18</th>
    <th>TwiIh-emo</th>
  </tr>
  <tr>
    <td>Sem18<sub>MSA+Mixed</sub></td>
    <td>32.41 (Lv<sub>abz</sub>)</td>
    <td><b>22.01 (DZ<sub>abz</sub>)</b></td>
    <td>6.56 (Tn<sub>ar</sub>)</td>
  </tr>
  <tr>
    <td>TwiIh-emo<sub>en</sub></td>
    <td>35.81 (Lv<sub>abz</sub>)</td>
    <td>35.13 (Lv<sub>abz</sub>)</td>
    <td><b>9.78 (DZ<sub>abz</sub>)</b></td>
  </tr>
</table>

Table 10: Emotion detection transfer learning results. Cross-dataset: Best transfer learning vs. in-dataset performances in terms of macro F1-scores. Fine-tuning transfer: highlighted in yellow.

using DZ<sub>abz</sub>. However, in Cross-dataset transfer, the best results come from TSAC<sub>Tn</sub>'s top-performing model, AceGPT-chat8b with DZ<sub>abz</sub>, showing a significant 37% performance gap compared to Sem17<sub>MSA+Mixed</sub>'s best model, AceGPT8b with Tn<sub>ar</sub>.

Additionally, when these models encounter confusion, they tend to default to the base model language, typically MSA or Lv<sub>abz</sub>, as observed in cases such as Sem17<sub>MSA+Mixed</sub>.

A similar trend is observed in ED, where models fine-tuned on datasets with a predominant script struggle to generalize effectively when transferred to datasets using a different script.
<table>
  <tr>
    <th>Dataset</th>
    <th>Semi17<sub>MSA+MresponseData</sub></th>
    <th>Twilf-info_sent</th>
    <th>OCA<sub>MSA+TasGA</sub></th>
    <th>TSAAGa</th>
    <th>Semi18<sub>TasGA+MresponseData</sub></th>
    <th>Twilf-emo_sent</th>
    <th>IDAT<sub>MSA+MresponseData</sub></th>
    <th>Flood<sub>Eng+MLG</sub></th>
  </tr>
  <tr>
    <td>Best Transformer/Baseline</td>
    <td>79.06</td>
    <td>75.48</td>
    <td>91.08</td>
    <td>89.10</td>
    <td>91.23</td>
    <td>87.65</td>
    <td>78.78</td>
    <td>78.89</td>
  </tr>
  <tr>
    <td>Best open source LLM NFT</td>
    <td>82.18</td>
    <td>75.93</td>
    <td>91.09</td>
    <td>89.81</td>
    <td>91.57</td>
    <td>87.64</td>
    <td>83.45</td>
    <td>88.05</td>
  </tr>
  <tr>
    <td>Best open source LLM NFT Lang</td>
    <td>14.91</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>Total: Eng</td>
    <td>Total: Eng</td>
  </tr>
  <tr>
    <td rowspan="7">EmoLlama7s</td>
    <td>Eng</td><td>43.09</td><td>79.22</td><td>67.95</td><td>62.51</td><td>20.30</td><td>21.02</td><td>78.20</td><td>60.81</td>
  </tr>
  <tr>
    <td>MSA</td><td>40.72</td><td>76.62</td><td>69.07</td><td><b>94.28</b></td><td><b>26.75</b></td><td>17.50</td><td>75.36</td><td><b>62.51</b></td>
  </tr>
  <tr>
    <td>Dzaki</td><td>43.01</td><td>76.75</td><td>81.97</td><td>94.35</td><td>18.27</td><td>20.02</td><td>75.68</td><td>61.82</td>
  </tr>
  <tr>
    <td>Tain</td><td>40.53</td><td>76.11</td><td>64.91</td><td><b>94.72</b></td><td>17.43</td><td>17.85</td><td><b>76.71</b></td><td><b>62.96</b></td>
  </tr>
  <tr>
    <td>Lvald</td><td>40.30</td><td>77.00</td><td>33.04</td><td>62.75</td><td>17.68</td><td><b>23.99</b></td><td>74.66</td><td>61.83</td>
  </tr>
  <tr>
    <td>Lvar</td><td>36.16</td><td>74.44</td><td>53.83</td><td>94.27</td><td>20.00</td><td>18.55</td><td>76.19</td><td>60.19</td>
  </tr>
  <tr>
    <td>Eng</td><td>43.80</td><td>76.45</td><td>81.93</td><td>94.36</td><td>17.63</td><td>18.06</td><td>76.67</td><td><b>62.68</b></td>
  </tr>
  <tr>
    <td rowspan="7">EmoLlama-chat7s</td>
    <td>MSA</td><td>42.39</td><td>78.10</td><td>55.80</td><td>94.38</td><td><b>20.98</b></td><td>HSE</td><td>75.00</td><td>60.90</td>
  </tr>
  <tr>
    <td>Dzaki</td><td>43.11</td><td>77.81</td><td>81.49</td><td><b>94.48</b></td><td>18.27</td><td>20.02</td><td><b>78.42</b></td><td>61.84</td>
  </tr>
  <tr>
    <td>Tain</td><td>43.05</td><td>78.77</td><td>79.58</td><td>93.70</td><td>17.77</td><td>18.34</td><td>76.49</td><td>61.92</td>
  </tr>
  <tr>
    <td>Lvald</td><td>43.83</td><td>75.54</td><td>67.35</td><td>93.78</td><td>16.21</td><td><b>24.32</b></td><td>75.15</td><td>61.46</td>
  </tr>
  <tr>
    <td>Lvar</td><td>44.10</td><td>76.45</td><td>32.52</td><td>94.19</td><td>19.44</td><td>19.44</td><td>75.79</td><td>58.61</td>
  </tr>
  <tr>
    <td>Eng</td><td><b>44.18</b></td><td>77.06</td><td>66.37</td><td>94.06</td><td>18.81</td><td>20.56</td><td>76.18</td><td>67.04</td>
  </tr>
  <tr>
    <td>MSA</td><td>43.12</td><td>75.63</td><td>80.80</td><td>94.47</td><td>20.47</td><td>23.50</td><td>77.06</td><td><b>67.57</b></td>
  </tr>
  <tr>
    <td rowspan="7">EmoLlama-chat15s</td>
    <td>Dzaki</td><td>43.05</td><td>78.86</td><td>87.24</td><td><b>95.88</b></td><td><b>21.07</b></td><td>25.52</td><td>82.87</td><td><b>62.51</b></td>
  </tr>
  <tr>
    <td>Tain</td><td>43.99</td><td>78.55</td><td>78.88</td><td>93.35</td><td>18.29</td><td>17.52</td><td>77.60</td><td>64.19</td>
  </tr>
  <tr>
    <td>Lvald</td><td>43.59</td><td>79.47</td><td>44.53</td><td>95.04</td><td>16.75</td><td>32.84</td><td>76.66</td><td>62.30</td>
  </tr>
  <tr>
    <td>Lvar</td><td>43.91</td><td>79.58</td><td>73.38</td><td><b>95.41</b></td><td>16.35</td><td>27.05</td><td>79.22</td><td>64.00</td>
  </tr>
  <tr>
    <td>Eng</td><td><b>43.30</b></td><td>74.72</td><td>93.81</td><td><b>94.69</b></td><td>18.97</td><td>27.78</td><td>81.69</td><td><b>62.10</b></td>
  </tr>
  <tr>
    <td>MSA</td><td>34.98</td><td>78.02</td><td>78.02</td><td>92.75</td><td>17.51</td><td>33.36</td><td><b>80.89</b></td><td>58.14</td>
  </tr>
  <tr>
    <td>Dzaki</td><td>(Ad, c)</td><td>40.58</td><td>79.24</td><td><b>94.47</b></td><td>14.02</td><td>27.85</td><td>78.81</td><td>43.62</td>
  </tr>
  <tr>
    <td rowspan="7">Llama3では</td>
    <td>Tain</td><td>43.39</td><td>79.80</td><td>34.21</td><td>93.09</td><td>17.07</td><td>31.79</td><td>83.02</td><td>61.32</td>
  </tr>
  <tr>
    <td>Lvald</td><td>42.96</td><td>77.84</td><td>68.75</td><td>93.57</td><td><b>17.73</b></td><td>32.60</td><td>80.76</td><td><b>62.02</b></td>
  </tr>
  <tr>
    <td>Lvar</td><td>42.99</td><td>79.88</td><td>8.31</td><td>93.46</td><td>16.19</td><td>37.64</td><td>79.77</td><td>61.35</td>
  </tr>
  <tr>
    <td>Eng</td><td>43.99</td><td>79.84</td><td>97.18</td><td><b>94.42</b></td><td>18.60</td><td>25.80</td><td>82.50</td><td>58.48</td>
  </tr>
  <tr>
    <td>MSA</td><td>43.30</td><td>65.89</td><td><b>99.92</b></td><td>92.10</td><td><b>18.53</b></td><td>30.89</td><td>70.28</td><td><b>66.91</b></td>
  </tr>
  <tr>
    <td>Dzaki</td><td>43.42</td><td>68.90</td><td>99.90</td><td>91.60</td><td>11.98</td><td>24.23</td><td>79.97</td><td>61.69</td>
  </tr>
  <tr>
    <td>Eng</td><td>43.70</td><td>60.08</td><td>99.00</td><td>93.88</td><td><b>17.77</b></td><td>19.84</td><td>78.97</td><td>61.72</td>
  </tr>
  <tr>
    <td rowspan="7">Jannas3-insts</td>
    <td>Dzaki</td><td><b>8.48</b></td><td>22.16</td><td>11.43</td><td>92.10</td><td>6.29</td><td>7.15</td><td>75.92</td><td>45.66</td>
  </tr>
  <tr>
    <td>Lvald</td><td>43.22</td><td>73.94</td><td>91.11</td><td>94.06</td><td>16.11</td><td><b>35.81</b></td><td>77.17</td><td>65.96</td>
  </tr>
  <tr>
    <td>Lvar</td><td>42.31</td><td>81.00</td><td><b>99.89</b></td><td>94.51</td><td>16.80</td><td>22.23</td><td>85.00</td><td>58.01</td>
  </tr>
  <tr>
    <td>Eng</td><td>32.21</td><td>5.31</td><td>81.68</td><td>89.80</td><td>18.16</td><td>20.34</td><td>78.20</td><td>86.76</td>
  </tr>
  <tr>
    <td>MSA</td><td>17.93</td><td>60.11</td><td>97.12</td><td>87.50</td><td>17.39</td><td>17.13</td><td>84.77</td><td>57.11</td>
  </tr>
  <tr>
    <td>Dzaki</td><td>58.64</td><td>78.32</td><td>85.25</td><td><b>93.43</b></td><td>19.60</td><td>23.55</td><td>75.77</td><td>58.38</td>
  </tr>
  <tr>
    <td>Lvald</td><td>35.61</td><td>76.68</td><td>34.21</td><td>12.28</td><td>15.13</td><td>19.88</td><td>77.71</td><td>61.13</td>
  </tr>
  <tr>
    <td>Lvar</td><td>38.58</td><td>73.72</td><td>85.61</td><td>81.49</td><td>15.89</td><td>14.70</td><td>76.44</td><td>62.64</td>
  </tr>
  <tr>
    <td>Eng</td><td>21.77</td><td>76.28</td><td>81.22</td><td>90.63</td><td><b>18.71</b></td><td>15.85</td><td>74.01</td><td>66.57</td>
  </tr>
  <tr>
    <td>MSA</td><td>30.47</td><td>59.86</td><td><b>95.43</b></td><td>89.80</td><td>17.50</td><td>22.90</td><td>76.27</td><td>53.24</td>
  </tr>
  <tr>
    <td>Dzaki</td><td>(Ad, c)</td><td>35.63</td><td>75.17</td><td>64.33</td><td>91.74</td><td>12.97</td><td>20.06</td><td>77.16</td><td>60.19</td>
  </tr>
  <tr>
    <td>Mistral/1stL</td>
    <td>Tain</td><td>38.54</td><td>62.17</td><td>32.88</td><td>93.68</td><td>13.89</td><td>21.23</td><td>77.35</td><td>61.30</td>
  </tr>
  <tr>
    <td>Livald</td><td>12.45</td><td>63.27</td><td>36.55</td><td>94.23</td><td>18.15</td><td>15.42</td><td>72.36</td><td>61.30</td>
  </tr>
  <tr>
    <td>Lvar</td><td>33.56</td><td>72.67</td><td>85.61</td><td>81.49</td><td>15.89</td><td>14.70</td><td>76.44</td><td>39.53</td>
  </tr>
  <tr>
    <td>Eng</td><td>38.94</td><td>75.34</td><td>85.77</td><td>89.95</td><td>18.50</td><td>25.81</td><td>78.42</td><td>60.82</td>
  </tr>
  <tr>
    <td>MSA</td><td>33.95</td><td>73.87</td><td>95.33</td><td>89.04</td><td>20.80</td><td>22.47</td><td>83.94</td><td>65.69</td>
  </tr>
  <tr>
    <td>Dzaki</td><td>41.50</td><td>79.94</td><td><b>97.00</b></td><td>91.34</td><td>17.18</td><td><b>27.66</b></td><td><b>87.40</b></td><td>66.56</td>
  </tr>
  <tr>
    <td>SOLAR-inst-15s</td>
    <td>Tain</td><td>34.00</td><td>75.39</td><td>82.61</td><td>91.49</td><td>17.96</td><td>23.12</td><td><b>78.92</b></td><td>63.99</td>
  </tr>
  <tr>
    <td>Lvald</td><td><b>40.61</b></td><td>76.81</td><td>82.49</td><td>90.76</td><td><b>23.05</b></td><td><b>24.72</b></td><td>78.90</td><td><b>61.96</b></td>
  </tr>
  <tr>
    <td>Lvar</td><td>47.29</td><td>77.88</td><td>87.56</td><td>90.77</td><td>17.73</td><td>21.12</td><td>85.41</td><td><b>62.06</b></td>
  </tr>
  <tr>
    <td>AceComPTsubplott</td>
    <td>Eng</td><td>83.38</td><td><b>85.90</b></td><td>98.00</td><td>97.99</td><td>8.08</td><td>8.67</td><td>88.30</td><td>72.73</td>
  </tr>
  <tr>
    <td>MSA</td><td>51.39</td><td>81.81</td><td><b>99.92</b></td><td>89.85</td><td>30.07</td><td>6.36</td><td>87.34</td><td>52.45</td>
  </tr>
  <tr>
    <td>Dzaki</td><td>79.23</td><td>85.88</td><td>97.00</td><td>80.35</td><td>29.54</td><td>6.26</td><td>77.58</td><td>72.27</td>
  </tr>
  <tr>
    <td>Eng</td><td><b>85.23</b></td><td><b>85.23</b></td><td><b>99.98</b></td><td><b>99.29</b></td><td><b>30.02</b></td><td><b>6.26</b></td><td><b>77.30</b></td><td><b>74.35</b></td>
  </tr>
  <tr>
    <td>Lvald</td><td>78.36</td><td>78.50</td><td>34.21</td><td>96.21</td><td><b>32.41</b></td><td><b>3.59</b></td><td>83.86</td><td>34.45</td>
  </tr>
  <tr>
    <td>Lvalr</td><td>80.53</td><td>84.29</td><td><b>99.98</b></td><td><b>98.59</b></td><td>31.77</td><td><b>6.26</b></td><td><b>84.99</b></td><td>79.45</td>
  </tr>
  <tr>
    <td>AceGPT-chat</td>
    <td>Eng</td><td>79.48</td><td>84.82</td><td>98.00</td><td>97.28</td><td>30.84</td><td>6.26</td><td>82.11</td><td>64.70</td>
  </tr>
  <tr>
    <td>MSA</td><td>59.55</td><td>81.34</td><td><b>99.98</b></td><td>89.79</td><td>26.68</td><td>6.29</td><td>84.43</td><td>73.27</td>
  </tr>
  <tr>
    <td>Dzaki</td><td>78.05</td><td><b>87.15</b></td><td><b>94.01</b></td><td><b>94.96</b></td><td>27.00</td><td>6.26</td><td>85.05</td><td>75.47</td>
  </tr>
  <tr>
    <td>Laval</td><td><b>84.11</b></td><td>85.88</td><td>87.98</td><td>97.02</td><td>31.37</td><td>6.26</td><td>83.79</td><td>14.36</td>
  </tr>
  <tr>
    <td>Lvar</td><td>81.99</td><td>78.59</td><td>88.75</td><td>94.99</td><td>19.16</td><td>3.59</td><td>81.77</td><td>74.26</td>
  </tr>
  <tr>
    <td>Jais-family_gpt9</td>
    <td>Eng</td><td><b>13.28</b></td><td>20.52</td><td>34.21</td><td><b>43.19</b></td><td>1.56</td><td>6.26</td><td><b>34.21</b></td><td>7.27</td>
  </tr>
  <tr>
    <td>MSA</td><td><b>18.61</b></td><td>14.00</td><td>32.43</td><td>33.45</td><td><b>1.56</b></td><td>6.26</td><td>34.21</td><td><b>7.83</b></td>
  </tr>
  <tr>
    <td>Dzaki</td><td>13.12</td><td>14.00</td><td>12.85</td><td>33.45</td><td>1.56</td><td>6.26</td><td>34.21</td><td>7.43</td>
  </tr>
  <tr>
    <td>Tain</td><td>10.18</td><td>14.00</td><td>32.43</td><td>33.45</td><td>1.56</td><td>6.26</td><td>34.21</td><td>7.43</td>
  </tr>
  <tr>
    <td>Lvalld</td><td>18.61</td><td>14.00</td><td>12.85</td><td>33.45</td><td>1.56</td><td>6.26</td><td>34.21</td><td>7.83</td>
  </tr>
  <tr>
    <td>Lvar</td><td>18.61</td><td>14.00</td><td>24.83</td><td>33.45</td><td>1.56</td><td>6.26</td><td>34.21</td><td>7.43</td>
  </tr>
</table>

Table 11: Comparative performance of models across tasks, datasets, and dialects: Best performances are highlighted in green, worst in red, and each model's top scores are in bold. The first section of the table presents the best results for baseline and non-fine-tuned (NFT) models. The second and detailed section is dedicated to fine-tuned results.

Reviewer paper summaries: This paper investigates the effectiveness of dialect-specific prompting for subjective text classification in Arabic. The authors explore the impact of three key variables: (1) using dialectal prompts vs. MSA or English prompts, (2) varying prompt scripts (Arabic, Latin, and Arabizi), and (3) fine-tuning LLMs. The study evaluates several LLMs across four subjective classification tasks.

This paper presents an extensive empirical study on subjective text classification in Arabic, examining the impact of various modeling and prompting choices. The authors compare three categories of models—fine-tuned BERT-based models, vanilla large language models (LLMs), and fine-tuned LLMs—across different prompting languages (English, MSA, and dialectal Arabic), as well as orthographic variants (Arabic script and Arabizi). The models are evaluated on four benchmark tasks with varying degrees of dialectal content: sentiment analysis, emotion recognition, irony detection, and crisis detection. In total, 12 models are evaluated across a range of prompting strategies. Results show that fine-tuned LLMs, when prompted in dialectal Arabic or Arabizi, outperform others on five out of eight datasets. The paper also examines cross-dataset and cross-dialect generalization, highlighting the limited transferability of current models across domains and dialects.
Summary of strengths: - Great detailed study about the role of prompt for subjective text classification in Arabic varieties which would benefit the Arabic NLP community.
- Extensive experiments across different models, datasets and scenarios (without fine-tuning, with fine-tuning ...).

- Extensive investigation on subjective text classification tasks in Arabic, spanning a wide set of experimental variables, such as model types, prompting language variants, and scripts.  
- Providing comprehensive empirical results including in-domain performance, as well as out-of-domain performance, offering insights into how well the models generalize across different datasets and dialects.
- The inclusion of under-explored aspects such as Arabizi and dialectal prompting contributes valuable insights for Arabic NLP, especially in real-world scenarios.
Summary of weaknesses: - The paper claims SOTA results but lacks direct comparisons with previous SOTA methods on these datasets.
- The improvements from dialectal prompting over MSA prompting, while present, are sometimes modest (particularly for well-represented dialects), maybe the authors should consider performing a statistical significance test.
- No clear analysis about the efficiency of this approach, the paper could provide more concrete examples of where the approach succeeds and fails. This will help understanding more the reason why dialectal prompts help.

- While reporting the best results per setup highlights the top-performing combinations, the analysis would have benefited from additional aggregated results (e.g., averages across model types, prompting languages, or scripts). Such summaries could help reveal broader trends and make the findings more interpretable and generalizable.
- The difference between (3) cross-dataset and (4) fine-tuning transfer can be better clarified. Is the difference whether the evaluation is zero-shot or involves fine-tuning?
Comments, suggestions and typos: Some of the presented results tables are dense and could benefit from better organization. Also The transfer learning analysis could be more thoroughly explained as the process followed in the paper is not clear .

The authors can benefit from checking these publications about the same topic:

- https://aclanthology.org/W19-5606/
- https://aclanthology.org/2021.naacl-main.226/

- Line 036: Is MSA a really low-resourced language?
- Line 214: a well as -> as well as
- Line 306: dialects s -> dialects
Overall assessment scores: 2

3
Metareview score: 3
Metareview: This paper explores Arabic dialect-specific prompting, script variations, and fine-tuning in Arabic subjective text classification. It uses 12 models and 8 datasets, showing that dialectal prompts improve results, especially in low-resource settings.
Summary of reasons to publish: The paper offers interesting insights into dialectal prompting for NLP, with a thorough experimental setup. It provides new findings on fine-tuning LLMs and explores Arabic script variations, making it a strong contribution to multilingual NLP research.