Benchmarking LLaMA-3 on Arabic Language Generation Tasks (2024)

Abstract: Open-sourced large language models (LLMs) have exhibited remarkable performance in a variety of NLP tasks, often catching up with the closed-sourced LLMs like ChatGPT. Among these open LLMs, LLaMA-3-70B has emerged as the most recent and the most prominent one. However, how LLaMA-3-70B would situate itself in multilingual settings, especially in a rich morphological language like Arabic, has yet to be explored. In this work, we focus to bridge this gap by evaluating LLaMA-3-70B on a diverse set of Arabic natural language generation (NLG) benchmarks. To the best of our knowledge, this is the first study that comprehensively evaluates LLaMA-3-70B on tasks related to Arabic natural language generation. Our study reveals that LLaMA-3-70B lags behind the closed LLMs like ChatGPT, both in modern standard Arabic (MSA) and dialectal Arabic (DA). We further compare the performance of LLaMA-3-70B with our smaller and dedicated finetuned Arabic models. We find that both LLaMA-3-70B and ChatGPT are outperformed by comparatively smaller dedicated Arabic models, indicating the scope for potential improvement with Arabic-focused LLMs.

heading: 1 Introduction
text: In the last few years, the emergence of large language models (LLMs) (OpenAI, 2022, 2023; AI@Meta, 2024; Gemini et al., 2023; Anthropic, 2023; Jiang et al., 2023) has emerged as powerful systems to accomplish diverse set of natural language processing (NLP) tasks. Such success of these models primarily stems from pretraining them on large datasets with the next token prediction objective. The current state-of-the-art performance is however, dominated mostly by ‘closed’ models (OpenAI, 2023; Gemini et al., 2023; Anthropic, 2023). That is, little to no information about them is known. This includes details about model architectures, pretraining data, languages
involved, and training configurations. Such LLMs are also expensive both to pretrain and deploy. To alleviate these concerns, ‘open’ LLMs such as LLaMA-3 (Touvron et al., 2023a), Mistral (Jiang et al., 2023), and OLMo (Groeneveld et al., 2024) were introduced to facilitate research and (non-) commercial deployment. Rigorous evaluations (Qin et al., 2023; Gilardi et al., 2023; Laskar et al., 2023) have already been conducted to analyze LLMs performance on a wide range of English benchmarks. However, a limited number of studies can be found when it comes to evaluating LLMs on non-English languages (Bang et al., 2023a; Lai et al., 2023; Kasai et al., 2023; Huang et al., 2023a). This limitation becomes more acute, in the case of Arabic; a rich morphological language comprised of a wide range of dialects and varieties. Khondaker et al. (2023) evaluate closed models like ChatGPT and GPT-4 on a wide range of Arabic NLP tasks, while Kadaoui et al. (2023) experiment with different commercially available closed models (e.g., Bard) on various Arabic machine translation tasks. However, the performance of the open-sourced models on Arabic natural language generation (NLG) benchmarks is still an unexplored territory to this date. To address this gap, in this study, we benchmark LLaMA-3-70B (AI@Meta, 2024), the most recent and the best open LLM, on a diverse set of 20 Arabic NLG tasks. Notably, our study reveals that LLaMA-3-70B lags behind closed-sourced LLMs like ChatGPT. We further show that even smaller, dedicated, finetuned Arabic models can outperform LLaMA-3-70B.
Our contributions can be summarized as follows:
1. We present the first benchmark of opensourced LLM, LLaMA-3-70B on a wide range of 20 Arabic NLG tasks.
2. We perform a systematic evaluation of LLaMA-3-70B and demonstrate how the
283
model situate itself compared to closed-source models like ChatGPT.
3. We additionally conduct evaluation on Arabic dialects to show the inferiority of LLaMA-3-70B compared to ChatGPT.
4. Through our empirical analyses, we further demonstrate that the LLaMA-3-70B significantly lags behind much smaller Arabicfocused finetuned models. We believe such findings can motivate future studies on developing stronger open-sourced Arabic LLMs.

heading: 2 Related Work
text: This section reviews recent developments in Large Language Models (LLMs), focusing on their applications in Machine Translation, Question Answering, and Multilinguality. It particularly emphasizes tasks involving non-English languages such as Arabic. Please see Appendix A for a comprehensive guide.

heading: 2.1 Performance analysis on English
text: Open-sourced LLMs have advanced in machine translation yet faced challenges in domain-specific and low-resource scenarios. Innovations like LlamaIT framework (Zheng et al., 2024) reduce the need for detailed input examples, while the TER framework (Feng et al., 2024) enhances translation accuracy through systematic self-correction. Realtime adaptive translation capabilities have been demonstrated to be especially useful with limited in-domain data (Moslem, 2024), and improvements in instruction adherence through a two-stage finetuning algorithm (Zan et al., 2024) significantly enhance translation accuracy in low-resource languages. In question-answering (QA), open LLMs demonstrate significant potential, (Gramopadhye et al., 2024) enhance medical QA with a Chain of Thought approach, (Jiang et al., 2024) generate document-question pairs. Additionally, (Arefeen et al., 2024) introduce LeanContext, which uses reinforcement learning to tailor the amount of contextual information needed, (Kim et al., 2024) further refine open-domain (QA) through the SuRe framework, which supports answer validation. However, despite these advancements, (LLMs) still face challenges in achieving human-like performance. Majumdar et al. (2024) note that current models underperform in interactive environments such as Embodied (QA), where understanding and interact-
ing with real-world contexts remains a substantial hurdle. For text classification, Guo et al. (2024) surpass traditional supervised models in healthrelated classifications using social media data. Similarly, Abburi et al. (2023) employ LLMs to distinguish between AI-generated and human-written texts, (Miah et al., 2024) confirm the effectiveness of LLMs in multilingual sentiment analysis, translating and processing texts with high accuracy across different languages. However, despite these successes, challenges persist as Chae and Davidson (2023) point out that while (LLMs) perform well in minimal-example scenarios, they still require extensive fine-tuning to achieve optimal accuracy.

heading: 2.2 Performance on Multilingual Data
text: In assessing the capabilities of open-sourced LLMs within multilingual contexts, several studies have illuminated both the strengths and the limitations these models exhibit across diverse language environments. Shen et al. (2024) highlight significant safety challenges, particularly when LLMs process lower-resource languages, where they tend to generate unsafe or irrelevant responses. Alam et al. (2024) address similar challenge across multilingual, multimodal, and dialectal settings, emphasizing the importance of instruction tuning and reinforcement learning. Meanwhile, Li et al. (2024) propose integrating code elements during training to strengthen multilingual structured reasoning, showcasing substantial improvements in tasks requiring scientific commonsense across various languages. Additionally, Zhang et al. demonstrate the effectiveness of Self-Distillation from ResourceRich Languages (SDRRL) in leveraging the capabilities of LLMs trained on resource-rich languages to enhance performance more broadly. Liu et al. (2024) challenge the over-reliance on translation, advocating for native language prompting to capture cultural nuances better, and achieve more authentic multilingual functionality. Expanding on in-context learning, Cahyawijaya et al. (2024) explore its application in low-resource languages, emphasizing minimal training data’s potential to yield effective language processing and examine how LLMs handle multilingual processing within their architectures,. Together, these studies form a comprehensive examination of LLMs in multilingual contexts, revealing promising strategies for improvement and persistent challenges that necessitate further research.

heading: 2.3 LLMs with Low Resource Languages
text: Efforts such as the HELM project and BIG-Bench have expanded the evaluation of Large Language Models (LLMs) to encompass low-resource languages, shedding light on the models’ generalizability across a diverse range of tasks (Liang et al., 2022; Srivastava et al., 2022). Despite these extensive evaluations, studies including those by (Bang et al., 2023b; Ahuja et al., 2023) consistently report that LLMs underperform in low and extremely low-resource languages, highlighting a persistent challenge in achieving parity for non-English languages.
The representation of languages like Arabic in major datasets such as Common Crawl is notably limited, posing significant obstacles in LLM research. Even though models like Bloom are trained on data from numerous languages, they exhibit restricted effectiveness in non-English contexts, particularly for complex NLP tasks (Le Scao et al., 2022). A detailed evaluation by (Khondaker et al., 2023) further illuminates this issue, showing that while ChatGPT demonstrates strong performance in English, it is often surpassed by models that are fine-tuned specifically for Arabic. Their comprehensive analysis, which includes both automated and human evaluations on over 60 datasets, constitutes a significant exploration of ChatGPT’s capabilities with Arabic dialects and Modern Standard Arabic (MSA), emphasizing the model’s limitations in handling the nuanced characteristics of Arabic languages. Our research adds to these findings by assessing open LLM, LLaMA-3-70B across multiple Arabic datasets and tasks, providing deeper insights into the capabilities and limitations of LLaMA-3-70B in managing the distinct linguistic features of Arabic. This exploration helps to elucidate further the disparities in performance between general and specialized models in handling lowresource languages.

heading: 3 Dataset
text: We conduct a comprehensive evaluation of LLaMA-3-70B and ChatGPT on Arabic Natural Language Generation (NLG) tasks. This evaluation encompasses 20 distinct tasks, organized into 10 clusters, compiled from Dolphin benchmark (Nagoudi et al., 2023).
Machine Translation. This cluster evaluates translation performance across various languages, focusing on capturing nuanced meanings and contexts. Key tasks include translating from the six official UN languages to Modern Standard Arabic (MSA), using the UN parallel corpus (Ziemski et al., 2016) and the MultiUN corpus for training (Eisele and Chen, 2010). Another task involves translating Arabizi, a Romanized version of Arabic dialects, into English and French using the Darija (Outchakoucht and Es-Samaali, 2021) and NArabizi datasets (Seddah et al., 2020). Additionally, the cluster includes translating texts from six Arabic dialects into English utilizing the MDP corpus (Bouamor et al., 2018), with zero-shot training supported by additional MSA-English sentences. Code-Switching. This task cluster addresses translating Arabic dialect texts, including codeswitching, into the corresponding foreign languages. Specifically, it includes translating 300 code-switched Arabic-French tweets from Twitter users in Algeria, Morocco, and Tunisia, and 300 code-switched Arabic-English posts from users in Egypt, Jordan, and Palestine. These translations are conducted in a zero-shot setting, utilizing 50,000 MSA-English and MSA-French sentences from the AraOPUS-20 dataset for monolingual training (Nagoudi et al., 2022b). Text Summarization (TS). The Text Summarization cluster leverages five datasets featuring Arabic and multilingual content to enhance and evaluate summarization capabilities. The datasets include MassiveSum, which offers a vast collection of multilingual news articles designed for summarization (Varab and Schluter, 2021); XLSum, providing summaries across 44 languages with a focus on abstractive techniques (Hasan et al., 2021); CrossSum, which supports summarization over 1500 language pairs, enabling crosslingual abstractions (Bhattacharjee et al., 2021); ANT, a specialized corpus for extractive summarization in Arabic using news sources (Chouigui et al., 2021); and MarSum, targeting summarization in the Moroccan Arabic dialect with advanced AI methodologies (Gaanoun et al., 2022). News Title Generation (NTG). This task focuses on crafting suitable and grammatically correct titles for news articles. It employs the Arabic (NTG) dataset, which is dedicated to creating titles specifically for Arabic news content (Nagoudi et al., 2022c), and the XLSum dataset, used for generating titles across multiple languages (Hasan et al., 2021). Question Answering (QA). The QA cluster com-
prises four distinct tasks utilizing seven publicly available datasets. Extractive QA employs the Arabic QA dataset ARCD and Arabic sections from multilingual QA test sets like MLQA, XQuAD, and TyDiQA, with models fine-tuned using the GoldP multilingual TyDiQA training set (Mozannar et al., 2019; Lewis et al., 2019; Artetxe et al., 2020). Retrieval QA involves LAReQA, adapted from XQuAD into a retrieval task focusing on its Arabic version AraQuAD-R (Roy et al., 2020). Open-Domain QA answers fact-based questions using the DAWQAS, an Arabic Why QA dataset (Ismail and Nabhan Homsi, 2018). Multi-choice QA uses the EXAMS dataset, a cross-lingual multi-choice QA collection that includes Arabic, evaluated in a zero-shot scenario (Hardalov et al., 2020). Question Generation (QG). The Question Generation cluster focuses on creating questions from provided passages. The objective is for models to generate straightforward, yet pertinent questions along with their answers, based on the approach described in (Gehrmann et al., 2021). Paraphrase. The Paraphrase task focuses on generating paraphrases of Arabic sentences while preserving their original meaning. It utilizes four datasets: AraPara, a multi-domain Arabic paraphrase dataset (Nagoudi et al., 2022a); ASEP, from the Arabic SemEval paraphrasing challenge (Cer et al., 2017); the Arabic Paraphrasing Benchmark (APB), a key dataset for testing Arabic paraphrase systems (Alian et al., 2019); and TaPaCo, the Arabic portion of a multilingual paraphrase corpus (Scherrer, 2020). Transliteration. The transliteration task focuses on converting text from one writing system to another while maintaining the original pronunciation. It utilizes three word-level datasets: ANETA, for transliteration and classification of English-Arabic named entities (Ameur et al., 2019); ATAR, a parallel corpus for transliterating Jordanian Arabizi into Modern Standard Arabic (MSA) (Talafha et al., 2021); and NETransliteration, a comprehensive dataset for transliterating named entities from English to Arabic, Hebrew, Japanese (Katakana), and Russian, sourced from Wikidata (Merhav and Ash, 2018). Text Rewriting. The text rewriting task focuses on recreating text in a specific target style while maintaining the original content. It includes converting Arabic dialectal text into Modern Standard Arabic (MSA) using the Dial2MSA corpus, which covers dialects like Egyptian, Maghrebi, Levantine, and
Gulf (Mubarak, 2018), and adapting sentences to the opposite gender with the Arabic parallel gender corpus (Alhafni et al., 2022). Diacritization. The diacritization task restores missing diacritics in Arabic texts, utilizing the Arabic diacritization dataset (Fadel et al., 2019).

heading: 4 Prompt design
text: In this study, we define a prompt as a customized set of instructions that shapes the functionality and enhances the output of a large language model, aligning with the framework proposed by (White et al., 2023). The design of a prompt is critical as it influences how the model interacts with users and the quality of its responses by establishing specific operational rules. Therefore, to ensure the model’s responses meet desired outcomes, it’s essential to articulate these rules and objectives.
In our initial experiments, we employed a variety of prompts in both English and Arabic to evaluate their efficacy across several tasks. These included dialect identification, detection of machinegenerated texts, and toxicity detection. Our observations indicated that prompts in English consistently outperformed their Arabic counterparts.
We developed a standardized English prompt template to optimize performance. This template systematically guides the model by (1) Assigning a specific role to the model, such as acting as a translator for natural language generation (NLG) tasks; (2) Naming the task to be undertaken, like machine translation; (3) Describing the expected outcome, such as accurately translating text from one language to another; (4) Incorporating k-shot learning examples if applicable, and; (5) Providing a test input to generate an output.
We showcase the structure of our prompts in Figure 1. This structured approach allows us to fully utilize the capabilities of LLMs by ensuring they operate within well-defined parameters that meet the complexities of the tasks at hand.

heading: 5 Experiments
text: Setup. We run experiments under zero-shot, 3- shot, and 5-shot settings. For the evaluation, we use the same 200 test samples from Khondaker et al. (2023) to ensure fair comparison. Similarly, we also use the exact same training samples from Khondaker et al. (2023) for k-shot settings. For all the models, we use free-form generation
together with simple posthoc-processing, e.g., removing whitespace. Models Comparisons. For benchmarking, we use LLaMA-3-70B1 from HuggingFace (Wolf et al., 2020) library. We set the maximum length of the newly generated tokens to 256. We use 4 Nvidia A100 40GB GPUs to generate the responses. Additionally, we obtain the performance of SoTA finetuned AraT52 model (Nagoudi et al., 2022a; Elmadany et al., 2023) on the sampled 200 test set and the full test set from Khondaker et al. (2023). We also obtain the performance of ChatGPT (gpt3.5-turbo)3 from Khondaker et al. (2023).

heading: 6 Evaluation Result
text: We present the results of our evaluation on natural language generation tasks in Table 1. We observe that ChatGPT outperforms LLaMA-3-70B on most of the tasks. On the other hand, the finetuned AraT5 outperforms both LLaMA-3-70B and ChatGPT on the majority of tasks. We now briefly describe our results for the different NLG tasks. Text Rewriting. For text rewriting, LLaMA-3-70B achieves BLEU scores of 41.83 with 3-shots and outperforms ChatGPT. However, AraT5 outperforms all the models. Paraphrase. For the paraphrase generation task, LLaMA-3-70B is outperformed by ChatGPT in all k-shot by large margins. Furthermore, both models are outperformed by AraT5. Question Generation and Question Answering. For the question generation task, ChatGPT outperforms LLaMA-3-70B on all the respective zero- and
1meta-llama/Meta-Llama-3-70B-Instruct 2https://huggingface.co/UBC-NLP/
AraT5v2-base-1024 3Snapshot of gpt-3.5-turbo from March 1st 2023
k-shots. However, both models are significantly outperformed by AraT5. For QA, LLaMA-3-70B outperforms ChatGPT in all the respective few-shot settings. Specifically, LLaMA-3-70B achieves the best score of 70.53 with 3-shot learning, whereas ChatGPT achieves 54.14 at best with 5-shot learning. However, both models are outperformed by AraT5 (81.45). Summarization. For summarization, LLaMA-3-70B with 0-shot (with 18.83 BLEU) beats ChatGPT in the corresponding shot. However, ChatGPT with 3- and 5-shots outperforms LLaMA-3-70B in the respective few-shots. Both models are outperformed by finetuned AraT5. News Title Generation. For news title generation, LLaMA-3-70B with 0-shot achieves the best scores (4.99 BLEU) and outperforms ChatGPT. Again, AraT5 shows better performance than both (7.72). Diacritization and Transliteration. For diacritization, LLaMA-3-70B is outperformed by ChatGPT with lower error rates, while AraT5 (CER=0.03) still outperforms both LLMs. For transliteration, LLaMA-3-70B (0.41) is outperformed by ChatGPT (0.23). Again, AraT5 (0.18) achieves the lowest CER scores. Machine Translation. ChatGPT outperforms LLaMA-3-70B on all the X (English, Spanish, French, Russian) → Arabic MT tasks. Both models perform better when the source language is English, signifying the availability of English data the models have been pretrained. Nevertheless, AraT5 still exibits superior performance over all the models. Code-Switched Translation. For Jordanian Arabic (Jo) mixed with English → English codeswitched translation task, LLaMA-3-70B (38.80 BLEU) with 5-shot outperforms ChatGPT by a small margin (0.25). Similarly, for Algerian
Arabic (Dz) mixed with French → French codeswitched translation task, LLaMA-3-70B (39.71) with 3-shot outperforms ChatGPT. Noticeably, both LLaMA-3-70B and ChatGPT exibit higher performance than the finetuned AraT5.

heading: 7 Performance on Dialectal Arabic
text: To determine the capability of open LLM for dialectal Arabic generation task, we compare the performance of LLaMA-3-70B and ChatGPT on dialectal machine translation task. Specifically, we use MSA and five Arabic dialects (Egyptian, Jordanian, Palestinian, Syrian, and Tunisian) → English MT tasks from the Multi-dialectal Parallel Corpus (MDPC) Bouamor et al. (2014) and analyze the performance of the models for few-shot settings.
As Figure 2 shows, ChatGPT outperforms LLaMA-3-70B on MSA and all the dialects (except
Palestinian and Syrian) across all the setups. Moreover, the performance gap tends to reduce as the number of shots increases. This indicates that with enough dialectal training data, open LLMs can improve their performance and close the gap with the closed LLM like ChatGPT.
Table 2 shows samples of generation outputs from ChatGPT and LLaMA-3-70B for translation tasks involving Jordanian-English code-switching translation, Egyptian dialect tranlation to English, as well as for diacritization and transliteration tasks. Following (Rust et al., 2020), we compute the fertility score to assess the capability of Llama-3 in handling Arabic languages, and compare it with the Arabic-based model. 4 We found that Llama-3 enhances its representation for Arabic compared to Llama-2, where Llama-3’s fertility score is 2.24 and Llama-2’s is 5.07. However, AraT5 outperforms both with a fertility score of 1.25.

heading: 8 Conclusion and Future Work
text: In this work, we conduct a systematic evaluation of LLaMA-3-70B on a wide range of Arabic NLG tasks. We benchmark LLaMA-3-70B and comparing against closed LLM like ChatGPT, we show that open LLMs like LLaMA-3-70B noticeably lags behind its counterpart. Additionally, we compare both closed and open-sourced LLMs with smaller Arabic-focused finetuned model to demonstrate the superiority of such smaller but finetuned models.
4The fertility score measures the average number of subwords produced per tokenized word. A minimum fertility of 1 indicates that the tokenizer’s vocabulary encompasses every single word in the text.
Finally, we extend our analysis on dialectal Arabic to reveal the inferiority of LLaMA-3-70B against ChatGPT. We believe our effort will encourage researchers to develop and improve Arabic-focused LLMs. In future, we aim to scale our benchmark to other open LLMs as well as Arabic language understanding tasks.

heading: 9 Limitations
text: Model Variation. We conduct our evaluation on the most recent and most prominent open LLM, LLaMA-3-70B. However, we do not include other open LLMs such as AceGPT-7B (Huang et al., 2023b), jais-13B (Sengupta et al., 2023), gemma-7B (Gemini et al., 2023). We acknowledge that incorporating a large number of models can facilitate the comparative analysis and we intend to include them in our future work. Experimental Setup. In this work, we use the performance of ChatGPT benchmarked by Khondaker et al. (2023). However, ChatGPT’s version can be updated, as a consequence, the performance of ChatGPT can vary. Therefore, the results and the corresponding analyses reported in this work should be treated accordingly, since the model’s
responses can change over time (Chen et al., 2023). Evaluation. We suspect that the inferior performance of LLaMA-3-70B can potentially be attributed to insufficient Arabic data during the pretraining stage as well as the lack of Arabic instruction data for finetuning. Although a different sets of prompt may improve the performance, we attempt to keep the prompt template same across all the models for a fair comparison. Results. We observe that LLaMA-3-70B’s performance does not necessarily increase as with the increased number of shots. This is because the improvement with few-shot learning is model- and task-dependent, and often sensitive to the order of the shots Wei et al. (2021); Brown et al. (2020).

heading: 10 Ethics Statement
text: Data Collection. We collect the NLG evaluation datasets from Dolphin (Nagoudi et al., 2023). To ensure proper credit assignment, we refer users to the original publication (Section 3). Intended Use. We believe our findings will encourage further research on studying open LLMs on Arabic NLG benchmarks, as we show that LLaMA-3-70B still lags behind compared to the
smaller finetuned model. Therefore, our work can spur the interest among the researchers to develop Arabic language dedicated LLMs that can match or outperform the SOTA finetuned models. Potential Misuse and Bias. LLMs can produce potentially harmful and biased contents (Laskar et al., 2023). Therefore, we recommend that these models not be used in applications without careful prior consideration of potential misuse and bias.

heading: A Literature Review
text: A.1 Machine Translation
The challenges in machine translation (MT) of Arabic stem from its complex morphology and the diversity of dialects, which deviate significantly from Modern Standard Arabic (MSA), influencing phonetics, lexicon, and grammar. Durrani et al. (2014) addressed these challenges by proposing a method to map Egyptian Arabic to MSA, aiming to standardize input data to enhance the effectiveness of translation models. This foundational work paved the way for further explorations into language-specific issues, such as segmentation challenges, which Sajjad et al. (2017) investigate, emphasizing the importance of tailored segmentation strategies in improving both MT and partof-speech tagging. Recognizing the need for specialized resources to support the development of robust MT systems, Sajjad et al. (2020) introduce AraBench. This extensive dataset is designed to benchmark the performance of dialectal ArabicEnglish MT systems, encompassing a wide range of dialect categories and covering diverse genres. Such comprehensive data resources are crucial for training MT systems to handle the spectrum of dialectal variations encountered in real-world applications.
Recent advancements also highlight the role of hybrid models that combine proprietary and opensource technologies, which have significantly improved translating dialectal Arabic (Kazemi and Li, 2024). In parallel, studies like those by Zhu et al. (2023) evaluate the capabilities of large language models (LLMs), such as ChatGPT, XGLM (Lin et al., 2022), OPT (Zhang et al., 2022), and BLOOMZ (Muennighoff et al., 2022). These evaluations demonstrate that while ChatGPT excels in zero-shot and in-context few-shot scenarios, it still trails behind fully supervised models such as those in the ’No Language Left Behind’ initiative (Team et al., 2022).
Further expanding the resources for Arabic MT, Abdelali et al. (2024) introduce LAraBench, a comprehensive benchmark for Arabic AI that evaluates various LLMs, including GPT-3.5 and GPT-4, along with speech models like Whisper and USM. Their extensive testing across 61 datasets for 33 tasks reveals that while state-of-the-art (SOTA)
models typically outperform LLMs in zero-shot learning, the gap narrows considerably with applying few-shot techniques. This suggests a promising potential for LLMs in Arabic language applications, particularly when leveraged with appropriate learning strategies.
A.2 Text Classification Text classification has witnessed significant advancements with the use of Large Language Models (LLMs) like ChatGPT, which excels in zeroshot and in-context few-shot settings. (Zhong et al., 2023) highlight that ChatGPT performs comparably to fully supervised models on the GLUE NLU benchmark, outperforming BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) on specific tasks such as MNLI, SST2, and RTE, although it lags in other areas. The importance of localized training datasets is also emphasized, as demonstrated by Ahmed et al. (2024), who find that models trained on dialect-specific data significantly surpass those trained on generalized Arabic datasets, underscoring the value of tailored data in text classification.
Further developments include instruction-tuned models like SOCIALITE-LLAMA, which have shown improved performance across specialized NLP tasks (Dey et al., 2024). Another innovative approach is seen in the DRG-LLaMA model, which uses fine-tuning on clinical notes to enhance DRG assignment accuracy and efficiency, substantially outperforming established models like ClinicalBERT and CAML (Wang et al., 2023b). Despite these advancements, challenges remain, particularly in handling Arabic dialects. For instance, even sophisticated models like GPT-4 encounter difficulties generating precise dialectal sentiment text, highlighting ongoing issues within specialized text classification contexts (Al-Thubaity et al., 2023).
A.3 Question Answering (QA) Advancements in QA have seamlessly integrated traditional NLP techniques with Large Language Models (LLMs), enhancing the handling of idiomatic expressions in Arabic as demonstrated by Singh and Patel (2024). Further exploring crosslingual capabilities, Alyafeai and Ahmed (2023) investigate the effectiveness of mBERT (Devlin et al., 2019) in zero-shot language transfer to Arabic. Their study focuses on tasks such as natural language inference and question answering, emphasizing the impact of morphological similarities be-
tween languages like Russian and Arabic on model performance. Additionally, integrating retrievalaugmented generation with models like Llama2 (Touvron et al., 2023b) has shown substantial improvements in complex, domain-specific QA settings (Alawwad et al., 2024).
Domain-specificity also plays a crucial role in enhancing QA systems, as evidenced by Yagnik et al. (2024), who evaluate the performance of both general and medical-specific distilled models in medical QA. Their work highlights the significant benefits of domain-specific fine-tuning in improving model accuracy. Complementing these domainspecific approaches, Wang et al. (2023a) introduce the QA-Eval task and the EVOUNA dataset, designed to advance the evaluation of Open-QA systems. Their research underscores the ongoing challenges by comparing AI-generated answers with standard responses and highlights potential improvements in automatic evaluation methods. In the broader scope of open-domain question answering, Zheng et al. critically assess ChatGPT’s performance, revealing its capabilities in handling complex user queries where no context is provided. Their analysis demonstrates the model’s impressive results and proposes methods to enhance the faithfulness of its answers, addressing key areas where ChatGPT may falter.
A.4 Multilinguality Research in multilingualism has highlighted significant variations in the performance of Large Language Models (LLMs) across different languages, particularly focusing on Arabic. Moreno and Zhang (2024) examine the efficacy of multilingual BERT variants, which show enhanced capabilities across various Arabic dialects. Further studies, such as those by Sallam and Mousa (2024), assess ChatGPT’s handling of Tunisian and Jordanian Arabic, revealing gaps in dialect understanding where both ChatGPT-3.5 and ChatGPT-4 perform variably, underscoring the necessity for dialectspecific model enhancements. Alkaoud (2024) contributes to this discourse by introducing a bilingual benchmark leveraging the General Aptitude Test (GAT), which illustrates substantial improvements in GPT-4’s Arabic capabilities compared to its predecessors, offering insights into bilingual language processing. Furthermore, Lai et al. (2023) evaluate ChatGPT across seven tasks in 37 languages from low, medium, and high-resource families. Their findings suggest that ChatGPT matches or even
exceeds fully supervised state-of-the-art models, particularly in high-resource languages. Interestingly, they note that providing task descriptions in a high-resource language enhances performance for low-resource languages.
Additionally, Koubaa et al. (2024) develop ArabianGPT, a native Arabic LLM designed specifically to tackle the complex linguistic features of Arabic, demonstrating improvements in tasks such as sentiment analysis and summarization. Moreover, expanding the scope of multilingual studies, (Tonja et al., 2024) introduce EthioLLM and Ethiobenchmark—a multilingual suite and corresponding dataset for Ethiopian languages and English. This initiative significantly enhances resource availability and task performance, marking considerable progress in multilingual NLP for lowresource languages.

heading: Future Work
text: We acknowledge that incorporating a large number of models can facilitate the comparative analysis and we intend to include them in our future work. Experimental Setup. In this work, we use the performance of ChatGPT benchmarked by Khondaker et al. (2023). However, ChatGPT’s version can be updated, as a consequence, the performance of ChatGPT can vary. Therefore, the results and the corresponding analyses reported in this work should be treated accordingly, since the model’s
responses can change over time (Chen et al. , 2023). Evaluation. We suspect that the inferior performance of LLaMA-3-70B can potentially be attributed to insufficient Arabic data during the pretraining stage as well as the lack of Arabic instruction data for finetuning. Although a different sets of prompt may improve the performance, we attempt to keep the prompt template same across all the models for a fair comparison. Results. We observe that LLaMA-3-70B’s performance does not necessarily increase as with the increased number of shots. This is because the improvement with few-shot learning is model- and task-dependent, and often sensitive to the order of the shots Wei et al. (2021); Brown et al. (2020)