Arabic Train at NADI 2024 shared task: LLMs’ Ability to Translate Arabic Dialects into Modern Standard Arabic (2024)

Abstract: Navigating the intricacies of machine translation (MT) involves tackling the nuanced disparities between Arabic dialects and Modern Standard Arabic (MSA), presenting a formidable obstacle. In this study, we delve into Subtask 3 of the NADI shared task (Abdul-Mageed et al., 2024), focusing on the translation of sentences from four distinct Arabic dialects into MSA. Our investigation explores the efficacy of various models, including Jais, NLLB, GPT-3.5, and GPT-4, in this dialect-to-MSA translation endeavor. Our findings reveal that Jais surpasses all other models, boasting an average BLEU score of 19.48 in the combination of zeroand few-shot setting, whereas NLLB exhibits the least favorable performance, garnering a BLEU score of 8.77.

heading: 1 Introduction
text: The diverse array of Arabic dialects poses a significant challenge for machine translation (MT) systems aiming to connect the different spoken varieties with Modern Standard Arabic, the standardized written form of the language. In recent years, there has been a notable surge of interest in developing effective MT systems capable of accurately translating Arabic dialects into MSA. Such advancements not only foster smoother communication across diverse Arabic-speaking communities but also expand access to a wide array of Arabic content, enriching cultural exchange and understanding.
In our endeavor to confront this challenge and foster progress in Arabic dialect MT, we embarked on an exploration within the framework of the NADI shared task. This task specifically focuses on the sentence-level translation of four major Arabic dialects - Egyptian, Emirati, Jordanian, and Palestinian - into MSA.
Our primary objective lies in evaluating and comparing the performance of three distinct ap-
proaches, GPT, Jais, and NLLB, in the task of translating Arabic dialects into MSA at the sentence level. In addition, Google Translate assesses the performance of translating dialectal text into English and French, and subsequently translating it back into Arabic (MSA). To facilitate this evaluation, we meticulously curated our own datasets for model refinement. Our approach involved Google Translate translations to generate parallel datasets along with the gold MSA translations. This dataset shifts the translation task to a translation-correction task.
Subsequently, all models underwent a rigorous evaluation using the test data set, which includes samples drawn from the four specified dialects. Our observations yielded noteworthy insights, with the performance analysis revealing that Jais exhibited superior translation proficiency compared to GPT, NLLB, and Google Translate as evidenced by higher BLEU scores.
These findings underscore the importance of continued exploration and innovation in the realm of Arabic dialect MT, as we strive to develop robust systems that not only bridge linguistic divides, but also foster cross-cultural exchange and understanding. Through collaborative efforts and thoughtful refinement, we aim to chart a path towards more inclusive and accessible language technologies for Arabic speakers worldwide

heading: 2 Related Work
text: The landscape of Arabic dialect Machine Translation (MT) has undergone significant evolution, transitioning from earlier rule-based and statistical methods to more sophisticated neural network models. Sawaf (2010) laid the groundwork by exploring these traditional approaches together. However, with the advent of deep learning, neural network-based methods have taken center stage.
Recent studies have delved into the capabilities
729
of Large Language Models (LLMs) in translating Arabic dialects. Kadaoui et al. (2023) conducted an extensive investigation into the translation proficiency of instruction-tuned LLMs across various Arabic varieties. Their research illuminated the challenges faced by LLMs like Bard and ChatGPT when dealing with dialects lacking substantial public datasets. Nevertheless, these models demonstrated remarkable performance, surpassing existing commercial systems and affirming their effectiveness as dialect translators.
Conversely, Farhan et al. (2020) pioneered unsupervised dialectal Neural Machine Translation (NMT), presenting two systems for this novel challenge. The Dialectal to Standard Language Translation (D2SLT) system leverages dialectal similarities via unique strategies, achieving promising results. Notably, the highest BLEU score in the unsupervised setting reached 32.14, at bar with the best score obtained in the supervised setting, which stands at 48.25.
Many proposed methods depend on accurately identifying the dialect before translation, posing several challenges in the process. Keleg and Magdy (2023) address the limitations of current Automatic Arabic Dialect Identification (ADI) systems in distinguishing between Arabic micro-dialects. They critique the prevailing approach of treating ADI as a single-label classification problem, arguing that it contributes to system failures. Through manual error analysis conducted by seven native Arabic speakers, approximately 66% of identified errors were found to be inaccuracies. Consequently, the paper proposes reframing ADI as a multi-label classification task and offers recommendations for enhancing future ADI datasets.
This paper focuses on exploring the capabilities of the most recent models, such as Jais, GPT-3.5, and GPT-4, for dialect to MSA translation.

heading: 3 Data

heading: 3.1 Shared Task Data
text: We conducted thorough evaluations on both the validation and test sets provided for the shared task.

heading: 3.1.1 Validation Dataset
text: The validation data set included in this shared task comprised 400 sentences, with 100 sentences from each dialect, and was used for both model tuning and evaluation purposes.
In some of the provided examples, there is a metaphorical expression that adds complexity to the translation process. For example, the phrase below employs imagery and cultural nuances that may not directly translate into other languages. It conveys a sense of agreement among the three individuals involved, using the metaphor of baking dough and leavening to suggest a collaborative effort. The metaphorical usage of words like " IÊË@" and " èXAK QË @ áj. ªË@" adds depth and richness to the expression, but also presents a challenge for translators in capturing the essence of the message accurately in the target language.
{ "dialect": "Palestinian", "source": ", èXAK QË @ áj. ªË@ ð IÊË@ ÊË á ® ® JÓ Õº J KC K AÖÏ A£
A JJ Ê« é ®k éËP Y Jª K ð éÖÏ QËA« iJ ¢ ð A JË Ag I. j A JJ Ê g", "target":
" A Kñ«X , ̈ PA ®Ë @ ÐC¾Ë@ ú ̄ ñ mÌ'@ @ XAÖÏ , àñ ® ® JÓ Õº J KC K à @ AÖß.
. A JJ Ê« é ®k éË P Y Jª Kð Ég. QË@ úÍ@ È Q Kð I. ë Y K"
"English translation": "Since the three of you agree, why engage in nonsense, let’s go down to the man and apologize
to him for his right against us."}

heading: 3.1.2 Test Dataset
text: The test set for Subtask 3 comprises 2,000 sentences, with 500 sentences representing each of the four dialects: Egyptian, Emariti, Jordanian, and Palestinian. Numerous lengthy sentences pose further challenges in translation, illustrated by the below example for a sentence of 58 words: { ", ½J Ê« I. J « ,ÉK @ñªË@
K Y A JË @ K Y éK A JJ. Ë @ H. @ñK. é J K @ ,I. J « , Ij Ij Ij . é J K @ ú æÓ á Ë ú k @ AK
ú æ Õ»Y J« AÓ ú kAK , é<Ë @ ék. ñË Q g ñ ½ KAJ k ú ̄ èYgð èQÓ , @Pñº B ð Z@ Qk. Õº JÓ YK Q K B Õ»Y J« AÓ , é<Ë @ ék. ñË éÖÞ @ èQÓ I. J £ iÊ¢ Ó ¼Y J« AÓ
áK P I. Ó . . ,I. J « , èYgð" }̈

heading: 3.2 Fine-tuning Dataset

heading: 3.2.1 MADAR Dataset
text: We utilized the MADAR Arabic Dialect Corpus and Lexicon (Bouamor et al., 2018) in our study to fine-tune and Few-shot Fine-tuning models, serving as a crucial resource for machine translation research, particularly in handling Arabic dialects’ complexities. This dataset includes 25 parallel translations, covering 2,000 sentences from 25 cities across the Arab world. In our experiments, we grouped the dataset into three categories, reflecting major geographical and dialectal regions. We focused on the cities of Salt, Amman, Cairo, Alexandria, and Jerusalem.

heading: 3.2.2 Custom Dataset
text: In our endeavor to create a dataset for fine-tuning, we conducted a series of "back translation" experiments leveraging Google Translate to generate translations between Arabic and English, as well as between Arabic and French. We translated Arabic sentences into English/French using Google Translate, subsequently back-translating the English/French translations into Arabic. This process aimed to introduce variation and diversity into the dataset, potentially capturing different nuances and interpretations of the original Arabic sentences. The resulting back-translated Arabic sentences served as the source (src) for the subsequent fine-tuning experiments. The rationale behind choosing French is twofold: French is widely spoken in several Arab countries, providing relevant dialectal nuances, and French translations can introduce further linguistic diversity, potentially capturing different structural and lexical variations that might not be present in English translations.

heading: 4 Methodology

heading: 4.1 Zero- and Few-Shot
text: We conduct a comprehensive evaluation of GPT3.5 (Ouyang et al., 2022), GPT4 (OpenAI, 2023), and Jais (G42, 2023) focusing on their ability to translate various dialects into modern standard Arabic. Specifically, we assess GPT3.5, GPT-4 in both zero-shot and few-shot scenarios. Following the findings of Kadaoui et al. (2023), which identify the few-shot setting as the most effective for a broad spectrum of Arabic to English translation tasks, we select two examples in this setting. Additional details about our experimental prompt are provided in Table 3 for GPT3.5 and GPT-4, and in Table 4 for Jais. Zero-shot. We evaluate GPT models and Jais in a zero-shot setting with a simple prompt asking the model to translate dialectal Arabic into MSA. Specifically, for the models, we use a temperature of 0.1 and top_p value of 0.4 to ensure a stable and consistent text generation. Few-Shot. We also use GPT3.5 and GPT-4 in the 2-shot setting and Jais in the 1-shot by providing examples from each dialect. We keep the example static throughout the dialect. Combination approach. Furthermore, we noticed that Jais already has a pre-existing familiarity with Jordanian and Palestinian dialects, as evidenced by Table 1. Therefore, we implement a new strategy
exclusively for Emirati and Egyptian dialects (i.e. One-shot for Egyptain and Emirati, and Zero-shot for Jordanian and Palestinian). This particular approach has yielded optimal outcomes across all methodologies.

heading: 4.2 Model Fine-tuning

heading: 4.2.1 NLLB
text: The NLLB model was subsequently fine-tuned using the custom dataset to evaluate whether initially providing it with a basic translation from Google Translate would enhance its performance by simply making corrections to this preliminary output. This approach aimed to determine whether starting with an approximate translation and refining it could lead to more accurate results than other methods.

heading: 4.2.2 Jais
text: We undertook the task of fine-tuning Jais, a pretrained model, utilizing a custom dataset using a LoRa PEFT method. LoRa enables fine-tuning a small number of additional weights in the model while freezing most of the parameters in the pretrained network. One of the main advantages of LoRa is that it preserves the original weights, mitigating the risk of catastrophic forgetting.
This attempt was pursued to evaluate Jais’ efficacy, as it was exclusively trained on Arabic data. However, our findings revealed that Jais model exhibited superior performance without additional training, thereby showcasing its inherent robustness and adaptability across linguistic domains. The results are presented in Table 1.

heading: 5 Results
text: For our results’ analysis, translations were assessed using two main criteria: (i) the performance of MT systems on each dialect separately, and (ii) the average score across all four dialects. This dual evaluation strategy enables a thorough evaluation of system performance, taking into account both dialect-specific intricacies and overall translation quality.
As we can see in Table 1, Jais, with a combination approach setup with Zero- and Few-shot, provides the highest 19.48 average BLEU score in our project. It means this model is already enhanced during the initial training process and provides optimal performance without fine-tuning. Additionally, we obtained results closest to our upper bound us-
ing GPT versions 3.5 and 4 models - with 16.67 and 16.53 respectively.

heading: 6 Discussion
text: In machine translation (MT) across Arabic dialects and Modern Standard Arabic (MSA), our study within the NADI shared task’s Subtask 3 illuminates the challenges and advancements in this domain. The final results for the test set (Table 2) show that a combination approach setup of Jais provides the highest 20.44 average BLEU score. Moreover, the selected GPT-3.5 model within a 2-shot approach obtained 0 for every BLEU score segment, indicating the need for further refinement or exploration of alternative methodologies. The divergence in results of GPT-3.5 model could potentially be attributed to the diverse nature of the test set and the propensity for the model to generate hallucinations, leading to discrepancies in BLEU scores. By evaluating models such as Jais, NLLB, GPT-3.5, and GPT-4, our findings underscore Jais’ remarkable proficiency, surpassing all others in the combination approach setting. A significant factor contributing to Jais’ superior performance is its specific pre-training for the Arabic language, which enhances its ability to handle various dialects effectively. In contrast, models like GPT-3.5 and GPT-4, while powerful, are general-purpose language models not specifically optimized for Arabic. This lack of specialization likely contributes to their lower performance in this specific task. Our study did not include other Arabic-specific models due to the focus on widely-used, general-purpose models to benchmark against a specialized model like Jais. Future work should consider incorporating additional Arabic-specific models to provide a
more comprehensive evaluation and better understand the capabilities and limitations of different approaches in translating Arabic dialects.
Error Analysis. After examining samples of the translated source sentences, we observed that named entities or country-specific words, including dish names, were the primary sources of errors, as illustrated in Figure 1.

heading: 7 Conclusion
text: Subtask 3 of the NADI shared task is a crucial avenue for advancing Arabic dialect machine translation, encouraging exploration and comparison of diverse translation techniques. By fostering the development of novel datasets and evaluating system efficacy across varied dialects, we’ve unearthed robust and universally applicable machine translation approaches, poised to enhance communication for Arabic speakers worldwide. Our findings underscore the profound influence of training data on model proficiency, elucidating the superior performance of models such as Jais in contrast to those not primarily tailored for Arabic, such as GPT and NLLB. As an additional achievement of our project, we took second place among all participants in the competition.

heading: A Appendix

heading: Future Work
text: Future work should consider incorporating additional Arabic-specific models to provide a
more comprehensive evaluation and better understand the capabilities and limitations of different approaches in translating Arabic dialects. Error Analysis. After examining samples of the translated source sentences, we observed that named entities or country-specific words, including dish names, were the primary sources of errors, as illustrated in Figure 1