CUFE at NADI 2024 shared task: Fine-Tuning Llama-3 To Translate From Arabic Dialects To Modern Standard Arabic (2024)

Abstract: LLMs such as GPT-4 and LLaMA excel in multiple natural language processing tasks, however, LLMs face challenges in delivering satisfactory performance on low-resource languages due to limited availability of training data. In this paper, LLaMA-3 with 8 Billion parameters is finetuned to translate among Egyptian, Emirati, Jordanian, Palestinian Arabic dialects, and Modern Standard Arabic (MSA). In the NADI 2024 Task on DA-MSA Machine Translation, the proposed method achieved a BLEU score of 21.44 when it was fine-tuned on the development dataset of the NADI 2024 Task on DA-MSA and a BLEU score of 16.09 when trained when it was fine-tuned using the OSACT dataset.

text: Proceedings of The Second Arabic Natural Language Processing Conference, pages 769–773 August 16, 2024 ©2024 Association for Computational Linguistics
LLMs such as GPT-4 and LLaMA excel in multiple natural language processing tasks, however, LLMs face challenges in delivering satisfactory performance on low-resource languages due to limited availability of training data. In this paper, LLaMA-3 with 8 Billion parameters is finetuned to translate among Egyptian, Emirati, Jordanian, Palestinian Arabic dialects, and Modern Standard Arabic (MSA). In the NADI 2024 Task on DA-MSA Machine Translation, the proposed method achieved a BLEU score of 21.44 when it was fine-tuned on the development dataset of the NADI 2024 Task on DA-MSA and a BLEU score of 16.09 when trained when it was fine-tuned using the OSACT dataset.

heading: 1 Introduction
text: The transformer architecture has revolutionized natural language processing, leading to the development of large self-supervised neural networks with tens or hundreds of billions of parameters trained on trillions of tokens. Recent models based on the transformer architecture include GPT series (Brown et al., 2020), LLaMA (Touvron et al., 2023), and others. As of late 2024, the release of LLaMA-3 sparked the growth of a thriving opensource community, resulting in the development of near-state-of-the-art models like Mistral (Jiang et al., 2023) and Mixtral (Jiang et al., 2024).
These LLMs can adapt to new tasks and instructions with minimal finetuning. To save on finetuning costs, the NLP community uses ParameterEfficient Fine-Tuning (PEFT) techniques instead of full-parameter fine-tuning. These PEFT techniques are shown to significantly reduce a model’s computational footprint (Houlsby et al., 2019). LowRank Adaptation (LoRA) (Hu et al., 2021), an innovation among these PEFT techniques, effectively balances performance and efficiency through
low-rank matrix approximations during the finetuning phase. Other PEFT techniques, include LLM model weights compression through quantization techniques, cutting down storage and memory usage during both training and inference. By employing techniques such as NormalFloat Quantization (Chen et al., 2023), these methods aim to preserve model performance while considerably reducing LLMs’ resource requirements, opening avenues for resource-efficient fine-tuning through their integration with PEFT methods like LoRA.
Large Language Models (LLMs) can be finetuned and used for translation of low-resource languages (Lankford et al., 2023). The adaptMLLM system outperformed baselines from the LoResMT2021 Shared Task (Ojha et al., 2021) in translating low-resource languages such as Irish. The improvements were 14% greater in the English to Irish direction (5.2 BLEU points) compared to the Irish to English direction (an increase of 117%, or 40.5 BLEU points).
In this paper, a LLAMA-3 model with an 8- Billions parameters model was finetuned to develop a sentence-level machine translation (MT) of four Arabic dialects into MSA. The rest of the paper is organized as follows. The related work is summarized in Section 2. The dataset used for training and validation was detailed in Section 3. In Section 4, the system is presented. Section 5 summarizes the study’s key findings.

heading: 2 Related Work

heading: 2.1 Arabic Machine Translation
text: Arabic ranks fifth in the number of native and total speakers. In the early nineteenth century, Modern Standard Arabic (MSA) was developed from Classical Arabic as the standardized and academic variant of the language. Modern Standard Arabic, having a standard orthography and used in formal contexts, contrasts with Dialectal Arabic,
769
commonly used unofficially and increasing in online usage. Complex words and certain expressions in Arabic dialects differ greatly from Modern Standard Arabic, evidenced by the use of letter concatenation, character repetition, and emoticons for emphasis (Baniata et al., 2018).
Different techniques were used to perform DA to MSA translation, the ELISSA system (Salloum and Habash, 2013), utilizes morphological analysis, transfer rules, dictionaries, and language models, adopts a rule-based approach for DA to MSA translation, and functions as a general preprocessor for DA when working with MSA NLP tools. 93% of ELISSA’s MSA translations are correct. Improving MSA-pivoted DA-to-English MT with ELISSA for producing MSA versions of DA sentences increases BLEU score by 0.6% to 1.4% on various blind test sets.
In (Alnassan, 2023), the author distinguishes between rule-based and statistical machine translation methods, emphasizing the challenge of accurately translating dialects compared to standard or contemporary Arabic. They propose an “automatic standardization” solution employing machine translation techniques to generate standard Arabic text from dialect inputs. Instead of creating linguistic rules for each dialect, the authors employ statistical models. The goal is to develop integrated software for automatic standardization and translation of Arabic dialects. The authors propose that transforming dialectal text into standard Arabic could aid in the comprehension of various Arabic dialects.
For low-resource languages, different approaches were employed, (Lample et al., 2017) explored the possibility of translating without utilizing any parallel data. This model maps sentences from bilingual corpora into the same latent space. By mastering the ability to rebuild in two languages from a common feature space, the model can translate accurately without relying on labeled data. This model attained BLEU scores of 32.8 on Multi30k and 15.1 on WMT English-French without being trained on any parallel data.

heading: 2.2 LLM Applications
text: Utilizing LLMs for a wide range of NLP tasks is currently popular. The most effective and efficient utilization of these models is yet to be determined. Three primary methods exist for constructing applications utilizing LLMs.
• Zero-shot prompting It is querying prompts that aren’t in LLM’s training data to elicit responses. These prompts often include detailed instructions and a primary question. Crafting precise prompts is essential for maximizing the effectiveness of large language models.
• Few-shot learning Few-shot learning involves giving LLMs a limited number of examples to produce appropriate responses. Zero-shot prompting refers to a method without providing any examples. In few-shot learning, examples are incorporated into the prompt template to guide the model’s response.
• Fine-tuning. The two methods above enable task adaptation without the requirement for additional training on the LLMs, while fine-tuning necessitates further training of the LLMs with task-specific data. When tailored datasets are available, this method is especially advantageous.
According to (Yang et al., 2024), LLMs perform effectively in most NLP tasks based on their examination of ’use cases’ and ’no use cases’ for particular downstream tasks using the mentioned methods.

heading: 2.3 LLMs for Machine Translation
text: LLMs have started to gain prominence in machine translation research. (Hendy et al., 2023) emphasized ChatGPT’s advantage in machine translation through prompting. These models may not always surpass the performance of SOTA MT systems and commercial translators. According to (Zhu et al., 2023), decoder-only LLMs, though competitive, fall behind encoder-decoder-based multilingual NLLB (Costa-jussà et al., 2022).
(Bawden and Yvon, 2023) reveal challenges in machine translation through prompting, including copying issues, mistranslation of entities, and hallucinations. However, in a few-shot learning setting, these limitations can be mitigated.

heading: 3 Data
text: The development dataset provided by OSACT 2024 shared task (Atwany et al., 2024) on DA-MSA translation was employed for fine-tuning the Llama3 8-Billions parameters model. The development
dataset includes comprised a total of 1001 sourceto-target examples, evenly distributed among dialects as follows: 200 Egyptian, 200 Maghrebi, 200 Levantine, 201 Gulf, and 200 Iraqi examples.
The performance of the developed method was assessed using the development and test dataset provided by NADI 2024 shared task (Abdul-Mageed et al., 2024) on DA-MSA translation. The development dataset includes 100 Egyptian, 100 Emirati, 100 Jordanian, and 100 Palestinian sentences, totaling 400 sentences, and the test dataset includes 500 Egyptian, 500 Emirati, 500 Jordanian, and 500 Palestinian sentences, totaling 2000 sentences. The dataset statistics are summarized in table 3.

heading: 4 Methodology
text: The Llama-3 model was fine-tuned with LoRA keeping the Llama-3 weight frozen updating only the LoRA matrices using the following formatted prompt-response triplets: Dialect [dialect], Arabic Dialect Sentence [source] and Modern Standard Arabic Sentence [target]. The dev set, like the training set, is formatted similarly. This prompt customizes the model for the DA-MSA machine translation assignment. The prompt is formatted as follows:

 
 «SYS» You are an expert Arabic proofreader! «/SYS» Translate the following text from the [dialect] dialect to Modern Standard Arabic. [source] 
 [target] 

For example, the first annotation in the OSACT DA-MSA dataset will have the following prompt

 
 «SYS» You are an expert Arabic proofreader! «/SYS» Translate the following text from the Egyptian dialect to Modern Standard Arabic. ø @ P@ ÕÎª
J K 
 ÕÎª J K J » 

During the finetuning of the Llama-3 model, the learning rate of the Adam optimizer was set to 1e-4. The evaluation is conducted every 50
steps with a batch size of 1. The low-rank approximation rank is set to 64, and its scaling factor for adaptation is set to 16 in the LoRA configurations. The model trainable parameters are all linear layers: “q_proj”, “up_proj”, “o_proj”, “k_proj, ‘down_proj”, “gate_proj”, and “v_proj”. A 0.05 dropout is applied in the LoRA layer. The model’s weights are quantized to 4-bit precision and mixedprecision training with float16 and float32 is enabled to reduce memory requirements and accelerate the training process. This model was trained on Google Colab with a single NVIDIA A100 GPU with 40GB of memory. The finetuning of the model and the generation of the test results took almost one hour. The code used for training and generating the results can be found on the following GitHub repository. 1
When the model was fine-tuned using the OSACT dataset, the system achieved a BLUE score of 15.52 on the NADI development dataset, and a BLUE score of 16.09 on the NADI testing dataset. The system results are summarized in table 2. When the model was fine-tuned using the NADI development dataset, the system achieved a BLUE score of 21.44 on the NADI testing dataset. The system results are summarized in table ??
One final comment about the model, when the model is fine-tuned using the OSACT dataset, the OSACT dataset doesn’t have any “Emirati”, “Jordonain” or “Palestinian” sentences, so, during the generation of the translated sentences, the “Emirati” sentence were considered as “Gulf” dialect, and
1https://github.com/MichaelIbrahim-GaTech/NADI2024subtask-3.git
the “Jordanian” and “Palestinian” sentences were considered as “Levantine” dialect.

heading: 5 Conclusions and Future Work
text: In this paper, PEFT methods like quantization and LoRA were used to finetune a LLama-3 model with 8 billion parameters, this model was adapted to translated between 4 different Arabic dialects and Modern Standard Arabic. The proposed method is efficient and it achieved a BLEU score of 21.44 on the NADI 2024 Task on DA-MSA Machine Translation when finetuned using the development dataset of the NADI shared task, and a BLEU score of 16.09 when finetuned on the OSACT dataset.
In the future, three interesting topics can be further explored: (1) We can further explore the effect of changing the model parameters on the results (i.e. learning rate, batch size, LoRA rank, etc.), (2) We can try different LLMs other than Llama-3, and (3) We can assess the proposed method on other Arabic dialects.

heading: Future Work
text: In the future, three interesting topics can be further explored: (1) We can further explore the effect of changing the model parameters on the results (i. e. learning rate, batch size, LoRA rank, etc. ), (2) We can try different LLMs other than Llama-3, and (3) We can assess the proposed method on other Arabic dialects