Dallah: A Dialect-Aware Multimodal Large Language Model for Arabic (2024)

Abstract: Recent advancements have significantly enhanced the capabilities of Multimodal Large Language Models (MLLMs) in generating and understanding image-to-text content. Despite these successes, progress is predominantly limited to English due to the scarcity of highquality multimodal resources in other languages. This limitation impedes the development of competitive models in languages such as Arabic. To alleviate this situation, we introduce an efficient Arabic multimodal assistant, dubbed Dallah, that utilizes an advanced language model based on LLaMA-2 to facilitate multimodal interactions. Dallah demonstrates state-of-the-art performance in Arabic MLLMs. Through fine-tuning six Arabic dialects, Dallah showcases its capability to handle complex dialectal interactions incorporating both textual and visual elements. The model excels in two benchmark tests: one evaluating its performance on Modern Standard Arabic (MSA) and another specifically designed to assess dialectal responses. Beyond its robust performance in multimodal interaction tasks, Dallah has the potential to pave the way for further development of dialect-aware Arabic MLLMs.

heading: 1 Introduction
text: Large language models (LLMs) have revolutionized how machines understand and generate human language. Recent developments have significantly expanded the scope of these models by integrating multimodal data, enabling sophisticated interaction with both textual and visual information. Despite these advances, applying NLP in linguistically diverse environments presents unique challenges, particularly in processing dialectal variations of languages and the integration of these variations within multimodal contexts. These challenges are especially pronounced in Arabic, a collection of languages and varieties characterized by a rich tapestry of dialects that vary significantly
across different regions. Arabic dialects enrich the cultural landscape and present complex linguistic variations that standard NLP models, primarily designed for MSA, often fail to solve. This linguistic diversity requires the development of specialized models that can navigate the rich world of dialectal Arabic and its integration with visual data. Addressing these needs is crucial for improving user interaction and preserving linguistic heritage, as many dialects are underrepresented. Some may be at risk of diminishing in the face of globalization and cultural homogenization.
Multi-cultural and multi-modal LLMs (Alwajih et al., 2024; Huang et al., 2023; Sengupta et al., 2023) are vital against cultural homogenization, where globalization tends to favour dominant languages like Arabic. This can potentially lead to the marginalization or even extinction of less widely spoken dialects (Barnet and Cavanagh, 2014; Ahmedov and Shaxrizoda, 2024; Fahmi and Liska, 2024). By including these low-resource dialects, we can ensure their continued relevance and preserve the rich linguistic diversity of the Arabicspeaking world.
To this end, we introduce a powerful multimodal language model that specifically targets the unique aspects of Arabic dialects. Our model, dubbed Dallah, is built on the foundations of LLaVA (Liu et al., 2023b), an advanced multimodal language model
320
framework. We enhance LLaVA with the linguistic capabilities of AraLLaMA (Alwajih et al., 2024), an LLM proficient in Arabic and English. Dallah is designed to understand and generate content in Arabic, navigating the complex interplay between different dialects and visual information effectively.
The following are the main contributions of our work:
1. We present Dallah, which combines the robust multimodal processing power of LLaVA with the dialectal versatility of AraLLaMA, creating a model uniquely equipped to handle the linguistic and visual challenges presented by Arabic dialects.
2. We introduce a novel data filtering method that optimizes the selection and usage of training data, ensuring that Dallah is fine-tuned with high-quality, relevant multimodal datasets that reflect the linguistic diversity found within the Arab world.
3. Dallah supports wide dialectal coverage, successfully fine-tuning over six major Arabic dialects using limited but highly representative dialectal data.
4. We introduce Dallah-Bench evaluation benchmark for Arabic dialects tailored to assess the efficacy of multimodal language models in real-world applications that require an understanding of dialectal variations.
5. We have also built an understanding of which model from the set {GPT4, GPT4-Turbo, Command-R+} is best suited for evaluating MSA and dialectal data compared to Human evaluation.
The remainder of this paper is structured as follows: In Section 2, we provide an overview of related work. Section 3 details our methodology, including the processes for Arabic dataset translation and filtering, construction of dialectal Arabic datasets, the architecture of Dallah, and the training procedures employed. In Section 4, we describe our implementation details and the benchmarks used for evaluation. Section 5 presents our experimental results, including both quantitative and qualitative analyses. We conclude in Section 6 with a discussion of our findings and future work.

heading: 2 Related Work

heading: 2.1 Large Language Models
text: Recent progress in NLP has been driven by advances in LLMs, starting with the foundational Transformer model (Vaswani et al., 2017). This innovation paved the way for language models like the encoder-based BERT (Devlin et al., 2018), and the decoder-based Generative Pre-trained Transformer (GPT) (Brown et al., 2020), as well as encoder-decoder-based models like T5 (Raffel et al., 2020), which have significantly improved linguistic understanding and performance in complex language processing tasks. The development of models such as OPT (Zhang et al., 2022), LLaMA (Touvron et al., 2023a), LLaMA-2 (Touvron et al., 2023b), GPT-2 (Radford et al., 2019), GPT-3 (Brown et al., 2020), GPT-4 (Achiam et al., 2023), and ChatGPT (OpenAI, 2023) Mistral (Jiang et al., 2023), Mixtral (Jiang et al., 2024), Phi-2 (Javaheripi et al., 2023), Phi-3 (Abdin et al., 2024), and instruction-tuned variants of LLaMA-2 like Alpaca (Taori et al., 2023) and Vicuna (Chiang et al., 2023), have demonstrated the rapid evolution of the field. These models benefit from extensive training on large datasets and tailored instruction sets, enhancing their effectiveness.
Arabic LLMs Building on the global momentum, the scope of LLMs has extended into Arabic language processing. The introduction of Jasmine (Nagoudi et al., 2023) marked a significant milestone, followed by AceGPT (Huang et al., 2023) and Jais (Sengupta et al., 2023), which have enhanced Arabic conversational AI. Recently, AraLLaMA (Alwajih et al., 2024) set a new standard with its proficiency in the Egyptian Arabic dialect, showcasing the flexibility of LLMs in handling linguistically diverse data.

heading: 2.2 Multimodal Large Language Models
text: The integration of computer vision and natural language processing has given rise to Visual Language Models (VLMs). These models merge visual and linguistic data, enhancing tasks that require visual perception and language abilities. Models like CLIP (Radford et al., 2021) bridge the gap between visual recognition and language tasks, demonstrating the effectiveness of cross-modal applications.
Recent advancements show that LLMs improve VLMs. Innovations such as Flamingo (Alayrac et al., 2022), Blip-2 (Li et al., 2023), and
LLaVA (Liu et al., 2023b) have leveraged large image-text pair datasets, enhancing cross-modal coordination and learning efficiency. These models also employ specialized architectural features for better integration. For instance, Flamingo utilizes a perceiver resampler to integrate visual data, and Blip-2 introduces a Q-Former (Li et al., 2023) for aligning visual and language modalities. LLaVA (Liu et al., 2023b) adjusts a linear projection layer to synchronize vision and language modalities. Meanwhile, LLaVA1.6 (Liu et al., 2023a) incorporates extensive instruction tuning and a high-resolution vision encoder, achieving outstanding results across multiple benchmarks.
Arabic Multimodal LLMs In Arabic NLP, Peacock (Alwajih et al., 2024) represents the first work in Arabic-centric MLLM capable of handling Arabic multimodal interaction effectively. Additionally, the multilingual PALO (Maaz et al., 2024) has demonstrated the ability to process and integrate multiple languages, including Arabic, into multimodal contexts.

heading: 2.3 Multimodal Instruction Tuning Datasets
text: Development of MLLMs typically involves two phases. The first phase focuses on aligning visual and linguistic features, utilizing datasets such as COCO (Lin et al., 2014), LLaVA-Pretrain (Liu et al., 2023b), and Laion (Schuhmann et al., 2022). The subsequent visual instruction fine-tuning phase enhances the models’ capabilities to follow complex multimodal instructions. This phase often involves transforming existing datasets into more conversationally relevant formats using advanced LLMs such as GPT-4, as seen in models like LLaVA-Instruct (Liu et al., 2023b) and SVIT (Liu et al., 2024). Recent works utilized GPT-4V (OpenAI, 2023) to generate new captions and questionanswers, such as in ShareGPT4V (Chen et al., 2023), LVIS-instruct4v (Wang et al., 2023) and Allava (Chen et al., 2024). Arabic MLLMs utilized translated versions of LLaVA-Instruct (Alwajih et al., 2024; Maaz et al., 2024) using different tools for translation.

heading: 3 Methodology

heading: 3.1 Arabic Dataset Translation and Filtering
text: In the first step, we aimed to build an Arabic MLLM using Arabic datasets. A major obstacle facing Arabic MLLMs is the lack of resources.
This lack is largely due to the challenges of sourcing relevant Arabic image-text pairs on a large scale. To bridge this resource gap, we have implemented a careful translate-and-filter pipeline consisting of a translation stage and a filtering stage inspired by (Mohamed et al., 2023; Alwajih et al., 2024). This pipeline converts publicly available, English-centric image-text and visual instruction datasets into Arabic while maintaining data quality and preventing error propagation due to translation.
We utilize the latest version of the Google Translate API (Google Cloud) for the translation stage which is the best translation method as shown by (Zhu et al., 2023). We also conducted back translation as required by the subsequent filtering stage. During the filtering stage, we ensure the quality of our translations by employing a sentence embedding model (Meng et al., 2024; Wang et al., 2024). We assess the quality by calculating the similarity of embedding between the original and back-translated sentences for both question and answer pairs, retaining only those translations that meet our quality standards. Essentially, we keep examples with questions and answers above a predefined threshold, which we have empirically set at 80%. Figure 3 illustrates the translation and filtering process. Unlike the methods used in (Mohamed et al., 2023; Alwajih et al., 2024), we employ an English sentence embedding model based on Mistral-7b (Wang et al., 2024) to calculate the similarities. This last model is more powerful than embedding models used in Mohamed et al. (2023); Alwajih et al. (2024) as shown by MTEB leaderboard (Muennighoff et al., 2022). Refer to Figure 2 for examples illustrating our pipeline’s effectiveness.

heading: 3.2 Dialectal Arabic Dataset Construction
text: Due to the absence of Arabic dialectal data tailored for vision tasks, we randomly select six subsets from our translated LLaVA-instruct 150k dataset. We aim to ensure that each subset included diverse content types, such as conversations, complex reasoning, and detailed descriptions, from the LLaVAinstruct dataset. Our goal is to capture the dialects of Egypt, Mauritania, Morocco, Palestine, Saudi Arabia, and Yemen. These dialects represent a broad spectrum of Arabic dialects.
These subsets are then assigned to native professional translators from the aforementioned countries in order to translate them from MSA into their respective dialects. Table 1 displays the number of
samples per country, while Figure 1 illustrates the targeted countries on the map. Refer to A.2 for more details.

heading: 3.3 Architecture
text: The Dallah model follows the structure of LLaVA1.5 (Liu et al., 2023a) and comprises three key elements:
1. Vision Encoder: The vision encoder (Vφ) employs the CLIP-Large model (Radford et al., 2021) to process input images (X) into 576 visual tokens at a resolution of 336x336 with a patch size of 14, producing a sequence of patch features V = {vj ∈ Rdx}Mi=j .
2. Projector: A connector (Pϕ), designed as a two-layer multi-layer perceptron (MLP), maps the visual patch sequences {vj}Mj=1 to the text embedding space {hj}Mj=1, allowing for effective integration between the pretrained LLM and the vision encoder.
3. Language Model (LLM): The Arabic LLM (Fθ), based on AraLLaMA (Alwajih et al., 2024)1 processes sequences of text embeddings {hi}N−1i=0 in the d-dimensional space, outputting corresponding next predictions {hi}Ni=1. A tokenizer and embedding module maps text sequences {yi}N−1i=0 to the embedding space and back to output text sequences {yi}Ni=1.
This structure equips the model to handle various multimodal understanding tasks, taking an image and instruction text sequence as input and generating a text sequence as output. Figure 4 illustrates Dallah architecture.
1AraLLaMA is an enhanced version of LLaMA-2 (Touvron et al., 2023a).

heading: 3.4 Training
text: Training of Dallah consists of three stages: (i) pretraining using data LLaVA-Pretrain (MAS Arabic and English), (ii) visual instruction supervised finetuning using LLaVA-Instruct (Arabic MSA and English), and (iii) further visual instruction supervised fine-tuning using dialectal data. Table 1 details the datasets used in each stage. Training data comprises pairs of images and text (X,Y ), with the text sequence Y formatted as a single-turn in the pre-training phase and multi-turn conversation in the visual Instruction Supervised Fine-tuning stage. Y = (Y 1q , Y 1 a , . . . , Y T q , Y T a ). Here, T represents the number of conversation turns, Y tq the user’s prompt, and Y ta the model’s response.

heading: 3.4.1 Pre-training
text: During this phase, the goal is to enhance the alignment between the vision and text data within the embedding space. For this, image-caption style data (X,Ya) is extracted from the conversation, where X is the image and Ya is a corresponding text description. The probability of generating Ya given the image is calculated as:
p(Ya|X) = Na∏
i=1
Fθ(yi|Pϕ ◦ Vφ(X)),
where Na is the length of Ya. The training objective is to maximize the log-likelihood of Ya autoregres-
sively:
maxϕ
Na∑
i=1
logFθ(yi|Pϕ ◦ Vφ(X)),
This framework permits the adjustment of learnable parameters of projector layers during pre-training. LLM and vision encoder learnable parameters are frozen during pre-training, as shown in Figure 5.

heading: 3.4.2 Visual Instruction Supervised Fine-tuning
text: The full image-text pairs (X,Y ) in their multiturn conversation format are used for fine-tuning. The set of tokens corresponding to the model’s responses is denoted as A = {y|y ∈ Y ta , for any t = 1, . . . , T}. The training objective is to maximize the log-likelihood of the model’s responses autoregressively:
maxθ́,ϕ
N∑
i=1
I(yi ∈ A) logFθ(yi|Pϕ ◦ Vφ(X)),
where N is the total number of tokens in Y , θ́ is a subset of θ , and I(yi ∈ A) is 1 if yi belongs to A, and 0 otherwise. Training the projector layers’ learnable parameters while the vision encoder is kept frozen during this phase. For LLM, we utilize LoRA (Hu et al., 2021) to train the LLM. Figure 5 visualizes both pre-training and visual instruction supervised fine-tuning stages.

heading: 4 Experiments

heading: 4.1 Implementation Details
text: Model Configurations. In this work, we develop the Dallah model based on the LLaVA1.5 framework. Specifically, we employ the CLIP-ViT-L/14
as the visual encoder with a standard resolution of 336 × 336. We also use AraLLaMA, a language model tailored specifically for Arabic, and employ a two-layer MLP as the interface to connect the visual encoder with the LLM.
For the construction of Dallah, a three-stages training process is implemented. Initially, we establish a base MSA MLLM in two stages pretraining and MSA fine-tuning, followed by an adaptation stage for Arabic dialects. The following are details of these different stages:
Stage 1: Pre-training stage. During this initial stage, training is conducted for a single epoch only on the projector as described in 3.4.1 using the translated LCS-558 (Liu et al., 2023b) dataset. This dataset includes data filtered for Arabic and 300K samples of English samples. The optimization is done using the AdamW optimizer with a learning rate of 1× 10−3, combined with a cosine learning rate schedule. The overall batch size is maintained at 32. This phase required approximately 28 hours of training on a single A100 GPU.
Stage 2: Instruction-tuning stage. In this satge, the visual encoder is frozen, tuning is applied to the visual projector, and the LLM is fine-tuned using LoRA as described in 3.4.2. Here we employ the 150K LLaVA-Instruct dataset in English alongside a translated and filtered Arabic version. The learning rate is set to 2 × 10−4 with a batch size of 8, maintaining the same settings as in the first stage. This training phase took around 58 hours using a single A100 GPU.
Stage 3: Dialectal instruction-tuning stage. This stage is similar to stage 2, but is focused on dialectal data for six different Arabic dialects and parallel data for MSA. The settings remain the same as in the second stage with learning rate 2× 10−5, over five epochs. This training phase took approximately 2 hours using a single A100 GPU. Table 1 details the data used in the aforementioned stages.

heading: 4.2 Benchmarks
text: We evaluate our model using two benchmarks: LLaVA-Bench for MSA evaluation and comparison with counterpart Arabic MLLMs and DallahBench to assess the model’s capabilities in six Arabic dialects.

heading: 4.2.1 LLaVA-Bench for MSA
text: The Ara-LLaVA-Bench is the Arabic version of the LLaVA-Bench, translated using Google API
and reviewed by Arabic native annotator. The LLaVA-Bench includes 30 images selected from the COCO2014 validation dataset. Each image is accompanied by three types of questions: conversion, detailed descriptions, and complex reasoning, amounting to 90 questions.

heading: 4.2.2 Dallah-Bench for Dialects
text: We select a subset of 20 questions from the Henna (Alwajih et al., 2024) dataset to assess the model’s response to dialects, naming it DallahBench. We task native professionals from each dialect to translate these from MSA into the aforementioned six Arabic dialects.

heading: 5 Results

heading: 5.1 LLaVA-Bench for MSA
text: We evaluate Dallah using the LLaVA-Bench
benchmark described in 4.2.1, specifically designed for multimodal models. We compare Dallah against two baselines: Peacock, an Arabic MLLM based on the InstructBlip architecture integrated with AraLLaMA, an LLM based on LLaMA-2; and PALO, a multilingual MLLM based on the LLaVA architecture integrated with Vicuna, an LLM based on LLaMA-2. PALO supports Arabic, along with nine other languages. We utilize both model-based evaluation and human evaluation. We describe each of these next.
Model Evaluation. We follow the original methodology of LLaVA-Bench (Liu et al., 2023b) by calling the APIs of three different models: GPT4, Cohere Command R+ 2, and GPT-4-Turbo. We slightly modify the prompts to accommodate Arabic instead of English.
Human Evaluation. We conduct a human evaluation of three models using LLaVA-Bench. We present three well-educated annotators with images and questions-answer pairs generated by Dallah and two other baseline models, Peacock and PALO. To ensure integrity, the names of the models are hidden throughout the evaluation process. Annotators are asked to rate the models’ responses on a scale from 1 to 10, based on correctness3, helpfulness4 and question-answer consistency5. Figure 6 presents the average scores (%) of the models Peacock, PALO, and Dallah as evaluated by four different evaluators: GPT-4, Command R+, GPT-4-Turbo, and Human.

heading: 5.1.1 Analysis
text: We report the main results in Table 2. In the GPT4 evaluation, the scale of scores is higher than in other evaluations. The overall scores for Cohere Command R+ and GPT-4-Turbo are close to those of the human evaluation, with GPT-4-Turbo being the closest numerically to human evaluation.
From Table 2, it is observed that the Dallah model outperforms the baseline models in most dimensions of the LLaVa-Bench across all evaluation methods. Peacock generally showed the lowest performance, which could be attributed to multiple factors, including the scale of training data and
2https://docs.cohere.com/docs/command-r-plus 3Correctness: The accuracy and factual correctness of the model’s response to the given question. 4Helpfulness: The degree to which the model’s response provides useful and informative assistance to the user. 5Consistency: The coherence and logical flow within the model’s responses, ensuring they are free from contradictions.
the model architecture, where the best model in the Peacock suite is based on InstructBlip and was trained with frozen Q-former components.
Dallah and PALO show close results across all evaluations, with Dallah having a slight advantage. Dallah surpasses PALO by an average of 1.74% across all evaluations. Both models share the same LLaVA architecture but differ in their LLMs; Dallah uses AraLLaMA, an Arabic LLM, giving it an advantage, whereas PALO utilizes Vicuna, a multilingual LLM based on LLaMa-2 (Touvron et al., 2023a). The training data are almost identical, though Dallah’s data went through a careful filtering method to ensure quality. Also, Dallah is trained on high-quality human-translated dialectal data.
These results demonstrate Dallah’s effectiveness in MSA, exhibiting strong reasoning capabilities and substantial knowledge.

heading: 5.2 Dallah-Bench for Dialects
text: We assess Dallah’s performance in dialects using Dallah-Bench described in 4.2.2. We employ two types of evaluations: human evaluation and model-based evaluation.
Human Evaluation. To evaluate the model responses related to dialect questions about the image, we ask native speakers from each respective country to score the models on a scale from 1 to 10 using the following criteria:
• Context Accuracy Score: This criterion focuses on the accuracy of the model’s response in relation to the question posed, irrespective of the dialect or language used. It assesses how well the response addresses the content and context of the question.
• Dialect Authenticity Score: This criterion assesses the authenticity of the dialect used in the response, independent of the content’s accuracy. It evaluates how authentically the response represents the specific dialect in question.
Model Evaluation. We craft a prompt to assess Dallah’s responses and subsequently call the APIs of two different models, Cohere Command R+ and GPT-4 Turbo. In the prompt, we request that the evaluator models rate Dallah’s response on a scale from 1 to 10 based on the Dialect Authenticity Score and Content Accuracy Score. We utilize GPT4V to extract a detailed description of the image content and include this description in the prompt to give context to the model. Figure 7 illustrates the prompt used to instruct the models for evaluation.

heading: 5.2.1 Model vs. Human Evaluation
text: In our analysis in Table 3, we compare the performance of Dallah based on two evaluators, Cohere Command R+ and GPT-4-Turbo, against human evaluations across several Arabic dialects. The mean absolute differences in scores for dialect authenticity and content accuracy are calculated to quantify the closeness of model evaluations to human judgments. Cohere Command R+ consistently shows a smaller deviation from human scores, with an average difference of 1.47 in dialect authenticity and 1.36 in content accuracy, compared to GPT-4Turbo’s 1.64 and 1.68, respectively. This suggests that Cohere Command R+ is better aligned with human evaluations, offering a more accurate reflection of human perception in dialect authenticity and content accuracy assessments.

heading: 5.2.2 Dallah Performance on Dialects
text: The evaluation of Dallah’s performance on various Arabic dialects using model-based and human evaluators in Table 3 provides crucial insights into its linguistic capabilities. The dialect authenticity and content accuracy scores indicate that Dallah can generate generally well-received outputs, with some variations across different evaluators and dialects.
The higher ratings from Cohere Command R+ indicate that Dallah excels in producing authentic dialect responses that align well with Command R+’s evaluation framework. However, the lower scores from GPT-4-Turbo reveal that some dialects are underrepresented in its training data, leading to
misunderstandings of dialectal responses and lower scores in content accuracy.
Moreover, the variation within individual dialects and the representation of each dialect in the LLM, along with the limited data used in the finetuning, affect the performance of these systems. It is important to note that spoken dialects often differ from written forms. Written dialects are typically closer to MSA as they lack acoustic cues, which may influence human evaluators. When reading the written model responses, evaluators might perceive them as MSA. When spoken aloud, however, the text may sound dialectal.
Furthermore, we attempt to compare our results with GPT-4V. However, GPT-4V consistently responded in MSA even when prompted with dialects. This highlights our model’s superiority over GPT4V in handling dialectal variation as our model responds in dialect.
Additionally, the human evaluation highlights the importance of incorporating cultural and linguistic subtleties in model assessments. Future work could explore more sophisticated evaluation metrics or integrate additional human feedback to refine Dallah’s performance further. We provide qualitative analysis in A.3. This analysis offers
insights into the model’s handling of MSA and various dialects.

heading: 5.2.3 Feasibility of Model Evaluations for Arabic Dialects
text: Given the findings from the comparative analysis, model evaluations, particularly using Cohere Command R+, demonstrate potential as useful tools for assessing dialect authenticity in Arabic dialects. While these evaluations do not completely align with human judgments, they offer a sufficiently close approximation that can be valuable in scenarios where rapid or large-scale evaluations are necessary. However, for applications requiring accurate understanding and cultural sensitivity, human assessments should ideally complement these model evaluations to ensure accuracy and relevance in the context of specific Arabic dialects.

heading: 6 Conclusion
text: The paper introduces Dallah, an advanced multimodal large language model tailored for Arabic dialects, which demonstrates superior performance in processing both standard Arabic and regional dialects. Developed through innovative data filtering and training, Dallah achieves state-of-the-art performance in the LLaVA-benchmark. Dallah maintains dialect authenticity and content accuracy, showing promising results in benchmarks and evaluations. Extensive testing shows the model’s robustness in MSA and across various dialects and contexts. This model marks a significant step in enhancing Arabic NLP, with future goals to expand dialect coverage and refine evaluation metrics for better user interaction insights.
Limitations
We identify a number of limitations for our work, as follows:
• Representation of Arabic Culture: Vision models, LLMs, and datasets used in building MLLMs inadequately represent Arabic figures, places, and culture. As shown in Figure 10, Dallah struggles with recognizing Arabic figures, unlike those from the USA. This highlights the need for more diverse cultural datasets.
• Hallucination Control: Dallah, like many LLMs, is prone to hallucinations, generating inaccurate information. Advanced techniques
and more robust datasets are needed to mitigate this issue and ensure reliability.
• Dialect Variation and Mixing: The model sometimes mixes similar dialects, such as Yemeni and Saudi, and struggles with dialects close to MSA. This can be improved with more extensive data collection and finetuning.
• Arabic Text Recognition in Images: Dallah cannot effectively recognize Arabic text within images due to the lack of annotated datasets. Developing such datasets is essential to enhance the model’s multimodal capabilities.
Ethics Statement
Energy Efficiency. Our Dallah models, like many large MLLMs, require significant pre-training time and are not energy-efficient. We acknowledge this critical issue and support continued research towards developing energy-efficient models. Data. Our pre-training datasets are translated from publicly available English data, encompassing diverse genres, communities, and varieties. Our Dallah models demonstrate potential in applications involving several Arabic varieties, serving broad populations. Human Annotation. The human annotators involved in this project are Arabic native speakers and well-educated individuals with PhD degrees and extensive NLP experience. No Institutional Review Board (IRB) review or approval was required for this project since we only used publicly available data, which does not require access to any social networking account or password. Applications. While Dallah, like many MLLMs, can be misused. It also holds promise for beneficial applications in education, health, and more. Responsible deployment and use are crucial to maximizing its positive impact. It would also help keep Arabic varieties in use in written form in the digital age.

heading: Acknowledgments
text: We acknowledge support from Canada Research Chairs (CRC), the Natural Sciences and Engineering Research Council of Canada (NSERC; RGPIN2018-04267), the Social Sciences and Humanities Research Council of Canada (SSHRC; 435-2018- 0576; 895-2020-1004; 895-2021-1008), Canadian
Foundation for Innovation (CFI; 37771), Digital Research Alliance of Canada,6 and UBC ARCSockeye.

heading: A Appendices
text: We organize content here as follows:
• Translation and Filtering Details (A.1)
• Dialect Translation Examples (A.2)
• Qualitative Analysis (A.3)
A.1 Translation and Filtering Details
As described in Section 3.1, we employed a careful translation and filtering process to ensure the quality of the Arabic dataset used for training the Dallah model. Figure 2 demonstrates this process, highlighting the importance of maintaining high translation accuracy. Examples with low similarity scores between the original English text and backtranslated English text were removed, as shown in the red rows. These include mistranslations such as “What does my browser load?” and “The bird in the photo is an ice skater,” which were incorrectly back-translated from the original prompts. Conversely, examples with high similarity scores, as shown in the green rows, were retained, ensuring that both the questions and answers remained consistent and accurate. This careful filtering process was crucial in developing a robust and reliable Arabic multimodal language model capable of handling complex dialectal interactions.
A.2 Dialect Translation Examples
Figure 8 showcases examples of translations from MSA to regional dialects from six Arabic-speaking countries: Egypt, Mauritania, Morocco, Palestine, Saudi Arabia, and Yemen. These translations were performed by native speakers, ensuring cultural and contextual accuracy. Such examples highlight the complexities involved in developing a dialectaware multimodal language model like Dallah.
A.3 Qualitative Analysis
The qualitative evaluation of Dallah’s responses showcases its effectiveness in generating accurate and contextually relevant answers across various Arabic dialects. This evaluation is based on examples illustrated in Figures 9, 10, 11, and 12 , highlighting Dallah’s capability to handle MSA
and diverse dialectal interactions in both textual and visual contexts.
In the context of food descriptions in Figure 9, Dallah was asked to describe a traditional dish, including the preparation steps. The response provided a detailed and coherent step-by-step guide, demonstrating the model’s understanding of culinary terms. This example highlights Dallah’s proficiency in generating detailed content. Additionally, in the same figure, Dallah demonstrated the ability to generate detailed and accurate descriptions of an image containing a group of children and was capable of providing potential risks about the activity in the image when asked about potential risks.
As shown in Figure 10, Dallah illustrates its ability to describe the appearance of persons and Arabic figures but fails to identify these figures. In contrast, the model was capable of identifying the US president, which is due to the lack of representation for Arabic figures and culture.
Dallah also demonstrated its ability to manage dialectal variations and maintain contextual accuracy as shown in Figure 11 and 12. When addressing questions in a specific dialect, the model accurately reflected local linguistic features and idiomatic expressions. The ability to switch between dialects and maintain contextual accuracy is crucial for multilingual and multicultural applications, highlighting Dallah’s comprehensive training on diverse dialectal datasets.
Dallah’s qualitative performance underscores its potential as a robust multimodal language model tailored for Arabic dialects. Its capability to generate accurate, contextually relevant, and dialectspecific responses makes it a valuable tool for various applications, from education to cultural preservation. The model’s strength in handling diverse dialectal variations and integrating visual and textual information is particularly noteworthy, paving the way for further advancements in Arabic NLP.

heading: Future Work
text: This model marks a significant step in enhancing Arabic NLP, with future goals to expand dialect coverage and refine evaluation metrics for better user interaction insights. Limitations
We identify a number of limitations for our work, as follows:
• Representation of Arabic Culture: Vision models, LLMs, and datasets used in building MLLMs inadequately represent Arabic figures, places, and culture. As shown in Figure 10, Dallah struggles with recognizing Arabic figures, unlike those from the USA. This highlights the need for more diverse cultural datasets. • Hallucination Control: Dallah, like many LLMs, is prone to hallucinations, generating inaccurate information. Advanced techniques
and more robust datasets are needed to mitigate this issue and ensure reliability. • Dialect Variation and Mixing: The model sometimes mixes similar dialects, such as Yemeni and Saudi, and struggles with dialects close to MSA. This can be improved with more extensive data collection and finetuning. • Arabic Text Recognition in Images: Dallah cannot effectively recognize Arabic text within images due to the lack of annotated datasets. Developing such datasets is essential to enhance the model’s multimodal capabilities