Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks (2024)

Abstract: Multimodal large language models (MLLMs) have proven effective in a wide range of tasks that require complex reasoning and linguistic comprehension. However, due to a lack of high-quality multimodal resources in languages other than English, success of MLLMs remains relatively limited to English-based settings. This poses significant challenges in developing comparable models for other languages, even those with large speaker populations, such as Arabic. To alleviate this challenge, we introduce a comprehensive family of Arabic MLLMs, dubbed Peacock, with strong vision and language capabilities. Through comprehensive qualitative and quantitative analysis, we demonstrate the solid performance of our models on various visual reasoning tasks and further show their emerging dialectal potential. Additionally, we introduce Henna, a new benchmark specifically designed for assessing MLLMs on aspects related to Arabic culture, setting the first stone for culturallyaware Arabic MLLMs. The GitHub repository for the Peacock project is available at https: //github.com/UBC-NLP/peacock.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12753–12776 August 11-16, 2024 ©2024 Association for Computational Linguistics
Multimodal large language models (MLLMs) have proven effective in a wide range of tasks that require complex reasoning and linguistic comprehension. However, due to a lack of high-quality multimodal resources in languages other than English, success of MLLMs remains relatively limited to English-based settings. This poses significant challenges in developing comparable models for other languages, even those with large speaker populations, such as Arabic. To alleviate this challenge, we introduce a comprehensive family of Arabic MLLMs, dubbed Peacock, with strong vision and language capabilities. Through comprehensive qualitative and quantitative analysis, we demonstrate the solid performance of our models on various visual reasoning tasks and further show their emerging dialectal potential. Additionally, we introduce Henna, a new benchmark specifically designed for assessing MLLMs on aspects related to Arabic culture, setting the first stone for culturallyaware Arabic MLLMs. The GitHub repository for the Peacock project is available at https: //github.com/UBC-NLP/peacock.

heading: 1 Introduction
text: Empowered by progress in large language models (LLMs) and foundation models of other modalities, multimodal large language models (MLLMs) now have a remarkable understanding (Alayrac et al., 2022; Li et al., 2023e; Dai et al., 2023; Liu et al., 2023b,a; Zhu et al., 2023). For example, they can handle various complex reasoning tasks spanning from visual question answering to understanding sarcastic comics (Achiam et al., 2023; Yang et al., 2023). These capabilities, however, are mostly seen in models serving the English language. This leaves behind the majority of the world’s languages, furthering an already acute technological divide. We alleviate this challenge for Arabic, a diverse
collection of languages and dialects with a native population of more than 400 million speakers.
More concretely, we draw inspiration from English counterparts (Dai et al., 2023; Liu et al., 2023b) to present a robust family of Arabic MLLMs with powerful vision and language capabilities. Our models adopt the approach of integrating an image encoder with an Arabic text decoder. In our experimental setup, we explore two popular directions for aligning the vision and the language components: one involves employing a fully connected layer as a projection head on top of the vision encoder (Liu et al., 2023b), while the other utilizes a Q-former transformer (Dai et al., 2023). All models are trained in two stages, a pretraining stage and an instruction fine-tuning stage. For the first stage, we curate high-quality pretraining data from publicly available English datasets. We translate these datasets into Arabic and apply a carefully designed pipeline to ensure the quality of our training data. Similarly, we curate and translate an instruction fine-tuning dataset which is essential for achieving reasoning and conversational capabilities.
12753
We showcase the performance of our models across different tasks such as visual question answering (VQA) and visual reasoning. Our models perform much better than a multilingual baseline mBlip (Geigle et al., 2023) on different tasks and datasets, and we set the first comprehensive Arabic vision-language benchmark to facilitate future work in this area. Finally, we demonstrate the promising capabilities of our Peacock models in interacting in dialectal Arabic by conducting a case study on the Egyptian dialect. When fine-tuned on a small set of Egyptian dialect data, our models exhibit an interesting level of proficiency in their dialectal responses when prompted in the same dialect. We hope this acts as a spark for future works in dialectal Arabic vision language models.
To summarize, our contributions in this paper are as follows: (1) We introduce a suite of Arabic MLLMs, dubbed Peacock, capable of instruction following and visual reasoning, in addition to their intriguing dialectal affinity. For developing Peacock, we use existing vision and language models. We also offer a new language model, AraLLaMA, based on LLaMA2-7B (Touvron et al., 2023). (2) We introduce a diverse collection of Arabic translated datasets carefully curated for the training and evaluation of Arabic MLLMs. (3) We adapt the popular LLaVA (Liu et al., 2023b) benchmark and SEED-Bench (Li et al., 2023d) for Arabic MLLMs evaluation. (4) We present Henna, a benchmark designed to measure model capabilities in interpreting images related to Arabic culture.
The rest of this paper is organized as follows: In Section 2, we provide an overview of related work. Section 3 introduces Peacock, our family of MLLMs. In Section 4, we describe our evaluation strategies and benchmarks. In Section 5, we present our experiments, human evaluation, and a comprehensive analysis of our models. We conclude in Section 6.

heading: 2 Related Work

heading: 2.1 Multimodal Large Language Models
text: Progress in MLLMs is largely dependent on advances in LLMs. Refer to Appendix A.2 for more details on LLM-related works. The common trend in recent MLLMs involves integrating an LLM as their text decoder alongside a vision encoder for image understanding. Several approaches were proposed for aligning the vision encoder with the text decoder. Flamingo (Alayrac et al., 2022) and
Otter (Li et al., 2023c), for example, blend a vision encoder with a resampler and a cross-gated attention layer, reducing the computational load in vision-text cross-attention, and enhancing instruction optimization. While BLIP-2 (Li et al., 2023e) and InstructBLIP (Dai et al., 2023), combine a vision encoder with a Q-former and a linear layer, streamlining the cross-modality projection and utilizing learnable query vectors for feature extraction. LLaVA (Liu et al., 2023b,a), on the other hand, pairs a vision encoder with multilayer perceptron (MLP), retaining all visual tokens for comprehensive visual information processing. Finally, the simplest form, illustrated by models such as Fuyu (Bavishi et al., 2023) and OtterHD (Li et al., 2023a), relies solely on a linear layer, operating as basic decoder-only transformers without specialized vision encoders. This diversity in design showcases the innovative approaches in integrating vision and language in MLLMs.

heading: 2.2 Visual Instruction Tuning
text: Following the success of instruction tuning in LLMs, recent works in MLLMs transitioned to visual instruction tuning. MULTIINSTRUCT (Xu et al., 2022) pioneered this transition by creating a multi-modal instruction tuning benchmark dataset that transforms 62 different multi-modal tasks into a unified sequence-to-sequence format. Building on this, LLaVA (Liu et al., 2023b) leveraged GPT4’s adeptness in understanding multimodal textual representations to reformulate image-text pairs into an instruction-following format. Similarly, MIMIC-IT (Li et al., 2023b) focused on generating instruction-response pairs using multi-modal incontext information and a variety of visual scenes. Most recently, M3IT (Li et al., 2023f) converted traditional vision-language tasks into a unified visionto-text framework through manual instruction writing and dataset pre-processing. This includes tasks such as captioning, visual question answering, visual conditioned generation, reasoning, and classification. In their comprehensive survey, Yin et al. (2023) provide an extensive overview of MLLMs, including an evaluation of their performance and capabilities. This paper serves as a valuable resource for researchers interested in the field of MLLMs.

heading: 2.3 Arabic MLLMs
text: The majority of research in Arabic MLLMs focuses mainly on image captioning (ElJundi et al., 2020; Attai and Elnagar, 2020; Afyouni et al., 2021;
Emami et al., 2022; Eddin Za’ter and Talafha, 2022; Elbedwehy and Medhat, 2023; Mohamed et al., 2023). Other areas, VQA for example, remain largely unexplored. This is primarily due to scarcity of publicly available datasets in these areas. As far as we know, the only significant work in Arabic VQA is by Kamel et al. (2023) and explores closedform VQA without attempting generative VQA. We also know of no native Arabic datasets for either image captioning or VQA, with two exceptions: AraCOCO (Mohamed et al., 2023) for image captioning, which is mainly used for evaluation, and AVQA (Kamel et al., 2023) for VQA, which was automatically generated from MSCOCO for Arabic VQA. In many works, translations of either MSCOCO or Flickr8k are utilized (ElJundi et al., 2020; Afyouni et al., 2021; Mohamed et al., 2023).

heading: 3 Peacock

heading: 3.1 Architectures
text: The Peacock family is designed based on the vision components of two architectures, that of InstructBlip (Dai et al., 2023) and LLaVA1.5 (Liu et al., 2023a). For language, our models are integrated with one of two powerful Arabic LLMs, AceGPT (Huang et al., 2023)1 and a new model based on LLaMA2-7B, dubbed AraLLaMA, that we further pretrain on a large Arabic dataset and fine-tune using diverse instructions. Our motivation behind introducing AraLLaMA is to create a model with strong knowledge of the Arabic language and
1In all our experiments, we use the AceGPT-7B-chat. We also limit ourselves to LLMs with 7B parameters due to computational constraints.
culture. More information about AraLLaMA and how it compares to AceGPT is in Appendix A.4. InstructBlip-Based Peacock. Here, our models consist of four key components: (1) A vision encoder based on the ViT (ViT/G-14) model (Dosovitskiy et al., 2020), operating at a resolution of 224×224 and employing a patch size of 14. (2) A Querying Transformer (Q-former) (Li et al., 2023e), designed to link the pretrained vision encoder with the LLM, using the BERT base model (Devlin et al., 2018) as its foundation. (3) A linear layer projector, tasked with aligning the output of the Q-former with the LLM embedding space. (4) An LLM, incorporating one of the two forenamed models, AceGPT or AraLLaMA, both of which are derivatives of the LLaMA2 architecture enhanced for Arabic. Figure 2 illustrates this architecture. LLaVA-Based Peacock. For this setting, models are structured around three primary components: (1) A vision encoder employing the CLIP-Large model (Radford et al., 2021), capable of processing images at a resolution of 336x336 and a patch size of 14, converting these images into 576 tokens. (2) A two-layer MLP projector that aligns the output of the visual and language modalities. (3) And either AceGPT or AraLLaMA. The architecture is shown in Figure 3

heading: 3.2 Pretraining
text: Our models are trained in two stages, a pretraining stage and an instruction fine-tuning stage. The pretraining stage aims to train the alignment module, which projects the visual and textual features into a common embedding space. The models are trained
using our carefully curated text-image pairs dataset. In the case of InstructBlip-based models, only the projection layer, which is the alignment module, is trainable. In contrast, the Q-former, vision encoder and language model parameters are frozen. Meanwhile, for the LLaVA-based models, only the MLP connector is the trainable part, with the CLIP encoder and LLM being frozen.

heading: 3.3 Visual Instruction Fine-tuning
text: After the pretraining stage, the model will only be capable of generating simple captions and descriptions of an image. To give the models the ability to function on tasks requiring visual reasoning and engage in an intelligible visual conversation, we further fine-tune them using instruction datasets. To keep computational costs manageable, we employ the parameter-efficient fine-tuning technique LoRA (Hu et al., 2021). Similar to the previous stage, in addition to the LoRA parameters, only the linear layer is trainable in the case of InstructBlip models, while for LLaVA models, we fine-tune the MLP and apply LoRA to the LLM, following the LLaVA 1.5 training scheme (Liu et al., 2023a). We provide in Table A.7 the number of parameters for the main components of each model in Appendix A.5.

heading: 4 Datasets and Benchmarks

heading: 4.1 Translation and Filtering Pipeline
text: A significant challenge for Arabic MLLMs is lack of available resources, which is due to the difficulty of retrieving relevant Arabic image-text pairs from the internet at scale and absence of suitable image-text relevance filtering methods similar to that of CLIP (Radford et al., 2021)2. To address this resource gap, we introduce a careful translateand-filter pipeline for converting publicly available image-text datasets into Arabic without losing data quality. To this end, we adopt the latest version of Google translate API (Google Cloud). We follow Mohamed et al. (2023) in further assuring high quality of acquired translations using a multilingual sentence embedding model LaBSE (Feng et al., 2020). We calculate the similarity of embeddings between the original and translated sentences (questions and answers), retaining translations that meet a minimum similarity threshold of 80% or greater
2CLIP was used in filtering many English web scraped large-scale datasets (Ordonez et al., 2011; Sharma et al., 2018; Changpinyo et al., 2021).
for both the question and the answer. Figure 4 demonstrates the filtering pipeline. We provide details about our datasets and translation method in Appendix A.3, and sample translations illustrating variations in quality ranging from good to moderate to poor in Figure A.9 (also in Appendix A.3).

heading: 4.2 Pretraining Data
text: Aligning with recent work showing that the quality of LLMs pretraining data is more important than quantity (Gunasekar et al., 2023; Lee et al., 2023), we curate a high-quality text-image pairs dataset collected from publicly available sources. Specifically, we utilize LCS-558K (Liu et al., 2023b) and COCO (Lin et al., 2014) as our pretraining data. LCS-558K encompasses 558k textimage pairs carefully curated from three datasets: LAION (Schuhmann et al., 2021), Conceptual Captions (Sharma et al., 2018), and SBU (Ordonez et al., 2011). COCO is a high-quality dataset comprising 118k images, covering 80 different objects, with five captions per image, all human-annotated. As stated in Section 4.1, all the datasets are translated into Arabic using Google API and further filtered based on their semantic similarity with the original English text.

heading: 4.3 Instruction Fine-tuning Data
text: For the second training stage, we curate another dataset that follows the instructions tuning paradigm as in Liu et al. (2023b). Concretely, the model is asked to respond to a specific instruction or question for each image in the dataset. The first dataset we include is the multi-modal instructions dataset by Liu et al. (2023b). It comprises 150k samples covering conversations, detailed image descriptions, and complex reasoning instructions and responses. This dataset was created us-
ing GPT-4 (Achiam et al., 2023), and the images were taken from the COCO dataset. Additionally, we incorporate the VQAv2 dataset (Goyal et al., 2017) after transforming it to the same instructions and responses format. To incorporate further diverse instructions, we utilize 60k multi-choice questions extracted from LLaVA1.5 mixed instruction dataset (Liu et al., 2023a). This exposes the model to different scenarios, giving it better generalization capabilities. Similar to the pretraining stage, all the datasets are translated using Google API and filtered following our data-cleaning pipeline.

heading: 4.4 Evaluation Benchmarks
text: SEED-Bench. SEED-Bench (Li et al., 2023d) consists of 19K multiple-choice questions, each meticulously annotated by humans. These questions cover 12 evaluation dimensions, addressing the comprehension of both image and video modalities. This study exclusively focuses on the image-only subset of SEED-Bench comprising 14K multiplechoice questions. SEED-Bench is translated via our translation and filtering pipeline as described in Section 4.1. LLaVA-Bench. The LLaVA-Bench (Liu et al., 2023b) comprises 30 images, which the authors randomly select from the COCO-Val-2014 dataset. For each image, three questions are generated, resulting in 90 instances. These questions fall into three categories: conversational, detailed description, and complex reasoning. This benchmark evaluates the model’s performance across conversation, description, and reasoning, using GPT-4 scoring. Henna Benchmark. As Arabic culture may be underrepresented in current English MLLMs datasets, we develop Henna, a new benchmark for testing purposes only. Henna comprises attractions, food, events, and other Arabic-relevant objects, consisting of 1,132 samples that have been manually curated and reviewed to ensure quality and relevance. More details about how we create Henna and how we use it for evaluation are in Section 5.2.4.

heading: 5 Experiments

heading: 5.1 Implementation Details
text: In the first training stage, we use the 916k imagetext pairs described in Section 4.2 to train Peacock models. The pretraining phase spans three epochs with a batch size of 32 and a learning rate of 1e-3. As previously described, all model parameters are kept frozen except for the projection layer in the
case of InstructBlip-based models and the MLP connector for LLaVA-based models. During the second training phase, we utilize the instructions dataset introduced in Section 4.3. The models are further fine-tuned for three epochs with a batch size of eight and a learning rate of 2e-5. As mentioned before, only the introduced LoRA parameters are trainable, with the addition of the projection layer in the case of InstructBlip-based models and MLP connector for LLaVA-based models.
For the training objective, we follow the language modeling approach where the model predicts the next text token given previously predicted text tokens and the visual features. Concretely, our goal is to maximize the probability of the next token or, for mathematical convenience, minimize the negative log-likelihood. The loss is calculated only on the response generated by the model. The instructions and visual tokens are skipped during these calculations.

heading: 5.2 Results and Discussion
text: We evaluate our suite of models on a range of typical vision-language tasks and benchmarks. In addition, we show our models’ performance on our novel Arabic cultural dataset, Henna. Since this is the first work on Arabic MLLMs, we adapt popular benchmarks in the literature to our case. Mainly, these are a VQA-tasks benchmark, LLaVABench (Liu et al., 2023b), and SEED-Bench (Li et al., 2023d). We also evaluate the performance on Henna benchmark and conduct a case study focusing on the Egyptian dialect. This establishes the first comprehensive benchmark for future works in Arabic MLLMs. We further compare our models with the multilingual mBlip model (Geigle et al., 2023) as a baseline for completeness. The mBlip model is trained on 96 languages, including Arabic.

heading: 5.2.1 General VQA
text: In general VQA tasks, the challenge involves answering textual questions about images, requiring learning and integrating visual and textual information. This demonstrates a deep understanding of the interconnectedness between the two modalities. To evaluate performance in general VQA, our validation process includes three datasets: VQAv2 (Goyal et al., 2017), OKVQA (Marino et al., 2019), and GQA (Hudson and Manning, 2019). Notably, evaluation of English VQA tasks is typically performed through online platforms by submitting results. However, this option is un-
available for Arabic-translated data because these platforms only support English and the original datasets. Since the test sets do not contain groundtruth labels, we evaluate held-out validation sets. We follow Geigle et al. (2023) in using “exact match accuracy with open generation" to evaluate our models’ output. The metric considers an answer correct if it matches any of the ground-truth answers. Table 1 shows model accuracy on these datasets under the zero-shot setting for both the filtered and unfiltered(All) versions of the datasets.
It is evident from Table 1 that the top-performing Peacock model, InstructBlip with AraLLaMA LLM, significantly outperforms the best version of mBlip, which integrates the BLOOMZ-7B LLM, by an average margin of 4.5 points. A comparative analysis of all models also reveals significant performance improvements when only the filtered high-quality data is included. This enhancement is consistently observed across all models and tasks, highlighting the crucial role of data quality in the effectiveness of these models.
Furthermore, we observe that the choice of the underlying LLM is has a significant impact on performance. This is the case if we compare integrating AraLLaMA to AceGPT in our overall MLLMs. Specifically, the InstructBlip model integrated with AraLLaMA demonstrates superior performance across all tasks and datasets when using either filtered or unfiltered(All) data in training. This performance disparity is likely attributable to the inherent differences in how these LLMs handle Arabic, with AraLLaMA being more effective due to its extensive training and its ability to align with visual information. In addition, it is worth noting that the performance of Peacock models varies considerably depending on the task, with a general trend of models performing better on

heading: Architecture LLM Conv DD CR Avg
text: the VQAv2 task than on OKVQA and GQA. Such variations can be attributed to each task’s inherent complexities and specific requirements, including the sophistication of the presented questions and the nature of the required visual understanding.

heading: 5.2.2 LLaVA-Bench
text: For evaluation using LLaVA-Bench, we follow the method of Liu et al. (2023b). Table 2 displays our models’ successful performance across the three metrics of the LLaVA-Bench. Despite the limited data and resources, this suggests a burgeoning capability for multi-modal comprehension in Arabic. Under the same training conditions, the integration of InstructBlip with AraLLaMA notably excels within the Peacock suite. It achieves an average score of 82.27 on the GPT-4 scale, a significant 9.4 margin over the LLaVA model combined with AceGPT. As shown in Table 2, all Peacock models surpass the mBlip-BLOOMZ-7B baseline in the three metrics of the LLaVA-Bench.

heading: 5.2.3 SEED-Bench
text: For our third benchmark, we adapt the SEEDBench for Arabic and use it to evaluate our models. Table 3 and Figure 1 present an evaluation of Peacock models across a broad spectrum of visual understanding dimensions within SEED-Bench,

heading: Architecture LLM IA II IN IL IC SU SR VR
text: where a diverse range of performance efficiencies is observed. LLaVA-AraLLaMA emerges as a particularly robust model, excelling in visual reasoning and scene understanding with accuracy scores of 68.88% and 65.52%, respectively. However, it displays weaknesses in spatial relations and instance location. Mirroring this trend, LLaVA-AceGPT showcases strengths in scene understanding and visual reasoning (66.72% and 71.6%, respectively), but marginally underperforms in instance interaction and spatial relations compared to LLaVAAraLLaMA. In contrast, InstructBlip-AraLLaMA, while proficient in scene understanding and Visual Reasoning (59.34% and 64.95%), falls short in Instance Attributes and Counting, resulting in a lower overall accuracy of 46.43%. InstructBlipAceGPT, the model with the most modest performance, achieves its best results in visual reasoning and instance interaction (51.06% and 43.3%), but struggles significantly with instance counting and scene understanding. In contrast to mBlip, which outperforms Peacock models only in one dimension (instance interaction) and achieves the same score in the spatial relation dimension as InstructBlip-AraLLaMA.
This comparative analysis underscores the superiority of LLaVA-based models in the Peacock family on SEED-Bench, especially those with AraLLaMA, over InstructBlip models in most tasks. This could be attributed to the capability of AraLLaMA in understanding and ability of align information coming from the vision encoder on the one hand and the input questions about the input image, on the other hand. Meanwhile, the InstructBlip models, particularly those with AceGPT LLM, reveal limitations in broader visual understanding tasks. The marked variation in performance between AraLLaMA and AceGPT within the same model base highlights the significant impact of language model selection on visual task performance,

heading: Architecture LLM Helpfulness Relevance Accuracy Level of Details
text: offering valuable insights into the inherent abilities (and limitations) in contemporary MLLMs.

heading: 5.2.4 Henna Benchmark
text: Henna was developed to establish a standard for evaluating Arabic MLLMs on elements particularly related to Arabic culture, such as food, customs, and landmarks. The dataset was created by prompting GPT-4V (OpenAI, 2023) to generate descriptions of images based on questions, while providing it with relevant Wikipedia context. Images were carefully selected to represent the culture of 11 Arab countries.
To achieve this, we selected images from Wikipedia and corresponding articles to create the context for GPT-4V during the generation process. The images represent a range of countries, including Algeria, Egypt, Iraq, Jordan, Morocco, Palestine, Saudi Arabia, Syria, Tunisia, the United Arab Emirates, and Yemen. We identified ten top attractions from each country within categories such as traditional food and cuisine, local customs, historical monuments and sites, common activities and lifestyles, and architectural styles and notable buildings. Figure A.10 demonstrates selected examples from the dataset’s images.
For each attraction, we used GPT-4V to generate ten questions. Each image was accompanied by its Wikipedia article to provide comprehensive context. This approach yielded a minimum of ten images per country, resulting in a total of 1,132 question-answer pairs across all countries. An example of the dataset generation process is illustrated in Figure 5, and Figure A.11 demonstrates four randomly selected images with a generated pair of a question and their answers. Moreover, Figure A.13 shows examples of questions and answers generated by the Henna pipeline from the image depicted in Figure A.12. These questions and answers were translated into English to provide a qualitative assessment of the dataset’s quality.
The evaluation process utilizes GPT-4 to assess each model’s responses based on four criteria: Helpfulness, Relevance, Accuracy, and Level of Details. Each criterion is rated on a scale from one to ten, with higher scores indicating better responses.
The evaluation process involves GPT-4 reviewing a question and its correct answer in Arabic, followed by the model’s response, which is then rated according to the aforementioned criteria. The results are formatted as a JSON object with keys corresponding to each criterion. Figure A.17 illustrates examples where GPT-4’s evaluations varied in quality, showing high, medium, and low evaluations for different models’ responses. An example of the prompt used in the evaluation is shown in Figure A.18.
The leading model from the Peacock suite was evaluated against the multilingual model mBlip following our benchmark. The data presented in Table 4 demonstrates the superiority of the InstructBlip model paired with AraLLaMA, setting a benchmark for future models in terms of their ability to comprehend and recognize aspects of Arabic culture. Figure 6 shows an example response from Peacock along with a response from GPT-4V, illustrating the practical application of these findings.
This structured evaluation method, where GPT-4 serves both as the subject and the evaluator, facilitates a quantitative analysis of the model’s performance in understanding and responding to visual questions in Arabic.

heading: 5.2.5 Qualitative Analysis
text: In our qualitative analysis of Peacock models, we select two random samples from each question type previously described in Section 5.2.2, totaling six samples. Figure A.23 displays the answers by all Peacock models to these six questions, accompanied by their corresponding images.
For the conversion type questions, one direct question involves asking about the color of an elephant. While InstructBlip integrated with AceGPT fails to provide the correct answer, all other models succeed. In the second conversion example, the LLaVA-based models are unable to answer a question about counting donuts. In the detail type questions, all models provide answers that are closely related to the details of the objects in the images, albeit with some hallucinations. For the complex type questions, all models provide subjective answers, which, despite offering slightly different conclusions about the image, can still be considered correct. In summary, InstructBlip integrated with AraLLaMA, provides accurate and more helpful answers for most of the three types of questions.

heading: 5.2.6 Case Study with Egyptian Dialect
text: Attention to dialectal Arabic in the NLP research community is not sufficient to date, with complete absence when it comes to MLLMs. Addressing
this gap, we conduct the first study on the capabilities of MLLMs in generating dialectal Arabic, focusing the study on the Egyptian dialect. Out of the box, our fine-tuned models were able to understand questions posed in the Egyptian dialect but responds in MSA. Following this observation, we transform a subset of 1k random samples from our instruction tuning dataset into Egyptian dialect by a professional Egyptian translator. This small dataset is then used to further fine-tune our InstructBlip based Peacock models following the previously mentioned experimental setup. Surprisingly, as seen in Figure 7, our Peacock models are capable of generating solid answers in Egyptian dialect when instructed on this small sample, while keeping their MSA fluency intact. This could be due to the fact that our LLMs have seen dialectal Egyptian Arabic during their pretraining.
To provide a measurable evaluation, we further transform 20 samples from our instruction tuning evaluation set into Egyptian dialect. Using these samples, we appoint four native Egyptian speakers to anonymously score the responses of our models against GPT-4. The evaluation was based on two criteria: the accuracy of the model’s response to the question (rated on a scale from 1 to 10) and the authenticity of the Egyptian dialect (rated on a scale of 1 to 10).
To ensure transparency, the answers from models were anonymized before being presented to the annotators. As shown in Figure 8, Peacock models exhibited greater closeness to the Egyptian dialect compared to GPT-4V, even when the latter was specifically instructed to respond in the Egyptian dialect. On the other hand, our dialectal models lag slightly in the accuracy of the answers, which we assume can be alleviated by providing sufficient training data, a task we leave for future work. More details and examples on the case study are provided in Appendix A.8.
InstructBlip-AraLLaMA model, and "IB-AceGPT"
refers to our InstructBlip-AceGPT model. "GPT-4V DI" is the GPT-4V model explicitly instructed to respond in the Egyptian dialect."GPT-4V" represents the GPT-4V model, which is given a question in the Egyptian dialect, similar to how Peacock models are instructed.

heading: 6 Conclusion
text: In this work, we present the family of Peacock models. Peacock demonstrates significant advancements in Arabic MLLMs, showcasing remarkable abilities in interpreting visual data in Arabic language. These models bridge the gap in multimodal understanding capabilities for Arabic and Egyptian dialects by introducing a suite of models, with various reasoning skills, accompanied by a diverse collection of datasets and benchmarks carefully prepared. This includes our Henna benchmark, designed to assess MLLM tasks focused on the Arabic culture. The development of Peacock sets strong baselines and a new benchmark for future work in Arabic MLLMs, highlighting the importance of high-quality data processing and the selection of language models for multimodal task performance.

heading: 7 Limitations
text: We identify a number of limitations for our work, as follows:
• Peacock models have demonstrated remarkable abilities in interpreting visual data in Arabic. However, these models can struggle with object hallucination, where the generated descriptions or answers may include references to objects that do not exist in the input image, along with unnecessary details.
• Additionally, translation errors can significantly impact the model’s performance and propagate through the training data. We have identified several such errors in the model’s responses. For example, the English word ’sitting’ typically indicates the location of an object. However, the Google API often mistranslates it to suggest that the object is lying down, as seen in the translation of ’The train is sitting at the station’ to é¢jÖÏ @ ú
̄ ËAg. PA¢ ®Ë@, where ËAg. inaccurately implies that the train is lying down.
• Moreover, the Peacock model’s capabilities are further limited in recognizing text within images. This limitation stems from the fact that our training datasets do not include imagetext pairs.

heading: 8 Ethics Statement
text: Energy Efficiency. Our Peacock models, like many large MLLMs, require significant pretraining time and are not energy-efficient. We acknowledge this critical issue and support continued research towards developing energy-efficient models. Data. Our pretraining datasets are translated from publicly available English data, encompassing diverse genres, communities and varieties. Our Peacock models demonstrate potential in applications involving several Arabic varieties, serving broad populations. Human Annotation. Three authors of this paper, all Arabic native speakers with PhD degrees and extensive NLP experience, conducted the human annotation. They are full-time employees of our research group, with data annotation as part of their job duties. No IRB review was necessary as the project used publicly available data without requiring access to any private accounts. Applications. While Peacock, like many MLLMs,
can be misused, it also holds promise for beneficial applications in education, health, and more. Responsible deployment and use are crucial to maximizing its positive impact. It would also help keep Arabic varieties in use in written form in the digital age. AI Usage. ChatGPT was used to corect grammar in some early stages of the paper writing by some of the authors. This utilization was strictly for the purpose of enhancing the linguistic precision. Our research team independently carried out the fundamental research, analysis, development, and paper writing.

heading: Appendices
text: We provide an addition organized as follows:
• Image Attribution A.1.
• Large Language Models A.2.
• Translation and Filtering A.3.
• AraLLaMA A.4.
• Models Parameters A.5.
• Henna Generation Pipeline Examples A.6.
• Qualitative Analysis Examples A.7.
• Case Study with Egyptian Dialect Details and Examples A.8.
A.1 Image Attribution All figures presented in this document are sourced from the COCO dataset and Wikipedia (Wikipedia contributors, 2024), unless otherwise specified.

heading: A.2 Large Language Models
text: The field of LLMs has seen significant growth due to increased data availability and computational capabilities. Initial models such as BERT (Devlin et al., 2018) and T5 (Raffel et al., 2020), along with decoder-focused models like GPT (Radford et al., 2019), utilized the Transformer architecture (Vaswani et al., 2017) to achieve remarkable results in various NLP tasks. The advent of GPT3 (Brown et al., 2020) marked a shift towards decoder-only structures, emphasizing auto-regressive decoding for prediction generation. Following models, including PaLM (Anil et al., 2023), expanded the scope of parameters and dataset sizes. Meanwhile, models like InstructGPT (Ouyang et al., 2022), mistral (Jiang et al., 2023), ChatGPT (ChatGPT, 2024) integrated fine-tuning and reinforcement learning to enhance conversational abilities (Ouyang et al., 2022). More recently, Direct Preference Optimization (DPO) technique has been developed to finetune language models (LMs) to align with human preferences (Rafailov et al., 2023). The advancements in LLMs were not limited to the English language only, recent works in Arabic LLMs showed promising performance. (Nagoudi et al., 2022) presented Jasmine, a strong Arabic text decoder capable of performing Arabic text generation and classification tasks. While (Huang et al., 2023; Sengupta et al., 2023) followed (ChatGPT, 2024) instructions fine-tuning and introduced language decoders based on LLaMA2 (Touvron et al., 2023)
capable of following instructions and holding a coherent conversation in Arabic. These advancements, coupled with contributions from the opensource community, have established new standards and created new opportunities for research in the NLP field.

heading: A.3 Translation and Filtering
text: filtered datasets highlighted in green and cyan are used for pretraining and instruction finetuning, respectively.
In the literature, ElJundi et al. (2020) found that Google translates API translations to be suboptimal, while Mohamed et al. (2023) showed that No Language Left Behind (NLLB) project (Costajussà et al., 2022) performs better than the free Google translate API integrated into Google Sheets. However, our investigation reveals that the latest Google Translate API has superior performance and provides higher-quality translations than NLLB. This observation aligns with the recent findings of Kadaoui et al. (2023).
To evaluate the effectiveness of filtering in our study, we randomly selected a subset of 250 samples from the VQAv2 dataset and performed human annotations to assess the accuracy of question-andanswer (Q&A) translations in both filtered and unfiltered (All) segments. The analysis revealed that out of the 250 samples in the unfiltered segment, 35 were incorrectly translated. In contrast, within
the filtered segment, which comprised 207 samples after the application of a filtering mechanism, only 7 were incorrectly translated. The filter effectively removed 43 samples, of which 31 were inaccurately translated, indicating that the majority of removed items were indeed problematic, while 12 were accurately translated.
To quantify the effectiveness of the filtering process, we calculated several metrics as shown in Table A.6. These results, presented in Table A.6, demonstrate that the filtering approach substantially enhances the quality of the dataset by effectively identifying and removing incorrectly translated Q&A pairs.
Table A.5 presents the statistics of the translated data before and after the filtering process. As the table indicates, the reduction ratio due to filtering is approximately 20%

heading: A.4 AraLLaMA
text: Pretraining Data. AraLLaMA, is an autoregressive pretrained language model based on LLaMA27B. We further train it on a large and diverse Arabic dataset, including all categories of Arabic, namely Classical Arabic (CA), Dialectal Arabic (DA), and MSA. This data is aggregated from various sources: AraNewsv2 (Nagoudi et al., 2020), El-Khair (ElKhair, 2016), Gigaword,3 OSCAR (Suárez et al., 2019), OSIAN (Zeroual et al., 2019), Wikipedia Arabic, and Hindawi Books.4 We also derived ArabicWeb22 (A) and (B) from the open source Arabic text 2022.5 Training Strategy. LLaMA-2’s original vocabulary wasn’t specifically optimized for Arabic and had a limited selection of Arabic words (it includes only 28 Arabic alphabet), leading to inadequate comprehension of Arabic language data. The initial step to address this involved expanding LLaMA2’s vocabulary. Expanding the vocabulary significantly enhanced the efficiency of encoding string sequences and enriched these sequences with more meaningful information, which was especially beneficial for document-level understanding and en-
3LDC Catalog Link 4OpenITI corpus (v1.6) (Nigst et al., 2020). 5ArabicText-2022 data
coding (Li et al., 2023g). However, the limited amount of data available for continual pre-training meant that a substantial increase in vocabulary could introduce words or phrases that lack practical significance, posing a challenge in learning them effectively from the pre-training dataset and negatively affecting the model’s performance. Moreover, a larger vocabulary size would increase the number of embedding-related parameters, which could impact training efficiency. Therefore, after conducting numerous experiments and considering the balance between training quality and efficiency, we increased the vocabulary size from the original 32,000 words in LLaMA-2 to 60,000 for the Arabic version. With the expanded vocabulary, the next step involved initializing the embeddings for the new vocabulary based on the original LLaMA-2 (7B) model. To ensure a smooth transition of capabilities from the original LLaMA-2 to the Arabic LLaMA-2 while keeping the English proficiency unaffected in the initial phase, we used a mean initialization method for the new embeddings, utilizing the weights from the original LLaMA-2. This approach not only preserved the English language capabilities but also facilitated the effective transfer of these capabilities to the Arabic language model, enabling LLaMA-2 to function efficiently in both English and Arabic. Moreover, 30 GB of English and Arabic data, was utilized during pre-training. Instruction Fine-tuning. To enhance capabilities of our AraLLaMA we instruct-tuning it on three datasets: Alpaca-GPT4, Evol-instruct, ShareGPT extracted from MultilingualSIFT datasets (Chen et al., 2023).

heading: A.5 Models Parameters

heading: Model LLM Vision Encoder Q-Former Projection Layer(s) LLM Total
text: In Table A.7, we provide details of the number of parameters for the main components of the Peacock models, including the distinction between trainable and non-trainable parameters for both the pertaining and instruction finetuning stages. These
distinctions are vital for understanding the model’s structure and optimization strategy.

heading: A.6 Henna Generation Pipeline Examples
text: This appendix section includes Figures A.10, A.11, A.13, A.17, and A.18 described in Section 5.2.4.

heading: A.7 Qualitative Analysis Example
text: In this section, we provide an example of qualitative analysis as described in Section 5.2.5, illustrated in Figure A.23.

heading: A.8 Case Study with Egyptian Dialect Details and Examples
text: Figure A.24 displays three randomly selected examples of question-answer pairs translated from MSA into the Egyptian dialect. Figure A.25 presents four examples—three from Henna and one from LLaVA-Bench—along with the correct MSA answers and responses from two models: InstructBlip-AraLLaMA and
InstructBlip-AceGPT , demonstrating the Peacock model’s proficiency in responding in the Egyptian dialect. In Figure A.26, we highlight the superiority of Peacock over GPT-4 when posed with questions in the Egyptian dialect, where GPT-4 responds in MSA while Peacock provides answers in the Egyptian dialect. Conversely, when both models are queried in MSA, they reply in MSA.
Figure A.17: Henna evaluation samples showing GPT-4’s scoring range for responses from different models. Sub-figures illustrate the spectrum of scoring from high to low, based on the accuracy and relevance of the modelgenerated answers to the benchmark questions.
Figure A.23: Selected examples from the LLaVA benchmark include two from the conversational type, two from the detail type, and two from the complex question type. The top-left responses are from InstructBlip-AraLLaMA, the top-right responses from InstructBlip-AceGPT, the bottom-left responses from LLaVA-AraLLaMA, and the bottom-right responses from LLaVA-AceGPT.

heading: Future Work
text: The development of Peacock sets strong baselines and a new benchmark for future work in Arabic MLLMs, highlighting the importance of high-quality data processing and the selection of language models for multimodal task performance