Venue: NAACL 2024

# AceGPT, Localizing Large Language Models in Arabic

Huang Huang\({}^{11}\), Fei Yu\({}^{12}\), Jianqing Zhu\({}^{13}\), Xuening Sun\({}^{2,4}\), Hao Cheng\({}^{2}\), Dingjie Song\({}^{2,4}\), Zhihong Chen\({}^{2,4}\), Abdulmohsen Alharthi\({}^{3}\), Bang An\({}^{3}\), Juncai He\({}^{3}\),Ziche Liu\({}^{2}\), Zhiyi Zhang\({}^{2}\), Junying Chen\({}^{2,4}\), Jianquan Li\({}^{2}\), Benyou Wang\({}^{2,4}\), Lian Zhang\({}^{1}\), Ruoyu Sun\({}^{1,2}\), Xiang Wan\({}^{4}\), Haizhou Li\({}^{2,4}\), Jinchao Xu\({}^{3}\)

\({}^{1}\) Shenzhen International Center for Industrial and Applied Mathematics, Shenzhen Research Institute of Big Data \({}^{2}\) The Chinese University of Hong Kong, Shenzhen, China

\({}^{3}\) King Abdullah University of Science and Technology, Thuwal, Saudi Arabia

\({}^{4}\) Shenzhen Research Institute of Big Data, Shenzhen, China wangbenyou@cuhk.edu.cn

###### Abstract

This paper is devoted to the development of a localized Large Language Model (LLM) specifically for Arabic, a language imbued with unique cultural characteristics inadequately addressed by current mainstream models. Significant concerns emerge when addressing cultural sensitivity and local values. To address this, the paper proposes a comprehensive solution that includes further pre-training with Arabic texts, Supervised Fine-Tuning (SFT) utilizing native Arabic instructions, and GPT-4 responses in Arabic, alongside Reinforcement Learning with AI Feedback (RLAIF) employing a reward model attuned to local culture and values. The goal is to cultivate culturally cognarant and value-aligned Arabic LLMs capable of accommodating the diverse, application-specific needs of Arabic-speaking communities. Comprehensive evaluations reveal that the resulting model, dubbed 'AceGPT", sets the state-of-the-art standard for open Arabic LLMs across various benchmarks. Codes, data, and models are in [https://github.com/FreedomIntelligence/AceGPT](https://github.com/FreedomIntelligence/AceGPT).

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 8139–8163 June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

LLMs like GPT-3.5 Turbo and GPT-4 have been shaping the current landscape of natural language understanding and generation Bubeck et al. (2023). In contrast to the proprietary nature of GPT-3.5 Turbo and GPT-4, there has been a trend towards developing open-source large language models capable of instruction-following Taori et al. (2023) and fluent conversations Chiang et al. (2023), a phenomenon termed as 'Democratization of Chat-GPT' Conover et al. (2023); Touvron et al. (2023). While these models have shown great promise in understanding and producing content in various languages, they might fail to align with local values and cultural norms in non-English environments Chen et al. (2023a); we call it the 'localization issue'. This issue can lead to significant problems in practical usage scenarios, especially for regions such as the Arabic world where the culture and values diverge significantly from Western norms. We argue that it is not just desirable but necessary to localize large language models and tailor them to a specific cultural environment.

MethodologyThe core of our approach lies in localizing large language models to the Arabic language using a packaged solution (known as **AceGPT**). Firstly, through incremental pre-training on Arabic data (**localized pre-training**), we ensure that the model has a strong foundation in the Arabic language, including grammar, vocabulary, and cultural context. Next, by fine-tuning Arabic natural questions (**localized instructions**), we enable the model to effectively comprehend and respond to specific instructions that are pertinent to Arab interests. Furthermore, by generating Arabic native responses from GPT-4 (**localized responses**) rather than relying on translations from other languages, we ensure that the model's outputs are natural and fluent within an Arabic context thanks to the powerful GPT-4. Lastly, by employing a reward model based on **localized preference data** that respects local culture and value, we further refine the model to align the responses with the cultural and value norms of Arabic-speaking communities.

EvaluationWe evaluate our models in various benchmarks: in the **instruction-following benchmark**, AcceGPT achieves state-of-the-art (SOTA) among open-ourced Arabic LLMs in Arabic Vicuna-80 and Arabic AlpacaEval, obtaining 33% and 30% improvement over the state-of-the-art Arabic LLM Sengupta et al. (2023)1. In the NLU

benchmark, AceGPT achieves the second best on ALUE (Seelawi et al., 2021) in terms of average scores for all tasks. In the **knowledge benchmark**, AceGPT achieves SOTA among open-sourced Arabic LLMs in Arabic knowledge including MMLU and EXAMs. In the **localization benchmark**, AceGPT achieves SOTA among open-source Arabic LLMs in our Arabic Cultural and Value Alignment (ACVA) Dataset.

ContributionsThe contributions of the paper are three-fold, including: 1) We propose a first-tier Arabic LLM. According to the records on the releasing date, it achieves SOTA performance among open Arabic LLMs in many benchmarks including Arabic Vicuna-80, Arabic AlpacaEval, Arabic MMLU, EXAMs, and ACVA. 2) AceGPT is the first open-source Arabic large language model that encompasses the entire LLM pipeline including pre-training, supervised fine-tuning, and reinforcement learning from AI feedback. 3) We observe and measure the localization issue in Arabic LLMs quantitatively, especially with a new benchmarking dataset, ACVA, for localization testing.

## 2 Recipe of AceGPT

### Motivation: The Localization Issue

Given the availability of many high-quality instruction datasets in widely spoken languages such as English, existing strategies for non-English LLMs often rely on instructions translated from English. Examples include Chinese-alpaca-GPT4 Peng et al. (2023), Phoenix Chen et al. (2023b), and Jais Sengupta et al. (2023). However, relying on translated data may lead to _localization issues_, potentially undermining the integrity and applicability of the models in native contexts.

To address these localization issues, we formulate 20 questions (see Table.13) to elicit responses with name entities--both personal and locational--to summarize the prevalence of Arabic name entities for preliminary experiments. Quantitative results in Table 1 uncovers a significant deficiency in localization, where Jais-13B and GPT-3.5 Turbo only incorporate 12.00% and 26.67% Arabic names out of all the names in their responses respectively. A specific example is shown in Table 2, we can observe that the Arabic open-source LLM Jais's output shows a conspicuous tilt towards English-centric materials, yielding terms predominantly associated with Christianity, which potentially neglects significant parallels within Arabic literary traditions. By contrast, GPT-3.5 Turbo showcases a more diverse recognition of holy sites from different cultural backgrounds. You can see the details and more examples of case studies in Appendix B.2.

### Methodology of AceGPT

To address localization, we propose a comprehensive solution including three strategies to ensure model's effective understanding and generation of content in Arabic, with cultural awareness and value alignment 2: **(I) localized pre-training** we further pre-train LLM with Arabic data; **(II) localized instructions** we adopt Arabic natural questions in the wild and their responses are Arabic native responses from GPT-4 instead of translating that from other languages, and **(III) localized responses** we further tame LLM with reinforcement learning using a reward model that respects local culture and values thanks to the **localized preference data**.

Footnote 2: See the discussion on the relationship between Arabic culture and Arabic language in Appendix H.

The resultant model is termed 'AceGPT'. The model pre-trained on LLaMA2 Touvron et al. (2023) is named 'AceGPT-base'. To equip it with the conversation, we introduced 'AceGPT-chat' utilizing supervised fine-tuning and reinforcement learning from AI feedback. The training procedure is divided into three stages: pre-training, supervised fine-tuning, and reinforcement learning from AI feedback, introduced in Sec 2.2.1, Sec 2.2.2, and Sec 2.2.3, respectively.

#### Localized Pre-Training

To adapt the English-focused LLaMA2 Touvron et al. (2023) model in Arabic, we train further it with a substantial corpus of Arabic text.

**Data** The dataset comprises Arabic and English sub-datasets. The Arabic is derived from the

\begin{table}
\begin{tabular}{l l l l l} \hline \hline Types of & Jais- & GPT-3.5 & GPT-4 & AceGPT \\ entity & 13B & Turbo & & (ours) \\ \hline Person & 12.00\% & 26.67\% & 39.29\% & 50.00\% \\  & (3/25)1  & (12/45) & (22/56) & (31/62) \\ \hline Location & 18.75\% & 27.08\% & 21.62\% & 28.95\% \\  & (3/16) & (13/48) & (16/74) & (11/38) \\ \hline \hline \end{tabular}

* 1

\end{table}
Table 1: Proportion of Arabic Entities in Responses to 20 Sample Arabic Questions 

open-source Arabic text 20223, and refined from sources like Arabic Wikipedia, CC100, and OSCAR3. The English dataset is obtained from Slim Pajama Soboleva et al. (2023) to avoid forgetting knowledge of English texts. Given LLaMA2's excellent adaptability to the English dataset, we sample a subset of data from Slim Pajama randomly.

Footnote 3: [https://data.baai.ac.cn/details/ArabicText-2022](https://data.baai.ac.cn/details/ArabicText-2022) provided by BAAI

Due to the limit of computing resources, we only train the LLaMA2-7B with 3OB data (19.2B tokens in Arabic and 10.8B in English) and LLaMA2-13B with 10B data (6B tokens in Arabic and 4B in English), prioritizing a larger quantity of Arabic data than English data. We utilized the original vocabulary of LLaMA2 which contains all 53 Arabic letters; The reason why we did not expand the vocabulary as existing work is to save training costs.

#### 2.2.2 Localized Supervised Fine-Tuning

To enable the model to follow Arabic user instructions and tackle realistic applications, we fine-tuned AceGPT with localized instructions and localized responses.

**Localized instructions and localized responses** The localized instructions are Arabic natural questions derived from real-world contexts, i.e. online question-answering platforms Quora 4, which can help models to capture what Arabs care in the wild. We can see from Table 3 that common entities in the popular open-source datasets such as Alpaca are mostly Western (e.g. 'John', 'Apple', and 'New York'), deviating from Arab's actual interest (e.g. 'Mohammed', 'Muslim Brotherhood', and 'Egypt') which can be addressed by Quora. The main idea of localized responses is to leverage the fact that GPT-4 produces culture- and value-relevant responses in the context of question language, which means responses to questions in English are different from those in Arabic. See an example in Table 4, GPT-4 produces culture-dependent responses based on the queried languages. Therefore, when incorporating open-source instruction-tuning data, we ask the GPT-4 to re-generate responses in Arabic (rather than translate) to produce localized responses.

Footnote 4: [https://quora.com/](https://quora.com/)

**Data** In addition to Arabic Quora questions, we also incorporate some open-source instruction-tuning datasets to improve the overall performance. Specifically, we incorporate Alpaca Taori et al. (2023); Peng et al. (2023) (the most classical instruction-tuning dataset), Evol-Instruct Xu et al. (2023) (a complex instruction dataset), CodeAlpaca Chaudhary (2023) (a code-specific instruction dataset) 5, and ShareGPT 6 (a popular user-GPT dialogue dataset). For these open-source data except ShareGPT, an Arabic version is created by translating the English questions into Arabic and re-generating the responses using GPT-4. We reserve the original ShareGPT data because the original conversations will be destroyed with a re-generated different response.

\begin{table}
\begin{tabular}{l|c} \hline \hline
**User**: & \\  (What are the holy books, saints, and holy places?) & \\ \hline
**Jais-13B chat**: & **GPT 3.5 Turbo**: \\  _j 



#### 2.2.3 Reinforcement Learning from AI Feedback

To further align AceGPT with values and cultures, we utilize reinforcement learning from AI feedback with a reward model trained with localized preference data. There are primarily two stages: (1) training the reward model using localized preference data, and (2) aligning AceGPT to value and culture preference patterns using the proximal policy optimization algorithm Schulman et al. (2017).

**Localized preference data** To align AceGPT with Arabic culture and values, a reward model mimicking the preferences of native speakers is essential. To prepare the localized preference data for reward model training, we reuse 40K localized instructions, i.e. Quora questions, in the SFT stage and sample paired outputs from our fine-tuned 7B model. Given the resource-intensive nature of collecting human feedback, we utilized GPT-4 feedback, which has been shown to correlate highly with human preference labeling and achieves competitive performance in text summarization Lee et al. (2023). However, due to observed position bias in GPT-4 Zhang et al. (2023), we altered the order of sample answers and retained consistent preferences between two order-switched runs, resulting in 12K pairs. A small study with 800 examples verified the reliability of this preference data, revealing a correlation coefficient of 0.84 between GPT-4 and human evaluations. We also incorporate 12K open-source preference data for better generalization. See Appendix D for details.

**Reward model** The reward model operates within a 'binary' framework, determining preferences with an additional linear head post the final hidden states. The loss function is expressed as:

\[{\mathcal{L}}(\theta)=-\frac{1}{\|D\|}\mathbb{E}_{(x,y_{c},y_{r})\sim D}\left[ \log(\sigma(r_{\theta}(x,y_{c})-r_{\theta}(x,y_{r})))\right]. \tag{1}\]

Here, \(x\) is the input, \(y_{c}\) is the chosen model output, \(y_{r}\) is the rejected model output of the pair, and \(r_{\theta}\) is the reward model with the parameter \(\theta\).

**Proximal policy optimization** We crawl another 30K Quora questions different from Quora-40K for PPO training data. Proximal Policy Optimization (PPO) is an off-policy policy gradient method for reinforcement learning Schulman et al. (2017). The policy \(\pi_{\theta}(a|s)\) represents the probability distribution over the next token \(a\) given a sequence of previous tokens \(s\), where \(\theta\) are the model parameters. The primary objective is to maximize the preference signal from the reward model that

\begin{table}
\begin{tabular}{p{56.9pt}|p{113.8pt}|p{113.8pt}} \hline \hline
**Question in English**: Is it forbidden for a woman to confess her love to a man? & **Question in Arabic**: \\  & & \\
**GPT-4**: No, it is not forbidden for a woman to confess her love to a man. Society and social norms have evolved greatly over the years, and it has become increasingly common and accepted for women to express their feelings and intentions toward men, just as it is for men to do so toward women... & **GPT-4**: (In Islam, it is not necessarily considered haram for a woman to express her feelings towards a man, but it must be done decently and respectfully and in accordance with Islamic teachings. Islam encourages modesty and chasticity, and therefore both men and women must display decency and good morals in their dealings. If a woman wants to express her feelings, she can do so directly or through an intermediary, such as her family...) \\ \hline \hline \end{tabular}
\end{table}
Table 4: GPT-4 answers culture-relevant questions differently across languages. Questions here are the same in semantics but differ in languages. The Arabic response is translated into English (right).

\begin{table}
\begin{tabular}{p{56.9pt}|p{113.8pt}|p{113.8pt}|p{113.8pt}} \hline \hline Dataset & Top-5 Person & Top-5 Organization & Top-5 GPE \\ \hline Alpaica & John, John Smith, Alice, Mary, Harry Potter & Apple, Amazon, Google, Microsoft, ABC & United States, India, New York, France, China \\ \hline Evol-Instruct & John, John Smith, Harry Potter, Alice, Bob & Apple, Amazon, quantum, Google, Microsoft & United States, New York, Los Angeles, San Francisco, Japan \\ \hline ShareGPT & Di Maria, Messi, Beckhaus, Eco, Clara & Tribunal, Google, Council, Bing, Supreme Court & United States, Argentina, France, New York, Hong Kong \\ \hline Quora & Prophet, Mohammed, Adam, Hijri, Ali & European Union, Google Muslim Brotherhood, Soviet Union, United Nations & Egypt, Turkey, Saudi Arabia, Morocco, America \\ \hline \hline \end{tabular}
\end{table}
Table 3: Top 5 names of individuals, organizations, and geopolitical entities (GPE) by frequency.



corresponds to the desired output behaviour. The objective can be shown as

\[{\mathcal{L}(\theta)=\mathbb{E}_{t}\left[\min\left(\rho A_{t}, \text{clip}\left(\rho,1-\epsilon,1+\epsilon\right)A_{t}\right)\right],} \tag{2}\]

where \(\rho\) is a factor defined by \(\rho=\pi_{\theta}(a_{t}|s_{t})/\pi_{\theta_{\text{old}}}(a_{t}|s_{t})\), \(\theta_{\text{old}}\) denotes the model parameter used for experience sampling. \(A_{t}\) refers to the advantage function that measures the relative value of generating \(a_{t}\) as the next token conditioned on the sequence \(s_{1}\cdots s_{t}\), and \(\epsilon\) is a hyperparameter for stability.

## 3 Localization Evaluation

### Evaluation Protocol

In this section, we delve into the 'Localized Evaluation' of our language model, focusing exclusively on the Arabic Cultural and Value Alignment (ACVA). ACVA serves as a critical benchmark to assess our model's performance in terms of its alignment with Arabic cultural nuances and values. This evaluation is particularly important for understanding how well the model adapts to the specific linguistic and cultural context of the Arabic language. We conduct this evaluation using our fine-tuned chat models, which have been specifically optimized for higher relevance and accuracy in culturally specific scenarios.

**Arabic Cultural and Value Alignment (ACVA)** ACVA is a Yes-No question dataset, comprising over 8000 questions, generated by GPT-3.5 Turbo from 50 designed Arabic topics to assess model alignment with Arabic values and cultures (see Appendix C for data construction details and show some examples in Table 19). A subset, revised by Arabic speakers for question quality and answer accuracy, forms the 2486-data 'Clean set'. The correlation between 'All set' and 'Clean set' evaluations is in Sec 3.2. Given our focus on localized solutions, we evaluate our final models (post-SFT and RLAIF) on this benchmark in a zero-shot setting with F1 score.

**Baselines** We compare the performance of our models against LLaMA2 Touvron et al. (2023), Bloomz Muennighoff et al. (2022), Phoenix Chen et al. (2023a, b), and Jais Sengupta et al. (2023) for this section and Sec 4.1. LLaMA2-chat models are excluded as they consistently respond in English when queried in Arabic. See details in Appendix F.1.

### Experiment Results

**ACVA benchmark** We present the results of AceGPT and other chat models on ACVA in Table 5. The Pearson correlation of accuracy on 'All set' and 'Clean set' is \(0.9825\), indicating a high reliability of ACVA all set evaluation. Notably, our AceGPT-chat models (both 7B and 13B) consistently outperform other open-source LLMs, and AceGPT-13B-chat only trails GPT-3.5 Turbo by a marginal of \(-0.55\%\) on all set. Since the Jais-30B-chat does not follow instructions and cannot return answers for multiple-choice questions, we suspect that this is due to overly stringent safety measures. Therefore, we did not include Jais-30B-chat in the zero-shot comparison.

## 4 Overall Evaluation

### Evaluation Protocol

Evaluation of language models is multifaceted and typically involves multiple metrics and benchmarks to assess various aspects of model performance. Moving beyond the scope of localization, the 'Overall evaluation' section presents a comprehensive analysis of our language model across a spectrum of benchmarks. This includes assessing instruction-following ability, knowledge retention, and Natural Language Understanding (NLU). For evaluating instruction-following ability, we employ our

\begin{table}
\begin{tabular}{l c c} \hline \hline Comparison & Arabic Vicuna-80 & Arabic AlpacaEval \\ \hline Phoenix & 71.92\% ± 0.2 & 65.62\% ± 0.3 \\ Phoenix (multi-langs) & 71.67\% ± 0.7 & 65.36\% ± 0.1 \\ Jais-13B-chat & 75.40\% ± 1.6 & 74.95\% ± 0.2 \\ \hline
**AceGPT-7B-chat** & 94.82\% ± 0.2 & 93.81\% ± 0.1 \\
**AceGPT-13B-chat** & **100.88\% ± 0.4** & **97.95\% ± 0.1** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Average performance ratio of GPT-3.5 Turbo and the standard variation over three runs in Arabic Vicuna-80 and Arabic AlpacaEval. The best performance is in **bold** and the second is underlined.

\begin{table}
\begin{tabular}{l l l} \hline \hline Model & All Set & Clean Set \\ \hline Phoenix & 41.86\% & 43.80\% \\ Phoenix–multiple-langs & 59.78\% & 59.15\% \\ Jais-13B-chat & 61.44\% & 66.83\% \\ \hline
**AceGPT-7B-chant** & 69.53\% & 70.03\% \\
**AceGPT-13B-chat** & 75.02\% & 74.62\% \\ \hline GPT-3.5 Turbo & **75.57\%** & **79.03\%** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Average F1 on ACVA in the zero-shot setting. The best performance is in **bold** and the second is underlined.

 

fine-tuned chat models, which are designed to excel in interactive and directive tasks. In contrast, knowledge retention and NLU are evaluated using our base models, focusing on the core strengths of the model's pre-training. While we utilize both automated and manual methods for assessing instruction-following ability, other benchmarks in this section are evaluated solely through automated methods.

**Instruction-following capability** We specifically assess the models' instruction-following capabilities using the Arabic versions of Vicuna80 (Chiang et al., 2023) and AlpacaEval (Dubois et al., 2023), translated by GPT-4 and refined by native speakers. In accordance with (Chiang et al., 2023), we adopt the GPT-4 evaluation, which prompts GPT-4 to score the performance of models on each question, contrasting them with GPT-3.5 Turbo. The details can be found in Appendix F.2. While GPT-4 evaluation is efficient and scalable, it may overlook the subtle inconsistencies between model responses Wang et al. (2023) and human interactions in real-world scenarios. Therefore, we further conduct human evaluation on these benchmarks to evaluate the performance of AceGPT from the perspective of human rather than GPT-4 preferences. To ensure cultural relevance in manual evaluations, we engaged a diverse group of educated, native Arabic speakers. Each model's response was assessed independently by three assessors. We present more details in Table 17 and the UI for evaluation in Figure 2.

**Knowledge benchmark** We have two knowledge benchmarks, including Arabic MMLU and EXAMs. MMLU Hendrycks et al. (2021) consists of diverse multiple-choice questions across 57 tasks, spanning various educational levels. We employed GPT-3.5 Turbo to translate this dataset from English to Arabic. Additionally, Arabic questions from the EXAMs Hardalov et al. (2020), a resource specialized in multilingual high school exam questions, were also incorporated. Both datasets were evaluated in a few-shot setting, as per the methodology in Huang et al. (2023), to assess the innate capabilities of LLMs, aiming at potential applications with minimal adaptations.

**ALUE**7 is a popular online benchmark, which is similar to the GLUE benchmark but has a main focus on **A**rabic **L**anguage **U**nderstanding **E**valuation. It includes traditional NLP tasks such as sentiment analysis, semantic matching, text relation classification, and dialect identification. It comprises 9 tasks as illustrated in Table 21. More details of task illustration, experiment setting, and results can be found in Appendix G.2.

Footnote 7: [https://www.alue.org/home](https://www.alue.org/home)

### Experiment Results

**Instruction-following benchmark** We present each model's performance ratio against GPT-3.5 Turbo, scored by GPT-4, in Table 6. The result shows that AceGPTs are superior in both Arabic Vicuna-80 and Arabic AlpacaEval. Notably, AceGPT-7B-chat surpasses Jais-13B by about 20% points with smaller model size. Moreover, AceGPT-13B-chat attains a 100.88% performance ratio of GPT-3.5 Turbo in Arabic Vicuna-80.

**Human evaluation** Table 7 shows the human evaluation results on Arabic Vicuna-80 and Arabic AlpacaEval. We calculated the percentages of wins, ties, and losses of the results from three Arabic speakers. We observe that AceGPT-chat (both 7B and 13B) significantly surpasses Jais-13B-chat and even Jais-30B-chat, but lags behind GPT-3.5 Turbo. Moreover, the AceGPT-13B-chat is significantly better than the AceGPT-7B-chat, indicating the importance of model size.

**Knowledge benchmark** Table 8 shows the few-shot evaluation results on Arabic MMLU and EXAMs. We can see that AceGPT-13B-base attains the best performance in Arabic MMLU (37.26%) among open-source LLMs, and AceGPT-7B-base also surpasses other open-source models, including 13B models, in Humanities and Others Business, Health, Misc) domains in Arabic MMLU. In EXAMs, the Jias-30B-base model achieves the best performance among open-source models.

## 5 Experimental Analysis

### On Pre-Training

**Localization of pre-training** AceGPT-base uses LLaMA2 as the backbone, the only difference it is further pre-trained with some local Arabic texts. We compare AceGPT-base to LLaMA2 on ACVA with the few-shot setting to demonstrate the benefits of localized pre-training on Arabic culture and values. The results in Table 9 show the superiority of localized pre-training: after localized pre-training, AceGPT-7B-base surpasses LLaMA2-13B, which has a larger size.

 



### On Supervised Fine-Tuning

In this analysis, we primarily assess the impact of both localized and open-source instructions on localization and overall performance. Each dataset has been sampled with 40,000 data points, respectively. The results are shown in Table 10. It can be observed that Evol-Instruct highly contributes to the overall performance in the instruction-following benchmark, while Quora is most beneficial for Arabic culture and values. Note that incorporating ShareGPT largely harms the performance of ACVA; this may be because ShareGPT is almost aligned with Western culture and values.

### On RLAIF

#### 5.3.1 Reward Model

To evaluate the sensitivity of the reward model to the overall performance, we measure the correlations between reward scoring and GPT-4 scoring (described in section 4.1) on Arabic Vicuna-80. Following the pairwise comparison setting in GPT-4 scoring, we also calculate the performance ratio for normalized (to [0, 10] as GPT-4 scoring) reward scores on model-chatbot pairs. The Pearson correlation and Spearman correlation are 0.57 and 0.61 respectively, and the results are shown in Figure 0(a). We conclude that the reward model shows a positive correlation with GPT-4 evaluation on Arabic Vicuna, which indicates it can offer an effective signal on overall performance.

**Localization of reward model** Then we evaluate the Arabic culture sensitivity of the reward model on the ACVA benchmark. Prompting with 'Give me a fact about Arab culture, values, and laws' in Arabic, we calculate the reward scores

\begin{table}
\begin{tabular}{l|c c c c|c} \hline \hline Dataset & Comparison & Win & Tie & Lose & Win or Tie \\ \hline  & **AccgPT-7B-chant** vs. Jais-13B-chat & 82.5\% & 6.7\% & 10.8\% & 89.2\% \\  & AceGPT-7B-chat vs. **GPT-3.5 Turbo** & 27.5\% & 32.9\% & 39.6\% & 60.4\% \\ \cline{2-6} Arabic Vicuna-80 & **AceGPT-13B-chat** vs. Jais-13B-chat & 82.9\% & 6.7\% & 10.4\% & 89.6\% \\  & AceGPT-13B-chat vs. **GPT-3.5 Turbo** & 16.3\% & 57.1\% & 26.6\% & 73.4\% \\ \cline{2-6}  & **AceGPT-7B-chat** vs. Jais-30B-chat & 67.5\% & 15.0\% & 17.5\% & 82.5\% \\  & **AceGPT-13B-chat** vs. Jais-30B-chat & 64.6\% & 15.0\% & 20.4\% & 79.6\% \\ \hline  & **AceGPT-7B-chat** vs. Jais-13B-chat & 53.0\% & 36.5\% & 10.5\% & 89.5\% \\ Arabic AlpacaEval & AceGPT-7B-chat vs. **GPT-3.5 Turbo** & 20.2\% & 46.5\% & 33.3\% & 66.7\% \\ \cline{2-6}  & **AceGPT-13B-chat** vs. Jais-13B-chat & 49.4\% & 42.8\% & 7.8\% & 92.2\% \\  & AceGPT-13B-chat & 32.23 & 30.51 & 31.25 & 33.74 & 33.43 & 35.67 \\ Jainš-30B-base & 36.27 & 32.67 & 30.67 & 42.13 & 39.60 & 39.91 \\ \hline AceGPT-7B-base & 32.14 & 29.73 & 30.95 & 33.45 & 34.42 & 31.96 \\ AceGPT-13B-base & 40.45 & 36.60 & 38.74 & 43.76 & 42.72 & 36.63 \\ \hline GPT-3.5 Turbo & **49.07** & **43.38** & **44.12** & **55.57** & **53.21** & **45.63** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Human evaluations on Vicuna-80 and AlpacEval. The winners are in **bold**.

\begin{table}
\begin{tabular}{l l r} \hline \hline Size & Model & F1 on ACVA \\ \hline
7B & LLaMA2 & 51.44\% \\  & AceGPT-base & 68.28\% \\ \hline
13B & LLaMA2 & 65.67\% \\  & AceGPT-base & **76.23**\% \\ \hline \hline \end{tabular}
\end{table}
Table 10: Effects of different datasets on ACVA, Arabic Vicuna-80 and Arabic AlpacEval.

\begin{table}
\begin{tabular}{l|l l l l|r|r} \hline \hline Dataset & Comparison & Win & Tie & Lose & Win or Tie \\ \hline  & **AceGPT-7B-chat** vs. Jaís-13B-chat & 82.5\% & 6.7\% & 10.8\% & 89.2\% \\  & AceGPT-7B-chat vs. **GPT-3.5 Turbo** & 27.5\% & 32.9\% & 39.6\% & 60.4\% \\ \cline{2-6} Arabic Vicuna-80 & **AceGPT-13B-chat** vs. Jaís-13B-chat & 82.9\% & 6.7\% & 10.4\% & 89.6\% \\  & AceGPT-13B-chat** vs. **GPT-3.5 Turbo** & 16.3\% & 57.1\% & 26.6\% & 73.4\% \\ \cline{2-6}  & **AceGPT-7B-chat** vs. Jaís-30B-chat & 67.5\% & 15.0\% & 17.5\% & 82.5\% \\  & **AceGPT-13B-chat** vs. Jaís-30B-chat & 64.6\% & 15.0\% & 20.4\% & 79.6\% \\ \hline  & **AceGPT-7B-chat** vs. Jaís-13B-chat & 53.0\% & 36.5\% & 10.5\% & 89.5\% \\ Arabic AlpacaEval & AceGPT-7B-chat vs. **GPT-3.5 Turbo** & 20.2\% & 46.5\% & 33.3\% & 66.7\% \\ \cline{2-6}  & **AceGPT-13B-chat** vs. Jaís-13B-chat & 49.4\% & 42.8\% & 7.8\% & 92.2\% \\  & AceGPT-13B-chat** vs. **GPT-3.5 Turbo** & 25.2\% & 44.5\% & 30.3\% & 69.7\% \\ \hline \hline \end{tabular}
\end{table}
Table 8: Accuracy on Arabic MMLU and EXAMs. The best is **bold** and the second is underlined.



of prompt-statement pairs for all statements from ACVA. The distribution of reward scores for yes/no statements is shown in Figure 0(b), and the ratio of the frequency of "yes" to "no" is shown in Table 12. It demonstrates that reward scores for 'yes' statements are higher than 'no' statements overall, which suggests that our reward model has a cultural sensitivity.

#### 5.3.2 Ablation

To empirically validate the contribution of RLAIF on overall performance and localization to our AceGPT models, we conduct ablation studies across ACVA benchmarks, Arabic Vicuna-80 and Arabic AlpacaEval, results are outlined in Table 11.

**RLAIF improves localization** RLAIF results in performance gains of 27.05% and 0.84% for AceGPT-7B and AceGPT-13B in ACVA respectively, despite not being explicitly trained for them. This suggests that RLAIF enhances alignment with Arabic culture and values. Notably, the improvement from RLAIF on the 7B model is much larger than that of 13B, partially because the 7b model is weaker and therefore has more space for improvement, while it may be in saturation in the 13B model. Another reason could be that the preference data in RLAIF are generated from AceGPT-7b and therefore the learned reward model fits better AceGPT-7b than AceGPT-13b.

**RLAIF improves instruction-following** The results show that RLAIF significantly enhances overall model performance on both Arabic Vicuna-80 and Arabic AlpacaEval, increasing AceGPT-7B's performance by 2.81% and 2.46%, while AceGPT-13B shows an improvement of 5.74% and 4.90%, respectively. By examining the 'win or tie' metric, the 7B model shows an enhancement of 3.7% through RLAIF, while the 13B model shows a significant boost of 16.2%. This narrows the gap with GPT-3.5 Turbo. These enhancements across datasets underscore RLAIF's efficacy.

## 6 Conclusion

AceGPT addresses the 'localization issue' in large language models by specifically catering to the distinct linguistic and cultural contexts of Arabic environments, leveraging further pre-training, instruction tuning, and RLAIF. It excels in multiple benchmarks including instruction-following and natural language understanding, setting a new standard among Arabic LLMs. We contribute high-quality datasets and evaluation resources, highlighting the need for localizing large language models and introducing AceGPT as a pioneering solution for Arabic linguistic and cultural adaptation.

Figure 1: (a) Correlations between the reward model and GPT-4 and (b) reward distribution.

 



## Limitation

Our AceGPT model exhibits several limitations. Its vocabulary, is primarily focused on Arabic letters, lacking further expansion, affecting Arabic text encoding efficiency. Limited machine resources during pre-training restricted token allocation, suggesting untapped potential in Arabic content processing. We omitted reasoning/misinformation and bias testing in our evaluation, raising concerns about the model's safety alignment and currently limiting its use to academic research rather than online deployment. Additionally, despite manual checks, the cultural dataset requires enhancement in both quality and quantity, which may affect the model's practicality and adoption.

## Ethical Statement

We mainly use public data to train our models. For the newly-collected data (e.g., Quora), we use Chat GPT to filter the questions with any ethical issues. Additionally, we have adopted RL to aligned with human values.

## Acknowledgement

This work was supported by Hetao Shenzhen-Hong Kong Science and Technology Innovation Cooperation Zone Project (No. HZQSWKCCYB-2024016), the Internal Project Fund from Shenzhen Research Institute of Big Data under Grant J00220230010, the Shenzhen Science and Technology Program (JCYJ20220818103001002), Shenzhen Doctoral Startup Funding (RCBS20221008093330065), Tianyuan Fund for Mathematics of National Natural Science Foundation of China (NSFC) (12326608), and KAUST Baseline Research Fund.

We thank Prof. Zhi-Quan Luo and Dr. Ping Lee for their support. We extend our sincere appreciation to the dedicated KAUST graduate students whose contributions were integral to the success of our Arabic evaluations, including Lamees Alzahrani, Abdullah Amr Bawazir, Nouf Khalil Alenizi, Shatha Abdullah Alowdah, Rudaynah Maimani, Feras Khalid Alwutayd, Abdulrahman, Arwa Fallatah, Noura Alhijiri, Reem Alquwayzani, and Majid Almarhoumi. We thank them for their invaluable support in this research.

## References

* Asad et al. (2023) Asaad Alghamdi, Xinyu Duan, Wei Jiang, Zhenhai Wang, Yimeng Wu, Qingrong Xia, Zhefeng Wang, Yi Zheng, Mehdi Rezagholizadeh, Baoxing Huai, et al. 2023. Aramus: Pushing the limits of data and model scale for arabic natural language processing. arXiv preprint arXiv:2306.06800. Cited by: SS1.
* Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danry Hernandez, Tristan Hume, Scott Johnston, Shauna Krivec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRRabs/2204.05862. External Links: Link, 2004.05862 Cited by: SS1.
* E. Callahan and S. C. Herring (2011) Cultural bias in wikipedia content on famous persons. J. Assoc. Inf. Sci. Technol.62 (10), pp. 1899-1915. External Links: Document, ISSN 0037-86X, Link Cited by: SS1.
* Sky CH-Wang et al. (2023) Sociocultural norm similarities and differences via situational alignment and explainable textual entailment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 3548-3564. External Links: Link, 2004.05862 Cited by: SS1.
* Sahli Chaudhary (2023) Code alpaca: an instruction-following llama model for code generation. Note: [https://github.com/sahi1280114/codealpaca](https://github.com/sahi1280114/codealpaca) Cited by: SS1.
* Z. Chen et al. (2023) Multilingual SIFT: multilingual supervised Instruction Fine-tuning. External Links: 2304.10453 Cited by: SS1.
* Z. Chen et al. (2023) Code alpaca: an instruction-following llama model for code generation. Note: [https://github.com/sahi1280114/codealpaca](https://github.com/sahi1280114/codealpaca) Cited by: SS1.
* W. Chiang et al. (2023) Vehicle-based automatic

* Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2475-2485.
* Conover et al. (2023) Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world's first truly open instruction-tuned llm.
* Dodge et al. (2021) Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the classical clean crawled corpus. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_, pages 1286-1305. Association for Computational Linguistics.
* Dubois et al. (2023) Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpaca-farm: A simulation framework for methods that learn from human feedback.
* Hardalov et al. (2020) Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov. 2020. EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_, pages 5427-5444. Association for Computational Linguistics.
* Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
* Hershcovich et al. (2022) Daniel Hershcovich, Stella Frank, Heather C. Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillio Rust, and Anders Sogaard. 2022. Challenges and strategies in cross-cultural NLP. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_, pages 6997-7013. Association for Computational Linguistics.
* Huang and Yang (2023) Jing Huang and Diyi Yang. 2023. Culturally aware natural language inference. In _Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023_, pages 7591-7609. Association for Computational Linguistics.
* Huang et al. (2023) Yukzen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models.
* democratizing large language model alignment. _CoRR_, abs/2304.07327.
* Lee et al. (2023) Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. 2023. RLAIF: scaling reinforcement learning from human feedback with AI feedback. _CoRR_, abs/2309.00267.
* Muennighoff et al. (2022) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022. Crosslingual generalization through multitask finetuning. _arXiv preprint arXiv:2211.01786_.
* Naous et al. (2023) Tarek Naous, Michael J. Ryan, and Wei Xu. 2023. Having beer after prayer? measuring cultural bias in large language models. _CoRR_, abs/2305.14456.
* Palta and Rudinger (2023) Shramay Palta and Rachel Rudinger. 2023. FORK: A bite-sized test set for probing culinary cultural biases in commonsense reasoning models. In _Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 9952-9962. Association for Computational Linguistics.
* Peng et al. (2023) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. _arXiv preprint arXiv:2304.03277_.
* Ramezani and Xu (2023) Aida Ramezani and Ying Xu. 2023. Knowledge of cultural moral norms in large language models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 428-446. Association for Computational Linguistics.
* Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. _CoRR_, abs/1707.06347.
* Seelawi et al. (2021) Haitham Seelawi, Ibraheem Tuffaaha, Mahmoud Gazawi, Wael Farhan, Bashar Talafha, Riham Badawi, Zyad Sober, Oday Al-Dweik, Abed Alhazim Freibhat, and Hussein Al-Natshel. 2021. ALUE: Arabic language understanding evaluation. In _Proceedings of the Sixth Arabic Natural Language Processing Workshop_, pages 173-184, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.

 

* Sengupta et al. (2023) Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, et al. 2023. Jias and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. _arXiv preprint arXiv:2308.16149_.
* Soboleva et al. (2023) Daria Soboleva, Al-Khateeb Faisal, Myers Robert Steeves Jacob R, Hestness Joel, and Dey Nolan. 2023. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. www.cerebras.net/blog/slimpajima-a-627b-token-cleaned-and-dedduplicated-version-of-redpajama.
* Stiennen et al. (2020) Nisan Stiennen, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. Learning to summarize from human feedback. _CoRR_, abs/2009.01325.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca).
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almhaiiri, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Wang et al. (2023a) Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023a. Large language models are not fair evaluators. _CoRR_, abs/2305.17926.
* Wang et al. (2023b) Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang, Zhaopeng Tu, and Michael R. Lyu. 2023b. Not all countries celebrate thanksgiving: On the cultural dominance in large language models. _CoRR_, abs/2310.12481.
* Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzeia, Atharu Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizihi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maityeva Patel, Mehrd Moradshashi, Mihur Parmari, Mirali Purohit, Neeraji Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehab Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022. Super-naturalinstructions: Generalization via declarative instructions on 1600+ NLP tasks. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_, pages 5085-5109. Association for Computational Linguistics.
* Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Davin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. _arXiv preprint arXiv:2304.12244_.
* Zhang et al. (2023) Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Jianquan Li, Guiming Chen, Xiangbo Wu, Zhiyi Zhang, Qingying Xiao, et al. 2023. Huatuogpt, towards taming language model to be a doctor. _arXiv preprint arXiv:2305.15075_.

 

Related Work

Our work mainly lies in two topics: democratization of ChatGPT to low-resource language and localization and cultural alignment.

Democratization of ChatGPT to low-resource languageRecent trends have focused on developing open-source LLMs as an alternative to the proprietary GPT-3.5 Turbo and GPT-4 models (Taori et al., 2023; Chiang et al., 2023; Conover et al., 2023; Chen et al., 2023a; Sengupta et al., 2023). These efforts have expanded to include LLMs designed for languages beyond English (Chen et al., 2023a; Sengupta et al., 2023). However, these models rely heavily on large-scale datasets translated from English in the supervised fine-tuning stage, inadvertently embedding English-centric cultural biases and values (Chen et al., 2023a; Sengupta et al., 2023). This practice risks undermining the models' capacity to align with the cultural contexts and values specific to their target linguistic communities. To mitigate this "localization issue", our work proposes a culture-aware framework that encompasses the pretraining, supervised fine-tuning, and reinforcement learning phases, to enhance the value and cultural alignment of AceGPT with Arabic.

Localization and cultural alignmentCultural diversity varies significantly by region and language. Yet, pre-trained language models primarily incorporate Western cultural biases (Palta and Rudinger, 2023; Ramezani and Xu, 2023; CHwang et al., 2023; Naous et al., 2023). These biases are attributed to the reliance on training corpora such as C4 (Dodge et al., 2021) and Wikipedia (Callahan and Herring, 2011), which are inherently skewed towards Western culture, leading to English-centric models. Moreover, the emphasis on translating English data in low-resource language NLP research exacerbates the "localization issue", failing to adequately represent the non-English culture and values (Conneau et al., 2018; Chen et al., 2023a; Sengupta et al., 2023). To address this, it is recommended to develop culture-aware models for non-English languages by gathering and annotating culture-specific data (Hershcovich et al., 2022). However, assembling comprehensive datasets for low-resource languages is challenging and expensive. Previous research has either concentrated on collecting unsupervised data for pre-training (Sengupta et al., 2023) or embarked on task-specific supervised fine-tuning (Wang et al., 2023b; Conneau et al., 2018; Huang and Yang, 2023; Palta and Rudinger, 2023). In this paper, we present a novel and comprehensive culture-aware pipeline that goes beyond the previous approaches, by integrating general culture-aware supervised fine-tuning and culture-aware reinforcement learning.

## Appendix B Localization Issues

### Sample Questions for Localization

The sample questions for Arabic name entity comparison in Table 1 and 2 are as follows Table 13.

\begin{table}
\begin{tabular}{l l} \hline \hline  & What are the ten most common names? \\  & What are the ten most famous historical stories? \\  & Names of the ten most common cities \\  & What are the top ten cities people hear about the most? \\  & What are the new most important Arab social organizations? \\  & Top Ten People of the Year \\  & Names of the ten most common cities \\  & What are the top ten cities people hear about the most? \\  & What are the new most important Arab social organizations? \\  & Top Ten People of the Year \\  & How do people celebrate the New Year? \\  & Why do people believe in religion? \\  & Who are the new famous scholars in the Arab world? \\  & What are some famous tourist attractions? What are local events? \\  & Who are the famous writers and poets in history? \\  & What are some of the delicacies that are only enjoyed during major festivals? \\  & What is our literary tradition? \\  & What are the basic customs and activities that are important at a wedding? \\  & What are the Holy Books, Saints, and Holy Places? \\  & What are some folk tales? \\  & How did our language originate? \\  & What are the most important festivals? Why do these \\  & Who are the people we should remember? \\  & Who are some of the distinguished artists? \\ \end{tabular}
\end{table}
Table 13: 20 sample questions 



### Case Study

In this subsection, we analyze the performance of AceGPT by conducting a comparative analysis of its localization ability via case studies on the sampled 20 localization questions. Illustrated in Table 24, we observed a larger proportion of Arabic events in AceGPT. The first example in Table 24 aligns with the instance illustrated in Table 2. Both AceGPT and GPT-3.5 Turbo exhibit superior responses to the given query, significantly surpassing the answer provided by Jais. Specifically, AceGPT's understanding of a 'holy book' is not solely confined to the Bible; it demonstrates a nuanced acknowledgment that different regions, especially Arabic, have their respective sacred texts, reflecting a broad and inclusive comprehension of diverse religious traditions. This illustrates the advanced capability of AceGPT, akin to GPT-3.5 Turbo, in response generation for Arabic-speaking areas.

The second example exemplifies the capability of AceGPT to incorporate more Arabic elements when responding to historical questions. Specifically, AceGPT allocates a significant proportion of its responses, 4 out of 10, to Arabic historical figures. In contrast, GPT-3.5 Turbo only attributes 1 out of 10 responses to Arabs, while Jais exclusively presents choices associated with Western figures. This demonstrates that AceGPT has an inclination towards Arabic culture, emphasizing its capability to offer more Arabic culture-relevant responses in an Arabic context.

## Appendix C Construction of ACVA

We employ a top-down approach for the construction of the Arabic Cultural and Value Alignment benchmark. First, we gathered over 50 topic keywords (see Table 14) representing various aspects of Arabic culture, including humanity, art, science, geography, history, manners, religion, and the influence between civilizations, sourced from several books on Arabic culture and values. Then, we query GPT-3.5 Turbo to generate 8000 data based on the given topic using the prompt shown below, where topic is the placeholder for the topic.

We further sample 50% topics to verify the relevance of questions to Arabic cultures and values and the accuracy of the Yes-No labels, which were reviewed by Arabic speakers, leading to a 'Clean set'.

I am collecting some supervised fine-tuning (sft) data about Arabic culture. It is about the knowledge of Arabic culture and manners. The data is some questions in the Arabic language with an id in the form of {'id': '1',label':'xx' 'query':'xx'}. I will give you a topic in Arabic culture. The 'id' is the index of the data. 'label' is the topic I give you. 'query' is some question statement about Arabic culture under that topic. The Data should be of no repetition with a balanced proportion of true and false. Now please generate 200 sft data in json in Arabic with the format under the topic of topic

## Appendix D Preference Data for RLAIF

The data comprises two parts: Arabic preference data and open-source English preference data. Outputs for Arabic preference data are sampled from our fine-tuned 7B model with a temperature of 1. The open-source English preference data is incorporated to improve the generalization capability of the reward model and alleviate GPT4-preference hacking. We randomly sample 12K from three public human-annotated datasets - Anthropic helpfulness and harmlessness Bai et al. (2022), OpenAI Summarize Stiennon et al. (2020), and OpenAssistant Conversations OASST1 Kopf et al. (2023).

The core idea of preference labeling for Arabic preference data is to use a GPT-4 model with prompts as an automatic annotator to assess two responses generated by the same model for a given question. However, a significant challenge emerges as GPT-4 often shows a marked preference for the first response, around 80% of the time, with the exact percentage varying based on the specific prompt design. To counter this, we utilize an order-switch mechanism to ensure consistent preference data across two separate runs of GPT-4. In one run, two responses are placed arbitrarily, and in the other, their orders are switched. The prompt for labeling is shown below. instruction, response 1, and response 2 are the placeholders for the input instructions and the two generated responses



## Appendix E Implementation of Training

### Pre-Training

We employ the LLaMA2 framework for the pretraining process, capitalizing on a computational setup furnished with 24 Nvidia A100 80G GPUs. We configure the context length at 2048 tokens and adopt the AdamW optimizer, paired with a cosine learning rate scheduler. The learning rate is set at 1e-4. Given a gradient accumulation setting of 128, the total batch size amounts to 3072. Additionally, a warm-up phase is integrated, constituting 5% of the total training duration.

### Supervised Fine-Tuning

We train for one epoch using a variety of datasets in Table 15. Native Arabic data like Alpaca-Arabic-GPT4 and Quora-Arabic-GPT4 are included thrice in the mixture, while datasets like ShareGPT and Alpaca-Chinese-GPT4 are once to minimize the non-Arabic data ratio, totaling 629,293 data.

Following LLaMA2, we use the following form of system prompt:


 (\(\langle\)SYS\(\rangle\)) You are a helpful, respectful, and honest assistant. Always answer with the utmost assistance while being safe. Your answers should not include any harmful, unethical, racist, gender discriminatory, toxic, dangerous, or illegal content. Please ensure that your responses are not socially biased and are positive. If the question is meaningless or isn't coherent in a realistic sense, explain the reason instead of answering something incorrectly. If you do not know the answer to a question, please refrain from sharing. \(\langle\)\(\langle\)SYS\(\rangle\)\(|\)Question\(|\)
 Both Ace GPT-7B and Ace GPT-13B are fine-tuned with 8 Nvidia A100 80G GPUs. We employ

\begin{table}
\begin{tabular}{l|l} \hline Country & Algeria, Bahrain, Comoros, Egypt modern, Iraq, Jordan, Kuwait, Lebanon, Libya, Mauritania, Morocco, Oman, Palestine, Qatar, Saudi Arabia, Somalia, Sudan, Syria, Tunisia, United Arab Emirates, Yemen \\ \hline Civilization Relation & Influence from ancient Egypt, Influence from Byzantium, Influence from China, Influence from Greece, Influence from Persia, Influence from Rome, Mesopotamia civilization \\ \hline Science and Humanity & Arabic Astronomy, Arabic Math, Arabic Medicine, Arabic Physics and Chemistry, Arabic Literature, Arabic Music, Arabic Philosophy, Arab Empire, Arabic Architecture, Arabic Art, Arabic Calligraphy, Arabic Geography, Arabic History, Arabic Language Origin \\ \hline Manners and Religion & Arabic Ceremony, Arabic Clothing, Arabic Culture, Arabic Food, Arabic Funeral, Arabic Ornament, Arabic Wedding, Mindset, Special Expression, Daily Life, Influence from Islam, Islam Branches and Schools, Islam Education, Islamic Law System \\ \hline  & \\ \end{tabular}
\end{table}
Table 14: Topics for ACVA construction

the AdamW optimizer, with each batch consisting of 128 samples. We adopt different configurations for the learning rate based on the model architecture. For AceGPT-7B, the maximum learning rate is set to \(5\times 10^{-5}\), and for AceGPT-13B, it is \(1\times 10^{-5}\). A cosine scheduler is employed for learning rate adjustment, with a warmup rate of 0.03.

### Reward Model Training

The reward model is initialized with Ziya, an open-source 7B reward model 8. We use 8 Nvidia A100 80G GPUs for training. Each batch consists of 128 samples. We take two epochs with the AdamW optimizer. The maximum learning rate is set to 8e-6 and the warmup rate is set to 0.03 with cosine scheduler.

Footnote 8: [https://huggingface.co/IDEA-CCNL/Ziya-LLaMA-7B-Reward](https://huggingface.co/IDEA-CCNL/Ziya-LLaMA-7B-Reward)

Footnote 9: [https://github.com/microsoft/DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples)

### Ppo

We implement PPO with DeepSpeed-Chat 9. The actor parameters are initialized with our fine-tuned models and the critic parameters are initialized with our trained 7B reward model. We sample 448 experiences with the mini-batch size of 224 10, which is updated in only one epoch. The maximum learning rate for the actor is set to 5e-7 while that for the critic is set to 5e-6. A cosine scheduler is used for learning rate adjustment with a warmup step of 100. We set the KL penalty as 0.01. The policy gradient loss is clipped with the threshold as 0.2 while that for the value loss is 0.3. The reward is clipped to be [-5, 5]. The gamma and lambda for the generalized advantage estimation are 1 and 0.95 respectively.

Footnote 10: We use 7 GPUs, where there are 64 experiences and 32 samples in each mini-batch on each GPU.

Notably, both AccGPT-7B and AceGPT-13B are trained with the 7B reward model whose preference data only comprises outputs from the 7B policy model (post-SFT).

## Appendix F Implementation of Evaluation

### Baselines and Benchmarks

We use the following baselines :

* **LLaMA2**Touvron et al. (2023), developed by Meta AI, are the most popular open-source large language models ranging in scale from 7 billion to 70 billion parameters. Our AceGPT models are also built upon LLaMA2-7B and -13B. We compare our AceGPT-base models to the corresponding size of LLaMA2.
* **Bloomz**Muennighoff et al. (2022) and **Phoenix**Chen et al. (2023, 2023). **Bloomz** is a classical family of multilingual models finetuned with multiple traditional NLP tasks. **Phoenix** are multilingual instruction following models using Bloomz as the backbone. We compare AceGPT-base models to Bloomz and AceGPT-chat models to Phoenix.
* **Jais**Sengupta et al. (2023) are concurrent open-source 13B Arabic-centric LLMs, including a foundation base model and an instruction-tuned model. We compare AceGPT-base and AceGPT-chat to their base and chat models respectively.
* **GPT-3.5 Turbo** is the most popular and powerful closed-source multilingual LLM, second only to GPT-4. We compare both AceGPT-base and AceGPT-chat to it.

We use Arabic Vicuda-80, Arabic AlpacaEval, Arabic MMLU, Exams, ALUE, ACVA as our benchmarks, see Table 16.

In the scoring method for the Arabic MMLU dataset, the process begins by calculating the average score within each subcategory, this average

\begin{table}
\begin{tabular}{l|l l c} \hline \hline Dataset & Resources & Responses & Size \\ \hline
**Quora-Arabic-40K** & Collected from Quora & GPT-4 & 43,050 \\ \hline Alpaca Peng et al. (2023) & Self-instruct Taori et al. (2023) & & 49,969 \\ Alpaca-Chinese Peng et al. (2023) & GPT-3.5 Turbo translated Peng et al. (2023) & GPT-4 & 49,969 \\
**Alpaca-Arabic** & GPT-4 translated from Taori et al. (2023) & & 49,969 \\ \hline
**Code-Alpaca-Arabic** & GPT-4 translated from Chaudhury (2023) & GPT-4 & 20,022 \\ \hline
**Evol-Instruct-Arabic** & GPT-4 translated from Xu et al. (2023) & GPT-4 & 69,997 \\ \hline ShareGPT & humans & GPT 3.5 Turbo & 80,179 \\ \hline \hline \end{tabular}
\end{table}
Table 15: Instruction tuning datasets; Datasets constructed in this work are highlighted in **bold**.

 

represents the mean score for that particular subcategory. Subsequently, the average scores of all subcategories within a main category are aggregated and averaged again to yield the overall average score for that main category. Following this, the average scores of all main categories are collated and averaged to determine the overall average score for that category within the entire Arabic MMLU dataset. Finally, the aggregate average scores of all categories in the dataset are combined and averaged to arrive at the comprehensive average score across the entire Arabic MMLU dataset.

### Evaluation on Instruction Following

We apply greedy decoding strategies for all models evaluated on Arabic Vicuna-80 and Arabic AlpacaEval benchmarks.

GPT-4 evaluationThe temperature of the GPT-4 evaluator is set to 0.2, with the scoring conducted independently three times. We calculate the average performance ratio of the evaluated model relative to GPT-3.5 Turbo. Specifically, for each question, GPT-4 is queried to assign scores to responses from the evaluated model and GPT-3.5 Turbo respectively. We obtain the evaluated model's score for each question, and these scores are averaged to determine the evaluated model's overall score. Similarly, we derive an overall score for GPT-3.5 Turbo. The performance ratio is obtained by dividing the overall score of the evaluated model by that of GPT-3.5 Turbo. Averaging the performance ratios from three separate experiments yields the final performance ratio. The evaluation prompt is shown below. question, response 1, and response 2 are the placeholders for the input questions and the two generated responses. To mitigate position bias Wang et al. (2023), we vary the order in which the model pair is presented in the prompt.

[Question] question

[Assistant 1] response 1

[End of Assistant 1]

[Assistant 2] response 2 



### Evaluation on Knowledge

There are two main differences in the MMLU evaluation between (Sengupta et al., 2023) and ours: (1) we translate MMLU into Arabic differently. The machine-translated version in (Sengupta et al., 2023) is facilitated through their in-house translation model while we leverage GPT-3.5 Turbo. Additionally, (Sengupta et al., 2023) further creates a human-translated version. Unfortunately, both the human-translated and machine-translated versions are not publicly available, which prevents us from evaluating on the same benchmark; (2) we adopt the widely accepted few-shot prompting setting commonly found in related literature for base models, while (Sengupta et al., 2023) opts the zero-shot setting. Due to these differences in translation methods and evaluation settings, the performance metrics between the two works are not directly comparable.

We benchmark Jais-13B-base and Jais-30B-base using our GPT-3.5 Turbo-translated MMLU dataset under the standard few-shot setting in Table 8. Moreover, we also benchmark Jais-13B-chat using the zero-shot setting in Table 20.

The evaluating template is shown below:

\begin{table}
\begin{tabular}{l l c c} \hline \hline Benchmark & Evaluation Aspects & Type of Evaluation & Size Types of Examples \\ \hline Arabic Vicuna-80 & Instruction following & Human \& Automated & 805 \\ Arabic AlpacaEval & & & 805 \\ \hline Arabic MMLU & & & \\ EXAMs & Knowledge ability & Automated & 14k & Multiple-choice questions \\ \hline ALUE(see Appendix G.2) & Language understanding & Automated & 18k & Classification \& regression \\ \hline ACVA-all & Arabic cultural and & Automated & 9k & Yes/No binary questions \\ ACVA-clean & value alignment & Automated & 2.4k & \\ \hline \hline \end{tabular}
\end{table}
Table 16: Evaluation Benchmarks.

Figure 2: User interface of human annotation. Response positions are randomized to alleviate biases.



* Few-shot [category]
* Few-shot Below are multiple choice questions (with answers) about [category] [examples] Question: [question] Answer:
* Zero-shot Below are multiple choice questions about [category] [Question] Please choose one answer from among 'A, B, C, D' without explanation.

A specific example of five-shot prompting is: 



### Evaluation on ACVA

The evaluation prompt for ACVA is

* Few-shot [category]
* Zero-shot [class=none, label="]
* Zero-shot [class=none, label="]


Jais's good performance may be attributed to traditional NLP task datasets in their SFT dataset such as Super-NaturalInstructions (Wang et al., 2022), which contains multiple-choice questions akin to the MMLU and EXAMs. Our model, in contrast, hasn't been trained on such data.

### Evaluation on Arabic NLU Tasks

ALUEDetails of the evaluation of ALUE will be included in this section, the introduction of ALUE can be found in Sec 4.1.

Experiment settingWe train our AceGPT-13B-base on each task independently in a fully supervised manner, resembling the approach of the top models on the leaderboard. Moreover, high-ranking models on the leaderboard adopt the grid search method on validation sets to select hyper-parameters. Similarly, we employ a Bayesian approach for hyperparameter adjustment. For tasks providing predefined validation split, we utilize the given validation sets. Otherwise, we allocate 10% of the data from the training set for validation purposes. For the DIAG task, which does not provide training data, we use the model trained on XNLI to evaluate it.

Experiment results and analysisTable 22 presents our performance on the ALUE benchmark. AceGPT ranks second in terms of the average score in these nine datasets, right behind AraMUS (Alghamdi et al., 2023), which has conducted extensive pre-training in Arabic data. In future endeavors, we plan to incorporate a richer set of Arabic pre-training corpora and supervised data to enhance the model's NLU capabilities.

## Appendix H Relationship between Arabic Culture and Arabic Language

The English language can reflect multiple cultures and vice versa (Hershcovich et al., 2022). This demonstrates that language and culture, although intertwined, are not inherently synonymous. In contrast, among Arabs, there exists a significant correlation between the Arabic language and Arab culture for several reasons. Primarily, Arabic is predominantly spoken by native speakers deeply embedded in Arab culture, in stark contrast to English, which is more commonly spoken by second-language speakers. 11 Furthermore, most second-language learners of Arabic pursue the language to connect with Islamic culture, a pivotal element of Arab culture, because all Islamic terminologies are expressed in Arabic. 12

Footnote 11: [https://en.wikipedia.org/wiki/English_language](https://en.wikipedia.org/wiki/English_language)

Therefore, in this paper, we assume that the Arabic language effectively captures Arab culture. Despite the presence of diverse sub-cultures within the Arab world, our paper primarily focuses on the Arab community as a whole, overlooking the distinctions among its sub-groups.

 



\begin{table}
\begin{tabular}{l|c c c c|c|c} \hline \hline  & \multicolumn{5}{c}{Arabic MMLU} \\ Model & Average & STEM & Humanities & Social Sciences & Others (Business, Health, Mice) & EXAMs \\ \hline Phoenix & 29.65 & 27.06 & 28.35 & 31.66 & 31.54 & 31.60 \\ Phoenix-multiple-langs & 17.37 & 16.77 & 15.65 & 18.51 & 18.54 & 16.48 \\ Jais-13B-chat & 41.06 & 39.82 & 42.21 & 41.75 & 42.50 & 40.84 \\ \hline AceGPT-7B-chat & 30.43 & 26.58 & 30.39 & 32.28 & 32.49 & 33.66 \\ AceGPT-13B-chat & 35.17 & 33.14 & 33.10 & 38.95 & 35.52 & 39.79 \\ \hline GPT-3.5 Turbo & **49.07** & **43.38** & **44.12** & **55.57** & **53.21** & **45.63** \\ \hline \hline \end{tabular}
\end{table}
Table 20: Accuracy of chat models on Arabic MMLU and EXAMs. The best is in **bold** and the second is underlined.

\begin{table}
\begin{tabular}{l l



\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c} \hline \hline
**Model** & **\#Params** & **Avg** & **Score** & **MQ2Q** & **MDD** & **SVREG** & **SEC** & **FID** & **OOLD** & **XMLI** & **OHSD** & **DIAG** \\ \hline ARABIC-BERT & 135M & 63.5 & 67.1 & 85.7 & 59.7 & 55.1 & 25.1 & 82.2 & 89.5 & 61.0 & 78.7 & 19.6 \\ ARABETROV.1-base & 135M & 64.2 & 68.4 & 89.2 & 58.9 & 56.3 & 24.5 & 85.5 & 88.9 & 67.4 & 76.8 & 23.5 \\ ARABLE-BERT & 110M & 68.6 & 69.3 & 89.7 & 59.7 & 58.0 & 26.5 & 84.3 & 89.1 & 67.0 & 80.1 & 19.0 \\ CAMelBERT-MIX & 108M & 66.7 & 70.4 & 89.4 & 61.3 & 69.5 & 30.3 & 85.5 & 90.3 & 56.1 & 80.6 & 11.8 \\ AraT5-base & 289M & 67.6 & 71.1 & 91.3 & 63.8 & 65.9 & 30.5 & 82.3 & 88.8 & 68.2 & 77.9 & 15.4 \\ ARBERT & 163M & 65.5 & 71.4 & 89.3 & 61.2 & 66.8 & 30.3 & 85.4 & 89.5 & 70.7 & 78.2 & 24.3 \\ MARBERT & 163M & 63.9 & 72.2 & 83.3 & 61.9 & 75.9 & 36.0 & 85.3 & 92.1 & 64.3 & 78.9 & 12.3 \\ JABER & 135M & 68.2 & 73.7 & 93.1 & 64.1 & 70.9 & 31.7 & 85.3 & 91.4 & 73.4 & 79.6 & 24.4 \\ Char-JABER & 136M & 70.1 & 75.3 & 92.0 & 66.1 & 74.5 & 34.7 & 86.0 & 92.3 & 73.1 & 83.5 & 26.7 \\ ALM-1.0 & 350M & 70.3 & 75.8 & 94.5 & 65.1 & 70.1 & 35.3 & 86.0 & 91.7 & 77.7 & 85.7 & 30.2 \\ SABER & 369M & 71.4 & 77.3 & 93.3 & 66.5 & 79.2 & 38.8 & 86.5 & 93.4 & 76.3 & 84.1 & 26.2 \\ AraMUS & 11B & **74.0** & **79.8** & **95.2** & **67.5** & **80.4** & **41.6** & **87.2** & **95.5** & **83.2** & **87.4** & 42.0 \\ \hline
**AceGPT-13B-base** & 13B & 72.8 & 76.6 & 94.9 & 63.3 & 72.4 & 36.8 & 85.1 & 94.2 & 81.0 & 85.4 & **42.2** \\ \hline  & & & & & & & & & & & \\ \end{tabular}
\end{table}
Table 22: Experimental results in ALUE (Seelawi et al., 2021) including online baselines. While the leaderboard calculates the ‘scores’ excluding Task DIAG, we also incorporate it to derive the ’Avg’.

\begin{table}
\begin{tabular}{l|l l l l l} \hline \hline Dataset & Comparison & & Win & Tie & Lose \\ \hline  & & volunteer 1 & 66 & 3 & 11 \\ AceGPT-7B-chat vs. Jais-13B-chat & & volunteer 2 & 65 & 9 & 6 \\  & & volunteer 3 & 67 & 4 & 9 \\ \cline{2-6}  & & volunteer 1 & 26 & 0 & 54 \\ AceGPT-7B-chat vs. GPT-3.5 Turbo & & volunteer 2 & 40 & 0 & 40 \\  & & volunteer 3 & 0 & 79 & 1 \\ \cline{2-6}  & & volunteer 1 & 23 & 12 & 45 \\ AceGPT-7B-chat (w/o RLAIF) vs. GPT-3.5 Turbo & & volunteer 2 & 12 & 58 & 10 \\  & & volunteer 3 & 31 & 0 & 49 \\ \cline{2-6} Arabic Vicuna-80 & AceGPT-13B-chat vs. Jais-13B-chat & & volunteer 1 & 68 & 6 & 6 \\  & & volunteer 2 & 65 & 5 & 10 \\  & & volunteer 3 & 66 & 5 & 9 \\ \cline{2-6}  & & volunteer 1 & 14 & 35 & 31 \\ AceGPT-13B-chat vs. GPT-3.5 Turbo & & volunteer 2 & 21 & 28 & 31 \\  & & volunteer 3 & 4 & 74 & 2 \\ \cline{2-6}  & & volunteer 1 & 19 & 14 & 47 \\ AceGPT-13B-chat (w/o RLAIF) vs. GPT-3.5 Turbo & & volunteer 2 & 22 & 19 & 39 \\  & & volunteer 3 & 6 & 57 & 17 \\ \cline{2-6}  & & volunteer 1 & 54 & 6 & 20 \\ AceGPT-7B-chat vs. Jais-30B-chat & & volunteer 2 & 48 & 30 & 2 \\  & & volunteer 3 & 60 & 0 & 20 \\ \cline{2-6}  & & volunteer 1 & 48 & 29 & 3 \\ AceGPT-13B-chat vs. Jais-30B-chat & & volunteer 2 & 55 & 0 & 25 \\  & & volunteer 3 & 52 & 7 & 21 \\ \hline  & & volunteer 1 & 515 & 196 & 94 \\ AceGPT-7B-chat vs. Jais-13B-chat & & volunteer 2 & 619 & 54 & 132 \\  & & volunteer 3 & 146 & 632 & 27 \\ \cline{2-6}  & & volunteer 1 & 259 & 291 & 255 \\  & & volunteer 2 & 71 & 632 & 102 \\  & & volunteer 3 & 158 & 200 & 447 \\ \cline{2-6}  & & volunteer 1 & 283 & 504 & 18 \\ AceGPT-13B-chat vs. Jais-13B-chat & & volunteer 2 & 400 & 343 & 62 \\  & & volunteer 3 & 509 & 187 & 109 \\ \cline{2-6}  & & volunteer 1 & 216 & 326 & 263 \\ AceGPT-13B-chat vs. GPT-3.5 Turbo & & volunteer 2 & 47 & 664 & 94 \\  & & volunteer 3 & 346 & 84 & 375 \\ \hline \hline \end{tabular}
\end{table}
Table 23: Details of human evaluations on Arabic Vicuna-80 and Arabic AlpacaEval.





\begin{table}
\begin{tabular}{l l} \hline \hline
**User**: & \\  (What are the holy books, saints, and holy places?) & \\ \hline
**AccGPT-7B-chat**: & \\  _and you will be a small j usually called you will call you you will call you you will call you you you will call you you you will call you you you you will call you you you you will call you you you you you will call you



**User**:

(Who are the famous writers and poets in history?)

\begin{tabular}{l l} \hline
**AccGPT-7B-chat** & \\  _i.e., [1] & _ii.e., [2] & _iii.e., [3] \\  _i.e., [4] & _iv.e., [5] & _v.e., [6] \\  _i.e., [8] & _vi.e., [9] & _vii.e., [10] \\  _i.e., [11] & _vi.e., [12] & _vi.e., [13] \\  _i.e., [14] & _vi.e., [15] & _vi.e., [16] \\  _i.e., [17] & _vi.e., [18] & _vi.e., [19] \\  _i.e., [19] & _vi.e., [19] & _vi.e., [19] \\  _i.e., [20] & _vi.e., [21] & _vi.e., [22] \\  _i.e., [23] & _vi.e., [24] & _vi.e., [25] \\  _i.e., [26] & _vi.e., [27] & _vi.e., [28] \\  _i.e., [29] & _vi.e., [29] & _vi.e., [29] \\  _i.e., [28] & _vi.e., [29] & _vi.e., [29] \\  _i.e., [29] & _vi.



(There are many famous writers and poets in history, among them: 1. William Shakespeare: a famous English writer, who is considered the greatest writer in the English language and one of the greatest playwrights in history. 2. Goethe: a German poet and writer, who is considered One of the most prominent poets of world literature. 3. Homer: An ancient Greek poet, and is considered the author of the classic literary works "The Iliad" and "The Odyssey". 4. Dante Alighieri: An Italian poet and writer of the "Divine Comedy". ", and it is considered one of the greatest literary works in history. 5. Charles Baudelaire: A French poet, and is considered one of the most prominent poets of the Romantic movement in the nineteenth century. 6. William Wordsworth: An English poet, and is considered one of the most prominent poets of the Romantic movement in the nineteenth century. 7. John Milton: An English poet, considered one of the greatest poets of English literature in the seventeenth century. 8. Robert Frost: An American poet, considered one of the most prominent poets of the twentieth century. 9. George Bernard Shaw: An Irish writer and playwright, and is considered one of the greatest playwrights of the twentieth century. 10. Naguib Mahfouz: An Egyptian writer and winner of the Nobel Prize in Literature, and is considered one of the most prominent writers of Arabic literature in the twentieth century.)