ALLaM: Large Language Models for Arabic and English

## Abstract

Abstract We present ALLaM : A rabic L arge La nguage M odel, a series of large language models to support the ecosystem of Arabic Language Technologies (ALT). ALLaM is carefully trained considering the values of language alignment and knowledge transfer at scale . Our autoregressive decoder-only architecture models demonstrate how second-language acquisition via vocabulary expansion and pretraining on a mixture of Arabic and English text can steer a model towards a new language (Arabic) without any catastrophic forgetting in the original language (English). Furthermore, we highlight the effectiveness of using parallel/translated data to aid the process of knowledge alignment between languages. Finally, we show that extensive alignment with human preferences can significantly enhance the performance of a language model compared to models of a larger scale with lower quality alignment. ALLaM achieves state-of-the-art performance in various Arabic benchmarks, including MMLU Arabic, ACVA, and Arabic Exams. Our aligned models improve both in Arabic and English from their base aligned models.

## 1 Introduction

Language modeling has significantly progressed from its humble origins, transitioning from fundamental probabilistic methods to complex neural priors. The foundational work by Shannon () on the information theory of language laid the groundwork for predicting the next word in a sequence, which was subsequently tackled by Bengio et al. () in neural networks. The field experienced a substantial leap with the introduction of LSTMs (Hochreiter & Schmidhuber, ) in language models (LM) (Peters et al., ), which could capture longer dependencies in LMs but proved difficult to scale. The emergence of scalable and distributed architectures like Transformers (Vaswani et al., ) and the potential for precisely (Kaplan et al., ; Hoffmann et al., ) compressing web-scale data has resonated in recent years with the advancements of *Generative Pretraining* (Radford et al., ; Brown et al., ; Anil et al., ).

With the release of ChatGPT (OpenAI, ), followed by the introduction of more frontier class models Gemini (Google, ), Claude (Anthropic, ), Reka (Ormazabal et al., ), Mistral (Mistral, ), Llama-3 (Meta, ) and recently released Qwen-2 (Yang et al., ), large language models have demonstrated significant leaps over each generation of models (Laskar et al., ). This exponential growth in performance has raised hope in the possibility of achieving Artificial General Intelligence (Hendrycks & Mazeika, ; Marcus, ). This rapid advancement has spurred discussions across various fields, including ethics, economics, and technology (Weidinger et al., ). Judging from the initial capabilities (Bubeck et al., ), the potential of these frontier models are reinventing the way humans interact with machines, impacting social norms, productivity, trends, and culture on a broader scale (Zhou et al., ). However, most of these frontier-class models are primarily trained on English and often lack a connection to localized regional cultures and norms (Naous et al., ). This gap has the potential to result in slow and irreversible manipulation of regional identities and lead to cultural homogenization.

The natural course to reverse this trend is to invest resources in curating data and building models to support the diversity of languages and cultures represented in the modern world. While this is possible, the significant training costs of LLMs and their environmental impact have become major concerns in recent years (Strubell et al., ). The vast computational resources required to train LLMs contribute to substantial carbon emissions (Luccioni & Hernandez-Garcia, ). Governments (^1^11[https://www.cnrs.fr/en/update/jean-zay-supercomputer-recycling-its-heat](https://www.cnrs.fr/en/update/jean-zay-supercomputer-recycling-its-heat)) and non/for-profit organizations (Dodge et al., ; Google, ; Amazon, ), are increasingly aware of these issues. This awareness has led to discussions about the ethical implications of AI development and the need for sustainable practices concerning “*When and how to scale the training of these models*”. In addition, curating data for each language/region at pretraining scale is also an extremely difficult task, since most available data comes from a few high-resource languages.

Figure: Figure 1: Performance on Arabic (Koto et al., ) and English (Hendrycks et al., ) MMLU Benchmarks. ALLaM (red line) shows impressive improvement from its base model Llama-2 (yellow line). All evaluations were done on the latest version of the fine-tuned (chat or instruct) models. The ALLaM 7B from scratch model also shows significant improvement over the ALLaM 7B continued pretraining model.
Refer to caption: x1.png

Figure: Figure 2: Comparison of tokenizer fertility rates. The chart illustrates the fertility rates across four tokenizers: Llama-2, ALLaM Arabic only, ALLaM merged with Llama-2, and ALLaM Arabic/English (from scratch model). We calculate the fertility over a random subsample of the entire English, Arabic, and code training corpus.
Refer to caption: x2.png

To address these concerns, we consider the problem of adapting strong, but potentially under-trained, open pretrained models, rather than starting from a randomly initialized model. Technically, this involves continuing training of a model in a new language to facilitate Second Language Acquisition (SLA) (Swain & Lapkin, ), popularized by Bari et al. () in NLP and recently adapted to LLMs by Nguyen et al. (). This process involves the challenging task of incorporating an additional language distribution without compromising the source language(s). For instance, if a pretrained model was initially trained in English, expanding to an additional language presents challenges related to tokenization. Figure 2 gives an overview of ALLaM’s tokenizers compared to a tokenizer primarily trained on English.

Our approach to building ALLaM, large language models developed specifically for fluency and understanding in Arabic and English, can be outlined as follows. We first demonstrate the feasibility of adapting an existing pretrained English model (Llama-2 (Touvron et al., )) to fluency in both Arabic and English through tokenizer and vocabulary expansion. Then, we apply our learnings to train a stronger model from scratch (random initialization) in a similar fashion, i.e., pretraining on English followed by training in mixed Arabic and English. The resulting model exhibits impressive performance and has favorable tokenization properties compared to other models.
This approach aligns with both our technical goals and our commitment to sustainable practices.

Our overall contributions are summarized below:

- •
The ALLaM model series, with the goal of supporting the cultural values of the Arabic-speaking world. We train four models at three different scales: 7B, 13B, and 70B models initialized by Llama-2 weights and a 7B model from scratch/random initialization.
- •
Our model achieves state-of-the-art results in Arabic, as well as improving overall English performance of the original Llama-2 model. Refer to Figure 1 for an overview.
- •
We demonstrate that it is possible to train highly-performant models in low-resource languages from publicly available model weights using our continued pretraining recipe with tokenizer expansion, presenting a path for better representation of low-resource languages.
- •
The training methodology and decision-making involved in training the LLM. We provide necessary ablation studies for most crucial decisions.

## 2 Pretraining

Figure: Figure 3: Measuring the effect of adding machine translated Arabic data to pretraining. Although the two loss curves look normal (left), adding the translated Arabic reduced the frequency of gradient spikes during training (center). Adding translated Arabic data also clearly helps align the Arabic and English capabilities of the model and reduce catastrophic forgetting (right).
Refer to caption: x3.png

Pretraining language models on trillions of natural language tokens represents the bulk of the cost required to build an effective language model. This large investment of time and compute precludes experimentation or ablation for every decision. Thus, before starting to train ALLaM from random initialization, or “scratch”, we experiment in the continue-pretraining regime. As the name implies, continue pretraining is the practice of warm-starting a pretraining experiment from an already pretrained LM.

We begin by discussing our entire pretraining corpus, describe experiments conducted with continue-pretraining, and finally describe pretraining from scratch.

### 2.1 Pretraining Data

For English, many high quality and large scale datasets are available for pretraining (Computer, ; Soldaini et al., ; Gao et al., ; Penedo et al., ). We harnessed subsets from Dolma-v1 (Soldaini et al., ) and Pile (Gao et al., ) datasets e.g., Dolma-CC, The Stack (Kocetkov et al., ), PeS2o, PubMed, DM-Math (Saxton et al., ) and StackExchange (Soboleva et al., ). In total, we had access to 4T high to medium quality English tokens for pretraining.

Pretraining data in the Arabic language is much more limited, thus we undertook large scale collection and curation of Arabic language data.
This includes in-house crawled sources covering Web documents, news articles, books (literature, religion, law and culture, among others), Wikipedia (over 1M articles), and audio transcripts (books and news)(^2^22We are currently working on systematic auditing of our pretraining data. We do not have any timeline or visibility on when or if we can share our data for research.). To ensure high quality Web data, we applied the following processing steps: *(i)* drop documents with language identification score below 95%,
*(ii)* drop short documents that are less than 30 words, *(iii)* drop documents with duplicate URLs or high ratio of spam and stop words,
and *(iv)* drop duplicate documents using exact matching. We experimented with fuzzy matching but opted against using it as it was too restrictive given the scarcity of Arabic data.

Figure: Figure 4: We determine the optimal Arabic/English language mixture that balances between acquiring Arabic understanding while retaining English proficiency by conducting ablations over 6 Arabic/English ratios (trained up to 20B tokens). We found that a 45/55 Arabic/English ratio achieves the best performance, as measured by English and translated Arabic MMLU.
Refer to caption: x4.png

Additionally, we extended our Arabic data with translated English content using an in-house machine translation system. We translated the following English datasets from Dolma: Wikipedia, books, C4 and peS2o, which also are part of our English data. The hypothesis is that this will improve Arabic-English language alignment, leading to a better Arabic model. Figure 3 demonstrates the impact of Arabic translated datasets in the pretraining data mixture. While models trained without translated data exhibit lower training loss, those trained with translated data show more stable training, as evidenced by fewer spikes in gradient norms. Incorporating Arabic translated data in the pretraining dataset mitigates catastrophic forgetting in English. In total, we curate 540B Arabic tokens(^3^33Tokens counted by our merged tokenizer.) of which 270B are natural Arabic tokens and 270B are translated Arabic tokens.

#### Data Mixture

To build a performant model in both Arabic and English, we conducted experiments to determine the optimal language mix. Figure 4 gives an overview of data-mixture experiments on our curated Arabic-English corpus. We conducted the experiments with the same sampling ratio (Table 1) and data order. We observe best trend in performance with 45/45 Arabic/English data mix.

Table 1 shows the language and category mixing distributions for English, Arabic natural, Arabic translated and final mix. Following mainstream work (Touvron et al., ; Chowdhery et al., ; Rae et al., ), web data constitutes the highest ratio with 71%, 65% and 48% of the Arabic natural, Arabic translated and overall mix, respectively. We limited the contribution of English web data to 31%, as the Llama-2 base model was trained on a significant amount of English web data already, and we expected that increasing its ratio might degrade performance. We ensured that high quality sources, such as books, news articles, and code, are well represented in our mixture.

**Table 1: ALLaM’s pretraining data mixtures. The first four columns summarize the distribution of the continued pretraining mixed Arabic/English data. The English only pretraining from scratch mixture is shown in the last column. We upsample data to match the mixture rates when needed.**
| Domain | Mixed Arabic & English | English Only |  |  |  |
| --- | --- | --- | --- | --- | --- |
| English | Arabic | Mixed |  |  |  |
| Natural | Translated |  |  |  |  |
| Web | 31% | 71% | 65% | 48% | 71% |
| Books | 9% | 13% | 12% | 11% | 3% |
| Wiki | — | 0.70% | 0.61% | 0.3% | 0.1% |
| News | — | 14% | — | 3% | — |
| Science | 16% | — | 22% | 14% | 6% |
| Code | 39% | — | — | 21% | 17% |
| Math | 5% | — | — | 2.5% | 0.9% |
| Other | — | 1.3% | 0.39% | 0.2% | 2% |
| Lang Mix | 55% | 22.5% | 22.5% | 100% | 100% |
| Tokens | 660B | 270B | 270B | 1.2T | 4T |

### 2.2 Continued Pretraining

Open source and open weight models present an attractive option to conduct pretraining experiments cheaply. However, they also present challenges, since most such models do not natively support Arabic or other languages. We develop a simple approach to enhance any language model with capabilities in new languages (i.e., language expansion). The approach relies on two steps: *(i)* tokenizer augmentation, and *(ii)* expanded vocabulary learning. We demonstrate that this approach leads to minimal degradation of capabilities in the original language.

#### Tokenizer Augmentation

Existing open weight language models (e.g., Llama-2) tokenize Arabic (and other languages) poorly, often splitting words down to the character level or even relying on byte-fallback mechanisms for tokenization. This results in: *(i)* inefficient training, as the pretraining corpus size is inflated, *(ii)* unoptimized inference, since the model must generate more tokens per word, and *(iii)* the effective context length is reduced, because it is based on a fixed number of tokens. To address these issues, we use a corpus of text in the target language to train a tokenizer specialized in that language. We then merge the original tokenizer with the language-specific tokenizer. Merging is accomplished by adding all tokens from the language-specific tokenizer that do not exist in the original tokenizer. As shown in Figure 2, this effectively reduces the fertility rate in the target language of the merged tokenizer to the level of the language-specific tokenizer.

#### Expanded Vocabulary Learning

Newly added tokens in the merged tokenizer have no associated embedding representations in the pretrained language model’s weights. To learn these representations, we experiment with two approaches: *(i)* random initialization and *(ii)* initialization from combined representations of tokens in the original tokenizer. Approach *(ii)* is accomplished by tokenizing each token in the vocabulary of the new tokenizer using the original tokenizer. The associated embedding representations of this tokenization are then averaged and assigned as the vector representation of the new token. Since we work with tokenizers with byte-fallback, such a tokenization is guaranteed to exist. Figure 6 provides an overview of our initialization method. Initializing the new embeddings from the combination of previously learned embeddings gives a significant boost to the learning of a new language.

#### Experiment Details

Starting from Llama-2 pretrained model weights, we continue pretraining the ALLaM-7B and ALLaM-13B models on 1.2T tokens, covering both English and Arabic languages. For the ALLaM-70B model, we only train up to 600B tokens (using the same data mixture). In all of our continued pretraining experiments, we used the final learning rate of the pretrained language model (usually $3\times 10^{-5}$). We experimented with approaches to gradually increase and decay the learning rate with limited success, as such models typically exhibited catastrophic forgetting, indicated by significant drops in performance in the original language. We also considered optimizer state warm up, as open-weight models typically do not include the optimizer states, but found this had little effect on performance. Figure 6 provides an overview of adding dropout during continued pretraining. We observe that adding dropout helps the Arabic language, as it acts as a regularizer for the new distribution. However, Llama-2 was pretrained on 2T tokens without any dropout, and adding dropout negatively impacts the source language performance. Considering this trade-off, we decided not to add dropout in the continued pretraining stage. Unlike recent trends  (AI2, ), we did not add any alignment data in this stage of training.

Figure: Figure 5: Effect of *random initialization* vs. *embedding initialization* during the start of continued pretraining. We find that initializing the embeddings for new tokens from combinations of existing embeddings speeds up learning dramatically.
Refer to caption: x5.png

### 2.3 Pretraining From Scratch

Following (Hoffmann et al., ; Touvron et al., ), training a high-quality model from scratch requires a substantial amount of tokens. Even when pretraining from random initialization, we find it beneficial to train with a high-resource language for trillions of tokens (English) and then continue training with a mixture of Arabic and English tokens.
On small scale experiments (with 1B parameter models) we find that beginning training with two languages can sometimes degrade the performance in English or result in slow learning of both language distributions. From this, we hypothesize that low-resource languages are diluted in the large volume of high-resource language data when pretraining from scratch, even with upsampling and careful tuning.

#### Training Recipe

Our pretraining from scratch recipe consists of two steps: training on 4T English tokens followed by training on 1.2T mixed Arabic/English tokens. This retains the English capabilities of the model without catastrophic forgetting, effectively transferring knowledge from one language distribution to another. The only difference between pretraining from scratch and continued pretraining from an existing model is that vocabulary expansion is not required.

We match hyperparameters and architecture for pretraining from scratch with Touvron et al. (), including 4M tokens per batch and max LR $3\times 10^{-4}$ decayed to $3\times 10^{-5}$ with a cosine schedule.

#### English Data Mixture

The last column of Table 1 shows the domain mix for the English only pretraining dataset. As expected, web data represent the bulk of the mixture, followed by code and scientific articles.

### 2.4 Compute and Training Infrastructure

Over the course of our development of ALLaM, we had access to 128-1024 A100 GPUs. Our GPU cluster was equipped with InfiniBand connections to enable high-speed communication between nodes. The all-reduce test on the cluster ranges around 1200-1400 Gbps (node-node interconnect (RoCE)). The entire training period of the models is estimated to be 5M GPU hours.

At the start of the project, we forked Megatron-LM(^4^44[https://github.com/NVIDIA/Megatron-LM](https://github.com/NVIDIA/Megatron-LM)) and applied our own customizations (including improving data iterators, adding metadata in the checkpoints, and custom data pipelines). We utilized data, tensor, and pipeline parallelism supported by Megatron-LM to efficiently train at a large scale as well as FlashAttention (Dao et al., ; Dao, ). By leveraging these techniques, we achieved significant improvements in training speed. The throughput per GPU varied from 135 to 167 TFlop/s/GPU depending on the number of GPUs, number of nodes, batch size, and parallelism strategy. We trained ALLaM with bf16 mixed-precision.

## 3 Alignment

Building useful LLMs requires ensuring they are able to follow instructions while adhering to ethical standards and user expectations. This alignment process is especially crucial for models used in diverse linguistic and cultural contexts. In our setting, this means aligning models to the Arabic language and cultural context while also supporting English.

Supervised Fine-Tuning (SFT) (Section 3.1) refines a pretrained model using a carefully selected dataset relevant to specific tasks and domains. Preference training (Section 3.2), on the other hand, aligns the model’s outputs with human values and preferences by prioritizing responses that meet user expectations and ethical guidelines. These methods work together to create reliable and ethically sound LLMs for real-world use.

### 3.1 Supervised Fine-Tuning

#### Data

Our SFT data is curated from a diverse array of sources. Given a piece of context from a source, we utilize humans and/or generative models (Ding et al., ) to identify if the text can be considered suitable for supervised fine-tuning or if we can generate instructions to create an SFT example from the context. For English, we primarily use public web content as our main source, offering a broad range of high-quality and especially diverse prompts. In contrast, our Arabic data comes from a combination of public and proprietary sources to ensure comprehensive coverage and relevance. To gather data from the source, we collect seed websites or data sources, which involves utilizing domain experts, prompt librarians, local institutes specializing in areas such as Arabic language, history, and politics, the use of commercially permissible licensed LLMs to generate data, and machine translation models to convert rich English SFT data into Arabic. Our datasets cover various domains and capabilities, ensuring the model’s proficiency in handling tasks across education, history, Arabic linguistics, politics, religion, computer science, and other fields. The entire Arabic/English collection is called Ultra-Instinct, which is not *human generated*, but rather, *human driven*.

#### Quality Filtering

Unlike Zhou et al. (); AI et al. (), we hypothesized that scaling SFT data can unlock diverse capability, as well as improve responsiveness to the prompts. Initially, we crawled the public web for SFT samples. The first version (v1) of Ultra-Instinct includes 12M samples evenly split between English and Arabic, while the second version (v2), is a reduced version with half the samples. Compared to v1, v2 underwent strict quality checks and human assessments of random subsamples. Our quality checks for v2 included *(i)* assessments based on instruction/response word length, *(ii)* lexical (^5^55Lexical diversity is calculated by taking the ratio of the total number of unique words to the total number of words across all samples, excluding stop words.) and semantic diversity, exact and near-exact lexical deduplication, *(iii)* removal of low quality machine-translated Arabic data from English sources, and *(iv)* ensuring diversity in questions and commands.
For detailed metrics on instruction and response lengths and lexical diversity, see Table 2.

Figures 7(a) and 7(b) show the distribution of the prompts and responses in v2, respectively. We focused on maximizing the number of multi-turn conversations in our dataset. Figure 7(c) shows the distribution of conversation turns from Ultra-Instinct.

Figure: (a) Prompt word count.
Refer to caption: x7.png

**Table 2: Comparison of average word length and lexical diversity of prompts and responses.**
| Quality Metric | Ultra-Instinct v1 | Ultra-Instinct v2 |  |  |
| --- | --- | --- | --- | --- |
| Prompt | Response | Prompt | Response |  |
| Avg # of Word | 146.94 | 97.19 | 60.81 | 136.47 |
| Lexical diversity | 76.34 | 75.25 | 85.29 | 69.53 |

**Table 3: Comparative results of Ultra Instinct v1 and v2, across various evaluation datasets.**
| Version | MMLU | Exams (ar) | ACVA | ETEC |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Huang et al. () | Koto et al. () | en |  |  |  |  |
| Ultra-Instinct v1 | 51.0 | 68.0 | 63.8 | 56.8 | 79.8 | 66.8 |
| Ultra-Instinct v2 | 51.4 | 68.5 | 63.3 | 56.8 | 76.7 | 65.9 |

To extrinsically evaluate the impact of higher quality SFT data, we trained two 13B models using our v1 and v2 SFT datasets. Even though v2 has half as many samples and v1, both versions performed equally well on English and Arabic evaluation benchmarks, as shown in Table 3. This reduction in data volume led to faster training times and reduced costs without compromising performance. It also clearly demonstrates the value of quality filtering for alignment.

#### Training Details

We fine-tune our base model, which was trained on 3.2 trillion (2T Llama-2 + 1.2T ALLaM) tokens, for 3 epochs using Ultra-Instinct-v2 with a learning rate of $5\times 10^{-6}$ and a batch size of 1024. The model is not trained to generate the prompt, as we mask out our prompt tokens when calculating the loss. Ultra-Instinct-v2 contains a substantial number of multi-turn conversations. To train on these multi-turn conversations, we performed turn-augmentation. Figure 8 visually explains the process of turn augmentation.

While training the SFT model, we encountered tokenization issues. Specifically, Llama-2’s tokenizer was trained using sentencepiece(^6^66[https://github.com/google/sentencepiece](https://github.com/google/sentencepiece)), which breaks the beginning and end of sequence tokens into multiple tokens and adversely affects long multi-turn conversations. To address this issue, we patched sentencepiece using the HuggingFace LlamaTokenizer wrapper (Wolf et al., ). Over many iterations of training, we saw that even having 1% noisy samples (e.g., empty responses or formatting issues) in alignment data can noticeably affect model quality.

### 3.2 Preference Training

After SFT, models are able to converse in multi-turn conversations. However, they are not fully aligned with human preferences. For example, our SFT models were terse and had limited guardrails. To circumvent these issues, we performed preference tuning with human verified samples via Direct Preference Optimization (DPO) (Rafailov et al., ).

Figure: Figure 8: Augmentation process for conversations. The original conversation (left) is expanded into one sample per turn (right), with user prompts and assistant replies marked for training (red) and masking (orange) to enhance ALLaM’s language understanding and multi-turn response generation capabilities.
Refer to caption: x10.png

#### Data

The inputs were sourced from early model testers and a manually curated selection of prompts from various domains or attack vectors. These include ethical dilemmas, middle eastern culture, religions, illegal activities, human rights, locale awareness, and personality.

Preference training necessitates both negative and positive outputs for each input. We relied on the testers’ feedback to identify the positive outputs. In the absence of positive outputs, we used a model to generate an output and manually verified that the output was aligned. While (Tunstall et al., ) utilized preference data from AI Feedback (AIF) at scale, we adopt a more cautious approach in creating preference data. We generate a smaller volume of data, ensuring it is fully reviewed, edited, and verified by humans.

There are two approaches for generating negative outputs: *(i)* on-policy: use the generations of the model we are tuning as negative outputs, and *(ii)* off-policy: use another similar model to generate the negative outputs. We did not verify that the negative outputs were worse than the positive. However, we ensured that the positive outputs were of the highest quality, such that they were almost always better than the negative outputs.

Khan et al. () demonstrated that model outputs can vary significantly depending on the sampling mechanism used. Building on this insight, we generate additional samples for each instance by varying temperature and nucleus sampling techniques. These additional samples are utilized to produce rejected samples, ensuring that ALLaM provides more grounded responses and generalizes well across various sampling mechanisms.

In total, we collected 25,854 samples (triplets of {prompt, accepted, rejected}) in English and Arabic language. Using the technique mentioned above, we sample 10 different response from the model to generate additional rejected responses for each sample. This results in a dataset of 245K samples (after filtering) for preference training.

#### Training

For DPO, we used a batch size of 512 with $\texttt{KL}_{penalty}=0.1$ and a learning rate of $9\times 10^{-7}$ decayed to $5\times 10^{-7}$ using a cosine annealing learning rate schedule. We train ALLaM for a single epoch using all the preference data.

From our initial experiments with small datasets, we observed issues with model quality even when a small fraction (0.1%) of the data was noisy. In this context, noise can be improper labeling of positive/negative pairs or low quality positive outputs. It is not clear, however, if after scaling up the DPO data whether the model can ignore this type of noise. In early DPO models, trained on data where we did not verify all the samples, we found that even a few moderately noisy samples resulted in broken models that repeatedly generate the same text or output incoherent text.

#### DPO vs. PPO

One of the fundamental differences between DPO and PPO is that PPO is always on-policy with an external reward model. In our experience with DPO, we did not encounter any significant issues with off-policy experiments. Additionally, DPO allows for faster iteration and easier understanding of the training dynamics. The decision to use DPO over PPO was based on logistical constraints rather than a performance comparison of the algorithms. Given our compute setup and time constraints, we chose to proceed with DPO. We plan to explore PPO in the future for alignment.

## 4 Evaluation

Figure: Figure 9: Selected benchmark evaluated through ALLaM’s training. Using HellaSwag as a proxy for language understanding, the performance of smaller models degrades when introducing Arabic, while larger models (70B) have enough capacity to improve simultaneously in English and Arabic. Arabic language acquisition is rapid in all models, as indicated by Arabic MMLU.
Refer to caption: x11.png

Table: Table 4: Arabic benchmark results for instruction tuned models.

Table: Table 5: English benchmark results for instruction tuned models.

In this section, we describe the evaluation of our model and report the results of ALLaM 7B, 13B, and 70B models, as well as other relevant models, such as GPT-4, Command-R+ (Gomez, ), and Jais-30B (Sengupta et al., ). Our evaluations encompass three main types: *(i)* automatic evaluations, *(ii)* LLM-based evaluations, and *(iii)* human evaluations.

#### Limitations

Recently, (Alzahrani et al., ) showed that multiple choice or cloze test based evaluation may not be robust. In addition, MT-Bench uses an LLM as a judge, and has likely leaked into training datasets. Unfortunately, human evaluation is time-consuming and requires well-trained human evaluators. In this work, we try to ensure robust evaluation and attain a balanced assessment of the quantitative metrics and qualitative effectiveness of models in various applications and domains.

### 4.1 Automatic Evaluations

The automatic evaluations cover Arabic and English benchmarks grouped into the categories listed below:

- 1.
Multi-domain: MixEval (Ni et al., ), MMLU-Pro (Wang et al., ), and BBH (Suzgun et al., ).
- 2.
Reasoning and Commonsense: HellaSwag (Zellers et al., ), PIQA (Bisk et al., ), WinoGrande (Sakaguchi et al., ), and AraSwag (Nagoudi et al., ).
- 3.
World Knowledge and Language Understanding: MMLU (Hendrycks et al., ),ARC Easy and Challenge (Clark et al., ), TriviaQA (Joshi et al., ), BoolQ (Clark et al., ), NQ Open (Kwiatkowski et al., ), AGIEval (Zhong et al., ), Exams-Ar (Hardalov et al., ), MMLU Arabic (tr) (Huang et al., ), MMLU Arabic (MBZU) (Koto et al., ) , and ETEC (in-house curated).
- 4.
Safety and Alignment: Hendrycks Ethics (Hendrycks et al., ), ACVA (Huang et al., ), TruthfulQA (Lin et al., ), and AraTruthfulQA (in-house curated).
- 5.
Conversation: MT-Bench (Zheng et al., ), and Arabic domain capability dataset (in-house curated).
- 6.
Math: Minerva MATH (Lewkowycz et al., ; Hendrycks et al., ), GSM8K (Cobbe et al., ) and AraMath (in-house curated).
- 7.
Coding: HumanEval (THUDM, )

The following benchmarks were curated and developed in-house:

- •
ETEC: a collection of 1891 multiple choice questions covering different exams performed by the Education and Training Evaluation Commission in Saudi Arabia(^7^77[https://etec.gov.sa/home](https://etec.gov.sa/home)).
- •
AraMath: a set of 600 test samples that were post-processed and prepared from the AraMath (Alghamdi et al., ) dataset. These samples focus on testing the models’ performance on Arabic math problems.
- •
AraTruthfulQA: a dataset created using similar methodology to the TruthfulQA (Lin et al., ) dataset. It comprises a total of 541 samples, 285 samples were translated directly from TruthfulQA using GPT-4 and carefully validated and localized by human verifiers. Additionally, 256 questions were curated by humans to ensure their contextual relevance and cultural appropriateness.

While serving as a good test bench, observing the dynamics of automatic evaluations during training is also interesting. Figure 9 shows the behavior of selected benchmarks during mixed Arabic/English pretraining while scaling up model size. In particular, we observe that smaller models tradeoff between capability in the new and original languages. However, larger models can simultaneously improve in both languages.

Another observation from automatic evaluations is that some evaluations provide more signal for training decisions than others, e.g., Hellaswag smoothly improves during training while improvements in GSM8k occur in discontinuous jumps. Other benchmarks show no improvement until 1.5T tokens have been seen (i.e., grokking) making them unreliable for early training decisions.

Tables 4 and 5 give an overview of the performance of ALLaM instruct models compared to other relevant models.

In Arabic benchmarks, we can see that ALLaM-70B scores are the best in five (MMLU Arabic (natural and translated), Exams, ETEC, araTruthfulQA) out of the eight benchmark sets. On English, ALLaM is the second-best model in the majority of cases, following Llama 3-Instruct. We highlight the excellent performance of ALLaM on benchmarks released after training was completed (MMLU-Pro, MixEval) and benchmarks the training team did not have access to (ETEC), since they provide a clean evaluation signal.

Table 7 and Table 8 in the appendix contain more detailed evaluation results for Arabic and English, respectively.

#### Evaluation Framework

All evaluations were completed using the Language Model Evaluation Harness (Gao et al., ) with the following exceptions: HumanEval was evaluated using BigCode Evaluation Harness (Ben Allal et al., ). MMLU-Pro, MixEval, and Arabic MMLU (Koto et al., ) were evaluated using the repositories of the dataset creators.

### 4.2 LLM-Based Evaluations

MT-Bench (Zheng et al., ) consists of 80 multi-turn questions to evaluate models’ capabilities on complex instruction-following. In addition to the English version, MT-Bench Arabic was created using GPT-4 to translate the original dataset and human annotators to review and align the prompts to Arabic culture. GPT-4 serves as the LLM judge, scoring responses as recommended in (Zheng et al., ). Model performance is compared turn by turn, with results shown in Table 6, where ALLaM-70B achieves the best Arabic performance.

**Table 6: MT-Bench scores for Arabic and English. Each score is an average over 80 samples of the score ranging from 0 to 10 returned by the judge (GPT-4).**
| Model | English | Arabic |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Avg. | Turn 1 | Turn 2 | Avg. | Turn 1 | Turn 2 |  |
| AceGPT 13B-chat | 5.44 | 6.76 | 4.12 | 6.33 | 7.01 | 5.64 |
| ALLaM 13B Instruct | 7.34 | 7.67 | 7.01 | 7.57 | 7.9 | 7.23 |
| ALLaM 70B Instruct | 7.44 | 7.91 | 6.96 | 8.19 | 8.4 | 7.97 |
| Jais 13B Chat | 4.18 | 4.39 | 3.96 | 4.72 | 5.07 | 4.36 |
| Jais 30B Chat v1 | 3.89 | 4.13 | 3.64 | 3.54 | 4.13 | 2.95 |
| Jais 30B Chat v3 | 5.86 | 6.25 | 5.47 | 6.28 | 6.78 | 5.78 |
| Cohere Command R+ | 7.41 | 7.63 | 7.18 | 7.97 | 8.28 | 7.65 |
| Cohere Command R | 6.99 | 7.19 | 6.79 | 7.47 | 7.82 | 7.12 |
| DBRX Instruct | 7.16 | 7.33 | 6.98 | 7.83 | 8.19 | 7.46 |
| GPT 3.5 Turbo | 7.55 | 7.79 | 7.31 | 8.12 | 8.39 | 7.84 |

### 4.3 Human Evaluation

We developed an Arabic multi-turn dataset that covers seven domains: Arabic linguistics, history, health, politics, coding, entertainment, and ethics, each domain contains ten questions with two turns.

Human evaluators compared the responses from two models and were asked to choose the winning response with the following instructions:

- •
Choose a response as the winner if it is the best, tie if both responses are equally good, and both-bad if both responses are not good.
- •
A response is considered good if it is coherent, grammatically correct, and is a reasonable response to the question or previous turn in the conversation.
- •
Good responses should be in the correct language (the response should be in the same language as the previous turn, unless another language was requested).
- •
Good responses should not contain toxicity, hate speech, or bias.

Each pair of responses was inspected by three evaluators, and the winner was determined by majority voting. In case of a tie, a fourth evaluator was used to break the tie. Figure 11 presents the human evaluation results of the pair-wise comparisons of these models: ALLaM-13b, Jais-30b-chat-v3, Command-R-plus, and Command-R-v01. ALLaM-13b’s win rate was always higher than its loss rate compared with other models.

Finally, we gather votes from the human evaluators and calculated ELO scores for each model. ELO scoring had two configurations, the default scoring rewards the model for good responses with 1 point, tied responses (good and both-bad) with 0.5 points, and penalizes for bad responses with 0 point. The custom configuration penalizes the model with the bad response and both models if both provided bad responses with 0 point. Figure 11 shows the ELO scores based on the human evaluations. From the figure, GPT-4 achieved the highest score, followed by ALLaM-13b with the second-highest score, outperforming (or matching) larger models such as CommandR+.

Figure: Figure 10: Pairwise win rates as judged by human evaluators. ALLaM-13B wins against many much larger models.
Refer to caption: x14.png

## 5 Related Work

Our work sits at the cross-section of research building language models that support multiple languages and scaling such techniques in terms of size and data. To successfully train a large language model in a language other than English requires a complete understanding of cross-lingual transferability between languages and a good understanding of scaling laws as well as the fundamentals of training large language models. In this section, we discuss work on language modeling from the perspectives of cross-lingual alignment, multitask learning, and Arabic specialization.

### 5.1 Language Modeling and Cross-lingual Representations

In early work, word representations were derived using basic forms of the skip-gram model (Mikolov et al., ), wherein each word is assigned a representation that does not account for varying contexts (Grave et al., ; Pennington et al., ). Further work in this area developed word representations that are adaptive to the context surrounding the words (McCann et al., ; Peters et al., ; Howard & Ruder, ; Devlin et al., ; Yang et al., ; Radford et al., ).

Peters et al. () introduced ELMo, a model built with a bidirectional LSTM-based language model (LM) for pretraining to obtain contextualized word representations. This technique combines the outputs from all layers linearly when targeting specific tasks. Nonetheless, the sequential nature of LSTM-based LM pretraining presents challenges in scaling training efficiently. Concurrently, Vaswani et al. () developed the Transformer architecture, which leverages multi-headed self-attention and positional encoding to handle long-range dependencies and enable parallel processing. Following that Radford et al. () introduced GPT, a model that pretrains a Transformer decoder using a conditional language model objective, with subsequent fine-tuning requiring only minimal modifications. Similarly, Devlin et al. () unveiled BERT, which uses a Transformer encoder pretrained via a masked language modeling (MLM) objective. This approach excelled at task adaptation and benefited from the MLM’s ability to encode context bidirectionally, unlike the traditional unidirectional (conditional) LM that processes either the left or right context. Later Raffel et al. () proposed a detailed hybrid encoder-decoder architecture based LLM with an implementation of many objective function via autoregressive structure.

During the release of BERT, mBERT, a multilingual version of BERT is trained on 102 languages using a shared vocabulary of 110K subword tokens.(^8^88[https://github.com/google-research/bert/blob/master/multilingual.md](https://github.com/google-research/bert/blob/master/multilingual.md)). Despite the lack of explicit cross-lingual supervision, mBERT has demonstrated the ability to learn cross-lingual representations that generalize well across languages. Wu & Dredze (); Pires et al. () evaluated the zero-shot cross-lingual transferability of mBERT on several NLP tasks and attributed its generalization capability to shared subword units. Pires et al. () additionally identified structural similarity (e.g., word order) as another crucial factor for successful cross-lingual transfer. K et al. (), however, argued that shared subwords contribute minimally, and instead, structural similarity between languages is more critical for effective transfer. Artetxe et al. () further showed that joint training might not be necessary and proposed an alternative method to transfer a monolingual model to a bilingual model by learning only the word embeddings in the target language. They also highlighted the vocabulary size per language as an important factor. Finally, Xue et al. () showed that joint training on a large multilingual vocabulary can robustly map multilingual language models to the same latent space.

In the early days, *Cross-lingual alignment* from mono-lingual embeddings was tricky and often required complex adversarial training (Conneau et al., ), careful orthogonal mapping (Artetxe et al., ) or semi-supervised learning (Mohiuddin et al., ; Bari et al., ). With the introduction of mBERT, it became evident that learning joint distribution makes it easier for LLM to achieve cross-lingual alignment at scale. Lample & Conneau () enhanced mBERT by incorporating a conditional LM and a translation LM (leveraging parallel data) objective along with a language embedding layer and trained a larger model utilizing more monolingual data. Huang et al. () suggested employing auxiliary tasks like cross-lingual word recovery and paraphrase detection for pretraining. Subsequent work by Conneau et al. () and Soltan et al. () scaled up the training of multilingual language models. As well, Xue et al. () scale the size and languages in the T5 architecture. In an effort to reproduce GPT-3, Scao et al. () trained the first auto-regressive multilingual LLM.

### 5.2 Multitask Learning And Alignment

Early work has demonstrated that multitask learning can enhance the performance of NLP models (Collobert & Weston, ). In explicit multitask learning, augmenting all samples during training may introduce noise due to differing output distributions in a traditional full-model fine-tuning setup (Weller et al., ; Bari et al., ). For implicit multitask learning, Radford et al. () showed that a language model can begin to learn downstream tasks without explicit supervision by pretraining alone. Large language models (Brown et al., ; Smith et al., ; Chowdhery et al., ) at scale can perform few-shot in-context learning, making them effective multitask models. Additionally, Sanh et al. (); Wei et al. (); Muennighoff et al. (); Chung et al. () found that these implicitly learned language models could be further improved by explicitly fine-tuning them with human instructions and prompts (Bach et al., ; Wang et al., ) in a multitask fashion. Unlike previous template-based prompting approaches, Ouyang et al. () applied preference tuning with reinforcement learning (Stiennon et al., ) using naturally written prompts. Subsequently, Bai et al. () introduced *Constitutional AI* to automate alignment using AI feedback. Recently, following the work of Rafailov et al. (), various efforts (Azar et al., ; Ethayarajh et al., ; Hong et al., ; Park et al., ; Meng et al., ) have been directed towards preference tuning without explicit reward models.

### 5.3 Language Models for Arabic

As of the time of writing, the most prominent Arabic-focused LLMs are:

- 1.
Jais (Sengupta et al., ): 13B and 30B base and chat models trained from scratch using a combination of natural and translated Arabic data along with English and code data.
- 2.
AceGPT (Huang et al., ): 7B and 13B base and chat models trained from Llama-2 *without* vocabulary expansion.

While Jais and AceGPT are currently the most prominent models, early open models such as AraGPT (Antoun et al., ), AraT5 (Elmadany et al., ), AraBART (Eddine et al., ), and Noon (Lakim et al., ) (^9^99[https://huggingface.co/Naseej/noon-7b](https://huggingface.co/Naseej/noon-7b)) pioneered the area with models developed with limited resources to serve Arabic.

Other models such as Jasmine (Abdul-Mageed et al., ) and Aramus (Alghamdi et al., ) also showed the need for building a language model for over 400 million speakers worldwide.

In addition to the language adaptation of models and multilingual models reviewed above, recent work has focused on building multilingual/bilingual language models from open weight language models. For example, Ruciński () adapted Mistral 7B for the Polish without vocabulary expansion. Mala-500 is another effort to expand to 534 languages by expanding the vocabulary to 260K tokens and further pretrained Llama-2 using LoRA adaptors (Lin et al., ). Due to the number of languages they aimed to support, a small amount of data was included for each language and the evaluation of the approach was limited to measuring perplexity and automatic classification benchmarks. (Cui et al., ) introduced a Chinese Language adaptation of Llama and Alpaca models, where the vocabulary was increased to 50K tokens, then continued to pretrain the models and finally fine-tune them.

## 6 Conclusion

The ALLaM model series marks a significant advancement in Arabic Language Technologies, achieving state-of-the-art performance across various Arabic benchmarks while maintaining or enhancing English performance. Through careful training that emphasizes language alignment and transferability, our models demonstrate effective second-language acquisition without catastrophic forgetting. The strategic use of translated data, knowledge encoding, and alignment with human preferences have been crucial in this success.

## 7 Limitations

ALLaM was trained on data that may potentially include toxic language, unsafe content, and societal biases originally sourced from the internet, leading to the possible amplification of these biases and toxic responses. Although ALLaM underwent comprehensive safety training during the alignment phase, more community feedback is needed to iteratively improve ALLaM. Additionally, inherent uncertainties in generative models mean that trials cannot encompass every possible use case, making it impossible to predict the model’s responses in all contexts. This can occasionally result in inaccurate, biased, or socially unacceptable outputs, even if the prompt itself is not explicitly offensive. Developers must conduct thorough safety evaluations and make specific adjustments to ensure that ALLaM is suitable for their intended purposes. Furthermore, the output generated by ALLaM should not be considered a statement from ALLaM’s creators or any affiliated organization.

## 8 Ethical Statement

While conducting and presenting this research, we are committed to upholding the highest ethical standards. We recognize the potential impact of large language models on society and the importance of ensuring their responsible development and deployment. Our work adheres to principles of fairness, transparency, and inclusivity, striving to mitigate biases and ensure diverse representation in our training data. We are mindful of privacy concerns and have taken steps to anonymize and secure data used in our research. Additionally, we acknowledge the potential for misuse of language technologies and advocate for their ethical application, promoting beneficial use cases while being vigilant about unintended consequences. ALLaM models are made openly available to foster collaboration and further research, with the aim of contributing positively to the advancement of language technologies and supporting the cultural and technological growth of the Arabic-speaking world.

## 9 Risk Statement

The deployment and use of LLMs in various applications poses significant risks, including data privacy and security concerns due to the inadvertent inclusion of sensitive information in training datasets. LLMs can perpetuate or amplify biases, resulting in unfair treatment and discrimination in critical decision-making processes. They can also generate convincing but inaccurate content, spreading misinformation and potentially influencing public opinion negatively. Over-reliance on LLMs may diminish human judgment, and the models’ susceptibility to adversarial attacks can compromise system integrity. To mitigate these risks, we follow robust governance, continuous monitoring, and iterative improvements. We also adhere to best practices in data handling and model training, fostering transparency and accountability in LLM development.

## Appendix A Appendix

### A.1 Detailed Arabic Evaluation

Table: Table 7: Comprehensive Arabic benchmark results.

### A.2 Detailed English Evaluation

Table: Table 8: Comprehensive English benchmark results.

## Appendix B Author Contributions

In no particular order,

- •
M Saiful Bari: Research lead for alignment at scale (second language acquisition). Wrote the basic orchestration and optimization of the LLM training framework, data, and training deployment. Trained all the models explained in the paper. Helped build the evaluation infrastructure and orchestrate the entire LLM engine. Implemented the alignment training pipeline and sandboxing environments. Served as a prompt librarian. Supported any project requirements regardless of training, data, or evaluation. Wrote a major part of the paper.
- •
Yazeed Alnumay: Implemented pretraining data translation at scale. Worked on vocabulary embeddings augmentation and initialization, as well as the model framework conversion logic. Built the automatic evaluation infrastructure. Acted as a prompt librarian and built the SFT data infrastructure, including SFT and preference data acquisition, curation, processing, cleaning, and lifecycle management. Supported any project requirements regardless of training, data, or evaluation. Contributed to the figures and manuscript writing.
- •
Haidar Khan: Guided and reviewed all aspects of the work, with a focus on pretraining and alignment decisions. Contributed directly to optimizing the training infrastructure and trained the 7B model from scratch. Edited and wrote majority of the paper.
- •
Areeb Alowisheq: Managed the ALLaM training team and wrote the configurations for v0 models. Contributed to all research and logistics decisions for ALLaM. Edited and reviewed the paper, wrote the Arabic LLM related works section.
- •
Nora Al-Twairesh: Managed the data and evaluation effort of ALLaM. Assisted in building the complete data and evaluation ecosystem for the ALLaM project. Edited and reviewed the paper.
- •
Abdalghani Abujabal: Worked on SFT and alignment data creation, quality metrics for data, and model red teaming. Standardized human evaluation for alignment data and led a major part of the data effort. Wrote the data sections of the paper.
- •
Jeril Kuriakose: Worked on MT bench evaluation and maintained all deployment and pretraining data preparation. Supported any project requirements regardless of training, data, or evaluation. Wrote the MT bench evaluation section of the paper.
- •
Ahmed Abdelali: Worked on the Arabic pretraining data. Led the Arabic OCR effort and book corpus acquisition and creation. Worked on Arabic SFT data.
- •
Abdulmohsen Al-Thubaity: Worked on Arabic SFT data creation. Prepared a major part of Arabic SFT data.
- •
Zaki Alawami: Lead the HPC (LLM infrastructure) development and model deployment.
- •
Ali Alammari: Worked on the acquisition of pretraining Arabic data and the creation of the data processing pipeline.
- •
Majed Alrubaian: Supported the entire data, training, and evaluation teams. Worked on Arabic Education SFT data.
- •
Norah A. Alzahrani: Implemented all private Arabic evaluations. Lead the human evaluation and prompt acquisition. Maintained the human evaluation pipeline. Wrote the Arabic human evaluation section and prepared the figures.
- •
Nouf M. Alotaibi: Trained the v0 version of ALLaM on NeMo. Trained all the tokenizers and prepared training data for the v0 and v3 versions. Prepared the initial SFT figures of the paper. Worked on the red-teaming, SFT quality metrics, and generating visualizations for SFT data.
- •
Hisham A. Alyahya: Trained the v0 version of ALLaM on NeMo, identified bugs in NeMo checkpointing and resuming from checkpoints. Worked on model deployment, measured latency and throughput, and evaluated trade-offs between frameworks like TRT-LLM, vLLM, and HF-TGI. Deployed the preference data collection frontend and wrote the data sampling code for v0 models.
- •
Sultan AlRashed: Worked on Megatron-LM checkpointing and maintained the v0 version of the SFT and RLHF code. Implemented the preference data pipeline in Megatron-LM. Worked on the alignment specialization. Profiled Megatron-Deepspeed.
- •
Faisal A. Mirza: Trained the Arabic-only tokenizer and worked on model creation with an expanded vocabulary.
- •
Shaykhah Z. Alsubaie: Maintained evaluation results for the paper, worked on synthetic SFT data creation, specialized model finetuning, few-shot evaluation, and model capability evaluation. Trained router models and worked on the tokenizer. Worked on red-teaming ALLaM models.
- •
Hassan A. Alahmed: Worked on pretraining data pipeline for Arabic.
- •
Ghadah Alabduljabbar: Worked on data pipeline for Arabic, SFT quality metrics. Worked on red-teaming ALLaM models.
- •
Raghad Alkhathran: Worked on data pipeline for Arabic. Worked on red-teaming ALLaM models.
- •
Yousef Almushayqih: Worked on lm-harness evaluation framework. Worked on red-teaming ALLaM models.
- •
Raneem Alnajim: Worked on training team. Worked on red-teaming ALLaM models.
- •
Salman Alsubaihi: Worked on deployment team. Worked on red-teaming ALLaM models.
- •
Maryam Al Mansour: Worked on the Arabic Data pipeline.

#### Note:

In this work we used the v2 version of our data and all the model mentioned in the paper are the v1 version trained on Megatron-LM. Our v0 model series were trained on NeMo.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: ...
soundness: 3
strengths: Focus on Arabic Language: The paper specifically addresses the need for high-quality language models for Arabic, a language with a significant number of speakers worldwide and relatively fewer resources compared to English.  This focus fills a gap in the current landscape of large language models.    

Comprehensive Training and Alignment Strategy: The authors employ a thorough approach to training and aligning ALLAM. They experiment with different data mixtures, vocabulary expansion techniques, and use both supervised fine-tuning and preference training to achieve optimal performance.    

State-of-the-art Performance: ALLAM achieves state-of-the-art results on various Arabic benchmarks, demonstrating the effectiveness of the proposed techniques.  The authors also show that their model maintains or enhances English performance compared to the base Llama-2 model.    

Exploration of Second Language Acquisition: The paper investigates the use of second language acquisition techniques for LLMs, which is a promising area of research.  The authors show how pretraining on a mixture of Arabic and English text can lead to effective learning of Arabic without catastrophic forgetting in English.    

Detailed Methodology: The paper provides a detailed description of the training methodology, data curation process, and evaluation setup.  This transparency allows for reproducibility and facilitates future research in the field.   

Commitment to Openness: The authors express their intention to make the ALLAM models openly available to the community, which promotes collaboration and further development of Arabic language models.
summary: This paper introduces ALLAM, a large language model for Arabic and English. The authors demonstrate the effectiveness of their training and alignment strategy for achieving state-of-the-art performance on several Arabic benchmarks.
weaknesses: The authors should clearly state the originality of their work. While the paper builds on existing techniques such as second language acquisition for LLMs, it is not clear what specific novel components are being introduced. Is it the particular training recipe, the alignment strategy, or the focus on Arabic? A clear statement of originality is essential for establishing the contribution of this work.

Error Analysis:
The error analysis presented is currently insufficient. The authors need to go beyond simply stating that some evaluations provide more signal than others. A deeper dive into the types of errors made by the model on different benchmarks is necessary. What are the common patterns in incorrect predictions? Are there specific linguistic constructs or knowledge domains where the model struggles? Addressing these questions will provide valuable insights into the limitations of ALLAM and potential avenues for future research.


Additional Areas for Improvement:
Motivation for changing training data distribution: The authors mention changing the training data distribution for the 30B model. The motivation behind this change is not entirely clear and should be elaborated upon.
Fair comparison of models: The paper touches upon the difficulty of comparing models due to variations in training data size, architecture, and other factors. While comparing to larger models is a reasonable approach, the authors could consider additional metrics or analyses to further ensure a fair comparison.
Instruct model vs. base model results: The authors justify reporting results for the instruct model instead of the base model. However, providing some comparison or insights into the performance difference between the base and instruct models could be beneficial.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: none
soundness: 3
strengths: The paper gives the details of the training of Arabic LLMs, including training from scratch and contunue training from Llama2.  The result Arabic LLMs achieve SoTA on Arabic benchmarks.
summary: This paper presents a series of LLMs for Arabic adapted from Llama2 as well as training from scratch.  The paper gives technical details of the pre-training, alignment and evaluation of these models.
weaknesses: This is mainly a technical report rather than a research paper.  I do not see novel ideas or methods proposed.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: - How many Arabic tokens already existed in the LLama2 backbone, and how many were added after the expansion?

- Why did the training mix change for the 34B model specifically? (lines 286-292) 

- I would be interested to compare allam against Silma Arabic llms

- Please consider evaluating on more dataset like AlGhafa benchmark or including ALLAM on the huggingface Arabic leaderboard.
soundness: 4
strengths: Originality: As far as I know, this is not the first work to fine-tune llama-based models for Arabic or other language acquisition. In my opinion, the main originality source in this paper might be the data curation effort, and the vocabulary expansion and continual pre-training without loss of original language. For example, the finding that initializing the embedding of the new tokens from averages of smaller tokens in the original tokenizer is an very useful finding and I expect to see it being used more in the future.

Quality: The effort spent in human evaluation and curation of the data is notable, and I hope this dataset gets released, at least partially for the research community to build on. The comparison against other Arabic-enabled LLMs are also very comprehensive and useful.

Clarity: While many typos and non-standard terminology exist and are highlighted in the weaknesses section. The paper is still mostly easy to follow and read. The description of the datasets and training/evaluation procedure is particularly clear and concise.

Significance: Arabic is definitely an excellent language choice for this exploration of second language acquisition, given that the original LLama2 family of models do not train on Arabic at all (as shown in Table 10 in the original Lama2 report), and that means that the tokenizer and model are not very well trained for Arabic. Also, Arabic is definitely an underserved language in the LLM, especially given the size of its native speakers population (fifth most in the world, roughly 400M people).
summary: This paper presents a ALLaM: a family of large language models for Arabic and English, based on LLama-2 family of models.

The paper shows a method that can customize an existing LLM (like Llama) to acquire a much better performance at another language that was not included at all the original training of the model, without destroying the LLM's performance on previous language (i.e. second language acquisition).

The paper shows that second language acquisition can be done by augmenting the model in two major steps: 1-vocabulary expansion, and 2- carefully pre-training on mixture of new language and old language. This way the second language acquisition is not accompanied by catastrophic forgetting of the original language.

The paper includes a lot of experiments and comparisons with different baselines, including training from scratch. Showing that their second language acquisition method beats training from scratch. Many evaluations, both automated and human-evals are included in the paper.
weaknesses: Even though the paper is mostly well-written, it's littered with typos, non-standard terminology and missing references:
- "Fertility rate": very confusing term, it's not really a rate, and is usually referred to "fertility score": https://arxiv.org/pdf/2310.08754 
- Figure 8 is not referenced anywhere as far as I can see.
- line 475: missing reference
-  line 294: missing reference
- line 291: lower case "we"
- line 072: training of "these" 
- lines 151-153: DA identification is a hard task ... classifying the data to DA is not very difficult (opposite claim)  
- line 155: in-house in-house  

- I would have loved to see some newer Arabic models like Silma included in the comparisons for this paper, or a comparison against the models in the huggingface Arabic leaderboard.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. Page 3 says 4T English tokens then and 5.2T tokens for 30B pre-training. It was a bit confusing what is the 5.2T. Also Table 1 lists English as 660B tokens from mixed corpora -- is this a subset from the 4T English Only column?

2.  Fig 5/6: for my display, there are black lines in the legend (e.g. Arabic/English MMLU) but only red/orange lines in the figure.

3. I am curious if you have considered using something other than Llama2 as your base model. If so, why did you pick Llama2 in the end?
soundness: 4
strengths: This is a timely paper, as many research groups are working on building LLMs for languages other than English. The paper presents useful details on their experience and will serve as a good reference.
summary: This paper describes the process for building ALLAM, an LLM trained on Arabic and English. Details about the training data and each stage of the training process are presented. Various evaluations are performed, including both automatic benchmarks like Arabic MMLU and human evaluation.
weaknesses: There are no major weaknesses in my opinion. I would recommend adding discussing about other efforts in building LLMs for non-English languages and explaining how yours compare, i.e. moving some of the Related Work into the main paper. Especially it would be useful for the reader (who isn't referring to the appendix) to know about Jais and any other Arabic-centric model.

Meta-review: This paper presents the process used to train ALLaM, a series of Arabic and English LMs finetuned on top of Llama-2. The paper considers several adaption techniques to adapt the model to a new language and provides insights into best practices for adapting English LMs to other languages in the future. The resulting models achieve good performance on the considered downstream tasks.

Strengths:
- The paper is well-motivated (Arabic is a relatively underserved language among LLMs), and the paper clearly describes the process and methodology used to adapt an existing LM to a new language.
- Based on the experimental results, ALLaM models achieve state-of-the-art performance on multiple Arabic language benchmarks.

Weaknesses:
The main contribution of this paper is (and is stated to be) the development of a family of Arabic language models that achieve the state-of-the-art on Arabic language benchmarks.  However, limited artifacts from the paper have been released: the 7B and 13B models are only available through (separate) proprietary platforms (Appendix B), and there is no firm agreement to fully release the models upon publication. The data collected for Arabic pretraining, supervised finetuning (SFT), and preference optimization (DPO)—another major contribution, in my opinion—has also not been released. I encourage the authors to fully release their models and data, which will provide a significant contribution to the Arabic NLP community.

Some issues raised by the reviewers, such as presenting the novel aspects of the paper more clearly (Reviewer hSix) and the typos raised by 9J4g, have been addressed in the revised version of the paper.

Decision: Accept (Poster)