ALLaM: Large Language Models for Arabic and English
Authors: M Saiful Bari
Yazeed Alnumay
Norah A. Alzahrani
Nouf M. Alotaibi
Hisham Abdullah Alyahya
Sultan AlRashed
Faisal Abdulrahman Mirza
Shaykhah Z. Alsubaie
Hassan A. Alahmed
Ghadah Alabduljabbar
Raghad Alkhathran
Yousef Almushayqih
Raneem Alnajim
Salman Alsubaihi
Maryam Al Mansour
Saad Amin Hassan
Dr. Majed Alrubaian
Ali Alammari
Zaki Alawami
Abdulmohsen Al-Thubaity
Ahmed Abdelali
Jeril Kuriakose
Abdalghani Abujabal
Nora Al-Twairesh
Areeb Alowisheq
Haidar Khan
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model
English
Arabic
Second Language Acquisition
Primary area: foundation or frontier models, including LLMs
TL;DR: We trained a foundational Arabic LLM by leveraging English LLMs.

Abstract: In this work, we present ALLaM: Arabic Large Language Model, a series of large language models to support the ecosystem of Arabic Language Technologies (ALT). ALLaM is carefully trained, considering the values of language alignment and transferability of knowledge at scale. The models are based on an autoregressive decoder-only architecture and are pretrained on a mixture of Arabic and English texts. We illustrate how the second-language acquisition via vocabulary expansion can help steer a language model towards a new language without any major catastrophic forgetting in English. Furthermore, we highlight the effectiveness of using translation data and the process of knowledge encoding within the language model's latent space. Finally, we show that effective alignment with human preferences can significantly enhance the performance of a large language model (LLM) compared to less aligned models of a larger scale. Our methodology enables us to achieve state-of-the-art performance in various Arabic benchmarks, including MMLU Arabic, ACVA, and Arabic Exams. Our aligned models improve both in Arabic and English from its base aligned models.

Published as a conference paper at ICLR 2025
ALLAM: LARGE LANGUAGE MODELS FOR ARABIC AND ENGLISH
M Saiful Bari* Yazeed Alnumay"
Norah A. Alzahrani
Nouf M. Alotaibi Hisham A. Alyahya
Sultan AlRashed Faisal A. Mirza Shaykhah Z. Alsubaie
Hassan A. Alahmed Ghadah Alabduljabbar Raghad Alkhathran
Yousef Almushayqih Raneem Alnajim Salman Alsubaihi
Maryam AI Mansour Majed Alrubaian Ali Alammari Zaki Alawami
Abdulmohsen Al-Thubaity
Ahmed Abdelali
Jeril Kuriakose
Abdalghani Abujabal* Nora Al-Twairesh" Areeb Alowisheq' Haidar Khan"
National Center for AI (NCAI) Saudi Data and Authority (SDAIA) Riyadh, Saudi Arabia
ABSTRACT
We present ALLaM: Arabic Large Language Model, series of large language models to support the ecosystem of Arabic Language Technologies (ALT). ALLaM is carefully trained considering the values of language aligument and knowledge transfer scale. Our autoregressive decoder-only architecture models demon- strate how second-language acquisition via vocabulary expansion and pretraining on a mixture of Arabic and English text can steer a model towards a new language (Arabic) without any catastrophic forgetting in the original language (English). Furthermore, we highlight the effectiveness of using paralle//translated data to aid the process of knowledge alignment between languages. Finally, we show that extensive alignment with human preferences can significantly enhance the per- formance of language model compared to models of larger scale with lower quality alignment. ALLaM achieves state-of-the-ard performance in various Arabic benchmarks, including MMLU Arabic, ACVA, and Arabic Exams. Our aligned models improve both in Arabic and English from their base aligned models. Ara- bic assets are released in Hugging Face."
INTRODUCTION
Language modeling has significantly progressed from its humble origins, transitioning from funda- mental probabilistic methods to complex neural priors. The foundational work by Shannon (1951) on the information theory of language laid the groundwork for predicting the next word in sequence, which was subsequently tackled by Bengio et al. (2003) in neural networks. The field experienced substantial leap with the introduction of LSTMs (Hochreiter & Schmidhuber, 1997) in language models (LM) (Peters et al., 2018b), which could capture longer dependencies in LMs but proved dif- ficult to scale. The emergence of scalable and distributed architectures like Transformers (Vaswani et al., 2017) and the potential for precisely (Kaplan et al., 2020; Hoffmann et al., 2022) compress- ing web-scale data has resonated recent years with the advancements of Generative Pretraining (Radford et al., 2018; Brown al., 2020a; Anil al., 2023).
With the release of ChatGPT (OpenAI, 2022), followed by the introduction of more frontier class models Gemini (Google, 2024), Claude (Anthropic, 2022), Reka (Ormazabal et al., 2024), Mistral (Mistral, 2024), Ll a.lama-3 (Meta, 2024) and recently released Qwen-2 (Yang et al., 2024), large language models have demonstrated significant leaps over each generation of models (Laskar et al., 2023). This exponential growth in performance has raised hope in the possibility of achieving Artificial General Intelligence (Hendrycks & Mazeika, 2022; Mareus, 2022). This rapid advancement has spurred discussions across various fields, including ethics, economics, and technol-
*Core contributors
httpps: 'https://hugee htttps://hugginngfeee co/ AliiaM-A1

---

Published as a conference paper at ICLR 2025
Arabic MMLU
English MMLU
Allrn Inzzstt
70
Orica 722
50
Alart Insruct thee arreca)
Liuira chas
Lite 3-708
73
Cwee2-7
Ocark Caraannd
A0eursay (80
Cappnaad-n
Matrra
70
Sg
ChiRx
Jade I3D
Mittii-- lack
strisee
KGA 7
40
y 3.270
50
45
100
10
100
dog)
Figure I: Performance on Arabic (Koto et al., 2024) and English (Hendrycks al., 2020) MMLU Benchmarks. ALLaM (red line) shows impressive improvement from its base model Llama-2 (yellow line). All evaluations were done on the latest version of the fine-tuned (chat or instruct) models. The ALLaM 7B from scratch model also shows significant improvement over the ALLaM 7B continued pretraining model.
ogy (Weidinger et al., 2021). Judging from the initial capabilities (Bubeck et al., 2023), the potential of these frontier models are reinventing the way humans interact with machines, impacting social norms, productivity, trends, and culture on a broader scale (Zhou et al., 2024). However, most of these frontier-class models are primarily trained on English and often lack a connection to localized regional cultures and norms (Naous et al., 2024). This gap has the potential to result in slow and irreversible manipulation of regional identities and lead to cultural homogenization.
The natural course to reverse this trend is to invest resources in curating data and building models to support the diversity of languages and cultures represented in the modern world. While this is possible, the significant training costs of LLMs and their environmental impact have become major concerns in recent years (Strubell et al., 2019). The vast computational resources required to train LLMs contribute to substantial carbon emissions (Luccioni & Hernandez-Garch 2023). Govern- ments 2 and non/for-profi organizations (Dodge et al., 2022; Google, 2021; Amazon, 2021), are increasingly aware of these issues. This awareness has led to discussions about the ethical implica- tions of Al development and the need for sustainable practices concerning "When and how to scale the training of these models". In addition, curating data for each language/region at pretraining scale is also a difficult task, since most available data comes from a few high-resource languages.
To address these concerns, we consider the problem of adapting strong, but potentially under-traiie under-trained, open pretrained models, rather than starting from randomly initialized model. Technically, this involves continuing training of model in a new language to facilitate Second Language Acquisition (SLA) (Swain & Lapkin, 1995), popularized by Bari et al. (2020) in NLP and recently adapted to LLMs by Nguyen et al. (2023). This process involves the challenging task of incorporating an additional language distribution without compromising the source language(s). For instance, if a pretrained model was initially trained English, expanding to an additional language presents challenges related tokenization. Figure 2 gives an overview of ALLaM's tokenizers compared to tokenizer primarily trained on English.
Eoglssh
Arabic
Code
Rerality aosec
1.3mma2
A.LaM Araliic
All WM inerged
A.L.M [ovon scrricc
Figure 2: Comparison of tokenizer fortility scores. The chart illustrates the fertility scores across four tokenizers: Llama-2, ALLaM Arabic only, ALLaM merged with Llama-2 2, and ALLaM Arabic/English (from scratch model). We calculate the fertility over random subsample of the entire English, Arabic, and code training corpus.
There are various Arabic LLMs that have been developed, such as Jais (Sengupta al., 2023) and ACeGPT T (Huang et al., 2023), which trained from scratch or continued training L1 ama-2, respectively. more detailed description and other relevant works can be found in Appendix C.
Our approach to building ALLaM, large language models developed specifically for fluency and un- derstanding in Arabic and English, can be outlined as follows. We first demonstrate the feasibility
Phttps/// wtt.rre....eeeeeeeeeeeeeeeeeeeee wugercorgut reccye ing fie beat
2

---

Published as a conference paper at ICLR 2025
Loss Curves
Gradient: Norms
Benchmark Evaluations
2.2
No translation Transktion
1.00
HellaSwen accuracy (%)
76
MMMLU accuacy 0%)
Training loss
Gradient sprm
0.75
1.8
0.50
33
1.6-
72
1.4
0.25
30
70
1.2
25
100
200
300
400
500
100
200
300
400
500
100
2.00
300
400
500
Training tokens (B)
Figure 3: Measuring the effect of adding machine translated Arabic data to pretraining. Although the two loss curves look normal (left), adding the translated Arabic reduced the frequency of gradient spikes during training (center). Adding translated Arabic data also clearly helps align the Arabic and English capabilities of the model and reduce catastrophic forgetting (right).
of adapting existing pretrained English model (Llama - (Touvron et al., 2023a)) to fluency in both Arabic and English through tokenizer and vocabulary expansion. Then, we apply our learnings to train a stronger model from scratch (random initialization) in similar fashion, i.e., pretraining on English followed by training in mixed Arabic and English. The resulting model exhibits impressive performance and has favorable tokenization properties compared other models. This approach aligns with both our technical goals and our commitment to sustainable practices. Our overall con- tributions are summarized below:
1. The ALLaM model series, with the goal of supporting the cultural values of the Arabic- speaking world. We train four models at four different scales: 7B, 13B, 34B and 70B models initialized by Llama-2 weights along with 7B and 34B models trained from scratch.
2. Our model achieves state-of-the-ar results in Arabic, as well as improving overall English performance of the original L1ama-2 model. Refer Figure for an overview.
3. We demonstrate that possible to train highly-performant models low-resource lan- guages from publicly available model weights using continued pretraining recipe with tokenizer expansion, presenting a path for better representation of low-resource languages.
4. The training methodology and decision pecision-making involved in training the LLM. We provide necessary ablation studies for most crucial decisions.
PRETRAINING
Pretraining language models trillions of natural language tokens represents the bulk of the cost required to build an effective language model. This large investment of time and compute precludes experimentation ablation for every decision. Thus, before starting train ALLaM from random initialization, or "scratch", we experiment in the continue-pretrainiig regime. As the name implies, pertinueprettaiingggge. the practice of warm-starting an experiment from an already pretrained LM.
We begin by discussing our entire pretraining corpus, describe experiments conducted with continue- pretraining, and finally describe pretraining from scratch.
2.1 PRETRAINING DATA
For English, many high quality and large scale datasets are available for pretraining (Together Com- puter, 2023; Soldaini et al., 2024; Gao et al., 2021; Penedo et al., 2023). We harnessed subsets from RedPajama (Together Computer, 2023), Fibe (Penedo al., 2024), Dolma-v1 (Soldaini al., 2024) and Pile (Gao et al., 2021) datasets e.g., Dolma- The Stack (Kocetkov et al., 2022), PeS2o, PubMed, DM-Math (Saxton al., 2019) and StackExchange (Soboleva et al., 2023). In total, we had access to 4T high to medium quality English tokens for pretraining. For our 30B pre-training we sample 5.2T tokens from the RedPajamaV2 (Together Computer, 2023). We provide detailed description of our data filtering method in the Appendix G.
Pretraining data in the Arabic language is much more limited, thus we undertook large scale col- lection and curation of Arabic language data. This includes in-house crawled sources covering Web documents, news articles, books (literature. religion, law and culture, among others), Wikipedia (over 1M articles), and audio transcripts (books and news). To ensure high quality Web data, applied the following processing steps: (i) drop documents with language identification score be- low 95% (ii) drop short documents that less than 30 words, (iii) drop documents with duplicate
3

---

Published as a conference paper at ICLR 2025
URLs high ratio of spam and stop words, and (iv) drop duplicate documents using exact match- ing. We experimented with fuzzy matching but opted against using it, as it was too restrictive given the scarcity of Arabic data. During Arabic data processing, characters was performed. For language identification, we used fastText embeddings. For spam and stop words selection, we used Mubarak al. (2020). The majority of the Arabic data is from the web, which naturally contains Arabic varieties; however, no specific Dialectal Arabic (DA) datasets were intentionally included in the corpus, Knowledge of the distribution of Arabic varieties would require the data to be classified into DA, and since, to the best of our knowledge, DA identification a hard task that is still under research. According to Abdul- Maggee et al. (2024); Bouamor et al. (2019); Abdelali et al. (2021), classifying the data to DA is difficult. Nonetheless, for future work, we plan to evaluate ALLaM's s ability to understand and generate DA via ALDi (Keleg et al., 2023). For Audio transcription, we used the SauTech ASR system.
standardizing
normalization of
Additionally, we extended our Arabic data with translated English content using an in-house machine translation system. We translated
Language Ratio Evaluations
100
the following English datasets from Dolma:
Annuracy (96)
English (90
Wikipedia, books, C4 and peS2o, which also part of our English data. The hypothesis is that this will improve Arabic-English language alignment, leading to a better Arabic model. Figure 3 demonstrates the impact of Arabic translated datasets in the pretraining data mix- ture 3. While models trained without translated data exhibit lower training loss, those trained with translated data show more stable training, as evidenced by fewer spikes in gradient norms. Incorporating Arabic translated data in the pre- training dataset mitigates catastrophic forget- ting in English. In total, curate 540B Ara- bic tokens of which 270B are natural Arabic tokens and 270B are translated Arabic tokens.
335
Acuble KNEL
60
linglsh MMU
Selected (50% es)
-40
30
20
25
Training tokens (B)
Figure 4: We determine the optimal Arabic/English language mixture that balances between acquiring Ara- bic understanding while retaining English proficiency by conducting ablations over 6 Arabic/English ratios (trained up to 20B tokens). We found that 45/55 Arabic/English ratio achieves the best performance. as measured by English and translated Arabic MMLU.
For Arabic, we have a total of 540B tokens, and for English, we have a total of ~660 distinct tokens. Based on the sampling ratio from Table 1, we collected the training data from the corpus.
Data Mixture To build a performant model in both Arabic and English, we conducted experiments determine the optimal language mix. Figure 4 gives an overview of data- mixture experiments on our curated Grabic-Englishe corpus. We conducted the experiments with the same sampling ratio (Table 1) and data order. We observe best trend in performance with 45/55 Arabic/Figlish data mix.
Table 1: ALLaM'S pretraining data mixtures. The first four columns summarize the distribution of the continued pretraining mixed Ara- bic/English data. The English only pretraining from scratch mixture is shown in the last column. We upsample data to match the mixture rates when needed. Mixed English is a subset of English only.
Mixed Arabic & English
Domain
English Only
Arabic
English
Mixed
Natural
Translated
Web
31%
71%
65%
48%
71%
Books
9%
13%
12%
11%
3%
Wiki
0.70%
0.61%
0.3%
0.1%
News
14%
3%
Science
16%
22%
14%
6%
Code
39%
21 %
17%
Math
5%
2.5%
0.9%
Other
1.3%
0.39%
0.2%
2%
Table 1 shows the language and category mixing distri-
Lang Mix
55%
22.5%
22.5%
100%
100%
butions for English. Ara- bic natural, Arabic trans- lated and final mix. Following mainstream work (Touvron et al., 2023b; Chowdhery et al., 2022;
Tokens
660B
270B
270B
1.2T
4T
Rae al., 2021), web data constitutes the highest ratio with 71%, 65% and 48% of the Arabic nat-
$ OOur in-house translation system's 's BLEU score is around 25.23 on IWSLT test set.
*Tokens counted by our merged tokenizer.
4

---

Published as a conference paper at ICLR 2025
ural, Arabic translated and overall mix, respectively. We limited the contribution of English web data to 31,, as the Llama-2 base model was trained on a significant amount of English web data already, and we expected that increasing its ratio might degrade performance. We ensured that high quality sources, such as books, news, code etc. are well represented in our mixture.
2.2 CONTINUED PRETRAINING
Open source and open weight models present an attractive option to conduct pretraining experiments cheaply. However, they also present challenges, since most such models do not natively support Ara- bic or other languages. We develop a simple approach to enhance any language model with capa- bilities in new languages (i.e., language expansion). The approach relies on two steps: (i) tokenizer augmentation, and (ii) expanded vocabulary learning. We demonstrate that this approach leads to minimal degradation of capabilities in the original language.
Tokenizer Augmentation Existing open weight language models (e.g.. allam-2) tokenize Ara- bic (and other languages) poorly, often splitting words down to the character level or even relying on tyte-fallback mechanisms for tokenization. This results in: (i) inefficient training, the pretrain- ing corpus size is inflated, (ii) unoptimized inference, since the model must generate more tokens per word, and (iii) the effective context length is reduced, because it is based on a fixed number of tokens. To address these issues, we use corpus of text in the target language to train a tokenizer specialized in that language. We then merge the original tokenizer with the language-specific tok- enizer. Merging is accomplished by adding all tokens from the language-specific tokenizer that do not exist in the original tokenizer. As shown in Figure 2. this effectively reduces the fertility score in the target language of the merged tokenizer to the level of the language-speciffs tokenizer. We expanded Llama-2's s 32, 000 token vocabulary to 61, 568 tokens.
Expanded Vocabulary Learning Newly added tokens in the merged tokenizer have no associ- ated embedding representations in the pretrained language model's weights. To learn these repre- sentations, we experiment with two approaches: combined representations of tokens in the original tokenizer. Approach (ii) is accomplished by tok- enizing each token in the vocabulary of the new tokenizer using the original tokenizer. The associated embedding representations of this tokenization are then averaged and assigned as the vector repre- sentation of the new token. Since we work with tokenizers with byte- fyte-fallback, such a tokenization is guaranteed exist. Figure provides overview of our initialization method. Initializing the new embeddings from the combination of previously learned embeddings gives a significant boost to the learning of a new language.
(i) random initialization and (ii) initialization from
Experiment Details Starting from amaa-a- 22 pretrained model weights, continue pretraining the ALLaM-7B and ALLaM-13H models on 1.2T tokens, covering both English and Arabic lan- guages. For the A ALLaM-70B model, we only train up 600B tokens (using the same data mixture). In all of our continued pretraining experiments, we used the final learning rate of the pretrained lan- 10-5). We experimented with approaches to gradually increase and decay the learning rate with limited success, as such models typically exhibited catastrophic forgetting, in- dicated by significant drops in performance in the original language. We also considered optimizer state warm up, as open-weight models typically do not include the optimizer states, but found this had little effect on performance. Figure 6 provides an overview of adding dropout during continued pretraining. We observe that adding dropout helps the Arabic language, as it acts regularizer for the new distribution. However, a1a-a 2 was pretrained on 2T tokens without any dropout, and adding dropout negatively impacts the source language performance. Considering this trade-off, decided not to add dropout in the continued pretraining stage. For Vocabulary expanded model with LLaMa-2 (7B, 13B, 70B models), unlike recent trends (A12, 2024), we did not add any alignment data in this stage of training.
guage model (usually
2.3 PRETRAINING FROM SCRATCH
Following (Hoffmann et al., 2022; Touvron et al., 2023a), training a high-quality model from scratch requires substantial amount of tokens. Even when pretraining from random initialization, we find it beneficial to train with a high-resource language for trillions tokens (English) and then con- tinue training with mixture of Arabic and English tokens. On small scale experiments (with IB parameter models) we find that beginning training with two languages can sometimes degrade the performance in English or result in slow learning of both language distributions. From this, hy- pothesize that low-resource languages are diluted in the large volume of righuresourre language data when pretraining from scratch, even with upsampling and careful tuning.
5

---

Published as a conference paper at ICLR 2025
Effect of Embedding Initialization
Effect Dropout
Ancoruay (%)
Arabic MMLII (selid)
35
English MMLU (dashed)
10% drepout
40-
Arabic MMLU (solid)
dzopout.
Englishh MMLU (dashed)
30
Random embedding
Instialzeed ambodding
28
10
20
30
40
50
60
70
80
Thaining tokens
Traising toleens (B)
Figure 5: Effect of random initialization vs. em- bedding initialization during the start of con- tinued pretraining. We find that initializing the embeddings for new tokens from combinations of existing embeddings speeds up learning dra- matically.
Figure 6: Effect of dropout during the start of continued pretraining experiments. While in- troducing dropout marginally improve the second language acquisition, it negatively im- pacts the model's capabilities in the original language.
Training Recipe Our pretraining from scratch recipe consists of two steps: training on 4T English tokens followed by training on 1.2T mixed Arabic/English tokens. This retains the English capa- bilities of the model without catastrophic forgetting, effectively transferring knowledge from one language distribution to another. The only difference between pretraining from scratch and contin- ued pretraining from an existing model is that vocabulary expansion is not required.
We match hyperparameters and architecture for pretraining from scratch with Touvron et al. (2023a), including 4M tokens per batch and max LR 3 x 10 decayed to 3 10 with cosine schedule.
During the training of the 7B model (from scratch and vocab expanded models), we found that reducing the learning rate made it difficult to recover without hurting knowledge retrieval/retention capability of the model. To address this, we maintained constant learning rate during the initial English training phase for the 30B model. In the second stage, we applied cosine decay learning rate to fine-tune on a mix of English, Arabic, and SFT data. This approach allows for potential extended training with constant learning rate, followed by shorter but reasonably large-scale cross- lingual alignment using high-quality English, Arabic, and SFT data.
Data Mixture The last column of Table shows the domain mix of ALLaM-7B of the English only pretraining data. For ALLaM-34B, we made slight changes to the domain distribution and improved the quality of the Web data by deploying more aggressive filtering. As expected, web data represent the bulk of the mixture, followed by code and scientific articles.
While our 7B model followed the Arabic-English data mix depicted in Table 1, we made the follow- ing changes in ALLaM-34B. We introduced English and Arabic instruction data in our mix (12%). Additionally, we reduced Arabic translated content to 16% from 22.5%. Domain distribution re- mains close to the one shown in the table. Refer to Appendix H.4 for the training details.
For our 7B model, we adhered to the Arabic-English data mix as outlined in Table 1. However, in AML4M- we made several adjustments. Specifically, we introduced 12% English and Arabic instruction data into the mix and reduced the proportion of Arabic translated content from 22.5% to 16%. The domain distribution remains largely consistent with what is presented in the table. For further details on the training process, please refer to Appendix H.4.
ALIGNMENT
Building useful LLMS requires ensuring they are able to follow instructions while adhering to ethical standards and user expectations. This alignment process is especially crucial for models used in diverse linguistic and cultural contexts. In our setting, this means aligning models to the Arabic language and cultural context while also supporting English.
Supervised Fine-Tuning Tuning (SFT) (Section 3.1) refines a pretrained model using a carefully selected dataset relevant to specific tasks and domains. Preference training (Section 3.2), the other hand, aligns the model's outputs with human values and preferences by prioritizing responses that meet user expectations and ethical guidelines. These methods work together create reliable and ethically sound L.LMs for real-world use.

---

Published as a conference paper at ICLR 2025
354
Tritts
101
2008
100k
Dep
Shodl: Sep
(a) Prompt word count.
(b) Response word count.
(c) Number of conversation turns.
Figure 7:
Word count and turn distributions of SFT data. There are two main differences in our Arabic and English SFT datasets: shorter responses are more frequent in our Arabic SFT dataset, while our English SFT dataset contains more dialogues with more than 8 turns.
3.1 SUPERVISED TUNNNT
Data Our SFT data is curated from diverse array of sources. Given a piece of context from source, we utilize humans and/or generative models (Ding et al., 2023) to identify if the text can be considered suitable for supervised fine-tuning or if we can generate instructions to create an SFT example from the context. For English, we primarily use public web content as our main source, offering broad range of high-quality and especially diverse prompts. In contrast, our Arabic data comes from a combination of public and proprietary sources to ensure comprehensive coverage and relevance. To gather data from the source, we collect seed websites or data sources, which involves utilizing domain experts, prompt librarians, local institutes specializing in areas such as Arabic lan- guage, history, and politics, the use of commercially permissible licensed LLMs to generate data, and machine translation models to convert rich English SFT data into Arabic. Our datasets cover various domains and capabilities, ensuring the model's proficiency in handling tasks across edu- cation, history, Arabic linguistics,, politics, religion, computer science, and other fields. The entire Arabic/English collection is called Ultra- Inst iisstiinen intt, which is not human generated, but rather, human driven.
Table 2: Comparison of average word length and lexical diversity of prompts and responses.
Quality Filtering Unlike Zhou al. (2023); AI et al. (2024), we hypothesized that scal- ing SFT data can unlock diverse capabil- ity, as well as improve responsiveness to the prompts. Initially, we crawled the public web for SFT samples. The first version (v1) of includes 12M samples evenly split between English and Arabic, while the second version (v2), is reduced version with half the samples. Compared to v1, v2 underwent strict quality checks and human assessments of random subsamples. Our quality checks for v2 included (i) assessments based instruction/re- sponse word length, (ii) lexical and semantic diversity, exact and near-exact lexical deduplication, (ii)) removal of low quality machine-translated Arabic data from English sources, and (iv) ensuring diversity in questions and commands. For detailed metrics on instruction and response lengths and lexical diversity, see Table 2.
Quality Metric
Dltra
-nast inct vl Ulitaa Tnat inoe V2
Prompt
Response
Prompt
Response
.Ang A Word
146.94
97.19
60.81
136.47
Lesical divensty
76.34
75.25
85.29
69.53
Ultra-Instinel
Figures 7a and 7b show the distribution of the prompts and responses in v2, respectively. We fo- cused on maximizing the number of multi-turn conversations in our dataset. Figure 7c shows the distribution of conversation turns from Ultra-Insting inct.
Table 3: Comparative results of Ultra Instinct vl and v2, across various evalu- ation datasets.
To extrinsically evaluate the impact of higher quality SFT data, we trained two 13B models using our vI and SFT datasets. Even though v2 has half as many samples and v1, both ver- sions performed equally well on English and Arabic evaluation benchmarks, shown in Ta- ble 3. This reduction in data volume led to faster training times and reduced costs without compromising performance. It also clearly demonstrates the value of quality filtering for alignment.
MMLU
Hangg Q303a
(300))
DECEE Trans
513
<6.3
033
75
Lexiccl diversity is calculated by taking the ratio of the total number of unique words to the total number of words across all samples, excluding stop words.
7

---

Published as a conference paper at ICLR 2025
HelaSSwag
Arabic MMLU
English MMLU
84
02-
60
/
00
35
78-
76-
40
250
1000
250
500
750
1000
25.0
300
750
1000
Training takens (B)
70B
from scratch
Figure 8: Selected benchmark evaluated through ALLaM's training. Using WellaSwag a proxy for language understanding, performance smaller models degrades when introducing Arabic, while larger models (70B) have enough capacity to improve simultaneously in English and Arabic. Arabic language acquisition is rapid in all models, as indicated by Arabic MMLU,
3.2 PREFERENCE TRAINING
After SFT, models are able to converse multi-turn conversations. However, they are not fully aligned with human preferences. For example, our SFT models were terse and had limited guardrails. To circumvent these issues, we performed preference tuning with human verified samples via Direct Preference Optimization (DPO) (Rafailov al., 2024).
The inputs were sourced from early model testers and manually curated selection of prompts from various domains or attack vectors. These include ethical dilemmas, middle eastern culture, religions, illegal activities, human rights, locale awareness, and personality.
Data
Preference training necessitates both negative and positive outputs for each input. We relied on the testers' feedback to identify the positive outputs. the absence of positive outputs, we used a model to generate an output and manually verified that the output was aligned. While (Tunstall et al., 2023) utilized preference data from AI Feedback (AIF) at scale, we adopt more cautious approach in creating preference data. We generate a smaller volume of data, ensuring it is fully reviewed, edited, and verified by humans.
There are two approaches for generating negative outputs: (i) on on-policy: use the generations of the model we are tuning as negative outputs, and (ii) off-policy: use another similar model to generate the negative outputs. We did not verify that the negative outputs were worse than the positive. However, we ensured that the positive outputs were of the highest quality, such that they were almost always better than the negative outputs.
Khan et al. (2023) demonstrated that model outputs can vary significantly depending on the sam- pling mechanism used. Building this insight, generate additional samples for each instance by varying temperature and nucleus sampling techniques. These additional samples are utilized to produce rejected samples, ensuring that ALLaM provides more grounded responses and generalizes well across various sampling mechanisms.
In total, we collected 25,854 samples (triplets of (prompt, accepted, rejected)) in English and Arabic language. Using the technique mentioned above, we sample 10 different response from the model generate additional rejected responses for each sample. This results dataset of 245K samples (after filtering) for preference training.
4 EVALUATION
In this section, we describe the evaluation of our model and report the results of ALLaM 7B, 13B, 34B, and 70B models, as well as other relevant models, such as 4, Command R+ (Gomez, 2024), and f (Sengupta et al., 2023). Our evaluations encompass three main types: (i) automatic evaluations, (ii) LLMM-bsee evaluations, and (iii) human evaluations.
Limitations Recently, (Alzahrani et al., 2024) showed that multiple choice or cloze test based evaluation may not be robust. addition, ATT-Bench uses an LLM as a judge, and has likely leaked into training datasets. Unfortunately, human evaluation is time-consuming and requires well- well-trained human evaluators. In this work, we try to ensure robust evaluation and train a balanced assessment of the quantitative metrics and qualitative effectiveness of models in various applications and domains.
8

---

Published as a conference paper at ICLR 2025
Table Arabic benchmark results for instruction tuned models. Follow Table for detailed results.
araSwng
ACVA
MMLU
(ar)
ETBC
TrurhfilQQA
Kato et al. (2024)
Huang et (2023)
10-shot
S-shot
O-shot
O-shot
S-shot
O-sbot
O-shox
S-shot
ALLaM-Instruet
78
49.28
80.33
66.9
49.6
52.7
62.95
36.4
36.5
AceGPT-Chat
7B
43.4
59.35
37,9
Llana 2-Chat
78
24.44
52.46
33.33
26.45
25.33
26.69
29.9
21.5
Mistral- Mistral-Instruct-OO.. Instruct-vO.3
7B
30.59
60.7
44.3
34,06
31.1
34.41
30.3
26.0
I-Instract
SB
33.99
75.21
34.0
ALLaM- Instruct
13B
54.77
78,59
68.11
$1.03
$4,93
65.59
37.5
46.8
Llama 2-Chat
138
25.75
60.
35,84
28,73
22.91
30.44
31.4
Jais-Chat
13B
77.12
70.68
51.8
41.43
46.93
48.68
31.6
25.3
ALLaM-Instruct
34B
59.74
81,00
75,98
60.2
$8.66
74.26
35.49
46.5
Chat-33
88.37
62.37
ALLaM Instruct
70B
57.91
79.01
75.92
62.23
$8.47
78.38
38.4
56.8
Iamra 2-Chat
708
30.72
59.49
40.77
32,86
28.68
30.6
32.3
25.5
Llana 3-Instract
708
45.75
80.26
36.27
60.11
58.47
71.41
37.7
59.70
4.1 AUTOMATIC EVALUATIONS
The automatic evaluations cover Arabic and English benchmarks grouped into many categories. Detailed description of the evaluation dataset and benchmark can be found in Appendix 1.1.
While serving as good test bench, observing the dynamics of automatic evaluations during training also interesting. Figure 8 shows the behavior of selected benchmarks during mixed Arabic/English pretraining while scaling up model size. In particular, we observe that smaller models tradeoff be- tween capability in the and original languages. However, larger models simultaneously improve in both languages.
Another observation from automatic evaluations is that some evaluations provide more signal for training decisions than others, e.g., Hellaswag smoothly improves during training while improve- ments in GSMSk occur in discontinuous jumps. Other benchmarks show improvement until 1.5T tokens have been seen (i.c., grokking) making them unreliable for early training decisions. Tables and give an overview of the performance of ALLaM instruct models.
In Arabic benchmarks, we can see that ALLaM- 70B scores are the best in five (MMLU Ara- bic (natural and translated), Exams, ETEC, araTruthfulQA) out of the eight benchmark sets. On English, ALLaM is the second-best model in the majority of cases, following Llama 3 r-ct. We highlight the ex- cellent performance of ALLaM on benchmarks released after training completed (MMLU- Pro, MixEval) and benchmarks the training team did not have access (ETEC), since they provide a clean evaluation signal. Table 11 and Table 12 in the appendix contain more detailed evaluation results for Arabic and English.
Table 5: MT-Beech scores for Arabic and English. Each score is an average over samples of the score ranging from 0 to 10 returned by the judge 4P)
English
Avg.
Tum 1
Turn
Avg.
Turn
Turn
AscEPT 13B-chal
5.44
6.76
4.12
6.33
7.01
5.61
ALLaM 13B Instrcct
7.34
7.67
7.01
7.57
7.9
7.23
AL.LaM Instuet
7.44
13B Chat
Jais 30B Chat vi
3.89
4.13
3.64
3.54
4.13
2.95
lois 30B Chot v3
5.85
6.25
5.47
6.28
6.78
5.78
Cohere Command R
7.41
7.63
Command
DBRX Instruct GPT 3.5 Turbu
7.16
7.33
6.98
7.83
8.19
7.46
7.55
7.79
7.31
8.12
8.39
7.84
4.2 LLM-BASED EVALUATIONS
MT-Bench (Zheng et al., 2024) consists of 80 multi-turn questions to evaluate models' capabilities on complex instruction following. inslluction-followingg In addition to the English version, MT-Bench Arabic was created using GPT-4 to translate the original dataset and human annotators to review and align the prompts Arabic culture. GPT-4 serves as the LLM judge, scoring responses as recommended in (Zheng et al.. 2024). Model performance is compared turn by turn, with results shown in Table 5, where ALLaM-700 achieves the best Arabic performance.
4.3 HUMAN EVALUATION
We developed an Arabic multi-turn dataset spanning seven domains: linguistics, history, health, pol- itics, coding, entertainment, and ethics, with each domain featuring ten two-turn questions. Human evaluators compared the responses from two models and were asked to choose the winning response with the following instructions:

---

Published as a conference paper at ICLR 2025
Table English benchmark results for instruction tuned models. (Follow Table 12 for detailed results.)
AGiisval
MMLU
MMLU-Prs
Ehiis
TrubfulQA
ARC Challenge
HellaSwag
MixFaall
Average
Hurd
Standard
D-shot
(base/it)
540-shor (bave/))
ALI ai--ssmnett
7B
47.09
58.31
27.78
69.8
42.11
51.45
75.2
28.9
67.6
AceGPT-CChr
26.33
44.53
38
49.54
Lhma 2-Chot
Mismal-lostm---.333
7R
73.59
82.88
36.2
70.0
Llnma 3-Instmet
8B
44.35
63.82
41.32
68.07
51.72
56.83
75.81
45.6
75.0
ALLaM ALsuM--nntzuet
13B
48.42
61.8
34.05
76.47
$7.69
$5.89
81.14
37.2
72,8
Lhaaa 2-Chor
15B
37.73
53.3
27.19
70.52
43,95
30.17
79,66
64.92
ALLaM-Instruet
Jass Chat-33
30B
36.78
57.57
26.45
68.03
42.34
51.02
78.91
708
65.67
75.43
48.61
76 16
58.78
59.56
84.97
$1.60
83.5
I Llama 2-Char
708
46.0
61.15
35.16
68.5
52:77
54.27
82.14
38.0
74.6
Llama 3-Enstreet
70B
63.78
78.38
77.09
61.79
64.33
82.49
55,90
54,00
ALLaM 1LB
Eva loution
Elo Scores Huraa Evaluation
11.00
Commanci
45
1188
2034
4%.4%
Elo Rating (Oustoms
Rating (Defuul))
1050
CommandR+
34.25
16.0%
1000
ALLAN-330 GF74
12:39
20.76
ALLaV-135
d3
30.0%
25
37.0%
23.9%
900
GPTS
ALLaM-136
CommandR++
Jale platss3
Commasdift
los
both ted
Figure 9: Pairwise win rates judged by human evaluators. ALLaM-13B wins against many much larger models.
Figure 10: ELO scores from human eval- uator preferences. ALLaM is tied with Command-R+ and lags only behind GPT-4. Choose response as the winner if is the best, tie if both responses are equally good, and both-bad if both responses are not good.
A response considered good it is coherent, grammatically correct, and a reasonable response the question previous turn in the conversation.
Good responses should be in the correct language (the response should be in the same language as the previous turn, unless another language was requested).
Good responses should not contain toxicity, hate speech, or bias.
Each pair of responses was inspected by three evaluators, and the winner determined by majority voting. In case of a tie, fourth evaluator was used to break the tie. Figure 9 presents the human eval- uation results of the pair-wise comparisons of these models: ALLaM- 13b, Jai 3-30--30ttt chat-v3, Command- R R-plus, and Command- Beieaand---yye R-v01. ALLaM- 13b's 's win rate was always higher than its loss rate compared with other models.
Finally, we gather votes from the human evaluators and calculated ELO scores for each model. ELO scoring had two configurations, the default scoring rewards the model for good responses with point, tied responses (good and both-bad) with 0.5 points, and penalizes for bad responses with point. The custom configuration penalizes the model with the bad response and both models both provided bad responses with 0 point. Figure 10 shows the ELO scores based on the human evaluations. From the figure, GPT-4 achieved the highest score, followed by ALLaM-1 3b with the second- highest score, outperforming (or matching) larger models such as CommandR+.
5 CONCLUSION
The ALLaM model series marks significant advancement in Arabic Language Technologies, achieving state-of-the-art performance across various Arabic benchmarks while maintaining or en- hancing English performance. Through careful training that emphasizes language alignment and transferability, models demonstrate effective second-language acquisition without catastrophic forgetting. The strategic use of translated data, knowledge encoding, and alignment with human preferences have been erucial in this success.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Ahmed Abdelali, Hamdy Mubarak, Younes Samih, Sabit Hassan, and Kareem Darwish. QADI: Arabic dialect identification in the wild. In Nizar Habash, Houda Bouamor, Hazem Hajj. Walid Magdy, Wajdi Zaghouani, Fethi Bougares, Nadi Tomeh, Ibrahim Abu Farha, and Samia Touileb (eds.). Proceedings of the Sixth Arabic Natural Language Processing Workshop, PP. 1-10, Kyiv, Ukraine (Virtual), April 2021. Association for Computational Linguistics. URL ptttps://aclattho...e org/ 2 2221..1222 wanlp-1.1.
Muhammad Abdul-Mageed, Abdelrahim Elmadany, Alcides Inciarte, Md Tawkat Islam Khondaker, et al. Jasmine: Arabic gpt models for few-shot learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 16721-16744, 2023.
Muhammad Abdul-Mageed, Amr Keleg, AbdelRahim Elmadany. Chiyu Zhang, Injy Hamed, Walid Magdy, Houda Bouamor, and Nizar Habash. Nadi 2024: The fifth nuanced arabic dialect identi- fication shared task, 2024. URL https: // /awxiiiivv//../////v//.../a/aaia org/ abs 000s//4000 240024070000 04910.
01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01.ai, 2024.
A12. Olmo 1.7-7b: 24 point improvement on mmlu, 2024. URL https://bbloo allenai.. -ww.eanttt---mmm------a mm1--mmmu-9000 -0010-00000001-----23337225
Asaad Alghamdi, Xinyu Duan, Wei Jiang, Zhenhai Wang, Yimeng Wu, Qingrong Xia, Zhefeng Wang. Yi Zheng, Mehdi Rezagholizadeh, Baoxing Huai, et al. Aramus: Pushing the limits of data and model scale for arabic natural language processing. arXiv preprint 23XXx:2306006800 2023.
Reem Alghamdi, Zhenwen Liang, and Xiangliang Zhang. ArMATH: a dataset for solving Arabic math word problems. In Nicoletta Calzolari. Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerek, Sara Goggi. Hitoshi Isahara, Bente Maegaard, Joseph Mari- ani, Hélène Mazo, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 351-362, Marseille, France, June 2022. European Lan- guage Resources Association. URL https: / aaaalcaaaa cllanthllogyyy wwrgorgg///// 2022. 1rc-11.3.
Norah Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Al- subaie, Yusef Almushaykeh, Faisal Mirza, Nouf Alotaibi, Nora Altwairesh, Areeb Alowisheq, M Saiful Bari, and Haidar Khan. When benchmarks targets: Revealing the sensitivity of large language model leaderboards, 2024.
Amazon. Sustainability in the cloud., 2021. URL https::/ aboutamazon. www.coo.e/ee... enviiiiiiiii www.irrrtoeeeeeeeeeeeeeeeeeeee httettheee.c...tee
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Lau- rent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Brad- bury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christo- pher A. Choquerte-Choo, Aakanksha Chowdhery, Clément Crepy. Shachi Dave, Mostafa De- hghani. Sunipa Dev. Jacob Devlin, Mark Díaz, Nan Du Ethan Dyer, Vlad Feinberg, Fangxi- aoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann. Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Ke- nealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li. Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Freder- ick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Mous- salem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Mar- tin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone,
11

---

Published as a conference paper at ICLR 2025
Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahallii, Xuezhi Wang. Pidong Wang. Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm technical report, 2023.
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2022. URL htwps:ttp//// http:///wwwwwtttt cdn. anthropics c. com/ de 9b0 hbtpc///aaccbbaccaaaaabb0 185 081768887777111 /Model]
Mard_Claude__a 3. pdff.
Wissam Antoun, Fady Baly, and Hazem Hajj. Aragpt2: Pre-trained transformer for arabic language generation. arXiv preprint arXiv:2012.15520 2020.
Mikel Artetxe, Gorka Labaka, and Eneko Agirr,, Generalizing and improving bilingual word em- bedding mappings with a multi-step framework of linear transformations. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pp. 5012-5019, 2018.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of mono- lingual representations, 2019.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences, 2023. URL https: //tpxvaa.vvvvv/aa///aa org/ abs/2310, 12036.
Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang. Jason Alan Fries, Maged S. Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Xiangru Tang, Mike Tian-Jian Jiang, and Alexander M. Rush. Promptsource: An integrated development environment and repository for natural language prompts, 2022.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li. Eli Tran- Johnson, Ethan Perez, Jamie KeTr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Con- erly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022. URL https: /tt//aaiii.///////////e///l//ata org/ abs 2212. 08073.
M Saiful Bari, Shafiq Joty, and Prathyusha Jwalapuram. Zero-resource cross-lingual named entity recognition. In Proceedings of the aaai conference on artificial intelligence, volume 34, pp. 7415- 7423, 2020.
M Saiful Bari, Tasnim Mohiuddin, and Shafiq Joty. Uxla: A robust unsupervised data augmentation framework for cross-lingual nip. In Proceedings The Joint Conference of the 59th Annual Meet- ing of the Association for Computational Linguistics and the I Ith International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021), Online, 2021. Association for Computa- tional Linguistics.
Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra. A framework for the evaluation of code generation models. https: /ttp::///tthub..... com/ 2022.
bigcode-project. www.ooe--ealaaaainnn--oooee-ee--
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilis- tic language model. J. Mach. Learn Res., 3:1137-1155, 2003. URL https://apt.
semanticschol larl.rr.ao haagg//oruuaaae ID: 221275765.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty- Fourth AAAI Conference on Artificial Intelligence, 2020.
12

---

Published as a conference paper at ICLR 2025
Houda Bouamor, Sabit Hassan, and Nizar Habash. The MADAR shared task Arabic fine- grained dialect identification. In Wassim El-Hajj. Lamia Hadrich Belguith, Fethi Bougares, Walid Magdy, Imed Zitoumi. Nadi Tomeh. Mahmoud El-Haj, and Wajdi Zaghouani (eds.), Proceed- ings of the Fourth Arabic Natural Language Processing Workshop, pp. 199-207, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10. (0.18653/v1//199440 URL ht https: httpp:://aaaaath...eo
org/olgg///.oooooooooo..o WW9 4622.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Liwwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020a.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020b. URL https: /tas:///axxiii org/ alt//bbs////// 14165.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski. Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph. Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern Dou- glas Eck, Jeff Dean, Slay Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. CoRR, Abs/2204.0231., 2022. doi: 10.485550arxx..22440031 URL https: //doi. org/10. 48550/ 2204 02311.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghami, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros. Marie Pel- lat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling instruction-finetune language models, 2022. URL https: /ttpx/xxivv/////////////////.avl org/ /aab//22100 11416.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL-HLT 2019, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/ 10s./883.,5457, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reñcchrr Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021.
13

---

Published as a conference paper at ICLR 2025
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Confer- Machine Learning, ICML '08, pp. 160-167, New York, NY, USA, 2008. Associa- tion for Computing Machinery. ISBN 978 9781605582054, doi: 10.1 14114413391156 1390177. URL. htt.ps://doi http://oo..orgg/////
1390177.
Alexis Conneau, Guillaume Lample, Marc' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. CoRR, abs/1710.04087, 2017. URL /ttt:://arxiin
org/ bxg/abs// 1110..10.... 04087.
Alexis Conneau, Kartikay Khandelwal. Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zentemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. arXiv preprint urXiv:1911.000 2019.
Yiming Cui, Ziqing Yang, and Xin Yao. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint wrXiv:204.08117, 2023.
Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In Inter- national Conference on Learning Representations (ICLR), 2024.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Process- ing Systems (NeurIPS), 2022.
Jacob Devlin, Ming-Wir Weng-Weii Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL- HLT 2019, Minneapolis. MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171-4186. Association for Computational Linguistics, 2019. doi: 10. 10.18653/v1/119-4225 URL https: / /oi. orgg D.D...86533000 11853//1/n19-1111
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.1 14233, 2023.
Jesse Dodge, Taylor Prewitt, Remi Tachet Des Combes, Erika Odmark, Roy Schwartz, Emma Strubell, Alexandra Sasha Luccioni, Noah A. Smith, Nicole DeCario, and Will Buchanan. Mca- suring the carbon intensity of cloud instances, 2022.
Moussa Kamal Eddine, Nadi Tomeh, Nizar Habash, Joseph Le Roux, and Michalis Vazirgiannis. Arabart: pretrained arabic sequence-to-sequene model for abstractive summarization. arXiv preprint arXiv:2203. 10945, 2022.
AbdelRahim Elmadany, Muhammad Abdul-Mageed, et al. Arat5: Text-to-text transformers for ara- bic language generation. In Proceedings of the 60th annual meeting of the association for com- putational linguistics (Volume Long papers), pp.
2022.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model alignment prospect theoretic optimization, 2024. URL htsttps:/aaxivvorrtt 2402. 01306.
Kto:
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027, 2021. URL https:
/ //arxiv.org/ableeb.e 22 00027.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https: : zenodo. org/ recordsc 02..6.36.
14

---

Published as a conference paper at ICLR 2025
Aidan Gomez. Introducing Command R+: A Scalable LLM Built for Business, April 2024. URL https: /oceeee.///eeeeee/eeeeoo http:////cccmmm------------ooooo
Google. Carbon free energy for google cloud regions, 2021. URL com/ www.staaablityy/eene..aaaaaalaae carbon..
Google. Gemini: A family of highly capable multimodal models, 2024.
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning word vectors for 157 languages, In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018), 2018.
Momehil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov. Exams: A multi-subject high school examinations dataset for cross-lingual and multilin- gual question answering. arXiv preprint
2020.
Dan Hendrycks and Mantas Mazeika. X-risk analysis for ai research, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 2Xiv:2009.033000 2020.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Stein- hardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021b.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9:1735- 80, 12 1997. doi: 10. 10.1162201 1..1166//ccc.1999
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 2022. URL
https: org/ abs 2203//2203..m 155556.
Jiwoo Hong, Noah Lee, and James Thorne Orpo: Monolithic preference optimization without ref- erence model, 2024. URL https: http///axvvvvvv////v///////aaav/ wwtp///////b/b//////eoee 07691.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 328-33--33 339, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18. 1031. wogp///-///111111
URL https: / /at/aacaannthoooo
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. Unicoder: universal language encoder by pre-training with multiple tross-lingual tasks. In Ken- taro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence Natural Language Processing (EMNLP-IJCNLP), pp. 2485-2494, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1222 URL attps://aclanthor
org 1aag/D19-220 1252.
Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingiie Song, Zhihong Chen, Ab- dulmohsen Alharthi, Bang An, Ziche Liu, et al. Acegpt, localizing large language models in arabic. arXiv preprint arXiv:2309. 12055, 2023.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics, Vancouver, Canada, July 2017. Association for Computational Linguistics.
15

---

Published as a conference paper at ICLR 2025
Karthikeyan K, Zihan Wang. Stephen Mayhew, and Dan Roth. Cross-lingual ability of multilingual [bert]: An empirical study. International Conference on Learning Representations, 2020. URL forummidddteeeyyaeeea. 1et.errrrre.......o..oooaeoioeoe
https:: wttp://eeppeeeeeeeeemmeeeeeeeee. .eet
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Amr Keleg, Sharon Goldwater, and Walid Magdy, ALDi: Quantifying the Arabic level of dialectness of text. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10597-1061) Singapore, December 2023. Association for Computational Linguisties. doi: 1.. 10.186533v1/2023.0 .em.lp- main.655. URL https: http///clannttttt
/rg/2023330200 2023....22 emnlp in.. 655.
Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. xcodeeval: A large scale multilingual multitask benchmark for code understand- ing, generation, translation and retrieval, 2023.
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The stack: tb of permissively licensed source code, 2022.
Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, et al. Arabicmmlu: Assessing massive multitask language understanding in arabic. arXiv preprint arXiv:2402. 12840, 2024.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Al- berti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wes Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
Imad Lakim, Ebtesam Almazrouei, Ibrahim Abualhaol, Merouane Debbah, and Julien Launay. A holistic assessment of the carbon footprint of noor, very large arabic language model. In Pro- ceedings of BigScience Episode# 5-Workshop on Challenges & Perspectives in Creating Large Language Models, pp. 84-94, 2022.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Xiangji Huang. systematic study and comprehensive evaluation of chatgpt on benchmark datasets, 2023.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman -Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with lan- guage models, 2022.
Peigin Lin, Shaoxiong Ji. Jörg Tiedemann, André FT Martins, and Hinrich Schütze. Mala-500: Massive language adaptation of large language models. arXiv preprint arXiv:2401.13303 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human
falsehoods. arXiv preprint arXiv:2109.07955. 2021.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic hu- man falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Pupers), pp. 3214-3252, Dublin, Ireland, May 2022. As- URL https:
sociation for Computational Linguistics. doi: 10. 10011665/11120222aalloooo / //aclanthology
acn- 229.
Alexandra Sasha Luccioni and Alex Hermandez-Grin Garcii.. Counting carbon: A survey of factors influ- encing the emissions of machine learning, 2023.
16

---

Published as a conference paper at ICLR 2025
Gary Marcus. Is chatgpt really "code red" for google search?, 2022. garymarous substack. cww/c//m/////// http:///aaa-ccttttttttaaal------
URL https:// red- for.
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Con- textualized word vectors. In Advances Neural Information Processing Systems, pp. 6297-6308, 2017.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward, 2024. URL https: / /aar/vivvv////////av////a///aaaa www.zii.oog/aaao .205.1105..1 14734.
Meta. Introducing meta Ilama 3: The most capable openly available Ilm to date, 2024. URL https: / //ai. meta. com hblp//llg///.... /meta-/eee momea-llamma 11ama 3/.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. In C. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26, pp. 1199-3119. Curran Associates, Inc., 2013.
Mistral. Au large, 2024. URL https: / ai htt////w//sssssee /tt.s//iiissaralaarree.
Tasnim Mohiuddin, M Saiful Bari, and Shafiq Joty. Lnmap: Departures from isomorphic assumption in bilingual lexicon induction through non-linear mapping latent space, 2020. URL https: / faraiiv org/ abs 2004. 13889.
Hamdy Mubarak, Ahmed Abdelali, Sabit Hassan, and Kareem Darwish. Spam Detection Arabic Twitter, pp. 237-251. 10 2020. ISBN 978-3-030-6097440. doi: 11100/978-3-030-609777
Niklas Muennighoff, Thomas Wang. Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong. Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafcai, Albert Webson, Ed- ward Raff, and Colin Raffel. Crosslingual generalization through multitask finetuning, 2022.
El Moatez Billah Nagoudi. Muhammad Abdul-Mageed, AbdelRahim Elmadany, Alcides Alcoba Inciarte, and Md Tawkat Islam Khondaker. Jasmine: Arabic gpt models for few-shot learning. arXiv preprint 22 12. 10755, 2022.
Tarek Naous, Michael J. Ryan, Alan Ritter, and Wei Xu. Having beer after prayer? measuring cultural bias large language models, 2024.
Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. Seallms large language models for southeast asia, 2023.
Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. Mixeval: Deriving wisdom of the crowd from Ilm benchmark mixtures, 2024.
OpenAl. Chatgpt: Optimizing language models for dialogue, 2022. URL https: //oppaii. com/blog/ chapog//ccaatp/.....
Aitor Ormazabal, Che Zheng, Cyprien de Masson d' Autume, Dani Yogatama, Deyu Fu, Donovan Ong. Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, Kaloyan Aleksiev, Lei Li, Matthew Henderson, Max Bain, Mikel Artetxe, Nishant Relan, Piotr Padlewski, Qi Liu, Ren Chen, Samuel Phua, Yazheng Yang, Yi Tay, Yuqi Wang, Zhongkai Zhu, and Zhihui Xie. Reka core, flash, and edge: A series of powerful multimodal language models, 2024.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL https: /tt///axxiv .rgg abs 2203. 02155.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization, 2024. URL https: /attps://aee //txx//avvvvv///////a//ea org. awt///b///// 24003. 19159.
17

---

Published as a conference paper at ICLR 2025
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.0/ / 16, 2023. URL https:: arxii. v. org/ abs///////// 2306. 01166.
Guilherme Penedo, Hynek Kydlīček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweh datasets: Decanting the web for the finest text data scale, 2024.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP 14, pp. 1532-1543, Doha, Qatar, October 2014. URL http:// www. acl.aalweet wwt.:aaw.wb...rg/atttooo.oo
Matthew F. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL, 2018a.
Matthew F. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations, 2018b.
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT? In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4996-5001, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 18655//11/1991444 URL https: / //aclanthology wtt/p1s--4:0
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. OpenAl blog, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song. John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hen- nigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Suther- land, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazari- dou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux. Mantas Pajarskas, Toby Pohlen, Zhitao Gong. Daniel Toyama, Cyprien de Mas- son d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew J. Johnson, Blake A. Hecht- man, Laura Weidinger, lason Gabriel, William Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals. Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Has- sabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs/2 1146, 2021. URL https: waaaaiiivvoor abs/211 2bs22112.. 19..11444 11446.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2023. URL sattps://arxae http://wwww....g/bb///.....
Rafael Rafailov, Arehhi Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to- text transformer. J. Mach. Learn. Res., 21:140:1-140:67, 2020. URL http:/ jjmr.. org/ papers/v21/20-000 html.
18

---

Published as a conference paper at ICLR 2025
Szymon Ruciñski. Efficient language adaptive pre-training: Extending state-of-the-aan large lan- guage models for polish. arXiv preprint arXiv:2402.09759, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. arXiv preprint urXiv::107, 10641, 2019.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, An- toine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim. Gunjan Chh- ablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Mat- teo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization, 2021. URL https: //arxiv. abtp//as./11......./.../aa 0220000777....0...
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical rea- soning abilities of neural models, 2019.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Ro- man Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexan- der M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff. Albert Villanova del Moral, Olatunji Ruwase, Rachel Baw- den, Stas Bekman, Angelina SoMMilaan-Majo Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christo- pher Klamm, Colin Leong, Daniel van Strien, David Ifcoluwa Adelani, Dragomir Radev, Ed- uardo González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhat- tacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Ma- soud, María Grandury, Mario Saško, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Vil- legas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebas- tian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lep- eroq, Vrinda Prabhu, 7aid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut Emre Taşar, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lec, Abheesht Sharma, An- drea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szczechla, Gunjan Chh- ablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim. Tali Bers, Thibault Fevry, Trishala Neeraj. Urmish Thakker, Vikas Raunak, Xiangru Tang. Zheng- Xin Yong. Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung.Jesuung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper. Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anas- tasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurélie Névéol, Charles Lovering, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, Jan- Jan-Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal,
Beltagy, Huu Nguyen, Lucile Saulnier, Samson
19

---

Published as a conference paper at ICLR 2025
Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shav- rina, Thomas Scialom, Tian Yun. Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdeněk Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj. Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh Haji- Hosseini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Daniel McDuff, Danish Contractor, David Lansky, Davis David, Douwe Kiela, Duong A. Nguyen, Ed- ward Tan, Emi Baylor, Ezinwanne Ozoani, Fatima Mirza, Frankline Ononiwu, Habib Rezane- jad, Hessie Jones, Indrani Bhattacharya, Irene Solaiman, Irina Sedenko, Isar Nejadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim Elbadri, Mar- got Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Rajani, Nour Elkott, Nour Fahmy, Olanre- waju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy. Sylvain Viguier, Thanh Le. Tobi Oyebade, Trieu Le. Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap. Alfredo Palasciano, Alison Callahan, Anima Shukla, An- tonio Sirrada-Escclaade Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñinn Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec. Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madelcine Hahn de Bykhovetz, Maiko Takeuchi, Mare Pamies. Maria A Castillo, Marianna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf, Mina Mihaljeic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee Sang-aroonsin Srishti Kumar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gi- gant, Tomoya Kainuma, Wojeiech Kusa, Yanis Labrak, Yash Shailesh Bujaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. Bloom: A 176b-parameter open-access multilingual language model, 2023. URL https::///// / /awxivvvvvv..////////a.////aaava org/ abs /211/ 05100.
Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, et al. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint XrXiv:2308. 16149, 2023.
Claude E Shannon. Prediction and entropy of printed english. Bell system technical journal, 30(1): 50-64, 1951.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using deepspeed and megatron to train megatron-turing nlg 530b, large-scale generative language model, 2022. URL https: 22....199..........
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama, June 2023. URL hattps:/hhuee https///hggggggaggggeee co/ ///aaaatsss///eaaa wwtpbrereebaaeeeeeeeeee Sww.m/mmmm.... jama- 627B.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani. Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. CoRR, abs/2402.00159, 2024. doi: 0044550/ARXXI..40200000 URL ht attps: / /oooi / 10. 4 85500 www.iv..vvvv. 2402. 00159.
20

---

Published as a conference paper at ICLR 2025
Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, Chandana Satya Prakash, Mukund Sridhar, Fabian Triefenbach, Apurv Verma, Gokhan Tur, and Prem Natarajan. Alexatm 20b: Few-shot learning using a large-scale multilingual seq2seq model, 2022. URL https: //arxiv. wrg. abs/ 2208 01448.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize from human feedback. CoRR, abs/2009.01325, 2020. URL https://arxiv org/ abs/ /009/ 200.//1105........
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. 2019.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.0926.: 2022.
Merrill Swain and Sharon Lapkin. Problems in Output and the Cognitive Processes They Generate: A Step Towards Second Language Learning. Applied Linguistics, 16(3):371-391, 09 1995. ISSN 0142-6001. doi: 10. 10.1093/applin/11..337 URL 361..371.
THUDM. Codegeex: A multilingual code generation model. httpp:://gthtb..cmm////
CodeGeeX, 2022.
Together Computer. Redpajama: an open dataset for training large language models, October 2023. URL https: wwt../cweeeeeeeeeeeeeeeeeeoooooe
/aaeeerreeeeee RRedP a jama -ata.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet. Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint (tlo:2330..19444 2023.
21

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2017.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, An- jana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap. Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, Mehrad Moradshahi, Mihir Par- mar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddhartha Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi, and Daniel Khashabi. Superr-naturalinsttrciioss Generalization via declarative instructions on 1600+ nlp tasks, 2022.
org.orgg//ooooo. abs http//bbs/22...22 22220000775
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv: 2024.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners, 2021. URL wweeiii.orrg//ee alss//s//2// 22009, 01652.
ficts:://ariis
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, 7ac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel. Ethical and social risks of harm from language models, 2021.
Orion Weller, Kevin Seppi, and Matt Gardner. When to use multi-task learning vs intermediate fine-tuning for pre-trained encoder transfer learning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 272-282, Dublin, Ireland, May URL
2022. Association for Computational Linguistics. doi: https: http://caaaatttthacceeee org/2022.lccc..o.oooo.2
htt5://////2222allsshrrrr 2022. acl-short. 30.
Guillaume Wenzek, Marie-Anne Lachaux. Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data, 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. As- sociation for Computational Linguistics. URL https:////// /twww//www///...o///// ac wew........... org/ hotooolyy/ 2020. emn 1p demas. 6.
Shijie Wu and Mark Dredze. Beto, bentz, becas: The surprising cross-linguse effectiveness of BERT. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 833-844, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 18.136 1118655//1/11111010/ 1/D19-1077. UtL www.. ac 1 web. org/ anthol ogy /D1 9-1077.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5; massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North Americun Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v 1/502///2222mmmm www.cl-mainnn.... / //aclanthology org/2021 naacl -ma-m.aain..........
https:::
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang,
22

---

Published as a conference paper at ICLR 2025
Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang. Peng Wang, Ru Peng. Rui Men, Ruifze Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xi- aohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Zeyu Cui, Zhemru Zhang, and Zhihao Fan. Qwen2 technical report, 2024. URL https: /arpiii///avvv.../aaa org/ //abs//////// / 2 077 10671.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32. pp. 5753-5763. Curran Associates, Inc., 2019.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yeji Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, 2019.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023.
Xuhui Zhou, Zhe Su, Tiwalayo Eisape, Hyunwoo Kim, and Maarten Sap. Is this the real life? is this just fantasy? the misleading success of simulating social interactions with llms, 2024.
23

---

Contents
1
Introduction
2
Pretraining
2.1
Pretraining Data
2.2
Continued Pretraining
2.3
Pretraining From Scratch
3
Alignment
3.1
Supervised Fine- Fine-Tuning
3.2
Preference Training
4
Evaluation
4.1
Automatic Evaluations
4.2
LLM-Based Evaluations
4.3
Human Evaluation
5
Conclusion
10
A Acknowledgment
25
Frequently Asked Question
25
Related Work
28
C.I Language Modeling and Cross- lingull Representations
28
C.2 Multitask Learning And Alignment
29
C,3 Language Models for Arabic
29
Limitations
29
Ethical Statement
30
F Risk Statement
30
G Processing Of English data for 30B parameter model
30
Training Details
32
H.I Compute and Training Infrastructure
32
H.2 Pretraining Details
32
H.3 Alignment Training Details
33
H.4 DPO PPO
33
I
Evaluation
33
LI
Details of the evaluation datasets
33
I.2
Detailed Arabic Evaluation
34
1.3 Detailed English Evaluation
1.4 Examples
34
24

---

Published as a conference paper at ICLR 2025
A ACKNOWLEDGMENT
This release is funded by Saudi Data & Al Authority. We thank Mishari Almishari and Yaser Alonaizan for leading the ALLaM project. We list below the different teams that supported the development of ALLaM. AI Team who joined after the first release of ALLaM models:
Alhanoof Althnian, Iman Albilali, Jubair Sheikh, Ibrahim Mammu, Ghadah Alsaif, Abdulhameed Alothaimeen, Hadeel Alnegheimish, Emad Alghamdi, Khalid Almubarak, Atheer Al-Barqi, Ba- tool AI-Otaibi, Muhammad Al-Hajji, Abdulraouf Al-Maasoumi, Abdulmajeed Alrowithi, Faisal Qarah, Sakhar Alkhereyf, Arwa Omayrah, Mohammed Al-Salcem, Amal Al-Shammari, Maryam Saif, Raeda Al-Marzoog. Lamyaa Alghatani, Anas Al-Manie, Ghassan Al- Ward, Shahad Al-Zaidi, Batool Al Al-Ghoraibi, Alanoud Al-Shammari, Saad AIrazouq.
Data Acquisition, Management and Engineering:
Ali Alotaibi, Abdulrahman Alsaudi, Abdulaziz Asheyban, Areej Alokaili, Norah Alangari, Hamad Alnamazi, Fatima Alsalman, Salch Alrusayyis, Moneerah Alballa, Omar AlGhamdi, Rancem Alqah- tani, Amirah Albinhar, Nour Al-Aldahan, Khalid Alharbi, Hanouf Aljlayl, Fatimah Alzubaidi, Lama Aldossary, Rania Alzahrani, Shoug Alkhalaf.
Development and Infrastructure:
Saud AlHamoud, Fahad Alothaimeen, Ahmed Alrowatie, Saad Alajlan, Hassan Almitib, Abdinasir Mohamed, Sultan Alotaibi, Mohammed Alanazi Mohammed Alsharani, Khalid Aljandal, Faisal Abulnaja, Ahmed Alosaimi, Muhannad Abu Hussain, Nasser Almoffarej, Muhammad Al-Hashem, Ahmed Al-Ghamdi, Amer Ashraf, Abeer Al-khars, Nawaf Babgy, Fevicks Kumar, Islam Gamal, Layla Al-Mutairi, Shroq Al-Ghamdi, Amjad Al-Zahrani, Tjad Clark, Ali bjorn, Meshari Alyami, AbduIrahman Bahkali, Reema Alomair.
Project Management:
Naif Shalhoub, Esshaq Almotawa, Sara AlRasheed, Mohammed Alshaalan, Mohammed Albreeh, Nezar Kaaki, Muna Alsahli, Abdullah Aldahami, Faisal Al-Tamimi, Tariq Alrouqi. Sahar Alghamdi, Raghad Abuznadah, Naif Almohammed, Oraib Alhemmyaine, Sarah Binlibdah, Shrooq Almo- hamdi.
Human Evaluators:
Amal Almohammadi. Badr almalki, Amjad Alsaeed, Alhanouf Alotaibi, Rajaa Almakki, Maryam Alasmari, Maha Awaji. Fatima Alshehri, Maryam ALshuwaiman. Ebtesam Alzahrani, Yasmeen Al- gahtani, Atheer Almusallam, Rehab Almalky, Shahad Alsulami, Abdullah Albalawi, Abeer Alqah- tani, Lama Alrajeh, Shahad Aqeel, Yasir Alharbi, Rassil Al-Otaibi, Khulud Alawadh, Fatimah Almu- tairi, Suad Alkhamshi, Abdulrahman Alasmari, Goot Alqahtani, alhanouf Alfoaim, Rawan Aljohani, Aisha Almutairi.
B FREQUENTLY ASKED QUESTION
What is the difference between Alignment at Scale vs. Post-training ALLaM is trained from pretrained English language model (LLM). In pretrained English-only language model, the lan- guage alignment between English and Arabic isn't sufficient, By the phrase"... ... language alignment and transferability of knowledge at scale.." " we mean that we focused on English and Arabic co- learning (alignment and knowledge transfer) during the pretraining stage by jointly training both languages together. Note that when we mention "language alignment at scale" here, it refers to the pretraining stage, not the post-pretraining stage (fine-tuning preference tuning).
Why did we change the training data distribution for 30B experiments Not all the training was done at the same time. As the training progressed, we gained more knowledge about our process, data, and the entire ecosystem of our training engine. Iterating over single training run incurred significant costs, so we always prioritized quality over ablations for large-scale training runs. Given the available compute and deadline, were able to conduct only one training of the 34B model. We discovered that we could apply custom filters to a large data collection based on our use cases and preferences. In the first phase, we used an open-source data collection, and in the second phase,
25

---

Published as a conference paper at ICLR 2025
Training Log
Loaa Adive
Downtine (5alum)
Dowriree das setodlutons 39 day,, 1644322 s1805
Dowrinne (Cannelitor)
1x12
2016
202488-3
2004-8811
2014-9-3
179-4707
2224-7-45
2004-98-1
2014-8-3
Date
Figure 11: Annotated training log for ALLaM-30B showing uptime and downtime. Downtime due to cancellation indicates the training was paused to accommodate other jobs. Failures are marked in red. The model recovered from several loss spikes through the course of training without degrading performance.
we filtered a collection of 84 CC snapshots down to approximately 30T tokens. After filtering, we performed manual check to verify the data quality.
What was the uptime of the cluster We didn't 't log the uptime of the cluster for all training runs, but we tracked the uptime for our 34B training runs. Figure 11 shows the cluster's uptime.
Will you opensource/openweiihr our models? We released our 13B model on the IBM WatsonX platform May 2024. Our model, pretrained from scratch, is now available on Microsoft Azure. As time progresses and we build more risk assessment tools with scalable oversight pipeline, will open up more models to the community.
Why did we report Instruct model result instead of base model result in the main paper?
Blurred distinction between base and instruct models: Modern pretraining often incorpo- rates supervised fine-tuning (SFT) data, including alignment data aimed at improving user interactions. As result, the clear separation between the base model and the instruct model has become less distinct. Many models today pre- trained with some degree of align- ment, making difficult to evaluate them purely as base models.
For instance, as shown in Table 12, the Qwen2-7B-base achieves score of 77.94 on GSM8k, while ALLaM-7B-base (trained from scratch) achieves 16.98, with a significant delta of 60.96. After supervised fine-tuning, 53.6, while Qwen2-7B -i-isstuutt scores 77.86, reducing the delta to 24.26. Although AL- LaM shows improvement of 36.62 points during SFT, Qwen2 experiences notable gain in performance in this phase. We suspect this is due to the inclusion of alignment data during Qwen2's pretraining phase. Thus, while Qwen2 models inherently better than ALLaM, the performance gap between Qwen2 and ALLaM at the base level does not nec- essarily reflect true model capabilities due the suspected presence of alignment data in the base model.
CaaaM--B-isstuue (from scratch) scores
Focus on user interaction: The primary goal of building these models is optimize them for user interaction. Since users will interact with the instruct version of the model, makes sense to report the results of the model in its instruct phase. This ensures that the reported performance is reflective of the actual experience users will have, making the results more relevant and impactful for the paper's audience.
26

---

Published as a conference paper at ICLR 2025
What is fair comparison of different models? Many models are released without information on the training dataset size, and of those that do, most do not provide a breakdown of tokens per language. Additionally, it is not clear how we should categorize models, based on (i) training FLOPs, (ii) model size, (iii) inference FLOPs, (iv) number of training tokens, or (v) quality of training tokens. Therefore, it is extremely difficult control for all these factors. Instead, we opted to ensure fair assessment by comparing our 13B models to larger models, such as Jais (30B), Command R+ (104B), and GPT-4 4 (unknown, but almost certainly larger than 13B), which actually puts our model at a disadvantage (see Figure 9).
What are failure cases of ALLaM? We identified several patterns where the model's performance could be refined through extensive human evaluation. These observations can be summarized as follows:
1. Repetition Issues:
In certain versions, the model exhibited tendency to overgenerate or repeat content. This behavior varied with temperature settings. After extensive testing, we found that temperature of 0.6 yielded the most balanced outputs.
2. Translation Challenges: The model sometimes regenerates the translation instruction within the translated output, or attempts summarize the shared context while translating it, rather than focusing solely on the translation itself.
3. Variation in Outputs: When tasked with summarization or translation, the model provides multiple outputs.
4. Precise Instruction Following:
The model does not always adhere to specified output length structure. For instance, when requested to summarize text in 100 words, it might exceed this limit or produce a summary longer than the input text.
5. Complex Arabic Proofreading: The model faces challenges with certain Arabic-language tasks, such as applying proper punctuation, grammar, and diacritics. While it may provide partially correct answers, it often fails deliver fully accurate or comprehensive outputs in these areas.
Table 7: ALLaM failure examples. For brevity, we omitted parts of the questions/answers with "[...]" to focus on the failure cases
Question
ALLaM's Answer
Error
السررروي المطي الأين المسامنs داي
اللll a/ا تابل السردي الأيي الأبن الححمما 4ill i,les يشق رطام [--.] حليوط رصوة ef االراا الأطي الأسن عاات في ولأاادد السجية
Excessive repetition of the same phrase sالاe الليبl البب iهاة yomé, ولبر through- out the response. We noticed that the KL divergencc penalty scalar in DPO impacts this. This issue occurs in fewer than 1% of our total evaluation samples.
Sزا اللية كفان وصس و وططي ولملرااتت repeated many times]
Have you ever struggled with persuading somcone? [...] First you need to work on your confidence [...] Finally, build trust I.d. Reyرl Will Vor 4 تاعيز %/2
In اسمك القاا inj ف مااممد بددن بلطيي shas une بلااا تبيي في سخاا الامعادة اا
The model asked for context that was al- ready provided.
war Bc Revl dl تررز عن عب باnا القاا
و [..]] I. اعر 30 uns ااري wekn on K 231 (id السددوة gسيt legets ins
السييي الدون غزنن |..] الكركة ملاة الغسس ف w f is الإلااي المه of إق كي ا551 مندد إن [..]
Instead translating the provided Tafseer text, the model created a new explanation/- summary it.
|..|] العبر plz R,
الهس Us iij ل inss فنيع ككاك jن Mai..
Please, can you do me the faver of trans- lating this text English? Rhetoric is dofined the art of persuasion [-.].
The model translated the request for trans- lation instruction along the content text
الراا فن 4 و الإاة مرق yacke' will Jl [..]
Write poem in arabic
Pa اiر Kelly تسييكة 2/87 3 chacclue giيي ااطبي الايييي :Jets شسيةة اديية اله
ALLaM provided Arabic introduction instead of starting with English response before the poem.
How many Arabic tokens were in Llama 2, and how many did you add to ALLaM? We expanded Llama 2's vocabulary from 32,000 to 61,586 tokens with all the added tokens being Arabic. We define an Arabic token any token that contains at least one Arabic letter by using the regular expression "* (?=.* Aar.**ApArraaa (Arabbcc)) ) ) + $ Llama 2 contained 46 Arabic Tokens (mostly single Arabic characters), while ALLaM contained 29,552 Arabic tokens post-expansion.
27

---

Published as a conference paper at ICLR 2025
Note that Llama 2's tokenizer does not include all Arabic letters, and thus has to rely on byte fallback for less common Arabic letters. For instance, it tokenizes the letter "<U+0624> as two bytes (<<0DDB>', ><0xA4>'].
C RELATED WORK
Our work sits at the cross-section of research building language models that support multiple lan- guages and scaling such techniques in terms of size and data. To successfully train a large language model in language other than English requires complete understanding of cross cross-lingual transfer- ability between languages and a good understanding of scaling laws, as well as the fundamentals of training large language models. In this section, discuss work on language modeling from the perspectives of tross-lingual alignment, multitask learning, and Arabic specialization.
C.I LANGUAGE MODELING AND CROSS-LINGUAL REPRESENTATION:
In early work, word representations were derived using basic forms of the skip-gram model (Mikolov et al., 2013), wherein each word is assigned a representation that does not account for varying contexts (Grave al., 2018; Pennington et al., 2014). Further work in this area developed word representations that are adaptive to the context surrounding the words (McCann et al., 2017: Peters et al., 2018a; Howard & Ruder, 2018; Devlin al., 2019; Yang et al., 2019; Radford et al., 2019).
Peters et al. (2018a) introduced ELMo, model built with bidirectional LSTM-based language model (LM) for pretraining to obtain contextualized word representations. This technique combines the outputs from all layers linearly when targeting specific tasks. Nonetheless, the sequential nature of LSTM--ased LM pretraining presents challenges in scaling training efficiently. Concurrently, Vaswani al. (2017) developed the Transformer architecture, which leverages multi-headed self- attention and positional encoding to handle long-range dependencies and enable parallel processing. Following that Radford al. (2019) introduced GPT, a model that pretrains a Transformer decoder using a conditional language model objective, with subsequent fine-tuning requiring only minimal modifications. Similarly, Devlin al. (2019) unveiled BERT, which uses a Transformer encoder pretrained via a masked language modeling (MLM) objective. This approach excelled at task adap- tation and benefited from the MLM's ability encode context bidirectionally, unlike the traditional unidirectional (conditional) LM that processes either the left or right context. Later Raffel et al. (2020) proposed a detailed hybrid encoder-decode architecture based LLM with an implementation of many objective functions via autoregressive structure.
During the release of BERT, mBERT, a multilingual version of BERT trained on 102 languages using a shared vocabulary of 110K subword tokens. Despite the lack of explicit cross-lingual su- pervision, MBERT has demonstrated the ability to learn cross- looss-lingual representations that generalize well across languages. Wu & Dredze (2019); Pires et al. (2019) evaluated the zero-shot tross-lingual transferability of mBERT on several NLP tasks and attributed its generalization capability to shared subword units. Pires al. (2019) additionally identified structural similarity (e.g., word order) as another crucial factor for successful cross- lingull transfer. K et al. (2020), however, argued that shared subwords contribute minimally, and instead, structural similarity between languages is more critical for effective transfer. Artetxe et al. (2019) further showed that joint training might not be necessary and proposed an alternative method transfer a monolingual model to bilingual model by learning only the word embeddings in the target language. They also highlighted the vocabulary size per language as an important factor. Finally, Xue et al. (2021) showed that joint training on a large multilingual vocabulary can robustly map multilingual language models to the same latent space.
In the early days, Cross-lingual alignment from mono-lingual embeddings was tricky and often re- quired complex adversarial training (Conneau et al., 2017), careful orthogonal mapping (Artetxe et al., 2018) or semi-supervised learning (Mohiuddin et al., 2020; Bari et al., 2020). With the in- troduction of mBERT, it became evident that learning joint distribution makes it casier for LLM to achieve cross-lingual alignment at scale. Lample& & Conneau (2019) enhanced mBERT by incorporat- ing a conditional LM and a translation LM (leveraging parallel data) objective along with a language embedding layer and trained larger model utilizing more monolingual data. Huang al. (2019)
httpps:// thub. com/ com/googl goog] e resea htw.chheeettbb////ecccbeec
ti til 1
28

---

Published as a conference paper at ICLR 2025
suggested employing auxiliary tasks like cross-lingual word recovery and paraphrase detection for pretraining. Subsequent work by Conneau et al. (2019) and Soltan et al. (2022) scaled up the train- ing of multilingual language models. As well, Xue et al. (2021) scale the size and languages in the TS architecture. In an effort to reproduce GPT-3, Scao et al. (2023) trained the first auto-regressive multilingual LLM.
C.2 MULTITASK LEARNING AND ALIGNMENT
Early work has demonstrated that multitask learning can enhance the performance of NLP models (Collobert & Weston, 2008). In explicit multitask learning, augmenting all samples during train- ing may introduce noise due to differing output distributions in a traditional full-model fine-tuning setup (Weller et al., 2022: Bari et al., 2021). For implicit multitask learning, Radford al. (2019) showed that a language model can begin to learn downstream tasks without explicit supervision by pretraining alone. Large language models (Brown et al., 2020b; Smith et al., 2022; Chowdh- ery et al., 2022) at scale can perform few-shot in-context learning, making them effective multitask models. Additionally, Sanh et al. (2021); Wei et al. (2021); Muennighoft et al. (2022); Chung et al. (2022) found that these implicitly learned language models could be further improved by explicitly fine-tuning them with human instructions and prompts (Bach et al., 2022; Wang et al., 2022) multitask fashion. Unlike previous template-basee prompting approaches, Ouyang et al. (2022) ap- plied preference tuning with reinforcement learning (Sticnnon et al., 2020) using naturally written prompts. Subsequently, Bai et al. (2022) introduced Constitutional Al to automate alignment using AI feedback. Recently, following the work of Rafailov et al. (2023), various efforts (Azar et al., 2023; Ethayarajh et al., 2024; Hong et al., 2024; Park et al., 2024; Meng al., 2024) have been directed towards preference tuning without explicit reward models.
C.3 LANGUAGE MODELS FOR ARABIC
As of the time of writing, the most prominent Arabic-focused LLMSs are:
1. Jais (Sengupta et al., 2023): 13B and 30B base and chat models trained from scratch using a combination of natural and translated Arabic data along with English and code data.
2. AceGPT (Huang et al., 2023): 7B and 13B base and chat models trained from Lla 1mamm22 without vocabulary expansion.
While Jais and AceGPT are currently the most prominent models, early open models such as AraGPT (Antoun al., 2020), AraT5 (Elmadany et al., 2022), AraBART (Eddine et al., 2022), and Noon (Lakim et al., 2022) 7 pioneered the area with models developed with limited resources to serve Arabic.
Other models such as Jasmine (Abdul-Mageed et al., 2023) and Aramus (Alghamdi et al., 2023) also showed the need for building language model for over 400 million speakers worldwide.
In addition to the language adaptation of models and multilingual models reviewed above, recent work has focused on building multilingual/blinguaall language models from open weight language models. For example, Ruciúski (2024) adapted Mistral 7B for the Polish without vocabulary expan- sion. Mala-500 is another effort to expand to 534 languages by expanding the vocabulary to 260K tokens and further pretrained I.1 ama-2 using LoRA adaptors (Lin et al., 2024). Due to the number of languages they aimed to support, a small amount of data was included for each language and the evaluation of the approach was limited to measuring perplexity and automatic classification bench- marks. (Cui et al., 2023) introduced a Chinese Language adaptation of L1 ama and Alpaca models, where the vocabulary was increased to 50K tokens, then continued to pretrain the models and finally fine-tune them.
D LIMITATIONS
ALLaM trained on data that may potentially include toxic language, unsafe content, and societal biases originally sourced from the internet, leading to the possible amplification these biases and
Phttps://huge aggaace.e....o. co/ Nasee /ncn//cnntt
29

---

Published as a conference paper at ICLR 2025
toxic responses. Although ALLaM underwent comprehensive safety training during the alignment phase, more community feedback is needed to iteratively improve ALLaM. Additionally. inherent uncertainties in generative models mean that trials cannot encompass every possible use case, mak- ing it impossible to predict the model's responses in all contexts. This can occasionally result in inaccurate, biased, socially unacceptable outputs, even if the prompt itself is not explicitly offen- sive. Developers must conduct thorough safety evaluations and make specific adjustments to ensure that ALLaM is suitable for their intended purposes. Furthermore, the output generated by ALLaM should not be considered a statement from ALLaM's creators or any affiliated organization.
E ETHICAL STATEMENT
While conducting and presenting this research, we are committed to upholding the highest ethical standards. We recognize the potential impact of large language models society and the impor- tance of ensuring their responsible development and deployment. Our work adheres to principles of fairness, transparency, and inclusivity, striving mitigate biases and ensure diverse representation in our training data. We are mindful of privacy concerns and have taken steps to anonymize and secure data used in our research. Additionally, we acknowledge the potential for misuse of language technologies and advocate for their ethical application, promoting beneficial use cases while being vigilant about unintended consequences. ALLaM models are made openly available foster collab- oration and further research, with the aim of contributing positively to the advancement of language technologies and supporting the cultural and technological growth of the Arahic-speaking world.
F RISK STATEMENT
The deployment and use of LLMs in various applications poses significant risks, including data privacy and security concerns due to the inadvertent inclusion of sensitive information in training datasets. LLMs can perpetuate or amplify biases, resulting in unfair treatment and discrimination in critical decision-making processes. They can also generate convincing but inaccurate content, spreading misinformation and potentially influencing public opinion negatively. Over-reliance on LLMs may diminish human judgment, and the models' susceptibility to adversarial attacks can com- promise system integrity. To mitigate these risks, we follow robust governance, continuous moni- toring, and iterative improvements. We also adhere to best practices in data handling and model training. fostering transparency and accountability in LLM development.
G PROCESSING OF ENGLISH DATA FOR 30B PARAMETER MODEL
The RedPajama V2 (Rp dataset is large resource for training large language models:
Size and Composition: RpV2 is an open dataset that includes over 100B text documents sourced from 84 CommonCrawl snapshots. These documents have been processed using the CCNet Wenzek et al. (2019) pipeline, which is known for preparing web-crawled data. The dataset encompasses a massive 30T tokens, making it, to the best of current knowledge, the largest public dataset released specifically for training language models.
Quality Signals and Deduplication: Of the documents included, 30B comes with 40+ pre- computed quality signals, and 20B documents are deduplicated. This ensures that the data is not only vast but also of high quality and relevance for training purposes, reducing redundancy, and improving the efficiency of learning.
Please refer to https: / http:/wwwtttttteeeee/ www.:/ogge.eee...boooo/o/o// http///ag//edpaa//ee// jama data-v2 for the com- plete list of existing quality signals.
Multilingual Coverage: The dataset includes documents in multiple languages, although the pri- mary focus seems to be on English. The other languages are German (DE), French (FR), Spanish (ES), and Italian (IT). Table Appendix G shows detailed statistics of the multilingual breakdown of V2 dataset.
Shtpp: thub. com httphh/cww/ooooeeeeeccrrrreee//e
jama Data
30

---

Drop 0.9 Published as a conference paper at ICLR 2025
Table 8: Document and token counts for different languages
Language
Document count
Estimated token count (deduped)
English
14.5B
20.5T
German
1.9B
3.0T
French
1.6B
2.7T
French
1.8B
2.8T
Italian
0.9B
1.5T
Total 20.8B
30.4T
Processing Steps Applied:
Out of the 40+ pre-computed quality signals, we applied the following create high-quality subset. Note that we focused on English only.
Table 9: Preprocessing steps to filter high-quality data from the Red-Pajama V2 dataset.
Annotation Tag
Description
Threshold
ccnet_language_ score ccnet_length ccnet_nlide mlines rps_dos _l__do_____ palm_score
Language identification model score. Number characters in the document. Number of lines in the document.
Keep 2 0.6
Drop < 150 characters
Drop < 3 lines
FastText classifier prediction for doc- ument classification as Wikipedia, OpenWebText, or RedPajama-VV VI book (English only).
Sample according to dis- tribution
rps_doc_frac_lines end wit_withhellisss
Fraction of lines ending with ellipsis ("..." or
Drop 0.8
rps =ps_doc_frac_nt alph_words
Fraction of words without any alpha- Drop 2 0.9 betical characters.
ipe_doc_loren_ipsun
Ratio of occurrences "lorem ipsum" to total characters in content (after nor- malization).
Drop 0.5
"ps_doc_stoop_wrrd fraction
Ratio of stop words to total words the document, using stop words from here.
rrs_doc_symbol.1 word_ratio
Ratio of symbols ("#",, 4 or "..") to words in content.
Drop 2 0.9
rps_doc_ldnoobw
_words
Count of sequences from the List- oa-Dirt--Nuught-OOee Soachtt--Obccee--an Otherwise-Bar Bad-Words blocklist (see
Drop 0.9 (oanoobx_ wttpd//tttttaaaaaaaa
here).
minhash_aignan
Minhash signature for fuzzy deduplica- tion at Jaceard similarity of 0.7, based 128 hash functions grouped into 14 bands rows for L.S..
0.7 (Enochxx words/total
Stats of the filtered data: Applying all the steps above removed 85.9% of the documents, and 80% of the words.
In addition to the above steps, the CC-NET pipeline, which was used to download CC snapshots, applied language detection and exact-match paragraph deduplication.
31

---

Published as a conference paper at ICLR 2025
Table 10: Document and word counts after each data processing Step
Step
Input (kdocs)
Outpat (#docs)
Remooval Ratio doc level (%)
Input (thords)
Output (#words)
Removal Ratio word level (%)
minhash minhashh_signtturee..
24,38B
24.36B
0.11
2.311
2.301
0.07
24.36B
14,49B
40.5
2.30T
1.30T
43,51
conet_ianguage__cc
14,49
14,30B
1.301
1.301
0.44
conet_lengta
14.30B
13 80B
3.51
1.30T
1.30T
0.05
conet_niines
13,80B
12.87B
6.79
1.301
1,281
1.37
12,87B
12.87B
1.28T
1.28T
12.87
12.87B
1.281
1,281
_pa_dde_Loren_ipste
12.87B
12.87B
1.28T
1.28T
sps_doc_stop_woree
Fraciin
12.87
12.87B
1.281
1,281
12,87B
12.87B
1.28T
1.28T
rps_
gos_doe_ldnoob___ winooo_noo...oooooo M__rds
12.84
12.84B
1.281
1,281
joa_doe_ml _p_______p____corr
12.84B
343.12M
73.29
1.28T
459B
64,03
After applying all processing steps outlined above, we ended up with:
3,451,217,579 (4.3B) total documents, with a total of 4,587,781,98, 18,49, (4.5T) words, and 5.2T
tokens.
The average doe length words and characters.
H TRAINING DETAILS
H.1 COMPUTE AND TRAINING INFRASTRUCTURE
Over the course of our development of ALLaM, we had access to 128-1024 A100 GPUs. Our GPU cluster was equipped with InfiniBand connections to enable high-speed communication between nodes. The all-reduce test on the cluster ranges around 1200-1400 Gbps (node-node interconnect (RoCE)). The entire training period of the models is estimated to be 5M GPU hours.
At the start of the project, we forked Megatron-LMD and applied our customizations (including improving data iterators, adding metadata in the checkpoints, and custom data pipelines). We uti- lized data, tensor, and pipeline parallelism supported by Megatron-LM to efficiently train at large scale as well as FlashAttention (Dao et al., 2022; Dao, 2024). By leveraging these techniques, we achieved significant improvements in training speed. The throughput per GPU varied from 135 to 167 TFlop/s/GPU depending on the number of GPUs, number of nodes, batch size, and parallelism strategy. We trained ALLaM with bf1 mixed-precision.
H.2 PRETRAINING DETAILS
We fine-tune our base model, which was trained on 3.2 trillion (2T Llama-2 1.2T ALLaM) tokens, for 3 epochs using Ultra- Inst inct -22 with learning rate of x 10-6 and batch size of 1024. The model not trained to generate the prompt, as we mask out our prompt tokens when calculating the loss. Intra-Instinct.. conversations. train on these multi-turn conversations, we performed turn www.anttttttaiia Figure 12 visually explains the process of turn augmentation.
contains substantial number of multi-turn
While training the SFT model, we encountered tokenization issues. Specifically, I. t.lama-22 22's tok- enizer was trained using sent sentencepiece 10 which breaks the beginning and end of sequence tokens into multiple tokens and adversely affects long multi-turn conversations. To address this is- sue, we patched sent sentencepiece using the HuggingF et al., 2020). Over many iterations of training, we saw that even having 1% noisy samples (e.g., empty responses or formatting issues) in alignment data noticeably affect model quality.
LlamaTokenie
wrapper (Wolf
/ //gi thub. https////g..tt...ooooo//////////
32

---

Published as a conference paper at ICLR 2025
H.3 ALIGNMENT TRAINING DETAILS
For DPO, we used a batch size of 512 with Peeeaatt = 0.1 and a learning rate of 9 x 10-7 decayed to 5 x 10-7 using a cosine annealing learning rate schedule. We train ALLaM for a single epoch using all the preference data.
From our initial experiments with small datasets, we observed issues with model quality even when small fraction (0.1%) of the data was noisy. In this context, noise can be improper labeling of positive/negative pairs or low quality positive outputs. It is not clear, however, if after scaling up the DPO data whether the model can ignore this type of noise. early DPO models, trained on data where we did not verify all the samples, we found that even few moderately noisy samples resulted in broken models that repeatedly generate the same text or output incoherent text.
H.4 DPO VS. PPO
One of the fundamental differences between DPO and PPO is that PPO is always on-policy with an external reward model. In our experi- ence with DPO, we did not encounter any sig- nificant issues with off-policy experiments. Ad- ditionally, DPO allows for faster iteration and easier understanding of the training dynamics. The decision to use DPO over PPO based on logistical constraints rather than a perfor- mance comparison of the algorithms. Given our compute setup and time constraints, we chose to proceed with DPO. We plan explore PPO in the future for alignment.
Cocvenatis
Augneticd
Aupezated
Augroaatd Tum
pronpe
pourpe
proepe
proege
Aeoroeeppyy 1
Antissa rppy
Aunezat may
procept
Aoittar spple
hoply
Amisust roly
Uier
Uar
Rviitted
Auiiiant reply
Figure 12: Augmentation process for conversa- tions. The original conversation (left) is expanded into one sample per turn (right). with user prompts and assistant replies marked for training (red) and masking (orange) to enhance ALLaM's language understanding and multi-turn response generation capabilities.
I EVALUATION
Evaluation Framework All evaluations were completed using the Language Model Evaluation Harness (Gao et al., 2023) with the following exceptions: HumanEval was evaluated using BigCode Evaluation Harness (Ben Allal et al., 2022). MMLU-Pro, MixEval,
and Arabic MMLU (Koto et al., 2024) were evaluated using the repositories of the dataset creators.
1.1 DETAILS OF THE EVALUATION DATASETS
The automatic evaluations cover Arabic and English benchmarks grouped into the categories listed below:
1. Multi-domain: MixEval (Ni et al., 2024), MMLU-Pro (Wang al., 2024), and BBH (Suz- gun et al., 2022).
2. Reasoning and Commonsense: HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), WinoGrande (Sakaguchi et al., 2019), and AraSwag (Nagoudi et al., 2022).
3. World Knowledge and Language Understanding: MMLU (Hendrycks et al., ARC Easy and Challenge (Clark et al., 2018), TriviaQA (Joshi et al., 2017), BoolQ (Clark et al., al., 2019), AGIEval (Zhong et al., 2023), Exams-Ar
2019), NQ Open (Kwiatkowski (Hardalov et al., 2020), MMLU Arabic (tr) (Huang et al., 2023), MMLU Arabic (MBZU) (Koto al., 2024). and ETEC (in-house curated).
4. Safety and Alignment: Hendrycks Ethics (Hendryeks al., 2021a), ACVA (Huang al., 2023), TruthfulQA (Lin et al., 2022), and AraTruthfulQA (in-house curated).
5. Conversation: MT-Bench (Zheng et al., 2024). and Arabic domain capability dataset (in- house curated).
33

---

Published as a conference paper at ICLR 2025
Sesskj nntry wilten' trightn
Seakkrrrenten "Getwrllng"
Wuig Ui on slemy prosont lolecte je mile tmoigges
orta by denos 10 way salan devenngue
Figure 13: Example of Arabic cultural alignment between ALLaM and GPT-4.
6. Math: Minerva MATH (Lewkowyez et al., 2022; Hendrycks et al., 2021b), GSM8K (Cobbe et al., 2021) and AraMath (in-house curated).
7. Coding: HumanEval (THUDM, 2022)
The following benchmarks were curated and developed in-house:
collection of 1891 multiple choice questions covering different exams performed by the Education and Training Evaluation Commission in Saudi Arabia!!
ETEC:
AraMath: set of 600 test samples that were post-processed and prepared from the Ara- Math (Alghamdi et al., 2022) dataset. These samples focus on testing the models' perfor- mance on Arabic math problems.
AraTruthfulQA:: dataset created using similar methodology the TruthfulQA (Lin al., 2021) dataset. It comprises total 541 samples, 285 samples were translated directly from TruthfulQA using GPT-4 4 and carefully validated and localized by human verifiers. Additionally, 256 questions were curated by humans to ensure their contextual relevance and cultural appropriateness.
I.2 DETAILED ARABIC EVALUATION
Follow the Table 11 for details.
1.3 DETAILED ENGLISH EVALUATION
Follow the Table 12 for details.
1.4 EXAMPLES
example of Arabic culture alignment improvements illustrated in Figure 13.
lhnttps://etcc gov. sa /ooe
34

---

Published as a conference paper at ICLR 2025
Table 11: Comprehensive Arabic benchmark results.
ACVA
KIMLU (1))
Z0ome (ary
ITEEC
Heang (200))
22-dat
D-dadt
Dolox
Todar:
Oolat
Sodau
Poettraled
ALLaheBum San Urom acnatch)
7B
12:15
10C48
44.43
38.25
12.73
41.7
24.b
ALLaM-Baae
7B
91.53
(AL
41.02
54.42
36.35
19.00
25.3
11.
Matni-ba
70
31.33
40.51
16.3
0L310477
70
26.11
37.8
30,57
25.
25.7
27.17
23.5
16.8
78
21.88
31.7
Gana
78
25.38
54.32
46.11
20.04
22.01
20.3
40
38.55
71.34
(7.07
26.00
41.99
13:20
15.9
40%
11.
11.75
12.55
IT3
20.75
Liaa 2
198
20.53
68.52
58.51
310
23.89
15.5
haee
138
29.25
D1.15
22.2
25.25
$3.33
27.00
25.3
28.5
ALLaXCIsse
78.35
JST
JalesG
51.66
71.10
50.35
25.2
Qjemt.5
320
17.7%
730
25.91
18.07
18.53
17..1
310
45.3
Vod.
318
02 16
00.25
0231
33.71
19.21
020
2
008
04.33
16.17
36.97
12.31
11.02
20.3
Quee
220
05.54
7600
01.35
310
13.01
01.5
01.75
60.0
06.00
62.3
Abcmnk-t22B--0
1418
45.1
43.32
48.03
$3.00
A1.
Vine-tuned
ALLaMcliattut (Tro scralch)
7B
10155
10:35
05.35
11.35
10.31
$7.54
12.3
7B
20.25
01.35
09.0
81.0
02.1
12.66
15.0
10.5
70
12:4
22.5
70
31.5
34.05
20.0
015te- liste-ltatrae
70
23.35
06.11
12.74
505
31.77
27.33
20.5
2655
Queak
78
37.75
193
42.3
51.
Ann-2
40
01
73.65
24.37
56.39
15.75
11.50
3.6
20.0
ALL Instrrut
138
54.77
76.15
16.11
51.01
14.31
$3.00
33.5
15.2
138
20.41
64.55
01.
50.20
40.94
41.01
22.5
Liems 24chal
138
20.75
00.14
36.64
28.75
22.31
33.48
3-44
22.5
77.18
14.00
05.3
S
908
85.37
10.66
60.37
61.21
15.05
82.6
Qwea iuChhat
228
35.30
15.55
0121
0009
08.23
19320
3500
630
Anw-3
338
06.55
70.00
0121
47.70
15.10
65.
470
17.53
17.27
1240
13.00
42.04
13.17
32.5
207
25.5
Queal ACChat
228
46.1
80.09
54.12
$3.21
61.0
05.
728
33.3
71115
11.
05.18
08.00
15.6
47.7
61.7
Oomovulle
09:35
2012
05381
005
Mical ac3lll Iecteut vo
-3.79
4671
45.7
35

---

Published as a conference paper at ICLR 2025
Table 12: Comprehensive English benchmark results.
10.44 1h00 10.00 1000E1
030
15
160
90300
0000
2016
1000
1008
2800
1230
2200
231e
235
63.0
130
Bext
267
361
450
6111
9215
222
672
6511
5.8
11:11
255
210
64.70
13.50
305
122
1:30
0.11
68.0
8/24
341
1.5m
32
2234
2114
2224
48.45
2035
2229
10
128
22
618
2012
4612
0.80
11:0
30:0
1100
170
100
11.01
1391
370
8044
0044
6.15
6.21
0008
115
1138
1224
1273
903
0100
211
575
11.6
251
4000
215
2100
0450
210
0150
610
6111
2514
450
220
6040
834
6130
612
6.1
6.3
221
335
310
512
67
634
570
1234
441
1811
11-
140
11
642
6010
335
625
650
615
625
221
Tay
0331
1937
124
600
400
4000
3330
0006
724
180
660
4000
220
513
825
810
462
602
6331
242
354
631
512
521
8848
652
623
000
610
312
110
44
16.6
12
618
842
633
02
015
12
055
0.84
130
1230
1500
504
322
334
2336
1000
6530
622
60m
945
1150
352
625
350
961
433
22
2.5
022
011
38
13
680
2116
6.1
007 S.A
312
5114
611
3556
271
411
139
0550
331
551
9/18
N0
65
506
374)
611
6427
6328
330
635
2101
5112
61.
1130
the
34
6.1
0-6
8:34
693
6339
2714
9413
631
873
321
5.5
614
861
49
2014
68.5
4036
7720
28.0
233
2319
366
455
776
113
200
218
12-40
272
10000
0/6
8:31
64.0
13.00
00.0
8:55
110
6.19
32.35
34.0
9.0
0.20
286
6-3
6:1
4110
06.8
31.8
19.00
60.00
01.5
574
80
5726
1.5
0.3
32
011
2-
6:4
28
632
74
130
146
1138
722
75
1:0
54t
248
100
1.30
36


Reviews:
[{"review_id": "KDnWSQvvuV", "reviewer": "Reviewer_hSix", "summary": "This paper introduces ALLAM, a large language model for Arabic and English. The authors demonstrate the effectiveness of their training and alignment strategy for achieving state-of-the-art performance on several Arabic benchmarks.", "questions": "...", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Focus on Arabic Language: The paper specifically addresses the need for high-quality language models for Arabic, a language with a significant number of speakers worldwide and relatively fewer resources compared to English.  This focus fills a gap in the current landscape of large language models.    \n\nComprehensive Training and Alignment Strategy: The authors employ a thorough approach to training and aligning ALLAM. They experiment with different data mixtures, vocabulary expansion techniques, and use both supervised fine-tuning and preference training to achieve optimal performance.    \n\nState-of-the-art Performance: ALLAM achieves state-of-the-art results on various Arabic benchmarks, demonstrating the effectiveness of the proposed techniques.  The authors also show that their model maintains or enhances English performance compared to the base Llama-2 model.    \n\nExploration of Second Language Acquisition: The paper investigates the use of second language acquisition techniques for LLMs, which is a promising area of research.  The authors show how pretraining on a mixture of Arabic and English text can lead to effective learning of Arabic without catastrophic forgetting in English.    \n\nDetailed Methodology: The paper provides a detailed description of the training methodology, data curation process, and evaluation setup.  This transparency allows for reproducibility and facilitates future research in the field.   \n\nCommitment to Openness: The authors express their intention to make the ALLAM models openly available to the community, which promotes collaboration and further development of Arabic language models.", "weaknesses": "The authors should clearly state the originality of their work. While the paper builds on existing techniques such as second language acquisition for LLMs, it is not clear what specific novel components are being introduced. Is it the particular training recipe, the alignment strategy, or the focus on Arabic? A clear statement of originality is essential for establishing the contribution of this work.\n\nError Analysis:\nThe error analysis presented is currently insufficient. The authors need to go beyond simply stating that some evaluations provide more signal than others. A deeper dive into the types of errors made by the model on different benchmarks is necessary. What are the common patterns in incorrect predictions? Are there specific linguistic constructs or knowledge domains where the model struggles? Addressing these questions will provide valuable insights into the limitations of ALLAM and potential avenues for future research.\n\n\nAdditional Areas for Improvement:\nMotivation for changing training data distribution: The authors mention changing the training data distribution for the 30B model. The motivation behind this change is not entirely clear and should be elaborated upon.\nFair comparison of models: The paper touches upon the difficulty of comparing models due to variations in training data size, architecture, and other factors. While comparing to larger models is a reasonable approach, the authors could consider additional metrics or analyses to further ensure a fair comparison.\nInstruct model vs. base model results: The authors justify reporting results for the instruct model instead of the base model. However, providing some comparison or insights into the performance difference between the base and instruct models could be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hdFN1kLTlj", "reviewer": "Reviewer_gnB6", "summary": "This paper presents a series of LLMs for Arabic adapted from Llama2 as well as training from scratch.  The paper gives technical details of the pre-training, alignment and evaluation of these models.", "questions": "none", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper gives the details of the training of Arabic LLMs, including training from scratch and contunue training from Llama2.  The result Arabic LLMs achieve SoTA on Arabic benchmarks.", "weaknesses": "This is mainly a technical report rather than a research paper.  I do not see novel ideas or methods proposed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PUhBvBqAAm", "reviewer": "Reviewer_9J4g", "summary": "This paper presents a ALLaM: a family of large language models for Arabic and English, based on LLama-2 family of models.\n\nThe paper shows a method that can customize an existing LLM (like Llama) to acquire a much better performance at another language that was not included at all the original training of the model, without destroying the LLM's performance on previous language (i.e. second language acquisition).\n\nThe paper shows that second language acquisition can be done by augmenting the model in two major steps: 1-vocabulary expansion, and 2- carefully pre-training on mixture of new language and old language. This way the second language acquisition is not accompanied by catastrophic forgetting of the original language.\n\nThe paper includes a lot of experiments and comparisons with different baselines, including training from scratch. Showing that their second language acquisition method beats training from scratch. Many evaluations, both automated and human-evals are included in the paper.", "questions": "- How many Arabic tokens already existed in the LLama2 backbone, and how many were added after the expansion?\n\n- Why did the training mix change for the 34B model specifically? (lines 286-292) \n\n- I would be interested to compare allam against Silma Arabic llms\n\n- Please consider evaluating on more dataset like AlGhafa benchmark or including ALLAM on the huggingface Arabic leaderboard.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality: As far as I know, this is not the first work to fine-tune llama-based models for Arabic or other language acquisition. In my opinion, the main originality source in this paper might be the data curation effort, and the vocabulary expansion and continual pre-training without loss of original language. For example, the finding that initializing the embedding of the new tokens from averages of smaller tokens in the original tokenizer is an very useful finding and I expect to see it being used more in the future.\n\nQuality: The effort spent in human evaluation and curation of the data is notable, and I hope this dataset gets released, at least partially for the research community to build on. The comparison against other Arabic-enabled LLMs are also very comprehensive and useful.\n\nClarity: While many typos and non-standard terminology exist and are highlighted in the weaknesses section. The paper is still mostly easy to follow and read. The description of the datasets and training/evaluation procedure is particularly clear and concise.\n\nSignificance: Arabic is definitely an excellent language choice for this exploration of second language acquisition, given that the original LLama2 family of models do not train on Arabic at all (as shown in Table 10 in the original Lama2 report), and that means that the tokenizer and model are not very well trained for Arabic. Also, Arabic is definitely an underserved language in the LLM, especially given the size of its native speakers population (fifth most in the world, roughly 400M people).", "weaknesses": "Even though the paper is mostly well-written, it's littered with typos, non-standard terminology and missing references:\n- \"Fertility rate\": very confusing term, it's not really a rate, and is usually referred to \"fertility score\": https://arxiv.org/pdf/2310.08754 \n- Figure 8 is not referenced anywhere as far as I can see.\n- line 475: missing reference\n-  line 294: missing reference\n- line 291: lower case \"we\"\n- line 072: training of \"these\" \n- lines 151-153: DA identification is a hard task ... classifying the data to DA is not very difficult (opposite claim)  \n- line 155: in-house in-house  \n\n- I would have loved to see some newer Arabic models like Silma included in the comparisons for this paper, or a comparison against the models in the huggingface Arabic leaderboard.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B1shCQTRmD", "reviewer": "Reviewer_SGGa", "summary": "This paper describes the process for building ALLAM, an LLM trained on Arabic and English. Details about the training data and each stage of the training process are presented. Various evaluations are performed, including both automatic benchmarks like Arabic MMLU and human evaluation.", "questions": "1. Page 3 says 4T English tokens then and 5.2T tokens for 30B pre-training. It was a bit confusing what is the 5.2T. Also Table 1 lists English as 660B tokens from mixed corpora -- is this a subset from the 4T English Only column?\n\n2.  Fig 5/6: for my display, there are black lines in the legend (e.g. Arabic/English MMLU) but only red/orange lines in the figure.\n\n3. I am curious if you have considered using something other than Llama2 as your base model. If so, why did you pick Llama2 in the end?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This is a timely paper, as many research groups are working on building LLMs for languages other than English. The paper presents useful details on their experience and will serve as a good reference.", "weaknesses": "There are no major weaknesses in my opinion. I would recommend adding discussing about other efforts in building LLMs for non-English languages and explaining how yours compare, i.e. moving some of the Related Work into the main paper. Especially it would be useful for the reader (who isn't referring to the appendix) to know about Jais and any other Arabic-centric model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)