To Code or Not To Code? Exploring Impact of Code in Pre-training
Authors: Viraat Aryabumi
Yixuan Su
Raymond Ma
Adrien Morisot
Ivan Zhang
Acyr Locatelli
Marzieh Fadaee
Ahmet Üstün
Sara Hooker
Venue: ICLR 2025 Poster (2025)
Keywords: code data
pre-training
code pre-training
Primary area: foundation or frontier models, including LLMs
TL;DR: Exploring the impact of code data in pre-training

Abstract: Including code in the pre-training data mixture, even for models not specifically designed for code, has become a common practice in LLMs pre-training. While there has been anecdotal consensus among practitioners that code data plays a vital role in general LLMs' performance, there is only limited work analyzing the precise impact of code on non-code tasks. In this work, we systematically investigate the impact of code data on general performance. We ask “what is the impact of code data used in pre-training on a large variety of downstream tasks beyond code generation”. We conduct extensive ablations and evaluate across a broad range of natural language reasoning tasks, world knowledge tasks, code benchmarks, and LLM-as-a-judge win-rates for models with sizes ranging from 470M to 2.8B parameters. Across settings, we find a consistent results that code is a critical building block for generalization far beyond coding tasks and improvements to code quality have an outsized impact across all tasks. In particular, compared to text-only pre-training, the addition of code results in up to relative increase of 8.2% in natural language (NL) reasoning, 4.2% in world knowledge, 6.6% improvement in generative win-rates, and a 12x boost in code performance respectively. Our work suggests investments in code quality and preserving code during pre-training have positive impacts.

Published as a conference paper at ICLR 2025
To CODE, OR NOT TO CODE? EXPLORING IMPACT OF CODE IN PRE-TRAINING
Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadace, Ahmet Üstün, Sara Hooker {viraat, ahmezustun, earabookery@coherc com
ABSTRACT
Including code in the pre-training data mixture, even for models not specifically designed for code, has become a common practice in LLMs pre-training. While there has been anecdotal consensus among practitioners that code data plays a vi- tal role in general LLMs' performance, there is only limited work analyzing the precise impact code on non-code tasks. In this work, we systematically in- vestigate the impact of code data general performance. We ask "what is the impact of code data used in pre-training on a large variety of downstream tasks beyond code generation". We conduct extensive ablations and evaluate across a broad range of natural language reasoning tasks, world knowledge tasks, code benchmarks, and LLM-as-a-judge win-rates for models with sizes ranging from 470M to 2.8B parameters. Across settings, we find a consistent results that code is a critical building block for generalization far beyond coding tasks and im- provements to code quality have an outsized impact across all tasks. In particular, compared to text-only pre-training, the addition of code results in up to relative increase of 8.2% in natural language (NL) reasoning, 4.2% in world knowledge, 6.6% improvement in generative win-rates, and a 12x boost in code performance respectively. Our work suggests investments in code quality and preserving code during pre-training have positive impacts.
INTRODUCTION
The role of data has taken on critical significance in recent breakthroughs. State-of-the-art models highlight the importance of the pre-training data mixture, diversity of data sources (Brown et al., 2020; Longpre et al., 2023; Singh et al., 2024) combined with compute availability as key drivers on performance (Dubey et al., 2024; Üstün et al., 2024; Team et al., 2023; Aryabumi et al., 2024). A eritical question is what properties of data impart the best general performance?
Perhaps surprisingly, code is often included in pre- pre-training even if model is not explicitly intended generate high-quality code. Code datasets differ significantly in terms of structure and textural characteristics from high-quality web datasets (Wikimedia; Raffel et al., 2019). Despite this, several previous generations of LLMs like PaLM (Chowdhery al., 2022), Gopher (Rae al., 2022) and Bloom (Workshop et al., 2023) that were not explicitly intended to support code generation, included code data together with high-quality natural language data in their pre-training mixture.
In current state-of-the-art models, it is an accepted norm to not only include code data but further increase the proportion - for instance, Llama (Dubey et al., 2024) has four times more code data in proportion (17%), of its pre- pre-training mixture than Llama 2 (4.5%) (Touvron et al., 2023). While there has been consensus anecdotally among practitioners that code data plays vital role in LLMs' performance, there has been only limited work analyzing the precise impact of code on non-code tasks. Prior work shows particular side benefits of the inclusion of code data, such as impact on scaling in limited data regime (Muennighoff et al., 2023a), entity tracking capabilities (Kim et al., 2024), and mathematical reasoning (Razeghi et al.). However, there has been no exhaustive study to date that systematically investigates the impact of code data on general performance. In this work, we ask "what is the impact of code data used in pre-training on large variety of downstream tasks beyond code generation?""

---

Published as a conference paper at ICLR 2025
We embark exhaustive set of large-scale controlled pre-training experiments. This includes consideration of where in the training process adding code is beneficial, code proportions, the role of scaling, and the quality and properties of code added. While costly endeavor to perform these ablations in a rigorous way, we find consistent and valuable results that code provides critical im- provements non-code performance. In particular, compared text-only pre-training, for our best variant, the addition of code results in relative increase of 8.2% in natural language (NL) reasoning, 4.2% in world knowledge, 6.6% improvement in generative win-rates, and a 12x boost in code per- formance respectively. Further performing cooldown with code, improves NL reasoning by 3.7%, World knowledge by 6.8%, and code by 20%, relative to cooldown without code and leads 4. % additional win-rate increase.
Here, several factors matter including getting the proportion of code correct, improving the quality of code by including synthetic code and code adjacent data such as commits, and leveraging code across multiple stages of training including during cooldown. Our results suggest code is critical building block for generalization far beyond coding tasks and improvements to code quality have outsized impact on performance. We conduct an extensive evaluation on a broad range of benchmarks, which cover world knowledge tasks, natural language reasoning, and code generation, well LLM-as- a- a-judge win-rates. Across experiments on models ranging from 470 million to 2.8 billion parameter models, we find the following detailed results:
1. Code provides critical improvements to non-code performance. Initialization with code pre- trained models results in improved performance for natural language tasks. In particular, com- pared to text-only pre-training, for our best variant, the addition of code results in a relative increase of 8.2% in NI. reasoning. 4.2% in world knowledge, 6.6% improvement in generative win-rates, and a 12x boost in code performance respectively.
Code quality and properties matter. Using markup-style programming languages, code- adjacent datasets such as GitHub commits and synthetically generated code improves the per- formance in pre-training. dataset results in a 9% and 44% increase in natural language reasoning and code performance, re- spectively, compared to web-based code data in pre-training. Additionally, continual pre-training from code model that includes synthetic data results in 1.9% and 41% relative increases in natural language reasoning and code performance respectively, compared to initialization from a code model that does not include code data.
particular, training on a higher quality synthetically generated code
3. Code in cooldown enables further improvement across all tasks. Including code data in pre- training cooldown, where high-quality datasets are up-weighted, leads to an increase of 3.6% in NL reasoning, 10.1 %% in world knowledge, and 20% in code performance relative to no cooldown. More significantly, cooldown with code beats the baseline (no cooldown) by 52.3% win-rates, where win-rates are 4.1 % higher compared to cooldown without code.
2 METHODOLOGY
We describe the details of our Pre-training Data ( 2.1), Evaluation (S 2.2), Training and Model details ($ 2.3). Figure/ shows the high-level experimental framework. Precise details for each experiment and their results are presented in Section 3.
2.1 PRE-TRAINING DATA
In this section, we describe the details of our pre-training and cooldown datasets. We aim to evaluate the role of code in pre-training, following current state-of-art practices. Hence, we consider pre- training runs that consist of two phases: 1) continued pretraining and 2) cooldown. Continued pre-training refers to training a model that initialized from a pre-trained model and trained for a fixed token budget. Cooldown (Team et al., 2023; Parmar et al., 2024) involves up- up-weighting high- quality datasets and annealing the learning rate for relatively small number of tokens during the final stages of training. This up-weighting of high-quality datasets for smaller amount of steps at the end of training can significantly boost model quality.
Text dataset. We use the SlimPajama pre-training corpus (Soboleva al., 2023) as source of natural language text data. SlimPajama is a de-duplicated, quality-filtered, multi-corpora, open- source dataset based RedPajama-I (Computer, 2023). SlimPajama consists of documents
2

---

Published as a conference paper at ICLR 2025
Evaluation
Model Initialization
Natural Language Passortig
Model
Rundom
Gode
Balanced
Weiec kinededge
Ablations
Data mixture
Data Ablations
Modal intia lastion from Gode IM
Proportions of code
Mocill State
Datasets
Codo-date coaldown
Phase pre-traniin
Teioen busdge
style/
Synthetic
Figure 1: Overview of our experimental framework: We exhaustively evaluate the impact of code by varying: 1) the proportion of code in pre-trate pre-training, 2) code quality and properties, 3) model initialization, 4) model scale, and stage of training at which code introduced. We evaluate the resulting model on wide-ranging set of tasks, including natural language reasoning, world knowledge, code, and open-ended generations.
from CommonCrawl, C4, GitHub, Books, ArXiv, Wikipedia, and StackExchange. We filter out all documents from GitHub and StackExchange to remove code and code-adjacent data sources and ensure this is text-only source. SlimPajama has total of 627B tokens. After removing all code sources, this results in our text pre-training corpus with a total 503B tokens.
Code datasets. To explore the impact of different properties of code data, use multiple sources of code in our experiments:
WEB-BASS BEB-BASED CODE DATA: For our main source of code data, we start with the Stack dataset (Kocetkov al., 2022) that used train StarCoder (Li et al., 2023a). The Stack consists of permissively licensed code data scraped from GitHub. We apply quality filters' and restrict to the top 25 programming languages based on document count 2 After all filtering steps. the size of the code-only and markup subset is 139B tokens.
MARKDOWN DATA We also separately process markup-style languages such as Markdown, CSS, and HTML. After all filtering steps, the size of this markup subset is 180B tokens.
SYNTHETIC CODE DATA: To ablate the quality of the code dataset, synthetically generated code dataset that consists of Python programming problems that have been formally verified. We treat this high-quality source of code data (See the details & 3.4). The final synthetic dataset consists of 3.2B code tokens.
proprictary
Finally, to explore different properties of code data, we include version of the code data which includes auxiliary data such GitHub commits, jupyter notebooks, StackExchange threads. For GitHub commits, and jupyter notebooks we use the datasets provided as part of the Stack (Kocetkov et al., 2022). We the version of StackExchange that part of SlimPajama (Soboleva et al., 2023). In total we have 21.4B tokens of code-adjacent data.
CODE ADIACENT DATA:
Pre-training cooldown datasets. Cooldown involves up- up-weighting higher quality datasets for the final steps of pre-training and has been found to improve performance on downstream tasks (Parmar al., 2024; Team al., 2023), partieular to impart cnstruction-folloviin capabilities. We choose a cooldown mixture comprising high-quality text, math, code, and instruct-style text datasets.
'See Appendix C.1 for details about quality filters
Reefr Appendix C.2, C.3 for the full list of programming and markup-style languages included

---

Published as a conference paper at ICLR 2025
2.2 EVALUATION
Our goal is to systematically understand the impact of code on general performance, which requires broad evaluation suite that extends to large variety of downstream tasks beyond code generation. To achieve this, we evaluate models on benchmarks that are reasonable proxies for model ability on 1) world knowledge, 2) natural language reasoning, and 3) code performance. In addition, we report win-rates as evaluated by an LLM-as-a-judge Table 1 (Appendix A) shows the full evaluation suite and their respective grouping, along with the metric used.
For World knowledge, we use benchmarks to measure knowledge memorization, retrieval, and question answering capability given context. We include Natural Questions Open (Kwiatkowski et al., 2019), and TriviaQA (Joshi et al., 2017) as the datasets. Natural language reasoning suite consists of benchmarks that involve natural language based reasoning such as Question Answer- ing, natural language inference (NLI), sentence completion, co-reference resolution, and general intelligence. We include the full list of the constituent benchmarks with references Table 1. Fi- nally, while our main focus is general performance, generation performance. For Code benchmarks, we focus on the function completion task where use Human
also want to measure any changes to code
(Chen et al., 2022) and MBPP (Austin et al., 2021).
We evaluate performance at two scales: 470M to 2.8B parameter models. At 470M scale, model capabilities are limited, thus ensure fair comparisons, we only compare benchmarks for which all models achieve scores above random similar to Muennighoff et al. (2023a); Lozhkov et al. (2024).
LLM-as-a-judge win-rates. In addition to task-specific discriminative performance, allow for a more holistic view across all performance measures, we also evaluate generative performance using LLM--s-a-jjude win-rates. This particularly valuable given recent work that has shown that as performance on open-ended generations improves, there is deterioration in traditional academic tasks (Üstün et al., 2024; Ouyang et al., 2022; Iyer et al., 2022; Muennighoff et al., 2023c). The use of LLLMs-as-a-Judge benchmarks (Fu et al., 2023; Liu et al., 2023; Chiang & yi Lee, 2023; Shimabucoro et al., 2024) has gained traction as an alternative to performing human evaluation, which tends to be laborious and expensive (Wang et al., 2023; Boubdir et al., 2023). I.LSs as evaluators compare two completions based upon detailed prompts and are reasonable proxies aligned with human preference (Üstún et al., 2024; Dubois et al., 2024).
We use the Dol English dataset (Üstün et al., 2024; Singh et al., 2024), which consists 200 hand-picked examples from the DolKy-15K dataset (Conover et al., 2023). These prompts are open- ended and capture general-purpose purpose non-code use cases making them a valuable proxy for how code impacts more fluid and often open-ended tasks. For our win-rate evaluations, we use Command-R++ as the L.LM judge. Details about the prompt are provided in Appendix D.
2.3 TRAINING AND MODEL DETAILS
We use 470M and 2.8B parameters decoder-only auto-regressive Transformer models (Radford et al., 2019) that are trained with a standard language modeling objective. We use parallel atten- tion layers. (Chowdhery et al., 2022; Wang & Komatsuzaki, 2021), SwiGLU activation (Shazeer, 2020), no biases in dense layers, and byte-pair-encoding tokenizer with a vocabulary size of 256K. All models are pre- pre-trained using Adam W (Loshchilov & Hutter, 2019) with a max sequence length of 8192, batch size of 512 and a cosine LR schedule with warmup of 1325 steps.
Infrastructure. We use TPU v5e chips (Jouppi et al., 2017) for training and evaluation. All models trained using Jax (Bradbury et al., 2018) framework. We pre-train 64 models in total. This is an enormous endeavour given the scale and computational resources required. Each pre- pre-training run for 200B tokens takes 4736 TPU-chip hours for 470M and 13824 TPU-chip-hours for 2.8B parameter models. Each cooldown run for 40B tokens takes 1024 TPU-chip hours for 470M models.
3 RESULTS AND DISCUSSION
In this section, we will report descriptions and results for each experimental variants. We systemati- cally investigate, (1) initializing LLM with code pre-trained models ($ 3.1), and (2) the impact of
4

---

Published as a conference paper at ICLR 2025
balanced-only
balanced text
code text
text-only
60
9.0
11
55
average pass@]
53.0
53.3
Accuracy
51.8
10
9.9
9.5
4.8
4.1
45
8.1
8
Reasoning
World knowledge
Code
Figure 2: Impact of initialization using code pre-trained models: Initializing model training with code pre-trained models improves reasoning and code generation compared to text-only models, where the improvement is the most when continued pre-training with high percentage text (Bal- anced-Teen Code-+Text). Note that these variants are designed to isolate the role of initialization, so do not include cooldown.
model scale ($ 3.2). (3) varying proportion of code in pre- pre-training data ($ 3.3), (4) quality and prop- erties of the code data ($ 3.4), (5) code data pre-training cooldown 3.5). Finally, we compare the resulting pre-training recipes ($ 3.6). Figure shows the key levers of our experimental design.
3.1 INITIALIZING AN LLM WITH CODE PRE-TRAINED MODELS
We explore different initializations pre-trained models to understand if using an LM with large portion of code data initialization improves the performance. These key ablations, along with their token counts, are summarized Table 2. We briefly describe below:
Text LM (TEXT-ONLY BASELINE): Pre-trained model from scratch using glorot-norma initialization (Glorot al., 2) on the text-only data for 400B tokens.
Balanced LM (BALANCED-ONLY) A model is trained with an equal ratio of code and text data (50% and 50% code) in pre-training for 400B tokens.
Ralance-initialized Text LM (BALANCED TEXT): This model is initialized with bal- anced LM (50% text and 50% code) and further pre-trained using text data for 200B tokens.
Code-initialized Text LM (CODE + TEXT): Different from other variants, this model is code dataset for 200B tokens. The
initialized with code-LM which is pre-trained code dataset contains mixture of 80% code data and 20% markup-style code data. We for another 200B tokens.*
then continually pre-train this model
Natural Language Reasoning As seen in Figure initializing with 100% code pre-trained model (code=text) has the best performance for NL Reasoning benchmarks, and closely followed by the balanced+text model. The code+tex: extt model and balanced=text model beat the text-only baseline on NL reasoning tasks by 8.8% and 8.2% relative improvement respectively. The balanced-only model improves upon the bascline by 3.2%. This shows that initialization from pre-trained model with a mix of code has a strong positive effect on NI. reasoning tasks. Further using text mix with small percentage of code for continual pre-training results in the best performance as evidenced by both the code-text and balanced+t models.
World Knowledge For World Knowledge tasks, we see that the balanced+text model has the best performance over all other variants, beating the code+text by 21% and text-only -onnoony by 1% relative improvement. This suggests that performance on world knowledge tasks depends on a more balanced data mixture for initialization and a larger proportion of text in the continual pre- training stage. Overall, code data still beneficial compared text-only pre-training for world knowledge tasks.
'We use 10% of code in text mixture data during continual pre-training of code-initialized models (ba anced=text, code+text) avoid full distribution shift and maintain the henefits of code.

---

Published as a conference paper at ICLR 2025
balanced-only
balanced text
code text
470M
2.8B
60
25
30
21.8
58
57.9
27.1
25.6
20
25
23.6
average pass@1
Accuracy
56
54.5
55.8
20
15
12.4
11.7
54
15
10
53.0
52
51.8
10
8.1
9.9
9.5
50
5
NL Reasoning
World Knowledge
Code
Figure 3: Impact of model scale on different tasks. We observe that scale provides pronounced gains across tasks of up-to 2.7x increase, however the overall trend remains the same across scales showing consistency of findings across model sizes.
Trade-offs between NI tasks and code generation For code generation, balanced-onl] achieves the best performance, where we see a 46.7% and 54.5% relative improvement over balanced+text and code+text. This is expected as balanced only includes 50% code throughout pre-training. However, this model trades off better code generation with lower perfor- mance in NL tasks. code-text and ba balanced=text achieves 2.9% and 2.3% relative increase in reasoning, and 17.3% and 22.2% relative increase world knowledge respectively compared balanced-only..
Generative quality win-rates comparison Additionally, we compare the generative performance of each code variant (code=text. and ba anced-only against the text-onl, model. We report win-rates and observe that the presence of code has a strong positive impact on generation quality. Both code+text and balanced onnly) models beat the text text-only variant by 6.6% difference in win-loss rates. We again note that Polly-200-Englinn win-rate calculation curated to reflect open ended questions and non-code evaluation. This confirms that code data in the pre-training mix does not only improves reasoning but also helps the model produce better quality generations."
evaluation set we use for
3.2 IMPACT OF SCALE
To understand if the findings of Section 3.1 transfer to larger models, train 2.8B parameters models with the same token budget following the same model variants at 470M seale. Figure shows the results of 2.8B models in comparison with 470M results.
Comparison between 2.8B and 470M models Scaling model size to 2.8B enables higher perfor- mance for all model variants in all task categories, compared to 470M results. In terms of average performance across NL reasoning and world knowledge, balanced+texx model benefits from scaling-up by 33.1% % increase relative to the same model with 470M size. The improvement for code-text and balanced-only are 31.7% and 30% relative increase.
We find that the improvements in NL reasoning are relatively modest with 5.3%, 9.2%, and 5.2% relative gains for balanced#text: ext, code-text ext, and balanced-on world knowledge and code performance nearly triples for all the model variants. In particular, 2.8B balanced=text results increase by 2.7x in world knowledge and 2.5x in code evaluation compared to 470M
respectively. However,
Trends between model variants in 2.8B Notably, in terms of initialization with code pre-trained models, the same trends seen in 470M parameter seale hold at 2.8B models. code+text and balance selanced+teett models improve over ba anced models by 6.9% and 6.1% relative gain, how- ever, fall significantly behind in code generation performance with 43.1% and 46.3% relative drop. These results show that the trade-off between NL tasks and code generation increases with the model size. Overall our experiments scaling to a larger size shows that our results hold and are consistent with the trends observe at 470M parameter ablations.
We include the extended Win-rates for these experiments in Appendix

---

Published as a conference paper at ICLR 2025
Natural language Reasoning
World knowledge
52.5
10
10.0
AccuraCy
50.0
7.5
47.5
45.0
2.5
50
Increasing Proportion of Code
Figure 4: Impact of the proportion of code in pre-training for different tasks: We observe that as the code proportion of pre-training increases, the performance on code tasks increases linearly. In contrast, there more sensitivity for NL reasoning and World knowledge tasks and an optimal range of code proportions where benefits are most tangible. Model size is 470M parameters and trained for 200B tokens.
3.3 CODE DATA PROPORTION IN PRE-TRAINING
In these experiments, we ablate the proportions of code data in the pre-training mixture to understand what is the optimal amount of code to maximize performance on non-code tasks. Here, we focus on the first phase of pre-training with random initialization. We train six models for 200B tokens with increasing code proportions: 0%, 25%, 50%, 75%, 90%, and 100%. The remaining proportion filled with text data. For each variant, we train a new model independently in order to carefully ablate the impact of varying code proportions.
Natural Language Reasoning and World Knowledge For NL Reasoning, as the amount of code increase in performance compared to text-only (0% code) model.
increases, in Figure 4 The best performance from a model with 25% code and 75% text, with a 3.4% relative improve- ment over model with 0% code. While performance is maintained up 75% code, it starts to rapidly erode at higher proportions with a sharp relative drop of 18.3% when the model is trained on 100% code compared model with no code.
For World Knowledge tasks, we see an inverse relationship with increasing the amount of code. As seen in Figure 4 middle inset, there is slight relative drop of 3.4% 25% code and this relative drop worsens to 31% at 75% code compared to the no-code model. The fully code model (100% code) unable to perform world knowledge task (86% drop relative to text-only) as there are no data sources to acquire the required knowledge in the pre-training mix.
Performance on Code For code evaluation, there a linear increase in performance as the amount of code increases, with the best model being a code-only model. As observable in Figure 4 right inset, the 100% code leads to a 2.6x increase in code benchmarks compared to the 25% code model. As expected, for the model with 0% code, the average pass @ score drops to 0.
3.4 INFLUENCE OF CODE QUALITY AND PROPERTIES ON GENERAL PERFORMANCE
In this section, we investigate the properties of code data by varying its quality and composition. We study this firstly (a) from the perspective of training from scratch, we want to isolate the exact effects of different properties of code data. Secondly (b), we incorporate the best variant of the code data (high-quality synthetic code), in our continual pre-training experiments to see if the impact of the code quality transfer. We report performance NI reasoning and Code tasks.
We study the effect of the following properties: (1) MARKUP-STYLEE DATA: we separate markup- style programming languages ($ 2.1) from the rest of web-based code (Appendix C.3). We replace 20% of code-only tokens with markup-style tokens. (2) CODE ADJACENT DATA: Instead of us- ing purely web-based code data, replaced 15% percentage of code tokens with code-adjacent datasets GitHub issues (5%), StackExchange (5%) and Jupyter Notebooks (5%), resulting a code-ad jadent model. (3) CODE QUALITY: To control the quality of the code, replaced 10% of existing code tokens with synthetically generated high-quality code dataset. The remain- ing proportions of web- web-based code data are kept the same, resulting in code -ynhh model.

---

Published as a conference paper at ICLR 2025
code
code + markup
code+ cadjaadl
code + synth
balanced text
balanced + synth text
46
16
53.5
53.0
53.0
4.8
Accuracy
42.4
52.5
42
11.5
3.4
41.0
10.7
52.0
40
39.9
10
51.5
2
Reasoning
Code
NL Reasoning
Code
(a) Code-only Pre-training
(b) Continual Pre-training
Figure 5: Impact of using different properties of code data: (a) As the most impactful code data source, synthetically generated high-quality code improves NL reasoning and code performance for code pre- pre-training. (b) These improvements with synthetically generated high-quality code data also transfer the continual pre-training setting. All models are of size 470M parameters.
Code-only pre-training We compare the above variants to a model that trained only on web- based code data (code) from the stack dataset (Kocetkov al., 2022), which forms baseline model. All the variants are pre-trained using the same amount of tokens (200B) for fair comparison.
In Figure 5a, evaluate the impact of code quality and code composition. We observe that across all variants, including diverse code sources and also synthetic code leads to gains in natural language performance relative to code, however, only synthetically generated code improves the code bench- marks. We relate this to our code evaluation where we measure performance in python, thus different programming languages or code-adjacent data slightly decrease the results. Here, code umarkup and codetad a cent. leads to 2.8% and 6.3% relative improvement in NI. reasoning compared to code (web-code-only), but cause 15.7% 7% and 9.4% drop code evaluation.
Our synthetic code data (sode+synth)) is the most impactful ablation. It is particularly impres- sive given its relatively small share of the overall dataset. Despite small weighting of 10%, the inclusion of synthetic data leads to relative improvements of 9% on NL reasoning, and 44.9% on code benchmarks compared to the baseline of web-code-only. We note that the lifts observed for synthetic data even more impressive given the limited amount of synthetic data available com- pared to code-adjacent data (3.2B tokens vs 21.4B tokens) or code+markup data (3.2B tokens vs 40B tokens), and the weighting during pre-training allocation (10% 15% 20% for synthetic data, code-adjacent, code-markup respectively). This suggests a key future lever of improvement is increasing the proportion of such high-quality code data sources.
Continual pre-training Here, based on the findings from code-only pre-training, we incorporated code +yynth into best continual pre-training variant (balanced+ synt We com- pare this against the same variant without synthetic code data bbalanced-tees to evaluate the benefits of synthetic data. We use the same amount code and text tokens in these experiments.
As shown in Figure 5b, slllanee++ynne synth-text achieves 2% and 35% relative improvement over bxlanced+text in NL reasoning and code, respectively. This further confirms that even small percentage of high-quality code data, not only improves performance in code pre-training but also increases code and non-code performance after continual pre-training with data.
3.5 CODE PRE-TRAINING COOLDOWN
In this section, we evaluate the impact of code at the final stage of pre-training, Here, we consider cooldown, where we up-weight high-quality text, math, code, and instruct-style datasets. We change the learning rate schedule from cosine-based to linear annealing with final learning rate of 1e - 6. We evaluate the impact code in cooldown by comparing 3 models: a pre-trained model before cooldown, cooldown without code data, and cooldown with 20% code data. For our pre-trained model, we use salanced=text as it is our best pre-trained variant. We preserve the same token budget across variants 40B tokens which is 10% of the token budget of the pre-trained model.
Impact of code used during cooldown different tasks In Figure 6a, we evaluate the impact of code in cooldown model performance in NI. Reasoning, world knowledge, and code benchmarks.
8

---

Published as a conference paper at ICLR 2025
cooldown
coordown wil
coolcoon
50
57
11.5
20
56
11.0
14.9
201
3.0
70
9.5
10
52
9.0
8.5
coddoner
50
NL Reasoning
8.0
World kndw ooge
Code
Downstream tasks
(h) Generative win-rates
Figure 6: Impact of code data in pre-training cooldown: Including code data in the cooldown phase improves downstream relative to cooldown with no code. All cooldown variants benefit for downstream tasks and especially generative quality. We find the largest gains from cooldown with code, with the highest win-rates of 52.3 % over a model with no cooldown.
Across tasks, find that cooldown with code data is most beneficial with 3.6%, 10.1%, and 20% in NL reasoning, world knowledge, and code relative to the model without cooldown.
In contrast, find that cooldown without code does provide any increases for both NI. reason- ing and Code, while providing a relative improvement of 3.1 % in World Knowledge tasks compared no cooldown, showing the critical role of code data in also cooldown phase of pre-training.
Generative win-rates after cooldown As expected, cooldown has a significant impact on gener- ative performance measured by win-rates (seen in Figure 6b). This is because we up-weight high-quality data sources in pre-training mix including instruction-style datasets such as Dolly v2 (Conover al., 2023). Both cooldown variants (cooldown w/o code, cooldown w/ code) beat no- pol-cooldoww model by large win-rates (48.2% and 52.3%) as seen in Figure 6b. Compar- ing the cooldown variants, including code leads an additional 4. % generative win-rates against no no-cool down compared to cooldown without code.
3.6 COMPARING PRE-T THEETININING RECIPES
Considering all our experiments, we summarize our findings and recommend recipes for pre-training with code data. Table 2 (Appendix B) shows the different variants along with pre-training phases.
Best recipe for natural language tasks As seen Sections 3.1, 3.3, and 3.5, including code in all phases of pre-training provides improvements across all task categories. When looking the final recipes, we find that salanced+teett model followed by cooldown that includes code data corresponds the best overall performance in natural language tasks considering NL reasoning. world knowledge, and generative performance. Notably this model achieves the highest generative win-rates with 37.7% 33.7 against text-only as shown in Figure
Best recipe for code performance Among complete recipes shown in Table 2, balanced-on only provides the best performance in code benchmarks. This model achieves 20% relative gain com- pared to second best code-text and 55% relative gain compared to balanced+text. How- ever, ba lanced-on
falls behind in natural language performance by 2.5% relative difference and 5.0% win-rate difference (vs text-only both compared to ba balanced-text.
Including code in phases of pre-training is beneficial across our three task categories and gener- ative performance. Our recommendation for the best overall performance is to include balanced mixture of code and text data during pre-training from scratch ($ 3.3), use a relatively lower code percentage during continual pre-training ($ 3.1), and include code data into cooldown mixture. Fur- ther, we recommend including high-quality code data during all phases of pre-training ($ 3.4).
RELATED WORK
Understanding the impact of pre-training mixes Several works have studied the effects of data age, quality, toxicity and domain of pre-training data (Longpre al., 2023; Üstün et al., 2024). Sev- eral works have looked at the impact of filtering (Raffel et al., 2020: Rae et al., 2021; Penedo et al., 2023), de-duping (Zhang et al., 2022) and data pruning (Lozhkov al., 2024; Marion et al., 2023;

---

Published as a conference paper at ICLR 2025
37.7
Generative Winrate
34.7
32.7
33.77
31.7
30
27.6
25
20
text-only - cooldown
text-only cooldown
text-only cooldown code text cooldown
balanced-only cooldown
balanced cooldown
ties
Figure Generative performance as measured by win-rates for variants with full-cooldown.
Chimoto et al., 2024; Boubdir et al., 2023). Furthermore, several works have considered the role of synthetic data at improving performance (Shimabucoro et al., 2024; Dang et al., 2024; Aakanksha al., 2024) and helping bridge the gap in performance between open weights and proprietary mod- els (Gunasekar et al., 2023; Li et al., 2023b). In contrast to our work which focuses explicitly on understanding the role of code, these studies focus on characteristics of training data as a whole.
Understanding the role of code Including code in the pre-training data mixture, even for mod- els not specifically designed for code, has been common practice in LLMs pre-training (Dubey et al., 2024; Gemini-Tean et al., 2024; Groeneveld et al., 2024). In addition to serving the pop- ular use case code completion and generation (Chen al., 2021), previous studies suggest that the addition of code improves the performance of LLMs on various NLP tasks, such as entity link- ing (Kim et al., 2024) and commonsense reasoning (Madaan al., 2022b)), mathematical reasoning tasks (Liang et al., 2022; Madaan et al., 2022a; Gao et al., 2023; Shao et al., 2024), and general reasoning capabilities (Muennighoff et al., 2023a; Fu & Khot, 2022; Ma et al., 2023). Muennighoff et al. (2023b) demonstrated Python code data can be used to improve pretraining performance. They focused on a low-resource pre-training regime with limited data and an evaluation set-up limited to perplexity evaluations. Zhang al. (2024) investigated the impact of code on LLMs' internal rea- soning capability across various tasks and model families. They only focus on the effect of code in the supervised fine-tuning stage (SFT) primarily measuring the impact reasoning. Zhu et al. (2024) report the performance of their DeepSeek- PeasSeek-Code---e V2 models on General Natural Language benchmarks. They compare chat and instruct models, and do not investigate different phases of pre-training and properties of code.
To the best of our knowledge, this work is the first study that presents a thorough investigation of the impact of code in pre-training non-code tasks. Our experiment spans several axes and exhaustive evaluation suite, with costly ablations scale including model initialization strategies, different proportions and properties of code data, and model scales.
5 CONCLUSION
We perform first-of-its-ind systematic study answer "what is the impact of code data used in pre-training on large variety of downstream tasks beyond code generation". focus, not just on code performance but downstream natural language performance, well generative quality using LLM-as-a-judge win-rates. We conduct ablations that look at initialization, proportions of code, quality and properties of code, and role of code in pre-training cooldown. We find across all scales experiments that code provides critical improvements to performance on non-code tasks. Compared text-only pre-traiee pre-training, for best variant, the addition of code results in relative increase of 8.2% in natural language (NL) reasoning, 4.2% in world knowledge, 6.6% improvement in generative win-rates, and 12x boost in code performance respectively. Further performing cooldown with code, improves 3.6%, 10.1%, and 20% in NL reasoning, world knowledge, and code relative to the model before cooldown and leads 52.3% generative win-rates. Finally, find that adding small amount of high-quality synthetic data can have outsized impact on both NI. reasoning (9% relative increase) and code performance 44.9% relative increase).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Boldfarb-Tarrany Julia Kreutzer, Marzich Fadaee, and Sara Hooker. The multilingual alignment prism: Aligning global and local prefer- ences to reduce harm, 2024. URL https: /wwx/aavvv.o//////..a org/ abs/ / /20/2446...2 18682.
Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Se- bastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan Gomez, Phil Blunsom, Marzieh Fadace, Ahmet Üstün, and Sara Hooker. Aya 23: Open weight releases to further multilingual progress, 2024. URL https:t////c /ttp:///aaaivviieeee org/ abs 24 2450444440 15032.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https: / http//rxivv.rr.. abs 2108 07732.
Meriem Boubdir, Edward Kim, Beyza Ermis, Marzich Fadace, and Sara Hooker. Which prompts make the ?ifference? data prioritization for efficient human Ilm evaluation, 2023. URL https:
//arxiv. org abs /tabbs/23100 2310. 12.311111444... 14424.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github com/ google/jes jax.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv, abs/2005. 14165, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021. URL https: //tpx//axvvvv///a////a//eea/.awa org/ www.aa/a/a0 033744 4.
Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding the mixture- -f-exers layer in deep learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: /epenrevviii ew net/ / forum? d=MaY daMaazuugMgggggg ZugDnQV.
Cheng-Han Chiang and Hung yi Lee. Can large language models be an alternative to human evalu- ations?, 2023.
Everlyn Asiko Chimoto, Jay Gala, Orevaoghene Ahia, Julia Kreutzer, Bruce A. Bassett, and Sara Hooker. Critical learning periods: Leveraging early training dynamics for efficient data pruning. 2024. URL https:
/ttp///xxvv/v.///////e///..a...a org/ abs abs//ab5. 241.
Eunsol Choi, He He. Mohit Iyyer. Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. QuAC: Question answering in context. In Proceedings of the 2018 Con- ference Empirical Methods in Natural Language Processing, pp. 2174-2184, Brussels, Bel- gium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/ D18- 1241. URL https: /aa//aaaattttt woatthooogyy..oom www. 8
11

---

Published as a conference paper at ICLR 2025
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyyashchenko, Joshua Maynez, Abhishek Rao. Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph. Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstert Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022. URL https: arxiv. org wttg/aa/////eeeeee/o///eaaeaaa// wtwpaa/a/222.// 02311.
Christopher Clark, Kenton Lee, Ming-Wei Chang. Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ:
Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2924-2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/
v1/N19- 1300. URL https::p///......... / aa/.cccacccaaaa htt///aaaanntlcooeeoeeeeee org /Ngg/N19 9-1300.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXivv//80..05457vll 2018.
Together Computer. Redpajama: an open dataset for training large language models, 2023. URL https://git thtb://./gttutuut//eee com/ www.ccoootttttttceee
/RedP a ama-- Daama---tttt Da t.a
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tumed Ilm, 2023. URL https: /www databri cks. wttp:///oooo/o////////////////// 12//olly-firr open cowmercieccooo.o...o.m www.mmeerrlllllllaallaaao ly- inserre--nnstturttootottne
1mm.
John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. Rlhf can speak many languages: Unlocking multilingual preference optimization for Ilms, 2024. URI. https: / wtw.g//////////e 02552.
de Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentank:: Inves- tigating projection in naturally occurring discourse, 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux. Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar. Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yers Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Man- nat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova,
12

---

Published as a conference paper at ICLR 2025
Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajiwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sum- baly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa. Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Kam, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguct, Virginie Do, Vish Vogeti, Vladan Petro- vic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho. Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt. David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Di- ana Liskovich, Didem Foss, Dingkang Wang. Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Flaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Es- teban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Flo- rez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Her- man, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Igor Molybog. Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan. Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhic, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang. Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael Seltzer, Michal Valko, Michelle Restrepo. Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy- ata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa- tadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lind- say, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar. Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho,
13

---

Published as a conference paper at ICLR 2025
Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler. Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihaileseu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xi- aocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models, 2024. URL https://arxte org/ alos awws/247 //1./22777.2222.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani. Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2024. URL https://arxiii org/ abs/2305. 14387.
Hao Fu, Yao; Peng and Tushar Khot. How does gpt obtain its ability? tracing emergent abilities of language models to their sources. Yao Fu's Notion, Dec 2022. URL notion.
?pvs=25.
Jinlan Fu, See-Kiong Ng. Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire, 2023.
Yifu Gao, Yongquan He, Zhigang Kan, Yi Han, Linbo Qiao, and Dongsheng Li. Learning joint structural and temporal contextualized knowledge embeddings for temporal knowledge graph completion. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 417- 430, Toronto, Canada, July 2023, Association for Computational Linguistics. URL https: // //aclanthology wrgg 2023.023322222 findings sorddings-ac.220 ac1.28.
Gemini-Team, Rohan Anil, Sebastian Borgeaud, Bean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai. Anja Hauth, Katic Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lilli- crap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hen- nigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Jack Krawczyk, Cosmo Du Ed Chi. Heng-Tze Tze Cheng. Eric Ni, Purvi Shah, Patrick Kane, Betty Chan, Manaal Faruqui, Aliaksei Severyn, Hanzhao Lin, YaGuang Li. Yong Cheng. Abe Ittycheriah, Mahdis Mahdieh, Mia Chen, Pei Sun, Dustin Tran, Sumit Bagri, Balaji Lakshminarayanan, Jeremiah Liu, Andras Orban, Fabian Güra, Hao Zhou, Xinying Song, Aurelien Boffy, Harish Ganapa- thy, Steven Zheng. HyunJeong Choe, Ágoston Weisz, Tao Zhu, Yifeng Lu, Siddharth Gopal, Jarrod Kahn, Maciej Kula, Jeff Pitman. Rushin Shah. Emanuel Taropa, Majd Al Merey, Mar- tin Baeuml, Zhifeng Chen, Laurent El Shafey, Yujing Zhang. Olcan Sercinoglu, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anais White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lu- cas Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura Culp. Lev Proleev, Yi Luan, Xi Chen, James Lottes, Nathan Schucher, Federico Lebron, Alban Rrustemi, Natalie Clay, Phil Crone, Tomas Kocisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rae, Han Lu, Laurent Sifre, Marcello Maggioni, Fred Alcober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun Ahuja, Gaurav Singh Tomar, Evan Senter, Martin Chadwick, Ilya Kornakov, Nithya Attaluri, Iñaki Iturrate, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen. Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grimstad. Ale Jakse Hart- man, Xavier Garcia, Thanumalayan Sankaranaray Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adrià Puigdomènech Badia, David Re- itter, Mianna Chen, Jenny Brennan, Clara Rivera, Sergey Brin, Shariq Iqbal, Gabriela Surita, Jane Labanowski, Abhi Rao, Stephanie Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Ravi Addanki, Antoine Miech, Annie Louis, Denis Teplyashin, Geoff Brown, Elliot Catt, Jan Bala- guer, Jackie Xiang, Pidong Wang. Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Gana- pathy, Smit Sanghavi, Ajay Kannan. Ming- Wei Chang, Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang, Juliette Love, Junwhan Ahn, Dawn Bloxwich, Kehang Han, Peter Humphreys, Thibault
14

---

Published as a conference paper at ICLR 2025
Sellam, James Bradbury. Varun Godbole, Sina Samangooei, Bogdan Damoe, Alex Kaskasoli, Sébastien M. R. Arnlld, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanburn, Srivatsan Srinivasan, Hyeontack Lim, Sarah Hodkinson, Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Paine, Jian Li, Yujia Li, Minh Giang. Alexander Neitz, Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery, Dipan- jan Das, Dominika Rogozińska, Vitaliy Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu, Raoul de Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Albin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo, Craig Swanson, Dessie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Lan- don, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai Giménez, Legg Yeung, James Keeling, Petko Georgiev, Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputro, Kiran Vodrahalli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fer- nando, Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang, Fan Yang. Shuo yin Chang, Paul Komarek, Ross Mcllroy, Mario Lučić, Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakeri, Christina Butterfield, Justin Chung. Paul Kishan Rubenstein, Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Karel Lenc, Timothy Chung, Aedan Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang. Joshua Maynez, Mary Phuong. Taylor Tobin, Andrea Tacchetti, Maja Trebacz. Kevin Robinson, Yash Katariya, Sehastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora Aroyo, Ambrose Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara, Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu. Richard Powell, Vijay Bolina, Mariko linuma, Polina Zablotskaia, James Besley, Da- Woon Chung. Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su. Martin Polacek, Raphaël Lopez Kaufman, Simon Tokumine, Hexi- ang Hu, Elena Buchatskaya, Yingjie Miao, Mohamed Elhawaty, Aditya Siddhant, Nenad Toma- sev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashraf, Aurko Roy, Zizhao Zhang, Ada Ma, Angelos Filos, Milos Besta, Rory Blevins, Ted Klimenko, Chih-Kuan Yeh, Soravit Chang- pinyo, Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Sholto Douglas, Rajkumar Samuel, Mingqiu Wang. Sophia Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjösund, Sébastien Cevey, Zach Gleicher, Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, Léonard Hussenot, Livio Baldini Soares, Kate Baumli, Michael B. Chang, Adrià Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kass- ner, Subhrajit Roy, Ethan Dyer, Víctor Campos Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy, Elspeth White, Basil Mustafa, Oran Lang, Abhishck Jindal, Sharad Vikram, Zhitao Gong, Sergi Caelles, Ross Hemsley, Gregory Thortoon, Fangxiaoyu Feng. Wojciech Stokowiec, Ce Zheng, Phoebe Thacker, Çağlar Unlü, Zhishuai Zhang, Mohammad Saleh, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez, Tom Kwiatkowski, Samira Daruki, Keran Rong, Allan Dafoe, Nicholas FitzGerald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pel- lat, Vladimir Feinberg, James Kebon-Kerr, Tara Sainath, Maribeth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Hasson, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Paganini, Jean-Baptiste Lespiau, Alexandre Moufarek, Samer Has- san, Kaushik Shivakumar, Joost van Amersfoort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Sheahan, Vedant Misra, Cheng Li. Nemanja Rakićević, Mostafa Dehghani, Fangyu Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlie Chen, Sidharth Mudgal, Romina Stella, Kevin Brooks, Gautam Vasudevan, Chenxi Liu, Mainak Chain, Nivedita Melinkeri, Aaron Cohen, Venus Wang, Kristie Seymore, Sergey Zubkov, Rahul Goel, Summer Yue, Sai Krishnakumaran, Brian Albert, Nate Hurley, Motoki Sano, Anhad Mohananey, Jonah Joughin Egor Filonov, Tomasz Kepa, Yomna Eldawy, Jiawern Lim, Rahul Rishi, Shirin Badiezadegan, Taylor Bos, Jerry Chang, Sanil Jain, Sri Gayatri Sundara Padmanabhan, Subha Puttagunta, Kalpesh Krishna, Leslie Baker, Nor- bert Kalb, Vamsi Bedapudi, Adam Kurzrok, Shuntong Lei, Anthony Yu, Oren Litvin, Xiang Zhou, Zhichun Wu, Sam Sobell, Andrea Siciliano, Alan Papir, Robby Neale, Jonas Bragagnolo. Tej Toor, Tina Chen, Valentin Anklin, Feiran Wang, Richie Feng, Milad Gholami, Kevin Ling, Lijuan
15

---

Published as a conference paper at ICLR 2025
Liu, Jules Walter, Hamid Moghaddam, Arun Kishore, Jakub Adamek, Tyler Mercado, Jonathan Mallinson, Siddhinita Wandekar, Stephen Cagle, Eran Ofek, Guillermo Garrido, Clemens Lom- briser, Maksim Mukha, Botu Sun, Hafeezul Rahman Mohammad, Josip Matak, Yadi Qian, Vikas Peswani, Pawel Janus, Quan Yuan, Leif Schelin, Oana David, Ankur Garg. Yifan He, Oleksii Duzhyi, Anton Algmyr, Timothée Lottaz, Qi Li, Vikas Yadav, Luyao Xu, Alex Chinien, Rakesh Shivanna, Aleksandr Chuklin, Josie Li. Carrie Spadine, Travis Wolfe, Kareem Mohamed, Sub- habrata Das, Zihang Dai, Kyle He, Daniel von Dincklage, Shyam Upadhyay, Akanksha Mau- rya, Luyan Chi, Sebastian Krause, Khalid Salama, Pam G Rabinovitch, Pavan Kumar Reddy M, Aarush Selvan, Mikhail Dektiarev, Golnaz Ghiasi, Erdem Guven, Himanshu Gupta, Boyi Liu, Deepak Sharma, Idan Heimlich Shtacher, Shachi Paul, Oscar Akerlund, Trançois-Xavier Aubet, Terry Huang. Chen Zhu, Eric Zhu, Elico Teixeira, Matthew Fritze, Francesco Bertolini, Liana- Eleonora Marinescu, Martin Bölle, Dominik Paulus, Khyatti Gupta, Tejasi Latkar, Max Chang, Jason Sanders, Roopa Wilson, Xuewei Wu, Yi-Xuan Tan, Lam Nguyen Thict, Tulsee Doshi, Sid Lall, Swaroop Mishra, Wanming Chen, Thang Luong, Seth Benjamin, Jasmine Lee, Ewa Andrejezuk, Dominik Rabiej, Vipul Ranjan, Krzysztof Styre, Pengcheng Yin, Jon Simon, Mal- colm Rose Harriott, Mudit Bansal, Alexei Robsky. Geoff Bacon, David Greene, Daniil Mirylenka, Chen Zhou, Obaid Sarvana, Abhimanyu Goyal, Samuel Andermatt, Patrick Siegler. Ben Horn, Assaf Israel, Francesco Pongetti, Chih-Wei "Louis" Chen, Marco Selvatici, Pedro Silva, Kathie Wang, Jackson Tolins, Kelvin Guu, Roey Yogev, Xiaochen Cai, Alessandro Agostini, Maulik Shah, Hung Nguyen, Noah Ô Donnaile, Sébastien Pereira, Linda Friso, Adam Stambler, Adam Kurzrok, Chenkai Kuang, Yan Romanikhin, Mark Geller, ZJ Yan, Kane Jang, Cheng-Chun Lee, Wojciech Fica, Eric Malmi, Qijun Tan, Dan Banica, Damiel Balle, Ryan Pham, Yanping Huang, Diana Avram, Hongzhi Shi, Jasjot Singh, Chris Hidey, Niharika Ahuja, Pranab Saxena, Dan Doo- ley. Srividya Pranavi Potharaju, Eileen O'Neill, Anand Gokulchandran, Ryan Foley, Kai Zhao, Mike Dusenberry, Yuan Liu, Pulkit Mehta, Ragha Kotikalapudi, Chalence Safranek-Shrader An- drew Goodman, Joshua Kessinger, Eran Globen, Prateek Kolhar, Chris Gorgolewski, Ali Ibrahim, Yang Song, Ali Eichenbaum, Thomas Brovelli, Sahitya Potluri, Preethi Lahoti, Cip Bactu, Ali Ghorbani, Charles Chen, Andy Crawford, Shalini Pal, Mukund Sridhar, Petru Gurita, Asier Mu- jika, Igor Petrovski, Pierre-Louis Cedoz, Chenmei Li, Shiyuan Chen, Niccolò Dal Santo, Sid- dharth Goyal, Jitesh Punjabi, Karthik Kappaganthu, Chester Kwak, Pallavi LV, Sarmishta Velury, Himadri Choudhury, Jamie Hall, Premal Shah, Ricardo Figueira, Matt Thomas, Minjie Lu, Ting Zhou, Chintu Kumar, Thomas Jurdi, Sharat Chikkerur, Yenai Ma, Adams Yu, Soo Kwak, Victor Ähdel, Sujeevan Rajayogam, Travis Choma, Fei Liu, Aditya Barua, Colin Ji, Ji Ho Park, Vincent Hellendoom, Alex Bailey, Taylan Bilal, Huanjie Zhou, Mehrdad Khatir, Charles Sutton, Woj- ciech Rzadkowski. Fiona Macintosh, Konstantin Shagin, Paul Medina, Chen Liang, Jinjing Zhou, Pararth Shah, Yingying Bi, Attila Dankovies, Shipra Banga, Sabine Lehmann, Marissa Bredesen, Zifan Lin, John Eric Hoffmann, Jonathan Lai, Raynald Chung, Kai Yang, Nihal Balani, Arthur Bražinskas, Andrei Sozanschi, Matthew Hayes, Héctor Fernández Alcalde, Peter Makarov, Will Chen, Antonio Stella, Liselotte Snijders, Michael Mandl, Ante Kärrman, Pawel Nowak, Xinyi Wu, Alex Dyck, Krishnan Vaidyanathan, Raghavender R, Jessica Mallet, Mitch Rudominer, Eric Johnston, Sushil Mittal, Akhil Udathu, Janara Christensen, Vishal Verma, Zach Irving, Andreas Santucci, Gamaleldin Elsayed, Elnaz Davoodi, Marin Georgiev, Ian Tenney, Nan Hua, Geoffrey Cideron, Edouard Leurent, Mahmoud Alnahlawi, Ionut Georgescu, Nan Wei, Ivy Zheng, Dylan Scandinaro, Heinrich Jiang, Jasper Snoek, Mukund Sundararajan, Xuezhi Wang. 7ack Ontiveros, Itay Karo, Jeremy Cole Vinu Rajashekhar, Lara Tumeh, Eyal Ben- David, Rishub Jain, Jonathan Uesato, Romina Datta, Oskar Bunyan, Shimu Wu, John Zhang, Piotr Stanczyk. Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew Johnson, Adam Paszke, Chung-Cheng Chiu, Jaume Sanchez Elias. Afroz Mohiuddin, Faizan Muhammad, Jin Miao, Andrew Lee, Nino Vieil- lard, Jane Park, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abhijit Karmarkar, Zhe Dong, Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens, William Isaac, Geoffrey Irving, Edward Loper, Michael Fink, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Ivan Petrychenko, Zhe Chen, John- son Jia, Anselm Levskaya, Zhenkai Zhu, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao, Javier Snaider, Norman Casagrande, Evan Palmer, Paul Suganthan, Alfonso Castaño, Irene Giannoumis, Wooyeol Kim, Mikotaj Rybiński, Ashwin Sreevatsa, Jennifer Prendki, David So- ergel, Adrian Goedeckemeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Di- ana Gage Wright, Yawen Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Le, Lu Li, Chimezie Iwuanyanwu, Lu Liu, Kevin Ramirez, Andrey Khorlin, Albert Cui, Tian LIN, Mar- cus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Ginger Perng, Elena Allica Abellan, Mingyang Zhang, Ishita Dasgupta, Nate Kushman, Ivo Penchev, Alena Repina, Xihui Wu, Tom
16

---

Published as a conference paper at ICLR 2025
van der Weide, Priya Ponnapalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang. Jeff Piper, Nathan Ie, Rama Pasumarthi, Nathan Lintz, Anitha Vijayakumar, Daniel Andor, Pedro Valenzuela, Minnie Lui, Cosmin Paduraru, Daiyi Peng, Katherine Lee, Shuyuan Zhang, Somer Greene, Due Dung Nguyen, Paula Kurylowicz, Cassidy Hardin, Lucas Dixon, Lili Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang, Achintya Singhal, Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolukbasi, Orgad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffnny, Ken Franko, Anna Bu- lanova, Rémi Leblond, Shirley Chung, Harry Askham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Alek Dimitriev, Han- nah Forbes, Dylan Banarse. Zora Tung, Mark Omernick, Colton Bishop. Rachel Sterneck, Rohan Jain, Jiawei Xia, Ehsan Amid, Francesco Piccinno, Xingyu Wang. Praseem Banzal, Daniel J. Mankowitz, Alex Polozov, Victoria Krakovna, Sasha Brown, MohammadHossein Bateni, Den- nis Duan, Vlad Firoiu, Meghana Thotakuri, Tom Natan, Matthicu Geist, Ser tan Girgin, Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee- Lee-Thorp, Christopher Yew, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Kathy Wu, David Miller, Nico- las Sonnerat, Denis Vnukov, Rory Greig, Jennifer Beattie, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex Korchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu, Fan Yang. Rui Zhu, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Neje Trdin, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting Xue, Chen Elkind, Oliver Woodman, John Carpenter, George Papamakarios. Rupert Kemp, Sushant Kafle, Tanya Grunina, Rishika Sinha. Alice Talbert, Diane Wu, Denese Owusu-Afriyie, Cosmo Du, Chloe Thortoo,, Jordi Pont-Tuset, Pradyumna Narayana, Jing Li, Saaber Fatehi, John Wieting, Omar Ajmeri, Be- nigno Uria, Yeongil Ko, Laura Knight, Amélie Héliou, Ning Niu, Shane Gu, Chenxi Pang. Yeqing Li, Nir Levine, Ariel Stolovich, Rebeca Miitammrra-FFrnanderr Sonam Goenka, Wenny Yustalim, Robin Strudel, Ali Elqursh, Charlie Deck, Hyo Lee, Zonglin Li, Kyle Levin, Raphael Hoffmann, Dan Holtmann- Rice, Olivier Bachem, Sho Arora, Christy Koh, Soheil Hassas Yeganeh, Siim Pöder, Mukarram Tariq, Yanhua Sun, Lucian Ionita. Mojtaba Seyedhosseini. Pouya Tafti, Zhiyu Liu, Anmol Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszez, Lily Wang, Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Vinod Koverkathu, Christo- pher A. Choquette- Sanaz Bahargam, Rob Willoughby, David Gaddy, Guillaume Desjardins, Marco Cornero, Brona Robenek, Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Rivière, Alanna Walton, Clément Crepy, Alicia Parrish, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Eri Latorre-Chimoto, Hanna Klimezak- Piiiriik--Pu David Bridson, Dario de Cesare Tom Hudson, Piermaria Mendolicchio, Lexi Walker, Alex Morris, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe. Anastasia Petrushkina, Tom Ducrig, Antonio Sanchez, Steve Yad- lowsky, Amy Shen, Amir Globerson, Lynette Webb, Sahil Dua, Dong Li, Surya Bhupatiraju. Dan Hurt, Haroon Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Anuj Khare, Shreyas Ram- mohan Belle, Lei Wang. Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Saeta, Tyler Liechty, Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuyyura. John Aslanides, Nidhi Vyas. Martin Wicke, Xiao Ma. Evgenii Eltyshev, Nina Martin, Hardie Cate, James Manyika, Keyvan Amiri. Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nilesh Tripuraneni, David Madras. Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslic, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang. Riham Mansour, Jason Gelman, Yang Xu, George Polovets, Ji Liu, Honglong Cai, Warren Chen, XiangHai Sheng, Emily Xue, Sherjil Ozair, Christof Angermueller, Xiaowei Li, Anoop Sinha, Weiren Wang, Julia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Iyer, Madhu Gurumurthy, Mark Gold- enson, Parashar Shah, MK Blake, Hongkun Yu, Anthony Urbanowicz, Jennimaria Palomaki, Chrisantha Fernando. Ken Durden, Harsh Mehta, Nikola Momchev, Elahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sebastian Ruder, Morgan Redshaw, Jinhyuk Lee, Denny Zhou, Komal Jalan, Dinghua Li, Blake Hechtman, Parker Schuh, Milad Nasr, Kieran Milan, Vladimir Miku- lik, Juliana Franco, Tim Green, Nam Nguyen, Joe Kelley, Aroma Mahendru, Andrea Hu, Joshua Howland, Ben Vargas, Jeffrey Hui, Kshitij Bansal, Vikram Rao, Rakesh Ghiya, Emma Wang. Ke Ye. Jean Michel Sarr, Melanie Moranski Preston. Madeleine Elish, Steve Li, Aakash Kaku, Jigar Gupta, Ice Pasupat, Da-Cheng Juan, Milan Someswar, Tejvi M.. Xinyun Chen, Aida Amini, Alex Fabrikant, Eric Chu, Xuanyi Dong, Amruta Muthal, Senaka Buthpitiya, Sarthak Jauhari, Nan Hua, Urvashi Khandelwal, Ayal Hitron, Jie Ren, Larissa Rinaldi, Shahar Drath, Avigail Dabush,
Yunjie Li, TJ Lu, Abe Itycheriih,, Prakash Shroff, Mani Varadarajan,
17

---

Published as a conference paper at ICLR 2025
Nan-Jiang Jiang, Harshal Godhia, Uli Sachs, Anthony Chen, Yicheng Fan, Hagai Taitelbaum, Hila Noga, Zhuyun Dai, James Wang. Chen Liang. Jenny Hamer, Chun-Sung Ferng, Chenel Elkind, Aviel Atias, Paulina Lee. Vít Listik, Mathias Carlen, Jan van de Kerkhof, Marcin Pikus, Krunoslav Zaher, Paul Müller, Sasha Zykova, Richard Stefanec, Vitaly Gatsko, Christoph Hirn- schall, Ashwin Sethi, Xingyu Federico Xu, Chetan Ahuja, Beth Tsai, Anca Stefanoiu, Bo Feng. Keshav Dhandhania, Manish Katyal, Akshay Gupta, Atharva Parulekar, Divya Pitta, Jing Zhao, Vivaan Bhatia, Yashodha Bhavnani, Omar Alhadlaq, Xiaolin Li, Peter Danenberg, Dennis Tu, Alex Pine, Vera Filippova, Abhipso Ghosh, Ben Limonchik, Bhargava Urala, Chaitanya Krishna Lanka, Derik Clive, Yi Sun, Edward Li, Hao Wu, Kevin Hongtongsak, Ianna Li, Kalind Thakkar, Kuanysh Omarov, Kushal Majmundar, Michael Alverson, Michael Kucharski, Mohak Patel, Mu- dit Jain, Maksim Zabelin, Paolo Pelagatti, Rohan Kohli, Saurabh Kumar, Joseph Kim, Swetha Sankar, Vineet Shah, Lakshmi Ramachandruni, Xiangkai Zeng, Ben Bariach, Laura Weidinger, Amar Subramanya, Sissie Hsiao, Demis Hassabis, Koray Kavukcuoglu, Adam Sadovsky, Quoc Le. Trevor Strohman, Yonghui Wu, Slav Petrov, Jeffrey Dean, and Oriol Vinyals. Gemini: A family of highly capable multimodal models, 2024.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp- 315-323, 2011.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838, 2024.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks all you need. arXiv preprint arXiv:2306. 11644, 2023.
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasumuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint 2rxiv:2212.12017 2022.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale dis- tantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18553/v//11--111 URL https: /aaaaaaaannttcoooooeeeooo htt.llatthooooooooo//e------
org/p17-1147.
Norman Jouppi, Cliff Young. Nishant Patil, David A. Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghani, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg. John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alck Jaworski, Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. In In-Datacenter Performance Analysis of a Tensor Processing Unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, ISCA, 2017.
Najoung Kim, Sebastian Schuster, and Shubham Toshniwal. Code pretraining improves entity track- ing abilities of language models. arXiv preprint arXiv:2405.21068, 2024.
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The stack: th of permissively licensed source code. Preprint, 2022.
18

---

Published as a conference paper at ICLR 2025
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-War Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le. and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453 466, 2019. doi: a) 00076 URL
https://doo http://ooi.ooo http:///oooooogg/..///o
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pp. 6086-6096, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1000 1612. URL RtLhttps:// /wwww aclweb org/ anthology /P1 PP1 1612.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06 2023a.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi- 1.5 technical report. arXiv preprint arXiv:2309.05663 2023b.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:221 20.-222111111100 2022.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu, G-eval: Nlg evaluation using gpt-4 with better human alignment, 2023.
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, toxicity. arXiv, abs/2305.13169, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https: / //taxiiv.rrg/aaaaaaeaaaaaaa 0111.051100.1
Anton Lozhkov, Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu, May 2024. URL
Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. At which training stage does code data help 1lms reasoning? arXiv preprint 230.::2939..19999 2023.
Aman Madaan, Dheeraj Rajagopal, Niket Tandon, Yiming Yang. and Antoine Bosselut. Condi- tional set generation using seq2seq models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 4874-4896, Abu Dhabi, United Arab Emirates, De- cember 2022a. Association for Computational Linguistics. URL org/ 2002. emnlp-main 324.
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners. arXiv preprint arXiv:2210.07/28, 2022b.
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang. Marzich Fadace, and Sara Hooker. When less is more: Investigating data pruning for pretraining Ilms at scale, 2023. URL https:// a arxiv. orroorgaaaao abs / 230 9 04564.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Van- derwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper under- standing of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp- 839-849, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/vI/N16 1098. URL https: /aaaaaaaannnoooyooooooooooo org/No6 1098.
Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Noua- mane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language 2023a.
models. arXiv preprint 235.v:2305.
19

---

Published as a conference paper at ICLR 2025
Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Noua- mane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models, 2023b. URL https:/ /ttp/v/aaivvvvv//aa//aa org/ abs
Niklas Muennighoff. Thomas Wang. Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong. Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generalization through multitask fine- tuning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers), pp. 15991- 15991-16115 11, Toronto, Canada, July 2023c. Association for Computational Linguis- tics. doi: 10.18653//1/0023.acll ............ acl-long. 891.
URL https: hww.aaaaaaaattooo..oooooooooolle org/2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Mostofa Patwary, Sandeep Subramanian, Dan Su, Chen Zhu, Deepak Narayanan, Aastha Jhunjhunwala, Ayush Dattagupta, Vibhu Jawa, Jiwei Liu, Ameya Mahabaleshwarkar, Osvald Nitski, Annika Brundyn, James Maki, Miguel Martinez, Jiaxuan You, John Kamalu, Patrick LeGresley. Denys Fridman, Jared Casper, Ash- wath Aithal, Oleksii Kuchaiev, Mohammad Shoeybi, Jonathan Cohen, and Bryan Catanzaro. N 1 15b technical report, 2024. URL https: / ///aaiivvvorr/ abs/ 220/222...10 16819, 9.
Guilherme Penedo. Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: Outperforming curated corpora with web data, and web data only. 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11444 2021.
Jack W. Rae, Sehastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang. Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kun- coro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki I Lazaridou, Arthur Men- sch, Jean-Baptisse Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yu- jia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Au- relia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Benneut Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training go- pher, 2022. URL https: araii.iiiv....i. org/ abs 22122 11446.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, abs/1910. 10683, 2020.
20

---

Published as a conference paper at ICLR 2025
Yasaman Razeghi, Hamish Ivison, Sameer Singh, and Yanai Elazar. Backtracking mathematical reasoning of language models to the pretraining data. In The Second Tiny Papers Track at ICLR 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale, 2019.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Common- sense reasoning about social interactions. arXiv, abs/1904.09728, 2019.
Minjoon Seo, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, and Hannaneh Hajishirzi. Phrase- indexed question answering: A new challenge for scalable document comprehension. In Pro- ceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 559-564, Brussels, Belgium, October-November 2018, Association for Computational Linguis- tics. doi: 10.1 (0.18653/1/DD11-1033
https::: /aaaaaaaaanttoooooooeeoeaa wttp:/y..rr///-100000000000 1052..
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Noam Shazeer. Glu variants improve transformer, 2020. URL httpss//axxiv.orr///aaaa 2002. 05202.
Luísa Shimabucoro, Sebastian Ruder, Julia Kreutzer, Marzieh Fadace, and Sara Hooker. LIm see, Ilm do: Guiding data generation to target non-differentiable objectives, 2024. URL https:
//arxiv. orgg abs 20..2400.0000 01490.
Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje Karlsson, Abinaya Mahendiran, Wei- Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzemińíski, Hakimeh Fadaei, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadace, and Sara Hooker. Aya dataset: An open-access collection for multilingual instruction tuning. arXiv preprint 2XXv:2402200669, 2024.
Daria Soboleva, Faisal Al-Khateed Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama, 2023. URL https: / //huggingface co /ataaaaaases/s/eee/eeeeaeaaaaaee baatasess/eeeebrassettteeeee s imP a ama 627B.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint 23xv:23312.//880 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv, abs/2307.09288. 2023.
Alex Wang, Yada Pruksachatkun. Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for purpose language understanding systems, 2020. URL. https:/:
905 00537.
21

---

Published as a conference paper at ICLR 2025
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https: httppp://ggthubb /gggtutthhb....gggggggggg....h www./iiggooolll....cmcco httpeoolllllllhehheeeeeeeeaaahea http///esssttssssseee formeforrerrrr jax, May 2021.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruet: Aligning language models with self self-generated instructions. In Proceedings of the s Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484-13508, Toronto, Canada, July 2023. Association for Computational Linguisties. URL ht hts hat///aaaannnttooogooaoaoo org/2023 acl - long. 754.
Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. pp. 90-1994,60000 September 2017. doi: 10. 111186551//W17-44133 httpstps:: /atp://alllllaaaa aatt ho hog.....o....o...oooo........... org/#17-445
Wikimedia. Wikimedia downloads. URL hntttss///dups. wi wikkiiedia....
BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammana- manchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lu- cile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo I Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Al- ham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong. Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Eduardo González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang. Jörg Frohberg. Joseph Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Šaško, Max Huang. Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subra- mani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen. Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, So- maich Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lep- eroq. Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut Emre Taşar, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, An- drea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szezechla, Gunjan Chh- ablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Tee- han, Samuel Albanic, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Tacwoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj. Urmish Thakker, Vikas Raunak, Xiangru Tang. Zheng-Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myr- iam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers. Ahmed Baruwa, Amanpreet Singh, Anas- tasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurélic Névéol, Charles Lovering, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Hli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, Jan-Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shav- rina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun. Yonatan Belinkov, Zachary Bamberger, Zdenčk Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony
22

---

Published as a conference paper at ICLR 2025
Hevia, Antigona Unldreaj. Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh Haji- Hosseini, Bahareh Behroozi, Benjamin Ajibad, Bharat Saxena, Carlos Muñoz Ferrandis, Daniel McDuff, Danish Contractor, David Lansky, Davis David, Douwe Kicla, Duong A. Nguyen, Ed- ward Tan, Emi Baylor, Ezinwanne Ozoani, Fatima Mirza, Frankline Ononiwu, Habib Rezane- jad, Hessie Jones, Indrani Bhattacharya, Irene Solaiman, Irina Sedenko, Isar Nejadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim Elbadri, Mar- got Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nañi Abrar, Nazneen Rajani, Nour Elkott, Nour Fahmy, Olanre- waju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy. Sylvain Viguier, Thanh Le, Tobi Oyebade, Trrieu Le, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, An- tonio Miranda-Fscalann
Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fubrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Mare Pamies, Maria A Castillo, Marianna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf. Mina Mihaljeic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Out, Sinee Sang-aroonsiri, Srishti Kumar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gi- gant, Tomoya Kainuma, Wojciech Kusa, Yanis Labrak, Yash Shailesh Bajaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. Bloom: A 176b -oro-prreetter open-access multilingual language model, 2023. URL httph:tp///cmmc / /awxivivivvv////aa//aa.///aaatva org/ abs / 2211.22111111 05100.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv, abs/1905.00 1995.0783000 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
Xinlu Zhang, Zhiyu Zoey Chen, Xi Ye, Xianjun Yang, Lichang Chen, William Yang Wang, and Linda Ruth Petzold. Unveiling the impact of coding data instruction fine-tuning large language models reasoning. arXiv preprint arXiv:2405.2000 205555 2024.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang. Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder-v22 Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406. /931, 2024.
Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Yei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blun- som, Shayne Longpre. Niklas Muennighoff, Marzich Fadace, Julia Kreutzer, and Sara Hooker. Aya model: An instruction finetuned apcessccess multilingual language model, 2024.
23

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT AND LIMITATIONS
While we systematically study the impact of code data on downstream natural language tasks, we do not study its impact safety and bias. Additionally, given the nature of pre-training and the number of ablations we have conducted we were limited by the scale of larger model sizes due to prohibitive compute costs.
REPRODUCIBILITY
We provide details about data mixture (Section 2.1), data filtering (Appendix C.I, C.2, C.3), evaluation (Section 2.2, Appendix A) and training (Section 2.3) setups. We believe these details provide clear picture how obtain our data setup, model ablations and evaluation results.
A EVALUATION DETAILS
We briefly describe the details of our evaluation benchmarks and the composite datasets used for each category below:
1. World knowledge. These benchmarks aim measure world knowledge, testing knowl- edge memorization, retrieval, and question answering capability given context. We include Natural Questions Open (Kwiatkowski et al., 2019), and TriviaQA (Joshi et al., 2017) as the datasets. We report the average exact match scores for both these benchmarks.
2. Natural language reasoning. The Natural language (NL) reasoning suite consists of 11 benchmarks that involve natural language based reasoning such as Question Answering (Clark et al., 2019; Seo et al., 2018; Welbl et al., 2017; Sap et al., 2019; Choi et al., 2018), natural language inference (NLI) (Wang et al., 2020; de Marneffe al., 2019; Wang al., 2020), sentence completion (Mostafazadeh et al., 2016; Zellers et al., 2019), co-reference resolution (Sakaguchi et al., 2019) and general intelligence (Clark et al., 2018). We include a full list of the constituent benchmarks in Table 1. We report the average accuracy scores across all benchmarks.
3. Code. While main foeus is general performance, also want measure any changes to code generation performance. For code benchmarks, we focus on the function comple- tion task. We evaluate HumanFyyll-Python (Chen et al., 2022) and MBPP (Austin al., 2021). We report the average pass@] 1 scores of these benchmarks.
B SUMMARY RESULTS FOR PRE- TRE-TRAINING RECIPES
Summary results are shown in Table 2.
C CODE-DATASETS FILTERING
C.1 QUALITY FILTERS
In addition the deduplication and quality filtering applied on the GitHub scrapes by Starcoder for The Stack dataset (Li et al., 2023a), we apply filters to remove documents with greater than 1000 float numbers, with instances of the string 0x, that are lists of top-level domains, and with 'generated by' in the first 400 characters
C.2 PROGRAMMING LANGUAGES PRESENT IN WEB-BASED CODE DATASET
Programming languages included in our version The Stack dataset are present in Table 3
C.3 MARKLP-STYLE PROGRAMMING LANGUAGES PRESENT IN WEB-BASED BASED CODE DATASET
Markup-style languages included in version of The Stack dataset in Table
24

---

Published as a conference paper at ICLR 2025
Tisk
Dataset
Metric
WORLD KNOWLEDGE TASKS
Question Answering
NaturalQuestionnOpen (Lec et al., 2019)
TriviaQA (Joshi al., 2017)
O-shol
0-shot
Acc.
Acc.
NATURAL LANGUAGE REASONING
BoolQ (Clark 2019)
0-shot
Question Answering
PIQA (Sco et al,, 2018)
0-shot
Acc.
SciQ (Welb) al.,
0-shot
SocialIQA (Sap al. 201,9
0-shot
Acc.
QUAC (Choi al., 2018)
0-shot
Apc.
Natural Language Inference
SuperGLLTF.COP. (Wang et al. 2020)
SuperGLUEE-CB (Wang et 2020; Mamelle al., 2019)
O-shol
0-shot
Acc.
Acc.
Sentence Completion
HellaSwag (Zellers 2019)
StoryCloze (Mostalazadch et al., 2016)
O-shol
0-shot
Acc.
Acc.
Coreference Resolution General Intelligence
Winogramde (Sakaguchi el al.., 2019)
O-shol
Avc.
ARC-Easy (Clark al., 2018)
0-shot
Acc.
Open-Ended Generation
Dolly-200 (English) (Singh et 2024)
O-shot
win-rate
CODE GENERATION
Function completion
HumanFval (Chen et al., 2021)
O-shot
pass @
(Austin 2021)
O-shoi
pass
Table 1: Datasets considered for evaluation: We conduct extensive evaluations benchmarks detailed above. These provide valuable proxies for performance in natural language reasoning, world knowledge, open ended text generation, and code generation tasks.
Token Count
Natural Language
Model Variant
Recipe
Code
Total Avg.
Text
Avg.
Pre-training
400B
49.0
9.5
29.2
0.4
19.6
TEXT-ONLY
Cooldown
+32B
+BB
54.1
11.1
32.6
4.4
23.2
Pre-training
200B
200B
$1.8
8.1
30.0
9.0
23.0
BALANCED-ONLY
Cooldown
+32B
+8B
53.2
32.1
8.4
24.2
Pre Pre-training Init.
1008
100B
$2.0
7.4
29.6
7.8
22.4
BALANCED TEXT
Continue Pre-train.
+180B
+00B
$3.0
9.9
31.5
4.8
22.6
Coeldown
32B
$4.9
32.9
5.8
23.9
Pre-training Init.
200B
44.7
23.1
15.5
20.6
CODE TEXT
Continue Pre-truin.
180B
+20B
53.3
31,4
4.1
22.3
Cooldown
+328
+8B
52.1
10.3
31.2
7.5
23.3
Pre-training recipes in- clude initial pre-training. continued pre-training, and cooldown phases. Balanced-<Tex achieves the best NL performance while Balanced-only performs significantly better in code generation.
Table 2: Model variants with the corresponding pre-training recipes:
D LLM JUDGE PROMPT AND PREAMBLE FOR WIN-RATES
Preamble
You are a helpful following assistant whose goal is select the preferred
(least wrong) output for given instruction.
Prompt
Which of the following answers is the best one for the given instruction.
good answer should follow these rules:
1) It should have correct reasoning,
2) It should answer the request in the instruction,
3) It should be factually correct and semantically comprehensible,
4) It should be grammatically correct and fluent.
Instruction:
instruction
25

---

Published as a conference paper at ICLR 2025
Language Name
Proportion of total code documents
java
15.54
javascript
15.29
php
12.46
python
9.60
c-sharp
8,30
typescript
7.92
6.63
cpp
4,91
go
3,49
ruby
2.69
shell
1.82
kotlin
1.76
Swift
1.52
Vue
1.48
rust
1.00
scals
0.94
JSX
0.83
sql
0.74
dart
0,72
makefile
0,53
lua
0.47
haskell
0.45
smalltal
0,43
tex
0.37
clojure
0.10
Table Programming languages included in our version of The Stack dataset
Language Name
Proportion of total code documents
markdown
54.23
yaml
10.77
json
9.97
html
8.57
cSs
6,86
SCSS
5.84
restructuredtext
2.26
1.25
kmarkdown
0.02
Sass
0.22
Table 4: Markup-style languages included in our version of The Stack dataset
Answer (A) completion_a
Answer (B) completion_b
FIRST provide concise comparison of the two answers which explains
which answer you prefer and why.
SECOND, on a new line, state exactly one
PPreferred Answer (A) or >preferred: Answer (B) to indicate your choice of preferred response.
Your response should use the format:
Comparison: <concise comparison and explanation>
Preferred: <'Answer (A) \nnwwe (B)>>
26

---

Published as a conference paper at ICLR 2025
E
GENERATIVE WIN-RATES FOR IMPACT OF INITIALIZATION
Generative Winrate
42.2
40
34.737.2
32.2
28.1
25.6
20
text-only code + text
text-only
balanced + text
ties
ties
Figure 8: Impact of initialization on generative quality as judged by LLM--s-a-judge.
F EVALUATION OF 470M COOLDOWN MODELS ON GSM8K
Mathematical Evaluation
4.05
4.12
4
5-shot (maj@01)
3.60
3
2.90
2
1
0
GSM8K
text - no-code cooldown
text - cooldown w/ code
balanced text no-code cooldown
balanced text cooldown w/ code
Figure 9: Evaluation of 470M cooldown models on GSM8K Including code any stage of the pre-training improves performance compared to the model where code has been seen in any of the training stages: pre-training, continual pre-training and cooldown. The most performant model in this comparison has code in all stages including cooldown where it leads significant improvement (from 2.9 to 4.12, +42% relative gain).
27


Reviews:
[{"review_id": "1zdK1E0XGD", "reviewer": "Reviewer_PKei", "summary": "The paper investigates the impact of incorporating code data into pre-training for large language models (LLMs).\nIt explores how code data influences various tasks, including natural language reasoning, world knowledge, and open-ended generation. \nThrough extensive ablation studies, the authors reveal that adding code data yields substantial benefits across tasks beyond coding. \nThe study emphasizes code quality and finds that synthetic, high-quality code data further enhances general performance. \nThe results imply that code data should be included in the pre-training process as it significantly enhances generalization across tasks.", "questions": "- In the sentence \"Figure 8 shows the results of xxx\" in Section 3.2, it seems that \"Figure 8\" should actually be \"Figure 3.\" May be a typo?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses a crucial question: does code data improve performance on non-code tasks? This question has been relatively unexplored, and the paper offers a fresh perspective on the potential benefits of code data for models primarily intended for natural language tasks.\n\n- The methodology is rigorous, employing controlled experiments that examine various factors, such as code quality, proportions, and stages of pre-training. The large-scale experiments (up to 2.8B parameters) provide an in-depth understanding of how code data impacts model performance.\n\n- This work has significant implications for pre-training dataset design, suggesting that code data improves generalization and that high-quality synthetic code can be particularly impactful. This insight can guide future LLM development, especially in choosing and refining pre-training data.", "weaknesses": "- For the scaling experiments, some key information could be explored further. For instance, does the trend observed in Figure 4 hold consistently across models of different scales? Additionally, how can the trends identified be extended or generalized to larger models?\n\n- I am also curious about the model's performance on mathematical tasks, as these, like coding, are highly logical and reasoning-intensive. Including a benchmark evaluation on mathematical tasks could provide valuable insight into whether similar gains extend to other logic-focused domains.\n\n- In terms of writing, certain critical details could be clarified. For instance, it may not be immediately clear in the text which model size (e.g., 470M or 2.8B) is being evaluated in figures such as Figures 4 and 5. Adding such context in the captions or main text would enhance clarity for readers.\n\n**Given the close relationship between scaling laws and research on dataset composition, there is still room to refine and expand upon some of the paper's exploratory findings.**", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xvrUNz4RsV", "reviewer": "Reviewer_SkF7", "summary": "The authors examine the role of code data in the pre-training of large language models (LLMs), a practice that has become increasingly common, even for models not specifically designed for code. Despite anecdotal consensus among practitioners regarding the importance of code in enhancing the performance of general LLMs, there has been limited research analyzing its precise impact on non-code tasks. This study systematically investigates the influence of code data in pre-training on a diverse array of downstream tasks beyond code generation. The authors pose the question: “What is the impact of code data used in pre-training on a large variety of downstream tasks?” Through extensive ablation studies, they evaluate models ranging from 470M to 2.8B parameters across natural language reasoning tasks, world knowledge tasks, code benchmarks, and LLM-as-a-judge win rates. The results consistently demonstrate that code serves as a critical building block for generalization well beyond coding tasks. Improvements in code quality significantly enhance overall task performance, with findings indicating up to an 8.2% relative increase in natural language reasoning, a 4.2% boost in world knowledge, a 6.6% improvement in generative win rates, and a 12x enhancement in code performance when compared to text-only pre-training. The authors suggest that investing in code quality and retaining code during pre-training can yield substantial benefits across various tasks.", "questions": "The investigation into the role of code in LLM pre-training addresses a crucial aspect of model performance with good implications for both research and practical applications. The exploration of the impact of code data and strategies for leveraging it provides valuable insights into enhancing model effectiveness, contributing to a deeper understanding of LLM capabilities.\n\nI have two concerns:\n\n1. Limited Technical Contribution: The study primarily consists of empirical analyses without introducing new methodologies or frameworks. While the findings are good, the technical contribution could be perceived as limited due to the absence of novel approaches or theoretical advancements.\n\n2. Limited Scale: The analysis is restricted to models with 470M and 2.8B parameters, which may not capture the performance dynamics across a broader range of model sizes and architectures.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Important Area.\n+ Interesting Finding.\n+ Analysis of the Code Quality is Good.", "weaknesses": "- Technical Contribution is Weak.\n- Limited Scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bUJtzyHARz", "reviewer": "Reviewer_ujyE", "summary": "This paper investigates the impact of code data in pre-training for NLP tasks. The authors pre-train Transformer models with sizes ranging from 470M to 2.8B parameters with ablations of code and non-code data. The pre-trained models are evaluated on downstream tasks such as NL reasoning, world knowledge, and code benchmarks. Experimental results show that incorporating code into pre-training has a positive impact on non-coding tasks. In addition, improvements to code quality have an outsized impact across all tasks. In particular, compared to text-only pre-training, the addition of code results in up to relative increase of 8.2% in natural language (NL) reasoning, 4.2% in world knowledge, 6.6% improvement in generative win-rates, and a 12x boost in code performance respectively. The results suggest future investments in code quality and preserving code during pre-training.", "questions": "1. Given the observed performance drop on certain NL tasks, how does the study reconcile this finding with the broader claim of code data’s benefits for NLP tasks?\n\n2. Have the authors considered evaluating the pre-trained models on additional natural language tasks to better understand the generalization effects of code data?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "• The paper provides insights into the effects of code data on pre-training across a range of NLP tasks.\n\n• Experiments are conducted with multiple configurations, such as balancing code and text data, adding depth to the analysis.", "weaknesses": "• The findings are somewhat unsurprising, as incorporating diverse data sources (like code) in pre-training is known to improve model performance—a principle demonstrated by previous studies, including PaLM (Chowdhery et al., 2022), Gopher (Rae et al., 2022), and Bloom (Workshop et al., 2023). This paper extends these findings by analyzing code quality and proportion but does not significantly depart from established conclusions, such as the importance of data quality in pre-training (https://arxiv.org/abs/2207.05579).\n\n• Figure 4 suggests that including code data may actually reduce performance on some natural language tasks, which appears contradictory to the paper’s main claim. This inconsistency weakens the overall reliability of the conclusions.\n\n• The pre-training models used in this study are relatively small (470M and 2.8B parameters), which limits the generalizability of the findings. Larger models may be necessary to substantiate conclusions regarding the effects of code data on NLP pre-training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Og6vgwg1Y0", "reviewer": "Reviewer_YwMY", "summary": "## Summary:\nThe authors conduct a systematic investigation into the role of code data in the pre-training phase of LLMs, assessing the impact on three task categories: natural language reasoning, world knowledge, and code generation. They analyze three pre-training stages, including initial pre-training, continual pre-training, and a cooldown phase; evaluate the effect of code data’s proportion, quality, and properties on the model’s performance across the three task categories. Their results demonstrate that incorporating code data enhances model performance across all the pre-training stages, suggesting a balanced approach with optimized code data quality can boost the model’s ability in reasoning, general knowledge, and coding.", "questions": "- Could the authors clarify whether the Balanced LM (50% text and 50% code) incorporates markup-style data in its training mix? The paper specifies that the Code-initialized model includes 80% code data and 20% markup-style data, but it remains unclear whether this type of data is also included in the Balanced LM.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- _Novelty_: This work is a first-of-its-kind, comprehensive study on how code data in pre-training affects LLMs across various natural language and coding tasks, offering new insights into data requirements for LLM pre-training.\n\n- _Comprehensiveness_: The authors carry out a broad set of experiments, exploring various scales of model size, data quantity, and training stages. Their work provides actionable guidance for future LLM development.", "weaknesses": "1. **Limited Details on High-Quality Code Data**: A significant portion of the paper’s conclusions, particularly in Section 3.4, emphasize the positive impact of incorporating high-quality code data on LLM performance across different task categories. However, the description of the high-quality code data is limited to that it is proprietary and synthetically generated. The lack of transparency about the dataset's characteristics, such as the source, data format, or complexity raises questions about what specifically constitutes \"high quality\" and how these qualities directly influence performance. More details on criteria for data quality would strengthen the paper and enable the research community to apply similar quality standards in future work.\n\n2. **Potential Bias in Generative Ability Evaluation**: The evaluation of generative ability relies on the LLM-as-a-Judge approach, specifically through win-rate scores on the Dolly-200-English dataset, which is designed to reflect open-ended, non-code tasks. The authors attribute improvements in generative quality to the inclusion of code data in pre-training, arguing that code data enriches the model’s ability to generate preferred outputs. However, using LLMs to evaluate other LLM outputs has known limitations[1], including potential biases toward verbosity, structured formatting, and positional advantage. Moreover, the inclusion of markup-style code may enhance readability and formatting, making responses more appealing to the LLM evaluator. To mitigate ambiguity, qualitative examples of output differences would be helpful in illustrating how the inclusion of code data contributes to generative improvements beyond format and readability alone.\n\n3. **Minor Issues**: \n   - In Section 3.2, Figure 3 is referenced incorrectly as Figure 8, which could confuse readers. \n   - The legend for Figure 8 lacks a label for the gray bar, which reduces clarity in interpreting the data presented.\n\n[1] Zheng, L., Chiang, W. L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., ... & Stoica, I. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 46595-46623.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)