How Does Code Pretraining Affect Language Model Task Performance?
Authors: Jackson Petty
Sjoerd van Steenkiste
Tal Linzen
Venue: Accepted by TMLR ()

Abstract: Large language models are increasingly trained on corpora containing both natural language and non-linguistic data like source code. Aside from aiding programming-related tasks, anecdotal evidence suggests that including code in pretraining corpora may improve performance on other, unrelated tasks, yet to date no work has been able to establish a causal connection by controlling between language and code data.

Here we do just this. We pretrain language models on datasets which interleave natural language and code in two different settings: competitive, in which the total volume of data seen during pretraining is held constant; and  additive, in which the volume of language data is held constant. We study how the pretraining mixture affects performance on (a) compositionality, measured by generalization accuracy on semantic parsing and syntactic transformation tasks, and more broadly on (b) downstream non-code-related objectives, measured by performance on tasks from the BigBench benchmark. We find that pretraining on higher proportions of code improves performance on compositional tasks involving structured output (like semantic parsing), and mathematics. Conversely, increase code mixture can harm performance on other tasks, including on tasks that requires sensitivity to linguistic structure such as syntax or morphology, and tasks measuring real-world knowledge.

Published in Transactions on Machine Learning Research (01/2025)
How Does Code Pretraining Affect Language Model Task Performance?
Jackson Petty'
research@jccksonpetttyyg
Deportment of Linguistics New York University
Sjoerd van Steenkiste Google Research
swansternkiiste com
Tal Linzen
linzen droeelenggoole.coom
Google Research
Reviewed OpenReview: https: http::///pppeeeee.eeeeeeee
forum? id= p
Abstract
Large language models are increasingly trained on corpora containing both natural lan- guage and non-linguistic data like source code. Aside from aiding programming-relater tasks, anecdotal evidence suggests that including code in pretraining corpora may improve performance other, unrelated tasks, yet to date work has been able establish causal connection by controlling between language and code data. Here we do just this. We pretrain language models on datasets which interleave natural language and code in two different settings: competilive, in which the total volume of data seen during pretraining held constant; and additive, which the volume of language data is held constant. We study how the pretraining mixture affects performance on (a) compositionality, measured by generalization accuracy on semantic parsing and syntactic transformation tasks, and more broadly on (b) downstream non-code-related objectives, measured by performance tasks from the BigBench benchmark. We find that pretraining higher proportions of code improves performance on compositional tasks involving structured output (like semantic parsing), and mathematics. Conversely, increased code mixture can harm performance on other tasks, including on tasks that require sensitivity to linguistic structure such as syntax or morphology, and tasks measuring real-world knowledge.
1 Introduction
Large language models (LLMs) are increasingly used not only natural-language assistants, but also for programming. LLMs which are trained corpora containing code in various programming languages are used as programming assistants capable of generating code from natural- descriptions (Chen et al. 2021), translating code between programming languages (Roziere et al. 2020 decompilation of machine code into human-rondable source code (Hosscini & Dolan-Gavitt, 2022). repairing vulnerabilities in existing code (Pearce et al. 2023). and even acting as programming agents when paired with tools (Yang et al., 2024a). These use cases have motivated adding code to pretraining corpora (see, inter alia, Gemini Team et al. 2024 OpenAl et al. 2024 Anthropic Al 2024 Groeneveld et al. 2024
Concomitant to the inclusion of code in pretraining corpora, the performance of LLMs many tasks has improved. Relevant for our purposes, many of the best-performing models include code in their pretraining
corpus (see, inter alio, Fu & Khot 2022 Ye & Durrett 2022 Ye et al. 2023 Zhang et al. 2023 Zhou et al.
2023 Kim et al. 2024 Ma et al. 2024 Yang et al. 2024b Razeghi et al. 2024 Coda-Forno et al. 2024 Longpre
Work done while student researcher at Google.

---

Published in Transactions on Machine Learning Research (01/2025)
et al. 2024. That models trained in part code perform well on several non-programming benchmarks raises intriguing questions: Does pretraining code confer an advantage on non-programming tasks? If so, given & fixed compute budget, how much data should be allocated to code instead of natural-anguuge data?
Establishing a causal relationship between code pretraining and downstream performance is difficult. Earlier studies have tackled these questions by comparing off-the-shelf code and no-code models (see, inter alia. Kim et al. 2024; Cora--ooro et al. 2024). Such observational studies are limited by the design choices of model creators and the availability of information about hyperparameters and training data. Many of the models typically surveyed are proprietary and don't disclose this information. While pairs of open-source models differing only in their pretraining corpora do exist, such as Llama 2 & Code Llama (Touvron et al.) 2023; Roziere et al. 2023) or Gemma. & CodeGemma (Gemmma Team et al. 2024 Google 2024), they often come with two important caveats: first, the
of the models are derived by taking the non-code variants and conducting additional pretraining on code data, meaning the comparisons cannot control for total data volume; second, cach pair treats the inclusion of code data as a binary variable, either present or absent, frustrating altempls to explore how changes in the amounl of code influence downstream behavior.
We address these issues directly. We construct datasets that mix natural- and source-code data at varying ratios, treating code inclusion as a continuous variable. We then pretrain language models of equal size on these parameterized datasets in two different experimental setups: a competiliiee setting where we keep the total volume of training data constant and vary the percentage allocated between code and natural language; and an additive setting where we keep the volume of language data constant and add additional amounts of code on top.
Previous work has found that augmenting training data with synthetic formal languages instantiating com- positional patterns can improve compositional generalization (Papadimitriou & Jurafsky 2023 Yao & Koller 2024 Lindemaan et al.) 2024). Like formal languages, source code has a number of qualities that may aid models on scomingly unrelated tasks: it is highly structured, by virtue of its conformance to the syntax of the programming language it's written in; it is generally high-quality, owing to the use of linting and bug-checking tools and programming methodologies employed by its authors; it has interpretable semantics which is grounded by the functionality it describes; and, notably for compositionality, it contains instances of identical arguments and functions (c.g., variable names and method signatures). Informed by these ob- servalions, we evaluate our trained models for compositional generalization by finetuning them on three compositional generalization benchmarks (COGS, COGS-vf, and English Passivization). We also measure their performance on a broad array of tasks from BigBench to see how well code helps or hurts performance on unrelated domains.
We find that including code in a model's pretraining corpus has noticcable impacts on its performance on downstream tasks, in varying directions. Higher code mixtures improve performance in arithmetic and compositionality in domains whose output has formal structure (like semantic parsing). Conversely, increased exposure to code can harm language model performance on purely-linguistic tasks and tasks involving factual knowledge. We conduct permutation tests to study the impact of pretraining on downstream tasks and show that code pretraining increases the variance on task performance while raising the performance on the upper-quartile of tasks.
2 Related Work
Earlier work has studied whether pretraining on code is beneficial for non-programming tasks. Observational studies have looked at the impact of code downstream performance post-hoc. Fu & Khot 2022 speculated that code pretraining is at least partially responsible for the improvement in capabilities between the -001 and -002 scries of GPT-3(.5) models, specifically highlighting chain-of-thought reasoning, long-term dependency sensitivily, and "complex reasoning" as likely resulting from code pretraining. Yang et al. (2024b) provides a broad study of how code impacts language model capabilities, arguing that code improves complex reasoning and structured data understanding. Mueller et al. (2024) shows that code protraining improves generalization on syntax-sensitiv in-context learning tasks. By contrast, [ooa-Forno et al. (2024), in an observational study, conclude that code pretraining does not improve model performance on a benchmark of behavioral
2

---

Published in Transactions on Machine Learning Research (01/2025)
tasks motivated by cognitive psychology. Kim et al. (2024) show that code pretraining improves models' entity-tracking capabilities.
Several experimental studies on the impact of code pretraining have also been conducted. Ma et al. (0224) attempt to verify the impact of code experimentally, comparing the 2.6 parameter CodePanGu2.66 model trained on a mixture of natural-languee PanGu models of the same architecture trained only on natural language data. They conclude that code exposure, both during pretraining and instruction finetuning, is beneficial for performance on logical, legal, analogical, and scientific reasoning, and for chain-of- thought capabilities, though their experimental design does not control for data volume (~26.5 B tokens for PanGu2.6/13 versu=~~42 B tokens for CodePanGu2..6) and does not quite control for model and training hyperparameters (models differ in the number of attention heads and use slightly different optimizer settings, which are magnified by the large difference in the number of training steps due to the difference dataset size). Ma et al. (2024) also show exposing code to models early on during training can be helpful for some tasks. [ongpre al.) (2024) show experimentally that removing code from model's pretraining corpus harms performance on question answering in number of different domains, though their experimental setup does not control for data volume and, consequently, other training hyperparameters sensitive to this.
and code data to Zeng et al. 2222)'s s 2.6 and 13 parameter
The closest study to ours is the concurrent, Aryabumi et al. (2024), which provides a thorough examination of the impact of mixed text-code training on reasoning and natural language tasks when controlling for data volume. We differ from Aryabuui et al. (2024) in three ways. First, in addition to comparing training setups where total data volume fixed (our competitive case, and all training recipes studied in Aryabumi et al. al.[22210 2024). : we also consider training mixtures which vary the amount of code while keeping the volume of language data fixed (our additive case). Second, we treat dataset composition as continuous range and conduct regression analysis to quantify the impact of code mixture. Third, we focus particularly on studying how code pretraining impacts compositional generalization, motivated by the hypothesis that code instantiates compositional patterns useful for genoralization. We compare our results and those of Aryabumi et al. 2024)) in Section G
3 Dataset Construction
To study how the amount of code in a language model's pretraining corpus impacts downstream performance, we construct datascts which interloave natural language and code sequences. The ingrodients for our datasets are the English portion of the Colossal Cleaned Common Crawl (C4; [Rffel et al. 2020) and a version of the code portion of The Pile (Cao et al.] 2020), itself taken from public GitHub repositories; we use version which has been cleaned to include only non-binary files smaller than 1MB with common code-related file extensions.
Each dataset, which we refer to as a 'code mixture,' is parameterized by a single value m E [0, 1] representing the percentage of code in the training data, under the assumption that the C4 dataset has been fully cleaned of any code data. The mixture m relates the number of total tokens Niotal in the datasct to the number of code Noode and language Ning tokens via
Neude = m Niotal:
Nang = (1 m) Nuotal-
We construct families of training datasots in two different settings: competitive, in which the total amount of data is held constant while m varies, redueing the number of language tokens as the number of code tokens increases; and additive, in which the number of language tokens is held constant while the number of code tokens increases proportional to m (see fig.
'There is some ambiguity in the way Ma et al. (2024) describe their dataset: first, they cite that PanGu13 is trained
1TB of data, but Zeng et al. [202]] report that trained on 100GB of data while their far larger 200 B parameter model is the trained
ITB of data; second, Ma al. (2024) detail the individual data sources in GB but report the total dataset
size in terms of tokens. It is unclear from phrasing whether their sampling strategy yiclds a datasct of 100 GB in total, contains 100 GB of text data in addition to 50 GB of code data, but in cither case the Table 4 in Zeng et al. (021 shows that the 100 GR natural-language dataset used for the PanGu comparison models contains only ~26.5 B tokens, compared to CodePanGu's 422 tokens.
3

---

Published in Transactions on Machine Learning Research (01/2025)
Competitive
Additive
0
Code
10
Language
20
10
Language-onf Baseline
30
Mixture
40
50
90
50
100
132
50
100
152
264
Takens (billions)
Tokens (billions)
Figure 1: Code mixtures for the competitive and additive settings; for comparability between settings, we set the 0% mixture in both settings to have 132 B tokens of natural-language data (so Ntotal 132 B tokens in the competitive sctting, and Mang = 132 B tokens in the additive setting). Note that these sequences are shuffled during training, so models see code and language data at the same time.
Competitive: Here, Nootal is held constant while varies between 0 and 0.9. This means that models trained the 0% code mixture see Nootal Niang language tokens and 0 code tokens, while those trained on the 90% mixture see 0.1 x Ntotal tokens of language data and 0.9 x Niotal tokens of code data.
This setting provides the clearest way to quantify the marginal utility of training on code instead of language, since we control for the total volume of data seen and consequently the total compute cost. However, the interpretability of results on mixtures with high values of m may be diminished since removing nearly all natural-language training data from model's training corpus will lessen its ability to interpret and generate language; this, in turn, may greatly reduce its utility, even code-related tasks, since the model will have far less ability to understand prompts or follow instructions. Additionally, the applicability of any results here to established pretraining setups may be limited by the fact that it will always be better in an absolute sense (and may be in a compute-optimal sense) to train a model on more data rather than less data (sec, for instance, the conclusions of Hoffmann et al. 2022). Given this incentive, artificially limiting the amount of either code or language data provided a model may not accurately reflect the considerations of model developers who, they want to improve the code performance of a model, will simply add additional code data to the training corpus. To mitigate these issues, we also consider sccond sctting:
Here, Mang is held constant while varies between 0% and 50%. In order to keep Nang fixed while varies, we increase the number of total tokens proportionally:
Additive:
1
Neotal = Naang X
m
Since Niotl increases unboundedly in m, we limit our study to consider additive mixtures of at most 50% code, which have twice as many tokens as the 0% mixture, which is identical to the 0% competitive mixture. This setting guarantees that all models have seen the same amount of natural language data, ameliorating the concern that any degradation in performance may result from insufficient exposure to natural language, but at the cost. of failing to control for total data volume or compute. To further ensure that we can adequately compare code and non-code models across, we construct language-only bascline datasets for each code mixture. These datasets have the same number of total tokens, but with 100% of those tokens coming from natural language.
Though both the code and language datascts we use are intended to be distinct in content type from one another, it is likely that there is a degree of overlap in content lype between them. For instance, source code often contains comments and string literals containing natural-language data; in the other direction, though C4 is cleaned using a varioty of houristics, some of which are explicitly designed to exclude code-like data, it is likely that such clcaning attempts are imperfect and therefore there may be (relatively) small amount of code data in the natural-langune data source. We do not perform any additional filtering or cleaning of these
4

---

Published in Transactions on Machine Learning Research (01/2025)
data sources. As such, the code data source almost certainly contains some natural language data, just the natural language source likely contains some un-filtered source code. We leave it to future work to explore what impact, if any, fully removing code comments has on downstream performance, though we speculate that the pairing of a code block with a natural language description of its function and implementation may be meaningfully important to language model performance on non-code tasks whose performance is aided by code pretraining.
4 Experimental Setup
4.1 Model Construction & Training
We use the datasets constructed in section 3 as pretraining corpora for causally-masked decoder-only trans-
former language models (Vaswani et al. 2017 Radford et al. 2019 We construct 12-layer decoder-only models in t5x E5xeRts et al. (2023). ). Model hyperparameterss were chosen following the methodology of Wang et al. 2022)) and Petty et al. (2024) to approximate decoder-only versions of T5-large, resulting in models with roughly 374M parameters; see Appendix A for hyperparameter details We pretrain these models with base natural language data volume of 132 B tokens. This means that all models in the competitive setting were trained with Neotal = 132 B lokens, while the models in the additive setting were trained with Mang 132 B tokens, and hence Neotal varying between 132 B tokens and 264 B tokens depending on the mix- ture; we use a batch size of 128, meaning that models were trained for between M and 2 M steps, depending on the mixture and setting. For each combination of code mixture and setting, we pretrain models from five different random seeds. We pretrain models on TPUs. We estimate that full replication of the pretraining procedure outlined here would take roughly 750 TPU-days of compute.
4.2 Evaluation
We mcasure performance on three compositional generalization benchmarks and, more generally, on Big- Bench lasks. For each uvaluation domain, we quantify the impact that code pretraining has on performance by calculating lines of best fit between performance (e.g., generalization accuracy for the compositional gencralization benchmarks or multiple-choice grade for BigBench multiple choice tasks) and code mixture.
4.2.1 Compositional Generalization
Compositional generalization is measure of how well learner can generate and interpret novel, licit combinations of primitive pieces which have been previously learned. Originally motivated to describe human linguistic faculty-sucd as the ability of speakers to produce and understand an infinite number of novel, grammatical sentences conpiintiicoompiiiiiiitittt is also a relevant property of many formal systems, like mathematics or programming languages. We hypothesize that the presence of source code in pretraining data may aid models in making this kind of generalization since source code often contains sequences in which a finite set of primitives (e.g., variable and method identifiers) are broadly combined.
To evaluate whether increased code mixture enables compositional generalization, we finetune pretrained models on suite of compositional generalization datasets: COGS (Kim & Linzen 2020), a semantic parsing task in which natural-langusl sentences are transformed into a formal semantic representation; COGS-vf (Qiu et al. 2022), a variant of COGS which simplifies the output format; and English Passivization (Mueller) let al. 2022), a natural- languagg transduction task in which aeahhetically-generate active-voice sentences are transformed into passive variants. Each dataset contains training, validation, and generalization splits, where the generalization split is constructed to test liaatbeut combinations of familiar primitives. Table shows examples of the input and output sequences for each of the datasets.
COGS and COGS-vf both divide their generalization split into two parts based on generalization type: either lezical, in which a known primitive is used in a grammatical position it has not been seen in before (e.g..
We don't view paper exploring the merits of the specific architecture in question; our results here are of interest irrespective of the specific transformer model architecture, unless there reason to suspect that change architecture confers inductive bias which is particularly sensitive to source code as a training distribution.
5

---

Published in Transactions on Machine Learning Research (01/2025)
COGS
2:: A hedgehog ate the cake
y: (raa hedgehog(#1) AND cat.agent((3, 21) AND eat.theme
COGS-vf
A hedgehog the cake the bed
y: eat (agent. = hedgehog, theme = "cake(nmod.on = "bed))
English Passivization
our vultures admired her walrus above some zebra
y: her walrus above some zebra was admired by our vultures
Table 1: Examples of inputs (x)) and targets (y) from each compositional generalization dataset.
hedgehog in subject position, when had only been seen during training as an object); or structural, which a known grammatical structure is used in a novel position (e.g., a prepositional phrase such as on the mat modifying the subject, when in training such phrases only modified objects). Previous studies involving COGS and COGS-vf have found the structural generalization examples in COGS to be much harder than the lexical generalization examples. Reducing the complexity of the output form, as is done in COGS-vf, makes the structural (asks somewhat easier, though nol easy. [Pelly et al. (2024) found that models of a comparable size could attain accuracies near 90% on the lexical generalization examples from COGS but near 0% on the structural examples; on COGS-vf, models were able to attain accuracies greater than 95% on lexical cases and 10% on structural cases.
For all compositional generalization datasets, finetune models for 10K steps and report the mean full- sequence accuracy (i.e., if every sutlorggeesiveely-geeereee token is correct, 0 otherwise) over all examples in the generalization split for each random pretraining seed. In general, we do not observe any meaningful effect that code mixture has on training dynamics over the course of finetuming, nor do we see evidence of over- or fitting to the validation or generalization sets; see Appendix C for a full discussion.
4.2.2 BigBench
We also evaluate models on BigBench Srrvastava et al. 2023 benchmark of 204 diverse and challenging tasks presented in a common format. We evaluate models in a zero-shot setting, where a question is given in context (e.g., What is 697 times 205? from the 8-digit multiplication task) and the model must either generale the correct label (e-g, (a).) from a provided list of responses (for multiple-choice lasks) or generate the correct answer (for generative tasks). Since our focus is on the effect of code in pretraining on non-code tasks, we exclude from consideration tasks which explicitly designed to test the capabilitios of models underslanding generating source code. Table 2 shows examples of the input and outpul sequences for the BigBench tasks we discuss in detail.
5 Results
Code improves compositional generalization for structured outputs. When we finetune on COGS and COGS-vf, where the output domain has a formal structure, we find that performance improves as the proportion code increases in both the competitive and additive settings (see fig. 2 and table g in Appendix B The effect is most pronounced for the structural generalization examples from COGS-vf the competitive and additive settings (regression coefficients 3 = 0.147 and 3 = 0.165, respectively; this indicates that the best-fit line predicts an accuracy increase of 14.7% as the proportion of code increases from 0% to 100%), though all code-mixture models show a non-negative relationship between code mixture and generalization accuracy. Code helped the least on the structural generalization examples from COGS, where absolute performance remained near-zero. In the additive setting, find that code-mixture models perform well (on lexical generalization examples) or better (on structural generalization examples) than the equivalent language-only baseline models.
In order for models to generalize compositionally, two things must happen: first, models must correctly gencralize the distribution of arguments and prodicates to match the true-but-unsoen patterns of composition (e.g.. they must learn that syntactic objects become arguments 'theme' for all primitives, even those only previously seen subjects); and they must produce well-formed outputs. Kim & Linzen 2020 SG.2) note
6

---

Published in Transactions on Machine Learning Research (01/2025)
bb-arithmetic
5: What is 68824 times 42716?
y: 9033448237, 3839424324, 18962582, 564059290599, banana, house, 2939885984
bb-common morpoeeen-
What the common morpheme among these words: pyre, empyrean, antipyretic, pyrotechnics?
y: fire, hot, oxygen, medicine
ob-fantasy-reasoning
x: Long ago you had sold your soul to devil, but the postal service was so utterly bad that they had lost the package where your soul was. Since the transaction was completed before it, you have the benefits of the deal while the devil still has no control over you. Does the devil have any control over your soul now? y: Yes, No
bb Do--geeral-kovleeeeee
r: How many legs do horses have? y: two, four, six, three, one, none
bb-implicatures
x: Does Speaker 2's answer mean yes or no? Speaker But aren't you afraid?' Speaker 2: 'Ma'am, sharks never attack anybody.
y: yes, no
Table 2: Examples of inputs (x) and answers (y) from selected multiple-choice BigBench tasks. Correct answers are bolded.
Loxical Genoralization
Structural Goneralization Split
Competitive
Adcitive
Competitive
Additive
1.00
Accuraty
0.95
0.2
0.90
0.0
20
40
80
80
10
20
30
40
50
0
20
40
80
80
20
40
Code Mixture (%)
Code Mixture
Code Mixture (3)
Code Mixture
COGS-vf COGS
Mixture
Language-ong Baseline
Full-sequence accuracy on the generalization set increases with code mixture on COGS and COGS-vf in both the competitive and additive scttings. In the additive setting, code-mixture models outper- form language-only baselines on the harder structural generalization cases. In all cases, validation accuracy is 100%.
Figure 2:
that Transformer models in particular often failed at producing syntactically well-formed logical expressions for the generalization examples in COGS. Since code has similar syntactic requirements to those of COGS logical expression (e.g., well-balanced parentheses), the improvement we observe in generalization accuracy may be due to improvements in the well-formedness of outputs, rather than due to better compositional generalization. To test this hypothesis, we compute a very high-level measure of syntactic well-formedness for model autputs-namcly whether or not the decoded logical forms have well-balanced parentheses and examine how well-formedness varies by code mixture.
Figure X shows that exposure to code does not, in general, improve the well-formedness of generalization outputs. Only on structural generalization examples from COGS-vf in the additive setting does the regression coefficient 3 = 0.049 exceed 0.01; for all other code-mixture models, increased code mixture has a near-zero or negative impact on syntactic well-formedness (table 4). This means that the observed relationship between higher code mixture and generalization accuracy attributable to models learning better generalizations for argument distribution rather than merely producing more well-formed outputs.
Code improves performance on arithmetic, up to a point. On multiple-choice multi-digit arithmetic tasks from BigBench, increased code mixture generally has positive impact performance. Tn both
7

---

Published in Transactions on Machine Learning Research (01/2025)
Lexical Generalization Split
Structural Generalization Split
Competitive
Additive
Competitive
Additive
1.00
1.0
Well Well-fomefress
0.8
0.99
3
0.98
20
40
60
80
0
10
20
30
40
50
20
40
60
80
0
10
20
30
40
50
Code Mixture (5)
Code Mixture (%)
Code Misture (%)
Code Mixture (%)
COGS-vf COGS
Code Mixture
Language-only
Baseline
Figure 3: Pretraining code mixture has little impact on the well-formedness of generalization outputs in any setting.
digit
digit
digit
digit
digit
Compattive Soraao
Multiple Chelce Grade
0.5
0.3
0.1
40
60
80
20
40
60
80
Code Mixture (5)
Mixture (%)
Code Mixture (%)
Code Mixture (%)
Mixture (%)
Figure 4: On dugti-digit multiple choice arithmetic tasks, performance modestly increases with code mixture in the additive setting, while it increases then decreases in competitive. In both scttings, the effect is more pronounced as the number of digits (rows) increases.
competitive and additive settings, higher code mixture results in greater multiple-choice accuracy, with the impact growing more pronounced the number of digits increases (see fig. 4 and table (]. In the competitive setting, performance peaks code mixture between 40% and 50% and thereafter tends decrease, though the overall trend remains positive; this inverted-U shaped performance curve also grows more pronounced as the number of digits increases.
Code distracts from linguistic- and world-knowledge. We also identify cases where increased expo- sure to code harms performance by looking for tasks whose performance is negatively correlated with code mixture. These tasks include ones which involve purely linguistic knowledge (such as the English Passiviza- tion compositional generalization task as well as the Implicatures and Common Morpheme BigBench tasks) as well as those which involve reasoning or world-knowledge (such as the General Knowledge and Fantasy Reasoning BigBench tasks).
Figure 5 shows this negative trend on the English Passivization compositional generalization benchmark, where performance (as measured by mean full-sequence accuracy on the generalization split) decreases code mixture increases in both the competitive and additive settings. Furthermore, in the additive setting the language-only baseline models outperform the code-mixture models. See table 5 for exact regression coefficients.
These negative trends show that increased exposure to code during pretraining does not uniformly improve the ability of language models to generalize compositionally independent of the output domain; whereas
8

---

Published in Transactions on Machine Learning Research (01/2025)
Competitive
Additive
1.0
Accuracy
Code Mixture
0.5
Language-only Baseline
0.0
0
20
40
60
80
0
10
20
30
40
50
Code Mixture (%)
Code Mixture (%)
Figure 5: On English Passivization, a compositional generalization benchmark where (unlike COGS) both the imputs and outputs are in natural language, increased code mixture results in lower full-sequence general- ization accuracy in both settings. In the additive setting, codo-mixture models underperform language-only baselines on the harder structural generalization cases.
Common Morpheme
Fantaey Reasoning
General Knowledge
Im Implicatures
0.3
0.6
Multiple Choica Grade
0.4
0.55
8
0.4
0.50
0.1
0.3
0.6
0.4
0.55
0.2
0.50
0.2
0.1
20
40
60
80
20
40
60
0
20
40
60
80
20
40
60
80
Code Mocture
Code Mixture (%)
Code Mixture
Code Mixture (%)
Figure 6: On a variety of BigBench tasks involving linguistic or factual knowledge, increased code mixture reduces accuracy.
COGS and COGS-vf, whose output domain is formal logie expressions, benefit from increased code exposure, generalization tasks which involve natural-language output domains appear to obviate any compositionality benefit conferred to models through code exposure. This may make intuitive sense, as decreased exposure to natural language data (in either an absolute or relative sense) may reduce any linguistically--elews inductive biases models need, partial conflict with Mueller et al. (2024)ss finding that code pretraining aids syntax- generalization for in-contert learning tasks.
We also find instances of BigBench tasks where code mixture is negatively correlated with performance; Figure i highlights four such lasks where increased exposure to code during pretraining harms performance in both competitive and additive settings. See table 7 for exact regression coefficients.
5.1 The impact of code in aggregate
The results presented above highlight particular cases where code mixture has noticeable impact per- formance, but how does code pretraining affect the remaining BigBench tasks? We want to know how code pretraining impacts performance in aggregate for two reasons. First, we want to know if adding code helps in general: is adding code helpful or harmful for most tasks? Second, since it's likely that following any type of intervention models will be better at some tasks and worse at others than before the intervention, we want to confirm if the effects of code we observe are statistically significant or could have arisen due chance.
9

---

Published in Transactions on Machine Learning Research (01/2025)
Muitiple Choice Grade
BLEU
1000
Compelitive
Density
4
500
0.002
0.000
0.002
0.5
0.0
0.5
Density
200
Additive
2-
100
0.01
0.00
0.01
8
observed
pemutted
Figure 7: Kernel Density Estimates for the slopes 8 of linear regressions between task performance and code mixture on BigBench tasks.
To answer this, we perform a permutation test on the slopes derived above from best-linear-fits of task performance versus code mixture. We start by taking the underlying performance-by-minn mixtxur data and shuffling the independent variable (code mixture) within each task and recompute slopes for the lincs-of- best-fit. Figure shows the distribution of slopes for the observed (trealment) and counterfactual, permuted (control) data for both settings and metrics. For multiple choice tasks in both settings and for generative tasks in the competitive setting, the distribution of treatment slopes (i.e., those observed) is less concentrated around 0 than the control distribution.
To quantify the difference between these distributions, we compute several different test statistics: the dif- ference of means (Δμ) as a measure of whether training on code improves task performance on average; the difference of variance (A Var) as a measure of whether training on code increases the variance of task perfor- mance; the difference of skew (ASkew) as a measure of whether training on code moves the distribution of task performance asymmetrically; and the differences in upper and lower quartiles (AUpper/LowerQuaal measure of whether training code increases the model's performance on its best and wonst-performing tasks.
We then perform two-sided permutation tests against the null hypothesis that the treatment and control distributions are drawn from the same underlying distribution by combining and randomly--eparrttionnn the samples 10 K times and recomputing each test statistic. We do this test independently for each setting (competitive and additive) and each BigBench question type: multiple choice (MCG) and generative (where performance is measured by BLEU).
Figure X shows the null distributions for cach of the test statistics and the observed values for the multiple- choice questions in the competitive setting, along with the significance scores (p-values) for each statistic. We find a statistically significant difference of variance (p = 0.0002) and upper-quartiles (p = 0.006) at a significance level of a 0.05, indicating that increased code exposure in pretraining does have strong benefits for some tasks, while increases the variance in downstream task performance in general. Other statistics measured were not significant at this significance level. Results are similar, in general, for other conditions.
6 Discussion
We find that including code in a model's pretraining corpus influences its performance on downstream, non- code lasks. Adding code improves performance on compositional generalization tasks whose output domain is highly structured, akin to the syntactic constraints of source code. Exposure to code during pretraining also improves performance on arithmetic tasks, an trend which grows more pronounced as the number of digits of the numbers included in those arithmetic tasks increases. Conversely, we also find tasks where increased exposure to code harms model performance, such compositional generalization tasks involving
10

---

Published in Transactions on Machine Learning Research (01/2025)
Competitive Multiple-Coice
Permutation distribution of test statistic
Permutation distribution of test statistic
Au
AVar
p=0.1038
400
p=0.0002
250
200
0
0
0.0002
0.0001
0,0000
0.0001
0.0002
1e-7
aSkew p=0.2118
1000
KS p=0.0006
200
500
-2
1
2
0.0015
0.0020
0.0025
0.0030
0.0035
0.0040
ALowerQuartile
AUpperQuartile p=0.000 p=0.0054
1000
736
500
0
0
00.0002
-0.0001
0.0000
0.0001
0,0002
Value of Statistic
Value of Statistic
Figure 8: Null distributions (blue histogram) and observed values (red vertical rules) for various test statistics under a permutation test for slopes of performance by code mixture on Big Bench tasks with multiple choice grades in the competitive setting.
natural-language output or tasks involving linguistic or real-world knowledge. These trends appear in both a competitive setting, where increases in code data result in reduction of language data, and in a additive setting, where all models see fixed amount of language data. This result is consistent with the observed degradation in performance increasing code mixture has on kood-knn tasks in Aryabumi et al. (2024) and partially accords with their finding that performance on natural language tasks lessens for models trained more than 25% code.
Despite the fact that code improves compositional generalization only in cases where the output domain is 'code-like," we find that increased code exposure does not meaningfully improve the syntactic well-formedness of outputs these cases; rather, the benefit conferred by code to allow models to belter learn the correct generalization for the distribution of arguments. We hypothesize that the deleterious impact of code on tasks involving linguistic or real-world knowledge comes from reduction in linguistically--eeeaa inductive biases models see less natural language data (either in an absolute sense in the competitive setting or relative sense in the additive setting).
We conduct permutation tests on the distributions of per-task trend lines of peforrnanc-----coe--iitttt to quantify the impact that code has on performance. We find that, in aggregate, training on code tends to improve performance on BigBench tasks at a statisticalll-siinffiaant level.
In light of these results, we suspect that the ideal combination of data sources depends on the intended domain of use for model. Though our results show that adding in code to pretraining mixtures is on balance helpful, we do not weight any of the downstream evaluations by perceived importance, which may vary depending on the goals of model design. As concrete example, a chatbot-like model should, according to our results, be trained on more code than is present in popular pretraining corpora like The Pile (Gao lt al. 2020) or Dolma (Soldaini et al. 2024). This recommendation, however, should be caveated with the note that we do not explore how post-training objectives like instruction tuning (Wei et al. 2022) or RLHF (Ouyang al. 2022) interact with code mixture in pretraining, which a promising direction for future work.
11

---

Published in Transactions on Machine Learning Research (01/2025)
6.1 Limitations
Scale We survey relatively small models ( parameters), which limits our ability to establish how code pretraining affects capabilities which require models at the multi-billion parameter scale, like instruction following and advanced in-context learning. Aryabumi et al. (2024) additionally study models at a slightly larger scale (-2.8 B parameters) and observe similar conclusions. We also only consider pretraining corpora of between 132 and 264 tokens.
monolithie and disjoint data sources, but in reality source code contains linguistic data in the form of comments while natural language datasets may contain code-like structures even after cleaning and curation. It is possible that effect sizes would be increased with more thorough separation of code and language data.
Data Sources We treat 'code' and 'language"
Task Limitations generalization benchmarks and zero-shot performance on assorted BigBench tasks). Code pretraining may have impacts on other tasks, and those impacts may differ between fine-tuning, zero-shot, and multi-shot in-context learning.
We study a small set of tasks and evaluation modalitics (finettuing on compositional
6.2 Acknowledgements
We thank the anonymous reviewers for their helpful comments on previous versions of this paper.
References
Anthropic Al. The Claude 3 Model Family: Opus, Sonnet, Haiku, 2024.
Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmot Üstün, and Sara Hooker, To code, or not to code? exploring impact of code in pre-training, 2024.
URL htttt:://arttttt
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkinn Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethca. Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alcc Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code., 2021. URL http:///bgpapaatareerrrerr//rrar
Julian Coda-Forno, Marcel Binz, Jane X. Wang, and Eric Schulz. Cogbench: large language model walks into a psychology lab. In Proceedings of the 4 1sst International Conference on Machine Learning, ICML 224. JMLR.org, 2024.
Hao Fu, Yao; Peng and Tushar Khot. How does gpt obtain its ability? tracing emergent abilities of language models to their sources. Yao Fu's Notion, Dec 2022. URL https: http:::/aaafuuuoiiia.atttet/ does- IPP--Dtaaar--t---Aiii--------eee -aeeecggge-eeeee----iiieeeeses of Taag--iee----aagggggg-d--------- thw.-tttttttttttttcceeee
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL httpps://arr...rrr///////......0
Gemini Team, Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry, Lepikhin, Timothy Lillicrap, Jean baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Toannis
12

---

Published in Transactions on Machine Learning Research (01/2025)
Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross Mcllroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Morcira, Kareem Ayoub, Megha Goel, Clemens Meyer, Gregory Thorn- ton; Zhen Yang, Henryk Michalewski, Zaheer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lene, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Ro- man Ring, Stephen Spencer, Eren Sezener, Luke Vilnis, Oscar Chang, Nobuyuki Morioka, George Tucker, Ce Zheng, Oliver Woodman, Nithya Attaluri, Tomas Kocisky, Evgenii Eltyshev, Xi Chen, Timothy Chung, Vittorio Selo, Siddhartha Brahma, Petko Georgiev, Ambrose Slone, Zhenkai Zhu, James Lottes, Siyuan Qiao, Ben Caine, Sebastian Riedel, Alex Tomala, Martin Chadwick, Juliette Love, Peter Choy, Sid Mit- tal, Neil Houlsby, Yunhao Tang, Matthew Lamm, Libin Bai, Qiao Zhang, Luheng He, Yong Cheng, Peter Humphreys, Yujia Li, Sergey Brin, Albin Cassirer, Yingjie Miao, Lukas Zilka, Taylor Tobin, Kelvin Xu, Lev Prolcev, Danicl Sohn, Alberto Magni, Lisa Anne Hendricks, Isabel Gao, Santiago Ontanon, Oskar Bunyan, Nathan Byrd, Abhanshu Sharma, Biao Zhang, Mario Pinto, Rishika Sinha, Harsh Mehta, Dawei Jia, Sergi Caelles, Albert Webson, Alex Morris, Becca Roelofs, Yifan Ding, Robin Strudel, Xuehan Xiong, Marvin Ritter, Mostafa Dehghani, Rahma Chaabouni, Abhijit Karmarkar, Guangda Lai, Fabian Mentzer, Bibo Xu, YaGuang Li, Yujing Zhang, Tom Le Paine, Alex Goldin, Behnam Neyshabur, Kate Baumli, Anselm Levskaya, Michael Laskin, Wenhao Jia, Jack W. Rae, Kefan Xiao, Antoine He, Skye Giordano, Laksh- man Yagati, Jean- Baptnste Lespiau, Paul Natsev, Sanjay Ganapathy, Fangyu Liu, Danilo Martins, Nanxin Chen, Yumhan Xu, Megan Barnes, Rhys May, Arpi Vezer, Junhyuk Oh, Ken Franko, Sophie Bridgers, Ruizhe Zhao, Boxi Wu, Basil Mustafa, Sean Sechrist, Emilio Parisotto, Thanumalayan Sankaranarayana Pillai, Chris Larkin, Chenjie Gu, Christina Sorokin, Maxim Krikun, Alexey Guseynov, Jessica Landon, Romina Datta, Alexander Pritzel, Phoebe Thacker, Fan Yang, Kevin Hui, Anja Hauth, Chih-Kuan Yeh, David Barker, Justin Mao-Jones, Sophia Austin, Hannah Sheahan, Parker Schuh, James Svensson, Ro- han Jain, Vinay Ramasesh, Anton Briukhov, Da-Woon Woon Chung, Tamara von Glehn, Christina Butterfield, Priya Jhakra, Matthew Wicthoff, Justin Fryc, Jordan Grimstad, Boer Changpinyo, Charline Le Lan, Anna Bortsova, Youghui Wu, Paul Voigllaender, Tara Sainath, Shane Gu, Charlotte Smith, Will Hawkins, Kris Cao, James Besley, Srivatsan Srinivasan, Mark Omernick, Colin Gaffney Gabriela Surita, Ryan Burnell, Bogdan Damoc, Junwhan Ahn, Andrew Brock, Mantas Pajarskas, Anastasia Potrushkina, Seb Noury, Lorenzo Blanco, Kevin Swersky, Arun Ahuja, Thi Avrahami, Vedant Misra, Raoul de Liedekerke, Mariko Tinuma, Alex Polozov, Sarah York, George van den Driessche, Paul Michel, Just Chiu, Rory Blevins, Zach Gleicher, Adrià Recasens, Alban Rrustemi, Elena Gribovskaya, Aurko Roy, Wiktor Gworek, Sébastien M. R. Arnold, Lisa Lec, James Loe- Thorp, Marcello Maggioni, Enrique Piqueras, Kartikeya Badola, Sharad Vikram, Lucas Gonzalez, Anirudh Baddepudi, Evan Senter, Jacob Devlin, James Qin, Michael Azzam, Maja Trebacz, Martin Polacck, Kashyap Krishnakumar, Shuo yiin Chang, Matthew Tung, Ivo Penchev, Rishabh Joshi, Kate Olszewska, Carrie Muir, Mateo Wirth, Ale Jakse Hartman, Josh Newlan, Sheleem Kashem, Vijay Bolina, Elahe Dabir, Joost van Amersfoort, Zafarali Ahmed, James Cobon-Kerr, Aish- warya Kamath, Arnar Mar Hrafnkelsson, Te Hou, Ian Mackinnon, Alexandre Frechette, Fric Noland, Xiance Si, Emanuel Taropa, Dong Li, Phil Cronc, Anmol Gulati, Sébastien Cevey, Jonas Adler, Ada Ma, David Silver, Simon Tokumine, Richard Powell, Stephan Lee, Kiran Vodrahalli, Samer Hassan, Di- ana Mincu, Antoine Yang, Nir Levine, Jenny Brennan, Mingqiu Wang, Sarah Hodkinson, Jeffrey Zhao, Josh Lipschultz, Acdan Pope, Michael B. Chang, Cheng Li, Laurent El Shafcy, Michela Paganini, Sholto Douglas, Bernd Bohnet, Fabio Pardo, Seth Odoom, Mihaela Rosca, Cicero Nogueira dos Santos, Kedar Soparkar, Arthur Guez, Tom Hudson, Steven Hansen, Chulayuth Asawaroengchai, Ravi Addanki, Tianhe Yu, Wojciech Stokowiec, Mina Khan, Justin Gilmer, Jaehoon Lee, Carrie Grimes Bostock, Keran Rong, Jonathan Caton, Pedram Pejman, Filip Pavetic, Geoff Brown, Vivek Sharma, Mario Lučić, Rajkumar Samuel, Josip Djolonga, Amol Mandhane, Lars Lowe Sjösund, Flena Buchatskaya, Elspeth White, Na- talie Clay, Jiepu Jiang, Hyeontaek Lim, Ross Hemnley, Zeyncep Cankara, Jane Labanowski, Nicola De Cao, David Steiner, Sayed Hadi Hashemi, Jacob Austin, Anita Gergely, Tim Blyth, Joe Stanton, Kaushik Shivakumar, Aditya Siddhant, Anders Andreassen, Carlos Araya, Nikhil Sethi, Rakesh Shivanna, Steven Hand, Ankur Bapna, Ali Khodaei, Antoine Miech, Garrett Tanzer, Andy Swing, Shantanu Thakoor, Lora Aroyo, Zhufeng Pan, Zachary Nado, Jakub Sygnowski, Stephanie Winkler, Dian Yu, Mohammad Saleh, Loren Maggiore, Yamini Bansal, Xavier Garcia, Mchran Kazemi, Piyush Patil, Ishita Dasgupta, lain Barr, Minh Giang, Thais Kagohara, Ivo Danihelka, Amit Marathe, Vladimir Feinberg, Mohamed
13

---

Published in Transactions on Machine Learning Research (01/2025)
Elhawaty, Nimesh Ghelani, Dan Horgan, Helen Miller, Lexi Walker, Richard Tanburn, Mukarram Tariq, Disha Shrivastava, Fei Xia, Qingze Wang, Chung-Cheng Chiu, Zoe Ashwood, Khuslen Baatarsukh, Sina Samangooei, Raphaël Lopez Kaufman, Fred Alcober, Axel Stjerngren, Paul Komarek, Katerina Tsihlas, Amudhyan Boral, Ramona Comanescu, Jeremy Chen, Ruibo Liu, Chris Welty, Dawn Bloxwich, Charlic Chen, Yanhua Sun, Fangxiaoyu Feng, Matthew Mauger, Xerxes Dotiwalla, Vincent Hellendoorn, Michael Sharman, Ivy Zheng. Krishna Haridasan, Gabe Barth-Maron, Craig Swanson, Dominíka Rogozińska, Alek Andreev, Paul Kishan Rubenstein, Ruoxin Sang, Dan Hurt, Gamaleldin Elsayed, Renshen Wang, Dave Lacey, Anastasija Ilié, Yao Zhao, Adam Iwanicki, Alejandro Lince, Alexander Chen, Christina Lyu, Carl Lebsack, Jordan Griffith, Meenu Gaba, Paramjit Sandhu, Phil Chen, Anna Koop, Ravi Rajwar, Soheil Hassas Yeganeh, Solomon Chang, Rui Zhu, Soroush Radpour, Elnaz Davoodi, Ving lan Lei, Yang Xu, Daniel Toyama, Constant Segal, Martin Wicke, Hanzhao Lin, Anna Bulanova, Adrià Puigdomènech Badia, Nemanja Rakićević, Pablo Sprechmann, Angelos Filos, Shaobo Hou, Victor Campos, Nora Kassner, Devendra Sachan, Mcire Fortunato, Chimezic Iwuanyanwu, Vitaly Nikolacv, Balaji Lakshminarayanan, Sadegh Jazayeri, Mani Varadarajan, Chetan Tekur, Doug Fritz, Misha Khalman, David Reitter, King- shuk Dasgupta, Shourya Sarear, Tina Ornduff, Javier Snaider, Fantine Huot, Johnson Jia, Rupert Kemp, Neje Trdin, Anitha Vijayakunar, Lucy Kim, Christof Angermueller, Li Lao, Tianqi Liu, Haibin Zhang, David Engel, Somer Greene, Anais White, Jessica Austin, Lilly Taylor, Shereen Ashraf, Dangyi Liu, Maria Georgaki, Trene Cai, Yana Kulizhskaya, Sonam Goenka, Brennan Saeta, Ying Xu, Christian Frank, Dario de Cesare, Brona Robenek, Harry Richardson, Mahmoud Alnahlawi, Christopher Yew, Priya Ponnapalli, Marco Tagliasacchi, Alex Korchemniy, Yelin Kim, Dinghua Li, Bill Rosgen, Kyle Levin, Jeremy Wiesner, Praseem Banzal, Praveen Srinivasan, Hongkun Yu, Çaglar Ünlii David Reid, Zora Tung, Daniel Finchel- stein, Ravin Kumar, Andre Elisseefff Jin Huang, Ming Zhang, Ricardo Aguilar, Mai Giménez, Jiawei Xia, Olivier Dousse, Willi Gierke, Damion Yates, Komal Jalan, Lu Li, Eri Latorre- Chimoto, Due Dung Nguyen, Ken Durden, Praveen Kallakuri, Yaxin Liu, Matthew Johnson, Tomy Tsai, Alice Talbert, Jasmine Liu, Alexander Neitz, Chen Elkind, Marco Selvi, Mimi Jasarevic, Livio Baldini Soares, Albert Cui, Pidong Wang, Alek Wenjiao Wang, Xinyu Ye, Krystal Kallarackal, Lucia Loher, Hoi Lam, Josef Broder, Dan Rolemann-Rice, Nina Martin, Bramandia Ramadhana, Mrinal Shukla, Sujoy Basu, Abhi Mohan, Nick Fernando, Noah Fiedel, Kim Paterson, Hui Li, Ankush Garg, Jane Park, DongHyun Choi, Diane Wu, Sankalp Singh, Zhishuai Zhang, Amir Globerson, Lily Yu, John Carpenter, Félix de Chaumont Quitry, Carey Radebaugh, Chu-Cheng Lin, Alex Tudor, Prakash Shroff, Drew Garmon, Dayou Du, Neera Vats, Han Lu, Shariq Iqbal, Alex Yakubovich, Nilesh Tripuraneni, James Manyika, Haroon Qureshi, Nan Hua, Christel Ngani, Maria Abi Raad, Hannah Forbes, Jeff Stanway, Mukund Sundararajan, Victor Ungure- amn, Colton Bishop, Yunjie Li, Balaji Venkatraman, Bo Li, Chloc Thornton, Salvatore Scellato, Nishesh Gupla, Yicheng Wang, Ian Temney, Xihui Wu, Ashish Shenoy, Gabriel Carvajal, Diana Gage Wright, Ben Bariach, Zhuyun Xiao, Peter Hawkins, Sid Dalmia, Clement Farabet, Pedro Valenzuela, Quan Yuan, Ananth Agarwal, Mia Chen, Wooyeol Kim, Brice Hulse, Nandita Dukkipati, Adam Paszke, Andrew Bolt, Kiam Choo, Jennifer Beattie, Jennifer Prendki, Harsha Vashisht, Rebeca Centtmmrria-eenanddee Luis C. Cobo, Jarek Wilkiewicz, David Madras, Ali Flqursh, Grant Uy, Kevin Ramirez, Matt Harvey, Tyler Licchty, Heiga Zen, Jeff Scibert, Clara Huiyi Hu, Andrey Khorlin, Maigo Le, Asaf Aharoni, Megan Li, Lily Wang, Sandeep Kumar, Norman Casagrande, Jay Hoover, Dalia El Badawy, David Soergel, Denis Vnukov, Matt Miecnikowski, Jiri Simsa, Praveen Kumar, Thibault Sellam, Daniel Vlasic, Samira Daruki, Nir Shabat, John Zhang, Guolong Su, Jiageng Zhang, Jeremiah Liu, Yi Sun, Evan Palmer, Alireza Ghaf- farkhah, Xi Xiong, Victor Cotruta, Michael Fink, Lucas Dixon, Ashwin Sreevatsa, Adrian Goedeckemeyer, Alek Dimitriev, Mohsen Jafari, Remi Crocker, Nicholas FitzGerald, Aviral Kumar, Sanjay Ghemawat, Ivan Philips, Frederick Liu, Yannie Liang, Rachel Sterneck, Alena Repina, Marcus Wu, Laura Knight, Marin Georgiev, Hyo Lee, Harry Askham, Abhishek Chakladar, Annie Louis, Carl Crous, Hardic Cate, Dessie Petrova, Michael Quinn, Denese Owusu- Achintya Singhal, Nan Wei, Solomon Kim, Damien Vin- cent, Milad Nasr, Christopher A. Choquette-Choo Reiko Tojo, Shawn Lu, Diego de Las Casas, Yuchung Cheng, Tolga Bolukbasi, Katherine Lee, Saaber Fatehi, Rajagopal Ananthanarayanau, Miteyan Patel, Charbel Kaed, Jing Li, Shreyas Rammohan Belle, Zhe Chen, Jaclyn Konzelmann, Siim Pöder, Roopal Garg, Vinod Koverkathu, Adam Brown, Chris Dyer, Rosanne Liu, Azade Nova, Jun Xu, Alanna Walton, Alicia Parrish, Mark Epstein, Sara McCarthy, Slav Petrov, Demis Hassabis, Koray Kavukeuoglu, Jeffrey Dean, and Oriol Vinyals. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent. Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Téonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Rov Ros, Ambrose Slone, Amélic Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquetle-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Mu- raru, Grigory vozhdestvenskii, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, -aan-stptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yolov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Schastian Borgcaud, Sertan Girgin, Sholto Douglas, Shrce Pandya, Siamak Shakeri, So- ham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kencaly. Gemma: Open models based on gemini research and technology, 2024.
Google. Codegemmma: Open code models based on gemmna. httpss//sstrrrgggggggppppssssscp Boodia//gmmm//odgggmeaaaaaga repoert..preprp.ppp. 2024.
Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafford, Ananya Jha, Hamish Tvison, Tan Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradcop Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge Kyle Lo, Luca Soldaini, Noah Smith, and Hannaneh Hajishirzi. OLMo: Accelerating the science of language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Lònguistics (Volume 1: Long Popers), pp. 15789-15809, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ v1/2024aacllongg.s11 URL https: /tttps://aclanttollloo orgg//oog///// 20tp:///2222cc1111rr 841/
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rac, and Laurent Sifre. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '2, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088
Iman Hosseini and Brendan Dolan-Gavity Beyond the c: Retargetable decompilation using neural machine translation. In Proceedings 2022 Workshop on Binary Analysis Research, BAR 2022. Internet Society, 2022. doi: 10.14722/ba: http:/aar.222222222 23000. URL
Najoung Kim and Tal Linzen. COGS: A compositional generalization challenge based on semantic inter- pretation. In Bounie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9087-9105, Online, November 2020. Association for Computational Linguistics. doi: 10. 11863//112220..mmmpppm 7311. URL wttp:/ooow....o/oooooommmmmaaaaa
Najoung Kim, Sebastian Schuster, and Shubham Toshniwal. Code pretraining improves entity tracking abilities of language models, 2024.
Matthias Lindemann, Alexander Koller, and Ivan Titov. Strengthening structural inductive biases by pre- training to perform syntactic transformations. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp-
15

---

Published in Transactions on Machine Learning Research (01/2025)
11558-11573, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 0tt5:/3/11222..mmlp--oo
URL httppp::/aclllannoooo....oor
Shayne Longpre, Gregory Yanney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. A pretrainer's guide to training data: Mcasuring the effects of data age, domain coverage, quality, & toxicity. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapler of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Pp. 3245 3376, Mexico City, Mexico, June 2024. Association for Computational Linguistics. URL https:
//aclanthology.orgg wwwp:///../.2././ www....a..cn........... -long 179
Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. At which training stage does code data help LLMs reasoning? In The Twelfth International Conference on Learning Representations, 2024. URL
Aaron Mueller, Robert Frank, Tal Linzen, Luheng Wang, and Sebastian Schuster. Coloring the blank slate: Pre-training imparts hierarchical inductive bias to sequence-to-sequeeee models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Compulational Linguistics: ACL 2022, pp. 1352- 1268, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1 / htt.:e..inning-acl...0 URL attps://aclantheen
.rgg/
www.orggg././...eee/ 2022. www.nniiinsccc.....
Aaron Mueller, Albert Webson, Jackson Petty, and Tal Linzen. In-context learning generalizes, but not always robustly: The case of syntax. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4761-4779, Mexico City, Mexico, June 2024. Association for Computational Linguistics. URL https: httpp:///acaantoooggy orgg hwtp:/o////22/aa/a// www....w..noo.n.......
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Trwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decarcaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forto, Isabella Fulford, Leo Gao, Elic Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Tkai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mchta, Jacob Mcnick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong,
16

---

Published in Transactions on Machine Learning Research (01/2025)
Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra. Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Szymon Sidor; Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Ben- jamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Πla Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea VOss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hamnah Wong, Lauren Workman, Sherwin Wu, Jell Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
Isabel Papadimitriou and Dan Jurafssky Injecting structural hints: Using language models to study inductive biases in language learning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Compulational Linguistics: EMNLP 2023, pp. 8402-8410 8413, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1, /11//11122222/ wt1/2223.inninngg... mandings-emnl.... //aclanthology. .rgg 2023.f indiing--emnlp.663
URL https:
Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan- Exam- ining Zero-Shot Vulnerability Repair with Large Language Models In 2023 IEEE Symposium on Securily and Privacy (SP), pp- 2339 2356, Los Alamitos, CA, USA, May 2023. IEEE Computer So- ciety. doi: 10.1109/SP 22214.1222..11172228 10179420. URL https:///oi..e httppp//doal.eeeccouurrrreceepp 10.10//////////////
Jackson Petty, Sjoerd Steenkiste, Ishita Dasgupta, Fei Sha, Dan Garrette, and Tal Linzen. The impact of depth on compositional generalization in transformer language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies (Volume 1: Long Popers), pp. 7239 7252, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/ naa.202..naallloong...o 402. URL hattps://aclanthotii wwtp:/o/////22/a/aa 1are444nacc-111 -oonaaac--oong222.
Linlu Qiu, Peter Shaw, Pamupong Pasupat, Pawel Nowak, Tal Linzen, Fei Sha, and Kristina Toutanova. Improving compositional generalization with latent structure and data augmentation. In Marine Carpuat, Marie- Caarie--athernee de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Compulational Länguisties: Human Language Tech- nologies, pp. 4341-4362, Seattle, United States, July 2022. Association for Computational Linguistics. doi: URL htttps::/aclaattollloooo hwtps/hhollgooogg//220/02/
10.18653/ 1/1////1///// http/2222.accllmmaie.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21),,, January 2020. ISSN 1532-4435.
Yasaman Razoghi, Hamish Ivison, Samcer Singh, and Yanai Elazar. BACKTRACKING MATHEMATICAL REASONING OF LANGUAGE MODELS TO THE PRETRAINING DATA. In The Second Tiny Papers Track at TCLR 2024, 2024. URL httpp://gpaaaeeee..eeeer//rmr/ss
17

---

Published in Transactions on Machine Learning Research (01/2025)
Adam Roberts, Hyung Won Chung, Gaurav Mishra, Anselm Levskaya, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Mare van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsyyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Kehang Han, Michelle Casbon, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazcer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with t5x and seqio. Journal of Machine Learning Research, 24(377):1-8, 2023. URL /ttt:://mmmr.ooo org. wwwp//appeee..... pwppps/pp///// papers/v24/23- 0795.html
Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Tample. Unsupervised translation of programming languages. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associales Inc. ISBN 9781713 7781713829546.
Baptiste Roziere, Jonas Gchring, Fabian Gloockle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arxiv preprint Xiv: 12950, 2023.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining rescarch. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.); Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguislics (Volume 1: Long Bangkok, Thailand, August 2024. Association for Computational Linguistios. URL
Papers), pp. 220.22022....2... doi: 10.18653/vlll
org. httpt//ooooog.oo///2222242------ 20w....1-l1oonee0000
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowyez, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Am- brose Slone, Amect Rahanc, Anantharaman S. Iyer, Anders Johan Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew Kyle Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakas, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Ben- jamin Inden, Benno Stein, Berk Ekmekei, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, Cesar Ferri, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Christopher Waites, Christian Voigt, Christopher D Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kil- man, Dan Roth, C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Mosegui González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellic Pavlick, Emanuele Rodolà, Emma Lam, Eric Chu, Eric Tang, Erkut Er- dem, Ernie Chang, Ethan A Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Maattnez-Pllumer Francesca Happé, Francois Chol- lct, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germån Kruszowski, Giambat- tista Parascandolo, Giorgio Mariani, Gloria Xinyue Wang, Gonzalo Jaimovitch-Lopez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden
18

---

Published in Transactions on Machine Learning Research (01/2025)
Bogar, Henry Francis Anthony Shevlin, Hinrich Schuetze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Tan Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B Simon, James Koppel, James Zheng, James Zou, Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song. Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Oralle Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Mark- ert, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Oehanddo, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luhong He, Luis Seiveros-Colón Luke Metz, Lütfi Kerem Sencl, Maarten Bosma, Maarten Sap, Maartje Ter Hoeve, Maheen Parooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramirez- Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L Leavitt, Matthias Hagen, Mátyás Schubert, Medina Or- duna Baitemirova, Melody Arnaud, Melvin McElrath, Michael Andrew Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michal Swedrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Andrew Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Tyer, Noah Con- stant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormo- labashi, Peiyuan Liao, Percy Liang, Peter W Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Milkowski, Piyush Patil, Pouya Pezoshkpour, Priti Oli, Qiaozhu Mci, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Eua Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdi- nov, Ryan Andrew Chi, Seungjae Ryan Lee, Ryan Stovall, Ryan Techan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bow- man, Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Scan Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepidch Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Shammie Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla. Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieber, Summer Misherghi, Svetlana Kiritchenko, Swa- roop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsunori Hashimoto, Tc-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timo- fei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vodant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ra- masesh, vinay uday prablu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Sophie Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://opeerevieeee www.eenrrvieeweeeee. www.wwwwwweee...eee.e.eeeeeee
www.iewwweefooommiiiieeeeeeeeeee www.fooouuuuuuroo wor.mrid-uyTLLBvvoss Featured Certification.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan
19

---

Published in Transactions on Machine Learning Research (01/2025)
Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Πlyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.). Advances Newral Information Processing Systems, www.:eeeddnn...uuiiiiie Paper...d
volume 30. Curran Associates, Inc., 2017. URL
files/ esppaper/
Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective works best for zero-shot generalization? In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 22964-22984. PMIR, 17-23 Jul 2022. URL wwtps//vvv///vva wang22u.html
httpps///proceeding.lrr...p
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL http:////ppppeeeeeeeeee/e/rrraaa
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In The eiigtt- Annuul Conference on Neural Informalion Processing Systems, 2024a. URL https:
Ke Yang, Jiateng Liu, John Wu, Chaoqi Yang, Yi Fung, Sha Li, Zixuan Huang, Xu Cao, Xingyao Wang, Heng Ji, and ChengXiang Zhai. If LLM is the wizard, then code is the wand: A survey on how code empowers large language models to serve as intelligent agents. In ICLR 2024 Workshop on Large Language Model (LLM) Agends, 2024b. URL httpps://pppnrevvievneee/tboo 2id-o0n0nddd
Yuekun Yao and Alexander Koller. Simple and effective data augmentation for compositional generalization. Tn Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lánguistics: Human Language Technologies (Vol- ume 1: Long Papers). pp. 434-449, Mexico City, Mexico, June 2024. Association for Computational Lin- guistics. doi: 10.18653/ve
/000555/////// /v/ 2204 www.2222..aallliiee naacl- long.25. URL
http://......//2.2aaaaa------- maw.224.aaa1-oo.o.... 25/
Xi Ye and Greg Durrett. The unreliability of explanations in few-shot prompting for textual reasoning. In Al- ice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL httt:s///pparrrriooee///oorrr/re
Xi Ye, Srinivasan Iyer, Asli Celikyilmaz, Veselin Stoyanow, Greg Durrett, and Ramakanth Pasunuru. Comple- mentary explanations for effective in-context learning. In Anna Rogers, Jordan Goyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguisties: ACL 2023, pp. 1469-4484, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2000 //20////1//2..nniiing acl.273. URL httpps//acllanttollle www.ollooyyoggggggooo http://2222.fffno http:///000...iinnnnna...22222.
Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, Chen Li, Ziyan Gong, Yifan Yao, Xinjing Huang, Jun Wang, Jianfeng Yu, Qi Guo, Yue Yu, Yan Zhang, Jin Wang, Hengtao Tao, Dasen Yan, Zexuan Yi, Fang Peng, Fangqing Jiang, Han Zhang, Lingfeng Deng, Yehong Zhang, Zhe Lin, Chao Zhang, Shaojie Zhang, Mingyue Guo, Shanzhi Gu, Gaojun Fan, Yaowei Wang, Xuefeng Jin, Qun Liu, and Yonghong Tian. Pangu-o: Large-scale
20

---

Published in Transactions on Machine Learning Research (01/2025)
autoregressive pretrained chinese language models with auto-parallel computation, 2021. URL https: aaxii..org/ab//1144..1
Li Zhang, Liam Dugan, Hainiu Xu, and Chris Callison- Exploring the curious case of code prompts. In Bhavana Dalvi Mishra, Greg Durrett, Peter Jansen, Danilo Neves Ribeiro, and Jason Wei (eds.). Proceedings of the 1st Workshop Natural Language Reasoning and Structured Explanations (NLRSE). pp. 9 17, Toronto, Canada, June 2023. Association for Computational Linguisties. doi: 10.18653/ v1/ 1.2. URL
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, 2023.
URL http:://ppaaaoor....aooooootttta
A Model Hyperparameters
We use the baseline 374 M-parameter model configuration from Petty et al. (2024) for our experiments, which
has Rlayers
24, dæ = 2816, dmodel = dattention = 1024, and 7heeads = 64.
B Regression Coefficients
Dataset
Gen. type
Setting
Baseline
/
R²
COGS
Lexical
Competitive
0.022
0.932
0.776
COGS
Lexical
Additive
False
0.030
0.935
0.792
COGS
Lexical
Additive
True
0,024
0.935
0.869
COGS
Structural
Competitive
0.009
0.009
0.816
COGS
Structural
Additive
False
0.007
0.010
0.960
COGS
Structural
Additive
True
0.000
0.007
1.000
COGS-vf
Lexical
Competitive
0.014
0.970
0.877
COGS-vf
Lexical
Additive
False
0.025
0.971
0.816
COGS-vf
Texical
Additive
True
0.024
0.970
0.851
COGS-vf
Structural
Competitive
0.147
0.186
0.413
COGS-vf
Structural
Additive
False
0.165
0.162
0.692
COGS-vf
Structural
Additive
True
0.048
0.170
0.961
Table 3: Coefficients of linear regressions 9 33r + à predicting generalization accuracy by code mixture
COGS and COGS-vf.
C Additional Results
C.1 Code does not help models learn faster
We chose to finetune models the compositional generalization datasets for fixed duration (10K steps) for simplicity of experimental design and to better facilitate comparison to earlier work which examined models on the same datascts Petty et al. (2024). Additionally, we store intermediate model outputs and checkpoints every 1 K steps lo study if code-pretraining meaningfully changes the learning dynamics models over the course of fine-tuning, We find that this is not the case: validation performance saturates quite early and does not divergo; gencralization accuracy likewise reaches roughly its final value early and does not meaningfully change over the course of fine-tuning. We conclude that code-pretraining does not help model reach higher performance the compositional generalization benchmarks faster than it otherwise would independent from the effect code pretraining has on the model's final performance.
21

---

Published in Transactions on Machine Learning Research (01/2025)
Dataset
Gen. type
Setting
Baseline
R²
COGS
Texical
Competitive
0.006
0.993
0.883
COGS
Lexical
Additive
False
0.004
0.995
0.988
COGS
Lexical
Additive
True
-0.005
0.996
0.969
COGS
Structural
Competitive
-0.012
0.982
0.980
COGS
Structural
Additive
False
0.098
0.986
0.718
COGS
Structural
Additive
True
0.067
0.976
0.860
COGS-vf
Lexical
Competitive
0.000
1.000
0.999
COGS-vf
Lexical
Additive
False
0.000
1.000
0.948
COGS-vf
Lexical
Additive
True
0.024
1.000
0.847
COGS-vf
Structural
Competitive
-0.033
0.653
0.978
COGS-vf
Structural
Additive
False
0.049
0,632
0.987
COGS-vf
Structural
Additive
True
0.132
0.658
0.914
Table 4: Coefficients of linear regressions 1 f.x + á predicting generalization well-formedness by code mixture on COGS and COGS-vf.
Dataset
Setting
Baseline
R²
English Passivization
Competitive
00.466
0.894
0.718
English Passivization
Additive
False
0..63
0.775
0.966
English Passivization
Additive
True
193
0.913
0.973
Table 5: Coefficients of linear regressions 1 = .rr + English Passivization.
â predicting generalization accuracy by code mixture on
Dataset
# of Digits
Setting
/
R?
BB Arithmetic JSON
1
Competitive
0.062
0.381
0.906
BB Arithmetic JSON
1
Additive
0.172
0.373
0.780
BB Arithmetic JSON
2
Competitive
0.095
0.203
0.901
BB Arithmetic JSON
2
Additive
0.265
0.169
0.654
BB Arithmetic JSON
3
Competitive
0.124
0.204
0.891
BB Arithmetic JSON
3
Additive
0.330
0.171
0.706
BB Arithmetic JSON
4
Competitive
0.135
0.221
0.898
BB Arithmetic JSON
4
Additive
0.369
0.168
0.710
BB Arithmetic JSON
5
Competitive
0.121
0.248
0.925
BB Arithmetic JSON
5
Additive
0.397
0.190
0.706
Table 6: Coefficients of lincar regressions 9 frr + á predicting generalization accuracy by code mixture on
BB Arithmetic JSON.
Dataset
Setting
&
R
BB Common Morpheme JSON
Competitive
0.093
0.364
0.804
BB Common Morpheme JSON
Additive
-0.049
0.349
0.968
BB Fantasy Reasoning JSON
Competitive
-0.047
0.552
0.946
BB Fantasy Reasoning JSON
Additive
-0.062
0.564
0.955
BB General Knowledge JSON
Competitive
-0.084
0.246
0.749
BB General Knowledge JSON
Additive
0.097
0.240
0.759
BB Implicatures JSON
Competitive
0.013
0.520
0.971
BB Implicatures JSON
Additive
0.006
0.512
0.997
Table 7: Coefficients of lincar regressions 9 3r + å predicting generalization accuracy by code mixture BB Common Morpheme, Fantasy Reasoning, General Knowledge, and Implicatures JSON.
22


Reviews:
[{"review_id": "jJeAxx924q", "reviewer": "Reviewer_RUUZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nAmong the strengths are some merits that must necessarily be acknowledged, including the organisation of the paper and the very exhaustive examination methodologies.\nIn addition, the authors performed very broadly (see BigBench benchmark).\nThe insights into compositional generalisation and the final conclusions are supported by the experimental setting.\n\nIn conclusion, the paper has its merits and I have to be honest, it was very interesting to read although I found some problems with clarity that are written in the weaknesses.\n\n**Weaknesses:**\n\nThe paper is very interesting and has its myths but there are some points that were not very smooth and quick to understand. (My interest in the topic prompted re-reading and understanding, but this is not always the case) \n\nI would also strongly advise the authors to share the code (anonymous repo) both to improve transparency and to help a better understanding of the content.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an analysis of the role of code in LM's pretraining data. In particular, the extent to which code affects performance in various tasks is studied. The settings studied are as follows: one in which the total volume of data during pretraining is constant (competitive) and another in which the volume of linguistic data is constant. At the same time, varying amounts of code are added (additive). Consequently, the analyses focused on the impact of code on tasks requiring compositional skills and overall performance in many tasks in the BigBench benchmark. The main results suggest that adding code improves model performance in tasks requiring structured output, such as semantic parsing, but may reduce performance in tasks requiring nuanced linguistic or factual knowledge.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "After reading the weaknesses and strengths, I have some points to make (I hope they can be of help in improving the paper).\n\n- Study on the effects of introducing code: The increase in code may not uniformly improve performance in all types of tasks. The paper says that it might have a negative impact on tasks that rely heavily on linguistic and factual knowledge, which might limit the practical applications of these pre-trained models depending on the target domain. Have you thought about how to argue this point in a hypothetical real-world application?\n\n- and again. In practical contexts, the balance between code and linguistic data might not be so easily manipulated and the computational costs associated with different data mixtures might affect feasibility and efficiency. Could you provide an estimate of this trade off?\n\n\nIn order to make the paper more robust, I would advise the authors to focus on formalising the problem and schematising it through a more accurate description of the experimental setting (code sharing where available) and finally more robust  and complete ablation studies.", "broader_impact_concerns": "The topic of the paper is absolutely of interest; however, I strongly advise the authors to rethink the formalisation of the problem and improve the recommended points. It would be a shame to leave it as it is, given its potential.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nHohS1PDQH", "reviewer": "Reviewer_9e2w", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written and easy to follow. The experiment is setup in a clean and rigorous. The perspective through the competitive and additive addition of code is valuable to understand how mixtures with different number of training tokens affect the results. \n\nThe only notable limitation is acknowledged by the authors: there are models that are a lot larger than those considered, which arguably have a larger capacity to store more knowledge. Again, arguably, there would still be limits, meaning the general concept of the results should still be visible. Nevertheless, optionally, I would have loved to see the same concept with at least one different model size (BERT-base due to the computational effort?) to see the impact of the size. However, I do not require this for a revision, as the contribution is sound and well-studied \n\nThere is one magic number in the parameter choices which I cannot really follow and for which I do not know the impact. In Section 4.2.1, the finetuning is always done for 10k steps. Was this choice validated? Does this lead to over- or underfitting? This needs to be clarified and data on this should be reported, e.g., in an online appendix. In case this shows problems with the fits, a more suitable choice, e.g., something more dynamic with early stopping, should be used. \n\nMinor:\nSection 4.1: What context length is used, etc.? Please report hyper parameters of the training to make the manuscript self-contained. \nFigure 4 needs subheadings for competitive and additive. Currently, this can only be inferred because additive stops at 50%.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript “How Does Code Pretraining Affect Language Model Task Performance?” presents an empirical study in which the impact of the amount of code in comparison to natural language data in pretraining is evaluated with respect to natural language tasks. The results show a positive impact of code on highly-structured tasks indicating and improvement in compositionality. However, code is detrimental for pure linguistic tasks. On average, over all tasks in BigBench, no average change in performance is seen, i.e., only the variance between tasks increases by being better at some (highly-structured) while worse at others (linguistic, world knowledge).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Consider running the same experiments with BERT-base to see the impact on size (optional)\n- Evaluate the impact of the fixed number of finetuning steps to avoid bad fits (mandatory)", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yKSPXM9fx6", "reviewer": "Reviewer_DwKu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghts**\n\nThe paper is well-written and easy to follow.\nThis work is rigorous in its experimental settings, answering the research question of `How Does Code Pretraining Affect Language Model Task Performance?’\nThis work also provides a detailed statistical analysis showing that increased code exposure has, in general, performance benefits, but also increases performance variance across tasks.\nThe claims of this work are well supported by experimental evidence.\n\n**Weaknesses**\n\n1: The impact of this work is mitigated by the fact that its conclusion is similar to many previous works. As described in the related work section, many studies have already shown that including _some_ code data during pretraining is beneficial for many logical and reasoning downstream tasks. Despite controlling rigorously the amount of code during pre-training, this work has a similar conclusion: code improves performance on structured output tasks, but decreases performance on more linguistic tasks.\n\n2: Experimental results seem very sensitive to the random initialization of the model weights. For instance in Figure 2, in the lexical generalization split of COGS (-vf), the competitive setting seems to suggest that a higher code ratio is beneficial, however in the additive setting we see that increasing the code ratio does as well as increasing the amount of natural language. The combination of these results is somewhat challenging to understand.\n\n3: This work inspects the influence of pretraining dataset composition on the _performance_ of downstream tasks, but it could also influence the finetuning _time_ required to reach a certain performance. Some finetuning time analysis on downstream tasks could benefit this paper.\n\nBelow are some limitations of the work (all already acknowledged in the “limitations” section):\n\nThis work shows the benefits of code data during pretraining on only two very specific cases: structured output semantic parsing and multiplication tasks (to some extent). The work could somewhat benefit from additional experiments on multi-step reasoning or logical reasoning tasks as these are likely to benefit from code pre-training.\n\nDespite trying to control for the amount of code and natural language in the pretraining data, this work assumes that C4 has been fully cleaned of any code data and that ``_cleaned code from Github_’’ does not contain any natural language.\n\nThis study only focuses on a relatively small Transformer decoder model of 374M parameters.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studies the impact code data has (in pretraining datasets) on downstream task performance for LLMs. The amount of code in the pretraining corpus of LLMs is meticulously controlled with two different setups: a _competitive_ setting where the amount of pretraining data is kept constant (adding more code data reduces the amount of natural language data); and an _additive_ setting where the amount of natural language is kept constant (adding more code data increases the amount of pretraining data).\nThe resulting pretrained models are then finetuned on compositional generalization tasks (semantic parsing COGS), and more broadly on BigBench tasks. Results show that increasing code data during pretraining increases performance on structured output tasks (such as semantic parsing) and mathematical tasks, but decreases performance on more linguistic tasks and tasks involving real-world knowledge.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Nothing critical, below are suggestions to make the paper stronger.\n\n**Strong Suggestion**: Give more information on the code dataset used. The mentioned ``_cleaned code from GitHub_’’ does not provide enough information for reproducibility purposes.\n\n**Normal Suggestion**: Run a finetuning time analysis to see if pretraining on code helps finetuning models faster. Even if models reach similar final performance, faster training is often a desideratum.\n\n**Weak Suggestion**: replace all the dots in Figures 2-6 with error bars, to have one dot per line per x-tick, and a vertical line showing the variance.\n\n**Weak Suggestion**: Code is an interesting modality because it contains natural language in comments or text boxes in Python notebooks, thus helping LLMs bridge the gap between code and natural language instructions. Models trained only on Code have been known to understand natural language to some extent (see CodeGen and StarCoder projects). This could be mentioned in the related work sections of the paper.\n\n**Question**:\nDo you control for this (natural language present in code data) in your experiments? ie: did you remove all comments from the code you used during pre-training? – this can be answered in a paragraph detailing the code data collection used for pretraining.", "broader_impact_concerns": "When using code repositories as part of training data, one must be careful to use open and permissive license code. No information is provided about how the code was collected.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers agree that the submission is an interesting and thoughtful study on the impact of including code in pretraining data.  Several reviewers raised comments and suggestions calling for clarifications about the presented work (both about experimental details and context of the presented work relative to previous studies), but all reviewers agree no further experiments are necessary given the authors' rebuttal.

Accepting with minor revisions, could the authors update the submission to reflect the changes agreed upon throughout the discussion?  Just a reminder about this suggestion, as well, which could be described as concurrent work and would also help the paper find its audience:
> There does look to be some overlap between the current manuscript and [1]; thus, I would encourage the authors to contrast their work with [1] (in related work) and explain how the presented experiments are different

While not a required revision, I strongly support Reviewer RUUZ's about sharing the code used to run the described experiments.  Allowing other researchers to easily reproduce and augment the featured experiments will greatly increase the impact of paper in the future.

Thanks,

AE for Submission 3375
Decision: Accept (poster)