How Does Code Pretraining Affect Language Model Task Performance?

Authors: jackson  petty
gemini  team
machel  reid
nikolay  savinov
de- nis  teplyashin
  dmitry
timothy  lepikhin
jean  lilli- crap
radu  baptiste alayrac
angeliki  soricut
orhan  lazaridou
julian  firat
ioannis  schrittwieser
rohan  antonoglou
sebastian  anil
an- drew  borgeaud
katie  dai
ethan  millican
mia  dyer
thibault  glaese
benjamin  sottiaux
fabio  lee
mal- colm  viola
yuanzhong  reynolds
james  xu
jilin  molloy
michael  chen
paul  isard
tom  barham
ross  hennigan
melvin  mcilroy
johan  johnson
eli  schalkwyk
eliza  collins
erica  rutherford
ka- reem  moreira
megha  ayoub
clemens  goel
gre- gory  meyer
zhen  thornton
henryk  yang
zaheer  michalewski
nathan  abbas
ankesh  schucher
richard  anand
james  ives
karel  keeling
salem  lenc
siamak  haykal
pranav  shakeri
aakanksha  shyam
roman  chowdhery
stephen  ring
eren  spencer
luke  sezener
oscar  vilnis
nobuyuki  chang
george  morioka
ce  tucker
oliver  zheng
nithya  wood- man
tomas  attaluri
evgenii  kocisky
xi  elty- shev
timothy  chen
vittorio  chung
sid- dhartha  selo
petko  brahma
ambrose  georgiev
zhenkai  slone
james  zhu
siyuan  lottes
ben  qiao
sebastian  caine
alex  riedel
martin  tomala
juliette  chad- wick
peter  love
sid  choy
neil  mittal
yunhao  houlsby
matthew  tang
libin  lamm
qiao  bai
luheng  zhang
yong  he
peter  cheng
yujia  humphreys
sergey  li
albin  brin
yingjie  cassirer
lukas  miao
taylor  zilka
kelvin  tobin
lev  xu
daniel  proleev
alberto  sohn
lisa anne magni
isabel  hendricks
santiago  gao
oskar  ontanon
nathan  bunyan
abhanshu  byrd
biao  sharma
mario  zhang
rishika  pinto
harsh  sinha
dawei  mehta
sergi  jia
albert  caelles
alex  webson
becca  morris
yifan  roelofs
robin  ding
xuehan  strudel
marvin  xiong
mostafa  ritter
rahma  dehghani
abhijit  chaabouni
guangda  karmarkar
fabian  lai
bibo  mentzer
yaguang  xu
yujing  li
tom  zhang
alex  le paine
behnam  goldin
kate  neyshabur
anselm  baumli
michael  lev- skaya
wenhao  laskin
jack w jia
kefan  rae
antoine  xiao
skye  he
lakshman  giordano
jean-baptiste  yagati
paul  lespiau
sanjay  natsev
fangyu  ganapathy
danilo  liu
nanxin  martins
yunhan  chen
megan  xu
rhys  barnes
arpi  may
junhyuk  vezer
ken  oh
sophie  franko
ruizhe  bridgers
boxi  zhao
basil  wu
sean  mustafa
emilio  sechrist
thanumalayan  parisotto
chris  sankara- narayana pillai
chenjie  larkin
christina  gu
maxim  sorokin
alexey  krikun
jessica  guseynov
romina  landon
alexander  datta
phoebe  pritzel
fan  thacker
kevin  yang
anja  hui
chih- kuan  hauth
david  yeh
justin  barker
sophia  mao-jones
hannah  austin
parker  sheahan
james  schuh
rohan  svensson
vinay  jain
anton  ramasesh
da-woon  briukhov
tamara  chung
christina  von glehn
priya  butterfield
matthew  jhakra
justin  wi- ethoff
jordan  frye
chang- pinyo  grimstad
charline  le lan
anna  bortsova
yonghui  wu
paul  voigtlaender
tara  sainath
shane  gu
charlotte will hawkins
kris  cao
james  besley
srivatsan  srinivasan
mark  omernick
colin  gaffney
gabriela  surita
ryan  burnell
bogdan  damoc
junwhan  ahn
andrew  brock
anastasia  petrushkina
seb  noury
lorenzo  blanco
kevin  swersky
arun  ahuja
thi  avra- hami
vedant  misra
raoul  de liedekerke
mariko  iinuma
alex  polozov
sarah  york
george  van den driessche
paul  michel
justin  chiu
rory  blevins
zach  gleicher
adrià  recasens
alban  rrustemi
elena  gribovskaya
aurko  roy
wiktor  gworek
sébastien m r arnold
lisa  lee
james  lee- thorp
marcello  maggioni
enrique  piqueras
kar- tikeya  badola
sharad  vikram
lucas  gonzalez
anirudh  baddepudi
evan  senter
jacob  devlin
james  qin
michael  azzam
maja  trebacz
martin  polacek
kashyap  krishnakumar
yiin  shuo
matthew  chang
ivo  tung
rishabh  penchev
kate  joshi
carrie  ol- szewska
mateo  muir
ale  wirth
hart- man  jakse
josh  newlan
sheleem  kashem
vijay  bolina
elahe  dabir
joost  van amersfoort
zafarali  ahmed
james  cobon-kerr
aishwarya  kamath
arnar mar hrafnkelsson
le  hou
ian  mackinnon
alexandre  frechette
eric  noland
xiance  si
emanuel  taropa
dong  li
phil  crone
anmol  gulati
sébastien  cevey
jonas  adler
ada  ma
david  silver
si- mon  tokumine
richard  powell
stephan  lee
kiran  vodrahalli
samer  hassan
diana  mincu
antoine  yang
nir  levine
jenny  brennan
mingqiu  wang
sarah  hodkinson
jeffrey  zhao
josh  lipschultz
aedan  pope
michael b chang
cheng  li
lau- rent  el
michela  paganini
sholto  douglas
bernd  bohnet
fabio  pardo
seth  odoom
mihaela  rosca
cicero  nogueira
kedar  santos
arthur  soparkar
tom  guez
steven  hudson
chu- layuth  hansen
ravi  asawaroengchai
tianhe  addanki
wojciech  yu
mina  stokowiec
justin  khan
jae- hoon  gilmer
carrie  lee
keran  grimes bostock
jonathan  rong
pedram  caton
filip  pejman
ge- off  pavetic
vivek  brown
mario  sharma
rajku- mar  lučić
josip  samuel
amol  djolonga
lars lowe mandhane
elena  sjösund
elspeth  buchatskaya
natalie  white
jiepu  clay
hyeontaek  jiang
ross  lim
zeyncep  hemsley
jane  cankara
nicola  labanowski
david  de cao
hadi  steiner
jacob  hashemi
anita  austin
tim  gergely
joe  blyth
kaushik  stanton
aditya  shivakumar
anders  siddhant
carlos  an- dreassen
nikhil  araya
rakesh  sethi
steven  shiv- anna
ankur  hand
ali  bapna
an- toine  khodaei
garrett  miech
andy  tanzer
shan- tanu  swing
lora  thakoor
zhufeng  aroyo
zachary  pan
jakub  nado
stephanie  sygnowski
dian  winkler
mohammad  yu
loren  saleh
yamini  maggiore
xavier  bansal
mehran  garcia
piyush  kazemi
ishita  patil
iain  dasgupta
minh  barr
thais  giang
ivo  kagohara
amit  danihelka
vladimir  marathe
mohamed  feinberg
nimesh  elhawaty
dan  ghelani
helen  horgan
lexi  miller
richard  walker
mukarram  tanburn
disha  tariq
fei  shrivas- tava
qingze  xia
chung-cheng  wang
zoe  chiu
khuslen  ashwood
sina  baatarsukh
raphaël lopez saman- gooei
fred  kaufman
axel  alcober
paul  stjerngren
katerina  komarek
anud- hyan  tsihlas
ramona  boral
jeremy  comanescu
ruibo  chen
chris  liu
dawn  welty
char- lie  bloxwich
yanhua  chen
fangxiaoyu  sun
matthew  feng
xerxes  mauger
vincent  dotiwalla
michael  hellendoorn
ivy  sharman
krishna  zheng
gabe  haridasan
craig  barth-maron
dominika  swanson
alek  ro- gozińska
paul kishan andreev
ruoxin  rubenstein
dan  sang
gamaleldin  hurt
ren- shen  elsayed
dave  wang
anastasija  lacey
yao  ilić
adam  zhao
alejandro  iwanicki
alexander  lince
christina  chen
carl  lyu
jordan  lebsack
meenu  griffith
paramjit  gaba
phil  sandhu
anna  chen
ravi  koop
  rajwar
hassas  soheil
solomon  yeganeh
rui  chang
soroush  zhu
elnaz  radpour
ving ian davoodi
yang  lei
daniel  xu
martin  toyama
hanzhao  wicke
anna  lin
adrià  bulanova
nemanja  puigdomènech badia
pablo  rakićević
angelos  sprechmann
shaobo  filos
víctor  hou
nora  campos
devendra  kassner
meire  sachan
chimezie  fortunato
vitaly  iwuanyanwu
balaji  nikolaev
sadegh  lakshminarayanan
mani  jazay- eri
chetan  varadarajan
doug  tekur
misha  fritz
david  khalman
kingshuk  reitter
shourya  dasgupta
tina  sarcar
javier  ornduff
fan- tine  snaider
johnson  huot
rupert  jia
nejc  kemp
anitha  trdin
lucy  vijayakumar
christof  kim
li  anger- mueller
tianqi  lao
haibin  liu
david  zhang
somer  engel
anaïs  greene
jessica  white
lilly  austin
shereen  taylor
dangyi  ashraf
maria  liu
irene  georgaki
yana  cai
sonam  kulizhskaya
brennan  goenka
ying  saeta
christian  xu
dario  frank
brona  de cesare
harry  robenek
mahmoud  richard- son
christopher  alnahlawi
priya  yew
marco  ponnapalli
alex  tagliasacchi
yelin  korchemniy
dinghua  kim
bill  li
kyle  rosgen
jeremy  levin
praseem  wiesner
praveen  banzal
hongkun  srini- vasan
çaglar  yu
david  ünlü
zora  reid
daniel  tung
ravin  finchelstein
andre  kumar
jin  elisseeff
ming  huang
ricardo  zhang
mai  aguilar
jiawei  giménez
olivier  xia
willi  dousse
damion  gierke
komal  yates
lu  jalan
eri  li
  latorre-chimoto
dung  duc
ken  nguyen
praveen  durden
yaxin  kallakuri
matthew  liu
tomy  johnson
alice  tsai
jasmine  talbert
alexander  liu
chen  neitz
marco  elkind
mimi  selvi
  jasarevic
baldini  livio
albert  soares
pidong  cui
wenjiao  wang
xinyu  wang
krystal  ye
lucia  kallarackal
hoi  loher
josef  lam
dan  broder
nina  holtmann- rice
bramandia  martin
mrinal  ramadhana
sujoy  shukla
abhi  basu
nick  mohan
noah  fernando
kim  fiedel
hui  paterson
ankush  li
jane  garg
donghyun  park
diane  choi
sankalp  wu
zhishuai  singh
amir  zhang
lily  globerson
john  yu
félix  carpenter
carey  de chaumont quitry
chu-cheng  radebaugh
alex  lin
prakash  tudor
drew  shroff
dayou  garmon
neera  du
han  vats
shariq  lu
alex  iqbal
nilesh  yakubovich
james  tripura- neni
haroon  manyika
nan  qureshi
christel  hua
maria abi ngani
hannah  raad
jeff  forbes
mukund  stanway
victor  sundararajan
colton  ungure- anu
yunjie  bishop
balaji  li
bo  venkatra- man
chloe  li
salvatore  thornton
nishesh  scellato
yicheng  gupta
ian  wang
xihui  tenney
ashish  wu
gabriel  shenoy
diana  carvajal
wright  gage
ben  bariach
zhuyun  xiao
peter  hawkins
sid  dalmia
clement  farabet
pedro  valenzuela
quan  yuan
ananth  agarwal
mia  chen
wooyeol  kim
brice  hulse
nandita  dukkipati
adam  paszke
andrew  bolt
kiam  choo
jennifer  beattie
jen- nifer  prendki
harsha  vashisht
rebeca  santamaria- fernandez
luis c cobo
jarek  wilkiewicz
david  madras
ali  elqursh
grant  uy
kevin  ramirez
matt  harvey
tyler  liechty
heiga  zen
jeff  seibert
clara huiyi hu
andrey  khorlin
maigo  le
asaf  aharoni
megan  li
lily  wang
sandeep  kumar
norman  casagrande
jay  hoover
dalia el badawy
david  soergel
denis  vnukov
matt  miecnikowski
jiri  simsa
praveen  kumar
thibault  sellam
daniel  vlasic
samira  daruki
nir  shabat
john  zhang
guolong  su
jiageng  zhang
jeremiah  liu
yi  sun
evan  palmer
alireza  ghaffarkhah
xi  xiong
vic- tor  cotruta
michael  fink
lucas  dixon
ashwin  sreevatsa
adrian  goedeckemeyer
alek  dimitriev
mohsen  jafari
remi  crocker
nicholas  fitzgerald
aviral  kumar
sanjay  ghemawat
ivan  philips
fred- erick  liu
yannie  liang
rachel  sterneck
alena  re- pina
marcus  wu
laura  knight
marin  georgiev
hyo  lee
harry  askham
abhishek  chakladar
an- nie  louis
carl  crous
hardie  cate
dessie  petrova
michael  quinn
denese  owusu-afriyie
achintya  singhal
nan  wei
solomon  kim
damien  vincent
milad  nasr
christopher a choquette-choo
reiko  tojo
shawn  lu
diego  de
las  casas
yuchung  cheng
tolga  bolukbasi
katherine  lee
saaber  fatehi
rajagopal  ananthanarayanan
miteyan  patel
charbel  kaed
jing  li
rammohan  belle
zhe  chen
jaclyn  konzelmann
siim  põder
roopal  garg
vinod  koverkathu
adam r brown
chris  dyer
rosanne  liu
azade  nova
jun  xu
alanna  walton
alicia  parrish
mark  epstein
sara  mccarthy
slav  petrov
demis  hassabis
koray  kavukcuoglu
dirk  groeneveld
iz  beltagy
pete  walsh
akshita  bha- gia
rodney  kinney
oyvind  tafjord
ananya  harsh jha
hamish  ivison
ian  magnusson
yizhong  wang
shane  arora
david  atkinson
russell  au- thur
raghavi  khyathi
arman  chandu
jen- nifer  cohan
yanai  dumas
yuling  elazar
jack  gu
tushar  hes- sel
william  khot
jacob  merrill
niklas  morrison
aakanksha  muennighoff
crystal  naik
matthew e nam
valentina  peters
abhilasha  pyatkin
dustin  ravichander
saurabh  schwenk
will  shah
emma  smith
nishant  strubell
mitchell  subramani
pradeep  wortsman
nathan  dasigi
kyle  lambert
luke  richardson
jesse  zettlemoyer
kyle  dodge
luca  lo
noah a soldaini
hannaneh  smith
 2024 hajishirzi
  olmo
colin  raffel
noam  shazeer
adam  roberts
sharan  narang
michael  matena
yanqi  zhou
wei  li
peter j 2023 liu
aarohi  srivastava
abhinav  rastogi
abhishek  rao
abu  awal
md  shoeb
abubakar  abid
adam  fisch
adam  santoro
aditya  gupta
adrià  garriga-alonso
agnieszka  kluska
aitor  lewkowycz
akshat  agarwal
alethea  power
alex  ray
alex  warstadt
alexander w kocurek
ali  safaya
ali  tazarv
alice  xiang
alicia  par- rish
allen  nie
aman  hussain
amanda  askell
amanda  dsouza
ambrose  slone
ameet  rahane
anantharaman s iyer
anders  andreassen
andrea  madotto
andrea  santilli
andreas  stuhlmüller
an- drew  dai
andrew  la
andrew  lampinen
andy  zou
angela  jiang
angelica  chen
anh  vuong
animesh  gupta
anna  gottardi
antonio  norelli
anu  venkatesh
arash  gholamidavoodi
arfa  tabas- sum
arul  menezes
arun  kirubarajan
asher  mullokandov
ashish  sabharwal
austin  herrick
avia  efrat
aykut  erdem
ayla  karakaş
ryan  roberts
bao sheng loe
barret  zoph
bartłomiej  bojanowski
batuhan  özyurt
behnam  hedayatnia
behnam  neyshabur
benjamin  inden
benno  stein
berk  ekmekci
yuchen  bill
blake  lin
bryan  howald
cameron  orinion
cameron  diao
catherine  dour
cedrick  stinson
césar  argueta
chandan  ferri ramírez
charles  singh
chen- lin  rathkopf
chitta  meng
chiyu  baral
chris  wu
chris  callison- burch
christian  waites
christo- pher d voigt
christopher  manning
cindy  potts
clara e ramirez
clemencia  rivera
colin  siro
courtney  raffel
cristina  ashcraft
damien  garbacea
dan  sileo
dan  garrette
dan  hendrycks
dan  kilman
daniel  roth
daniel  freeman
daniel  khashabi
daniel  levy
danielle  moseguí gonzález
danny  perszyk
danqi  hernandez
daphne  chen
dar  ippolito
david  gilboa
david  dohan
david  drakard
debajyoti  jurgens
deep  datta
denis  ganguli
denis  emelin
deniz  kleyko
derek  yuret
derek  chen
dieuwke  tam
diganta  hupkes
dilyar  misra
dimitri coelho buzan
diyi  mollo
dong-ho  yang
dylan  lee
ekaterina  schrader
do- gus  shutova
elad  cubuk
eleanor  segal
eliz- abeth  hagerman
elizabeth  barnes
ellie  donoway
emanuele  pavlick
emma  rodola
eric  lam
eric  chu
erkut  tang
ernie  erdem
ethan a chang
ethan  chi
ethan  dyer
ethan  jerzak
eunice engefu kim
evgenii  manyasi
fanyue  zheltonozhskii
  xia
fernando  siar
francesca  martínez-plumed
francois  happé
frieda  chollet
gaurav  rong
  mishra
indra  genta
gerard  winata
ger- mán  de melo
giambattista  kruszewski
gior- gio  parascandolo
gloria  mariani
gonzalo  wang
gregor  jaimovitch- lópez
guy  betz
hana  gur-ari
hannah  galijase- vic
hannah  kim
ha- jishirzi  rashkin
harsh  mehta
hayden  bogar
henry  shevlin
hinrich  schütze
hiromu  yakura
hongming  zhang
hugh mee wong
ian  ng
jason  wei
jason  yosinski
jekaterina  novikova
jelle  bosscher
jennifer  marsh
jeremy  kim
jeroen  taal
jesse  engel
jesujoba  alabi
jiacheng  xu
ji- aming  song
jillian  tang
joan  waweru
john  miller
john u balis
jonathan  batchelder
jonathan  berant
jörg  frohberg
jos  rozen
jose  hernandez-orallo
joseph  boudeman
joseph  guerr
joseph  jones
joshua b tenenbaum
joshua s rule
joyce  chua
kamil  kanclerz
karen  livescu
karl  krauth
karthik  gopalakrishnan
katerina  ig- natyeva
katja  markert
kaustubh d dhole
kevin  gimpel
kevin  omondi
kory  mathewson
kris- ten  chiafullo
ksenia  shkaruta
kumar  shridhar
kyle  mcdonell
kyle  richardson
laria  reynolds
leo  gao
li  zhang
liam  dugan
lianhui  qin
lidia  contreras-ochando
lütfi kerem metz
maarten  şenel
maheen  bosma
manaal  farooqi
mantas  faruqui
marco  mazeika
marco  baturan
marco  marelli
maria jose maru
marie  ramírez quintana
mario  tolkiehn
martha  giulianelli
martin  lewis
matthew l potthast
matthias  leavitt
mátyás  ha- gen
  schubert
orduna  medina
melody  baitemirova
melvin  arnaud
michael a mcelrath
michael  yee
michael  cohen
ivan- itskiy  gu
michael  starritt
michael  strube
michał  swędrowski
michele  bevilacqua
michihiro  ya- sunaga
mihir  kale
mike  cain
mimee  xu
mirac  suzgun
mitch  walker
mo  tiwari
mohit  bansal
moin  aminnaseri
mor  geva
mozhdeh  gheini
mukund  varma
nanyun  peng
nathan a chi
nayeon  lee
neta  gur-
ari  krakover
nicholas  cameron
nicholas  roberts
nick  doiron
nicole  martinez
nikita  nangia
niklas  deckers
niklas  muennighoff
shirish  nitish
niveditha s keskar
noah  iyer
noah  constant
nuan  fiedel
oliver  wen
omar  zhang
omar  agha
omer  elbaghdadi
owain  levy
pablo  evans
moreno  antonio
parth  casares
pascale  doshi
paul pu fung
  liang
vi- col  paul
pegah  alipoormolabashi
peiyuan  liao
percy  liang
peter  chang
peter  eckersley
mon  htut
pinyu  hwang
piotr  miłkowski
piyush  patil
pouya  pezeshkpour
priti  oli
qiaozhu  mei
qing  lyu
qin- lang  chen
rabin  banjade
rachel etta rudolph
raefer  gabriel
rahel  habacker
ramon  risco
rui  jacobs
ruslan  zhang
ryan  salakhutdinov
ryan  chi
ryan  lee
ryan  stovall
rylan  teehan
sahib  yang
saif m singh
sajant  mohammad
  anand
Venue: EMNLP_2024_Workshop_BlackBoxNLP
Published: 2024-08-16 03:19:37

Abstract: Large language models are increasingly trained on corpora containing both natural language and non-linguistic data like source code. Aside from aiding programming-related tasks, anecdotal evidence suggests that including code in pretraining corpora may improve performance on other, unrelated tasks, yet to date no work has been able to establish a causal connection by controlling between language and code data.
Here we do just this. We pretrain language models on datasets which interleave natural language and code in two different settings: competitive, in which the total volume of data seen during pretraining is held constant; and additive, in which the volume of language data is held constant. We study how the pretraining mixture affects performance on (a) a diverse collection of tasks included in the BigBench benchmark, and (b) compositionality, measured by generalization accuracy on semantic parsing and syntactic transformations. We find that pretraining on higher proportions of code improves performance on compositional tasks involving structured output (like semantic parsing), and mathematics. Conversely, increase code mixture can harm performance on other tasks, including on tasks that requires sensitivity to linguistic structure such as syntax or morphology, and tasks measuring real-world knowledge.

Main paper: 
 introduction: 
 Large language models (LLMs) are increasingly pretrained on corpora containing both natural language text and programming source code (see, e.g., Gemini Team et al. 2024; OpenAI et al. 2024; Anthropic AI 2024; Groeneveld et al. 2024) . While code pretraining straightforwardly improves performance on code-related tasks, its impact in other domains is less clear. Fu and Khot (2022) speculated that code pretraining may be partly responsible for the concomitant rise in reasoning performance in large language models. Observational studies have lent credence to this hypothesis (Yang et al., 2024; Mueller et al., 2024; Kim et al., 2024) , but establishing a causal relationship between code pretraining and non-code downstream task performance is frustrated by the lack of experimental studies which control for relevant hyperparameters to directly compare the utility of training on source code.Here we do just this, constructing pretraining datasets which parametrically vary the amount of data coming from natural-language and sourcecode sources. We then pretrain families of identical models on these datasets and measure their performance on three compositional generalization tasks (COGS, Kim and Linzen 2020; COGS-vf, Qiu et al. 2022; and English Passivization, Mueller et al. 2024 ) and on BigBench (Srivastava et al., 2023) against the amount of code seen during pretraining.We find that code pretraining can significantly improve model performance on some downstream tasks, though other tasks are harmed. Code pretraining strongly benefits compositional generalization in COGS and COGS-vf, where outputs have strict formal syntactic constraints; and on arithmetic tasks from BigBench. Conversely code pretraining can harm compositional generalization in purely * Work done while a student researcher at Google. natural-language domains like English Passivization, and is harmful for some tasks in BigBench which rely on linguistic-or world-knowledge. In aggregate, we find that code pretraining is helpful on average for BigBench tasks. 
related work: 
 Fu and Khot (2022) speculated that code pretraining is partially responsible for the improvement in reasoning capabilities between the -001 and -002 series of GPT-3(.5) of models. Several observational studies have argued code-pretrained models outperform non-code-pretrained models in several domains including reasoning (Yang et al., 2024) , syntax-aware generalization in in-context learning (Mueller et al., 2024) , and entity tracking (Kim et al., 2024 ), though Coda-Forno et al. (2024) find code pretraining does not improve performance on tasks motivated by cognitive psychology.Several experimental studies on the impact of code pretraining have also been conducted. Ma et al. (2024) argues that code pretraining is beneficial for performance on logical reasoning and chain-of-thought capabilities, while Longpre et al. (2024) find that removing code from pretraining corpora is harmful for performance on question answering in a number of different domains; both experimental designs, however, do not fully control for data volume as an experimental hyperparameter (and consequently, any other hyperparameters which are influenced by volume) which prevents the establishment of a clear causal impact. 
methodology: 
 To establish the causal impact of code pretraining, we construct datasets which mix language and code sequences. The ingredients for our datasets are the English portion of the Colossal Cleaned Common Crawl (C4; Raffel et al. 2023 ) and a cleaned corpus of source code from GitHub. Each dataset is pa-rameterized by a single value ∈ [0, 1] denoting the percentage of code data in each mixture.We consider two different experimental setups. In the competitive setting, each dataset has the same number of total tokens; increasing the number of code tokens then amounts to reducing the number of language tokens. This setup provides the clearest way to study the impact of code volume in pretraining since it directly controls for all other factors; however, for high values of the interpretability of results may be skewed by the fact that models will have seen very little language, and so may not be able to adequately understand the tasks themselves.To remedy this, our second additive setting holds constant the number of language tokens across datasets, and additional code tokens are added on top such that the total code-to-language ratio is . This setting ensures that all models have an equal chance to learn from language, but at the cost of controlling for data volume. To address this, we also pretrain models on equal-volume languageonly baselines to compare a counterfactual scenario in which models were simply trained for additional time on non-code data.On these datasets we pretrain 374 M-parameter decoder-only transformer language models with a base data volume of 132 B tokens (total volume in the competitive setting, language volume in additive). We evaluate models in two ways: first, we fine-tune models on the training portions of three compositional generalization datasets (COGS, COGS-vf, and English Passivization) and then measure performance via full-sequence generalization accuracy on the generalization splits of each. Second, we measure zero-shot performance on Big-Bench tasks. For each setting and code mixture we pretraining 5 models from different random seeds and report mean performance across these models for each evaluation. Using these independent trials, we calculate lines-of-best-fit between performance and code mixture and compare slopes ( ˆ ) between tasks. 
results: 
 In a controlled setup, code pretraining does have a meaningful impact on non-code downstream task performance. On compositional generalization tasks whose output has a formal structure, like COGS and COGS-vf, code pretraining significantly improves model performance in both competitive and additive settings. Code pretraining likewise improves performance on the BigBench arithmetic tasks. Conversely, we find cases where code pretraining can harm performance: on the English Passivization compositional generalization task, where models produce natural-language outputs, increased code exposure is harmful in both settings; similar patterns emerge on several BigBench tasks, including common morpheme identification, implicature judgement, and general world knowledge.To quantify the impact of code pretraining in aggregate across BigBench tasks, we consider a null hypothesis that any correlation between task performance and code pretraining could arise from chance. We construct a counterfactual sample by shuffling the code mixture values and recomputing lines-of-best-fit for all BigBench tasks. We then conduct a 10 K-trial two-sided permutation test measuring the difference in mean, variance and upper-and lower-quartile ranges between our treatment and control samples. At a significance level of = 0.05, the differences of variance and upperquartile range are significant. In particular, we find a positive difference in upper-quartile range (and no difference in lower-quartile range), showing that code pretraining results in stronger increases in performance for the best-performing tasks without harming the least-performing tasks more than would otherwise be expected. 
 
 Appendix:  


Decision: Condensed version of full-length arXiv paper. Thorough set of experiments, and the subset chosen here is a good fit for the workshop.