How Do Large Language Monkeys Get Their Power (Laws)?
Authors: Rylan Schaeffer
Joshua Kazdan
John Hughes
Jordan Juravsky
Sara Price
Aengus Lynch
Erik Jones
Robert Kirk
Azalia Mirhoseini
Sanmi Koyejo
Venue: ICML 2025 oral (2025)
Keywords: scaling laws
inference compute
scaling inference compute
language models
evaluations
scaling-predictable evaluations
Primary area: general_machine_learning->evaluation

Abstract: Recent research across mathematical problem solving, proof assistant programming and multimodal jailbreaking documents a striking finding: when (multimodal) language model tackle a suite of tasks with multiple attempts per task -- succeeding if any attempt is correct -- then the negative log of the average success rate scales a power law in the number of attempts.
In this work, we identify an apparent puzzle: a simple mathematical calculation predicts that on each problem, the failure rate should fall exponentially with the number of attempts.
We confirm this prediction empirically, raising a question: from where does aggregate polynomial scaling emerge?
We then answer this question by demonstrating per-problem exponential scaling can be made consistent with aggregate polynomial scaling if the distribution of single-attempt success probabilities is heavy tailed such that a small fraction of tasks with extremely low success probabilities collectively warp the aggregate success trend into a power law - even as each problem scales exponentially on its own.
We further demonstrate that this distributional perspective explains previously observed deviations from power law scaling, and provides a simple method for forecasting the power law exponent with an order of magnitude lower relative error, or equivalently, ${\sim}2-4$ orders of magnitude less inference compute.
Overall, our work contributes to a better understanding of how neural language model performance improves with scaling inference compute and the development of scaling-predictable evaluations of (multimodal) language models.

How Do Large Language Monkeys Get Their Power (Laws)?
Rylan Schaeffer'
Joshua Kazdan
John Hughes Jordan Juravsky' Sara Price* Aengus Lynch
Erik Jones
Robert Kirk
Azalia Mirhoseini
Sanmi Koyejo
Abstract
1. Introduction
Scaling behaviors of large neural language models have surprised and fascinated engineers, scientists and society alike (Hestness al., 2017; Kaplan al., 2020; Brown et al., 2020a; Hoffmann et al., 2022; Ganguli et al., 2022; Sorscher et al., 2022; Wei et al., 2022b; Schaeffer et al., 2023; OpenAl et al., 2024), shaping engineering, economic and governmental interests in frontier Al systems (Bom- masani al., 2021; Eloundou al., 2023; Anderljung et al., 2023; Wang et al., 2023; Reuel et al., 2024; Besiroglu et al., 2024a; Maslej et al., 2024). For more thorough exposition of relevant literature, please see Related Work (Section 6).
Recent research across mathematical problem solving, proof assistant programming and mul- timodal jailbreaking documents a striking finding: when (multimodal) language model tackle a suite of tasks with multiple attempts per task succeed- ing if any attempt correct then the negative log of the average success rate scales power law in the number of attempts. In this work, we identify an apparent puzzle: simple mathematical calcu- lation predicts that on each problem, the failure rate should fall exponentially with the number attempts. We confirm this prediction empirically. raising question: from where does aggregate polynomial scaling emerge? We then answer this question by demonstrating per-problem exponen- tial scaling can be made consistent with aggregate polynomial scaling if the distribution of single- attempt success probabilities heavy tailed such that small fraction of tasks with extremely low success probabilities collectively warp the aggre- gate success trend into power law even each problem scales exponentially on its own. We fur- ther demonstrate that this distributional perspec- tive explains previously observed deviations from power law scaling, and provides a simple method for forecasting the power law exponent with an order of magnitude lower relative error, or equiva- lently, ~2 4 orders of magnitude less inference compute. Overall, our work contributes to a better understanding of how neural language model per- formance improves with scaling inference com- pute and the development of gealing-predictue evaluations of (multimodal) language models.
One direction of renewed interest is inference-time compute scaling, whereby compute is controllably increased at infer- ence to improve the performance of a model, e.g., Pachocki et al. (2024). In this direction, recent research discovered that language model success rates scale predictably with the number of independent attempts made at accomplish- ing task. Specifically, in a paper titled, "Large Language Monkeys: Scaling Inference Compute with Repeated Sam- pling," " Brown et al. (2024) studied how language model performance changes at mathematical problem solving and coding problems when independent attempts are sampled per problem. Performance on the i-th problem was mea- sured using the expected (over attempts) success rate (Kulal et al., 2019; Chen et al., 2021), defined as:
pass; @k def E Attempes
[[Any attempt on i-th problem succeeds]]. (1)
Using the unbiased and numerically stable estimator of Chen et al. (2021) (for details, see Appendix B), Brown et al. (2024) found that the negative log averaged-over problems success rate falls as a power law with the number of independent attempts per problem k:
Stanford Computer Science 2Stanford Statistics SSpeechmatics 'ML Alignment & Theory Scholars 'University College London "Anthropic. Correspondence to: Ry- lan Schacffer sechaff@cs.stanforr.co <sanmi @ es. stanford.eduz.
1 P
Sanmi Koyejo
-log
pass; @k
ak-b
for model-specific and benchmark-specinn speciffc constants a, (Fig. Top). Soon after, on a separate topic of jailbreak- ing multimodal language models via text, image and audio
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

How Do Large Language Monkeys Get Their Power (Laws)?
Large Language Monkeys
papers, see Appendix. C. As minor matter of terminology. both papers frame their results in terms of "coverage" the fraction of problems that can be solved after k attempts per problem - but as Brown et al. (2024) pointed out, coverage is equivalent to the average success rate (Appendix D); we prefer this latter framing as avoids the binary implication that each problem either is or is not solved after attempts.
Model
Pythia 70M
Pythia 160M
-log (peeep6ii))
Pythia 410M
10P
Pythia IB
Pythia 2.8B
Pythia 6.9B
Pythia 12B
Benchmark
2. Should Power Law Scaling Be Expected?
10-1
MATH
10a
10
10a
10
10
Should we expect large language monkeys to have such power (laws)? That is, should the negative log of the av- erage success rate scale polynomially with the number of independent attempts k:? As we now explain mathematically and demonstrate empirically, such polynomial scaling with k is perhaps surprising because, for any single problem, the negative log success rate at & should fall exponentially with k; the intuition is that pass; @k is 1 unless all attempts fail, and since attempts are independent, the probability that all fail is exponentially unlikely with the number of attempts.
Num. Attempts per Problem
Best-of-N Jailbroaking
Model
Claude 3.5 Somnet
-oog (ig(A))))
Claude 3.5 Opus
100
Gemini 1.5 Flash
Gemini 1.5 Pro
GPT4o Mini
GPT4o
10-1
Llama 8B IT
Mudality Text
Mathematically, on any given attempt, the model has proba- bility pass; @1 of solving the i-th problem. Recalling that pass; @k is defined
10
10a
Num, Attempts per Prompt.
if any of the k attempts succeed, 0 otherwise, by linearity of expectation and by independence of the k attempts, we can rewrite pass; @k as:
Figure I: Power Law Scaling in Language Models from Repeat Sampling. Top: Brown al. (2024) found the negative log average pass rate log(passp solving mathematical problems scales polynomially (i.e., as power law) with the number of independent attempts per problem k. Bottom: Hughes et al. (2024) similarly found the negative log average attack success rate- D (kk when jail- breaking multimodal language models scales polynomially with the number of jailbreak attempts per prompt. Should such power law scaling be expected? From where do large language monkeys obtain their power (laws)?
pass @k
E k AAleeppss
[ 1[All k Attempts Fail]
(5)
k
E Attempt
Iiith Attempt Fails]]
(6)
j=l
The probability that the j-th attempt fails is one minus the probability that the j-th attempt succeeds. Since each at- tempt is i..d. with success probability pass; @1, we find
attacks, independent work by Hughes et al. (2024) studied jailbreaking success rates when i independent attempts are made per harmful prompt. Performance was measured using Attack Success Rate (ASR) at k:
pass, @k (1 passs @1)*
(7)
For large k. (1 - pass; @1)* will be small. Recalling that the Taylor Series expansion of log(1 + x) for small æ is (-1) x² /i ze x, we have:
ASR, @k def E k Attempts
(3)
[[nyy attack on i-th prompt succeeds]].
log(pass; @k) = - log (1 -1-(1aass@11)
(8)
This "Best-of-N Jailbreaking" attack similarly discovered that the negative log averaged-over wwwo----eer-rppppe P-prompts attack suc- cess rate fell as a power law with the number of jailbreak attempts per prompt k::
(1 @1)+
(9)
Thus, for any single problem, we should expect the negative log expected (over attempts) success rate to fall exponen- tially with k. not polynomially with k:
P M P
log
ASR
Ck ~ -b
(4)
To confirm this claim, we plotted the scaling of model performance on each problem measured either by log(pass, @k) or by - (og(R - against the num- ber of independent attempts k. We specifically used Brown
for model-specific and modality-specifie constants a, b > 0 (Fig. Bottom). For the specific coefficients from both
2

---

How Do Large Language Monkeys Get Their Power (Laws)?
Average Power Law Scaling
Per-Problem Exponential Scaling + Pass
Distribution Over Problems
10a
10a
10a
Po(pass;01 @1)
(pg(passpkk) x k--
α (pass, a1 )0-1
log (pessp(ikk
loogppass @kk
PP ppaass (1))
10
109
108
10
10-1
10-1
10
102 Num. Attempts per Prohlem &
10
10a
10a
10
10-5
10
10
Num. Attempts per Problem
pass @01
Figure 2: Schematic: The Origin of Power Laws from Scaling Inference Compute via Repeat Sampling, The
log(passp @k) scales as a power law with the number of attempts per problem & (left). This arises from a combination of two factors: (1) for each problem, - -log @k) scales exponentially with & (center), and (2) the distribution (over problems in the dataset) of single-ateempt success rates pass; @1 itself has a left power-law tail of small values (right).
al. (2024)'s data of the Pythia language model family (Biderman et al., 2023) solving 128 mathematical problems from MATH Hendryeks et al. (2021) as well as Hughes et al. (2024)'s data from jailbreaking frontier AI systems Claude, GPT4 (OpenAl et al., 2024), Gemini (Team et al., 2024a;b) and Llama 3 8B Instruction Tuned (IT) (Grattafiori et al., 2024) - on 159 prompts from HarmBench (Mazeika al., 2024). For each individual mathematical problem and jailbreaking prompt,
and k, as well as what properties of the distributions set the scaling exponent. In Appendices E.3-E.8, we derive that several simple distributions yield power law scaling with different exponents whereas others do not:
log pas$tnitorm (51) @k
x k-
log (asSBBet(a.,9) @k
x ka
found the negative log expected (over attempts) success rates fall exponentially with & as expected (Fig. 3), including on Llama 3 8B IT which does not exhibit an aggregate power law (Fig. 1).
log p
α k
log [paStcootinoonseeeooolll//////aa @k
x k1
a))
log
3. Distribution of Per-Problem Single-Attempt Success Rates Creates Power Law Scaling
k
To test this understanding, we examined whether the data of Brown et al. (2024) and Hughes et al. (2024) had per-problem single-attempt success rate distributions that matched one of these simple distributions (Fig. 4). We found that the distributions could indeed be well fit by a 3- 3-parameter Kumaraswamy (a, B, a 0, distribution with scale parameter c (Fig. black dashed lines); we found the scale parameter was critical obtain good fits be- cause the standard 2- 2-parameter Kumaraswamy distribution is supported on (0, 1) whereas most single-attempt success distributions have a smaller maximum such as 0.01 or 0.1.
How does polynomial scaling of the negative log average success rate emerge from exponential scaling of the negative log per-problem success rate? The answer to this question must lie in the distribution D over benchmark problems of single attempt (i.e., k- 1) success rates because this distribution's s density PD ( (pass;01 links the per-problem scaling behavior to the aggregate scaling behavior via the definition of the aggregate success rate passp @k:
passp @k def
pass, @k(pass, @1)]
pass, WIND
More generally, what are the distributional properties that create such power law scaling and that set the specific power law exponent? As now show, the negative log average success rate will exhibit power law scaling in k with exponent b if and only if the distribution over problems of single-attempt success probabilities itself behaves like a power law near 0 with exponent 1:
=1 1 - f (1- pass 2
PD (pass d pass, @1.
(10)
Based on known result that power laws can originate from an appropriately weighted sum of exponential functions (Appendix 1) we begin by considering simple distribu- tions for the single-attempt success probabilities and asking which yield power law scaling between log(pass @k)
Theorem 3.1 (Sufficiency of Power-Law Left Tail in Dis-
3

---

How Do Large Language Monkeys Get Their Power (Laws)?
Large Language Monkeys
Model Pythia 70M
Model Pytlii 160M
Model Pythia 410M
Model Pythia
10
10
10-
107
10
10
10
10
Num. Attenntt per Problem
Model Pythia 288
Model Pythia 6,98
Model = Pythia 123
10
Model
Pythia 70M
lon (k3)
Pythia
109
Pythia 410M
Pythia 1B
10
Pythia 288
Pythia 6.9
10
Pythia 12B
10"
10
10
10
10!
10"
10
10
10a
10
10²
10
10"
10a
Num. Attempts Proclem &
Num. Astemppls per Problem k
Num. Atterapts per Problema &
Best-of-N Jailbreaking
Clundo Soomet
Madel Opts
Cenini
Mode Gemini 1.5
10
logl ASD, (46)
10F
10-3
10:
10
Num. Attemupts per Promut e
Model GPTlo Mini
Model GPTlo
Model Liana SB IT
10
Model
Clande
log (8)))
Clande Oprs
10a
Gemini
Gemini
10-1
GPTo Mini
GPTSo
10-2
Llama SB IT
10"
10
10P
10
10
10"
10a
10a
10
10a
10 10F
10
10
Attemnts per Promapt
Attemupts per Prompt
Attecupts per Prompt
Figure 3: Per-problem performance scales exponentially with the number of attempts per problem A. Top: Pythia language models 128 problems from MATH, with performance the i-th problem measured as - log(ppass @k). Bottom: Frontier AI models on jailbreaking prompts from HarmBench, with performance on the i-th problem measured as - llog(AS)) (ogg,RR, @k). In both settings, on each problem, the negative log per-problem success rate falls exponentially with the number independent attempts k. However, the negative log average success rate falls as a power law with (black).
tribution of Single-Attempt Success Rates), Let be a probability distribution on [0, 1] with PDF pp(pass; @1). Suppose there exist constants b 0, C 0, > and
distribution over pass; @1 E [o, 1] with PDF pD pp(asss @1). Suppose there exist constants b > and A such that for large k.
6 > 0 such that, for all pass @ 1 < 8, we have
log (passp@k) Ak
PD (passs @1) = C-(pass;01) 6-1 + + ((pass(11)--1H
Then, under mild regularity assumptions, the probability density must satisfy
Then, for large k,
A T(b)
Po(pass;@1 @1)
(pass, @1)"-1 as pass; @1 0+.
log passp
~ CT(b) b-
Theorem 3.2 (Necessity of Power-Law Left Tail Dis- tribution of Single-Attempt Success Rates), Let be a
In Fig. 2, we illustrate this connection schematically. For proofs, see Appendices E.8 and E.9. These results clarify

---

How Do Large Language Monkeys Get Their Power (Laws)?
Large Language Monkeys
Model Pythis 70M
Model Pyslia 160M
Madel Pytlii 410M
Model Pythia
100
Coant
60
401
0
10
10a2
10
paess 10
Madel Pythia 2
Model Pythia 6.0B
Model Pythia 12R
100
Model
Pytlii 70M
80
Pythia 100M
Cotaal
Pythia 410M
Pythia 1B
Pythia 288
20
Pythis 6.9
Pythia 12B
10-1
10-2
10p
10
10
10
10
10-2
10
pars 121
paasy AM
pass
Best-of-N Jailbreaking
Clonde
Model Clondi 3.5 Opes
Mode Gemini 1.5 Flush
Model
100
Dbunt
60
0
10
pass
Model GP Mini
Model GPTlo
Model Llama 3 SB
100
Model
Clande 3.5
80
Clande 3.5 Opns
Count
Goraini Flash
Gerzini 1.5 Pro
GPT3o Mini
GPT4o
0
10-11
10
10
10
10--2
100
10
10
109
Llarna 8B IT
80
post
piss:
Figure 4: Single-Attempt Success Rates Distributions Possess Power Law-Like Left Tails. Pythia language models on 128 MATH problems (top) and frontier AI systems 159 HarmBench prompts (bottom) exhibit distributions (over problems) of pass;O1 @1 and ASR; with power law-like tails that are well fit by scaled Beta-Binomial distributions (black dashed lines), which produce aggregate power law scaling. Note that Llama 3 8B Instruction Tuned (IT) does not possess a power law tail, explaining why the model did not exhibit aggregate power law scaling under Best-of-N jailbreaking (Sec. 4).
that whenever- - loogoo(aa (passp @k) exhibits power-law decay in k with exponent b, the distribution over problems of single-attempt success rates must have "polynomial weight" near pass; @1 1 0, i.e. (p( Opp² 6-1).
mial density near "piles up" enough hard problems in just the right way such that even though each of those problems is being solved exponentially quickly, the aggregate success rate over problems decreases at only power-law rate in k. A more succinct mathematical summary is that, for compound binomial distribution, the lower tail probability controls the upper tail of the marginal survivor function.
To offer intuition, we know that each problem is being solved by the model (or equivalently, each prompt is jail- breaking the model) exponentially quickly. If one looks across all problems in the benchmark, some have pass, @1 so small that they remain unsolved for many, many attempts. Whether these "tiny-pase -aay-pas; @1" problems still matter at large k depends on how many such problems there are. Polyno-
5

---

How Do Large Language Monkeys Get Their Power (Laws)?
A
B
[
Score Samples
Estimate pass;
Fit Estimated passplik
12
26
23
52
10
logg peeplll
k
34
45
104
Least Squares Estimator
Pooblenn
104
130
78
100
56
67
78
10a
208
182
10-3
89
234
Estimated
100
Preblen
111
10a
100
1000
100000
Number
10
Astempts &
10a
122
Number of Attempts &
D
E
144
155
100
Fit pass, @1 Distribution
Fit Simulated passp
166
177
188
10
Distributional Estimator
1rpl 7rpppesy/11) (1))
20
10" Migete
Rd à b-
199
210
10
221
232
Estimated Simulated
243
254
Samples
Sucressps
10
10-4
10
10 10 Astempts &
pass 01
Number
Figure 5: Schematic: Two Estimators of Power Law Parameters for Scaling Inference Compute via Repeat Sampling. (A) Both estimators begin by generating many samples per prompt, then computing the number of successes per prompt. In the standard least squares power law parameter estimator (top), (B) pass; @k is estimated for each i-th problem at multiple values, then (C) averaged over problems and fit with linear regression in log-log space. In the distributional power law parameter estimator (bottom), (D) a distribution D is fit to estimates of pass,@11 then (F) the single-attempt success probability distribution is used to simulate passp @k at arbitrary & values for linear regression in log-log space.
4. Lack of Distributional Structure Explains Deviations from Power Law Scaling
Pv (pass, @1) and then simulate how passp Ok will scale with & (Fig. 5) using the relationship:
Notably, previous papers observed that not every model exhibits power law scaling in every setting. To highlight one, Hughes et al. (2024) observed that when jailbrakiigg Meta's Ilama 3 8B Instruction Tuned (TT) model (Grattafiori et al., 2024), the log (ASlR @k) fell faster than any power law (Fig. 1), i.e., the ASR @k rose much more quickly than mathematical
passp @k def
1- 1/ (a-pas;;01)) - pass, @1)*
(11)
pn (pass; @1) d pass;
To empirically test this claim, we compared the standard least squares regression estimator (in log-log space) (Hoff- mann et al., 2022: Caballero et al., 2022; Besiroglu et al., 2024b) against a distributional estimator: To motivate our distributional estimator, we first need explain key ob- stacle and how the distributional estimator overcomes it. The obstacle is that there are problems or prompts whose single-attempl success probabilities pass; @1 lie between (0, 1 //Number of Samples) such that, due to finite sampling, we lack the resolution to measure. While we do not know the true single-attempt success probability for the problems that lie in this interval, we do know how many problems fall into this left tail bucket, and can fit a distribution's 's parameters such that the distribution's probability mass in the interval (0, 1 /umbber of Samples) matches the empirical fraction of problems in this tail bucket. Thus, our distributional esti- mator works by first selecting a distribution (e.g., a scaled 3- 3-parameter Beta distribution), discretizing the distribution
the other frontier AI systems. Based insights and the empirical per-problem single-attempl attack success rates (Fig. 4), we can understand why: Llama 3 8B IT could be successfully jailbroken on every prompt within the permitted sampling budget and thus had no heavy left tail necessary to create the aggregate power law scaling.
5. A New Distributional Estimator for Predicting Power Law Scaling
natural consequence of this connection between the seal- ing of log(passp@k) and the left tail of the distribution pp pp(pass; @1) is that the distribution of single-attempt suc- cess rates can be used to predict whether power-law scaling will appear and if so, what the intercept and exponent of the power law will be. To do this, one can fit the distribution
6

---

How Do Large Language Monkeys Get Their Power (Laws)?
Power Exponent [Leost Squares)
Large Langunge Monkeys
Power Law Exponent [Leost Squares)
Best-of-N Jailbroaking
0.6
0.6
Model
Model
0.5
Pythia
0.5
Claude 3.5 Sounet
Pythia 160M
Clande 3.5 Opos
0.4
Pythia 410M
Gemini 1.5 Flash
0.3
Pythia
Gemini Pro
Pythia 2.8B
GPT4o Mini
0.2
Pythia 6.98
0.2
GPT40
Pythia 12B
Llama
0.1
0.1
Beachmark
Modality
0.0.
0.0
0.6
MATH
0.0
0.2
0.4
Text
Power Law Exponent (Kumnraswann- Binomiill)
Power Law Exponent (Kumnraswamy- Bimomiill)
Figure 6: Comparing Estimators of Power Law Exponents. We compare two estimators of the power law exponent in
log (passps @k) ak- h (1) the standard least-squares estimator between and log (passp@k) in log-log space, and (2) the distributional estimator of pass; @1 assuming scaled Kumaraswamy-Binomin distribution. Using all available data to fit both estimators, we find agreement between the least-squares estimate (ordinate) and the distribution-derived estimate (abscissa) for both Pythia models on MATH (left) and for frontier Al systems on HarmBench (right). For an explanation of why the two estimators match more closely for Large Language Monkeys than for Best-of-N Jailbreaking, see Appendix A.
Thue Distribution
Fit Method
Betaj0.15 15, 3.5. 0 0.1)
Beaa 15, 5, 3.0,0 0.1)
Loast Squares
Dolaiie Eave [6- b/jo
100
Discresioed
Disceesaeed Kummzewwayy
Nuar. Problems
51
Betaj0 0 08)
Beta 08)
Rolative Eror 6- 0//6
10P
Nom: Sumples per
Nom. Sumples per
Figure 7: Comparing Two Estimators of Power Law Exponents via Backtesting. On synthetic data with known ground-truth power law -b, compare how well the least squares and the distributional estimator recover the scaling exponent as measured by the relative error 1B - bl/b b by backtesting: subsampling the number of problems and the number of samples per problem. We find that the distributional estimator obtains significantly better sample efficiency.
according to the sampling resolution 1 I/Number of Samples and performing maximum likelihood estimation under the discretized distribution's probability mass function.
Second, the distributional estimator also comes with an- other benefit: it directly provides an estimate the power law's exponent 6 in k-6 Estimating the power law's ex- ponent especially valuable because the exponent dictates how success rates are improving with increasing inference compute. To test how the distributional estimator and least squares estimator compare at recovering the true asymptotic power law exponent, we generated synthetic data so that we would have ground-truth knowledge of the true power law exponent, then backtested how the scaling estimators compare recovering the true exponent (Alabdulmohsin et al., 2022a; Owen, 2024) by subsampling data with fewer
We tested this distributional estimator in two different ways. First, focusing on Large Language Monkeys, we used all available real data from all problems and all samples per problem to compare the standard least squares regression estimator against the distributional estimator. We found close agreement between the two estimators (Fig. 6), giving us sense that the estimators yield reasonably consistent estimates under large sampling budgets.

---

How Do Large Language Monkeys Get Their Power (Laws)?
problems and fewer samples per problem. We found that the distributional estimator obtains significantly better sample efficiency, with approximately order of magnitude lower relative error
has explored emerging phenomena such as inverse scaling (McKenzie et al., 2024), unique functional forms (Caballero et al., 2022), scaling patterns across model families (Ruan et al., 2024; Polo et al., 2024), and downstream capabili- ties (Srivastava et al., 2023; Wei et al., 2022a; Hu et al., 2024; Schaeffer et al., 2024b; Snell et al., 2024b; Wu & Lo, 2024). Researchers have also investigated critical chal- lenges including data contamination (Schaeffer, 2023; Jiang et al., 2024; Dominguez-Olmedd et al., 2024), model-data feedback loops (Dohmatob et al., 2024; Gerstgrasser et al., 2024; Kazdan et al., 2024), and overtraining effects (Gao et al., 2023; Gadre et al., 2024). Additional contributions include studies in sparse autoencoders (Gao et al., 2024), biologically-plusibbb backpropagation (Filipovich et al., 2022), and self-supervised learning for vision (Schaeffer et al., 2024a). Recent efforts have also focused on reconeil- ing apparent contradictions in scaling behaviors (Besiroglu et al., 2024b; Porian et al., 2024).
def _ - bl/b compared with the least squares estimator (Fig. 7), or equivalently, ~2 - 4 orders of magni- tude less inference-compuce The distributional estimator performs well even under distributional mismatch.
6. Related Work
Research into scaling laws of deep neural networks has a rich history spanning theoretical foundations, empirical vali- dations, and diverse applications. The earliest investigations discovered power law scaling in simple machine learning settings (Barkai al., 1993; Mhaskar, 1996; Pinkus, 1999). However, the modern era of scaling laws began with break- through studies in neural language models (Hestness et al., 2017; Kaplan et al., 2020; Brown et al., 2020b), catalyzing extensive research across multiple directions. The theoreti- cal understanding of scaling laws has advanced significantly (Spigler et al., 2020; Bousquet et al., 2020; Hutter, 2021; Sharma& & Kaplan, 2022; Maloney et al., 2022; Roberts et al., 2022; Bahri et al., 2024; Michaud et al., 2024; Paquette et al., 2024; Atanasov et al., 2024; Bordelon et al., 2024a;b; Lin et al., 2024; Brill, 2024), complemented by comprehensive empirical studies (Rosenfeld et al., 2020; Henighan et al., 2020; Gordon et al., 2021; Tay et al., 2021; Ghorbani et al.,
7. Discussion and Future Directions
This work advances our mathematical understanding of how and why language model performance improves with additional inference compute through repeat sampling. By establishing rigorous theoretical foundations for these empirically- power laws, our work provides prac- titioners with principled ways to understand and predict model performance when scaling inference compute. The distributional perspective we develop explains previously puzzling deviations from power law scaling and enables more efficient estimation of scaling parameters.
2021; Tay et al., 2022b; Zhai et al., 2022; Alabdulmohsin et al., 2022b; Dehghani et al., 2023; Bachmann et al., 2023). In the context of language models, researchers have explored scaling behaviors in various aspects: context length (Xiong et al., 2023), in-context learning (Chan et al., 2022; Agarwal et al., 2024; Arora et al., 2024), vocabulary size (Tao et al., 2024), and jailbreaking attempts (Anil et al., 2024; Hughes et al., 2024). Studies have also investigated scaling dynam- ics in fine-tuning (Kalajdzievski, 2024; Zhang al., 2024), transfer learning (Hernandez et al., 2021), and the impact of repeated data (Hernandez et al., 2022; Muennighoff et al., 2023). Architectural considerations have been extensively studied, including network design (Tay et al., 2022a; Clark et al., 2022), nested models (Kudugunta et al., 2023), prun- ing strategies (Rosenfeld et al., 2021), and precision require- ments (Dettmers& & Zettlemoyer, 2023; Kumar et al., 2024; Sun al., 2025). Research has also addressed multimodal extensions (Aghajanyan et al., 2023; Cherti et al., 2023) and inference optimization (Sardana et al., 2023; Brown et al., 2024; Snell et al., 2024a; Wu et al., 2024; Chen et al., 2024). The field has expanded to encompass diverse domains in- cluding reinforcement learning (both single-agent (Jones, 2021; Hilton et al., 2023; Neumann Gros, 2024) and multi-agent (Neumann & Gros, 2022)), graph networks (Liu al., 2024), diffusion models (Mei et al., 2024; Liang et al., 2024), and associative memory models (Romani et al., 2013; Cabannes et al., 2024; Schaeffer et al., 2024c). Recent work
Two related questions are why such distributional structure exists in the single-attempt success rates and whether one should expect such structure to appear in future benchmarks. We conjecture there are at least two reasons: (I) benchmark design, in that benchmarks are intentionally crafted that problems have a spread of difficulty without being too easy too hard, and (2) selection bias, in that more interesting patterns such as power law scaling are more likely to garer more interest from the research community.
Despite focusing on scaling inference compute, our paper contributes a new hypothesis for an open question in scaling pretraining compute: why are neural scaling laws power laws? Just as the scaling behavior of logg (rassp@k) @k) only becomes clear for large k. so too might the scaling behav- ior of pretraining cross entropy with pretraining compute C. Specifically, suppose the pretraining cross entropy L as a function of pretraining compute C' is a sum of many functions which decay at different rates:
((2)))
A Ca
[(C) -
+
+
C
where a is the smallest (positive) polynomial exponent and
8

---

How Do Large Language Monkeys Get Their Power (Laws)?
(1/CC% represents functions that decay more slowly than any polynomial. Initially, for small C, the dominant term may be unclear, but pretraining compute is scaled up across 8 - 10 orders of magnitude, the leading order term dominates and an approximate power law emerges:
[(C) const +
as C OC.
Thus, a power law relationship may only be reasonable for sufficiently large pretraining compute C, which in turn may require excluding the lowest pretraining compute models in order to obtain good predictions, justifying a widespread empirical practice (Kaplan et al., 2020). We designate possi- ble functions hiding (1/C²) and o(1/C") as the dark matter of neural scaling laws.
Acknowledgments
RS acknowledges support from Stanford Data Science and the OpenAl Superalignment Fast Grant. SK acknowledges support by NSF 2046795 and 2205329, IES R305C240046, ARPA-H, the MacArthur Foundation, Schmidt Sciences, OpenAl, and Stanford HAL.
Impact Statement
Our findings have important practical implications for the deployment of large language models, as they can help or- ganizations more accurately forecast compute requirements and make informed trade-offss between model size, inference costs, and performance targets. The mathematical frame- work we develop could also generalize beyond language models to other domains where similar scaling phenom- ena emerge. While our work is primarily theoretical, we acknowledge that advances in language model capabilities can have broad societal impacts. We hope that better under- standing these fundamental scaling behaviors will help the research community develop more efficient and reliable AI systems.

---

How Do Large Language Monkeys Get Their Power (Laws)?
References
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. Ex- plaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27):62311888122 2024.
Agarwal, R., Singh, A., Zhang. L. M., Bohnet, B., Rosias, L., Chan, S. C., Zhang, B., Anand, A., Abbas, Z., Nova, A., Co-Reyes, J. D., Chu, E., Behbahani, F., Faust, A., and Larochelle, H. Many-shot in-context learning. In The Thirty-eighth Annual Conference on Neural In- formation Processing Systems, 2024. URL https: //openreview wwt./e//oouuie weeeiniddAAABBMA AB6XpMZvQH.
Barkai, N., Seung, H. S., and Sompolinsky, H. Scaling laws in learning of classification tasks. Physical review letters, 70(20):3167, 1993.
Besiroglu, T., Emery-Xu, N., and Thompson, N. Economic impacts of ai-augmented r&d. Research Policy, 53(7): 105037, 2024a.
Aghajanyan, A., Yu, L.., Conneau, A., Hsu, W.-N., Ham- bardzumyan, K., Zhang, S., Roller, S., Goyal, N., Levy, O., and Zettlemoyer, L. Scaling laws for generative mixed-mooee modall language models. In International Confer- ence on Machine Learning, pp. 269-279. PMLR, 2023.
Besiroglu, T., Erdil, E., Barnet,, M., and You, J. Chinchilla scaling: A replication attempt, 2024b. URL https: //arxiv. org/ httpaa////24...000
Alabdulmohsin, I., Neyshabur, B., and Zhai, X. Revisiting neural scaling laws in language and vision, 2022a. URL https: / arxiv. org/ abs 2209.022222 00000.066640 06640.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., Orien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for ana- lyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Alabdulmohsin, I. M., Neyshabur, B., and Zhai, X. Revisit- ing neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300- 22312, 2022b.
Bochud, T. and Challet, D. Optimal approximations of power-laws with exponentials, 2006. URL https: arxiv, arxiv.org/and haasss/hyssio ics ctnoyss4cs/0054455 0605149.
Anderljung, M. Barnhart, J.. Korinek, A., Leung, J., OeKeefe, C., Whittlestone, J., Avin, S., Brundage, M., Bullock, J., Pass-BBggss D., Chang, B., Collins, T., Fist, T., Hadfield, G., Hayes, A., Ho, L., Hooker, S., Horvitz, E., Kolt, N., Schuett, J., Shavit, Y., Siddarth, D., Trager, R., and Wolf, K. Frontier ai regulation: Managing emerging risks to public safety, 2023. URL https: // /wwxiv..vvv. orgg abs/ 2307.03000 03718.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07755, 2021. Bordelon, B., Atanasov, A., and Pehlevan, C. A dy- namical model of neural scaling laws. arXiv preprint arXiv:2402.01092, 2024a.
Anil, C., DURMUS, E., Rimsky, N., Sharma, M., Ben- ton, J., Kundu, S., Batson, J., Tong, M., Mu, J., Ford, D. J. Mosconi, F, Agrawal, R., Schaeffer, R., Bashkan- sky. N., Svenningsen, S.. Lambert, M., Radhakrishnan, A., Denison, C., Hubinger, J., Bai, Y., Bricken, T., Maxwell, T., Schiefer, N., Sully, J., Tamkin, A., Lan- ham, T., Nguyen, K., Korbak, T., Kaplan, Ganguli, D., Bowman, S. R., Perez, E., Grosse, R. B., and Duvenaud, D. Many-shot jailbreaking. In The eighth- Annual Conference on Neural Information Processing Systems, 2024. URL wwwps:tt///.... http://ppereeeeeeeeeeeee forum? id=cw5mgd71jj.
Bordelon, B., Atanasov, A., and Pehlevan, C. How feature learning can improve neural scaling laws. arXiv preprint arXiv:2409.1783 2024b.
Bousquet, O., Hanneke, S., Moran, S., van Handel, R., and Yehudayoff, A. A theory of universal learning, 2020. URL https:tp////.. https::/aaxiv..r/ wwwaaaxxvv.oo.....a abs 20111. 2011.0001 04483.
Brill, A. Neural scaling laws rooted in the data distribution. arXiv preprint arXi::2412.07942. 2024.
Arora, A.. Jurafsky, D., Potts, C., and Goodman, N. D. Bayesian scaling laws for in-context learning, 2024. URL https:/ //tp://aaavv.//aa//eeeeeeeeaaevr org/ h abs 2410. 16531.
Brown, B., Juravsky. J., Ehrlich, R., Clark, R., Le. Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling, 2024. URL /ttp///axiii http:////xxvvvoroeeee /2007 2407.21787.
Atanasov, A., Zavatone- Veth, J. A., and Pehlevan, C. Scal- ing and renormalization in digh--imensional regression. arXiv preprint N0N.:244500000222 2024.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020a.
Bachmann, G., Anagnostidis, S., and Hofmann, T. Scaling mlps: A tale of inductive bias, 2023. URL ht htsp:: //arxiv marxiv..o org/ abs 2206. 13575.
10

---

How Do Large Language Monkeys Get Their Power (Laws)?
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-282)) 2023.
Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler. D. M., Wu,
Clark, A., de Las Casas, D., Guy, A., Mensch, A., Paganini, M., Hoffmann, J., Damoc, B., Hechtman, B., Cai, T., Borgeaud, S., et al. Unified scaling laws for routed lan- guage models. In International conference on machine learming, pp. 4057-4086. PMLR, 2022.
J., Winter, C., Hesse, C., Chen, M., Sigler, E.,, Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners, 2020b. URL https:// arxiv. org/ abs 2005 14165.
Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., et al. Scaling vision transformers 22 billion parameters. In International Conference on Machine Learning, pp- 7480-7512. PMLR, 2023.
Caballero, E., Gupta, K., Rish, I., and Krueger, D. Broken neural scaling laws. arXiv preprint arXiv:2210.1488 2022.
Cabannes, V., Dohmatob, E., and Bietti, A. Scaling laws for associative memories, 2024. URL https: arxiv. org/abs/2311 02984.
Dettmers, T. and Zettlemoyer, The case for 4-bit preci- sion: k-bit inference scaling laws. In International Con- ference Machine Learning, pp. 7750-7774. PMLR, 2023.
Chan, S., Santoro, A., Lampinen, A., Wang. J., Singh, A., Richemond, P., McClelland, J., and Hill, Data distributional properties drive emergent in-context learning in transformers.
Dohmatob, E., Feng, Y., Yang, P., Charton, F,, and Kempe, J. A tale of tails: Model collapse as a change of scal- ing laws, 2024. URL https: /aaaaaaxvvvorr../eaa abs/ 2402.07043
Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho. K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 18878-18891, Curran Associates, Inc., 2022. URL https: /ttppe::/pprcceeedingg... neurips. cc co/paper [eapper_fieessapprr//2//////// (rc6cacf3362222277f666666663 pdf.
Dominguez-Olmedo, R., Dorner, F. E., and Hardt, M. Train- ing on the test task confounds evaluation and emer- www.eennneeen22222 2024. URL
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto,
Elkies, N. Is there way to express an power law de- cay as series of exponentials? MathOverflow, 2016. URL pttps::/mattoovrrffte URLLhht.///mattoeefoooweee/22221 (version: 2016- 10-08).
H. P., Kaplan, J.. Edwards, H., Burda, Y.. Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder. N., Pavlov, M., Power, A., Kaiser, L., Bavar- ian, M., Winter, C., Tillet, P., Such, P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji. S., Jain, S.. Saun- ders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., Me- Grew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained code, 2021. URL https: /aaiiiiiiivv//..v//aaaaaaaiiavva org/ abs/ 2107. 03374.
net/q/25/
Eloundou, T., Manning, S., Mishkin, P., and Rock, D. Gpts are gpts: An early look at the labor market im- pact potential of large language models, 2023. URL https: /ttp///axvvv//v///////eee/e/aae/ wttp://wwwrr/ba///.....o
Filipovich, M. J., Cappelli, A., Hesslow, D., and Launay, J. Scaling laws beyond backpropagation, 2022. URL https: arxii. wtw.vvorrgggass/ee wttpavv..rr/bbb//21..14444 12599
Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.085400 2024.
Chen, Y., Pan, X., Li, Y., Ding, B., and Zhou, J. A simple and provable scaling law for the test-time compute of large language models, 2024. URL https: /arxiv.. org/ arg/abs/ 221..111..1111 19477.
Ganguli, D., Hernandez, D., Lovitt, L.., Askell, A., Bai, Y., Chen, A., Conerly, T., Dassarma, N., Drain, D., Elhage, N., et al. Predictability and surprise in large generative models. In 2022 ACM Conference on Fairness, Account- ability, and Transparency, pp. 1747-1764, 2022.
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J. Reproducible scaling laws for con- trastive language-image learning. Proceedings of the

---

J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Prasad, K., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., Iyer, K. Malik, K., Chiu, K., Bhalla, K., Lakhotia, K., Rantala- Yeary, L., van der Maaten, L., Chen, L.. Tan, L., Jenkins, L., Martin, L.,, Madaan, L.. Malo, L.., Blecher, L.., Landzaat, L.. de Oliveira, L.. Muzzi, M.. Pasupuleti, M., Singh, How Do Large Language Monkeys Get Their Power (Laws)?
Gao, L.,, Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 10835-10866. PMLR, 23-29 Jul 2023. URL https: /prrceeeeidggss... mlr, press/ v202/ 3h. html.
Fu, J., Chi, J., Huang, J., Liu, J., Wang. J., Yu, J., Bitton,
M., Paluri, M., Kardas, M., Tsimpoukelli, M., Oldham, M. Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N.. Chatterji. N., Zhang, N., Duchemne, O., Çelebi, O., Airassy, P., Zhang, P., Li, P., Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, S., Xu, P., He, Q. Dong, Q., Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S., Stojnic, R., Raileamu, R., Maheswari, R., Girdhar. R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang. R., Hosseini, S., Chennabasappa, S.. Singh, S., Bell, S., Kim. S. S., Edunov, S., Nie, S., Narang, S., Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang, S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Herman, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speck- bacher, T., Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do, V., Vogeti, V., Albiero, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Wang, X., Tan, X. E., Xia, X., Xie, X., Jia, X., Wang, X., Gold- schlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y.. Zhang, Y., Li, Y., Mao, Y., Coudert, Z. D., Yan, Z.., Chen, Z.,, Papakipos, Z., Singh, A., Srivastava, A., Jain, A., Kelsey, A., Shajnfeld. A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Teo, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poul- ton, A., Ryan, A., Ramchandani. A., Dong, A.. Franco, A., Goyal, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, Liu, C., Wang, C., Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C., Feichtenhofer, C., Gao, C., Civin, D., Beaty, D.,, Kreymer. D., Li, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D.., Foss, D., Wang, D., Le, D., Holland, D., Dowling, E., Jamil, E., Mont- gomery, E., Presani, E., Hahn, E.,, Wood, E., Le, E.-T., Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, F., Kreuk, F., Tian, F., Kokkinos, F., Ozgenel, F., Cag- gioni, F,, Kanayet, F,, Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpem, G., Herman, G., Sizov, G., Guangyi, Zhang, Lakshminarayanan, G., Inan, H.,
Gao, L., la Tour, T. D., Tillman, H., Goh, G. Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scal- ing and evaluating sparse autoencoders. arXiv preprint arXiv:2406. 04093, 2024.
Gerstgrasser, M., Schaeffer, R., Dey. A., Rafailov, R., Sleight, H., Hughes, J., Korbak, T,, Agrawal, R., Pai, D., Gromov, A., Roberts, D. A., Yang. D., Donoho, D. L., and Koyejo, S. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data, 2024. URL https:::: arwii.iiiiiv... org/ abs / 2404. 01413.
Ghorbani, B., Firat, O., Freitag, M., Bapna, A., Krikun, M., Garcia, X., Chelba, C., and Cherry, C. Scaling laws for neural machine translation. In International Conference on Learning Representations, 2021.
Gordon, M. A., Duh, K., and Kaplan, J. Data and parameter scaling laws for neural machine translation. In Moens, M.-F,, Huang, X., Specia, L.,, and Yih, S. W.-t. (eds.), Pro- ceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 5915-5922, On- line and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 www.-- aclanthology. wrg. wttp://22221.1mmmaa1mm1mmmmmm www.p-main......e..m.....m..mmam 478.
URL https://
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshom, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B. Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C. McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C. Ferrer. C. C., Nikolaidis, C., Allonsius, D., Song, D., Pintz, D., Livshits, D., Wyatt, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia- Olano, D., Perino, D.. Hupkes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F,, Guzmán, F., Zhang, F., Synnaeve, G., Lee, G., Anderson, G. L., Thattai, G., Nail, G. Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Ko- revaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, I., Misra, I., Evtimov, I., Zhang, J.. Copet, J., Lee, J.. Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong, J., Lee, J.,
12

---

How Do Large Language Monkeys Get Their Power (Laws)?
Shojanazeri, H., Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Zhan, H., Damlaj. I., Molybog, I., Tufanov, L., Leontiadis, L,, Veliche, I.-E.. Gat. I., Weissman, J., Geboski, J., Kohli, J. Lam, J., Asher, J., Gaya, J.- Marcus, J., Tang, J., Chan, J., Zhen, J., Reizenstein, J., Teboul, J., Zhong, J., Jin, J., Yang, J., Cummings, J., Carvill, J., Shepard, J., McPhie. J., Torres, J., Ginsburg. J.. Wang, J., Wu, K., U, K. H., Saxena, K., Khandelwal, K., Zand, K., Matosich, K. Veeraraghavan, K., Michelena, K., Li, K., Jagadeesh, Huang, K., Chawla, K., Huang. K., Chen, I.., Garg, L, L., Silva, L., Bell, L., Zhang, L., Guo, L., Yu, L., Moshkovich, L., Wehrstedt, L.., Khabsa, M., Avalani, M., Bhatt, M., Mankus, M., Hasson, M., Lennie, M., Reso, M.. Groshev, M., Naumov, M., Lathi, M., Keneally, M., Liu, M., Seltzer, M. L., Valko, M., Restrepo, M., Patel, M.. Vyatskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Mehta, N., Laptev, N. P. Dong, N.. Cheng, N., Chernoguz. O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, Bontrager, P., Roux, P.,
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14700) 2020.
Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Sealing laws for transfer, 2021. URL https:// arxiv. org/ abs 2202.10220. 01293.
Hernandez, D., Brown, T., Conerly, T., DasSarma, N., Drain, D., El-Showk, S., Elhage, N., Hatfield Hatfiel-Dodds, Z., Henighan, T., Hume, T., al. Scaling laws and inter- pretability of learning from repeated data. arXiv preprint arXiv:2205. 10487, 2022.
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M., Ali, M., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically, arXiv preprint arXiv: /tt.iv::711..0 1712.00499 2017.
Hilton, J., Tang, J., and Schulman, J. Scaling laws for single-agent reinforcement learning, 2023. URL https: //arxiv. org/ abs 10.33111.4433
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A.. Welbl, J., Clark, A.. Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L. Training compute-optimal large language models, 2022. URL https: http///xxivv..o/////. abs / 2203.223..222 0122203..55.... 15556.
Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj. P., Liang, Q. Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Parthasarathy, R., Li, R., Hogan, R., Battey, R., Wang, R., Howes, R., Rinott, R., Mehta, S., Siby, S., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Mahajan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C., Patil, S., Shankar, S., Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajuyigbe, S., Chintala, S., Max. S., Chen. S., Kehoe, S., Satter- field, S., Govindaprasad, S., Gupta, S., Deng, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T. Glaser, T., Best, T., Koehler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T. Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V. Mohan, V., Kumar, V. S., Mangla, V., Ionescu, V., Poenaru, V., Mi- hailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W. Constable, W., Tang, X., Wu, X., Wang X., WU, X., Gao, X., Kleinman, Y., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Zhao, Y., Hao, Y., Qian, Y., Li, Y., He. Y., Rait, Z., DeVito, Z., Rosnbrick, Z., Wen, Z., Yang, Z., Zhao, Z., and Ma, Z. The llama herd of models, 2024. URL https: / aaaaiiiii..///vv o abs 2407 21783.
Hu, S., Liu, X., Han, X., Zhang, X., He, C., Zhao, W., Lin, Y., Ding, N., Ou, Z., Zeng, G., Liu, Z., and Sun, M. Predicting emergent abilities with infinite resolution evaluation, 2024. URL https: http::://axxxv.rrgp/p /bss/ 2310 03262.
Hughes, J., Price, S., Lynch, A., Schaeffer, R., Barez, F.. Koyejo, S., Sleight. H.. Jones, E., Perez, E., and Sharma, M. Best-of-n jailbreaking, 2024. URL https:
/ //arxiv. wwwpxii.orggbb///411110 2335.
Hutter, M. Learning curve theory, 2021. URL https:
//arxiv. org abs////////aa htt..////22...... 04074.
Jiang, M., Liu, K. Z., Zhong, M., Schaeffer, R., Ouyang, S., Han, J., and Koyejo, S. Investigating data contamination for pre-training language models, 2024. URL https: / aaaai.vvvv.... org/ abs 22012 06059.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. In Thirty-fifhh Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 2), 2021.
Jones, A. L. Scaling scaling laws with board games. arXiv preprint arXiv:2104.03: 13, 2021.
Kalajdzievski, D. Scaling laws for forgetting when fine- tuning large language models, 2024. URL https:/ arxiv. www.vv..rgrbb//eeeeee /2401. 05605.
13

---

How Do Large Language Monkeys Get Their Power (Laws)?
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language mod- els, 2020. URL https::///// /ttpe///aaxiviiee org/ /bsp/bbb//22... 08361.
McKenzie, I. R., Lyzhov, A., Pieler, M., Parrish, A., Mueller, A., Prabhu, A., McLean, E., Kirtland, A., Ross, A., Liu, A., Gritsevskiy, A., Wurgaft, D., Kauffman, D., Recchia, G., Liu, J., Cavanagh, J., Weiss, M., Huang, S., Droid, T. F. Tseng, T., Korbak, T., Shen, X., Zhang, Y., Zhou, Z.,, Kim, N., Bowman, S. R., and Perez, E. Inverse scaling: When bigger isn't better, 2024. URL https: /arxiv. org/ abs 22022 09479.
Kazdan, J., Schaeffer, R., Dey, A., Gerstgrasser, M., Rafailov, R., Donoho, D. L., and Koyejo, S. Collapse or thrive? perils and promises of synthetic data a self- generating world, 2024. URL Utthtp::: httptuusttttts///aaaaaa htap:s:s//rxivooror orgg/ abs/2410 ...00.6660 16713.
Mei, K., Tu, Z., Delbracio, M., Talebi, H., Patel, V. M., and Milanfar, P. Bigger is not always better: Scaling properties of latent diffusion models, 2024. https:::: / //aaxii..org abs 2204 01367.
Kudugunta, S., Kusupati, A., Dettmers, T., Chen, K., Dhillon, I., Tsvetkov, Y., Hajishirzi, H., Kakade, S., Farhadi, A., Jain, P., et al. Matformer: Nested transformer for elastic inference. arXiv preprini arXiv:2310.07707 2023.
Mhaskar, H. N. Neural networks for optimal approximation smooth and analytic functions. Neural computation, (1):164-177, 1996.
Michaud, E., Liu, Z., Girit, U., and Tegmark, M. The quantization model of neural scaling. Advances in Neural Information Processing Systems, 36, 2024.
Kulal, S., Pasupat, P., Chandra, K., Lee, M., Padon, O.,
Aiken, A., and Liang, P. S. Spoc: Search-based pseu- docode to code. Advances in Neural Information Process- ing Systems, 32, 2019.
mpmath development team, T. mpmath: Python library arbitrary-precision for arbitrary precision floating- point arithmetic (version
Kumar, T., Ankner, Z., Spector, B. F., Bordelon, B., Muen- nighoff, N., Paul, M., Pehlevan, C., Ré, C., and Raghu- nathan, A. Scaling laws for precision. arXiv preprint arXiv:2411.04:: 2024.
1.3.0), 2023. http:
org/
Muennighoff, N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. A. Scaling conttrconst language models. Advances in Neural Information Processing Systems, 36:50358-50371 2023.
Liang, Z., He, H., Yang, C., and Dai, B. Scaling laws for diffusion transformers, 2024. URL https://arxis Atg/abs/2413 08184.
Neumann, O. and Gros, C. Scaling laws for a multi- agent reinforcement learning model. arXiv preprint arXiv:2210.00844 2022.
Lin, L., Wu, J., Kakade, S. M., Bartlett, P. L., and Lee, D. Scaling laws in linear regression: Compute, parameters, and data, arXiv preprint arXiv:2406.08466, 2024.
Neumann, O. and Gros, C. Alphazero neural scaling and zipf's law: a tale of board games and power laws, 2024. URL https: / arxiv org/ /abs//////// / 11//2211.111911
Liu, J.. Mao, H., Chen, Z., Zhao, T., Shah, N., and Tang, J. Towards neural scaling laws on graphs, 2024. URL https:: //axivv. org/ abs/ 2402 02054.
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Bal- aji. S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bogdonoff, L., Boiko, O., Boyd, M., Brakman, A.-L., Brockman, G., Brooks, T., Brundage, M., Button, K., Cai, T., Campbell, R., Cann, A., Carey, B., Carlson, C., Carmichael, R., Chan, B., Chang, C., Chantzis, F,, Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B., Cho, C., Chu, C., Chung, H. W., Cummings, D., Cur- rier, J., Dai, Y., Decareaux, C., Degry, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P., Forte, J., Fulford, I. Gao, L., Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., GontijoLopps, R., Gordon, J., Grafstein, M., Gray, S., Greene, R., Gross, J., Gu, S. S., Guo, Y., Hallacy, C., Han,
Maloney, A., Roberts, D. A., and Sully. J. A solvable model of neural scaling laws. arXiv preprint 1rXiv:2210.16852 2022.
Maslej, N., Fattorini, L., Perrault, R., Parli, V., Reuel, A., Brynjolfsson, E.,, Etchemendy, J., Ligett, K., Lyons, T., Manyika, J., Niebles, J. C., Shoham, Y., Wald, R., and Clark, J. Artificial intelligence index report 2024, 2024. URL httph::tp///m /aw.vv.vvvvv.////a// org hbtp///..//2.... 19522.
Mazeika, M., Phan, L., Yin, X., Zou, A., Wang. Z., Mu, N., Sakhaee, F., Li, N., Basart, S., Li, B., Forsyth, D., and Hendrycks, D. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal, 2024. URL https: /at.xia...org../...............a abs/2402. 04249.
14

---

How Do Large Language Monkeys Get Their Power (Laws)?
J., Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K., Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun, H., Kaftan, T., tukasz Kaiser, Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L..
Paquette, E., Paquette, C., Xiao, L.,, and Pennington, J. 4+ 3 phases of compute-optimal neural sealing laws. arXiv preprint arXiv:2405.15007 2024.
Pinkus, A. Approximation theory of the mlp model in neural networks. Acta numerica, 8:143-195, 1999.
Kim, J. W. Kim, C., Kim, Y., Kirchner, H., Kiros, J., Knight, M., Kokotajlo, D., Lukasz Kondraciuk, Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, I., Lee, T., Leike, J., Leung, J., Levy, D., Li, C. M., Lim, R., Lin, M., Lin, S., Litwin M., Lopez, T., Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T. Markovski, Y. Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S. M., McLeavey, C.,, McMillan, P., McNeil, J., Medina, D., Mchta, A., Menick, J., Metz, L., Mishchenko, A., Míshkin, P., Monaco, V., Morikawa, E., Mossing, D., Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang, L., O' Keefe, C., Pachocki, J., Paino, A., Palermo, J., Pantuliano, A., Parascandolo,
Polo, F. M., Somerstep, S., Choshen, L., Sun, Y., and Yurochkin, M. Sloth: scaling laws for ilm skills pre- dict multi-benchmark performance across families, 2024. URL https::/ /atpivvaaaiiiiaa/aaa/aae org abs /212 06540.
Porian, T., Wortsman, M., Jitsev, J., Schmidt, I.,, and Car- mon, Y. Resolving discrepancies in compute-optimal scaling of language models, 2024. URL https:// arxiv, /aggabb/////eaao/ 2206 914..
Reuel, A., Buckmall, B., Casper, S., Fist, T., Soder, L., Aarne, O., Hammond, L.., Ibrahim, I.., Chan, A., Wills, P., Anderljung, M. Garfinkel, B., Heim, L., Trask, A., Mukobi, G., Schaeffer, R., Baker, M., Hooker, S., So- laiman, I., Luccioni, A. S., Rajkumar, N. Moës, N. Ladish, J., Guha, N., Newman, J., Bengio, Y., South, T., Pentland, A., Koyejo, S., Kochenderfer, M. J., and Trager, Open problems in technical governance, 2024. URL https: http//axivvv.. org/ abs 220...1491
G., Parish, J., Parparita, E., Passos, A., Pavlov, M., Peng, A., Perelman, A., de Avila Belbute Peres, F., Petrov, M., de Oliveira Pinto, H. P. Michael, Pokorny, Pokrass, M., Pong. V. H., Powell, T., Power, A.. Power, B., Prochl, E., Puri, R., Radford, A., Rae, J., Ramesh A., Raymond, C., Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schnurr, D., Schulman, J., Selsam, D., Sheppard, K., Sherbakov, T., Shieh, J., Shoker, S., Shyam, P. Sidor, S., Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl, Sokolowsky, B., Song, Y., Staudacher, N., Such, F. P., Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M. B., Tillet, P., Tootoonchian, A., Tseng, E., Tuggle, P., Turley, N., Tworek, J., Uribe, J. F. C., Vallone, A. Vijayvergiya, A., Voss, C., Wainwright, C., Wang, J.J., Wang, A., Wang, B., Ward, J., Wei, J., Weinmann, C., Welihinda, A., Welinder, P., Weng, J.. Weng, L.. Wi- ethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H. Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q. Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang, J., Zhuk, W., and Zoph, B. Gpt-4 technical report, 2024. URL htw.h:tts///eete /ttp://aaxvvvvv orgg abs/ 2303 08774.
Roberts, D. A., Yaida, S., and Hanin, B. The principles of deep learning theory, volume 46. Cambridge University Press Cambridge, MA, USA, 2022.
Romani, S., Pinkoviezky, I., Rubin, A., and Tsodyks, M. Scaling laws of associative memory retrieval. Neural computation, 25(10):2523-2544 2013.
Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. A constructive prediction of the generalization error across scales. In International Conference on Learning Representations, 2020.
Rosenfeld, J. S., Frankle, J., Carbin, M., and Shavit, On the predictability of pruning across scales. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference Machine Learn- ing, volume 139 of Proceedings of Machine Learn- ing Research, pp. 9075-9083, PMLR, 18-24 Jul 2021. www.tttss//pprceeeeee www.rrreeeiing.m......... mlr.. ..ress/ v139 htt////oeneeeelaaaaaat html.
Owen, D. How predictable is language model benchmark performance?, 2024.
Ruan, Y., Maddison, C. J., and Hashimoto, T. Observational scaling laws and the predictability of language model per- formance, 2024. URL https: 2405. 10938.
Pachocki, J., Tworek, J., Fedus, L., Kaiser, L., Chen, M., Sidor, S., and Zaremba, W. Learning rea- son with LLMs. Technical report, OpenAI, Septem- ber 2024. URL https://openai wwwccoooioooeeeeeeeee index//d////// woo--ressowwtth---1e
Sardana, N., Portes, J., Doubov, S., and Frankle, J. Beyond chinchilla-optine optimall: Accounting for inference in language model scaling laws. In Forty-first International Confer- ence on Machine Learning, 2023.
Contributors include the ol Contributions team, Core Contributors, and multiple research and safety teams.
15

---

Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agar- wal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., Xiang, A., Parrish, A., Nie, A., Hussain, A., Askell, A., Dsouza, A., Slone, A., Rahane, A., Iyer, A. S., Andreassen, A., Madotto, A., Santilli, A., Stuhlmüller, A., Dai, A., La, A., Lampinen, A.. Zou, A.. Jiang, A., Chen, A., Vuong. A., Gupta, A., Gottardi, A., Norelli, A., Venkatesh, A., Gholamidavoodi, A., Tabas- sum, A., Menezes, A., Kirubarajan, A., Mullokandov, A., Sabharwal, AA., Herrick, A., Efrat, A., Erdem, A., Karakaş, How Do Large Language Monkeys Get Their Power (Laws)?
Schaeffer, R. Pretraining on the test set is all you need, 2023. URL htap:p://aaxiv..o
abs 2.009 08632.
Schaeffer, R., Miranda, B., and Koyejo, S. Are emer- gent abilities of large language models a mirage? In Oh, A., Naumann, T.. Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, vol- ume 36, pp. 55565-55581, Curran Associates, Inc., 2023. URL httpa//eeieeeappe///////////// co cc/paperr_files//eeeeeeeeeereeep adc96226f45005544333111116ee8882 pdf.
A., Roberts, B. R., Loe, B. S., Zoph, B., Bojanowski, B., Özyurt B., Hedayatnia, B., Neyshabur, B., Inden, B., Stein, B., Ekmekci, Lin, B. Y., Howald, B., Orinion, B., Diao, C., Dour, C., Stinson, C., Argueta, C., Ramírez, C. Singh, C., Rathkopf. C.,, Meng, C., Baral, C., Wu, C., Callison-Burch, C.,, Waites, C., Voigt, C., Manning, C. D., Potts, C., Ramirez, C., Rivera, C. E., Siro, C., Raffel, C., Ashhraftt C., Garbacea, C., Sileo, D., Gar- rette, D., Hendrycks, D., Kilman, D., Roth, D., Freeman, D., Khashabi, D., Levy, D., González, D. M., Perszyk, D., Hernandez, D., Chen, D., Ippolito, D., Gilboa, D., Dohan, D., Drakard, D., Jurgens, D., Datta, D., Gan- guli, D., Emelin, D., Kleyko, D., Yuret, D., Chen, D., Tam, D., Hupkes, D., Misra, D., Buzan, D., Mollo, D. C., Yang, D., Lee, D.-H., Schrader, D., Shutova, E., Cubuk, E. D., Segal, E., Hagerman, E., Barnes, E., Donoway, E., Pavlick, E., Rodola, E., Lam, E., Chu, E.,, Tang. E., Erdem, E., Chang. E., Chi, E. A., Dyer, E., Jerzak, E., Kim, E., Manyasi, E. E.,, Zheltonozhskii, E., Xia, F, Siar, F. Martinez-Plumed, F., Happé, F., Chollet, F., Rong, F. Mishra, G., Winata, G. I., de Melo, G., Kruszewski, G., Parascandolo, G., Mariani, G., Wang, G., Jaimovitch- López, G., Betz, G., Gur-Ari, G., Galijasevic, H., Kim, H., Rashkin, H., Hajishirzi. H., Mehta, H., Bogar, H., Shevlin, H., Schütze, H Yakura, H. Zhang, H., Wong, H. M.. Ng, I., Noble, I., Jumelet, J., Geissinger, J., Kernion, J., Hilton, J., Lee, J., Fisac, J. F. Simon, J. B., Koppel, J., Zheng. J., Zou, J., Kocoñ, J., Thompson, J., Wingfield, J., Kaplan, J., Radom, J., Sohl-Dickstein, J., Phang, J., Wei, J., Yosinski, J., Novikova, J., Bosscher, J., Marsh, J., Kim, J., Taal, J., Engel, J., Alabi, J., Xu, J., Song, J., Tang, J., Waweru, J., Burden, J., Miller, J., Balis, J. U., Batchelder, J., Berant, J., Frohberg, J., Rozen, J., Hemandez-Orallo, J., Boudeman, J., Guerr, J., Jones, J., Tenenbaum, J. B., Rule, J. S., Chua, J., Kanclerz, K., Livescu, K., Krauth, K., Gopalakrishnan, K., Ignatyeva, K., Markert, K., Dhole, K. D., Gimpel, K., Omondi, K., Mathewson, K., Chiafullo, K., Shkaruta, K., Shridhar, K., McDonell, K., Richardson, K., Reynolds, L.., Gao, L., Zhang, L., Dugan, L., Qin, L., Contreras- -chhnndo. L.. Morency, L.-P., Moschella, L., Lam, L., Noble, L., Schmidt, L., He, L., Colón, L. O., Metz, L., Şenel, L. K.,
Schaeffer, R., Lecomte, V., Pai, D. B., Carranza, A., Isik, B., Unell, A., Khona, M., Yerxa, T., LeCun, Y., Chung, S., Gromov, A., Shwartz-Ziv, R., and Koyejo, S. Towards an improved understanding and utilization of maximum manifold capacity representations, 2024a. URL https: //arxiv. org/ abs/ / 2406. 09366.
Schaeffer, R., Schoelkopf, H., Miranda, B., Mukobi, G., Madan, V.. Ibrahim, A., Bradley, H., Biderman, S., and Koyejo, S. Why has predicting downstream capabilities of frontier models with scale remained elusive?, 2024b. URL https: / arxiv. org/ abs 6406. 04391.
Schaeffer, R., Zahedi, N., Khona, M., Pai, D., Truong, S., Du, Y., Ostrow, M., Chandra, S., Carranza, A., Fiete, I. R., Gromov, A., and Koyejo, S. Bridging associative memory and probabilistic modeling, 2024c. URL https://
arxiv. org/ abs / 220222022222 22242210202..
Sharma, U. and Kaplan, J. Scaling laws from the data mani- fold dimension. Journal of Machine Learning Research, 13(9):1334, 2022.
Snell, C., Lee, J.. Xu, K., and Kumar, A. Scaling Ilm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint arXiv:2408.03314, 2024a.
Snell, C., Wallace, E., Klein, D., and Levine, S. Predicting emergent capabilitiess by finetuning, 2024b. URL https: // // //axxiivv org/ abs/2411 16035.
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Mor- cos, A. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 11119523-19534 2022.
Spigler, S., Geiger, M., and Wyart, M. Asymptotic learn- ing curves of kernel methods: empirical data versus teacher-student paradigm. Journal of Statistical Mechan- ics: Theory and Experiment, 2020(12):124000 Decem-
ber 2020. ISSN 1742-5468. doi: 10.1088/1742-54668 abc61d. URL http:/ /xpp/xxx///////// doi wtt.oi.org/1..100
1742-546 5468 /bc
16

---

How Do Large Language Monkeys Get Their Power (Laws)?
Bosma, M., Sap, M., ter Hoeve, M., Farooqí, M., Faruqui, M., Mazeika, M., Baturan, M., Marelli, M., Maru, M., Quintana, M. J. R., Tolkiehn, M., Giulianelli, M., Lewis, M.. Potthast, M.. Leavitt, M. L., Hagen, M., Schubert, M., Baitemirova, M. O., Arnaud, M., McElrath, M., Yee, M. A., Cohen, M., Gu, M., Ivanitskiy, M., Starritt, M., Strube, M., Swędrowski, M., Bevilacqua, M., Yasunaga, M.. Kale, M., Cain, M., Xu, M., Suzgun, M.. Walker, M., Tiwari, M., Bansal, M., Aminnaseri, M., Geva, M., Gheini, M., T. M. V. Peng. N., Chi, N. A., Lee, N., Krakover, N. G.-A., Cameron, N., Roberts, N., Doiron, N., Martinez, N., Nangia, N., Deckers. N., Muennighoff, N., Keskar, N. S., Iyer, N. S., Constant, N., Fiedel, N., Wen, N., Zhang, O., Agha, O., Elbaghdadi, O., Levy, O., Evans, O., Casares, P. A. M., Doshi, P., Fung, P., Liang, P. Vicol, P., Alipoormolabashi, P., Liao, P., Liang, P., Chang. P., Eckersley, P., Htut, P. M., Hwang, P., Milkowski, P., Patil, P., Pezeshkpour, P., Oli, P., Mei, Q., Lyu, Q. Chen, Q. Banjade, R., Rudolph, R. E., Gabriel, R., Habacker, R., Risco, R., Millière, R., Garg, R., Barnes, R., Saurous, R. A., Arakawa, R., Raymackers, R., Frank, R., Sikand, R., Novak, R., Sitelew, R., LeBras, R., Liu, R., Jacobs, R., Zhang. R., Salakhutdinov, R., Chi, R., Lee, R., Stovall, R., Techan, R., Yang. R., Singh, S., Mohammad, S. M., Anand, S., Dillavou, S., Shleifer, S., Wiseman, S., Gruetter, S., Bowman, S. R., Schoenholz, S. S., Han, S., Kwatra, S., Rous, S. A., Ghazarian, S., Ghosh, S., Casey, S., Bischoff. S., Gehrmann, S., Schuster, S., Sadeghi. S., Hamdan, S., Zhou, S., Srivastava, S. Shi, S., Singh, S., Asaadi, S., Gu, S., Pachchigar, S., Toshniwal, S., Upad- hyay, S., Shyamolima, Debnath, Shakeri, S., Thormeyer, S., Melzi, S., Reddy, S., Makini, S. P., Lee, S.-H., Torene, S., Hatwar, S., Dehaene, S., Divic, S., Ermon. S., Bider- man, S., Lin, S., Prasad, S., Piantadosi, S. T. Shieber, S. M.. Misherghi, S., Kiritchenko, S., Mishra, S., Linzen, T., Schuster, T., Li, T., Yu, T., Ali, T., Hashimoto, T., Wu, T.-L., Desbordes, T., Rothschild, T., Phan, T., Wang. T., Nkinyili, T., Schick, T., Kornev, T., Tunduny, T., Gersten- berg, T., Chang, T., Neeraj, T., Khot, T., Shultz, T., Sha- ham, U., Misra, V., Demberg, V., Nyamai, V., Raunak, V., Ramasesh, V., Prabhu, V. U., Padmakumar, V., Srikumar, V., Fedus, W., Saunders, W., Zhang, W., Vossen, W., Ren, X., Tong, X., Zhao, X., Wu, X., Shen, X., Yaghoobzadeh, Y., Lakretz, Y., Song, Y., Bahri, Y., Choi, Y., Yang, Y., Hao, Y., Chen, Y., Belinkov, Y., Hou, Y., Hou, Y., Bai, Y., Seid, Z., Zhao, Z., Wang, Z., Wang, Z. J., Wang, Z., and Wu, Z. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023. URL http::/ /awxiviivvvv....//aaaaaaaaavv.ae org/ 2200. 04615.
Tao, C., Liu, Q. Dou, L., Muennighoff, N., Wan, Z., Luo, P., Lin, M., and Wong, N. Scaling laws with vocabulary: Larger models deserve larger vocabularies. arXiv preprint arXiv:2407. 13623, 2024.
Tay, Y. Dehghani, M., Rao. J., Fedus, W., Abnar, S., Chung, H. W., Narang, S., Yogatama, D., Vaswani, A.. and Metzler, D. Scale efficiently: Insights from pre- training and fine-tuning transformers. arXiv preprint arXiv:2109, 10686, 2021.
Tay, Y., Dehghani, M., Abnar, S., Chung, H. W., Fedus, W., Rao, J., Narang, S., Tran, V. Q., Yogatama, D., and Met- zler, D. Scaling laws vs model architectures: How does inductive bias influence scaling? In The 2023 Conference on Empirical Methods in Natural Language Processing, 2022a.
Tay, Y., Wei, J., Chung. H. W., Tran, V. Q., So, D. R., Shak- eri, S., Garcia, X., Zheng. H. S., Rao, J., Chowdhery, A., Zhou, D., Metzler, D., Petrov, S., Houlsby, N., Le, Q V., and Dehghani, M. Transcending scaling laws with 0.1 URL https::///// arwivv wwwpeiii.org/eooe
org/ /abs///b///// /aabb//22210 11399.
Team, G., Anil, R., Borgeaud, S., Alayrac, J.-.., Yu, Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., Silver, D., Johnson, M., Antonoglou, I., Schrittwieser, J., Glaese, A., Chen, J., Pitler, E., Lillicrap. T., Lazari- dou, A., Firat, O., Molloy, J., Isard, M., Barham, P. R., Hennigan, T., Lee. B., Viola, F., Reynolds, M., Xu, Y., Doherty, R., Collins, E., Meyer, C., Rutherford, E., Mor- eira, E., Ayoub, K., Goel, M., Krawezyk, J., Du, C., Chi, E., Cheng, H.-T., Ni, E.,, Shah, P., Kane, P., Chan, B., Faruqui, M. Severyn, A., Lin, H., Li, Y., Cheng, Y., It- tycheriah, A., Mahdieh, M., Chen, M., Sun, P., Tran, D., Bagri, S., Lakshminarayanan, B., Liu, J., Orban, A., Güra, F., Zhou, H., Song, X., Boffy, A., Ganapa- thy, H., Zheng, S., Choe, H., Ágoston Weisz, Zhu, T., Lu, Y., Gopal, S., Kahn, J., Kula, M., Pitman, J., Shah, R., Taropa, E., Merey, M. A., Baeuml, M., Chen, Z., Shafey, L. E., Zhang, Y., Sercinoglu, O., Tucker, G., Pi- queras, E., Krikun, M., Barr, I., Savinov, N., Danihelka, I.. Roclofs, B. White, A.. Andreassen, A., von Glehn, T., Yagati, L., Kazemi, M., Gonzalez, L., Khalman, M., Sygnowski, J., Frechette, A., Smith, C., Culp, L., Proleev, L.. Luan, Y., Chen. X. Lottes, J., Schucher, N.. Lebron, F. Rrustemi, A., Clay, N., Crone, P., Kocisky, T., Zhao, J., Perz, B., Yu, D., Howard, H., Bloniarz, A., Rae, J. W., Lu, H., Sifre, L., Maggioni, M., Alcober, F., Garrette, D.,, Barnes, M., Thakoor, S., Austin, J., Barth- G., Wong, W., Joshi, R., Chaabouni, R., Fatiha, D., Ahuja, A., Tomar, G. S., Senter, E., Chadwick, M., Kornakov, I., Attaluri, N., Iturrate, I., Liu, R., Li, Y., Cogan, S.,
Sun, X., Li, S., Xie, R., Han, W., Wu, K., Yang, Z., Li, Y., Wang, A., Li, S., Xue, J., Cheng, Y., Tao, Y., Kang, Z., Xu, C., Wang, D., and Jiang, J. Scaling laws for floating point quantization training, 2025. URL https:
17

---

How Do Large Language Monkeys Get Their Power (Laws)?
Chen, J., Jia, C., Gu, C., Zhang, Q., Grimstad, J., Hart- man, A. J., Garcia, X., Pillai, T. S., Devlin, J., Laskin, M., de Las Casas, D., Valter, D.. Tao, C., Blanco, L., Badia, A. P., Reitter, D.. Chen, M., Bremnan, J.. Rivera, C., Brin, S., Iqhal, S., Surita, G., Labanowski, J., Rao, A., Winkler, S., Parisotto, E., Gu, Y., Olszewska, K., Addanki, R., Miech, A., Louis, A., Teplyashin, D., Brown, G., Catt, E., Balaguer. J., Xiang. J.. Wang. P., Ashwood, Z., Briukhov, A., Webson, A., Ganapathy, S., Sanghavi, S., Kannan, A., Chang, M.-W., Stjerngen,, A., Djolonga, J.. Sun, Y., Bapna, A., Aitchison, M., Pejman, P., Michalewski, H., Yu, T., Wang, C., Love, J., Ahn, J.. Bloxwich, D., Han, K., Humphreys, P., Sellam, T., Bradbury, J.. Godbole, V., Samangooei, S., Damoc, B., Kaskasoli, A., Arnold, S. M. R., Vasudevan, V., Agrawal, S., Riesa, J.. Lepikhin, D., Tanburn, R., Srinivasan, S., Lim, H., Hodkinson, S., Shyam, P., Ferret, J., Hand, S., Garg, A., Paine, T. L., Li, J., Li, Y., Giang, M., Neitz, A., Abbas, Z., York, S., Reid, M., Cole, E., Chowdhery, A., Das, D., Rogozinska, D., Nikolaev, V., Sprechmann, P., Nado, Z., Zilka, L., Prost, F., Hc, L., Monteiro, M., Mishra, G., Welty, C., Newlan, J., Jia, D., Allamanis, M., Hu, C. H., de Liedek- erke, R., Gilmer, J., Saroufim, C., Rijhwani, S., Hou, S., Shrivastava, D., Baddepudi, A., Goldin, A.. Ozturel, A., Cassirer, A., Xu, Y., Sohn, D., Sachan, D., Amplayo, R. K. Swanson, C., Petrova, D., Narayan, S., Guez, A., Brahma, S., Landon, J., Patel, M., Zhao, R., Villela, K., Wang. L., Jia, W., Rahtz, M., Giménez, M., Yeung, L., Keeling, J., Georgiev, P., Mineu, D., Wu, B., Haykal, S., Saputro, R., Vodrahalli, K. Qin, J.. Cankara, Z., Sharma, A., Fernando, N., Hawkins, W., Neyshabur, B., Kim, S., Hutter, A., Agrawal, P., Castro-Ros, A., van den Driess- che, G., Wang, T., Yang, F., yiin Chang, S., Komarek, P., Mcllroy, R., Lučić M., Zhang, G., Farhan, W., Sharman, M., Natsev, P., Michel, P., Bansal, Y., Qiao, S., Cao, K., Shakeri, S. Butterfield, C., Chung, J., Rubenstein, P. K., Agrawal, S., Mensch, A., Soparkar, K., Lene, K., Chung, T., Pope, A., Maggiore, L., Kay, J., Jhakra, P., Wang. S., Maynez, J., Phuong. M., Tobin, T., Tacchetti, A., Trebacz, M., Robinson, K. Katariya, Y., Riedel, S., Bailey, P., Xiao, K., Ghelani, N., Aroyo, L., Slone, A., Houlsby, N., Xiong, X., Yang, Z., Gribovskaya, E., Adler, J., Wirth, M., Lee, L., Li, M., Kagohara, T., Pavagadhi, J., Bridgers, S., Bortsova, A., Ghemawat, S., Ahmed, Z., Liu, T., Pow- ell, R., Bolina, V., linuma, M., Zablotskaia, P., Besley, J., Chung. D.-W., Dozat, T., Comanescu, R., Si, X., Greer, J., Su, G., Polacek, M., Kaufman, R. I.., Tokumine, S., Hu, H., Buchatskaya, E., Miao, Y., Elhawaty, M., Sid- dhant, A., Tomasev, N., Xing, J., Greer, C., Miller, H., Ashraf, S., Roy, A., Zhang, Z., Ma, A., Filos, A., Besta, M., Blevins, R., Klimenko, T., Yeh, C.-K., Changpinyo, S., Mu, J., Chang, O., Pajarskas, M., Muir, C., Cohen, V., Lan, C. L., Haridasan, K., Marathe, A., Hansen, S., Dou- glas, S., Samuel, R., Wang, M., Austin, S., Lan, C., Jiang,
J., Chiu, J., Lorenzo, J. A., Sjösund, L. L.., Cevey, S., Gle- icher, Z., Avrahami, T.,, Boral, A., Srinivasan, H., Selo, V., May, R., Aisopos, K., Hussenot, L., Soares, L. B., Baumli, K., Chang, M. B., Recasens, A., Caine, B., Pritzel, A., Pavetic, F,, Pardo, F., Gergely, A., Frye, J., Ramasesh, V., Horgan, D., Badola, K., Kassner, N., Roy, S., Dyer, E., Campos, V. C., Tomala, A., Tang, Y., Badawy, D. E., White, E., Mustafa, B., Lang. O., Jindal, A.. Vikram, S., Gong, Z., Caelles, S., Hemsley, R., Thornton, G., Feng, F., Stokowiec, W. Zheng, C., Thacker, P., Çaglar Ünlüi Zhang, 7.,, Saleh, M., Svensson, J., Bileschi, M., Patil, P., Anand, A., Ring. R., Tsihlas, K., Vezer, A., Selvi, M., Shevlane, T., Rodriguez, M., Kwiatkowski, T., Daruki, S., Rong, K., Dafoe, A., FitzGerald, N., Gu-Lemberg, K., Khan, M., Hendricks, L. A., Pellat, M., Feinberg, V., Cobon-Kerr, J., Sainath, T., Rauh, M., Hashemi, S. H., Ives, R., Hasson, Y., Noland, E.,, Cao. Y., Byrd, N.,, Hou, I.., Wang, Q. Sottiaux, T., Paganini, M., Lespiau, J.-.., Moufarek, A., Hassan, S., Shivakumar, K., van Amers- foort, J., Mandhane, A., Joshi, P., Goyal, A., Tung, M., Brock, A., Sheahan, H., Misra, V., Li, C., Rakiéević, N., Dehghani, M., Liu, F., Mittal, S., Oh, J., Noury, S., Sezener, E., Huot, F., Lamm, M., Cao, N. D., Chen, C., Mudgal, S., Stella, R., Brooks, K., Vasudevan, G., Liu. C.,, Chain, M., Melinkeri, N., Cohen, A., Wang. V., Seymore, K., Zubkov, S., Goel, R., Yue, S., Krishnakumaran, S., Albert, B., Hurley, N., Sano, M., Mohananey, A., Joughin, J., Filonov, E., Kepa, T., Eldawy. Y.. Lim, J., Rishi, R., Badiezadegan, S., Bos, T., Chang, J., Jain, S., Padman- abhan, S. G. S., Puttagunta, S., Krishna, K., Baker, L., Kalb, N., Bedapudi, V., Kurzrok, A., Lei, S., Yu, A., Litvin, O., Zhou, X., Wu, Z., Sobell, S., Siciliano, A.. Papir, A., Neale, R., Bragagnolo, J., Toor, T., Chen, T. Anklin, V., Wang, F., Feng, R., Gholami, M., Ling, K., Liu, L.., Walter, J., Moghaddam, H., Kishore, A., Adamek, J., Mercado, T., Mallinson, J., Wandekar, S., Cagle, S., Ofek, E., Garrido, G., Lombriser, C., Mukha, M., Sun, B., Mohammad, H. R.,, Matak, J., Qian, Y., Peswani, V., Janus, P., Yuan, Q. Schelin, L., David, O., Garg, A., He, Y., Duzhyi, O., Älgmyr, A., Lottaz, T., Li, Q., Yadav, V., Xu, L., Chinien, A., Shivanna, R., Chuklin, A., Li, J., Spadine, C., Wolfe, T., Mohamed, K., Das, S., Dai, Z., He, K., von Dincklage, D.,, Upadhyay, S., Maurya, A., Chi, L.. Krause, S., Salama, K., Rabinovitch, P. G., M. P. K. R., Selvan, A., Dektiarev, M., Ghiasi, G., Guven, E., Gupta, H., Liu, B., Sharma, D., Shtacher, I. H., Paul, S., Akerlund, O., Aubet, F.-X., Huang, T., Zhu, C., Zhu, E., Teixeira, E., Fritze, M., Bertolini, F., Marinescu, L.- E., Bölle, M., Paulus, D., Gupta, K., Latkar, T., Chang, M., Sanders, J., Wilson, R., Wu, X., Tan, Y.-X., Thict, L. N., Doshi, T., Lall, S., Mishra, S., Chen, W., Luong, T., Benjamin, S., Lee, J., Andrejezuk, E., Rabiej, D., Ranjan, V. Styre, K., Yin. P., Simon, J., Harriott, M. R., Bansal, M., Robsky, A., Bacon, G., Greene, D.,, Mirylenka, D.,
18

---

How Do Large Language Monkeys Get Their Power (Laws)?
Zhou, C., Sarvana, O., Goyal, A., Andermatt, S., Siegler, P., Horn, B., Israel, A., Pongetti, F., Chen, C.-W. L., Sel- vatici, M., Silva, P., Wang, K., Tolins, J., Guu, K., Yogev, R. Cai, X., Agostini, A., Shah, M., Nguyen, H., Don- naile, N. O., Pereira, S., Friso, L., Stambler, A., Kurzrok, A., Kuang, C., Romanikhin, Y., Geller, M., Yan, Z., Jang, K., Lee, C.-C., Fica, W., Malmi, E., Tan, Q. Banica, D., Balle, D., Pham, R., Huang, Y., Avram, D., Shi, H., Singh, J., Hidey, C., Ahuja, N., Saxena, P., Dooley, D., Potharaju, S. P., O'Neill, E., Gokulchandran, A., Foley, R., Zhao, K., Dusenberry, Liu, Y., Mehta, P., Kotikala- pudi, R., fiiranck-Shrader C., Goodman, A., Kessinger, J., Globen, E., Kolhar, P. Gorgolewski, C., Ibrahim, A., Song, Y., Fichenbaum, A., Brovelli, T., Potturi, S., La- hoti, P., Bactu, C., Ghorbani, A., Chen, C., Crawford, A., Pal, S., Sridhar, M., Gurita, P., Mujika, A., Petro- vski, I. Cedoz, P.-L., Li, C., Chen, S., Santo, N. D., Goyal, S., Punjabi, J., Kappaganthu, K., Kwak, C., L.V, P. Velury, S., Choudhury, H., Hall, J., Shah, P., Figucira, R., Thomas, M., Lu, M., Zhou, T., Kumar, C., Jurdi, T., Chikkerur, S., Ma, Y.. Yu, A., Kwak, S., Ähdel, V., Ra- jayogam, S., Choma, T., Liu, F., Barua, A., Ji, C., Park, J. H., Hellendoorn, V., Bailey, A., Bilal, T., Zhou, H., Khatir, M., Sutton, C., Rzadkowski, W., Macintosh, F., Shagin, K., Medina, P., Liang, C., Zhou, J.. Shah, P., Bi, Y., Dankovics, A., Banga, S., Lehmann, S., Bredesen, M., Lin, Z., Hoffmann, J. E., Lai, J., Chung, R., Yang, K., Balani, N., Bražinskas, A., Sozanschi, A., Hayes, M., Alcalde, H. F., Makarov, P., Chen, W., Stella, A., Snijders, L., Mandl, M., Kärrman, A., Nowak, P.. Wu, X., Dyck, A., Vaidyanathan, K., R, R., Mallet, J., Rudominer, M., John- ston, E., Mittal, S., Udathu, A., Christensen, J., Verma, V., Irving, Z., Santucci, A., Elsayed, G., Davoodi, E., Georgiev, M., Tenney, I., Hua, N., Cideron, G., Leurent, E., Alnahlawi, M., Georgescu, I., Wei, N., Zheng, I., Scandinaro, D., Jiang, H., Snoek, J., Sundararajan, M., Wang, X., Ontiveros, Z., Karo, I., Cole, J.. Rajashekhar, V., Tumeh, L., Ben-David, E., Jain, R., Uesato, J., Datta, R., Bunyann O., Wo, S., Zhang. J.. Stanczyk, P., Zhang, Y., Steiner, D., Naskar, S., Azzam, M., Johnson, M., Paszke, A., Chiu, C.-C., Elias, J. S., Mohiuddin, A., Muhammad, F, Miao, J., Lee, A., Vieillard, N., Park, J., Zhang, J., Stanway, J., Garmon, D., Karmarkar, A., Dong, Z.. Lee, J., Kumar, A., Zhou, L., Evens, J., Isaac, W., Irving, G., Loper, E., Fink, M., Arkatkar, I., Chen, N., Shafran, I., Petrychenko, I., Chen, Z., Jia, J.. Levskaya, A., Zhu, Z., Grabowski, P., Mao, Y., Magni, A., Yao, K., Snaider, J.,
A., Perng, G., Abellan, E. A., Zhang, M., Dasgupta, I., Kushman, N., Penchev, I., Repina, A., Wu, X., van der Weide, T., Ponnapalli, P., Kaplan, C., Simsa, J., Li, S., Dousse, O., Yang, F., Piper, J., Ie, N., Pasumarthi, R., Lintz, N., Vijayakumar, A., Andor, D., Valenzuela, P., Lui, M., Paduraru, C., Peng, D., Lee, K., Zhang, S., Greene, S., Nguyen, D. D., Kurylowicz, P., Hardin, C., Dixon, L., Janzer, L.. Choo. K., Feng, Z., Zhang, B.. Singhal, A., Du, D., McKinnon, D., Antropova, N., Bolukbasi, T., Keller, O., Reid, D., Finchelstein,, D., Raad, M. A., Crocker, R., Hawkins, P., Dadashi, R., Gaffney, C., Franko, K., Bu- lanova, A., Leblond, R., Chung, S., Askham, H., Cobo, L. C., Xu, K., Fischer, F., Xu, J., Sorokin, C., Alberti, C., Lin, C.-C., Evans, C.,, Dimitriev, A., Forbes, H., Ba- narse, D., Tung, Z., Omernick, M.. Bishop. C., Sterneck, R., Jain, R., Xia, J., Amid, E., Piccinno, F., Wang, X., Banzal, P., Mankowitz, D. J., Polozov, A., Krakovna, V., Brown, S., Bateni, M., Duan, D., Firoiu, V., Thotakuri, M., Natan, T., Geist, M., tan Girgin, S., Li, H., Ye, J., Roval, O., Tojo, R., Kwong, M., Therph J., Yew, C., Sinopalnikov, D.. Ramos, S., Mellor, J., Sharma, A., Wu, K., Miller, D., Sonnerat, N., Vnukov, D., Greig. R., Beattie, J., Caveness, E., Bai, L.,, Eisenschlos, J., Ko- rchemniy, A., Tsai, T., Jasarevic, M.,, Kong, W., Dao, P., Zheng, Z., Liu, F., Yang. F., Zhu, R., Teh, T. H., Sanmiya, J., Gladchenko, E., Trdin, N., Toyama, D., Rosen, E., Tavakkol, S. Xue, L.,, Elkind, C., Woodman, O., Carpen- ter, Papamakarios, G., Kemp, R., Kafle, S., Grumina, T., Sinha, R., Talbert, A., Wu, D., Owusu-Afriyie, D., Du, C., Thortoon, C.,
J., Narayana, P., Li, J., Fatchi, S., Wieting, J., Ajmeri, O., Uria, B., Ko, Y., Knight, L., Héliou, A., Niu, N., Gu, S., Pang. C., Li, Y., Levine, N., Stolovich, A., Santamaria- Centamaria-Fermaneez R., Goenka, S., Yustalim, W., Strudel, R., Elqursh, A., Deck, C., Lee, H., Li, Z., Levin, K., Hoffmann, R., Holtmann-Rice, D., Bachem, O., Arora, S., Koh, C., Yeganeh, S. H., Pöder, S., Tariq, M.. Sun, Y., Ionita, L., Seyedhosseini, M., Tafti, P., Liu, Z., Gulati, A., Liu, J., Ye, X., Chrzaszez, B., Wang, L., Sethi, N., Li, T., Brown, B., Singh, S., Fan, W., Parisi, A., Stanton, J., Koverkathu, V., Choquette-Choon C. A., Li, Y., Lu, T., Ittycheriah, A., Shroff, P., Varadarajan, M., Bahargam, S., Willoughby, R., Gaddy, D., Desjardins, G., Cornero, M., Robenek, B., Mittal, B., Albrecht, B., Shenoy, A., Moiseev, F., Jacobsson, H., Ghaffarkhah, A., Rivière, M., Walton A., Crepy, C., Parrish, A., Zhou, Z., Farabet, C., Radebaugh, C., Srinivasan, P., van der Salm, C., Fidjeland, A., Scellato, S., Latorre-Chimoto, E., Klimezak-Pluciifsk, H., Bridson, D., de Cesare, D., Hudson, T., Mendolicchio, P., Walker, L., Morris, A., Mauger, M., Guseynov, A., Reid, A., Odoom, S., Loher, L., Cotruta, V., Yenugula, M., Grewe, D., Petrushkina, A., Duerig, T., Sanchez, A., Yadlowsky, S., Shen, A.. Globerson, A., Webb, L., Dua, S., Li, D.,, Bhupatiraju, S., Hurt, D., Qureshi, H., Agarwal, A., Shani, T., Eyal,
Casagrande, N., Palmer, E., Suganthan, P., Castaño, A., Giannoumis, I., Kim, W., Rybiński, M., Sreevatsa, A., Prendki, J., Soergel, D., Goedeckemeyer, A., Gierke, W., Jafari, M., Gaba, M., Wiesner, J., Wright, D. G., Wei, Y., Vashisht, H., Kulizhskaya, Y., Hoover, J., Le, M., Li, L., Iwuanyanwu, C., Liu, L., Ramirez, K., Khorlin, A., Cui, A. LIN, T., Wu, M., Aguilar, R., Pallo, K., Chakladar,
19

---

How Do Large Language Monkeys Get Their Power (Laws)?
M., Khare, A., Belle, S. R., Wang, L., Tekur, C., Kale, M. S., Wei, J., Sang, R., Saeta, B., Liechty, T., Sun, Y., Zhao, Y., Lee, S., Nayak, P., Fritz, D., Vuyyuru, M. R., Aslanides, J., Vyas, N., Wicke, M., Ma, X., Eltyshev, E., Martin, N., Cate, H., Manyika, J., Amiri, K., Kim, Y.,
intiranon, K., Firat, O., Gu, Y., Zhang, Y., Rahtz, M., Faruqui, M., Clay, N., Gilmer, J., Co-Reyes, J., Penchev, I., Zhu, R., Morioka, N., Hui, K., Haridasan, K., Cam- pos, V., Mahdich, M.. Guo, M., Hassan, S., Kilgour, K., Vezer, A., Cheng, H.-T., de Liedekerke, R., Goyal, S., Barham, P., Strouse, D., Noury, S., Adler, J., Sun- dararajan, M., Vikram, S., Lepikhin, D., Pagannni, M., Garcia, X., Yang. F., Valter, D., Trebacz, M.. Vodrahalli, K., Asswaroengghai, C., Ring. R., Kalb, N., Soares, L.. B., Brahma, S., Steiner, D., Yu, T., Mentzer, F., He, A., Gon- zalez, I.., Xu, B., Kaufman, R. I.., Shafey, I.. E., Oh, J., Hennigan, T., van den Driessche, G., Odoom, S., Lucic, M., Roelofs, B., Lall, S., Marathe, A., Chan, B., Ontanon, S., He, L.. Teplyashin, D., Lai, J., Crone, P., Damoe, B., Ho. L.., Riedel, S., Lene, K., Ych, C.-K., Chowdhery, A.. Xu, Y., Kazemi, M., Amid, E., Petrushkina, A., Swersky, K., Khodaei, A., Chen, G., Larkin, C., Pinto. M., Yan, G., Badia, A. P., Patil, P., Hansen, S., Orr, D., Arnold, S. M. R., Grimstad, J.. Dai, A., Douglas, S., Sinha, R., Yadav, V., Chen, X., Gribovskaya, E., Austin, J., Zhao, J., Patel, K., Komarek, P., Austin, S., Borgeaurddd S., Friso, L., Goyal, A., Caine, B., Cao, K., Chung, D. Lamm, M., Barth-Maron, G., Kagohara, T., Olszewska, K., Chen, M., Shivakumar, K., Agarwal, R., Godhia, H., Rajwar, R., Snaider, J., Dotiwalla, X., Liu, Y., Barua, A., Ungure- anu, V., Zhang, Y., Batsaikhan, B.-O., Wirth, M., Qin, J., Danihelka, I., Doshi, T., Chadwick, M., Chen, J., Jain, S., Le, Q. Kar, A., Gurumurthy, M., Li, C., Sang. R., Liu, F., Lamprou, L.,, Munoz, R., Lintz, N., Mehta, H., Howard, H., Reynolds, M., Aroyo, L., Wang, Q.. Blanco, L., Cassirer, A., Griffith, J.,, Das, D., Lee, S., Sygnowski, J., Fisher, Z., Besley, J., Powell, R., Ahmed, Z., Paulus, D., Reitter, D., Borsos, Z., Joshi, R., Pope, A., Hand, S., Selo, V., Jain, V., Sethi, N., Goel, M., Makino, T., May, R., Yang, Z., Schalkwyk, J., Butterfield, C., Hauth, A., Goldin, A., Hawkins, W., Senter, E., Brin, S., Woodman, O., Ritter, M., Noland, E., Giang, M.. Bolina, V., Lee, L., Blyth, T., Mackinnon, I., Reid, M., Sarvana, O., Silver, D., Chen, A., Wang, L., Maggiore, L., Chang, O., At- taluri, N., Thornton, G., Chiu, C.-C., Bunyan, O., Levine, N., Chung, T., Eltyshev, E., Si, X., Lillicrap, T., Brady, D.,, Aggarwal, V., Wu, B., Xu, Y., M.Mclooy R., Badola, K., Sandhu, P., Moreira, E., Stokowiec, W., Hemsley, R., Li, D., Tudor, A., Shyam, P., Rahimtoroghi, E., Haykal, S.. Sprechmann, P., Zhou, X., Mincu, D., Li, Y., Addanki, R., Krishna, K., Wu, X., Frechette, A., Eyal, M., Dafoe, A., Lacey, D., Whang. J., Avrahami, T., 7hang, Y., Taropa, E., Lin, H., Toyama, D., Rutherford, E., Sano, M., Choe, H., Tomala, A., Sefranek-Shradee C., Kassner, N., Pajarskas, M., Harvey, M., Sechrist, S., Fortunato, M., Lyu, C., El- sayed, G., Kuang, C., Lottes, J., Chu, E., Jia, C., Chen, C.-W., Humphreys, P., Baumli, K., Tao, C., Samuel, R. dos Santos, C. N., Andreassen, A., Rakićević, N., Grewe, D., Kumar, A., Winkler, S., Caton, J., Brock, A., Dalmia,
Xiong, X., Kang, K., Luisier, F., Tripuraneni, N., Madras, D. Guo, M., Waters, A., Wang. O., Ainslie, J., Baldridge, J., Zhang. H., Pruthi, G., Bauer, J., Yang. F., Mansour, R., Gelman, J., Xu, Y., Polovets, G., Liu, J., Cai, H., Chen, W., Sheng, X., Xue, E., Ozair, S., Angermueller, C., Li, X., Sinha, A., Wang, W., Wiesinger, J.. Koukoumidis, E., Tian, Y.. Iyer, A., Gurumurthy, M., Goldenson, M., Shah, P.. Blake, M., Yu, H., Urbanowicz, A., Palomaki, J., Fernando, C., Durden, K., Mehta, H., Momchev, N., Rahimtoroghi, E., Georgaki, M., Raul, A., Ruder, S., Red- shaw, M., Lee, J., Zhou, D., Jalan, K., Li, D., Hechtman, B., Schuh, P., Nasr, M., Milan, K., Mikulik, V., Franco, J., Green, T., Nguyen, N., Kelley, J., Mahendru, A., Hu, A., Howland, J., Vargas, B., Hui, J., Bansal, K., Rao, V., Ghiya, R. Wang. E.,, Ye, K., Sarr, J. M., Preston, M. M., Elish, M., Li, S., Kaku, A., Gupta, J., Pasupat, I., Juan, D.-C., Someswar, M., M., T., Chen, X., Amini, A., Fab- rikant, A., Chu, E.,, Dong, X., Muthal, A., Buthpitiya, S., Jauhari, S., Hua, N., Khandelwal, U., Hitron, A., Ren, J., Rinaldi, L., Drath, S., Dabush, A., Jiang, N.-J., God- hia, H., Sachs, U., Chen, A., Fan, Y., Taitelbaum, H., Noga, H., Dai, Z., Wang, J., Liang, C., Hamer, J., Ferng, C.-S., Elkind, C., Atias, A., Lee, P., Listik, V., Carlen, M., van de Kerkhof, J., Pikus, M., Zaher, K., Müller, P., Zykova, S., Stefanec, R., Gatsko, V., Hirnschall, C., Sethi, A., Xu, X. F., Ahuja, C.,, Tsai, B., Stefanoiu, A., Feng, B., Dhandhania, K., Katyal, M., Gupta, A., Parulekar, A., Pitta, D., Zhao J.. Bhatia, V., Bhavnani, Y., Alhadlaq, O., Li, X., Danenberg, P., Tu, D., Pine, A., Filippova, V. Ghosh, A., Limonchik, B., Urala, B., Lanka, C. K., Clive, D., Sun, Y., Li, E., Wu, H., Hongtongsak, K., Li, I. Thakkar, K., Omarov, K., Majmundar, K., Alver- son, M., Kucharski, M., Patel, M., Jain, M., Zabelin, M., Pelagatti, P., Kohli, R., Kumar, S., Kim, J., Sankar, S., Shah, V., Ramachandruni, I.., Zeng, X., Bariach, B., Weidinger, L., Vu, T., Andreev, A., He, A., Hui, K., Kashem, S., Subramanya, A., Hsiao, S., Hassabis, D., Kavukcuoglu, K., Sadovsky, A., Le, Q., Strohman, T., Wu, Y., Petrov, S., Dean, J., and Vinyals, O. Gemini: A family of highly capable multimodal models, 2024a. URL https: / arxivv.vii.././ii org/ abs/ /et///31222 11805.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., Mariooryad, S., Ding, Y., Geng, X., Alcober, F., Frostig, R., Omernick, M., Walker, L., Paduraru, C., Sorokin, C., Tacchetti, A., Gaffney, C., Daruki, S., Sercinoglu, O., Gleicher, Z., Love, J., Voigtlaender, P.. Jain, R., Surita,
G., Mohamed, K., Blevins, R., Ahn, J., Zhu, T., Kaw-
20

---

How Do Large Language Monkeys Get Their Power (Laws)?
S., Sheahan, H., Barr, I., Miao, Y., Natsev, P., Devlin, J. Behbahani, F., Prost, F., Sun, Y., Myaskovsky, A., Pil- lai, T. S., Hurt, D., Lazaridou, A., Xiong, X., Zheng, C.,
N., Glaese, M., Lakshninarayanan, B., Schucher, N., Tang, Y., Collins, E.,, Lim, H., Feng, E,, Recasens, A., Pardo, F., Li. X., Horgan, D., Stanton, J., Ambar, M., Xia, F,, Lince, A., Wang, M., Mustafa, B., Webson, A., Lee, H., Anil, R., Wicke, M., Dozat, T., Sinha, A., Piqueras, E., Dabir, E., Upadhyay, S., Boral, A., Hendricks, L. A., Fry, C., Djoloogaa, J., Su, Y., Walker, J., Labanowski. J., Huang. R., Misra, V. Chen, J., Skerry- Skerry-Ryan, R., Singh, A. Rijhwani, S., Yu, D., Castro-Ros, A., Changpinyo,
Lai, G., Magni, A., Cao, N. D., Siddhant, A., Ashwood, Z. Orbay. J., Dehghani, M., Brennan, J., He, Y.. Xu, K., Gao, Y. Saroufim, C.,, Molloy, J., Wu, X., Arnold, S., Chang, S., Schrittwieser, J., Buchatskaya, E., Radpour, S., Polacek, M., Giordano, S., Bapna, A., Tokumine, S., Hel- lendoom, V. Sottiaux, T., Cogan, S., Severyn, A., Salch, M. Thakoor, S., Shefey, L.., Qiao, S., Gaba, M., yinn Chang. S., Swanson, C., Zhang, B., Lee, B., Rubenstein, P. K., Song, G., Kwiatkowski, T., Koop, A., Kannan, A., Kao, D., Schuh, P., Stjerngren, A., Ghiasi, G. Gibson, G., Vilnis, L., Yuan, Y., Ferreira, F. T.,, Kamath, A., Kli- menko, T., Franko, K. Xiao, K., Bhattacharya, I., Patel, M.. Wang, R., Morris, A., Strudel, R., Sharma, V., Choy, P., Hashemi, S. H., Landon, J., Finkelstein, M., Jhakra, P., Frye, J., Barnes, M., Mauger, M.. Daun, D., Baatarsukh, K., Tung, M., Farhan, W., Michalewski, H., Viola, F., de Chaumont Quitry, F., Lan, C. L., Hudson, T., Wang, Q.. Fischer, F., Zheng, I., White, E.,, Dragan, A., baptiste Alayrac, J., Ni. E., Pritzel, A., Iwanicki, A., Isard, M., Bulanova, A., Zilka, L., Dyer, E., Sachan, D., Srinivasan, S., Muckenhirn, H., Cai, H. Mandhane, A., Tariq, M.. Rae, J. W., Wang, G., Ayoub, K., FitzGerald, N., Zhao, Y., Han, W., Alberti, C., Garrette, D., Krishnakumar, K., Gimenez, M., Levskaya, A., Sohn, D., Matak, J., Itur- rate, I., Chang, M. B., Xiang, J., Cao, Y., Ranka, N., Brown, G., Hutter, A., Mirrokni, V., Chen, N., Yao, K., Egyed, Z., Galilee, F., Liechty, T., Kallakuri, P., Palmer, E.,, Ghemawat, S., Liu, J., Tao, D., Thomtoo C.,, Green, T., Jasarevic, M., Lin, S., Cotruta, V., Tan, Y.-X., Fiedel, N., Yu, H., Chi, E., Neitz, A., Heitkaemper, J., Sinha, A., Zhou, D., Sun, Y., Kaed, C., Hulse, B., Mishra, S., Georgaki, M., Kudugunta, S., Farabet, C., Shafran, I., Vlasic, D., Tsitsulin, A., Ananthanarayanan, R., Carin, A., Su, G., Sun, P., V. S., Carvajal, G., Broder, J., Comsa, I.. Repina, A., Wong, W., Chen, W. W., Hawkins, P., Filonov, E., Loher, L.,, Himsshall, C., Wang, W., Ye, J., Burns, A., Cate, H., Wright, D. G., Piccinini, F., Zhang, I.., Lin, C.-C., Gog. I., Kulizhskaya, Y., Sreevatsa, A., Song, S., Cobo, L. C., Iyer, A., Tekur, C., Garrido, G., Xiao, Z., Kemp, R., Zheng, H. S., Li, H., Agarwal, A., Ngani, C., Goshvadi, K., Santamaria- Fernandez, R., Fica, W., Chen, X., Gorgolewski, C., Sun, S., Garg, R., Ye, X., Eslami, S. M. A., Hua, N., Simon, J., Joshi, P., Kim, Y., Temney, I., Potluri, S., Thiet, L. N., Yuan, Q., Luisier, F. Chronopoulou, A., Scellato, S., Srinivasan, P., Chen, M., Koverkathu, V., Dalibard, V., Xu, Y., Saeta, B., An- derson, K. Sellam, T., Fernando, N., Huot, F., Jung. J., Varadarajan, M., Quinn, M., Raul, A., Le, M., Habalov, R., Clark, J., Jalan, K., Bullard, K., Singhal, A., Lu- ong, T., Wang, B., Rajayogam, S., Eisenschlos, J., Jia, J., Finchelstein, D.,, Yakubovich, A., Balle, D.,, Fink, M., Agarwal, S., Li, J., Dvijotham, D., Pal, S., Kang, K.,
B., Datta, R., Bagri, S., Hrafnkelsson, A. M., Maggioni, M., Zheng, D., Sulsky, Y., Hou, S., Paine, T. L.. Yang, A., Riesa, J., Rogozinska, D., Marcus, D., Badawy, D. E.,, Zhang, Q., Wang, L.,, Miller, H., Greer, J., Sjos, L. L.., Nova, A., Zen, H., Chaabouni, R., Rosca, M., Jiang, J., Chen, C., Liu, R., Sainath, T., Krikun, M., Polozov, A., Lespiau, J.-B., Newlan, J., Cankara, Z., Kwak, S., Xu, Y., Chen, P., Coenen, A., Meyer, C., Tsihlas, K., Ma, A., Gottweis, J., Xing, J., Gu, C., Miao, J., Frank, C., Cankara, Z., Ganapathy, S., Dasgupta, I., Hughes- -iith, S., Chen, H., Reid, D., Rong. K., Fan, H., van Amersfoort, J. Zhuang, V., Cohen, A., Gu, S. S., Mohananey, A., Ilie, A., Tobin, T., Wieting, J., Bortsova, A., Thacker, P., Wang, E., Caveness, E., Chiu, J., Sezener, E.,, Kaskasoli, A., Baker, S., Millican, K., Elhawaty, M., Aisopos, K., Lebsack, C., Byrd, N., Dai, H., Jia, W., Wiethoff, M., Davoodi, E., Weston, A., Yagati, L., Ahuja, A., Gao, I., Pundak, G., Zhang. S., Azzam, M., Sim, K. C., Caelles, S., Keeling, J., Sharma, A., Swing, A., Li, Y., Liu, C., Bostock, C. G., Bansal, Y., Nado, Z., Anand, A.. Lip- schultz, J., Karmarkar, A., Proleev, L.., Ittycheriah, A., Yeganeh, S. H., Polovets, G., Faust, A., Sun, J.. Rrustemi, A,, Li, P., Shivanna, R., Liu, J., Welty, C., Lebron, F., Bad- depudi, A., Krause, S., Parisotto, E., Soricut, R., Xu, 7 Bloxwich, D., Johnson, M., Neyshabur, B., Mao-Jones, J., Wang, R., Ramasesh, V., Abbas, Z., Guez, A., Segal, C., Nguyen, D. D., Svensson, J.. Hou, L., York, S.. Milan, K. Bridgers, S., Gworek, W., Tagliasacchi, M., Lee-Thorp J. Chang, M., Guseynov, A., Hartman, A. J., Kwong, M., Zhao, R., Kashem, S., Cole, F., Miech, A., Tanbur,, R., Phuong, M.. Pavetic, F., Cevey, S., Comanescu, R., Ives, R., Yang, S., Du, C., Li, B., Zhang, Z., linuma, M., Hu, C. H., Roy, A., Bijwadia, S., Zhu, Z., Martins, D., Saputro, R., Gergely, A., Zheng, S., Jia, D., Antonoglou, I. Sadovsky, A., Gu, S., Bi, Y., Andreev, A., Samangooei, S., Khan, M., Kocisky, T., Filos, A., Kumar, C., Bishop, C., Yu, A., Hodkinson, S., Mittal, S., Shah, P., Moufarek, A., Cheng, Y. Bloniarz, A., Lee, J., Pejman, P., Michel, P., Spencer, S., Feinberg, V., Xiong, X., Savinov, N., Smith, C. Shakeri, S., Tran, D., Chesus, M., Bohnet, B., Tucker, G. von Glehn, T., Muir, C., Mao, Y., Kazawa, H., Slone, A. Soparkar, K., Shrivastava, D., Cobon-Kerr, J., Shar- man, M., Pavagadhi, J.. Araya, C., Misiunas, K., Ghelani, N. Laskin, M., Barker, D., Li, Q., Briukhov, A., Houlsby,
21

---

How Do Large Language Monkeys Get Their Power (Laws)?
Konzelmann, J., Beattie, J., Dousse, O., Wu, D., Crocker, R., Elkind, C., Jonnalagadda, S. R., Lee, J., Holtmann- Rice, D., Kallarackal, K., Liu, R., Vnukov, D., Vats, N., Invernizzi, L., Jafari, M., Zhou, H., Taylor, L., Prendki, J., Wu, M., Eccles, T., Liu, T., Kopparapu, K., Beaufays, F., Angermueller, C., Marzoca, A., Sarcar, S., Dib, H., Stan- way, J., Perbet, F., Trdin, N., Sterneck, R., Khorlin, A., Li, D., Wu, X., Goenka, S., Madras, D., Goldshtein, S., Gierke, W., Zhou, T., Liu, Y., Liang, Y., White, A., Li, Y., Singh, S., Bahargam, S., Epstein, M., Basu, S., Lao, L.. Ozturel, A., Crous, C.,, Zhai, A., Lu, H., Tung, Z., Gaur, N., Walton, A., Dixon, L.. Zhang, M., Globerson, A., Uy G., Bolt, A., Wiles, O., Nasr, M., Shumailov, I., Selvi, M., Piccinno, F., Aguilar, R., McCarthy, S., Khalman, M., Shukla, M., Galie, V., Carpenter, J., Villela, K. Zhang, H., Richardson, H., Martens, J., Bosnjak, M., Belle, S. R., Seibert, J., Alnahlawi, M., McWilliams, B., Singh, S., Louis, A., Ding, W., Popovici, D., Simicich, I.., Knight, L., Mehta, P., Gupta, N., Shi, C., Fatehi, S., Mitrovic, J., Grills, A., Pagadora, J., Munkhdalai, T., Petrova, D., Eisenbud, D., Zhang. Z., Yates, D., Mittal, B.. Tripura- neni, N., Assael, Y., Brovelli, T., Jain, P., Velimirovic, M., Akbulut, C., Mu, J., Macherey, W., Kumar, R., Xu, J., Qureshi, H., Comanici, G., Wiesner, J., Gong, Z., Rud- dock, A., Bauer, M., Felt, N., GP, A., Arnab, A., Zelle, D., Rothfuss, J., Rosgen, B., Shenoy, A., Seybold, B., Li, X., Mudigonda, J., Erdogan, G., Xia, J., Simsa, J., Michi, A., Yao, Y. Yew, C., Kan, S., Caswell, I., Radebaugh, C., Elisseeff, A., Valenzuela, P., McKinney, K., Paterson, K., Cui, A., Chiimoo,,,oooo, E., Kim, S., Zeng, W., Dur- den, K., Ponnapalli, P., Sosea, T., Pntoquette-Choo, C. A., Manyika, J., Robenek, B., Vashisht, H., Pereira, S., Lam, H., Velic, M., Owusu-Afriyie, D., Lee, K., Bolukbasi, T., Parrish, A., Lu, S., Park, J., Venkatraman, B., Tal- bert, A., Rosique, L., Cheng. Y., Sozanschi, A., Paszke, A., Kumar, P., Austin, J., Li, L., Salama, K., Perz, B., Kim, W. Dukkipati, N., Baryshnikov, A., Kaplanis, C., Sheng, X., Chervonyi, Y., Unlu, C., de Las Casas, D., Askham, H., Tunyasuvunakool, K., Gimeno, F., Poder, S., Kwak, C., Miecnikowski, M., Mirrokni, V., Dimitriev, A., Parisi, A., Liu, D., Tsai, T., Shevlane, T. Kouridi, C., Gar- mon, D., Goedeckemeyer, A., Brown, A. R., Vijayakumar, A., Elqursh, A., Jazayeri, S., Huang, J., Carthy, M., Hoover, J., Kim, L., Kumar, S., Chen, W., Biles, C., Bingham, G. Rosen, E., Wang, L., Tan, Q., Engel, D., Pongetti, F., de Cesare, D., Hwang, D.,, Yu, L., Pullman, J.,
D., Hsieh, C.-J,, Cai, I., Iqbal, S., Sundermeyer, M., Chen,
Z., Bursztein, E.,, Malaviya, C., Biadsy, F., Shroff, P., Dhillon, I., Latkar, T., Dyer, C., Forbes, H., Nicosia, M.. Nikolaev, V. Greene, S., Georgiev, M., Wang. P., Martin, N., Sedghi, H., Zhang, J., Banzal, P., Fritz, D., Rao, V., Wang. X., Zhang, J., Patraucean, V., Du, D., Mordatch, I., Jurin, I., Liu, L.,. Dubey, A., Mohan, A., Nowakowski, J., Ion, V.-D., Wei, N. Tojo, R., Raad, M. A.. Hudson, D. A., Keshava, V., Agrawal, S., Ramirez, K., Wu, Z., Nguyen, H., Liu, J., Sewak, M., Petrini, B., Choi, D., Philips, I., Wang. 7., Bíca, I., Garg, A., Wilkiewiez, J., Agrawal, P., Li, X., Guo, D., Xue, E., Shaik, N., Leach, A., Khan, S. M., Wiesinger, J., Jerome, S., Chakladar, A., Wang. A. W., Ornduff, T., Abu, F., Ghaffarkhah, A., Wainwright, M., Cortes, M., Liu, F., Maynez, J., Terzis, A., Samangouei, P., Mansour, R., Kepa, T., Aubet, F.-X., Algymr, A., Banica, D., Weisz, A., Orban, A., Senges, A., Andrejezuk, E., Geller, M., Santo, N. D., Anklin, V., Merey, M. A., Bacuml, M., Strohman, T., Bai, J.. Petrov, S., Wu, Y., Hassabis, D., Kavukcuoglu, K., Dean, J., and Vinyals, O. Gemini 1.5: Unlocking multimodal un- derstanding across millions of tokens of context, 2024b. URL https: / arxiv. orgg abs///////// /20.//433022 05530.
Wang. H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P., Deac, A., et al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47-60, 2023.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abili- ties of large language models, 2022a. URL https: / //arxiv. org/ argg/bbs/2000 /2066. 07682.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Met- zler, D., et al. Emergent abilities of large language models. 2022b.
arXiv preprint arXiv:2206.0760
Wu, T.-Y. and Lo, P.-Y. U-shaped and inverted-u scaling behind emergent abilities of large language models, 2024. URL https: arxiv. org/ aba/ /2wl0 01692.
Wu, Y., Sun, Z., Li, S., Welleck, S., and Yang. Y. Infer- ence scaling laws: An empirical analysis of compute- optimal inference for problem-solving with language models, 2024. URL 2408. 00724.
Narayanan, S., Levin, K., Gopal, S., Li, M., Aharoni, A., Trinh, T., Lo, J., Casagrande, N., Vij. R., Matthey, L., Ra- madhana, B., Matthews, A., Carey, C., Johnson, M., Gora- nova, K., Shah, R., Ashraf, S., Dasgupta, K., Larsen, R., Wang, Y., Vuyyuru, M. R., Jiang. C., Ijazi, J., Osawa, K., Smith, C., Boppana, R. S., Bilal, T., Koizumi, Y. Xu, Y., Altun, Y., Shabat, N., Bariach, B., Korchemniy, A., Choo, K., Ronneberger, O., Iwuanyanwu, C., Zhao, S., Soergel,
orb//////
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., Khabsa, M., Fang, H., Mehdad, Y., Narang, S., Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M., Wang, S., and Ma, H. Effective long-context scaling
22

---

How Do Large Language Monkeys Get Their Power (Laws)?
of foundation models, 2023. URL https://arxii.. 6039.
Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12104-12113 13, 2022.
Zhang, B., Liu, Z., Cherry, C., and Firat, O. When scaling meets IIm finetuning: The effect of data, model and fine- tuning method, 2024. URL https:/ / //aaxixxvvoor.//... org/
23

---

How Do Large Language Monkeys Get Their Power (Laws)?
A. Clarification of How Large Language Monkeys and Best-of-N Jailbreaking Sampled Data
In this manuscript, we used the phrasing of "independent attempts," which not fully correct. In this appendix section, we clarify why chose this terminology. what likely impacts we believe this inaccuracy may have had on our results, and how correct the paper accordingly.
Large Language Monkeys (Brown et al., 2024) indeed drew 10, 000 independent attempts per problem, but Best-of-N Jailbreaking (Hughes al., 2024) sampled data slightly different: for each problem, jailbreaking attempts were drawn until either a successful jailbreak was obtained or until a maximum limit of 10, 000 attempts was hit. Samples were also drawn in minibatches of size 60, making the (in)dependence of samples a bit tricky.
We omitted this nuance because it offers second-order correction our paper's main story while offering little additional insight. Neither of our theorems and none of our main figures change. We suspect that this slightly different sampling procedure explains why, in Fig. 6, the estimated power law exponents between the least squares power law estimator and the distributional power law estimator deviate more significantly from identity for Best-of-N Jailbreaking than for Large Language Monkeys. A natural way to correct for this is to use beta-negative binomial distribution rather than beta-binomial distribution, with an additional correction for the maximum number of attempts. For more information, please see Appendix H.
24

---

How Do Large Language Monkeys Get Their Power (Laws)?
B. Estimating Success Rates Using Chen et al. (2021)'s Estimator
In this manuscript, we defined pass; Ok and ASR; @k as:
pass; @k
def
E A Allempts
[[At least 1 attempt by the model solves the i-th problem]]
ASR,OL @k
def
E A Attempes
[[At least 1 attempt jailbreaks the model on the i-th prompt]
Throughout this manuscript, to estimate pass; Ok and ASR@k, we used the unbiased and lower variance estimator introduced by Chen al. (2021): for the i-th problem, we sampled n > attempts per problem, counted the number of successful attempts e, and then swept & to compute estimate of pass; @k for different & values:
pass Ok
(12)
Two comments: Firstly, n as used here no relationship with the number of problems in the benchmark (Sec. 1), and secondly, our notation differs slightly from that of Chen et al. (2021), but the ideas are consistent. A numerically stable Python implementation of the estimator provided in Fig. 8:
def #*"
perrt____err___ee per_ (n: int, e: int, k: int) -> float:
:param number of total attempts this problem.
:param C: number of correct attempts on this problem.
:param k: k in pass_i0$k$.
if n c k: return 1.0
return 1.0 0 prod (1.0 k / ap.arange (n c 1, + 1))
Figure A numerically stable unbiased estimator of pass; @k, introduced by Chen et al. (2021).
To reiterate point made by Chen et al. (2021), estimating pass; @k (1 passss @1) y is biased (Fig. 9).
True pase 0.001
True pess 91: 0.01
True pass 0.1
10P
10a
3.0
10.0
32.0
paseziic
10-1
6 6x 10-1
100.0
316.0
10
10-1
10-2
10000
3 10-1
Estimator
1 (1 - 2)
10'
10a
10P
10a
10a
10a
/'/9/()
True
Figure 9: Bias of Estimators of pass; @k. Numerical simulations show that estimating pass; Ok as 1 - (1 - pass; @1) is biased whereas the estimator Chen et al. (2021) not. For a mathematical proof of unbiasedness, see the original paper.
25

---

How Do Large Language Monkeys Get Their Power (Laws)?
C. Fitting Power Laws to Large Language Monkeys and Best-of-N Jailbreaking
fit power laws to a subset of data from Large Language Monkeys (Brown et al., 2024) and from Best-of-N Jailbreaking (Hughes al., 2024), specifically Pythia language models (Biderman et al., 2023) on the MATH benchmark (Hendrycks al., 2021) and frontier AI models Claude, GPT4 (OpenAI et al., 2024), Gemini (Team al., 2024a;b) and Llama the HarmBench jailbreaking benchmark (Mazeika et al., 2024). We show the functional
(Grattafiori al., 2024) forms and the fit parameters in Table I and Table 2 respectively. To fit the parameters, for Large Language Monkeys, we simply minimized the squared enror between the actual and predicted - log (og(paspp@uu @k), and for Best-of-N Jailbreaking, we similarly minimized the squared error between the actual and predicted - log(ASRp @k)).
Note: Llama 8B IT does not exhibit power law scaling under Best-of-N Jailbreaking (shown in Fig. 1, bottom).
Model
Benchmark
b
Model
Modality
b
Pythia 70M
MATH
8.026
0.194
Claude 3.5 Opus
Text
2.630
0.448
Pythia 160M
MATH
6.591
0.280
Claude 3.5 Sonnet
Text
3.436
0.312
Pythia 410M
MATH
5.524
0.286
GPT4o
Text
3.639
0.395
Pythia 1B
MATH
5,452
0.315
GPT4o Mini
Text
3.637
0.492
Pythia 2.8B
MATH
4.104
0.336
Gemini 1.5 Flash
Text
6.158
0.303
Pythia 6.9B
MATH
4.255
0.348
Gemini 1.5 Pro
Text
6.296
0.256
Pythia 12B
MATH
4.113
0.370
Llama 3 8B IT
Text
Table I: Large Language Monkeys (Brown et al., 2024)
Table 2: Best-of-N Jailbreaking (Hughes et al., 2024) fitted power law parameters on text jailbreak prompts from Harm- Bench (Mazeika al., 2024).
fitted power law parameters on 128 mathematical problems from MATH (Hendrycks al., 2021).
Functional Form: (pg(passp @k) = a -
Functional Form: log(ASB, a- Note: Llama 3 8B Instruction Tuned (IT) does not exhibit power law scaling.
26

---

How Do Large Language Monkeys Get Their Power (Laws)?
D. Mathematical Equivalence Between Coverage and Average Success Rate
Brown et al. (2024) and Hughes al. (2024) phrase their research in terms of "coverage", defined as the fraction of problems that can be solved or the fraction of prompts that can jailbreak model, but as Brown al. (2024) comment and we here derive, the coverage is mathematically equivalent the average pass; @k (equivalently, ASR@k. due to two simple probabilistic primitives: (1) linearity of expectation, (2) the expectation of indictor random variable of some event is the probability of said event and (3) the definition of pass; Øk:
Coverage
def
E
Fraction of Problems Solved After k Attempts
Prompts
Problems Allemps
Attempts
E
E Attempes|Problee Probblem
[[Pobblem Solved After k Attempts]]
Problems
E
DaSSSroblem @k
Problems
passp Ok
In our work, we prefer phrasing along the lines of "success rate" over "coverage" because success rate avoids coverage's binary implication that each problem/prompt is either "solved" or "not solved".
27

---

How Do Large Language Monkeys Get Their Power (Laws)?
E. A ggreegate Power Laws from a Probability Distribution over Exponential Functions
E.1. Preliminaries: Power Laws from Weighted Exponential Functions
known result is that power laws can emerge from appropriately weighted sums of exponential functions, e.g., (Bochud Challet, 2006; Elkies, 2016; Bousquet al., 2020). For concrete example with short proof:
r(r) Jo p e pe dp,
(13)
where ((r) def Jo s--1e- ds is the Gamma function. The proof is via u-substitution def
p
e du p-l e p dp = T(r) (u/z)--le
(14)
[(r) 0
100
((r) x
w=1 E du
(15)
(T(r)))
(16)
[(r)
(17)
In our particular context, we are interested in the scaling with it of the expected success rate over problems sampled from the benchmark's 's data distribution:
passp Ok def
E
pass; @k
(18)
pass & I-D
distribution (over problems in a benchmark) of pass; @k scores that yields power law scaling with respect the number of attempts l::
log
pass Ük
~ ak-b
(19)
1=1
for constants a, > 0.
E.2. Delta Distribution: pass; @1 ~ d(p),p (0,1)
To start with negative result, will show that not all distributions of the per-problem success probabilities pass; @1 yield aggregate power law scaling. Suppose that the model's pass, @1 probabilities across the benchmarks' problems are all exactly E (0, 1). For brevity, let p; pass;@1. Then the aggregate success rate is:
del
E B (pass; Kk)
Ep, [1 =1
(20)
/" 6(p) (1 - pa)* dps
(21)
(1 -p)*
(22)
Recalling that the expansion of log(-) for small x log(1 x) = x + O(x2). ) in our case, we obtain:
- log (1 - Ep, [pass@k]) = -1 p)* + 0 - p)) = (1 - p)* + (1
(23)
Thus, in the large &o regime, we find the negative log aggregate success rate exhibits exponential scaling with k as we intuitively expect.
E.3. Uniform Distribution: pass; @1 Uniform(a,, (a, 8)
Suppose pass:() @1 probabilities follow a uniform distribution Uniform (a, B) where 0 S a < 8 < 1. The aggregate success rate after l attempts is defined as:
ak def TE (1 p)*1.
pas$Uniform(a..3
28

---

How Do Large Language Monkeys Get Their Power (Laws)?
If p Unilorm(a,,3) the expectation of (1 p)* is:
1
1- (1 (1-p)* dp.
3
E (1 - p)
B 3-cd
Evaluating the integral gives:
a) )kt - (1 33
[[(1-)) - }]
(3 a) : (k: + 1)
Thus, the aggregate success rate becomes:
(1 a))tt (1 - 3)m-1
pasSUniform(a..)
@k
(B a) (k: + 1)
Case A: a > a > O, then both (1 - a) and (1 - (3) are strictly less than As k - oc, (1 a)ht1 and (1 B)h+1
decay exponentially. Hence:
(1 - a:)*t1
E[(1 p)*]
(B a)) (k 1)
and )) @k approaches 1 exponentially fast:
pasSCnlform(a.B) @k (8-a) (k: + 1)
Thus, the negative log of the aggregate success rate decays exponentially:
logg(oossssso pasStnnioomm(...) @k) (R))
Case a 0
When O, the uniform distribution is over [0, 3]. In this case:
1 (1 3)++1
[11 p)
B k 1
For large k; (1 (1-B)) 3)++1 becomes exponentially small, and:
1
E (11 p)
+1
The aggregate success rate is then:
1
pasStniform(0..) Ck 1
B.
The negative log exhibits power-law scaling:
1 1
(og(pasSUniform(..) @k)
3 Te
Special Case: Uniform 1) If B = 1, the distribution is uniform on [o. 1]. In this case:
E - -p) ke 6+1"
and the success rate becomes:
1
pasSUniiomm00.1) @k
+
For large k:
log (passsttniioom
@k) k
29

---

How Do Large Language Monkeys Get Their Power (Laws)?
E.4. 2-Parameter Beta Distribution: pass; @1 Beta(a,
Suppose that the model's pass; @1 probabilities across the benchmark problems follow Beta distribution:
pass; Beta(a, 3)
The probability density function of this distribution over the support & E (0, 1) is:
fax; a,
def
xa (1 xx)-1
(24)
B(a B)
where ar > 3 > 0 and B(,,) is the Beta function. For brevity, let def @a Under our assumed Beta distribution:
passBeta(a,
@k def
1 - Ep Betala -Pp)* pi)*1
(25)
[ Pi (1 11-pp pi)
B(a, B) (1 - pi)* dps
(26)
I(a + (3) (( +k)
(27)
[()) I( + B + k)
where I(-) is again the Gamma function, The [a) terms cancel, and standard asymptotic result of the gamma function for large & tells that:
T(3 +))
k
(28)
T(a + ++
and thus:
I(a + 3)
I(3 +k)
T(a + 3) k-a
(29)
I(3) +
I((3)
Recalling again that the expansion of log(-) for small r is - log(1 x) + O(²²), in our case, we obtain:
[(a + B) k O 0(k-2n) I(a + 3) k + o(k I(B)
- log passp
(30)
I(3)
From this final result, we see that under a Beta distribution and in the large is regime, the negative log aggregate success rate exhibits polynomial (power-law) scaling with & for exponent a
E.5. Kumaraswamy Distribution: pass; @1 ~ Kumaraswamy(a, B)
Next, suppose the model's pass;01 probabilities follow a Kumaraswamy distribution. The probability density function of this distribution over the support & E (0, 1) is:
fl f def a3xx011 (1-12)33
(31)
Again for brevity, let p; pass;01 @1. Under our assumed Kumaraswamy distribution:
ManKaaaaawaaaya..... @k 1 - Ep Kmmaravaamya..))l =p
(32)
=1 1 (1 - p)* ...pp (1-2- 1
(33)
Define the integral
I del ()))1)))* = (000)))(1100000))0000 w) B a-1 (1 --)3-1 dr.
(34)
We aim to analyze I& for large Notice that (1 x1) * is exponentially small in k unless x is very close to Thus, intuitively. most of the contribution Tg arises from [0. (11/)))
30

---

How Do Large Language Monkeys Get Their Power (Laws)?
Step 1: Split the integral into two parts.
Fix constant 0. Write
/ oo/k
[ [..1dr Jo
Is 1...dr +
Io, + rigihhtt
where [...] indicates the same integrand. In the region [e/k, 1), we have (1 2)} S e- S < Hence
rightt ( Since can be made arbitrarily large, righh becomes negligible compared to any polynomial in 1/k.
Step 2: Approximate the integrand in the small-r region. On [0, c/kj, we use the approximation log(1 x)
-I - (a²)) Thus
(1 x)* = exp (k: log(1 xx)) = exp ( + Okkz²))
Since < c/k implies 2x² S 2// (()/k), and exp(e) = 11 Oce), we get
(1 - x²) x)) (0w(O(1)p))00cepeee11 = (1)
Furthermore, since (1 y)mh = 1 my + 0vr²). ) for small r
- -- = 1 (B 1)x² O(x2a) 0(x)).
In the region c/k, that error is O(k Hence, within the small-2 region, the integrand
(1 2² a aa-1 (1 xa)
can be approximated by
a 0-1 e + za-
Thus
Ik.lett = fell a 100 -k dr + 0
po/k (oth x xa-le- (o-le-kadr) dx) + 0
Step 3: Substitution kr. To handle po/k aa-le-ke dæ, we substitute kæ Then = u/k, dx du/k, and the upper limit x c/k becomes = Hence,
o/k gel
l" (4)
a-1
du k
xa-1 c kx dx =
e
0
= k- 1-" a- -- du.
As 00, So w--ee- du T(a), and for finite the remainder is O 9). Therefore,
S. xa-le- k I(a) 0
and absorbing the constant into big-O notation gives
S. - c ck dz dz= + 0(k-a99) for some > 0.
Multiplying by the factor B, we deduce that
1 = + aa--a
31

---

How Do Large Language Monkeys Get Their Power (Laws)?
Step 4: Final conclusion for the success rate.
Recall p @k /.- Hence
pass unsssaaaaaaaa @k a (T(())) O(.
Since this tends to 1, its negative log is governed by the magnitude of a Using the expansion - -log(1 - y) y + Ovy²) as - 0, we get
llog p @k a. k o(k
That is precisely polynomial (power-law) decay in the negative log success rate with exponent a.
E.6. Continuous Bernoulli Distribution: pass; @1 ~ ContinousBernoull(A)
Next, suppose model's pass; @1 probabilities follow a Continuous Bernoulli distribution. function of this distribution over the support æ E [0, 1] is:
The probability density
f(z; A) def (11)A=(1-2) -
(35)
2
A 1/2
((2)
(36)
tanh (1-2A)
otherwise
1-2A
The density can equivalently be rewritten in more convenient form for our purposes:
-11 - 1)- C())
(37)
Because the individual success probability low in our data, shall consider the small 1/2 regime. We follow the same approach with the Kumaraswamy distribution.
Step 1: Write the aggregate pass rate.
The aggregate pass rate is defined as:
DaSSContiiuonneeeenollllll @k - 1 /x, where IN del S/0 (1- p)*h f(p; A) dp.
Substituting the density f(; A), get:
I = I (1- (1) A)^ dp.
Step 2: Simplify using exponential form. Using the exponential rewriting:
XP (1 - --=-1--- (1 A) exp p log }
the integral becomes:
4 = C1- / (1 p)* exp (p log (1x)((=))) dp.
Step 3: Dominance of the small-p region. For large k: (1 11-pl p²* decays exponentially unless is close to 0. Thus, the main contribution to the integral arises from the region E [0, c/k], where e > is a constant. Decompose the integral:
co/k |...) + 1' 1..10-.111
1 =
dp. def Tkllen + Jk.rihtt
o/k
In the region p E lc/k, 1]. we have (1 p)* / -kp S < l/k. Thus, we focus on Itlett
2e-,mmaing /nrrihht O(e which is negligible compared to
I C())(1 2) 1/0
c0/k
(1 p)* (ppp)olo(TT)) dp.
32

---

How Do Large Language Monkeys Get Their Power (Laws)?
Step 4: Approximate the integrand. Forp E (0,c/k) use the same approximations from the Kumaraswamy derivation:
(1 p)* =e kp (1)), (sp(plog(1x)) -1p)
Thus, the integrand becomes:
(1 p) exp p poog (1))((1x)) = = (1+0
def = kp, sopsop p = u/k ls and dp = du/k. The integral becomes:
Step 5: Change of variables.
Let u
It.left = C(A) (1- 2 //" e e" (1+ (1+0(u/H)) O(u/k)) du
Split the integral:
C(2) - 2) (e e du + 0
Jg,len
4
k2
As + 0o, To e du + Thus:
C()) 2)
+ +0
Ikllett
k
Since rightt = ()) is negligible, we have:
C1 A) + 0
I
h
12
Step 7: Final conclusion for the success rate. Recall:
paSSCCantiaoouseennolllly @k In.
For large k this implies:
DaSSCContinonsseeroollllll @k = C(A) (1 A)
+ +0
k
k2
Using the expansion - log(( y) + O(y²) for small y, we find:
- log (pasSsconttuuuoueeenollllll @k) = C(A) - (1-)k + o(k
That is precisely polynomial (power-law) decay in the negative log success rate with exponent -1. As side comment, recall that tanh -1 (x)) log [1+2), the normalizing constant C(A) can be rewritten as:
2 log
1 (1-11)
1 (1>)) A A
C(A) = - 22 2
log
(38)
(1-11 1- 1 2A
Thus, for small X, note that C(A) ~ log(1/A) = - log(A). For i < - log(())) A), the 1/k formula is valid. However, near - log(A), the leading term - log(A)/k becomes of order 1, and for k > - logg() (A), the success rate is now very close to 1. Consequently, we see that X is very small, there is a soft cutoff scale around ie = - log(A).
E.7. Any Continuous Distribution with p(pass, @1) = c > 0
Suppose that the distribution over pass;@] @1 continuous and has constant non-zero density near O:
f(0) c > 0
(39)
Because the density is continuous at with /(0) = c 0, there exist some > such that:
f()) = c O(p)
for all p E [0.
(40)
33

---

How Do Large Language Monkeys Get Their Power (Laws)?
Because the small pass; @1 region dominates for large k, similar argument the Kumaraswamy argument and Continuous Bernoulli argument yields power law scaling with respect to with exponent 1:
- llog pass Kk - ck + o(k )
(41)
This result is consistent with the Continuous Bermoulli, where is given by (Continuos (0; = C()) - X) for A 1/2. This result reveals that the Continous Bernoulli is just one instance of a larger family: any continuous distribution with non-zero constant density at pass; @1 = 0 will exhibit power law scaling with exponent -1.
E.8. Reciprocal Distribution: pass, @1 Reciprocal(a.) b)
Next, suppose the model's pass;01 @1 Reciprocal(a, b) distribution with 0 < a < b < 1. The probability density function of this distribution over the support [a, bj is:
[ a) (log(b) - log(a)) x
(42)
As with the other distributions, the aggregate success rate after k attempts is:
| ama
pasSRecirocaal(a.b)
@k
EE pass, @kk
Ts. where T
def
(1- (1-2) x²** (log b log a) x
dx.
We aim show that I& is on the order of (1=a)* The main contribution to the integral arises from the vicinity of r a. because (1 - . decays rapidly grows away from
Step 1: Change of variable. Define y def x a, so the domain x: E ja, b] becomes y E [0, - a]. Then
(1 (1-28 x)) ((1 a) -))=
and
1 b-a fr ((1 a) 4)* 1 dy. log(b/a)
1-=0
a + y
Step 2: Expansion near y 0. For small y. write (1 a) y (1 - a) - T hence
log((1 - a) - y) log(1 a) + log =-
Using log(1 -2) = -2 0(23) for small 2, get
log((1 (1 - a) - y) = -log(1 - a) - I - + 0
y
so
(1 (1-a a 4)* = exp l log(1 - a) - + 0 01 [1-0}=
In partieular, for y up to c/k, the term kyz = =1(1) remains bounded, so
(1 (1-a- y) = (1 a)* exp (-
ky 1-a
[1t]][++
Step 3: The integral is dominated by y E [0, 0 For large k, exp(- ky decays quickly once y exceeds a multiple 1-a of T Consequently, the integral from y co/k: to b is exponentially small in k. On [0, co/ /k), we also have (a + 1 Thus
cafk
1 = log(b/a) y=0
(1 - a - y**
dy + (exponentially small tail).
a a
34

---

How Do Large Language Monkeys Get Their Power (Laws)?
Substitute our approximation from Step 2 into the integrand:
(1 - a- u)
= (1 - a)* exp
Rv I-a) _ + +
a a+ y
Step 4: Change variable kv
Then (1-a)u and dy
- du. The upper limit y co/k corresponds
I-a"
co (TT)) so
co/k
exp
Ry 1-e
dy
cou (1-a)
e 1 du.
y=
==
Letting co + oo only contributes e -0((-1) factor to the tail, which vanishes. Hence
exp
(xp(-110)) -
dy = h u=0
1 a
e e du
y=
Putting all factors together,
log(b/a) (1 (1-a)² a)* [
1-a b
+ (exponentially small in k).
Thus in big-Theta form,
I θ 0("-aa")
Conclusion. Since b) @k Ik, get
@k = (1-a)*
pa
Moreover, using log(1 y) + Oz²²) for small it follows that
log ( @k) = 0 (1-a)+
Hence the negative log aggregate success rate converges to exponentially fast in k which is not a power law in k.
35

---

How Do Large Language Monkeys Get Their Power (Laws)?
Sufficient Condition for Power-Law Scaling in Negative Log of Aggregate Success
Theorem E.1. Let DD be a probability distribution on [0, with PDF f(p). Suppose there exist constants > C
0 0 and 8 >> O such that, for all 6, have
f(p) - Cp b-1 + 0010
Then, for large
1 passp @k CT(b) + O(k-b-min(1)) 1,0)).
which implies
log passp@k) cT + o(k
Equivalently, including the leading constant),
log(passp CT(b) Eーt
Proof. Step 1. Decompose the key integral. Define
passp (((-p)) p)* f(p)
Ig def
For positive constant c > 0, split Tg:
Is = S. (1 (1-p)* f(p) + S ofk
(1 (1-p)* f(p) dp def
IkJettt + Iarightt
Right Tail Bound (Lg.righ))
For p 2 k/ki observe (1 pp- kp Se Hence
rigitt jefk (1-))) S e S s(D) dp = e
Since e can be made arbitrarily large, e can be driven below any power of 1/k. Thus Thrrghtt = =(k-a) for any a > 0. We may therefore focus on
o/k Tk.loft - (1- f(p) dp,
knowing that right is negligible in polynomial-type estimates.
Step 2. Use the assumed behavior of (p) near p O.
By hypothesis, for up to some d > 0,
f(p) = Cp 0pp--140)
Choose c/k 6, p S c/k for the left integral. Then
= S
ofk (1-p)* + +0(1--100 dp.
Igllef =
Split it into main term and error term:
Iklleft = C ofk (1 f(
c/k |e((1 (1-p)* p)* 0(p0-1+0) dp.
p b- dp dp +
Denote these Tmain and Torr. respectively.
Step 3. Approximate (1 f** by e kp and control the error.
For in [0, c/k], expand log(1 - p) -p -p+0 O(p²). Thus
k)* exp(> p)) = -k exp(O()p3)) -kp [1 [1+ 0(kp)))
36

---

How Do Large Language Monkeys Get Their Power (Laws)?
Since p c/ki get k2² s e /kk which is bounded for large k. Consequently,
p) = kp + -
We will use this in both maain and Terr-
Step Main term main.
main =c c//k (1 p)* 1--1 -p.
Substituting (1-p) p)* +k(kpp c
=c l/
o// k-kp p--lddd dp C Je of Olkpbtl dp.
T Tmain
Call these integrals Ti and T2.
Ti term.
=C J
o/f D pb-le e dp.
Make the substitution def kp. Then p = u/k, dp du/ki and p 1-1 k-b b-1, The upper limit p = c/k becomes
Thus
[ e e- du T = C (N) (x)
k =Ck-b / c
As + 00, To u- + l(b). So
Ti = ((()) - Ro), where |R 0(e-).
By choosing large after k + we conclude
Ti = = +o(
T3 term.
T. C o/k Olkputzg kk) dp.
Inside the integral, kp b+1 ck the main factor. Substituting u det kp again,
p b+1 = =# 6+1 k: -- u b+1
Hence
T2 =2 / c/k kp E -kp k = O(ki) / p b kk dp.
calk
Substitute u kp and dp du/ki ki Then
1 (#) -- du k-k))-ba b-2 1 u b+1 e- du O(k-b-1
T2 = O(k)
dr
Thus T2 is of strictly smaller order than bb Combine and T2:
T Trnin = CT(b))-b
+
37

---

How Do Large Language Monkeys Get Their Power (Laws)?
Step 5. Error term T Ter Recall
Torr = / c/k (1 (1-p)* Opp h- dp.
Exactly the same substitution (1 -p)x c-p + e(kkpp=kk)) plus u = k shows
Terr ofo
1/ p p * dpd + off o/k kp b krd).
When substituting u = k p,, the exponent on p increases by +1 each time if we multiply by k. so each term of order A or smaller. Concretely.
6- -
c//k 1eth pb-1t0 e - h (( ub-1to - du -
and similarly for the second term, which even smaller. Hence
Terr = 06k
Step Putting all together. Summarize:
Taain
Terr CT(b) k- 0(k + 0(k
Thus
Itlleft = CT(b) + 0(k-b-min(1,))
Recalling the tail piece right. = - = =o(k-a) for any we obtain
Is = Tk,left + Ik.right = C] + ol-b-)m)))))))
Hence
paspasp==ll Is ~ kT
Final negative-log argument.
Since
passp @k = 1 14 = -1-(0))))) + + =kk-bm-min()0)))
for large l it very close to 1. Then
- log ppsss
= = -
Using the expansion - llog(1 x) ((22) as 0. and here x CT(b) k get
log passp @k) CT(b) +o( :
In the "~"" notation including the leading coefficient:
log passp
CT(b) b
This completes the proof.
38

---

How Do Large Language Monkeys Get Their Power (Laws)?
E.9. Necessary Condition for Power Law Scaling from Distribution over pass; @1
Theorem E.2. Let be probability distribution over [0, 1] with PDF f(p) satisfying the following regularity near p 0:
No point mass at p 0. So Jo f(p) and is genuine PDF on (0, 1].
Continuity and nonnegative behavior near p 0. There exist > 0 such that f is continuous on [0, 6] and has no pathological oscillations or singularities that violate integrability.
Define the aggregate success rate attempts:
passp @k def S.' [ (1-)) d(
and relatedly
Tk S [11- -)* dl 1 pass Ök
Assume that there exist constants
and such that for large k:
log passs @k) ~ Ak
Then
LN Ak- o(k-)), and under the mild regularity assumptions above,
A
f(p)
p b-1 as p -
r(b)
Proof. Step 1. Relating 4 to loggpassspp @k).
By definition,
1-Ia. Ik. In f(a-p)) ()- f(x) dp.
passp @k
1
Since
- llog ppassp0kkk ~ Ak
we have, for large k.
passp @k exp(-Ak (1 (l+--A)-((+o(())) o(1)))
When is small, exp(-x) = x + ( Thus
I 1 - passp Ok Ak- okk
So
I ~ A
Step 2. Restricting to a small interval near p = 0.
Since (1 1-p) f** decays exponentially once p is on the order of 1 1/k: ls or larger, we split:
Ib def ( (((1-))* -1-pp* f(p) dp = 1oth
o/k (1- f(p) dp + I (1 (1-p) p)* f(p) dp def
I + Le.ight.
o/k
39

---

How Do Large Language Monkeys Get Their Power (Laws)?
for positive constant c. In the region p 2 c/k, have (1 -p) <ekpp e
rightt S E [ dp e
Since 0 can be made large, e can be driven below any fixed power of 1 /k. Hence for the O(k-9) behavior, the main contribution comes from [0, c/k].
Thus
4. = Je + o(k for every > 0
Step 3. Change of variables and controlling the ratio of (1 t to e kkp
(a) Ratio to e kk, For p E [0, f define the ratio
(1 p)*
(p)
def
kp
We will show that (p) stays close to 1 uniformly in E [0, c/ki for large k. Indeed,
(1- p)* exp kk log(1 p)]
n²
log(1 p) pp
2
Hence
pp²
pa
log(1 - p) +p =
= =O(p²)
as p - 0.
2
3
Multiplying by k we get
1: [log(1 - p) + p. = (1pp²).
Since S p S f implies kp² S
² E.
which - - oo, it follows that
ke log(11 - p) = - 0
Exponentiating:
(1 11-p) p)* - exp (o(t)) e *D (-+x
Thus
Re(p) (1 p1
1+0
kp
with the O bound uniform for all E [0, c/k].. In other words, there is some constant M > 0 (independent of k) such that
M for all p E 0, T
[R((p) - 1| s
(b) Integral expression using R2 (p). Hence on c/k]..
(1 - 4)* f(p) -kp Ra(p) f(p)-
Thus
Ik lalk-- --p f()) Rx(p)
off
Define (p) def Ra 1, which satisfies [a(p)] S M/k. Then
1klett = felt k f(p) dp + (ahe - f(p) Ae(p) dp.
(43)
40

---

How Do Large Language Monkeys Get Their Power (Laws)?
Step Substitution w = p and deriving ('() ~ 6-1,
(a) The leading part. Focus on the first term of equation 43:
c/k
e kk f(p)
del p. so = and dp Substitute v
x du. The upper limit
becomes a = c. Thus
yatl kk k
dp (-
du k
Jo E
Hence
1 {-- E e- f( du. e kp dp = k J
(b) The error part. The second equation 43 has 4x(p) (p( satisfying Ax(p)) # So
Jou of
M oo/k e e f(p) dp. <
e kr 5(p) (x)))
But the integral relk --*-((p) dp is precisely the leading part we just considered. Thus the error bounded by M times k
term that will turn out to be O(k b). Hence the error is subleading if is not the case but even then, we can keep
track of it systematically.
Overall, combining both terms, get
Te (:= c- (#) du + O (leading integral))
(44)
(c) Matching O(k-b), Since Is Ik.left + tigrtght with righttht negligible, we have
du + (small corrections). I 1e (ee e" s (x
1
But by hypothesis, 4 ak Thus
k-
/ e- ee"5 (("s(H du + (smaller terms)
a k--b
(45)
0
Hence the expression
[ c du
must be (k: 1-b) for large k. Since z is small for S u S c, we are effectively sampling f near 0. For the integral to
produce b- we deduce
o (()))
i.c. if must behave like n01 near p 0. Rewriting the constant in front, one obtains
(xx)
=
12 k
6-1
some positive constant
(We then identify that constant with r(6) by matching the integral precisely, just in the prior argument.)

---

How Do Large Language Monkeys Get Their Power (Laws)?
Step 5. Conclusion. We have thus shown that over p c/k], one has
(1 (1- e [1+ [
and upon integrating, the required - form for I forces
f()) r(b) p p01 + o(p"-1), a -
A
This completes the necessity proof.
Remark (Mild Regularity). f had bizarre oscillations or nonintegrable singularities near 0, the integral fo (1-p)* f(p) dp might not produce a clean k_b Typically, we impose monotonicity or at least continuity near p = O, no atom at 0, and ʃ(0) = 0 b > or (() > 0 if = 1, etc. These assumptions exclude pathological behaviors and guarantee that the local shape of f(p) drives a clean power law.
42

---

How Do Large Language Monkeys Get Their Power (Laws))
F. Maximum Likelihood Estimation of Scaled Beta-Binomial Distribution
To model the distribution pass; @1, we can perform maximum likelihood estimation on a scaled three-parameter Beta- Binomial distribution, which we chose because each attempt on the i-th problem is i.idd Bernoulli random variable with success probability pass; @1, and we introduced scale parameter because the largest pass; @1 I values were typically 1-2 orders of magnitude less than 1.0 (the maximum of the unscaled beta distribution's 's support).
In greater detail, as background, the 4-parameter Beta distribution has PDF
py (yr, a,,,,,,, def (U 10-a)=-((c y)
(46)
(c- 1--a-tt--1 B(a, 9)
where -B: )) is the Beta function. If the minimum is fixed at 0 and the maximum e is constrained to < e < 1, then the scaled three parameter Beta distribution simplifies to:
0,c) = pe- p))
fplp 3,a -,0,,,)
(47)
ca+ B(a,
We want the PMF of a three-parameter Beta-Binomial distribution based on this scaled Beta distribution. For n samples and a successes, the PMF is:
P(X B,c,B, def {-"
n2 2 p² (1 (p 3,
(48)
ca+s B(a, B) 1 10tta-111 (1 -p) p)" (c- (c-p)8-1 dp.
(49)
Using a change variable def = cz, the PMF can be rewritten as
n2
B(a, B) L z-ta-1 (1 (1- (1-2)8-1 (1-2))-(((cc))) cz)m==a dz
P(X = x;a,3,cm))
(50)
e ("B(x +a, 3)
2fi (-(n x a x 3: B:
(51)
B(a, 3)
where () is the (Gauss) hypergeometric function.
43

---

How Do Large Language Monkeys Get Their Power (Laws)?
G. Maximum Likelihood Estimation of Scaled Kumaraswamy-Binonia Distribution
To model the distribution pass;11, we can perform maximum likelihood estimation on a scaled turee-parameter Kumaraswamy--innnt distribution, which we chose because each attempt on the i-th problem is an i..d. Kumaraswamy random variable with success probability pass and we introduced scale parameter because the largest pass; @1 values typically 1-2 orders of magnitude less than 1.0 (the maximum of the unscaled beta distribution's 's support).
In greater detail, the scaled three parameter Kumaraswamy distribution simplifies to:
a3 ca pa
3, 0,
(52)
over the support (0, c). The rescaled Kumaraswamy--inomial distribution then has PMF:
n12 a ca pata-i(1- (1 (1-p)" (1-p)"~= (1 (1- (1-(2) (2) P(X 3,c,, = /
8-1
dp.
(53)
One can perform change of variable but simplifying yields sums of hypergeometrie functions that add little conceptual clarity and so we resort to numerical integration using Python's 's mpmath library (mpmath development team, 2023).
44

---

How Do Large Language Monkeys Get Their Power (Laws)?
H. Maximum Likelihood Estimation of Scaled Beta-Negative Binomial Distribution
To model the distribution of pass; @1, we can perform maximum likelihood estimation on a scaled three-parameter Beta- Negative Binomial distribution. Recall that the scaled three parameter Beta distribution is:
p= 1 8-8 felp a, 3, - =0,)= ca+ B(a, 3)
(54)
We want the PMF of three-parameter Beta-Negative Binomial distribution based on this scaled Beta distribution. For desired successes, the PMF that we first draw failures is:
1/¹ (xttr=))
pa- (c p) dp
P(X r,,,3,c,r)
p' (1 (1-1p) p)"
(55)
ca+ B(a, 3)
NegBin(T.pp
scaled Beta
-(=rr-1 x
J p prrt-1 pp) (c 2)3-111 dp.
(56)
cat--1 (0t--1((a,B) B)
Next, substitute p
c2 = dp- = edz which rescales the domain [0, cl to [0, 1]. Under this change:
p prta-1 (ta)) crta-1 1 zrta-1
(c (c-p) (c-p) 8-1 (c c2) 0-1 =cc(1 2))50+= c (1-2) 2)0-1
(1-p)² (1-
Putting these into the integrand:
r (1 (1-p))(c-))) (c- dp (cttal ((1-cz)*) (c0-1 (1 (1-1)0-1) (edz).
Factor out the constants in c:
= c cll c x zta-1 (1 - c2)* (1 2)8-1 dz.
Since cr r-a-1 .0-1.c. -cta+t-1 we get
p prta-1 (1 - 2)² (c p) dp c++a+B-1 2 rta-1 (1 2)8-1 (1 cz)² dz.
Plugging back into P(X x;; a, B,,, and simplifying:
,(x=x;a,,,,7)) xx + (
L rrta- (1-2)8-1 (1 cz)² dz.
(57)
x
B(a, B)
We can re-express this using the (Gauss) hypergeometric function 2F((,
(8 + (M+r-1)) ) cf B(r + a, 3)
P(X ==;;,,,,cc))
B(a, 8) 2F1 2F((-z,
r-,, r r+a+ + B)
(58)
x
45


Reviews:
[{"review_id": "5J5XLgVhGe", "reviewer": "Reviewer_UNs5", "summary": "This paper investigates the negative log of the average success rate scales as a power law with the number of attempts when LLMs make multiple independent attempts at a task (mathematical problems or jailbreaking). The authors identify a paradox that for any individual problem, success rates should improve exponentially (not as a power law) with more attempts. The paper resolves this paradox by demonstrating that power law scaling emerges from the distribution of per-problem single-attempt success probabilities. Specifically, the authors prove that a power law left tail in this distribution is necessary and sufficient for the emergence of aggregate power law scaling. The paper provides a theoretical framework that explains previously observed deviations from power law scaling and introduces a more sample-efficient method for estimating power law exponents.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and thoughtful review of our work. We will correct the typo you identified in line 255, changing \"Kuamraswamy\" to \"Kumaraswamy.\" We address other points below:\n\n### Origins of Heavy Left Tailed Distributions\n\n> The brief discussion of benchmark design and selection bias could be expanded.\n\n> Your explanation focuses on the statistical properties of problem distributions that lead to power laws. Could you elaborate on potential causal factors that might create these heavy-tailed distributions in natural language tasks?\n\nIf our paper is accepted, we will use the additional page in our camera-ready version to expand on benchmark design and selection bias as factors leading to heavy-tailed distributions. Your question about causal factors creating these distributions touches on an important insight: power law scaling emerges in a \"Goldilocks zone\" of problem difficulty. For heavy left-tails to appear, we need problems that are challenging but not impossible—difficult enough to require many attempts yet still solvable. This explains why we wouldn't observe power law scaling when applying state-of-the-art models like GPT-4.5 or Claude 3.7 Sonnet to relatively simple benchmarks like GLUE (too easy), nor when applying these same models to extremely difficult tasks like Millennium Prize problems (effectively impossible). The power law phenomenon manifests precisely in this intermediate difficulty range.\n\nIf you can think of a more rigorous way to investigate causal factors, we would welcome your suggestions!\n\n> The empirical analyses are limited to specific model families and benchmarks. Verification across a broader range of models and tasks would strengthen generalizability.\n\nWe agree that verification across a wider range of models and tasks would strengthen generalizability. We relied on existing datasets from Brown et al. (2024) and Hughes et al. (2024) because generating 10,000+ attempts per model per problem involves substantial computational costs, e.g., drawing 10k attempts from 10 models across 5 benchmarks with 100 problems each would require 50 million samples. While computational constraints limited the scope of our current study, we view this as an important direction for future work and are exploring more efficient experimental designs to validate our theoretical framework more broadly.\n\n### Dark Matter of Neural Scaling Laws\n\nRegarding your question about the \"dark matter\" of neural scaling laws, this is indeed the focus of our ongoing follow-up work! The experimental approach involves training numerous small models on scaling ladders and running scaling predictions in reverse to identify deviations from expected power law functional fits. This allows us to fit more complex functional forms and better understand deviations. We're particularly excited about this direction because experimenting with small models enables cheaper and faster iteration, but is currently not useful because extremely small models are poorly predictive of massive models. If we can figure out the appropriate scaling corrections, this would accelerate experimentation with larger models.\n\nThank you again for your insightful comments and strong support for our work.", "claims_and_evidence": "- The claim that individual problems scale exponentially is supported by mathematical derivation in Section 2 and empirical evidence in Figure 3, showing negative log success rates falling exponentially for each problem.\n- The necessary and sufficient conditions for power-law scaling are rigorously established through formal mathematical proofs (Theorems 3.1 and 3.2) and validated empirically.\n- The explanation of why Llama 3 8B IT deviates from power law scaling (because its success distribution lacks the required heavy left tail) is empirically validated.", "methods_and_evaluation_criteria": "- The authors leverage existing datasets from prior work (Brown et al. 2024, Hughes et al. 2024), ensuring comparability with published results.\n- The distributional models (Beta, Kumaraswamy, etc.) used to characterize success probability distributions are appropriate given the bounded nature of probabilities.\n- he evaluation of the distributional estimator includes both agreement with least-squares on real data (Figure 6) and superior performance on synthetic data with known ground truth (Figure 7), providing a comprehensive analysis.\n\n[1] Large Language Monkeys: Scaling Inference Compute with Repeated Sampling,  Brown et al., 2024\n\n[2] Best-of-N Jailbreaking, Hughes et al., 2024", "theoretical_claims": "I verified the key theoretical claims\n\n- Theorem 3.1 (sufficiency): The proof correctly shows that power-law behavior near zero in the distribution yields aggregate power-law scaling.\n- Theorem 3.2 (necessity): The proof correctly establishes that aggregate power law scaling requires a power-law left tail in the distribution.", "experimental_designs_or_analyses": "- The authors appropriately visualize both per-problem exponential scaling and aggregate power law scaling.\n- The distribution fitting and parameter estimation methods are appropriate.\n- The backtesting approach for comparing estimators is rigorous, showing the distributional estimator achieves lower relative error.\n- The authors appropriately account for sampling limitations and edge cases (problems with extremely low success probabilities).", "relation_to_broader_scientific_literature": "- This work extends recent work by Brown et al. (2024) on \"Large Language Monkeys\" and Hughes et al. (2024) on \"Best-of-N Jailbreaking\" by providing a theoretical explanation for their empirical findings.\n\n- It connects to the broader literature on scaling laws in neural networks (Kaplan et al., 2020; Hoffmann et al., 2022) by revealing distributional foundations for observed scaling patterns.\n\n[1] Training Compute-Optimal Large Language Models, Hoffmann et al., 2022\n\n[2] Large Language Monkeys: Scaling Inference Compute with Repeated Sampling,  Brown et al., 2024\n\n[3] Best-of-N Jailbreaking, Hughes et al., 2024\n\n[4] Scaling Laws for Neural Language Models, Kaplan et al., 2020", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Weaknesses:**\n\n- While the paper explains how power law scaling emerges, it offers limited insight into why single-attempt success rates have heavy-tailed distributions in the first place. The brief discussion of benchmark design and selection bias could be expanded.\n\n- The empirical analyses are limited to specific model families and benchmarks. Verification across a broader range of models and tasks would strengthen generalizability.", "other_comments_or_suggestions": "- Typo in line 255: \"Kuamraswamy\" -> \"Kumaraswamy\"", "questions_for_authors": "1. Your explanation focuses on the statistical properties of problem distributions that lead to power laws. Could you elaborate on potential causal factors that might create these heavy-tailed distributions in natural language tasks?\n2. In Section 7, you speculate about connections to pretraining compute scaling laws. Have you found any empirical evidence that supports the \"dark matter\" hypothesis for neural scaling laws?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VQFaQUH9xb", "reviewer": "Reviewer_5fN8", "summary": "This paper explores the scaling behavior of LLMs when inference-time compute is increased through repeated sampling. While failure rates for individual problems should decrease exponentially with multiple attempts, the authors observe that the aggregate success rate across many problems follows a power law. They resolve this paradox by demonstrating that this phenomenon arises from the distribution of single-attempt success probabilities, which is heavy-tailed that a small fraction of extremely difficult tasks disproportionately influences the overall trend. Through empirical analysis on math problem-solving and multimodal jailbreaking tasks, they confirm that while individual tasks improve exponentially, the global trend follows a power law due to the nature of the problem distribution. This work introduces a new distributional estimator that predicts the power law exponent with far less compute than traditional methods, improving efficiency by 2-4 orders of magnitude. Furthermore, they explain why some models deviate from power law scaling, attributing it to the lack of a heavy-tailed success rate distribution. Ultimately, this research enhances the understanding of inference-time scaling and provides a more accurate framework for forecasting LLM performance, offering practical implications for model evaluation and optimization.\n\n## update after rebuttal\n\nI appreciate the clarification and insight from the authors. My concerns have been addressed. I have updated the scores accordingly.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We appreciate your recognition of our work's strengths, particularly that our paper \"presents a novel theoretical framework explaining why per-problem success rates follow exponential decay, while aggregate success rates exhibit power law behavior\" and that our distributional estimator \"significantly improves compute efficiency, making it a practical contribution for LLM evaluation.\"\n\n### Models and Tasks Are Diverse\n\n> However, the study is somewhat limited in scope, as the sample sizes for both benchmarks (128 math problems, 159 jailbreaking prompts) may not fully capture model behavior across diverse tasks.\n\nWe believe our study offers substantial diversity.\n\n1. Our analysis spans leading frontier models from four major AI companies (OpenAI, Google, Anthropic, Meta), open-parameter models ranging from 17M to 12B parameters, and fundamentally different tasks (mathematical problem solving and multimodal jailbreaking). This diversity strengthens our confidence in the generalizability of our findings.\n\n2. We agree that verification across an even wider range of models and tasks would further strengthen generalizability. We relied on existing datasets from Brown et al. (2024) and Hughes et al. (2024) because generating 10,000+ attempts per model per problem involves substantial computational costs. For perspective, an experiment with 10 models across 5 benchmarks with 100 problems each would require 50 million sampled outputs.\n\n3. From a statistical perspective, sample sizes matter in order to make precise statistical statements, e.g., determining confidence intervals. If you feel like 128 and 159 problems with >=10k samples per problem are inadequate for specific claims, could you please tell us which claims you find inadequately justified so we can better assess?\n\n### Real-World Applications\n\n> The proposed estimator for power law exponents is validated on synthetic data and limited benchmarks, but its performance on real-world applications (e.g., machine translation, conversational AI) remains uncertain.\n\n\nThis is a valid concern. While our current validation on both synthetic and real-world data demonstrates the estimator's effectiveness, we acknowledge the need for broader validation across diverse applications. Our method's foundations in statistical theory provide confidence in its generalizability, but we agree that testing across additional domains would be valuable. We view this as an important direction for future work and are exploring partnerships to apply our estimator to machine translation, conversational AI, and other practical applications.\n\n### Deviations from Power Law Scaling\n\n> The study notes that some models (e.g., LLaMA 3 8B IT) do not follow power law scaling. Do you have hypotheses on why \n\nOur theoretical framework provides a clear explanation: power law scaling emerges only when the distribution of single-attempt success probabilities has a heavy left tail, which Llama 3 8B IT lacks when tested on jailbreaking (as shown in Figure 4).\n\nWhat this means practically is that Llama 3 8B IT has lower robustness against adversarial attacks than the other models Hughes et al. 2024 tested. This could be attributable to many reasons. This could stem from several factors, including its smaller size, potentially less extensive safety training, or absence of defense mechanisms likely present in API-based models like GPT, Claude, and Gemini. Unfortunately, the proprietary nature of these other models limits our ability to investigate these hypotheses further.\n\n### Deeper Theoretical Analysis of Why Heavy Left Tails Appear\n\n> While the authors demonstrate empirically that single-attempt success rates follow a heavy-tailed distribution, they do not provide a deeper theoretical justification for why this occurs in practice. They speculate that benchmark design and selection bias may contribute, but these points are not rigorously analyzed.\n\nThe best answer we can think of is that power law scaling emerges in a \"Goldilocks zone\" of problem difficulty. For heavy left-tails to appear, we need problems that are challenging but not impossible—difficult enough to require many attempts yet still solvable. This explains why we wouldn't observe power law scaling when applying state-of-the-art models like GPT-4.5 or Claude 3.7 Sonnet to relatively simple benchmarks like GLUE (too easy), nor when applying these same models to extremely difficult tasks like Millennium Prize problems (effectively impossible). The power law phenomenon manifests precisely in this intermediate difficulty range.\n\nIt is not clear to us what a more compelling or more rigorous investigation would look like. If you have suggestions, we would greatly appreciate them!\n\nThank you again for your insightful feedback, which will help strengthen both this work and our future research directions.", "claims_and_evidence": "### Theoretical Justification for Heavy-Tailed Distributions:\nWhile the authors demonstrate empirically that single-attempt success rates follow a heavy-tailed distribution, they do not provide a deeper theoretical justification for why this occurs in practice. They speculate that benchmark design and selection bias may contribute, but these points are not rigorously analyzed.", "methods_and_evaluation_criteria": "Yes. The paper employs a rigorous mathematical framework to establish that while individual problems exhibit exponential failure rate decay, aggregate success rates follow a power law due to the heavy-tailed distribution of single-attempt success probabilities. Empirical validation is conducted on two key tasks: mathematical problem-solving using the MATH benchmark and multimodal jailbreaking using HarmBench, both of which effectively illustrate how repeated sampling impacts model performance.\n\nThe paper also introduces a distributional estimator that predicts power law scaling exponents more efficiently than traditional regression-based methods, reducing computational costs by 2-4 orders of magnitude. The evaluation criteria, particularly the use of negative log success rate (−log(pass@k)), are well-motivated and provide clear insights into model scaling behavior. However, while the chosen benchmarks are appropriate, the study does not explore whether similar scaling laws hold across a broader range of NLP tasks such as summarization or question answering. Additionally, the underlying causes of heavy-tailed success probability distributions remain speculative.", "theoretical_claims": "The key theoretical contributions involve proving that per-problem failure rates decay exponentially while the aggregate success rate across problems follows a power law due to the distributional properties of single-attempt success rates.\n\n- 1. Exponential Decay of Per-Problem Failure Rates\n   - Claim: If each attempt at solving a problem is independent with a fixed success probability, then the failure probability over $k$ attempts follows an exponential decay.  \n   - This confirms that the failure rate decreases exponentially as $k$ increases. The proof is valid and follows standard probability theory, particularly the Bernoulli trial model where repeated independent attempts lead to geometric or exponential-like decay.\n\n- 2. Power Law Scaling of Aggregate Success Rates\n   - Claim: Despite individual problems following exponential failure rate decay, the overall success rate across problems follows a power law if the distribution of single-attempt success probabilities is heavy-tailed.  \n   - The authors show that if the distribution $p_D(pass_i@1)$ has a power-law-like left tail near zero, the resulting negative log success rate follows a power law in $k$. They provide sufficiency and necessity theorems, proving that this scaling occurs if and only if the distribution of $pass_i@1$ behaves in a certain way. The derivation follows known statistical results about sums of exponentials forming power laws in appropriate conditions. The use of Gamma functions and integral approximations aligns with established results in scaling law analysis.\n\n- 3. Connection Between Distributional Shape and Power Law Exponents\n   - Claim: The power law exponent $b$ of the aggregate scaling behavior is directly determined by the shape of the distribution of single-attempt success probabilities.  \n    - The authors analyze different statistical distributions (e.g., Beta, Kumaraswamy, Continuous Bernoulli) and derive their impact on the resulting power law exponent. They prove that a heavy-tailed distribution of single-attempt success rates naturally leads to power law scaling. The derivations match well-known properties of compound binomial distributions, where a sum of many exponentially decaying functions with varying rates can form a power law.", "experimental_designs_or_analyses": "The experimental design is well-structured to investigate the scaling behavior of LLMs under inference-time compute scaling. The authors conduct two core experiments: mathematical problem-solving using Pythia models on the MATH benchmark and Best-of-N jailbreaking on HarmBench, analyzing how success rates improve with multiple attempts. The negative log success rate (-log(pass@k)) is used as a primary metric, effectively distinguishing between exponential and power law scaling trends. Additionally, the authors fit success rate distributions using Beta and Kumaraswamy distributions, demonstrating that the heavy-tailed nature of single-attempt success probabilities explains power law behavior. Their proposed distributional estimator for power law exponents significantly reduces compute requirements by 2-4 orders of magnitude, showing clear advantages over traditional methods.\n\nHowever, the study is somewhat limited in scope, as the sample sizes for both benchmarks (128 math problems, 159 jailbreaking prompts) may not fully capture model behavior across diverse tasks.", "relation_to_broader_scientific_literature": "This work builds upon and extends key areas in the scientific literature on scaling laws, inference-time compute strategies, and power law behaviors in LLMs. It connects to scaling laws which showed that model performance improves predictably with increased compute, data, and parameter count, and later being refined by emphasized data efficiency over sheer model size. However, while previous work primarily focused on pretraining compute scaling, this paper shifts the focus to inference-time compute scaling, showing how repeated sampling affects model success rates. The discovery that per-problem failure rates decreases exponentially while aggregate success follows a power law introduces a new perspective, linking task difficulty distributions to inference efficiency.\n\nAdditionally, the study relates to Best-of-N sampling strategies which demonstrated that generating multiple outputs and selecting the best significantly improves performance. This paper extends those insights by providing a theoretical framework explaining why repeated inference exhibits power law scaling, depending on the distribution of single-attempt success probabilities.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "### Strengths\n\n- The paper presents a novel theoretical framework explaining why per-problem success rates follow exponential decay, while aggregate success rates exhibit power law behavior due to heavy-tailed task difficulty distributions.\n\n- The introduction of a distributional estimator for power law exponents is innovative and significantly improves compute efficiency, making it a practical contribution for LLM evaluation.\n\n- The work has security implications, particularly in adversarial robustness and jailbreaking prevention, by explaining how increased attack attempts affect model vulnerabilities.\n\n- The paper is well-organized, with a clear presentation of mathematical derivations and strong empirical validation.\n\n### Weaknesses\n\n- The experiments focus on MATH (128 problems) and HarmBench (159 adversarial prompts), which may not fully generalize to other NLP tasks (e.g., summarization, question answering, commonsense reasoning).\n\n- The proposed estimator for power law exponents is validated on synthetic data and limited benchmarks, but its performance on real-world applications (e.g., machine translation, conversational AI) remains uncertain.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "The study notes that some models (e.g., LLaMA 3 8B IT) do not follow power law scaling. Do you have hypotheses on why these models deviate from the expected trend? Could this be due to model architecture, training objectives, or tokenization differences? Understanding these deviations would help clarify when power law inference-time scaling can and cannot be expected.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GCg13Du5rO", "reviewer": "Reviewer_oCti", "summary": "The paper demonstrates that power law behaviour in “pass at k” metrics originates from a power law tail in the distribution of the “pass at 1” probability across the test set. Furthermore, it argues that directly modeling the “pass at 1” distribution leads to more accurate predictions for the values of “pass at k.”\n\n## Update after rebuttal\n\nI maintain my positive assessment of the paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback. We address your points below.\n\n### Improvements to Related Work\n\n>  Could they provide a parametric form for that distribution? Polo et al. (2024) likely also deserve more detailed discussion due to the pass@k experiments they describe in Section 4.5.\n\nWe will incorporate a more thorough discussion of Polo et al. (2024)'s Sloth and the related papers Ruan et al. (2024)'s Observational Scaling Laws, and Owen (2024)'s predictability analysis. Their latent variable regression models offer complementary perspectives to our approach. Interestingly, Polo's Section 4.5 pass@k experiments show relatively poor fits (their Figure 7), possibly because (as best as we can tell) they use the biased estimator of $\\operatorname{pass_i@k}$ that Chen et al. (2021) caution against. That said, while Polo et al. (2024) don’t define a functional form for scaling, our mathematical analysis could be combined with their estimated per-problem single-attempt success probabilities $\\operatorname{pass_i@1}$. It’s possible that their cross-benchmark fitting method gives better estimates of these per-problem single-attempt success probabilities, which would improve our method.\n\nWe will expand on this connection and highlight how per-problem analyses could potentially combine with such cross-benchmark approaches to further improve predictability.\n\n> What I expect to find in a related work section is insight about how these works relate to the paper. Here specifically, I am missing a discussion about prior work attempting to explain the origin of scaling laws currently listed in lines 390 to 395 - do any of these models predict a power law tail for the distribution of difficulties of individual test data?\n\nWe will revise the related work section to discuss contributions from key works beyond just listing scaling law analyses. To the best of our knowledge, no prior work has specifically attempted to explain the power law emergence with repeat sampling in the manner we propose, perhaps due to the recency of works like Brown et al. (2024) and Hughes et al. (2024).\n\n### Quantification of Measurement Error\n\nThis is an excellent suggestion. For Large Language Monkeys, each problem had 10,000 attempts sampled, making the per-problem single-attempt success rate $\\operatorname{pass_i@1}$ a Bernoulli estimator with well-understood standard error. The Best-of-N jailbreak case is more nuanced due to varying sample sizes across problems (as detailed in Appendix A).\n\nWe will add this measurement precision information to the main text and try to develop an appropriate visualization to represent the uncertainty in Figure 4.\n\n### Optimality of Discretized ML Estimator\n\n> Is the discretize+ML described in lines 370-379 optimal in any sense? More specifically, is it the true maximum likelihood estimator of the pass@1 distribution parameters given the observations?\n\nRegarding whether our discretize+ML approach is optimal: we cannot make such a strong claim. While our empirical results demonstrate its effectiveness for the specific task of power law exponent estimation, a formal proof of optimality would require additional theoretical analysis.\n\nThe approach may be particularly well-suited for estimating parameters that best describe the distribution's left tail, which is crucial for our application. However, as you correctly suggest, this is a potentially complex statistical estimation question that warrants dedicated investigation. We will clarify these limitations in our revision and position this as an opportunity for future research.\n\n### Invitation for Ways to Improve\n\nWe believe addressing these points will substantially improve the paper's clarity and impact.\n\nWe are committed to making this paper as strong as possible and would value your guidance on what specific improvements would strengthen the manuscript further. If you have any additional suggestions that would elevate your assessment to a 'Strong Accept,' we would be grateful for that feedback and would make every effort to address those points in our revision.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I did not check the proofs and derivations in the appendix, but overall the analytical claims in the paper make sense and agree with similar calculations I have made in the past in different contexts.", "experimental_designs_or_analyses": "I have checked the experimental design and analysis to the level with which it is described in the main text. Overall, I found it satisfactory; see a question and a suggestion below.", "relation_to_broader_scientific_literature": "This paper provides a simple but valuable insight into the shape of scaling laws with respect to success probability after multiple attempts. Prior work observed these sometimes behave as a power law, but this appears to be the first work to reconcile this finding with the fact that for an individual data point, the pass at k probability must decay exponentially.", "essential_references_not_discussed": "I am not aware of any gross omission of related work, but I am also not satisfied with the related work section in the submitted paper - see “Comments Or Suggestions” below.", "other_strengths_and_weaknesses": "Covered by my answers to the other questions.", "other_comments_or_suggestions": "1. While overall the paper is well written, Section 6 (Related Work) is sub-par: it is a single paragraph spanning over more than a column that reads as a laundry list of papers about scaling laws. I can find better lists of this sort online. What I expect to find in a related work section is insight about how these works relate to the paper. Here specifically, I am missing a discussion about prior work attempting to explain the origin of scaling laws currently listed in lines 390 to 395 - do any of these models predict a power law tail for the distribution of difficulties of individual test data? Could they provide a parametric form for that distribution? Polo et al. (2024) likely also deserve more detailed discussion due to the pass@k experiments they describe in Section 4.5.\n\n2. Figure 4 is missing some indication of what is the measurement error of pass@1. I am not sure what is the best way to visualize it - at the very least, you should indicate the (inverse of the) sample size used to obtain the pass@1 estimates.", "questions_for_authors": "Is the discretize+ML described in lines 370-379 optimal in any sense? More specifically, is it the true maximum likelihood estimator of the pass@1 distribution parameters given the observations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MdNiyVAIGI", "reviewer": "Reviewer_wsdP", "summary": "This work tries to explain a curious phenomenon in LLM test-time scaling via repeated sampling and verification, as well as in Best-of-N jailbreaking:\nwhile the per-problem failure probability should decay exponentially with the number of attempts,\nit is often observed in practice that the average success rate on a task (which contains multiple problems) exhibits a power law instead.\nThe authors prove theoretically that this should be the case if (and only if) the distribution of per-problem single-attempt success probability satisfies certain long-tailed property; \nsuch a condition is validated empirically for multiple tasks and LLMs.\nBased on such analyses, this work also proposes a distributional estimator for the coefficients in the power laws of repeated sampling, and validates its efficacy numerically.\n\n**Update after rebuttal:** I have read the authors' rebuttal (as well as other reviews), and will maintain my positive evaluation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and thorough review of our work. We appreciate your thoughtful assessment of our theoretical and numerical analyses, as well as your recognition of our contribution in applying the mathematical insight about power laws emerging from weighted sums of exponential functions to this specific domain. We will certainly fix the identified typo in Line 413, changing \"contributes is a new hypothesis\" to \"contributes a new hypothesis.\"\n\nWe are committed to making this paper as strong as possible and would value your guidance on what specific improvements would strengthen the manuscript further. If you have any additional suggestions that would elevate your assessment to a 'Strong Accept,' we would be grateful for that feedback and would make every effort to address those points in our revision.\n\nThank you again for your constructive engagement with our work.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence, both theoretically and numerically.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense to me.", "theoretical_claims": "I read through the theoretical analyses in the main text, which make sense to me.\nI only skimmed through the technical proofs in the appendix.", "experimental_designs_or_analyses": "I have checked all experimental designs and analyses, which are mostly standard statistical analyses for supporting the developed theories.\nI don't see any serious issue in the results.", "relation_to_broader_scientific_literature": "This work offers some mathematical insights for LLM inference scaling laws that have been extensively studied recently.\nThe key to solving the puzzle under consideration can be easily explained in one sentence (in Line 231 Left): \n\"(a known result that) power laws can originate from an appropriately weighted sum of exponential functions\".\nAlthough the solution becomes obvious once it has been presented,\nit is the essential contribution of this work to bring this to light.", "essential_references_not_discussed": "Not that I'm aware of.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "A typo in Line 413 Right: \"contributes is a new hypothesis\" --> \"contributes a new hypothesis\"", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper aims to understand scaling laws of the success rate when LLMs make multiple attempts to solve a problem through multiple trials. The work demonstrates why/when one observes a power-law (opposed to say, an exponential law) through a mix of theory and experiments and provide a sample efficient method to estimate the exponent. 

All reviewers unanimously liked the submission due to its strong theoretical proofs (Theorems 3.1 and 3.2) and the rigorous design of the empirical evaluations. Moreover, reviewers also appreciated the explanations of cases in which power-laws are not observed (Llama 3 8B IT). 

Reviewer 5fN8's concerns about a lack of deeper theoretical justifications on why power laws arise were cleared during the rebuttal phase. 

As inference-time scaling is becoming more relevant to modern LLMs, this work is of high interest to the ICML audience. Given the high quality of the paper and overall interest in the work, I recommend acceptance.
Decision: Accept (oral)