Pretraining Scaling Laws for Generative Evaluations of Language Models
Authors: Rylan Schaeffer
Noam Itzhak Levi
Brando Miranda
Sanmi Koyejo
Venue: ICLR 2026 Poster (2026)
Keywords: language models
large language models
scaling laws
evaluations
generative evaluations
sampling
Primary area: foundation or frontier models, including LLMs
TL;DR: Scaling laws for generative evals of language models during pretraining

Abstract: Neural scaling laws have driven the field's ever-expanding exponential growth in parameters, data and compute. While scaling behaviors for pretraining losses and discriminative benchmarks are well established, generative benchmarks such as mathematical problem-solving or software engineering remain under-explored.
We propose and evaluate three different pretraining scaling laws for fitting pass-at-$k$ on generative evaluations and for predicting pass-at-$k$ of the most expensive model using cheaper models.
Our three scaling laws differ in the covariates used: (1) pretraining compute, (2) model parameters and pretraining tokens, (3) log likelihoods of gold reference solutions.
First, we demonstrate that generative evaluations introduce new hyperparameters (in our setting, $k$) that act as a control lever for scaling behavior, modulating both the scaling law parameters and the predictability of performance.
Second, we identify a stark difference in parameter stability: while the compute and parameters+tokens laws stabilize for only the last $\mathord{\sim}1.5\mathord{-}2.5$ orders of magnitude, the gold reference likelihood law is uniquely stable, converging across $\mathord{\sim}5$ orders. Third, in terms of predictive performance, we find all three scaling laws perform comparably, although the compute law predicts slightly worse for small $k$ and the gold reference law predicts slightly worse for large $k$. Finally, we establish a theoretical connection, proving that the compute scaling law emerges as the compute-optimal envelope of the parameters-and-tokens law. Our framework provides researchers and practitioners with insights and methodologies to forecast generative performance, accelerating progress toward models that can reason, solve, and create.

Published as a conference paper at ICLR 2026
PRETRAINING SCALING LAWS FOR GENERATIVE EVALUATIONS OF LANGUAGE MODELS
Rylan Schaeffer' Computer Science
Noam Levi' AI Center
Brando Miranda
Computer Science
Stanford University
EPFL.
Stanford University
Sanmi Koyejo
Computer Science
Stanford University
ABSTRACT
Neural scaling laws have driven the field's ever-expanding exponential growth in parameters, data and compute. While scaling behaviors for pretraining losses and discriminative benchmarks are well established, generative benchmarks such as mathematical problem-solving or software engineering remain under-explored. We propose and evaluate three different pretraining scaling laws for fitting pass- at-k on generative evaluations and for predicting pass-at-k: of the most expen- sive model using cheaper models. Our three scaling laws differ in the covariates used: (1) pretraining compute, (2) model parameters and pretraining tokens, (3) log likelihoods of gold reference solutions. First, demonstrate that generative evaluations introduce new hyperparameters (in our setting, k) that act as a control lever for scaling behavior, modulating both the scaling law parameters and the predictability of performance. Second, we identify a stark difference in parame- ter stability: while the compute and tarameters+tokens laws stabilize for only the last ~1.5-2.5 orders of magnitude, the gold reference likelihood law is uniquely stable, converging across ~5 orders. Third, in terms of predictive performance, we find all three scaling laws perform comparably, although the compute law pre- dicts slightly worse for small & and the gold reference law predicts slightly worse for large k. Finally, we establish theoretical connection, proving that the com- pute scaling law emerges as the compute-optimal envelope of the parameters-and tokens law, Our framework provides researchers and practitioners with insights and methodologics to forecast generative performance, accelerating progress to- ward models that can reason, solve, and create.
INTRODUCTION
Neural scaling laws, which predictably map resources to the performance of neural networks, are foundational for developing frontier Al systems (Kaplan et al. [220bb] Hoffmann et al. 2022b). Such empirical regularities for improving model performance from scaling parameters, data, and compute have driven pursuit of ever-larger models. Much of the research in neural scaling laws has focused on fitting and predicting model performance on pretraining losses, e.g., (Kaplan et al. [2020b] Hoffmann
al. 2022b; Clark et al. 2022, Hernandez al. 2022] Sardanaa al.) [024] Muennighofi et al.)
2023
Porian et al. 20244 Pearce & Song] 2024) and on "downstream" discriminative evaluations
like multiple-choice question-
question-answering answering. e.g., (Schaeffer et al. [023 Gadre et al. 2024: Schaeffer
et al.
2024b Grattafiori et al.) 2024 Chen et al. 2025 Bhagia et al. 2025).
While pretraining losses and discriminative evaluations are undeniably useful, many capabilities we care about are generative: writing books, autoformalizing mathematics, or conducting cutting- edge research. Scaling laws for generative tasks remain undercharacterized (Appendix (A) (but see
Denotes equal authorship. Contact:
sta.efeccssaafod.eeeee noam. 1evi @epf1 ch,
sanmi @cs stanford..ddd

---

Published as a conference paper at ICLR 2026
Sum Altomapi Prolicss &
Num Altezipia Prokken A 130
Num Altemppi Prolien & 10200
31 C-13lees
1082
10
Pr Wodmaalliu Chasriie
13-1
1y
4x 107
10"t
10
109
10P2
1011
11PT
10
Pestralting Computie
Prstecinig Couputte
Protrining Computt
Figure 1: Scaling of GSM8K Pass Rates with Pretraining Compute (Full Fit), Each panel fits
Eqn. 3 log log(passy (anessgk)aaa (C,k) k) = (k) + Co(k) C-a(k) to GSM8K pass rates for Pythia check- points across ~5 orders of magnitude of pretraining compute. Increasing k drives two major shifts: (i) The irreducible error (k) vanishes (from R 2 ++ 0), removing the performance plateau, and (ii) the power law steepens, with the exponent a (k)) rising from = 00.121 0.375. Takeaway: Larger climinate irreducible error and yield steeper pass-at-k rates with respect to pretraining compute.
OpenAl et al. 2024a; Hu al. 2024) ); for example, Gadre et al. 2024) studied 46 tasks, none of
which were generative. Generative evaluations differ from discriminative evaluations in many ways, most crucially that performance is calculated from the model's open-ended samples, which intro- duces new considerations such as how attempts are sampled from the model, including the sampling temperature and the sampling algorithm, and which metric is used measure performance.
In this work, we propose a framework for fitting scaling laws and predicting performance on gener- ative evaluations, focusing on particular setting: benchmarks with verifiable binary rewards, with multiple attempts per problem, with performance scored using the "pass-at-k" metric (Kulal et al.)
2019
Chen al. 2021p. We chose this setting due its widespread use in the literature, e.g., (First 2023
et al.
Brown et al. 22024: Hassid et al. [024] |Hughes et al.) [024] Chen et al. 2024 Ehrlich
et al.
2025
Kwok et al. 2025). this setting, we make the following contributions:
1. We propose three scaling laws that fit and predict pass-at-k k from three different covariates:
(1) pretraining compute (Sec. 3. (2) model parameters and pretraining tokens (Sec. 4 and
(3) likelihoods of gold reference solutions (Sec.5) 5)
2. We reveal that a hyperparameter specific to this setting the number of attempts per prob- lem - offers& a new lever to shape the scaling and predictability of model performance.
3. We quantify the stability of each scaling law's parameters. We find that parameters for the compute scaling law and the parameters tokens scaling law stabilize by the last 11.5~ orders of magnitude of pretraining compute, whereas parameters for the gold reference likelihood scaling law are stable over the last ~-5 orders of magnitude.
4. We quantify how predictive each scaling law is, finding that all three comparubly predict the performance of the most expensive model based on cheaper models. As lesser differ- ence, we find that the compute scaling law has slightly higher predictive eтror for small k whereas the gold reference likelihoods law has slightly higher predictive error for large k.
5. We prove that the compute-only scaling law is the compute-optimal envelope of the scaling law, obtained by minimizing the objective under a fixed compute budget. From this, derive dimensionless misallocation penalty that explains quantitatively when and why pretraining recipes underperform compute optimaal scaling.
parameters-and-toke
2 METHODOLOOGY
Language Model Family: For our experiments, we used the Pythia family (Biderman et al. 2023 of 8 models from 14M to 12B parameters pretrained on up to 300B tokens from The Pile (Gao et al. 2020). We restricted our analysis to Pythia family because it is, our knowledge, the only public model family meeting three criteria: (1) densely sampled across both model parameters (N) and pretraining tokens (D); (2) spanning many orders of magnitude in pretraining compute (5); and (3)
2

---

Published as a conference paper at ICLR 2026
10:0
107
Itrecuctall Error Eack)
Compine Prediction Ca(A)
Comgrete Expomert ((f))
109
5.0
109
10-1
2.3
10
0.0
109.
10
10
10
10
10a
10
10"
10
Num. Attempts/
Problem l
Nam. Attempts
Problem &
Num. Attempts Proisem
Figure 2: Number of Attempts per Problem lc Shapes Scaling Law Parameters (Full Fit). We fit Eqn. to each i (hue); see Appendix [ for the next two scaling laws. Left: Irreducible error (o(k) decays roughly exponentially with & and is a by k z 10 Center: Compute prefactor (okk) increases monotonically with k, indicating that once Eo(k) + O, the compute-dependent term dominates. Right: Compute exponent a<(k) increases smoothly, from 0.15 at k= to 0.3
at k=100, indicating that larger sampling budgets yield steeper, more favorable scaling behaviors.
known token budgets per checkpoint. Other families (e.g.. Llama 3, Qwen 3) either lack sufficient intermediate checkpoints to robustly 5-parameter scaling law or do not disclose specific per- model pretraining token budgets. We used 8 checkpoints per parameter size of the non-deduplicated
variants, but excluded extremely overtrained checkpoints identified as abnormal (Godey et al.
2024).
We approximated pretraining compute C' 6 D (Kaplan et al. [2020b] Hoffmann et al. 2022b
Pearce & Song, 2024; Gadre et al. 2024) Schaeffer et al.) 2024b Porian et al. 2024) and used intermediate checkpoints in lieu of fully trained models.
Benchmarks: We used two well-established generative tasks: GSM8K (Cobee et al. 202 I) with
8, 500 grade school math word problems, and MATH (Hendrycks et al. 2021 with 12, 500 compe- tition mathematics problems. Each attempt solving problem yields a binary outcome: success or failure. Because generative evaluations are computationally expensive, to enable a rigorous analysis across orders of magnitude of compute with high sample counts, on a randomly selected subset of 128 problems from GSM8K and 128 problems from MATH; see Appendix B for the number of samples per problem per model, and Appendix C for the costs.
evaluated all checkpoints
Metric: We measured the performance of each model using the pass-at-k metric (Kulal et al. 2019), defined as the probability that if model makes & attempts the ith problem, at least one attempt successful. We estimate this per-problem pass rate using the unbiased, low variance and numerically stable estimator of Chen et al. (2021), , which, for each problem, draws n, > ie samples, counts the number of successes 8; E [0, n.). and then averages over all subsets of size k:
pass; @k
def
For a benchmark B of [B] problems, a model's benchmark pass rate at ls attempts is estimated as:
def
passg@k
pass @k
B
Pass rates are sometimes phrased as the fraction of problems in the benchmark solved ("coverage") which is mathematically equivalent (Brown et al., 2024 Schaeffer et al.] 2025b). Importantly, pass- at-k is fundamentally different from (discriminative) accuracy, as ptss-at-k a continuous proba- bility derived from the model's generative distribution whereas accuracy is discriminative function of the correct and incorrect multiple choices (Appendix A).
Generative evaluation introduces vast hyperparameter space (e.g., temperature (Ack- ley et al.) 1985). top-p (Holtzman et al., 2020), top-k (Fan et al. 20088). We deliberately isolated the number of attempts per problem (k) our primary variable because is simple yet effective lever for scaling inference-ting time compute (OpenAl et al. 2024b; Snell et al. 2024 Brown et al. 2024] DeepMind et al. 2025 DeepSeek-Al et al. 2025a) and because previous work has studied the effect of sampling algorithm and sampling temperature on GSM8K (Chen et al. 202T Schaeffer et al, 2025a). We used temperature-only sampling 1.0, holding the decoding algorithm fixed.
Generation:
3

---

Published as a conference paper at ICLR 2026
10a
10
Target Model's (pg(pps
10'
Irreducible Ervor Ex
Relative Eror of
10
100
10-1
1000
10-2
10000
2
10-3
10-4
1012
10
Compute Prefactor Calk)
1040
Compute Exponent ((k)
10a
109
106
10a
10-1
10a
10"
10-2
10-4
10-3
10-2
10 Target C
100
10-4
10
10-2
10-1
109
Max C for Predicting
Max C for Predicting Targer C
Figure 3: Predicting GSM8K Pass Rates from Scaling Pretraining Compute (Backtesting). We evaluate predictability via backtesting: iteratively fitting Eq. 3 on subsets models (C < Cmax)
to predict the most expensive model (Pythia 12B- 12B-parameter #00B-token: ss 2.16 x 10 FLOP)'s
-og (og( The æ-axis denotes the compute horizon relative to the target (Cmax/ (targett) Top Left: Relative error decreases and then plateaus: reliable prediction requires checkpoints within ~2 orders of magnitude of the target for k E {1, 10²} and 1=.5 for k = 10a. Other Three Panels: Backtested estimates of the scaling law parameters are initially unstable but converge their full-fit values once fits include models within ~2 orders of magnitude of the target.
3 FITTING AND PREDICTING PASS RATES FROM PRETRAINING COMPUTE
For our first scaling law, we assessed how well the benchmark passy ok can be (1) fit as a function of pretraining compute C' and (2) predicted from pretraining compute.
3.1 FITTING PASS RATES FROM PRETRAINING COMPUTE
Motivated by the GPT-4 Technical Report (OpenAl al. 2024a), we posited that the negative log of benchmark pass rates follows a scaling law as function of model pretraining compute C:
Co(k) - log [passs@k) (C, = (k()) + Ca(A)
(3)
where (k) 2 0 is the irreducible error, Co(k) > 0 is the compute scaling prefactor, and a<k) > 0 the compute scaling exponent. In contrast with prior research on scaling laws, parameters that were previously constant are now parameterized by the number of attempts per problem k. Thus, k can be viewed as a hyperparameter that offers new lever change the predictability of scaling laws in way that is not available for pretraining loss or discriminative evaluation scaling laws.
We fit the scaling law parameters based on the model checkpoints' pretraining compute and their corresponding passg Visually, the fit scaling laws captured the trend reasonably well for differ- ent values of number of attempts per problem & (Fig. []. We next evaluated what role the number of attempts per problem A has on the scaling law parameters: As & increases, the irreducible error term (k) falls roughly exponentially with and effectively by A = 1 x 102 (Fig. 2 Left). This is consistent with the probability problem goes unsolved falling exponentially with the number of attempts per problem (Lvii, 2025 Schaefffr et al.] 20256). Thus, by increasing k Eqn. [3 becomes a pure power law with irreducible error. In comparison, increases, the compute prefactor and
4

---

Published as a conference paper at ICLR 2026
Sum. Altomapis Prolicss &
Namt Alterupla Prokden 100
Num Altecupii Prokless A 10200
Mitro/V-411--- --11--1 5.5e22 Clie
Bo0te-1 J.tee.N 20h--1 2000577-120
10R2
Py Medmaaliue Clasruie
6xx 10
10"
100
1011
101
10:1
10p0
101
Pestratzing Computte
Prstecining
Protriniig Compptt
Figure 4: Scaling of GSMSK Pass Rates with Parameters and Tokens (Full Fit). Each panel fits Eqn. log [passg Oik (N, k) = &o(k) + (k) N + Do(k) D->(k), Decomposing compute into parameters N and tokens D instead yields tighter in-range fits than the compute law for all k. Consistent with Fig. 2 the irreducible error decreases sharply with (E) 3.87 + by 300),, after which variation is dominated by (N, D) terms. However, despite the better global fit, the largest-compute model checkpoint in each panel exhibits comparatively large relative error.
compute exponent also increase, with the compute prefactor rising 44 orders of magnitude (Fig. Center) and the compute exponent rising moderately from 1.21 3.75 10 (Fig. 2 Right).
3.2 PREDICTING PASS RATES FROM PRETRAINING COMPUTE
Previously, we fit Eqn. 3 using all available checkpoints. However, a key motivation for scaling laws is prediction: can we forecast the performance an expensive model using cheaper models?
We evaluated the predictability of pass rates via backtesting, cross coooss-valiattioo-oo like process for eval- uating forecasts in which cheaper models are used to predict the performance of the most expensive model. We fixed target model checkpoint pretrained
Ctarget FLOP; in work, used Pythia-12B trained for 300B tokens, pretrained with approximately 2.16 x 10 FLOP. For quence of compute caps Cmax S Ctarget. we: (i) fit Eq. 3 on all model checkpoints with compute C < Cmax obtain (k). (k)), and @(k); (ii) extrapolate to the target compute horizon, and (iii) measure the absolute relative еπтог to the target model's log (pass @kicarget):
del' abs| log (pass @ (target) (o( (k( Oranget) -((k)) I
RelativeError(k: Cmaax
- -log pass @ Niarggot)
We report results as function of the compute ratio Cmaxx/ /Cargett.
Relative errors typically decrease and then plateau as higher-compute checkpoints are included in the fit (Fig top left). For {1, le2), reliable extrapolation requires that the largest checkpoint used for fitting is within ~22 orders of magnitude of the target's compute, and the relative errors plateau at 1 x 10-1 In comparison, for = 1e4, reliable extrapolation requires ~155 order of magnitude, and the relative emrors plateau below 1. Similarly, the backtested estimates of the param- eters (k), (k) and ā(k) converge their full-fit values once models within ~1.5- orders of magnitude of Crarget are included (Fig. 3 other three panels). For these models on this bench- mark, the compute scaling law provides useful forecasts long forecasts are made using models within ~2 orders of magnitude of compute as the target model; when only smaller checkpoints are available, the predictive error increases and the fitted parameters are unreliable.
FITTING AND PREDICTING PASS RATES FROM PARAMETERS AND TOKENS
In our second approach, we assessed how well the benchmark passg @k can be (1) fit and (2) pre- dicted function of the number of model parameters N and pretraining tokens D.
5

---

Published as a conference paper at ICLR 2026
Target Model's - logipase Blk)
10a
10
10'
Itrecheibble Error Eg(R)
Relative Error
10"
10-1
10-2
10-4
109
10.
Purumeter Prefector No(k)
10
Paruneter Exponent 3(k)
107
10
10a
10a
10
109
10
108
101
Data Prefacton Dotk)
Dota Exponent ((8)
10"
1
10a
10
104
1007
1000
10000
10a
10-4
10
10-2
10
10P
10-4
10-3
10-2
10
10a
Max C. for Prodicting / Target C
Max C. for Prodicting Target
Figure 5: Predicting GSMSK Pass Rates from Scaling Parameters and Tokens (Backtesting). We evaluate how accurately the parameters + tokens scaling law (Eqn.E) predicts the most expensive model's - log (pgssas @k). Top Left: The relative error decreases as higher-compute checkpoints are included, plateauing once the fit includes models within ~22 orders of magnitude of the target. Other Five Panels: Estimates of the five scaling law parameters are initially unstable but converge to their full-fit values once included models are within ~2 orders of magnitude of the target.
4.1 FITTING PASS RATES FROM MODEL PARAMETERS AND PRETRAINING TOKENS
Motivated by Hoffmann et al. (202b)), we posited that the negative log of benchmark pass rates might instead follow scaling law as function of model parameters N and pretraining tokens D:
(o) NB
Do(k) D
log (pass @k) (N, D, = Eo(k)
(4)
where (ok is the irreducible error, (okk) is the parameter scaling prefactor, B(k) is the parameter scaling coefficient, (k) is the token scaling prefactor and (kk) is the token scaling coefficient.
We fit this parameters + tokens scaling law. Visually, this law provides better characterization of performance for the full range of models, with the fitted curves appearing tighter and reducing the residual scatter (Fig compared against the pretraining compute law (Fig[). Consistent with findings for the compute-only law, the irreducible error term (K( decreases with the number of attempts per problem k although it does so more gradually, reaching zero by k (Fig. 12). However, despite the improved global fit, we observed that the largest-compute model
102

---

Published as a conference paper at ICLR 2026
Sum Altomppi per Probiess & m1 1.te=11 1- GoolPloll)) 1h
Nom Alterapla per Prokler A 120 2.01e-0 200c-22 2 GolCCli
Num. Altemupli per Prokiess A 10000
0.00e-10 1og
1082
101
Pr mumiiii Canmrrae
tof
10
1y
10P
Figure 6: Scaling of GSM8K Pass Rates with Gold Reference Likelihood (Full Fit). Each panel fits Eqn. - lggspas @k) ) = fo Ko+ log (GoldProbg)]" ]=((), Regressing the gold reference log-likes log-likelfhooos against the pass rates produces visually tighter fit with less residual scatter than the compute scaling law (cf. Fig. T). As k increases, the irreducible error (k) falls toward zero and the exponent xc(k) increases, making the relationship a steeper, purer power law.
checkpoints exhibit comparatively large relative error, suggesting that while this law better explains the performance of cheaper models, may struggle with extrapolation to the frontier.
4.2 PREDICTING PASS RATES FROM MODEL PARAMETERS AND PRETRAINING TOKENS
We evaluated how predictive the parameters + tokens scaling law is using the same backtesting ap- proach (Section 3.2) We again used Pythia 12B model trained for 300B tokens as the target, and iteratively fit Eqn. The relative error in predicting the target model's performance decreases and then plateaus as more capable models are included in the fit (Fig. S Top Left). For all values of k, these extrapolations become reliable once the largest checkpoint used for fitting is within ap- proximately ~2-2.5 2.5 orders of magnitude of the target's compute. Similarly, the five backtested scaling parameters converge their full-fit values once models within this
compute range of ~15 2.5 orders of magnitude are included (Fig. 5 Other Five Panels). While decomposing pre- training compute into parameters and tokens provides tighter in-range fit, doing so does not yield a significant improvement in predicting the performance of the most expensive model. For small k the parameters+tokens scaling law has slightly lower relative error at predicting the most expensive model's 's performance, but the advantage disappears for larger l (Fig. 5 UUpee Left).
5 FITTING AND PREDICTING PASS RATES FROM GOLD REFERENCE LIKELIHOODS
Generative evaluations oftentimes contain not just problems and correct answers, but also "gold ref- erences", high quality responses that reach the correct answer for the given problem. We calculated the average log- likelihood of these gold reference sequences to use predict pass rates:
GoldProbg
def
Po (Gold Reference, PProblem.).
(5)
B
IEB
For our gold reference likelihood scaling laws we assessed how well the benchmark passg @k can be (1) fit and predicted function of the log likelihoods of the gold references.
5.1 FITTING PASS RATES FROM GOLD REFERENCE LIKELTHOODS
Motivated by [chaetfer et al. 2024b), we tested whether this average likelihood of the gold re- sponses can be accurately regressed fit and predict benchmark pass rates passg
log @kk (GoldProbg, k) = to (k) + Ko(k) log (GoldProbg) k(k)
(6)
We find that this straightforwardly computable quantity serves as an exceptionally accurate predictor of benchmark pass rates. As shown in Figure 6. regressing against the negative log likelihood of

---

Published as a conference paper at ICLR 2026
10a
10
Target Model's (og
10'
Ineeucibll Ervor Sock)
Relative Eroo of
10
100
10-1
1000
10-2
10000
10-3
10-4
10
10a
GoldProbg Preficter Ka(k)
GoldProbs Exponent x(k)
10a
104
10a
10a
10a
10-2
10-1
10-1
10-6
10-2
10-4
10-3
10-2
10 Target C
10P
10-4
10
10-2
10-1
10
Max C for Predicting
Max C for Predicting Target C
Figure 7: Predicting GSM8K Pass Rates from Gold Reference Likelihoods (Backtesting). Top Left: While the overall predictive error is comparable to the other laws, it is slightly lower for small k and slightly higher for large k. Other Three Panels: The most striking result is the stability of the scaling law's parameters. In stark contrast to the compute and parameters+tokens laws, whose parameters stabilized only when fit models within ~2 orders of magnitude of the target, the backtested estimates for (A)). (k)), and R(k) converge to their final values using models up to ~5 orders of magnitude cheaper than the target. This remarkable stability suggests that the relationship between gold reference likelihoods and pass rates provides robust signal for long-range forecasts.
the gold references produces visually tighter fit than the pretraining compute law (Fig. D. The residual scatter is reduced, and the relationship between - (og(pass::@k and log (oodsrobb)) has smaller reducible error and more closely resembles a power law. We next examined how the number of attempts per problem & influences the scaling law's s parameters (Fig. 13). Consistent with our findings for the prior two scaling laws, the irreducible error term (k( collapses toward zero as k increases and the exponent k(k) increases smoothly.
5.2 PREDICTING PASS RATES FROM GOLD REFERENCE LIKELIHOODS
Using our backtesting approach once more, we evaluated how well the gold references scaling law can predict the performance of our most expensive model. For small k, the gold reference likeli- hoods scaling law has slightly lower relative error at predicting the most expensive model's 's perfor- mance, but worsens for larger (Fig. 7 Upper Left). Remarkably, the gold reference law exhibits exceptional parameter stability. Unlike compute parameter token scaling laws, the backtested estimates for the gold reference scaling parameters stabilize when fitting on models up to ~55 of magnitude cheaper than the target (Fig. 7 Other Three Panels) and for all values of k, suggesting this relationship is robust signal for long-range forecasting.
How THE COMPUTE LAW EMERGES FROM THE PARAMETERS + TOKENS LAW, WITH IMPLICATIONS FOR OVERTRAINING
The approximately equivalent performance between the compute scaling law (Eqn. 3 and the pa- rameters + tokens scaling law (Fqn. led ask whether deeper connection might exist. We
8

---

Published as a conference paper at ICLR 2026
discovered that the compute scaling law is the compute-optingg envelope of the parameters+tooens scaling law. We summarize the insight here; see Appendix E for a deeper discussion.
Fix benchmark B and attempts per problem k. Consider compute budget C' cND. The best achievable error is obtained by minimizing the right-hand side of Eqn. Jover all (N, D) with ND C/c. Evaluating the minimizer yields Eqn. The mapping of exponents and constants is:
a(k)
Eo(k) Eo(k).
((R)
Compute-Optimal Allocation: Along the envelope (i.e., when Eqn. 3 tight), the optimal split of compute between parameters and tokens is
N (C, k) = Dokk)
D* k)
Dolk) No(k)
347
Thus the compute-optimal tokens and parameter shift with scale according to B(k), (k); when B(k) (A:), both grow like C1/2
Departures from the Envelope: Deviating from the compute-optimal allocation (N*, D*) intro- duces multiplicatiie penalty (r) 2 where r(C)- -00
is the misallocation ratio: N-Cey
N
-log (psss @k)(C, (k) + Co(k) CC-a(k) +
و B+9
(r: ((k), ((k), 2 1
with equality i = 1. Intuitively, (r) represents a "tax" on performance caused by inefficient compute allocation: small deviations are benign (Φ = + 81 (logr) + O(logr )) but persistent off-ridge
10"
oovetranood
undertraane
Eleetive Compate Mulcaplier
10-1
2
sealing significantly reduces "effective" pretraining compute:
10-2
Effective Pretraining Compute
C(r -1r=1 1) C(r 1) (7)
10-3
Pretraining Compute
100
10000
(Bk)),())))))) ((k))(((()))))))))) (//(k)
10
10:
10
10"
10a
100
useful case: one scales N and D at a fixed ra- tio (i.e., without re-optimizing with C). the effective
Misallocation Ratio r(C) Relative to Compete-Optimal
compute slope degrades to O'path min(8.2)
Figure 8: Quantifying the Cost of Non- Optimal Scaling. We plot the "effective" compute multiplier (derived in Eq, 7 against the misallocation ratio rC). The peak at r = corresponds to the compute-optimal allo- cation; deviating from this optimum either by overtraining (r 1, left) undertrain- ing (r > 1, right) rapidly reduces the effec- tive compute by orders of magnitude. Bench- mark: GSMSK.
Takeaways: (i) The compute law is not an inde- pendent phenomenology: is the optimal-allocation shadow of the parameters tokens law. (ii) Use the above allocation to stay on-envelope to maximize compute efficiency. (iii) Implementation constants relating C' to ND shift (k),), but do not change a (k). All statements hold pointwise in k.
Relationship to Overtraining: Our misallocation ratio r(C) is the inverse of the overtraining multi- plier M studied by Gadre et al. (2024). Our results extend their overtraining scaling law to generative evaluations and the downstream metric passg Ok for k: 2
How SCALING LAWS DEPEND ON THE BENCHMARK: MATH vs GSM8K
We repeated the same scaling analyses for MATH (Hendrycks al. 2021 as for GSM8K. Due to space constraints, we defer the detailed results to Appendix F and focus here on the salient similari- ties and differences between the benchmarks.
First, irreducible error reflects benchmark hardness. For GSM8K, the irreducible error E, (k) vanishes quickly, approaching zero by l 100. In contrast, MATH retains significant irreducible

---

Published as a conference paper at ICLR 2026
error even at high sample counts: (k) xe 0.45 at l 10* (Figure 14). This quantifies the intuition that MATH is a significantly harder benchmark.
Second, harder benchmarks yield steeper power laws at high sample counts. While both bench- marks show that increasing &o steepens the scaling curve, MATH exhibits a more dramatic shift. The compute exponent (<(a) for GSM8K rises = 0.38 at k 10 whereas for MATH it rises to x 0.58. Because GSM8K performance saturates near 100% coverage, the returns on compute di- minish. Conversely, because MATH performance remains far from perfect, the "headroom" allows for steeper scaling returns as compute increases.
Third, scaling exponents for MATH are sometimes more stable and sometimes less stable when backtesting. The compute-only scaling law's a(k) and the parameters+ooees''s ((k) stabilize across 3 orders of magnitude of compute. but the gold reference likelihood's 's k(k) stabilizes across 2 order of magnitude and the parameters+tokens s ((k) stabilizes across 1 order of magnitude. We do not know what exactly determines this the hardness of the task, the distribution shift between training data and the task, or some other factor.
DISCUSSION
Key Findings: We investigated pretraining scaling laws for generative evaluations of language mod- els. Our work was motivated by OpenAl et al. (2024a)ss 's success, as well as by the insights of Hu jet al.) (2024). Our primary contribution is the proposal and rigorous backtesting of three distinct scaling laws for pass-at-k, utilizing (1) pretraining compute, (2) model parameters and pretraining tokens, and (3) gold reference log likelihoods as covariates. We demonstrate that the number of attempts per problem, k. acts critical hyperparameter in generative evaluation, offering novel lever to control both the parameters of the scaling laws and their predictive reliability. Empirically, while all three laws exhibit comparable predictive accuracy, the gold reference likelihood law is uniquely stable, with parameters converging across nearly five orders of magnitude of compute. Fi- nally, we establish theoretical connection between the compute law and the parameters- parameters-and-toleee and-tokens law, proving that the former emerges as the compute-optmal envelope of the latter. This derivation yields a dimensionless misallocation penalty, quantifying the efficiency loss when training deviates from the optimal parameter-token ratio.
Limitations: The primary limitation of this work is its empirical focus on single model fam- ily (Pythia). As detailed in Section 2 this scope was necessary constraint to enable a rigorous, high-resolution analysis (fitting 5-parameter laws requires dense checkpoints) while managing the substantial computational costs of generative sampling. Additionally, we focus exclusively on pass- at-k metrics; other generative evaluation metrics and tasks warrant similar investigation.
Related Work: We defer detailed discussion of related work to Appendix A
Future Directions: Several avenues for future research are clear. First, subsequent work should de- termine whether predictive performance can be further improved using controlled pretrained models or more extensive sampling budgets. Second, OpenAl et al. (2024a) suggest that subsetting and/or stratifying problems enables better predictions; future work should isolate exactly which techniques yield the highest signal-too signal-to-noise. Third, the strong predictive power of gold reference log-likelihooke warrants theoretical investigation: given that next-token sampling is a branching process with likely exponentially many valid solution paths, it is not immediately obvious why the likelihood of the spe- cific sanchharr--rovidder gold reference correlates so strongly with the pass rate. Two key questions arise: (i) how generalizable is this relationship across diverse benchmarks? and (ii) to what extent does this signal remain robust under heavy optimization pressure? Fourth, while held decoding strategies constant to isolate the effect of k. understanding how different sampling algorithms (c.g., top-p) and temperature interact with scaling laws remains an open and valuable question.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
RS acknowledges support from Stanford Data Science and from the OpenAl Superalignment Fast Grant. NL is supported by the EPFL AlAscience/AI Center program. BM acknowledge support by Schmidt Sciences, Stanford EDGE Scholar Fellowship. SK acknowledges support by NSF 2046795 and 2205329, the MacArthur Foundation, Stanford HAL, OpenAl and Google Inc.

---

Published as a conference paper at ICLR 2026
REFERENCES
David H Ackley, Geoffrey E. Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 9114:16-6,, 1985.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27))e2311888) 121, 2024.
Akshita Bhagia, Jiacheng Liu, Alexander Wettig, David Heineman, Oyvind Tafjord. Ananya Harsh Jha, Luca Soldaini, Noah A. Smith, Dirk Groeneveld, Pang Wei Koh, Jesse Dodge, and Han- naneh Hajishirzi. Establishing task scaling laws via compute-efficient model ladders. In Second Conference on Language Modeling, 2025. URL https::/ hwtp:/wwwwwwweeeeeeeeeeeeeeeee herumoommm? did= FeAM2RVO8
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le. Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, 2024. URLpttts:://arxttt
/ abs /2407 21787
Lingjiao Chen, Jared Quiney Davis, Boris Hanin, Peter Bailis, lon Stoica, Matei Zaharia, and James Zou. Are more Ilm calls all you need? towards scaling laws of compound inference systems, 2024. URL wrw.aaivv....... org/ abs/ / 2403. 02419 99
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021. URL URLhttpps: : /ttp://aaxivvvv.
http:/ttss//ari..ooooo www.aawwwv.....gaaaaaaaa abs / 03374
Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang, Jingfeng Yang, and Heng Ji. Scal- ing laws for predicting downstream performance in LLMs. Transactions on Machine Learn ing Research, 2025. ISSN 2835-8856. URL https: http:://pppeeeeeeeeeeeee net forum?id=
PJUbMDKQVY
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoff- mann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et Unified scaling laws for routed language models. In International conference on machine learning, pp. 4057- 4086. PMLR, 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10. /4168, 2021.
Google DeepMind, Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobs- son, Idan Szpektor, Nan-Jiang Jiang, Krishna Haridasan, Ahmed Omran, Nikunj Saunshi, Dara Bahri, Gaurav Mishra, Eric Chu, Toby Boyd, Brad Hekman, Aaron Parisi, Chaoyi Zhang, Korn- raphop Kawintiranon, Tania Bedrax-Weiss Oliver Wang, Ya Xu, Ollie Purkiss, Uri Mendlovic, Ili Deutel, Nam Nguyen, Adam Langley, Flip Korn, Lucia Rossazza, Alexandre Ramé, Sagar Waghmare, Helen Miller, Nathan Byrd, Ashrith Sheshan, Raia Hadsell Sangnie Bhardwaj. Pawel Janus, Tero Rissa, Dan Horgan, Sharon Silver, Ayzaan Wahid, Sergey Brin, Yves Raimond, Kle- men Kloboves, Cindy Wang, Nitesh Bharadwaj Gundavarapu, Ili Shumailoy, Bo Wang, Mantas
12

---

Published as a conference paper at ICLR 2026
Pajarskas, Joe Heyward, Martin Nikoltchev, Maciej Kula, Hao Zhou, Zachary Garrett, Sushant Kafle, Sercan Arik, Ankita Goel, Mingyao Yang, Jiho Park, Koji Kojima, Parsa Mahmoudieh, Koray Kavukcuoglu, Grace Chen, Doug Fritz, Anton Bulyenov, Sudeshna Roy, Dimitris Paparas, Hadar Shemtov, Bo-Juen Chen, Robin Strudel, David Reitter, Aurko Roy, Andrey Vlasov, Chang- wan Ryu, Chas Leichner, Haichuan Yang. Zelda Mariet, Denis Vnukov, Tim Sohn, Amy Stuart, Wei Liang, Minmin Chen, Praynaa Rawlani, Christy Koh, JD Co-Reyes, Guangda Lai, Praseem Banzal, Dimitrios Vytiniotis, Jieru Mei, Mu Cai, Mohammed Badawi, Corey Fry, Ale Hartman, Daniel Zheng, Eric Jia, James Keeling, Annie Louis, Ying Chen, Efren Robles, Wei-Chih Hung, Howard Zhou, Nikita Saxena, Sonam Goenka, Olivia Ma. Zach Fisher, Mor Hazan Taege, Emily Graves, David Steiner, Yujia Li, Sarah Nguyen, Rahul Sukthankar, Joe Stanton, Ali Eslami, Glo- ria Shen, Berkin Akin, Alexey Guseynov, Yiqian Zhou, Jean-Baptiste Alayrae, Armand Joulin, Efrat Farkash, Ashish Thapliyal, Stephen Roller, Noam Shazeer, Todor Davchev, Terry Koo, Han- nah Forbes-Pollard, Kartik Audhkhasi, Greg Farquhar, Adi Mayrav Gilady, Maggic Song, John Aslanides, Piermaria Mendolicchio, Alicia Parrish, John Blitzer, Pramod Gupta, Xiaoen Ju, Xi- aochen Yang. Puranjay Datta, Andrea Tacchetti, Sanket Vaibhav Mehta, Gregory Dibb, Shubham Gupta, Federico Piccinini, Raia Hadsell, Sujee Rajayogam, Jiepu Jiang, Patrick Griffin, Patrik Sundberg, Jamie Hayes, Alexey Frolov, Tian Xie, Adam Zhang, Kingshuk Dasgupta, Uday Kalra, Lior Shani, Klaus Macherey, Kuo Huang, Liam MacDermed, Karthik Duddu, Paulo Zac- chello. Zi Yang, Jessica Lo, Kai Hui, Matej Kastelic. Derek Gasaway, Qijun Tan, Summer Yue, Pablo Barrio, John Wieting, Weel Yang. Andrew Nystrom, Solomon Demmessie, Anselm Lev- skaya, Fabio Viola, Chetan Tekur, Greg Billock, George Necula, Mandar Joshi. Rylan Schaeffer, Swachhand Lokhande, Christina Sorokin, Pradeep Shenoy, Mia Chen, Mark Collier, Hongji Li, Taylor Bos, Nevan Wichers, Sun Jae Lee, Angéline Pouget, Santhosh Thangaraj. Kyriakos Axi- otis, Phil Crone, Rachel Sterneck, Nikolai Chinaev, Victoria Krakovna, Oleksandr Ferludin, Ian Gemp. Stephanie Winkler, Dan Goldberg, Ivan Korotkov, Kefan Xiao, Malika Mehrotra, Sandeep Mariserla, Vihari Piratla, Terry Thurk, Khiem Pham, Hongxu Ma, Alexandre Senges, Ravi Ku- mar, Clemens Meyer, Ellie Talius, Nuo Wang Pierse, Ballie Sandhu, Horia Toma, Kuo Lin, Swa- roop Nath, Tom Stone, Dorsa Sadigh, Nikita Gupta, Arthur Guez, Avi Singh, Matt Thomas, Tom Duerig, Yuan Gong. Richard Tanburn, Lydia Lihui Zhang, Phuong Dao, Mohamed Hammad, Sirui Xie, Shruti Rijhwani, Ben Murdoch, Duhyeon Kim, Will Thompson, Heng-Tze Cheng, Daniel Sohn, Pablo Sprechmann, Qiantong Xu, Srinivas Tadepalli, Peter Young, Ye Zhang, Hansa Srinivasan, Miranda Aperghis, Aditya Ayyar, Hen Fitoussi, Ryan Burnell, David Madras, Mike Dusenberry, Xi Xiong, Tayo Oguntebi, Ben Albrecht, Jörg Bornschein, Jovana Mitrović, Ma- son Dimarco, Bhargav Kanagal Shamanna, Premal Shah, Eren Sezener, Shyam Upadhyay. Dave Lacey, Craig Schiff, Sebastien Baur, Sanjay Ganapathy, Eva Schnider, Mateo Wirth, Connor Schenck, Andrey Simanovsky, Yi-Xuan Tan, Philipp Fränken, Dennis Duan, Bharath Mankalale, Nikhil Dhawan, Kevin Sequeira, Zichuan Wei, Shivanker Goel, Caglar Unlu, Yukun Zhu, Haitian Sun, Ananth Balashankar, Kurt Shuster, Megh Umekar, Mahmoud Alnahlawi, Aäron van den Oord, Kelly Chen, Yuexiang Zhai, Zihang Dai, Kuang-Huct Lee, Eric Doi, Lukas Zilka, Rohith Vallu, Disha Shrivastava, Jason Lee, Hisham Husain, Honglei Zhuang, Vincent Cohen-Addad Jarred Barber, James Atwood, Adam Sadovsky, Quentin Wellens, Steven Hand, Arunkumar Ra- jendran, Aybuke Turker, CJ Carey, Yuanzhong Xu, Hagen Soltau, Zefei Li, Xinying Song, Con- glong Li, lurii Kemaev, Sasha Brown, Andrea Burns, Viorica Patraucean, Piotr Stanczyk, Renga Aravamudhan, Mathieu Blondel, Hila Noga, Lorenzo Blanco, Will Song. Michael Isard, Mandar Sharma. Reid Hayes, Dalia EI Badawy, Avery Lamp, Itay Laish, Olga Kozlova, Kelvin Chan, Sahil Singla, Srinivas Sunkara, Mayank Upadhyay, Chang Liu. Aijun Bai, Jarek Wilkiewiez, Martin Zlocha, Jeremiah Liu, Zhuowan Li, Haiguang Li, Omer Barak, Ganna Raboshchuk, Jiho Choi, Fangyu Liu, Erik Jue, Mohit Sharma, Andreea Marzoca, Robert Busa-Fekete, Anna Ko- rsun, Andre Elissefff, Zhe Shen, Sara Mc Carthy, Kay Lamerigts, Anahita Hosseini. Hanzhao Lin, Charlie Chen, Fan Yang. Kushal Chauhan, Mark Omernick, Dawei Jia, Karina Zainul- lina, Demis Hassabis, Danny Vainstein, Ehsan Amid, Xiang Zhou, Ronny Votel, Eszter Vértes, Xinjian Li, Zongwei Zhou, Angeliki Lazaridou, Brendan McMaaan, Arjun Narayanan, Hubert Soyer, Sujoy Basu, Kayi Lee, Bryan Perozzi, Qin Cao, Leonard Berrada, Rahul Arya, Ke Chen, Katrina, Xu, Matthias Lochbrunner, Alex Hofer, Sahand Sharifzadeh, Renjie Wu, Sally Gold- man, Pranjal Awasthi, Xuezhi Wang, Yan Wu, Claire Sha, Biao Zhang, Maciej Mikula, Filippo Graziano, Siobhan Mcloughlin, Irene Giannoumis, Youhei Namiki, Chase Malik, Carey Rade- baugh, Jamie Hall, Ramiro Leal-Cavazos, Jianmin Chen, Vikas Sindhwani, David Kao, David Greene, Jordan Griffith, Chris Welty, Ceslee Montgomery. Toshihiro Yoshino. Liangzhe Yuan, Noah Goodman, Assaf Hurwitz Michaely, Kevin Lee, KP Sawhney, Wei Chen, Zheng Zheng,
13

---

Published as a conference paper at ICLR 2026
Megan Shum, Nikolay Savinov, Etienne Pot, Alex Pak, Morteza Zadimoghaddam, Sijal Bhat- nagar, Yoad Lewenberg, Blair Kutzman, Ji Liu, Lesley Katzen, Jeremy Selier. Josip Djolonga, Dmitry Lepikhin, Kelvin Xu, Jacky Liang, Jiewen Tan, Benoit Schillings, Muge Ersoy, Pete Blois, Bernd Bandemer, Abhimanyu Singh, Sergei Lebedev, Pankaj Joshi, Adam R. Brown, Evan Palmer, Shreya Pathak, Komal Jalan, Fedir Zubach, Shuba Lall, Randall Parker, Alok Gun- jan, Sergey Rogulenko, Sumit Sanghai, Zhaoqi Leng, Zoltan Egyed, Shixin Li, Maria Ivanova, Kostas Andriopoulos, Jin Xie, Elan Rosenfeld, Auriel Wright, Ankur Sharma, Xinyang Geng, Yicheng Wang, Sam Kwei, Renke Pan, Yujing Zhang, Gabby Wang, Xi Liu, Chak Yeung, Eliz- abeth Cole, Aviv Rosenberg. Zhen Yang. Phil Chen, George Polovets, Pranav Nair, Rohun Sax- ena, Josh Smith, Shuo yiin Chang. Aroma Mahendru, Svetlana Grant, Anand Iyer, Irene Cai, Jed McGiffin, Jiaming Shen, Alanna Walton, Antonious Girgis, Oliver Woodman, Rosemary Ke, Mike Kwong, Louis Rouillard, Jinmeng Rao, Zhihao Li, Yuntao Xu, Flavien Prost, Chi Zou, Ziwei Ji, Alberto Magni, Tyler Liechty, Dan A. Calian, Deepak Ramachandran, Igor Krivokon, Hui Huang, Terry Chen, Anja Hauth, Anastasija Ilié, Weijuan Xi. Hycontack Lim. Vlad-Doru Ion, Pooya Moradi, Metin Toksoz-Exley Kalesha Bullard, Miltos Allamanis, Xiaomeng Yang, Sophie Wang, Zhi Hong, Anita Gergely, Cheng Li, Bhavishya Mittal, Vitaly Kovalev, Victor Un- gureanu, Jane Labanowski, Jan Wassenberg, Nicolas Lacasse, Geoffrey Cideron, Petar Dević, Annie Marsden, Lynn Nguyen. Michael Fink, Yin Zhong, Tatsuya Kiyono, Desi Ivanov, Sally Ma, Max Bain, Kiran Yalasangi. Jennifer She, Anastasia Petrushkina, Mayank Lunayach, Carla Bromberg. Sarah Hodkinson, Vilobh Meshram, Daniel Vlasic, Austin Kyker, Steve Xu, Jeff Stan- way, Zuguang Yang, Kai Zhao. Matthew Tung, Seth Odoom, Yasuhisa Fujüi, Justin Gilmer, Eunyoung Kim, Felix Halim, Quoc Le, Bernd Bohnet, Seliem El-Sayed, Behnam Neyshabur, Malcolm Reynolds, Dean Reich, Yang Xu, Erica Moreira, Anuj Sharma, Zeyu Liu, Moham- mad Javad Hosseini, Naina Raisinghani, Yi Su, Ni Lao, Daniel Formoso, Marco Gelmi, Almog Gueta, Tapomay Dey, Elena Gribovskaya, Domagoj Čevid, Sidharth Mudgal, Garrett Bingham, Jianling Wang. Anurag Kumar, Alex Cullum, Feng Han, Konstantinos Bousmalis, Diego Cedillo, Grace Chu, Vladimir Magay, Paul Michel, Ester Hlavnova, Danicle Calandricllo, Setarch Ari- afar, Kaisheng Yao, Vikash Schwag, Arpi Vezer, Agustin Dal Lago, Zhenkai Zhu, Paul Kishan Rubenstein, Allen Porter, Anirudh Baddepudi, Oriana Riva, Mihai Dorin Istin, Chih-Kuan Yeh, Zhi Li, Andrew Howard, Nilpa Jha, Jeremy Chen, Raoul de Liedekerke, Zafarali Ahmed, Mikel Rodriguez, Tanuj Bhatia, Bangju Wang, Ali Elqursh, David Klinghoffer, Peter Chen, Pushmeet Kohli, Te Weiyang Zhang. Zack Nado, Jilin Chen, Maxwell Chen, George Zhang, Aayush Singh, Adam Hillier, Federico Lebron, Yiqing Tao, Ting Liu, Gabriel Dulac-Aroldd, Jingwei Zhang, Shashi Narayan, Buhuang Liu, Orhan Firat, Abhishek Bhowmick, Bingyuan Liu, Hao Zhang, Zizhao Zhang, Georges Rotival, Nathan Howard, Anu Sinha, Alexander Grushetsky, Ben- jamin Beyret, Keerthana Gopalakrishnan, James Zhao, Kyle He, Szaboles Payrits, Zaid Nabulsi, Zhaoyi Zhang, Weijie Chen, Edward Lee, Nova Fallen, Sreenivas Gollapudi, Aurick Zhou, Filip Pavetić, Thomas Köppe, Shiyu Huang, Rama Pasumarthi, Nick Fernando, Felix Fischer, Daria Čurko, Yang Gao, James Svensson, Austin Stone, Haroon Qureshi, Abhishek Sinha, Apoorv Kulshreshtha, Martin Matysiak, Jieming Mao, Carl Saroufim, Aleksandra Faust, Qingnan Duan, Gil Fidel, Kaan Katircioglu, Raphaël Lopez Kaufman, Dhruv Shah, Weize Kong, Abhishek Bapna, Gellért Weisz, Emma Dunleavy, Prancet Dutta, Tianqi Liu, Rahma Chaabouni. Carolina Parada, Marcus Wu, Alexandra Belias, Alessandro Bissacco, Stanislav Fort, Li Xiao, Fantine Huot, Chris Knutsen, Yochai Blau, Gang Li, Jennifer Prendki, Juliette Love, Yinlam Chow, Pichi Charoenpanit, Hidetoshi Shimokawa, Vincent Coriou, Karol Gregor, Tomas Izo, Arjun Akula, Mario Pinto, Chris Hahn, Dominik Paulus, Jiaxian Guo, Neha Sharma, Cho-Jui Hsieh, Adaeze Chukwuka, Kazuma Hashimoto, Nathalie Rauschmayr, Ling Wu, Christof Angermueller. Yulong Wang, Sebastian Gerlach, Michael Pliskin, Daniil Mirylenka, Min Ma, Lexi Baugher, Bryan Gale, Shaan Bijwadia, Nemanja Rakićević, David Wood, Jane Park. Chung-Ching Chang, Babi Seal, Chris Tar, Kacper Krasowiak, Yiwen Song, Georgi Stephanov, Gary Wang, Marcello Maggioni, Stein Xudong Lin, Felix Wu, Shachi Paul, Zixuan Jiang, Shubham Agrawal, Bilal Piot, Alex Feng, Cheolmin Kim, Tulsee Doshi, Jonathan Lai, Chuqiao, Xu, Sharad Vikram, Ciprian Chelba, Sebastian Krause, Vincent Zhuang, Jack Rae, Timo Denk, Adrian Collister, Lotte Weerts, Xi- anghong Luo, Yifeng Lu, Håvard Garnes, Nitish Gupta, Terry Spitz, Avinatan Hassidim, Lihao Liang, Izhak Shafran, Peter Humphreys, Kenny Vassigh, Phil Wallis, Virat Shejwalkar, Nicolas Perez-Nieves, Rachel Hornung, Melissa Tan, Beka Westberg, Andy Ly, Richard Zhang, Brian Farris, Jongbin Park, Alec Kosik, Zeynep Cankara, Andrii Maksai, Yunhan Xu, Albin Cassirer, Sergi Caelles, Abbas Abdolmaleki, Mencher Chiang, Alex Fabrikant, Shravya Shetty, Luheng
14

---

Published as a conference paper at ICLR 2026
He, Mai Giménez, Hadi Hashemi, Sheena Panthaplackel, Yana Kulizhskaya, Salil Deshmukh, Daniele Pighin, Robin Alazard, Disha Jindal, Seb Noury, Pradeep Kumar S. Siyang Qin, Xerxes Dotiwalla, Stephen Spencer, Mohammad Babaeizadeh, Blake JianHang Chen, Vaibhav Mchta, Jennie Lees, Andrew Leach, Penporn Koanantakool, Ilia Akolzin, Ramona Comanescu, Junwhan Ahn, Alexey Svyatkovskiy, Basil Mustafa, David D' Ambrosio, Shiva Mohan Reddy Garlapati, Pascal Lamblin, Alekh Agarwal, Shuang Song. Pier Giuseppe Sessa, Pauline Coquinot, John Maggs, Hussain Masoom, Divya Pitta, Yaqing Wang, Patrick Morris-Suzuki Billy Porter, John- son Jia, Jeffrey Dudek, Raghavender R, Cosmin Paduraru, Alan Ansell, Tolga Bolukbasi, Tony Lu, Ramya Ganeshan, Zi Wang, Henry Griffiths, Rodrigo Benenson, Yifan He, James Swirhun, George Papamakarios, Aditya Chawla, Kuntal Sengupta, Yan Wang. Vedrana Milutinovic, Igor Mordatch, Zhipeng Jia, Jamie Smith, Will Ng. Shitij Nigam, Matt Young, Eugen Vušak, Blake Hechtman, Sheela Goenka, Avital Zipori, Kareem Ayoub, Ashok Popat, Trilok Acharya, Luo Yu, Dawn Bloxwich, Hugo Song, Paul Roit, Haiqiong Li, Aviel Boag, Nigamaa Nayakanti, Bilva Chandra, Tianli Ding, Aahil Mchta, Cath Hope, Jiageng Zhang, Idan Heimlich Shtacher, Kartikeya Badola, Ryo Nakashima, Andrei Sozanschi, Iulia Comşa, Ante Žužul, Emily Cave- ness, Julian Odell, Matthew Watson, Dario de Cesare Phillip Lippe, Derek Lockhart, Siddharth Verma, Huizhong Chen, Sean Sun, Lin Zhuo, Aditya Shah, Prakhar Gupta, Alex Muzio, Ning Niu, Amir Zait, Abhinav Singh, Meenu Gaba, Fan Ye, Prajit Ramachandran, Mohammad Saleh, Raluca Ada Popa, Ayush Dubey, Frederick Liu, Sara Javanmardi, Mark Epstein, Ross Richard Green, Nishant Ranka, Eden Cohen, Chuyuan Kelly Fu, Sanjay Ghemawat, Jed Borovik, James Martens, Anthony Chen, Pranav Shyam, André Susano Pinto, Ming- Hiugg Yang. Alexan- dru Tifrea, David Du Boqing Gong, Ayushi Agarwal, Seungycon Kim, Christian Frank, Saloni Shah, Xiaodan Song, Zhiwei Deng, Ales Mikhalap, Kleopatra Chatziprimou, Timothy Chung, Toni Creswell, Susan Zhang, Yennie Jun, Carl Lebsack, Will Truong, Slavica Andačić, Itay Yona, Marco Fornoni, Rong Rong, Serge Toropov, Afzal Shama Soudagar, Andrew Audibert, Salah Zaiem, Zaheer Abbas, Andrei Rusu, Sahitya Potluri, Shitao Weng. Anastasios Kementsietsidis, Anton Tsitsulin, Daiyi Peng, Natalie Ha, Sanil Jain, Tejasi Latkar, Simeon Ivanov, Cory McLean, Anirudh GP, Rajesh Venkataraman, Canoee Liu, Dilip Krishnan, Joel D'sa, Roey Yogev, Paul Collins, Benjamin Lee, Lewis Ho, Carl Doersch, Gal Yona, Shawn Gao, Felipe Tiengo Ferreira, Adnan Ozturel, Hannah Muckenhirn. Ce Zheng, Gargi Balasubramaniam,, Mudit Bansal, George Van den Driessche, Sivan Eiger, Salem Haykal, Vedant Misra, Abhimanyu Goyal. Danilo Martins, Gary Leung, Jonas Valfridsson, Four Flynn, Will Bishop, Chenxi Pang, Yoni Halpern, Honglin Yu, Lawrence Moore, Yuvein, Zhu Sridhar Thiagarajan, Yoel Drori, Zhisheng Xiao, Lucio Dery. Rolf Jagerman, Jing Lu, Eric Ge, Vaibhav Aggarwal, Arjun Khare, Vinh Tran, Oded Elyada, Ferran Alet, James Rubin, Ian Chou, David Tian, Libin Bai, Lawrence Chan, Lukasz Lew, Karo- lis Misiunas, Taylan Bilal, Aniket Ray, Sindhu Raghuram, Alex Castro-Ros, Viral Carpenter, CJ Zheng, Michael Kilgore, Josef Broder, Emily Xue, Praveen Kallakuri, Dheeru Dua, Nancy Yuen, Steve Chien, John Schultz, Saurabh Agrawal, Reut Tsarfaty, Jingcao Hu, Ajay Kannan, Dror Marcus, Nisarg Kothari, Baochen Sun, Ben Horn, Matko Bošnjak, Ferjad Naeem, Dean Hirsch, Lewis Chiang, Boya Fang. Jie Han, Qifei Wang, Ben Hora, Antoine He, Mario Lučié, Beer Changpinyo, Anshuman Tripathi, John Youssef, Chester Kwak, Philippe Schlattner, Cat Graves, Rémi Leblond, Wenjun Zeng. Anders Andreassen, Gabriel Rasskin, Yue Song. Eddie Cao, Junhyuk Oh, Matt Hoffman, Wojtek Skut, Yichi Zhang, Jon Stritar, Xingyu Cai, Saarthak Khanna, Kathie Wang, Shriya Sharma, Christian Reisswig. Younghoon Jun, Aman Prasad, Ta- tiana Sholokhova, Preeti Singh, Adi Gerzi Rosenthal, Anian Ruoss, Françoise Beaufays, Sean Kirmani, Dongkai Chen, Johan Schalkwyk, Jonathan Herzig, Been Kim, Josh Jacob, Damien Vincent, Adrian N Reyes, Ivana Balazevic, Léonard Hussenot, Jon Schneider, Parker Barnes, Luis Castro, Spandana Raj Babbula, Simon Green, Serkan Cabi, Nico Duduta, Danny Driess, Rich Galt, Noam Velan, Junjie Wang, Hongyang Jiao, Matthew Mauger. Du Phan, Miteyan Patel, Vlado Galić, Jerry Chang, Eyal Marcus, Matt Harvey, Julian Salazar, Elahe Dabir, Suraj Satishku- mar Sheth, Amol Mandhane, Hanie Sedghi, Jeremiah Willcock, Amir Zandieh, Shruthi Prab- hakara, Aida Amini, Antoine Miech, Victor Stone, Massimo Nicosia, Paul Niemezyk, Ying Xiao, Lucy Kim, Slawek Kwasiborski, Vikas Verma, Ada Maksutaj Oflazer, Christoph Hirnschall, Pe- ter Sung, Lu Liu, Richard Everett, Michiel Bakker, Ágoston Weisz, Yufei Wang, Vivek Sam- pathkumar, Uri Shaham, Bibo Xu, Yasemin Altun, Mingqiu Wang, Takaaki Saeki, Guanjie Chen, Emanuel Taropa. Shanthal Vasanth, Sophia Austin, Lu Huang, Goran Petrovic, Qingyun Dou, Daniel Golovin, Grigory Rozhdestvenskiy, Allie Culp, Will Wu, Motoki Sano, Divya Jain, Julia Proskurnia, Sébastien Cevey, Alejandro Cruzado Ruiz, Piyush Patil. Mahdi Mirzazadeh, Eric Ni, Javier Snaider, Lijie Fan, Alexandre Fréchette, AJ Pierigiovanni, Shariq lgbal, Kenton Lee, Clau-
Hemsley,
15

---

Published as a conference paper at ICLR 2026
dio Fantacci, Jinwei Xing, Lisa Wang, Alex Irpan, David Raposo, Yi Luan, Zhuoyuan Chen, Har- ish Ganapathy, Kevin Hui, Jiazhong Nie, Isabelle Guyon, Heming Ge, Roopali Vij. Hui Zheng, Dayeong Lee, Alfonso Castaño, Khuslen Baatarsukh, Gabriel Ibagon, Alexandra Chronopoulou, Nicholas FitzGerald, Shashank Viswanadha, Safeen Huda, Rivka Moroshko, Georgi Stoyanov, Prateek Kolhar, Alain Vaucher, Ishaan Watts, Adhi Kuncoro, Henryk Michalewski, Satish Kam- bala, Bat-Orgil Batsaikhan, Alck Andreev, Irina Jurenka, Maigo Le, Qihang Chen, Wael Al Jishi, Sarah Chakera, Zhe Chen, Aditya Kini, Vikas Yadav, Aditya Siddhant, Ilia Labzovsky, Balaji Lakshminarayannnn, Carrie Grimes Bostock, Pankil Botadra, Ankesh Anand, Colton Bishop, Sam Conway-Rahman Mohit Agarwal, Yani Donchev, Achintya Singhal, Félix de Chaumont Quitry, Natalia Ponomareva, Nishant Agrawal, Bin Ni, Kalpesh Krishna, Masha Samsikova, John Karro, Yilun Du, Tamara von Glehn, Caden Lu, Christopher A. Choquette-Choo, Zhen Qin, Tingnan Zhang. Sicheng Li, Divya Tyam, Swaroop Mishra, Wing Lowe, Colin Ji, Weiyi Wang. Man- aal Faruqui, Ambrose Slone, Valentin Dalibard, Arunachalam Narayanaswamy, John Lambert, Pierre-Antoine Manzagol, Dan Karliner, Andrew Bolt, Ivan Lobov, Aditya Kusupati, Chang Ye, Xuan Yang, Heiga Zen, Nelson George, Mukul Bhutani, Olivier Lacombe, Robert Riachi, Gagan Bansal, Rachel Soh, Yue Gao, Yang Yu, Adams Yu, Emily Nottage, Tania Rojas-Esponda, James Noraky, Manish Gupta, Ragha Kotikalapudi, Jichuan Chang, Sanja Deur, Dan Graur, Alex Mossin, Erin Farnese, Ricardo Figueira, Alexandre Moufarek, Austin Huang. Patrik Zochbauer, Ben Ingram, Tongzhou Chen, Zelin Wu, Adria Puigdomenech, Leland Rechis, Da Yu, Sri Gay- atri Sundara Padmanabhan, Rui Zhu, Chu ling Ko, Andrea Banino, Samira Daruki, Aarush Sel- van, Dhruva Bhaswar, Daniel Hernandez Diaz, Chen Su, Salvatore Scellato, Jennifer Brennan, Woohyun Han, Grace Chung, Priyanka Agrawal, Urvashi Khandelwal, Khe Chai Sim, Mor- gane Lustman, Sam Ritter, Kelvin Guu, Jiawei Xia, Prateek Jain, Emma Wang, Tyrone Hill, Mirko Rossini, Marija Kostelac, Tautvydas Misiunas, Amit Sabne, Kyuyeun Kim, Ahmet Is- cen, Congchao Wang. José Leal, Ashwin Sreevatsa, Utku Evci, Manfred Warmuth, Saket Joshi, Daniel Suo, James Lottes, Garrett Honke, Brendan Jou, Stefani Karp, Jieru Hu, Himanshu Sahni, Adrien Ali Taïga, William Kong, Samrat Ghosh, Renshen Wang. Jay Pavagadhi, Natalie Axels- son, Nikolai Grigorev, Patrick Siegler, Rebecca Lin, Guohui Wang. Emilio Parisotto, Sharath Maddineni, Krishan Subudhi, Eyal Ben-David, Elena Pochernina, Orgad Keller, Thi Avrahami, Zhe Yuan, Pulkit Mehta, Jialu Liu, Sherry Yang, Wendy Kan, Katherine Lee, Tom Funkhouser, Derek Cheng, Hongzhi Shi, Archit Sharma, Joe Kelley, Matan Eyal. Yury Malkov, Corentin Tal- lec, Yuval Bahat, Shen Yan, Xintian, Wu, David Lindner, Chengda Wu, Avi Caciularu, Xiyang Luo, Rodolphe Jenatton, Tim Zaman, Yingying Bi, Ilya Kornakov, Ganesh Mallya, Daisuke Ikeda, Itay Karo, Anima Singh, Colin Evans, Praneeth Netrapalli, Vincent Nallatamby, Isaac Tian, Yannis Assael, Vikas Raunak, Victor Carbune, Ioana Bica, Lior Madmoni, Dee Cattle, Snchit Grover, Krishna Somandepalli, Sid Lall, Amelio Vázquez-Reina, Riccardo Patana, Jiaqi Mu, Pranav Talluri, Maggie Tran, Rajeev Aggarwal, RJ Skerry-Ryan, Jun Xu, Mike Burrows, Xiaoyue Pan, Edouard Yvinec, Di Lu, Zhiying Zhang, Duc Dung Nguyen. Hairong Mu, Gabriel Barcik, Helen Ran, Lauren Beltrone, Krzysztof Choromanski, Dia Kharrat, Samuel Albanic, Scan Purser-haskell, David Bieber, Carrie Zhang, Jing Wang. Tom Hudson, Zhiyuan Zhang, Han Fu, Johannes Mauerer, Mohammad Hossein Bateni, AJ Maschinot, Bing Wang, Muye Zhu, Arjun Pillai, Tobias Weyand, Shuang Liu, Oscar Akerlund, Fred Bertsch, Vittal Premachandran, Ali- cia Jin, Vincent Roulet, Peter de Boursac, Shubham Mittal, Ndaba Ndebele, Georgi Karadzhov, Sahra Ghalebikesabi, Ricky Liang. Allen Wu, Yale Cong, Nimesh Ghelani, Sumeet Singh, Ba- har Fatemi, Warren, Chen, Charles Kwong, Alexey Kolganov, Steve Li, Richard Song, Chenkai Kuang. Sobhan Miryoosefi, Dale Webster, James Wendt, Arkadiusz Socala, Guolong Su, Artur Mendonça, Abhinav Gupta, Xiaowei Li, Tomy Tsai, Qiong. Hu, Kai Kang, Angie Chen, Ser- tan Girgin, Yongqin Xian, Andrew Lee, Nolan Ramsden, Leslie Baker, Madeleine Clare Elish, Varvara Krayvanova, Rishabh Joshi, Jiri Simsa, Yao Yuan Yang. Piotr Ambroszczyk. Dipankar Ghosh, Arjun Kar, Yuan Shangguan, Yumeya Yamamori, Yaroslav Akulov, Andy Brock, Hao- tian Tang. Siddharth Vashishtha, Rich Munoz, Andreas Steiner, Kalyan Andra, Daniel Eppens, Qixuan Feng, Hayato Kobayashi, Sasha Goldshtein, Mona El Mahdy, Xin Wang. Jilei, Wang, Richard Killam, Tom Kwiatkowski, Kavya Kopparapu, Serena Zhan, Chao Jia, Alexei Bende- bury, Sheryl Luo, Adrià Recasens, Timothy Knight, Jing Chen, Mohak Patel, YaGuang Li, Ben Withbroe, Dean Weesner, Kush Bhatia, Jie Ren, Danielle Fisenbud, Ebrahim Songhori, Yanhua Sun, Travis Choma, Tasos Kementsietsidis. Lucas Manning, Brian Roark, Wael Farhan, Jie Feng, Susheel Tatineni, James Coron-Kerr, Yunjie Li, Lisa Anne Hendricks, Isaac Noble, Chris Breaux, Nate Kushman, Liqian Peng, Fuzhao Xue, Taylor Tobin, Jamie Rogers, Josh Lipschultz, Chris Alberti, Alexey Vlaskin, Mostafa Dehghani, Roshan Sharma, Tris Warkentin, Chen Yu Lee, Be-
16

---

Published as a conference paper at ICLR 2026
nigno Uria, Da-Cheng Juan, Angad Chandorkar, Hila Sheftel, Ruibo Liu, Elnaz Davoodi, Borja De Balle Pigem. Kedar Dhamdhere, David Ross, Jonathan Hoech, Mahdis Mahdieh, Li Liu, Qiu- jia Li, Liam McCafferty, Chenxi Liu, Markus Mircea, Yunting Song, Omkar Savant, Alaa Saade, Colin Cherry. Vincent Hellendoorn, Siddharth Goyal, Paul Pucciarelli, David Vilar Torres, 7o- har Yahav, Hyo Lee, Lars Lowe Sjoesund, Christo Kirov, Bo Chang, Deepanway Ghoshal, Lu Li, Gilles Baechler. Sébastien Pereira, Tara Sainath, Anudhyan Boral, Dominik Grewe, Afief Halumi, Nguyet Minh Phu, Tianxiao Shen, Marco Tulio Ribeiro, Dhriti Varma, Alex Kaskasoli, Vlad Fein- berg, Navneet Potti, Jarrod Kahn, Matheus Wisniewski, Shakir Mohamed, Arnar Mar Hrafnkels- son, Bobak Shahriari, Jean- -aptisttttt Lespiau, Lisa Patel, Legg Yeung, Tom Paine, Lantao Mei, Alex Rammrez, Rakesh Shivanna. Li Zhong, Josh Woodward, Guilherme Tubone, Samira Khan, Heng Chen, Elizabeth Nielsen, Catalin Ioneseu, Utsav Prabhu, Mingcen Gao, Qingze Wang, Sean Augenstein, Neesha Subramaniam, Jason Chang, Fotis Iliopoulos, Jiaming Luo, Myriam Khan, Weicheng Kuo, Denis Teplyashin, Florence Perot, Logan Kilpatrick, Amir Globerson, Hongkun Yu, Anfal Siddiqui, Nick Sukhanov, Arun Kandoor, Umang Gupta, Marco Andreetto, Moran Ambar, Donnie Kim, Pawef Wesotowski, Sarah Perrin, Ben Limonchik, Wei Fan, Jim Stephan, Ian Stewart-Binks, Ryan Kappedal, Tong He, Sarah Cogan, Romina Datta, Tong Zhou, Jiayu Ye, Leandro Kieliger, Ana Ramalho, Kyle Kastner, Fabian Mentzer, Wei-Jen Ko, Arun Suggala, Tianhao Zhou, Shiraz Butt, Hana Strejček, Lior Belenki, Subhashini Venugopalan, Mingyang Ling, Exgenii Eltyshev, Yunxiao Deng, Geza Kovacs, Mukund Raghavachari, Hanjun Dai, Tal Schuster, Steven Schwarez. Richard Nguyen, Arthur Nguyen. Gavin Buttimore, Shrestha Basu Mallick, Sudeep Gandhe, Seth Benjamin, Michal Jastrzebski, Le Yan, Sugato Basu, Chris Apps, Isabel Edkins, James Allingham, Immanuel Odisho, Tomas Kocisky, Jewel Zhao, Linting Xue, Apoorv Reddy, Chrysovalantis Anastasiou, Aviel Atias, Sam Redmond, Kieran Milan, Nico- las Heess, Herman Schmit, Allan Dafoe, Daniel Andor, Tynan Gangwani, Anca Dragan, Sheng Zhang, Ashyana Kachra, Gang Wu, Siyang Xue, Kevin Aydin, Siqi Liu, Yuxiang Zhou, Mahan Malihi, Austin Wu, Siddharth Gopal, Candice Schumann, Peter Stys, Alek Wang, Mirek Olšák, Dangyi Liu, Christian Schallhart, Yiran Mao, Demetra Brady. Hao Xu, Tomas Mery, Chawin Sitawarin, Siva Velusamy, Tom Cobley, Alex Zhai, Christian Walder, Nitzan Katz, Ganesh Jawa- har, Chinmay Kulkarni, Antoine Yang, Adam Paszke, Yinan Wang, Bogdan Damoc, 7alán Bor- sos, Ray Smith, Jinning Li, Mansi Gupta, Andrei Kapishnikov. Sushant Prakash, Florian Luisier, Rishabh Agarwal, Will Grathwohl, Kuangyuan Chen, Kehang Han, Nikhil Mchta, Andrew Over, Shekoofch Azizi, Lei Meng, Niccolò Dal Santo, Kelvin Zheng, Jane Shapiro, Igor Petrovski, Jeffrey Hui, Amin Ghafouri, Jasper Snoek, James Qin, Mandy Jordan, Caitlin Sikora, Jonathan Malmaud, Yuheng Kuang, Aga Swietlik, Ruoxin Sang, Chongyang Shi, Leon Li, Andrew Rosen- berg, Shubin Zhao, Andy Crawford, Jan- Thersterr Peter. Yun Lei, Xavier Garcia, Long Lc, Todd Wang, Julien Amelot, Dave Orr, Praneeth Kacham, Dana Alon, Gladys Tyen, Abhinav Arora, James Lyon, Alex Kurakin, Mimi Ly. Theo Guidroz, Zhipeng Yan, Rina Panigrahy, Pingmei Xu, Thais Kagohara, Yong Cheng, Eric Noland, Jinhyuk Lee, Jonathan Lee, Cathy Yip. Maria Wang. Efrat Nehoran, Alexander Bykovsky, Zhihao Shan, Ankit Bhagatwala, Chaochao Yan, Jie Tan, Guillermo Garrido, Dan Ethier, Nate Hurley, Grace Vesom, Xu Chen, Siyuan Qiao, Ab- hishek Nayyar, Julian Walker, Paramjit Sandhu, Mihaela Rosca, Danny Swisher, Mikhail Dek- tiarev. Josh Dillon, George-Cristian Muraru, Manuel Tragut, Artiom Myaskovsky, David Reid, Marko Velie, Owen Xiao, Jasmine George, Mark Brand, Jing Li, Wenhao Yu, Shane Gu, Xiang Deng, François-Xavier Aubet, Soheil Hassas Yeganeh, Fred Alcoher, Celine Smith, Trevor Cohn, Kay McKinney, Michael Tschannen, Ramesh Sampath, Gowoon Cheon, Liangchen Luo, Luyang Liu, Jordi Orbay, Hui Peng, Gabriela Botca, Xiaofan Zhang, Charles Yoon, Cesar Magalhaes, Pawel Stradomski, Ian Mackinnon, Steven Hemingray, Kumaran Venkatesan, Rhys May, Jaeyoun Kim, Alex Druinsky, Jingchen Ye, Zheng Xu, Terry Huang, Jad AI Abdallah, Adil Dostmohamed, Rachana Fellinger, Tsendsuren Munkhdalai, Akanksha Maurya, Peter Garst, Yin Zhang, Maxim Krikun, Simon Bucher, Aditya Srikanth Veerubhotla, Yaxin Liu, Sheng Li, Nishesh Gupta, Jakub Adamek, Hanwen Chen, Bernett Orlando, Aleksandr Zaks, Joost van Amersfoort, Josh Camp, Hui Wan, HyunJcong Choe, Zhichun Wu, Kate Olszewska, Weiren Yu, Archita Vadali, Martin Scholz, Daniel De Freitas, Jason Lin, Amy Hua. Xin Liu, Frank Ding, Yichao Zhou, Boone Severson, Ka- terina Tsihlas, Samuel Yang, Tammo Spalink, Varun Yerram, Helena Pankov, Rory Blevins, Ben Vargas, Sarthak Jauhari, Matt Miecnikowski, Ming Zhang, Sandeep Kumar, Clement Farabet, Charline Le Lan, Sebastian Flennerhag, Yonatan Bitton, Ada Ma, Arthur Bražinskas, Eli Collins, Niharika Ahuja, Sneha Kudugunta, Anna Bortsova, Minh Giang, Wanzheng Zhu, Ed Chi, Scott Lundberg, Alexey Stern, Subha Puttagunta, Jing Xiong, Xiao Wu, Yash Pande, Amit Jhindal, Daniel Murphy, Jon Clark, Mare Brockschmidt, Maxine Deines, Kevin R. McKee, Dan Bahir,
17

---

Published as a conference paper at ICLR 2026
Jiajun Shen, Minh Truong, Daniel McDuff, Andrea Gesmundo, Edouard Rosseel, Bowen Liang, Ken Caluwaerts, Jessica Hamrick. Joseph Kready, Mary Cassin, Rishikesh Ingale, Li Lao, Scott Pollom, Yifan Ding, Wei He, Lizzetth Bellot, Joana Iljazi, Ramya Sree Boppana, Shan Han, Tara Thompson, Amr Khalifa, Anna Bulanova, Blagoj Mitrevski, Bo Pang, Emma Cooney, Tian Shi, Rey Coaguila, Tamar Yakar, aurelio Ranzato, Nikola Momchev, Chris Rawles, Zachary Charles, Young Maeng, Yuan Zhang, Rishabh Bansal, Xiaokai Zhao, Brian Albert, Yuan Yuan, Sudheendra Vijayanarasimhan, Roy Hirsch, Vinay Ramasesh, Kiran Vodrahalli, Xingyu Wang, Arushi Gupta, DJ Strouse, Jianmo Ni, Roma Patel, Gabe Taubman, Zhouyuan Huo, Dero Gharib- ian, Marianne Monteiro, Hoi Lam, Shobha Vasudevan, Aditi Chaudhary, Isabela Albuquerque, Kilol Gupta, Sebastian Riedel, Chaitra Hegde, Avraham Ruderman, András György, Marcus Wainwright, Ashwin Chaugule, Bureu Karagol Ayan, Tomer Levinboim, Sam Shleifer, Yogesh Kalley, Vahab Mirrokni, Abhishek Rao, Prabakar Radhakrishnan, Jay Hartford, Jialin Wu, Zhen- hai Zhu, Francesco Bertolini, Hao Xiong. Nicolas Serrano, Hamish Tomlinson, Myle Ott, Yifan Chang, Mark Graham, Jian Li, Marco Liang. Xiangzhu Long, Sebastian Borgeaud, Yanif Ahmad, Alex Grills, Diana Mincu, Martin Izzard, Yuan Liu, Jinyu Xie, Louis O' Bryan, Sameera Ponda, Simon Tong, Michelle Liu, Dan Malkin, Khalid Salama, Yuankai Chen, Rohan Anil, Anand Rao, Rigel Swavely, Misha Bilenko, Nina Anderson, Tat Tan, Jing Xie, Xing Wu, Lijun Yu, Oriol Vinyals, Andrey Ryabtsev, Rumen Dangovski, Kate Baumli, Daniel Keysers, Christian Wright, Zoe Ashwood, Betty Chan, Artem Shtefan, Yaohui Guo, Ankur Bapna, Radu Soricut, Steven Pecht, Sabela Ramos, Rui Wang. Jiahao Cai, Trieu Trinh, Paul Barham, Linda Friso, Eli Stickgold, Xiangzhuo Ding, Siamak Shakeri, Diego Ardila, Eleftheria Briakou, Phil Culliton, Adam Raveret, Jingyu Cui, David Saxton, Subhrajit Roy, Javad Azizi, Pengcheng Yin, Lucia Loher, Andrew Bun- ner, Min Choi, Faruk Ahmed, Eric Li, Yin Li, Shengyang Dai, Michael Elabd, Sriram Ganapathy, Shivani Agrawal, Yiqing Hua, Paige Kunkle, Sujeevan Rajayogam, Arun Abuja, Arthur Conmy, Alex Vasiloff, Parker Beak, Christopher Yew, Jayaram Mudigonda, Bartek Wydrowski. Jon Blan- ton, Zhengdong Wang, Yann Dauphin, Zhuo Xu, Martin Polacek, Xi Chen, Hexiang Hu,, Pauline Sho, Markus Kunesch, Mehdi Hafezi Manshadi, Eliza Rutherford, Bo Li, Sissie Hsiao, Lain Barr, Alex Tudor, Matija Keeman, Arsha Nagrani, Vladimir Pchelin, Martin Sundermeyer, Aishwarya P S, Abhijit Karmarkar, Yi Gao. Grishma Chole, Olivier Bachem, Isabel Gao, Arturo BC, Matt Dibb, Mauro Verzetti, Felix Hernandez-Campos. Yana Lunts, Matthew Johnson, Julia Di Trapani, Raphael Koster, Idan Brusilovsky, Binbin Xiong. Megha Mohabey, Han Ke, Joe Zou, Tea Sabolić, Víctor Campos. John Palowitch, Alex Morris, Linhai Qiu, Pranavaraj Ponnuramu, Fangtao Li, Vivek Sharma, Kiranbir Sodhia, Kaan Tekelioglu, Aleksandr Chuklin, Madhavi Yenugula, Erika Gemzer, Theofilos Strinopoulos, Sam EI- El-Husseini, Huiyu Wang, Yan Zhong, Edouard Leurent, Paul Natsev, Weijun Wang. Dre Mahaarachchi, Tao Zhu, Songyou Peng. Sami Alabed, Cheng- Chun Lee, Anthony Brohan, Arthur Szlam, GS Oh, Anton Kovsharov, Jenny Lee, Renee Wong, Megan Barnes, Gregory Thornton, Felix Gimeno, Omer Levy, Martin Sevenich, Melvin Johnson, Jonathan Mallinson, Robert Dadashi, Ziyue Wang, Qingchun Ren, Preethi Lahoti, Arka Dhar, Josh Feldman, Dan Zheng, Thatcher Uirich, Liviu Panait, Michiel Blokzijl. Cip Bactu, Josip Matak, Jitendra Harlalka, Maulik Shah, Tal Marian, Daniel von Dincklage, Cosmo Du, Ruy Ley- Wild, Bethanie Brownfield, Max Schumacher, Yury Stuken, Shadi Noghabi, Sonal Gupta, Xiaoqi Ren, Eric Malmi, Felix Weissenberger, Blanca Huergo, Maria Bauza, Thomas Lampe, Arthur Douillard, Mojtaba Seyedhosseini, Roy Frostig, Zoubin Ghahramani, Kelvin Nguyen, Kashyap Krishnakumar, Chengxi Ye, Rahul Gupta, Alireza Nazari, Robert Geirhos, Pete Shaw, Ahmed Eleryan, Dima Damen, Jennimaria Palomaki, Ted Xiao. Qiyin Wu, Quan Yuan, Phoenix Mead- owlark, Matthew Bilotti, Raymond Lin, Mukund Sridhar, Yannick Schroecker, Da- Da-Woon Chung, Jincheng Luo, Trevor Strohman, Tianlin Liu, Anne Zheng, Jesse Emond, Wei Wang, Andrew Lampinen, Toshiyuki Fukuzawa, Folawiyo Camphell-Ajala, Monica Roy, James Lee-Thorp. Lily Wang. Iftekhar Naim, Tony. Nguy ên, Guy Bensky, Aditya Gupta, Dominika Rogozińskka, Justin Fu, Thanumalayan Sankaranarayana Pillai, Petar Veličković, Shahar Drath, Philipp Neubeck, Vaibhav Tulsyan, Arseniy Klimovskiy, Don Metzler, Sage Stevens, Angel Yeh, Junwei Yuan, Tianhe Yu, Kelvin Zhang. Alec Go, Vincent Tsang. Ying Xu, Andy Wan, Isaac Galatzer-Levy, Sam Sobell, Abodunrinwa Toki, Elizabeth Salesky, Wenlei Zhou, Diego Antognini, Sholto Dou- glas, Shimu Wu, Adam Lelkes, Frank Kim, Paul Cavallaro, Ana Salazar, Yuchi Liu, James Besley, Tiziana Refice, Yiling Jia, Zhang Li, Michal Sokolik, Arvind Kannan, Jon Simon, Jo Chick, Avia Aharon, Meet Gandhi, Mayank Daswani, Keyvan Amiri, Vighnesh Birodkar, Abe Ittycheriah, Peter Grabowski, Oscar Chang, Charles Sutton, Zhixin, Lai, Umesh Telang, Susie Sargsyan, Tao Jiang. Raphael Hoffmann, Nicole Brichtova, Matteo Hessel, Jonathan Halcrow, Sammy Jerome, Geoff Brown, Alex Tomala, Elena Buchatskaya, Dian Yu, Sachit MenOn, Pol Moreno, Yuguo
18

---

Published as a conference paper at ICLR 2026
Liao, Vicky Zayats, Luming Tang. SQ Mah, Ashish Shenoy, Alex Siegman, Majid Hadian, Okwan Kwon, Tao Tu, Nima Khajehmouri, Ryan Foley, Parisa Haghani, Zhongru Wu, Vaishakh Keshava, Khyatti Gupta, Tony Bruguier, Rui Yao, Danny Karmon, Luisa Zintgraf, Zhicheng Wang, En- rique Piqueras, Junehyuk Jung, Jenny Brennan, Diego Machado, Marissa Giustina, MH Tessler, Kamyu Lee, Qiao Zhang. Joss Moore, Kaspar Daugaard, Alexander Frömmgen, Jennifer Beat- tie, Fred Zhang, Daniel Kasenberg. Ty Geri, Danfeng Qin, Gaurav Singh Tomar, Tom Ouyang, Tianli Yu, Luowei Zhou, Rajiv Mathews, Andy Davis, Yaoyiran Li, Jai Gupta, Damion Yates, Linda Deng, Elizabeth Kemp, Ga- Young Joung, Sergei Vassilvitskii, Mandy Guo, Pallavi LV, Dave Dopson, Sami Lachgar, Lara McConnaughey. Himadri Choudhury, Dragos Dena, Aaron Cohen, Joshua Ainslie, Sergey Levi, Parthasarathy Gopavarapu, Polina Zablotskaia, Hugo Val- let, Sanaz Bahargam, Xiaodan Tang, Nenad Tomasev, Ethan Dyer, Daniel Balle, Hongrae Lee, William Bono, Jorge Gonzalez Mendez, Vadim Zubov, Shentao Yang. Ivor Rendulic, Yanyan Zheng, Andrew Hogue, Golan Pundak, Ralph Leith, Avishkar Bhoopchand, Michael Han, Mis- lav Zanić, Tom Schaul, Manolis Delakis, Tejas Iyer, Guanyu Wang. Harman Singh, Abdelrah- man Abdelhamed, Tara Thomas, Siddhartha Brahma, Hilal Dib, Naveen Kumar, Wenxuan Zhou, Liang Bai, Pushkar Mishra, Jiao Sun, Valentin Anklin, Roykrong Sukkerd, Lauren Agubuzu, An- ton Briukhov, Anmol Gulati, Maximilian Sieb, Fabio Pardo, Sara Nasso, Junquan Chen, Kexin Zhu, Tiberiu Sosea, Alex Goldin, Keith Rush, Spurthi Amba Hombaiah, Andreas Noever, Al- lan Zhou. Sam Haves, Mary Phuong, Jake Ades, Yi ting Chen, Lin Yang, Joseph Pagadora, Stan Bileschi, Victor Cotruta, Rachel Saputro, Arijit Pramanik, Sean Ammirati, Dan Garrette, Kevin Villela, Tim Blyth, Canfer Akbulut, Neha Jha, Alban Rrustemi, Arissa Wongpanich, Chi- rag Nagpal. Yonghui Wu, Morgane Rivière, Sergey Kishchenko, Pranesh Srinivasan, Alice Chen, Animesh Sinha, Trang Pham, Bill Jia, Tom Hennigan, Anton Bakalov, Nithya Attaluri, Drew Garmon, Daniel Rodriguez, Dawid Wegner, Wenhao Jia, Evan Senter, Noah Fiedel, Denis Pe- tek, Yuchuan Liu, Cassidy Hardin, Harshal Tushar Lehri, Joao Carreira, Sara Smoot, Marcel Prasetya, Nami Akazawa, Anca Stefanoiu, Chia-Hua Ho, Anelia Angelova, Kate Lin, Min Kim, Charles Chen, Marein Sieniek, Alice Ii, Tongfei Guo, Sorin Baltateanu, Pouya Tafti, Michael Wunder, Nadav Olmert, Divyansh Shukla, Jingwei Shen, Neel Kovelamudi, Balaji Venkatraman, Seth Neel, Romal Thoppilan, Jerome Connor, Frederik Benzing. Axel Stjerngren. Golnaz Ghi- asi, Alex Polozov. Joshua Howland, Theophane Weber, Justin Chiu, Ganesh Poomal Girirajan, Andreas Terzis, Pidong Wang, Fangda Li, Yoav Ben Shalom, Dinesh Tewari, Matthew Denton, Roee Aharoni, Norbert Kalb, Heri Zhao, Junlin Zhang, Angelos Filos, Matthew Rahtz, Lalit Jain, Connie Fan, Vitor Rodrigues, Ruth Wang, Richard Shin, Jacob Austin, Roman Ring, Mariella Varghaz-Vargas, Mehadi Hassen, Ido Kessler, Uri Alon, Gufeng Zhang, Wenhu Chen, Yenai Ma, Xiance Si, Le Hou, Azalia Mirhoseini, Mare Wilson, Geoff Bacon, Becca Roelofs, Lei Shu, Gautam Vasudevan, Jonas Adler, Artur Dwomik, Tayfun Terzi, Matt Lawlor, Harry Askham, Mike Bernico, Xuanyi Dong, Chris Hidey, Kevin Kilgour, Gael Liu, Surya Bhupatiraju, Luke Leonhard, Siqi Zuo, Partha Talukdar, Qing Wei, Aliaksei Severyn, Vít Listík, Jong Lee, Aditya Tripathi, SK Park, Yossi Matias, Hao Liu, Alex Ruiz, Rajesh Jayaram, Jackson Tolins, Pierre Marcenac, Yiming Wang, Bryan Seybold, Henry Prior, Deepak Sharma, Jack Weber, Mikhail Sirotenko, Yunhsuan Sung, Dayou Du, Ellie Pavlick, Stefan Zinke, Markus Freitag, Max Dylla, Montse Gonzalez Arenas, Natan Potikha, Omer Goldman, Connie Tao, Rachita Chhaparia, Maria Voitovich, Pawan Dogra. Andrija Ražnatović, Zak Tsai, Chong You, Oleaser Johnson, George Tucker, Chenjie Gu, Jae Yoo, Maryam Majzoubi, Valentin Gabeur, Bahram Raad, Rocky Rhodes, Kashyap Kolipaka, Heidi Howard, Geta Sampemane, Benny Li, Chulayuth Asawaroengchai, Duy Nguyen, Chiyuan Zhang, Timothee Cour, Xinxin Yu, Zhao Fu, Joe Jiang, Po-Sen Huang, Gabriela Surita, Iñaki Iturrate. Yael Karov, Michael Collins, Martin Bacuml, Fabian Fuchs, Shilpa Shetty, Swaroop Ramaswamy, Sayna Ebrahimi, Qiuchen Guo, Jeremy Shar, Gabe Barth-Maron, Sravanti Addepalli, Bryan Richter, Chin- Yi Cheng. Eugénie Rives, Fei Zheng, Johannes Griesser, Nishanth Dikkala, Yoel Zeldes, I1kin Safarli, Dipanjan Das, Himanshu Srivastava. Sadh MNM Khan, Xin Li, Aditya Pandey, Larisa Markeeva, Dan Belov, Qiqi Yan, Mikotaj Rybiński, Tao Chen, Megha Nawhal, Michael Quinn, Vineetha Govindaraj. Sarah York, Reed Roberts, Roopal Garg, Namrata Godbole, Jake Abernethy, Anil Das, Lam Nguyen Thiet, Jonathan Tompson, John Nham, Neera Vats, Ben Caine, Wesley Helmholz, Francesco Pongetti, Ycongil Ko, James An, Clara Huiyi Hu, Yu-Cheng Ling, Julia Pawar, Robert Leland, Keisuke Kinoshita, Waleed Khawaja, Marco Selvi, Eugene Ie, Danila Sinopalnikov, Lev Proleev, Nilesh Tripuraneni, Michele Bevilacqua, Seungji Lee, Clayton Sanford, Dan Suh, Dustin Tran, Jeff Dean, Simon Baumgartner, Jens Heitkaemper, Sagar Gubbi, Kristina Toutanova, Yichong Xu, Chandu Thekkath, Keran Rong, Palak Jain, Annie Xie, Yan Virin, Yang Li, Lubo Litchev, Richard Powell, Tarun Bharti, Adam Kraft, Nan Hua,
19

---

Published as a conference paper at ICLR 2026
Marissa Ikonomidis, Ayal Hitron, Sanjiv Kumar, Loic Matthey, Sophie Bridgers, Lauren Lax, Ishaan Malhi, Ondrej Skopek, Ashish Gupta, Jiawei Cao, Mitchelle Rasquinha, Siim Pöder, Woj- ciech Stokowiec, Nicholas Roth, Guowang Li, Michaël Sander, Joshua Kessinger, Vihan Jain, Ed- ward Loper, Wonpyo Park, Michal Yarom, Liqun Cheng, Guru Guruganesh, Kanishka Rao, Yan Li, Catarina Barros, Mikhail Sushkov, Chun-Sung Ferng. Rohin Shah, Ophir Aharoni, Ravin Ku- mar, Tim McConnell, Peiran Li, Chen Wang, Fernando Pereira, Craig Swanson, Fayaz Jamil, Yan Xiong, Anitha Vijayakumar, Prakash Shroff, Kedar Soparkar, Jindong Gu, Livio Baldini Soares, Eric Wang, Kushal Majmundar, Aurora Wei, Kai Bailey, Nora Kassner, Chizu Kawamoto, Goran Zužié, Victor Gomes, Abhirut Gupta, Michael Guzman, Ishita Dasgupta, Xinyi Bai, Zhufeng Pan, Francesco Piccinno, Hadas Natalie Vogel, Octavio Ponce, Adrian Hutter, Paul Chang. Pan-Pan Jiang. lonel Gog, Vlad Ionescu, James Manyika, Fabian Pedregosa, Harry Ragan, Zach Behrman, Ryan Mullins, Coline Devin, Aroonalok Pyne. Swapnil Gawde, Martin Chadwick, Yiming Gu, Sasan Tavakkol, Andy Twigg, Naman Goyal, Ndidi Elue, Anna Goldie, Srinivasan Venkatachary, Hongliang Fei, Ziqiang Feng, Marvin Ritter, Isabel Leal, Sudeep Dasari, Pei Sun, Alif Raditya Rochman, Brendan O' O'Donoghue, Yuchen Liu, Jim Sproch, Kai Chen, Natalie Clay, Slav Petrov, Sailesh Sidhwani, Ioana Mihailescu, Alex Panagopoulos, AJ Piergiovanni, Yunfei Bai, George Powell, Deep Karkhanis, Trevor Yacovone, Petr Mitrichev, Joe Kovac, Dave Uthus, Amir Yaz- danbakhsh, David Amos, Steven Zheng, Bing Zhang. Jin Miao, Bhuvana Ramabhadran, Soroush Radpour, Shantanu Thakoor, Josh Newlan, Oran Lang, Orion Jankowski, Shikhar Bharadwaj, Jean-Michel Sarr, Shereen Ashraf, Sneha Mondal, Jun Yan, Ankit Singh Rawat, Sarmishta Velury, Greg Kochanski, Tom Eccles, Franz Och, Abhanshu Sharma, Ethan Mahintorabi, Alex Gumey, Carrie Muir, Vered Cohen, Saksham Thakur, Adam Bloniarz, Asier Mujika, Alexander Pritzel, Paul Caron, Altaf Rahman, Fiona Lang, Yasumasa Onoe, Petar Sirkovic, Jay Hoover, Ying Jian, Pablo Duque, Arun Narayanan, David Soergel, Alex Haig, Loren Maggiore. Shyamal Buch, Josef Dean, Ilya Figotin, Igor Karpov, Shaleen Gupta, Denny Zhou, Muhuan Huang, Ashwin Vaswani, Christopher Semturs, Kaushik Shivakumar, Yu Watanabe, Vinodh Kumar Rajendran, Eva Lu, Yanhan Hou, Wenting Ye, Shikhar Vashishth, Nana Nti, Vytenis Sakenas, Darren Ni, Doug DeCarlo, Michael Bendersky, Sumit Bagri, Nacho Cano. Elijah Peake, Simon Tokumine, Varun Godbole, Carlos Guía, Tanya Lando, Vittorio Selo, Scher Ellis, Danny Tarlow, Daniel Gillick, Alessandro Epasto. Siddhartha Reddy Jonnalagadda, Meng Wei, Meiyan Xie, Ankur Taly, Michela Paganini, Mukund Sundararajan, Daniel Toyama, Ting Yu, Dessie Petrova, Aneesh Pappu, Rohan Agrawal, Senaka Buthpitiya, Justin Frye, Thomas Buschmann, Remi Crocker, Marco Tagliasacchi, Mengchao Wang. Da Huang, Sagi Perel, Brian Wieder, Hideto Kazawa, Weiyue Wang, Jeremy Cole, Himanshu Gupta, Ben Golan, Seojin Bang, Nitish Kulkarni, Ken Franko, Casper Liu, Doug Reid, Sid Dalmia, Jay Whang, Kevin Cen, Prasha Sundaram, Johan Ferret, Berivan Isik, Lucian Ionita, Guan Sun, Anna Shekhawat, Muqthar Mohammad, Philip Pham, Ronny Huang, Karthik Raman, Xingyi Zhou, Ross Mcilroy, Austin Myers. Sheng Peng, Jacob Scott, Paul Covington, Sofia Erell, Pratik Joshi, João Gabriel Oliveira, Natasha Noy, Tajwar Nasir, Jake Walker, Vera Axelrod, Tim Dozat, Pu Han, Ch Chu, Eugene Weinstein, Anand Shukla, Shreyas Chandrakaladharrnn Petra Poklukar, Bonnie Li, Ye Jin, Prem Eruvbetine, Steven Hansen, Avigail Dabush, Alon Jacovi, Samrat Phatale, Chen Zhu, Steven Baker, Mo Shomrat, Yang Xiao, Jean Pouget-Abadie, Mingyang Zhang, Fanny Wei, Yang Song, Helen King. Yiling Huang. Yun Zhu, Ruoxi Sun, Juliana Vicente Franco, Chu-Cheng Lin. Sho Arora, Hui, Li, Vi- vian Xia, Luke Vilnis, Mariano Schain, Kaiz Alarakyia, Laurel Prince, Aaron Phillips. Caleb Habtegebriel, Luyao Xu, Huan Gui, Santiago Ontanon, Lora Aroyo, Karan Gill, Peggy Lu, Yash Katariya, Dhruv Madeka, Shankar Krishnan, Shubha Srinivas Raghvendra, James Freed- man, Yi Tay, Gaurav Menghani, Peter Choy, Nishita Shetty, Dan Abolatia, Doron Kukliansky, Edward Chou, Jared Lichtarge, Ken Burke, Ben Coleman, Dee Guo, Larry Jin, Indro Bhat- tacharya, Victoria Langston, Yiming Li, Suyog Kotecha, Alex Yakubovich, Xinyun Chen, Pe- tre Petrov, Tolly Powell, Yanzhang He, Corbin Quick, Kanav Garg, Dawsen Hwang, Yang Lu, Srinadh Bhojanapalli, Kristian Kjems, Ramin Mehran, Aaron Archer, Hado van Hasselt, Ash- win Balakrishna, JK Kearns, Meiqi Guo, Jason Riesa, Mikita Sazanovich, Xu Gao, Chris Sauer, Chengrun Yang. XiangHai Sheng. Thomas Jimma, Wouter Van Gansbeke, Vitaly Nikolaev, Wei Wei, Katie Millican, Ruizhe Zhao, Justin Snyder, Levent Bolelli, Maura O'Brien, Shawn Xu, Fei Xia, Wentao Yuan, Arvind Neelakantan, David Barker, Sachin Yadav, Hannah Kirkwood, Fa- rooq Ahmad, Joel Wee, Jordan Grimstad, Boyu Wang, Matthew Wiethoff, Shane Settle, Miaosen Wang, Charles Blundell, Jingjing Chen, Chris Duvarney Grace Hu, Olaf Ronneberger, Alex Lee, Yuanzhen Li, Abhishek Chakladar, Alena Butryna, Georgios Evangelopoulos, Guillaume Des- jardins, Jonni Kanerva, Henry Wang, Averi Nowak, Nick Li, Alyssa Loo, Art Khurshudov, Lau-
20

---

Published as a conference paper at ICLR 2026
rent EI Shafey, Nagabhushan Baddi, Karel Lene, Yasaman Razeghi, Tom Lieber, Amer Sinha, Xiao Ma. Yao Su, James Huang, Asahi Ushio, Hanna Slimczak-Plucinsae Kareem Mohamed, JD Chen, Simon Osindero, Stav Ginzburg, Lampros Lamprou, Vasilisa Bashlovkina, Duc-Hieu Tran, Ali Khodaei, Ankit Anand, Yixian Di, Ramy Eskander, Manish Reddy Vuyyuru, Jasmine Liu, Aishwarya Kamath, Roman Goldenherg, Mathias Bellaiche, Juliette Pluto, Bill Rosgen, Has- san Mansoor, William Wong, Suhas Ganesh, Eric Bailey, Scott Baird, Dan Deutsch, Jinoo Baek, Xuhui Jia, Chansoo Lee, Abe Friesen, Nathaniel Braun, Kate Lee, Amayika Panda, Steven M. Hermandez, Duncan Williams, Jianqiao Liu, Ethan Liang, Arnaud Autef, Emily Pitler, Deepali Jain, Phoebe Kirk, Oskar Bunyan, Jaume Sanchez Elias, Tongxin Yin, Machel Reid, Aedan Pope, Nikita Putikhin, Bidisha Samanta, Sergio Guadarrama, Dahun Kim, Simon Rowe, Mar- cella Valentine, Geng Yan, Alex Salcianu, David Silver, Gan Song, Richa Singh, Shuai Ye, Han- nah DeBalsi, Majd Al Merey, Eran Ofek, Albert Webson, Shibl Mourad, Ashwin Kakarla, Silvio Lattanzi, Nick Roy, Evgeny Sluzhaev, Christina Buttcrfield, Alessio Tonioni, Nathan Waters, Sudhindra Kopalle, Jason Chase, James Cohan, Girish Ramchandra Rao, Robert Berry, Michael Voznesensky, Shuguang Hu, Kristen Chiafullo, Sharat Chikkerur, George Scrivener, Ivy Zheng, Jeremy Wiesner, Wolfgang Macherey, Timothy Lillicrap, Fei Liu, Brian Walker, David Welling, Elinor Davies. Yangsibo Huang, Lijie Ren, Nir Shabat, Alessandro Agostini, Mariko linuma, Dustin Zelle, Rohit Sathyanarayana, Andrea D' D'olimpio, Morgan Redshaw, Matt Ginsberg. Ash- win Murthy, Mark Geller, Tatiana Matejovicova, Ayan Chakrabarti, Ryan Julian, Christine Chan, Qiong Hu, Daniel Jarrett, Manu Agarwal, Jeshwanth Challagundla, Tao Li, Sandeep Tata, Wen Ding, Maya Meng, Zhuyun Dai, Giulia Vezzani, Shefali Garg, Jannis Bulian, Mary Jasarevic, Honglong Cai, Harish Rajamani, Adam Santoro, Florian Hartmann, Chen Liang, Bartek Perz, Apoory Jindal, Fan Bu, Sungyong Seo, Ryan Poplin, Adrian Goedeckemeyer, Badih Ghazi, Nikhil Khadke, Leon Liu, Kevin Mather, Mingda Zhang, Ali Shah, Alex Chen, Jinliang Wei, Ke- shav Shivam, Yuan Cao. Donghyun Cho, Angelo Scorza Scarpati, Michael Moffitt, Clara Barbu, Ivan Jurin, Ming- Wei Chang, Hongbin Liu, Hao Zheng, Shachi Dave, Christine Kaeser-Chen, Xiaobin Yu, Alvin Abdagic, Lucas Gonzalez, Yanping Huang, Peilin Zhong, Cordelia Schmid, Bryce Petrini, Alex Wertheim, Jifan Zhu, Hoang Nguyen, Kaiyang Ji. Yanqi Zhou, Tao Zhou, Fangxiaoyu Feng, Regev Cohen, David Rim, Shubham Milind Phal, Petko Georgiev, Ariel Brand, Yue Ma, Wei Li, Somit Gupta, Chao Wang. Pavel Dubov, Jean Tarbouriech, Kingshuk Majumder, Huijian Li. Norman Rink, Apury Suman, Yang Guo, Yinghao Sun, Arun Nair, Xiaowei Xu, Mo- hamed Elhawaty, Rodrigo Cabrera, Guangxing Han, Julian Eisenschlos, Junwen Bai, Yuqi Li, Yamini Bansal, Thibault Sellam, Mina Khan, Hung Nguyen, Justin Mao-Jones, Nikos Parot- sidis, Jake Marcus, Cindy Fan, Roland Zimmermann, Yony Kochinski, Laura Graesser, Feryal Behbahani, Alvaro Caceres, Michael Riley, Patrick Kane, Sandra Lefdal, Rob Willoughby, Paul Vicol, Lun Wang. Shujian Zhang, Ashleah Gill, Yu Liang, Gautam Prasad, Soroosh Mariooryad, Mehran Kazemi, Zifeng Wang, Kritika Muralidharan, Paul Voigtlaender, Jeffrey Zhao Huanjie Zhou, Nina Douua, Aditi Mavalankar, Séb Amold, Nick Young. Obaid Sarvana, Chace Lec, Milad Nasr, Tingting Zou, Seokhwan Kim, Lukas Haas, Kaushal Patel, Neslihan Bulut, David Parkinson, Courtney Biles, Dmitry Kalashnikov, Chi Ming To, Aviral Kumar, Jessica Austin, Alex Greve, Lei Zhang, Megha Goel, Yeqing Li, Sergey Yaroshenko, Max Chang, Abhishek Jindal, Ge- off Clark, Hagai Taitelbaum, Dale Johnson, Ofir Roval, Jeongwoo Ko, Anhad Mohananey, Chris- tian Schuler. Shenil Dodhia, Ruichao Li, Kazuki Osawa, Claire Cui, Peng Xu, Rushin Shah, Tao Huang. Ela Gruzewska, Nathan Clement, Mudit Verma, Olcan Sercinoglu, Hai Qian, Viral Shah, Masa Yamaguchi, Abhinit Modi, Takahiro Kosakai, Thomas Strohmann, Junhao Zeng, Beliz Gunel, Jun Qian, Austin Tarango. Krzysztof Jastrzebski. Robert David, Jyn Shan, Parker Schuh, Kunal Lad, Willi Gierke, Mukundan Madhavan, Xinyi Chen, Mark Kurzeja, Rebeca Santamaria- Fernandez, Dawn Chen, Alexandra Cordell, Yuri Chervonyi, Frankie Garcia, Nithish Kannen, Vincent Perot, Nan Ding, Shlomi Cohen-Ganor, Victor Lavrenko, Junru Wu, Georgie Evans, Cicero Nogueira dos Santos, Madhavi Sewak, Ashley Brown, Andrew Hard, Joan Puigcerver, Zeyu Zheng, Yizhong Liang, Evgeny Gladchenko, Reeve Ingle, Uri First, Pierre Sermanet, Char- lotte Magister, Mihajlo Velimirović, Sashank Reddi, Susanna Ricco, Eirikur Agustsson, Hartwig Adam, Nir Levine, David Gaddy, Dan Holtmann-Rice. Xuanhui Wang, Ashutosh Sathe, Abhi- jit Guha Roy, Blaž Bratanič, Alen Carin, Harsh Mehta, Silvano Bonacina, Nicola De Cao, Mara Finkelstein, Verena Rieser, Xinyi Wu, Florent Altché, Dylan Scandinaro, Li Li, Nino Vieillard, Nikhil Sethi, Garrett Tanzer, Zhi Xing, Shibo Wang. Parul Bhatia, Gui Citovsky, Thomas An- thony, Sharon Lin, Tianze Shi, Shoshana Jakobovits, Gena Gibson, Raj Apte, Lisa Lee, Mingqing Chen, Arunkumar Byravan, Petros Maniatis, Kellie Webster, Andrew Dai, Pu-Chin Chen, Jiaqi Pan, Asya Fadeeva, Zach Gleicher, Thang Luong, and Niket Kumar Bhumihar. Gemini 2.5: Push-
21

---

Published as a conference paper at ICLR 2026
ing the frontier with advanced reasoning, multimodality,, long context, and next generation agentic capabilities, 2025. URL attts::/arxxis wwtp/oo////.........o...ooo.e//o ha/abbb/2500.. 06261
DeepSeek-AI, Daya Guo, Dejian Yang. Haowei Zhang. Junxiao Song, Peiyi Wang. Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in Ilms through reinforcement learning. Nature, 645(8081):63336333 2025a.
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangho Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song. Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang. Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang. Ruizhe Pan. Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shut- ing Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng. Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang. Xiaojin Shen, Xiaokang Chen, Xi- aokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song. Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su. Xuheng Lin, Y. K. Li, Y. Q. Wang. Y. X. Wei, Y. X. Zhu, Yang Zhang. Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao. Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong. Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang. Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2025b. URL
https::/arxiv... org/ abs// 2412 19437
Ryan Ehrlich, Bradley Brown, Jordan Juravsky, Ronald Clark, Christopher Ré, and Azalia Mirho- seini. Codemonkeys: Scaling test-time compute for software engineering, 2025. URI. https:
14723
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint arXiv:1805.048. 048333 2018
Emily First, Markus N. Rabe, Talia Ringer, and Yuriy Brun. Baldur: Whole-proof generation and repair with large language models, 2023. URL httt:::://axxx..rrtee org/ abs/ 2303. 04910
Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Worts- man, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Luca Soldaini, Alexandros G. Dimakis, Gabriel Ilharco, Pang Wei Koh, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muen- nighoff, and Ludwig Schmidt. Language models scale reliably with over-training and on down- stream tasks, 2024. ( https:t///// /ttp//aaxivvv/v////vv//ea. org/abs abs/ 2403.4033..0 08540
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL
org/
abs/2101. 00027
Leo Gao, Jonathan Tow, Baber Abbasi. Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang
22

---

Published as a conference paper at ICLR 2026
Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model httttp:://zenod.org///o
evaluation harness, 07 2024.
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. In International Conference on Learning Representations, 2021.
Nathan Godey, Éric de la Clergerie, and Benoit Sagot. Why do small language models un- derperform? studying language model saturation via the softmax bottleneck, 2024. URL
https://arxiv: wttp/rr/////oo/oooo 0 07647
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Ko- renev, Arthur Hinsvark. Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dicuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee. Georgia Lewis Anderson, Govind That- tai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Kore- vaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang. Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Ject Shah, Jelmer van der Linde. Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jong- soo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafell, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Kantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Ku- mar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoy- chev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic. Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ra- mon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Ro- hit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mi- haylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Gold- schlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song. Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi. Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, An- drew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, An- nie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leon- hardi, Bernie Huang. Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Mon- talvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao
23

---

Published as a conference paper at ICLR 2026
Zhou, Chester Hu, Ching- Hisingg Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Flaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smoth- ers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayannn, Hakan Inan, Hamid Shojanazeri. Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harri- son Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman. Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang. Jen- nifer Chan. Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Jun- jie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer. Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Ro- driguez, Rafi Ayub, Rughotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan. Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto. Sharadh Ra- maswamy, Shaun Lindsay. Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang. Shuqiang Zhang. Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen. Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Ku- mar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen liang. Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiao- jian Wu, Xiaolan Wang. Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi. Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Vito, Zef Rosnbrick, Zhao- duo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL btttps:/arxxe org/ htw.gg/ab//2400 21783
Michael Hassid, Tal Remez, Jonas Gehring, Roy Schwartz, and Yossi Adi. The larger the better? im- proved Ilm onde-generation via budget reallocation. In First Conference on Language Modeling, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701, 2020.
24

---

Published as a conference paper at ICLR 2026
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nel- son Elhage, Zac Surfield-Doddee Tom Henighan, Tristan Hume, et al. Scaling laws and inter- pretability of learning from repeated data. arXiv preprint arXiv:2205. 10487, 2022.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017. URL /tttps://arxtin www.voorr//aooo abs/ 1712. 00409
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoe, Au- relia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Lau- rent Sifre. An empirical analysis of compute-optimal large language model training. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 30016-30038 Curran Associates, Inc., 2022a. URL httppss//proceeedingss.stt www.::inn.........ee.ee..e..r.ee cc/ paper Serapr__files/ee fwwes//appr/ Sow.aaaeeeeee.a.eaaaaaeeeeaaeee.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Flena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022b. URL https:// wttp:///axvvvvvo org/ abs/ 22033.1555
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference Learning Representations, 2020.
Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, Zhiyuan Liu, and Maosong Sun. Predicting emergent abilities with infinite resolution evaluation, 2024. URL httpp:s//aaxvv.ooom htt.:teee//aaatooooo//////////// 032262
John Hughes, Sara Price, Aengus Lynch, Rylan Schaeffer, Fazl Barez, Sanmi Koyejo, Henry Sleight, Erik Jones, Ethan Perez, and Mrinank Sharma. Best-of-n jailbreaking, 2024. URL https:
//arxiv.org/aabe waaprr//bbb//a0 /2412 03556
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020a.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020b. URL http::///a//x.......t.
08361
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems, 32, 2019.
Jacky Kwok, Christopher Agia, Rohan Sinha, Matt Foutter, Shulu Li, Ion Stoica, Azalia Mirho- seini, and Marco Pavone. Robomonkey: Scaling test-time sampling and verification for vision- language-action models, 2025. URL http:/tsss/s.ii......a
/2500 17811
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Noam Itzhak Levi. A simple model of inference scaling laws. In Forty-second International Con- ference Machine Learning, 2025.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling cons-constr language models. Advances in Neural Information Processing Systems, 36:50358-50376 2023.
25

---

Published as a conference paper at ICLR 2026
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji. Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadert-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brock- man, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gib- son, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha GontijoLopps, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hal- lacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Ka- mali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim. Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis. Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy. Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz. Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing. Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang. Cullen O' OeKeefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder. Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song. Natalie Staudacher. Felipe Petroski Such, Natalic Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng. Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich. Hannah Wong, Lauren Work- man, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024a. URL. https://erxiv org/ abs /tt./////3333 2203 08774
OpenAl, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16721 2024b.
Tim Pearce and Jinyeop Song. Reconciling kaplan and chinchilla scaling laws. Transactions on Machine Learning Research, 2024.
Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-optimal scaling of language models. Advances in Neural Information Processing Systems, 37:100535-1005777 2024.
26

---

Published as a conference paper at ICLR 2026
Qwen, An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan. Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL
https::/arxiv.
htpp///s//2//00 09388
Daniel A Roberts, Sho Yaida, and Boris Hanin. The principles of deep learning theory, volume 46. Cambridge University Press Cambridge, MA, USA, 2022.
Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond ohinchilll-optimall Ac- counting for inference language model scaling laws. International Conference on Machine Learning, pp. 43445-43460. PMLR, 2024.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Pro- cexsing Systems, volume 36, pp. 55565-55581. Curran Associates, Inc., 2023. URL
large language models mirage?
Rylan Schaeffer, Hailey Schoelkopf, Brando Miranda, Gabriel Mukobi, Varun Madan, Adam Ibrahim, Herbie Bradley, Stella Biderman, and Sanmi Koyejo. Why has predicting down- stream capabilities of frontier ai models with scale remained elusive?, 2024a. URL https:
//arxiv.org/aaneeev /206 04391
Rylan Schaeffer, Hailey Schoelkopf. Brando Miranda, Gabriel Mukobi, Varun Madan, Adam Ibrahim, Herbie Bradley, Stella Biderman, and Sanmi Koyejo. Why has predicting down- stream capabilities of frontier ai models with scale remained elusive?, 2024b. URL https:
//atxiv.org///e/ee...e....... http///aa./44... 04391
Rylan Schaeffer, Joshua Kazdan, and Yegor Denisov-Blanche Min-p, max exaggeration: critical analysis of min-p sampling in language models, 2025a. URL htttp::/aaxiv..ortt
2506. 13681
Rylan Schaeffer, Joshua Kazdan, John Hughes, Jordan Juravsky, Sara Price, Aengus Lynch, Erik Jones, Robert Kirk, Azalia Mirhoseini, and Sanmi Koyejo. How do large language monkeys get their power (laws)? In Forty- International Conference on Machine Learning, 2025b. URL
net/ for.m/ooummmmmmm -dd2221222Qemmm
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024. URL. htttps://arxiv.ot
abs / 2208.40... 03314
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lilli- crap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hen- nigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Jack Krawezyk, Cosmo Du Ed Chi. Heng-Tze Cheng, Eric Ni, Purvi Shah, Patrick Kane, Betty Chan, Manaal Faruqui, Aliaksei Severyn, Hanzhao Lin, YaGuang Li. Yong Cheng, Abe Ittycheriah, Mahdis Mahdieh, Mia Chen, Pei Sun, Dustin Tran, Sumit Bagri, Balaji Lakshminarayanan, Jeremiah Liu, Andras Orban, Fabian Güra, Hao Zhou, Xinying Song, Aurelien Boffy, Harish Ganapa- thy, Steven Zheng, HyunJeong Choe, Ágoston Weisz, Tao Zhu, Yifeng Lu, Siddharth Gopal, Jarrod Kahn, Maciej Kula, Jeff Pitman. Rushin Shah, Emanuel Taropa. Majd Al Merey, Mar- tin Baeuml, Zhifeng Chen, Laurent El Shafey, Yujing Zhang, Olean Sercinoglu, George Tucker,
27

---

Published as a conference paper at ICLR 2026
Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anais White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lu- cas Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura Culp, Lev Proleev, Yi Luan, Xi Chen, James Lottes, Nathan Schucher, Federico Lebron, Alban Rrustemi, Natalie Clay, Phil Crone, Tomas Kocisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rae, Han Lu, Laurent Sifre, Marcello Maggioni, Fred Alcober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun Ahuja, Gaurav Singh Tomar, Evan Senter, Martin Chadwick, Ilya Kornkkov, Nithya Attaluri, Iñaki Iturrate. Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjje Gu, Qiao Zhang. Jordan Grimstad, Ale Jakse Hart- man, Xavier Garcia, Thanumalayan Sankaranarayana Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adrià Puigdomènech Badia, David Re- itter, Mianna Chen, Jenny Brennan, Clara Rivera, Sergey Brin, Shariq lqbal, Gabricla Surita, Jane Labanowski, Abhi Rao, Stephanie Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Ravi Addanki, Antoine Miech, Annie Louis, Denis Teplyashin, Geoff Brown, Elliot Catt, Jan Bala- guer, Jackie Xiang, Pidong Wang. Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Gana- pathy, Smit Sanghavi, Ajay Kannan, Ming-Wei Chang, Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang, Juliette Love, Junwhan Ahn, Dawn Bloxwich, Kehang Han, Peter Humphreys, Thibault Sellam, James Bradbury. Varun Godbole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, Sébastien M. R. Arold, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanburn, Srivatsan Srinivasan, Hyeontaek Lim, Sarah Hodkinson, Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Paine, Jian Li, Yujia Li, Minh Giang, Alexander Neitz, Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery, Dipan- jan Das, Dominika Rogozińskka, Vitaliy Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu, Raoul de Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Albin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo, Craig Swanson, Dessie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Lan- don, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai Giménez, Legg Yeung, James Keeling, Petko Georgiev, Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputro, Kiran Vodrahalli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fer- nando, Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang, Fan Yang, Shuo yünn Chang, Paul Komarek, Ross Mcllroy, Mario Lučic. Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakeri, Christina Butterfield, Justin Chung. Paul Kishan Rubenstein, Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Karel Lenc, Timothy Chung, Acdan Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang. Joshua Maynez, Mary Phuong. Taylor Tobin, Andrea Tacchetti, Maja Trebacz, Kevin Robinson, Yash Katariya, Sebastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora Aroyo, Ambrose Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara, Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko linuma, Polina Zablotskaia, James Besley, Da- WooWoon Chung, Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su. Martin Polacek, Raphaël Lopez Kaufmann Simon Tokumine, Hexi- ang Hu, Elena Buchatskaya, Yingjie Miao, Mohamed Elhawaty, Aditya Siddhant, Nenad Toma- sev, Jinwei Xing. Christina Greer, Helen Miller, Shereen Ashraf, Aurko Roy, Zizhao Zhang, Ada Ma, Angelos Filos, Milos Besta, Rory Blevins, Ted Klimenko, Chih-Kuan Yeh, Soravit Chang- pinyo, Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Sholto Douglas, Rajkumar Samuel, Mingqiu Wang, Sophia Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjösund, Sébastien Cevey, Zach Gleicher, Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, Léonard Hussenot, Livio Baldini Soares, Kate Baumli, Michael B. Chang, Adrià Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kass- ner, Subhrajit Roy, Ethan Dyer, Victor Campos Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy. Elspeth White, Basil Mustafa, Oran Lang, Abhishek Jindal, Sharad Vikram, Zhitao Gong, Sergi Caelles, Ross Hemsley, Gregory Thomton, Fangxiaoyu Feng, Wojciech Stokowiec,
28

---

Published as a conference paper at ICLR 2026
Ce Zheng, Phoebe Thacker, Çağlar Ünlü, Zhishuai Zhang, Mohammad Saleh, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez. Tom Kwiatkowski, Samira Daruki, Keran Rong, Allan Dafoe, Nicholas FitzGeral,, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pel- lat, Vladimir Feinberg, James Cobon-Kerr, Tara Sainath, Mariheth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Hasson, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Paganini, Jean-Baptiste Lespiau, Alexandre Moufarek, Samer Has- san, Kaushik Shivakumar, Joost van Amersfoort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Sheahan, Vedant Misra, Cheng Li, Nemanja Rakićević, Mostafa Dehghani, Fangyu Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlie Chen, Sidharth Mudgal, Romina Stella, Kevin Brooks, Gautam Vasudevan, Chenxi Liu, Mainak Chain, Nivedita Melinkeri, Aaron Cohen, Venus Wang, Kristic Seymore, Sergey Zubkov, Rahul Goel, Summer Yue, Sai Krishnakumaran, Brian Al- bert, Nate Hurley, Motoki Sano, Anhad Mohananey. Jonah Joughin, Egor Filonov, Tomasz Kepa, Yomna Eldawy, Jiawern Lim, Rahul Rishi, Shirin Badiezadegan, Taylor Bos, Jerry Chang, Sanil Jain, Sri Gayatri Sundara Padmanabhan, Subha Puttagunta, Kalpesh Krishna, Leslie Baker, Nor- bert Kalb, Vamsi Bedapudi, Adam Kurzrok, Shuntong Lei, Anthony Yu, Oren Litvin, Xiang Zhou, Zhichun Wu, Sam Sobell, Andrea Siciliano, Alan Papir, Robby Neale, Jonas Bragagnolo, Tej Toor, Tina Chen, Valentin Anklin, Feiran Wang. Richie Feng. Milad Gholami. Kevin Ling. Lijuan Liu, Jules Walter, Hamid Moghaddam, Arun Kishore, Jakub Adamek, Tyler Mercado, Jonathan Mallinson, Siddhinita Wandekar. Stephen Cagle, Eran Ofek. Guillermo Garrido. Clemens Lom- briser, Maksim Mukha, Botu Sun, Hafeezul Rahman Mohammad, Josip Matak, Yadi Qian, Vikas Peswani, Pawel Janus, Quan Yuan, Leif Schelin, Oana David, Ankur Garg, Yifan He, Oleksii Duzhyi, Anton Algmyr. Timothée Lottaz, Qi Li, Vikas Yadav, Luyao Xu, Alex Chinien, Rakesh Shivanna, Aleksandr Chuklin, Josie Li, Carrie Spadine, Travis Wolfe, Kareem Mohamed, Sub- habrata Das, Zihang Dai, Kyle He, Daniel von Dincklage, Shyam Upadhyay, Akanksha Mau- rya, Luyan Chi, Sebastian Krause, Khalid Salama, Pam G Rabinovitch, Pavan Kumar Reddy M, Aarush Selvan, Mikhail Dektiarev, Golnaz Ghiasi, Erdem Guven, Himanshu Gupta, Boyi Liu, Deepak Sharma, Idan Heimlich Shtacher, Shachi Paul, Oscar Akerlund, trançois-Xavier Aubet, Terry Huang. Chen Zhu, Eric Zhu, Elico Teixeira, Matthew Fritze, Francesco Bertolini, Liana- Elconora Marinescu, Martin Bölle, Dominik Paulus, Khyatti Gupta, Tejasi Latkar, Max Chang, Jason Sanders, Roopa Wilson, Xuewei Wu, Yi-Xuan Tan, Lam Nguyen Thiet, Tulsee Doshi, Sid Lall, Swaroop Mishra, Wanming Chen, Thang Luong, Seth Benjamin, Jasmine Lee, Ewa Andrejczuk, Dominik Rabiej. Vipul Ranjan, Krzysztof Styre, Pengcheng Yin, Jon Simon, Mal- colm Rose Harriott, Mudit Bansal, Alexei Robsky, Geoff Bacon, David Greene, Daniil Mirylenka, Chen Zhou, Obaid Sarvana, Abhimanyu Goyal, Samuel Andermatt, Patrick Siegler, Ben Horn, Assaf Israel, Francesco Pongetti, Chih- Wei "Louis" Chen, Marco Selvatici, Pedro Silva, Kathie Wang, Jackson Tolins, Kelvin Guu, Roey Yogev, Xiaochen Cai, Alessandro Agostini, Maulik Shah, Hung Nguyen, Noah O Donnaile, Sébastien Percira, Linda Friso, Adam Stambler, Adam Kurzrok, Chenkai Kuang, Yan Romanikhin, Mark Geller, ZJ Yan, Kane Jang, Cheng- Chun Lee, Wojciech Fica, Eric Malmi, Qijun Tan, Dan Banica, Daniel Balle, Ryan Pham, Yanping Huang, Diana Avram, Hongzhi Shi, Jasjot Singh. Chris Hidey, Niharika Ahuja, Pranab Saxena, Dan Doo- ley, Srividya Pranavi Potharaju, Eileen O'Neill, Anand Gokulchandran, Ryan Foley, Kai Zhao, Mike Dusenberry, Yuan Liu, Pulkit Mehta, Ragha Kotikalapudi, Chalence Seffanek-Shradde An- drew Goodman, Joshua Kessinger, Eran Globen, Prateck Kolhar, Chris Gorgolewski, Ali Ibrahim, Yang Song. Ali Eichenbaum, Thomas Brovelli, Sahitya Potluri, Preethi Lahoti, Cip Bactu, Ali Ghorbani, Charles Chen, Andy Crawford, Shalini Pal, Mukund Sridhar, Petru Gurita, Asier Mu- jika, Igor Petrovski, Pierre-Louis Cedoz, Chenmei Li, Shiyuan Chen, Niccolò Dal Santo, Sid- dharth Goyal, Jitesh Punjabi, Karthik Kappaganthu, Chester Kwak, Pallavi LV, Sarmishta Velury, Himadri Choudhury, Jamie Hall, Premal Shah, Ricardo Figueira, Matt Thomas, Minjie Lu, Ting Zhou, Chintu Kumar, Thomas Jurdi, Sharat Chikkerur, Yenai Ma, Adams Yu, Soo Kwak, Victor Ähdel, Sujeevan Rajayogam. Travis Choma, Fei Liu, Aditya Barua, Colin Ji, Ji Ho Park, Vincent Hellendoom, Alex Bailey, Taylan Bilal, Huanjie Zhou, Mehrdad Khatir, Charles Sutton, Woj- ciech Rzadkowski, Fiona Macintosh, Konstantin Shagin, Paul Medina, Chen Liang, Jinjing Zhou, Pararth Shah, Yingying Bi, Attila Dankovics, Shipra Banga, Sabine Lehmann, Marissa Bredesen, Zifan Lin, John Eric Hoffmann, Jonathan Lai, Raynald Chung, Kai Yang. Nihal Balani, Arthur Bražinskas, Andrei Sozanschi. Matthew Hayes, Héctor Fernández Alcalde, Peter Makarov, Will Chen, Antonio Stella, Liselotte Snijders, Michael Mandl, Ante Kärrman, Pawel Nowak, Xinyi Wu, Alex Dyck, Krishnan Vaidyanathan, Raghavender R, Jessica Mallet, Mitch Rudominer, Eric
29

---

Published as a conference paper at ICLR 2026
Johnston, Sushil Mittal, Akhil Udathu, Janara Christensen, Vishal Verma, Zach Irving, Andreas Santucci, Gamaleldin Elsayed, Elnaz Davoodi, Marin Georgiev, Ian Tenney, Nan Hua, Geoffrey Cideron, Edouard Leurent, Mahmoud Alnahlawi, lonut Georgescu, Nan Wei, Ivy Zheng, Dylan Scandinaro, Heinrich Jiang, Jasper Snoek, Mukund Sundararajan, Xuezhi Wang. 7ack Ontiveros, Itay Karo, Jeremy Cole, Vinu Rajashekhar, Lara Tumeh, Eyal Ben-David, Rishub Jain, Jonathan Uesato, Romina Datta, Oskar Bunyan, Shimu Wu, John Zhang, Piotr Stanczyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew Johnson, Adam Paszke, Chung-Cheng Chiu, Jaume Sanchez Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, Andrew Lee, Nino Vieil- lard, Jane Park, Jiageng Zhang. Jeff Stanway. Drew Garmon, Abhijit Karmarkar, Zhe Dong, Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens, William Isaac, Geoffrey Irving, Edward Loper, Michael Fink, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Ivan Petrychenko, Zhe Chen, John- son Jia, Anselm Levskaya, Zhenkai Zhu, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao, Javier Snaider, Norman Casagrande, Evan Palmer, Paul Suganthan, Alfonso Castaño, Irene Giannoumis, Wooyeol Kim, Mikolaj Rybiński, Ashwin Sreevatsa, Jennifer Prendki, David So- ergel, Adrian Goedeckemeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Di- ana Gage Wright, Yawen Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Le, Lu Li, Chimezie Iwuanyanwu, Lu Liu, Kevin Ramirez, Andrey Khorlin, Albert Cui, Tian LIN, Mar- cus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Ginger Perng. Elena Allica Abellan, Mingyang Zhang, Ishita Dasgupta, Nate Kushman, Ivo Penchev, Alena Repina, Xihui Wu, Tom van der Weide, Priya Ponnapalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li. Olivier Dousse, Fan Yang, Jeff Piper, Nathan Ic. Rama Pasumarthi. Nathan Lintz, Anitha Vijayakumar, Daniel Andor, Pedro Valenzuela, Minnie Lui, Cosmin Paduraru, Daiyi Peng, Katherine Lee, Shuyuan Zhang, Somer Greene, Due Dung Nguyen, Paula Kurylowicz, Cassidy Hardin, Lucas Dixon, Lili Janzer, Kiam Choo, Ziqiang Feng. Biao Zhang, Achintya Singhal, Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolukbasi, Orgad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Ken Franko, Anna Bu- lanova, Rémi Leblond, Shirley Chung, Harry Askham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Alek Dimitriev, Han- nah Forbes, Dylan Banarse, Zora Tung. Mark Omernick, Colton Bishop. Rachel Sterneck, Rohan Jain, Jiawei Xia, Ehsan Amid, Francesco Piccinno, Xingyu Wang, Praseem Banzal, Daniel J. Mankowitz, Alex Polozov, Victoria Krakovna, Sasha Brown, MohammadHossein Bateni. Den- nis Duan, Vlad Firoiu, Meghana Thotakuri, Tom Natan, Matthieu Geist, Ser tan Girgin, Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee-Thorp, Christopher Yew, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Kathy Wu, David Miller, Nico- las Sonnerat, Denis Vnukov, Rory Greig, Jennifer Beattie, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex Korchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu, Fan Yang. Rui Zhu, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Neje Trdin, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting Xue, Chen Elkind, Oliver Woodman, John Carpenter, George Papamakarios, Rupert Kemp, Sushant Kafle, Tanya Grunina, Rishika Sinha, Alice Talbert, Diane Wu, Denese Owusu-Afriyie, Cosmo Du, Chloe Thornton, Jordi Pont-Tuset, Pradyumna Narayana, Jing Li, Saaber Fatehi, John Wieting, Omar Ajmeri, Be- nigno Uria, Yeongil Ko, Laura Knight, Amélie Héliou, Ning Niu, Shane Gu, Chenxi Pang, Yeqing Li, Nir Levine, Ariel Stolovich, Rebeca Santamariaa-Fernaddez Sonam Goenka, Wenny Yustalim, Robin Strudel, Ali Elqursh. Charlie Deck. Hyo Lee, Zonglin Li, Kyle Levin, Raphael Hoffmann, Dan Holtmann- Rice, Olivier Bachem, Sho Arora, Christy Koh, Soheil Hassas Yeganeh, Siim Pöder, Mukarram Tariq. Yanhua Sun, Lucian Ionita. Mojtaba Seyedhosseimi. Pouya Tafti, Zhiyu Liu, Anmol Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszc7, Lily Wang, Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Vinod Koverkathu, Christo- pher A. Choquette-Choo, Yunjie Li, TJ Lu, Abe Ittycheriah, Prakash Shroff, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Guillaume Desjardins, Marco Cornero, Brona Robenek, Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Rivière, Alanna Walton, Clément Crepy, Alicia Parrish, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Eri Latorre-Chimoto, Hanna Ulimezak-Plucinset Dario de Cesare, Tom Hudson, Piermaria Mendolicchio, Lexi Walker, Alex Morris, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe, Anastasia Petrushkina, Tom Ducrig, Antonio Sanchez, Steve Yad- lowsky, Amy Shen, Amir Globerson, Lynette Webb, Sahil Dua, Dong Li, Surya Bhupatiraju, Dan Hurt, Haroon Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Anuj Khare, Shreyas Ram-
David Bridson,
30

---

Published as a conference paper at ICLR 2026
mohan Belle, Lei Wang, Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Saeta, Tyler Liechty, Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuyyuru, John Aslanides, Nidhi Vyas, Martin Wicke, Xiao Ma, Evgenii Eltyshev, Nina Martin, Hardie Cate, James Manyika, Keyvan Amiri, Yelin Kim, Xi Xiong. Kai Kang, Florian Luisier, Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang, Riham Mansour, Jason Gelman, Yang Xu, George Polovets, Ji Liu, Honglong Cai, Warren Chen, XiangHai Sheng, Emily Xue, Sherjil Ozair, Christof Angermueller, Xiaowei Li, Anoop Sinha, Weiren Wang, Julia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Iyer. Madhu Gurumurthy, Mark Gold- enson, Parashar Shah, MK Blake, Hongkun Yu, Anthony Urbanowicz, Jennimaria Palomaki, Chrisantha Fernando, Ken Durden, Harsh Mehta, Nikola Momehex, Elahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sebastian Ruder, Morgan Redshaw, Jinhyuk Lee, Denny Zhou, Komal Jalan, Dinghua Li, Blake Hechtman, Parker Schuh, Milad Nasr, Kieran Milan, Vladimir Miku- lik, Juliana Franco, Tim Green, Nam Nguyen, Joe Kelley, Aroma Mahendru, Andrea Hu, Joshua Howland, Ben Vargas, Jeffrey Hui, Kshitij Bansal, Vikram Rao, Rakesh Ghiya, Emma Wang, Ke Ye, Jean Michel Sarr, Melanie Moranski Preston, Madeleine Elish, Steve Li, Aakash Kaku, Jigar Gupta, Ice Pasupat, Da-Cheng Juan, Milan Someswar, Tejvi M.. Xinyun Chen, Aida Amini, Alex Fabrikant, Eric Chu, Xuanyi Dong, Amruta Muthal, Senaka Buthpitiya, Sarthak Jauhari, Nan Hua, Urvashi Khandelwal, Ayal Hitron, Jie Ren, Larissa Rinaldi, Shahar Drath, Avigail Dabush, Nan-Jiang Jiang, Harshal Godhia, Uli Sachs, Anthony Chen, Yicheng Fan, Hagai Taitelbaum, Hila Noga, Zbuyun Dai. James Wang. Chen Liang, Jenny Hamer, Chun-Sung Ferng, Chenel Elkind, Aviel Atias, Paulina Lee, Vít Listík, Mathias Carlen, Jan van de Kerkhof, Marcin Pikus, Krunoslav Zaher, Paul Müller, Sasha Zykova, Richard Stefanec, Vitaly Gatsko, Christoph Hirn- schall, Ashwin Sethi, Xingyu Federico Xu, Chetan Ahuja, Beth Tsai, Anca Stefanoiu, Bo Feng, Keshav Dhandhania, Manish Katyal, Akshay Gupta, Atharva Parulekar, Divya Pitta, Jing Zhao, Vivaan Bhatia, Yashodha Bhavnani, Omar Alhadlaq, Xiaolin Li, Peter Danenberg, Dennis Tu, Alex Pine, Vera Filippova, Abhipso Ghosh, Ben Limonchik, Bhargava Urala, Chaitanya Krishna Lanka, Derik Clive, Yi Sun, Edward Li, Hao Wu, Kevin Hongtongsak, Ianna Li, Kalind Thakkar, Kuanysh Omarov, Kushal Majmundar, Michael Alverson, Michael Kucharski, Mohak Patel, Mu- dit Jain, Maksim Zabelin, Paolo Pelagatti, Rohan Kohli, Saurabh Kumar, Joseph Kim, Swetha Sankar, Vineet Shah, Lakshmi Ramachandruni, Xiangkai Zeng, Ben Bariach, Laura Weidinger, Tu Vu, Alek Andreev, Antoine He, Kevin Hui, Sheleem Kashem, Amar Subramanya, Sissie Hsiao, Demis Hassabis, Koray Kavukcuoglu, Adam Sadovsky, Quoc Le, Trevor Strohman, Yonghui Wu, Slav Petrov, Jeffrey Dean, and Oriol Vinyals. Gemini: A family of highly capable multimodal models, 2024. URL http::tttt//t/ //tpiv///xvvvvv.a
11805
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models, 2024.
00724
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 12104-12113, 2022.
Zhipu-AI, Aohan Zeng, Xin Lv. Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xic, Cunxi- ang Wang, Da Yin, Hao Zeng, Jiajie Zhang. Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, Yean Cheng, Yifan An, Yilin Niu, Yuanhao Wen, Yushi Bai, Zhengxiao Du, Zihan Wang, Zilin Zhu, Bohan Zhang, Bosi Wen. Bowen Wu, Bowen Xu, Can Huang, Casey Zhao, Changpeng Cai, Chao Yu, Chen Li, Chendi Ge, Chenghua Huang, Chenhui Zhang, Chenxi Xu, Chenzheng Zhu, Chuang Li, Congfeng Yin. Daoyan Lin, Dayong Yang. Dazhi Jiang, Ding Ai. Erle Zhu, Fei Wang, Gengzheng Pan, Guo Wang, Hailong Sun. Haitao Li, Haiyang Li. Haiyi Hu, Hanyu Zhang, Hao Peng, Hao Tai, Haoke Zhang, Haoran Wang, Haoyu Yang, He Liu, He Zhao, Hongwei Liu, Hongxi Yan, Huan Liu, Hui- long Chen, Ji Li, Jiajing Zhao, Jiamin Ren, Jian Jiao, Jiani Zhao, Jianyang Yan, Jiaqi Wang. Jiayi Gui, Jiayue Zhao, Jie Liu, Jijie Li, Jing Li. Jing Lu, Jingsen Wang, Jingwei Yuan, Jingxuan Li, Jingzhao Du, Jinhua Du, Jinxin Liu, Junkai Zhi, Junli Gao. Ke Wang, Lekang Yang, Liang Xu, Lin Fan, Lindong Wu, Lintao Ding, Lu Wang, Man Zhang, Minghao Li, Minghuan Xu, Mingming Zhao, Mingshu Zhai, Pengfan Du, Qian Dong, Shangde Lei, Shangqing Tu, Shangtong Yang, Shaoyou Lu, Shijie Li, Shuang Li. Shuang-Li, Shuxun Yang, Sibo Yi, Tianshu Yu. Wei Tian, Weihan Wang, Wenbo Yu, Weng Lam Tam, Wenjie I Liang, Wentao Liu, Xiao Wang, Xiaohan Jia,
31

---

Published as a conference paper at ICLR 2026
Xiaotao Gu, Xiaoying Ling, Xin Wang. Xing Fan, Xingru Pan, Xinyuan Zhang, Xinze Zhang, Xiuqing Fu, Xunkai Zhang, Yabo Xu, Yandong Wu, Yida Lu, Yidong Wang, Yilin Zhou, Yiming Pan. Ying Zhang, Yingli Wang, Yingru Li. Yinpei Su, Yipeng Geng, Yitong Zhu, Yongkun Yang, Yuhang Li, Yuhao Wu, Yujiang Li, Yunan Liu, Yunqing Wang. Yuntao Ii, Yuxuan Zhang, Zezhen Liu, Zhen Yang, Zhengda Zhou, Zhongpei Qiao, Zhuoer Feng, Zhuorui Liu, Zichen Zhang, Zi- han Wang, Zijun Yao, Zikang Wang, Ziqiang Liu, Ziwei Chai, Zixuan Li, Zuodong Zhao, Wen- guang Chen, Jidong Zhai, Bin Xu, Minlie Huang, Hongning Wang, Juanzi Li, Yuxiao Dong, and Jie Tang. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025. URL https://arxiv org/ /208. 06471
32

---

Published as a conference paper at ICLR 2026
A RELATED WORK
The study of neural scaling laws, which characterize how model performance predictably improves with resources, has become a cornerstone of modern large-scale machine learning (Hestness al.) 2017 Kaplan et al. 2020a Hoffmmann et al. 2022a Roberts et al. [022 Bahri et al., 2024). Foun- dational work demonstrated that pretraining loss follows power law with respect to model size, dataset size, and training compute (Kaplan et al.] 2020a Henighan et al.] 2020: Hoffmann et al.)
2022a). These principles now guide the development of nearly all state-of of-ee---ttetttt language mod- els (OpenAl et al. 2024a Team et al. 2024 Grattafiori et al., 2024 DeepSeek-Al et al., 2025b Qwen et al. 2025] Zhipu-Al et al. 2025). This research area has since expanded to explore scaling properties in diverse domains, including computer vision (Zhai et al. 2022), machine translation (Ghorbani et al. 2021), and even for optimizing inference-time compute rather than training (Wu et at 2024 Snell et al. 2024). Despite this breadth, scaling laws for complex, multi-step generative tasks which often represent the ultimate goal of language modeling- have remained compara- tively underexplored due to the challenges of evaluation. Our work builds most directly on recent efforts to forecast performance on such generative evaluations. The GPT-4 Technicul Report pio- neered this direction by showing that the negative log pass rate subset of the HvmanEval coding benchmark scaled predictably as power law of pretraining compute (OpenAl et al. 2024a); they successfully predicted GPT-4's performance by extrapolating from smaller models over approxi- mately three orders of magnitude (OpenAl et al. 2024a). We adopt their proposed functional form for our compute-basee based law and extend their work by systematically analyzing how the number of attempts, k, reshapes the law's parameters. Furthermore, our rigorous backtesting reveals the brittle- ness of this compute-only law, finding it reliable only across 1-2 orders of magnitude in our setting. Separately, work shares philosophical motivation with Hu et al. (2024). who investigate "infi- nite resolution" sampling for emergent abilities.
Relationship Between Pass-at-k and Accuracy: In the context of language model evaluations, accuracy and pass k are fundamentally different. "Accuracy" is a likelihood comparison: does the correct multiple-choice answer have the highest probability relative the incorrect alternative answers? (Gao et a 2024) [Schafffer et al.] (024a).. In contrast, pass@k measures the success of free-form generative sampling. To make the distinction clear, here are three obvious differences:
1. Accuracy is either binary 0 or 1, whereas pass@k k is continuous quantity in [0, I] and practically almost always in (0, 1). The two metrics have different output spaces.
2. As functions, the two metries have different inputs: accuracy answer and the incorrect answer, whereas pass@k k does not depend on the incorrect answer.
function of the correct
3. Accuracy is computed using the exact sequence of the correct answer via teacher forcing. In comparison, pass @ can sample any sequence of tokens.
Detailed Comparison with Gadre et al. (2024): On the empirical side, the 46 downstream tasks in Gadre et al. (2024) fall into two categories (see their Table 5): (1) pretraining losses (i.e., next- token prediction), (2) discriminative evaluations: measuring accuracy multiple-choice tasks (c.g., MMLU). Our work focuses on a third, distinct category: generative evaluation, scored with pass @ k.
On the theoretical side, starting point is the standard separable (N, D)) scaling law of Hoffmann et al. (202aa) (L(N, D) z E AN-a + BD- 9). Gadre et al. 2024) show in their Appendix B that this form implies compute power law with exponent (ax3/(a (a + B)), and that departures from the compute-optimal allocation modify only the prefactor as function of an over-training parameter (m). Our derivation Appendix is a direct generalization of this result to the relevant quantity for inference scaling (-log pass@k) and to arbitrary (k): we retain the same ((NDD) ansatz, derive the compute-optimal envelope and its exponent (a(k) (L/3(k) + 1/>(k)) and explicitly express how sub-optimal choices of ((N, D)) enter as a multiplicative penalty depending on the ratio (N/N*).
33

---

Published as a conference paper at ICLR 2026
B METHODOLOGY; NUMBER OF SAMPLES PER PROBLEM PER MODEL
Because sampling from language models is expensive and because our compute budget limited, chose a compule-aware sampling strategy designed balance resolution with cost. A key con- sideration is that the number of samples per problem sets the resolution, i.e., we draw n samples per problem, then any pass rate on that problem below 1/n will likely appear be O.
For each model checkpoint, we took the following strategy: we drew minimum of 2 samples per model per problem, and then continued sampling until 10 successes were obtained until 2 maximum of 2 samples were drawn. 2 was chosen so that we would have more samples per problem than the largest number of attempts per problem we considered (1e4).
Fig. 9 shows how many samples per GSM8K problem per model we had at the time our analyses were conducted, and Fig. [0 shows how many samples per MATH problem per model had at the time our analyses were conducted. In total, we had 500M total samples for GSM8K and 400M total samples MATH. Additional sampling would have likely benefited our analyses.
Pyla,,
Priw RCM
Pelogen zaM
Prtte 16554
Pyllag wilglt
Py Pyllemeii
Pebba p00
www.mm
Potti 2010
11 2010
tntiiga00.00 2010
Figure 9: Number of Samples per Model per GSM8K Problem. This heatmap visualizes the
number samples per model (y-axis) per problem (x-axis) at the time our analyses were conducted.
34

---

Published as a conference paper at ICLR 2026
Pycaa 4L94
jaw.eaa.. pas
29020190.0
Pettag 200000
Drtwisa.10 2194
Prtbul
Figure 10: Number of Samples per Model per MATH Problem. This heatmap visualizes the number of samples per model (y-axis) per problem axis) at the time our analyses were conducted.
35

---

Published as a conference paper at ICLR 2026
C ESTIMATION OF TOTAL SAMPLING COST
Sampling was performed using vLLM (Kwon et al. 2023). We generated 128 samples of the same problem at time make use of prefix caching for more efficient sampling.
C.I ESTIMATION GSM8K SAMPLING COSTS
Estimates of the GPU time per 128 samples per model size for GSM8K are provided in Table 1
Pythia Model Parameters
A100-s Per 128 Samples (Approximate)
14M
70M
160M
410M
IB
9
1.4B
11
2.8B
13
6.9B
12B
36
Table 1: A A100-Seconds Per 128 Samples by Model Size GSM8K Problems.
At minimum 3.2e4 samples per problem, 128 problems per model checkpoint, 8 model check- points per model size, our total compute costs are lower bounded by:
Checkpoints
128 Problems
32e4 Samples XModel Size A100-s
Total Chip-Time
Model Size
Checkpoint
Problem
128 Samples
Model Size
292c6 A100-s
81e3 A100-hr
Thus, sampling our GSM8K data is lower bounded by 81e3 A100-hr. To estimate the total financial cost of this sampling, we looked up pricing of compute providers. As of November 2025, Lambda offers A100s at AI at $1.40/A100-hhr In total, the financial of our GSM8K sampling data is more than $1 $112k.
RunPod at 1..39/A100-hr Tensordock at /.////3/-011m and Vast-
C.2 ESTIMATION MATH SAMPLING COST
Estimates of the GPU time per 128 samples per model size for MATH are provided in Table
Pythia Model Parameters
A100-s Per 128 Samples (Approximate)
14M
70M
160M
410M
IB
10
1.4B
10
2.8B
15
6.9B
12B
39
Table 2: A100-Seconds Per 128 Samples by Model Size MATH Problems.
36

---

Published as a conference paper at ICLR 2026
D GSM8K SCALING LAW PARAMETERS BY NUMBER OF ATTEMPTS PER PROBLEM
For the three scaling laws consider, visualize how the fit GSM8K scaling law parameters change as function of the number of attempts per problem & (Fig. 11 Fig. 12 Fig. 13p.
10.0
107
109
Iredueible Eareg B.(0)
Compate Prefactor C6()
Compuce Exgonetz o.
7.5
10P
5.0
109
2.5
10
0.0
10a
10-2
10
10
10
10a
10
10P
102
10
Nun. Attempls/ Prohlem &
Nau. Allempia / Problem k
Num. Attetapls / Proideea k
Figure 11: GSM8K Pretraining Compute Scaling Law Parameters by Number of Attempts per
Problem (Full Fits), Scaling law parameters of Eqn. aas functions of k. Hue corresponds to k.
107
Parmunters Pvelactor No(k)
Pacameters Exponent 8(k)
Irrecuetble Ervot Salk)
7.5
5.0
0.0
10
19
10'
10
10"
102
10
10
10
Num. Attemptt/ / Problem k
Nom. Attempos / Problern k
Num. Attempts / Problem
Data Poelacor Dalk)
Data Exporent 7(k)
107
10a
101
10
10
10*
Num. Attempts /
Problem
Num. Attempts
Problem
Figure 12: GSM8K Parameters and Tokens Scaling Law Parameters by Number of Attempts
per Problem (Full Fits). Scaling law parameters of Eqn. 4 as functions k. Hue corresponds to k.
10.0
GodPs Profectrr Kick)
GoliProbg Expocent m(R)
Ireedaable Eroor 6,00)
7.5
10
5.0
10
10P
10-3
10
10
10"
10
10
10
102
10
10'
102 Nun. Attenpls / Proideen
10
Nun. Altempls/
Prohlem k
Nam. Allempiaa / Problem k
Figure 13: GSM8K Gold Reference Log Likelihoods Scaling Law Parameters by Number of Attempts per Problem (Full Fits).
Scaling law parameters of Eqn. 5 as functions of k. Hue
corresponds to
37

---

Published as a conference paper at ICLR 2026
E RELATION BETWEEN THE COMPUTE SCALING LAW AND THE PARAMETERS + TOKENS SCALING LAW
Here, we provide concrete analytic relation between Eqn. and Eqn. A provided in the main text. For fixed benchmark B and number of attempts problem k, we model
No(k) Do(k) log[pass, @N) A) = Eo(k) + NB(K) DN(R)
(8)
and, when training under compute budget C cND,
Co(k)
log passs @k) (C, F(k) Ca(k)
(9)
Compute-optimal envelope. separable in (N, D) as in Eq. fixed k; define
Assume the excess loss beyond the irreducible term additive and with (o(k), Do > 0 and exponents (k)) >> 0. For each
A No(k), B= Do(k,, == 3(k), = >k)),
(10)
and
B
F(N,D) No + D
F(C) = min F(N, D).
(11)
N N.DS0: cND=C
Then the compute law Eq. iis the compute-optimal envelope of Eq. 8 with
B(k) E(k) (k:). a(k) B(k) + (1k)
(12)
3(R) 7(k)
(k(k) (8 (B+7) गー म Artz Buir (Kk)
(13)
The compute compute-optimal allocation along the envelope is
[3A ग B
(7B) Ji.
(C,
D*(C,k)
(14)
BA
Derivation Sketch. Form the Lagrangian L = AN-8 BD + X - C/c). First-order conditions give 3AN-8 = -BD-> and ND = C/c, from which (A.5) follows. Substituting N D" into F(N, D) yields F(C) Co (k) C-a(k) with a(k) and (k as in Eqs. 12 and E(k) (k).
Remarks.
1. The compute exponent is the parallel sum of the (N, D) exponents: a = (1/3 + 1/1)-1
2. If B then B/2 and N D* C1/2 with C. 2 2,8 (AB)
3. If scales N and D fixed ratio (i.e., without re-optimizing with C), the effective compute exponent degrades to Opath = min/8,
4. All quantities may depend on A; the mapping in Eqs. 14 applies point-wise in k. The conversion constant enters only via Co. not a.
E.I MULTIPLICATIVE DEVIATION FROM THE OPTIMAL COMPUTE ENVELOPE
Let (N (C), (C)) denote the compute-optimal allocation in Eq. 14 and define the misallocation ratio
r(C) = N(C) D²
(15)
NC(C) D(C)
38

---

Published as a conference paper at ICLR 2026
A direct algebraic manipulation using the first-order optimality condition (A(N") )- =D((D") gives the exact factorization
p (C) - Ed (r: = 3 3+7 log(pase g @k) ope (C) Eg
8
r
1,
(16)
3
with equality i = 1. Equivalently, this produces multiplicative correction to Eq.
log(pass; @k @k)(C) Eo Co C
: 3,7)
(17)
misallocation factor 21
Thus the deviation from equality fully parameterized by r via the dimensionless penalty 2
Local and asymptotic behavior of the penalty. Let logr. A Taylor expansion of equation I6
around r = gives
87 0(+)), 2
(r)
(18)
so small misallocations incur only quadratic penalty in log- log-space. Along power-law path N x C (with constants absorbed into r), (p - log C' + O(1); hence if * 347 the penalty grows polynomially:
B4. rO)', r(C) - xo (over-allocating N), B r(C) 0 (under-allocating N).
+ (r(())
(19)
This isolates the compute-optines scaling C and expresses all off-ridge effects through single, dimensionless factor depending only on the allocation ratio N/N (or equivalently D DD)
We conclude that if one matches the compute law without optimizing (N, the result is not single power law in general: it is compute compute-optimal optimal power multiplied by a misallocation penalty (NN/N* 2 1.
39

---

Published as a conference paper at ICLR 2026
F HOW SCALING DEPENDS ON THE BENCHMARK: THREE PRETRAINING SCALING LAWS FOR MATH
In the main text, presented results for GSM8K (Cobbe et al. 2021). Here, present corre- sponding results for MATH (Hendrycks et al. 2021 J. For a focused discussion of the similarities and differences between the two benchmarks, please see Section 7 in the main text.
F.1 FITTING AND PREDICTING PASS RATES FROM PRETRAINING COMPUTE
Nim Attempptgge per Prefiam
Atherppte per Priblen
Num Attemppo per Profiam
34T 2,406 C C-2
10
5,44
4.47e11 1 1.72011 (-iatees
10
1013
1090
1027
20
1221
1019
10
10
1077
Prst Paetmining Compate
Prstrrinnn Chemgete
Pretzaining Comgrain
Figure 14: Scaling of MATH Pass Rates with Pretraining Compute (Full Fit). Each panel fits Eqn. 3
(pass @k) (C, k) ((())) <<k) MATH pass rates for Pythia checkpoints orders magnitude of pretraining compute for & {1, 10,100014
across
Target Modells llgg (passs (ike
10a
10
10!
Irreducible Erron En (0)
Relotive Error of
10"
10-1
4
10-2
10-3
10-4
10t6
10a
Compute Prefactor Ga(k)
Compute Exponent a(k)
10
10a
10
100
1040
1000
10000
10a
104
10
10
10-1 Max C for Predicting / Target C
10-a
10-2
10P
10
10
10-2
10-1
10
Max C. for Predicting Target
C
Figure 15: Predicting MATH Pass Rates from Scaling Pretraining Compute (Backtesting). We evaluate predictability via backtesting: we iteratively fit Eq.3oon subsets of models up to a maximum pretraining compute, then extrapolate to predict the expensive model's - -log (passg @k). The most expensive model is Pythia 2a-parrmteer 300B-token (z= 2.16 10 FLOP).
40

---

Published as a conference paper at ICLR 2026
F.2 FITTING AND PREDICTING PASS RATES FROM PARAMETERS AND TOKENS
Nun. Prohásu
Problen
Num. Prolicm 1000
4:55 3. 3.4268--aai
100 Webb 22-0-----1
1.2te-1 2.00er 1:00et/00-21k1
Partninia Cassule
101
2
x 10p
Typ
1011
1880
11t
109
20
1991
10
101
102
Pretraning Compane
Pretriiigg Corpeete
Preszaming Conganie
Figure 16: Scaling of MATH Pass Rates with Parameters and Tokens (Full Fit). Each panel fits Eqn. 4 - log(logsssae @ (N, k) (k) + (o(k) N-8(k) + Do(k) D-7(k)
10a
10
Target Model's - (pgepassggii lkk
10'
Imeednciblle Error Ed(3)
Relative Error
10e
10-1
10-2
10-3
10-4
10
101
Parameter Prefactor Nalk)
108
Parameter Exponent ((k)
10a
10
10a
10
10a
10a
10-1
102
1055
Data Prefactor Dy(k)
Data Exponent (()
1088
10a
10
1024
10a
10
10
10
10
10-3
10-2
10
10P
10
10
10-2
10-1
10a
Max for Predicting
Target C
Max C for Predicting
Targer
Figure 17: Predicting MATH Pass Rates from Scaling Parameters and Tokens (Backtesting). We evaluated how accurately the parameters + tokens scaling law (Eqn. 4 predicts the most expen- sive model's log(pass @k.).
41

---

Published as a conference paper at ICLR 2026
F.3
FITTING AND PREDICTING PASS RATES FROM GOLD REFERENCE LIKELIHOODS
Nun. Prolisu
Proble
Nun. Prolism 1000
345 - 0.lle=7 ---- GodP
700mm1 E.lo-0 (- Prd(
240e-1i 10 120-22 [- legg Sriiilin
107
2
x 107
Typp
10"4
4 13
6x 101
6 10
4xx 5x 30
6. 101
xx
6.x.
Log GabilPooo))
Figure 18: Scaling of MATH Pass Rates with Gold Reference Likelihood (Full Fit). Each panel to + Ko(k) |- log [GoldProbg)]/tke [=(8), for & E (1, 10 10*}.
fits Eqn. log(pass @k)
Target Model's log loggpassge passgelll
10a
10
10
Irreducible Error Eolk)
Relotive Error of
109
6
100
10-1
1000
10-2
10000
10-4
10-4
10a
10a
GokiProbg Prefactor Ko(k)
GoldProbg Exponent k(k)
104
10
10a
10a
10a
10-2
10-1
10-4
10-6
10-2
10-4
10-1 Max C for Predicting / Target C
10-3
10-2
100
10-4
10
10-2
10-1
10
Max C for Predicting Targes C
Figure 19: Predicting MATH Pass Rates from Gold Reference Likelihoods (Backtesting). We evaluated how accurately the gold reference likelihoods scaling law (Fqn. (6) predicts the most ex- pensive model's (og(passpák))
42

---

Published as a conference paper at ICLR 2026
G MATH SCALING LAW PARAMETERS BY NUMBER OF ATTEMPTS PER PROBLEM
For the three scaling laws consider, visualize how the fit MATH scaling law parameters change as a function of the number of attempts per problem & (Fig. 20 Fig. 21 Fig. 22
10.0
109
Ireduebble Eareg B.(0)
Compate Profuctor Colk)
10"1
Compuce Exgoneet o.
7.5
10P
5.0
2.5
0.0
10-2
10
10
10
10P
10
10"
102
10
Nun. Attempls/ Problem &
Nau. Allempia / Problem k
Num. Attetapls / Proileea k
Figure 20: MATH Pretraining Compute Scaling Law Parameters by Number of Attempts per
Problem (Full Fits), Scaling law parameters of Eqn. ass functions of k. Hue corresponds to k.
10
10P
Paraneterss Prefucter No(ee
Pacamaters Expaneal 3(K)
Irrecuelble Ervot Salk)
7.5
10
10'
5.0
10a
0.0
10'
10
10f
10'
10
10
10
10
10
Nom. Attempts/ / Problem
Nom. Attempos / Problern k
Num. Attempas Problem k
10
Data Prcluctor Dalk)
1095
Data Exposent (()
10"7
4 10
1075
2 109
1040
1020
109
104
10F
10
Num. Attempts
Problemn
Num. Attempta Problem k
Figure 21: MATH Parameters and Tokens Scaling Law Parameters by Number of Attempts
per Problem (Full Fits). Scaling law parameters of Eqn. 4 as functions k. Hue corresponds to k.
10.0
GodPsobg Prdfectrr Kickk
GoliProbg Expocent m(R
Ireeduable Eroor 6,00)
7.5
10
5.0
10:
108
2.5
10-3
10
10
10
10
10
10
102
10
10'
102 Nun. Attenplss / Proiseen
10
Nun. Altempls// / Prohlem k
Nam. Allempia / Problem k
Figure 22: MATH Gold Reference Log Likelihoods Scaling Law Parameters by Number of Attempts per Problem (Full Fits).
Scaling law parameters of Eqn. 5 as functions of k. Hue
corresponds to
43


Reviews:
[{"review_id": "95jGfjEZxi", "reviewer": "Reviewer_GH31", "summary": "This paper extends the study of neural scaling laws from pretraining losses and discriminative benchmarks to generative evaluations, tasks where models are assessed by their ability to produce correct outputs such as math solutions or code completions.\nThe authors propose and compare three scaling laws for predicting pass@k performance across pretrained checkpoints: (1) A Compute-based scaling law, relating log(pass@k) to pretraining FLOPs; (2) A Parameters + Tokens law, decomposing compute into model size N and dataset size D; (3) A Gold Reference Likelihood law, regressing pass@k against the log-likelihoods of reference (“gold”) solutions. \nThrough backtesting on the GSM8K benchmark using Pythia checkpoints, they show that all three scaling laws achieve comparable predictive accuracy, but the gold-likelihood-based law is notably more stable, maintaining parameter convergence over up to five orders of magnitude in compute.\nFinally, the authors establish a theoretical connection showing that the compute scaling law emerges as the compute-optimal envelope of the parameters+tokens law, explaining how optimal allocation of compute between parameters and data induces the observed power-law behavior.", "questions": "It would be interesting to test whether the fitted α(k) exponents are consistent across different generative benchmarks or model families.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is the first (as far as I know) to systematically analyze generative scaling laws for pass@k-based evaluations, extending the classical scaling law framework beyond pretraining loss or discriminative accuracy.\n2. Evaluating three distinct covariates (compute, parameters+tokens, and gold log-likelihoods) under a unified methodology is conceptually clean and methodologically rigorous.\n3. The derivation showing that the compute law is the compute-optimal envelope of the parameters+tokens law offers a theoretical explanation of empirical scaling behaviors and clarifies the relationship between prior empirical findings.", "weaknesses": "1. The study is restricted to a single model family (Pythia) and one generative benchmark (GSM8K).\nThe conclusions, especially regarding the stability of the gold-likelihood law, should be validated on more diverse tasks (e.g., code generation, open-ended reasoning).\n2. While figures show relative stability, a numerical summary (e.g., mean relative error across backtests) would help quantify the magnitude of differences between the three scaling laws.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jawHydDHIk", "reviewer": "Reviewer_Cuov", "summary": "This paper investigates pretraining scaling laws for generative evaluations of language models and corresponding metrics such as pass@k on verifiable tasks. The author introduces a new parameter to the scaling law and identify gold reference likelihood to be a better proxy metric for stabilizing scaling laws.", "questions": "Please see my comments above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is clearly written and contains the necessary details for reproducibility.\n2. The direct comparison of different types of scaling laws answers the question of how to develop scaling laws for generative evaluations well. \n3. The analysis of including k as a parameter and its role in the scaling law is clear. \n4. The finding of the stability of the gold reference likelihood scaling law is a strong result and well founded, and explained.", "weaknesses": "1. The scaling law is developed on a single downstream task. While Pythia is an older generation model, where other tasks might be too difficult to obtain meaningful accuracy, why not average more tasks where Pythia could obtain a meaningful accuracy on, and expand beyond math?\n\n2. What is the practical implication of the penalty factor derived in Section 6? What does it look like for the studied model checkpoints?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ALvm8Pdhpt", "reviewer": "Reviewer_XVLF", "summary": "This paper studies scaling laws for generative evaluations, which is a relatively unexplored area compared to pretraining loss or discriminative benchmarks. The authors propose three types of scaling laws that relate model performance (measured by pass@k) to:\n1. Pretraining compute\n2. Model parameters and pretraining tokens\n3. Log-likelihoods of gold reference solutions\n\nThey empirically evaluate these laws on GSM8K using Pythia models (14M-12B parameters, 300B tokens), showing that: All three scaling laws achieve comparable predictive performance.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper expands scaling law studies beyond pretraining or discriminative settings to generative evaluations, which are highly relevant for coding, reasoning, and mathematical tasks.\n2. The authors clearly define three scaling formulations, use consistent benchmarks (GSM8K), and employ backtesting to rigorously evaluate predictive ability.\n3. The results show stable behavior over large dynamic ranges, and the study identifies when extrapolation becomes unreliable", "weaknesses": "1. The writing quality of the paper could be improved. For example, Figures 3 and 5 omit certain points, and the legend partially obscures the plotted lines.\n2. All experiments are conducted on GSM8K using Pythia checkpoints\n3. The paper focuses only on pass@k with temperature sampling = 1.0. Other decoding strategies (top-p, nucleus sampling, self-consistency) might influence scaling behavior.\n4. The results are derived solely from internal checkpoints of a single architecture. Evaluating additional model families such as LLaMA or Qwen could help assess the robustness of the proposed scaling laws. Moreover, since Pythia is a relatively dated model, it may not be well suited for evaluating mathematical reasoning capabilities.\n5. I believe that pass@k can be viewed as a variant of accuracy. Could the authors clarify in detail how this work differs from Language Models Scale Reliably with Over-Training and on Downstream Tasks https://arxiv.org/abs/2403.08540.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)