Pre-training under infinite compute
Authors: Konwoo Kim
Suhas Kotha
Percy Liang
Tatsunori Hashimoto
Venue: ICLR 2026 Oral (2026)
Keywords: scaling laws
data efficiency
pre-training
Primary area: foundation or frontier models, including LLMs
TL;DR: Since compute grows faster than the web, we design simple recipes that improve the asymptote of compute scaling laws to be 5x data efficient, offering better performance with sufficient compute.

Abstract: Since compute grows much faster than web text available for language model pre-training, we ask how one should approach pre-training under fixed data and no compute constraints. We first show that existing data-constrained approaches of increasing epoch count and parameter count overfit, and we improve upon such recipes by tuning regularization, finding that the optimal weight decay is $30\times$ larger than standard practice. Since our regularized recipe monotonically decreases loss following a power law in parameter count, we estimate its best possible performance via the \textbf{asymptote} of its scaling law rather than the performance at a fixed compute budget. We then identify that ensembling independently trained models achieves a significantly lower loss asymptote than the regularized recipe. Our best intervention combining epoching, regularization, parameter scaling, and ensemble scaling achieves an asymptote at 200M tokens using $5.17\times$ less data than our baseline, and our data scaling laws predict that this improvement persists at higher token budgets. We find that our data efficiency gains can be realized at smaller parameter counts as we can distill an ensemble into a student model that is 8$\times$ smaller and retains $83$% of the ensembling benefit. Finally, our interventions designed for validation loss generalize to downstream benchmarks, achieving a $9$% improvement for pre-training evals. Our results show that simple algorithmic improvements can enable significantly more data-efficient pre-training in a compute-rich future.

TRAINING UNDER INFINITE COMPUTE
Konwoo Kim Suhas Kotha Percy Liang, Tatsunori Hashimoto Stanford University
ABSTRACT
Since compute grows much faster than web text available for language model pre-training, we ask how one should approach pre-training under fixed data and no compute constraints. We first show that existing data-constrained approaches of increasing epoch count and parameter count overfit, and we improve upon such recipes by tuning regularization, finding that the optimal weight decay is 30 larger than standard practice. Since regularized recipe monotonically decreases loss following power law in parameter count, we estimate its best possible performance via the asymptote of its scaling law rather than the performance at fixed compute budget. We then identify that ensembling independently trained models achieves a significantly lower loss asymptote than the regularized recipe. Our best intervention combining epoching, regularization, parameter scaling, and ensemble scaling achieves an asymptote at 200M tokens using 5.17x less data than our baseline, and our data scaling laws predict that this improvement persists at higher token budgets. We find that our data efficiency gains can be realized at smaller parameter counts can distill ensemble into student model that is 8x smaller and retains 83% of the ensembling benefit. Finally, our interventions designed for validation loss generalize to downstream benchmarks, achieving a 9% improvement for pre- training evals. Our results show that simple algorithmic improvements can enable significantly more data-efficient pre- pre-training in compute-rice -ric future.
INTRODUCTION
Language model pre-training has historically been studied under compute constraints at training (Kaplan et al.] 2020 Hoffmann et al. (202) and inference (Brown et al., 2024 [Seell et al.) 2024) while assuming access unlimited web text. However, web data grows by 1.03x per year, whereas
compute spent on pre-training grows by 4x per year (Villalobos et al. 2024 Sevilla and Roldán 2024
In anticipation of regime where compute vastly exceeds data, ask:
How should one approach pre-training under fixed data and no compute constraints?
To establish a baseline, we fix a seed training corpus of 200M tokens of web text and evaluate a standard recipe following existing data-constrained approaches of repeating data (Muennighofi et al.) 2023) and increasing parameter count (Kaplan et al. 2020) (Section 2 We find that either too many epochs or too many parameters results in the loss eventually increasing due to overfitting. This bounds the performance improvements we can get from tuning this recipe, even if we were willing to spend more compute in exchange for better model.
We instead get predictable monotone scaling in parameter count by considering a regularized recipe (Section 3) Currently, regularization used in pre-training is often adopted from existing recipes, defaulting to weight decay of 0.1 from Brown et al. (2020). We find this amount to be inadequate for preventing overfitting under data constraints as the optimal weight decay is 30x larger than standard practice for our most weerpparameterized models. After jointly tuning weight decay, learning rate, and epoch count at each parameter count N, loss closely follows a power law in N for parameter-to- to-keen ratios 140x larger than Chinchilla, shown in Figure [
Normally, we would compare two recipes by evaluating performance at different train or inference compute budgets (Hoffmann et al. 2022 Snell al.) 2024). However, this does not reflect
Equal contribution.

---

Comparing scaling recipes with compute constraints
3,8
100
37-
LXLLM validialaaaannn koe
36-
efficiecyy
35
2.29
3,4
Standard necipe
Regalleized recipe (Titz 0.06/NN 1.02- 3,43)
3.0
33
Eneembiliigg recipe ( 1-122 3.54)
scaling recipe
asymplote (N, -.))
150M
300M
600M
Total parameter count
Figure 1: Comparing scaling recipes with no compute constraints. To simulate a data-constrained
future, we restrict models to 200M tokens. (1) Standard recipes overfit with too many epochs or parameters, even if we tune the epoch count at each parameter count N [2) By correctly tuning
regularization for each N, loss monotonically decreases following power law in N. We predict the
best possible loss of the regularized recipe by the asymptote of its power law.
Instead of scaling
N, we achieve lower asymptote by ensembling K models of size 300M as K + 00. Composing parameter and ensemble scaling improves the asymptote, and we estimate that the baseline would need 5.17x more data to match its loss, even with infinite compute. These data efficiency wins hold for larger token counts, distilled models, and downstream benchmarks (Sections
interest in the best possible performance under fixed data and no compute constraints. Since the loss of the regularized recipe continues decrease as N increases, we are interested in the limit of the loss as N + oc. More generally, we propose evaluating monotone scaling recipes by the asymptote of their scaling law (e.g. 3.43 for the regularized recipe as seen Figure []. By preferring recipes with lower loss asymptotes, we can train better models at sufficiently high compute budgets.
Though taking the parameter count to infinity one possible limit under infinite compute, we design recipes with even lower asymptotes. We consider alternative ensembling recipe where we average the logits of K independently trained models of the same size (Section (4). The ensembling recipe achieves lower asymptote as K -> compared to the regularized recipe as N - oo (Figure T). At sufficiently high parameter counts, it better to train multiple smaller models instead of single larger model. We further show that ensembling and parameter scaling compose, achieving lower asymptote when following the joint scaling recipe of taking both K, N + xc.
on the scale of 200M tokens, we study how our recipes scale across higher token counts and find that the asymptotes themselves follow a scaling law (Section 5). Our estimates indicate that the joint scaling recipe achieves its 200M asymptote with 5.17x less data than the standard recipe. Importantly, extrapolation of our data scaling laws indicates that the data efficiency improvements will persist at higher token counts.
Since our previous experiments
Though the asymptotes of our recipes benefit the most from large parameter counts, we find that distillation (Hinton al. 2015 Kim and Rush 2016) allows us retain most of the loss improve- ments without increasing inference parameter count. Distilling an 8-ensemble into single 300M model retains 83% of the ensembling loss improvement over the best regularized 300M model and outperforms the asymptote of the regularized recipe. We also find that self-distiling 300M teacher into a student the same size reduces loss, improving data efficiency without ever explicitly training model of higher parameter count.
Finally, we confirm that improvements on validation loss translate to improvements on downstream benchmarks (Section 7). Ensembles with better validation loss perform better downstream benchmarks, with our best ensemble outperforming our best unregularized model by 9% on average over PIQA, SciQ, and ARC Easy (standard benchmarks for models at scale (Thrush et al. 2025
2

---

2 STANDARD PRE-TRAINING
Historically, pre-training has focused on training the best possible models subject to compute or parameter constraints. Under train compute constraints, recipes like Chinchilla recommend jointly scaling data and model size with 20x more tokens than parameters (Kaplan et al. 2020 Hoffmannn et al. 2022). Under parameter constraints for cheaper inference, current practice opts for over-training language models relative to Chinchilla with token counts 2000 larger than parameter counts or
distilling from preexisting larger models (Gadre et 2024 Grattafiori al. 2024 Sardana et al.) 2025 Busbridge et al. )025).
Prior works prescribe such scaling recipes by always training on fresh data. In this paper, we instead study data-constrained pre-trnin where we cannot jointly scale data and model size. We analyze the purest form of the problem by lifting all other constraints (including compute) besides data. We formalize standard pre-training as training routine A that accepts arguments such token count D, parameter count N, epoch count to produce a model M with loss L(M). Unspecified arguments passed through hyperparameter tuple H. Our constrondt pre-trate pre-training objective becomes D = min C (D).
We construct controlled pre-training environment with limited amount of web data from DCLM et al. 2025). Since our algorithms spend more compute than Chinchilla scaling at a fixed data budget, default 200M tokens and test whether For evaluation, we defer to loss on a held-out i.i.d. validation set which is shown to correlate with
findings hold across higher token counts in Section 5
downstream capabilities in Section 7 and prior work (Chen al. 2025 Thrush al., 2025 Gadre et al. 2024). To best represent standard practice, we follow a standard auto-regressive recipe (full
details in Appendix B)
2.1 EVALUATING EXISTING DATA- CONSTRAINED RECIPES
Since the amount of fresh data is limited, we build standard recipe of increasing repetition count (Muennighoff et al., 2023) and parameter count (Kaplan et al. 2020). Since there un- limited compute, depart from compute-efficies practice by training models that are larger relative to the token count, defaulting to 300M parameter models for 200M tokens.
We first increase the epoch count E at fixed parameter count, taking E' more training compute. Figure 2 (left) shows for high E, overfitting occurs and loss increases. These findings contradict the functional form of the decay-based scaling law in Muennighoff et al. (2023). which posits that loss monotonically decreases in E. Their work acknowledges this discrepancy and removes most overfit runs from their scaling law (see their Appendix D).
Since increasing epoch count arbitrarily hurts loss, we turn to increasing parameter count. To establish competitive baseline. we jointly tune epoch count and learning rate for each parameter
Increasing epach count
Increasing parameter count
5.0
4.-
4.5
Lo55
1,00
4.4
4.2
4.0
3.9
2
15
32
128
150M
300M
600M
1.00
1pochs
l'ammnttrr count
Tuned
128
Tuned
150M
300M
608M
Learning rate
16-3 16-3
3o-3
Leaminng
3e-3
1e:3
1e:3
Se-4
Epoch count
Figure 2: Evaluating standard recipe of epoching and parameter scaling for 200M tokens. Left: Though repeating the data lowers the loss, too many repetitions results overfitting for 300M models. Right: We try increasing parameter count, tuning the epoch count at each parameter count. We similarly find that loss starts increasing. Moreover, increasing the parameter count 10x improves the loss by less than 0.1.
3

---

count (Appendix C.1
We find minimal improvement in loss with higher model size, with our 1.4B model performing worse than our 600M model. This is consistent with the single-pass findings of Kaplan et al. (2020), Figure 9 which show that increasing parameter count eventually starts likely that both higher epoch and parameter count result in overfitting the train set, detailed in Appendix C.5
increasing loss for fixed data.
REGULARIZED PARAMETER SCALING
We show that to get the best performance from these over-parameterized. epoched models, it is critical regularize pre-training with much higher weight decay than standard practice. To jointly tune weight decay, learning rate, and epoch count, we perform an extensive search for "locally optimal" hyperparameters using coordinate descent algorithm inspired by Wen et al. (2025) (details in Appendix C.T). We find that aver-parametrized models need much higher weight decay, over x0x larger than the standard practice of 0.1 (Figure 3 right table).
With this tuning, loss follows monotone scaling in parameter count for models up to 140x larger than Chinchilla as shown in Figure 3 This agrees with theory for over-parameterized regression that predicts that even when loss does not monotonically decrease due to double descent, the loss will monotonically decrease when regularization is optimally tuned (Advani and Ganguli, [2016] Nakkiran et al., [02T]] Canatar et al. 2021] Simon et al. (024).. In Appendix C.2 tuning critical to achieve monotone scaling.
we show how locally-opting
To capture how increasing parameter count improves loss, we fit power law with an asymptote as LD,
An NoE + En where we fit free variables An, or En Our fit across four parameter counts 0.05 results in L200M Nroz + 3.43 The exponent of 1.02 for parameter scaling is high given that Chinchilla finds a parameter scaling exponent of 0.34. This suggests that when we better leverage the data, there is faster improvement from larger models.
Our monotone scaling law differs from compute-optimal prescriptions where increasing N can hurt performance due to training on less data. We characterize our best possible performance unconstrained by compute as lim LD e.g. the asymptote Ep- The asymptote for the regularized recipe law predicts that the best possible model achieves loss 3.437
Regularized parameter scaling
Tuned
150M
300M
600M
1.4B
3.55
Learning rate
3e:3
3e-3
1e:3
le.3
3.50
Fpoch
Weight detay
0.8
322
322
375
Figure 3: Power law scaling from jointly tuning regularization. We compare the standard recipe in Figure (red line) to our regularized recipe that jointly tunes learning rate, epoch count, and weight decay (purple line). After regularization, the loss decreases proportional to = N The power law predicts that as N oc, the best model achieves 3.43 loss.
Epoched rodipe Repulartzed (N13/N/NN) 3,43)
3
3.50
3.5
3.0
345
Parumeter count
4 ENSEMBLE SCALING
The regularized recipe offers straightforward way to improve performance by taking N - oo. Can different training algorithms better leverage the data under infinite compute? In this section, we consider ensembling (Dietterich) 2000); independently train models and average their logits for generation (Section 4.1) In Section 4.2 show how ensembling can outperform parameter scaling at fixed parameter counts and under the limit as total parameter count approaches infinity.
For this power law and all following ones, the units of parameters and tokens will he in billions for cleaner visualization and comparison. This only affects the numerator the scaling law, not its asymptote or exponent. Because of run-to-run variance of the points forming the power law, we share a sensitivity analysis in Appendix I.I showing that the asymptotes vary by most 0.02 loss across seeds.
4

---

In Section 4.3
we construct best recipe composing regularized parameter scaling and ensemble scaling by taking the limit as both N, - cc.
4.1 DEFINING ENSEMBLES
The ensembling pre-training algorithm & accepts pre-training algorithm A, trains K members that are identical up to random seed Z; controlling data order and model initialization, and returns a model that averages the logits of the K members. See Appendix [. for a full formal definition.
The number of FLOPs needed to generate from or evaluate an ensemble is simply the sum of the costs for all members. Since the number of FLOPs in forward pass is approximately linear in parameter count (Kaplan et al. 2020; Hoffmann et al. 2022). ) we will consider an ensemble's 's total parameter count as K when comparing it to standard pre-training.
Ersemble memboer scaling
soddis kulling (TA 007/ 02 345) ensembls 0220
Figure 4: Comparing scaling parameter count scaling ensemble member count. Instead of scaling the parameter count of a single model, we train an ensemble of smaller models and scale the number of ensemble members (resulting in NK total parameters for K ensemble members). Scaling up member count K can similarly be fit by power law with exponent approximately 1. Importantly, this law achieves a better asymptote than scaling N.
3.00
340
340
3.01
348
3.01
3.5
199
X00M
1.40
Trraa parattoa court
4.2 SCALING MEMBER COUNT INSTEAD OF PARAMETER COUNT
We compare the regularized and ensembling recipes under the best regularized hyperparameters from Section 3 In Figure find that the ensembling recipe's s excess loss decreases close K. similar to how the regularized recipe's excess loss decreases at a rate close to N. Under infinite compute. the ensembling recipe's 's (N 300M, K + oo) asymptote is 3.34, which is lower than the regularized recipe's (N + 20, K = 1) asymptote of 3.43. Thus, for large N, it is better to train multiple small models instead of a single large model. In fact, even the K ensemble outperforms the regularized recipe's asymptote.
of
Why does ensembling improve over parameter scaling? Allen and Li (202)) shows that ensem- bling helps when the data can be well-classified with one of many features but is best classified using all such features. Under this "multi-view" structure, they find that a single model is only learns one feature, whereas each ensemble member learns different features.
4.3 JOINT SCALING RECIPE COMPOSING PARAMETER AND ENSEMBLE SCALING
Although the ensembling recipe outperforms parameter scaling, we can compose both by taking the number of members and the size of each member infinity (N, K oo). To estimate the best possible loss of a joint scaling recipe, we take two limits:
ED
lim lim min C (E,,,,, N K-000 H
As long as ming (EA (D, N, K, H)) monotonically decreases in N and when fixing the other variable, the value does not depend on the order of the limits. We choose this order as it results in the most convenient hyperparameter tuning (Appendix D.6). For the inner limit, we cannot fully find locally optimal hyperparameters due to experimental constraints. Instead, we use the heuristic of taking the optimal regularized hyperparameters with 2x epochs and 0.5x weight decay (Appendix D.4).
In Figure 5 we show how we take this double limit. Our final estimate for the joint scaling recipe's loss is 3.17. which is much better than the regularized and unregularized losses of 3.43 and the 75.
II Appendix D.2 we find optimally tuned ensembles match this intuition. Slightly overfitting each ensemble member beats ensemble using the best regularized hyperparameters.
5

---

220M
22004
Saadad
Regelataed
31
37
15-
16
3
c0Rd pacamaelite 839-8-110-
14B (T1 2.0/1//11
3CM
4CM
144
Figure 5: Composing the regularized and ensembling recipes under the double limit. Left: For each N, we fit a power law on the loss as K increases. We select hyperparameters for low asymptotes instead of loss at small K. Right: We take the asymptotes from the left plot and fit power law to capture how the asymptote changes for bigger ensemble members. This law's asymptote estimates the best possible loss under the joint scaling recipe.
SCALING THE SEED TOKEN COUNT UNDER INFINITE COMPUTE
Do our loss improvements 200M tokens generalize to larger scales? In Sections 5.1 and 5.2 we first measure the best possible loss of our recipes at higher token counts up to 1.6B tokens. We contextualize the loss improvement and data efficiency of recipe by interpolating how much data the standard recipe would need to match performance. In Section 5.3 we fit data-scaling laws to extrapolate how our recipes would perform at even higher token counts.
Taking panamette craat
Verving soed bolum onunt
Stankesl page
Iels LD3/0022 1JK)
2014 biorre 2.85/""1.2
41254 bioeee 2.8/
I08M4 dft 2. -15)
1.70 8.20/N11..7-
Figure 6: Scaling the seed token count for single models. We first consider the best loss of the standard recipe tuning epochs and parameters (red points, right). We then consider the best loss of the regularized recipe by fitting parameter scaling laws across four token counts (left) and taking their asymptotes (purple points, right). We fit data-scaling laws (red and purple lines) to extrapolate performance as seed token count increases.
5.1 DATA SCALING LAWS FOR SINGLE MODEL RECIPES
As shown in Section 2 the standard recipe overfits and does not admit a monotone scaling law. Instead, we search for the best parameter count and hyperparameters at each of our four data scales (Appendix E..I. Given these four estimates of the best loss at each token count (Figure 6 right), we fit data scaling power law shown as the red line using Lp = A + E.
We characterize the best possible loss of the regularized recipe by estimating lim min L (AA N, H)) as shown in Section 3 Since we have to compute asymp- totes to build the points for the data scaling law, we Tollow a two step procedure shown in Figure 6
Measuring data efficiency. We measure the data efficiency between two recipes at fixed token count D. We first compute the effective data D' that A, would need to match Ag. After interpolating D' via the data scaling law of A,, we report the data efficiency as z. This metric characterizes the regularized recipe asymptote as 2.29x more data efficient than the standard recipe at 200M tokens.

---

Even without any extrapolation of the asymptote, the best 1.4B model at 200M tokens is 2.09x more data efficient than our baseline.
5.2 DATA SCALING LAWS FOR ENSEMBLES
We repeat the above procedure for ensembles by following Section 4.3 and estimating lim -ioo minH L A (D, N, K, H)) for each seed token count D. We visualize the three step procedure in Figure At 200M tokens, the asymptote of the joint scaling recipe is 5.17x more data efficient than the standard recipe. Without taking asymptotes, our best ensemble of five 1.4B models itself 3.75x more data efficient.
Taking court
Taking pazameler
Varying loken count
3
5.-
(Fa 1.0 - 1581
3
2.-
Reg bolaad segeppans
aesppiio (ht 025 DP0224 1150
34-
544
SPAC
8 022
12
32
ES0C bakens
10
178
30
3.-
1M00 portes
eand parzeo
2000 puzzeo
26-
2.-
1440 parzen
26
152M
XCM
GBM
246
300M
BIM
Prosinter cound N ensorrloo
Seal blle coant
Figure 7: Scaling the seed token count for ensembles. Left: For fixed parameter and token count, we fit a power law in K, with hyperparameters optimized for the asymptote. Middle: We take the asymptote of the left 16 laws and fit power law to measure how the asymptote changes in N. Right: We take the asymptote of the middle 4 laws and fit power law to mcasure how the asymptote of asymptotes changes in D. At all token counts, we find over 2x and 5x data efficiency wins over the regularized and standard recipes respectively.
5.3 DATA SCALING ANALYSIS
Although the data scaling laws are expected be noisy, they predict that all recipes decay similar rate with exponents between 0.23 and 0.24 and asymptotes between 1.89 and 1.96. Asymptotic statistics suggests that the asymptotes are equal if the algorithms achieve Bayes-optimal error under infinite data and compute, in which case their loss would be the entropy of text (Shannon I951 Van der Vaart 2000). When the asymptote E and exponent a of the laws are the same for two algorithms, there is constant data efficiency improvement at all token counts determined by the numerators A,, A2, equal to ) Our preliminary analysis suggests that our data efficiency wins will not disappear across all data scales even if they perform similarly under infinite data.
6 DATA EFFICIENCY UNDER PARAMETER CONSTRAINTS
The asymptotes of the regularized and ensembling recipes rely on arbitrarily high parameter models We study whether large models are necessary for data efficiency, either for the final model for training. In Section[6.1] we distill an 8-ensemble of 300M members into a 300M student, preserving 83% of the loss improvement with 8x smaller final model. In Section 6.2 show self-distiling 300M model into a student of the same size outperforms the teacher, removing the need for large parameter counts training.
6.1 REDUCING FINAL PARAMETER COUNT VIA ENSEMBLE DISTILLATION
Even if our best scaling recipe helps in the limit N, K + can it help train models that are small relative to D It is known that better large models can improve the performance of smaller models through knowledge distillation (Hinton et al. 2015 Yang et al. 2025 Team et al. 2025a). Since we are not bound by train compute, we can first pre-train a data-efficient teacher M/' on D tokens using our existing recipes. Then, sample from M unconditionally (i.e. with prompt) to generate D' tokens. We train our distilled student model M from scratch on the mixture of and D' (Kim and Rush| 2016).
7

---

Dosilling a (200 seed
Besslatecc wuppe the ozn 20122
Figure 8: Ensemble distillation and self- distillation. We can compress our data efficiency gains into smaller models through distillation. Dis- tilling an 8-ensemble teacher into a 300M student retains most of the loss improvement (pink star) and outperforms the regularized asymptote. Self- distillation with a 300M teacher and 300M student (green star) is surprisingly effective, matching the asymptote the regularized recipe without increas- ing parameter count at training.
sntenble 44/821L 222)
Deetling uodie craudi
Sensemblo dast
34
6KM
In Figure 8. we show the student model (pink star) obtained from using an 8-ensemble of 300M models (right-most blue point) with loss 3.32. Despite the 8x smaller size, our distillation model attains a loss of 3.36, preserving 83% of the ensemble improvement over the regularized 300M model loss of 3.57 (purple point). Our student outperforms the regularized recipe asymptote and matches the loss of a 4-ensemble (details in Appendix F.
6.2 REDUCING TRAIN PARAMETER COUNT VIA SFLF-DDITIILLATION
Is it possible to train data-efficien, model without high parameter count at train time as well? We consider this question for seff-istilllation where the teacher and student are of the same size and architecture. Many recent papers discuss how training a new student model model generations can result in model collapse (Shumailov et al. 2024 Taori and Hashimoto 2022).
Gerstgrasser et al. 2024 Dohmatob et al. 2024
On the contrary, by mixing together the D real tokens and D' synthetic tokens, we avoid collapse and can train a fresh student that vastly outperforms its teacher. In Figure we show how using 300M model as a teacher (blue point) results in a 300M student model (green star) that outperforms the best regularized 300M model (purple point). Why does self-distillation help and Li (2023) provide theory interpreting seff-distillation as implicitly ensembling the teacher and freshly initialized student.
7 DOWNSTREAM TASKS
Validation loss
Downstremm benchrark erre
Epoched
Epoched
soilnn
ensmmble
ensembiee
35
300 ensemble
30M ensemble
600M eommble
ensemblee
3.6
Arouage Eneor
Selidioill
0.42
Sel-dittl
s-tnsemblee drettl
d-Izsembiee duall
3.5
32
0.36
10'
11P
10tal pacsmetet court (ptlione))
Totsl pocatotee Coust (rtlbone)
Figure 9: Performance of pre-trained models downstream tasks. We have thus far been using validation loss (left) to seperate whether models are better pre-traiin trainrdddd models or not. We evaluate the same models and ensembles on downstream benchmarks (right). Models with lower validation loss have lower average error across downstream benchmarks.
Although validation loss is known to correlate with capabilities of interest (Chen et al. 2025 Thrush et al. 2023 Gadre et al. 2024), we further test our models' general capabilities using downstream benchmarks. For evaluations that are informative for models at scale, we take all of the accuracy- based benchmarks from Thrush et al. (2025), namely PIQA (Bisk et al., 2019). SciQ (Welbl et al. 2017), and ARC Easy (Clark et al. 2018). Notably, did not evaluate any benchmarks until
8

---

the end of the project after selected the best recipes following validation loss, making these benchmarks a strong test of generalization.
In Figure 9 we show the validation loss (left) and downstream benchmark enror (right) of models for 200M tokens. Without regularization, the standard recipe does not benefit much from parameter scaling. Regularization (purple points) makes downstream accuracy scale smoothly with diminishing returns, similar to validation loss. Ensemble error mirrors loss with increasing and K improving performance. Overall, our best ensemble outperforms best unregularized model by over 9% on average and our best distilled model outperforms the unregularized 300M model by 7%. See Appendix for full breakdown results.
RELATED WORK
We cite additional related work on wver-parametrized maching learning, distillation algorithms, synthetic data, and classical data-constrained deep learning Appendix
Scaling laws. Much of the success of language model pre-training was built upon scaling laws which
accurately predict performance a given resource budget (Hestness et al. 2017 2019 Rosenfeld
et al.
2019 Henighan et al. 2220 Kaplan et al. 2020 Sorscher et al. [023] Hoffmann et al.) 2022]
Ruan et |024| Cortes al.) 1993). Past work has studied scaling laws under constraints such as
data and compute (Muennighofi et al. [023 Goyal et al. (024), hardware precision (Kumar et al.
2024), parameter count (Sardana et al. 2025 Springer et al. 2025| Gadre et al. 2024), and test-time
compute (Brown et al. 2024] Snell et al. 2024). We show that past work (Muennighoff et al. 2023)
does not account for over-fitting, fix this via regularization, and propose asymptote estimation as a new metric.
Ensembling. Ensembling (Dietterich) 2000 known to boost performance across settings in-
cluding uncertainty estimation (Lakshmiinrryyanan al. 2017), image classification (Huang et al.) 2017 Garipov et al. 2018), and reinforcement learning (van Hasselt et al.) 2015). Deep ensembles shown to follow power laws (Lobacheva al. 202 1) and not believed to outperform parameter
scaling in certain theoretical models (Vyas et al. [2023] Ruben al. 2024). We show how ensembling be adopted for pre-tral pre-training and build scaling laws characterize loss. See Appendix D.5 for discussion on related alternatives.
Distillation. Distillation spends compute produce strong models with lower inference costs (Hnn ton et al. 2015) which we show with sequence knowledge distillation (Kim and Rush| 2016). For self-distillation there is recent work showing how training self- geferaeneetee inputs can be harm- ful (Shumailov et al., 2024) Dohmatob et al. 2024 Taori and Hashimoto 2022). Gerstgrasser et al. (2024) suggests that training on self-generated data can be helpful in limited scenarios, though their comparisons are neither compute-matched nor data-matched. The success of self-distillation aligns with prior evidence from constronstr deep learning (Mobahi et al.) 2020; Zhang al. 2019). Notably, |Allen and Li (2023) show how self-distillation can be viewed as implicitly performing ensembling and distillation.
Modern data-constrained pre-training. There are several recent works which study data-efficient pre-training and show the benefit of epoching (Muennighoff et al., 2023), rephrased synthetic
data (Maini et al.) 2024 Yang et al. 2024 DatologyAI et al. 2025 Ruan et al. (024). diffusion language models (Prabhudesai et al. 2025
Ni et al., 2025). and energy- based models (Gladstone
al.] (025). These recent works do not aggressively regularize for optimal epoching nor build scaling laws to estimate infinite compute performance.
DISCUSSION
The success of classical ideas from data-constrained deep learning like regularization and ensembling suggests that there is free lunch on the table, encouraging us to revisit pre-training design decisions. We are also excited by methods that can better leverage extra compute for performance, in line with The Bitter Lesson (Sutton, 2019). We hope that evaluating scaling recipes via their asymptotes inspires more data-efficient algorithms for the future.

---

10 ACKNOWLEDGEMENTS
We thank Steven Cao, Sam Park, Jacob Mitchell Springer, Kaiyue Wen, Yu Sun, Nathan Hu, Meena Jagadeesan, Luke Bailey, Neil Band, Sally Zhu, Ben Spector, and Audrey Xie for their helpful discussions or feedback on the paper draft.
This work is a part of the Marin Project and the compute is supported by the Google TPU Research Cloud (TRC). TH was supported by grant by HAI, DSO labs, gifts from Open Philanthropy, Amazon, Schmidt Sciences, the Tianqiao and Chrissy Chen Foundation and a grant under the NSF CAREER I1S-2338866, ONR
and DARPA Cooperative Agreement HR001 12520013. PI. was supported by DARPA Cooperative Agreement HR001 12520013. 3. This work does necessarily reflect the position policy of the govermment and no official endorsement should be inferred.
11 ETHICS
We hope that our work may be applied to settings beyond pre-training to improve data efficiency. We acknowledge our work may increase the amount of compute used for language model pre-training. We believe most other harms specific to our work apply to general language modeling research.
12
REPRODUCIBILITY
We open-source all of our runs on WandB and our cod/ on Github.
10

---

REFERENCES
M. Advani and S. Ganguli. Statistical mechanics of optimal convex inference in high dimensions. Phys. Rev. X, 6:031034, Aug 2016. doi: 10. 1001100/yyeeeex...051000 URL https://link
aps. org.org//oil dot/ /000oo1110./ 11w3///yseeee///e.e 631003
R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, Ramos, M. Geist, and O. Bachem. On-policy distillation of language models: Learning from self-generated mistakes, 2024. URL https
//arxiv. org/ abs/ 13344
S. K. Ainsworth, J. Hayase, and S. Srinivasa. Git re-basin: Merging models modulo permutation 04836
symmetries, 2023.
Z. Allen-Zhu and Y. Li. Towards endersttndinggessmmbbeee.nnnn www.eeeeeeeeeeeeeeeeeeeeeeeeeeee distillation and self-distillation in deep learning, 2023.
Z. Allen-Zhu and Y. Li. Physics of language models: Part 3.1, knowledge storage and extraction, 2024. URL
A. Amini, S. Gabriel, Lin, R. Koncel-Kedziorski, Choi, and H. Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms, 2019. URL https
//arxiv.org/
13319
M. Belkin, D. Hsu, S. Ma, and S. Mandal. Reconciling modern machine-learning practice and the classical bias variance trade-off. Proceedings of the National Academy of Sciences, 116 (32):15849-15854) July 2019. ISSN 1091-6490. doi: 0t007///naa.190377000 URL URLhttp!
//dx. doi wowp 1073 / ppas. httpppaa.........11111111 1903070116
T. Besiroglu, E. Erdil, M. Barnett, and J. You. Chinchilla scaling: A replication attempt, 2024. URL
https://arxix: wwwp://wwwwwwroo/////o.//////
Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. Piqa: natural language, 2019. URL
Reasoning about physical commonsense in
B. Brown, J. Juravsky, R. Ehrlich, R. Clark, Q. V. Le, C. Ré, and A. Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, 2024. URL ltttps://aaxx: org/abs 2407 .17877
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, Clark, C. Berner, McCandlish, A. Radford, Sutskever, and Amodei. Language models are few-shot learners, 2020. http:::/aazxv..orrgaa///////
D. Busbridge, A. Shidani, Weers, Ramapuram, Littwin and Webb. Distillation scaling laws, 2025. URL https: /ttp::///aaxivvva/ wttp:///.rrg/////////a/aass
08606
A. Canatar, B. Bordelon, and C. Pehlevan. Spectral bias and task- alignment explain generalization in kernel regression and infinitely wide neural networks. Nature Communi- cations, 12(1), May 2021. ISSN 2041-1723. doi:
URL
http: //t:://xxx doi. www.oogg.1..... 10000/541110 021002101-------1
23103-1
Y. Chen, B. Huang, Gao, Z. Wang, Yang, and H. Ji. Scaling laws for predicting downstream performance in 1lms, 2025. URL htttss://aaxiiiii org/ ht///bb//2410 08527
Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018. URL https:
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021. URL htllhttps:/// hhttps://arxiv.. .rg./oorrooooooo..oooooooooo www.oorgg//aaa/e..e abs//a10........... 14168
11

---

C. Cortes, L. D. Jackel, S. Solla, V. Vapnik, and J. Denker. Learning curves: Asymp- totic values and rate of convergence. In Cowan, G. Tesauro, and J. Alspector, edi- tors, Advances in Neural Information Processing Systems, volume 6. Morgan-Kaufmamn. 1993. URL htttps://proceeddnnss neur cctp File/Taa48 Fiee/// /
D' Angelo, M. Andriushchenko, A. Varre, and N. Flammarion. Why do we need weight decay in modern deep learning?, 2024. URL htttps://arxivv.rrr///e// 0310. 04415
DatologyAl, :, Maini, V. Dorna, Doshi, A. Carranza, Pan, J. Urbanek, P. Burstein, A. Fang, A. Deng, A. Abbas, B. Larsen, C. Blakeney, C. Bannur, C. Back, D. Teh, D. Schwab, H. Mongstad, H. Yin, J. Wills, K. Mentzer, L. Merrick, R. Monti, R. Adiga, S. Joshi, S. Das, Z. Wang. B. Gaza,
A. Morcos, and M. Leavitt. Beyondweb: Lessons from scaling synthetic data for trillion-scale pretraining, 2025.
abs/ 1097///00//555
J. Deng. W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248 255, 2009. doi:
200..5206855
T. G. Dietterich. Ensemble methods in machine learning. In Proceedings of the First International Workshop Multiple Classifier Systems, MCS '00, page 1-15, Berlin, Heidelberg, 2000. Springer- Verlag. ISBN 3540677046.
E. Dohmatob, Y. Feng, A. Subramonian, and J. Kempe. Strong model collapse, 2024. URL
petps::/arrivy http:////axxii.../////////////// wttp//////////////// 04840
K. Everett, L. Xiao, M. Wortsman, A. A. Alemi, R. Novak, J. Liu, Gur, J. Sohl-Dickstein, L. P. Kaelbling, J. Lee, J. Pennington. Scaling exponents across parameterizations and optimizers, 2024. URL /tt.ctp::/aaiiv...o. wwwpaxxwii.or.gaaaaa /abs///////// httpr//g/b///4.....a// 224000052000
S. Y. Gadre, G. Smyrnis, Shankar, Gururangan, M. Wortsman, R. Shao, J. Mereat, A. Fang,
J. Li, S. Keh, R. Xin, M. Nezhurina, I. Vasiljevic, J. Jitsev, L. Soldaini, A. G. Dimakis, G. Ilharco, W. Koh, S. Song, T. Kollar, Carmon, A. Dave, R. Heckel, N. Muennighoff, and L. Schmidt. Language models scale reliably with over-training and on downstream tasks, 2024. URL https 7/arxiv.
Y. Gal and Z. Ghahramani. A theoretically grounded application of dropout in recurrent neural networks, 2016.
www.oorgg///..o..o..ooo...ooooo/ alsp///bb/1111 05287
L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac'h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang. L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang. and A. Zou. The language model evaluation harness, 07 2024. URL ttt=ps://zeendo.orr http://nnnboooorrrrrrreaaae/ 12608602
Garipov, Izmailov, D. Podoprikhin, D. Vetrov, and A. G. Wilson. Loss surfaces, mode connectiv- ity, and fast ensembling of dnns, 2018.
M. Gerstgrasser, Schaeffer, A. Dey, R. Rafailov, H. Sleight, J. Hughes, T. Korbak, R. Agrawal, D. Pai. A. Gromov, D. A. Roberts. D. Yang, D. L. Donoho, and S. Koyejo. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data, 2024. URL
pettss:/aaxxie orgg alss 2404.440 01413
A. Gladstone, G. Nanduru, M. M. Islam, P. Han, H. Ha, A. Chadha, Y. Du, H. Ji, J. Li, and
T. Iqbal. Easrgy-based transformers are scalable learners and thinkers, 2025. URL https 02092
//axxiv.rrg/abs/220
S. Goyal, P. Maini, Z. C. Lipton, A. Raghunathan, and J. Kolter. Scaling laws for data filtering - data curation cannot be compute agnostic, 2024. URL BBLL://ppss///axt..or//////////
07177
S. Goyal, D. Lopez-Paz, and K. Ahuja. Distilled pretraining: A modern lens of data, in-context learning and test-time scaling, 2025. URL htttps://erxi...rrrttt http://wwww..rrr///5/50.........
12

---

A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sra-
vankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Wyatt, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hup- kes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovie, F. Guzmán, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang. G. Cu- curell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, A. Ibarra, Kloumann, Misra, I. Evtimov, J. Zhang. J. Copet, J. Lee, J. Geffert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, van der Linde, J. Billock, J. Hong, J. Lee, Fu, J. Chi, J. Huang, J. Liu, J. Wang. J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, J. Jía, K. V. Alwala, K. Prasad, K. Upasani, K. Plaw- iak, K. Li, K. Hcafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, L. Rantala- Yeary, L. van der Maaten, L. Chen, L. Tan, L. Jenkins, L. Martin, L. Madaan, L. Malo, L. Blecher, L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kar- das, M. Tsimpoukelli, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji. N. Zhang, O. Duchenne, O. Çelebi, P. Alrassy, P. Zhang, P. Li, P. Vasic, P. Weng. P. Bhargava, P. Dubal, P. Kr- ishnan, P. S. Koura, P. Xu, Q. He. Q. Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Maheswari, R. Girdhar, R. Patel, R. Sauvestre. R. Polidoro. R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang. S. Hosseini, S. Chennabasappa S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Georgiou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao, U. Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti, V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang, X. Wang, X. E. Tan, X. Xia, X. Xie, X. Jia, X. Wang, Y. Goldschlag, Y. Gaur, Y. Babaei. Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos. A. Singh, A. Srivastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Baevski, A. Fein- stein, A. Kallet, A. Sangani, A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. D. Paola, B. Paranjape, B. Liu, B. Wu, Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido. B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Liu, C. Wang, C. Kim, C. Zhou, C. Hu, C.-H. Chu, C. Cai, C. Tindal, C. Feichtenhofer, C. Gao, D. Civin, D. Beaty, D. Kreymer, D. Li, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang, D. Le, D. Holland, F. Dowling, E. Jamil, F. Montgomery, E. Presani, E. Hahn, E. Wood, E. Le, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Kokkinos, F. Ozgenel, F. Caggioni, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, G. Sizov, Guangyi, Zhang, G. Lakshminaraaaaann H. Inan, H. Shojanazeri, H. Zou, H. Wang, H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, H. Zhan, Damlaj. I. Molybog, I. Tufanov, I. Leontiadis, I.-E. Veliche, Gat, J. Weissman, J. Geboski, J. Kohli, J. Lam, J. Asher, J.. Gaya, J. Marcus, J. Tang. J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong, J. Jin, J. Yang, J. Cummings. J. Carvill, J. Shepard, J. McPhie. J. Torres, J. Ginsburg.
J. Wang, K. Wu, K. H. U, K. Saxena, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang, K. Chawla, K. Huang, L. Chen, L. Garg. L. A.
L. Silva, I. Bell, I.. Zhang, I. Guo, I.. Yu, I.. Moshkovich, I.. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Keneally, M. Liu, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang. M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. San- thanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. Mehta, N. P. Laptev, N. Dong, N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, P. Saab, P. Balaji, P. Rittner, P. Bontrager, P. Roux, P. Dollar, P. Zvyagina, P. Ratanchandani, P. Yuvraj, Q. Liang, R. Alao, R. Rodrigue7, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li, R. Hogan, R. Battey, R. Wang, R. Howes, R. Rinott, S. Mehta, S. Siby, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Mahajan, S. Verma, S. Yamamoto, S. Ramaswamy, S. Lindsay, S. Lindsay, S. Feng, S. Lin, S. C. Zha. S. Patil, S. Shankar, S. Zhang, S. Zhang, S. Wang, S. Agarwal, S. Sajuyigbe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Satterfiel, S. Govindaprasad, S. Gupta, S. Deng, S. Cho. S. Virk, S. Subramanian, S. Choudhury, S. Goldman,
13

---

T. Remez, T. Glaser, T. Best, T. Koehler, T. Robinson, T. Li, T. Zhang, T. Matthews. T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihaileseu, V. Ivanov, W. Li, W. Wang. W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wu, X. Wang, X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi, Y. Nam, Yu, Wang, Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He, 7. Rait, Z. DeVito, Z. Rosnbrick, Z. Wen, Z. Yang, Z. Zhao, and Z. Ma. The llama 3 herd of models, 2024. abcp/ab//2000 wtt../orrr/aaa/20.2111
URL
E. Grave, A. Joulin, and N. Usunier. Improving neural language models with continuous cache, 2016. URL/ttpss: http:::://axxvvvvo/a/ /ttp://aa......raba/b//22222....
Y. Gu, L. Dong, F. Wei, and M. Huang. Minillm: Knowledge distillation of large language models, 2024. URL
arxiaaaivv...... wttp:/vv.rg/ab//////00 08543
Hastie, A. Montanari, S. Rosset, and R. J. Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation, 2020. URL https://arxiv.orgg
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang. D. Song, and Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. http::/ /atpss///axxii //arxiv arg/abs//a htg/abs/2103 2203. 03874
T. Henighan, J. Kaplan, M. Katz, M. Chen, C. Hesse, J. Jackson, H. Jun, T. B. Brown, P. Dhariwal, S. Gray, C. Hallacy, B. Mann, A. Radford, A. Ramesh, N. Ryder, D. M. Ziegler, J. Schulman, D. Amodei, and S. McCandlish. Scaling laws for autoregressive generative modeling. 2020. URL
https: /taps:///axiiv org/ 201.
Hestness, Narang, Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. M. Patwary, Y. Yang, and Y. Zhou. Deep learning scaling is predictable, empirically, 2017. URL wwt.hhttts//rtte
org/abs hkg/abs/1712. 00409
J. Hestness, N. Ardalani, and G. Diamos. Beyond human-level accuracy: Computational challenges in deep learning, 2019. URL htt.hhttp:://tm //arxivv///rxv.... htttttss:///axx httpppp////aaaill.rr///////00000
G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network, 2015, URL https: /atp:s///xxivvve/aa org. httpg//bb//11333 1503.11330000 02531
Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifree Training compptte-pptimal large language models, 2022. URL https::/arxiv..trm org. abs// 2203. 155566
G. Huang, Y. Li, G. Pleiss, Z. Liu, E. Hopcroft, and K. Q Weinberger. Snapshot ensembles: Train 1, get m for free, 2017. URL https:/ http/raxiivvooo// wwtp:///////bb///.........
A. Q. Jiang, A. Sablayrolles, A. Roux, Mensch, B. Savary, C. Bamford, D. S. Chaplot, de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, I.. Saulnier, M.-.. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mixtral of experts, 2024. URL https: /ttp://axxivvooooo
04088
K. Jordan. On the variance of neural network training with respect test sets and distributions, 2024.
URL https:t///// httphtttp//cxixi/// http///xxivvorrr/ abs/ /304 01910
Juravsky, A. Chakravarthy, R. Ehrlich, S. Eyuboglu, B. Brown, J. Shetaye, C. Ré, and
A. Mirhoseini. Tokasaurus: An Ilm inference engine for high-throughput workloads. https
// //ccalingineelll welllliggncee gence stanford. edu/ http://cccc..boooo//oo//uuaaaaaa
2025.
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models, 2020. URL https: http:p//a/axxi..orr... //tpxxi.ox///////// abs/ /200 08361
14

---

N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang. On large-batch training for deep learning: Generalization gap and sharp minima, 2017. URL https: hwtttp://aaxii..//// abs/ / 1609. 04836
Y. Kim and A. M. Rush. Sequence-level knowledge distillation, 2016. URL https://arxii org/ abs/ 1 1606. 07947
B. Krause, E. Kahembwe, Murray, and S. Renals. Dynamic evaluation neural sequence models, 2017. URL CRLbttts::/// http://rxxvv.r.ro 0abs/1709.074
Kumar, Z. Ankner, B. Spector, B. Bordelon, Muennighoff, M. Paul, C. Pehlevan, C. Ré, and
A. Raghumathan. Scaling laws for precision, 2024. URL
04330
B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles, 2017. URL httttt:://axxi..ottttt wwwp//////bbbba..ee.eeeeeeeeee/e 01474
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11)::278-232 1998. doi: 10.1 10.1109/..72671..
Y. LeCun, L. Bottou, G. B. Orr, and K. R. Müller. Efficient BackProp, pages 9-50. Springer Berlin Heidelberg, Berlin, Heidelberg, 1998. ISBN 978-3-540-4943004 doi: 1007773-540-44430--222
URL (tRpsttps: /tt::://dooooooo org/ 0 1007000-544--945333000
Li, A. Fang, G. Smymiss M. Ivgi, M. Jordan, S. Gadre, Bansal, E. Guha, Keh, K. Arora, S. Garg, R. Xin, N. Muennighoff, R. Heckel, J. Mercat, M. Chen, S. Gururangan, M. Wortsman, A. Albalak, Y. Bitton, M. Nezhurina, A. Abbas, C.-Y. Hsich, D. Ghosh, Gardner, M. Kilian, H. Zhang, R. Shao, S. Pratt, S. Sanyal, G. Ilharco, G. Daras, K. Marathe, A. Gokaslan, J. Zhang,
K. Chandu, T. Nguyen, I. Vasiljevic, Kakade, S. Song. S. Sanghavi, F. Faghri. S. Oh, L. Zettle-
moyer, K. Lo, A. El-Nouby, H. Pouransari, A. Toshev, S. Wang. D. Groeneveld, L. Soldaini, P. W. Koh, J. Jitsev, T. Kollar, A. G. Dimakis, Y. Carmon, A. Dave, L. Schmidt, and V. Shankar. Datacomp-lm: In search of the next generation of training sets for language models, 2025. URL
https: a/aaxiiivv/// org. abs/ 2406. 11794
J. Liu, J. Su, X. Yao, Z. Jiang, G. Lai, Y. Du Y. Qin, W. Xu, E. Lu, J. Yan, Y. Chen, H. Zheng, Y. Liu, S. Liu, B. Yin, W. He, H. Zhu, Y. Wang, J. Wang, M. Dong, Z. Zhang, Y. Kang, H. Zhang, X. Xu, Y. Zhang, Y. Wu, X. Zhou, and Z. Yang. Muon is scalable for Ilm training, 2025. URL
URL http:////aaxi..oo//aa////// htt.z//ab///55222 16982
E. Lobacheva, N. Chirkova, M. Kodryan, and D. Vetrov. On power laws in deep ensembles, 2021.
08483
Maini, S. Seto, H. Bai, D. Grangier, Y. Zhang, and N. Jaitly. Rephrasing the web: A recipe for compute and data-efficient language modeling, 2024. URL hRR/httss///axivvorrr www.axiww.org//bb.oaoo//as//bbre
24C1.16388
M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):33-330, 1993. URL https
//aclanthology.
M. Marek, S. Lotfi, A. Somasundaram, A. G. Wilson, and M. Goldblum. Small batch size training for language models: When vanilla sgd works, and why gradient accumulation is wasteful, 2025.
URL httlhhtt////// /tttpp://arxivv arxiv. org. wttp:xiv.rgg/b//25...oooo 07101
S. McCanddlihh, J. Kaplan, D. Amodei, and O. D. Team. An empirical model of large-batch training, 2018. URL/tttps///axxivccot wwg.:/ow.rgggo....... waazz////bb/la1/ 06162
S. Merity, N. S. Keskar, and R. Socher. Regularizing and optimizing Istm language models, 2017. URL htt.hhttss://axiive wttp://wwwwrg//////////// 02182
T. Mikolov, M. Karafiát, L. Burget, J. Cernocky, and Khudanpur. Recurrent neural network based language model. volume 2, pages 1045-1048, 09 2010. doi: (0.21437/interspeech.hl 2011-0033...
15

---

H. Mobahi, M. Farajtabar, and P. L. Bartlett. Self-distillation amplifies regularization in hilbert space, 2020. URLpttps: //tpssi///xxvv.. http://wrrr/aaab/2....0.5 05715
N. Muennighoff, A. M. Rush, B. Barak, T. L. Scao, A. Piktus, N. Tazi, S. Pyysalo, T. Wolf, and
C. Raffel. Scaling data-constrained language models, 2023. /RLp:ttp:///axxv org. abs/ 23C5.16262
N. Muennighoff, L. Soldaini, D. Groeneveld, K. Lo, J. Morrison, S. Min, W. Shi, P. Walsh, O. Tafjord, N. Lambert, Y. Gu, S. Arora, A. Bhagia, D. Schwenk, D. Wadden, A. Wettig, B. Hui, T. Dettmers,
D. Kiela, A. Farhadi, N. A. Smith, P. W. Koh, A. Singh, and H. Hajishirzi. Olmoe: Open
mixture-of-experts language models, 2025.
P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak, and Sutskever. Deep double descent: Where bigger models and more data hurt, 2019. URL
991. 02290
Nakkiran, P. Venkat, S. Kakade, and T. Ma. Optimal regularization can mitigate double descent, 2021. UtLhtts::
/aaorg//b///// 01897
J. Ni, the, and team. per data learners.
Diffusion language
models
wot.ieeeinnooioonoo. site/
Diffusi cersiion-ie Language Ltffusio--lnnguggeeWaaae Models are are--Suerr otteerceerrrrre22222000006653111 2025. Notion Blog.
M. Prabhudesai, M. Wu, A. Zadeh, K. Fragkiadaki,
and and D. Pathak. Diffusson beats autoregressive in
data-constrained settings, 2025, UK/httpps//aaxxx../////////// 15857
S. Rosenfeld, A. Rosenfeld, Y. Belinkov, and N. Shavit. A constructtive prediction of the general- ization error across scales, 2019. UKt_httpe://axxxxv.rrr///bb//101
Y. Ruan, C. Maddison, and Hashimoto. Observational scaling laws and the predictability of language model performance, 2024. www.lhttpas./aaix...rrrr///
10938
Y. Ruan, N. Band, C. J. Maddison, and T. Hashimoto. Reasoning learn from latent thoughts, 2025. URL htttpp::/arvvl..orr///b/////
B. S. Ruben, W. L. Tong. H. T. Chaudhry, and C. Pehlevan. No free lunch from random feature ensembles, 2024.
V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, /ttp////////////aa/oooo///////// 01108
N. Sardana, J. Portes, S. Doubov, and Frankle. Beyond chinchcihhhnclllla-ppptmaaa Accounting for inference in language model scaling laws, 2025. URL https: abs
00448
J. Sevilla and E. Roldán. grows by 4-5x per year, 2024. training at ootpr:/aooooooolorrrroooo------- Accessed: 2025-08-21.
Training
compute of frontier ai models
URL http:://epoch
C. Shannon. Prediction and entropy of printed english. Bell system technical journal, journal, 30(1):50-64, 1951.
N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gen gated mixture-of-experr layer, 2017. URL httpps///axxx org/abs/1701. 06338
H. Shi, K. Livescu, and K. Gimpel. Substructure substitution: Structured data augmentation for nlp, 2021. /uBl/ttpp:://xx....rog///// wwtp:////o////....o////// 00411
I. Shumailov, Z. Shumaylov, Y. Zhao, N. Papernot, R. Anderson, and Y. Gal. Ai models collapse when trained recursively generated data. Nature, 631 631(8002):755-759) 2024.
J. B. Simon, D. Karkada, N. Ghosh, and M. Belkin. More is better in modern machine learning: when infinite overparameterizztion is optimal and overfitting is obligatory, 2024. URL https 7/arxive //axxiv..rrr//bb//331.111
16

---

S. P. Singh and M. Jaggi. Model fusion via optimal transport, 2023. URL mtlLttaa//axxx..or abs/ abs/1910 1910.05653
L. Smith, E. Elsen, and S. De. On the generalization benefit noise in stochastic gradient descent, 2020. httpp:://a/axiii http://///aaaiiivooora//a/////// 10000
C. Snell, J. Lee, K. Xu, and A. Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024.
03314
B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning, 2023. URL httphttt///xxx...crrtt 14486
2206
J. M. Springer, S. Goyal, K. Wen, T. Kumar, X. Yue, S. Malladi, G. Neubig, and A. Raghunathan. Overtrained language models are harder to fine-tune, 2025. URL http:tttt/aa...orpp///// http///rxxxv..r//bb..s.s.
2503. 1.5.1995 19206
D. Su, K. Kong, Y. Lin, J. Jennings. B. Norick, M. Kliegl, M. Patwary, M. Shoeybi, and B. Catanzaro. Nemotron-cc: Transforming common crawl into refined long-horizon pretraining dataset, 2025. www.:/ogggab////// 02595
htRlhhtps:://aat
C. Summers and M. J. Dinneen. Nondeterminism and instability in neural network optimization,
2021. http:::://a.....//////ta//
R. Sutton. The bitter lesson.
http://wwy/ http://www.nnccmmlleteeeeee..d wtw.:/daaaa.eettnneeeeeeeeeeee
BitterLesson html 2019. Blog post.
Takase, J. Suzuki, and M. Nagata. Direct output connection for high-rank language model, 2018.
URL httht:tts///iiii/ wwg. httpg/ab/18880 10143
R. Taori and T. B. Hashimoto. Data feedback loops: Model-driven amplification of dataset biases, 2022. URL htttl::t////atttt /ttp///avvvv/.. orq/ aa/////////// 22..222999999
G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova,
A. Ramé, M. Rivière, I.. Rouillard, T. Mesnard, G. Cideron, J. bastien Grill, S. Ramos, E. Yvinec, M. Cashon, E. Pot, I. Penchev, G. Liu, F. Visin, K. Kenealy, L.. Beyer, X. Zhai, A. Tsitsulin, R. Busa-Fekete, A. Feng, N. Sachdeva, B. Coleman, Y. Gao, B. Mustafa, I. Barr, E. Parisotto, Bhupatiraju, R. Agarwal, M. Kazemi, D. Malkin, R. Kumar, D. Vilar, I. Brusilovsky. J. Luo, A. Steiner, A. Friesen, A. Sharma, A. Sharma,
D. Tian, M. Eyal, C. Cherry, J. Peter, D. Sinopalnikov,
A. M. Gilady, A. Goedeckemeyer, A. Saade, A. Feng, A. Kolesnikov, A. Bendebury, A. Abdagic, A. Vadi, A. György, A. S. Pinto, A. Das, A. Bapna, A. Miech, A. Yang, A. Paterson, A. Shenoy, A. Chakrabarti, B. Piot, Wu, B. Shahriari, B. Petrini, C. Chen, L. Lan, C. A. Choquette-Chon
C. Carey, C. Brick, D. Deutsch, D. Fisenbud, D. Cattle, D. Cheng. D. Paparas, D. S. Sreepathihalli,
D. Reid, D. Tran, D. Zelle, E. Noland, E. Huizenga, E. Kharitonov, F. Liu, G. Amirkhanyan, H. Singh, H. Mehta, H. T. Lehri, H. Hazimeh, Ballantyne, I. Szpektor, I. Nardini, J. Pouget-Abadies J. Chan, J. Stanton, J. Wieting, J. Lai, J. Orbay, J. Fernandez, J. Newlan, J. yeong Ji, J. Singh, K. Black, K. Yu, K. Hui, K. Vodrahalli, K. Greff, L. Qiu, M. Valentine, M. Coelho, M. Ritter, M. Hoffmma,, M. Watson, M. Chaturvedi, M. Moynihan, M. Ma, N. Babar, N. Noy, N. Byrd, N. Roy, N. Momchev, N. Chauhan, N. Sachdeva, O. Bunyan, P. Botarda, P. Caron, P. K. Rubenstein, P. Culliton, P. Schmid, P. G. Sessa, P. Xu, Stanczyk, P. Tafti, R. Shivanna. Wu, R. Pan, Rokni, R. Willoughby, R. Vallu, S. Jerome, S. Smoot, S. Girgin, S. lgbal. S. Reddy. S. Sheth, S. Pöder, S. Bhatnagar, S. R. Panyam,
G. Cameron, H. Hashemi, H. Siimczak-Plucifsseee
Mullins,
S. Figer, S. Zhang, T. Liu, T. Yacovone, T. Liechty, U. Kalra, U. Evei, V. Misra, V. Roseberry, V. Feinberg, V. Kolesnikov, W. Han, W. Kwon, X. Chen, Y. Chow, Y. Zhu, Z. Wei, Z. Egyed,
V. Cotruta, M. Giang, P. Kirk, A. Rao, K. Black, N. Babar, Lo, E. Moreira, L. G. Martins,
O. Sanseviero, L. Gonzalez, Z. Gleicher, T. Warkentin, V. Mirrokni, E. Senter, E. Collins, J. Barral,
Z. Ghahramani, R. Hadsell, Y. Matias, D. Sculley. S. Petrov, N. Fiedel, N. Shazeer, O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu. C. Farabet, E. Buchatskaya, J.-B. Alayrac, R. Anil. Dmitry, Lepikhin, S. Borgeaud, O. Bachem, A. Joulin, A. Andreev, C. Hardin, R. Dadashi, and L. Hussenot. Gemma 3 technical report, 2025a. URL http:/p://.aaai...ggaaa///////// 19786
17

---

K. Team, Y. Bai, Y. Bao, G. Chen, J. Chen, N. Chen, R. Chen, Y. Chen, Y. Chen. Y. Chen, Z. Chen, Cui, H. Ding, M. Dong, A. Du, C. Du, D. Du, Y. Du, Y. Fan, Y. Feng, K. Fu, B. Gao, H. Gao, P. Gao, T. Gao, X. Gu, L. Guan, H. Guo, J. Guo, H. Hu, X. Hao, T. He, W. He, W. He, C. Hong, Y. Hu, Z. Hu, W. Huang, Z. Huang, Z. Huang, T. Jiang, Z. Jiang, X. Jin, Y. Kang, G. Lai, C. Li,
R. Li, H. Li, M. Li, W. Li, Y. Li, Y. Li, Z. Li, Z. Li, H. Lin, X. Lin, Z. Lin, C. Liu, C. Liu, H. Liu, J. Liu, J. Liu, L. Liu, S. Liu, T. Y. Liu, T. Liu, W. Liu, Y. Liu, Y. Liu, Y. Liu, Y. Liu, Z. Liu,
E. Lu, L. Lu, S. Ma, X. Ma. Y. Ma. S. Mao, J. Mei, X. Men, Y. Miao. S. Pan, Y. Peng, R. Qin,
B. J.,u Z. Shang, L. Shi, S. Shi, F. Song. J. Su, Z. Su, X. Sun, F. Sung, H. Tang. J. Tao, Q. Teng,
C. Wang, D. Wang. F. Wang, H. Wang. J. Wang, Wang. Wang. S. Wang, S. Waang Y. Wang.
Y. Wang, Y. Wang, Y. Wang. Y. Wang. Z. Wang, Z. Wang, Z. Wang, C. Wei, Q Wei, W. Wu,
X. Wu, Y. Wu, C. Xiao, X. Xie, W. Xiong, B. Xu, J. Xu, J. Xu, L. H. Xu, L. Xu, Xu, W. Xu,
X. Xu, Y. Xu, Z. Xu, J. Yan, Y. Yan, X. Yang, Y. Yang. Z. Yang. 7. Yang. 7. Yang. H. Yao, X. Yao, W. Ye, Z. Ye, B. Yin, L. Yu, E. Yuan, H. Yuan, M. Yuan, H. Zhan, D. Zhang, H. Zhang, W. Zhang,
X. Zhang, Y. Zhang. Y. Zhang, Y. Zhang. Y. Zhang. Y. Zhang, Y. Zhang, Z. Zhang, H. Zhao,
Y. Zhao, H. Zheng, S. Zheng. J. Zhou, X. Zhou, 2. Zhou, Z. Zhu, W. Zhuang, and X. Zu. Kimi k2: Open agentic intelligence, 2025b. URL / arxiv. org/ alss 2507 20534
T. Thrush, C. Potts, and T. Hashimoto. Improving pretraining data using perplexity correlations, 2025. BRRttts:///axxii wwwpa/iwww.gg..///.///a..a.. /2009. 05816
A. W. Van der Vaart. Asymptotic statistics, volume 3, Cambridge university press, 2000.
H. van Hasselt, Guez, and Silver. Deep reinforcement learning with double q-learning, 2015. URL https:://axxi: org/
66
P. Villalobos, A. Ho, J. Sevilla, T. Besiroglu, L. Heim, and M. Hobbhahn. Will we run out of data? limits of Ilm scaling based on human-generated data, 2024. URL hattps://arxa www.aiiwww.aa...aaaaaaaaaaooo org/orgg///a... abs//
24325
N. Vyas, A. Atanasov, B. Bordelon, D. Morwani, S. Sainathan, and C. Pehlevan. Feature-learning networks are consistent across widths at realistic scales, 2023. URL htttps://aaxiv..o/ abs 2205.2355.o 18411
Z. Wang, F. Zhou, X. Li, and P. Liu. Octothinker: Mid-training reneeeennnnneeeeeeeeeneen learning scaling, 2025. URL /ttt::://aaaai..or//bbb////////s
reinforcement
20512
A. Warstadt, L. Choshen, A. Mueller, A. Williams, E. Wilcox, and C. Zhuang. Call for papers the babylm challenge: Sample-efficien pretraining on developmentally plausible corpus. 2023. URL https: /wtpiv..ivv///////e//eeeeeaaaev httgg//bb//33311 .1..3011..71...
Welbi, N. Liu, and M. Gardner. Crowdsourcing multiple choice science questions, 2017. URL https://arxxt org/ wwtp/b/./////ooooo./t..///aaaara /tt.///110 1707. 06209
K. Wen, D. Hall, T. Ma. and P. Liang. Fantastic pretraining optimizers and where to find them, 2025.
URL httptttss/s//ttteeee org/ abs 25t../509..0 02046
M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, and L. Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, 2022. URL https: /
/ttps://arxiv.. arxiv. org/ abc//as//220 wwwp/////22222 05482
Z. Xie, S. I. Wang. J. Li, D. Lévy, A. Nic, D. Jurafsky, and A. Y. Ng. Data noising as smoothing in neural network language models, 2017. URL https: /ttravivvvv...///// wttpw/wwwwwww////////..//// 02573
A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang. Tu, J. Zhang, J. Yang. J. Yang. J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang. X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu. Qwen3 technical report, 2025. URL https: /taps:///arivv ww.. als /2505.0938
18

---

G. Yang, J. Hu, I. Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder. J. Pachocki, W. Chen, and Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer, 2022. URL org/ abs 22t./223....52 03466
Z. Yang, Z. Dai, R. Salakhutdinov, and W. W. Cohen. Breaking the softmax bottleneck: A high-rank mn language model, 2018. URL ht potttps///e /ttppi///axvvv.o/ wttpaaiww..gga..aaaaoooooooooott abs/ 7711 03953
Z. Yang, N. Band, S. Li, E. Candès, and T. Hashimoto. Synthetic continued pretraining, 2024. URL https://arxiv. httpvvv....//////05553333 07431
W. Zaremba, Sutskever, and O. Vinyals. Recurrent neural network regularization, 2015, URL
L. Zhang, J. Song, A. Gao, J. Chen, C. Bao, and K. Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation, 2019. URL https://arxxi org/abs/ 1905. 08094
G. Zilly, R. K. Srivastava, Koutník, and Schmidhuber. Recurrent highway networks, 2017. URL https::/arxiv.crm abs 607. 03474
19

---

CONTENTS
Introduction
2 Standard pre-training
2.1
Evaluating existing data-constrained recipes
Regularized parameter scaling
Ensemble scaling
4.1
Defining ensembles
Scaling member count instead of parameter count
4.3
Joint scaling recipe composing parameter and ensemble scaling
5
Scaling the seed token count under infinite compute
6
5.1
Data scaling laws for single model recipes
5.2
Data scaling laws for ensembles
5.3
Data scaling analysis
Data efficiency under parameter constraints
6.1
Reducing final parameter count via ensemble distillation
6.2
Reducing train parameter count via self-distillation
Downstream tasks
8
Related Work
9
Discussion
10 Acknowledgements
10
Ethics
10
12 Reproducibility
10
Continued pre-training
22
B Problem setting
22
Standard pre-training details
23
Locally optimal hyperparameters
23
Ablating coordinate descent
24
Tuned hyperparameters)
24
Hyperparameter ablations
25
20

---

C.5 Overltting analysis
25
D
Ensembling details
27
D.I
Ensembling formalization,
27
D.2 Overview of ensembling tuning
27
D.3 Seed science
28
D.4
Hyperparameter tuning ensembles
28
D.5
Alternatives to ensembling
29
D.5.1 -fxture
29
D.5.2 Model soups
29
D.6 Order of limits.
Data scaling
32
Epoch tuned baseline
32
E,2 Scaling parameter count
32
Scaling member and parameter count
32
Distillation details
Data generation
33
Hyperparameters
33
F3
Mixing data ablation
34
Downstream task details
34
Downstream tasks
34
G.2
Hyperparameter tuning
34
Continued pre-training
35
H.I Hyperparameters
35
H22 CPT soups
Power laws
36
Sensitivity analysis
1.2 Fitting laws
36
Additional related work
36
K Updates during rebuttals
38
New model sizes
38
K.2
Mixture-of-Expett
38
Goodness-offfit and extrapolation
39
K.3.1 Goodness-of-ftt
39
21

---

K.3.2 Extrapolation
39
More evaluations
39
A CONTINUED PRE-TRAINING
We demonstrate the immediate applicability of our findings in settings outside of pre-training from scratch by improving the data efficiency of existing CPT recipes. We adopt the setup from Wang eet al. (2025) of performing continued pre-training on Llama 3.2 3B Base (Grattafiori et al.) 2024) with the MegaMath Bee---eb-PPp---- Pro mid-training dataset. To simulate data-constrained setting, restrict ourselves to only 4B seed tokens of the full 73B tokens. We evaluate math performance by via accuracy
subset of representative benchmarks from Wang et al. (2025): GSM8K (Cobbe et al. 2021 MATH (Hendrycks et al. 2021), and MathQA (Amini et al. 2019).
In Table we show that many of our results for pre-training from seratch transfer to continued pre-traiming setting. We start with CPT baseline (Default) from the reference hyperparameters of Wang et al. (2025) which gives 6.34% improvement over the base model. We then apply our interventions from Appendix C.4 Section by decreasing batch size and epoching, and we find that weight decay was not helpful for CPT. Our final single model baseline provides an additional 5.23% lift in average accuracy on top of the reference hyperparameters. See Appendix H for details.
We further show that ensembling eight epoched models provides an additional 4.76% gain average accuracy just single epoched CPT model (K 1). We observe that average accuracy scales with increasing ensemble member count, and our best ensembles exceed the performance of a baseline continually pre- trained the full 73B tokens, providing 17.5 7..5x data efficiency win.
Table 1: Data efficiency improvements OctoThinker. We take reasoning mid- mid-training data from Wang et al. (2025) and apply our data efficiency interventions of reducing batch size (Ap- pendix C.4), epoching data, and ensembling multiple models. Our best ensemble utilizing 4B tokens outperforms vanilla CPT on 73B tokens following the original paper's 's training hyperparameters, resulting x data efficiency improvement.
CPT (4B tokens)
e-enemmbles
Benchmarks
Llama 3B
CPT (73B tokens)
Default
Lower BS
Epoching (K
1)
K =
K
4
=
GSM8K
28.23
38.44
44.50
44.05
49.28
51.80
52.99
49.51
MATH; sloo)
6.90
14.38
19.74
21.84
23.04
MATHQA,p-a
35.07
38.96
41.31
42.58
45.12
46.06
45.26
44.79
Averuge
24.25
30.59
34,48
35.82
38.79
40.35
40.58
39.23
B PROBLEM SETTING
Pre-training algorithm. We instantiate the pre-training algorithm using a standard pre-training recipe developed through the Marin project (https: :/marri///r iimarinn community) by following the best practices shared publicly and found internally.
Optimizer. We train with AdamW, either with a default of 0.1 or a tuned weight decay. We set other hyperparameters to standard defaults (31 = 0.9, B2 0.95, 10-8). We clip the norm of the gradient at 1.
Learning rate. We use a cosine learning rate schedule with a warmup for the first 1% of training, decaying to 0 by the end of training. We always tune learning rate for all of our experiments. Every run has own learning rate schedule and we never report the loss before the learning rate anneals zero in the main body.
Architecture. We train Llama-style auto-regressive language models. We specify the main architectural choices Table 2 When scaling models, we change the initialization scheme to have variance inversely proportional to the hidden dimension (this is known com/
to outperform pP (Yang et al.) 2022) within our framework: https:: For other architectural choices, we default
mar farii n communitmotooooommmm wttammunnty/marrrr
621
22

---

SiLU activations, untied word embeddings, and rotary position embeddings. We note that the non-standare scaling for the 1.4B model a consequence of using presets in our pre pre-training framework.
Systems. We train in mixed eeeedippecision with parameters in fp32 and compute + output in 6. Most of our jobs were run on v4-64 or v4-128 TPUs, with bitwise-determinism for handling preemption and promoting reproducibility.
Data. We pre-train using DCLM data (Li et al. 2025). We keep a fixed validation set of 1024 sequences (4 million tokens) across all experiments for clear comparison. When increasing the size of the train pool, we ensure that smaller pools are a subset of larger pools.
Data order. We generate a random permutation of the windows after tokenization and use this same permutation across epochs. We note that performance might have been better if we used a unique permutation every epoch but keep this fixed across models which further reduces randomness of training.
Parameter
150M
300M
600M
1.4B
Context Length
4096
4096
4096
4096
Hidden Dimension
512
768
1024
2048
Intermediate Dimension
1792
2688
3584
7168
Attention Heads
8
12
16
16
KV Heads
8
12
Layers
6
12
24
16
Table 2: Model architecture configurations for different model sizes. if not specified.
We default to the 300M model
C STANDARD PRE-TRAINING DETAILS
C.I LOCALLY OPTIMAL HYPERPARAMETERS
We are interested finding the best setting of hyperparameters (e.g. learning rate, epoch count, and weight decay) for a fixed parameter count and token count in the data-efficient pre-training setting. To make this search problem tractable, we first discretize the space of hyperparameters (e.g. only try epoch counts that powers of Under this discretization, it prohibitively expensive to try every possible hyperparameter selection within our grid. Therefore, we search for locally-optimal hyperparameters as defined below.
Definition 1 (Locally-optimal hyperparameters). We define the neighborhood B(H) of hyperparame- ter tuple H containing mn variables as the 2m neighbors from incrementing/lecreeennng exactly one of the variables. We say H is locally optimal if and only if
WH' B(H), L (A,,,,, H)) S ((A(,N,,
Under certain assumptions, the locally-optimal hyperparameters are also globally optimal (for example, if the loss was convex in each dimension when the other variables are fixed). Though this may seem like big assumption, did not observe counter-examples to this in early experiments. One can verify this property for the single model losses presented in Figure 17
Assuming this property,
the following search procedure
Seed the search with initial runs around best guess for optimal hyperparameters (heuristi- cally set by us, using all runs until this point in time)
2. Take the best run so far and run its neighbors.
3. If any of its neighbors is better than the current candidate, the current candidate is sub- optimal. Repeat step 2 with the new best run.
4. If this run is better than all of its neighbors, we consider it "certified" and terminate the search procedure.
23

---

Abloting hyperpaaraettrr tuning
3.5
Figure 10: Ablating joint tuning procedure. We gener- ally jointly tune learning rate, epoch count, and weight decay separately for each parameter count (purple). We show that trying to naively transfer hyperparameters across scales is a bad idea. Red: fixing weight decay to 0.1 (default regularization). Green: assuming that the optimal weight decay for 150M models (0.8) is optimal across N. Blue: assuming that the optimal epoch count for 150M models (16) optimal across N.
2.5
3.6
162
333
speche Ooneline)
150
Fodig 58 wighi decry
Faing ppoch
345
Though this procedure is quite expensive, we found it better than other natural heuristics which don't rigorously follow this procedure separately for each parameter and token count (Appendíx C.2). Furthermore, it seems to give clean scaling in both loss and hyperparameters, suggesting that the hyperparameter optimization landscape nice enough for this coordinate descent algorithm work. We note that this is simplified version of the assumptions used in Wen et al. (2025)
Our specific discretization was forcing learning rate to be or 3 times a power of 10, epoch count be an integer power of 2, and weight decay to be either 0.0 or an integer power of 2 times 0.1. To further restrict the search space, we bounds based on initial experiments tuning these hyperparameters. with maximum learning rate of 3e-3, maximum weight decay of 6.4, and maximum epoch count of 64 (these bounds only triggered for three of our searches).
C.2 ABLATING ON COORDINATE DESCENT
For the parameter scaling experiments, we jointly tune learning rate, epoch count, and weight decay. We find this joint tuning necessary for clean scaling. To demonstrate this, consider three alternatives and show their scaling in Figure TO
1. Fixing weight decay 0.1 and tuning learning rate and epoch count (red). This baseline has already been shown to fail in Section 2.1
2. Jointly tuning weight decay only for 150M (green). Here, we jointly tune the weight decay for the 150M model, finding the optimal value is 0.8. We then assume this is the optimal value for higher parameter counts and correspondingly tune learning rate and epoch count. We find that this scaling is not even monotonic.
3. Jointly tuning epoch count only for 150M (blue). Here, we jointly tune the epoch count for the 150M model, finding the optimal value is 16. We then assume this is the optimal value for higher parameter counts and correspondingly tune learning rate and weight decay. We find that this scaling plateaus much faster than the regularized recipe.
This shows the importance of jointly tuning both weight decay and epoch count at each model scale instead of blindly hoping for transfer.
C.3 TUNED HYPERPARAMETERS
In Figure IT we share the locally optimal hyperparameters we found for 4 different token counts and 4 different parameter counts. We notice few trends when looking at the resulting hyperparameters and power laws.
The optimal learning rate decreases for larger models, noted by prior work (Yang et al. 2022 Everett et al. 2024). The optimal learning rate does not strongly depend on the number of tokens.
The optimal weight decay increases for larger models. Similarly, the optimal weight decay decreases for larger token counts. When fixing the garameter-to-tokee ratio, the weight decay stays around 0.8.
24

---

201
419M seed tokanss
RIRM
seed tokens
37x
355
36 10
35
74-
0.09:N00977
10
33
3,6x 10
35xt0
53x
32-10
34
10
235
32 10
31-11P
31
3+122
325
KSMM
150M
656M
150M
S0UM
603M
Madkd DERLITOIIS
paranaters
DROM
5XM
3KM
WD
1A
W0
15
16
Se-23
Le433
TedEs
1F
3--11
Se-th
1e45)
164
16
2--33
3e4t5
5e40
1e40
LK
1440
Te40
In40
Figure 11: Tuned hyperparameters for regularized scaling. We show the optimal hyperparameters tuned seperately for each parameter and token count. We find that parameter count increases, optimal weight decay goes up, optimal epoch count goes down, and optimal learning rate goes down. We find the trends for weight decay and epoch count hold when token count decreases.
The optimal epoch count decreases for larger models. Similarly, the optimal epoch count increases for larger token counts. When fixing the parameter-to-ton to-kee ratio, the epoch count stays around 16.
The power laws fit across all token counts share similar scaling exponents close to This holds even though almost every model is aver-parameterized at 200M tokens and under- parameterized at 1.6B tokens.
C.4 HYPERPARAMETES ABLATIONS
We perform additional ablations on hyperparameters to understand their role beyond their optimal values. We start from recipe of single epoch pre-training with 0.1 weight decay and tuned learning rate, and build our way up to tuning all hyperparameters.
Batch size. is known that when optimizing for throughput, it is best to train at the "critical batch size" to best utilizes hardware (McCandlish et al. 2018). However if we drop this constraint and instead measure performance for fixed number of data points, we find that it best use smaller batch sizes, as shown in Figure T2 left, corroborating prior work in optimization (Smith et al. 2020 Keskar al. 2017 7 LeCun et al. 1998 Marek al. 2025). We use batch size of 64, which the smallest size that is practical for our hardware.
Weight decay. It is known that regularization can further improve generalization when repeating data (D Angelo et al. 2024.. Figure 12 right shows how varying the weight decay impacts epoched models (1.4B parameters for 8 epochs vs 300M parameters for 16 epochs). More over wver-paameerrized models require a larger amount of regularization (3.2 1.6). Without optimally tuning weight decay, one may draw the incorrect conclusion that larger models are worse than smaller models in the data-constrained setting. We also reproduce findings that higher weight decay enables higher optimal learning rate and epoch count, shown by contrasing Figure and Figure 3
We find that increasing weight decay strongly changes the training dynamics. Though the train and validation losses decrease much slower at the start of training with high weight decay, they decrease rapidly by the end of training, eventually beating the run with low weight decay. In Figure [3] we visualize this for the best run with weight decay 0.1 and the best run tuning weight decay. This phenomenon also holds when using higher weight decay on top of the best 0.1 weight decay hyperparameters. This is in line with previous findings in optimization research (Wen et al., 2025 Liu et al. 2025 | Angelo al. 2024) and suggests that we should only look at the performance at the end of training.
C.5 OVERFITTING ANALYSIS
In Section 2.I we discuss how introducing too many epochs or parameters results in validation loss going up which we believe is due over-fitting. In Figure we track train loss for the interventions
25

---

Batch size
Tuning weight decay
40
70
37-
6.0
300 pammetee
1.40 parameioos
Batch size
Weighht decry
Figure 12: Re-evaluating hyperparameters for standard pre-training. Left: smaller batch sizes
better, stop at 64 since this is the smallest our hardware practically supports (shown for epoch training with 0.1 weight decay and a fixed learning rate of 3e-3). Right: weight decay helps, and the optimal weight decay higher for larger models (300M is 16 epochs 3e-3 learning rate, 1.4B is 8 epochs lc-3 learning rate).
Volidation
Weight Weight decay 16
40
40
3.8
12
10
(bilioos))
Loss trajectories for different weight decays. We compare the best run with default weight decay (8 epochs, 1e-3 learning rate, 0.1 weight decay) and the best run with tuned weight decay (16 epochs, 3e-3 learning rate, 1.6 weight decay) for 200M tokens and 300M parameters. We find that loss for runs with high weight decay decreases much more slowly the start of training, but quickly decreases near the end of training.
Figure 13:
26

---

Increasing epuch count train loss
Increasing parameter count trein loes
5.0
4.5
3.-
4.0
36-
355
30-
Tuned epochs train loss
25
Validation
epochu validation
Train loss
appoche train lows
30
Epochs
Paremcter count
Figure 14: Train losses for epoching and parameter scaling. Left: Increasing epoch count results in train loss decreasing while validation loss starts increasing. Right: Increasing parameter count does not always decrease loss, potentially due to optimal epoch count changing (8, 4). Train loss monotonically goes down when restricted to 4 epochs.
of increasing epoch count and parameter count. On the left, we show how increasing epoch count monotonically decreases train loss but eventually results in validation loss going up. On the right, we show how increasing parameter count results erratic changes in train loss. We hypothesize this is because the optimal epoch count changes from 8 for the first two models to 4 for the last two models. We find that when we restrict all models to only use epochs, train loss decreases monotonically and validation loss still goes up, suggesting fvet-fitting.
We note that another reason over- fitting may be happening is because our 1.4B model trades depth model scaling
for width. We did not recognize experiments had finished because these were the default settings in our pre-training framework. We do not think this is severe issue since correctly tuning weight decay seems to correct for the fact that this architecture has less layers. Moreover, the large improvement from weight decay is also suggestive of the fact that larger models over-fitting.
non-standard until the majority of
D ENSEMBLING DETAILS
D.1 ENSEMBLING FORMALIZATION
The ensembling pre-training algorithm & accepts standard pre-training algorithm A and trains K members that are identical except for random seed Zr controlling the data order and model initialization. The output of the ensembling algorithm is model that averages the logits of the K models, computed by querying all K models. More formally, we define the ensembling algorithm as
EA(D, K, H) = LogitAvg (IA,D,N, Z,, H))
for randomness Z,, where Logit.vg produces model with likelihood for sequence æ given by
LogitAvg (MiE(K) (x) x exp
Σ KE
log (M)
K
D.2 OVERVIEW OF ENSEMBLING TUNING
Given the success of ensembles, study how to tune their hyperparameters. In Figure 4 the ensemble members were trained using the best hyperparameters for a single model, but this may not be optimal for large K ensembles. More formally,
argmin C (A(D,N,H) H)) = argmin N, 1, H,H + argmin lim ((E,(D,N,K,H))
K-00
best single model
for ensemble member asymptote
27

---

Verying creemble hypeeperramdte
Single
membber asyasptole
Best contprate Reppaaa 8. eeeedoo (2t:3125k7 3390
3.383
3:023
40-
38-
16-
3:64
3,399
3,307
-30
3,296
3,02
3.36
3
32-
3,500
355
3271
3
4
24431
32065
3,333
3333
32521
35
06
Weglh Docry
Mogll Doury
Figure 15: Tuning hyperparameters of ensemble members for lowest asymptote under K - co. We construct ensembles for different K when varying epoch count and weight decay. We find that the ranking between hyperparameters changes across K (left) and that the infinite member asymptote benefiting from more epochs and less weight decay per member.
In Figure 15 train ensembles for K [5) the best hyperparameters as K + 0o, we fit a power law and refer to the asymptote. We find that the ranking between hyperparameters changes depending on K (Figure 15] left). Importantly, the optimal hyperparameters for K = (black) are not the best asymptote under K + 2o (pink), which benefits from more epochs and less weight decay. This intuitively corresponds each member being more overfit to the data, which might be helpful to learn different views of the data. Selecting the hyperparameters when considering K - oc improves the ensembling asymptote from 3.34 to 3.27 (Figure 15 right).
vary weight decay and epoch coun To estimate
Another design choice for our ensembles is the source of randomness Z,, Though we vary both the data order and model initialization, we find that either one by itself obtains most of the benefits of ensembling and reserve a full analysis to Appendix D.3
D.3
SEED SCIENCE
For our training runs, the randomness only comes from the sampled initialization (train seed) and shuffled data order (data seed). We first characterize the run-to-run variance when varying both seeds, only train seed, or only data seed. We train 5 models for each of the three randomness options. When using the optimal hyperparameters for a 300M model with 200M tokens, the standard deviation is estimated to be 0.008207 for both seeds, 0.007605 for only train seed, and 0. 0.007213 for only data seed. This is in line with prior work that shows how only small amount of instability is needed to induce the majority of the variance over a final model's loss (Summers and Dinneen 2021 Jordan 2024).
We now measure the loss of ensembles with these sources of variance in Figure 16 Either of these
sources delivers most of the benefit of ensembling, with data order helping more. Considering the marginal benefit of introducing additional sources of randomness, we didn't strongly explore adding more sources of randomness during training.
D.4 HYPERPARAMETER TUNING FOR ENSEMBLES
Similar to parameter scaling, we hope to find the locally optimal hyperparameters for a given parameter count and token count. However, we care about the hyperparameters as K -> co, not at K = (as discussed in Bection[4.2). Since it is too experimentally prohibitive to search for locally optimal hyperparameters for the asymptote, we study how the hyperparameters change relative to the optimal H for single models.
We repeat the analysis in Figure 15 for different parameter counts as well as 2 different learning rates at 300M parameters, shown in Figure 17 We find that for all of the displayed settings, our best run comes from an ensemble with the same learning rate, half weight decay, and double epoch count relative to the optimal single model hyperparameters. For the three parameter counts display,
We find that optimal learning is generally consistent across K, Appendix D.4
28

---

pandommess
3.5
100
3.00
tral und L1--120//11155 + 330
Figure 16: Sources of randomness for ensembling. Only varying train or data seed is enough to induce the benefits of ensembling. Varying data seed (i.e. order) is better between these two.
verify that these are locally optimal hyperparameters. In fact, across all of the ensembles we trained across our scales (including many that are not directly visualized), we find only one counter-example to this heuristic. This occurs for 1.4B models trained on 200M tokens, where the hyperparameters that minimize the asymptote do not halve the weight decay. We suspect this change occurs because this is our most over wer-ppaamettrizee setting, and our scaling laws use this best run over the heuristic for this single setting.
D.5 ALTERNATIVES ENSEMBLING
The success ensembling suggests alternative parameterizations that might also boost data efficiency. We discuss commonly considered ones here, which we were not able to get to outperform ensembling.
D.5.1 MIXTURE-O1-EXPERTT
Ensembling may qualitatively seem similar training with Mixture-of -f-eeeests (MoE). However, we find an important distinction: when training an ensemble, the learning trajectory for each model completely independent of each other, whereas for MoE, is still single learning trajectory. Unfortunately, the intuition from Allen-Zhu and Li (023) suggests that the sparsity of the MoE architecture is not guaranteed benefit from "multi-view" data in the way ensembles do. In their paper, they consider simplified analogue where they construct a model that runs the ten models parallel and takes the gradient step through the ensemble jointly. They find that this barely improves performance over single model. In early experiments, we were able to reproduce this phenomenon, with jointly trained 10 ensemble of models outperforming a single model by only 0.02 loss. Therefore, we hypothesize that if MoE's were to help, their benefits would come from the drop-out aspect instead of the sparsity aspect, which does not require MoE's (note that we did not tune consider drop-out in this work, though expect it to further help performance). We hope future experiments can settle this intuition more concretely.
D.5.2 MODEL SOUPS
Prior works have shown that averaging the weights of independent training runs can result in better models (Wortsman et al., 2022). However, we note that most success from averaging weights comes at fine-tuning, not pre-training. We replicate these results in own settings, with model soups achieving close to random performance on downstream benchmarks (Table 5) but slightly outperforming ensembles in continued pre-training (Table 7
One intuition for this discrepancy is that models need to be in the same "loss basin" for averaging to help final performance, and pre-trained models enter different loss basins (Singh and Jaggi [202) Ainsworth et al.) 2023). Past studies also design compute-efficient algorithms for merging models trained from scratch, but they find that the more expensive procedure of distillation outperforms
29

---

Ensemble tuning for N 150M, D 209M
Stigle member
member sepmptote (220))
3.00
A77
A30
3014
195
387
3.5
1.0
N/A
N/A
N/A
A86
NIA
8G Hoggh Dowy
24
1485
Reght
Ensemble tuning for N - 300M, D 209M
Single member Koe.
Intont mumbee soympiete
0.000)
1777
35.0
3.58
3372
3005
3731
1428
A30
2336
1.88
4170
2324
136
64
16
24
06
Weigh Doaa
Dacay
mncaler
contripltt (LK 0.003
(LB 1.l8-..000 0.005
3.4
3.8
334
64-
2.00
Height
Welge
Ensemble tuning for N = 600M, D 209M
minglemmmbee
Single mumbhr
nember aepmppitee
N/A
N/A
N/A
330
N/A
322
Heigtt
Height
Figure 17: Single model and asymptote loss when varying epoch count and weight decay for different model sizes, token counts, and learning rates. We display an extended version of Figure [5 for 200M tokens with 150M, 300M (sub-optimal and optimal LR), and 600M parameter models. For these settings, the "double epoch, half weight decay" heuristic correctly predicts the best ensemble. This heuristic consistent with all of our parameter and token counts except for our most over-parameterized setting.
30

---

their method (Singh and Jaggi 2023 Since in the infinite compute regime, opt to distillation over model merging for the best performance.
D.6 ORDER OF LIMITS
In Section 4.3 we are interested computing the best possible performance of ensembles as and K both go to co. There are a couple of different ways to compute this, some of which are enumerated below
Double Limit 1 (Our Approach). We first solve for the limit as K oo by tuning asymptotes and then solve for the outer limit via a second power law over the inner asymptotes.
lim lim min C CEA (D.N, N K H
Hypothetical Double Limit 2.
We can flip the above order and instead take N oo before
K + 0o, corresponding to
Jim min C (Ex lim K N N H
Hypothetical Double Limit 3. Following literature in compute-optimal scaling, we can find the best possible performance for a given compute budget C and take C + DC.
lim C
min H.N.K
(D,N,
S.L K.H
We believe that our approach experimentally much more convenient than the hypothetical ap- proaches even though they are equivalent in output under assumptions. We share our reasoning by answering the following questions comparing the approaches.
The core assumption that we will make is that f(N, K) min (EA ND, is monotone N and K when the other fixed. Across all of our experiments, we do not observe any contra- dictions to this assumption as long we tune regularization (for examples, refer to Figure and Figure
Are Double Limit 1 and mathematically equivalent? Mathematically, both limits are equivalent. For quick proof, define k; := limg- f(D,N) and n, lim f(D, .N. By monotonicity, both k; and n, exist and are non-increasing sequences. Define lim k; and lim n, which exist for the same reason. Note that f(V, K) S ns. and since limits preserve inequality, it follows that k; < and further follows that k S By repeating this argument in the reverse direction, follows that n1.
Are Double Limits 1 and 2 mathematically equivalent to 3? we make the same mono- tonicity assumption as earlier, then we have that the minimization problem is monotonic in C. With this, we can apply similar argument as above to show that the limit converges and is equivalent to l and n.
How do we solve for the inner limit in Double Limit 1? It computationally prohibitive to tune all the hyperparameters for each choice of N and K. Therefore, we would prefer searching for optimal hyperparameters once per choice of N to fit the outer limit. Since are only interested in the performance as K - oo increases, we can reasonably approximate this via our hyperparameter heuristic from Appendix D.4 without ever determining the optimal hyperparameters at lower values of K.
Why do we prefer Double Limit 1 to Double Limit 2? We first make the observation that tuning hyperparameters depends N, but if we follow the previous bullet's asymptote tuning heuristic, tuning hyperparameters does not depend on any finite value of K. Therefore, Approach 2 would still have to fit hyperparameters for each N, K, whereas Approach 1 avoids this by fitting it once per N.
Why do we not use Double Limit 3? In Double Limit 1, we keep the hyperparameter as the scaling axis, instead of Double Limit 3 which sets compute as the scaling axis. When
31

---

choose to scale the hyperparameter, algorithm to find the best possible performance for that hyperparameter. This is difficult when scaling compute, since our hyperparameterss such model size and epoch count influence the compute spent during pre-training. Our preference reflects how practitioners typically Chinchilla Approach 3 (fitting loss for best run given N, D, closer to Double Limit 1) over Chinchilla Approach (fitting the envelope of the runs, closer to Double Limit 3) (Besiroglu et al. 2024
our locally optimal hyperparameter search
E DATA SCALING
E. EPOCH TUNED BASELINE
Unlike regularized parameter scaling and ensemble scaling where we estimate the best possible loss via asymptotes (Section [5. epoch tuning eventually over-fits. To estimate the best possible loss under epoch tuning for each token count, we use the following procedure.
For a fixed token count D and fixed parameter count N, search for locally optimal hyperpa- rameters while fixing weight decay to be 0.1
2. Perform this for all parameter counts N and token counts D
After following this procedure, we found that 600M models and 1.4B models were within = 0.02 loss of each other and were much better than the other models, as shown in Figure T8
Across all our token
scales, 600M models slightly outperformed 1.4B models, which we discuss in C.5 Therefore, we take the 600M performance an estimate of the best possible loss under regularized parameter scaling. We believe the performance of this algorithm would be better under better widhh//epth/archiccctures,, though we expect this benefit to translate to other recipes as well.
E.2 SCALING PARAMETER COUNT
For each token count D, we characterize the best possible loss of the regularized recipe by estimating lim min, L (D), N, H)) via the asymptote of the power law as described in Section 3 Since we now have to compute asymptotes to build the points for the data scaling law, we follow a two step procedure visualized in Figure
1. Varying parameter count (left plot). We first follow the locally- hyperparameter search detailed in Section 3 to find the best possible models for four parameter counts across four seed token counts. These are the 16 points in Figure left.
2. Varying token count (right plot). For each seed token count, we can fit a power law and the asymptote ED estimate of L3 These four (D, ED) tuples form the purple points in Figure [] right. We then a data scaling power law over these asymptotes, shown the purple line.
E.3 SCALING MEMBER AND PARAMETER COUNT
we repeat the above procedure for ensembles by taking lim limg-+o minH (E (D,N, K, H)) for each seed token count D. This results in the following three step procedure, visualized in Figure 7
Following Section 4.3
1. Varying ensemble member count (left plot). In Figure 7 left, we picture the losses of ensembles across our 4 token counts, 4 parameter counts, and 5 member counts with hyperparameters selected for better asymptotes. For each of the 16 pairs of D and N, we use power law extrapolate the performance as take member count K co.
2. Varying member parameter count (middle plot). The asymptotes of the above 16 limits are visualized Figure middle. Given these asymptotes, we can fit a second set of power laws that extrapolate the loss as we take parameter count N + co.
3. Varying token count (right plot). The asymptotes of these limits are visualized by the gold points Figure right. We then fit data scaling law, depicted by the gold line.
32

---

Figure 18:
Scaling seed token count for epoching and parameter scaling, For each seed token count, train models of varying with jointly tuned learning rate and epoch count with weight decay of 0.1. We take the best possible model at each token count as an estimate of the best performance under the standard recipe.
32-
32-
14-
COOL wdcm
M6.14
F DISTILLATION DETAILS
F.1 DATA GENERATION
For all distillation experiments, we generate teacher data from the same model family: K K-ensembles of 300M models with optimal hyperparameters for asymptotic performance. We choose to perform self-distillation with a 1-ensemble from this family rather than the 300M model from the regularized recipe to cleanly isolate the effect of distilling from stronger teacher. In practice, don't observe significant difference: self-distillation from the l-ensemble (blue point, Figure gives a loss of 3.43 while self-dist distiislillaion from the 300M regularized recipe (purple point, Figure 3) gives loss of 3.44.
Because we are unconstrained by train compute, optimal distillation should never epoch on teacher data and instead generate more. We pre- pre-generate large pool of teacher distillation data by sampling unconditionally with temperature using high-throughput inference engine designed for batched workloads (Juravsky et al., 2025). For generating ensemble teacher distillation data, experiment with inferencing both the logit averaged ensemble as well as the individual members. We observe better student performance using the individual members.
F.2 HYPERPARAMETERS
For both ensemble distillation and self-distillation, we search for optimal hyperparameters using a procedure similar to Appendix C.I Our distillation recipe also introduces a new hyperparameter which we refer to as the mixing ratio: the ratio of batches of real data to synthetic data. A mixing ratio of 1 1 indicates that data. For example, if we have 209M tokens of real data that we wish to epoch on 16 times, a 1 mixing ratio would require 209 106 16 3.3B tokens of teacher-generated distillation data. We find tuning the mixing ratio to be important for performance.
take the same number of gradient steps real data as teacher generrted
We detail the exact values of hyperparameters in Table 3 Interestingly, observe that optimal weight decay for distillation is lower than that of our regularized recipes, in line with standard practice. In addition, the greater diversity from the teacher's synthetic data. Our ensemble distillation run trains on a total of 16 209 10 (1 + 9) 33.4B tokens, while our self-distillation run trains on total of 16 209 10t x (1 + 3) = 13.4B tokens. Due to limitations in inferencing, we only generate 10B tokens each of ensemble distillation and self-distillation data, so our ensemble distillation may epoch up to 3 times on the teacher data.
find that ensemble distillation admits higher optimal mixing ratio, likely due to
Parameter
Ensemble Distill
Self-Distill
Learning Rate Weight Decay
3e-3
3e-3
0.1
0.1
Mixing Ratio
1 9
1 3
Epochs
16
16
Table 3: Optimal hyperparameters for ensemble and self-distillation.
33

---

Weight decay overfitting (Loss)
Weight decay overtitting (Downstream benchmarks)
Weight daeaa 0.1
066
Weightt dacay 0.1
Weight
Wiightt
500
Nrssage.ooo
0.55
4.25
0.90
4.00
0.43
16
129
Epochs
Opodie
Figure 19: Effect of regularization on overfitting for downstream benchmarks. Downstream benchmarks also reflect the benefit of heavy regularization performance. The effect of overfitting on downstream benchmarks (right) appears at twice the epoch count compared to validation loss (left).
F.3 MIXING DATA ABLATION
We provide token- matched ablation for the effect of mixing in the real pre-training data when doing self-distillation. As in Appendix F.2] we start with the same pool of 10B pre-generated tokens from 300M 1-ensemble. Perfect distillation into a student model of the same size (with infinite amount of teacher data) would achieve the same loss as the teacher.
We compare self-distillation with and without mixing in real data. For mixing in real data, we epoch the real data 16 times and use a mixing ratio so that the total number of tokens we train on is less than 10B. For mixing, we simply train subset of the pre-generated pool. Both methods use the same learning rate and batch size, train on a total of 16 x 209 10th (1+ 1) = 6.688B tokens, and never repeat the synthetie teacher data. For mixing, additionally search over weight decay.
Table shows that without mixing in real pre-training data, self-distillation is substantially worse than the teacher model (as might expect). Mixing data allows for eeff-distillation exceed the teacher model.
Teacher Model
Self-Distill (1 mixing)
Self-Distill (No mixing)
Val Loss
3.7103
3.4373
4.0693
Table 4: Effect of mixing real pre-training data for self-distillatio.
G DOWNSTREAM TASK DETAILS
G.1 DOWNSTREAM TASKS
We provide full breakdown of downstream benchmark scores per model type in Table 5 We use 1m inreevaluation harness (Gao el al. 2024) for our evaluations.
G.2 HYPERPARAMETER TUNING
We find that hyperparameter tuning from validation loss transfers to downstream benchmarks well. Figure 19 (left) shows how adding heavy regularization with weight decay (with fixed learning rate of 3)-3) shifts the overfitting point based validation loss the right and down. We observe similar effect in Figure [9) (right), although the overfitting threshold (in epochs) twice the threshold observed for validation loss.
34

---

Model type
ARC-Easy (%)
PIQA (%)
SciQ (%)
Avg (%)
150M
40.95+1.01
59.6841.14
62.4041.68
54.3540.72
300M
41.96+1.01
61.15+1.14
62.90 1.53
55. 55.34+0072
Unregularized model scaling
600M
39.86+1.00
59.90+1.14
60.50+1.66
53.4240.72
1.4B
40.61
60.39
61.40.1..4
54.14400.22
150M
41.29
60.17+1.14
63.90+1.52
55.12 12.772
300M
44.28+1102
61.81
69. 69.10+1..
58 58.394u.m0
Model scaling
600M
47.10+1.02
63.0641.18
69.70
50.95
1.4B
45.66
63.82
12.70
60.63
42.85
60.88
64.90 11.51
56.21 1.0722
K 2
44.61 11.02
62.02
65.80
57.48 10.71
150M ensembles
= 3
44.91+1.02
62.08
68.30
58. 48.77
K 4
45.83+1.00
62.1941.13
69. 69.20
50.07
= 5
45.50+1.02
62.3041.18
70.40+1.44
59.40.10.70
K = 1
44.36+1.02
62.95
68. 104:1.47
58.47+u.71
2
46.72+1.02
63.87+1.12
70.70+1.44
60.43+40.70
300M ensembles
K = 3
47.77
64.74
72.90
61.8010.66
= 4
48.36
65.89
73. 1..40
62.45.10.69
K 5
49.33
65.67
74.00
63.00 10.68
K 1
45.92
62.84
68.50 11.47
59.09 10.71
K = 2
47.56+1.02
64.0441.12
71.80
61. 01.1330.00
600M ensembles
K 3
48.44+1.03
64.25+11.2
73.30+1.440
62.00+0.06
K = 4
49.03+1.08
64.80+1.11
73.70
62.51
5
50.34+1.03
64.80+1.11
75.30
63. 63.48+0.68
= 1
43.56+1002
64.20++1.2
68.80+1.47
58. 20.85
K 2
47. 47.26+1.02
65.13
75.30
62.56 62.5640.0
1.4B ensembles
= 3
49.33
65.40
76.50
63.7410.67
K 4
48.8611.00
11.10
77.80 11.31
64.35 10.67
K = 5
49.71+1.03
66.38
77. 77.10+1.33
64.39+00.7
Self
46.68+1.02
62.35
72.60
60.5440.09
Distillation (300M)
Ensemble
48.44+1.03
62.84
75.30
62. 62.1940.08
= 2 K
26.5640.91
54.84 55.28
24. 14.70+11.0
35.37 35.3740.07 40.67
Model soups
24.96
23.90
34.71 34.7140.66 40.66
Table 5: Benchmark accuracies of all methods using 200M tokens on ARC-Easy, PIQA, and SciQ with averages. Entries are value sE in percentage points.
H CONTINUED PRE-TRAINING
H.1 HYPERPARAMETERS
Hyperparameters for continued pre-training baselines are shown in Table The 73B CPT run uses the default hyperparameters from /Wang et al. (025). except for learning rate which we tuned ourselves. The individual members of the K -ensemblls use the same hyperparameters as the standard recipe.
H.2 CPT SOUPS
We ablate the performance of model soups compared to ensembling in our continued pre-training setting by averaging the weights of the members instead of ensembling them. Unlike standard pre-training, CPT soups perform strongly and slightly outperform ensembles as we increase the number of averaged models (T.
35

---

Parameter
Default
Lower BS
Epoching
Learning Rate
3e-5
3e-5
3e-5
Weight Decay
0.1
0.1
0.1
Batch Size
512
64
64
Epochs
Table 6: Hyperparameters for continued pre-training.
Table 7: Continually pre-trained ensembles vs. soups
K-ensembles
K-soups
Benchmarks
Llama 3B
=
2
K ==
K =
K =
=
K 8
GSM8K(B-shoo
28.23
49.28
51.80
52.99
49.73
53.83
54.96
MATH (4-shot)
(4-hot-shhtt
6.90
21.84
23.04
23.50
22.40
23.02
23.72
MATHQA
35.07
45.12
46.06
45.26
44.59
46.10
45.33
Average
24.25
38.79
40.35
40.58
38.91
40.98
41.34
I POWER LAWS
1.1 SENSITIVITY ANALYSIS
To test whether our asymptote estimation is reliable due to run-to-run variance, we conduct sensitivity analysis for regularized parameter scaling and ensembling, shown in Figure 20
To test parameter scaling, we fit three power laws to all the models trained where each power law uses different seed (governing data order and model initialization). Though the scaling laws change per seed, they remain relatively consistent, with the asymptotes staying close together. This is encouraging, as the standard deviation in asymptotes is close to the run-to-run standard deviation for 300M models (Appendix) D.3p.
Since have runs for ensembling, test the reliability of ensembling by subsampling the number of members. When fitting a power law using up to four ensemble members, we find an extremely similar law using up eight ensemble members. Qualitatively, over the course of experiments, we found that the scaling law for ensembling is a lot more stable than the scaling law for parameter scaling.
We note that this is a limited stress-test and that it is likely our asymptote estimation procedure is quite noisy. Furthermore, note that this does not test our two-tier and three-tier power laws for joint scaling of parameters, members, and data, nor does it test our settings where our best run is further from the asymptote. We advise taking these asymptotes with a grain salt and interpreting them as rough estimates.
1.2 FITTING LAWS
To fit our power laws, we use scipy. opttpptmm peptimize. curve_ fit, either with no initial condi- tions and bounds or with p0- [1.0, 0, 0.5, 2. 2.0] and bounds- ( (IO, 0, 0], [np inf, np. inf, np inff) *. We note that unlike prior work where such parameters have been found be important (Besiroglu et al. [024 Hoffmann et al. (002), we did not find them be eritical considering how our fits are simple and over dimension.
J ADDITIONAL RELATED WORK
Over-parai prer-parametrized machine learning. We show that even though early work in double descent suggests that over parameterized deep learning does not have clean scaling laws due to double descent (Belkin al. 2019 Hastie al. 2020 Nakkiran al. 2019). can get clean scaling via
36

---

Sensitivity analysis for regularieed parameter scaling
Subssumpling for ensembling
375
Maain body power law U-I: L10/N-1.2 343)
3.0
Using 5 runs (1it 10 3.27
Sced power law (2 0.00/N-124
Using (TLL 14/0--104--3
3770
Seed 2 power law G11 343)
3.23
360
3.50
3.5
333
345
Paremcter count
Ereemble member count
K
Figure 20: Sensitity analysis. Left: When re-fitting the regularized power law across two additional seeds, we find that the asymptote stays relatively stable. Right: When subsampling the number of points for the ensemble scaling law, we find that the power law barely changes.
tuning regularization. agreeing with theory in over-veearattt werrpaaaaettrrieed regression (Advani and Ganguli 2016 Nakkiran et al. 202 Canatar et al. 2021 Simon et al. 2024).
Distillation algorithms Though we use sequence knowledge distillation (Kim and Rush 2016) as a preliminary demonstration, there are many better distillation algorithms such as using more supervision via logits (Sanh et al., 2020) and minimizing different divergences between the teacher and student (Agarwal et al.) 2024, Gu et al.) 2024). Past work has further quantified the scaling properties of logit distillation (Busbridge et al. 2025). and logit distillation is increasingly used to
pre-train the most performant small language models (Goyal et al.) 2025 Yang et al.) 2025 Team
ct al. 2025a).
Synthetic data. We can interpret distillation form of synthetic data, and unlike recent work on synthetic data to improve data efficiency (Maini et al. (2024] Allen-Zhu and Li, [024] Su et al. 2025 Team al.) 20255 Yang al.) [2024] |Raan et al. (025). distillation requires minimal human priors such trusted reward function or known augmentation invariance like rephrasing.
Classical data-constrained deep learning. Historically, many machine learning benchmarks
before the of large language models data-constrained (Marcus al. 1993 Warstadt et al.)
2023 Deng et al.; 2009 Lecun et al.) 1998), resulting in many effective algorithms. For example.
the best models on Penn Tree Bank utilized dynamic evaluation (Mikolov et al.,
2010 Krause et al.
2017). ensembling and model averaging (Takase et al. 2018 Zaremba et al. 2015), regularization (drop-out, weight decay, weight tying. lower batch size) (Merity et al. 2017 Zaremba al. 2015
Gal and Ghahramani] 2016), data augmentation (Shi et al., 2021 Xie et al. 2017). ) and novel
architectures (Zilly et al. 2017] Grave al. 2016 Yang et 2018). We revisit few of these algorithms and advocate for future work to explore all others.
37

---

K UPDATES DURING REBUTTALS
K.1 NEW MODEL SIZES
Our 1.4B parameter model is too shallow and does not follow a consistent aspect ratio compared to smaller models. We pre-train 1.5B and 3.2B parameter models with fixed aspect ratios the 200M token scale. We detail the new model configurations Table g optimal regularized and unregularized models Figure 21
We show the losses of locally
Parameter
150M
300M
600M
1.4B
1.5B
3.2B
Context Length
4096
4096
4096
4096
4096
4096
Hidden Dimension
512
768
1024
2048
1536
2048
Intermediate Dimension
1792
2688
3584
7168
5376
7168
Attention Heads
16
16
16
KV Heads
12
8
8
8
Layers
24
36
48
Table 8: New model architecture configurations for different model sizes.
K.2 MIXTURE-OF-EXPERTT
We perform some intial experiments the data efficiency of alternative architectures, specifically (Shazeer et al.) 2017). We pre-train an MoE with 300M active parameters following standard recipes from Mixtral (Jiang We use token choice router, load balancing and z losses with default hyperparameters of 0.01 and 0.001 respectively, 2 active experts per token, and 8 total experts. See Table for additional details.
2024 and OLMoE (Muennighofi et al. (025).
We search for locally optimal hyperparameters of this architecture (including regularization) and achieve 3.57 loss, slightly outperforming the regularized 300M dense model with loss 3.58. This initial experiment suggests that the data efficiency of MoEs is similar to dense models, but we leave more thorough study to future work.
Parameter
Dense 300M
MoE Active 300M
Context Length
4096
4096
Hidden Dimension
768
768
Intermediate Dimension
2688
1344
Attention Heads
12
12
KV Heads
12
12
Layers
12
12
Experts per token
Total experts
Table 9: Model architecture configuration for 300M dense vs. Mixture-of-Exxperss.
Model
Val Loss
Learning Rate
Epoch Count
Weight Decay
Dense 300M
3.5868
3e-3
16
1.6
MoB Active 300M
3.5767
3e-3
1.6
Table 10: Loss and hyperparameters of the 300M dense vs. Mixture-of-Expertt.
38

---

Scaling law
R²
Figure
Regularized parameter scaling
300M regularized recipe
0.9999
Figure
Ensemble scaling
300M ensemble (standard hparams)
0.9999
Figure
Joint scaling
150M ensemble (tuned)
1.000
300M ensemble (tuned)
1.000
600M ensemble (tuned)
1.000
1.4B ensemble (tuned)
0.9998
Parameter scaling double limit
0.9983
Figures
and
Data scaling for all recipes
Standard recipe
1.000
Regularized recipe
0.9998
Joint scaling recipe
0.9999
Table 11: R² for our main scaling laws.
K.3 GOODNESS-OF-RI AND EXTRAPOLATION
K.3.1 GOODNESS-OF-FTT
We report R² for the important scaling laws display in our paper in Table 11 We note that all of these values are quite high (above 0.999). likely due to the low sample count. Nonetheless, these simple fits are much stronger than prior work that use more expressive scaling laws (e.g. Muennighoff et al. 2023): Prabhudesai et al. (2025)).
K.3.2 EXTRAPOLATION
Due to the nature of our fits, we believe that the above Rt analysis is insufficient to test our scaling laws. Therefore, perform extrapolation test to predict the losses larger model. For this, train 1.5B and 3.2B models following the config described in K. with locally optimal hyperparameters. We find that our regularized power law (purple dotted line) predicts the loss of the 1.5B model with error 0.005 and the 3.2B model with error 0.008, suggesting that our scaling laws have reasonable predictive power. Further, find that larger unregularized models (red stars) eventually overfit, when we fix the aspect ratio.
Regularized parameter scaling
Tuned F
150M
300M
600M
1.4B
1,58
3.28
3.50
Learning rate
3e-3
3e.3
16:3
16.3
16:3
16-3
Epoch count
3.7
Weight decay
0.8
1.6
3.2
3.2
3.2
3.2
3.70
Eputchad naipe
Eporcbud entropeiation
Figure 21: Extrapolation to larger model sizes. We extrapolate the predictions of our power law from Figure 3 We find that our power law predicts the loss of the 1.5B model with error 0.005 and the 3.2B model with error 0.008. We also find that even when fixing aspect ratio, large enough models without regularization (red stars) overfit.
Regelarivad miips (Hi /NV 543)
Regubatzzed extrapelation
3.58
330
3.85
Parombber connt
K.4 MORE EVALUATIONS
We evaluate additional benchmarks to test the reliability of our improvements in validation loss. In addition to ARC-Easy, PIQA, and SciQ, now evaluate ARC-Challenge. HellaSwag, LAMBADA, and Winogrande. We visualize the results in Figure 22 and find a consistent ranking over regulariza- tion. parameter scaling, ensemble scaling, and distillation. Additionally, the unregularized 1.5B and 3.2B models described in Appendix K.1 still underperform smaller models.
39

---

Downstream benchmark error
Epoched recipe
0.60
Model scaling
150M ensembles
0.59
300M ensembles
600M ensembles
1.4B ensembles
8-Ensemble distill
Average Eroor
Self-distill
0.56
0.55
10c
10a
Total parameter count
Figure 22: Performance of pre-trained models on downstream tasks. We re-evaluate our models and ensembles on total downstream benchmarks. Models with lower validation loss have lower average eтror across downstream benchmarks.
40

---

Table 12: Benchmark accuracies all methods using 200M tokens on ARC Challenge, ARC Easy, HellaSwag, LAMBADA PIQA, SciQ. and Winogrande, Entries
40.25 42
40.67.0.44 42
39 40.41
39.95++.42
39.68
40.00
11.35
43.58 10.41
44.33
45.1620.41
41.5920.2 42
12.2542-42
42.81+0.41
42.96+n.4:
43.044n.41
43.78
44.70
45. 15.12
45. $5.4640.41
45.76+0-41
44.30+0.42
45.39
45.97
46.56
16.0610.41
44.31 42.42
16.20 41
47.0440.41
47. 47.50+0-41
47.5440-41
44.8740-41
45.50
Arg
Winogrande
50.04+1.4 1/111
49 19.0941.41
50.0441.4 1.1
50.75
51.0741.00
50. 50.12
51.07
1.140
51. 51.78
50.04
50.75
50.59
51. 31.3881..8 1400
52.25
51.7041.44
50.67+1.41
52.09
51.4611.00
50. 50.28
51.78+1.0 10
51. 51.70 1400
52. 12.3321.40 1.40
50.5141.41
51 51.38 1.440
52. 32.49+1.40
51.3811.40
52.57+1.00
51 $1.3811.00
51.7011.00
10
51. 51.30+1-4
52.01+1-00
51. 31.6221-440
52. 22.49
SeiQ (%)
62. 12.00.1.51
62. 22.5021.53
60. 20.50
61.00
57.00+1.87
59. 1.55
61.0011.2
60. 1..60.1.46
62 12.68 60.6041.46
72. 22.70
51.10
65 15.9021-50
61.9741.12 68 08.30
62.02+1.14 69 09.30+1..40
62.5741.19 10.4401.44
68.0 18.00
70. 1..0
72. 72.70
73.30+1.40
74.00
02. 12.30 11.407
71.70
64.25-1.12 73.20
73.80 13.80
64.74.1.11 75.10
68. 10.
75. 16.1021.37
76. 13.40.1.34
77. 77.80
66.3221.10 77-10
72. 22.30
75.10
64.15 71.
PIQA (%
5 59.5221.15
61.04
59.90+114
60.07+1.14
61.21 61.21+1.14
61.21
60.28
62. 02.1911.15
03.87+1122
60 10.18
02. 02.13+1-13
63.06
63.98
64. 11.12
65 35.56+1.11
65 65.89+1-11
64.42+1.2
64.25+1.1
65. 13.111
65. 05.6141.11
65 65.89+1-11
62 62-46-1-13
62.51
LAMBADA
23 13.93+0.0
23 73.09220.5 50
21. 20.09
50.57
21 31.9340.5
21.04
21 21.50
26 26.55 10.62
28. 13.9511..8
31 #1.81
32 22.6022.66 65
25 60.66
25. 10.111
25. 35.4240.m
25.52
25.58 40.61
28.08 10.00
30 10.06.10.64
12..64
30 30.25
30 00.39+4.60
28. 20.66
30 19..44
32 10.00+0.66
32. 22.62+42 +0.68
32 20.65
27.38 10.62
31 10.665
32.76 92.7620.65
33.65 33.65+4.00
34.00
32.04 22.04+00.05
32.52
Easy (%) wag
26 40.44
27. 77.7320.48 45
27 200.44
27 40.
27 27.39+0.40
27. 27.45
28 65 1..45
20 1644.6
40
27 24.44
27. 27.1340.44
27.27 27.27+0.44
27. 27.42+n.40
27 07.39+mm 44.45
29 19.5410..00
20 .7.37. 11.3710.4 10.45
20 10.44
29 20.40.
29 20-46
31.03 03.344
31 204046
31 +0.00
31 11..03+m.44 +0.46
31. 12..66
31 18.88.077
31 10.40
32. 15
32 41.447
32.5 32.51 40.47
28. 27.444
28.5410.44
27. 27.01
40 10.91
15.701
40. 40.11
40. 19.49.1.01
41 11.9611.01
40 1.01
41 1..00
44 1..02.002
47. 17.1441.02 02
45. 13.8321.00 02
42 1..052 1.00
M.5321.00 .0
14.9541.02
45.83 15.83
45.41++102
4 1..28.1.02
46.55 46.55.1.00
47. 17.8121.02
48 23.2311.03
49. 11.41.10
(6.0021.02
47. 17.60+...00
48.27 18.27
18.95
50.51s1.03 1.00
43. 1.02
47. 1..5511022 .2
49.2 10.24
49. 10.122100
49. 19.7921.03
46. 10.68&1.00
48.1 15
ARC- Challleleee
18 11.4
19 16.11
17.31
18.00
18.09+1.12
18.94
19.37 11.15
19 19.80 11.10
19 11.17
20. 10.14
18.94 15.9411.15
19.5421-18
19.5441.10
18.94+1.15
19.28
21.42
20 11.19
20 19
10.8021.16
19 19.4511.16
11.08
21.67 120
22.61+11 1::22
20.9 1.
21 21.6711.2
21.6 11.6711.20 20
21.42.1.00 20
22. 72.3541.22
21 21.08
20.0521.17
19.20
21.67
Self Enseeblee
150M
300M
600M
1,4B
1.5B
3.2B
150M
300M
600M
1.4B
K =
K = 2
K = 3
K = K =
K 11-1
K =
3
K 4
K =
K =
K =
K = 3
K =
K -
K =1=1
K 2
K 3
K ==4 4
K = 5
are value Si in percentage points.
Unregularized model scaling
Distillation (300M)
150M ensembles
300M ensembles
600M ensembles
1.4B ensembles
Model scaling
Model type
41


Reviews:
[{"review_id": "ut6coQ5cXo", "reviewer": "Reviewer_kN1H", "summary": "This work studies how to pre-train when compute is abundant but data is limited. The authors show that standard approaches of increasing epochs or model size overfit, and that stronger regularization, especially much larger weight decay, helps sustain gains. They further show that ensembling independently trained models improves the loss asymptote, and that their best setup, combining regularization, epoching, and ensembling achieves the same performance with 5× less data. Finally, they find that most of their ensembling recipe benefit can be retained through distillation into a smaller model, with gains also reflected on downstream tasks.", "questions": "1. The novelty feels incremental in places. Ensembling and stronger regularization are well‑known tools; the main new piece seems to be “optimize for and compare asymptotes under fixed data.” Could you sharpen the claim of conceptual novelty beyond careful tuning + ensembling, and clarify what, if anything, is surprising in the results relative to prior data‑constrained scaling work?\n2. Compute‑matching across alternatives could be clearer. For a fixed training FLOP budget, how do (i) a single larger model, (ii) K‑member ensembling, and (iii) spending compute to synthesize more training text compare?\n3. Weight decay is pushed high and clearly helps, but other classic knobs (dropout, data noising, etc.) are not explored. Would be interested in knowing your thoughts on whether similar monotone‑scaling and asymptote reductions hold with these (or other) alternatives?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem addressed is timely. The main ideas like stronger regularization, evaluating by scaling-law asymptotes, and using simple ensembling, are easy to follow and reproduce. The experiments consistently show clear gains and improved data efficiency, and the distillation results make the approach more useful. The paper is well-written, with clear experiments and a natural, convincing link between the sections.", "weaknesses": "See Questions section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4Dc4jjTLus", "reviewer": "Reviewer_GYnT", "summary": "This paper presents a compelling and timely investigation into language model pre-training for a future where high-quality data is the primary bottleneck, not compute. The authors argue that as compute continues to scale faster than data generation, the community must shift from compute-efficient to data-efficient training paradigms. The work first demonstrates that standard recipes, which involve increasing epochs or model parameters, inevitably lead to overfitting and performance degradation. As a solution, the authors propose a \"regularized recipe\" using aggressive weight decay (up to 30x standard practice) to achieve stable, monotonic performance scaling. Building on this, they show that ensembling multiple smaller models is a more data-efficient use of compute than scaling a single large model. By combining these methods, they claim a significant 5.17x improvement in data efficiency. Finally, the paper grounds these compute-intensive findings in practice, showing that the gains from ensembling can be largely preserved by distilling them into a single, smaller model, and that improvements in validation loss translate to meaningful gains on downstream tasks.", "questions": "The paper convincingly shows that tuned regularization eliminates the initial overfitting peak seen in standard recipes. However, the double descent literature suggests that with sufficient over-parameterization, an unregularized model's performance might enter a \"second descent\" and improve again. Could you comment on whether this second descent could, in the infinite limit, outperform the regularized recipe? Is there a theoretical or empirical reason to believe the regularized performance curve is strictly superior at all points in the over-parameterized regime?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This is a strong paper that could become a foundational empirical work for the emerging data-constrained training regime. The experiments are thorough, and the insights are valuable for both theoretical and practical applications. Specifically, I appreciate the following contributions of this paper.\n1. **Extensive Experiments:** The claims are backed by an impressive scale of experimentation, reportedly over 2000 training runs. I like the scientific approach of diagnosing the failure of a baseline, introducing a robust regularized recipe, and then demonstrating the superiority of ensembling, which builds a convincing, step-by-step argument.\n2. **Practical and Generalizable Insights:** The work successfully connects the \"infinite compute\" thought experiment back to practical applications. The demonstration that an 8-member ensemble can be distilled into a single student model that retains 83% of the performance gain is a crucial result, proving these methods are not purely theoretical. Furthermore, the validation on downstream benchmarks  confirms that the observed improvements in pre-training loss correspond to genuine enhancements in model capabilities.", "weaknesses": "1. **Architectural Specificity:** The empirical results are derived exclusively from Llama-style decoder-only architectures. It remains unclear whether the core findings, particularly the superiority of ensembling over scaling a single model, generalize to different architectures such as Mixture-of-Experts (MoE). More critically, under the infinite-compute regime, transformer architectures may not represent the optimal choice. The paper would benefit from discussion on architectural considerations within data-constrained training regimes.\n2. **Reliability of Power-Law Extrapolation:** The paper's headline quantitative claims (e.g., 5.17× data efficiency) depend on extrapolating power laws fitted to limited data points (typically four model sizes). While the authors provide a sensitivity analysis and appropriately advise caution, the absence of formal statistical goodness-of-fit metrics (such as R² or confidence intervals) makes it difficult to assess the true reliability of these asymptotic estimates. The broader literature demonstrates that simple power laws can break down or exhibit sub-scaling behavior at larger scales, making long-range extrapolation risky. This concern is compounded by the non-standard width/depth configuration of the 1.4B model, which introduces a potential confounder that could affect the quality of the fit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AbJz53E6Wj", "reviewer": "Reviewer_rfVe", "summary": "This paper investigates language model pre-training in a fixed data infinite compute setting. To address overfitting in the data-constrained scenario, the authors propose a recipe with 30x larger weight decay and the asymptote of the scaling law. The final joint scaling recipe yields a 5x data efficiency gain over the baseline.\n\nThe authors then propose and demonstrate that model ensembles can be trained and distilled into a single student model while preserving most of the loss improvement.\n\nLastly, the authors show that their metrics generalize well to downstream benchmarks.", "questions": "1. Is there any intuition behind distilling from a single model working better than distilling the ensemble?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The motivation for the paper is practical, where internet data is exhausted. \n2. The empirical studies are thorough and clear. The authors produced a recipe that achieves 5x higher data efficiency and demonstrates the benefit of a much larger weight decay.\n3. The level of attention to experiment details is commendable, and experiment setups are described in detail.\n4. The settings the authors take into account are comprehensive, with results ranging from pretraining, model ensembling, and distillation.", "weaknesses": "1. Due to the shallow setup for the 1.4B model, it underperforms the 600M version without the proposed weight decay. While in Figure 3, the authors show how the regularized recipe fixes this, this still casts doubt on the following conclusions, given how sensitive power law fits are, and that 1.4B is the largest model variant that the author uses for curve fitting.  For instance, in Figure 9, 1.4B's downstream eval results also break the trend, most likely due to model architecture configuration. Since the authors only train on 200M tokens, this is not prohibitively expensive to fix.\n\n2. The ensemble scaling part requires a heuristic that breaks in the over-parametrized setting. The heuristic itself, when and why it works well or not, could use more explanation to make the ensemble tuning recipe simpler and more principled.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lb5roZA43l", "reviewer": "Reviewer_7Xg1", "summary": "The paper explores a forward-looking problem: how to design pre-training strategies when compute grows faster than data availability. The authors simulate a future where web text becomes the limiting resource and propose several methods to make pre-training data-efficient under fixed data but infinite compute.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper identifies an under-explored but realistic future scenario where compute scales faster than data. This is a fresh take beyond the traditional “compute-optimal” (Chinchilla-style) scaling laws.\n2. The authors construct a controlled 200M-token benchmark and systematically vary parameters, epochs, and regularization. The coordinate-descent tuning of weight decay, LR, and epochs shows strong empirical rigor.\n3. The evaluation results demonstrate the effectiveness of the proposed approach.", "weaknesses": "1. The question raised in this paper is quite interesting. However, I am curious whether the conclusion would still hold if the dataset size were fixed between 30B and 100B tokens while training a 1B-parameter model. According to the Chinchilla paper, the optimal number of training tokens is approximately 20 x N, and over-training is quite common nowadays. For instance, the Qwen3 model was pretrained on about 36T tokens. It would be interesting to investigate, when training with more than 20 x N tokens, how to appropriately adjust the learning rate, weight decay, and number of epochs.\n2. Could the authors briefly clarify how this paper differs from the Tensor Programs V: Tuning Large Neural Networks via Zero‑Shot Hyperparameter Transfer (https://arxiv.org/pdf/2203.03466)? Additionally, the μP paper shows that hyperparameters are closely tied to the model architecture, and the more recent Scaling Inference‑Efficient Language Models (https://arxiv.org/pdf/2501.18107) builds a scaling law that explicitly incorporates architecture. Could the authors elaborate on the impact of model architecture in their study?\n3. The evaluation over three downstream tasks is limited. Is it possible for the authors to show the evaluation over Arc-Challenge, Lambada, HellaSwag, Winogrande, etc?\n\nI would be willing to raise my score if the authors can address the above questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)