Don't be lazy: CompleteP enables compute-efficient deep transformers
Authors: Nolan Simran Dey
Bin Claire Zhang
Lorenzo Noci
Mufan Li
Blake Bordelon
Shane Bergsma
Cengiz Pehlevan
Boris Hanin
Joel Hestness
Venue: NeurIPS 2025 poster ()
Keywords: Deep learning
depth
transformer
LLM
compute-efficiency
parameterization
stability
hyperparameter transfer
Primary area: deep_learning
TL;DR: We introduce CompleteP, which offers depth-wise HP transfer, FLOP savings when training deep models, and a larger range of compute-efficient width/depth ratios.

Abstract: We study compute efficiency of LLM training when using different parameterizations, i.e., rules for adjusting model and optimizer hyperparameters (HPs) as model size changes. Some parameterizations fail to transfer optimal base HPs (such as learning rate) across changes in model depth, requiring practitioners to either re-tune these HPs as they scale up (expensive), or accept sub-optimal training when re-tuning is prohibitive. Even when they achieve HP transfer, we develop theory to show parameterizations may still exist in the lazy learning regime where layers learn only features close to their linearization, preventing effective use of depth and nonlinearity. Finally, we identify and adopt the parameterization we call CompleteP that achieves both depth-wise HP transfer and non-lazy learning in all layers. CompleteP enables a wider range of model width/depth ratios to remain compute-efficient, unlocking shapes better suited for different hardware settings and operational contexts. Moreover, CompleteP enables 12-34% compute efficiency improvements over the prior state-of-the-art. All experiments were run on Cerebras CS-3 systems. A minimal implementation is available at https://github.com/EleutherAI/nanoGPT-mup/tree/completep.

Don't be lazy: CompleteP enables compute-efficient deep transformers
Nolan Dey* Cerebras Systems
Bin Claire Zhang" Cerebras Systems
Lorenzo Noci ETH Zurich Princeton University
Mufan Li
Princeton University
Blake Bordelon Harvard University
Shane Bergsma
Cengiz Pehlevan Harvard University
Boris Hanin Princeton University
Cerebras Systems
Kempner Institute
Joel Hestness Cerebras Systems
Abstract
We study compute efficiency of LLM training when using different parameteri- zations, i.e., rules for adjusting model and optimizer hyperparameters (HPs) as model size changes. Some parameterizations fail to transfer optimal base HPs (such as learning rate) across changes in model depth, requiring practitioners to either re-tune these HPs as they scale up (expensive), training when re-tuning is prohibitive. Even when they achieve transfer, we develop theory to show parameterizations may still exist in the lazy learning regime where layers learn only features close to their linearization, preventing effective use of depth and nonlinearity. Finally, identify and adopt the parameteriza- tion we call CompleteP that achieves both depth-wise HP transfer and non-lazy learning in all layers. CompleteP enables wider range of model width/depth ratios to remain compute-efficient, unlocking shapes better suited for different hardware settings and operational contexts. Moreover, CompleteP enables 12-34% compute efficiency improvements over the prior state-of-the-art. All experiments were Cerebras CS-3 systems. A minimal implementation is available https: / / thub. com/ Ell wtw.///eeeeeerr//aaaaoe-----.--. http///nnnnnmmm////mm ttappp/treee//eeeee htt.p//peee////peteeeeeeeeeeeeee completep.
accept sub-optimal
Stable HPs across Depth
Improved deep models
Larger range of efficient N:L
4.8
33%
Depth
Paromelars
Opsma
4.5
SimPJ Viikalioon Lose
300M
CompleteP ja
1.58
16
84
19%
10%
9%
LPOP
(scr00 Captal
3.5
10
Learring Rate 12
Depth
Figure 1: We introduce CompleteP, which offers depth-wise HP transfer (Left), FLOP savings when training deep models (Middle), and a larger range of compute-efficien width/depth ratios (Right).
"quual contribution. Email: fnolan, claire. zhang, joel) (ooee)@8cccc Ecerebras.net
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
hallmark of modern deep learning that training larger models leads to better performance [1]. In Large Language Models (LLMs), for instance, the paradigm of pre- pre-training larger models on larger datasets has led to remarkable results in a wide range of downstream evaluations. However, these gains come at substantial increase in computational cost. As compute budgets grow, practitioners must navigate a complex design space to choose model width and depth, dataset size, batch size, number of training steps, and variety of other hyperparameters (HPs) in order find an optimal allocation of resources to minimize a pretraining objective, given a fixed compute budget [2, 3].
The prohibitive cost of naively conducting such a search in large models can result in suboptimal HPs and hence an inefficient use of computational resources. This motivated techniques such as the maximal update parameterization (μP), which ensures that optimal HPs remain approximately constant when scaling model width [4, 5] and enables a "tune small and train large" strategy.
We refine and thoroughly compare extensions of uP for simultancously scaling depth and width [6-8]. These scaling strategies define parameterizations- - sets of rules for how scale model and optimizer hyperparameters with model size. The core difference between uu and these depth-awar refinements is how they re-scale the outputs of the transformer's [9] residual block (as a function of depth L) before the output gets added to the residual stream variables h":
= - Fe(h'), E {1, L},
where Fe is the l'th residual block (for transformers, these are MLP and attention blocks). The septh-dependent re-scaling factor is governed by a single parameter E [0.5, 1].
Yang et al. [7] argue a 0.5 works best in practice and that HP transfer is not possible at any a, while Bordelon et al. [8] find a = allows better learning as + co. Since a E {0.5, 1] are the two most promising potential candidates for depth scaling [6-8], we compare these
values and through better HP transfer and faster pre-training (Figure 1). Setting a gives a parameterization we call CompleteP because it is the unique a ensuring complete feature learning as width and depth are scaled (Section 6).
find a = is consistently more compute-efficie
To realize these gains. however, we must extend the parameterization to include principled re-scalings of LayerNorm (LN) and bias learning rates, AdamW's weight decay X, and Adam e (Table 1). CompleteP is simple to implement, yet yields superior upstream and downstream performance, with gains over alternative approaches increasing with model depth. To summarize, our contributions are:
HP transfer across depth. We compare transfer of learning rate and weight initialization standard deviation across depth (2-128 128 layers) for the standard parameterization (SP), μPP. and {0.5, 1}. when training Pre-LN transformers (Figure 2, 3). We find only 1 enables depthwise HP transfer, a feat thought impossible by Yang et al. [7].
Optimal shape and compute efficiency. For SP, μP, and {0.5, 1). construct scaling laws over models ranging from 75M to 1.9B total parameters, trained at a compute-optimal frontier of 20 tokens per parameter (TPP) [3]. We this to revisit the question of compute- efficient transformer shapes (i.e. width-to-depth ratio L.). The 1.9B parameter models (i.e. with 1.5B non-embedding parameters) trained with a 1 achieve 11.8% and 34.4% FLOP savings over μP for optimallysshaped and 179-layer models, respectively (Figure 4).
Refined desiderata for HP transfer. In Section 6, we describe the differences between SP, μP, and E {0.5, J.. Given the strong empirical evidence for HP transfer for a = 1, we devise refined set of desiderata for HP transfer that includes the notion of complete feature learning: we require that the learned representation in every layer of the model remains non-lazy [10] (i.e. non-linear) with respect the parameter in both that layer and all earlier ones. Only a = ensures complete feature learning, thus we name it CompleteP.
Extended parameterizations for modern training. Modern LLMs train with pre-LN and AdamW. In Table 1 and Appendix D. we extend existing parameterizations [8] to include preseriptions for LayerNorm learning rates 1), weight decay, bias, and Adam e as functions of depth and width. These changes are essential for stable training with 0.5 (Figure 7).

---

2
Related work
Theoretical approach to HP tuning. Early methods for selecting HPs analyzed networks initial- ization to ensure numerical stability in the initial forward and backward passes [11-28]. Subsequent work devised two parameterizations with consistent training dynamies infinite width: the Neural Tangent Kernel (NTK) paramcterization in which the model converges to its linearization around
initialization [29-31] and the mean-field/uP parameterization [4, 6, 8, 10, 32-39J. Training at the simultaneous limits of width and depth was explored in [40, 41] for fully connected networks, in [6. 7] for vanilla ResNets and in [8] for transformers. Infinite-limit descriptions of training in the mean- field/uP parameterization are difficult to study analytically but give clear proposal for HP transfer by requiring consistent dynamics of hidden layer representations across model scale [41-44. 32].
This approach to HP transfer was taken up in [6-8], which derive a family of mean-field parameter- izations for ResNets indexed by [0.5, 1] (see Table 1). Based on heuristic related to feature diversity, [7] argued HP transfer was not truly possible at any a but that a = 0.5 was the best in practice. In contrast, we show CompleteP (a- = 1) yields both HP transfer and FLOP savings during pre-training and provide theoretical justification in Section 6. Other interesting mean-field approaches to HP transfer include adaptations to sharpness-awan
optimization [45], to low-precision training [46], to sparse training [47], and finally to state space models [48].
Empirical approaches to depth scaling. Empirical approaches to HP transfer in [49, 50] normalize layer outputs and learning rates to yield HP transfer across width, consistent with the μP prescription. HP transfer across depth is often more ad hoc: [49] proposes something resembling a 0.5, while Large et al. [50] proposes h f+ = -_- I+ L Fr (h'))) per layer, reminiscent of setting a = 1. Several other works, though not directly focused on HP transfer, aim stabilize training in deep ResNets through LN modifications. Sun et al. [51] advocates multiplying pre-LN output in layer & by E-0.5, similar to taking a 0.5. Similarly, [52] applies LN both the input and output of each residual block, avoiding exponential-in -nntiill-----ephh activation and gradient variance. Finally, [53] adjusts the scale of weights in each residual block, up-weights residual branch contributions. and inserts LN after the residual add. While this allows one to train very deep networks, it does not achieve depth HP transfer.
L-
Compute-optimal transformer N: L ratios. Theoretical and empirical work have studied the optimal transformer shape. Notably Kaplan al. [2], using SP and large-scale tests, showed wide range of N: L. were close to compute-optimalit, with N: L ze 100 being the optimal ratio. This finding guides modern LIM shapes, which often fix N: L, R 100 when parameters and tokens scaled [54-72]. Yet SP does not fairly admit stable width and depth scaling, which undermines the prior conclusion. McLeish et al. [73] adopt a 17 Thase / / parameterization resembling incomplete = 0.5 and study compute-optimal N: L. However, their parameterization does not admit training stability (Figure 7) or HP transfer so their deeper models were disadvantaged, causing them to conclude shallower models are optimal. In Section we revisit the transformer N study with proper width and depth scaling control with CompleteP and show that even N: L x 10 remains close to compute optimality. Rather than fixing N: I., other works perform empirical searches to estimate scaling exponents for N and 7. [74, 75]. Levine et al. [76] propose theory for transformer capacity based on separation rank, predict that optimal N L increases with parameter count, and provide empirical validation. Mixture of experts (MoE) [77] enable transformer capacity to scale without increasing depth. Similarly, [78] propose parallel sub-networks and argue that large depth isn't 'n't necessary for competitive performance. We focus on dense transformers and consider MoE and
parallel sub-networks as out of scope.
3 Methodology
For all experiments in this work, we train decoder-only Transformer language models [79] with pre- normalization, untied embeddings, ALiBi position embeddings [80] and ReLU* nonlinearity [81, 82]. using the A optimizer with decoupled weight decay [83] with 3 = 0.9, 32 = 0.95, c = 1e- 16. The learning rate v schedule follows a linear warmup of min(10% of steps, 375M tokens), then a linear decay to zero [84]. We pretrain our models using an autoregressive loss (i.e. the next token prediction objective) on the SlimPajama dataset [85] with a maximal sequence length of 2048 tokens using the GPT-2 tokenizer [79]. All models use dhead 64 for each attention head and
3

---

Q'K/ feedforward dimension 4N. For all parameterizations, we use Q J/KK N as proposed in [5] to account for correlation between Q and K. We use Nave 256, Luse and scale parameters according to Table 1. All experiments were performed using Cerebras CS-3 systems. We refer the reader to Appendix G for full methodology of all experiments.
Table provides an overview of how to implement the parameterizations we test in this paper. We consider models with depth (or 2L residual blocks account for both MLP and attention blocks), and width N (e.g., residual activations / E IRA). We define adjustments in terms of width multiplier my N/Neuse and depth multiplier my L/ Lowse. where Nusee Lusee are the width and the depth of base model. For the base model, i.e., when m. = m, 1, all parameterizations are equivalent. We use Nusse = 256, Loase 2. In experiments where only my = 1, SP is equivalent to μPP such results are labelled "SP/UP"". We extend the infinite depth parameterizations in the literature [6-8] with corrections for bias learning rate 7, LayerNorm n, AdamWa and weight decay X. See Appendix D for derivations of these extensions and practical advice for applying CompleteP to different architectures. Appendix E for empirical verification of e scaling'. We provide minimal implementation of Table which reproduces Figure at: https://8 gi thub. com/ Www.EEeeeeeeeeeeeee www.rr///nnaanPT---aa- mwpptmpataeeeeeeee tree/ compp/teppppppp
Table 1:
Summary of SP, JP,, and a E {0.5, 1} for a pre-LN transformer language model. Terms related to width and depth control are highlighted in orange and green respectively. Additional tunable parameters are highlighted in blue. Hidden refers to all linear layers in the transformer backbone.
Parameterization
SP
μP
E (0.5, 1]
Emb. Init. Var. Emb. LR (AdamW)
o
Thhse
J/hae
Thuse
Pre-LN Init. Var. Pre-LN LR (AdamW)
base
Case
Tuse
Thase
Thuse
Thuse a-1
Hidden Init. Var.
cuuse
2 -77
Basse mi
Hidden LR (AdamW)
Thase
Teuse 12
Thuse m
y -181." a=1
Hidden Bias LR (AdamW)
Thass
Those
Hidden WD (AdamW)
Anasa
Apase
Abase -0N
MHA Residual
X ) X + MHA(LN(X')
X mE MHA(LN(X.))
MLP Residual
Z + MMP(LN(Z')
Z' + MLP(LN(Z'))
Z'
+ MLP(LN(Z'))
Final-LN Init. Var.
Phuse
Final-LNN I.R (AdamW)
Thase
Thase
Those
Unemb. Init. Var. Unemb. LR (AdamW)
2
Euse
Puse
Thase
Thase
Thuse
Unemb. Fwd.
X <WWT.. unemb
X- W unecab M
X W uncmbb 177
AdamW (Residual blocks)
Chase
Chase
-72.
Cbose
AdamW (Emb. & Unemb.)
Ebase
Cbase -70
Fbose
4 Depth-wise HP transfer and a
Here, we investigate the HP transfer abilities of pP and a E (0.5, 1} as model depth is varied.
Traditional HP transfer First, we investigate the desth-wise transfer of Thase and Ohasc while training all models for 300M tokens with batch size B = 128 and Absse O. We remark that this is the traditional setting where the phenomenon of HP transfer was originally observed [5]. Figure shows that SP, μP, and 0.5 do not have stable optimal MJaae Ousse depth L is varied. When
Forr the embedding layer, we choose an initialization variance scale independent of input dimension due to the input data being one-hot encoded. If the input data is dense, then we would require a pre-factor of --1/ In practice we observe that a "small enough" is sufficient to achieve consistent dynamics and transfer across width depth. See Appendix G for details.
*Thanks to [86] for finding typos in our Table 1 AdamW e emb. & unemb. prescriptions and mistakes in our official code! We have updated Table I and our official code to reflect the crose my prescriptions from Equation (40) and [87].

---

using the optimal HPs for L = Lase 2, the right column shows these parameterizations do not achieve consistent loss improvements with depth.
Finding I: With SP, pP, and 0.5, as I. is varied, models do not share the same optimal HPs.
For CompleteP (a 1), the optimal Mhsse and base remain stable across depths evidenced by concentric curves with stable minima. It is the only parameterization to consistently improve loss for deeper models without HP tuning. as shown in Figure 2 right column. Such stable HP transferability dramatically reduces HP tuning budgets for deep models. We demonstrate HP transfer from 2 to 128 layers, exceeding the depth of LLaMA-708E (80 layers) and LLaMA-405B (126 layers) [58].
UP
0.5
CompletoP (a
Loss optimal
Dash
SlimP) Validation
4.5
Completa low
4.0
2.5
2-5
2-5
2.5
2.5
2
Dopth (L)
Slimi) Valldation 1885
SP 1P
= 0.5
CompleteP (a 11
Loss at optimal Oue for Lasse
5.0-
Complenst (e-
2
0xt
0u7
Depth (.)
Figure 2: Desth-wise HP transfer, with 300M training tokens. Top: Learning rate (r) transfer.
Bottom: Initialization standard deviation (aiitt transfer. See Table 4 for experimental details.
Compute-efficient HP transfer The parametcrizations we test are grounded in theories derived under fixed token count [8]. We now investigate whether the depth-wise transfer of Thuse extends to the compute-optiman setup prescribed in Hoffmann et al. [3]. where all the models are trained for 20 TPP. We also select compute-efficiat
batch sizes based on total training FLOPS [2, 88-90), and ensure a well-tuned Atsee [91] (see Appendix G for further details). Figure: 3 shows this compute- optimal setup much less sensitive the choice of Thase compared to training for 300M tokens with B = 128 and Abase = 0 in Figure 2. Despite this reduced sensitivity, the right column shows a = 1 achieves superior losses SP, μP, and a 0.5 without additional Thase tuning from Lrase- These results lead us to conclude:
Finding 2: Only CompleteP (a = 1) achieves reliable depth-wise HP transfer.
Simii) Validation Loss
SP / uP
a 0.5
CompleteP (a -
Loss at optimal / for -aaase SP/
Deeh
3.5
2.9
Depth (L)
Figure 3: Learning rate transfer test under compute-optimal setting (20 TPP, batch size based on BS-FLOP power law, optimal weight decay A See Table for experimental details.
Finding 3: Empirically, we observe larger TPP reduces HP sensitivity (Figure vs. 3).
Re-examining compute-optimal N:L ratio
We now turn to the study of the optimal N: ratio in transformers, as its tuning can significantly impact compute efficiency. Kaplan et [2] popularized the practice of fixing N T. 100 and
scaling both model parameters and tokens [54-72], but without adopting parameterization that
5

---

allows for infinite width and depth control. The consequences of this are exemplified in Section 4, where we have shown that deeper models suffer under SP/uP due hyperparameter detuning and instability, leading a = 1 to outperform. This depth-disadvaantag represents an important confounding variable in these studies on the optimal N L.. In other words, without an adequate depth parameterization, deeper models are not given their best chance to succeed. In this section, we conduct the first study of compute-optimal N: L with simultancous control of infinite depth and enables wider range of clos=:0-coompte--ppinall aspect ratios, where even N L R 10 remain close to compute-optimal.
width and show that
Experimental setup We train models in the compute-optines setting of 20 TPP, select compute- efficient batch sizes based on FLOPs [2, 88-90], and ensure well-tuned 1), X, a for Thase = 2 [91] as in the previous section. See Appendix G for extensive experimental details and plotting methodology. This time, we vary N. L to approximately maintain the total number of non embedding parameters Pnon-emb 12N L, for values of P nonnembb E {50M, 300M, 1.5B). This setup matches the model sizes from Kaplan et al. [2]. but while they use the same number of tokens for all runs (compute- inefficient), we follow the compute-optimal prescription of Hoffmann al. [3]. Instead of pure web text (2), we train on SlimPajama: a mix of web text, academic prose, and code. We use the maximal update parameterization meaning our HPs have superior tuning with respect to changing width compared to Kaplan et al. [2] who use the standard parameterization. These differences modernize our setup and make our results more applicable to contemporary LLM training.
JIP
Af 0
Comm yr 1.0g
Outindl
-A-
SIM
-B-
500M
15A
(a) Pron
50M
(b) Pasn-are 300M
(c) Pron-ene 1.58
0%
Loos incressse VS. Xg=
8%%
Locs incressoe Ke-
07
Loss intresse 18,
4%
4%
2%
0%6
0%
10"
10"
Depth (M:4)
Wedth:Depth (FF-)
Depth (N:4)
(d) XiF) for differ ent pa parameterization
Comples saxings vs UP
(e) Completet "LOP savings vs. J.P
(1) NLL rarge within 1% aptimal
GInFU Valicatoo Loss 00
84xx10 10
32x 10
Aye (PTL.40277
Ravis 1P/3 me2ry 053
35%
Mith Dapith (M4:4) (4)
Optrall
3x 10'
Raw IF/2 -143
20%
10"
Compaede
28x10" 10"
26. 10*
24x 10*
0%
15
10
10"
10"
Traiiing FLOPS (F)
Depth (L)
Pron
Figure 4: Optimal N:L across model sizes. (a)-(c) For models of size (50M, 300M, 1.5B) we see an
increase in optimal N:L as size increases but optimal L is less than x 60 in this range of model sizes. (d) Scaling laws with FLOPs for the optimal aspect ratios in each parameterization. (e) FLOP savings of CompleteP compared to uP baseline, as a function of depth L. (f) Shaded regions represent the N range with < 1% loss increase relative to compute-optimat N.L.
Results We first fit the power law of the form X(F) = (F/a) describing how validation cross entropy loss (X) scales with power law in FLOPs (F) [1, 62). The scaling law X {FF), reported in Figure 4d is fitted with the most compute-efficient : from each parameter count and it defines the compute-efficient frontier in our setup. To fairly compare model loss obtained with different FLOP budgets, we then evaluate each N: L configuration in terms of the distance to the fitted X==1 (F). This approach is similar to Figure 5 from Kaplan et al. [2]; in our case it provides a measure of the loss difference with respect to the reference scaling law of models trained with CompleteP. We report N: : I. as a function of this measure in Figure 4a-c. Notice that the optimal aspect ratio is the same in uP and CompleteP (a = 1), while a = 0.5 prefers slightly wider models. In all cases, CompleteP (a 1) gives lower loss values. In particular, we observe the following trend, (sce also Figure 4f). :

---

Finding 4: Compute-optimall N: L trends larger with increasing scale, for all parumeterizations.
Advantages of CompleteP at large depth. The gap between μPP and CompleteP increases for deeper models. We quantify this gap in terms of FLOPs savings in Figure 4e, where we can see:
Finding 5: The deeper the model, the more FLOP savings CompleteP (a = has over pP..
We attribute this gap the fact that higher depth μP is more detuned due to lack of HP transfer. In the 1.5B parameter models, CompleteP saves 11.8% of FLOPs for optimal N L and 34.4% FLOPs in the deepest models (lowest N L). The shaded regions in Figure I-right and 4f represent the N:L range with I 1% loss increase relative to compute- optiimal N:L (see Appendix G.2 for more detail).
Finding 6: As model scale increases, CompleteP (a = 1) enables deep-narron models (small N: to remain close compute-opiie opttmall
enables N:L 11.8 to remain within 1% of compute-optimal. compared to N:L = 38.7 from μP. Hardware latency worsens with increasing depth, making shallow-wid models preferable for Vatency-sensinng applications. However, low-memory hardware benefit from narrow-deep models by streaming one layer at a time into memory. Prominent examples of weight streaming exist both training [92] and inference [93, 94] settings.
For Pmmn-emb=1.5BB
Downstream Performance. In Table 2 we evaluate the 20 TPP Puon-cmb=1.50 models at the minimum and optimal N: settings, and confirm the upstream gains also translate to gains across five downstream tasks [95-100] that collectively test for common sense reasoning, world knowledge, and reading comprehension. The details of downstream evaluation setup are in Appendix G.3.
Table 2: Zero-shot downstream evaluation accuracy for 20 TPP Puon-cmb R3 1.5B models at the optimal and minimum N: L ratios. We report average task accuracy + standard error. If the average task accuracy is within the standard error, both columns are bolded.
Optimal (N 1984, 7 32)
Deepest (N = 832, = 179)
Random
μP
CompleteP (a 1)
JP
a 0.5
CompleteP
SlimPI Val. xenr. (1)
4.701
2.302
2.326
2.286
2.386
2.417
2.330
FLOP savings vs. ul (?)
0%
-19.6%
11.8%
0%
-23.7%
34.4%
HellaSwag (*)
25.0
53.3 0.5
51.7
+ 0.5
54,2 0.5
49.1 + 0.5
46.7
52.7 + 0.5
ARC-Easy (f)
25.0
54.4 1.0
54.5 + 1.0
55.6 1.0
50.0 = 1.0
49.2
1.0
54.6 + 1.0
LAMBADA
54.3 0.7
51.7
+ 0.7
54.9-0.7 0.7
51.8 0.7
43.8
0.7
53.3 + 0.7
RACE (1)
25.0
34,9 1.5
34.7 = 1.5
34.1 1.5
33.5 1.5
32.7
7 1.5
35.6 1.5
PIQA
50.0
-0.7++1.1 1.1
71.6
+ 1.1
71.5
69.6
1.1
70.5
70.6 1.1
BoolQ (1)
50.0
58.4 2 0.9
61.3
= 0.9
60.7 007=00 0.9
$7.8 = 0.9
57.9
2 0.9
59.0 + 0.9
Downstrean Aug. (1)
54.3
0.3
55.2
$2.0
0.3
50.1
0.3
Given the popularity of TPP>20 training for enhanced parameter efficiency [56], we also compare compare parameterizations at 200 TPP for P pon-emb E (50M, 300M] in Table 3. CompleteP (a 1) consistently achieves the best loss across all parameterizations, even in the 200 TPP regime.
Table 3: SlimPJ validation loss for P Pnon-emb E 300M) models trained for 200 TPP.
S0M optimal (V 512, A= 169
50M deepest (N
256, f. 63)
300M optimal (N 1024, J. 24)
300M deepest (N 448, J. 125)
0-0.5
CompleteP (om1)
CompleteP (n=1)
a=0.5
CompleteP (mm1)
om0.5
CompleteP (mm1)
2.867
2.744
2,859
2.973
2.998
2,950
2.451
2.455
2.443
2.537
2.534
2.497
6 Desiderata for Hyperparameter Transfer
Our empirical results suggest that, among all the parameterizations we tested, a yields the best performance in terms of both HP transfer and pre-training efficiency. In this section, we provide a theoretical heuristic for why this might be and how one might have arrived a priori at the CompleteP (a = 1) parameterization. Our heuristic consists of proposing three desiderata for constructing good parameterization. While variants of our first two desiderata were already proposed in [4, 7, 8), Desideratum 3 is novel and distinguishes a = 1. To frame our discussion, recall that parameterizations are typically constructed ensure consistent, numerically stable, and meaningful updates for both the hidden layer representations and the network outputs, during training at any model size.
7

---

Setup Since we are interested in finding such a parameterization for transformers, let us consider a residual network with residual layers of width N in which the representation he RV of fixed input after & residual layers satisfies the following recursion
h + F,
l (= L}.
The residual block F2 is a fixed depth neural network such as attention or MLP block. The value of a determines at what scale residual blocks contribute the residual stream. The desiderata below constrain a E [0.5, 1I, consistent with prior work on depth transfer [6-8]. We will denote by dg the trainable parameters layer & [L] = (I....,L} .L} and consider updates of the form
+ 0 + A00
Ao =n
BAdamW
(3)
Here 9Adam is the Adam W update and, as we shall see from Desideratum 2, we shall have to take
= -
to ensure that the change in he entries is ((1) for any depth or width (see Appendix C.2 or [8)). We focus our presentation on the Adam optimizer because it is widely used for LLM training, but the desiderata below can be equally well applied to any optimizer (see [6-8]).
Stable Initialization. Our first desideratum is a numerical stability requirement for hidden layer representations h and network outputs f initialization.
Desideratum (Stable Initialization). Hidden layers and oulpul remain stable at initialization. More precisely, for all layers E [T], & I|bb112 O(1) and J E O(1), as N + 00, L, -> 0O.
This desideratum prescribes how scale weight variances (or pre-factors) as Table It also constrains the value of a to be at least 0.5 (see Appendix C.I). It can be viewed as numerical stability condition and has been well-studied in the signal- propagation literature [11, 16. 17, 19-21, 40J. We review why this desideratum imposes 2 1/2 in Appendix C.I.
Maximal Residual Stream Update. Many parameterization schemes, including those from works on signal propagation [16, 17, 17,27, 12], the NTK parameterization, and the Mean Field / μP approaches [29, 33, 4], satisfy Desideratum 1 for width scaling N - xo at fixed The core distinction between them is the presence or absence of feature learning. While there are several ways to make this precise, we follow the original μP work [4] and formalize feature learning by considering the change in hidden layer representations after each step of training.
Consider the simple case of fixed-depth fully connected networks + of in which the change Ah to first order in the learning rate be naturally written sum of two terms:
Ah AW' d(h)) W W' Aphh")
The first term captures the change in A from the immediately preceding weights W and the second term reflects the change in h from updates to representations in previous layers.
To derive parameterization for HP transfer across width, [4] required not only that the relative change || ++1 11/l|l///hh | 1/2 to pre-activations in layer be @(1) but also that the proportion of this change | w (12/))122/11
attributable directly to the parameters Wr must be O(1) as well. This excludes degenerate situations such as when one trains only the first few layers of network.
While the maximal update requirement leads to a unique parameterization for HP transfer across width in fixed-depth network. it is not directly applicable to the setting where one also scales the network depth. For example, in the case when a = 0.5 and (h(:; 0³) = a (hh), one must (LL-0.6) 0.5) in order for activations to remain stable in ((1) (see Appendix C.2 & 16. 7]). To cover residual networks of growing depth, we therefore require that the maximal update prescription hold per residual block.
actually require || W (07"((hh"))) (12 /h e+ +
the sense that || h ++1||
Desideratum 2 (Maximal Residual Stream Update). Each residual block's 's weights should contribute order L/L feature movements, and each non-residual block should contribute constant order. More precisely, for all L [L- 11. each block's 's parameter update 0f + oe 2 should contribute the N | (0achh+1 f+1 ++113 E (11LL. Moreover, for the embedding and unembedding layers we require | AW 1 E ((1) and 1 E O())
change 1
N
8

---

Desideratum 2 matches the maximal update prescription of [4] when depth L is finite, ((1/L) ((1). Furthermore, Desideratum 2 uniquely determines the depth-dependence of the learning rate in AdamW to be 1j ( for the update to satisfy the @ / (.) scale [6]. Note also that requiring F(h²; 0e) = ( with respect to L constrains a < 1. When combined with Desideratum 1 this explains why we consider 0.5 S a 1. We also emphasize that additional care must be taken to correctly determine learning rates for biases and LayerNorm (LN) parameters, otherwise a = 0.5 fails Desideratum 2 for pre-LN transformer (see Figure and Appendix D).
Complete Feature Learning. Variants Desiderata and have been proposed in prior work and satisfied by any a (0.5, 1]. In this section provide one possible intuition for what distinguishes = 1 and why it might work so well in our experiments. We do so by showing that only gives parameterization in which every layer remains uniformly non-linear its parameters, regardless of depth and width (see Desideratum 3). More precisely, we define the lincarization / lin of function h0) with respect about 0u is
h 1in.0 (0, 0o) h(o)) + (Voh(0)la. 0 0o).
We say is linear in 0 if, Definition. We say a layer he is lazy with respect to subset of parameters 0 C {0j)i<s if at finite depth and width, he is not linear in 6 and the change Agh' at initialization from updating only (i.e replacing + 0 + A0) is asymptotically the same as the change to the linearization of h;
hlin.8
Doh
Aohh A
o(1), as N, ~0000
Desideratum (Complete Feature Learning). The network parameterization satisfies complete feature learning. i.e. neither the hidden layers ( beelL) nor the model output f are lazy with respect any subset of model parameters.
Reaatvve Dilaance From Lieaatzaion
0.5 L - 0
19
V Virnn
SP!
10
Comp stoP 1.00
Activation Function
Desth
Faature Transform (Unntrained)
Figure 6: Only
achieves
Figure 5: As L + 00, = 0.5 asymptotically linearizes residual blocks (Eqn. 2). Left: Depth 2 MLP residual block. For a 0.5, this block is only nonlinear with respect to parameters when L is finite. Right: Linearized MLP residual block, with V FF non-trainable transform of h!.
complete feature learning with sta- bility. SP/μP diverges at large depth (triangles are NaNs), while a = 0.5 converges to the lincarization at rate of I/VL. The y axis is the left hand side of Eq. 4.
To put this definition into context, note that in the NTK regime, where we do not scale depth, the model output are lazy with respect to all non-output parameters [0e)/-] [101]. As we will illustrate through a simple example, only a = 1 gives complete feature learning for the - oo limit.
To illustrate why Desideratum 3 distinguishes between = 1 and other values of a, consider a width N toy model where we only scale depth L - xo
Simple Block Depth 2 Example
h = ht L w w h',
= (1....., L}
(5)
To satisfy Desiderata and 2, we need the weight update to satisfy Aw; O(L=-1) for both i 1, Next, consider the update to h from the parameters of preceding residual block parameters:

---

(W (i), W = 0 + 0 + A0". Taylor expanding yields:
10 ++ = (Vork A + 2
Voch A A
L = he AWD + + (1) h
AW +
AW AW
(6)
L0-1
L
La-2
Note the first term is exactly the change 1in,6; to the lincarization of h which has the correct (+1
0i order ((L The second term therefore captures the difference between updating h h and h&+1 and has order (LL=-)). The two terms have the same order only when a = 1. For all a < 1, as we increase the depth of the network, the contribution of h the non-linear term diminishes with scale. We empirically verify this, for the same toy model, in Figure 6 (see Appendix G.4 for details). Since we expect the optimal HPss in shallow models to depend on linear and non-linear dynamies of h we do not expect these HPs to be optimal when scaling depth with < 1.
While carried out computations above only in the simple toy model (Equation (5)), the form of the calculation extends to any arbitrary Fe( 6*) with bounded depth, including large wide MIPP Ms and self- blocks. The core observation that higher order terms in the Taylor expansion (k > 1) of Ag h (+1 will always have the form
4'h e+1 [[
O (L-a. I )
[A0 ((L((-)1)
We summarize the above discussion on lazy learning into the following statement.
Only CompleteP (a 1) ensures stable training, maximal updates, and complete feature learning as + xc, cc.
Finding 7:
Given that a = is the unique a that ensures complete feature learning while scaling both width and depth, we dub it CompleteP.
7 Limitations
Our theoretical analysis considers scaling width and depth at fixed token count, which limits the direct applicability the fixed TPP compute-optimal regime. However, the scaling predictions derived in the fixed token setting (Figure 2) have predicted success in the fixed TPP compute-optimal regime (Figure 3). The X fits in Section 5 were only fit to 3 points due to budget constraints with training models larger than 1.5B parameters. However the three points span a 256x FLOP range and are each the most compute-efficient point from each group of 7-10 N:L values for each parameter count (Table 5), so they are likely to be an accurate picture of the compute-effieee efficient frontier. We use the scaling law fits for interpolation rather than extrapolation, where fitting to limited data can make predictions unreliable.
8 Conclusion
We studied in large pre-LN transformers a variety of parameterizations, i.e., prescriptions for adjusting hyperparameters such as learning rates as functions of network architecture. Among them, we found that CompleteP (a = 1) gives HP transfer when varying depth and width. We showed that CompleteP achieves significant FL.OP savings during pre-training compared to other parameterizations, even in compute-optimal settings with jointly scaled batch, dataset, and model sizes. To achieve this, extended the theoretical analysis of CompleteP to include prescriptions for how to scale LN parameters and AdamW's e across depth and width.
To understand the salutary effects of CompleteP, we formalized set of desiderata for designing parameterizations. They include variants of desiderata from prior work as well as novel property, which we called complete feature learning. that distinguishes CompleteP. This parameterization is simple to implement, and we released public code base to facilitate reproduction and further study. Our empirical tests were conducted on models with up to 1.5B non-embedding parameters. In future work, we plan to train large, state-of-the-art L.LMs that leverage CompleteP's utility HP transfer and compute-efficient training.
10

---

Acknowledgments and Disclosure of Funding
BB and LN thank the Google PhD fellowship for support. BH is supported by a Sloan Fellowship, NSF DMS-2143754, and NSF DMS-2133806 BB and C.P. acknowledge support by NSF grant DMS-2134157, NSF CAREER Award IIS- 2239980, DARPA grant DIAL-FP-038, a Sloan Research Fellowship, and The William F. Milton Fund from Harvard University. C.P. and B.B's s work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
References
[1] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv: / /7l1:00099 2017.
[2] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv preprint arXiv:2001.08366 2020.
[3] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rac, and Laurent Sifre. An Empirical Analysis of Compute-optimal Large Language Model Training. In The Conference on Neural Information Processing Systems (NeurIPS), 2022.
[4] Greg Yang and Edward J Hu. Tensor programs IV: Feature learning infinite- width neural networks. In International Conference on Machine Learning, pages 11727-11737, PMLR, 2021.
[5] Greg Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems, 34: 17084-17097, 2021.
[6] Blake Bordelon, Lorenzo Noci, Mufan Bill Li, Boris Hanin, and Cengiz Pehlevan. Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit. In The Twelfth International Conference on Learning Representations, 2024. URL https: //pprrevie......e...
[7] Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Feature learning in infinite-depth neural networks. In NeurIPS 2023 Workshop Mathematics of Modern Machine Learning. 2023.
[8] Blake Bordelon, Hamza Tahir Chaudhry, and Cengiz Pehlevan. Infinite limits of multi- head transformer dynamics. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: htapp:://opeeee http///ppneeeeeeeeeeeeeeeeeeeeee http://errreeeeeeeeeeorooaaaaaaa .ett://orrmm...ppBBkaB......
[9] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https://proceett neureediddnnnnnnnnnroeeeeeeeeeer cctppaperffill//peeeeeeeeeeeeeee Papeee pdf.
[10] Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable program- ming. Advances in Neural Information Processing Systems, 32, 2019.
[11] Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions Neural Networks, 5(2):157-166. 1994.

---

[12] Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35:27198-27211 2022.
[13] Emily Dinan, Sho Yaida, and Susan Zhang. Effective theory of transformers at initialization. arXiv preprint arXiv:2304. 02034, 2023.
[14] Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pages 9-50. Springer, 2002.
[15] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
[16] Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. Advances in Neural Information Processing Systems, 29, 2016.
[17] Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. arXiv preprint arXiv: 2016.
[18] Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pen- nington. Dynamical isometry and a mean field theory of CNNs: How to train 10.000-layer vanilla convolutional neural networks. In International Conference on Machine Learning, pages 5393 5402. PMLR, 2018.
[19] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing numan--evee performance on ImageNet classification. In Proceedings of the IEEE International Conference on Computer Vision, pages 1026-1034, 2015.
[20] Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. Advances in Neural Information Processing Systems, 31, 2018.
[21] Boris Hanin. Which neural net architectures give rise exploding and vanishing gradients? Advances in Neural Information Processing Systems, 31, 2018.
[22] Boris Hanin and Mihai Nica. Products of many large random matrices and gradients in deep neural networks. Communications Mathematical Physics, 376(1):287-322, 2020.
[23] Lorenzo Noci, Gregor Bachmann, Kevin Roth, Sebastian Nowozin, and Thomas Hofmann. Precise characterization of the prior predictive distribution of deep ReLU networks. Advances in Neural Information Processing Systems, 14:20851-20866 2021.
[24] Mufan Li, Mihai Nica, and Dan Roy. The neural covariance SDE: Shaped infinite depth-and- width networks at initialization. Advances in Neural Information Processing Systems, 35: 10795-10808, 2022.
[25] Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint arXiv: 1901.0931,, 2019.
[26] Lorenzo Noci, Chuning Li, Mufan Li, Bobby He, Thomas Hofmann, Chris Maddison, and Dan Roy. The shaped transformer: Attention models in the infinite depth-and width limit. Advances in Neural Information Processing Systems, 36, 2024.
[27] Soufiane Hayou, Commutative scaling of width and depth in deep neural networks. Journal of Machine Learning Research, 25(299): 2024.
[28] Akhil Kedia, Mohd Abbas Zaidi, Sushil Khyalia, Jungho Jung, Harshith Goka, and Haejun Lec. Transformers get stable: An end-to-end signal propagation theory for language models. arXiv preprint arXiv:2403.09633. 2024.
[29] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization neural networks. Advances in Neural Information Processing Systems, 31, 2018.
12

---

[30] Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over- parameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis, 59:85-116, 2022.
[31] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 32, 2019.
[32] Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evo- lution in wide neural networks. Advances in Neural Information Processing Systems, 35: 32240-32256, 2022.
[33] Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33): 77665-EE667 2018.
[34] Grant Rotskoff and Eric Handen-Eijnden Trainability and accuracy of artificial neural net- works: An interacting particle system approach. Communications on Pure and Applied Mathematics, 75(9): 1889-1935, 2022.
[35] Lenaic Chizat and Francis Bach. On the global convergence gradient descent for over- parameterized models using optimal transport. Advances in Neural Information Processing Systems, 31, 2018.
[36] Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: A law of large numbers. SIAM Journal on Applied Mathematics, 80(2):725-752, 2020.
[37] Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: A central limit theorem. Stochastic Processes and their Applications, 130(3):1820- 182-11822, 2020.
[38] Katie Everett, Lechao Xiao, Mitchell Wortsman, Alexander A Alemi, Roman Novak, Peter Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, and Jeffrey Pennington. Scaling exponents across parameterizations and optimizers. arXiv preprint MxXi::2007.0587.. 2024.
[39] Lucas Lingle. A large-scale exploration of μ-transfer. arXiv preprint arXiv:2404.05728, 2024.
[40] Daniel A Roberts, Sho Yaida, and Boris Hanin. The principles of deep learning theory, volume 46. Cambridge University Press Cambridge, MA, USA, 2022.
[41] Sho Yaida. Meta-principled family of hyperparameter scaling strategies. arXiv preprint SXXi::2210004000 2022.
[42] Nikhil Vyas, Alexander Atanasov, Blake Bordelon. Depen Morwani, Sabarish Sainathan, and Cengiz Pehlevan. Feature-learning networks are consistent across widths at realistic scales. Advances in Neural Information Processing Systems, 36: 10:6000-1000, 2023.
[43] Lorenzo Noci, Alexandru Meterez, Thomas Hofmann, and Antonio Orvieto. Super consistency of neural network landscapes and learning rate transfer. In Advances in Neural Information Processing Systems, 2024.
[44] Blake Bordelon and Cengiz Pehlevan. Dynamics of finite width kernel and prediction fluctua- tions in mean field neural networks. Advances Neural Information Processing Systems, 36. 2024.
[45] Moritz Haas, Jin Xu, Volkan Cevher, and Leena Chennuru Vankadara. μP? Effective sharpness aware minimization requires layerwise perturbation scaling. arXiv preprint arX0v:2411.00075, 2024.
[46] Charlie Blake, Constantin Eichenberg, Josef Dean, Lukas Balles, Luke Y Prince, Björn Deiseroth, Andres Felipe Cruz-Salinas, Carlo Luschi, Samuel Weinbach, and Douglas Orr. u-uP: The unit-scaled maximal update parametrization. arXiv preprint 24Xiv:2407.1740 2024.
13

---

[47] Nolan Simran Dey, Shane Bergsma, and Joel Hestness. Sparse maximal update parameteriza- tion: A holistic approach sparse training dynamics. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[48] Leena Chennuru Vankadara, Jin Xu, Moritz Haas, and Volkan Cevher. On feature learning in structured state space models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[49] Edward Milsom, Ben Anson, and Laurence Aitchison. Function-space learning rates, arXiv preprint arXiv:2502.17405, 2025.
[50] Tim Large, Yang Liu, Minyoung Huh, Hyojin Bahng, Phillip Isola, and Jeremy Bernstein. Scalable optimization in the modular norm. arXiv preprint 1rXiv:2405.14873, 2024.
[51] Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng. and Shiwei Liu. The curse of depth in large language models. arXiv preprint arXiv:2502.05795, 2025.
[52] Jeonghoon Kim, Byeongchan Lee, Cheonbok Park, Yeontaek Oh, Beomjun Kim, Taehwan Yoo, Seongjin Shin, Dongyoon Han, Jinwoo Shin, and Kang Min Yoo. Peri-LN: Revisiting layer normalization in the transformer architecture. arXiv preprint arXiv:2502.00 02722222., 2025.
[53] Hongyu Wang. Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. DeepNet: Scaling transformers to 1,000 layers. arXiv preprint arXiv:2203.00555, 2022.
[54] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
[55] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open pre-trained transformer language models. arXiv preprint trXiv:2205.01066 2022.
[56] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient founda- tion language models. arXiv preprint arXiv: 2023.
[57] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. LLaMA 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[58] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, al. The Llama herd of models. arXiv preprint arXiv:2407.217833 2024. 2024.
[59] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, et al. Gemma: Open models based on Gemini research and technology. arXiv preprint wwXiv:2403.08295 2024.
[60] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00 200:000000000111 18, 2024.
[61] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Mer- hej. Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, et al. Gemma technical report. arXiv preprint arXiv:2503. 19786, 2025.
14

---

[62] Nolan Dey. Gurpreet Gosal, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, and Joel Hestmess. Cerebras-GPM Open compute-optimal language models trained on the Cerebras wafer-scale cluster. arXiv preprint arXiv:2304.088:
2023.
[63] Nolan Dey, Daria Soboleva, Faisal Al- Al-Khateeb, Bowen Yang, Ribhu Pathria, Hemant Khachane, Shaheer Muhammad, Zhiming, Chen, Robert Myers, Jacob Robert Steeves, Natalia Vassilieva, Marvin Tom, and Joel Hestness. BTL.-3B-8K: 7B parameter performance in a 3B parameter model. arXiv preprint arXiv:2309. /568, 2023.
[64] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord. Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, et al. OLMo: Accelerating the science of language models. arXiv preprint arXiv:2402.008838 2024.
[65] Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, et al. 2 OLMo Furious. arXiv preprint arXiv:2501 200.:500000066, 2025.
[66] Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A compact 7B SSM hybrid model. arXiv preprint arXiv:2405. /6712, 2024.
[67] Paolo Glorioso, Quentin Anthony, Yury Tokpanov, Anna Golubeva, Vasudev Shyam, James Whittington, Jonathan Pilault, and Beren Millidge. The Zamba2 suite: Technical report. arXiv preprint 1..i::24111152220 2024.
[68] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, arXiv preprint arXivv2334.0133 2023.
[69] Mosaic Research Team. Introducing MPT-7B: A new standard for open-source, commercially usable LLMs. https: //wp:///wwwwwwe/ databriaaai..o cks. com/ hoom/loo//ee httg//mpt77 May 2023.
[70] Mosaic Research Team. MPT-30B: Raising the bar for open-source foundation models. https:/ h/www//wwwww.co www.baiaatkkeeo http://dtaabrccs..oo//.oo/gggoo
June 2023.
[71] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023.
[72] Jeffrey Li. Alex Fang. Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, et al. DataComp-LLMe training sets for language models. arXiv preprint ar Xrxi::2406.11 1194, 2025.
search of the next generation of
[73] Sean McLeish, John Kirchenbauer, David Yu Miller, Siddharth Singh, Abhinav Bhatele, Micah Goldblum, Ashwinee Panda, and Tom Goldstein. Gemstones: A model suite for multi-faceted scaling laws. arXiv preprint arXiv:2502.0683 06857, 2025.
[74] Mingxing Tan and Quoe V. Le. EfficientNet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11900 2020.
[75] Ibrahim Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting VIT in shape: Scaling laws for compute-optimal model design. arXiv preprint arXiv:2305.13035 2024.
[76] Yoav Levine, Noam Wies, Or Sharir, Hofit Bata, and Amnon Shashua. The depth-to- width interplay in self-attention. arXiv preprint mrXiv22006.12467. 2021.
15

---

[77] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gatee mixture-of-experts layer. arXiv preprint arXiv: 1701.0658, 2017.
[78] Ankit Goyal, Alexey Bochkovskiy, Jia Deng, and Vladlen Koltun. Non-deep networks. arXiv preprint arXiv:2 10.06641, 2021.
[79] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. URL https: pesttss:/d4moo /tt////kmmcckyyw cloudfront www.tteeeeeeeteeeeeeeeeeeeeeee.. Saagtterr-annuugg--aaee soaddlsslanguggeen models pdf.
[80] Ofir Press, Noah Smith, and Mike Lewis. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. In International Conference on Learning Representations, 2022.
[81] David R. So, Wojciech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V. Le. Primer: Searching for efficient transformers for language modeling, arXiv preprint arXiv:2109.0866 2022.
[82] Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun Xiao, Chenyang Song. Zhiyuan Liu, Zeyu Mi, and Maosong Sun. ReLU2 wins: Discovering efficient activation functions for sparse LLMs.. arXiv preprint mwXiv:2402.0382 2024.
[83] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[84] Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, and Joel Hestness. Straight to zero: Why linearly decaying the learning rate zero works best for LLMs. In International Conference on Learning Representations, 2025.
[85] Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. slimpaj a-mma-a-67b--te wwl.eencleanee--dd-eeee ootededdddddeeeaaadd-eerraooaoda 2023. URL httph::/t///ttt htap:s////gggigaaceeeee co//datasets/cerebaaaa//tteeeeee
attps://wwx http:/wwwwweeebaaae//// hetpiaassnt//boog////.
[86] Bruno Mlodozeniec, Pierre Ablin, Louis Béthune, Dan Busbridge, Michal Klein, Jason Ramapuram, and Marco Cuturi. Completed hyperparameter transfer across modules, width, depth, batch and duration, 2025. URL https: /ttpps::/rrraee /awxiixxvvv.//// org/ abs////2//2.... 2bb.2222222282. 22382.
[87] Greg Yang and Etai Littwin. Tensor programs IVb: Adaptive optimization in the infinite-width limit. arXiv preprint arXiv:2308.01814, 2023.
[88] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. arXiv preprint arXiv: 18/2.061 2018.
[89] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong. Qiushi Du, Zhe Fu, et al. DeepSeek LLM: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.00 .00.99944 2024.
[90] Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Re- solving discrepancies in compute-optimal scaling of language models. arXiv preprint arXiv:2406. 01024406.91446 19146, 2024.
[91] Xi Wang and Laurence Aitchison. How set AdamW's weight decay as you scale model and dataset size. arXiv preprint arXiv:2405.13699 2024.
[92] Cerebras Streaming
Systems.
Linear
Scaling
Made Possible with Weight https://weet cerebras. ai/blog/ September 2022.
Cerebras.
[93] Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. L.MM a flash: Efficient large language model inference with limited memory. arXiv preprint
2024.
16

---

[94] Ying Sheng, Lianmin Zheng. Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Perey Liang, Christopher Ré, Ion Stoica, and Ce Zhang, FlexGen: High-throughput generative inference of large language models with single GPU. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[95] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? arXiv preprint
2019.
[96] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge. arXiv preprini / 180.055577 2018.
[97] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring broad discourse context. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics, 2016.
[98] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale reading comprehension dataset from examinations. arXiv preprint vrXiv:170.00460 2017.
[99] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language. arXiv preprint arXiv: 111199911//444 2019.
[100] Christopher Clark, Kenton Lee, Ming-Wer Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019.
[101] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jaseha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances Neural Information Processing Systems, 32, 2019.
[102] David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint 1xXi:22104.10500 2021.
[103] Emily M Bender, Timnit Gebru, Angelina McMillan- Majorr and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610-623, 2021.
[104] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. arXiv preprint arXiv: /906. 1906.0223. 2019.
[105] Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl- dickstein, Kelvin Xu, Jachoon Lee, Justin Gilmer, and Simon Kornblith. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309.14:: 14322, 2023.
[106] Saaketh Narayan, Abhay Gupta, Mansheej Paul, and Davis Blalock. juit scaling: Simple and scalable fp8 Imm training. arXiv preprint 2wxiv:2502.0966 2025.
[107] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa. Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https: org/ records/ 12608602..
[108] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 2911)1111114) 2012.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All of the claims made in the abstract and introduction are supported by both theoretical and empirical results we present in Section 4, 5, and 6.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section we discuss the limitations of our work. In Section 6 and Ap- pendix D we state our theoretical assumptions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [Yes]
Justification:
In Section 6 and Appendix D we state our theoretical assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: In Appendix G well as Section 4 and Section 5, disclose all the informa- tion needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental
Question:
material?
Answer: [Yes]
Justification: We provide a minimal open-source implementation of the parameterizations used in this work in our supplementary material. We also train all our models on SlimPajama [85], open-source dataset.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
cablic/guide /t/g//ddd//oo/S// Cooee/umiiissoonn olliy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. wctppblpccccccccccccccccccccccce hwmblicgguideg///o/ guwdes/ /ooe//oooeeeeseeeeeeeeeee wwwsssssssssieeeeeeeoo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: In Table 4 well Section 4 and Section 5, specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Table 2, along with task accuracy we report the standard error for each measurement. Other tests involve reporting upstream validation cross entropy loss single seed so we have no method of quantifying the variability.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Table as well as Section and Section 5, we provide extensive experimental details which can be used easily derive compute requirements based reader's specific hardware setting.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't It make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https /ttpar///////ee//eeeee cc/public/Ettiiei cttphi//eehhccciiieleeeeeeeeee
Answer: [Yes]
Justification:
This work conforms to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential broader impacts of our work in Appendix A.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do release any models or datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite and use all training and evaluation datasets in accordance with their licenses.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: The only new asset introduced in this work is minimal code implementation our parameterizations included in our supplementary material. This accompanied by "README. file containing clear documentation.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: do not conduct any research involving crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
We do conduct any research involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to LLM policy (https: : w/w./eurpps.../ for what should or should not be described.
A Broader impacts
As LLM compute budgets grow, compute efficient training methods are becoming increasingly important for reducing carbon emissions [102] and offisttingg environmental and financial costs of large model training [103]. This work presents methods which improve the compute efficiency of LLM training, especially for large deep models. There is also growing recognition that HP tuning is key contributor to these costs. HP tuning is costly, possibly undermining equity in AI research due to financial resources [104]. During model retraining, sensitivity to HPs also leads to downstream costs [104]. This work presents methods which can reduce these costs and sensitivities and thus improve equity.
B Coordinate check test for verification of stable training
To verify our theoretical expectations for training stability match empirical results, we perform "coordinate check" tests where we scale L. and measure the change in activation size the final residual addition h2 in Figure 7. Interestingly, a 0.5 as described in the literature [6-8] did not achieve transformer training stability as we initially expected (2nd column). We reasoned this was due to lack of consideration for how the bias and LayerNorm parameters can affect training stability. After adopting the bias and LayerNorm 7) adjustments proposed in Table and Appendix D, we achieved training stability with stability for any depth. This test is a necessary but not sufficient condition for satisfying Desiderata
0.5 (3rd column). Figure shows both E {0.5, 1} achieve
C Scalings that Guarantee Our Desiderata
In the following analysis, we argue why a 2 necessary for Desideratum 1 (Stable Initialization) and we argue for AdamW LR to be = - for each residual block to achieve Desideratum (Maximal Residual Stream Update).
C.1 Desideratum 1: Stable Initialization
! is necessary for stable signal propagation at initialization. In this section, we demonstrate that 2 2 For concreteness in the simplest model that captures this effeett, consider residual MLP network

---

SP
= 0.5 w/o Bias and Scalinn
0.5
CompleteR la 1.0)
100
Depth (1)
Depth
Figure 7: Coordinate check. Frobenius norm of activations after merged residual streams from MHSA and MLP blocks across models increasing depths after training step t. 1st column: SP/μP without depth control. 2nd column:
0.5 as described in literature [6-8]. 3rd column: 0.5 (Table 1).
4th column: a = 1.0. a E {0.5, 1} achieve stability for any depth.
with block depth 1, the model analyzed in [6].
h f+ h2 + Le
W' (hh).
(8)
The weights are initialized with sf ~ No, ). Defining He = 1n as the norm of the residual stream variables, we have the following recursion for H at large N
H = H" + L 20 1e
where the average E, represents an average over the hidden neurons at residual layer é [6]. Taking for concreteness (h)) ReLU(h), the above equation gives
H H² ow L 2 H² = 1
L-
OW 2
L
H
(10)
where 17 is the scale of the first residual variable. The last layer variance therefore has the following large depth limit
H
α1 >
lim H² +
exp
aiv
H
(1I)
0 <
We see that a 2 1 is necessary for the residual stream variance to converge to a finite value as 2 L For 1/ the residual stream covariance depends on the activation function and the initialization variance ofy while for the the signal propagation becomes trivial in the infinite depth limit as the contribution from the nonfmeerrtt and the initial weight variance disappears at large L.
C.2 Desideratum 2: Maximal Update
In this section, we consider the Adam learning rate necessary to achieve @(1) changes to the residual stream variables. First, we note that the backward pass variables g" Sh are strongly correlated across layers since, from the chain rule,
f+1
ohhh))
(ww) g g²*1].
(12)
La
We therefore note that any weighted average of over layers is O (1), such as
L
-ck=-0(1))
1 Σ c = O₂(1), I
(13)
where Ef is an arbitrary sequence with increments of scale C++1 C² (b Consider now the update under sign-GD update
(t+ W (t) 7
+ d(h))(hh). 2 -
1
9
(14)
7
25

---

where of is the learning rate for this hidden layer. Note that AdamW will contain the same scaling (with N, L) for the numerator and denominator but will also contain momentum (which does not change the scaling with N, L.) and an additional parameter which we discuss how to scale with N, L in Appendix D.5.
We now discuss how to choose of to obtain (1, updates to the residual stream (Desidcratum 2). First, we can express the residual variables layer + step & of training as
h (t) = (t) + w (0)6(h'()))
of
+
(
(()(b(()) (()(())
(15)
UCE
The entries of the above vector from the update to (t) have the following scaling with N and L
(()0()(()) (t)()
(t, =
(lh((t)) = Ox,(())
(16)
as the variables (t)) and ( (t)) are random variables with @(1) correlation. We desire the final layer of the residual stream to achieve (1)) updates
L-1 h² h h w (0))((())))
L-1 Σ " EV
(17)
C'
by 1/2
Desired scale @(1)
Since 2 3. the stability of the middle term is guaranteed from the DI analysis. The v² variables strongly correlated across layers, so sum all L layers will give result of order @(L). To make this final term (1) we must choose a learning rate of scale
T/hase N T,=1
(18)
This choice causes the final term above in the (1) formula be @(1), resulting in @(1) change to the residual variables due to the weight updates as desired.
D Derivation for bias, LN, WD, and AdamW parameterization adjustments
D.1 On Generalizability of the Parameterization
In the later parts of this subsection, we will discuss derivations of parameterizations for several different components of the architecture via heuristic method, which we find very accurately repre- sents the scaling important quantities. However, this remains limited as there are other architecture components that are not contained by manuscript, which may require further adjustments and derivations.
With this in mind, we can divide up the future generalizations into three categories:
No modifications required. This includes LayerNorm at any position (pre, post, QK-norm etc.), adding biases at different locations, LR schedule, and other well-normalized Adam-like algorithms (e.g. SignGD, Adagrad, Lion), or position embeddings (learned, ROPE, ALiBi, NoPE, etc.). The reason is that these do not change the scale of both the forward pass and hidden layer updates due to training, with respect to width and depth. LayerNorm, for example, maintains the (1) scale of each neuron, and long as the update maintains the (11) scale, then no changes are required.
2. Slight modifications are required and known. A good example of this is SGD, where the learning rate scaling derived in Table of Bordelon et [8] (up to an equivalent ABC- reparameterization), Notice here that compared to Adam, the learning rate of SGD needs be scaled up due to the prefactors front of both the weight matrices and the residual branch, where these extra factors canceled out in Adam due to normalization.
26

---

3. More theoretical derivations required. This includes MoE, long context, batch size (and schedule), gradient clipping. LAMB, and momentum. Some of these are a relatively straight forward exercise to derive (e.g. LAMB and other optimization algorithms), others are less straight forward. points and training steps along with width and depth remains theoretically challenging. Our approach in this work computes the theoretical scaling for finitely many data points and training steps, and testing whether not hyperparameter transfer empirically, which has been yielding very good results (see e.g. Figure 3).
particular, any scaling that requires increasing the number of data
For the third category, we would effectively need to derive the scale the updates and downstream effects for each new architectural modification. A good example is consider the effect of bias weights in the next subsection. Here biases b' are updated such that it contributes the same amount of changes to f the usual weights W In order to do that, need to choose the appropriate initialization and learning scale to satisfy this requirement, All other modifications follow the same mechanism, and the rest of this section serve example derivation for all future generalizations.
D.2 Bias Parameters
In this section, we consider the effect of bias parameter. To start, we will illustrate the effect of bias parameters depth MLP residual block
1
= f=N N W o(h"), h -h h
W (''(h')) +b] h W
(19)
La
The entries of the bias vectors b' are initialized with unit variance bd ~ No0 I) for all layers while We entries are NO0 }) for intermediate layers & E (1,..., L - 1} and (0, 1) for encoder and decoder layers E {0, L}. With this choice, stable signal propagation is guaranteed for 2 1 2' To obtain the desired 1 increment in the residual stream, the weight and bias parameters must be updated with scale
AW = (N-1L--1) AU; = =(L=-)), e [L,..., --1
(20)
For the Adam optimizer this directly sets the desired scaling of learning rate for each of these parameters
7/4 θ ONN (N- Th ((T
(21)
This argument extends to more complicated blocks such as self attention layers and deeper MLPs. However, for the first and last layer of the transformer which we want to achieve O(1) changes to their block outputs we need to set
AWG = (1), Ab, = e(1, E (0,
(22)
D.3 LayerNorm
While a static LayerNorm operation was examined in [8], most LayerNorm operations include trainable gain and bias parameters for each neuron.
1 LN Vol+e
hh- [h'- mml] og +
(23)
where nr and o are the mean and variance of the entries of h For any hidden block in the residual stream, the entries of g and must be updated with
vl (1,..., I. 1) Ag. = O(T Al = 1(.
(24)
while for the first and last layer (before and after the residual stream).
Ag, )( Ab O(L=-1), L ().
(25)
27

---

D.4 Weight Decay
In this section, we will perform a heuristic calculation the weight decay scaling required in AdamW, which we will approximate with SignGD-lik update of the type
w (1) " O Vw: (((()) f/ (1),
(26)
where Zf is collection of constant such that 1 (((tt have ( entries with respect to width N and depth L, À is the weight decay parameter.
Suppose in either an MLP or inside residual block, we have a fully connected layer (including the cases for Q, K, V blocks)
h 4-1 h = Woo(h") where W E NNN is initialized with (N0, (0, /N).
(27)
Suppose we want h change by order (1)) in an SignGID update with one data point, then we get that
(++ w'(t) Z ogg g s+ (t))((((t)) - (t),
(28)
where Z E R N is once again the normalization constants, and g is the backward pass variable [32, Equation 16]. However, in this case we have that 9 and (hh are already ((1) in the previous iterate so Z is itself @(1).
Next we will calculate Ah
(t (t+ 1)= = + (t (t+ +) h (t) to get
-h+ o(h)) w'Ap(h").
(29)
where we abuse notations slightly and drop the indices when clear. Here we will also assume as induction hypothesis that Ar(h' is well behaved so we will focus on the first term only, which is
AW o(h) -
2(h²)
<(h") n/ (hh)
Z
(30)
(0h
T
2(h) n h b
z
where the entries of h are (1) by the arguments of Appendix C.I. Following the arguments of Appendix C.2, the learning rate 17 must be set as
L La-1 N
7)
Nhsse
(31)
to ensure the correct change to the residual stream variables. Since we desire the final term to be on the same scale this means that O (NN) Our proposed in weight decay scaling rule (Table 1) is also used in a few recent works [91, 105, 106]. The contribution of this section is to show no additional depth correction to the weight decay parameter is required.
D.5 Adam Parameter
In principle, the e parameter for Adam may need to be scaled with width or depth to obtain stable behavior. The key problem can be seen from the update for a parameter /
of
A
m
(32)
where is moving average of squared gradients and m, is a moving average of gradients. Prior work outlined various strategies to achieve this stability as width is scaled [87, 38). We will now describe strategies to achieve of the parameter across depths.
Layer-wise c, y. One strategy is to utilize the current parameterization provided in the main text and to modify the parameter and learning rate for each type of layer. The parameterization is
wh') h f+1 = h + N
1
W (hh). h' = -
(33)
L
28

---

where the hidden weights W are initialized with random entries from
W N"),, L {l,..., L 1} and the first and last layer are initialized with unit variance
(34)
(0) ~ 1), () ~ ~
(35)
We note that the my and v, variables have the following scalings in the residual stream
m θ (L "N Vot e Le L- 1}
(36)
m = θ (N V = - {0
(37)
This parameterization demands the following learning rate for hidden layers for an e -+ limit to be stable
rf = To N La-1 f {l, L- 1 1}
(38)
rf = E {
(39)
Now, consider the effect of we desire it match the scale of the raw gradients m, take
é' =0 é' = be 40
(40)
This will ensure that is of the same order as vo for all hidden layers. For the read-in and readout we can pre-multipy gradients by constant.
Reparameterize the Defintion of the Model: Another strategy that one could adopt is to reparameterize the model that single value of be used for every layer. In this case,
W o(h") h NL N=
f=
1
= h
w ohh), h²
1
W
(41)
L
L
where the readin and readout weights are initialized as
W0( ~ No 2²),,
~ N10,
(42)
The global parameter can now be set as
6 L= -N
(43)
The learning rates must be set as
n = Lo 1 {}
(44)
nf a E {1, L 1}.
(45)
These rules will ensure the correct scale updates in each layer.
E Depth-wise transfer of AdamWa
Everett al. [38] show appropriate AdamW e scaling is important for compute-efficient large models. Building on the width adjustment for AdamW e introduced by Yang and Littwin [87], we add a depth correction for AdamW Figure shows this correction enables stable training for a wider range of Caase.
F Depth Scaling Training Dynamics
We illustrate the training dynamics in the form of loss curves in Figure and 10, which slice the n transfer test in Figure 2 at CompleteP (a = 1)'s and = 0.5's optimal respectively. CompleteP (a = 1) consistently achieves lower training loss when model depth scales up. In a 0.5, we point out the loss "crossing" behavior where deeper models continually take longer steps scale better than shallower models. This aligns with our observation in Figure 2 where a 0.5 desires larger with increasing model depth and laziness (defined in Section 6).
29

---

E- Eoas Ml
ELUs -pa
Depth
LoGs
4.5
18
4.0
126
3.8
10" 10
10"
10"
10"
10"
10
10"
10"
-5
10
70"
Figure 8: Using ComplteP (a 1), we ablate the effect of the depth- Adam scaling rule. The scaling rule enables wider range of ébase to achieve competitive loss.
/
CompleteP 1)
SlimP) Train Loss
10
Depth
120
200
400
600
800
1000
1200
200
400
600
800
1000
1200
200
400
600
800
1000
1200
Train Steps
Train Steps
Train Steps
Figure 9: Training dynamics of Figure for all parameterizations at CompleteP (a 1)'s optimal 2-8
7
G Additional experimental details
Table 4 contains extensive details for all experiments. The rest of this section describes the remaining experimental details.
G.1 Compute-efficient setup
In Figure to study the compute-optinal setting, first training setup must represent compute- optimal. Following [3], we train models for 20 tokens per parameter (TPP) to ensure a compute- optimal tradeoff of parameters and tokens. We also scale our batch size according FLOPS. Since, McCandlish et al. [88], Kaplan et al. [2] showed the critical batch size scales with a power law in loss, and loss scales with power law FLOPS, the critical batch size scales with power law in FLOPS [89, 90]. We follow Equation 46 based on empirical fits and rounded to the nearest multiple of where B, batch size and training FLOPS respectively.
power law in
max (32, 0.7857 0.157 306.8)
(46)
We adopt the Tbase 0.02 and the optimal Thase for I = Thuse 2 from Figure 2: Thusee 0.0039.
To ensure optimal weight decay we scale model and dataset size, we follow Wang and Aitchison A major shortcoming of Tema is that does not take the learning rate schedule into account, even though learning rates are decayed in practice. Despite this we observe reliable transfer of TEMA across model sizes, provided we use the same learning rate schedule. For SP and μP, Figure 11 validates that the claims of Wang and Aitchison [91] that the optimal TEMA remains stable across different learning rates and weight decay values. We then tune TEMA for SP, μP, a 0.5, and CompleteP (a 1). Figure 12 shows Tema 0. 1407 close to optimal for all parameterizations and this adopted throughout this paper. When non-zero weight decay is adopted, we use Tema: Thase Doteps to decide the Xhuse used in each
[91] by setting Absse to maintain the optimal TEMA = (Paesse)
run.
30

---

/ JP
CompleteP (a 1)
SlimP) Train Loss
Depth
128
0
200
400
600
800
1000
1200
0
200
400
600
800
1000
1200
0
200
400
800
800
1000
1200
Train Steps
Train Steps
Train Steps
Figure 10: Training dynamics of Figure for all parameterizations at 1) = 2 where a = 0.5 first demonstrates deeper-model-lower-loo ordering.
Table 4: Additional experimental details. Cells marked as "vary" mean the quantity is varied.
Figure
N
init
Those
Anase
TEMa
B
Steps
Tokens
TPP
left, 2 top, 9, 10
256
Vary
0.02
Vary
0
N/A
128
1144
300M
Vary
bottom
256
Vary
Vary
2H-08
0
N/A
128
1144
300M
Vary
256
Vary
0.02
Vary
Vary
0.1407
Eqn. 46
Vary
Vary
20
4. 13, middle right
Vary
Vary
0.02
2E-08
Vary
0.1407
Eqn. 46
Vary
Vary
20
256
Vary
0.06
2E-03
0
N/A
10
82K
Vary
256
Vary
0.02
2E-08
N/A
128
1144
300M
Vary
Vary
Vary
Vary
Vary
Eqn. 46
Vary
Vary
20
Vary
Vary
2E-08
Vary
Vary
Eqn. 46
Vary
Vary
20
Table
Vary
Vary
0.02
2F-08
Vary
0.1407
Eqn. 46
Vary
Vary
200
G.2 N:L study details
Here we include additional experimental details for Figure 4. We vary N, to approximately maintain P Ponn-emb 12N L. Table 5 contains all model shapes, token counts, and FLOPs used in Figure 4. Approximations training FLOPs like = 6N D do not account for embedding, attention, LN, nonlinearity bias FLOPs. We use very granular function to measure FLOPs. Although maintain Pnon-emb- the FLOPs are not equal mainly because of embedding and attention costs. We train Pa emb {50M, 300M, 1.5B) models, matching sizes from [2], but all at 20 TPP. [2] used similar study design, but with the same number tokens for all runs, meaning it didn' follow the compute-optine prescription of [3].
For each F, X) point in Figure 4d, we calculate the "Loss increase vs. Xa=1" as = X-X and plot it in Figure 4a-c. To understand how the FLOPP savings in Figure 4e are calculated, first recall the functional form of (F)) (F/a)=& b. Inverting yields F(X) aX t. Due to the scale invariance of power laws, we can calculate:
FLOP savings vs. uP 1- (1 (duP d da=1))-
(47)
SPIpP. 768,
SPIμP, N= 1088,0==
SP/μP, 1=3
SimPJ Vaiddaalltloor Lods
3.05
3=37
3.05
2.78
g=Btee00
2.78
3.00
3.00
7u 6e-t2
2g-42222
2.95
274
2.95
Twu 0.1407
2.90
10
10"
10
10
TONA (oassehss
Texx, (Qoncc
Tex Pose
Figure 11: When grid searching learning rate Thase and weight decay Xtuse the optimal TEMA remains stable.
31

---

SP|PP
0.5
CompletaP
L-10
1.00
0.1407
L-00
SteP Treen (aweh
1,075
TEM: 2.1407
SW/P Tran B8st
1,100
N-1008 1-14
N-250, L:50
Tax 0.1407
9.04
1.000
1.050
1,085
1.00
1,000
1,000
10
10"
10
10"
Figure 12: TEMA sweep for μP and E [0.5, TEMA changes minimally across parameterizations. The TEMA values we empirically chose sit the bowl of optimal loss values. TEMA for 0.5 seems good but for up and = 1 seems slightly off. Looks like aspect ratio affects the optimal TEMA-
Table 5: Details of model sizes used in the aspect ratio study. The compute-optimal configurations for uP and
are bolded. The compute-optinge optimall configurations for 0.5 are italicized
Label
P Poon-cmb (M)
P Puea (M)
Tokens (B)
Train FLOPS
Steps
N
L
N:L
50M
49.8
75.5
1.5
1.25E+18
4849
152
256
63
4.1
50M
49.3
81.5
1.6
1.26E+18
5235
152
320
40
8.0
50M
49.7
88.3
1.8
1.34E+18
5671
152
384
13.7
5OM
50.4
101.9
2.0
1.56E+18
5923
168
512
16
32.0
50M
49.2
113.6
2.3
1.76E+18
6301
176
640
10
64.0
50M
49.6
126.8
2.5
2.07E+18
6730
184
768
7
109.7
5OM
138.3
2.8
2.35E+18
7033
192
896
179.2
50M
50.4
153.3
3.1
2.82E+18
7198
208
1024
4
256.0
50M
47.8
163.6
3.3
3.11E+18
7397
216
1152
3
384.0
50M
47.6
189.1
3.8
4.02E+18
7696
240
1408
2
704.0
300M
301.8
346.8
6.9
2.38E+19
8301
408
448
125
3.6
300M
305.3
369.6
7.4
2.32E+19
8846
408
640
10.3
300M
299.4
383.1
7.7
2,27E+19
9352
400
832
36
23.1
300M
302.3
405.2
8.1
2.38E+19
9699
408
1024
24
42.7
300M
314.8
443.5
8.9
2.70E+19
10214
424
1280
16
80.0
300M
307.4
468.2
9.4
2.85E+19
10784
424
1600
160.0
300M
302.1
508.0
10.2
3,20E+19
11275
440
2048
6
341.3
300M
298.7
588.2
11.8
4.06E+19
12169
472
2880
3
960.0
1,5
1488.8
1572.5
31.4
4.10E+20
19195
800
832
179
4.6
1,5
1498.4
1614.2
32.3
3,96E+20
19903
792
1152
94
12.3
1.5B
1501.6
1656.0
33.1
3.91E+20
20418
792
1536
53
29.0
L,5
1512.3
1711.8
34.2
3.99E+20
21106
792
1984
62.0
1.5B
1487.9
1751.6
35.0
4,00E+20
21597
792
2624
18
145.8
1.5B
1487.3
1841.1
36.8
4,26E+20
22474
800
3520
10
352.0
1.5B
1487.0
1943.8
38.9
4.62E+20
23262
816
4544
6
757.3
Figure 13 shows how Figure 4f was created by fitting cubic functions data {a log(N L), pootimaly and finding intersections with y Cubie functions were used instead of quadratics to enable better fits to functions without an axis of symmetry.
G.3 Downstream evaluation details
We evaluated our models on downstream tasks using Eleuther Harness Evaluation (EEH) [107]. The evaluation details are reported in Table 6.
G.4 Figure details
To visualize the notion of laziness in (4):
Agh
Ao lin. e
o(1),
as N, 00,
(48)
Aolhtfling
32

---

0-05
0-10
Upane <A-
XMM
XAN
1,30
754
LoWk POWAAR Trom
12"
Deeth
Wethellpphh (N1L)
With (7:4)
Figure 13: Plotted loss increase from best point. Fit cubic functions to log-transformed data. Found intersections with y = for Figure 4f.
Table 6: Details downstream tasks evaluations. stands for accuracy. acc stands for accuracy normalized by target string's s byte-length.
Evaluation
EEH Task Name
Metric
Hellaswag
hellaswag are easy
acc_norm acc norm acc
ARC-easy
LAMBADA
lambada lambada_openall
RACE
race
acc
PIQA
piqa
acc porm
BoolQ
boolq
acc
we consider the toy architecture of residual network with two layers per residual branch:
h c+1 h L w W h1
(= -= L},
(49)
with width N 256, and perform the learning rate updates according our parameterizations, i.c.:
m a- a E [1/2,1]
(50)
μP.
We then perform a single gradient step only with respect W left hand side the laziness equation above varying depths across 50 seeds per run. report the median with lines and shade the region between the first and third quartiles. We compute 4gh isn
(2) for fixed layer L, and calculate the
explicitly by differentiating the block with respect to W and evaluate at the initial parameter values, i.c we compute:
(2)
lin.8 (0, 0n) h(0,) + (Voh(0)lo, 0 Oo):
where 0 W (2). We use to 0.0001 and train MNIST [108]. We provide the code to reproduce
this plot in supplementary material.
33


Reviews:
[{"review_id": "vj3FML35q3", "reviewer": "Reviewer_yU41", "summary": "Hyperparameter (HP) tuning is expensive at scale. Prior work like µP has shown how to enable zero-shot HP transfer when scaling model width. This work extends that line of research by providing a recipe for scaling HPs—such as bias learning rates, weight decay, Adam epsilon, and LayerNorm parameters, enabling a zero-shot HP transfer from shallow to deep networks. They show up to 34% FLOP savings.", "questions": "1. The authors demonstrate both empirically and theoretically that setting the residual scaling α = 1 enables full, non-lazy feature learning across depth. How does this finding relate to, or differ from, the s=1 regime described in the prior theoretical work Meta-Principled Family of Hyperparameter Scaling Strategies, ref. [41]? In what way is Desideratum 3 novel beyond what is implied by the s=1 maximal-update regime?\n\n2. There are many hyperparameters that influence training dynamics as model size increases. This work focuses on scaling a specific subset: bias learning rates, weight decay, Adam epsilon, and LayerNorm parameters. Could the authors elaborate on why these were prioritized over others—such as batch size, learning rate warmup schedules, gradient clipping, or optimizer momentum—and whether the proposed method is sensitive to those remaining hyperparameters?\n\n3. The work is developed and evaluated using pre-LayerNorm Transformer architectures, which are commonly used for deep models due to their favorable gradient dynamics. Could the authors comment on whether CompleteP generalizes to alternative normalization schemes, such as post-LN or sandwich-LN architectures? Would any modifications to the scaling rules (e.g., for residuals or LayerNorm) be required in those settings?\n\n4. The paper demonstrates the proposed scaling strategy using AdamW, but does not explore its interaction with other optimizers such as SGD, LAMB, or Lion. Could the authors comment on whether CompleteP generalizes to these optimizers, and whether additional tuning or scaling adjustments would be required?\n\n5. In Figure 4 (d), it appears that the scaling law fit is based on only three data points. Could the authors clarify whether this is the case, and if so, how robust the fit is to such limited data? Additionally, for Figure 4(f), how are the shaded region estimated?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "(+) practical recipe for training deep transformers\n\n(+) enabling zero-shot HP transfer, reducing HP tuning budget\n\n(+) unlocking model shapes better suited for different hardware\n\n(+) simple implementation, require only a few lines of code\n\n\t\n(-) lack of direct theoretical comparison to prior work\n\n(-) selective hyperparameter focus, the method only address a subset of HPs\n\n(-) unclear if proposed approach generalize beyond transformer and Adam optimizer", "was_revised": true, "final_justification": "This is a technically solid paper, with high impact for AI practitioners. I maintain my accept score.", "rebuttal": "Thank you for the detailed and constructive feedback, and your positive notes recognizing CompleteP as a simple and practical method for training deep transformers which enables zero-shot hyperparameter transfer and unlocks hardware-efficient model shapes.\n\n```\nDoes CompleteP generalize across batch size, learning rate warmup schedules, gradient clipping, optimizer momentum, post-LN, sandwich-LN, SGD, LAMB, or Lion? Would additional tuning or scaling adjustments be required?\n```\nThis is an excellent question relevant for many practitioners! We can definitely do a better job at explaining the generalizability of the CompleteP scaling, as well as explain the math for how to derive other cases. There are roughly three separate categories of generalizability: \n\n1. No modifications required. This includes LayerNorm at any position (pre, post, QK-norm etc.), adding biases at different locations, LR schedule, and other well-normalized Adam-like algorithms (e.g. SignGD, Adagrad, Lion), or position embeddings (learned, RoPE, ALiBi, NoPE, etc.). The reason is that these do not change the scale of both the forward pass and hidden layer updates due to training, with respect to width and depth. LayerNorm, for example, maintains the $\\Theta(1)$ scale of each neuron, and as long as the update maintains the $\\Theta(1)$ scale, then no changes are required. \n\n2. Slight modifications are required and known. A good example of this is SGD, where the learning rate scaling is derived in Table 1 of Bordelon et al. [8] (up to an equivalent ABC-reparameterization). Notice here that compared to Adam, the learning rate of SGD needs to be scaled up due to the prefactors in front of both the weight matrices and the residual branch, where as these extra factors are canceled out in Adam due to normalization. \n\n3. More theoretical derivations required. This includes MoE, long context, batch size (and schedule), gradient clipping, LAMB, and momentum. Some of these are a relatively straight forward exercise to derive (e.g. LAMB and other optimization algorithms), others are less straight forward. In particular, any scaling that requires increasing the number of data points and training steps along with width and depth remains theoretically challenging. Our approach in this work computes the theoretical scaling for finitely many data points and training steps, and testing whether or not hyperparameter transfer empirically, which has been yielding very good results (see e.g. Figure 3). \n\nTo improve the paper, we will include a more comprehensive and pedagogical section on generalizability, so can improve the accessibility of adopting CompleteP in a variety of training scenarios. Thank you for raising these concerns! \n\n```\nLack of direct theoretical comparison to prior work. The authors demonstrate both empirically and theoretically that setting the residual scaling $\\alpha=1$ enables full, non-lazy feature learning across depth. How does this finding relate to, or differ from, the $s=1$ regime described in the prior theoretical work Meta-Principled Family of Hyperparameter Scaling Strategies, ref. [41]? In what way is Desideratum 3 novel beyond what is implied by the $s=1$ maximal-update regime?\n```\nThank you for pointing out the connection of our work to Yaida [41]. The main contribution of Yaida [41] is identifying how to have consistent training dynamics (and hence HP transfer) as a function of a parameter $s\\in [0,1]$ in joint limits of diverging depth $L$ and width $n$ when fixing the ratio of $\\gamma = \\frac{L}{n^{1-s}}$ in MLPs. Since the roles of depth in MLPs and ResNets are quite different, it is hard to directly compare our parameterization to his. Moreover, since Yaida [41] considered non-shaped activations, he found that in the $\\mu P, s=1$ regime, one must keep the resulting ratio $\\gamma = L$ constant to have stable training. In contrast, we find that one can have diverging $n,L$ separately and still have HP transfer and stable training dynamics in ResNets. \n\nWe will add this discussion to the manuscript as this is an interesting and relevant connection to our work.\n\n```\nIn Figure 4 (d), it appears that the scaling law fit is based on only three data points. Could the authors clarify whether this is the case, and if so, how robust the fit is to such limited data?\n```\nYes the scaling law fit in Figure 4d is based on three data points. We acknowledge the limitations of this approach on lines 633-637.\n\nHowever each of three points are the most compute-efficient point from each group of points where we sweep 7-10 values of N:L for each parameter count, so they are likely to be an accurate picture of the compute-efficient frontier. The three points also span a 256x range of training FLOPs (1.56e18-3.99e20). Table 5 provides further details on this experimental design. We mainly use the scaling law fits for interpolation rather than a heavy extrapolation regime where fitting to limited data could make predictions unreliable. \n\nWe believe the above discussion provides important context so we will update the limitations section to include it. Thank you for raising this concern!\n\n```\nFor Figure 4(f), how are the shaded regions estimated?\n```\nThe shaded regions in Figure 4f (and Figure 1 right) represent the N:L range ensures <= 1\\% loss increase relative to the most compute-efficient point collected. The process for determining the shaded regions in Figure 4f is outlined on lines 817-819 and in Figure 13. On line 160 we do state \"See Appendix I.3 for extensive details on model shapes and plots\" but this seems woefully inadequate given that we include this plot on the first page of the paper. To remedy this we will update the manuscript to better explain this methodology near where the figure appears and have more informative captions. Thank you for noting this was unclear!\n\nWe will also clarify Figure 1 (middle) by changing the Y-axis label to \"FLOP savings vs. muP *at same depth*\" and noting the black points are \"Compute-optimal *(across all depths)*\"."}, {"review_id": "dixT2Xfzjg", "reviewer": "Reviewer_skfW", "summary": "This work studies depth-dependent parameterizations for hyperparameter transfer when training deeper models. The authors introduce the parameterization completeP, which they show guarantees non-lazy learning at each layer as models become deeper. They demonstrate that this parameterization enables effective hyperparameter transfer as model depth increases and improves performance relative to standard parameterization, μP, and another depth-dependent parameterization (i.e., α=0.5). They additionally demonstrate that completeP allows for a wider range of compute-efficient width/depth ratios. Their results are shown on large language models trained over a sweep of model sizes using large datasets.", "questions": "# Comments\n\n- I don’t understand what you are plotting in Figure 1(c) — the colored areas represent the “range of efficient N:L,” but how do you define this? I suggest you describe it clearly in the caption or within the figure itself.\n\n- Lines 35–37 don’t make sense to me as written. You introduce that the depth-dependent rescaling factor is governed by a single parameter α ∈ [0.5, 1], define α=1 as your algorithm completeP, and then state “Since α=0.5 and α=1 are the two most promising potential candidates for depth scaling...” but it’s unclear why α=0.5 is included. Has α=0.5 been previously defined in the literature? I think adding a sentence or two explaining what prior work has done with α=0.5 and how your work extends this will make the introduction clearer.\n\n- Since you continuously compare to the α=0.5 setting from the work “Feature Learning in Infinite-Depth Neural Networks,” I would suggest giving this setting a name.\n\n- I think you should also briefly discuss what would happen for α ∈ (0.5, 1.0), even though I agree with your logic that the boundary cases are the most important.\n\n- In Kaplan et al., they use a version of the WebText dataset, while your work uses SlimPajama, correct? It’s worth explicitly highlighting this difference.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\n- Tackles an extremely important problem for scaling large language models and improving the performance of state-of-the-art systems.\n- Presents an impressive display of empirical testing.\n- High-quality experiments, well written, and significant work overall.\n\n# Weaknesses\n\n- This work is not particularly original; it primarily extends prior ideas (references 6–8), but does an excellent job empirically testing and validating these results to demonstrate that completeP is the best setting. That said, I think this work could do a better job distinguishing its contributions from prior works (see my comments below)\n\n- Despite the impressive empirical sweep, there are still limitations. For example, as discussed in lines 79–88, there exist other layer-dependent rescalings that are not considered in this empirical study, and the results are only performed on the single dataset SlimPajama. I suggest moving the discussion of these limitations, such as the one you include in Appendix B, up into the main conclusion section.", "was_revised": true, "final_justification": "This is a strong paper with impressive empirical results and I recommend acceptance. While, not the most original, it is very well executed and the rebuttal helped clarify any remaining questions I had.", "rebuttal": "Thank you for the detailed feedback and your positive notes regarding the paper's impact, impressive empirical results, and clear presentation.\n\n```\nDespite the impressive empirical sweep, there are still limitations. For example, as discussed in lines 79–88, there exist other layer-dependent rescalings that are not considered in this empirical study, and the results are only performed on the single dataset SlimPajama. I suggest moving the discussion of these limitations, such as the one you include in Appendix B, up into the main conclusion section.\n```\nThis is a great suggestion! We will integrate the limitations section (Appendix B) into the main body and update it to highlight that we don't empirically test FLERM [49] or Modula [50] and update lines 629-630 to say \"Our empirical results only test the setting of next-token prediction decoder-only pre-LN transformers trained with AdamW on the SlimPajama text dataset.\"\n\n```\nI don’t understand what you are plotting in Figure 1(c) — the colored areas represent the “range of efficient N:L,” but how do you define this? I suggest you describe it clearly in the caption or within the figure itself.\n```\nThe colored areas in Figure 1-right (and Figure 4f) represent the N:L range ensures <= 1\\% loss increase relative to the most compute-efficient point collected. The process for determining the colored areas in Figure 1-right is outlined on lines 817-819 and in Figure 13. On line 160 we do state \"See Appendix I.3 for extensive details on model shapes and plots\" but this seems woefully inadequate given that we include this plot on the first page of the paper. To remedy this we will update the manuscript to better explain this methodology near where the figure appears and have more informative captions. Thank you for noting this was unclear!\n\nWe will also clarify Figure 1 (middle) by changing the Y-axis label to \"FLOP savings vs. muP *at same depth*\" and noting the black points are \"Compute-optimal *(across all depths)*\".\n\n```\nLines 35–37 don’t make sense to me as written. You introduce that the depth-dependent rescaling factor is governed by a single parameter $\\alpha \\in [0.5, 1]$, define $\\alpha=1$ as your algorithm completeP, and then state “Since $\\alpha=0.5$ and $\\alpha=1$ are the two most promising potential candidates for depth scaling...” but it’s unclear why $\\alpha=0.5$ is included. Has $\\alpha=0.5$ been previously defined in the literature? I think adding a sentence or two explaining what prior work has done with $\\alpha=0.5$ and how your work extends this will make the introduction clearer.\n```\nThank you for pointing out the unclear reference to $\\alpha=0.5$ in the introduction. On lines 72-76 we briefly mention how [7] argued $\\alpha=0.5$ was the best in practice and this paper shows CompleteP ($\\alpha=1$) is superior to $\\alpha=0.5$. However this information is not presented in the introduction, making the choice of $\\alpha=0.5$ on lines 35-37 unmotivated and weakening the introduction by assuming too much prior knowledge. We will update the introduction to provide more background on why both $\\alpha=0.5$ and $\\alpha=1$ are promising candidates for depth scaling.\n\n```\nSince you continuously compare to the $\\alpha=0.5$ setting from the work “Feature Learning in Infinite-Depth Neural Networks,” I would suggest giving this setting a name.\n```\nThank you for the suggestion. This is something we discussed while writing but ultimately decided against it. We considered naming the $\\alpha=0.5$ setting \"LazyP\" because it will be in the lazy learning regime in the infinite-depth limit. However this is not a unique name since any $\\alpha<1$ could also be called \"LazyP\". On the other hand, CompleteP is a unique name because only $\\alpha=1$ ensures complete feature learning. We were also concerned that \"LazyP\" would be too demeaning to the $\\alpha=0.5$ setting. We hope this clarifies our thinking!\n\n```\nI think you should also briefly discuss what would happen for $\\alpha \\in (0.5, 1)$, even though I agree with your logic that the boundary cases are the most important.\n```\nYes we agree and include a brief discussion in Section 6 (lines 271-275). We see now that some discussion of the $\\alpha \\in (0.5, 1)$ case should also be included in the introduction and related work to help motivate why we chose to study these specific boundary cases. Thank you for raising this point!\n\n```\nIn Kaplan et al., they use a version of the WebText dataset, while your work uses SlimPajama, correct? It’s worth explicitly highlighting this difference.\n```\nGood point. SlimPajama is a mix of web text (Common Crawl and C4), academic prose (Books, arXiv, and Wikipedia), and code (Github and Stack Exchange). Training on a more diverse and high quality dataset modernizes our setup and potentially makes our results more applicable to contemporary LLM training. We should also mention that Kaplan et al. [2] use the standard parameterization in their experiments. We will highlight this difference within lines 154-160. Thank you for the constructive feedback!"}, {"review_id": "Z5nCKtVBPW", "reviewer": "Reviewer_sp8Y", "summary": "The work builds on the maximal update parameterization (μP) framework by studying different values of the depth scaling parameter α in the residual connection formula: h^(ℓ+1) = h^ℓ + L^(-α) F^ℓ(h^ℓ).\n\nThe authors compare different parameterizations: Standard Parameterization (SP), μP, α = 0.5, and CompleteP (α = 1). Through extensive experiments on transformer language models ranging from 75M to 1.8B parameters trained on SlimPajama, they demonstrate that only CompleteP achieves reliable depth-wise hyperparameter transfer while maintaining training stability. Key empirical findings include: (1) Provides 11.8-34.4% FLOP savings over μP for deep models, and (2) Better accuracy (validation loss and zero shot results) at optimal points. (3) Wider range of depth to width ratios, enabling better choices across different HW platforms. \n\nThe authors also provide a theoretical justification of why setting α=1 works.", "questions": "1. Modern day LLMs are moving towards longer context, MoEs, Group Query Attention, RMSNorm and other enhancements to architecture. Would these analysis persists for these architectures too?\n\n2. Can the authors provide guidance on hyperparameter sensitivity analysis for practitioners for completeP?\n\n3. Will the benefits persist under different training regimes (e.g., longer training, different batch size schedules)?", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Comprehensive empirical evaluation: The paper provides extensive experiments across multiple model sizes, training settings, and evaluation metrics, including both upstream pretraining loss and downstream task performance. Studies also cover iso-tokens and compute optimal training (20 tokens per parameter) settings, accounting for different variations\n\n2. Theoretical grounding: The three desiderata provide a principled framework for understanding why CompleteP works, with the \"complete feature learning\" concept offering novel theoretical insight.\n\n\nWeakness:\n\n1. Adding α on top of μP to enable depth scaling has been covered by prior work. While this work, does consolidate discrepancies between various work and adopt it to transformers, but overall approach builds incrementally on existing μP and depth scaling work.\n        --- NOTE: I did not evaluate the theoretical justification provided in Section 6. The theoretical analysis falls beyond my expertise. So any novelty or originality there would have been missed by me.\n\n2. Largest scale of the model is 1.8B. Lack of validation on truly large-scale models (10B+ parameters) weakens the claims a bit\n\n3. While compute optimal settings provide good insights, modern day small LLMs are trained far beyond the 20TPP number. Impact of data can be seen from Figure 2 bottom to Figure 3. This makes the claim a little harder to verify.", "was_revised": true, "final_justification": "After reading the rebuttal and review from other reviewers, I have increased my rating by 1. Some of the questions around novelty and scale still remain, thus I did not push my rating further.", "rebuttal": "Thank you for your very useful suggestions, your positive comments regarding the comprehensive empirical evaluation and theoretical insights.\n\n```\nAdding $\\alpha$ on top of $\\mu P$ to enable depth scaling has been covered by prior work. While this work, does consolidate discrepancies between various work and adopt it to transformers, but overall approach builds incrementally on existing $\\mu P$ and depth scaling work. --- NOTE: I did not evaluate the theoretical justification provided in Section 6. The theoretical analysis falls beyond my expertise. So any novelty or originality there would have been missed by me.\n```\nYou are completely right that various choices of $\\alpha$ ranging from $1/2$ to $1$ were proposed in prior work [6-8] as ways to stabilize ResNet training across growing depth. There are, however, three sources of significant novelty related to the choice of parameterization in our work. Your comment helped us understand that we need to do a better job of highlighting them in our revision. We summarize them here:\n- **We showed depth-wise HP transfer for Transformers is possible at all.** Prior work -- notably [7] -- suggested that *no setting of $\\alpha$* can possibly give HP transfer across growing depth for Transformers. In contrast we believe that we've conclusively shown that $\\alpha = 1$ does in fact give good HP transfer and efficient pre-training when scaling both model depth and width. \n- **We refined prior $\\alpha$ parameterizations.** We found that to ensure stable training for infinite-depth models, we needed to *refine prior parameterizations for depth scaling* to include adjustments for the bias and LayerNorm learning rates (see Section C and Table 1). We also obtained new prescriptions for scaling the AdamW $\\epsilon$ parameter, which had been previously overlooked as a parameter that needs to be rescaled as a function of depth and width (see Table 1 and Figure 8).  \n- **We propose a criterion for constructing good parameterizations.** In Section 6, our desideratum 3 regarding *complete feature learning* is a novel conceptual idea. It gives the intuitive explanation for why $\\alpha = 1$ is most likely to give HP transfer in practice and achieve superior compute-efficiency to lazy parameterizations like $\\alpha=0.5$.\n\n```\nLargest scale of the model is 1.8B. Lack of validation on truly large-scale models (10B+ parameters) weakens the claims a bit\n```\nThank you for noting our comprehensive empirical evaluation. While we agree that testing at even larger scales (10B+ parameters) would be valuable, we emphasize that our current experiments already support the following key claims:\n\n1. CompleteP enables depth-wise HP transfer: We demonstrate HP transfer from 2 to 128 layers in Figures 2 and 3, exceeding the depth of LLaMA-70B (80 layers) and LLaMA-405B (126 layers), with additional theoretical grounding in the infinite-depth limit [8].\n\n2. CompleteP ensures complete feature learning: We provide both theoretical justification (Section 6) and empirical results extending to 4096-layer models (Figure 6).\n\n3. CompleteP improves the compute-efficiency of narrow-deep models: Kaplan et al. [2] performed a highly influential study that shaped the default $N:L \\approx 100$ design choice across nearly all subsequent LLMs, despite relying on standard parameterization, fixed-token training, and models up to only 1.5B parameters (line 91-93). In contrast to Kaplan et al. [2], we adopt a significantly more robust setup: $\\mu$P and CompleteP parameterizations, compute-efficient training (20 tokens per parameter) (Hoffmann et al. [3]) (lines 158-160). Our largest models reach 1.8B parameters, a scale that is already representative of widely-used small open models (e.g., Gemma-2B, Phi-1.5, and LLaMA-3.2-1B). With experiments spanning a 370x compute range (1.25e18-4.62e20 FLOPs, Table 5), CompleteP consistently improves the compute efficiency of deeper models.\n\nWe recognize many of these points could be made more clear in the paper and will revise the results and limitations sections to better highlight this context and clarify how our results substantiate these claims. Thank you for helping to improve our submission!\n\n```\nWhile compute optimal settings provide good insights, modern day small LLMs are trained far beyond the 20TPP number. Impact of data can be seen from Figure 2 bottom to Figure 3. This makes the claim a little harder to verify. Will the benefits persist under different training regimes (e.g., longer training, different batch size schedules)?\n```\nThis is a great point: we also believe that the practical utility of a parameterization such as CompleteP rests in large part on its usefulness across a range of models, datasets, and training procedures. While it is not computationally feasible to exhaustively test $\\alpha = 1$ across as many scenarios as we'd like, we do believe our work covers at least the settings below. We will emphasize them in the introduction of our revision:\n- **Varying token counts.** We demonstrate depth HP transfer for CompleteP at 300M tokens (most models under-trained) and 20 TPP (Figures 2 and 3). We also compare the performance of parameterizations across N:L ratios and show CompleteP is superior to $\\mu$P and $\\alpha=0.5$ for both 20 TPP (Figure 4 and Table 2) and 200 TPP (Table 3) settings. We believe this covers a reasonable range of practical scenarios, ranging from under-trained to compute-optimal to over-trained. We now see that the 200 TPP results (Table 3) are buried in the appendix with no references. We will move these to the main body.\n- **Varying batch-size prescriptions.** We demonstrate depth HP transfer for CompleteP both at constant batch size (Figure 2) and with batch size as a function of training FLOPs (Equation 47) to enhance compute-efficiency (Figure 3). Unfortunately we did not test *batch size schedules* in which batch size varies across optimization steps. This is one of the many configurations we would have liked to test but didn't have the compute to prioritize. \n\n```\nModern day LLMs are moving towards longer context, MoEs, Group Query Attention, RMSNorm and other enhancements to architecture. Would these analysis persists for these architectures too?\n```\nThis is a really good question! We can definitely do a better job at explaining the generalizability of the CompleteP scaling for practitioners, as well as explain the math for how to derive other cases. There are roughly three separate categories of generalizability: \n\n1. No modifications required. This includes LayerNorm at any position (pre, post, QK-norm etc.), adding biases at different locations, LR schedule, and other well-normalized Adam-like algorithms (e.g. SignGD, Adagrad, Lion), or position embeddings (learned, RoPE, ALiBi, NoPE, etc.). The reason is that these do not change the scale of both the forward pass and hidden layer updates due to training, with respect to width and depth. LayerNorm, for example, maintains the $\\Theta(1)$ scale of each neuron, and as long as the update maintains the $\\Theta(1)$ scale, then no changes are required. \n\n2. Slight modifications are required and known. A good example of this is SGD, where the learning rate scaling is derived in Table 1 of Bordelon et al. [8] (up to an equivalent ABC-reparameterization). Notice here that compared to Adam, the learning rate of SGD needs to be scaled up due to the prefactors in front of both the weight matrices and the residual branch, where as these extra factors are canceled out in Adam due to normalization. \n\n3. More theoretical derivations required. This includes MoE, long context, batch size (and schedule), gradient clipping, LAMB, and momentum. Some of these are a relatively straight forward exercise to derive (e.g. LAMB and other optimization algorithms), others are less straight forward. In particular, any scaling that requires increasing the number of data points and training steps along with width and depth remains theoretically challenging. Our approach in this work computes the theoretical scaling for finitely many data points and training steps, and testing whether or not hyperparameter transfer empirically, which has been yielding very good results (see e.g. Figure 3). \n\nTo improve the paper, we will include a more comprehensive and pedagogical section on generalizability, so can improve the accessibility of adopting CompleteP in a variety of training scenarios. Thank you for raising these concerns! \n\n```\nCan the authors provide guidance on hyperparameter sensitivity analysis for practitioners for CompleteP?\n```\nThis is a great question. First we note that Figures 2, 3, and 8 provide a hyperparameter sensitivity analysis for learning rate, initialization, and AdamW $\\epsilon$. However, ideally we would have tested the sensitivity of more hyperparameters. The theory on which we built CompleteP does not directly give any indication about the sensitivity of optimal HPs. However, we found empirically that *larger TPP leads to less sensitive optimal HP configurations* (see Figure 2 vs 3), and so that would be our main piece of advise to practitioners: it is likely that more over-trained models have more robust HPs. We didn't think to specifically formulate this point and will include it more prominently in our revision."}, {"review_id": "B3jlCEIWcV", "reviewer": "Reviewer_8H9A", "summary": "The paper studies compute-efficient hyper-parameter transfer for LLM models. A novel parameterization method called CompleteP is introduced, providing findings and guidance for efficient and reliable HP transfer. This method addresses the challenge of transferring optimal hyperparameters across varying model depths in deep transformers, which is crucial for efficient training and deployment of LLMs. Three major contributions include non-lazy depth-wise HP transfer, significant FLOPs saving when comparing with previous SOTA methods, and larger range of model width/depth ratios to remain compute-efficiency, making it suitable for various hardware settings and model deployment.", "questions": "1. In Figure 1, the FLOP saving is calculated based on equation (48). Are there any other hardware related metrics, such as power/ energy saving or latency profiling can be reported to show the effectiveness the compute efficiency?  \n2. For learning rate transfer and initialization standard deviation transfer, are there more plots beyond the task of SlimPJ? Is the trend consistent across different validation tasks?\n3. Could you please provide more details on the Methodology section, for example, which model are you using, and what are the steps to implement the parameterization? \n4. For Table 2, do you have results for the case of N:L > 100?", "limitations": "Yes, the authors have adequately addressed the limitations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThis paper is well studied on the topic of efficient HP transfer, with in-depth analysis, extensive experiments and empirical results, providing practical guidance for LLM model training under different hardware requirements. The paper is well written. \n\n\nWeaknesses:\n\n1. Although the paper has in-depth study on the topic, the result is limited to pre-LN transformers. Whether the findings still hold for other architectures or optimizers remains unknown. And how the method scale to models beyond 1.5B needs to be further verified. This may limit the application of such method. \n2. Some results and plots are based on empirical fitting instead of real profiling on certain hardware. This may also hurt the accuracy and generality of the report.", "was_revised": true, "final_justification": "This paper is well studied on the topic of efficient HP transfer, with in-depth analysis, extensive experiments and empirical results. All my concerns are well addressed, so I am pleased to raise the score to 5. One additional suggestion is to include a comprehensive analysis of the generalizability of the CompleteP scaling to enhance its practical applicability.", "rebuttal": "Thank you very much for the helpful comments, and for your kind words regarding the paper's extensive empirical results, practical guidance, and clear presentation.\n\n```\nThe result is limited to pre-LN transformers. Whether the findings still hold for other architectures or optimizers remains unknown.\n```\nThis is a really good question! We can definitely do a better job at explaining the generalizability of the CompleteP scaling for practitioners, as well as explain the math for how to derive other cases. There are roughly three separate categories of generalizability: \n\n1. No modifications required. This includes LayerNorm at any position (pre, post, QK-norm etc.), adding biases at different locations, LR schedule, and other well-normalized Adam-like algorithms (e.g. SignGD, Adagrad, Lion), or position embeddings (learned, RoPE, ALiBi, NoPE, etc.). The reason is that these do not change the scale of both the forward pass and hidden layer updates due to training, with respect to width and depth. LayerNorm, for example, maintains the $\\Theta(1)$ scale of each neuron, and as long as the update maintains the $\\Theta(1)$ scale, then no changes are required. \n\n2. Slight modifications are required and known. A good example of this is SGD, where the learning rate scaling is derived in Table 1 of Bordelon et al. [8] (up to an equivalent ABC-reparameterization). Notice here that compared to Adam, the learning rate of SGD needs to be scaled up due to the prefactors in front of both the weight matrices and the residual branch, where as these extra factors are canceled out in Adam due to normalization. \n\n3. More theoretical derivations required. This includes MoE, long context, batch size (and schedule), gradient clipping, LAMB, and momentum. Some of these are a relatively straight forward exercise to derive (e.g. LAMB and other optimization algorithms), others are less straight forward. In particular, any scaling that requires increasing the number of data points and training steps along with width and depth remains theoretically challenging. Our approach in this work computes the theoretical scaling for finitely many data points and training steps, and testing whether or not hyperparameter transfer empirically, which has been yielding very good results (see e.g. Figure 3). \n\nTo improve the paper, we will include a more comprehensive and pedagogical section on generalizability, so can improve the accessibility of adopting CompleteP in a variety of training scenarios. Thank you for raising these concerns!\n\n```\nHow the method scale to models beyond 1.5B needs to be further verified. This may limit the application of such method.\n```\nThank you for noting our extensive empirical evaluation. While we agree that testing at even larger scales (10B+ parameters) would be valuable, we emphasize that our current experiments already support the following key claims:\n\n1. CompleteP enables depth-wise HP transfer: We demonstrate HP transfer from 2 to 128 layers in Figures 2 and 3, exceeding the depth of LLaMA-70B (80 layers) and LLaMA-405B (126 layers), with additional theoretical grounding in the infinite-depth limit [8].\n\n2. CompleteP ensures complete feature learning: We provide both theoretical justification (Section 6) and empirical results extending to 4096-layer models (Figure 6).\n\n3. CompleteP improves the compute-efficiency of narrow-deep models: Kaplan et al. [2] performed a highly influential study that shaped the default $N:L \\approx 100$ design choice across nearly all subsequent LLMs, despite relying on standard parameterization, fixed-token training, and models up to only 1.5B parameters (line 91-93). In contrast to Kaplan et al. [2], we adopt a significantly more robust setup: $\\mu$P and CompleteP parameterizations, compute-efficient training (20 tokens per parameter) (Hoffmann et al. [3]) (lines 158-160). Our largest models reach 1.8B parameters, a scale that is already representative of widely-used small open models (e.g., Gemma-2B, Phi-1.5, and LLaMA-3.2-1B). With experiments spanning a 370x compute range (1.25e18-4.62e20 FLOPs, Table 5), CompleteP consistently improves the compute efficiency of deeper models.\n\nWe recognize many of these points could be made more clear in the paper and will revise the results and limitations sections to better highlight this context and clarify how our results substantiate these claims. Thank you for helping to improve our submission!\n\n```\nSome results and plots are based on empirical fitting instead of real profiling on certain hardware. This may also hurt the accuracy and generality of the report. In Figure 1, the FLOP saving is calculated based on equation (48). Are there any other hardware related metrics, such as power/ energy saving or latency profiling can be reported to show the effectiveness of the compute efficiency?\n```\nYes, we study FLOPs because it is hardware-agnostic and widely-used (e.g. Kaplan et al. [2], Hoffmann et al. [3]). However, we agree it is valuable to discuss other metrics and how they may inform potential hardware-specific limitations to running different N:L ratios. For example if a particular combination of code base and hardware cluster struggled to get consistent utilization across different N:L ratios, then a more hardware-specific training cost metric such as wall clock time on your cluster would provide a more accurate picture.\n\n```\nFor learning rate transfer and initialization standard deviation transfer, are there more plots beyond the task of SlimPJ? Is the trend consistent across different validation tasks?\n```\nWe agree that it is important to carefully quantify the impact of datasets on the performance of different parameterizations. Unfortunately we only evaluated our learning rate transfer and initialization standard deviation transfer tests using SlimPajama validation loss as the y-axis. However, in Table 2 we sho the SlimPajama validation loss improvements of CompleteP ($\\alpha=1$) over $\\alpha=0.5$ and $\\mu$P carry over to six downstream tasks: HellaSwag, ARC-Easy, LAMBADA, RACE, PIQA, and BoolQ. These results suggest the CompleteP benefits should generalize beyond the just the SlimPajama dataset.\n\n```\nCould you please provide more details on the Methodology section, for example, which model are you using, and what are the steps to implement the parameterization?\n```\nThank you for raising this point. The Methodology section does briefly explain we train decoder-only transformer language models with pre-normalization but then refers the reader to Appendix I which contains an extensive description of the models and training settings we used. In retrospect this seems to abrupt and we agree the Methodology section would benefit from repeating some of the Appendix I content. Regarding the steps to implement the parameterization, we do provide Table 1 in the Methodology section (line 108-109) which contains detailed steps to implement the parameterization and refer to an attached minimal code implementation on line 118. We will update line 118 to mention the code implements Table 1 and reproduces Figure 7. Please let us know if we could make the implementation steps more clear.\n\n```\nFor Table 2, do you have results for the case of N:L > 100?\n```\nThank you for the question. Unfortunately we don't have results for the case of N:L > 100 in Table 2 currently. However, as shown in Figures 4a, 4b, 4c, and 4e, models with higher N:L ratios (i.e., wider and shallower networks) tend to have CompleteP performance that closely tracks that of $\\mu$P, as the depth $L$ approaches the base depth $L_{\\text{base}}$. While we did not include specific results for N\\:L > 100 in Table 2, we focused on ranges that highlight two regimes of interest: the optimal N:L ratio from a compute-efficiency standpoint, and the low N:L regime, where CompleteP uniquely provides benefits not seen with $\\mu$P. We agree that further exploration of very high N:L values could be interesting and appreciate the suggestion."}]

Author rebuttal:

Meta-review: This submission proposes a parameterization for training neural networks that avoids the lazy regime (where features at intermediate layers of the network don't evolve during training) while still ensuring hyperparameter transfer across depth and width scaling (e.g., we can find the good stepsize with a small, cheap model and then use this on the big, expensive model).

Reviewers received the paper warmly, with all of them giving mostly positive initial reviews. They highlighted the paper's strengths as its rigorous approach, its practicality (few lines of code for implementation, easy HP tuning via transfer), and its comprehensive numerical experiments, which primarily cover language modeling.

The paper could have been improved by looking at optimization algorithms beyond Adam or by being more innovative/original in its theoretical aspects, since the parameterization ultimately used is quite closely related to what has already been proposed in some other works on parameterizations for hyperparameter transfer. I find that these flaws are not substantial enough to warrant rejection since the parameterization and study involved are novel, even if the idea that parameterizations can be chosen to ensure hyperparameter transfer was not novel per se.

During the rebuttal period, the authors addressed almost every concern brought up by the reviewers, with some reviewers improving their score as a result, and ultimately ended with all positive reviews. I agree with the reviewers that this paper should be accepted and that the results contained within are of great interest to the NeurIPS community.
Decision: Accept (poster)