Title: Don't be lazy: CompleteP enables compute-efficient deep transformers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Deep learning

depth

transformer

LLM

compute-efficiency

parameterization

stability

hyperparameter transfer
Abstract: We study compute efficiency of LLM training when using different parameterizations, i.e., rules for adjusting model and optimizer hyperparameters (HPs) as model size changes. Some parameterizations fail to transfer optimal base HPs (such as learning rate) across changes in model depth, requiring practitioners to either re-tune these HPs as they scale up (expensive), or accept sub-optimal training when re-tuning is prohibitive. Even when they achieve HP transfer, we develop theory to show parameterizations may still exist in the lazy learning regime where layers learn only features close to their linearization, preventing effective use of depth and nonlinearity. Finally, we identify and adopt the parameterization we call CompleteP that achieves both depth-wise HP transfer and non-lazy learning in all layers. CompleteP enables a wider range of model width/depth ratios to remain compute-efficient, unlocking shapes better suited for different hardware settings and operational contexts. Moreover, CompleteP enables 12-34% compute efficiency improvements over the prior state-of-the-art. All experiments were run on Cerebras CS-3 systems. A minimal implementation is available at https://github.com/EleutherAI/nanoGPT-mup/tree/completep.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies compute-efficient hyper-parameter transfer for LLM models. A novel parameterization method called CompleteP is introduced, providing findings and guidance for efficient and reliable HP transfer. This method addresses the challenge of transferring optimal hyperparameters across varying model depths in deep transformers, which is crucial for efficient training and deployment of LLMs. Three major contributions include non-lazy depth-wise HP transfer, significant FLOPs saving when comparing with previous SOTA methods, and larger range of model width/depth ratios to remain compute-efficiency, making it suitable for various hardware settings and model deployment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis paper is well studied on the topic of efficient HP transfer, with in-depth analysis, extensive experiments and empirical results, providing practical guidance for LLM model training under different hardware requirements. The paper is well written. \n\n\nWeaknesses:\n\n1. Although the paper has in-depth study on the topic, the result is limited to pre-LN transformers. Whether the findings still hold for other architectures or optimizers remains unknown. And how the method scale to models beyond 1.5B needs to be further verified. This may limit the application of such method. \n2. Some results and plots are based on empirical fitting instead of real profiling on certain hardware. This may also hurt the accuracy and generality of the report.\n\nQUESTIONS:\n1. In Figure 1, the FLOP saving is calculated based on equation (48). Are there any other hardware related metrics, such as power/ energy saving or latency profiling can be reported to show the effectiveness the compute efficiency?  \n2. For learning rate transfer and initialization standard deviation transfer, are there more plots beyond the task of SlimPJ? Is the trend consistent across different validation tasks?\n3. Could you please provide more details on the Methodology section, for example, which model are you using, and what are the steps to implement the parameterization? \n4. For Table 2, do you have results for the case of N:L > 100?\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work builds on the maximal update parameterization (μP) framework by studying different values of the depth scaling parameter α in the residual connection formula: h^(ℓ+1) = h^ℓ + L^(-α) F^ℓ(h^ℓ).\n\nThe authors compare different parameterizations: Standard Parameterization (SP), μP, α = 0.5, and CompleteP (α = 1). Through extensive experiments on transformer language models ranging from 75M to 1.8B parameters trained on SlimPajama, they demonstrate that only CompleteP achieves reliable depth-wise hyperparameter transfer while maintaining training stability. Key empirical findings include: (1) Provides 11.8-34.4% FLOP savings over μP for deep models, and (2) Better accuracy (validation loss and zero shot results) at optimal points. (3) Wider range of depth to width ratios, enabling better choices across different HW platforms. \n\nThe authors also provide a theoretical justification of why setting α=1 works.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Comprehensive empirical evaluation: The paper provides extensive experiments across multiple model sizes, training settings, and evaluation metrics, including both upstream pretraining loss and downstream task performance. Studies also cover iso-tokens and compute optimal training (20 tokens per parameter) settings, accounting for different variations\n\n2. Theoretical grounding: The three desiderata provide a principled framework for understanding why CompleteP works, with the \"complete feature learning\" concept offering novel theoretical insight.\n\n\nWeakness:\n\n1. Adding α on top of μP to enable depth scaling has been covered by prior work. While this work, does consolidate discrepancies between various work and adopt it to transformers, but overall approach builds incrementally on existing μP and depth scaling work.\n        --- NOTE: I did not evaluate the theoretical justification provided in Section 6. The theoretical analysis falls beyond my expertise. So any novelty or originality there would have been missed by me.\n\n2. Largest scale of the model is 1.8B. Lack of validation on truly large-scale models (10B+ parameters) weakens the claims a bit\n\n3. While compute optimal settings provide good insights, modern day small LLMs are trained far beyond the 20TPP number. Impact of data can be seen from Figure 2 bottom to Figure 3. This makes the claim a little harder to verify.\n\nQUESTIONS:\n1. Modern day LLMs are moving towards longer context, MoEs, Group Query Attention, RMSNorm and other enhancements to architecture. Would these analysis persists for these architectures too?\n\n2. Can the authors provide guidance on hyperparameter sensitivity analysis for practitioners for completeP?\n\n3. Will the benefits persist under different training regimes (e.g., longer training, different batch size schedules)?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies depth-dependent parameterizations for hyperparameter transfer when training deeper models. The authors introduce the parameterization completeP, which they show guarantees non-lazy learning at each layer as models become deeper. They demonstrate that this parameterization enables effective hyperparameter transfer as model depth increases and improves performance relative to standard parameterization, μP, and another depth-dependent parameterization (i.e., α=0.5). They additionally demonstrate that completeP allows for a wider range of compute-efficient width/depth ratios. Their results are shown on large language models trained over a sweep of model sizes using large datasets.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n- Tackles an extremely important problem for scaling large language models and improving the performance of state-of-the-art systems.\n- Presents an impressive display of empirical testing.\n- High-quality experiments, well written, and significant work overall.\n\n# Weaknesses\n\n- This work is not particularly original; it primarily extends prior ideas (references 6–8), but does an excellent job empirically testing and validating these results to demonstrate that completeP is the best setting. That said, I think this work could do a better job distinguishing its contributions from prior works (see my comments below)\n\n- Despite the impressive empirical sweep, there are still limitations. For example, as discussed in lines 79–88, there exist other layer-dependent rescalings that are not considered in this empirical study, and the results are only performed on the single dataset SlimPajama. I suggest moving the discussion of these limitations, such as the one you include in Appendix B, up into the main conclusion section.\n\nQUESTIONS:\n# Comments\n\n- I don’t understand what you are plotting in Figure 1(c) — the colored areas represent the “range of efficient N:L,” but how do you define this? I suggest you describe it clearly in the caption or within the figure itself.\n\n- Lines 35–37 don’t make sense to me as written. You introduce that the depth-dependent rescaling factor is governed by a single parameter α ∈ [0.5, 1], define α=1 as your algorithm completeP, and then state “Since α=0.5 and α=1 are the two most promising potential candidates for depth scaling...” but it’s unclear why α=0.5 is included. Has α=0.5 been previously defined in the literature? I think adding a sentence or two explaining what prior work has done with α=0.5 and how your work extends this will make the introduction clearer.\n\n- Since you continuously compare to the α=0.5 setting from the work “Feature Learning in Infinite-Depth Neural Networks,” I would suggest giving this setting a name.\n\n- I think you should also briefly discuss what would happen for α ∈ (0.5, 1.0), even though I agree with your logic that the boundary cases are the most important.\n\n- In Kaplan et al., they use a version of the WebText dataset, while your work uses SlimPajama, correct? It’s worth explicitly highlighting this difference.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nHyperparameter (HP) tuning is expensive at scale. Prior work like µP has shown how to enable zero-shot HP transfer when scaling model width. This work extends that line of research by providing a recipe for scaling HPs—such as bias learning rates, weight decay, Adam epsilon, and LayerNorm parameters, enabling a zero-shot HP transfer from shallow to deep networks. They show up to 34% FLOP savings.\n\nSTRENGTHS AND WEAKNESSES:\n(+) practical recipe for training deep transformers\n\n(+) enabling zero-shot HP transfer, reducing HP tuning budget\n\n(+) unlocking model shapes better suited for different hardware\n\n(+) simple implementation, require only a few lines of code\n\n\t\n(-) lack of direct theoretical comparison to prior work\n\n(-) selective hyperparameter focus, the method only address a subset of HPs\n\n(-) unclear if proposed approach generalize beyond transformer and Adam optimizer\n\nQUESTIONS:\n1. The authors demonstrate both empirically and theoretically that setting the residual scaling α = 1 enables full, non-lazy feature learning across depth. How does this finding relate to, or differ from, the s=1 regime described in the prior theoretical work Meta-Principled Family of Hyperparameter Scaling Strategies, ref. [41]? In what way is Desideratum 3 novel beyond what is implied by the s=1 maximal-update regime?\n\n2. There are many hyperparameters that influence training dynamics as model size increases. This work focuses on scaling a specific subset: bias learning rates, weight decay, Adam epsilon, and LayerNorm parameters. Could the authors elaborate on why these were prioritized over others—such as batch size, learning rate warmup schedules, gradient clipping, or optimizer momentum—and whether the proposed method is sensitive to those remaining hyperparameters?\n\n3. The work is developed and evaluated using pre-LayerNorm Transformer architectures, which are commonly used for deep models due to their favorable gradient dynamics. Could the authors comment on whether CompleteP generalizes to alternative normalization schemes, such as post-LN or sandwich-LN architectures? Would any modifications to the scaling rules (e.g., for residuals or LayerNorm) be required in those settings?\n\n4. The paper demonstrates the proposed scaling strategy using AdamW, but does not explore its interaction with other optimizers such as SGD, LAMB, or Lion. Could the authors comment on whether CompleteP generalizes to these optimizers, and whether additional tuning or scaling adjustments would be required?\n\n5. In Figure 4 (d), it appears that the scaling law fit is based on only three data points. Could the authors clarify whether this is the case, and if so, how robust the fit is to such limited data? Additionally, for Figure 4(f), how are the shaded region estimated?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}