Title: The Curse of Depth in Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: curse of depth

LLMs

Pre-LN
Abstract: In this paper, we re-introduce the Curse of Depth, a concept that re-introduces, explains, and addresses the recent observation in modern Large Language Models (LLMs) where deeper layers are much less effective than expected. We first confirm the wide existence of this phenomenon across the most popular families of LLMs, such as Llama, Mistral, DeepSeek, and Qwen. Our analysis, theoretically and empirically, identifies that the underlying reason for the ineffectiveness of deep layers in LLMs is the widespread usage of Pre-Layer Normalization (Pre-LN). While Pre-LN stabilizes the training of Transformer LLMs, its output variance exponentially grows with the model depth, which undesirably causes the derivative of the deep Transformer blocks to be an identity matrix, and therefore barely contributes to the training. To resolve this training pitfall, we propose LayerNorm Scaling, which scales the variance of output of the layer normalization inversely by the square root of its depth. This simple modification mitigates the output variance explosion of deeper Transformer layers, improving their contribution. Our experimental results, spanning model sizes from 130M to 7B, demonstrate that \ours significantly enhances LLM pre-training performance compared to Pre-LN. Moreover, this improvement seamlessly carries over to supervised fine-tuning. All these gains can be attributed to the fact that LayerNorm Scaling enables deeper layers to contribute more effectively during training.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSeveral researchers have indicated that deep layers in LLMs are ineffective and have limited contribution to their accuracy. This paper analyzes the cause of this ineffectiveness and attributes it to pre-layer norm (i.e., placing layer norm or RMSNorm before attention and FFN). Then the paper proposes placing a simple scaling after each layer norm, and shows that it makes deeper layers more effective and increase overall accuracy of the model.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- insights backed by rigorous mathematics\n- hyperparameter-free\n- easy to implement\n- confirmed to be stable in training (unlike other approaches proposed in literature)\n- provided results training LLMs with various architectures, sizes, training budgets as well as ViTs\n- Compared with a wide range of initialization / layernorm modification techniques and showed superior accuracy\n\nWeaknesses:\n- Contribution is simple\n\nQUESTIONS:\n- Equation 5: I didn't understand how this equation is derived. a) What is the relationship between Pre-LN(x) and LN(x)? b) why do we have an identity, I, here?\n- In preliminary analysis, drop in SQuAD on BERT seems to be small, like mostly less than 1%, while drop in MMLU on other models seems to reach almost 50%. Wondering if we can verify with another task on BERT to ensure a significant variance in accuracy.\n- Line 185: Even though the authors cite the experimental setup from another paper, I believe readers would appreciate if authors provide a few more details like what training set is used.\n- Table 1: Please specify what evaluation data set is used to measure perplexity, and what is the context length when measuring it.\n- Paper would be strong if authors could train on an architecture with MoE (e.g., Qwen3 or DeepSeekv3)\n- Line 310: \"Admin introduces additional parameters to control residual dependencies, stabilizing Post-LN.\": Missing citation?\n- Equation 19: please define n right after the equation \n- I didn't understand how we derive the 3rd step in Equation 23\n- if possible, it would be great if authors could do an early exit analysis similar to Figure 2 of LayerSkip paper ( https://arxiv.org/abs/2404.16710 ) to compare a model trained with LN vs. LN Scaled\n\nLIMITATIONS:\nThe author did not mention any limitations. However, I can't seem to think of a limitation.\nMaybe one limitation is that auhtors did not test on a MoE model, or VLM model.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper identifies and analyzes a phenomenon the authors call the Curse of Depth(CoD) in Pre-LayerNorm (Pre-LN) transformers. Empirically, the top-most blocks of several mainstream LLMs (up to 40 layers in 13 B models) can be pruned with negligible loss, indicating that these layers contribute little to representation learning. The authors attribute CoD to an exploding activation variance along residual paths, which forces the layerwise Jacobian toward the identity and suppresses gradient flow. They prove upper- and lower-bounds on this variance growth, then propose a one-line fix LayerNorm Scaling (LNS), i.e., dividing each LayerNorm output by $1/\\sqrt{\\ell}$. LNS theoretically limits variance to polynomial growth, restores gradient magnitudes, and empirically improves perplexity, downstream accuracy, and the functional importance of deep layers in 130 M–1 B models.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths  \n1. **Clear problem identification**  \n   * Shows that upper transformer blocks add almost no value.  \n   * Targets the industry-standard *Pre-LayerNorm + Residual* architecture, making the diagnosis immediately relevant.  \n\n2. **Tight theory-experiment loop**  \n   * Theorizes exponential activation-variance growth and Jacobian collapse.  \n   * Logs training stats to confirm the variance explosion and gradient/pruning patterns.  \n   * Introduces LayerNorm Scaling ($1/\\sqrt{\\ell}$) and re-applies the same analysis to show variance drops to polynomial growth and deep layers regain utility.  \n\n3. **Minimal, broadly applicable fix**  \n   * One-line code change, no extra parameters.  \n   * Consistently lowers perplexity and boosts accuracy on 130 M–1 B models, with potential to extend to larger checkpoints.  \n\n---\n\n### Weaknesses  \n1. **Limited experimental scale**  \n   * Remedy is tested only up to 1 B (24 layers). Behavior on 70 B–110 B models (80–120 layers) is unverified; the $1/\\sqrt{\\ell}$ schedule may need retuning.  \n\n2. **Incomplete factor isolation**  \n   * All runs keep RMSNorm, so gains from the new scaling vs. RMSNorm stability aren’t disentangled.  \n   * No comparisons with alternatives like DeepNorm or residual α-scaling.  \n   * Mean-field assumptions (Gaussian, independent channels) are not empirically checked late in training.  \n\n3. **Practical trade-offs and reproducibility gaps**  \n   * No data on speed, memory, or energy—so efficiency benefits are unclear.  \n   * Key hyper-parameters (batch size, grad clipping, weight decay) are under-documented.  \n   * Evaluation omits code generation, dialogue, and reasoning tasks where deep context might matter most.\n\nQUESTIONS:\n1. **Down-stream Metrics on 7B**\n    \n    In Section 5.3, you report perplexity improvements for a 7B (32-layer) model, but no MMLU, ARC-e, or other accuracy numbers. Did you run those tests and, if so, could you share the results? If not, please clarify why only PPL was measured at that scale.\n    \n2. **Ultra-deep Applicability (≥ 80 layers)**\n    \n    The deepest setting evaluated for LayerNorm Scaling (LNS) is 32 layers. Have you analyzed variance curves, pruning sensitivity, and task accuracy on checkpoints in the 80–120-layer range (e.g., LLaMA-2-70 B)? Even a post-hoc fine-tune would help confirm whether the proposed fix remains effective in the truly “deep” regime.\n    \n3. **Tail-layer Importance**\n    \n    The upward “tail” in pruning loss suggests that the last few blocks may perform specialized functions (e.g., output-projection alignment). Do you foresee that a uniform $1/\\sqrt{\\ell}$ schedule could dampen such specialization, and have you measured any task where the very last layer dominates after LNS is applied?\n    \n4. **Resource Foot-print Quantification**\n    \n    One motivation for LNS is to “avoid wasting compute on dead layers.” Could you provide wall-clock time, GPU-memory, and energy usage comparisons between Pre-LN and LNS models at 130 M, 400 M, and 1 B scales?\n    \n5. **Exponent Search and Adaptive Scaling**\n    \n    Your experiments use a fixed $1/\\sqrt{\\ell}$ factor. Did you explore alternative exponents $1/l^\\beta (\\beta\\in[0.3, 0.7])$ or a run-time variance-adaptive schedule? Please report any trends in stability and accuracy.\n    \n6. **RMSNorm ablation and cross-norm mixes**\n    \n    All results keep RMSNorm active. Could you (i) disable RMSNorm and apply LayerNorm + LNS only; (ii) combine LNS with DeepNorm or residual α-scaling; and (iii) report variance and accuracy in each setting? This would isolate the true contribution of LNS versus RMSNorm.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the \"curse of depth,\" which is that later layers seem to contribute less than earlier layers. This is demonstrated by experiment, and a theoretical analysis shows that gradients have a constant upper bound as the depth of the network increases. The authors propose an extremely simple modification that scales the contribution of layer l by 1/sqrt(l). They repeat the theoretical analysis and show that the gradient grows linearly in the depth, and they repeat the experiments showing that the layers contribute more equally. Finally, they show through extensive experiments that their modification improves language model performance.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\nBoth the experimental and theoretical results are convincing, and the LNS modification is so simple that the paper makes a very compelling case that it should be adopted.\n\n# Weaknesses\n\nI found the theoretical argument around Lemma 1 and Theorem 1 difficult to follow. If the goal is to prevent the later layers from becoming identity-like -- that is, dominated by the residual connections -- it seems counterintuitive that the solution would be to scale down the part of each layer that is not the residual connection. Could you provide some explanation to resolve this paradox?\n\n# Minor comments\n\n- line 243: the reference to Figure 3 is not right. I couldn’t find what the right referent should be.\n\nQUESTIONS:\n- I found the theoretical argument around Lemma 1 and Theorem 1 difficult to follow. If the goal is to prevent the later layers from becoming identity-like -- that is, dominated by the residual connections -- it seems counterintuitive that the solution would be to scale down the part of each layer that is not the residual connection. Could you provide some explanation to resolve this paradox?\n\n- Since each layer norm has a scaling factor built into it (usually called $\\gamma$), why couldn't it learn the $\\sqrt{\\ell}$ factor automatically?\n\n- In Figure 2b, what’s the reason for the spike around layer 27-28?\n\n- In Eq (23), $\\sigma_W^2$ should be $\\sigma_W^4$ and $\\sigma_W$ should be $\\Sigma_W^2$ (right?)\n\nThe proof of Lemma 3, seems, to me, to skip a lot of steps. \n- How do you get from (32) and (33) to (34)? It would help to write out the Jacobian of LN. \n- How do you get equation (37)? The only justification is a citation to an entire textbook [44].\n\nLIMITATIONS:\nThe limitations section is relegated to the appendix but is adequate.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
In this paper, the authors attempt to understand the realisation that deep layers in the network are less effective than actually expected. They confirm the finding across various models and experiments. They then identify the reasons related to the usage of the pre-layer normalisation. They then propose a rather simple modification of LayerNorm scaling and show that this enhances the LLM's performance. 

The paper is generally well written and nice to read. I would like to kindly ask the authors a couple of questions: 

1. What about other factors that affect deeper layer expressivity? Like Attention saturation, Batch sizes and so forth. Do you think those do not matter? Does it not matter as much as the cause found in the paper? Would a holistic solution consider all those together? 

2. I find the analysis interesting, but can you isolate the effects of Mix-LN versus the other variant? Is it due to instability or poor parameter tuning? What could we say about this? 

3. Do the results show improvements for formal reasoning problems as well? Can this help us in coding and/or formal reasoning tasks? 

4. Can the authors please discuss and experiment with how their proposed method plays with "newer" tricks that we use today, like gradient checkpointing, flash attention or other memory optimisation and so forth? Would this amplify or negate what has been suggested? 

5. Can the authors apply their method retroactively to high-performing models and see if they benefit? I find it lacking. It would be better to understand the actual gain we would get if possible. 

All in all, i think this is a good paper but please help me understand the points above.

STRENGTHS AND WEAKNESSES:
Please see above

QUESTIONS:
Please see above

LIMITATIONS:
OK
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02