The Curse of Depth in Large Language Models
Authors: Wenfang Sun
Xinyuan Song
Pengxiang Li
Lu Yin
Yefeng Zheng
Shiwei Liu
Venue: NeurIPS 2025 poster ()
Keywords: curse of depth
LLMs
Pre-LN
Primary area: deep_learning
TL;DR: In this paper, we introduce the Curse of Depth, a concept that re-introduces, explains, and addresses the recent observation in modern Large Language Models (LLMs) where deeper layers are much less effective than expected.

Abstract: In this paper, we re-introduce the Curse of Depth, a concept that re-introduces, explains, and addresses the recent observation in modern Large Language Models (LLMs) where deeper layers are much less effective than expected. We first confirm the wide existence of this phenomenon across the most popular families of LLMs, such as Llama, Mistral, DeepSeek, and Qwen. Our analysis, theoretically and empirically, identifies that the underlying reason for the ineffectiveness of deep layers in LLMs is the widespread usage of Pre-Layer Normalization (Pre-LN). While Pre-LN stabilizes the training of Transformer LLMs, its output variance exponentially grows with the model depth, which undesirably causes the derivative of the deep Transformer blocks to be an identity matrix, and therefore barely contributes to the training. To resolve this training pitfall, we propose LayerNorm Scaling, which scales the variance of output of the layer normalization inversely by the square root of its depth. This simple modification mitigates the output variance explosion of deeper Transformer layers, improving their contribution. Our experimental results, spanning model sizes from 130M to 7B, demonstrate that \ours significantly enhances LLM pre-training performance compared to Pre-LN. Moreover, this improvement seamlessly carries over to supervised fine-tuning. All these gains can be attributed to the fact that LayerNorm Scaling enables deeper layers to contribute more effectively during training.

The Curse of Depth in Large Language Models
Wenfang Sun
Xinyuan Song" Pengxiang Li
Lu Yin' Yefeng Zheng' Shiwei Liu
WWetllake University *University of Surrey
Emory University
DDalian University of Technology
Univeesityy of Oxford
Abstract
In this paper, we introduce the Curse of Depth, concept that highlights, explains, and addresses the recent observation in modern Large Language Models (LLMs) where nearly half of the layers are less effective than expected. We first confirm the wide existence of this phenomenon across the most popular families of LLMs such as Llama, Mistral, DeepSeek, and Qwen. Our analysis, theoretically and empirically. identifies that the underlying reason for the ineffectiveness of deep layers in LLMS is the widespread usage of Pre-Layer Normalization (Pre-LNN). While Pre-LN stabilizes the training of Transformer LLMs, its output variance exponentially grows with the model depth, which undesirably causes the deriva- tive of the deep Transformer blocks to be identity matrix, and therefore barely contributes to the training. To resolve this training pitfall, we propose LayerNorm Scaling (LNS), which scales the variance of output of the layer normalization inversely by the square root of depth. This simple modification mitigates the output variance explosion of deeper Transformer layers, improving their contribu- tion. Across a wide range of model sizes (130M to 7B), our experiments show that LNS consistently outperforms previous normalization and scaling techniques in enhancing LLM pre-training performance. Moreover, this improvement seamlessly carries over to supervised fine-tuning. All these gains can be attributed to the fact that LayerNorm Scaling enables deeper layers to contribute more effectively during training. Our code available at LayerNorm-SSaalnn
Introduction
Recent studies reveal that the deeper layers (Transformer blocks) in modern LLMs tend be less effective than the earlier ones [54] 16 32 26). On the one hand, this interesting observation provides an effective indicator for LLM compression. For instance, can compress deeper layers significantly more [54] 29 14 to achieve high compression ratios. Even more aggressively, entire deep layers can
be pruned completely without compromising performance
33 39
On the other hand, having many layers ineffective is undesirable as modern LLMs are extremely resource-intensive train, often requiring thousands of GPUs trained for multiple months, let alone the labor used for data curation and administration [ 44 Ideally, we want all layers model to be well-trained, with sufficient diversity in features from layer layer, to maximize the utility of resources [26]. The existence of ill-trained layers suggests that there must be something off with current LLM paradigms. Addressing such limitations pressing need for the community avoid the waste of valuable resources, as new versions of LLMs are usually trained with their previous computing paradigm which results in ineffective layers.
To seck the immediate attention of the community, we re-introduce the concept of the Curse of Depth (CoD) systematically present the phenomenon of ineffective deep layers in various LLM families,
"Equal contribution.
Corresponding author: shivei. liu@maths. ox. ac.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Ligeo
Liers
Liper
Ligers
Marr
Upe
Pre4.N
Pre e Scaled Initaaliantion
LassrNormn Scoling
175
17%
Aweraee Vartaoom (loc)
1.50
100
73
Avetuug
5
22
Update Sicy
Upalate
Updato Stop
Figure 1: Layerwise output variance. This figure compares the output variance across various layers for different setups: (1) Pre-LN; (2) Pre-LN with Scaled Initialization [38] [35;; and (3) LayerNorm Scaling. The experiments are conducted on the LLaM-130M model trained for 10,000 steps. The proposed LayerNorm Scaling effectively controls the variance across layers.
to identify the underlying reason behind it, and to rectify by proposing LayerNorm Scaling, We first state the Curse of Depth below.
The Curse of Depth. The Curse of Depth refers to the observed phenomenon where deeper layers in modern LLMs contribute significantly less (but not nothing) to learning and representation compared to earlier layers. These deeper layers often exhibit remarkable robustness to pruning and perturbations, implying they fail to perform meaningful transformations. This behavior prevents these layers from effectively contributing to training and representation learning, resulting resource inefficiency.
Empirical Evidence of CoD. To demonstrate that CoD is a common phenomenon across promi- nent LLM families, we perform layer pruning experiments on Qwen3, LLaMA2, and DeepSeek. Specifically, we prune one layer at time, without any fine-tuning, and directly evaluate the resulting pruned models the MMLU benchmark (17), as shown in Figure 2 Key findings: (1) Most models, including the latest Qwen3, exhibit surprising resilience to the removal of deeper layers; (2) The number of layers that can be removed without causing significant performance drop increases with model size; (3) Representations in deeper layers are significantly more similar to each other than those in earlier layers.
Identifying the Root Cause of CoD. We theoretically and empirically identify the root cause of CoD as the of Pre-Layer Normalization (Pre-LN) [3 8) which normalizes layer inputs before applying the main computations, such as attention or feedforward operations, rather than after. Specifically, while stabilizing training, observe that the output variance of Pre-LN accumulates significantly with layer depth (see Appendix F causing the derivatives of deep Pre-LN layers to approach an identity matrix. This behavior prevents these layers from introducing meaningful transformations, leading to diminished representation learning.
Mitigating CoD through LayerNorm Scaling. the output of Layer Normalization by the square root the depth 1
We propose LayerNorm Scaling (LNS), which scales
vI LayerNorm Scaling effectively scales down the output variance across layers of Pre-LN. LNS consistently delivers better pre-training performance than Pre-LN with various model sizes from 130M 7B. Unlike previous LayerNorm variants 26 28 LayerNorm Scaling is simple to implement, requires no hyperparameter tuning, and introduces no additional parameters during training Furthermore, we show that the model pre- trained with LayerNorm Scaling achieves better performance on downstream tasks in self supervueee fine-tuning, all thanks to the more diverse feature representations learned deep layers.
2 Empirical Evidence of the Curse of Depth
To empirically analyze the impact of layer normalization on the Curse of Depth in LLMs, we conduct a series of evaluations inspired by Li al. (26), to compare Pre-LN and Post-LN models.
2.1
Experimental Setup
Methodology: We evaluate Pre-LN and Post-LN models by assessing the impact of layer pruning at different depths. Our hypothesis is that Pre-LN models exhibit diminishing effectiveness in deeper layers, whereas Post-LN models have less effective early layers.
'e found that combining LNS with Scaled Initialization diminishes the effectiveness of LNS. Therefore. recommend removing the latter when applying LNS.

---

Figure 2:
Top: Performance drop after removing single layer without fine-tuning, Bottom:
Angular distance from the initial layer é (x-axis) and its subsequent n layer (y-axis). The results demonstrate that in Pre-LN LLMs, deeper layers produce highly similar representations to their adjacent layers, and their removal results in minimal performance degradation. In contrast, Post-LN models show the opposite trend: deep layers contribute more substantially model performance. See Appendix Dioor controlled experiments, where we train custom LLMs from scratch to isolate Layer Normalization as the sole varying factor.
Models: To verify this, we empirically quantify the contribution of individual layers to overall model performance across a diverse set of LLMs, including Qwen3 [42). LLaMA2 [44], DeepSeek [6]. and BERT-Large [IO]. These models were chosen to ensure architectural and application diversity. BERT-LLarge represents a Post-LN model, whereas the rest are Fre-LN-based. This selection enables
a comprehensive evaluation of the effects of layer normalization across varying architectures and model scales.
Evaluation Metrie: To empirically assess the impact of deeper layers in LLMs, we adopt two metrics, Performance Drop and Angular Distance, inspired by Gromov et al. [16. Li et al. (26).
Performance Drop (P(") quantifies the importance of each layer by measuring the performance change after its removal. A smaller (!() indicates that the pruned layer contributes less to the model's 's overall performance. For BERT-Large, we evaluate using the SQuAD v1.1 dataset [36], while for other models, we use MMLU [77. standard benchmark for multi-task language understanding.
Angular Distance d(x", xf+") quantifies the directional change between the input representations at layer and layer + n on a neutral pre-training dataset. Formally, given a token T, let aff. and -r4"" denote its input to layers l and l. + n, respectively. The angular distance is defined as:
1
she 1 1a 12
d(x', x (n)
arccos
where || denotes the ---om.. To reduce variance, report the average distance over 256K tokens sampled from the C4 dataset. Smaller values of d(x",x(44) indicate higher similarity between the two representations, suggesting limited transformation. Such layers can be considered redundant, as their removal minimally impacts the model's internal representations. Ideally, each layer should introduce meaningful representational shifts to fully leverage the model's 's capacity [53 [6].
Experimental Results: (1) Pruning deep layers in Pre-LN LLMs leads to negligible, and sometimes even positive, changes performance, as shown in Figure Top. Specifically, Figure (b)-(d) reveals that a wide range of deeper layers-parttiulart, beyond the 18th-can be pruned with minimal impact on performance. This indicates that deep layers in Pre-LN architectures contribute little the model's 's overall effectiveness. In contrast, Figure 2 (a) shows that pruning deep layers in BERT-Large (a Post-LN model) leads to substantial drop in accuracy, while pruning early layers has a relatively minor effect. (2) Pre-LN models exhibit decreasing angular distance in deeper layers, indicating highly similar representations, as shown in Figure2 DeepSeek-7B falls below 0.2 after the 18th layer. Qwen3-8B demonstrates higher similarity, with nearly half of its layers exhibiting distances below 0.2 from their preceding layers. In LLaMA2.
For instance, the angular distance in
13B, the angular distance approaches zero across the final one- one-third of the network. These similar
3

---

representations align with the pruning results in Figures 2 (b)-(d) where pruning later layers has little effect, while pruning early layers significantly degrades performance.
3 Analysis of the Curse of Depth
Preliminaries. This paper primarily focuses on Pre-LN Transformer 3 g. Let ze E IR be the input vector the f-th layer of Transformer, where denotes the feature dimension of each layer. For simplicity, we assume all layers to have the same dimension d. The layer output is calculated as follows:
= 2t+1 = 2/y FFN(LN((()),
(2)
2/2 xz Attn(..N(zz)),,
where LN denotes the layer normalization function. In addition, the feed-forward network (FFN) and the multi-head self-attention (Attn) sub-layers are defined as follows:
FFN(x) WWF(W
A1)n(z)) = Wo
(z),..., head, (x))),
(WQuz) (W (WvxX)",
head, (x) softmax
Vdhead
where F an activation function, concat concatenates input vectors, softmax applies the softmax function, and W1 E Id W2 E IRdddaa WQ: E Redaoad xd WK: E Rhda x Wvs E R.head x and Wo E Rdxd are parameter matrices, and deFN and duead are the internal dimensions of FFN and multi-head self-attention sub-layers, respectively. X Rdxx,, where the input sequence length.
The derivatives of Pre-In Transformers are:
ÖPre-LN(xx) -I+ (f())())) ØLN(x) aLN(x)
Ox
where f here represents either the multi-head attention function or the FFN function. the term @f(LN(=)) aLN(æ) becomes too small, the Pre-IN layer OPre-LN(a)) behaves like an identity map. Our 8z
8LN(z) de
main objective is to prevent identity map behavior for very deep Transformer networks. The first step
in this process is to compute the variance of of vector Zt.
3.1 Pre-LN Transformers
Let Ie and x'g denote the input and intermediate vectors of the E-th layer. Moreover, let We denote the model parameter matrix at the E-th layer. We assume that, for all layers, De, x'p and We follow normal and independent distributions with mean O.
Assumption 1.
Lemma 1. Let o² and o²,, denote the variances of x' and Vs, respectively. These two variances exhibit the same overall growth trend, which is:
2 =220,
II 1+
(6)
where the growth of 0² is sub-exponential, as shown by the following bounds:
(LL) s o2, I ((cxp(L)).
Here, the notation θ means: if f()) (g(x)), then there exist constants C1 C2 such that Gilg(=)] < [f()) C2lg(=)) - The lower bound O s of, indicates that o grows least linearly, while the upper bound o2 O(exp(L)) implies that its growth does not exceed an exponential function of L.
Based on Assumption I and the work of 41, we obtain the following:

---

Figure 3: Visualization of the LayerNorm Jacobian matrices across different layers of pre-trained LLaMA2-7B model. Each heatmap shows the token-averaged Jacobian at a specific layer. As depth increases, the Jacobians exhibit pronounced diagonal dominance with vanishing off diagonal entries, indicating that deep LayerNorm blocks increasingly approximate identity mappings.
Theorem 1. For a Pre-LN Transformer with L layers, using Equations and (3) the partial
derivative day
dug, can be written as:
Oyr. 0r1
L-1
Oye Ox't Oz'
II
(8)
0xl
8=1
OuL The Euclidean norm of de:
is given by:
dyL Oz
1
144-41 A+
-+
1
(9)
2
1=1
One
T
where and are constants for the Transformer network. Then the upper bound for this norm is given follows: when oz grows exponentially, (i.e., at its upper bound), we have:
Tyz
2 exp((),
$ S,,
(10)
0z1
where the gradient norm converges to a constant M. Conversely, when o², grows linearly (i.e., at its lower bound), we have
dyL Ox 2
2
l
< O((),
(11)
which means that the gradient norm grows linearly in L.
The detailed description of A and B, well the complete proof, are provided in Appendix A.2 we observe that when the variance grows exponentially, as the number of layers bounded above by a fixed constant M. This result implies that even
From Theorem
- xC, the norm
an infinitely deep Transformer remains stable, and by the Weierstrass Theorem, the network is guaranteed to converge. Consequently, this implies that for very large L, deeper layers behave nearly identity map from x; to yt. thereby limiting the model's expressivity and hindering its ability to learn meaningful transformations, This phenomenon is empirically illustrated in Figure 3 where we visualize the LayerNorm Jacobians of pre-trained LLaMA2-7B model across depth, revealing a clear collapse toward identity mappings in deeper layers. This outcome is undesirable, therefore,
we would instead prefer the variance to increase more gradually- e.g., linearly- so that 0yy. axm
exhibits linear growth. This observation highlights the necessity of appropriate variance control mechanisms, such as scaling strategies, to prevent excessive identity mappings and enhance network depth utilization.
LayerNorm Scaling (LNS)
To mitigate the abovementioned issue, we propose LayerNorm Scaling, a simple yet effective normalization strategy. The core idea of LayerNorm Scaling is to control the exponential growth of output variance in Pre-LN by scaling the normalized outputs according layer depth. Specifically,
5

---

Model Parameter Scaling
Seating
34-
Attention
Soading
24--
[4] Pre-N
(g) Layerform Scaling
10'
10th
12
(log
Figure 4: Left: Schematic diagrams of (a) Pre-LN and (b) LayerNorm Scaling. LayerNorm Scaling applies scaling factor inversely proportional the square root of the layer index t, preventing excessive variance growth. Right: Language modeling loss of scaling up parameter count up to 7B. All models trained for 20B tokens using OL [15]
we apply a scaling factor inversely proportional the square root of the layer index to scale down the output of LN layers, enhancing the contribution of deeper Transformer layers during training. LayerNorm Scaling is illustrated the left part of Figure
Formally, for a Transformer model with L layers, the output of Layer Normalization in each layer scaled by a factor of 1. Let (() denote the input to Layer Normalization at layer The modified Vi'
output is computed as:
h (8) LayerNorm(h (0) 1
(12)
where E (1,2..... ........... L}. a key limitation of Pre-LN. Unlike Mix-LN, which stabilizes gradients deeper layers but suffers from training instability caused by Post-LN 34 47), LayerNorm Scaling preserves the stability advantages of Pre-LN while enhancing the contribution of deeper layers to representation learning. Applying LayerNorm Scaling leads notable reduction of layerwise output variance shown in Figure resulting in a lower training loss. Morcover, compared with previous LayerNorm variants LayerNorm Scaling is hyperparameter--iie free, casy to implement, and does not introduce additional learnable parameters, making it computationally efficient and readily applicable to existing Transformer architectures. Our theoretical analysis in Appendix B shows that LNS slows down the variance growth from exponential to at most quadratic with depth, enabling more effective learning of deep layers.
This scaling prevents excessive variance growth with depth, addressing
226 28
Experiments
5.1 LLM Pre-training
To evaluate the effectiveness of LayerNorm Scaling, we follow the experimental setup of Li et al. using the identical model configurations and training conditions to compare LNS with widely used normalization techniques, including Post-LN (34), DecpNorm [47). and Pre-LN [8]. In line with Lialin et al. 27 and Zhao et al. [58), we conduct experiments using LLaMA- basee architectures with model sizes of 130M, 250M, 350M, and IB parameters.
(26,,
Table 1: Perplexity (() comparison of various layer normalization methods.
LLaMA-130M
LLaMA-250M
LLaMA-350M
LLaMA-1B
Training Tokens
2.2B
3.9B
6.0B
Post-I.N 2
26.95
1409.79
1368.33
1390.75
DeepNorm 47
27.17
22.77
1362.59
1409.08
Mix-LN (26
26.07
21.39
1363.21
1414.78
Pre- B
26.73
21.92
19.58
17.02
Pre-LN LayerNorm Scaling
25.76
20.35
18.20
15.71
The architecture incorporates RMSNorm 337 and SwiGLU activations [56],
which are applied
consistently across all model sizes and normalization methods. For optimization, we use the Adam

---

Table 2: Fine-tuning performance (1) of LLaMA with various layer normalizations.
Method
MMLU
BoolQ
ARC-e
PIQA
Hellaswag
OBQA
Winogrande
Average
1.MMA-250M
Post-LN DeepNorm 37
22.95
37.83
26.94
52.72
26.17
11.60
49.56
32.54
23.60
37.86
36.62
61.10
25.69
15.00
49.57
35.63
Mix-LN 20
26.53
56.12
41.68
66.34
30.16
18.00
30.56
41.34
Pre-IN
24.93
38.35
40.15
63.55
26.34
16.20
49.01
36.93
+ LayerNoem Scaling
27.08
58.17
32.81
43.14
LLaMA-1B
Post-LN
22.95
37.82
25.08
49.51
25.04
13.80
49.57
31.96
DeepNorm
23.35
37.83
27.06
52.94
26.19
11.80
49.49
32.67
Mix-LN 26
23.19
37.83
25.08
49.51
25.04
11.80
49.57
31.72
Pre LN
26.54
62.20
45.70
67.79
30,96
17.40
50.51
43.01
Pre LN + LayerNorm Scaling
28.69
61.80
48.85
67.92
33.94
18.60
54.30
44.87
optimizer [2] and adopt size-specific learning rates: 10-3 for models up to 350M parameters, and 5 x 10 for the 1B parameter model. All models share the same architecture, hyperparameters, and training schedule, with the only difference being the choice of normalization method. Unlike Mix-LN (26), which introduces an additional hyperparameter a manually set to 0.25, LayerNorm Scaling requires no extra hyperparameters, making simpler implement. Table T shows that LayerNorm Scaling consistently outperforms other normalization methods across different model sizes. While DeepNorm performs comparably to Pre-LN on smaller models, struggles with larger architectures like LLaMA-IB IB, showing signs of instability and divergence in loss values. Similarly, Mix-LN outperforms Pre-LN in smaller models but faces convergence issues with LLaMA-350M indicating its sensitivity to architecture design and hyperparameter tuning due to the introduction of Post-LN. Notably, Mix-LN was originally evaluated on LLaMA-1B with 50K steps (26), while our setting extends training to 100K steps, where Mix-LN fails to converge, highlighting its instability in large-scale settings caused by the usage Post-LN.
In contrast, LayerNorm Scaling solves the Curse of Depth without compromising the training stability. LayerNorm Scaling achieves the lowest perplexity across all tested model sizes, showing stable performance improvements over existing methods. For instance, on LLaMA-130M and LLaMA-1B, LayerNorm Scaling reduces perplexity by 0.97 and 1.31, respectively, compared to Pre-LN. Notably, LayerNorm Scaling maintains stable training dynamics for LLaMA-IB, a model size where Mix-LN fails to converge. These findings demonstrate that LayerNorm Scaling provides robust and computationally efficient normalization strategy, enhancing large-scale training of language models without additional implementation complexity.
5.2 Supervised Fine-tuning
To verifyy whether the gains in pre-training can be translated to the stage of post-training, perform SFT with the models obtained from Section 5.1 on the Commonsense 70KK dataset downstream tasks. We adopt the same fine-tuning configurations used in Li et al. 226] The results, presented Table 2 demonstrate that LayerNorm Scaling consistently surpasses other normalization techniques in all evaluated datasets. For the LLaMA-250M model, LayerNorm Scaling improves average performance by 1.80% and achieves a 3.56% gain on ARC-e compared to Mix-LN. Similar trends are observed with the LLaMA-1B model, where LayerNorm Scaling outperforms Pre-LN, Post-LNN, Mix-LN, and DeepNorm on seven out of eight tasks, with an average gain of 1.86% over the best baseline. These results confirm that LayerNorm Scaling enhances generalization on diverse downstream tasks by improving the representation quality of deep layers.
[18 across eight
5.3 Scaling Up Training
5.3.1
OLMo
Model Size Scaling. To further assess the scalability and robustness of LNS, we conduct additional experiments using the OLMo repository [15], scaling training across model sizes of 60M, 150M, 300M, IB, and 7B parameters. All models are trained on a fixed 20B-token budget to ensure comparability. These experiments are designed to evaluate whether the performance gains observed with LNS in smaller-scale settings extend to more challenging and state-of-the-ar art LLM training regimes. As shown in Figure LNS consistently and substantially outperforms the standard Pre-LN
7

---

baseline across all model sizes. Remarkably, for the 7B model, LNS reduces the final loss from 2.69 2.50. These results underscore the scalability LNS and its effectiveness in large-scale pre-training scenarios.
Comparison of Training Loss Curves for OLMo-78
Pre-LN
LayerNorm Scalig
3.2
33.0 3.0
2.8
2.6
5000
10600
15000
20000
25000
20000
35000
Step
Figure 5: Training loss of OLMo-7B with Pre-LN and LNS.
Loss Curve. F the training loss curves of 7B models trained with Pre-LN and LNS. While LayerNorm Scaling exhibits slightly slower convergence at the early stages of training, consistently outperforms Pre-LN as training progresses, ultimately achieving a substantial loss gap. We attribute this to the uncontrolled accumulation of output variance in Pre-LN,, which amplifies with depth and training steps, ultimately impairing the effective learning of deeper layers. In contrast, LNS mitigates this issue by scaling down the output variance in proportion depth, thereby enabling more stable and effective training across all layers during training.
Beating OLMo's Scaled Initialization. OLMo adopts the scaled initialization proposed in Zhang et al. (57 and used by Mehta et al. [11, which scales input projections by 1/v dmodell and output projections by 1/v2- dmmddd -7 at every layer. This method is designed to enhance training stability and to scale down variance initialization. To evaluate the effectiveness of LNS, compare against this state-of-the-art initialization by training OLMo- IB on 20B tokens. As shown Table 3 LNS achieves consistently lower training loss, indicating that it may offer more effective alternative for large-scale L.M training.
Table 3: Comparison with OLMo's Scaled Initialization.
Method
Model
# Tokens
Training Loss
Perplexity
OLMo's Scaled Initialization LayerNorm Scaling
OLMo-
20B 20B
2.96
19.30
OLMo-
2.85
17.28
5.4 LNS Effectively Scales Down Output Variance
As LNS is proposed to reduce output variance, empirically validate this claim during the pre- training of LLMs. We compare the layerwise output variance of three configurations: (1) the standard Pre-LN 2. (2) Pre-LN with Scaled Initialization (8 351, which scales the initialization of the feedforward layers' weights Wy and W2 by /ZL' where L is the total number of Transformer layers,
and (3) Pre-LN with LNS. The average output variance across layers is shown in Figure For both vanilla Pre-LN and Scaled Initialization, the output variance in shallow layers (blue) remains relatively stable throughout training, while variance in deeper layers (red) grows substantially after 2K iterations, reaching up to 175 in the final layer. Since Scaled Initialization only operates at initialization, is insufficient to constrain output variance during training. In contrast, LNS consistently suppresses the growth of output variance deeper layers, capping it at approximately 25.
5.5 LNS Enhances the Effectiveness of Deep Layers
Furthermore, to assess whether LNS enhances the effectiveness of deeper layers by promoting more diverse feature representations, we analyze the layerwise performance drop and the angular distance of LNS, as shown in Figure Compared to Pre-LN, the performance degradation in LayerNorm Scaling is more uniformly distributed across layers, indicating more balanced contribution from
8

---

100
-5. Layer
-150
Figure 6: Left:: Performance drop of layer pruning on LLaMA-130M Right: The angular distance between representations of subsequent layers is shown. LayerNorm Scaling enables deep layers to make a meaningful contribution to the model.
each layer. Notably, pruning the deeper layers of LNS results in a more significant accuracy drop. suggesting these layers play a more critical role in task performance. Additionally, features learned under LNS exhibit greater distinction: most layers show substantial angular distance. exceeding 0.6, from their adjacent layers, indicating more diverse representations. In sharp contrast, the layerwise angular distance in Pre-LN remains significantly lower and progressively decreases with depth, suggesting reduced feature diversity. We provide studies CoD in Vision Transformers and Vision Language Models Appendix H and Appendix G respectively.
6 Ablation Study
Comparing Against Other Scaling Methods. We first compare LNS with previous scaling ap- proaches, including (1) Scaled Initialization [38 (35, which scales the initialization of Wo and W₂ by the overall depth 1/v2L; (2) Depth-Scaled Initialization [57] scales the initialization of weight matrices by the current depth 1/ (3) Skiplnit ( introduces learnable parameter after FFN/Att layers, initialized as L/V: (4) LayerScale [43] applies per-channel weighting using diagonal matrix, diag(A (aigg(A,...... Aa), where each weight X; is initialized to a small value (e.g., A, = =). Table 4 presents the results of LLaMA-130M and LLaMA-250M.
First, we observe that methods involving learnable parameters, such as LayerScale and SkipInit, consistently degrade performance in LLMs. Among initialization-but based techniques, larger scaling factor proves beneficial: Scaled Initialization yields lower perplexity compared to Depth-Scaled Initialization. Notably, LNS achieves the best overall performance, underscoring the advantage of applying scaling dynamically during training. Interestingly, combining LNS with Scaled Initializa- tion results in worse performance than using LNS alone, highlighting the importance of removing conflicting initialization strategies prior to adopting LNS.
Table 4: Comparing LNS against other scaling methods. Perplexity (L) is reported.
I.L.MA-130
LLAMA-250M
Training Tokens
2.2B
3.9B
Pre-LN
26.73
21.92
+ LayerScale 4
27.93
23.45
+ Skiplnit [
27.41
22.29
+ Depth-Scaled Initialization (57
26.95
21.50
+ Scaled Initialization [38]
26.04
20.98
+ LayerNorm Scaling
25.76
20.35
LayerNorm Scaling Scaled Initialization
25.80
20.79
Comparison with Other Layer Normalization. In addition, we conducted comparisons using LLaMA-130M to evaluate LayerNorm Scaling against recently proposed normalization methods, including Admin 228, Sandwich-LN [T], Group-LN 50 30,, and Mix-LN [26]. Table 5 shows that Admin and Group-LN degrade performance. Sandwich-LN slightly outperforms Pre-LN. Both Mix-LN and LayerNorm Scaling improve over Pre-LN by good margins. However, Mix- fails to reduce perplexity under 26, falling short of LayerNorm Scaling and suffers from instability in large-scale scenarios as shown in Table
Table 5: Comparison against other normalization methods on LLaMA- 130M. All methods use the identical configurations. Perplexity (L) is reported.
Pre-LN
Admin
Group-LN
Sandwich-LN
Mix-LN
LayerNorm Sculing
26.73
27.91
28.01
26.51
26.07
25.76

---

Effect of Positions of LNS. The results in 6 that inserting the scaling factor at different points can have considerable influence on the model's 's performance. Placing after the residual connection ("After Residual") leads to perplexity of 1358. which indicates training divergence. In contrast, LNS incorporates the scaling factor after LN achieving the best perplexity of 25.76, surpassing both the baseline Pre-LN setting (26.73) and other placements. This suggests that modifying the LayerNorm to include the scaling factor can enhance training stability and final performance for this model configuration.
Table 6: Comparison against different scaling factor position LLaMA-130M.
Pre-LNN
Before LN
After Atta/FFN
After Residual
LNS Only After Atin
LNS Only After FFN
Ours (After LN
26.73
26.97
26.53
1358.11
26.89
26.43
25.76
7 Related Work
Ineffectiveness of Deeper Layers in Transformers, The ineffectiveness of deep layers in LLMs has been previously reported. Yin et al. [54] found that deeper layers of LLMs can tolerate significantly higher levels of pruning compared to shallower layers, achieving high sparsity. Similarly, Gromov et al. [6] and Men et al. [32 demonstrated that removing early layers causes a dramatic decline in model performance, whereas removing deep layers does not. Lad et al. [23 showed that the middle and deep layers of GPT-2 and Pythia exhibit remarkable robustness to perturbations such as layer swapping and layer dropping. Recently, Li et al. (25) highlighted that early layers contain more outliers and are therefore more critical for fine-tuning. While these studies effectively highlight the limitations of deep layers in LLMs, they stop short of identifying the root cause of this issue or proposing viable solutions to address it.
Layer Normalization in Language Models. LN (2 was initially applied after the residual connection in the original Transformer [45). which is known as Post-LN, Later on, Pre-LN [3] 34] dominated LLMs, due to its compelling performance and stability 7 44 21 G). Prior works have studied the effect of Pre-LN and Post-LN. Xiong et al. [51] proves that Post- LN tends to have larger gradients near the output layer, which necessitates smaller learning rates to stabilize training, whereas Pre-LN scales down gradients with the depth of the model, working better for deep Transformers. Wang et al. 48 empirically confirmed that Pre-LN facilitates stacking more layers and Post-LN suffers from gradient
vanishing. The idea of connecting multiple layers was proposed in previous works 5 12 48 Admin introduces additional parameters to control residual dependencies, stabilizing Post-LN. DeepNorm 477 enables stacking 1000-layer Transformers by upscaling the residual connection before applying LN. Additionally, Ding et al. [II] proposed Sandwich LayerNorm, normalizing both the input and output of each transformer sub-layer. Takase et al. 40) introduced B2T to bypass all LN except the final one in each layer. Li al. (6 recently combines Post-LN and Pre-LN to enhance the middle layers. Zhu et al. [60] introduces Dynamic Tanh (DyT) as normalization-fre free alternative in Transformers, delivering comparable performance. Zhuo al. 61 proposes HybridNorm, hybrid normalization scheme combining QKV normalization with Post-Norm FFN to stabilize training in deep transformers. De and Smith [ also states that normalized residual blocks in deep networks are close to the identity function and proposes Skiplnit to remove normalization by introducing learnable scalar multiplier the residual branch initialized to 1/ I/VL. Our experiments suggest that SkipInit's 's learnable parameter does not improve performance and sometimes harms training.
8 Conclusion
In this paper, we re-introduce the concept of the Curse of Depth in LLMs, highlighting an urgent yet often overlooked phenomenon: nearly half of the deep layers in modern LLMs are less effective than expected. We discover the root cause of this phenomenon is Pre-LN which is widely used in almost all modern LLMs. To tackle this issue, we introduce LayerNorm Scaling. By scaling the output variance inversely with the layer depth, LayerNorm Scaling ensures that all layers, including deeper ones, contribute meaningfully to training. Our experiments show that this simple modification improves performance, reduces resource usage, and stabilizes training across various model sizes. LayerNorm Scaling easy to implement, hyperparameter-fie free, and provides a robust solution to enhance the efficiency and effectiveness of LLMs.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[2] Jimmy Lei Ba. Layer normalization. arXiv preprint arXiv: 2016.
[3] Alexci Baevski and Michael Auli. Adaptive input representations for neural language modeling. ICLR, 2019.
[4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. /3923, 2025.
[5] Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. Training deeper neural machine translation models with transparent attention. EMNLP, 2018.
[6] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.00 029544, 2024.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020.
[8] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. ACL., 2019.
[9] Soham De and Samuel L. Smith. Batch normalization biases residual blocks towards the identity function in deep networks, 2020. URL https://arxiv bbrg/abs/2en 2002 10444
[10] Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.
[11] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. NeurlPS, 34:19822-19835, 2021.
[12] Zi-Yr Yi Dou, Zhaopeng Tu, Xing Wang. Shuming Shi. and Tong Zhang. Exploiting deep representations for neural machine translation. EMNLP. 2018.
[13] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling, arXiv preprint arXiv:2103. 10360, 2021.
[14] Razvan-Gabriel Dumitru, Vikas Yadav, Rishabh Maheshwary, Paul-Ioan Clotan, Sathwik Te- jaswi Madhusudhan, and Mihai Surdeamu. Layer-wise quantization: pragmatic and effective method for quantizing 1ims beyond integer bit-levels. arXiv preprint arXiv:2406.17413 2024.
[15] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, al. Olmo: Accelerating the science of language models. arXiv preprint 2rXiv:2402.008888 2024.
[16] Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403. /7887, 2024.
[17] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. ICLR, 2021.
[18] Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka- KeiWeill Lee. Llm-adapters: An adapter family for pfrfmeter-efficien fine-tuning of large language models. EMNLP, 2023.

---

[19] Tianjin Huang, Haotian Hu, Zhenyu Zhang, Gaojie Jin, Xiang Li, Li Shen, Tianlong Chen, Lu Liu, Qingsong Wen, Zhangyang Wang. et al. Stable-spam: How to train in 4-bit more stably than 16-bit adam. arXiv preprint urXiv: /7055, 2025.
[20] Tianjin Huang, Ziquan Zhu, Gaojie Jin, Lu Liu, Zhangyang Wang, and Shiwei Liu. Spam: Spike- aware adam with momentum reset for stable Ilm training. arXiv preprint arXiv:2501.0683 2025.
[21] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arX1v:2310.06825, 2023.
[22] Diederik Kingma, Adam: A method for stochastic optimization. ICLR, 2015.
[23] Vedang Lad, Wes Gurnee, and Max Tegmark. The remarkable robustness of Ilms: Stages of inference? arXiv preprint a4Xiv:2406. 19384, 2024.
[24] Michel Ledoux. The Concentration of Measure Phenomenon, volume 89 of Mathematical Surveys and Monographs. American Mathematical Society, Providence, RI, 2001.
[25] Pengxiang Li, Lu Yin, Xiaowei Gao, and Shiwei Liu. Owlore: Outlier layerwise sam- pled low-rank projection for memory-efficicn IIm fine-tuning, arXiv preprint wrXiv:2405.18388 2024.
[26] Pengxiang Li, Lu Yin, and Shiwei Liu. Mix-ln: Unleashing the power of deeper layers by combining pre-In and post-ln. arXiv preprint arXiv:2412.13795, 2024.
[27] Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. Relora: High- rank training through low-rank updates. In ICLR, 2023.
[28] Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. EMNLP, 2020.
[29] Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang. Michael W Mahoney, and Yaoqing Yang. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. NeurlPS, 2024.
[30] Xuezhe Ma, Xiaomeng Yang, Wenhan Xiong, Beidi Chen, Lili Yu, Hao Zhang, Jonathan May, Luke Zettlemoyer. Omer Levy, and Chunting Zhou. Megalodon: Efficient I1m pretraining and inference with unlimited context length. NeurlPS, 2024.
[31] Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, al. Openelm: An efficient language model family with open training and inference framework. arXiv preprint a4Xiv:2404. 14619, 2024.
[32] Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models more redundant than you expect. arXiv preprint arXiv:2403.03855 2024.
[33] Saurav Muralicharan, Sharath Turuvekere Sreenivas, Raviraj Bhuminand Joshi, Marcin Cho- chowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. In NeurIPS. 2024.
[34] Toan Q Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. /WSLT, 2019.
[35] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog. 1(8)9, 2019.
[36] P Rajpurkar. Squad: 100,000+ questions for machine comprehension of text. EMNLP, 2016.
[37] Noam Shazeer. Glu variants improve transformer. arXiv preprint arXi::2002.05222,,200 2020.
12

---

[38] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Img Imm Training multi-billion parameter language models using model parallelism. /CMI,, 2020.
[39] Shoaib Ahmed Siddiqui, Xin Dong, Greg Heinrich, Thomas Breuel, Jan Kautz, David Krueger, and Pavlo Molchanov. deeper look depth pruning of Ilms. ICML, 2024.
[40] Sho Takase, Shun Kiyono, Sosuke Kobayashi, and Jun Suzuki. connection:: Serving stability and performance in deep transformers. ACL, 2023.
[41] Sho Takase, Shun Kiyono, Sosuke Kobayashi, and Jun Suzuki. Spike no more: Stabilizing the pre-training of large language models. arXiv preprint arXiv:23 /6903, 2023.
[42] Qwen Team. Qwen3: Think deeper, act faster, April 2025. URL httttt:t////ennttt github 1o/blog/qwen3/ Accessed: 1025-05-11.
[43] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. ICCV, pages 32 42-42, 2021.
[44] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Amn Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977, 2023.
[45] Vaswani. Attention all you need. NeurlPS, 2017.
[46] Roman Vershynin. Digh--Dimesional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018
[47] Hongyu Wang, Shuming Ma. Li Dong, Shaohan Huang. Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. 7PAMI, 2024.
[48] Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. Learning deep transformer models for machine translation. ACL, 2019.
[49] E. T. Whittaker and Watson. A Course of Modern Analysis. Cambridge Mathematical Library. Cambridge University Press, 4 edition, 1996.
[50] Yuxin Wu and Kaiming He. Group normalization. In ECCV, pages 1-19,0018
[51] Ruibin Xiong, Yunchang Yang. Di He, Kai Zheng, Shuxin Zheng. Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization the transformer architecture. In ICML, pages 10524-10533. PMLR, 2020.
[52] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang. Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:24/2.f5 15115, 2024.
[53] Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou, Tensor programs vi: Feature learning in infinite-eppth neural networks. arXiv preprini 10X02:2310.02244, 2023.
[54] Lu Yin, You Wu, Zhenyu Zhang, Cheng- Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pech- enizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (owl); missing secret sauce for pruning Imms to high sparsity. ICML, 2024.
[55] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9556-9567, 2024.
[56] Biao Zhang and Rico Sennrich. Root mean square layer normalization. NewrIPS, 32, 2019.
13

---

[57] Biao Zhang, Ivan Titov, and Rico Sennrich. Improving deep transformer with depth-scaled initialization and merged attention. arXiv preprint arXiv:1908.1 /1365, 2019.
[58] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient IIm training by gradient low-rank projection. ICML, 2024.
[59] Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without normalization. arXiv preprint arXiv:2503. 10622, 2025.
[60] Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without normalization, 2025. URL https://axxi www.aiv..r.gbb///ee/eo/o/oo ab////033. 10622
[61] Zhijian Zhuo Yutao Zeng, Ya Wang, Sijun Zhang, Jian Yang, Xiaoqing Li, Xun Zhou, and Jinwen Ma. Hybridnorm: Towards stable and efficient transformer training via hybrid normal- ization, 2025. URL / arrxiv. httpaaawv...//a/aaa////aa abs 2253/55030053 04598
14

---

A Proofs of the Theorems of curse of depth
A.1 Proof of Lemma
Proof. Given Equation
from have:
y/ = m2+1 2ie + FFN(IN(E()),
(13)
z/- x4 Attn(LN(zz)).
Based on our Assumption Var(Auu((IN((z))) Amm. Then we can write:
Var(x')) Var(xz) + Var(Atta(LN((zz)) + Cov(Aut((LN(x:)) Var(a,))
(14)
= = + Aum + Oxe Aum-
where # is the correlation factor. Similarly, let Var(FFN(LN(())) -opEN- Then we have:
2 2 = aa(2)) + ofFN + P2 Ozf OPEN,
(15)
where P2 is the correlation factor. Thus, the relationship between Var(re+1) and Var(zz) becomes:
J xe + Amm OPEN + Pi -nn a + 02 Ong OFFN.
(16)
A.1.1 Variance of the Attention
The scaled dot-product attention mechanism is defined as:
QK
Attn(q. (Q, K, V) softmax
V.
(17)
Vogy
The softmax function outputs a probability distribution over the keys. Let the softmax output be = softmax [QK" where A is matrix with each row summing to The final attention output Vdl is obtained by multiplying the softmax output A with the value matrix V:
Attn Q K, V) AV.
(18)
Lemma 2 (24 Let (X+) be independent and identically distributed random variables with mean m and variance o² < co. Define the softmax weights p. Ex=,e and let = (pr...., .N.). Then, as N + oo, with high probability, the softmax vector p concentrates around the uniform distribution on N elements. In particular.
N
27 pi
lim N
E
N = 0,
(19)
which implies that the softmax output becomes asymptotically indistinguishable in expectation, from the uniform distribution.
According the above lemma, to simplify the analysis, we make the following additional assumptions: The softmax output A is approximately uniform, meaning each element of A is roughly 1/n, where is the number of keys/values. Given this assumption, the variance of the attention is:
Var(Attn(Q, ~ Var(AV) 1 Σ dhead Var( not dhead = cheado = oî d.
(20)
where W is the universal weight matrix defined as before.
15

---

A.1.2 Variance of the Feed-Forward Network
The feed-forward network (FFN) in transformers typically consists of two linear transformations with a ReLU activation in between. The FFN can be written as:
(TN W2 ReLU(W, x b1) b2.
(21)
where W1 and 22 are weight matrices, and bı and be are bias vectors. Using the result obtained by Wang et al. we get:
PFN ow, ow: =ow
(22)
In conclusion:
2
51 + Ow + P2
Osi
OW
o +
OW W
(23)
O
02
Zz
For simplicity, we set the numerator part to 1. Substitute Ozt = Oze V
1 +
+
into
02,
ore
Equation 16 we get:
0 21 + ow + of + p aze ow + P2 oz, owd
P20
p30
xe ow + of + pı oz ow + P2 oz owd +
2
20
2
(24)
0(1 + 1 24
From the result generally infer that the variance accumulates layer by layer. The variance with regard to ony
1+ (1+1) ozz (++)))
of = o θ II
(25)
We can also obtain a similar result for
We observe that for any o2 2 1, the sequence is increasing, meaning each term in the product is bounded. Consequently, the entire product is bounded above by:
) Ozi
1- 8-1
2
< 2
II
02
(1
O(L)
(26)
Ozi
k=
Taking the natural logarithm of both sides:
E-
1
log(o?,) =log
1
log
1
+ log(o2))
21
02
k=1
2t
1=1
(27)
2
log(o2))
o2
Exponentiating both sides to find the lower bound for o2 we obtain:
22
N 22
exp
202
16

---

This provides a tighter lower bound for o compared to the upper bound of Equation (26). Since we know the upper bound of variance grows exponentially, the lower bound must be sub-exponential. Therefore, for o = l, we must have:
1 2k ))
0z exp Σ
((exp(VT)) O(L).
Therefore, the increasing lower bound for o must grow faster than a linear function. So, the increase of variance is nub-exxonential. A large increase in such bound will lead to gradient spikes, which can connect to previous studies in Huang et al. [19
A.2 Proof of Theorem
In this proof, we will divide the argument into two parts: first, the calculation of the Lemma 3 and second, the analysis of Bur Uz
Lemma 3, For an L-layered Pre-LN Transformer,
duL Bag
using Equations
and (3) is given by:
dyL
L-1
Oye
Ox
II
(28)
0r
Orl
Ozg
n=1
The upper bound for the norm of U21
duL is:
OyL
< II
L-1
((1
o²
Jal ( vdrrN)?
2
1=1
(29)
(1 + 2dh
2
4/2
o² (o'd dhead + (1+ v dheaa/d)) ))
Vs) Ox
Here. h denotes the number of heads, is the sequence length, and d, drFN, and dhead the dimension of the embedding, FFN layer and multi-head attention layer, respectively. The standard deviation of Wq. WK. Wy. and WFFN layer is based on Assumption
A.2.1 Proof of Lemma 3
Proof. Our derivation follows results in specifically Equation (7), which provides an upper bound on the norm of
öye as:
0z1
Oye
L-1 Oye Ox' II
(30)
0z
Ox, dxy
Thus, we can estimate the upper bound of the gradient norm of Byz deg
by analyzing the spectral norms of the Jacobian matrices for the FFN layer and the self-attention layer, namely,
Oye
Oz
FFN:
Attention:
(31)
Ore
We now derive an upper bound of
ll2 as follows:
Oye Oz
(FFN(LN(E
OLN(x')
11+
(32)
aLN(x
Ox'
Let Owl, and ow2, be the standard deviations of W and W?. respectively. From Assumption spectral norms of W1 and W? are given by their standard deviations and dimensions 46 have:
1 the
so we
Will2 0z (v v dFFN).
(33)
17

---

For simplicity, we assume that d, and dEEN are equal, thus,
OFFN(LN(E3)) aLN()))
1IW) s a102(Vd vdn))
(34)
Finally, we have the following bound:
Oyle 0r
Owle wwel (vd+ ddFFN)
=1+ o2(v + d
+
(35)
2
Following a similar procedure for the FFN, we rewrite
||
0zt 8x
lla in Equation (31 as:
O.'
(A))(LN(())
|
[LN(()
(36)
0r 2
0LN(z)
8r
Let Z()) = concat(head, ()), ..... h-)d and J denote the Jacobian of the (()). We can now express the spectral norm of the Jacobian matrix of attntion as:
(Attn()((z)) (L)(
WP ((L(()) (Ze( -IW2 2- (L)( 2
(37)
2
From 46 we know that:
| 12 S (
o2v dd + of (va+ + V vdheus))
(38)
/s)
Here h is the number of heads, is the sequence length, and the standard deviation of Wo. WK, and Wy a.
By combining the inequalities 35). 38 and 36 and assuming that all a values are the same for simplicity. we obtain:
OyL Oc
L-1
II
((1
o2(Vd + vdFEN)?
S
2
1=1
(39)
(1 2dh
(V++2+
1
0² oddv dhead + (1+ vdhead/d) ))
V8]
Oxe
A.2.2 Analysis of the Upper Bound
should be sufficiently small, and the standard deviation, oz; or az should satisfy the condition o² < Ozy maintain the lazy training scheme. Thus, we obtain the following bound for the product over from T.
As discussed in 41
To find the bound for 0y: dum
with respect to f, we simplify the given inequality by approximating aze and Tg Based on Equation (23), Onz is only one layer ahead of agy, and this layer does not significantly affect the overall performance of deep Transformer networks. Furthermore, based on
Lemma I we assume that 024 Oxp-
Equation can be expressed in a traditional product form
49 for Oz
Tyr Ox
-++
1 A+
1+
(40)
2
1=1
Tae
0
where
0²
+ 2dh (V++2+ //) 0² (dr dhead (dVdnoudd
A
+ v dhcadd/d (41)
(vd+ 2
18

---

and
B 2dh (v++2+ //) o dd /hhed:
(42)
where A and B are independent of Oag. and under our assumption, are treated constants.
From classical infinite series analysis, it known that as dze grows faster rate, the upper bound of the product decreases. The proof is omitted here for brevity. For the upper bound on the convergence rate of o22; we assume of exp(E) without loss of generality. Under this condition, we can derive the following result:
Taking the natural logarithm of the product:
L-1 l
B 62k
log
II
log
1
22
e
Using the Taylor series expansion for log(1 + x), and applying this to our sum, we get:
1+
A ck
B
B 2h
B 22
log
+
22
ck
2k
ck
ek
R=
By evaluating the sums for each order of terms, we find that the result is a constant. Carrying this out each term, we obtain:
log
)
II
1
4 c
B
A
B
1
A
+2
A
B²
2k
2
-1
2
6
e
Am
Thus, the product is approximately:
θyr 0z 12
A
B
1
A
A
-1 ct )
B
S
exp
+2
+
M,
(43)
2²
2
2
1
where M a constant.
For the lower bound on the convergence rate of o² we assume o² = without loss of generality. Under this condition, derive the following result. Taking the logarithm of the product, applying the Taylor series expansion for log(1 + x),, and applying this to our sum:
20
log
1+
B ek2
30
A
B
A
B
2
A
3
)
)
k
2k2
k
h2
ek2
For the first-order terms:
30
A
B
The series Σx=1 is the harmonic series, which diverges. However, we approximate using the Maler-Mascheroni constant y and the fact recognize that the harmonic series grows logarithmically:
log + (for large n).
The other series such as E क converge because A² grows very rapidly.
19

---

For higher-order terms, they converge to constant, involving the series Ek=1 1st converges to x= they contribute constant. Exponentiating both sides, we get:
6. so
A k
B ek2
I
exp (A(log n 7) const).
Thus, the growth rate of the upper bound for
ppr.
is:
da
dyL 0x
S (L)).
(44)
12
Theoretical Analysis of LayerNorm Scaling
Lemma 4. After applying our scaling method, the variances of x'g and De, denoted as o and o² respectively, exhibit the same growth trend, which is:
{-1
o = θ
Vioae (L) S of ((((2-0))
II km
(1
))
1
(45)
with the following growth rate bounds:
(46)
where is a small number with 1/2 <
From Lemma we can conclude that our scaling method effectively slows the growth of the variance upper bound, reducing it from exponential polynomial growth. Specifically, it limits the upper bound to a quadratic rate instead of an exponential one. Based on Theorem | after scaling, we obtain the following:
Theorem 2. For the scaled Pre-LN Transformers, the Euclidean norm of day
duL, is given by:
dyL
T-1
Poo2 PozzD) 1)
II
1+
(47)
tdsz
2
t=1
where A and B are dependent on the scaled neural network parameters. Then the upper bound for the norm is given as follows: when o², grows at ((2-c), (i.e., at its upper bound), we obtain:
dyL
(2-c)
S w(1,,
(48)
0r
2
where denotes that if [(x) = w(g(x)), then lim1- linearly (i.e., at its lower bound), obtain:
f(x) g(z)
cc. Meanwhile, when o%, grows
Oyr.
22
O
(49)
dx 2
The detailed descriptions of A and B, and c, along with the full proof, are provided in Appendices B. and B.2
By comparing Theorem (before scaling) with Theorem 2 (after scaling), we observe a substan- tial reduction in the upper bound of variance. Specifically, it decreases from exponential growth O (exx(()) to at most quadratic growth ( ). In fact, this growth is even slower than quadratic expansion, as it follows (2-(2))) for some small > 0.
When we select a reasonable upper bound for this expansion, we find that
@dyz 8irr
no longer
possesses a strict upper bound. That is, as the depth increases,
Byg 021
continues to grow gradually.
Consequently, fewer layers act as identity mappings compared to original Pre-LN where nearly all deep layers collapsed into identity transformations. Instead, the after-scaled network effectively utilizes more layers, even as the depth approaches infinity, leading to improved expressivity and trainability.
20

---

B.1 Proof of Lemma 4
Proof. After scaling, the equation becomes:
y -t-1 x FFN 1 LN(EE))
Ve
(50)
5 = ze + Attn LN(1)))
Vt
Following the same analysis as before, we scale the Attention and FFN sub-layers, yielding:
1 7006 = ow nl n of
Aeta
oPE
Ow
W
(51)
L
82
In conclusion:
0 = oz2 + ow + P2 Ozi
ow
+ = o 01
(52)
/e
Vlo
Similarly, obtain:
1 = 01 0(1 +
(53)
Vlogs
From the result we can generally infer that the variance accumulates layer by layer. The variance with regard to ozz
2 = o +
[1+
(54)
Vkoze
k=
We can also obtain a similar result for 22
Taking the natural logarithm of both sides:
1
log(o2.) log
T
1
Σ log
1+
log(o
ko
ko
k=1
(55)
2
1
1
log(o%) )
ko2
2
ko
R=1
To establish a lower bound for oz,, we exponentiate both sides. Setting o E, we must have:
1 2k )
22 02 exp Σ
(cexp(log L)) (().
(56)
Therefore, the increasing lower bound of is greater than linear function.
Similarly, assuming o² ((2-c), we have:
(-1
E
H
(t//2-1
2 = 2
II
+ 12--12) exp
Σ
k2-112 k=
exp
c/2
1
(57)
k=1
< ( < (()))
Here e is a small constant with 1/2 < < 1. Therefore, the increasing upper bound of o2 is slower than the 8 function, leading to:
0
((L?)
21

---

B.2 Proof of Theorem 2
Proof. Similarly, after applying the scaling transformation, we derive an upper bound for
Oye Brg
ll as
follows:
dye Oxr
(FFN(LN(E))
aLN(x
- +
OLN(x()
V2
Oir
2
(58)
o7
=1+
20 (vd ddPEN)²
Similarly, rewriting Equation
31
after scaling, we have
0z'
(Atnn()((s)) @LN(x)
(LN(2)
(59)
0z
1
dz
By combining the bound obtain:
58 and inequality (59)
and assuming all equal for simplicity,
Oyr.
II
(11 20x,
o2(vd vdFFN)
0x l2
i=1
(60)
x (1 +1+ 2dh
[V++2+
1,) vs)
o²
(o'd dhead + (1 v Viuead/ ))
lo
Equation (60) is traditional product form
44 for Jze. After scaling, it becomes:
dyL de.
II
1
A+
B
(61)
td
Po2
t=1
where A and B retain their forms from Equation
(41 and Equation (42) and are treated as constants.
Regarding the upper bound on the convergence rate of 2. we assume of {(2-c) without loss of generality. For large L, the product can be approximated using the properties of infinite products:
1-1 II t=1
1+
p2-0/2
B pl-t
~ exp
Σ t=1
F2-e/2
A
)
B
(62)
Then, by evaluating the sum in the exponent, we obtain:
L-1
B
A. /6/2-1
fe3
II
11+ 22-e/2
+
exp
B
(63)
p4-c
c/2
3
t=1
Therefore, we establish the upper bound:
dyL
ge/2-1 -1
6033 -
< +
exp
A
+ B
w())
(64)
Om
c/2
c-3
2
where (1) denotes growth strictly greater than constant as defined before.
C
Qwen2.5
We further evaluate the generalizability of LNS by applying it to a state-of-the-art architecture, Qwen2.5-0.58 [52.. We train the model for 6B tokens and compare LNS against the standard Pre-LN setup. Consistent with previous findings, Table Table(7]illusrrate that LNS yields notable reduction in perplexity- from 20.62 to 19.57- highlighhtn its effectiveness even on strong, modern architectures.
The consistent benefits observed across increased model scales, larger training datasets, and diverse architectures suggest that LNS is promising technique for enhancing the training contemporary large language models, ensuring that deeper layers contribute more effectively to learning.
22

---

Table 7: Perplexity (PPL. .) comparison under scaled-up pre-training. For LLaMA-1B and 7B, training scheduled for 100B tokens but is terminated early report results. Qwen-2.5 trained with fixed budget of 6B tokens.
Model
# Params
Tokens
Pre-LN (PPL)
LNS (PPL)
Qwen2.5-0.5B
0.5B
6B
20.62
19.57
(a) Post-LN Angular Distance
(b) Pre-LN Angular Distance
0.4
Subsequent n Layer
0.4
Subsequent n
0.3
-03
0.2
0.2
0.1
Layer Index
Layer Index
(c) Post-LN Performance Drop
(d) Pre-LN Performance Drop
Ponginw
Pa Parighl
ARC ApD
ARC
-10
-10
-15
-15
0
Layer Index
Layer Index F
Figure 7: Results of in-house small-scale LLLMM-133MM Angular Distance (a, b): Each column represents the angular distance from the initial layer & (x-axis) and its subsequent n layer (y-axis). The distance scaled the range [O, 11, where yellow indicates smaller distances and purple indicates larger distances. Performance Drop (c, d): ARC-e performance drop of removing each single layer from LLaMa-130M.
D Results of In-house Small-scale LLaMa-130M
Figure 7 compares LLaMa- MMMMMMMMMMMMM models differing only Layer Normalization, clearly distinguishes Post-LN from Pre-LN. In Post-LN models, early layers exhibit high similarity (low angular distance,
Fig. a) and their removal causes minimal performance loss (Fig. 7ld), while deeper layers become
more distinct and critical. Post-LN also shows larger gradients in deeper layers but severe vanishing
in early layers at the start of training (Fig. (Fc.. Conversely, Pre-I.N 1.aaMa1330N demonstrates gradual decrease in angular distance with depth, resulting in highly similar deep layers (Fig. 7
b). Removing most layers after the first in Pre-LN causes negligible performance loss (Fig. d). d),
indicating their limited contribution. These consistent findings, observed in both open-weight and in-house LLMs, lead to the conclusion that the widespread use of Pre-LN is the root cause of the ineffectiveness of deep layers in LLMs.
E
Training Loss Curve
report the training loss curves of Pre-LN and LayerNorm Scaling Figure While LayerNorm Scaling exhibits slightly slower convergence at the early stages of training, it consistently outperforms Pre-LN as training progresses. attribute this the uncontrolled accumulation output variance in Pre-LN, which amplifies with depth and training steps, ultimately impairing the effective learning
23

---

of deeper layers. In contrast, LayerNorm Scaling mitigates this issue by scaling down the output variance in proportion to depth, thereby enabling more stable and effective training across all layers during training.
Com pariissson of Training Loss Curves on C4 Dataset for LLaMa-18
Pre-NN LayerNorm Scaling
Loss
10000
20020
30000
50000
Step
Figure 8: Training loss of LLaMA-1B with Pre-LN and LayerNorm Scaling.
F
Variance Growth in Pre-LN Training
To analyze the impact of Pre-LN on variance propagation, we track the variance of layer outputs across different depths during training.
Verience at 1000 Epochs Across Layers
Variance at 3000 Epochs Across Layars
Veriance at 6000 Epochs Across Layers
325
Varlance
Variance
Varfance
330
Layer Index
Layer Index
Layer Index
Figure 9: Variance growth across layers in LLaMA-130M, with Pre-LN. Each subplot shows the variance at different training stages (1000, 3000, and 6000 epochs). In all cases, the variance follows an exponential growth pattern as depth increases, indicating that deeper layers experience uncontrolled variance amplification regardless of training progress.
Figurr@illustratees illustrates the layer-wise variance in LLaMA-1300 with Pre-LN at 1000, 3000, and 6000 epochs. Across all stages, variance remains low in shallow layers but grows exponentially in deeper layers, confirming that this issue persists throughout training rather than being temporary effect. This highlights the necessity of stabilization techniques like LayerNorm Scaling to control variance and ensure effective deep-layer learning.
Figure 9
G Applicability to Vision-Language Models (Qwen 2.5-VL)
To examine whether the Curse of Depth also manifests in vasion-language models (VLMs), we perform layer-pruning experiments Qwen 2.5-VL-7B 4. For both its vision encoder and language decoder, we prune one transformer layer at a time and directly evaluate the pruned model on the MMMU benchmark [55] Figure 10 presents the resulting performance drops.
We observe that the language branch clearly suffers from the Curse of Depth, whereas the vision branch remains uniformly important. This suggests that the phenomenon is more pronounced in autoregressive language components of VLMSs and may not directly transfer to vision encoders. detailed modality-ppecific theoretical account is left to future work and community discussion.
24

---

(a) Qween2.5-V-788
(b) Qwen2 5--L-7---
Pergai
APIC
APIO
Figure 10: Performance drop of layer pruning on Qwen 2.5-VL-7B. (a) Vision branch shows relatively uniform sensitivity across layers. (b) Language branch exhibits clear Curse of Depth: deeper layers contribute much less than early ones.
H LayerNorm Scaling in Vision Transformer
To evaluate whether LayerNorm Scaling (LNS) also benefits architectures beyond language models, conduct experiments ViT-S for image classification. Since ViT-S includes LayerScale by default-which may interfere with the effect of LNS LNS- wwe remove LayerScale from all evaluated variants to ensure fair comparison. We then test different insertion positions of LNS. The top-1 accuracy results are summarized in Table g Whereas LNS in language models is typically most effective directly after normalization, Vision Transformers, the best position is after the attention and MIP blocks. We next examine whether this performance gain correlates with better control of layer-wise variance.
Table 8: Top-1 accuracy (%) of ViT-S model with and without LNS.
Model Variant
LNS Position
Top-1 Accuracy
ViT (with LayerScale)
70.30
VIT (w/o LayerScale)
67.91
ViT (w/o LayerScale)
after LayerNorm
66.43
VIT (w/o LayerScale)
after Attn/MIPP
68.75
Figure 11 plots the average output variance of each transformer block during training. Without 3,000 by 30K update steps.
LayerScale, variance in deeper layers grows rapidly-exceedie Applying LNS after Attn/MLP controls this growth to below 150, confirming that LNS stabilizes the forward signal even in vision transformers.
These preliminary findings indicate that the variance-control mechanism underlying LNS generalizes to vision transformers when the scaling is applied after Attn/MLP. We leave a more detailed theoretical understanding of this behavior future work and community discussion.
I Post-LN Transformers
For Post-LN Transformers, we continue to adopt Assumption In this setting, each layer followed by a layer normalization (LN) step, ensuring that the variances o² and o² remain fixed at across
25

---

vit w/o LayerScale
vit w/o LayerScale w LNS after mip
103
123
243
Aveenge Strrarce Chir
1203
60
0.33
Update
Undate
Figure 11: Laser-wise output variance of ViT-$ without LayerScale (left) and with LNS after Attn/MLP (right). LNS significantly reduces the variance growth compared to the baseline.
all layers. Consequently, the norm indicating stable gradient propagation.
Oyz Birg
exhibits minimal variation from one layer to the next,
Since the variance effectively controlled by LN Post-IN Transformers, an explicit variance- based analysis becomes less critical. Nonetheless, there remain other important aspects to investigate in deeper Post-LN architectures, such as the evolution of feature mappings and the behavior of covariance kernels over deep layers. These directions will be pursued in future work.
J Limitations
While this work offers a comprehensive analysis of the Curse of Depth in LLMs and proposes LayerNorm Scaling as effective remedy, several limitations remain:
Scope of Architectures. Our study primarily focuses on Transformer based LLMs using Pre-LN. Although Pre-LN dominates modern architectures, our theoretical study does not cover models em- ploying alternative normalization strategies (e.g., Post-LN only 133, normalization-fee architectures (59) or emerging paradigms such as mixture-of-experr or structured sparsity-base based models.
Task Coverage. Most empirical evaluations, including pruning and angular distance analyses, were general-purpor purpose benchmarks like MMLU. While these tasks reflect broad model capabilities, domain-specific long-context reasoning tasks may reveal different dynamics in deep layer contributions, which we leave for future work.
conducted using general
Fine-grained Representation Quality. While LNS improves angular distance and performance sensitivity across layers, a deeper analysis of what types of information are represented or lost in deeper layers remains unexamined. For example, whether LNS helps preserve syntactic, semantic, or factual knowledge across depth unclear.
K Broader Impact
The Curse of Depth phenomenon, identified and addressed in this work, has significant implications for the design, training, and deployment of LLMs. By revealing that deeper layers in modern Pre-LN Transformers often fail to contribute meaningfully to learning, how model capacity is allocated and utilized. This has both practical and ethical consequences.
study prompts reevaluation of
From computational efficiency perspective, the insights offered by this work can lead to more principled model pruning, layer reuse, or architecture design strategies that improve training and inference efficiency without compromising performance. In particular, LayerNorm Scaling enables deeper layers be trained more effectively, maximizing the utility of each parameter and reducing unnecessary resource expenditure. This can help democratize access to powerful models by reducing
26

---

the cost of pretraining and fine-tuning, especially for institutions or communities with limited computational resources.
From sustainability standpoint, addressing Col has the potential to lower the environmental impact of large-scale model training by mitigating wasteful computation. With LLMS increasingly deployed in industrial-scale applications, these gains can scale into substantial reductions in energy consumption and carbon footprint.
terms of scientific understanding, this work contributes to the growing body of research that seeks to interpret and improve the internal dynamics of deep neural networks. By identifying the failure modes induced by Pre-LN at depth, we provide both diagnosis and remedy that could influence future research in deep optimization, normalization strategies, and interpretability.
We also caution that increasing the efficiency of L.M training and deployment through techniques like LNS may further accelerate the proliferation of powerful LLMs, raising concerns around misuse, disinformation, labor displacement. As such, our findings should be accompanied by responsible deployment practices and continued ethical oversight in the broader Al community.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions and scope of this paper claimed in the abstract. Detailed information can be found in the introduction section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide Limitation Appendix I.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
28

---

Justification: We provide a complete set of assumptions and rigorous proof' for all theoretical results. Detailed information can be found Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide all experimental details in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
29

---

Answer: [Yes]
Justification: We provide all experimental details in the experiment section. The detailed LayerNorm Scaling codes of our method can be found in supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
hetpoolliigguddde/ooooooo http://uddde/ooeSuubntb ssicnlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all experimental details in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Following the standard experimental setup, we repeat each experiment over random seeds and report the mean of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
30

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computing resources experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We reviewed and followed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide the potential broader impacts in Appendix J.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
31

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The data and models pose no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original papers that produced the code package and datasets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
32

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Details of the datasetsscodeemmodll are provided in the supplemental materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "6seU1C7BKI", "reviewer": "Reviewer_JTrQ", "summary": "In this paper, the authors attempt to understand the realisation that deep layers in the network are less effective than actually expected. They confirm the finding across various models and experiments. They then identify the reasons related to the usage of the pre-layer normalisation. They then propose a rather simple modification of LayerNorm scaling and show that this enhances the LLM's performance. \n\nThe paper is generally well written and nice to read. I would like to kindly ask the authors a couple of questions: \n\n1. What about other factors that affect deeper layer expressivity? Like Attention saturation, Batch sizes and so forth. Do you think those do not matter? Does it not matter as much as the cause found in the paper? Would a holistic solution consider all those together? \n\n2. I find the analysis interesting, but can you isolate the effects of Mix-LN versus the other variant? Is it due to instability or poor parameter tuning? What could we say about this? \n\n3. Do the results show improvements for formal reasoning problems as well? Can this help us in coding and/or formal reasoning tasks? \n\n4. Can the authors please discuss and experiment with how their proposed method plays with \"newer\" tricks that we use today, like gradient checkpointing, flash attention or other memory optimisation and so forth? Would this amplify or negate what has been suggested? \n\n5. Can the authors apply their method retroactively to high-performing models and see if they benefit? I find it lacking. It would be better to understand the actual gain we would get if possible. \n\nAll in all, i think this is a good paper but please help me understand the points above.", "questions": "Please see above", "limitations": "OK", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Please see above", "was_revised": true, "final_justification": "Rebuttal addressed my comments, and I remain positive about this paper.", "rebuttal": "We sincerely appreciate your thoughtful feedback and positive evaluation. We address your remaining concerns below. \n\n**Q1. What about other factors that affect deeper layer expressivity? Like Attention saturation, Batch sizes and so forth. Do you think those do not matter? Does it not matter as much as the cause found in the paper? Would a holistic solution consider all those together?**\n\n**A1.** We appreciate the reviewer’s insightful observation regarding additional factors that may affect the expressivity of deeper layers. We do think other factors also matter for the expressivity of deeper layers, including but not limited to attention saturation [1], attention collapse [2,3,4], model dimension [5], initialization [6,7], etc. Our framework provide a simple explanation here, i.e., the (sub)-exponential growth of output variance with depth will cause deeper layers become an identify function for Pre-LN models. While other factors may affect the constant coefficient in the variance and gradient norm expressions, they do not change the growth rate. As the depth $L \\to \\infty$, these prefactors do not influence the asymptotic behavior.\n\n\nA holistic solution to training deep models would ideally integrate multiple factors. Our current work represents a step toward this broader goal by providing a principled intervention that is orthogonal to existing approaches and potentially complementary to them. These additional aspects are part of our ongoing research efforts and will be systematically explored in future publications.\n\n\n**Q2. I find the analysis interesting, but can you isolate the effects of Mix-LN versus the other variant? Is it due to instability or poor parameter tuning? What could we say about this?**\n\n**A2.** The divergence of Mix-LN stems from its reliance on Post-LN, which is notoriously unstable in large-scale Transformer training due to gradient vanishing issues [8,9]. While applying Post-LN in the early layers can enhance the representations of middle layers for small models compared to Pre-LN, Mix-LN becomes increasingly unstable with larger model and dataset sizes. For example, when increasing the number of training tokens from 5B (Mix-LN paper) to 10B (our paper), training LLaMa-1B diverges, as shown in Table 1.\n\n\n**Q3. Do the results show improvements for formal reasoning problems as well? Can this help us in coding and/or formal reasoning tasks?**\n\n**A3.** Thank you for your insightful question. Due to our limited computational resources, we pre-trained our models on a relatively small number of tokens, which is insufficient to achieve competitive performance on formal reasoning tasks such as code generation or mathematical problem solving.\n\nNevertheless, it is well-established that stronger base models tend to exhibit better reasoning capabilities. Moreover, model depth plays a crucial role in reasoning, as highlighted by the Physics of Language Models paper [10]. Our proposed LNS improves the effectiveness of deeper layers, thereby enhancing the model’s capacity for complex reasoning. We are confident that our approach can lead to improved performance on formal reasoning benchmarks when applied to larger-scale models. \n\n\n**Q4. Can the authors please discuss and experiment with how their proposed method plays with \"newer\" tricks that we use today, like gradient checkpointing, flash attention or other memory optimisation and so forth? Would this amplify or negate what has been suggested?**\n\n**A4.**  Our approach introduces only a simple scaling operation after LayerNorm, making it, in principle, fully compatible with existing memory optimization techniques such as gradient checkpointing, mixed-precision training, and activation offloading. Thanks to its simplicity, the resulting model remains functionally equivalent to one trained with the standard Pre-LN setup, it can be integrated seamlessly into existing training pipelines without introducing additional memory overhead or architectural modifications.\n\n\n**Q5. Can the authors apply their method retroactively to high-performing models and see if they benefit? I find it lacking. It would be better to understand the actual gain we would get if possible.**\n\n**A5.** Unfortunately, since high-performing models such as Qwen and LLaMA have been trained with Pre-LN on massive datasets, it is not straightforward to apply LNS in a post-training manner. To evaluate whether the performance gains from LNS extend to more challenging and state-of-the-art LLM training settings, we conduct additional scaling experiments using OLMO, a modern open-source framework.\n\nWe train models with 60M, 150M, 300M, 1B, and 7B parameters, all on a fixed 20B-token budget. As shown in the table below, LNS consistently and substantially outperforms the standard Pre-LN baseline across all model sizes. Notably, for the 7B model, LNS reduces the final training loss from 2.69 to 2.50. These results underscore the scalability of LNS and its effectiveness in large-scale pre-training regimes, highlighting its potential in large-scale pre-training scenarios.\n\n\n|  Model Size        | # Tokens | Pre-LN  | LNS  |\n|---------------|----------|--------------|-----------|\n| 60M      | 20B       |   3.67    | **3.56** |\n| 150M     | 20B       |    3.44   | **3.24** |\n| 300M     | 20B       |    3.29   | **3.14** |\n| 1B       | 20B       |   3.15    | **2.85** |\n| 7B       | 20B       |   2.69    | **2.50** |\n\nWe further evaluate the generalizability of LNS by applying it to a state-of-the-art architecture, Qwen2.5-0.5B. The following table illustrates that LNS yields a notable reduction in perplexity—from 20.62 to 19.57—highlighting its effectiveness even on strong, modern architectures.\n\n|   Model      | # Tokens | Pre-LN (PPL) | LNS (PPL) |\n|---------------|----------|--------------|-----------|\n| Qwen2.5-0.5B  | 6B       | 20.62        | **19.57** |\n\n[1] Xiao, G., Tian, Y., Chen, B., Han, S. and Lewis, M., 2023. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453.\n\n[2] Dong, Y., Cordonnier, J.B. and Loukas, A., 2021, July. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International conference on machine learning (pp. 2793-2803). PMLR.\n\n[3] Geshkovski, B., Letrouit, C., Polyanskiy, Y. and Rigollet, P., 2025. A mathematical perspective on transformers. Bulletin of the American Mathematical Society, 62(3), pp.427-479.\n\n[4] Noci, Lorenzo, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. \"Signal propagation in transformers: Theoretical perspectives and the role of rank collapse.\" Advances in Neural Information Processing Systems 35 (2022): 27198-27211.\n\n[5] Bao, Z., Yu, P., Jiang, H. and Li, Q., 2025. The Effect of Depth on the Expressivity of Deep Linear State-Space Models. arXiv preprint arXiv:2506.19296.\n\n[6] Yang, G. and Schoenholz, S., 2017. Mean field residual networks: On the edge of chaos. Advances in neural information processing systems, 30.\n\n[7] Takase, S., Kiyono, S., Kobayashi, S. and Suzuki, J., 2023. Spike no more: Stabilizing the pre-training of large language models. arXiv preprint arXiv:2312.16903.\n\n[8] Wang, H., Ma, S., Dong, L., Huang, S., Zhang, D. and Wei, F., 2024. Deepnet: Scaling transformers to 1,000 layers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(10), pp.6761-6774.\n\n[9] Takase, S., Kiyono, S., Kobayashi, S. and Suzuki, J., 2022. B2t connection: Serving stability and performance in deep transformers. arXiv preprint arXiv:2206.00330.\n\n[10] Ye, T., Xu, Z., Li, Y. and Allen-Zhu, Z., 2024. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process. arXiv preprint arXiv:2407.20311."}, {"review_id": "Tr8rGOtGOU", "reviewer": "Reviewer_skvN", "summary": "This paper explores the \"curse of depth,\" which is that later layers seem to contribute less than earlier layers. This is demonstrated by experiment, and a theoretical analysis shows that gradients have a constant upper bound as the depth of the network increases. The authors propose an extremely simple modification that scales the contribution of layer l by 1/sqrt(l). They repeat the theoretical analysis and show that the gradient grows linearly in the depth, and they repeat the experiments showing that the layers contribute more equally. Finally, they show through extensive experiments that their modification improves language model performance.", "questions": "- I found the theoretical argument around Lemma 1 and Theorem 1 difficult to follow. If the goal is to prevent the later layers from becoming identity-like -- that is, dominated by the residual connections -- it seems counterintuitive that the solution would be to scale down the part of each layer that is not the residual connection. Could you provide some explanation to resolve this paradox?\n\n- Since each layer norm has a scaling factor built into it (usually called $\\gamma$), why couldn't it learn the $\\sqrt{\\ell}$ factor automatically?\n\n- In Figure 2b, what’s the reason for the spike around layer 27-28?\n\n- In Eq (23), $\\sigma_W^2$ should be $\\sigma_W^4$ and $\\sigma_W$ should be $\\Sigma_W^2$ (right?)\n\nThe proof of Lemma 3, seems, to me, to skip a lot of steps. \n- How do you get from (32) and (33) to (34)? It would help to write out the Jacobian of LN. \n- How do you get equation (37)? The only justification is a citation to an entire textbook [44].", "limitations": "The limitations section is relegated to the appendix but is adequate.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\nBoth the experimental and theoretical results are convincing, and the LNS modification is so simple that the paper makes a very compelling case that it should be adopted.\n\n# Weaknesses\n\nI found the theoretical argument around Lemma 1 and Theorem 1 difficult to follow. If the goal is to prevent the later layers from becoming identity-like -- that is, dominated by the residual connections -- it seems counterintuitive that the solution would be to scale down the part of each layer that is not the residual connection. Could you provide some explanation to resolve this paradox?\n\n# Minor comments\n\n- line 243: the reference to Figure 3 is not right. I couldn’t find what the right referent should be.", "was_revised": true, "final_justification": "Thank your for your responses. I don't have anything to add and will keep my score the same.", "rebuttal": "We sincerely thank the reviewer for the positive evaluation. We are pleased that they find our experimental and theoretical results convincing and consider LNS compelling for adoption. We address your remaining comments below. \n\n\n**Q1. I found the theoretical argument around Lemma 1 and Theorem 1 difficult to follow. If the goal is to prevent the later layers from becoming identity-like -- that is, dominated by the residual connections -- it seems counterintuitive that the solution would be to scale down the part of each layer that is not the residual connection. Could you provide some explanation to resolve this paradox?**\n\n**A1.** Thanks for your great question. According to our theoretical analysis, the Jacobian matrix of a full Transformer layer—including both residual and non-residual components—tends to approximate the identity matrix. This means the entire layer contributes minimally during the backward pass. While scaling down non-residual part will amplify the contribution of the residual part, the Jacobian matrix of the whole layer becomes non-identity, therefore, making more meaningful contributions to training. \n\n\n**Q2. line 243: the reference to Figure 3 is not right. I couldn’t find what the right referent should be.**\n\n**A2.** Figure 3 is the illustration of Pre-LN and LayerNorm Scaling. Figure 3(a) is the Pre-LN and Figure 3(b) is the LayerNorm Scaling. We will revise the reference in the text to clarify this in the final version.\n\n\n\n**Q3. Since each layer norm has a scaling factor built into it (usually called ), why couldn't it learn the  factor automatically?**\n\n**A3.** Thank you for the insightful question. We appreciate your attention to the role of the learnable parameter $\\gamma$ in LayerNorm. Our analysis reveals that $\\gamma$ increases the output variance by amplifying the magnitude of outliers in LLM activations—a phenomenon also observed in Figure 1 of [1]. In other words, $\\gamma$ exacerbates the \"curse of depth\" in deep networks.\n\nOur plausible explanation is that Pre-LN together with skip connection is designed to improve the stability of deep Transformers training. However, it tends to take the short-cut and driving the network toward an extreme behavior, i.e., causing deeper layers to approximate identity functions. While this enable training very deep Transformers (i.e., allowing many layers to be stacked without vanishing gradients), it comes at the cost of reduced expressiveness and diminished learning capacity in deeper layers.\nTo mitigate this issue, we propose explicitly scaling down the output variance, enabling more effective learning in deeper layers.\n\n\n**Q4. In Figure 2b, what’s the reason for the spike around layer 27-28?**\n\n**A4.** This is due to the distinct roles played by different layers in the network. The last few layers are responsible for transforming high-level representations into final outputs (e.g., classification logits). As a result, any error or disruption in these layers directly affects the model’s performance, making it less tolerant to their removal compared to the middle layers. In contrast, the middle-to-deep layers often contain considerable redundancy, allowing the representational loss from their removal to be partially compensated by adjacent layers.\n\n\n\n\n**Q5. In Eq (23), $\\sigma_W^2$ should be $\\sigma_W^4$ and $\\sigma_W$ should be $\\sigma_W^2$ (right?)**\n\n**A5.** Thank you for your careful observation. Equation (23) characterizes the relationship between $x_\\ell$ and $x^\\prime_\\ell$, rather than between $x_\\ell$ and $x_{\\ell-1}$. In this case, we are considering the output of a single attention layer. Therefore, the variance and standard deviation terms remain as $\\sigma_W^2$ and $\\sigma_W$, respectively, rather than $\\sigma_W^4$ and $\\sigma_W^2$.\n\n\n**Q6. How do you get from (32) and (33) to (34)? It would help to write out the Jacobian of LN.**\n\n**A6**. Thank you for the comment. This part of the derivation is a simplified version of the analysis presented in [2], specifically Section 3.\n\nSpecifically, from Equation (33), we know that:\n$$\n\\left\\lVert \\frac{\\partial \\mathrm{FFN}(\\mathrm{LN}(x'))}{\\partial \\mathrm{LN}(x')} \\right\\rVert_{2} \\leq \\sigma_1 \\sigma_2 \\left( \\sqrt{d} + \\sqrt{d_{\\mathrm{ffn}}} \\right)^2, \n$$\n\nwhere the right-hand side satisfies the identity $\\sigma_1 \\sigma_2 (\\sqrt{d} + \\sqrt{d_{\\mathrm{ffn}}})^2 = \\lVert W_1 \\rVert_2 \\lVert W_2 \\rVert_2$.\n\nNext, regarding the LN part, the Jacobian matrix of LN can be written as:\n$$\n\\frac{\\partial \\mathrm{LN}(x')}{\\partial x'} \n= \\frac{\\sqrt{d}}{\\lVert x' \\rVert_2} \\left( I - \\frac{x' x'^\\top}{\\lVert x'\\lVert_{2}^2} \\right)\n= \\frac{\\sqrt{d}}{\\sigma_{x'} \\sqrt{d}} \\left( I - \\frac{x' x'^\\top}{\\sigma_{x'}^2 d} \\right)\n= \\frac{1}{\\sigma_{x'}} \\left( I - \\frac{z z^\\top}{d} \\right).\n$$\n\nThe second equation uses $\\lVert x'\\rVert_2 = \\sigma_{x'} \\sqrt{d}$, which can be obtained based on Assumption 1. The last equation is derived from the well-known formula of $z = (x' - \\mu_{x'}) / \\sigma_{x'}$, which converts a normal distribution, $x'$, to the standard normal distribution $z$, where $\\mu_{x'} = 0$ in Assumption 1. \n\n\nWe consider the variance ($\\mathrm{var}$) of each element in the matrix $zz^\\top$. Since $z_i z_i$ follows $\\chi^2$ with 1 degree of freedom, and $z_i z_j$ ($i \\neq j$) is the multiplication of two independent values following the standard normal distribution, the variances are as follows:\n\\begin{align}\n\\mathrm{var}(z_i z_j) = \n\\begin{cases}\n1 & \\text{if } i \\neq j \\\\\\\\\n2 & \\text{otherwise}\n\\end{cases}\n\\end{align}\n\nIt indicates that $\\frac{zz^\\top}{d} \\sim 0$ in LLMs due to $d \\gg 1$. Therefore, the spectral norm of the Jacobian matrix of LN can be written as:\n\\begin{align}\n\\left\\lVert \\frac{\\partial \\mathrm{LN}(x')}{\\partial x'} \\right\\rVert_{2}\n= \\frac{1}{\\sigma_{x'}}, \\quad\n\\text{where} \\quad \n\\frac{\\partial \\mathrm{LN}(x')}{\\partial x'} = \\frac{1}{\\sigma_{x'}} I. \n\\end{align}\n\nFinally, combining the above equations can be rewritten as:\n$$\n\\left\\lVert \\frac{\\partial y}{\\partial x'} \\right\\rVert_{2}\n\\leq 1 + \\frac{\\sigma_1 \\sigma_2}{\\sigma_{x'}} (\\sqrt{d} + \\sqrt{d_{\\mathrm{ffn}}})^2. \n$$\n\n\nAccording to the Assumption 1 and discussion in the paper, the standard deviations, $\\sigma_1$ and $\\sigma_2$, of $W_1$ and $W_2$, respectively, should be sufficiently small, and the standard deviation, $\\sigma_{x'}$, of the shortcut $x'$, should satisfy $\\sigma_1 \\sigma_2 \\ll \\sigma_{x'}$ in order to keep the upper bound small. Here the right-hand side satisfies the identity $\\sigma_1 \\sigma_2 (\\sqrt{d} + \\sqrt{d_{\\mathrm{ffn}}})^2 = \\lVert W_1\\rVert_{2} \\lVert W_2 \\lVert_{2}$.\n\n\n**Q7. How do you get equation (37)? The only justification is a citation to an entire textbook [44].**\n\n**A7.** Equation (37) is also derived from the analysis in [2], specifically from Appendix Equation (53). The analysis employs $\\lVert W\\rVert_2 \\sim \\sigma (\\sqrt{d_{in}} + \\sqrt{d_{out}})$ from Vershynin (2018), High-Dimensional Probability. The statement is located in Chapter 4, Section 4.4, around Theorem 4.4.5 and Exercise 4.4.7 the book. We additionally cite Vershynin 2018 here for completeness.\n\nSpecifically, \n\n\n$$\n\\begin{aligned}\n\\left\\lVert J_{kj} \\right\\rVert_2 \n&= \\left\\lVert \\sum_{l=1}^{L} \\left( \\frac{\\partial A_{kl}}{\\partial x_j} v_l^\\top \\right) + A_{kj} W_V \\right\\rVert_2 \\\\\\\\\n&\\leq \\sum_{l=1}^{L} \\left\\lVert \\left( \\frac{\\partial A_{kl}}{\\partial x_j} v_l^\\top \\right) \\right\\rVert_2 + \\lVert A_{kj} W_V \\rVert_2 \\\\\\\\\n&\\leq \\frac{1}{\\sqrt{L}} \\left( \\delta_{kj} + 1 + \\frac{2}{\\sqrt{L}} \\right) \\sigma^3 \\sqrt{d^3 d_{\\mathrm{head}}} + \\frac{\\sigma}{L} \\left( \\sqrt{d} + \\sqrt{d_{\\mathrm{head}}} \\right)\n\\end{aligned}\n$$\n\n$$\n\\begin{aligned}\n\\left\\lVert J^Z \\right\\rVert_2 \n&= \\left\\lVert \\sum_{k=1}^{L} \\sum_{i=1}^{h} J^i_{kj} W_i \\right\\rVert_2 \\\\\\\\\n&\\leq \\sum_{k=1}^{L} \\sum_{i=1}^{h} \\lVert J^i_{kj} \\rVert_2 \\lVert W_i \\rVert_2 \\\\\\\\\n&\\sim h \\left( \\frac{1}{\\sqrt{L}} \\left(1 + L + \\frac{2L}{\\sqrt{L}} \\right) \\sigma^3 \\sqrt{d^3 d_{\\mathrm{head}}} + \\frac{L\\sigma}{L} \\left( \\sqrt{d} + \\sqrt{d_{\\mathrm{head}}} \\right) \\right) \\\\\\\\\n&= h \\left( \\left( \\sqrt{L} + 2 + \\frac{1}{\\sqrt{L}} \\right) \\sigma^3 \\sqrt{d^3 d_{\\mathrm{head}}} + \\sigma \\left( \\sqrt{d} + \\sqrt{d_{\\mathrm{head}}} \\right) \\right)\n\\end{aligned}\n$$\n\nHere $\\lVert J^Z \\rVert_2$ is the norm of Jacobian matrix and $\\lVert J^i_{kj} \\rVert_2$ is a specific term of $\\lVert J^Z \\rVert_2$. Applying $\\lVert J_{kj} \\rVert_2$ to above equation we can get Equation (37) in our paper.\n\n[1] Wei, Xiuying, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. \"Outlier suppression: Pushing the limit of low-bit transformer language models.\" Advances in Neural Information Processing Systems 35 (2022): 17402-17414.\n\n[2] SPIKE NO MORE: Stabilizing the Pre-Training of Large Language Models (arXiv:2312.16903)."}, {"review_id": "V7FPgDsliw", "reviewer": "Reviewer_tprK", "summary": "The paper identifies and analyzes a phenomenon the authors call the Curse of Depth(CoD) in Pre-LayerNorm (Pre-LN) transformers. Empirically, the top-most blocks of several mainstream LLMs (up to 40 layers in 13 B models) can be pruned with negligible loss, indicating that these layers contribute little to representation learning. The authors attribute CoD to an exploding activation variance along residual paths, which forces the layerwise Jacobian toward the identity and suppresses gradient flow. They prove upper- and lower-bounds on this variance growth, then propose a one-line fix LayerNorm Scaling (LNS), i.e., dividing each LayerNorm output by $1/\\sqrt{\\ell}$. LNS theoretically limits variance to polynomial growth, restores gradient magnitudes, and empirically improves perplexity, downstream accuracy, and the functional importance of deep layers in 130 M–1 B models.", "questions": "1. **Down-stream Metrics on 7B**\n    \n    In Section 5.3, you report perplexity improvements for a 7B (32-layer) model, but no MMLU, ARC-e, or other accuracy numbers. Did you run those tests and, if so, could you share the results? If not, please clarify why only PPL was measured at that scale.\n    \n2. **Ultra-deep Applicability (≥ 80 layers)**\n    \n    The deepest setting evaluated for LayerNorm Scaling (LNS) is 32 layers. Have you analyzed variance curves, pruning sensitivity, and task accuracy on checkpoints in the 80–120-layer range (e.g., LLaMA-2-70 B)? Even a post-hoc fine-tune would help confirm whether the proposed fix remains effective in the truly “deep” regime.\n    \n3. **Tail-layer Importance**\n    \n    The upward “tail” in pruning loss suggests that the last few blocks may perform specialized functions (e.g., output-projection alignment). Do you foresee that a uniform $1/\\sqrt{\\ell}$ schedule could dampen such specialization, and have you measured any task where the very last layer dominates after LNS is applied?\n    \n4. **Resource Foot-print Quantification**\n    \n    One motivation for LNS is to “avoid wasting compute on dead layers.” Could you provide wall-clock time, GPU-memory, and energy usage comparisons between Pre-LN and LNS models at 130 M, 400 M, and 1 B scales?\n    \n5. **Exponent Search and Adaptive Scaling**\n    \n    Your experiments use a fixed $1/\\sqrt{\\ell}$ factor. Did you explore alternative exponents $1/l^\\beta (\\beta\\in[0.3, 0.7])$ or a run-time variance-adaptive schedule? Please report any trends in stability and accuracy.\n    \n6. **RMSNorm ablation and cross-norm mixes**\n    \n    All results keep RMSNorm active. Could you (i) disable RMSNorm and apply LayerNorm + LNS only; (ii) combine LNS with DeepNorm or residual α-scaling; and (iii) report variance and accuracy in each setting? This would isolate the true contribution of LNS versus RMSNorm.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths  \n1. **Clear problem identification**  \n   * Shows that upper transformer blocks add almost no value.  \n   * Targets the industry-standard *Pre-LayerNorm + Residual* architecture, making the diagnosis immediately relevant.  \n\n2. **Tight theory-experiment loop**  \n   * Theorizes exponential activation-variance growth and Jacobian collapse.  \n   * Logs training stats to confirm the variance explosion and gradient/pruning patterns.  \n   * Introduces LayerNorm Scaling ($1/\\sqrt{\\ell}$) and re-applies the same analysis to show variance drops to polynomial growth and deep layers regain utility.  \n\n3. **Minimal, broadly applicable fix**  \n   * One-line code change, no extra parameters.  \n   * Consistently lowers perplexity and boosts accuracy on 130 M–1 B models, with potential to extend to larger checkpoints.  \n\n---\n\n### Weaknesses  \n1. **Limited experimental scale**  \n   * Remedy is tested only up to 1 B (24 layers). Behavior on 70 B–110 B models (80–120 layers) is unverified; the $1/\\sqrt{\\ell}$ schedule may need retuning.  \n\n2. **Incomplete factor isolation**  \n   * All runs keep RMSNorm, so gains from the new scaling vs. RMSNorm stability aren’t disentangled.  \n   * No comparisons with alternatives like DeepNorm or residual α-scaling.  \n   * Mean-field assumptions (Gaussian, independent channels) are not empirically checked late in training.  \n\n3. **Practical trade-offs and reproducibility gaps**  \n   * No data on speed, memory, or energy—so efficiency benefits are unclear.  \n   * Key hyper-parameters (batch size, grad clipping, weight decay) are under-documented.  \n   * Evaluation omits code generation, dialogue, and reasoning tasks where deep context might matter most.", "was_revised": true, "final_justification": "1. **Clear problem identification**  \n   * Shows that upper transformer blocks add almost no value.  \n   * Targets the industry-standard *Pre-LayerNorm + Residual* architecture, making the diagnosis immediately relevant.  \n\n2. **Tight theory-experiment loop**  \n   * Theorizes exponential activation-variance growth and Jacobian collapse.  \n   * Logs training stats to confirm the variance explosion and gradient/pruning patterns.  \n   * Introduces LayerNorm Scaling ($1/\\sqrt{\\ell}$) and re-applies the same analysis to show variance drops to polynomial growth and deep layers regain utility.  \n\n3. **Minimal, broadly applicable fix**  \n   * One-line code change, no extra parameters.  \n   * Consistently lowers perplexity and boosts accuracy on 130 M–1 B models, with potential to extend to larger checkpoints.", "rebuttal": "Thanks for your time and your constructive comments! \n\n**Q1. Limited experimental scale\nBehavior on 70 B–110 B models (80–120 layers) is unverified; the schedule may need retuning. Even a post-hoc fine-tune would help confirm whether the proposed fix remains effective in the truly “deep” regime.**\n\n**A1:** We appreciate the reviewer’s comments. Unfortunately, training models of this size remains prohibitively expensive for academic researchers without access to industry-scale infrastructure. Moreover, since large pre-trained models have been trained with Pre-LN with massive data, it is difficult to directly apply LNS to them in a post-hoc fine-tuning manner.\n\nNevertheless, we conducted extra scaling experiments to provide **strong evidence for the scalability of our approach** through a scaling training of 60M, 150M, 300M, 1B, and 7B parameters under a fixed 20B-token budget. We use the same scaling factor i.e., $1/\\sqrt{l}$, for all models. LNS consistently and substantially outperforms the standard Pre-LN baseline across all model sizes.  These results highlight the robustness and scalability of our method, suggesting that its benefits are likely to persist—and potentially amplify—at even larger model scales.\n\n|  Model Size        | # Tokens | Pre-LN  | LNS  |\n|---------------|----------|--------------|-----------|\n| 60M      | 20B       |   3.67    | **3.56** |\n| 150M     | 20B       |    3.44   | **3.24** |\n| 300M     | 20B       |    3.29   | **3.14** |\n| 1B       | 20B       |   3.15    | **2.85** |\n| 7B       | 20B       |   2.69    | **2.50** |\n\n**Q2. Incomplete factor isolation.\nAll runs keep RMSNorm, so gains from the new scaling vs. RMSNorm stability aren’t disentangled...**\n\n**A2:** We want to clarify that the gain is indeed achieved by our LayerNorm Scaling **not by RMSNorm**, since RMSNorm is used for all approaches. As you requested, we further disable RMSNorm and apply LayerNorm with LNS. We can see that the standard LayerNorm+LNS also outperforms LayerNorm only, isolating the true contribution of LNS versus RMSNorm. LNS effectively scales down the output variance of LayerNorm from 67 to 11. \n\n\n| Model        | Method         | Perplexity | Variance (layer12) |\n|--------------|----------------|------------|------------|\n| LLaMA-130M   | Pre-LN (RMS)         | 26.73    |   71.00\n| LLaMA-130M   | Pre-LN (RMS) + LNS   | **25.76**    |    12.13\n| LLaMA-130M   | Pre-LN (LayerNorm)   | 26.71   | 67.00\n| LLaMA-130M   | Pre-LN (LayerNorm) + LNS   | **25.74**    |11.19\n\n**Combining LNS with Other Methods.** DeepNorm inherently avoids variance explosion since its outputs are always normalized. Consequently, applying LNS on top of DeepNorm causes training divergence. In contrast, α-scaling does not address variance explosion, whereas LNS effectively mitigates it, resulting in improved performance.\n\n\n| Model        | Method         | Perplexity | Variance (layer12) |\n|--------------|----------------|------------|------------|\n| LLaMA-130M   | DeepNorm    | 27.17 |  1.00\n| LLaMA-130M   | DeepNorm + LNS   | 1335.92 |    0.09\n| LLaMA-130M   | Pre-LN (RMS)   | 26.73    |   71.00\n| LLaMA-130M   | Pre-LN (RMS) + α-scaling     | 26.69   | 71.50\n| LLaMA-130M   | Pre-LN (RMS) + α-scaling + LNS | **25.73**   |11.44\n\n**Q3: Mean-field assumptions (Gaussian, independent channels) are not empirically checked late in training.**\n\n**A3:** We acknowledge that the proofs of Lemma 1 and Theorem 1 are based on standard idealized assumptions (Gaussian, independent channels). This setting is widely used in statistical machine learning analysis, including Takase et al. (2023) and Xiong et al. (2020) that we rely on. It allows us to precisely analyze the effect of variance scaling. It captures key behaviors relevant to gradient propagation, and extending to more general settings is an important direction for future work.\n\n\n**Q4. No data on speed, memory, or energy—so efficiency benefits are unclear. One motivation for LNS is to “avoid wasting compute on dead layers.”**\n\n**A4:** By \"avoiding wasting compute on dead layers,\" we **do not** mean that our primary goal is to reduce computational cost—a topic extensively explored in prior work on layer pruning [1,2,3]. Instead, our objective is to ensure that deeper layers contribute more effectively to learning, so that the substantial resources used for training these layers are not wasted, ultimately leading to stronger and more capable models. This is supported by our layerwise similarity analysis in our paper, which shows that LNS improves the feature diversity across layers, as well as the stronger evaluation performance achieved by LNS.\n\nApart from this, LNS is a simple scaling operation applied after LayerNorm, introducing almost no additional computational overhead compared to standard LayerNorm. \n\n[1] Men, X., Xu, M., Zhang, Q., Wang, B., Lin, H., Lu, Y., Han, X. and Chen, W., 2024. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853.\n\n[2] Gromov, Andrey, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. \"The unreasonable ineffectiveness of the deeper layers.\" arXiv preprint arXiv:2403.17887 (2024).\n\n[3] Yang, Y., Cao, Z. and Zhao, H., 2024. Laco: Large language model pruning via layer collapse. arXiv preprint arXiv:2402.11187.\n\n**Q5. Key hyper-parameters (batch size, grad clipping, weight decay) are under-documented.**\n\n**A5.** We closely follow the training configurations used in prior studies [1,2], ensuring a fair and consistent comparison across all baselines. A detailed description of the hyperparameters has been added in the revised version. \n\n[1] Zhao, Jiawei, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. \"Galore: Memory-efficient llm training by gradient low-rank projection.\" ICLM 2024.\n\n[2] Li, P., Yin, L. and Liu, S., 2024. Mix-ln: Unleashing the power of deeper layers by combining pre-ln and post-ln. ICLR 2025.\n\n**Q6. Evaluation omits code generation, dialogue, and reasoning tasks where deep context might matter most.**\n\n**A6.** Thank you for your insightful question. Due to our limited computational resources, we pre-trained our models on a relatively small number of tokens, which is insufficient to achieve competitive performance on formal reasoning tasks such as code generation or mathematical problem solving. \n\nNevertheless, it is well-established that stronger base models tend to exhibit better reasoning capabilities. Moreover, model depth plays a crucial role in reasoning, as highlighted by the Physics of Language Models paper (https://arxiv.org/pdf/2407.20311). Our proposed LNS improves the effectiveness of deeper layers, thereby enhancing the model’s capacity for complex reasoning. We are confident that our approach can lead to improved performance on formal reasoning benchmarks when applied to larger-scale models.\n\n**Q7. Down-stream Metrics on 7B\nIn Section 5.3, you report perplexity improvements for a 7B (32-layer) model, but no MMLU, ARC-e, or other accuracy numbers.**\n\n**A7:** As you requested, we further evaluate the zero-shot performance of our models on various reasoning tasks. The results show that the improvements in PPL effectively transfer to downstream metrics. Our method delivers a notable average improvement of 2.1%, demonstrating its practical benefit beyond pre-training loss/PPL.\n\n| Model/Metric   | ARC-easy | COPA | HellaSwag | PIQA | Winogrande | SciQ | **Average** |\n|----------------|---------|------|-----------|------|------------|------|------------|\n| **Pre-LN (7B)**| 59.8    | 71.0 | 55.6      | 71.7 | 53.9       | 84.8 | **66.1** |\n| **LNS (7B)**   | 61.4    | 76.0 | 57.1      | 74.8 | 55.2       | 84.9 | **68.2** |\n\n**Q6. Tail-layer Importance\n Do you foresee that a uniform schedule could dampen such specialization, and have you measured any task where the very last layer dominates after LNS is applied?**\n\n**A6.** Very good questions! We agree with you that the last few blocks perform specialized functions that are crucial for the performance. However, our uniform scaling is designed primarily to enhance the contribution of these layers. As a result, these final blocks actually become more crucial, rather than being suppressed. To verify this, we conducted an additional experiment where we excluded the last three layers from scaling. The results (shown below) indicate that this “no-scaling” variant performs worse than uniform scaling with $1/\\sqrt(l)$, although it still provides some improvement over the standard Pre-LN baseline.\n\n\n| Model        | Method           | Tail Layer Scaling  | Perplexity |\n|--------------|------------------|---------------------|------------|\n| LLaMA-130M   | Pre-LN           | ✗                   | 26.95      |\n| LLaMA-130M   | Pre-LN + LNS     | ✗ (last 3 layers)   | 26.61      |\n| LLaMA-130M   | Pre-LN + LNS     | ✓ (all layers)      | **25.76**  |\n\nRegarding your second question, our preliminary results show that the middle and deep layers indeed become more important for reasoning-intensive tasks such as GSM8K. However, we do not observe that the last few layers can dominate over the first few layers, likely because the model still uses a Pre-LN architecture, even though LNS enhances the effectiveness of deeper layers.\n\n**Q7. Exploring alternative exponents $1/l^\\beta, \\beta \\in[0.3, 0.7]$ or a run-time variance-adaptive schedule?**\n\n**A7.** As you requested, we run experiments with $\\beta=0.3$ and $\\beta=0.7$. The results below show that $\\beta=0.5$ (our setting) achieves the best performance. \n\n| Model        | Method         | Exponent        | Perplexity |\n|--------------|----------------|-----------------|------------|\n| LLaMA-250M   | Pre-LN         | -               | 21.92      |\n| LLaMA-250M   | Pre-LN + LNS   | 0.3             | 20.51      |\n| LLaMA-250M   | Pre-LN + LNS   | 0.5 (ours)    | **20.35**  |\n| LLaMA-250M   | Pre-LN + LNS   | 0.7             | 20.99"}, {"review_id": "UmvR60ntQx", "reviewer": "Reviewer_L6Vr", "summary": "Several researchers have indicated that deep layers in LLMs are ineffective and have limited contribution to their accuracy. This paper analyzes the cause of this ineffectiveness and attributes it to pre-layer norm (i.e., placing layer norm or RMSNorm before attention and FFN). Then the paper proposes placing a simple scaling after each layer norm, and shows that it makes deeper layers more effective and increase overall accuracy of the model.", "questions": "- Equation 5: I didn't understand how this equation is derived. a) What is the relationship between Pre-LN(x) and LN(x)? b) why do we have an identity, I, here?\n- In preliminary analysis, drop in SQuAD on BERT seems to be small, like mostly less than 1%, while drop in MMLU on other models seems to reach almost 50%. Wondering if we can verify with another task on BERT to ensure a significant variance in accuracy.\n- Line 185: Even though the authors cite the experimental setup from another paper, I believe readers would appreciate if authors provide a few more details like what training set is used.\n- Table 1: Please specify what evaluation data set is used to measure perplexity, and what is the context length when measuring it.\n- Paper would be strong if authors could train on an architecture with MoE (e.g., Qwen3 or DeepSeekv3)\n- Line 310: \"Admin introduces additional parameters to control residual dependencies, stabilizing Post-LN.\": Missing citation?\n- Equation 19: please define n right after the equation \n- I didn't understand how we derive the 3rd step in Equation 23\n- if possible, it would be great if authors could do an early exit analysis similar to Figure 2 of LayerSkip paper ( https://arxiv.org/abs/2404.16710 ) to compare a model trained with LN vs. LN Scaled", "limitations": "The author did not mention any limitations. However, I can't seem to think of a limitation.\nMaybe one limitation is that auhtors did not test on a MoE model, or VLM model.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strength:\n- insights backed by rigorous mathematics\n- hyperparameter-free\n- easy to implement\n- confirmed to be stable in training (unlike other approaches proposed in literature)\n- provided results training LLMs with various architectures, sizes, training budgets as well as ViTs\n- Compared with a wide range of initialization / layernorm modification techniques and showed superior accuracy\n\nWeaknesses:\n- Contribution is simple", "was_revised": true, "final_justification": "I would like to thank the authors for their rebuttal. I have read all the reviews and rebuttals and would like to keep my score.", "rebuttal": "We really thank the reviewer for recognizing the merits of our submission and valuable support! We address your remaining comments below. \n\n**Q1. Equation 5: I didn't understand how this equation is derived. a) What is the relationship between Pre-LN(x) and LN(x)? b) why do we have an identity, I, here?**\n\n**A1.** Equation (5) expresses the derivative of a residual block in a Pre-LN Transformer. In this setting, the output of each residual block can be abstractly written as: $\\text{Pre-LN}(x) = x + f(\\text{LN}(x))$,\nwhere f denotes the transformation function (e.g., attention or feed-forward sub-layer), and LN denotes the LayerNorm applied before f. Taking the derivative with respect to x, we apply the chain rule:\n$\\frac{\\partial\\, \\text{Pre-LN}(x)}{\\partial x}\n= \\frac{\\partial}{\\partial x} \\left[ x + f(\\text{LN}(x)) \\right]\n= I + \\frac{\\partial f(\\text{LN}(x))}{\\partial \\text{LN}(x)} \\cdot \\frac{\\partial \\text{LN}(x)}{\\partial x}$. \n\nOur analysis finds that this derivative of deep layers in LLMs tends to approximate identity functions. \n\n\n**Q2. In preliminary analysis, drop in SQuAD on BERT seems to be small, like mostly less than 1%, while drop in MMLU on other models seems to reach almost 50%. Wondering if we can verify with another task on BERT to ensure a significant variance in accuracy.**\n\n**A2.** As you requested, we evaluate BERT on SWAG dataset, where the performance drop is more significant. We observe a similar pattern where middle to deep layers suffer more from pruning than the first few layers. \n\n\n| Layer Index   | 0      | 1      | 2      | 3      | 4      | 5      | 6      | 7      | 8      | 9      | 10     | 11     |\n|---------------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|\n| Accuracy Drop | -1.74  | -1.61  | -7.51  | -8.03  | -7.41  | -8.02  | -8.12  | -10.73  | -8.34  | -6.25  | -4.38  | -2.99  |\n\n\n**Q3. Line 185: Even though the authors cite the experimental setup from another paper, I believe readers would appreciate if authors provide a few more details like what training set is used.**\n**A3.** Thanks for pointing out this! We provide the details of our training setting below. \n- Table 1: All models in Table 1 are trained from scratch on the English portion of the C4 corpus (C4-en).  We follow exactly the public training script provided in our supplementary material; the only change across runs is the choice of normalization method.  A global batch size of 512 sequences is used throughout.  For the 130M and 250M models we adopt a peak learning rate of 1 × 10⁻³, while the 350 M and 1B models use 5 × 10⁻⁴.  All other optimizer and scheduler hyper-parameters (Adam, β₁ = 0.9, β₂ = 0.95, cosine decay) remain identical across experiments.\n\n- Table 2: For the MMLU benchmark we fine-tune on its official training split.  The remaining tasks—BoolQ, ARC-easy, PIQA, HellaSwag, OBQA and Winogrande—are fine-tuned on CommonSense-170K following [1,2,3], which aggregates roughly 170k supervised examples covering these datasets.  To stabilise training, the learning rate is reduced to 1/10 of the values used during pre-training (i.e., 1 × 10⁻⁴ for < 350M models, 5 × 10⁻⁵ for larger models). Batch size is 16. \n\n- Table 3: The configuration mirrors Table 1 in every respect—same C4-en corpus, 512-sequence batches, and size-specific learning rates—but we extend the training schedule to accumulate a larger token budget.  No other hyper-parameters are altered, enabling an isolated assessment of how additional tokens (rather than different data or optimisation tweaks) influence the effectiveness of each normalization strategy.\n\n[1] Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E.P., Bing, L., Xu, X., Poria, S. and Lee, R.K.W., 2023. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933.\n\n[2] Li, P., Yin, L., Gao, X. and Liu, S., 2024. Owlore: Outlier-weighed layerwise sampled low-rank projection for memory-efficient llm fine-tuning. ACL 2025.\n\n[3] Li, P., Yin, L. and Liu, S., 2024. Mix-ln: Unleashing the power of deeper layers by combining pre-ln and post-ln. ICLR 2025.\n\n\n**Q4. Table 1: Please specify what evaluation data set is used to measure perplexity, and what is the context length when measuring it.**\n\n**A4.** We follow the evaluation protocol introduced in Galore [1]. Perplexity is computed on the C4 validation split (the “en” portion of the Colossal-Clean-Crawled-Corpus). Each document is chunked into 256-token contexts without overlap, and perplexity is averaged across all resulting sequences.\n\n[1] Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A. and Tian, Y., 2024. Galore: Memory-efficient llm training by gradient low-rank projection. ICML 2024.\n\n**Q5. Paper would be strong if authors could train on an architecture with MoE (e.g., Qwen3 or DeepSeekv3)**\n\n**A5.** We fully agree that applying LayerNorm Scaling to a modern MoE architecture would further validate the generality of our approach. Unfortunately, training a competitive MoE model from scratch exceeds the compute budget and time available during the short rebuttal window. We plan to explore this promising extension after the rebuttal period, when more resources and time are available, and we will include the results in our next version.\n\n\n**Q6. Line 310: \"Admin introduces additional parameters to control residual dependencies, stabilizing Post-LN.\": Missing citation?**\n\n**A6.** Thanks for your detailed comment! The Admin approach was proposed by Liu et al [1]. We will add it in our revision.\n\n[1] Liu, L., Liu, X., Gao, J., Chen, W. and Han, J., 2020. Understanding the difficulty of training transformers. arXiv preprint arXiv:2004.08249.\n\n\n**Q7. Equation 19: please define n right after the equation**\n\n**A7.** We appreciate your careful observation! In Equation (19), both 𝑛 and 𝑁 refer to the number of attention heads. We acknowledge the inconsistency and will revise the notation in the next version to improve clarity and maintain consistency.\n\n\n**Q8. I didn't understand how we derive the 3rd step in Equation 23**\n\n**A8.** Thank you for the question. The third step in Equation (23) uses asymptotic notation. Specifically, the expression $1 + \\frac{\\sigma_W}{\\sigma_{x_\\ell}} + \\frac{\\sigma_W^2}{\\sigma_{x_\\ell}^2}$ is approximated by $\\Theta\\left(1 + \\frac{1}{\\sigma_{x_\\ell}}\\right)$\n\nHere, the notation $\\Theta(\\cdot)$ denotes the dominant growth rate of the variance term defined in Lemma 1, with respect to $\\sigma_{x_\\ell}$. Both $\\frac{\\sigma_W}{\\sigma_{x_\\ell}}$ and $\\frac{\\sigma_W^2}{\\sigma_{x_\\ell}^2}$ decrease as $\\sigma_{x_\\ell}$ becomes large. In the final approximation, we retain only the dominant term and omit higher-order terms for simplicity, resulting in the asymptotic form $\\Theta\\left(1 + \\frac{1}{\\sigma_{x_\\ell}}\\right)$.\n\n\n\n\n\n**Q9. if possible, it would be great if authors could do an early exit analysis similar to Figure 2 of LayerSkip paper ( https://arxiv.org/abs/2404.16710 ) to compare a model trained with LN vs. LN Scaled**\n\n**A9.** We thank the reviewer for the thoughtful suggestion to evaluate our method in combination with LayerSkip. We agree that this is a very intriguing direction. However, applying LayerSkip effectively requires large-scale pretraining or continued pretraining (>=52B), which is unfortunately beyond our computational budget within the limited rebuttal period. We fully acknowledge the potential of combining these two approaches and plan to explore this promising direction in future work."}]

Author rebuttal:

Meta-review: The authors on this work start from a hypothesis that deep layers in LLM are not that effective and have limited contribution to their accuracy. They call this phenomenon 'the curse of depth'. They investigate why this is the case, and attribute this to the much used pre-layer normalization (e.g., putting the normalization before the attention as it is done in pretty much all modern LLMs). Then, they propose a solution/mitigation to this problem by scaling the contribution of layer l by 1/sqrt(l). The authors validate this both theoretically and empirically in some 'small' LLMs (130M–1B parameters).

The reviewers initially gave positive scores to the paper, giving two Borderline Accepts and 2 Accepts. In particular, most reviewers appreciate the paper's simplicity, its mathematical rigorousness and the strong experiments. However, they also find some weaknesses/unclarities that ask the authors to address. In particular:

- Reviewer JTrQ asks authors to do some analysis if this is related to other phenomenon that might make Transformers less efficient such as holistically exploring it with attention saturation and batch sizes, while asking for some other clarifications. The authors positively address these issues. Furthermore, the same reviewer asks for experiments in larger LLMs, and the authors are able to provide some preliminary results on a 7B model. The reviewer finds the arguments convincing and keeps their score.

- Reviewer skvN find some unclarities on Lemma 1 and Theorem 1, to which the authors provide a detailed explanation, that the reviewer finds convincing.

- Reviewer tprK also asks for experiments in larger networks in addition to some other experiments. The authors provide such experiments and the reviewer keeps their score.

- Reviewer L6Vr asks for some clarification on the relation of Pre-LN and LN and asks for some other experiments, that the authors happily provided, and makes the reviewer keep their score.

In general, all the reviewers remain positive about the paper and recommend it to be accepted. After reading the reviews and the paper, I agree with the reviewers and find the paper very interesting and with good results. The only thing missing is a stronger experimental setup in larger LLMs (7B+ parameters), but I acknowledge that not all labs are able to do such experiments. Nevertheless, I think that even without it, the paper easily crosses the threshold of acceptance, and the authors are encouraged to integrate the rebuttal in their camera-ready version of the paper. Congratulations to the authors!
Decision: Accept (poster)