Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
Authors: Jeonghoon Kim
Byeongchan Lee
Cheonbok Park
Yeontaek Oh
Beomjun Kim
Taehwan Yoo
Seongjin Shin
Dongyoon Han
Jinwoo Shin
Kang Min Yoo
Venue: ICML 2025 poster (2025)
Keywords: layer normalization
transformers
architecture
pre-training
Primary area: deep_learning->foundation_models
TL;DR: A comparative analysis of normalization layer in Transformers reveals that Peri-LN—adopted in recent large-scale architecture yet underexplored—effectively balances variance and stabilizes gradients, making it advantageous for large-scale training

Abstract: Selecting a layer normalization (LN) strategy that stabilizes training and speeds convergence in Transformers remains difficult, even for today’s large language models (LLM). We present a comprehensive analytical foundation for understanding how different LN strategies influence training dynamics in large-scale Transformers. Until recently, Pre-LN and Post-LN have long dominated practices despite their limitations in large-scale training. However, several open-source models have recently begun silently adopting a third strategy without much explanation. This strategy places normalization layer **peripherally** around sublayers, a design we term **Peri-LN**. While Peri-LN has demonstrated promising performance, its precise mechanisms and benefits remain almost unexplored. Our in-depth analysis delineates the distinct behaviors of LN strategies, showing how each placement shapes activation variance and gradient propagation. To validate our theoretical insight, we conduct extensive experiments on Transformers up to $3.2$B parameters, showing that Peri-LN consistently achieves more balanced variance growth, steadier gradient flow, and convergence stability. Our results suggest that Peri-LN warrants broader consideration for large-scale Transformer architectures, providing renewed insights into the optimal placement of LN.

Peri-LN:
Revisiting Normalization Layer in the Transformer Architecture
Jeonghoon Kim
Byeongchan Lee? Cheonbok Park 12 Yeontack Oh' Beomjun Kim
Taehwan Yoo* Jinwoo Shin 12 Kang Min Yoo"
Scongjin Shin 1 Dongyoon Han
Abstract
shown remarkable impact (Hoffmann et al., 2022; Guo et al., 2025; Yoo et al., 2024). As the demand for larger and more powerful models grows, various training stabilization tech- niques have been introduced (Yang et al., 2022; Zhai et al., 2023; Loshchilov et al., 2024). Among these, the choice of where and how to apply layer normalization (LN: Lay- erNorm or RMSNorm; Ba et al., 2016; Zhang & Sennrich, 2019) critically influences model convergence (Xiong et al., 2020; Kedia al., 2024; Wortsman al., 2024). However, their immense computational requirements have restricted deeper exploration of the underlying Transformer structure. Are we truly employing the optimal LN placement? In practice, fully revealing the results of massive resource in- vestments can be challenging (Rivière et al., 2024). Despite its importance, there is still no consensus on a single best L.N placement strategy.
Selecting layer normalization (LN) strategy that stabilizes training and speeds convergence in Transformers remains difficult, even for today's large language models (LLM). We present a com- prehensive analytical foundation for understand- ing how different LN strategies influence train- ing dynamics large-scale Transformers. Until recently, Pre-LN and Post-LN have long domi- nated practices despite their limitations in large- scale training. However, several open-source mod- els have recently begun silently adopting a third strategy without much explanation. This strategy places normalization layer peripherally around sublayers, a design we term Peri-LN. While Peri- LN has demonstrated promising performance, its precise mechanisms and benefits remain almost unexplored. Our in-depth analysis delineates the distinet behaviors of L.N strategies, showing how each placement shapes activation variance and gradient propagation. To validate our theoret- ical insight, we conduct extensive experiments on Transformers up to 3.2B parameters, showing that Peri-LN consistently achieves more balanced variance growth, steadier gradient flow, and con- vergence stability. Our results suggest that Peri- LN warrants broader consideration for large-scale Transformer architectures, providing renewed in- sights into the optimal placement of LN.
Two prominent LN placements have been widely explored. Post-LN (Vaswani al., 2017) normalizes the hidden state after adding the sub-layer output to the residual stream (that is, Norm + Module(z)) where æ is input hidden state. Norm is LN). This helps constrain the variance of hidden states but may inadvertently weaken gradient signals, partic- ularly in deeper models (Kedia al., 2024). Pre-LN (Dubey et al., 2024), by contrast, normalizes before passing the hid- den state to the sub-layer (that is, x Module(Normi (z))). While this can enhance gradient propagation, it also admits so-called "massive activations," where hidden states grow exponentially across layers (Sun et al., 2024).
Previous studies on deep convolutional neural networks (CNNs) have analyzed the impact of batch normalization on variance changes during the initialization stage of ResNet architectures, demonstrating its relationship to model per- formance (De & Smith, 2020). They noted that, in models without normalization, hidden activation growth at initial- ization can be exponential, leading to poor performance and stability. contrast, in pre-normalized CNNs, the vari- ance of hidden activations was shown to increase linearly as model depth grows. In the same vein, Kedia al. (2024) reported that, for Transformer architectures as well, the variance in the forward propagation of Transformer-bbseed language models at initialization increases linearly with depth. However, in the context of Transformer architectures, we observed that this variance growth initialization does
1. Introduction
Building on a rapidly expanding lineage of Transformer- based large language models, open-source models have
'Equal correspondence. 'NAVER Cloud Korea Advanced Insti- tute of Science and Technology (KAIST) "NAVER AI Lab. Corre- spondence to: Jeonghoon Kim <jeonghoon.samuu samuel @ emgai.cco>> Jinwoo Shin <jinwoos @ kaist Kang Min Yoo <kang- yoo www.yoooorror..ccooo navercorp.com>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
Post-LN
Pre-LN
1.15
2500
2. Background and Motivation
Initialization
2.18 Takens Trained
2000
Mean of Veriance
1.10
4.28 Takens Trained
1500
The analysis of activation variance at model initialization has long been central to understanding normalization layers and enhancing stability in convolutional neural networks (CNNs) (De & Smith, 2020; He et al., 2016; Brock et al., 2021a). Specifically, De & Smith (2020) showed that batch normalization in residual blocks can bias networks toward the identity function, thereby stabilizing gradients and im- proving overall training dynamics.
6.38 Trained
1000
1.05
500
0.25
0.00
10
20
20 Model Depth (Layer Index)
10
Model Depth (Layer Index)
Figure 1. Illustration of hidden-stace variance across different model depths and training iterations. From initialization through training on 6.3 billion tokens, observe the growth in hidden- state variance for both Pre-INN and Post-I.N architectures. The analysis is based on 1.ra-parrmeeer model. Detailed settings and additional results are provided in Section 5.1.
Similar investigations have emerged for Transformer ar- chitectures, examining how variance propagates and how gradients behave in both post-layer normalization (Post-LN) and pre-layer normalization (Pre-LN) configurations (Xiong et al., 2020; Kedia et al., 2024; Wortsman et al., 2024). Early work comparing Post- and Pre-LN primarily focused only on the initialization stage. Xiong et al. (2020) observed that Pre-LN architectures tend exhibit more stable gradients, but can still encounter issues such as gradient spikes and divergence, especially in deeper models or large-scale pre- training scenarios (Zhai et al., 2023; Wortsman et al., 2024; Fishman et al., 2024; Chung et al., 2024).
not persist as training progresses as shown in Figure 1. Sec- tions 3, 4, 5, and 6 provide a more detailed discussion of these hidden-state growth patterns.
Beyond these two common strategies, Post-LN and Pre-LN, a third LN placement has quietly emerged in large-scale open-source models: applying LN around the sub-layer, i.e., on both its input and output. Although recent open- open-source models (Team et al., 2025; Rivière et al., 2024; OLMo et al., 2024) have quietly adopted such designs and demonstrated promising performance on a large scale, these efforts often appeared isolated, lacking a conceptual unifying framework or a thorough investigation into their benefits. In this pa- per, we coin the term Peri-LA to unify these scattered approaches and highlight an underexplored avenue for sta- bilizing large-scale Transformer training. By dissecting the forward- and backward-pan pass dynamics of each LN strategy, we clarify how, when, and why they differ, interpreting these distinctions through the lens of training stability.
Among these challenges, the phenomenon of "massive acti- vations" has attracted attention (Dettmers et al., 2022; Yu et al., 2024; Fishman et al., 2024). Notably, Sun et al. (2024) identified that in Pre-LN architectures, large spikes in acti- vation magnitude can persist across layers due to residual connections. These massive activations act as fixed biases, potentially narrowing the model's ' focus to certain tokens and may influence generalization. However, the underlying mechanisms behind these large values, and their exact im- pact on the training process, remain not yet well understood.
Analytical work has provided theoretical frameworks to ex- plain phenomena like gradient explosion and vanishing in Transformers. For instance, Kedia et al. (2024) introduced signal propagation theory that details how activation vari- ance and gradient instability can evolve with depth, identi- fying critical factors that impair stability and performance. Recent studies have discussed how Pre-LN architectures can allow large values from Attention or MLP modules to flow unimpeded through residual connections (Csordás et al., 2024; Fishman et al., 2024; Zhai et al., 2023; Worts- man et al., 2024), but the precise impact of this behavior on large-scale training remains insufficiently explored.
Accordingly, this paper revisits LN placement in Trans- formers from both analytical and empirical perspectives. In particular, we:
1. Present an in-depth analysis of Post-LN and Pre-LN in large-scale Transformers, examining how variance and gradient properties evolve beyond initialization.
2. Investigate Peri-LN to understand how normalizing both the inputs and outputs of each module moderates hidden-state behavior during forward and backward propagation, providing a systematic perspective on this underexplored alternative.
These observations underscore the ongoing need to clar- ify how activation dynamics and normalization strategies interact, especially in large-scale training.
3. Provide quantitative evidence on how large activation influences training stability, benchmark performance, and model behaviors.
In response, this work aims deepen our understanding of how normalization strategies influence Transformer training, with particular attention to the emergence of large activa- tions and their implications for stability and performance.
Peri- (Prefix) means "around," reflecting that LN encapsulates the entire sub-layer. (e.g. peripherally)
2

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
3. Normalization Strategies
In this section, we discuss how different placements of layer normalization (LN 2) in Transformer architecture affect both training stability and the statisties of hidden states (activa- tions?) ).
A
B
C
Post-LN
x
X
Attention or MLP Module
Pre-LN
x
X
Peri-LN
/
X
3.1. Post-& & Pre-Normalization in Transtormers
Post-LN. The Post-Layer Normalization (Post-LNN) (Vaswani et al., 2017) scheme, normalization is applied after summing the module's output and residual input:
Figure 2. Placement of normalization in Transformer sub-layer.
studies have also reported large output magnitudes in both the pre-trained Attention and MLP modules (Dehghani et al., 2023; Wortsman et al., 2024; Fishman et al., 2024). To bridge the gap from initialization to the fully trained stage, we extend our empirical observations Figure 1 beyond initial conditions by tracking how these variance trends evolve intermediate points in training.
y/1 Norm [ + Module(z;)),
(1)
where zy is the input hidden state of [-th layer, y1 is the output hidden state of I-th layer, and Module denotes At- tention or Multi-Layer Perceptron (MLP) module in the Transformer sub-layer. Norm denotes normalization lay- such RMSNorm or LayerNorm. It is known that by stabilizing the activation variance at a constant scale, Post- LN prevents activations from growing. However, several evidence (Xiong et al., 2020; Kedia et al., 2024) suggest that Post-LN can degrade gradient flow in deeper networks, leading to vanishing gradients and slower convergence.
During training, we find that Post-LN maintains a roughly constant variance, which helps avert exploding activations. However, as models grow deeper and training progresses, consistently normalizing z + Module(az) can weaken gra- dient flow, occasionally causing partial vanishing gradients and slower convergence. In contrast, Pre-LN normalizes øt hefore the module but leaves the module output unnormal- ized, allowing hidden-state variance to accumulate exponen- tially once parameter updates amplify the input. Although Pre-LN preserves gradients more effectively in earlier stages, this exponential growth in variance can lead to "massive activations" (Sun et al., 2024), risking numeric overflow and destabilizing large-scale training.
Pre-LN. The Pre-Layer Normalization (Pre-LN) (Dubey et al., 2024) scheme, normalization is applied to the mod-
ule's input before processing:
V1 z + Module(Norm((r)
(2)
As for Llama 3 architecture, a final LN is applied to the network output. Pre-LN improves gradient flow during backpropagation, stabilizing early training (Xiong et al., 2020). Nonetheless, in large-scale Transformers, even Pre- LN architectures are not immune to instability during train- ing (Wortsman et al., 2024; Zhai et al., 2023). As shown in Figure 2, unlike Post-LNNwhice places LN at position C C-er-L,,, which places LN only at position A, can lead to a "highway" structure that is continuously maintained throughout the entire model if the module produces an out- put with large magnitude. This phenomenon might be related to the "massive activations" observed in trained mod- els (Sun et al., 2024; Fishman et al., 2024).
Takeaways from Pre-LN & Post-LN. (1) Keeping the Highway Clean: Post-LN's Potential for Gradient Vanishing and Slow Convergence. When layer normalization is placed directly on the main path (Placement C in Figure 2), it can cause gradient vanishing and introduce fluctuations in the gradient scale, potentially leading to instability (Xiong et al., 2020). (2) Maintaining a Stable Highway: Pre-LN May Not Suffice for Training Stability. Pre-LN does not normalize the main path of the hidden states, thereby avoiding the issues that Post-LN encounters. Nevertheless, structural characteristic of Pre-LN is that any large values arising in the Attention or MLP modules persist through the residual identity path. In particular, as shown in Figure 1, the ex- ponentially growing magnitude and variance of the hidden states in the forward path may lead to numerical instability.
3.2. Variance Behavior from Initialization to Training
As discussed by Xiong al. (2020) and Kedia et al. (2024), Transformer models at initialization exhibit near-constant hidden-state variance under Post-LN and linearly increasing variance under Pre-LN.. Most of the previous studies have concentrated on this early-stage behavior. However, Recent
3.3. Peri-Norratiiztion in Transformers
Recent open-source Transformer architectures have placed normalization layers in unconventional placements (Rivière et al., 2024; Team et al., 2025: OLMo et al., 2024). In particular, these models apply an additional normalization
Unless stated otherwise, LN refers to both LayerNorm (Ba al., 2016) and RMSNor (Zhang& & Sennrich, 2019). We use "hidden state" and "activation" interchangeably.
3

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
layer at the module output (Output-LNN,, yet the benefits of this design choice remain unclear. To assess the impact of Output-LN, we analyze the Peri-LN architecture.
in Peri-LN preserves the conditions of Eq. 4, enabling the model's hidden states remain better conditioned. By contrast, the rapid surge in variance observed in Pre-LN can trigger instability during the early stages of training, an effect we quantify empirically in Sections 5 and
The Peri-Layer Normalization (Peri-LN) applies LN twice within each layer- and after the module- and further normalizes the input and final output embed- dings. Formally, for the hidden state xy at layer I:
Peri-LN.
3.4. Stability Analysis in Normalization Strategies
Xiong et al. (2020) showed that, at initialization, Pre-LN exhibits smaller gradient scales at the final layer compared to Post-LN, with respect to model depth. In this study, we broaden our analysis beyond initialization to monitor hidden state variance over the full course of training. Building on the earlier observation that the deepest layer exhibits the largest activations, we focus on this surge in the final layer under the Peri-LN strategy to clarify its impact on training stability. To this end, we analyze stability by examining the gradient norm with respect to the final layer weights in the presence of massive activation. Formal statements and detailed proofs are presented in Appendix C.
1. (Optional) Initial Embedding Normalization:
Yo Norm(x.);
2. Input- & Output-Normalization per Layer:
yt = x + Norm (Normle(Norm(zz)))
(3)
3. Final Embedding Normalization:
yz = Norm(FI),,
Proposition 3.1 (Informal). Let L(-) be the loss function, and let ( denote the weight of the last layer of (LP(()- Let q be the scaling parameter in Norm(-), and let D be the dimension. Then, the gradient norm for each normalization strategy behaves as follows.
where x, denotes the output of the embedding layer, the hid- den input state. Yo represents the normalized input hidden state. Ty denotes the hidden state output by the final layer I of the Transformer sub-layer. This design unifies pre- and norput-noomalization to regulate variance from both ends. For clarity, the locations of normalization layers in the Post-, Pre-, and Peri-I.N architectures are illustrated in Figure
(1) Pre-LN (exploding gradient). Consider the following sequence of operations:
5 Norm(o), a MLP((),,
(5)
Both the latest Gemma (Team et al., 2025; Rivière et al., 2024) and OLMo (OLMo al., 2024) model families, which apply output layer normalization, adopt the same periinormalization strategy. However, neither line of work rigorously examines how this placement constrains variance or mitigates large residual activations. Our study extends these ppun-ed large-scale models by providing both theoretical and empirical insights into the Peri-LN scheme.
then
@C(o)
|1hall
(6)
aw??)
where h ReLU (#W() (EW(++60) +6V). In this case, when massive activation ||h|] occurs, exploding gradient ||C/0W(()) can arise, leading to training instability.
(2) Peri-LN (self-regularizing gradient). Consider the fol- lowing sequence of operations:
Controlling Variance Preserving Gradients. By nor- malizing both the input and output of each sub-layer, Peri- LN constrains the residual spikes commonly observed in Pre-LN, while maintaining a stronger gradient pathway than Post-LN. Concretely, if (Module exhibits near-constant variance So, then
a = ==oom((), MLP((F) Norm(a), 0=
(7)
then
aC(o)
4 4n D |blll
(8)
Var(I141) R Var(z:) + So
(4)
aw!21
||||
where h = ReLU (#W(1) + b()). In this case, even when massive activation ||A]| occurs, Norm(.) introduces damping factor factor ||a]]. which ensures that the gradient norm |ac//w remains bounded.
resulting in linear sub-exponential growth of activations, in contrast to the exponential growth patterns of Pre-LN.
Although Pre-LN and Peri-LN exhibit comparable, roughly linear variance growth at initialization (De Smith, 2020; Xie et al., 2023), their trajectories diverge once training begins. The additional normalization layer (Output-L.N)
The layer- wise amplification documented in $3.2, combined with the bounds in Proposition 3.1, naturally explains the
4

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
4.50
2.5
Perl-LN
Peri-LN
Peri-LN
Minimum Train Loss
Pre-LN
4.25
Pre-LN Post-LN
2.0
Pre-LN
2.9
4.00
Gradient Norm
Post-LN
1.5
3.75
2.8
3.50
2.7
X
400M
3.25
1.58
0.5
2.6
3.28
3.00
0.0
1e.4
5e.4
1e-3
2e-3
5e-3
3
2
Learning Rate
Tokens
1e10
Tokens
1c10
(a) Learning rate exploration
(b) Training loss
(c) Gradient- norm
Figure 3. Performance comparison of Post-, Pre-, and Peri-LN Transformers during pre-training. Figure 3(a) Ilustrates the pre-trat pre-training
loss across learning rates. Pre-training loss and gradient norm of best performing 400M size Transformers are in Figure 3(b) & 3(c).
4.5
5.0
Peri-LN
Peri-LN
Peri-LN
Peri-LN Pre-LN
Training Loss
Pre-LN
Training Loss
Pre-LN
Gradient Norm
Pre-LN
Training Loss
4.5
4.0
2
4.0
3.5
3.5
3.0
3.0
0
0
2
1
2
3
0.00
0.25
0.50
0.75
1.00 1e10
Tokens
1e10
Tokens
1e10
Tokens
1e10
Tokens
(a) Divergence at seed 2
(b) Loss spike at seed 3 (c) Gradient spikes at seed 5
(d) Loss spikes at seed 5
Figure 4. Common case of early stage instability in pre-training. In most of our experiments across different random seeds, the Pre-LN architecture exhibited early-stage instability. Although we initially suspected that a high learning rate might be the root cause, lowering it did not substantially mitigate these issues. By contrast, under the same settings, Peri-LN displayed stable training curves.
gradient spikes, and occasional divergences that arise in Pre-LN during large-scale pre-training. We revisit this phe- nomenon in $4.3. By contrast, the additional normalization in Peri-LN acts as a regfrragulriizing mechanism that damps variance growth, making the architecture less sensitive to large activations and therefore more stable in practice. The formal analysis for Post-LN is deferred to Appendix B.
pre- pre-train the Transformers under each LN strategy. We the M-basllle dataset (Li et al., 2024a), along with the "c1100k base" version of the TikToken tokenizer'. Unless otherwise noted, most training and model configurations follow those of the DCLM experiments(Li et al., 2024a). For normalization layer, we primarily employ RMSNorm. Further details are in Appendix D.
4. Experiments
4.2. Pre- Pre-Training Large Language Models
Figure 3(a) illustrates the pre-training loss across learning rates for models ranging size from 400M 3.2B parame- ters. Notably, the Peri-LN architecture consistently achieves superior loss curves over this entire model size. Since Pre- LN shows best performance at learning rate 2 x 10 across all model size, we set this to the default learning rate for Pre-LN and Peri-LN, Unlike Pre-LN, Post-LNN's appropriate learning rate lies in a lower range, so we provide separate summary in Appendix E.I. In Figures 3(b) and 3(c), we compare the pre-training loss and the gradient each LN strategy's 's best-performing learning rate of 400M size models. The same trend is observed across different model sizes (SE). In particular, when we sweep over train- ing seeds and learning rates, Pre-LN frequently exhibits spikes in the gradient- norm curve, whereas Peri-LN shows comparatively few, thereby supporting Proposition 3.1.
In this section, we provide a comprehensive comparison of Post-, Pre-, and Peri-Layer Normalization (LN) across large-scale Transformer pre-training, instruction-tuning and subsequent evaluations on the language domain.
4.1. Experimental Setting
Excluding the embedding parameters, the model size set to the parameters 400M, 1.5B and 3.2B, respectively. Each model is trained 30 billion tokens. To ensure reliable validation, pre-train each model with five different train- ing seeds in all experiments. We perform a exploration of the learning rates, ranging from 1 x 10-4 to 5 x 10-3 to identify the U-shaped pattern for each LN strategy. The sequence length is set to 8192, and the weight decay co- efficient is fixed at 0.033. We employ Megatron-LM* to
curve
chttps://gitee Com/NVIDIA, Meegaattaaaa-MMM
com/ com/oppena/ttkktok
5

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
Table 1. Average benchmark scores (with standard deviations) across 5 different training seeds for Post-, Pre-, and Larir Normaliza- tion language models. Each model size excludes the embedding parameters. Losy denotes the evaluation loss on random samples of the C4 dataset. Arch. denotes architecture, and Avg. denotes the averaged benchmark score actoss tasks. SFT avg. denotes the averaged
benchmark score across tasks of instruction fine-tuned models. Diverged checkpoints are excluded from the evaluation score computation.
Size
Arch.
ARC-Ease
HellaSwag
PIQA
SIQA
Winogrande
Avg. f
Loss
SFT Avg. f
Post-LN
35.70 +1.09
28.91 10.16
62.26
20.73
34.48
1..4 04
50.88
7..75
42.45
7.46
46.44
400M
Pre-LN
54.87
+1.03
34.17
1100
68.79
+1.34
39.73
+0.59
50.88
22.35
49.69
3.43
49.96
Peri-LN
57.51
10081
37.46
10.34
69.48
10.39
40.64
10.51
52.74
10.67
51.57
3.34
51.96
Post-LN
42.92
+0.93
31.69
=0.41
66.72
40.40
35.84
#0.61
50.30
+1.87
45.49
5.38
48.95
1.5B
Pre-LN
61.51
11.22
39.88
11.53
71.41
10.88
11.23
10.97
54.51
12.07
53.71
3.29
53.89
Peri-LN
66.17
10.21
43,94
10.34
73.63
1.024
42.34
10.83
56.64
10.44
56.55
3.18
56,94
Post-LN
45.30
+3.23
33.59
#0.14
66.45
22.86
35.82
+1.09
51,10
+1.0
46.45
4.43
49.33
3.2B
Pre-IN
65.24
12.32
44.23
22.32
73.86
11.19
42.68
10.07
57.42
42.51
56.69
3.20
57.08
Peri-LN
68.73 10.57
46.99
10.21
74.31 +0.41
43.00 10.73
59.76 10.78
58.56
3.11
59.02
0.4
except for the random seed. In Section 5.1, we examine this failure in greater depth and relate it to Proposition 3.1. The curves in Figure are obtained from 400M models.
Peri-LN
Peri-LN
Gradient Norm
Gradient Norm
0.10
Pre-LN
0.3
Pre-LN
Post-LN
Post-LN
0.2
0.05
4.5. Benchmark Evaluations & Instruction Tuning
0.1
To evaluate how well the pre-training loss aligns with its benchmark performance, we conduct five separate bench- mark evaluations. Furthermore, to investigate instruction- following capabilities under different layer normalization strategies, we conduct additional training using the LIMA dataset (Ouyang et al., 2022; Zhou et al., 2023). Diverged checkpoints are excluded from the evaluation score compu- tation (mostly occurs in Pre-LN). Additional training hyper- parameters for SFT are given in Appendix D.2. As shown in Table 1, Peri-LN consistently demonstrates superior per- formance across all model sizes. Additionally, we note that, beyond the improved scores, the standard deviation of the benchmark results across different training seeds is reduced by more than half with Peri-LN. From this, we observe that Peri-LN helps maintain consistency not only in gradient stability and final loss but also in benchmark performance. For the evaluation loss, we used 10K random samples from the C4 dataset (Raffel et al., 2020). Detailed settings and individual benchmark scores provided in Appendix J.
0.00
0.0
0
2
2
Tokens
1e10
Tokens
1e10
(a) Gradient-norm at seed
(b) Gradient- norm seed 4
Figure 5. Final-layer gradient norms for seeds and 5.
4.3. Early Stage Instability in Pre-Training
Early in pre- pre-training, Pre-LN models consistently show gra- dient spikes, loss surges, and occasional divergence across seeds and scales (Fig. 4). These issues are far less pro- nounced in Peri-LN. We posit that the instability of Pre-LN arises from three factors: (I) the hidden state variance ex- hibits sudden surge from initialization through the early stages of optimization, deviating from the linear trend pre- dicted by Eq. 4 (see $3.3); (2) the exponential growth of hidden state variance across both depth and training steps; and (3) the instability caused by the massive activations (Proposition 3.1). Among these, we highlight the vari- ance growth along the main path as the principal driver of the observed divergence. To corroborate this, Section 6 presents targeted experiments that manipulate weight decay and weight initialization schemes, demonstrating how curb- ing extreme variance mitigates the instability of each LN strategy. The curves in 4(a), 4(b), and 4(c) are from a 400M model, whereas 4(d) corresponds a 1.5B model.
5. Analysis
Despite emerging evidence that Peri-LN outperforms Post- and Pre-LN, key uncertainties remain: How do different LN placements shape hidden-stand statistics and gradient flow ($5.1, $5.2y? What role does the Output-LNN scale parameter 7 play ($5.3)? And why does Peri-LN produce more distinctive representations than its counterparts ($5.4)? The subsections below tackle these questions in turn.
4.4. Gradient Norm of the Final-layer
Motivated by Proposition 3.1. we track the final-layer gradient-norm in two representative runs selected from five training seeds. Figure 5(a), now including the newly added Peri-LN results, confirms the hierarchy reported by Xiong et al. (2020): whenever training remains stable, the gradient norms satisfy Post-LN > Pre-LN > Peri-LN. However, Fig- ure 5(b) shows a run in which the Pre-LN model diverges even though every hyperparameter matches the stable run
5.1. Growth of Hidden State
To examine in greater depth how Peri-LN affects forward propagation, we analyze the absolute magnitude and vari- ance of the hidden states using 1, 0,000 samples from the
6

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
1e4
Mean of Absoulte Mean
40
Peri-LN
Pre-LN
1.5
Pre-LN
Peri-LN
25
Training Tolens(Billion)
Post-LN
Mean of Varience
Post-LN
22
30
1.0
20
10
0.0
0
5
10
15
20
10
15
20
Model Depth (Layer Index)
Model Depth (Layer Index)
(a) Absolute magnitude growth
(b) Variance growth
Pigure 6. Forward hidden state growth patterns for each LN strategy a 1.58-prrreeter Transformer.
1e-4
0.8
Peri-LN
0.4
Peri-LN
Gradlent Variance
Peri-LN
Gradient Variance
Peri-LN
Gradient Norm
Pre-LN
Gradiennt Norm
Pro-LN
Pre LN
Pre-LN
0.6
Post-LN
0.3
Post-LN
Post-LN
Post-LN
0.4
0.2
0.2
0.1
0.0
0.0
0
8
12
16
20
12
16
20
24
0
4
8
12
16
20
24
12
20
24
Layer Index
Layer Index
Layer Index
Layer Index
(a) Grad-norm mood-norm init.
(b) Grad-norn final
(c) Grad-variance init.
(d) Grad-varianee at final
Figure Backward gradient norm and variance 1.5B Post-, Pre-, and Peri-I.N Transformers at initialization (init.) and final training.
Wikitext dataset (Merity et al., 2016). Figure shows how different normalization strategies influence forward-path hidden states over the course of training and across model depth. We observe the same pattern across all models trained with five different random seeds (SF).
layers initialization, potentially causing explosive gradi- in the early phase of training. Both strategies display non-uniform gradient distributions-eille vanishing ex- ploding-af different stages of training. On the other hand, Peri-LN demonstrates a consistent, layer-wise gradient dis- tribution at both initialization and the end of training. By maintaining comparatively uniform gradients with lower variance across layers, Peri-LN avoids the extremes of van- ishing or exploding behaviors. This stability is particularly beneficial in deeper architectures, where balanced gradient flow is essential for effective backpropagation.
Across layers, Post-LN maintains stable hidden state magni- tudes and variances because the main path includes nor- malization layer. In contrast, Pre-LN omits normalization after each attention and MLP sub-layer, so the magnitude and variance of the hidden states grow exponentially after the residual addition. For Peri-LN, which adds an Output- LN, these statistics remain comparatively well controlled. Across training iterations, Post-LN's block-level normaliza- tion continues to suppress large shifts, preventing substantial drift in magnitude or variance. Pre-LN starts with an approx- imately linear variance profile at initialization but escalates exponentially to extremely large values optimization pro- ceeds. Peri-LN again exhibits only moderate fluctuations, owing to Output-LN's consistent regulation of hidden-state statistics. Further discussion appears in Section 6.
5.3. Learnable Parameter y of RMSNorm
To investigate the impact of module output normalization training stability, proposed in the Proposition 3.1, we fix the learnable parameter y of RMSNorm to 1, isolating the effect normalization. As illustrated in Figure 8, adding output normalization to each sub-layer suppresses gradi- ent spikes and lowers the loss relative to Transformers that employ only pre-normalization. Nonetheless, we also con- firm that allowing to be learnale yields slightly better performance. The trend persists consistently across model scales and random seeds. In this experiment, we omit Peri- LN's embedding layer normalization in order to isolate and evaluate the precise role and benefits of output-LN.
5.2. Layer-wise Gradient Norm & Variance
Ensuring uniform gradient flow in large-scale model train- ing is crucial for balanced learning across the entire network (Yang & Hu, 2021; Yang et al., 2024). As shown in Figure 7, Post-LN, gradients decrease as they propagate backward through the layers in the final stage of training, which can lead to vanishing gradients in lower-index layers. In Pre-LN, gradients increase as they propagate backward through the
5.4. Hidden State Representation
To assess hidden state redundancy after training, we employ angular distance (Li et al., 2024b), which quantifies how sim-

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
4.2
Peri-LN
2.98
Peri-LN
1.50
Peri-LN
4.0
Perl-LN (Frozen y)
2.96
Peri-LN (Frozen y)
1.25
Peri-LN (Frozen
Pre-
Pre-LN
Gradient Norm
Pre-LN
3.8
Tranning Loss
2.94
1.00
Training
3.6
2.92
0.75
3.4
2.90
0.50
3.2
2.88
0.25
3.0
2.86
2.5
2.6
2.7
2.8
2.9
3.0
0.00
0
Tokens
le10
Tokens
1e10
Tokens
1e10
Training loss
(b) Loss in the final 5B token interval
(c) Gradient-norm
Figure Freezing learnable parameter y output normalization layer in Peri-LN, we set yy its initial value and keep fixed.
Post-LN
Pre-LN
Peri-LN
0.30
0
2.0
Starting Layer Index
Starting Layer index
0.25
Anguiar Distance
Laver Index
80
Starting Layer
8
0.20
1.5
Overagee
12
0.15
12
16
16
0.10
16
1.0
0.05
20
20
20
20.
0.5
12 16
20
4
8
12
16
20
4
8
12
16
20
0.00
Subsequent Layer Index
Subsequent Layer Index
Subsequent Layer Index
Trained Tokens (B)
(a) After 30B tokens training
(b) Learnable scale yy in Output-LN
Figure 9. Angular distance between hidden states after training. Fig. 9(b) monitor of every Output-LN in Peri-LN during training.
ilar or distinct the layer representations are. As Figure 9a) illustrates, Pre-LN produces markedly more redundant hid- den states than the other variants by the end of training. We attribute this effect to the exponential growth the main residual path in Pre-LN, which diminishes the relative contribution of individual sub-layers. In contrast, Peri-LN retains identity path whose learnable scale begins near and gradually adjusts with depth (Figure 9())), thereby mod- erating redundancy. These observations highlight the role of module-output normalization controlling hidden state similarity. All statistics are computed on 256 random sam- ples from RedPajama- Dattna---aaT1 IT (Computer, 2023). Appendix L. includes additional figures and initialization comparisons.
irrespective of the presence of massive activations, Peri-LN achieves better performance than Pre-LN.
To further probe stability under varying degrees of massive activation, we replicate the previously divergent run (seed 4) and repeat the same weight decay sweep. As Figure 10(b) demonstrates, raising the weight decay coefficient from the baseline 0.033 to 0.33 (a tenfold rise) prevents divergence, providing empirical support for Proposition 3.1. Nevertheless, strong weight decay can stabilize Pre-LN, it still fails to close the performance gap relative to Peri-LN.
6.2. Weight Initialization
As the initialization variance increases, the severity of mas- sive activations rises correspondingly for Pre-LN (Figure 10(c)); the largest variance, the model diverges outright (Appendix G.8.3). Pre-LN therefore displays marked sensi- tivity to its initial conditions. In contrast, across the same range of ablations, Peri-LN's 8 loss curves and activation vari- ances shift only marginally. This hyperparameter inseesstiveee robustness is corroborated by the low downstream standard deviations reported Table 1.
6. Ablation Study
To probe massive activations across conditions, we sweep weight decay coefficient and initialization variance for both Pre- and Peri-LN models, holding other settings fixed. Per- run results and detailed settings are in Appendix G.8.
6.1. Weight Decay
In Figure 10(a), stronger 1.2 regularization markedly lowers the variance curve, confirming that heavier weight decay directly curbs forward-pati -ettrd-path explosions in Pre-LN. In con- trast, the same increase in weight decay reduces Peri-LN's variance growth only marginally. We take the stable run initialized with seed 3 (Table 11) and sweep the weight de- cay coefficient. Table in the Appendix further shows that,
6.3. Additional Results
For brevity, we defer an extensive set of supplementary ex- periments to the appendix. Appendix G reports the core robustness checks: replacing RMSNorm with LayerNorm, varying sequence lengths, reducing pre-training budgets, and ablating embedding-lewe normalization. OLMo2-style
8

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
1000
Weight Decay
10
Weight Decay
Initialization
600
0.033
800
1/10d
0.0033
0.1
LeCun(1/d)
Variance
400
0.033
0.33
Training Loss
LN Method
0.33
Variance
600
10/d
He(2/d)
LN Method
Pre-LN
400
LN Method
200
Peri-LN
Peri-LN
Peri-LN
Pre-LN
200
Pre-LN
5
10
15
20
0
5
10
15
20
Model Death (Layer Index)
Tokens
1e10
Model Depth (Layer Index)
(a) Variance growth by weight decay
(b) Training loss by weight decay (c) Variance growth by weight init.
Figure 10. Effects of weight decay and initialization (init.) on massive activations. 10(a) Strong weight decay relieves the variance
explosion in Pre-LN. 10(b) Strong weight decay (0.33, which is 10x the baseline.) suppresses Pre-LN divergence. 10(c) Smaller-scale
initialization slightly curbs Pre-LN variance, while Peri-LN romains bounded regardless. d denotes the model's 's hidden dimensionality.
Peri-LN pre-training runs appear in Appendix H. Stochastic gradient descent (SGD) baselines are summarized in Ap- pendix I. Alternative LN placements, extending Figure 2, are provided in Appendix G.6. Across all settings, the re- sults consistent with the trends presented in the main Section 4 and 5, further substantiating our conclusions.
10
Global Max |h|
Pre-LN
Peri-LN
10
FP16 Max
7. Implications
0.5
1.0
1.5
2.0
2.5
3.0
Training Tokens
1e10
This section integrates our findings into practical guidance on variance-driven stability and precision constraints in large-scale Transformers.
Figure 11. Evolution of extreme hidden state absolute magnitudes during training. Colored bands trace the range of the global top- 100 absolute activations. Pre-LN (blue) quickly surpasses the FP 16 representable maximum, while Peri-LN (red) stays below it.
7.1. Mitigating Variance-Drived Instability via Peri-LN
for Pre-LN surpasses the FP16 maximum bound as early as 0.5B training tokens whereas Peri-LN (red band) consis- tently remains below this threshold. This pattern, echoing Sun et al. (2024), highlights that choosing FP16 BF16 is not just a hardware preference but is closely linked to how hidden state magnitudes evolve within the model. Earlier work on OPT (Zhang et al., 2022), which was pre-trained using FP16 precision, suggests that the training instabili- ties they observed were likely exacerbated by numerical overflows and gradient pathologies (Proposition 3.1) arising when activations exceeded the representable range of FP16.
Pre-LN, the prevailing normalization strategy, is inherently prone unchecked growth in activation variance ($5), which in turn induces numerical instability during train- ing. Our extensive empirical analysis shows that Peri-LN which normalizes the outputs of the Attention and MLP waalllyeesmarkkelhe curbs this variance and often pre- vents divergence (54 & $6). Proposition 3.1 formalizes how excessive variance amplifies gradient norms, clarifying its causal role in destabilizing large-scale pre-training. In Pre- LN, instability is further exacerbated when the statistical conditions assumed at initialization depart markedly from those observed later in training ($3.3). By contrast, Peri- IN alleviates this discrepancy, thereby improving training stability, and delivering additional performance gains.
8. Conclusion
We explore the placement of layer normalization within the Transformer architecture to better understand its role during training. By systematically comparing Post-LNN, Pre- LN, and newly termed Peri-LN, we highlight their distinct impacts stability, final performance, and optimization dynamics. Our findings suggest that placing LN on module outputs in addition to the Pre-LN can help manage large ac- tivations while preserving beneficial gradient flow, thereby offering a promising balance for stable optimization. By unifying these approaches under the term Peri-LN, we seek to consolidate existing variants and encourage deeper inves- tigation into this underexplored alternative.
7.2. Precision Constraints Imposed by Pre-LN
Both Pre-LN and Peri-LN architectures leave the main hid- den state path unnormalized, so once large values arise in earlier layers, they persist through to later layers. Conse- quently, Pre-LN's additive residual path might generates ac- tivations near beyond the FP16 limit. To gauge how often these values exceed FP16 yet remain within BF16, we track the 100 absolute hidden state values for 3.2B 3.2B-paramm ---aamameter Pre-LN and Peri-LN models. In Figure 11, the blue band

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
Acknowledgements
References
We thank our colleague Jeongin Bae for inspiring the un- derlying motivation for this research. We are also grateful to Jung Hyun Lee, Seonghyeon Kim, and Seunghyun Seo for their valuable assistance during the early stages discus- sions. Finally, we extend our gratitude to Gichang Lee, Lead of the Backbone Mission at NAVER Cloud, for his unwavering support. This work was partly supported by Institute for Information & communications Technology Technology Planning (valuationn(ITP)) grant funded by Artificial
Ba, L. J., Kiros, J. R., and Hinton, G. E. Layer normalization. CoRR, abs/1607 abs/1607.06450, 2016. URL http::/arxiit org/ abs//101/ 101.667700000 06450.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Pro- ceedings of the AAAI conference artificial intelligence, volume 34, pp. 7432-7439, 2020.
Brock, A., De, S., Smith, S. L., and Simonyan, K. High- performance large-scale image recognition without nor- malization. In Meila, M. and Zhang, T. (eds.), Pro- ceedings of the 38th International Conference chine Learning, ICML. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learn- ing Research, pp. 1059-1071. PMLR, 2021a. URL http: http:///ooeddddeeee igss ml r www.pprsssssse /v19/b rock21a a, html.
the Korea (OST Intelligence Graduate School Support (roST) No.RS-2021-11112066 Artificial Intelligence Innovation Hub, No. RS-2024-00509279, Global Al Frontier Lab)
Ma-
Impact Statement
The rapid advancement of Transformer-basee large lan- guage models (LLMs) has enabled remarkable break- throughs in natural language understanding and generation. However, these models also pose significant challenges, in- cluding concerns around safety, bias, and the computational cost associated with large-scale training. As L.MS become increasingly integral to various AI applications, ensuring their stability, efficiency, and accessibility remains a critical research focus.
Brock, A., De, S., Smith, S. I.., and Simonyan, K. High- performance large-scale image recognition without nor- malization. In Meila, M. and Zhang, T. (eds.), Pro- ceedings of the 38th International Conference on Ma- chine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learn- ing Research, pp. 1059-107/ PMLR, 2021b. URL http: http///oeeeeeee/eeo//o//oc wigeeeiiigg..... ml press /v139/vv1//bb rock2la.htm.
Our work addresses these challenges by proposing more stable and cost-effective large-scale training methodology. By improving training efficiency and reducing the associ- ated computational overhead, we lower the barrier to entry for organizations seeking to develop or fine-tune foundation models. This democratization of LLM technology fosters broader participation in Al research and development, accel- erating innovation while mitigating concerns resource concentration in few major players. Given the growing industry focus on optimizing LLM deployment costs, our contributions are particularly relevant in the current Al re- search landscape.
Chung, W., Hong, J., An, N. M., Thorne, J., and Yun, Sta- ble language model pre-training by reducing embedding variability. In Al-Onaizan, Y., Bansal, M., and Chen, Y. (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FI, USA, November 12-16, 2024, pp- 10852- 10863. Association for Computational Linguistics, 2024. URL https / aallaaahhlgggy....a/ wrg.l/oo..o.gg/oooo 2024 em Hip-main... 606.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018. URL https: /wwwivarivvv////////////////aaaa http:/axxxwvioraar..a./aa 5457.
Improving the cost-effectiveness of large-scale training si- multaneously lowers Al's environmental footprint by reduc- ing the vast energy consumption and carbon emissions inher- ent in wfte-offthe--a L.LM development. This efficiency not only aligns with global sustainability goals but also enables smaller research labs and academic groups to pursue cutting- edge AI without prohibitive resource demands, fostering a more inclusive and responsible ecosystem.
Computer, T. Redpajama: An open source recipe to re- produce llama training dataset, 2023. URL https:
http:a/www.m..ggggeeeeooonreee/e ama -ana-atta.
Csordás, R., Irie, K., Schmidhuber, J., Potts, C., and Manning, C. D. Moeut: Mixture-of-experts univer- sal transformers. CoRR, Abb/24405.16033 2024. doi: 10.48550/ARXIV.240...o 00lg//0.4855//a0
URL https: /dii.
10

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
De, S. and Smith, S. L. Batch normalization biases resid- ual blocks towards the identity function in deep net- works. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.. and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, viriual, 2020. URL wttpeeediinnnnseeereeeeeeee nwwrips oc wwwpspercrppeeaeeeeeeeeee paper 9cbcba
12517. URL https: pttttp:://oi.orggg /0org/10..00 10.48550/ark iv 1v.2409.12511
Gao, L.,, Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi A., Foster, C., Golding, L., Hsu, J., Le Noac A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang. E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL https: zenodo. http//ecor///////////ee//oo/ss 2608602.
/2020 72020/hash/eebb3388 6c1881d html.
Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J.. Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., Jenatton, R., Beyer, I.., Tschan- nen, M., Amab, A., Wang. X., Ruiz, C. R., Minderer, M., Puigcerver, J., Evci, U.. Kumar, M., van Steenkiste, S., Elsayed, G. F., Mahendran, A., Yu, F., Oliver, A.. Huot, F., Bastings, J., Collier, M., Gritsenko, A. A., Birodkar, V. Vasconcelos, C. N., Tay, Y., Mensink, T., Kolesnikov, A., Pavetic, F., Tran, D., Kipf, T., Lucic, M., Zhai, X., Keysers, D., Harmsen, J. J., and Houlsby, N. Scaling vision transformers to 22 billion parameters. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Ma- chine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii USA, volume 202 of Proceedings of Machine Learning Research, pp. 7480-7512. PMLR, 2023. URL https: http://pooceeennn....e/// mlr. www.ppresssssee /v22// ehchani Brnghani23.hhtne
Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. Tch, Y. W. and Titterington, M. (eds.), Proceedings of the Thir- teenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguma Resort, Sar- dinia, Italy, May 1 13-15, 2010, volume 9 of JMLR Proceed- ings, pp. 249 256. JMLR org. 2010. URL http: /pr o oceedi ngs. ml pre.sressvve.e 9 /g crot 0a html.
Guo, D., Yang, D., Zhang. H., Song. J., Zhang, R., Xu, R., Zhu, Q. Ma, S., Wang, P., Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12448 2025.
He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Con- ference on Computer Vision, 2015 Santiago, Chile, December 7-13, 2015, pp. 1026-1034. IEEE Computer Society, 2015. doi: 10. 109/ICVV 2015. 123. URL https: /tts:://doo.eee .rg wttpg///..1199//// 11..0.111991000 201...ccc......c..coo..c... 123.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3.int8(): 8-bit matrix multiplication for trans- formers at scale. In Koyejo, S., Mohamed, S., Agar- wal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Ad- vances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL http: //papers. hpapers..ips..oo cc/paper.ppppppeppeeee fiwes /aaee//pppr//2 022 022/haath/c3aa449220554439 Aoo8a-Abstract-tooo
He, K., Zhang, X., Ren, S., and Sun, Identity mappings in deep residual networks. In Leibe, B., Matas, J., Sebe, N., and Welling, M. (eds.), Computer Vision ECCV 2016- 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 Lecture Notes in Computer Science, pp. 630-645. Springer, 2016. doi: 10.10079978--3319-4449---9 URL https: / /oi. org /10 10010007///00 -46493- 0_38.
html. html.
Ding, M., Yang, Z., Hong, W., Zheng, W., Zhou, C., Yin, D., Lin, J., Zou, X., Shao, Z., Yang, H., et al. Cogview: Mastering text-to-image generation via transformers. Ad- vances in neural information processing systems, 34: 19822-19835, 2021.
Heo, J. H., Kim, J., Kwon, B., Kim, B., Kwon, J., and Lee, D. Rethinking channel dimensions to isolate outliers for low-bit weight quantization of large language models. Learning In The Twelfth International Conference Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //operropeeeee v view. www...ee//eeeee wwt.:eet//oori fortt//oru?? f d=JzG7kspjjkkk d=
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama herd of models. arXiv preprint 20X.::2407.21788.. 2024.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D.,, Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W.,
Fishman, M., Chmiel, B., Banner, R., and Soudry, D. Scaling FP8 training to tillion-token 1lms. CoRR, abs/2409.12517 2024. doi: 10.48550/ARXIV .2409.

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
Vinyals, O., and Sifre, L. Training compute-optinal large language models. CoRR, abbs/2233.55556 2022. doi: 11.A8550/ARXII.2233.55555 htt.o/..rg/....55////aaaaa. 4104448555/111 220.a.1............
Gokaslan, A., Zhang, J., Chandu, K. R., Nguyen, T.
Vasiljevic, I., Kakade, S. M., Song, S., Sanghavi, S., Faghri, F., Oh, S., Zettlemoyer. L., Lo, K., El-Nouby, A., Pouransari, H., Toshev, A., Wang, S., Groeneveld, D., Soldaini, I.., Koh, P. W., Jitsev, J., Kollar, T., Dimakis, A. G.G Carmon, Y., Dave, A., Schmidt, L., and Shankar, V. Datacomp-Im: search of the next generation of training sets for language models. CoRR, abs/2406. 1174,, 2024a. doi: 10. 10.48550/ARXIV/ 22w.. 1794. URL ht https: / 7/doi. (too..org/1...855//aa//... http://wwwiiv......... 11794.
URL https:
//doi. org 10.
Kedia, A., Zaidi, M. A., Khyalia, S., Jung, J., Goka, H., and Lee, H. Transformers get stable: An end-to-end signal propagation theory for language models. In Forty-first International Conference on Machine Learning, ICML
2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL httpt:tttttt..... httph:tt/tp/p/eeeeeeeeee http///pppreeee.eeoeeeeee. net/ forum ?id= 3Id=POWa
Li, P., Yin, L., and Liu, S. Mix-ln: Unleashing the power of deeper layers by combining pre-In and post-In. arXiv preprint 2t..::2//21139999 2024h.
Kim, J., Lee, J. H., Kim, S., Park, J., Yoo, K. M., Kwon, S. J., and Lee, D. Memory-efficieen fine-tuning of com- pressed large language models via sub-4-bit integer quan- tization. Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, (eds.), Advances in Neu- ral Information Processing Systems 36; Annual Confer- ence Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL petpl/ppeeee//22////s/////////// aper_files/ www.ffeeeppppeeee (eirffc6ee44b99714accbd wwt.ood-bbbsttttttte ference html.
Loshchilov, I., Hsieh, C., Sun, S., and Ginsburg, B. ngpt: Normalized transformer with representation learning on the hypersphere. CoRR, abs/2410.01131,, 2024. doi: 10. 10.48550AARXI2221000011 241 01131. org/10 org/10.48550//a .
URL https: /doi.
Merity, S., Xiong. C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016.
nips..iipp............ cc/p
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
Lee, J. H., Kim, J., Kwon, S. J., and Lee, D. Flexround:
Learnable rounding based on element-wise division for post-training quantization. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp- 18913-18939, PMLR, 2023. URL hwtphttps/.. : :/aps:///rrroo /procee sings.mlr.cg press/ / w20///22222/2222 httmmm..ee..................m...
OLMo, T., Walsh, P., Soldaini, L.,, Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M., et al. 2 olmo 2 furious. arXiv preprint 10Xiv:2501.00656, 2024.
Ouyang, L.,, Wu, J., Jiang, X.,, Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans,
Lee, J. H., Kim, J.. Yang, J. Y., Kwon, S. J., Yang. E., Yoo, K. M., and Lee, D. LRQ: optimizing post-training quan- tization for large language models by learning low-rank weight-scaling matrices. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associa- tion for Computational Linguisticx: Human Language Technologies, NAACL 2025- Volume 1: Long Papers, Al- buquerque, New Mexico, USA, April 29 May 4, 2025, pp- 7708-7743. Association for Computational Linguisties, 2025. URL https: / ac antlllathoooy...o oow.oorgg//000 2025. naacl - 1onollon..333 393/.
Systems 35: Annual Conference
LA, USA, November 28 December 9, 2022, 2022. URL http: http:/ppprr...ssscccc// e 2222 20/22/2/haas//bbeeffeeeeee333// pap 15500133--bbatractt-oofeeer
html.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5511, 2020.
Li, J., Fang, A., Smymiss G., Ivgi, M., Jordan, M., Gadre, S. Y., Bansal, H., Guha, E. K., Keh, S., Arora, K., Garg, S., Xin, R., Muennighoff, N., Heckel, R., Mercat, J., Chen, M., Gururangan, S., Wortsman, M., Albalak, A., Bitton, Y., Nezhurina, M., Abbas, A., Hsieh, C., Ghosh, D., Gardner, J., Kilian, M., Zhang, H., Shao, R., Pratt, S. M., Sanyal, S., Ilharco, G., Daras, G., Marathe, K.,
Rivière, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupati- raju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M.,
12

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
Ramos, S., Kumar, R., Lan, C. L.., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., Girgin, S., Momchev, N., Hoffmma, M., Thakoor, S., Grill, J., Neyshabur, B., Bachem, O., Walton, A., Severyn, A., Parrish, A., Ahmad, A., Hutchison, A., Abdagic, A., Carl, A., Shen, A., Brock, A., Coenen, A., Laforge, A., Paterson, A., Bastian, B., Piot, B., Wu, B., Royal, B., Chen, C., Kumar, C., Perry, C., Welty, C.,, Choquette- Choo, C. A., Sinopalnikov, D., Weinberger, D., Vijaykumar, D., Rogozinska, D., Her- bison, D., Bandy, E., Wang, E., Noland, E., Moreira, E., Senter, E., Eltyshev, E.,, Visin, F., Rasskin, G., Wei, G. Cameron, G., Martins, G., Hashemi, H., Klimezak- Plucinska, H., Batra, H., Dhand, H., Nardini, I., Mein, J., Zhou, J., Svensson, J., Stanway, J., Chan, J., Zhou, P. Carrasqueira, J., Ijjzii J., Beeker, J., Fernandez, J., van Amersfoort, J., Gordon, J., Lipschultz, J., Newlan, J.. Ji, J., Mohamed, K., Badola, K., Black, K., Millican, K., McDonell, K., Nguyen, K. Sodhia, K., Greene, K., Sjösund, L. L.. Usui, L., Sifre, L., Heuermann, L., Lago, L., and McNealus, L. Gemma 2: Improving open lan- guage models at a practical size. CoRR, abs/2408.001 2024. doi: 10. 10.48855/AARXIV222444 18. URL https: //doi.org 10. 48554 (a1//////xiv.v. 0011888........
Computational Linguisties: Human Language Technolo- gies, Volume I (Long and Short Papers), pp. 4149-4155 Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. N0.18653/11/N19-114
URL ptttps:://alaanthhloo
org /N1 9-1421.
Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A.. Rivière, M., et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L.,, and Polosukhin, I. Attention is all you need, In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008, 2017. URL https: : https: http:////pooceeeesseeeeeeeee www.isg.....niiin.occo...c. /201 / 72017/tash/3f5eee24555466 4a845aa- html.
Wang, A., Singh, A., Michael, Hill, F., Levy, O., and Bow- man, S. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyz- ing and Interpreting Neural Networks for NLP. pp. 353-
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106 August 2021. ISSN 0001-0782. doi: 10. 145/44444474381 URL https:
355, Brussels, Belgium, November 2018. 8. Association for Computational Linguistics. doi: 10.18653/v 1/41--4466 URL https: http:///aaaaaatttaaeeeeo /wgg//////// W18 8 44
//doi. woo.i.org/110 wwt./ogg//oo00
Sap, M., Rashkin, H., Chen, D., Bras, R. L., and Choi, Y. Social iqa: Commonsense reasoning about social interac- tions. In Inui, K., Jiang, J., Ng. V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 4462-4472. Association for Computational Linguistics, 2019. doi: 10. (0.18653/V/DDI9--44 URL attps://doi: org/ 111.//.111113///11111111
Wolf, T., Debut, L.,, Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J. Xu, C., Scao, T. L., Gugger, S., Drame, M.. Lhoest, Q., and Rush, A. M. Transformers: State-of- the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38- 45, Online, October 2020. Association for Computational Linguistics. URL https: www. ac web. org/a nthology 2200 emnlp demos. 6.
Sun, M., Chen, X., Kolter, J. Z., and Liu, Z. Massive activa- tions in large language models. CoRR, Abs/22402.17722 2024. doi: 10.48500AARXIV22222.776. URL https: / //doi. , orgg 10.
Wortsman, M., Liu, J., Xiao, L., Everett, K. E., Alemi, A. A., Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., Pennington, J., Sohl-Dickstein, J., Xu, K., Lee, J., Gilmer, J., and Kornblith, S. Small-scale proxies for large-scale transformer training instabilities. In The Twelfth International Conference on Learning Represen- tations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:/ http://oppeeeiieeeeee httt//formmiddddda0
Takase, S., Kiyono, S., Kobayashi, S., and Suzuki, J. Spike no more: Stabilizing the pre-training of large language models. CoRR, (bs/2311.11603 2023. doi: 10.48550/A RXIV 2XIV URLhttpss https:tppt.... : htt//oo.ooooooo org...0 8550/arXiv 2312. 16903.
Talmor, A., Herzig, J., Lourie, N., and Berant, J. Common- senseQA: A question answering challenge targeting com- monsense knowledge. In Proceedings of the 2019 Confer- ence of North American Chapter of the Association for
Xie, S., Zhang, H., Guo, J., Tan, X., Bian, J., Awadalla,
H. H., Menezes, A., Qin, T., and Yan, R. Residual:
13

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
Transformer with dual residual connections. CoRR, abs/2304. 14802, 2023. doi: 10.48550/ARXIV.22000 2304. 14802. URL httph:t/////mm ott.oo/.rrg1...85550333 1v.2304. 14802.
2024. doi: 10445500AARXXV2200000955. URL https:
/ / /tt.i/oor//.....////o http://1..8555///// 24944.1954.
Yu, M., Wang, D., Shan, Q. and Wan, A. The su- per weight in large language models. arXiv preprint urXiv:2411.0719,, 2024.
Xiong, R., Yang, Y., He. D., Zheng, K., Zheng, S., Xing. C., Zhang, H., Lan, Y., Wang, L., and Liu, T. On layer nor- malization in the transformer architecture. In Proceedings the 37th International Conference on Machine Learn- ing, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 10524-10533. PMLR, 2020. URL http:/ /procee dings. ml press / v111/v111111/ xiong20b. html..
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can machine really finish your sen- tence? In Korhonen, A., Traum, D., and Mårquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791 4800, Florence, Italy, July 2019. Association for Compu- tational Linguistics. doi: 10.18653/v1/P19-1477 URL https: http://aaaatththo hwwp:hhhw..oo...oo111--91991
Yang, G. and Hu, J. Tensor programs IV: feature learn- ing in -nfthite-width neural networks. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 11727-11737. PM......... 2021. URL http: http:://poccceeeeeeeeeeeee mlr. ppress// v139 2339/yang2ic. html.
Zhai, s., Likhomanenko, T., Littwin, E., Busbridge, D., Ramapuram, J., Zhang, Y., Gu, J., and Susskind, J. M. Stabilizing transformer training by preventing attention entropy collapse. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, (eds.). Inter- national Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 40770- 40803. PMLR, 2023. URL https: pptps//dinocceeingss.. mir. www.pprees/v000 httt///haaaaaam......
Yang, G., Hu, F. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs V: tuning large neural networks via zero-shot hyperparameter transfer. CoRR, 2e//2203303466 2022. doi: 10.48550/ARXII 220..22022.22200046666 //doi. . org. wtt.or///.....0.. 48550/4555///// 22444.0.
Zhang, B. and Sennrich, R. Root mean square layer nor- malization. In Wallach, H. M., Larochelle, H., Beygelz- imer, A., d' d'Alché-Bus F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Sys- tems 32: Anmual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 12360- 2019, URL htttp::: wtwpeeiiiinnnnn.nnnneee www.nnnrris... cc/p E47f1e7 html.
URL https:
Yang, G., Yu, D., Zhu, C., and Hayou, S. Tensor programs VI: feature learning in infinite depth neural networks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:/ / view. net. forum? idd 17VDnnpw1..
Yoo, K. M., Han, J., In. S., Jeon, H., Jeong, J., Kang, J., Kim, H., Kim, K., Kim, M., Kim, S., Kwak, D., Kwak, H., Kwon, S. J., Lee, B., Lee, D., Lee, G., Lee, J., Park, B., Shin, S., Yu, J., Back, S., Byeon, S., Cho, E., Choe, D., Han, J., Jin, Y., Jun, H., Jung, J., Kim, C.,, Kim, J. Kim, J., Lee, D., Park, D. W., Sohn, J. M., Han, S., Heo, J., Hong, S., Jeon, M.. Jung, H., Jung, J., Jung. W., Kim, C., Kim. H., Kim, J., Kim, M. Y., Lee, S., Park, J. Shin, J., Yang, S., Yoon, J., Lee, H., Bae, S., Cha, J., Gylleus, K., Ham, D., Hong, M., Hong, Y., Hong, Y., Jang, D., Jeon, H. Jeon, Y., Jeong, Y. Ji, M., Jin, Y., Jo. C., Joo, S. Jung, S., Kim, A. J., Kim, B. H., Kim, H., Kim, J., Kim, M., Kim, M., Kim, S., Kim, Y., Kim, Y., Kim, Y., Ko, D., Lee, D., Lee, H., Lee, J., Lee, J., Lee, J. Lee, J., Lee, M. Y., Lee, Y., Min, T., Min, Y., Moon, K., Oh, H., Park, J., Park, K., Park, Y., Seo, H., Seo, S., Sim, M., Son, G., Yeo, M., Yeom, K. H., and Yoo, W. Hyperclova X technical report. CORR, abs/2404.01
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M. T., Li, X., Lin, X. V., Mihaylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D.,, Koura, P. S., Sridhar, A., Wang, T., and Zettlemoyer, L. OPT: open pre-trained transformer language models. CoRR, abs/2205. 000000 2022. doi: 10.48550/ARXIV.2 205.01068. URL http:::://oi...gg//g/ /arXiv. 0.txxi.2205.00555 01068.
Zhang, Y., Chen, C., Ding, T., Li, Z., Sun, R., and Luo, Z. Why transformers need adam: A hessian perspec- tive. In Globersons, A., Mackey, L., Belgrave, D.. Fan, A., Paquet, U., Tomezak, J. M., and Zhang, C. (eds.), Advances in Neural Information Processing Sys- tems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024. Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL fctp://papene
cttps/cccpppppeeeeeeeeee files/p
14

---

Peri-LN; Revisiting Normalization Layer in the Transformer Architecture
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L.,, Zhang, S., Ghosh, G., Lewis, M., Zettlemoyer, L., and Levy, O. LIMA: less is more for alignment. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL TLLes//paper/222//aahhaaas
.ip
s. ccc/paper sccc//ppeee_fiieee//ppee//////// 642744003344471111114447440000-- t- E-Conference.htii html.
15

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
A. Related Work
Activation Dynamics in Large Language Models. Studies on the distribution and magnitude of activations in deep neural networks have revealed that certain outlier features can significantly affect model behavior and efficiency. Dettmers al. (2022) examined Transformer architectures, highlighting how specific feature dimensions may exhibit unusually large values (outliers) that disrupt quantization and overall system performance. Extending this line of work, Sun et al. (2024) identified the occurrence of "massive Unlike standard outliers, these massive activations remain relatively invariant to different inputs, effectively functioning as implicit bias terms in large language models (LLMs). Notably, such extreme values can skew the self-attention mechanism, causing the model to attend disproportionately to certain tokens. These observations demonstrate that even with standard normalization layers in place, hidden biases may linger in internal representations, underscoring the importance of deeper analyses of activation behavior in LLMs.
large activation values that persist across multiple layers.
Variance Control and Normalization in Convolutional Networks. The interplay between activation variance and training stability has also been extensively explored in convolutional neural networks (CNNs). De & Smith (2020) showed that Batch Normalization (BN) stabilizes the training of residual networks by effectively downscaling activation variance in the residual branches, thereby improving gradient behavior. However, BN imposes certain constraints, such as dependence on batch size and additional computational overhead for estimating batch statistics. Consequently, several normalization-fue free or alternative normalization approaches have been investigated. For instance, Brock et al. (2021b) introduced "Normalizer- FFeee ResNets," which manage activation variance through learnable scaling parameters. This approach achieved competitive performance without relying on BN, highlighting the critical role of effective variance control in fostering stable optimization and strong generalization in CNNs.
Layer Normalization in Transformers. Training stability in Transformer architectures is closely tied to the choice and placement of layer normalization (LN). Xiong et al. (2020) reported that Transformers employing Post-LN structure often suffer from gradient instabilities at initialization, requiring a careful learning-rate warm-up phase to mitigate these issues. In contrast, Pre-LN helps maintain more stable gradients during the early stages of training. However, Kedia et al. (2024) showed that while Post-LN can lead to vanishing or exploding gradients in deep Transformers, Pre-LN may induce hyperbolic gradient growth. These findings illustrate the nuanced trade-offs of normalization placement and draw parallels earlier CNN studies, where careful management of activation variance proved essential for stable deep learning, Ding et al. (2021) introduced Sandwich-La LN in the vision domain for the first time, yet they paid little attention to the structural characteristics and differences that distinguish one LN placement from another. In language domain, several major open-source language models (e.g., Olmo2 (OLMo et al., 2024), Gemma2 (Rivière et al., 2024), and Gemma3 (Team et al., 2025)) already employ a Peri-LN-like structure (see Section 3). Nevertheless, previous technical reports have not explained why this design might be advantageous compared with the more widely studied Pre-LN and Post-LN. By investigating Peri-LNN in detail, we aim to highlight the structural benefits that appear to underlie its empirical success in these implementations.
Gradient Propagation and Depth Scaling Ensuring consistent gradient propagation across many layers is pivotal for stable training in very deep models. Yang & Hu (2021) (Tensor Programs IV) introduced the concept of Maximal Update Parametrization (nP) in the infinite-width regime to preserve feature learning, preventing gradients from collapsing into kernel-like dynamics. Building on this, Yang et al. (2024) (Tensor Programs VI) proposed Depth-juP, which scales residual branches and learning rates according to network depth. Their theoretical analysis indicates that improper depth-dependent scaling leads to vanishing or exploding gradients, ultimately diminishing the diversity of learned representations. These insights highlight the necessity for principled scaling strategies and careful initialization to maintain robust gradient flow in deep architectures.
Summary. Taken together, these studies underscore the importance of managing activation variance and hidden biases achieve stable training and expressive internal representations in modern deep networks. In Transformer-basee models, normalization choice and placement-suck as Post-LN, Pre-LN, or other variants- pplay a significant role in controlling gradient dynamics and overall performance. While Post-LN and Pre-LN have received significant attention, we focus on a comparative analysis that includes Peri-LNN, an alternative normalization placement that has thus far been underexplored but holds potential for enhancing training stability and model performance.
16

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
B. Proposition 3.1 of Post-LN
Proposition B.1. Post-LN (yanishing gradient). Consider the following sequence of operations:
o= Norm((),
(9)
then
0C(
4- D DIDI ||
(10)
aw 1.j x +
where : ReLU (zWO)+ + bO)). Since Post-LN normalizes the hidden state after each residual addition along the main path, the activation norm [h] tends to remain relatively moderate. As a result, in Post-IN, the gradient scale is less sensitive to the magnitude of activations and more significantly influenced by model depth, as previously analyzed by Xiong et al. (2020) and Kedia et al. (2024), leading to vanishing gradients as depth increases.
C. Proof of Theoretical Insight
To support the claim that Peri-LN enhances the stability of training in such cases, we analyze the gradient norm in the final layer. For simplicity, we use RMSNorm and ReLU here. Here, we assume that 7, the scaling parameter used in RMSNorm, positive, and we empirically verified that it remains strictly positive across models of all sizes during training.
Proposition C.1. Consider the following sequence of operations:
I RMSNorm(z) (2),
a ReLU(aW + (()) + b02)
0 I + a.
Then,
OC(o)
h.(p, 4;).
(11)
aw! (2)
where & = ReLU (xW(1) + ((0), 0 := sootmaxx((()) (o), and y is the label (one-hot vector).
Proof. By the chain rule,
@C(o)
OC(0)
do
Oa
X
(12)
@w do
da
ow(22
(1XxD
(b)DDxD
(e:Dx1
(a) It is known that
aC(o)
pk ykk
(13)
Ooy
So,
OC(o)
10 91
D2 U22
ÅD yD
(14)
do
(b) Since a,
do
1.
(15)
da
(c) Recall that
a a: ReI.U(W(()) + (()) + (22)
(16)
For convenience, let
RcLUU(W(() + B())
(17)
17

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
Then, have
Zak
a
Σ
hp
W. Dik + (12) = h4 8k.5
(18)
aw 1
(2)
ow
(2) [.
In vector representation. Thus, by putting these all together,
da
0
h4
0] 0]T,
(19)
aw (2)
where the only nonzero entry is in the j-th component.
OC(o)
holps - yi).
(20)
aw 1
Proposition C.2. Consider the following sequence of operations:
I = RMSNorm (z),
a (eLU(aW(()+
ā RMSNorm(a).
0 x + a
Then,
OC(()
4yvDIM
(21)
aw
|all
ij
where y is the scaling parameter used RMSNorm(.) D the dimensionality, and h ReLU (xW()) + (()).
Proof. By the chain rule,
OC(o)
OC(o)
do
aā
da
aw:22 (2)
do
dā
X
da
X
aw (2)
(22)
(11xD
(b)
(o:DxD
(d:17x
(a) We have
aC(O)
|1i v/
S |2
|Bll
(23)
do
(b) We also have
do
= |||| 1.
(24)
0ā
(c) Recall that
= RMSNorm(a) =
a
(25)
V of +e
Then, 0a is the Jacobian matrix J of RMSNorm()). For brevity, let Ba
1 D
a =
Σ
aax 2.
vv
(26)
D
18

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
Then,
dāp
Op
p.4
Dag
7
dog
Va
(27)
dap
1
+7 ay dag
(28)
Va+e + dog
Va+e
apag
=7.
va + 6p.9 -Y' +
(29)
In matrix representation,
D(a + z)8/2 (a) T (a).
(30)
Va +
A
Then, we have
|A|
Va+e 1
|||||
(31)
va
va+e
and
|B]| =
D(a + E)3/2 (a) (a) (a)
7
?
x Da
70
(32)
+(a+=) 5)3/2
(a + =)3/2
So, we have
2y
2/vD
|| |A - 1IA-BII
|||||
+ |BI
(33)
Va
|a||
(d) Since
aa
00
h4
0]
(34)
aw (2)
1.j
we have
da
||A||-
(35)
aw-22) 1,j
Thus,
OC(o)
2 D |al|
<2 x
|||||
(36)
aw!?
||a||
Proposition C.3. Consider the following sequence of operations:
(eLU(=W(()) + b())) + ((2),
0 I + a,
Ô RMSNomm(o).
Then,
OC(o)
4 4r [
(37)
aw
ij
where y is the scaling parameter used in R.SNNrm((), D is the dimensionality, and h == ReLU (zW(1) + +b()).
Proof.
The proof is analogous the proof of the previous proposition.
19

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
D. Detailed Experimental Setting
In this section, we provide detailed configurations both the pretraining and supervised fine-tuning to reproduce our results.
D.1. Configurations on Pre-Training
The common training settings are provided in Table 2. Embedding settings for the language models are listed in Table 3. For the model architecture, we primarily follow the Llama 3 architecture (Dubey et al., 2024). In the MLP module, we use SwiGLU activations. Additional details regarding the model configurations are shown in Table Note that embedding parameters are excluded from the model size. Unless otherwise noted, most training and model settings follow those of the DCLM experiments (Li al., 2024a). During the pretraining stage, each model was trained under controlled random seed.
Table 2. Common configurations. LR Schedule denotes learning rate schedule.
Global Batch Size
Weight Decay
Iterations
Optimizer
LR Schedule
Warmup
Weight Initialization
256
0.033
14400
Adam
Cosine
10%
0.02
Table 3. Embedding configurations.
Max Position Embeddings
Position Embedding Type
8192
Rope
True
Table 4. Model configurations.
Size
Toyers
Tcheads
dmodel
dhead
400M
24
1024
128
1.5B
24
16
2048
128
3.2B
32
16
2560
160
D.2. Configurations on Supervised Fine-Tuning
To examine downstream task performance after instruction tuning, we employed high-quality LIMA alignment training set consisting of 1, 000 samples (Zhou al., 2023). Our supervised fine-tuning configuration was slightly modified from the original setup of LIMA: we fine-tuned the model for 15 epochs with batch size of 128. The optimizer was Adam with initial learning rate of 1e-5 and a cosine learning rate schedule. We selected the best checkpoints for each model by evaluating OpenBookQA (Mihaylov et al., 2018), CommonSenseQA (Talmor et al., 2019), and the NLI dataset in GLUE (Wang et al., 2018).
20

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
E. Additional Results on Pre-Training Study
E.1. Post-Layer Normalization Architecture & Learning Rate Exploration
In order to identify the optimal performance configuration for Post-LN within the experimental setup, we conducted a learning rate exploration as shown in Figure 12. Because the appropriate learning rate Post-LN fell into much lower range than those for Pre-LN and Peri-LN, treated it separately. For each Post-LN setting, the best learning rate was determined the yielding the lowest final training loss, with the random seed held constant during this selection process.
3.00
Post-LN
L939
2.95
2.90
X
400M
2.85
1.5B
3.2B
2.80 1e-4
2e-4
3e-4
4e-4 5e-4
Learning Rate
Figure 12. Learning rate explorations for Post-LN architectures.
E.2. Best Performing Checkpoints Comparisons of Other Model Sizes
As an extension of Section 4.2, we present below the results for additional model sizes that were omitted previously due to space constraints. Figures 13, we compare the pre- pre-training training loss and the gradient norm curve at each LN strategy's best-performing learning rate of 3.2B and 1.5B size models.
PeriL
Pei-UN
Pre-UN
2.0
Pre
Trahking Loos.
4.0
Gradant Norm
1.5
8.5
1.0
3.0
0.5
0.0
(a) Training loss for 3.2B
Gradient-norm for 3.2B
4.5
2.5
Pori LN Pre UN Post-LN
Peri LN
Pre
Tainin Loos
4.0
Morm
1.5
3.5
3.0
0.5
0.0
Tokens
1c10
Thicers
1c30
(c) Training loss for 1.5B
(d) Gradient-norm for 1.5B
Figure 13, Performance comparison of Post-LN, Pre-LN, and Peri-LN Transformers during pre-training for other two.
21

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
F. Additional Results on Growth of Hidden State
In this section, we examine the 400M- and 3. -.aaamarreeerr models, which were omitted in Section 5.1 due to space constraints. As illustrated in Figures 14 and 15, these models exhibit the same overall trend.
30
Mean of Absoulte Mean
100
Peri-LN
Peri-LN
25
Pre-LN
Pre-LN
80
Post-LN
Mean of Varlance
Post-LN
mmm aasslaaa
21
17
60
13
40
20
Model Depth (Layer Index)
Model Depth (Layer Index)
(a) Absolute magnitude growth
(b) Variance growth
Figure 14. The forward growth patterns of hidden state for different architectures highlight the structural impact of normalization placement. 3.2B size model.
1e3
30
Peri-LN
Mean of Absoulte Mean
Peri-LN
Pre-LN
25
Pre-LN
Tramin ng Tokens
20
Post-LN
Mean of Variance
Post-LN
21
17
13
10
5
10
15
20
10
15
20
Model Depth (Layer Index)
Model Depth (Layer Index)
(a) Absolute magnitude growth
(b) Variance growth
Figure The forward growth patterns of hidden state different architectures highlight the structural impact of normalization placement. 400M size model.
22

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
G. Additional Experimental Results Ablation Study
G.I. Amount Training Tokens
In order to investigate whether the learning behavior of each LN strategy varies with the number of training tokens, we conducted an additional round of learning-rate exploration for both the Pre-LN and Peri-LN architectures. As shown in Figure 16,
the number of training tokens increases, there is observable shift in the optimal learning-rate range. Based these findings, conclude that overall results remain consistent, even when the training token count is further increased. Furthermore, although learning rate of x 10-3 leads to divergence in the smaller-scale experiments with SB or 16B training tokens, it does not do so in the 30B-token setting. We attribute this discrepancy to the 10% warmup rate, suggesting that the warmup phase may be insufficient for the smaller-scale experiments.
3.0
Peri-LN
Minimum Train Loss
Pre-LN
2.9
2.8
30B
16B
2.7
x
8B
5e-4
1e-3
2e-3
5e-3
Learning Rate
Figure 16. Learning explorations of Pre-& Peri-L.N architecture with sequence length 2048 configuration.
G.2. Sequence Length
In language models, the number of iterations per token is influenced by the sequence length, which in turn, along with the batch size, affects training statistics. We conducted experiment determine whether the performance trend changes when the sequence length is reduced from 81 8192, as set in the main text, to 2048. As shown in Figure 17, Peri-LN still surpasses Pre-LN in the learning rate exploration.
2.74
Minimum Train Loss
2.72
2.70
2.68
Peri-LN
2.66
Pre-LN
-
10
Learning Rate
Figure 17. Learning rate explorations Pre-& Peri-LN architecture with sequence length 2048 configuration.
23

---

Peri-LN; Revisiting Normalization Layer in the Transformer Architecture
G.3. Warm-up
Warmup widely recognized to influence training stability. To investigate whether 10% warmup rate might unfairly disadvantage Pre-LN, conducted additional learning-rate exploration using 30% warmup rate. As illustrated in Figure 18, the overall trend remained unchanged, and Peri-LN continued to exhibit better performance than Pre-LN in terms of loss. Furthermore, we observed that increasing the warmup rate from 10% to 30% did not reduce the frequency of gradient norm spikes in Pre-LN.
2.95
Peri-LN
2.90
Pre-LN
2.85
2.80
2.75
2.70
X
400M
1.5B
2.65
5e-4
1e-3
2e-3
5e-3
Learning Rate
Figure 18. Learning rate explorations Pre-& Peri-I.N architecture with warmup 30% configuration.
G.4. RMSNorm & LayerNorm
As illustrated Figure 19, we conducted experiments which RMSNorm and LayerNorm were interchanged. Consistent with the findings reported in (OLMo al., 2024), we did not observe any notable performance differences our RMSNorm and LayerNorm replacement experiments. Learning rate to 2e-3 (best performance learning rate).
4.50
2.5
RMSNorm
RMSNorm
4.25
LayerNorm
2.0
LayerNorm
Training Loss
4.00
Gradient Norm
1.5
3.75
3.50
1.0
3.25
0.5
3.00
0.0
1
2
1
3
Tokens
1e10
Tokens
1e10
(a) Pre-training loss curve
(b) Gradient-nowe norm curve
Figure 19. LayerNorm vs. RMSNorm on Peri-LNN architecture. 400M size model.
24

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
G.S. Embedding Layer Normalization of Peri-Layer Normalization Transformers
Motivated by Takase al. (2023), we empirically explore the addition of embedding layer normalization improve training stability and overall model performance in Transformer architectures. As illustrated in Figures 20, 21, and 22, incorporating Embedding LN in the Peri-LN architecture yields slight improvement in pre-training loss. Furthermore, empirical observations suggest that this effect becomes more pronounced in smaller models.
2.5
2.94
Peri-UN w/o Embedding LN
Peri-LN w/o Embedding LN
Peri-UN
Peri-LN
2.92
2.5
Gradient
2.90
1.0
2.88
0.5
2.86
2.5
2.6
2.7
2.8
2.9
3.0
0.0
0
1
1e10
1c10
Pre-training loss
(b) Gradient-norm
Figure 20. Loss and Gradient- norm curves comparing the presence and absence of Embedding LN in the Peri-LN architecture, 400M size model.
2.5
2.74
Peri-LN w/o Embedding LN
2.0
Peri LN
2.72
1.5
Training
Gradient
2.70
2.68
Peri-LN w/o Embedding LN Peri-LN
0.5
2.68
2.5
2.6
2.7
2.8
2.9
3.0
0.0
0
1
Tokens
1e10
Tokens
1e10
(a) Pre-training loss curve
(b) Gradient-norm curve
Figure 21. Loss and Gradient-norm curves comparing the presence and absence of Embedding LN in the Peri-LN architecture. 1.5B size model.
2.68
Peri-LN Embedding
Peri-LN w/o Embedding LN Peri-LN
2.66
Peri-LN
2.0
Training Loos
Gradient Worm
1.5
2.64
1.0
2.62
0.5
2.60
2.5
2.6
2.7
2.8
2.9
3.0
0
Tokens
1e10
Takens
1e10
(a) Pre-training loss curve
(b) Gradient-norm curve
Figure 22, Loss and Gradient- norm curves comparing the presence and absence of Embodding LN in the Peri-LN architecture. 3.2B size model.
25

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
G.6. Ablation Study Additional Normalization Layer Placement
We additionally conduct further experiments on LN placements to compare different combinations (referred as A, B, and C positions in Figure 2). We add configurations where LN is placed at both A C (akin to combining Pre- and Post-LN), as well as only at B, to compare them with Peri-LN final training loss under the controlled same training seed. We pre-train the 400M-parameter Transformers on 30B tokens each, using the same training configurations described in Section 4. As aligned with Xiong et al. (2020), our new results confirm that placing LN exclusively at C leads to training instability or suboptimal performance. In particular, the + C configuration inherits characteristies of Post-LN (large gradient norm shifts), forcing the use smaller learning rates and still resulting in lower overall performance than Peri-LN architecture.
Table 5. Final training loss and additional normalization layer placement.
400M Final Training Loss
A+C
Post-LN
Peri-LN
3.01
3.05
Diverged
2.91
G.7. Peri-LN with QK-Norm
While Peri-LN alone provides robust training dynamics, QK-Norm can still enhance performance, We conducted additional experiments that confirm combining Peri-LN with QK-Norm yields slight improvements in training loss. We pre-train the 1.5B -ar--pareetter Transformers on 30B tokens each, using the same training configurations described in Section 4. As shown Table 6, adding QK-norm to Peri-LN indeed yielded better performance, consistent with Wortsman et (2024). this experiment, the Peri-LN variant equipped with QK-norm used LayerNorm instead of RMSNorm.
Table 6. Peri-LN with QK-Norm.
1.5B
Peri-LN +QK-Norm (Wortsman et al., 2024)
Final Training Loss
2.722
2.711
26

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
G.8. Weight Decay and Weight Initialization
G.8.1. COMMON SETTINGS
We pre-train the 400M parameeer Transformers on 30B tokens each under the controlled same training seed. We measure the training loss and averaged benchmark score for these experiments under the same evaluation settings used in Table I of the paper. Other configurations follow those outlined in Section 4. For the variance growth experiments in Figure 10, we adopt the same settings in Section 5.1, except that we use 100 samples for the forward-pass statistics.
G.8.2. WEIGHT DECAY
We conduct additional studies for various weight decay condition for both Pre-LN and Peri-LN architectures, As shown in the Table 7, Peri-LN continues to offer better performance than Pre-LN under the same settings. We provide per-run results as below:
Table Effect of weight decay on
Pre-LN and Peri-LN Transformers: Final training loss and averaged benchmark
400M
Weight Decay Coefficient
0
0.0033
0.033
0.33
Final Training Loss
Pre-LN
3.03
3.03
3.03
3
Peri-LN
2.94
2.94
2.93
2.90
Averaged Benchmark Score
Pre-LN
49.26
49.18
49.01
49.51
Peri-LN
51.41
51.14
50.68
52.13
4.50
2.5
Peri-LN
Peri-LN Pre-LN
4.25
Pre-LN
2.0
Training Loss
4.00
Gradient Norm
1.5
3.75
3.50
1.0
3.25
0.5
3.00
0.0
10
20
10
20
Tokens
Tokens
(a) Pre-training loss curve
(b) Gradient-norm curve
Figure Comparison of pre-training loss and gradient norms for Pre-LN and Peri-LN architectures with weight decay coefficient
fixed at 0, while all other hyperparameters are held constant.
4.50
2.5
Peri-LN Pre-LN
Peri-LN
4.25
2.0
Training Loss
4.00
Gradient Norm
1.5
3.75
3.50
1.0
3.25
www.uo we www.
0.5
3.00
0.0
10
10
Tokens
Tokens
(a) Pre-training loss curve
(b) Gradient-norm curve
Figure 24. Comparison of pre-training loss and gradient norms for Pre-LN and Peri-LN architectures with the weight decay coefficient fixed at 0. 0.0033, while all other hyperparameters are held constant.
G.8.3. WEIGHT INTIIALIZATION
We run an additional ablation on weight- weight-initializzn initialization schemes. For both Pre-LN and Peri-LN models, we first adopt Xavier initialization (Glorot& Bengio, 2010) and then compare it with He initialization (2/d) (He et al., 2015), LeCun initialization
27

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
4.50
2.5
Peri-LN Pre-LN
Peri-LN
4.25
2.0
Training Loss
4.00
Gradient Norm
1.5
3.75
3.50
1.0
3.25
0.5
3.00
0.0
10
10
Tokens
Tokens
(a) Pre-training loss curve
(b) Gradient-norm curve
Figure 25, Comparison of pre-training loss and gradient norms for Pre-LN and Peri-LN architectures with the weight decay coefficient
fixed at 0.033, while other hyperparameters are held constant.
4.50
2.5
Peri-LN
Peri-LN
Pre-LN
2.0
Pre-LN
Training Loss
4.00
Gradient Norm
1.5
3.75
3.50
1.0
3.25
0.5
3.00
0.0
0
10
20
30
10
20
30
Tokens
Tokens
Pre-training loss
(b) Gradient-norm
Figure 26. Comparison of pre-training loss and gradient norms for Pre-LN and Peri-LN architectures with the weight decay coefficient fixed at 0.33, while all other hyperparameters are held constant.
(1/d), and two scaled variants, 10/d and 1/(10d).
As Table 8 shows, Xavier initialization yields the strongest overall performance, improving on the configurations used in our earlier experiments. Crucially, our central finding remains intact: hidden-state variance sharply grows Pre-LN Transformers but stays bounded Peri-LN Transformers. Table 9 confirms the same pattern across all initialization scales, and detailed per-run results appear below.
Table Xavier initialization (Glorot & Bengio, 2010) yields better performance compared to our previous weight initialization configura- tions.
400M
Architecture
Baseline(O.02)
Xavier Initialization
Loss
Pre-LN
3.03
2.95
Peri-LN
2.93
2.91
Avg.
Pre-LN
49.01
51.25
Peri-LN
50.68
52.04
Table Effeec of weight--nitializen initializationn variance on final pre training loss for 400M 1114000000MMaee parameter Pre-LN and Peri-LN Transformers.
400M
Initialization Variance
10/d
He (2/d)
LeCun (1/d)
1/(10d)
Baseline (0.02)
Loss
Pre-LN
4.526
2.965
3.005
3.012
3.035
Peri-LN
3.027
2.929
2.915
2.902
2.916
28

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
5.5
Peri-LN Pre-LN
Perl-LN
5.0
Pre-LN
Gradient Norm
Training Loss
4.5
4.0
3.5
2
Takens
1e10
Tokens
1e10
(a) Pre-training loss curve
Gradient-norm curve
Figure 27. Comparison of pre-training loss and gradient norms for Pre-LN and Peri-LN architectures with the weight initialization variance
set 10/d, while other hynerparameters are held constant.
2.5
Peri-LN Pre-LN
Peri-LN Pre-LN
4.25
2.0
Gradient Norm
Training Loss
1.5
3.75
3.50
3.25
3.00
3
Tokens
Tokens
1e10
(a) Pre-training loss curve
Gradient-norm curve
Figure 28. Comparison of pre-training loss and gradient norms for Pre-LN and Peri-LN architectures with the weight initialization variance
set to 2/d (He init (He et al., 2015)) while all other hyperparameter ane held constant
4.50
2.5
Peri-LN
Peri-LN
4.25
Pre-LN
Pre-LN
Training Loss
4.00
Gradient Norm
3.50
3.25
0.5
3.00
0.0
Tokens
1e10
Tokens
1e10
(a) Pre-training loss curve
(b) Gradient-nomn curve
Figure Comparison pre-training loss gradient norms Pre-LN and Peri-LN architectures with the weight initialization variance
set 1/d (LeCun init), while all other hynernarameter are held constant
4.50
Peri-LN
Peri-LN
Pre-LN
Pre
Training Loss
4.00
Gradient Norm
3.75
3.50
1.0
3.25
0.5
3
Tokens
1e10
Tokens
le10
(a) Pre-training loss curve
(b) Gradient-norm curve
Figure 30. Comparison of pre-training loss and gradient norms for Pre-LN and Peri-LN architcctures with the weight initialization variance 1/(10d), while all other hyperparameter held constant.
29

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
H. Output-Layer Normalization with QK-Norm Architecture
Query and Key layer-normalization (QK-Norm) has been widely studied in modern Transformer architectures (Wortsman al., 2024; Zhai et al., 2023; OLMo al., 2024). In particular, OLMo et al. (2024) reported that QK-Norm combined with module output mayer-normalizatiin (output-LN, B Figure 31 referred to "reordered norm" in the OLMo2 paper) improves both training loss and stability. As shown in Figure 31, QK-Norm is applied after the Query and Key projections, similar to output-LN. From another perspective, QK-Norm is also applied immediately before the attention calculation, akin to Pre-LN approach. In our view, QK-Norm and Pre-LN (placed at A and A respectively in Figure 31) serve the same role but differ in certain details. As shown in Figures 32, 33, and 34, the two architectures exhibit comparable performance overall terms of both training loss and stability.. However, Peri-LN provides a slight performance advantage over the OL Mooo-ett Peri-LNN in the 400M- and 1B 1B-parameter models.
Attention Madule
E1+1
Åttn Out
Attention
B
Attention MLP Module
Q
K
V
Figure 31. QK-layer normalization in the Attention module.
2.96
OLMo2-style Peri-LN
OLMo2-style Peri-LN
Peri-LN
Peri-LN
Pre-Trein Loss
2.94
Gradient Norm
2.92
2.90
2.88
2.86
10
Takens (Billions)
Takens (Billions)
(a) Pre-training loss curve
(b) Gradient-norm curve
Figure 32, Comparison of pre-trannin loss and gradient norm between OLMo2-Sttle Peri-LN and the Peri-LN architecture. To ensure an accurate comparison, present the pre-training loss the final 5B tokens. 400M size model.
30

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
2.78
OLMol-style Peri-LN
3.0
OLMo2-style Peri-LN Peri-LN
2.76
Peri-LN
2.5
Gradient Norm
2.74
2.0
ReeTriin
1.5
2.72
1.0
2.70
0.5
2.68
0.0
26
27
28
29
30
0
10
20
30
Takens (Billioon)
(Billions)
Pre-training loss curve
(b) Gradient-norm curve
Figure 33. Comparison of pre- rreitrannig loss and gradient norm between L..Mo--Stthe Peri-LN and the Peri-LN architecture. To ensure accurate comparison, we present the pre training loss over the final 5B tokens. 1.5B size model.
2.68
OLMo2-style Peri-LN Peri LN
2.66
Pre-Train Loss
Vy MAMM
Gradient Norm
2.64
2.62
2.60
OLMo2-style Peri-LN
Perl-LN
2.58
26
10
20
Tokens (8 (llloo))))
Tokens (Billoos))
(a) Pre-training loss curve
(b) Gradient-norn curve
Figure 34, Comparison of pre-training loss and gradient norm between OL Peri-LN and the Peri-LN architecture. ensure an accurate comparison, present the pre-training loss the final 5B tokens. 3.2B size model.
Training the Transformer using Stochastic Gradient Descent
Using Stochastic Gradient Descent (SGD) for training Transformers is not a common practice. As Zhang et al. (2024) point out, Transformer-baat models tend to perform worse with SGD than with Adam by considerable margin. One reason is that SGD struggles to handle the heterogeneity across different blocks. Although these aspects are certainly intriguing and warrant further investigation, they lie beyond the scope of our current work, as Zhang al. (2024) also note.
Nonetheless, for somcone who might wonder to know, conduct additional experiments using SGD. We are searching for U-shaped patterns during the learning rate exploration for both Pre-LN & Peri-LN shown in the Figure 35. We observe that: (1) SGD performs worse than Adam, consistent with findings reported in (Zhang et al., 2024); and (2) Peri-LN demonstrates better performance than Pre-LN. In these results, we use 400 M-parameter Transformers and apply the same configurations as in the main experiments (Section 4.... We provide detailed training curves in Figures 36, 37, 38.
8.25
Pre-LN
8.20
Peri-LN
Final Training Loss
8.15
8.10
8.05
8.00
7.95
0.001
0.002 0.003 0.004 0.005
Learning Rate
Figure 35. Learning Rate Exploration of Pre-& Peri-LN architecture trained with SGD optimizer. The individual training-loss and gradicnt-norm curves appear in Figures 36, 37, and 38.
31

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
9.25
Peri-LN
7
Peri-LN Pre-LN
Pre-LN
6
Training Loss
9.00
1."
8.75
Gradient
4
8.50
8.25
2
8.00
0
1
0
Tokens
Tokens
(a) Pre-training loss curve
(b)
Gradient-norm curve
Figure Using SGD with learning rate 5e
Peri-LN
Peri-LN
Pre-LN
Pre-LN
Gradient Norm
Training Loss
9.0
8.5
8.0
2
Tokens
Tokens
(a) Pre-training loss curve
(b) Gradient-nore norm curve
Figure 37, Using SGD with learning rate 3e 3.
Peri-LN Pre-LN
Peri-LN
10.0
2.5
Training Loss
Gradient Norm
MM
9.5
2.0
9.0
1.5
1.0
8.0
Takens
Tokens
(a) Pre-training loss curve
(b) Gradient-norm curve
Figure 38, Using SGD with learning rate le 3.
32

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
J. Additional Details on Evaluation
J.1. Detailed Configurations
We utilized the Language Model Evaluation withh the HuggingFace Transformers library (Gao et al., 2024; Wolf al., 2020) assess overall performance. We employ five different evaluation benchmarks: ARC (Clark al., 2018), HellaSwag (Zellers al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap al., 2019), Winogrande (Sakaguchi et al., 2021). During the pretraining stage, each model was trained under controlled random seed. We used the training loss at iteration 14, coreesooegn to the completion of 30B tokens-as as main reference point. When calculating the evaluation score, diverged checkpoints were excluded.
J.2. Detailed Results on Benchmark Evaluations
In this section, present the evaluation results for each model trained with five different training seeds. We exclude any diverged scores and average the remaining values, which are then reported in Table I in the main text.
J.2.1. TRAINIGI
Table 10. Detailed results on pre-training the Peri-I.N architecture. These results are averaged to produce the values reported in Table 1.
SEED denotes pre- vee-trainnn seed.
Peri-LN
SEED
ARC-Easy
HellaSwag
PIQA
SIQA
Winogrande
0.5758
0,3803
0.6980
0.4115
0.5225
2
0,5728
0.3739
0.6915
0,4017
0.5367
400M
3
0.5842
0.3745
0.6986
0,4125
0.5249
0.5800 4
0.3722
0.6959
0.4038
0.5209
0.5627 5
0.3719
0.6899
0.4028
0.5320
0.6599
0.4437
0.7339
0.4304
0.5714
0.6591
0.4394
0.7399
0.4145
0.5699
1.5B
0.6625
0.4357
0.7372
0.4166
0.5627
0.6633
0,4367
0.7345
0,4222
0.5667
0.6637 5
0.4416
0.7361
0.4335
0.5612
0.6953
0,4734
0.7443
0,4417
0,5872
2
0.6839
0.4684
0.7427
0.4324
0.6054
3.2B
3
0.6902
0.4680
0.7486
0.4243
0.5967
0.6864
0.4700
0.7427
0.4273
0.5935
5
0.6806
0,4698
0.7372
0,4243
0.6054
Table 11. Detailed results on pre-training the Pre-LN architecture. These results are averaged to produce the values reported in Table 1. SEED denotes pre- preininin seed.
Pre-LN
SEED
ARC-Easy
HellaSwag
PIQA
SIQA
Winogrande
0.5669
0.3609
0.7008
0.4002
0.5359
Diverged
400M
0,5354
0.3328
0.6741
0.3905
0.4957
Diverged
0.5438
0,3314
0.6888
0.4012
0.4949
0.6326
0.4259
0.7242
0.4263
0.5691
0.6019
0,3924
0.7111
0.3992
0.5627
1.5B
0.6077
0,3932
0.7008
0.4125
0.5272
0.6111
0,3886
0.7187
0.4135
0,5225
0.6221
0.3941
0.7160
0.4099
0.5438
0.6688
0.4588
0.7470
0.4273
0,5919
Diverged
3.2B
Diverged
0.6359
0.4259
0.7301
0.4263
0.5564
Diverged
"https://github. www.w/....om..eeeeeeeeeeeee http:/ttt..../o/eeeeeeeer//////e rness
33

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
Table 12. Detailed results on pre-training the Post-LN architecture. These results are averaged to produce the values reported in Table 1.
SEED denotes pre-training seed.
Post-LN
SEED
ARC-Easy
HellaSwag
PIQA
SIQA
Winogrande
0.3413
0.2881
0.6311
0.3378
0,5067
2
0.3691
0.2886
0.6132
0.3337
0,5099
400M
3
0,3632
0.2889
0.6257
0,3603
0,5051
4
0.3603
0.2920
0.6262
0.3490
0.5012
0.3510 5
0.2880
0.6170
0.3434
0.5209
1
0,4268
0.3121
0.6659
0.3628
0.5185
2
0.4196
0.3150
0.6654
0.3639
0.5004
1.5B
3
Diverged
4 0.4285
0.3212
0.6730
0,3511
0.4775
5
0,4419
0.3193
0.6643
0.3557
0.5154
0.4731
0.3427
0.6774
0.3664
0.5343
2
0,4638
0.3326
0.6779
0.3577
0,4917
3.2B
3
0.3956
0.3321
0.6143
0.3408
0.5067
0.4663
0,3380
0.6692
0,3685
0,5178
5
0.4663
0.3340
0.6839
0.3577
0,5043
J.2.2. SUPERVISED FINE-TUNING
Table 13. Detailed results on SFT with Peri-LN architecture. These results are averaged to produce the values reported Table 1. SEED denotes pre-training seed.
Peri-LN
SEED
ARC-Easy
HellaSwag
PIQA
SIQA
Winogrande
0.5800
0.3819
0,6991
0.4145
0.5328
0.5783
0.3783
0.6921
0.4038
0.5391
400M
0.5888
0.3806
0.6980
0.4222
0.5288
0,5892
0,3738
0.6948
0,4089
0.5099
0.5783 5
0.3757
0.6991
0.4099
0,5312
0.6633
0,4502
0.7356
0,4304
0.5746
2
0.6641
0,4437
0.7405
0.4207
0.5706
1.5B
0.6671
0.4454
0.7454
0.4207
0,5620
0.6700
0.4455
0.7378
0.4284
0.5659
0.6688
0,4478
0.7421
0.4324
0.5620
0.7058
0.4810
0.7486
0.4422
0.5880
2
0.6898
0.4774
0.7437
0,4391
0.6054
3.2B
3
0.6995
0,4770
0.7481
0,4278
0,5912
0.6894 4
0.6911
0.4777
0.7432
0.4350
0.5943
0.4781
0.7448
0.4319
0.6046
34

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
Table 14. Detailed results on SFT with Pre-LN architecture. These results are averaged to produce the values reported in Table 1. SEED denotes pre-training seed.
Pre-LN
SEED
ARC-Easy
HellaSwag
PIQA
SIQA
Winogrande
0.5762
0.3625
0.7078
0.4058
0.5343
N/A
400M
3
0.5370
0.3339
0.6757
0.3905
0.4972
4
N/A
5
0.5509
0.3372
0.6893
0.4074
0.4886
1
0.6385
0.4310
0.7247
0.4227
0.5620
2
0.6035
0.3934
0.7095
0.4038
0.5572
0.6098
0.3944
0.7035
0.4150
0.5257
4 0.6208
0,3929
0.7182
0.4161
0.5272
5
0.6258
0.4017
0.7171
0.4181
0,5391
1
0.6785
0.4681
0.7568
0.4345
0.5825
2
N/A
3.2B
3
N/A
0.6427
0.4293
0.7274
0.4299
0.5580
5
N/A
Table 15. Detailed results on SFT with Post-LN architecture. denotes pre-training seed.
These results are averaged to produce the values reported in Table 1. SEED
Post-LN
SEED
ARC-Easy
HellaSwag
PIQA
SIQA
Winogrande
0.4428
0.3307
0.6583
0.3797
0.5099
0.4280
0.3208
0.6404
0.3746
0.5178
400M
0,4693
0.3241
0.6578
0.3905
0.5122
0.4680
0.3247
0.6610
0.3726
0.4830
0.4520
0.3283
0.6572
0.3849
0.5225
0.5316
0.3774
0.6980
0.3889
0.5359
0.4731
0,3316
0.6719
0,3813
0,5028
1.5B
N/A
0.5387
0.3546
0.6779
0.3864
0.4909
0.5261
0.3510
0.6752
0.3767
0.5209
0.5623
0.4029
0.7008
0.3920
0.5051
0,5417
0.3644
0.6823
0.3833
0.5264
3.2B
0.4444
0.3604
0.6333
0.3618
0.5043
4
0.5400
0.3645
0.6844
0.3823
0.5020
0.5341
0.3677
0.6942
0.3976
0.5012
35

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
K. Additional Discussions of Precision Constraints Imposed by Pre-LN
This section provide additional discussions of Section 7.2. Given that both Pre-LN and Peri-LN exhibit a structural property whereby large values do not readily disappear once they arise, it is important to monitor the occurrence of these extreme activations. Pre-LN's additive residual path often produces activations that approach, and occasionally exceed, the FP16 (float16) representable maximum'. To quantify how often these values would overflow FP16 yet remain within the BF16 (b range*, we measure the 100 largest absolute hidden-state values (the global top-100 tokens) for each Pre-LN and Peri-LN 3.2B-paree ---aaammetrr Transformer. The shaded region indicates the range of these global top-100 tokens. The blue curve (with shaded band) represents a Pre-LN model, the red curve a Peri-LN model, and the dashed orange line denotes the FPI6 representable maximum.
As shown Figure 11, consistent with the observations of Sun et al. (2024), activations in the Pre-LN model routinely exceed the FP16 bound from the very first 0.5B training tokens, with the overshoot becoming more pronounced in deeper layers- an indication of growing numerical instability. By contrast, Peri-LN consistently maintains comfortable margin below the FP16 limit throughout training. This finding underscores that the choice between FP16 and BF16 is not merely a hardware preference; it is tightly coupled to how hidden-state magnitudes evolve within the network.
Since NVIDIA V100 GPUs do not support BF16, these results imply that training and inference with Pre-LN models on such hardware are inherently disadvantaged. Moreover, from the standpoint of large-lagguage-morn modell quantization (Dettmers al., 2022; Lee al., 2023; Kim et al., 2023; Heo al., 2024; Lee et al., 2025), the prevalence of massive activations in Pre-LN can severely disrupt outlier-aware compression algorithms. When aggressive low-precision compression is the goal, the Pre-LN architecture's 's tendency generate extreme hidden state values therefore constitutes particularly challenging obstacle.
Meta Al's publicly released OPT training logbooks and chronicles document repeated episodes of gradient divergence and loss spikes encountered while pre-training entirely in FP16 precision 9. Since FP16 saturates at an absolute value of 65, 504, any hidden state activation that exceeds this threshold silently overflows, corrupting the forward pass and, through pack-propagations inducing erratic gradients. Earlier analyses of OPT (Zhang et al., 2022) therefore ascribe much of the observed instability to numerical overflow, view formalized in our Proposition 3.1, which shows how such out-of-range activations propagate into severe gradient pathologies. These external reports provide further corroboration that architectures prone to generating aage-magnitudd activations as Pre-LN does does-require either a wider numerical format (e.g., BF16) or explicit magnitude--egullazzzti to ensure stable large-scale training.
http::/:thttattoeeeeeeeeeesesees nt nt_update.md
36

---

Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
L. Additional Discussions of Hidden State Representations
As shown Figure 39(a), Post-LN exhibits smaller angular distances due to the LN being located on the main path, whereas Pre-LN and Peri-LN begin with very similar states. As shown in Figure 39(c). the end of training, Pre-LN tends to produce more redundant hidden state representations compared the others. This phenomenon may stem from Pre-LN's repeated residual additions, which amplify certain representations in this experiments. For dataset, we utilize 256 random samples of RedPajama-Daie Daatnn---tt-Tiaa IT (Computer, 2023) for this results.
others. We 30B tokens trained 400M size model
To investigate further, we focus on module-output normalization, the primary factor distinguishing Pre-LN from Peri-LN, As shown Figure 39(b), the learnable scale starts around 1 in the early stages of training and gradually changes with increasing depth. Because Peri-LN preserves the identity path, appears to adjust the scale of the module output accordingly. This suggests that the exponential growth of the main path's s magnitude in Pre-LN diminishes the relative contribution of individual modules, resulting in more redundant hidden representations. Figure (9(d) shows that fixing the learnable scale of Peri-LN's module output LN 1 causes the main path contribution to decrease in deeper layers. This finding confirms the role of module output normalization in controlling hidden state redundancy.
Post-LN
Pro-L
Peri-LN
0.30
0
Starting Layer Indlex
Starting Layer Inblex
0.25
4-
Layer Index
0.20
1.5
12.
0.15
16
0.10
16
1.0
20
20
20.
20
0.5
12
16
20
16
20
16
0.00
15
Subsequent Layer Index
Subsequent Layer Index
Subsequent Layer Index
Trained Tokens (B)
(a) At initialization
(b) Learnable scale 7 in Output-LN
Post-LN
Pre-LN
Peri-LN
0.30
Peri-LN
0.30
Starting Layer Index
Starting Layer Indox
0.25
Starting Layer Index
0.25
0.20
12 8
0.20
12.
12
16 12
0.15
Starting
Angulas
20 16
0.10
0.10
20
0.05
20
0.05
16
16
16 Subsequent Layer Index
0.00
12
0.00
Subsequent Layer Index
Subsequent Layer Index
Layer
(c) After 30B tokens training
(d) Frozen in Output-I.N
Figure 39, Angular distance hidden state is presented in Figure 39(a), 39(c), and (d))). Figure 39(b), we monitor y of every
Output-LN in Peri-LN during training. We use 30B tokens trained 400M size model in this experiments.
37


Reviews:
[{"review_id": "TWiV8gJ4bO", "reviewer": "Reviewer_AtH3", "summary": "This paper examines a layernorm \"layout\" in the transformer architecture called PeriLN. The PeriLN combines Prelayernorm with a module out layernorm (similar to post layernorm but before the residual stream). The authors provide intuition for how this addresses weaknesses in the Post layernorm and prelayernorm layouts along with supporting theoretical statements. The authors provide comprehensive experiments showing that this layout dominates the other layouts in performance and stability on LLM experiments.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **1) Confirming That Pre + Post LayerNorm is Worse Than Peri-LN**\n>*“It seems that pre + post layernorm should be worse than periLN, perhaps that can be confirmed experimentally?”*  \n\nIn response to the reviewer's comment, we additionally conduct further experiments on LN placements to compare different combinations (referred to as A, B, and C positions in Figure 2 of the paper). We add configurations where LN is placed at both A + C (akin to combining Pre- and Post-LN), as well as only at B, to compare them with Peri-LN at final training loss under the controlled same training seed. We pre-train the 400M-parameter Transformers on 30B tokens each, using the same training configurations described in the paper. As aligned with Xiong et al. [1], our new results confirm that placing LN exclusively at C leads to training instability or suboptimal performance. In particular, the A + C configuration inherits characteristics of Post-LN (large gradient norm shifts), forcing the use of smaller learning rates and still resulting in lower overall performance than Peri-LN architecture. We will include additional learning rate sweep results and detailed training loss curves for the additional A+C and B experiments in the revised manuscript to more comprehensively illustrate these differences.\n\n|400M|A + C|Post-LN|B|Peri-LN|\n|:-:|:-:|:-:|:-:|:-:|\n|Loss|3.01|3.05|Diverged|2.91|\n  \n[1] Xiong, Ruibin, et al. \"On layer normalization in the transformer architecture.\" ICML 2020.\n\n---\n### **2) Reconciling Gradients Blowup vs. Vanishing Gradients in Post-LN**\n>*“My takeaway from Xiong et al. [1] was that Postlayernorm will lead to gradient norm blowup, but from this paper it seems like the issue is actually gradient norm vanishing. How should I reconcile this?”*  \n\nThank you for raising this point. Our layer-wise observations in Post-LN (Figure 6 in the paper) indeed show signs of gradient vanishing through layers, yet we also observe strong gradient spikes (i.e., blowups in the total gradient summation) at various stages of training. This aligns with Xiong et al. [1], where the large shift in Post-LN gradients causes instabilities that lead to sudden spikes. In essence, both Xiong et al. and our Proposition 3.1 suggest that the gradient scale in Post-LN can swing dramatically. When examining training iteratively (step by step), we observe occasional gradient spikes (blowups). However, across the broader span of training, Proposition 3.1 and [2] show that Post-LN gradients ultimately exhibit a vanishing tendency overall—consistent with Figure 6.\nWe appreciate the chance to clarify further. In the revised manuscript’s supplementary material, we will include additional plots demonstrating both the micro-scale spikes in both gradient and loss over the course of training.\n\n[2] Kedia, Akhil, et al. \"Transformers get stable: an end-to-end signal propagation theory for language models.\" ICML 2024.\n\n---\n### **3) QK-Norm Discussion**\n\nWe agree that QK-Norm plays an increasingly important role in modern Transformers. As you suggest, we will move or more prominently feature the QK-Norm discussion from the supplementary section into the main paper. \n\n---\n### **4) Clarifying Olmo2 vs. Peri-LN**\n\nYes, as noted in Appendix G, the Olmo2 architecture slightly differs from Peri-LN. Peri-LN uses both a Pre-LN and an Output-LN, whereas Olmo2 relies on QK-Norm plus the output LN (No Pre-LN). From our experiments, applying only an output LN (or only a QK-Norm) proved insufficient to stabilize training under challenging hyperparameter settings, which is consistent with remarks in the Olmo2 paper [3].\n\n[3] OLMo, Team, et al. \"2 OLMo 2 Furious.\" arxiv 2024.\n\n---\n### **5) Is QK-Norm Unnecessary for Peri-LN?**\n\nWhile Peri-LN alone provides robust training dynamics, QK-Norm can still enhance performance. In response to reviewers comment, we conducted additional experiments that confirm combining Peri-LN with QK-Norm yields slight improvements in training loss. We pre-train the 1.5B-parameter Transformers on 30B tokens each, using the same training configurations described in the paper. This observation is consistent with prior work [4] indicating that QK-Norm can synergize with various LN placements. We will include these new results in the revised manuscript with detail, along with references to recent works like gemma3 [5], which successfully integrate Peri-LN and QK-Norm.\n| 1.5B | Peri-LN | + QK-Norm |\n|--|---|-|\n|Loss|2.722|2.711|\n\n[4] Wortsman, Mitchell, et al. \"Small-scale proxies for large-scale transformer training instabilities.\" ICLR 2024.  \n[5] Team, Gemma, et al. \"Gemma 3 Technical Report.\" arXiv 2025.  \n\n---\n### **Concluding Remarks**\nWe believe our additional experiments and clarifications—regarding LN placements, QK-Norm, gradient behavior—will strengthen the paper significantly. Your insightful guidance has been instrumental in refining our analysis. We will incorporate your valuable comments into the revised manuscript.", "claims_and_evidence": "Yes the claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes the methods and evaluation are solid.", "theoretical_claims": "I did not check correctness explicitly but they seem reasonable.", "experimental_designs_or_analyses": "Yes the designs are sound.", "relation_to_broader_scientific_literature": "This paper contributes to the literature on understanding training dynamics and layernorm in architectures. It identifies the weaknesses and strengths of post and prelayernorm, providing substantial evidence for a better alternative.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "The paper is written very clearly and the experiments are very solid, especially the learning rate sweeps (which I believe use muP).", "other_comments_or_suggestions": "I think QV norm discussion could be moved into the main paper because it is quite interesting and important in the \"layernorm in transformers\" design space.", "questions_for_authors": "In olmo2 there is only a module output layernorm and not the prelayernorm? So this differs from PeriLN?\n\nDoes the finding about QK normalization say that it is not needed with PeriLN?\n\nIt seems that pre + post layernorm should be worse than periLN, perhaps that can be confirmed experimentally?\n\nMy takeaway from Xiong et al. [1] was that Postlayernorm will lead to gradient norm blowup, but from this paper it seems like the issue is actually gradient norm vanishing. How should I reconcile this?\n\n\n[1]  On Layer Normalization in the Transformer Architecture - Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, Tieyan Liu", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D1qrj5lt38", "reviewer": "Reviewer_nnDw", "summary": "This paper investigates the effectiveness of  position where layer normalization (LN) (mainly its reduced version RMSNorm)  is placed in the Transformer architecture.  It also provides analyses from the perspective of activation/gradient propagation in the network to explain why a position LN  placed usually works better. In particular, it claims that the previous Pre-LN and Post-LN architecture are prone to vanishing gradients and \"massive activations\". It advocates to place LN peripherally around sublayer, termed Peri-LN (There exists similar usage of LN in the Transformer from previous work, as pointed in the paper). The experiments are conducted on Transformers up to 3.2B parameters, showing that Peri-LN achieves more balanced variance growth, steadier gradient flow, and convergence stability.\n\n\n## update after rebuttal:\n\nI have read the responses and other reviewers comments. My concerns on claims and Proposition 3.1 still hold. The two main claims of this paper:  1. Pre-LN  has exploding gradient (Proposition 3.1. (1);  2. Post-LN has  vanishing gradient are not well validated by experiments.  E.g., this paper should provide the results (gradients) to support the claims under different weight initialization and weight decay (if the theory and analyses hold, it should hold under different weight initialization and weight decay).  However, this paper does not provide the results for Post-LN even in rebuttal and  the provided results for the Pre-LN are also not convincing (i.e., I donot find exploding gradient).  \n\nI keep may score, and towards reject this paper.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **1) Considering $W^{(2)}$ in the MLP**\nFollowing Theorem 1 in [1], we analyze the last layer, as the gradient norm at the final layer is empirically known to be the most unstable (see Figure 1 in the paper). We choose $W^{(2)}$ (the final linear projection in the MLP) as a representative example because it most directly feeds into the residual connection $x+a$, making it clearer to illustrate how gradient norms can explode or vanish.\n\n[1] Xiong et al. \"On layer normalization in the transformer architecture.\" ICML2020.  \n\n---\n### **2) Post-LN and Vanishing Gradients**\nIn Proposition 3.1, the bound $\\frac{\\||h\\||}{\\||x+a\\||}$ involves not only $\\||h\\||$ but also $\\||x+a\\||$. Since $h$, $x$, and $a$ are interrelated, what ultimately matters is their relative magnitudes. The theory indicates that the Post-LN structure introduces $\\||x+a\\||$ into the bound, whereas Peri-LN introduces only $\\||a\\||$. Experimentally, we confirm that Post-LN exhibits vanishing gradients (see Figure 6(b, d) and [2]). As shown in Appendix D, the presence of a normalization layer significantly influences the gradient scale not only in Peri-LN but also in Post-LN—an observation consistent with [1] and [2]. For more discussion, we kindly refer to Response #2 of *Reviewer AtH3*.\n\n[2] Kedia et al. \"Transformers get stable: an end-to-end signal propagation theory for language models.\" ICML2024.  \n[3] Fishman et al. \"Scaling FP8 training to trillion-token LLMs.\" ICLR2025.  \n\n---\n### **3) Pre-LN and Exploding Gradients**\nAs discussed in our paper (Figures 1 and 6) and by Sun et al. [4], we observe that Pre-LN architectures can produce large activation values. Fishman et al. [3] and Wortsman et al. [5] note that quadratic computations in both the Attention [5] and MLP [3] modules can yield large activations. In Pre-LN architectures, these outputs are not regulated by normalization at the sub-layer level. Consequently, high-variance spikes in intermediate activations can lead to training instabilities because, in Proposition 3.1, $\\||h\\||$ appears in the numerator of the gradient bound, posing a risk of gradient explosion. Although Adam adaptively rescales gradients, raw gradient spikes may still destabilize updates or force Adam to make abrupt learning rate adjustments (see Section C.1 in the supplementary, where Post-LN uses a lower LR). Indeed, as our paper shows, Pre-LN experiences gradient spikes and training instability despite Adam’s adaptive rescaling. Our main point is that Peri-LN mitigates these large activations at the sub-layer output, reducing the risk of high-magnitude raw gradients before the optimizer’s moment-based rescaling takes effect.\n\n[4] Sun et al. \"Massive activations in large language models.\" COLM 2024.  \n[5] Wortsman et al. \"Small-scale proxies for large-scale transformer training instabilities.\" ICLR 2024.  \n\n---\n### **4) Weight Initialization & Decay**\n- **Weight Initialization**: Due to limited space, we kindly refer you to Response #3 of *Reviewer 1xqQ*.\n- **Weight Decay**: We conduct additional studies for various weight decay condition for both Pre-LN and Peri-LN architectures.  As shown in the table, Peri-LN continues to offer better performance than Pre-LN under the same settings. We will include this detailed ablation studies in the revised manuscript to solidify that our findings (especially large hidden state variance) still hold under varied weight decay and initialization methods. Experimental settings are in Response #3 of *Reviewer 1xqQ*.\n|400M||Decay=0|0.0033|0.033|0.33|\n|-|-|-|-|-|-|\n|Loss|Pre-LN|3.03|3.03|3.03|3|\n||Peri-LN|2.94|2.94|2.93|2.90|\n|Avg.|Pre-LN|49.26|49.18|49.01|49.51|\n||Peri-LN|51.41|51.14|50.68|52.13|\n---\n### **5) Theory references SGD, but experiments use Adam**\nThe key difference between Adam and SGD lies not in the gradients themselves but in how the learning rates are adjusted afterwards—Adam employs adaptive learning rates, while SGD uses a fixed one. Our theoretical analysis focuses on the structural characteristics of the raw gradients rather than assuming any specific optimizer behavior. Therefore, theoretical analyses on the gradients themselves remain valid regardless of whether SGD or Adam is used in the experiments.\n\n---\n### **6) Over-claimed novelty & Provide the background of LN**\nSince prior work has analyzed gradient behaviors of Post-&Pre-LN, our central contribution is to consolidate and extend these insights to a third placement—Peri-LN—that recent models (e.g., Gemma2 & 3, Olmo2) employ with limited understanding. We will moderate the overall expressions and provide a concise background on normalization layers.\n\n---\n### **Concluding Remarks**\nYour feedback has been instrumental in this process, and we sincerely extend our gratitude for your invaluable insights. Should you have any inquiries or require clarifications about our rebuttal, please don't hesitate to reach out. We are eager to address any concerns and elucidate potential ambiguities in greater depth.", "claims_and_evidence": "The main claims of this paper is that: (1)  it claims that the previous Pre-LN and Post-LN architecture are prone to vanishing gradients and \"massive activations\"; (2)  Peri-LN achieves more balanced variance growth, steadier gradient flow, and convergence stability.  I believe claims 2 are mostly correct and convincing, based on the experiments and my understanding. However,  I am not convinced by the claim 1.  Even though this paper provides informal (the so-called) theory and experiments to support claim 1, the experiments are not sufficient to support, e.g., it does not consider the affects of weight's variance and the optimizer, please see the comments in the experiments design, and I also have concerns on theory (see the comments in the theoretical claims).  \n\n\n\nBesides, some description is somewhat over-claimed, e..g, \"we provide fresh insights into where and when it may offer advantages over more widely adopted LN placements.\" .  I believe the analyses on the position of normalization using activation/gradient propagation are widely used in previous work (e..g, the paper introduces Pre-LN architecture), please see the survey paper [1] for details. \n\n\n\nRef:  [1] *Normalization Techniques in Training DNNs: Methodology, Analysis and Application*, TPAMI 2023.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria is overall make sense. But ithe experiments are not sufficient to support, e.g., it does not consider the affects of weight's variance and the optimizer, please see the comments in the experiments design.", "theoretical_claims": "The main (informal) theoretical claims is Proposition 3.1. \n\nIndeed, I have concerns on this Proposition:\n\n(1) why this paper only consider  the gradient of $W^{(2)}$, why not consider the gradient of   $W^{(1)}$ in the MLP, and further other weights in Self-Attention?\n\n\n\n(2)  The vanishing gradient of Post-LN is base on the description that \"when a massive activation $\\|h\\|$ occurs, Norm() introduces an overly suppressing factor $\\|x+a\\|$\", why is that,  note that the bound is relating to $\\frac{\\|h\\|}{\\|x+a\\|}$, when this paper assume $\\|h\\|$ is also massive. \n\n(3) The exploding gradient of Pre-LN is based on the description that \"when a massive activation $\\|h\\|$\"occurs,  $\\|\\frac{\\partial{L}}{\\partial{W^{(2)}}}\\|$ can arise? why $\\|h\\|$ definitely  occurs? even that, why leading to training instability? especially, the optimizer is Adam used in this paper, which can well remove out the scale of gradients.", "experimental_designs_or_analyses": "Even thought the experiments show a results supporting the claim 1 (that the previous Pre-LN and Post-LN architecture are prone to vanishing gradients and \"massive activations\") , I still have concerns on the experimental setups. My concerns are follows:\n\n(1) The activation and gradients is related to the variance of weight matrix initially, this paper should provide the details of initialization of  weight matrix, and further investigates whether  claim 1 holds when varying the  variance of weight matrix initially.  It is clear that normalization has the scale invariant property during forward process, but has the inverse scale property during back-propagation,  and the scale of weight matrix apparently affects the results.  Besides, this paper uses weight decay in the experiments,  I think this paper should further consider whether the claim 1 hold if weight decay removed or consistently hold if weight decay varied. \n\n(2) The analyses in theory is based on the gradients (e.g., SGD), not provide the analyses when using Adam optimizer. However, the experiments conducted in this paper only use the adam optimizer, not using SGD. It is not sufficient to support the theory.", "relation_to_broader_scientific_literature": "This paper provides clear routing how this paper builds on.", "essential_references_not_discussed": "I believe this paper provides overall references", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "The experiments are mainly conducted on RMSNorm, this paper should provide the background of Layer Normalization and RMSNorm, in case of the reader is not familiar to them.", "questions_for_authors": "See the questions in Theoretical Claims (comments)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DT8f8lzIs5", "reviewer": "Reviewer_1xqQ", "summary": "This paper focuses on how different LN strategies influence on training dynamics\nin transformer architectures training and present a LN strategy called Peri-LN ,\nwhich applies LN around the sub-layer. \n\nBy theoretical analysis and experiments, the authors suggest that Peri-LN can\nnot only improves gradient stability and final loss but also plays a critical role in\nreducing hidden-state redundancy, which shows better performance than Post-LN and Pre-LN.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **1) Extending Analysis to Other Layers**   \nThank you for highlighting this. As noted in Section 3.4 and Section D, our analysis focuses on the final layer. Following Theorem 1 in [1], we analyze the last layer because its gradients are often the largest in magnitude. We chose $W^{(2)}$ (the final linear projection in the MLP) as a representative example since it most directly feeds into the residual connection $(x + a)$, making it more transparent to illustrate how gradient norms can explode or vanish. This direct link to the residual path can significantly impact gradient stability in subsequent layers. Nonetheless, for a more comprehensive understanding of LLMs training dynamics, extending this theoretical foundation to other components (such as attention) is indeed important. We appreciate the reviewer’s insight on this matter and plan to pursue this direction as part of our future research.  \n\n[1] Xiong, Ruibin, et al. \"On layer normalization in the transformer architecture.\" ICML 2020.  \n\n---\n\n### **2) Extending Exploration to Vision or Multimodal Tasks**   \nDue to time and resource constraints, we could not run additional experiments in vision or multimodal settings. However, existing literature lends some support to the broader applicability of our findings. For instance, Sun et al. [2] reports that in ViT (Vision Transformer) architectures, massive activations can also emerge under a Pre-LN setup, paralleling what we observe in language models. This similarity suggests that the insights from our Peri-LN analysis could extend beyond pure language tasks. Given the trend of integrating LLMs into large-scale vision-language models, the reviewer’s question is indeed highly relevant. As outlined in our paper, we focused on large language models as the primary use case for Peri-LN. However, we see great potential in exploring vision or multimodal tasks in future work, building on the theoretical and empirical observations presented here.  \n\n[2] Sun, Mingjie, et al. \"Massive activations in large language models.\" COLM 2024.  \n\n---\n\n### **3) Weight Initialization: Additional Experiments & Clarification**\n- **Additional Experiments**: In response to the reviewer’s question, we conducted additional experiments to explore different weight initialization methods. In this study, for both Pre-LN and Peri-LN architectures, we apply Xavier initialization [3]. As shown in the table below, Xavier initialization yields better performance compared to our previous weight initialization configurations. We also confirm that our main observation still holds: *large variance occurs in Pre-LN Transformers but not in Peri-LN Transformers*. We will provide detailed results that gradient and loss spikes still occur in Pre-LN training curves. Thank you for your insightful guidance on improving the experimental quality of the paper.  \n- **Experimental Settings**: We pre-train the 400M-parameter Transformers on 30B tokens each under the controlled same training seed. We measure the training loss and averaged benchmark score for these experiments under the same evaluation settings used in Table 2 of the paper. Other configurations follow those outlined in Section 4.1.\n- **Clarification on the Weight Initialization**: We acknowledge that we did not provide sufficient detail about initialization methods in the original manuscript. In the experiments discussed in the paper, we initialized the weights using a zero-mean Gaussian distribution with a standard deviation of 0.02. We will clarify these details in the revised manuscript.  \n\n|400M|Architecture|Paper|Xavier Initialization [3]|\n|-|-|-|-|\n|Loss|Pre-LN|3.03|2.95|\n| |Peri-LN|2.93|2.91|\n|Avg.|Pre-LN|49.01|51.25|\n| |Peri-LN|50.68|52.04|\n\n[3] Glorot, Xavier, and Yoshua Bengio. \"Understanding the difficulty of training deep feedforward neural networks.\" international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2010.  \n\n---\n\n### **Concluding Remarks**  \nOnce again, we sincerely appreciate your insightful feedback. Your questions on extending Peri-LN’s theoretical analysis to other components and applying it to tasks beyond language have highlighted valuable directions for our future work. We are committed to further investigating these avenues—particularly how Peri-LN might generalize to vision or multimodal settings—and to incorporating additional details on initialization strategies to ensure that our results remain transparent and consistent. We hope these efforts will address your questions and more clearly convey the findings of our work. We will make sure to incorporate your valuable suggestions into the revised manuscript. If you have any further questions or topics you would like to discuss, please feel free to let us know.", "claims_and_evidence": "The claims are enlightening, which is supported by both theoretical and\nexperimental evidence. \n\nThe analysis and results are comprehensive, which offers a through\ncomparison of Post-LN, Pre-LN and Peri-LN.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are reasonable.\n\nHowever, the experiments only evaluate the performance on language\nbenchmarks. Assessing the effectiveness on other tasks would provide a more\ncomprehensive understanding of the utility of different LN strategies.", "theoretical_claims": "No errors in theoretical claims are found. This paper have solid theoretical\nanalysis, which provide convincing evidence of the conclusions. \n\nHowever, the theoretical analysis in Proposition 3 employs MLP for the analysis,\nwhich is different from the attention module used in the actual transformer\narchitectures.", "experimental_designs_or_analyses": "The experiments involve large scale experiments on multiple experiment\nsettings . The authors compare the performance of different LN strategies with\nseparate benchmarks and systematically analyse the mechanics of Peri-LN\nfrom different perspective. The analyses are comprehensive. \n\nMy concern is about the initilization method of the network, which may affect the conclusion of this paper. Please see **Questions For Authors** for details.", "relation_to_broader_scientific_literature": "The key contribution is a in-depth analysis of different LN strategies in large\u0002scale transformer structures. The author summarize a new LN strategy termed\nPeri-LN and bring a perspective to how we apply normalization technologies in\napplication.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Q1: Will the authors consider to provide further theoretical analysis, for example\nchanging MLP into attention?\n\nQ2: Could the advantage of Peri-LN preserve when it transfer to other tasks, for\nexample vision or Multimodal tasks?\n\nQ3: We know that LN can control the distribution of hidden neurons, but can not control the gradient strictly. Therefore, **Initialization Methods** are still essential in training a DNN. Which initialization method did the authors apply? He initialization (with variance 2/d), LeCun initialization (with variance 1/d) or others? Could the authors provide the results under different initialization methods? For example, change the variance to 10/d or 1/10d, which may address the issues of gradient vanishing or gradient exploding. I am curious about the results under these initializations, although they are not common in current trainings.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KfsNWjw4B9", "reviewer": "Reviewer_Rrms", "summary": "Extending existing works on the placement of layer normalization in transformers, such as Pre-LN and Post-LN, this study proposes specific positions to apply layer normalizations, called Peri-LN configurations. The authors claim improved theoretical properties such as variance accumulation and gradient behavior, which ensures advantages from existing placements. Experiments validate the improved performance from the Peri-LN.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **1) “$o$ seems intermediate; softmax is applied in an unnatural way.”**\nWe believe there may be a misunderstanding regarding the reviewer’s concern that $o$ is an intermediate representation and that softmax is applied in an unnatural manner. As noted in Section 3.4 and Section D, our theoretical analysis focuses on the final MLP layer. This choice is motivated by two reasons: (1) the gradient norm at the final layer is empirically known to be the most unstable (see Figure1); and (2) it allows for a rigorous mathematical analysis without requiring approximations or assumptions. Analyzing the final layer for theoretical insight is an established practice in the literature (see Theorem 1 in [1]).\nImportantly, our empirical observations show that other layers exhibit similar trends in gradient behavior and hidden-state variance (see Section 4.4), supporting the generality of our theoretical insights.\n\n[1] Xiong et al. \"On layer normalization in the transformer architecture.\" ICML 2020.\n\n---\n### **2) “No theoretical proof for MHSA”**\nAs discussed in our Response #1, we focus on the final MLP sub-layer rather than the intermediate MHSA layers. For these reasons, we introduce a proposition centered on the MLP layer, aiming to explain the phenomena observed in our experiments.\n\n---\n### **3) “Summation is necessary in multivariates”**\nWe respectfully note that the comment regarding explicit summation in multivariate expressions appears to stem from a misunderstanding: in our matrix notation, the summation is inherently handled by matrix multiplication. In componentwise notation, the multivariate chain rule involves a summation over the relevant indices. However, in matrix notation, the summation is taken care of by matrix multiplication. Please refer to Theorem 5.3 in [2].   \n\n[2] Colley, Susan Jane. Vector calculus. PEARSON EDUCATION LIMITED, 2012.\n\n---\n### **4) “$\\gamma$ is assumed positive, yet it can become negative in practice.”**\nWe appreciate this point. In theory, $\\gamma$ is a scaling parameter primarily intended to adjust magnitude, so the mathematical derivation naturally assumes $\\gamma > 0$. In this paper, we assume $\\gamma$ remains positive for theoretical simplicity, and we will include a clarification of this assumption in the revised manuscript. Empirically, we further verified $\\gamma$ during by monitoring the magnitude across all checkpoints for a 30B-token run; it stayed *strictly positive* in every layer. We would like to emphasize that, as discussed in Section 4.4.1, even when we freeze $\\gamma$ to 1, Peri-LN still retains its main benefits.\n\n---\n### **5) “RMSNorm <-> LN or ReLU <-> GeLU might not match exactly”**\nWe understand the concern that RMSNorm and ReLU may feel “approximate” compared to LN and GeLU. Our motivation was analytical tractability:  \n- RMSNorm omits mean-centering, which simplifies the Jacobian while still capturing the main effect of normalization (rescaling by the vector’s norm). LayerNorm introduces an additional term subtracting the mean, but in large dimensions, mean removal has a smaller relative effect—so the bounding principle remains largely the same [3].   \n- ReLU is piecewise-linear, making it straightforward to compute partial derivatives explicitly. Since we are only considering the gradient with respect to $W^{(2)}$ in the final layer, it makes no difference whether the preceding hidden activation function is ReLU or GeLU. In either case, the hidden activation $h$ is treated as a constant when differentiating with respect to $W^{(2)}$, so the conclusions regarding stability with respect to $\\||h\\||$ remain the same.    \n\nHence, these substitutions allow us to *cleanly* show how placing LN at different points can dampen or amplify the backpropagated gradients. We agree it is not a full 1:1 equivalence to LN or GeLU, but the essential mechanics are preserved. In the revised text, we will add disclaimers that our derivation is an instructive idealization—RMSNorm vs. LayerNorm and ReLU vs. GeLU do not qualitatively change the conclusion about how LN placement impacts model dynamics.  \n\n[3] Zhang et al. \"Root mean square layer normalization.\" Neurips 2019.\n\n---\n### **Concluding Remarks**\nIt seems there has been a misunderstanding, and we kindly ask you to reevaluate our work in light of its theoretical and experimental contributions. As Reviewer 1xqQ and AtH3 highlighted, we do believe that our theoretical contribution is solid. Your comments have prompted us to refine our exposition and add more detailed supporting materials. While we respectfully maintain that our core derivations—even with simplified assumptions (RMSNorm instead of LN, ReLU instead of GeLU)—provide a valid perspective on how LN placement influences Transformers, we will include additional clarifications and empirical validations in a revised manuscript. We hope these efforts will address your concerns and more clearly convey the findings of our work. Thank you again for your time.", "claims_and_evidence": "I think there are several problems in the theoretical parts. See the “Theoretical Claims” below.", "methods_and_evaluation_criteria": "Yes, the experiments look well-designed, and the authors provided extensive results.", "theoretical_claims": "- In Appendix D.1, the term $o$, which is the output of MLP and skip connection, would be a feature in a layer module. However, the authors apply softmax to $o$ to obtain $p$ for their proof, which looks significantly unnatural. The authors should clarify whether $o$ is an element of a layer or whether this proof is intended to narrow down to the scenario of the last layer.\n- The proof in Appendix D only considers MLP, and there is no theoretical proof for MHSA.\n- In the proof in Appendix D, when using the chain rule, I think summation is necessary in multivariates. Please check this point.\n- In Eq. 29 in Appendix D, the term $\\gamma$ is assumed to be positive. Although $\\gamma$ is commonly initialized to one, it frequently becomes negative during the training. I think this part is not strictly correct either.\n- The proof in Appendix D was performed using RMSNorm instead of layer normalization and ReLU instead of GELU. These approximations might be necessary for simplicity in theoretical proofs, but adopting them may give the impression that the proof is not perfect.\n- Overall, I evaluate the theoretical part of this manuscript as not enough for publication in ICML. The authors have omitted important parts in the actual proof compared to the wide claim range, and several parts of the proof look incorrect.", "experimental_designs_or_analyses": "I think the amount of experiments looks adequate to investigate the validity of the proposed method.", "relation_to_broader_scientific_literature": "The study on foundation models would advance general machine learning fields and further broader scientific literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Other strengths of the proposed method would be that it is easy to deploy in practical source code by injecting few lines of code.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See “Theoretical Claims” above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper provides an analysis of different layer normalization strategies and how they impact the training dynamics of large-scale transformers, finding that peripherally bracketing normalization layers around submodules (Peri-LN) can improve stability relative to the pre- and post-LN baselines.

This paper received mixed reviews, with some reviewers finding strength in the comprehensive empirical analysis and others questioning claims about exploding/vanishing gradients and highlighting gaps between theory and practice. Nevertheless, there is wide agreement that  the paper does provide convincing evidence that Peri-LN achieves more stability during training. Given the breadth and depth of empirical investigation, it is not necessary for the theoretical analysis to provide a rigorous explanation for all the observed phenomena and can instead simply serve to provide some useful guiding intuition to accompany the experiments. As such, I believe this paper will provide a valuable contribution to the ICML community and I recommend acceptance.
Decision: Accept (poster)