HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization
Authors: Zhijian Zhuo
Yutao Zeng
Ya Wang
Sijun Zhang
Xiaoqing Li
Jian Yang
zhou Xun
Jinwen Ma
Venue: NeurIPS 2025 poster ()
Keywords: transformer
pre-training
large language models
Primary area: deep_learning

Abstract: Transformers have become the de facto architecture for a wide range of machine learning tasks, particularly in large language models (LLMs). Despite their remarkable performance, many challenges remain in training deep transformer networks, especially regarding the position of the layer normalization. While Pre-Norm structures facilitate more stable training owing to their stronger identity path, they often lead to suboptimal performance compared to Post-Norm.  In this paper, we propose **HybridNorm**, a simple yet effective hybrid normalization strategy that integrates the advantages of both Pre-Norm and Post-Norm. Specifically, HybridNorm employs QKV normalization within the attention mechanism and Post-Norm in the feed-forward network (FFN) of each transformer block. We provide both theoretical insights and empirical evidence to demonstrate that HybridNorm improves the gradient flow and the model robustness. Extensive experiments on large-scale transformer models, including both dense and sparse variants, show that HybridNorm consistently outperforms both Pre-Norm and Post-Norm approaches across multiple benchmarks. These findings highlight the potential of HybridNorm as a more stable and effective technique for improving the training and performance of deep transformer models. Code is available at https://github.com/BryceZhuo/HybridNorm.

HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization
Zhijian Zhuo
Yutao Zeng2.
Ya Wang2 Sijun Zhang²
Jian Yang*
Xiaoqing Li
Xun Zhou² Jinwen Ma
School of Mathematical Sciences, Peking University 2 ByteDDnce Seed
3 Beihang University
Capital University of Economics and Business
Abstract
Transformers have become the de facto architecture for a wide range machine learning tasks, particularly in large language models (LLMs). Despite their remark- able performance, many challenges remain in training deep transformer networks, especially regarding the position of the layer normalization. While Pre-Norm structures facilitate more stable training owing to their stronger identity path, they often lead to suboptimal performance compared to Post-Norm. In this paper, we propose HybridNorm, a simple yet effective hybrid normalization strategy that integrates the advantages of both Pre-Norm and Post-Norm. Specifically, HybridNorm employs QKV normalization within the attention mechanism and Post-Norm in the feed-forward network (FFN) of each transformer block. We provide both theoretical insights and empirical evidence to demonstrate that Hy- bridNorm improves the gradient flow and the model robustness. Extensive ex- periments on large-scale transformer models, including both dense and sparse variants, show that HybridNorm consistently outperforms both Pre-Norm and Post-Norm approaches across multiple benchmarks. These findings highlight the potential of HybridNorm as a more stable and effective technique for improving the training and performance of deep transformer models. Code is available at https://gi ggt.ttttu.bb.... comm/Brryeehuoo/iyyrddoooor
Introduction
Transformers have become the backbone of large language models (LLMs) and a wide range of machine learning applications. These architectures capable of modeling long-range dependencies through self-attention mechanisms, which have made them the preferred choice for a variety of tasks, including language modeling, machine translation, and image processing [.3]. However, as transformer models become deeper and more complex, ensuring stable training remains significant challenge. One critical factor that influences training stability is the choice of normalization methods, which is crucial for mitigating issues such as internal covariate shift and gradient instability [4. Effectively addressing these challenges crucial for fully harnessing the potential of deep transformer models in large-scale applications.
In transformers, Layer Normalization (LayerNorm) (5) plays central role in stabilizing training by normalizing the activations within each layer. The two predominant strategies for applying LayerNorm Pre-Layer Normalization (Pre-Norm) and Post-Layer Normalization (Post-Norm), each with its respective benefits and trade-offs. In the Pre-Norm architecture, normalization is
Corresponding author: Yutao Zeng (yutao.zeng @ottoooooooo.coommm and Ya Wang (wangyazg @ gmail.com).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

applied before the residual addition, resulting in more prominent identity path that facilitates faster convergence and more stable gradients [4. This design is particularly advantageous when training deep models, as it helps mitigate gradient-related issues that can arise during backpropagation. However, while Pre-Norm can stabilize training, it often leads to inferior final performance compared to Post-Norm [6]. In contrast, Post-Norm applies normalization after the residual connection, resulting in stronger regularization effects, which contribute to improved model performance. This approach has been shown to improve the generalization ability of transformers, particularly in very deep networks 77. Further discussion related work is provided in Appendix
Despite the benefits of each approach, there is an inherent trade-off between training stability and final model performance. Pre-Norm structures typically stabilize training but may underperform in terms of generalization, while Post-Norm architectures provide better performance but can be more difficult to train, especially in deep models. To reconcile these trade-offs, we propose hybrid normalization method that applies QKV normalization in the attention mechanism and Post-Norm in the seed-forward network (FFN), which named as HybridNorm. The QKV V normalization in the attention mechanism stabilizes the flow of information between layers by normalizing the query, key, and value components, while Post-Norm in the FFN ensures the effective depth of the transformer.
Through extensive experiments on large-scale models, we validate the effectiveness of our approach. Our results show that the hybrid normalization method significantly outperforms both Pre-Norm and Post-Norm across multiple benchmarks, providing a stable training process and improved model performance. We believe that this hybrid approach offers a promising solution for enhancing the training stability and performance of deep transformer architectures, particularly in the rapidly evolving field of L.MSs. The main contributions of this paper can be summarized as follows:
We propose HybridNorm, a novel hybrid normalization structure that combines the advan- tages of Pre-Norm and Post-Norm, offering simple yet effective solution to enhancing performance in large transformer models. Our method designed exploit the strengths of both normalization approaches, ensuring robust convergence during training and superior final performance.
We present both theoretical and empirical analyses of HybridNorm, demonstrating its advantages in enhancing gradient flow stability and improving model robustness. Our findings underscore the method's effecctiveness in mitigating core challenges inherent to deep transformer architectures.
Through extensive experiments on large-scale models (550M - 7B), we empirically validate the effectiveness of our approach. Our results show that hybrid normalization significantly outperforms both Pre-Norm and Post-Norm across variety of tasks, leading to more stable training and improved model performance, particularly in the context of LLMs.
2 Preliminaries
Scaled Dot-Product Attention The scaled dot-product attention computes the attention scores between the Query (Q) and Key (K) matrices, scaled by the square root of the key dimension de, and applies these scores to the Value (V) matrix. The formulation is expressed as
QQKT
attn (Q, K, V) = softmax
V.
Vd
where Q, K, V E Ruxds represent the query, key, and value matrices respectively, and n is the sequence length.
Multi-Head Attention Multi-head attention (MHA) extends the sealed dot-product attention mechanism by splitting the query, key, and value matrices into h heads, each of size dg d/ h.h. Each head independently computes attention scores, and the outputs are concatenated and linearly projected to the original dimension,
MHA(X) = Concao Chonca(heeady...... head, where head, attnnQi;, K₁, V2) for i = 1, 1.2,,...h { = Split(XW.) for E (Q, VJ. and Wo, Wx. Wr, Wo E Rdxd are learnable parameters. By enabling the model to focus on different subspaces of the input representation, MHA enhances the transformer's capacity to capture diverse patterns in the input sequence.
(2)
2

---

Nor
L
L
FFN
FFN
FFN
Norm
Norm
Norm
Norm
Linear
Linear
Linear
Attention
Attention
Attention
Attention
Norm
Narm
Norm
Nom
Norm
A
+x
X
4V
Linear
Linear
Linear
Linear
Linear
Linear
Linear
Linear
Linear
Linaar
Norm
Norm
Figure Illustrations of different transformer layer structures: (a) Post-Norm architecture; (b) Pre-Norm architecture; (c) Pre-Norm with QK-Norm architecture; (d) HybridNorm architecture.
2.1 Post-Norm and Pre-Norm
The transformer architecture is composed of a stack of I. blocks, each consisting of two key compo- nents: MHA and FFN. Residual connections and normalization layers are applied around both the MHA and FFN in each block to facilitate effective training and improve model stability. Figure (b)&(b) illustrate Post-Norm and Pre-Norm, respectively.
Post-Norm Post-Norm applies the normalization layer after the residual connection in each trans- former sub-layer. Formally, the output of Post-Norm can be expressed as
y Norm(MHA(X")) + +X), x = Norm Norm(FFN((Y)) YY),
where Norm denotes RMSNorm or LayerNorm
Pre-Norm In contrast, Pre-Norm normalizes the input to the sub-layer, which allows for a more prominent identity path. The output of Pre-Norm is given by
y (HH((Nrr(XX))) + +X, x FFN(Nor((Y")) + Y
This structure facilitates better gradient flow and stable convergence, particularly for deep models. However, its reliance normalization before the residual connection lead to suboptimal perfor- mance compared to Post-Norm, as the normalization does not account for the interaction between the residual connection and the sub-layers output. An analysis of the fundamental differences between the two approaches is provided in the Appendix
3 HybridNorm
To address the trade-offs between Post-Norm and Pre-Norm, propose HybridNorm, hybrid normalization strategy that integrates their strengths. Specifically, HybridNorm combines QKV.
Norm 9 [0 in MHA and Post-Norm in FFN.
QKV Normalization in Attention In the attention mechanism, the query, key, and value matrices are normalized individually before computing the attention output. The normalized QKV matrices then used in the scaled dot- doo-put attention. QKV-Norm enhances the stability of model training

---

Pett
Poscul
0.12
Pybrishom
Hyor dh.om
Gradlent Norm
0.10
0.4
Gradleet
0.08-
0.4
0.00-
0.00
Layer Index
Layer Index
(a) Layer gradient norm at stop 1.
(b) Layer gradient norm at step 100.
Figure? 2: Gradient norm of Pre-Norm, Post-Norm, and HybridNorm at different training steps.
and leads to improved downstream performance. Formally, attention with QKV-Norm is defined as attnokv (Q, K, V) softmax
[Nrri(Q)oorm(K)
Norm(V).
(5)
vdk
And denote the multi-head attention with atlioky MHAQKV.
HybridNorm Architecture Combining the above, the overall output of a transformer block with HybridNorm can be expressed as
y (HA) x FFN(Norm(YY)) + Norm(Y').
(6)
The architecture illustration can be found in Figure d)) and the pseudocode shown in Algorithm [ By integrating QKV normalization in the attention mechanism and Post-Norm in the FFN, HybridNorm achieves stable training dynamics and enhanced final performance. The theoretical gradient analysis can be found in Appendix B
The method most closely related ours is Mix-LN [Лly, which applies Post-Norm to the earlier layers and Pre-Norm to the deeper layers, resulting in enhanced training stability and performance. In contrast, our proposed HybridNorm integrates Pre-Norm and Post-Norm within each transformer layer, thereby providing a more uniform approach across different layers to leverage the benefits of both normalization strategies. Moreover, experiments demonstrate that HybridNorm achieves superior downstream performance compared to Mix-LN (see T Table T2).
Remark 1.
Special Treatment of First Block Inspired by prior work [12]. which employs the Mixture of Experts (MoE) architecture with specialized handling of the first layer, we explore the impact of introducing specialized normalization to the first transformer block. In our approach, the first layer of the transformer is treated differently by applying Pre-Nome on MHA and FFN, while maintaining V- Specifically, the structure of our first layer defined as
Y MHAqkv (Norm(X")) X x FFN((orm((Y)) Y
We refer this variation of HybridNorm, which incorporates the specialized first block treatment, as HybridNorm". This design aims to stabilize the training of the first transformer block and boost overall performance by improving the flow of gradients in the early stages of training.
Theoretical Analysis
4.1 Benefits of Hybrid Method
To gain deeper insights into the stability introduced by HybridNorm, we follow the approach of B3 and analyze the evolution of gradient norms throughout training iterations. Suppose and F' are the input and the sublayer of Transformer, respectively. The output of Post-Norm is
PPost Norm(x + (x)) and the output of Pre-Norm is ypre + F(Norm(:)). Then we have
OyPost
@Nom + F(x))
1 aF(z)
(8)
dr
O(x + F(x))
dr

---

Øypre Ōx
OF (Fooomm())) @Norm(a) (Norm(x)
=I
dr
va where the gradient of normalization is @Norm(x) a TH 8x
1= The gradient of Post- Norm is the product of two gradients, one of which is the normalization. If the spectral radius of the normalization gradient is less than 1, it causes gradient vanishing. In Pre-Norm, the residual connection is isolated from the normalization, ensuring that gradients retain lower bound, thereby preventing gradient vanishing. To ensure relatively stable gradicnts, a natural idea is to use both types of normalization within Transformer block, leading to Pre-Post (Pre-Norm in MHA and Post-Norm in FFN) and Post-Pre. From T we find that Pre-Post achieves the best performance, which is why adopt this. And placing Pre-Norm in MHA with QKV-Norm further enhances performance.
(1- J.r
In Figure we compare the gradient norms of Pre-Norm, Post-Norm, and HybridNorm at steps and 100. The results indicate that Pre-Norm tends to exhibit gradient explosion in deeper models, while Post-Norm suffers from vanishing gradients, both which hinder effective optimization. In contrast, HybridNorm maintains well-balancee gradient flow throughout training, effectively mitigating these issues. An intuitive understanding is that Pre-Norm tends to amplify gradients, while Post-Norm diminishes them. HybridNorm alternates between these two normalization strategies, leading to more stable gradient propagation during backpropagation and effectively preventing gradient explosion or vanishing. This balanced gradient propagation contributes to smoother optimization dynamics and faster convergence, further reinforcing the effectiveness of HybridNorm in stabilizing training.
4.2 Benefits of QK QKV-Nor
Theoretically, we study how QKV-Norm affects the gradient flow during backpropagation, which is crucial for training stability deep transformer models. Our analysis reveals that QKV-Norm helps decouple gradients between different weight matrices, thereby stabilizing training.
Theorem (Informal version of Theorem
2. Suppose the the output of the attention S, the input X E Rxd parameters Wo, WK, Wv, W E Raxd, For the attention with Pre-Norm, we have
as O (11Wv112)
as
O ((IWoll=),
oW
as
as oW
O (1I (IlWklal/Wvlll (a
O ([WalallWv//a/WWoll))
@WQ
F
For the attention with Pre-Norm and QK-Norm, we have
as
as
as
as oWK
ow -(llo)))F)
=O(]WW/l/\Woll))
aW
OW F
For the attention with QKV-Norm, we have
as aw
as dw
as (Iloll)),
as
= -1),
O (IlWol22)
oW
=
aW
F
The above theorem an informal version of Theorem with a more precise statement and proof provided Appendix In attention with Pre-Norm, gradients of weights exhibit strong dependencies on other weights; for instance, Wo and WK are influenced by all three other weights but not by themselves. In contrast, with QK V-Nomm,, the gradient of each weight depends at most on itself and Wo. This indicates that the gradient in Pre-Norm is more tightly coupled with other weights compared to QKV-Nomm, while Pre-Norm with QK-Norm lies between the two. Consequently, during training. if the norm of certain weight becomes excessively large, it is harder to control in Pre-Norm, leading to increased gradient magnitude. This creates a vicious cycle that may cause model collapse. In contrast, V-oom alleviates this issue and significantly improves training stability. In summary, the degree of gradient coupling follows: Pre-Norm > Pre-Norm with QK-Norm > VKV-No:;; whereas training stability follows the reverse: Pre-Norm < Pre-Norm with QK-Nom < QK QKV-Norm.
5

---

Training loss
Validation loss (C4)
HellaSwag Acc. (%)
ARC-Easy Acc. (%)
3.05
HybrshNerm
2.75
60
70
2.70
2.50
30
2.55
2.50
56
54
2.00
2.65
52
Fra-Norm Myondhormm
Pytrolttm
Pre
Mybriditomm
bytoriMoomm
230
6:0
800
1030
2013
4010
500
5070
1300
250
430
650
800
1050
201
400 of Tokens (B)
600
500
1000
of Takens (8)
# of Tokenss (B)
of Tokens (B)
Figure 3: Training dynamics for 1.2B dense models with Pre-Norm, HybridNorm and HybridNorm" under IT training tokens. We present the training loss, validation loss, and downstream performance on HellaSwag and ARC-Easy, demonstrating that HybridNorm" achieves superior performance.
Table 1: Downstream evaluation results of 1.2B dense models with Pre-Norm, HybridNorm, and HybridNorm" under IT training tokens. OQA refers to OpenbookQA.
Methods
ritheeri HellaSwag SciQ ARC-C ARC-E PIQA OQA
COPA Avg.t
Pre-Norm
44.10
63.41 91.80
39.20
69.82
75.19 38.40
82.00 62.99
HybridNorm 44.12
64.22 91.88
39.13
71.05
74.72
38.88
82.00
63.25
HybridNorm"
47.21
65.12 91.38
37.06
71.79
75.72
39.16
85.78 64.15
Experiments
5.1 Experiment Settings
Baseline We evaluate HybridNorm across two series of models: dense models and Mixture of Experts (MoE) models. The dense models include two scales: 550M and 1B, with the latter containing approximately 1.27 billion parameters and utilizing an architecture similar to LLaMA 3.2 [1. All analytical experiments are conducted the OLMoE framework [15], which activates 1.3B parameters out of a total of 6.9B parameters (MoE-IB-7B). Both models are trained from scratch the OLMoE Mix dataset [15]
the 550M dense models. For the MoE model, we use
Model Configuration The 550M dense model has a model dimension of 1536, an FFN dimension of 4096, and utilizes 16 attention heads with 4 key/value heads per attention head. The 1.2B model features a larger model dimension of 2048 and an FFN dimension of 9192, with 32 attention heads and 8 key/value heads per attention head. The MoE-IB MoE-IB-7B model employs 16 attention heads, model dimension of 2048, and an FFN dimension providing more fine-grained distribution of computational resources. All models consist of 16 layers and are trained with a consistent context length of 4096. More details can be found in Appendix C
1024. Notably, it features experts out of 64,
Hy Hyperparameters Model weights are initialized using Megatron initialization [66] (See Section 5.4 for more details). For the optimization, we apply the AdamW optimizer with 31 = 0.9 and 32 0.95. All models are trained on sequences of 4096 tokens. For the dense model, we set the learning rate to 3e-4, decaying to 3e-5 using a cosine scheduler. The MoE model starts with a learning rate of 4e-4, decaying with cosine schedule. We summarize the hyperparameters in Table 10
Evaluation Metrics To evaluate the performance of LLMs with HybridNorm, we employ a diverse set of open benchmarks, including ARC-Easy (ARC-E) [17), ARC-Challenge (ARC-C) [17]
HellaSwag [18]. PIQA 19). SciQ (20), CoQA (21). Winogrande [22]. MMLU [23], BoolQ 24
COPA 25,. CSQA 26) OBQA (27), and SocialIQA (88. We leverage the LM Eval Harness 29
for standardized performance evaluation.
5.2 Main Results
Dense Models evaluate the performance of HybridNorm and HybridNorm" on 1.2B dense trans- former models. cinureBJconpares the training dynamics of dense models with different normalization methods. As shown in the figure, models with HybridNorm and HybridNorm" exhibit consistently

---

Training loss
Validation loss (C4)
HellaSwag Acc. (%)
MMLU Var Acc. (%6
2.42
Hybrishomn"
2.53
70
2.80
2.77
69
2.36
50
2.34
64
2.32
2.70
36
2.67
Pre-Nom
Pro-Marm
2.65
Hybridhuemm
100
200
300
400
500
100
200
300
400 500
100
200
300
400
50c
1200
200
300
400
500
of Takens (B)
# of Tokens (B)
of Tokens (B)
#F of Tokens (B)
Figure 4: Training dynamics for MoE-IB-7B models with Pre-Norm and HybridNorm" under 500B training tokens. We present the training loss, validation loss, and downstream performance on HellaSwag and MMLU Var, demonstrating that HybridNorm" achieves superior performance.
Table 2: Downstream evaluation results of MoE-IB-7B with Pre-Norm and HybridNorm" under 500B training tokens. OQA refers to OpenbookQA.
Methods
HellaSwag ARC-C ARC-E PIQA WinoGrande OQA
BoolQ COPA Avg.t
Pre-Norm 69,94
39,92
73.37
77.82
63.34
42.37
67.47
64.95 86.00
85.40
HybridNorm" 70.71
42.27
75.77
78.06
64.58
43.18
68.41
66.12
lower training loss and validation perplexity throughout training compared to Pre-Norm, highlighting their effectiveness enhancing training stability and convergence. Table presents the downstream evaluation results. HybridNorm" consistently outperforms Pre-Norm in most tasks, achieving the highest average score. Notably, it demonstrates substantial improvements in tasks such as Basi- cArithmetic (+3.11), HellaSwag (+1.71), and COPA (+3. 78), indicating enhanced generalization and robustness. These results underscore the scalability HybridNorm' * in larger transformer models, further validating its effectiveness in improving both training stability and downstream performance. More results can be found in Figure D In Appendix[E.] [.II we present additional comparisons with other approaches, such and entropy dynamics across different methods, as detailed Appendix
and Mix-LN. We further provide analyses of signal propagation
H.3
MoE Models For MoE models, we conduct experiments on MoE-1B-7h with 8 experts selected from a pool of 64. Figure T presents the training dynamics of MoE models under different normaliza- tion strategies. Throughout the training, HybridNorm' consistently achieves lower training loss and validation perplexity compared to Pre-Norm. These findings indicate that HybridNorm" effectively alleviates optimization difficulties in large-scale MoE models, resulting in more stable training and enhanced downstream performance. Further, shown in Table 2 HybridNorm' consistently out- performs Pre-Norm across various downstream tasks, achieving the highest average score. Notably, it demonstrates significant improvements in ARC-C (+2.35), ARC-E (+2.40), and OpenbookQA (+0.81) I), highlighting its ability to enhance generalization across diverse benchmarks.
5.3 More Experiment in 7B Dense Model
The experimental setup primarily follows the 7B model configuration outlined in (30] and the number of training tokens is 150B. To further enhance model stability, we introduce an additional normalization layer to the output of the attention module in 7B model, with its weight initialized to
1/v2L B1I, where L denotes the number of model layers. Table 3 presents the training loss and validation perplexity (PPL), while Table Table/4reports the downstream evaluation metrics.
The experimental results demonstrate the clear superiority of HybridNorm' over the traditional Pre-Norm approach in the 7B model. Firstly, HybridNorm achieves a lower training loss (2.430 vs. 2.469), indicating more efficient optimization during training. This improvement in training dynamics translates into consistently better performance across range of language modeling benchmarks. Specifically, HybridNorm" yields lower perplexity scores on all evaluated datasets, including C4, Books, Common Crawl, Wiki, and Wikitext 103. For instance, perplexity on the C4 dataset drops from 15.32 to 14.83, suggesting stronger generalization across both structured and unstructured corpora.

---

Table 3: Training loss and perplexity (PPL) comparison of Pre-Norm and HybridNomm" for 7B dense mddel across multiple datasets. CC means Common Crawl.
Methods
Loss
C4
Books
CC
peS2o
Reddit
Stack
Wiki
Pile
Wikitext
Pre-Norm
2.469
15.32
13.37
17.10
8.07
20.31
3.57
9.96
7.85
10.09
HybridNorm'
2.430
14.83
12.77
16.77
7.67
19.65
3.40
9.34
7.81
9.16
Table 4: Downstream evaluation results of 7B dense models trained with Pre-Norm and HybridNorm'. All numbers denote task accuracies (%) BA and OAQ mean Basic Arithmetic and Openbook QA, respectively.
Hella SciQ ARC-C ARC-E PIQA OQA COPA
Wino Grande BoolQ Avg.1
Methods
BA
Swag
Pre-Norm
43.50 69.03 46.57
41.47
74.95
76.71
39.40
84.00
63.00
67.43
60.61 70.06 63.06
HybridNorm"
50.67 70.77 47.44
43.82
75.82
78.93
43.77
86.01
63.32
Moreover, HybridNorm" shows consistent improvements on all downstream tasks, covering various domains such as arithmetic reasoning, commonsense QA, and natural language inference. Notably, performance on Basic Arithmetic improves significantly from 43.50% to 50.67%. Across the full suite of tasks, including ARC, PIQA, COPA, BoolQ, and WinoGrande, HybridNorm' outperforms Pre-Norm in every case, leading to an overall increase in average accuracy from 60.61% 61% to 63.06%.
5.4 Ablation Studies
Initialization To evaluate the sensitivity of Pre-Norm and HybridNorm to initialization schemes, conduct ablation studies comparing three widely used initialization strategies: Normal initialization Depth-Scalled initialization [3 [4], and Megatron initialization [16). Normal initialization initializes all weights of linear layers using a truncated normal distribution with mean zero and standard deviation 1/v2.5d, where d is the hidden dimension. Depth- initialization and Megatron initialization introduce scaling factors stabilize training in deep architectures. Specifically, Depth-Scaled initialization scales down the output projections of the attention and FFN by a factor of V21, where is the layer index. In contrast, Megatron initialization scales down these projections by VTT, where is the total number of layers, mitigating gradient variance accumulation in very deep transformers. As shown in Table Pre-Norm and HybridNorm exhibit sensitivity across different initialization methods, achieving the lowest training loss and perplexity under Normal initialization and Megatron initialization, respectively. Therefore, we set the default initialization method for Pre-Norm to Normal initialization and for HybridNorm to Megatron initialization in all experiments, respectively, which ensures that even under settings that may more favorable to baseline models, the superiority of our approach is effectively demonstrated.
32],
Normalization Position We investigate the impact of the position of normalization layers within the transformer block. First, we examine the effect of varying the placement of QKV normalization (e.g., normalization setting in attention). We extend the normalization setting by considering not only the Query (Q), Key (K), and Value (V) components but also the Context (C), which refers to the output of the attention mechanism. For instance, QKVC-Norm applies normalization to all four components: Query, Key, Value, and Context, while KV-Norm and KC-Norm focus on the normalization of the Key-Value and Key-Context pairs, respectively. QKVC-Post refers to transformer blocks that employ QKvC-Norm in the MHA while using Post-Norm in the FFN. Second, we explore the effect of integrating QKV-Norm into different transformer architectures. For instance, Pre- Pre-QKV-PPos refers to configuration where QKV-Norm is applied with Pre-Norm in the MHA layer, while the FFN layer utilizes Post-Norm. Other configurations follow similar definitions. Finally, we compare various hybrid combinations of Pre-Norm and Post-Norm. Pre-Post refers to transformer blocks that apply Pre-Norm in the MHA and Post-Norm in the FFN, whereas Post-Pre adopts the opposite configuration. Mathematical formulas for methods mentioned above can be found in Appendix G
8

---

Table 5: Training loss and validation perplexity of 550M dense models with Pre-Norm and Hybrid- Norm under various initialization methods and 400B training tokens.
Method Initialization Loss PPL on C44
Method
Initialization
Loss. PPL on C4.,
Normal
2.75
20.29
Normal
2.76
20.44
Pre-Norm Depth-Scaled
2.76
20.49
HybridNorm Depth-Scaled
2.76
20.40
Megatron 2.76
20.44
Megatron 2.74
20.00
Table 6: Abalation study of the position of normalization layers on 550M dense models with
400B tokens. We report the training loss, the perplexity on C4 and Pile, and the accuracy HS (HellaSwag).
Methods
Loss
C44
Pile|
HS f
Methods
Loss
C44
Pile
HS
QKVC-Post
2.74
20.05
10.34
52.68
Post-Norm
2.76
20.43
10.57
51.20
QKC-Post
2.73
20.00
10.31
52.26
Pre-Norm
2.75
20.30
10.48
51.97
QK-Post
diverge
QK-norm
2.75
20.22
10.43
52.29
KV-Post
2.74
20.11
10.38
52.10
Mix-LN
2.76
20.43
10.56
51.29
KC-Post
2.75
20.34
10.47
51.15
Post-Pre
2.75
20.26
10.46
51.19
Pre-QK --sst
2.74
20.13
10.37
52.65
Pre-Post
2.74
20.15
10.40
52.42
Pre-QK P-PrQ
2.74
19.97
10.33
53.05
Pre-QK-Pre
2.75
20.22
10.43
52.29
HybridNorm
2.74
20.00
10.29
53.35
QK V-Pre
2.74
19.96
10.33
52.57
HybridNorm"
2.73
19.85
10.25
53.36
As shown in Table 6 HybridNorm (a.k.a. QKV-Post) and variant HybridNorm" consistently surpass other methods. Notably. HybridNorm" achieves the lowest training loss and perplexity while attaining the highest accuracy on HellaSwag. Specifically, by comparing HybridNorm with the left first block in Table 6 we find that QKV-Norm is the most effective normalization. Similarly, comparing HybridNorm with the left second block, we observe that combining QKV-Norm with Post-Norm in the FFN yields superior performance. From the right table, one see that the Pre-Post configuration indeed leads to improved performance, while replacing Pre-Norm in the MHA with VKV-Norm to form HybridNorm further enhances performance, achieving the best results.
Special Treatment of First Block For the special treatment of the first block, we test different architectures, such as adding normalization layer after embedding (call EmbedNorm) and armed the first block with QK V-norm and Pre-Norm in FFN (call First-QK V-Pre), which formulations are:
MHAQKv(X*) + xº. x FFN(Norm(YY)) + y
(10)
As shown in Figure we can see that, except for EmbedNorm, the special treatment of the first block effectively reduces training loss and improves downstream performance.
5.5 Scaling Laws Experiments
We compare the loss scaling curves between Pre-Norm and HybridNorm' across range of dense model sizes, from 151M to 1.2B parameters. The model sizes used for the scaling law experiments detailed in Table 9 and all models are trained using the same setting and hyperparameters for fair comparison, as specified in Table TO Models with 151M, 285M, 550M, and 1 1.2B parameters trained 200B, 200B, 300B, and IT tokens, respectively. As shown in Figure 6 HybridNorm" exhibits superior scaling properties, demonstrating lower training loss as the model size increases. This highlights its capacity maintain both training stability and performance, even for extremely large models, thereby making it highly suitable for scaling to billion-pante -alloon-arramter regimes.
5.6
Deeper Models
To further evaluate the robustness of HybridNorm and HybridNorm" in deeper architectures, we conduct experiments on transformers with depths ranging from 16 to 29 layers while maintaining

---

Training loss
HellaSwag Acc. (%6
HybrioNem
Pre-Sem Hygh dlum
2,90
IybrioNomm
52
2.88
EmbedNom
Firt: QKV
50
2.85
2.83
46
2.80
46
Hybridorm Hybridltomm
2.77
44
EmbediNorm
2.75
42
100
200
300
400
100
200
300
400
128
of Tokens (B)
Takens (B)
Model Size
Figure 5: Training loss and accuracy on HellaSwag 550M dense models with different normalization methods for the first block.
Figure 6: Scaling law curves of Pre- Norm and HybridNorm'.
Table 7: Performance of dense models with different depths under 400B training tokens. We report the training loss, the perplexity on C4 and Pile, and the accuracy on HellaSwag and PIQA.
550M, 16 Layers
543M, 29 Layers
Methods
Loss
C4 Pilel HellaSwagt
PIQA Loss
C4 Pile HellaSwagt PIQAT
Post-Norm
2.76 20.43 10.57
51.20
71.80
diverge
Pre-Norm
2.75 20.30 10.48
51.97
71.14
2.73
19.88
10.31
53.86
71.63
HybridNorm
2.74 20.00 10.29
53.35
71.96
2.72
19.67
10.18
54.89
72.62
HybridNorm"
2.73 19.85 10.25
53.36
71.15
2.71
19.52
10.10
54.54
72.01
comparable parameter budget. This setup allows for a fair comparison of different normaliza- tion strategies in deep transformer architectures. As shown in Table 7 both HybridNorm and HybridNorm" consistently outperform Pre-Norm and Post-Norm across various depths. demonstrat- ing their effectiveness in stabilizing deep model training. A particularly striking observation is that Post-Norm fails to converge at 29 layers, reinforcing its dell-docu instability in deeper archi- tectures. In contrast, HybridNorm and HybridNorm not only ensure stable training across all depths but also achieve significantly lower training loss and perplexity on both the C4 and Pile datasets. These improvements indicate that HybridNorm-baat normalization strategies mitigate optimization difficulties that commonly arise in deep transformers. Furthermore, HybridNorm' achieves the highest accuracy on challenging downstream benchmarks such as HellaSwag and PIQA, suggesting that benefits extend beyond mere training stability to enhanced generalization on real-world tasks. These results provide strong empirical evidence that HybridNorm-baa normalization schemes enable deeper transformer training while preserving superior optimization efficiency and downstream task performance.
6 Conclusion
In this paper, have introduced HybridNorm, novel hybrid normalization strategy that has seamlessly integrated the advantages of both Pre-Norm and Post-Norm, thereby addressing the longstanding trade-off's in transformer training. We have provided both comprehensive theoretical and empirical analyses to demonstrate how HybridNorm has stabilized gradient propagation while preserving strong regularization effects, ultimately improving both convergence speed and final model performance. Extensive experiments across diverse benchmarks have substantiated the effectiveness of our approach, consistently showing that HybridNorm has outperformed conventional normalization schemes in terms of stability and accuracy. These findings have highlighted the importance of re- examining the role and placement of normalization within transformer architectures, paving the way for further exploration of hybrid normalization paradigms. We believe that HybridNorm has marked significant step forward in the development of more robust and efficient transformer models, offering practical ad advantages for training next-generation large-scale neural networks.
10

---

References
[1] Jacob Devlin, Ming-Woit Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
[2] Tom B Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
[3] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[4] Ruibin Xiong, Yunchang Yang. Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization the transformer architecture. In ICML, 2020.
[5] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1607 2016.
[6] Shufang Xie, Huishuai Zhang, Junliang Guo, Tan, Jiang Bian, Hany Hassan Awadalla, Arul Menezes, Tao Qin, and Rui Yan. Residual: Transformer with dual residual connections. arXiv preprint arXiv:2304. 14802, 2023.
[7] Hongyu Wang, Shuming Ma, Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet:
Scaling transformers to 1,000 layers. IEEE Transactions Pattern Analysis and Machine Intelligence 2024.
[8] Biao Zhang and Rico Sennrich. Root mean square layer normalization. NeurIPS, 2019.
[9] Stephen Menary, Samuel Kaski, and Andre Freitas. Transformer normalisation layers and the independence of semantic subspaces. arXiv preprint arXiv:2406.17833 2024.
2024.
[10] Oleg Rybakov, Mike Chrzanowski, Peter Dykas, Jinze Xue, and Ben Lanir. Methods of improving IIm training stability. arXiv preprint vxXiv22410116622 2024.
[11] Pengxiang Li, Lu Yin, and Shiwei Liu. Mix-LN: Unleashing the power of deeper layers by combining Pre-LN and post-LN. In The Thirteenth International Conference Learning Representations, 2025.
[12] DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024.
[13] Sho Takase, Shun Kiyono, Sosuke Kobayashi, and Jun Suzuki. B2T connection: Serving stability and performance in deep transformers. In Findings of the Association for Computational Linguistics: ACL, 2023.
[14] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
[15] Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Evan Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. OLMoe: Open mixture-of-experts language models. In The Thirteenth International Conference Learning Representations,
2025. 2025.

---

[16] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Img Im: Training multi- biltiibiliion parameter language models using model parallelism. arXiv preprint arXiv: 1xXiv:9909008000 2019.
[17] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint (7Xi::8803.05455 2018.
[18] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, 2019.
[19] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
[20] Johannes Welbl, Nelson Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94-106, 2017.
[21] Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019.
[22] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106 2021.
[23] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
[24] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2936,2936, 2019.
[25] Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. Semeval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In SEM 2012: The First Joint Conference on Lexical and Computational
Comantics-Volume I: Proceedings the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop Semantic Evaluation (SemEval 2012), pages 394-398, 2012.
[26] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4149 4158, 2019.
[27] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pages 2381-2391, 2018.
[28] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions. arXiv preprint arXiv: (9t:::904.0990 09728, 2019.
[29] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for shot language model evaluation, 12 2023.
12

---

[30]
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Ma- lik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christo- pher Wilhelm, Michael Wilson, Luke Zettlemyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 olmo 2 furious, arXiv preprint arxivv2501.0000 .06656, 2024.
[31] Ya Wang, Zhijian Zhuo, Yutao Zeng, Xun Zhou, Jian Yang, and Xiaoqing Li. Scale-distribation decoupling: Enabling stable and effective training of large language models. arXiv preprini arXiv:2502.15499 2025.
[32] Toan Q Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. In Proceedings of the 16th International Conference on Spoken Language Translation, 2019.
[33] Biao Zhang, Ivan Titov, and Rico Sennrich. Improving deep transformer with depth-scaled initialization and merged attention. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 999-9099170 2019.
[34] Suchin Gururangan, Mitchell Wortsman, Samir Yitzhak Gadre, Achal Dave, Maciej Kilian, Weijia Shi, Jean Mercat, Georgios Smyrnis, Gabriel Ilharco, Matt Jordan. Reinhard Heckel, Alex Dimakis, Ali Farhadi, Vaishaal Shankar, and Ludwig Schmidt. Openlm: minimal but performative language modeling (lm) repository, 2023. URL https://gitnn com/ olfoundations//pee Lm/ GitHub repository.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[36] Seniha Esen Yuksel, Joseph N Wilson, and Paul D Gader. Twenty years of mixture of experts. IEEE Iransactions on neural networks and learning systems, 2378):1 193, 2012.
[37] Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics: EMNIP 2020, pages 4246 4253, 2020.
[38] Mostafa Dehghani. Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In ICML, 2023.
[39] Zhuliang Yao, Yue Cao, Yutong Lin, Ze Liu, Zheng Zhang, and Han Hu. Leveraging batch nor- malization for vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021.
[40] Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5747-5763, 2020.
[41] Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. Learning deep transformer models for machine translation. In ACL, 2019.
[42] Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. Opennmt: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages 67-72, 2017.
[43] Ming Ding, Zhuoyi Yang. Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. In NewrIPS, 2021.
13

---

[44] Gemma Team, Morgane Riviere. Shreya Pathak. Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.000 0000080001/1 18, 2024.
[45] Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvicto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. In NeurlPS, 2022.
[46] Weronika Ormaniec, Felix Dangel, and Sidak Pal Singh. What does it mean to be a transformer? insights from a theoretical hessian analysis. In ICLR, 2025.
[47] Sidak Pal Singh, Gregor Bachmann, and Thomas Hofmann. Analytic insights into structure and rank of neural network hessian maps. In NeurlPS, 2021.
[48] Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
[49] Nandan Kumar Jha and Brandon Reagen. Relu's revival: On the entropic overload in normalization- free large language models. arXiv preprint arXiv:2410.09637, 2024.
14

---

Appendix
Table of Contents
Related Work
16
B
Thcoretical Gradient Analysis
Theorem 2
B.2
Proof Lemma
19
B3
Proof of Lemma
20
B 4
Proof of Theorem
20
B.5
Expand Theoretical Claritiatioons to LayerNorm
25
Details of Experiments
26
PyTorch Style Implementation HybridNomm
26
Architectures of Different Models
................................
26
Hyperparameters for Pretraining
26
D
Computational Overhead
27
Additional Experimental Results
28
E.1
Comparison with Other Methods
28
Signal Propagation
................................
29
E.3
Entropy Dynamics
29
E,4
Overall Results for Dense Models
29
E.5
Overall Results for MoF Models
29
F
Essential Differences Between Pre-Norm and Post-Norm
30
Formulas for Different Positions Normalization Layers
32
H
Broader Impacts and Limitations
35
H.1 Broader Impacts
35
112 Limitations
35
15

---

A Related Work
Architecture Modifications in Transformers Recent eftorts in transformer architecture modifica- tions have sought to optimize both the computational efficiency and the expressiveness of the model. These efforts include changes to the attention mechanism and feed-forward networks all aimed at improving performance variety of tasks, ranging from language modeling to vision tasks [5 3 For example, Multi-head Latent Attention (MLA) [12], Mixture of Experts (MoE) [66. While these modifications contribute more efficient training, they also require careful integration with other components, such as normalization layers, to maintain model stability and performance.
Normalization Types in Transformers Normalization layers integral to the success of deep learning models, and transformers are no exception. The most commonly used normalization tech- nique in transformers is LayerNorm (5). which normalizes the activations of each layer independently. However, alternative methods such as R.MNNrrm [8], which normalizes using root mean square statisties, have been proposed more effective alternatives in certain settings. These methods designed to mitigate the challenges of internal covariate shift and gradient instability, which are critical for the success of large-scale transformer models.
Normalization Settings in Attention For training stability, QK-Norm [37 38 modifies the stan- dard attention mechanism by applying normalization directly the query (Q) and key (K) components during attention computation. Building upon this, QKV-Norm [ [0] extends the approach by nor- malizing the Query (Q), Key (K), and Value (V) components. This comprehensive normalization ensures that all critical components of the attention mechanism are normalized, resulting in enhanced stability and improved performance.
Location of Normalization Layers Recent research has also explored the impact of normalization location in both Vision Transformers [39 40) and language models [41 40]. For example, the choice between Pre-Norm and Post-Norm architectures has been widely studied in the transformer literature 41 Pre-Norm, where normalization is applied before the residual connection, has been shown to be more stable in deep networks and accelerates convergence 44. Although Post-Norm more challenging to train, it tends to deliver better final performance by normalizing after the was proposed strategy address training instability in deep transformers, which scales the residual connections by carefully chosen factor to improve gradient flow and mitigate exploding vanishing gradients. Ding et al. [ introduced Sandwich-LN in multimodal settings to improve training stability, a strategy that has also been adopted by the
35
residual connection [40]. DeepNorm
Gemma team in their recent models 44). Similarly, OLMo-2 (30) applies the normalization layer after the sublayer but before the residual connection, differing from both traditional Pre-LN and Post-LN schemes. The method most similar to ours is Mix-LN [T], which applies Post-Norm to the earlier layers and Pre-Norm to the deeper layers, achieving improved training stability and better performance. In contrast, our HybridNorm integrates Pre-Norm and Post-Norm within each transformer block. This intra-layer hybridization offers several key advantages: (1) consistently improved model performance, (2) intra-layer hybridization ensures uniformity across all layers, facilitating other post-training such as pruning and quantization.
B Theoretical Gradient Analysis
For simplicity, we consider a single-head attention layer. The input is X E Raxd representing sequence of s tokens with dimension d. Throughout this section, we denote RMSNorT as Norm (.). ie., (orm(w) = a @ RMS(#) for x Rd, where RMS(x) = + + /dd. For further simplicity, set 1. The learnable parameters WΩ, WK, Wy E RRdx and Wo E Re Let XN NXrm(X), M = block with Pre-Norm then given by
= X X K XT and A = softmax(M) (M). The output of the attention
S AX
(11)
'Given that the vast majority of popular LLMS are based on RMSNorm, our experiments and conclusions are broadly applicable to standard LLM architectures. Futher, in Appendix B.5 we demonstrate that RMSNorm and LayerNorm exhibit no fundamental differences, both in theoretical analysis and empirical observations.
16

---

Defining Q= XWQ = XWK. and V = XWv. with their normalized counterparts QN Norm(Q). KN (K), and VN Norm(V), ) the output of the attention block with QKV- Norm is formulated as
SN = ANVNWo. Vd, QNKK
(12)
where AN softmax(Mx)) and MN
Defining Q X and KN NNNWa with their normalized counterparts QN (Q) and KN = Norm(R))))) (K), the output the attention block with Pre-Norm and QK-Norm formulated
ANXNWyWo.
(13)
where AN = softmax(MM) (Mx) and MN = Jdz QNKI.
Following the prior work of Noci et al. [5). Ormaniec et al. 46 we analyze the gradients by computing derivatives using row-wise vectorization and arranging the Jacobian in the numerator layout, i.c.
OY (Y)e,((Y Ovec, (X)T
0X
The following derivation primarily relies on the chain rule and the following rule
dAWB B
A
(14)
Ow
where A E Rmxn W E R"xp, B E IRXx and @ is the Kronecker product. The proof of Eq. 14 can
be found in 47
B.1 Theorem
We first present the following extension of Lemma 2 in Noci al. 455
Lemma (Extention of Lemma 2 in Noci et al. (5)). The gradients of the attention with Pre-Norm defined in Eq. (17) are given by
05
XNWaWTXN
aWo softmax
XNWy & La.
(15)
Vdk XNWOWAXN
0s
softmax
XN & Wo.
(16)
dWv
Vdh
0S
0A
XN & XNWK
(I, & wJW..x)
(17)
aWQ
AM
dd
0s
AA
(NNWQ 2 XN
(T, 3 WJWTXH
Kdd,d
(18)
aWk
aM
dds
where the gradients of the softmax with respect to its inputs is
0A = blockdiag
@AA.: iM.,
blockdiag (diag( (A.)) A.,
(19)
0M
As. is the i-th of A in column vector format, and the commuttation matrix Kd is permutation matrix that transforms the row-wise vectorization of WK into the column-wise vectorization of k. i.e.,
Vvec, (Wx) vec, (WC,.
The gradient of Xx with respect X
aXx 0x
@Norm(X)
(NNorm(X..) 0x
Vd [X,,,12
X
blockdiag
blockdiag
La
ax
XX,,,
(20)

---

where x is the i-th row of X represented as a column vector. The definitions of 8Qg. aQ
0Kg 8K
0Vx. ov
BQN and øR* follow similarly, i.e. for E 1Q, V, V, aq ak
NOn N
vdk
blockdiag
Id
(21)
12
Lemma 2. The gradients of the attention with QKV-Norm defined in Eq. 12 are
OSN
softmax
Norm(XWy) @ Id.
(22)
aWo
vdx
OSN
(orm(XWq)Nooe)
OVN
softmax
Wo
(X Idc),
(23)
dWy
dd
OV
OSN (I, & Norm((XWW
OAN
I. @ Norm(XWa
aQN (X @ Ida),
(24)
aWQ
0MN
dd
aQ
OSN (T, 6 W Norm(XWv) 0MN
0AN
Norm(XXQ)+
0KN (X 69 Id,); (25)
Kd OK
OWk
ddk
where the definition of 0My s.t., Kd vec, (KN) vec, (ec, N.
0AN is similar to
0A
and Ka is the commutation matrix
8M
Vec,
The proof of Lemma 2 is provided in Appendix B.2 Similarly, for the attention with Pre-Norm and QK-Norm, we derive the following lemma.
Lemma 3. The gradients of the attention with Pre-Norm and QK-Norm defined in Eq. 13 are
a$ softmax
Norm(Xx NoNN(((XWWW)
XNWV Ia-
(26)
awo
vdk
as softmax
Norm XN W (oVo))or((XXX) (XNWK)
XN @ WT.
(27)
DWv
Vdu
a$
DÂN
Ig Norm(Xx
aQN (XN Idc),
(L. 2 WJ W.
(28)
aWQ
0MN
dp Norm(XxWo)&11, & ddk
iQ
a$
dÂN
OkN (XN & lad,,
aWK (Is WW
Kan 0k
(29)
0MN
where the definition of aAv is similar to 8M and Kays is the commutation matrix RMN
s.l., vec, (Kx)= = vec, J.
The proof of Lemma can be found in Appendix
B.3
Armed with the above lemmas, we arrive at the following theorem, which characterizes the gradient norms of Pre-Norm, Pre-Norm with QK-Norn and QKV-Norm.
Theorem 2. For the attention with Pre-Norm, we have
as O (0sosssvvii (sVdlWwllz).
(30)
awo
as
O
(s) (oll)))
(31)
owv as
(s)3/2
| T
wwwlaalWwillllWoll
(32)
aw
2Vd
as
(s)3/2
WallallWilllWWol
(33)
aWx
Vdz
For the attention with Pre-Norm and QK-Norm, we have
a$
= (svalWwvll).
(34)
aWo
18

---

as
(s]lWoll))
(35)
dWv
0$
sv
|Wvlla/Woolk
(36)
min
a$
/vsdk
||WrllalWoolla]
(37)
θWk
o k
min
For the attention with QKV-Norm, we have
8SN
0 (sV)).
(38)
aWo
0SN
sdk
l|oll
(39)
dWy
min
0SN
sy Q min
|Woll2
(40)
@We
0SN
sy sdd
||oll2
(41)
oWK
o K
min
where o oin oiin oiin are minimal singular value of Wo. WK. Wv, respectively.
Theorem 2 presents the gradient norms of various methods, and its proof is provided in Appendix B.4 In the attention with Pre-Norm, the gradient of the weight matrix exhibits strong dependencies on other weights; for instance, WQ and Wx influenced by all three other weight matrices but not by themselves. contrast, in the attention with QKV-Nomm, the gradient of each weight matrix depends most itself and Wo. This suggests that the gradient of the attention with Pre-Norm more tightly coupled with other weight matrices compared to the gradient of the attention with QKV-Norm. Whereas the attention with Pre-Norm and QK-Norm lies between the methods. Therefore, during the gradient optimization process, if the norm of a certain weight becomes excessively large, it is more challenging control in the attention with Pre-Norm, leading an increase in gradient magniude. This, in turn, creates vicious cycle that may result in model collapse. In contrast, the attention with QKV-Norm alleviates this issue some extent, which significantly benefits the stability of model training. Regarding the degree of coupling, the relationship follows Pre-Norm > Pre-Norm with QK-Norm > QKV-Norm, whereas for training stability, the hierarchy is reversed: Pre-Norm < Pre-Norm with QK-Norm < QK
B.2 Proof of Lemma 2
Proof of Lemma
2
For aSN oWo
according to Eq. (14. obtain
ASN
Norm((W. Norm(XWv) & la- Vdz
AN La softmax
aW.
For asy, OWy
using the chain rule and Eq.
14 we have
OSN = aSN oV Ov awy OVN OV dwv
OVN OV
(A WT)
(X lde) Norm (omm(WWol) Nor))(o)( 8 W
OVN 2V (X 8 la.)
softmax
Vdz
For 88x
using the chain rule and Eq. 14).
we obtain
aWq
OSN
OSN OAN 0MN aQ, aQ
aW DAy OMN JQN
eQ We
19

---

OAN
( Is &T&KKN KN
(I, @ W
(X @ Idee
dMN
Vdkk
iQ
0AN
11 & N.NomXXWW))
0QN aQ
(I, @ (orNomXXWW))
(X @ la.)
ŌMN
dp
Similarly, for aSx OWx
we have
NS
OSN DAN dMN OKN 0K
OWk
DAN OMN OKN OK WK
OAN 0MN
QN 8 1.
Ovec, (vec,(KN)) 0KN
=
(L. & WIV))
du
Ovec, (KN) 0K
(X & Ide)
OAN 0MN
(Norm (WQ) @
ÖKN
= =I1 & Norm((WW)))
Kd 0K
(X @ Idz)
Vd
B.3 Proof of Lemma
3
Proof of Lemma
For a8 awo according Eq. (14). obtain
as
ANXNWy la softmax Norm(Xxx
Nor XNWV so la- ddk
awo
For as
awy, using Eq. 14). we have
a$
Norm(XxWa (orm(XXX )
= AN & W = softmax
XN @ WT.
dwv
vdk
For as 8Wa
using the chain rule and Eq. (4), obtain
as as dÂN aMN aQN aQ
aWQ dÂN aMN aQN aQ WQ
dÂN
1a & KN
JÂN
= =(
(XN 8 Ia,)
aMN
vdk
aQ
OÂN
I. 8 Norm(XxWkk)
0QN
-1, 60 W W.WTX X
(XN @ Idw)
0MN
d;
aQ
Similarly, for as DWk
have
as as DÂN OMN OKN 0k OWK
OÂN OMN OKN ok WK
dÂN
[N & I,
Ovec, Ovec,(KT) OKN Ovec,(KN))
-4. (1&WWW W.WWWW WJXW) X
(XN @ law)
dMN
dx
0K
dÂv
Norm(XxWoo @ Is Vdl.
ØkN (XN & lde).
= -1. W.oWWo.Jx
Kd 0k
0MN
B.4 Proof of Theorem
2
The proof is primarily based on the following facts
tr & C) tr(B)tr(C)
20

---

(B&C) =
- (BD) @ (CE)
[ C ||||||||||
[ Cl12 = |B||||C||
|BC _ |B||
|BC|| - || ||||||||||
[xx Fr v
If E R pi 0 and E Pi then |[dia(()) -pp 11 S
Proof. According to Gershgorin Circle Theorem 48 every eigenvalue of diag(p) pp'
lies within
UpI pi: p² - Σ PiPj. Pi -p² +
Σ
p.Pjl
1=1
j
UppiL -pi) - Pi Σ
Pj.Di(L p4) Ps Σ pjl
i=1
jzi
j&i
UlPi(1 -pr) -p(( p:). - p2) px(1 - p.)
i=1
O 1,,2pp/ 2p:(1 - P.)
i=1
C [0.
2
Therefore, [[diag(p) - pp T lla S 1. When pl = P2 2. the equality holds, indicating that this bound is tight.
E Rexe a stochastic matrix, ie., A1, 1, and each entry is nonnegative, then ||ll|2 S ||A||E S Vs.
Proof. Note that
|
ai
=
iml j=1
1ml
j=1
i=1
If A 1,61, then
|| = v Ammax(AT (ATx = (m = V8.
Hence, the bound tight.
Proof of Theorem According fundamental algebraic operations and Lemma I obtain
as
| S /W w sva |AXx & Talls
awo
as
= =I|AXN & Wolls ||Axl|l||Wolll= = |4Id|///|=//Wllll
|| s sloWol..
dwy F
as
11. N.oWWWW.xx'
0A
XN @ XNNK
oW
dM
Vdz
[|,,, @ Mo x blockdiag(diag/ (Ai AL.AT)((N @ XNWk)lle
Vdl
21

---

1 [bockding ( A..4 @ XNWK))lP
vd
XNI. T @ M (ing (A1.) A1.
1
vdk
XNo.
(A..) A.)
1
[xx]|:|Woll>//WVlll |
vd
$
2/dx
(s)8/2
2/
as OWk
0A
XN XN
(I. 2 Wo (J
Kie,d
ddx
P
0A
Nx & XN
(1. & WowTJ
aM
vdl
blockdiag(diag)) (A..) (X, We XN)IlF
Vixe
||blockdiag (W W (daagg(AA)))) (X.4 & XN)IlE
[XINWa) W W W A AT,
dds
(XxW & W A.. F
1
$ de
Wwlpl//lS//ll//Wollll////////// 111xN11
< 2/dx
||V]l/////l///oo////XXX//
(s)3/2
2v
Therefore,
as
0
(sVdlWvlla)
aWo 0s
(s|lWollr)
awv
as
(8)3/2
O
[ la Wy Welllllllllllllll
aWQ
2/d
as
(s)
WelllllWiillllWool
aWk F
Vdz
Since the attention with Pre-Norm and QK-Norm lies between Pre-Norm and QKV-Norm. its proof can be directly derived from those of the other two. Therefore, we defer its proof to the end. As for the attention with V- have
OSN awo
VN @ TallF = || & VallAwlellvvl S svd.
09x For DW. we have
0Sw = dwv F
@Vv Wo)
aV (x 0 Tan)
22

---

@V 0V F
||
(X @ ldt
JVN aV
S Welloollz
(X @ Ide
According to Eq. 21 we get
OVN
vdz
V1, 1.
(X & Ida)
blockdiag
Idz
(x Ian
0v
|V1,,11
1IVE,:13
X1.
vdl
Idu
VV
TIVi..1
X @
Ian
V.
TTV.. 2
VV.ill
| 11
(du
1)
V2.:
, 2
di (dz 1) Σ
W
X 12
Voda
V
min
Similarly, we can get
iQN iQ
(x s Ta.))
Voda
0KN OK
(X s Tan)
Voda
ok
o min
follows that
OSN aw
sdk
|Woll22 σ min
(42)
05x For
we have
aW
OSN aw
= (T. 0 WoVH)
DAN
1. & KN
OQN
(x & Tan)
2MN
Vdl
eQ
F
(I. @ WTVN)
DAN dMN
(I, @ KN)
DQQN (x 2 laaa
ViTa
vada (4. KN)ll, Q
[1 worW blockdiag(diag((A/i))
dd
min
Vodk
[ VN (diag((AN)).) (AW).,
min
v ssd
[oolll| KNl12
min
8y /dx
|olle
2 Q min
asn Similarly, for 0Wx
we have
0Sx @Wk K
sdk
|oll2
2 K
F
min
23

---

Therefore,
OSN
0
(vva
OWo
0SN
sdk
l|ollla]
OWv
0SN
sysdk
|Woll2
@WQ
mii
0SN
sysdy
||oll22
dWk
Finally, we present the proof for the attention mechanism with Pre-Norm and QK-Norm. For as_ and as aWo
awo
whose proofs are essentially identical to that of Pre-Norm, we have
as aWo
AN Ta =IWWwX s Val SvvilWvlll.
F
as
= AN @ Wo = ||AxxwllellWool= & 1l s|WWoles.
dWv
F
as For
and
as 3W
whose proof' are similar to that of QKV-QQQNoo we have
BWg
as
dAN
1. (LOKN) KN)
aÂN
(I. Wo
(XN @ law
aw
0MN
dk
8Q
DÂN (I, & KN) 0MN
|/
|Q2 (XN @ Idaa
(Is Wo W x
vdx
aQ
(T, & Wo W. x blockdiag(digg((AWW... - (Aw),. (1, @ RN)
Vsda
vd
σ min
ssdk
blockdiag( W WTJ (dia)((()))))) (AN)
min
/sdk
[WolaalWW KNll2
min
Vsdk
WWllallWillavs33
Q min
s/sd
WWlll
20 min
Q
Similarly,
a$ aW K
sV sd ||W]l2/|Wolk
20 2 min
Hence,
as
= (sVdlWivllz)
aWo
0$
0 (s|[Wol)
dWv
24

---

as
sy sdk
0
| Wwlll//ooll
@ a$ oWk K
? min
svsde
o
||WvlallWolll)
d K min
B.5 Expand Theoretical Clarifications to LayerNorm
In the following, we demonstrate that RMSNorm and LayerNorm exhibit no fundamental differences, both in theoretical analysis and empirical observations. It is worth noting that, given the vast majority of popular LLMs are based on RMSNorm, our experiments and conclusions are broadly applicable to standard LLM architectures.
Suppose the input given by E gaxd Let _ Then
1 2 Lald and - x EX X
xx- -XP.
x
For simplicity, we omit the affine transformation in the normalization layers. It follows that
-X
- EX
XP
LayerNorm(X)
RMSNorm(XP).
Var(X)
RMS(X
EX)
RMS(XP)
Hence, analogous to the gradient of RMSNorm (Eq.
20
the gradient of LayerNorm is given by
(LayerNorm(X)
@RMSNorm (XP
0X
0x
[RMSNorm((XP).
blockdiag
0x;
Vd
(XP) P) (XPP). p).
blockdiag
Ta
(XP))11
where X; means that the i-th column of X. Note that P positive semidefinite matrix with eigenvalues bounded between 0) and 1, hence |||||2 = 1. As result, the gradient norms of LayerNorm and RMSNorm differ only by constant factor. This implies that the main result, Theorem 2 also holds for LayerNorm.
On the experimental side, conducted controlled comparison using models of identical size (550M parameters), each trained on 400B tokens. Both models adopt the Pre-Norm architecture and employ the Megatron initialization scheme; the only difference lies in the normalization method one one uses RMSNorm, while the other uses LayerNorm. As shown in Table the training losses of RMSNorm and LayerNorm are nearly identical, with a marginal difference only 0.0008. In the context of large-scale models, a loss difference smaller than 0,001 is typically considered negligible.
Table 8: Training loss comparison between RMSNorm and LayerNorm under identical training settings.
Methods
Training Loss
RMSNorm
2.7631 2.7639
LayerNorm
25

---

Details of Experiments
C.1 PyTorch Style Implementation of HybridNorm
We provide PyTorch-style tion can be found at implemmttiinnnn of HybridNorm below, and more detailed implementa-
Algorithm PyTorch style pseudocode for Transformer block with HybridNorm
# q_norm, k_norm, v_norm, ffn_norm are normalization layers attn_pro] and attn_out are linear layers attn is the attention ffn is the feedforward network
def forward(x):
Attention block
shape (b, 8, d)
9, k, attn attn_proj (x) split d, d), dim=-1) # dk = =d / h is the number of attention heads 9, k, v ".veiewb,, s, h, dk)
# shape (b, d)
k, v +._oor((q), k_norm(k), v_norm(v) attn(q.k,v)
# shape (b, d)
x attn_out(x) + res
FFN block
ffn_norm (x) [xn(x)
return
C.2 Architectures of Different Models
For dense models, we adopt decoder-only transformer architecture akin LLaMA 3.2 14, with model sizes ranging from 151M to 1.2B parameters. For the MoE model, we follow the structure of OLMoE [15]. The specific architecture of models is summarized Table p All experiments are conducted on NVIDIA A100-80G GPUs, utilizing 32 GPUs for dense models with fewer than IB parameters, and 64 GPUs for the 1.2B dense model and MoE- IB-7B. Pretraining durations vary from one to ten days, depending on the model size and the number of training tokens.
Table 9: Model architecture for dense models and MoE models.
Dense L5IM Dense- Prense-550M, Pense-543k, Dense- 1.2B MoE- MB---BB-B
Model Dimension
768
1024
1536
1024
2048
2048
FFN Dimension
2048
4096
4096
4096
9192
1024
Attention heads
16
16
16
16
32
16
Key/Value Heads
4
4
4
8
16
Layers
12
12
16
29
16
16
Vocabulary Size
100278
100278
100278
100278
100278
50280
Weight Tying
True
True
True
True
True
False
Context Length
4096
4096
4096
4096
4096
4096
Expert Granularity
8 in 64
C.3 Hyperparameters for Pretraining
For the training hyperparameters, we primarily adopt the configuration outlined in OLMo (30 and OLMoE [15]. The training hyperparameters for our models across different sizes are presented
in Table TO The model trained using the AdamW optimizer with a learning rate (L.R) of 3e- 4 (4c-4 for MoE), which is scheduled decay following cosine function. The minimum LR
26

---

Table 10: Hyperparameters for Pretraining.
Dense Model
MoE-IB-7B
Optimizer
A
AdamW
Learning Rate (LR)
3e-4
4e-4
Minimum LR
3e-5
5e-5
LR Schedule
cosine
cosine
Weight Decay
0.1
0.1
B1
0.9
0.9
B2
0.95
0.95
Gradient Clipping
Batch Size
1024
1024
Warmup Tokens Init Distribution
1.388,608,000
10,485,760,000
Megatron
Megatron
Init std
1/v 1/v2.5d
1/v 2.5d
Init Truncation
3 std
3 std
RoPE
500000
10000
Activation
SwiGLU
SwiGLU
Load Balancing Loss Weight
0.01
Router z-loss Weight
0.001
is 3e-5 (5e-5 for MoE) prevent excessively small updates in the later stages of training. A weight decay of 0.1 is applied to regularize the model and prevent overfitting. The AdamW optimizer employs
0.9 and B2 0.95 control the first and second momentum estimates, respectively. Gradient clipping is utilized with a threshold of to mitigate the impact of large gradients during optimization. The model's training also incorporates a warmup phase with total of 8,388,608 668,00000000000000 tokens (10,485,766) 760,000 for MoE). The initialization of the model's 's parameters follows normal distribution with a standard deviation defined as 1/v2.5d, where d is the model dimension. Furthermore, the initialization truncated standard deviations to ensure a more stable starting point for training. The RoPE (Rotary Position Embedding) parameter @ is set to 500,000 (10000 for MoE), controlling the scale of position encodings. Finally, the activation function used the model is SwiGLU, which has been shown to outperform traditional activation functions in various tasks.
D Computational Overhead
The direct contribution RMSNorm to the overall parameter count and computational cost of large Transformer is, in fact, negligible. Below, we provide detailed analysis to support this claim.
For simplicity, we consider only the MHA and the SwiGLU activation function, excluding the Embedding and Output layers. Suppose the hidden dimension is d. the intermediate FFN size is d. the number of layers is T., and sequence length s. As shown in the table below, RMSNorm constitutes only negligible fraction of the actual runtime and memory consumption in Transformer under both Pre-Norm and HybridNorm configurations. And the ratio decreases as the model size increases. Moreover, when employing GQA, the relative overhead of RMSNorm in HybridNorm is further diminished.
Parameters Each RMSNorm layer introduces d parameters. In a Pre-Norm architecture, there are two RMSNorm layers per Transformer layer, resulting in total of 2dI. parameters. HybridNorm, there are four RMSNorm layers per Transformer layer, yielding 4d.. parameters in total. Each attention block has four weight matrices (for Q, K, and Ó projections), thereby contributing 4d parameters in total for the MHA component. Similarly, the FFN component introduces additional Sel L parameters.
The FLOPs for RMSNorm to process single token vector of dimension d is (4d++4. Since the constant term becomes negligible any reasonably large d, this can simplified to 4d. Accordingly, for Pre-Norm, the 2L RMSNorm operations incur a total cost of approximately SsdL,
Computation
27

---

Table 11: Parameter and computation comparison between Pre-Norm and HybridNorm architec- tures. denotes the number of layers, d the hidden dimension, and s the sequence length.
Main Transformer Components
Ratio (RMSNorm / Total
Type
Architecture
RMSNorm
(MHA & FFN)
2dL
1
Parameter
Pre-Norm
2dL
12d
12d2 L - 2dL
2
6d
4dL
HybridNorm
4dL
12d
~
12d I. - 4dT.
3d
8sdL
2
Computation
Pre-Norm
8sdL
(2Asd² +))
(FLOPs)
(24xd 4s
6d 8
16sd1.
HybridNorm
16sdL
(24ad² + d)
(24sd² +4s2d)L
6d +
whereas HybridNorm incurs 16sdL. In practice, however, the dominant computational overhead in
a Transformer stems from matrix-vector multiplications, primarily within MHA and FFN modules. Specifically, the FLOPs for MHA amount to (8sd² + 44s 4)), while the FLOPS for FFN total 16sd L.
E
Additional Experimental Results
E.1
Comparison with Other Methods
We compare the downstream evaluation results of 1.2B dense models trained on 200B tokens using five normalization strategies: Post-Norm, Pre-Norm. Mix-LN, HybridNorm, and HybridNorm". As shown in Table [2] HybridNorm' consistently delivers the highest average performance across eight downstream tasks, outperforming all other methods both on average and in the majority of individual cases.
In particular, HybridNorm' achieves the top scores HellaSwag (59.56), SeiQ (90.70), ARC-C (36.15), PIQA (73.83, and COPA (80.40), while remaining competitive on the remaining tasks. In contrast to Post-Norm and Pre-Norm, which show strong results on select benchmarks but suffer from variability elsewhere, HybridNorm" exhibits robust and consistently balanced performance. Notably, it attains average score of 60.67, surpassing the second-best method (Pre-Norm, 59.56) by over one point, underscoring its effectiveness in enhancing generalization across a diverse set of evaluation tasks.
We also observe that Post-Norm underperforms Pre-Norm in both Table 12 and Table 6 This can be attributed to the fact that, although Post-Norm generally offers higher performance upper bound, suffers from reduced training stability and requires extensive hyperparameter tuning 6 40]. In experiments, adopt the default hyperparameters from OLMo 2 without performing extensive hyperparameter search; hence, is unsurprising that Post-Norm does not outperform Pre-Norm under these settings. This observation motivates us combine the higher performance ceiling of Post-Norm with the superior training stability and hyperparameter robustness of Pre-Norm. Our experimental results show that HybridNorm achieves stronger robustness to hyperparameters, enabling balanced trade-off between stability and performance without the need for extensive tuning.
We also have extended Table to include DeepNorm [7]. Sandwich-LN 43 and OutputNorm (OLMo 2 (3)). as shown in Table 13 We have also clarified in the revised manuscript that OutputNorm refers to the normalization strategy employed in OLMo 2, where RMSNorm is applied to the outputs of attention and MLP sublayers.
It shows that HybridNorm consistently outperforms all other normalization strategies, including the aforementioned advanced methods, across all metries. This highlights the strength our approach in both generative perplexity tasks and HellaSwag.
28

---

Table 12: Downstream evaluation results of 1.2B dense models with Post-Norm, Pre-Norm, Mix-LN, HybridNorm, and HybridNorm' under 200B training tokens. OQA refers to OpenbookQA.
Methods
BasicArithmetic HellaSwag SciQ ARC-C
ARC-E PIQA OQA
COPA Avg.
Post-Norm
37.07
57.44 88.86
35.85
66.32
73.23
37.62
75.80
59.02
Pre-Norm
40.06
57.38 89.78
32.14
67.12
73.04
36.58
80.40 59.56
Mix-LN
33.23
56.91 89.20
33.78
69.30
72.25 37.00
79.00
58.83
HybridNorm
35.29
58.24 88.36
35.69
68.63
73.45
37.02
76.80
59.18
HybridNorm'
39.46
59.56 90.70
36.15
68.25
73.83
37.00
80.40
60.67
Table Comparison of normalization methods on 550M models.
Methods
Training Loss
C4 PPL.
Pile PPL.
HellaSwagt
Post-Norm
2.760
20.43
10.57
51.20
Pre-Norm
2.751
20.30
10.48
51.97
Mix-LN
2.760
20.43
10.56
51.29
DeepNorm
2.746
20.34
10.48
52.11
Sandwich-LN
2.751
20.45
10.58
52.07
OutputNorm (OLMo 2)
2.750
20.34
10.44
52.82
HybridNorm
2.737
20.00
10.29
53.35
HybridNorm"
2.731
19.85
10.25
53.36
E.2 Signal Propagation
Following 45
we plotted the evolution of the cosine similarity between tokens during pretraining. As shown in Figure 7 both Pre-Norm and Mix-LN exhibit notable increase in token similarity in certain layers, indicating a tendency toward representation degeneration. In contrast, HybridNorm maintains consistently lower similarity across most layers, suggesting better capabilities in informa- tion representation [45). This highlights the benefit of employing QK the attention module and Post-Norm in FFN to improve information flow.
E.3 Entropy Dynamics
we also examine the layerwise entropy evolution during pre-training. From Figure g HybridNorm maintains entropy within a relatively stable range, avoiding the sharp fluctuations observed in other methods. stable entropy distribution known to correlate with smoother training dynamics and improved optimization stability 449] providing further empirical support for the robustness of HybridNorm.
Following 49
E,4 Overall Results for Dense Models
Overall results for the dense model are presented Figure 9 depicting validation losses and downstream evaluations over IT training tokens. The comparison includes models with Pre-Norm, HybridNorm, and HybridNorm". One can see that both HybridNorm and HybridNorm" outperform Pre-Norm. with HybridNorm" achieving the lowest training and validation losses while delivering the best downstream performance on average.
E.5 Overall Results for MoE Models
Overall results for the MoE model are presented in Figure 10, illustrating validation losses and downstream evaluations over 500 billion training tokens. The comparison focuses on models employing Pre-Norm and HybridNorm*. From the figures, we can see that HybridNorm" achieves lower training loss and validation loss compared to Pre-Norm on all datasets, such as C4, Books, and Pile. Additionally, HybridNorm' outperforms Pre-Norm on most downstream tasks, though there

---

Layer
Layer 2
Layer
Layer
Cogine Similerito Ro
0.8
0-
0.8
0.0
Prab-orm
Pre Norm
2.4
Mi
Hetr choo
0.2
0.2
0.2
2000
4000 Step
6000
800D
2000
4060
6000
8000
2000
6000 Step
6000
BC00
2000
4030
6030
8020
Step
Step
Layer
Layer
Layer
Layer
Cosive Sim arty
Pa
Pest Norm
Paal-Norm
Fost Norm
Norm
Fre horm
Pre Nerm
horm
Nix UM
Mix
Mix Lt
Vix LH
2.6
Hyortsoomm
0.0
Hytro
Hyerssloom
0.0
Hytriclomm
0.4
0.4
0.0
0.0
0.0
0.0
Step
Layer
Layer 10
Layer 11
Layer 12
Cosine Similanty
4.0
Post Narm
1.0
Nom Frekoom
1.0
Norm
10
Fost Nom
0.8
Pre-korm
0.0
0.0
Prevlem
Fresbom
Mi-UN
Mi-UK
Mi-LM
6.6
Hyanssomm
Hytrdhhrrm
hyondtomm
Hytrohhamm
0.4
0-4
0.4
0.4
0.0
0.0
2000
4000 Step
0200
8000
4000 Step
8000
2000
6000
B000
2500
4000
0030
3000
Step
Step
Layer
Layer 14
Layer
Layer 16
Cosire Simillarty
2.6
0.2
0.8
0.0
VixUN
HyovisNorm
Hybridhorm
HybrissNom
Hytrichaom
0.7
0.2
0.2
0.7
2010
4000 Step
5000
8300
2000
4000
8000
5000
2000
4000
5000
8000
2000
4000
6000
5000
Step
Step
Figure 7: Evolution of the cosine similarity between tokens for Post-Norm, Pre-Norm, Mix-LN, and HybridNorm at different layers.
are some cases where it underperforms. average, however, HybridNorm demonstrates superior downstream performance.
Essential Differences Between Pre-Norm and Post-Norm
To facilitate a unified analysis of the various normalization variants, we propose categorization of Pre-Norm and Post-Norm based on their distinct approaches to residual connections (refer to Figure [T] particularly the sections highlighted by the dashed boxes). From this perspective, the FFN sublayer in HybridNorm can be considered as adopting a connection scheme similar to the Post-Norm.
30

---

Layer
Layer 2
Layer
Layer
Post Nort-Momm
Post Norm
Fost Norm
Fre-horm
Pre-Namm
Fre-homm
Entropy
Hybrddhermm
Myeridioo
3
Hy
Pre-ltrm NickN
Hyorlshomm
3000
Step
Step
Layer
Layer 6
Layer
Layer 8
Pout-Wemm
Pest/Mom
Entropy
2-
Stop
Step
Stop
Step
Layer
Layer 10
Layer
Layer 12
Entrpy
HyorldNom
Hybridoomm
HybridNom
2000
4000
5200
8D00
2000
4000
4000
8000
2000
6000
BC00
0
2003
4008
6020
8000
Step
Step
Shep
Step
Layer
Layer
Layer
Layer
Post
Post Norm Fre horm
Post Nomm
Norm MoxUN
borm
Mi-LNN
MoxUN
Mi-LN
Entropy
Hyorkshomm
Hybrddlarm
hyeridHom
Hybrichlrm
2000
4000
6000
0300
2000
2000
6000
8000
2000
6000
6000
8000
2000
4000 Step
6000
8000
Step
Step
Step
Figure 8: Evolution the layerwise entropy for Post-Norm, Pre-Norm, Mix-LN, and HybridNorm at different layers.
Norm
Subleyer
Sublayer
Sublayer
Norm
Norm
Sublayer
Post-Norm
Pro-Norm
Figure 11: unified view of Pre-Norm and Post-Norm
31

---

Training loss
Validation loss (C4)
Validation loss (Books)
Validation loss (CC)
2.0
Pybrilaam
Pro Nom
Nom
1,00
Iterrsham
damm
3.15
Hyondform
Morn
lybrisNomm
1.0
yyrcchom
Iyendhomm
3.10
Hyondsoam
2.7
2.30
2.80
(pe52o)
(Reddit)
(Stack)
(Wikkpedia)
1.55
2-37
Pelkom
lybrikam
3.30
Pre-oom
lnbrictiom
1.50
nyoo ohhemm
nodebm
Hyördhorm
Spmm
hybollomm
Hybrikiomm
1.45
Hyanttumn
222
1.40
2.20
3:15
2.15
1.0
200
400
2000
230
400
1000
100
1000
002
502
Validation loss (ICE)
Validation less (M202]
Validation loss (Pile)
Validation loss (Wikitext)
3.00
2.40
2.00
2.30
3.15
2.50
2.40
3.10
2.40
200
200
0.10
300
2010
3.05
250
250
400
300
1070
200
100
10
103
1010
200
105
605
500
1000
MMLU stem Acc. (%)
MMLU humanities Acc. (%)
MMLU social Acc. (%)
MMLU other Acc.
20
30
30
Pre-kom
60
Morm Hybrddhoom
NybridNarm
28
PybniSNamm
hylonchiiom
Hybrsdiorm
HellaSwag
Acc.
(%)
Pic Num
Rom
Pi fumm
Pre Mum myerdsoom Hyordshorm
Pybriciom Hybritiomm
Fybrccitom
86
ityondhom Itp moordhomm
Hybricticmm
430
230
1000
500
1000
502 500
ARC-Easy Acc. (5))
PIDA Acc. (%6
WinoGrande Acc. (%6
OpenbookOA Acc. (%)
60
56
34
hybelthomm
bybrisNum"
Hybeschimm
Hybritiiom
5d
56
Myardboom"
34
Hyandtamm
Hyarddtemm
200
630
620
800
1000
250
400
año
1000
260
460
cba
a6o
1000
200
402
60o
Róo
1000
BoalQ Ace. (%)
COPA Acc. (%)
Commonsense@L Acc. (%)
SocialiQA Acc. (%)
46
46
60
42
Nom PybrisNomm
40
45
Morm Hybrrdtoom
55
30
hyo-dhoomm
Hybrishoom"
430
350
1000
250
600
z0o
1010
500
505
of Tokens (B)
of Tokens
(B)
of Tokerns (6)
of Takenss 18)
Figure 9: Overall loss and downstream evaluations for the 1.2B dense models with IT training tokens.
G Formulas for Different Positions of Normalization Layers
In this section, we present the mathematical formulations for various normalization techniques. We begin by introducing the normalization layer within the attention mechanism.
32

---

Training loss
Validation loss (C4)
Validation loss (Books)
Validation loss (CC)
Pro Nom Pytriiaam"
Nom
9mm
2.00
Wern
2.40
Myl ioum
2.70
Hyo sPhumm
2.85
2.75
2.65
2.70
2.55
2.75
2.50
(pe52o)
(Reddit)
(Stack)
(Wiklpedia)
2.25
Pybrislam"
Ne-kom
3.00
Pr--tom
hytrcciimm
1.00
Pr--famm
idiammm
2.40
Hyandfamm
Narm
1.05
2.85
1.63
2.15
1.00
2.30
2.10
2.80
2:25
100
200
300
400
300
300
200
302
402
302
200
Validation loss (ICE)
Validation loss (M202)
Validation loss (Pile)
Validation loss (Wikitext)
3:25
2.35
2.70
2.60
2.50
11
2.00
2.05
2.35
2.30
100
200
300
400
500
100
205
305
405
503
180
200
300
400
505
2.25
1.00
250
350
400
500
MMLU stem Acc. (%)
MMLU humanities Acc. (%)
MMLU social Acc. (%)
MMLU other Acc.
52
50
30
25
PybnidNamm
Hybrisformm
Var (%)
HellaSwag
(16)
(%6
67.5
53
42.5
36
91
57.
Pre-iomm HybriSlamm
Pre-for
Pre-lamm ktyb IdWorm
Pe--Aorm Hybrldharm
60.0
Hybriciicm
100
200
300
400
300
300
202
302
402
302
ARC-Easy Acc. (56)
PIDA Acc. (%)
WinoGrande Acc. (56)
OpenbookQA Acc. (%)
77
70
76
Pre-icmm bybrisNamm
75
Hybrithiomm
59
Fre-Warm HybridWemm
100
200
300
400
500
100
200
300
605
500
56
130
200
300
600
100
1.0
200
300
600
500
BoolQ Acc. (%)
COPA Acc. (%)
Commonsense Acc. (%)
SocialiQA Acc. (%)
Norm
Hyarddoomm
65
85.0
48
82.5
80.0
PryonisMamm
77.
44
202
200
400
203
407
300
of Tokens (B)
of Takerss (B)
of Toleers (6)
of Takerss (8)
Figure 10: Overall loss and downstream evaluations for the MoE models with 500B training tokens.
Vanilla scaled dot-product attention are show in Eq. and attention with Q-V-Norm is defined in Eq. 5 Similarly, attention with QK-Norm is defined as
(orm(Q))orm V.
attnox(Q,K) K)) softmax
(43)
Vdk.
33

---

Attention with KV-Norm is defined as
QNorm(K)* Vdz
attngy K,V softmax
Norm(V).
(44)
As mentioned in Section 5.4 we extend traditional normalization approaches by considering not only the Query (Q), Key (K), and Value (V) components but also the Context (C), which refers to the output of the attention mechanism. And attention with QKVC-Norm is defined as
Norm (Q)Norm(())) Norm(V)). (45)
(I, = Norm softmax
Vdg
Attention with QKC-Norm is defined as attnqko(Q, K, V) Norm softmax
Norm Norm(Q)
v)
(46)
vdz
Attention with KC-Norm is defined as
@NNom K) v)
attuko(Q, V) = Norm softmax
(47)
vdk
Then we denote MHA with atiny as MHA # for # E (QKVC, QKV, QKV.QKC,KCCCOOOOOK KV, KC),
MHA(X), = Connca(heead,.... head,)
(48)
where head; = attn (Qi, K;, Vi) for 1, 2, h, 1o = (p.it(XW.) for e E (Q,K, V and Wo, WK, Wv, Wo E Rdxd are learnable parameters.
With the aforementioned definitions in hand, we present the mathematical formulations for the methods discussed the Ablation Study below (# E BQKVC,QKY QKCCCKKKKKKKKKKKKKKKKK KK,,Q, #-Post:
y = ( X
(49)
x = FFN(Norm(YY)) - Norm(Y').
(50)
#-Pre:
y' = MHA (X') + XX
(51)
x FFN((orm((Y))) y
(52)
Pre-:-PPsst:
r' = MHA (Norm(X') + X²,
(53)
x FFN(Norm(YY)) NYrm
(54)
Pre-//Prre:
Y' MHA (Norm(X')) X!,
(55)
x = FFN(Norm(YY)) + y
(56)
Pre-Post:
r = MHA((orm(X')) + +x',
(57)
x = FFN(Norm(YY)) - NYrm(Y')).
(58)
Post-Pre:
y' = MHA(Norm(X'))) + Norm(X)).
(59)
x /+1 FFN(Norm((Y)))) r!
(60)
34

---

H Broader Impacts and Limitations
H.1 Broader Impacts
This paper proposes HybridNorm, simple yet effective hybrid normalization technique that improves the training stability and performance of transformers. It has the potential to assist the LLM community in advancing transformer architectures and enhancing their overall effectiveness, While there may be societal implications of our work, none of which feel must be specifically highlighted here.
H.2 Limitations
First, due limited computational resources, our experiments are conducted on models ranging from 15IM to 7B parameters. While our method shows strong effectiveness on smaller models, its performance larger-scale models has not yet been empirically validated. Second, although our theoretical analysis demonstrates improved gradient stability, this does not directly guarantee better overall model performance.
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: We clearly discuss the contributions and scope of the paper in both the abstract and Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Appendix H
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
36

---

Justification: In Section and Appendix B we theoretically analyze the improved gradient stability of our method. All necessary notations and settings are clearly defined, and complete proofs are provided in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions of the experimental settings in Section 5.] and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
37

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Code is publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public/guides/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide detailed descriptions of the experimental settings in Section 5.] and Appendix C including information on baselines, datasets, model architectures, training hyperparameters, evaluation sets, and other implementation details necessary to understand and reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the high computational cost of pre- pre-training, did not repeat experi- ments to compute error bars. For evaluation, we follow standard practices in prior work by reporting results public benchmarks without including error bars statistical significance
tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
38

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the GPU types and provide detailed training information in Ap- pendix C.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpus//neurips.. http:///s...cpp..i...ttcccccaaaa
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses broader impacts of the work in Appendix H
Guidelines:
The answer NA means that there is no societal impact of the work performed.
39

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external assets used in the paper, including code, datasets, and models. are properly credited and used in accordance with their respective licenses. Appropriate citations and acknowledgments are provided in the manuscript.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
40

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines: The paper does not involve crowdsourcing nor research with human subjects.
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
41

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurips /att::///earrrpsssc//oooeereeee/ /c/////oeeeeeeccc//22////.. for what should or should not be described.
42


Reviews:
[{"review_id": "ek2MoLTzkL", "reviewer": "Reviewer_RJ3j", "summary": "This paper introduces HybridNorm, a normalization strategy for transformers that aims to combine the benefits of both Pre-Norm and Post-Norm approaches to address challenges in training deep transformer networks. HybridNorm specifically applies QKV normlization within the attention mechanism and Post-Norm in FFN of each block. The authors provide theoretical analysis and empirical evidence to demonstrate that HybridNorm improves gradient flow and model robustness.", "questions": "1. Could the authors elaborate on the specific differences in how RMSNorm and LayerNorm  might influence the theoretical gradient analysis and the empirical performance of HybridNorm? \n2. Please empirically validate HybridNorm on 7B or larger models \n3. Fig. 6 in [1] uses QKV-Norm to stably train a Transformer. It seems that you only changed pre-norm to post-norm. Both of the pre- and post-norm have advantages, they have their own shortcomings, too. You used post-norm, which introduces the Gradient Vanish problem mentioned in [2]", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n1. HybridNorm outperforms both Pre-Norm and Post-Norm across dense and Mixture of Experts models onmultiple benchmarks, demonstrating improved training stability and better downstream performance. \n2. This paper provides theoretical insights and gradient analysis to explain why HybridNorm improves gradient flow stability and model robustness.\n \nWeakness:\n1. The paper mentions that their theoretical analysis withRMSNorm. While RMSNorm and LayerNorm are common, the effectiveness of HybridNorm might be sensitive to the specific normalization function chosen, and the theoretical guarantees might not directly translate if other normalization techniques were considered\n\n2. The related work section mentions DeepNorm, Sandwich-LN, and OLMo-2 as normalization strategies. While the paper generally compares against Pre-Norm and Post-Norm, a more direct and in-depth empirical comparison with these advanced normalization methods could further solidify the superior performance of HybridNorm.\n\n3. This paper provides limited technical contribution. The method is very similar to Figure 6 in [1], merely replacing pre-norm with post-norm.", "was_revised": true, "final_justification": "Adjust accordingly after the rebuttal phase.", "rebuttal": "Thanks for your time and constructive comments. We appreciate that Reviewer RJ3j finds our method effective and consistent with our theoretical analysis. We address your concerns below and hope our responses adequately clarify your questions. \n\n---\n\nQ1.  Could the authors elaborate on the specific differences in how RMSNorm and LayerNorm might influence the theoretical gradient analysis and the empirical performance of HybridNorm?\n\nA1. In the following, we demonstrate that RMSNorm and LayerNorm exhibit no fundamental differences, both in theoretical analysis and empirical observations. It is worth noting that, given the vast majority of popular LLMs are based on RMSNorm, our experiments and conclusions are broadly applicable to standard LLM architectures.\n\n**On the theoretical side**, we adopt the notation introduced in Appendix B of the paper. Suppose the input is given by $X \\in \\mathbb{R}^{s \\times d}$. Let $P= I - \\frac{1}{d}1_d1_d^{\\top}$ .  Then $\\mathbb{E}X = X \\frac{1}{d}1_d1_d^{\\top}$ and  $X-\\mathbb{E}X =X - X \\frac{1}{d}1_d1_d^{\\top}=XP$. For simplicity, we omit the affine transformation in the normalization layers. It follows that \n\n$$\nLayerNorm(X)=\\frac{X-\\mathbb{E}X}{Var(X)}=\\frac{X-\\mathbb{E}X}{RMS(X-\\mathbb{E}X)}=\\frac{XP}{RMS(XP)}=RMSNorm(XP).\n$$\n\nHence, analogous to the gradient of RMSNorm (Eq. (20) in the paper), the gradient of LayerNorm is given by\n\n$$\n\\frac{\\partial LayerNorm(X)}{\\partial X} = \\frac{\\partial RMSNorm(XP)}{\\partial X} = \\mathrm{blockdiag}\\left(\\frac{\\partial RMSNorm((XP)_i)}{\\partial X_i}\\right)=\\mathrm{blockdiag}\\left( \\frac{\\sqrt{d}}{||(XP)_i||_2}\\left(I_d - \\frac{(XP)_i(XP)_i^\\top}{||(XP)_i||_2^2} \\right)P\\right).\n$$\n\nHere, $X_i$ means that the i-th column of $X$. Note that $P$ is a positive semidefinite matrix with eigenvalues bounded between 0 and 1, hence $||P||_2 = 1$. As a result, the gradient norms of LayerNorm and RMSNorm differ only by a constant factor. **This implies that the main result, Theorem 2 in the paper, also holds for LayerNorm.**\n\n**On the experimental side,** we conducted  controlled comparison using models of identical size (550M parameters), each trained on 400B tokens. Both models adopt the Pre-Norm architecture and employ the Megatron initialization scheme; the only difference lies in the normalization method—**one uses RMSNorm, while the other uses LayerNorm**. Detailed model configurations and experimental hyperparameters are provided in Table 6 and Table 7 of the paper. As shown in the table below, **the training losses of RMSNorm and LayerNorm are nearly identical**, with a marginal difference of only 0.0008. In the context of large-scale models, a loss difference smaller than 0.001 is typically considered negligible. \n\n|  | Training Loss |\n| --- | --- |\n| RMSNorm | 2.7631 |\n| LayerNorm | 2.7639 |\n\n---\n\nQ2. A more direct and in-depth empirical comparison with these advanced normalization methods could further solidify the superior performance of HybridNorm.\n\nA2. We have extended Table 4 to include **DeepNorm**, **Sandwich-LN**, and **OutputNorm (OLMo-2** [2]**)**, as shown below. We have also clarified in the revised manuscript that **OutputNorm** refers to the normalization strategy employed in **OLMo-2**, where RMSNorm is applied to the outputs of attention and MLP sublayers.\n\nIt shows that **HybridNorm consistently outperforms all other normalization strategies**, including the aforementioned advanced methods, across all metrics. This highlights the strength of our approach in both generative perplexity tasks and HellaSwag.\n\n| **Method** | **Loss ↓** | **C4 PPL ↓** | **Pile PPL ↓** | **HellaSwag ↑** |\n| --- | --- | --- | --- | --- |\n| Post-Norm | 2.760 | 20.43 | 10.57 | 51.20 |\n| Pre-Norm | 2.751 | 20.30 | 10.48 | 51.97 |\n| Mix-LN | 2.760 | 20.43 | 10.56 | 51.29 |\n| DeepNorm | 2.746 | 20.34 | 10.48 | 52.11 |\n| Sandwich-LN | 2.751 | 20.45 | 10.58 | 52.07 |\n| OutputNorm (OLMo-2) | 2.750 | 20.34 | 10.44 | 52.82 |\n| **HybridNorm** | **2.737** | **20.00** | **10.29** | **53.35** |\n| **HybridNorm*** | **2.731** | **19.85** | **10.25** | **53.36** |\n\n---\n\nQ3. This paper provides limited technical contribution. The method is very similar to Figure 6 in [1], merely replacing pre-norm with post-norm.. Both of the pre- and post-norm have advantages, they have their own shortcomings, too. You used post-norm, which introduces the Gradient Vanish problem.\n\nA3. First and foremost, we would like to emphasize that our method, HybridNorm, is a **simple yet effective** approach to enhancing both model performance and training stability. The core idea is to integrate Pre-Norm and Post-Norm within each Transformer layer, to balance training stability and model performance. Moreover, HybridNorm*  introduces a specialized treatment for a specific layer.\n\nThe primary difference between our method and the approach shown in Fig. 6 of [1] (which we refer to as QKV-Pre) lies in the integration of different normalization types within each transformer block. The layer-wise gradient norm of QKV-Pre, similar to Pre-Norm illustrated in Figure 2, tends to exhibit gradient explosion in deeper models, while Post-Norm suffers from vanishing gradients. **In contrast, HybridNorm adopts a hybrid strategy within each transformer block, leading to more stable gradient propagation during backpropagation and effectively mitigating both gradient explosion and vanishing.** Moreover, our method achieves the best performance. We also provides theoretical insights for QKV-Norm.\n\nBuilding on the foundation of the scaling laws, even seemingly minor modifications—such as changes to model architecture, hyperparameter configurations, initialization schemes, or training protocols—can lead to substantial shifts in performance, particularly in the context of large-scale LLMs. Techniques like GQA and MQA exemplify this phenomenon. Although our proposed changes may appear modest, **they induce significant alterations in training dynamics**, especially under a large training budget. Furthermore, we observe a clear improvement in convergence speedup. Extensive empirical studies and ablation analyses further validate the simplicity and effectiveness of our approach.\n\nTherefore, our contribution goes beyond a simple combination of existing components and offers a non-trivial advancement over prior approaches.\n\n---\n\nQ4.  Please empirically validate HybridNorm on 7B or larger models.\n\nA4. In fact, as described in Section 5.2 of the paper, we have already trained a 7B-parameter MoE model with 1B active parameters using 500B training tokens, and demonstrated that HybridNorm* achieves superior performance. Additionally, we have included experiments with a 7B dense model for further comparison.\n\n**New experiment in 7B dense model.** The experimental setup primarily follows the 7B model configuration outlined in [2] and the number of training tokens is 150B. The first table below presents the training loss and validation perplexity (PPL), while the second table reports the downstream evaluation metrics.\n\n|  | Training Loss **↓** | C4 PPL **↓** | Books PPL **↓** | Common Crawl PPL **↓** | peS2o PPL **↓** | Reddit PPL **↓** | Stack PPL **↓** | Wiki PPL **↓** | Pile PPL **↓** | Wikitext 103 PPL **↓** |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Pre-Norm | 2.469 | 15.32 | 13.37 | 17.10 | 8.07 | 20.31 | 3.57 | 9.96 | 7.85 | 10.09 |\n| HybridNorm* | **2.430** | **14.83** | **12.77** | **16.77** | **7.67** | **19.65** | **3.40** | **9.34** | **7.81** | **9.16** |\n\n|  | BasicArithmetic **↑** | HellaSwag **↑** | SciQ **↑** | ARC-C **↑** | ARC-E **↑** | PIQA **↑** | Openbook QA **↑** | COPA **↑** | WinoGrande **↑** | BoolQ **↑** | Avg. **↑** |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Pre-Norm | 43.50 | 69.03 | 46.57 | 41.47 | 74.95 | 76.71 | 39.40 | 84.00 | 63.00 | 67.43 | 60.61 |\n| HybridNorm* | **50.67** | **70.77** | **47.44** | **43.82** | **75.82** | **78.93** | **43.77** | **86.01** | **63.32** | **70.06** | **63.06** |\n\nThe experimental results demonstrate the clear superiority of **HybridNorm*** over the traditional **Pre-Norm** approach in the 7B model. Firstly, HybridNorm* achieves a lower training loss (2.430 vs. 2.469), indicating more efficient optimization during training. This improvement in training dynamics translates into consistently better performance across a range of language modeling benchmarks. Specifically, HybridNorm* yields lower perplexity scores on all evaluated datasets, including C4, Books, Common Crawl, Wiki, and Wikitext 103. For instance, perplexity on the C4 dataset drops from 15.32 to 14.83, suggesting stronger generalization across both structured and unstructured corpora.\n\nMoreover, HybridNorm* shows consistent improvements on all downstream tasks, covering various domains such as arithmetic reasoning, commonsense QA, and natural language inference. Notably, performance on BasicArithmetic improves significantly from 43.50% to 50.67%. Across the full suite of tasks—including ARC, PIQA, COPA, BoolQ, and WinoGrande—HybridNorm* outperforms Pre-Norm in every case, leading to an overall increase in average accuracy from 60.61% to 63.06%.\n\n---\n\nHope our explanations address your concerns. Please feel free to let us know if further clarification is needed.\n\n[1] Rybakov, et at. Methods of improving LLM training stability.  arXiv preprint arXiv: 2410.16682\n\n[2] OLMo Team, et al. 2OLMo 2 Furious. arXiv preprint arXiv:2501.00656"}, {"review_id": "Hp3iQru60C", "reviewer": "Reviewer_gyxV", "summary": "This paper proposes the HybridNorm method by combining the advantages of Pre-Norm and Post-Norm, it innovatively designs the normalization strategies in Transformer training, especially normalizing QKV in attention mechanism, and adopting Post-Norm in FFN, which in theory helps improve training stability and model performance. The author not only gave theoretical analysis, but also proved the effectiveness of HybridNorm through a large number of experiments, showing the application potential of this method in actual large-scale model training.", "questions": "1.From the overall perspective of the paper, the paper mainly focuses on performance improvement and training stability, but there is less interpretability analysis of the internal mechanism of HybridNorm. Why not you use some feature maps of different layers and experiments with heat maps to further verify the effectiveness of the method?\n2.The method proposed in this paper is carried out from the model with 151M to 7B parameters, but the model with a larger parameter quantity has not been fully experimentally verified. Why?", "limitations": "See the weaknesses.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe author conducted research in the normalization process of Transformer and proposed a training method that can efficiently transformer architecture. It systematically compared with existing mainstream normalization methods such as Pre-Norm, Post-Norm, QK-Norm, etc. The experimental part covers a variety of model structures and data sets, has a relatively comprehensive index, and is meticulous in the ablation experiment, which can fully verify the effectiveness of the method and the role of each component. It also displays a complete theoretical derivation process in the supplementary material, proving the effectiveness of the proposed method.\n\nWeaknesses:\n1.From the overall perspective of the article, the paper mainly focuses on performance improvement and training stability, but there is less interpretability analysis of the internal mechanism of HybridNorm, which can supplement some feature maps of different layers and experiments with heat maps to further verify the effectiveness of the method.\n2.The method proposed in this paper is carried out from the model with 151M to 7B parameters, but the model with a larger parameter quantity has not been fully experimentally verified. I hope that additional experimental verification can be carried out here in the future.", "was_revised": true, "final_justification": "I would like to commend the authors for their thorough and thoughtful response to my initial review. The rebuttal satisfactorily addresses the points I raised. In particular, the detailed explanations and additional experiments/analyses provided in the appendix have effectively alleviated my concerns.\n\nOverall, this paper presents a new normalization method with significant general applicability. The paper is well-structured. The high quality of both the research and the writing is evident.\n\nGiven the authors' successful addressing of my comments and the inherent strengths of the paper, I am now convinced of its merit and strongly recommend it for acceptance.", "rebuttal": "Thank you for your thoughtful feedback and constructive comments. We appreciate that Reviewer gyxV finds our method novel, our experiments well-executed, and our theoretical derivation complete. Below, we address your concerns point by point.\n\n---\n\nQ1. From the overall perspective of the paper, the paper mainly focuses on performance improvement and training stability, but there is less interpretability analysis of the internal mechanism of HybridNorm. Why not you use some feature maps of different layers and experiments with heat maps to further verify the effectiveness of the method?\n\nA1. Thank you very much for your insightful feedback. We fully agree that a more in-depth interpretability analysis could significantly enhance the understanding of HybridNorm’s internal mechanisms. **In fact, we have taken initial steps in this direction in Appendix D.2 and D.3 of our paper, where we examine information flow through the perspectives of signal propagation and entropy dynamics during pre-training.** The visualizations in Figure 7 and Figure 8 illustrate how HybridNorm influences layerwise entropy evolution and signal behavior across the network. We greatly appreciate your suggestion to incorporate feature map and heatmap-based analyses. We see this as a promising direction and plan to further expand our interpretability efforts along these lines in future work.\n\n---\n\nQ2. The method proposed in this paper is carried out from the model with 151M to 7B parameters, but the model with a larger parameter quantity has not been fully experimentally verified. Why?\n\nA2. In fact, as described in Section 5.2 of the paper, we have already trained a 7B-parameter MoE model with 1B active parameters using 500B training tokens, and demonstrated that HybridNorm* achieves superior performance. Additionally, we have included experiments with a 7B dense model for further comparison below.\n\n**New experiment in 7B dense model.** The experimental setup primarily follows the 7B model configuration outlined in [2] and the number of training tokens is 150B. The first table below presents the training loss and validation perplexity (PPL), while the second table reports the downstream evaluation metrics.\n\n|  | Training Loss **↓** | C4 PPL **↓** | Books PPL **↓** | Common Crawl PPL **↓** | peS2o PPL **↓** | Reddit PPL **↓** | Stack PPL **↓** | Wiki PPL **↓** | Pile PPL **↓** | Wikitext 103 PPL **↓** |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Pre-Norm | 2.469 | 15.32 | 13.37 | 17.10 | 8.07 | 20.31 | 3.57 | 9.96 | 7.85 | 10.09 |\n| HybridNorm* | **2.430** | **14.83** | **12.77** | **16.77** | **7.67** | **19.65** | **3.40** | **9.34** | **7.81** | **9.16** |\n\n|  | BasicArithmetic **↑** | HellaSwag **↑** | SciQ **↑** | ARC-C **↑** | ARC-E **↑** | PIQA **↑** | Openbook QA **↑** | COPA **↑** | WinoGrande **↑** | BoolQ **↑** | Avg. **↑** |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Pre-Norm | 43.50 | 69.03 | 46.57 | 41.47 | 74.95 | 76.71 | 39.40 | 84.00 | 63.00 | 67.43 | 60.61 |\n| HybridNorm* | **50.67** | **70.77** | **47.44** | **43.82** | **75.82** | **78.93** | **43.77** | **86.01** | **63.32** | **70.06** | **63.06** |\n\nThe experimental results demonstrate the clear superiority of **HybridNorm*** over the traditional **Pre-Norm** approach in the 7B model. Firstly, HybridNorm* achieves a lower training loss (2.430 vs. 2.469), indicating more efficient optimization during training. This improvement in training dynamics translates into consistently better performance across a range of language modeling benchmarks. Specifically, HybridNorm* yields lower perplexity scores on all evaluated datasets, including C4, Books, Common Crawl, Wiki, and Wikitext 103. For instance, perplexity on the C4 dataset drops from 15.32 to 14.83, suggesting stronger generalization across both structured and unstructured corpora.\n\nMoreover, HybridNorm* shows consistent improvements on all downstream tasks, covering various domains such as arithmetic reasoning, commonsense QA, and natural language inference. Notably, performance on BasicArithmetic improves significantly from 43.50% to 50.67%. Across the full suite of tasks—including ARC, PIQA, COPA, BoolQ, and WinoGrande—HybridNorm* outperforms Pre-Norm, leading to an overall increase in average accuracy from 60.61% to 63.06%.\n\n---\n\nThank you for pointing out our type. We have corrected it accordingly. Please feel free to let us know if further clarification is needed."}, {"review_id": "SYmqGuHZAl", "reviewer": "Reviewer_Cq1L", "summary": "This paper introduces a novel normalization technique that integrates post-normalization with QKV normalization to enhance the stability of transformer training. The authors demonstrate consistent performance improvements on dense LLMs (ranging from 150M to 1B parameters) and MoE models. Furthermore, the paper provides a theoretical analysis comparing the proposed solution with existing PreNorm and PostNorm approaches.", "questions": "Questions:\n* No questions regarding the technical aspects, as they are straightforward.\n* Have the authors explored scaling the model beyond 7B parameters?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* The development of improved normalization techniques is crucial for scaling base models, addressing a significant concern regarding training stability.\n* The proposed solution is not only empirically validated but also theoretically supported.\n* The experimental setup is extensive, although an evaluation on larger model scales (e.g., > 7B) would be beneficial.\n\nWeaknesses:\n* The method, primarily an exploration of normalization placement, has limited novelty.\n* Evaluating the solution solely on models under 1B parameters may not be sufficient to assess its scalability and effectiveness in real-world LLM training scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading and insightful comments. We appreciate that Reviewer Cq1L recognizes the importance of improved normalization techniques for stable transformer training and acknowledges the empirical strength and theoretical soundness of our proposed method. We address your concerns below.\n\n---\n\nQ1. The method, primarily an exploration of normalization placement, has limited novelty.\n\nA1. First and foremost, we would like to emphasize that our method, HybridNorm, is a **simple yet effective** approach to enhancing both model performance and training stability. The core idea is to integrate Pre-Norm and Post-Norm within each Transformer layer, to balance training stability and model performance. Moreover, HybridNorm*  introduces a specialized treatment for a specific layer.HybridNorm adopts a hybrid strategy within each transformer block, leading to more stable gradient propagation during backpropagation and effectively mitigating both gradient explosion and vanishing. Moreover, our method also achieves the best performance among all evaluated approaches.\n\nBuilding on the foundation of the scaling laws, even seemingly minor modifications—such as changes to model architecture, hyperparameter configurations, initialization schemes, or training protocols—can lead to substantial shifts in performance, particularly in the context of large-scale LLMs. Techniques like GQA and MQA exemplify this phenomenon. Although our proposed changes may appear modest, **they induce significant alterations in training dynamics**, especially under a large training budget. Furthermore, we observe a clear improvement in convergence speedup. Extensive empirical studies and ablation analyses further validate the simplicity and effectiveness of our approach.\n\nTherefore, our contribution goes beyond a simple combination of existing components and offers a non-trivial advancement over prior approaches.\n\n---\n\nQ2. Have the authors explored scaling the model beyond 7B parameters?\n\nA2. The models we have trained so far scale up to a maximum size of 7B parameters. In fact, as described in Section 5.2 of the paper, we have already trained a 7B-parameter MoE model with 1B active parameters using 500B training tokens, and demonstrated that HybridNorm* achieves superior performance. Additionally, we have included experiments with a 7B dense model for further comparison.\n\n**New experiment in 7B dense model.** The experimental setup primarily follows the 7B model configuration outlined in [2] and the number of training tokens is 150B. The first table below presents the training loss and validation perplexity (PPL), while the second table reports the downstream evaluation metrics.\n\n|  | Training Loss **↓** | C4 PPL **↓** | Books PPL **↓** | Common Crawl PPL **↓** | peS2o PPL **↓** | Reddit PPL **↓** | Stack PPL **↓** | Wiki PPL **↓** | Pile PPL **↓** | Wikitext 103 PPL **↓** |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Pre-Norm | 2.469 | 15.32 | 13.37 | 17.10 | 8.07 | 20.31 | 3.57 | 9.96 | 7.85 | 10.09 |\n| HybridNorm* | **2.430** | **14.83** | **12.77** | **16.77** | **7.67** | **19.65** | **3.40** | **9.34** | **7.81** | **9.16** |\n\n|  | BasicArithmetic **↑** | HellaSwag **↑** | SciQ **↑** | ARC-C **↑** | ARC-E **↑** | PIQA **↑** | Openbook QA **↑** | COPA **↑** | WinoGrande **↑** | BoolQ **↑** | Avg. **↑** |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Pre-Norm | 43.50 | 69.03 | 46.57 | 41.47 | 74.95 | 76.71 | 39.40 | 84.00 | 63.00 | 67.43 | 60.61 |\n| HybridNorm* | **50.67** | **70.77** | **47.44** | **43.82** | **75.82** | **78.93** | **43.77** | **86.01** | **63.32** | **70.06** | **63.06** |\n\nThe experimental results demonstrate the clear superiority of **HybridNorm*** over the traditional **Pre-Norm** approach in the 7B model. Firstly, HybridNorm* achieves a lower training loss (2.430 vs. 2.469), indicating more efficient optimization during training. This improvement in training dynamics translates into consistently better performance across a range of language modeling benchmarks. Specifically, HybridNorm* yields lower perplexity scores on all evaluated datasets, including C4, Books, Common Crawl, Wiki, and Wikitext 103. For instance, perplexity on the C4 dataset drops from 15.32 to 14.83, suggesting stronger generalization across both structured and unstructured corpora.\n\nMoreover, HybridNorm* shows consistent improvements on all downstream tasks, covering various domains such as arithmetic reasoning, commonsense QA, and natural language inference. Notably, performance on BasicArithmetic improves significantly from 43.50% to 50.67%. Across the full suite of tasks—including ARC, PIQA, COPA, BoolQ, and WinoGrande—HybridNorm* outperforms Pre-Norm in every case, leading to an overall increase in average accuracy from 60.61% to 63.06%.\n\n---\n\nPlease feel free to let us know if further clarification is needed."}, {"review_id": "oxwUMQ4SWX", "reviewer": "Reviewer_zMv1", "summary": "HybridNorm applies QKV-Norm in the attention mechanism and Post-Norm in the feed-forward network. The paper provides theoretical analysis of gradient flow benefits and validates the approach through experiments on both dense and MoE models.", "questions": "- The authors claim that HybridNorm improves the stability and performance of deep transformers. However, the evaluation is limited to models with up to 16 layers, which are relatively shallow compared to mainstream LLMs—such as the 28-layer 1.5B DeepSeek R1. Can the authors provide evidence that the observed stability gains generalize to deeper architectures commonly used in practice?\n- What is the computational overhead introduced by HybridNorm in terms of latency and memory consumption? How does this overhead scale with model size?\n- Table 3 shows Pre-Norm and HybridNorm have different sensitivities to initialization methods. What drives this divergence, and can the author provide an intuitive explanation?", "limitations": "While HybridNorm demonstrates improvements over Pre-Norm and Post-Norm under controlled experimental settings, the lack of direct comparisons with widely recognized foundation models (e.g., LLaMA 3.2, to which the authors’ tested architecture is closely related) limits the reader’s ability to contextualize the magnitude of these gains.\n  \nFor example, on the ARC-C benchmark, the 1.2B dense model with HybridNorm achieves a score of 37.06, which is lower than the Pre-Norm baseline at 39.2. Although the average score across benchmarks increases modestly from 62.99 (Pre-Norm) to 64.15 (HybridNorm∗), the relative improvements remain small. In contrast, mainstream foundation models such as Qwen2-1.5B and its enhanced variant Qwen2.5-1.5B achieve 43.7 and 54.7 on ARC-C, respectively—with the latter improvement largely attributed to higher-quality training data rather than architectural changes.\n\nCompared to such data-driven advancements, the performance gains offered by HybridNorm appear marginal. Without contextual baselines and further explanation, it remains difficult to assess whether the improvements are meaningful or competitive relative to more influential factors such as data quality or model scale.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- Theoretical analysis and empirical results offer clear insights into gradient stabilization.  \n- The paper lacks measurements of training and inference throughput, as well as memory overhead, associated with the added normalization steps. This makes it difficult to assess the practical efficiency of the proposed method.\n- Experimental comparison limited to Pre-Norm, Post-Norm, and Mix-LN, advanced schemes like DeepNorm and Sandwich-LN are mentioned but not evaluated.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns well.", "rebuttal": "Thank you for your thoughtful feedback. We appreciate that Reviewer zMv1 values our theoretical analysis. The concerns raised are mostly due to misunderstandings. We have addressed all concerns below.\n\n---\n\nQ1. What is the computational overhead introduced by HybridNorm? How does this overhead scale with model size?\n\nA1. **The direct contribution of RMSNorm to the overall parameter count and computational cost of a large Transformer is, in fact, negligible**. Below, we provide a detailed analysis to support this claim.\n\nFor simplicity, we consider only the MHA and the SwiGLU activation function, excluding the Embedding and Output layers. Suppose the hidden dimension is $d$, the intermediate FFN size is $\\frac{8}{3}d$, the number of layers is $L$, and sequence length is $s$. As shown in the table below, RMSNorm constitutes only a negligible fraction of the actual runtime and memory consumption in Transformer under both Pre-Norm and HybridNorm configurations. And the ratio decreases as the model size increases. Moreover, when employing GQA, the relative overhead of RMSNorm in HybridNorm is further diminished.\n\n|  | Architecture | RMSNorm  | Main Transformer Components (MHA & FFN) | Ratio (RMSNorm / Total) | Ratio (for d=s=4096) |\n| --- | --- | --- | --- | --- | --- |\n| Parameter | Pre-Norm | $2dL$ | $12d^2L$  | $\\frac{2dL}{12d^2L + 2dL} \\approx \\frac{1}{6d}$ | ~0.0041% |\n|  | HybridNorm | $4dL$ | $12d^2L$   | $\\frac{4dL}{12d^2L + 4dL} \\approx \\frac{1}{3d}$ | ~0.0081% |\n| Computation (FLOPs) | Pre-Norm | $8sdL$ | $(24sd^2 + 4s^2d)L$ | $\\frac{8sdL}{(24sd^2 + 4s^2d)L} = \\frac{2}{6d + s}$ | ~0.0068% |\n|  | HybridNorm | $16sdL$ | $(24sd^2 + 4s^2d)L$ | $\\frac{16sdL}{(24sd^2 + 4s^2d)L} = \\frac{4}{6d + s}$ | ~0.0137% |\n\n**Parameters.** Each RMSNorm layer introduces $d$ parameters. In a Pre-Norm architecture, there are two RMSNorm layers per Transformer layer, resulting in a total of $2dL$ parameters. In HybridNorm, there are four RMSNorm layers per Transformer layer, yielding $4dL$ parameters in total. Each attention block has four weight matrices (for Q, K, V, and O projections), thereby contributing $4d^2L$ parameters in total for the MHA component. Similarly, the FFN component introduces an additional $8d^2L$ parameters.\n\n**Computation.**  The FLOPs for RMSNorm to process a single token vector of dimension $d$ is $(4d + 4)$. Since the constant term becomes negligible for any reasonably large $d$, this can be simplified to $4d$. Accordingly, for Pre-Norm, the $2L$ RMSNorm operations incur a total cost of approximately $8sdL$, whereas HybridNorm incurs $16sdL$. In practice, however, the dominant computational overhead in a Transformer stems from matrix-vector multiplications, primarily within MHA and FFN modules. Specifically, the FLOPs for MHA amount to $(8sd^2 + 4s^2d)L$, while the FLOPs for FFN total $16sd^2L$.\n\n---\n\nQ2. Advanced schemes like DeepNorm and Sandwich-LN are not evaluated.\n\nA2. We have extended Table 4 to include DeepNorm, Sandwich-LN, and OutputNorm (OLMo-2), as shown below. \n\nWe observe that **HybridNorm consistently outperforms all other normalization strategies**, including the aforementioned advanced methods, across all metrics. This highlights the strength of our approach in both generative perplexity tasks and HellaSwag.\n\n| **Method** | **Loss ↓** | **C4 PPL ↓** | **Pile PPL ↓** | **HellaSwag ↑** |\n| --- | --- | --- | --- | --- |\n| Post-Norm | 2.760 | 20.43 | 10.57 | 51.20 |\n| Pre-Norm | 2.751 | 20.30 | 10.48 | 51.97 |\n| Mix-LN | 2.760 | 20.43 | 10.56 | 51.29 |\n| DeepNorm | 2.746 | 20.34 | 10.48 | 52.11 |\n| Sandwich-LN | 2.751 | 20.45 | 10.58 | 52.07 |\n| OutputNorm (OLMo-2) | 2.750 | 20.34 | 10.44 | 52.82 |\n| **HybridNorm** | **2.737** | **20.00** | **10.29** | **53.35** |\n| **HybridNorm*** | **2.731** | **19.85** | **10.25** | **53.36** |\n\n---\n\nQ3.  Provide evidence that the observed stability gains generalize to deeper architectures.\n\nA3. In fact, we have already conducted experiments on deeper models in **Section 5.5 (Deeper Models)**, where we evaluate HybridNorm on Transformers with depths ranging from 16 to **29 layers** under a comparable parameter budget. As shown in **Table 5**, both HybridNorm and HybridNorm* consistently outperform Pre-Norm and Post-Norm across all depths. Notably, **Post-Norm fails to converge at 29 layers**, while our method remains stable and achieves the best performance—demonstrating its superior scalability and training stability in deeper configurations.\n\nTo further validate the generalization of our method to large-scale, practically relevant models, we now provide **additional results on a 7B dense model with 32 layers**, closely following the configuration in [2].  It is important to note that the depth of our model is aligned with that of LLaMA3 and is comparable to the depth of other mainstream open-source models. Therefore, our model should not be considered shallow by any means. The results are summarized below.\n\nTraining and Validation Performance:\n\n|  | Training Loss **↓** | C4 PPL **↓** | Books PPL **↓** | Common Crawl PPL **↓** | peS2o PPL **↓** | Reddit PPL **↓** | Stack PPL **↓** | Wiki PPL **↓** | Pile PPL **↓** | Wikitext 103 PPL **↓** |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Pre-Norm | 2.469 | 15.32 | 13.37 | 17.10 | 8.07 | 20.31 | 3.57 | 9.96 | 7.85 | 10.09 |\n| HybridNorm* | **2.430** | **14.83** | **12.77** | **16.77** | **7.67** | **19.65** | **3.40** | **9.34** | **7.81** | **9.16** |\n\nDownstream Task Performance:\n\n|  | BasicArithmetic **↑** | HellaSwag **↑** | SciQ **↑** | ARC-C **↑** | ARC-E **↑** | PIQA **↑** | Openbook QA **↑** | COPA **↑** | WinoGrande **↑** | BoolQ **↑** | Avg. **↑** |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Pre-Norm | 43.50 | 69.03 | 46.57 | 41.47 | 74.95 | 76.71 | 39.40 | 84.00 | 63.00 | 67.43 | 60.61 |\n| HybridNorm* | **50.67** | **70.77** | **47.44** | **43.82** | **75.82** | **78.93** | **43.77** | **86.01** | **63.32** | **70.06** | **63.06** |\n\n---\n\nQ4. Table 3 shows Pre-Norm and HybridNorm have different sensitivities to initialization methods. What drives this divergence, and can the author provide an intuitive explanation?\n\nA4. For this issue, we attempt to provide a preliminary explanation here. In the **Pre-Norm** setup, the hidden state update follows the form $h_{t+1} = h_t + f(h_t)$. As the network depth increases, the norm of $h_t$ tends to grow, which leads to the residual branch $h_t$ increasingly dominating over the transformed signal $f(h_t)$.  In contrast, **HybridNorm** applies Post-Norm specifically to FFN, which helps to keep the norm of $h_t$ relatively stable across layers. As a result, the balance between $h_t$ and $f(h_t)$ remains more consistent during training. When using Megatron initialization, which scales down $f(h_t)$ by a factor of $1/\\sqrt{2L}$, the relative contribution of $f(h_t)$ diminishes. This shifts the update dynamics closer to that of Pre-Norm, where $h_t$ dominates. As a result, HybridNorm performs better when combined with Megatron initialization.\n\n---\n\nQ5. Limitation concerning the absence of contextual baselines.\n\nA5. Below, we provide clarifications and additional context regarding the positioning and contributions of our work.\n\n**Comparison with widely recognized foundation models.** We agree that direct comparisons with widely recognized foundation models (e.g., LLaMA 3.2, Qwen2.5) would offer valuable context. However, we would like to clarify that such comparisons are not straightforward due to substantial differences in training setups—**most notably, the total number of training tokens**. Our HybridNorm models were trained on only 1T tokens, while models like LLaMA 3.2 use up to 15T tokens, often combined with significantly higher-quality data, longer training schedules, and proprietary optimization pipelines. Under such vastly different conditions, direct performance comparisons would not be meaningful and could obscure the architectural contribution of HybridNorm. Instead, we chose to perform controlled, apples-to-apples comparisons between Pre-Norm and HybridNorm **under identical architecture, model size, token budget, and training setup**, ensuring that any observed improvements can be directly attributed to the normalization strategy. **Notably, the Pre-Norm baseline corresponds to the configuration used in LLaMA 3.2.** Since LLaMA 3.2 was trained on proprietary data, we conducted our comparisons using the same data and token budget across all settings to ensure fairness.\n\n**Orthogonality to data and scale improvements.** HybridNorm complements efforts to scale model size and data. As an architectural innovation, it can be seamlessly applied to any Transformer, regardless of scale or training corpus. Our extended experiments show **consistent performance gains even under longer training**, demonstrating strong scalability and potential for further benefits in larger, data-rich settings. Scaling laws suggest that modest performance gains often demand massive data increases—an expensive and unsustainable path. In contrast, **HybridNorm offers meaningful improvements with negligible training overhead**, enabling **faster convergence and reduced compute cost**. Given the limits of high-quality data, architectural enhancements like HybridNorm that **shift the slope or intercept of the scaling law** provide a valuable and efficient alternative.\n\n**Contribution and research focus.** Our focus is on understanding and improving the architectural design of normalization in transformers, an area that remains fundamental to the training dynamics, stability, and efficiency of large-scale models. While data and scale are undeniably critical, structure remains the foundation of model design, and our work contributes toward a better understanding of this foundational aspect.\n\n---\n\nHope our explanations above could ease your concerns. Please feel free to let us know if further clarification is needed."}]

Author rebuttal:

Meta-review: Pre-norm and post-norm are two norm approaches used in transformer blocks, each with a different set of benefits. This paper proposes a way to combine them by using pre-norm in the QKV transforms of the attention mechanism and post-norm in the FFN section. The authors show theory and experiments to show that this results in better gradient flow and robustness. 

Strengths
- Outperforms pre and post norm, and QK norm on dense and MoE models on multiple benchmarks. This directly improves the transformer architecture and is therefore, highly impactful.
- All reviewers appreciate the theoretical insights from this paper.

Weaknesses
- Many weaknesses have been reasonably addressed by the authors in the rebuttal 
    - New comparisons provided with DeepNorm, Sandwich LN and Olmo-2 norm
    - Testing on very deep architectures
    - Provided details of only negligible computational overheads
    - Figures illustrating how HybridNorm influences layerwise entropy evolution and signal behavior across the network.

For this paper, I am waiting for reviewers to provide their final justification. R1 seems to not have updated their score after rebuttal but have indicated that they are willing to do so.  Similarly, R4 has acknowledged that all their concerns have been addressed but have not updated their score. Once those scores are in, I would like to recommend weakly accepting this paper.
Decision: Accept (poster)