Title: HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization
Venue: NeurIPS 2025
Accepted: yes
Keywords: transformer

pre-training

large language models
Abstract: Transformers have become the de facto architecture for a wide range of machine learning tasks, particularly in large language models (LLMs). Despite their remarkable performance, many challenges remain in training deep transformer networks, especially regarding the position of the layer normalization. While Pre-Norm structures facilitate more stable training owing to their stronger identity path, they often lead to suboptimal performance compared to Post-Norm.  In this paper, we propose **HybridNorm**, a simple yet effective hybrid normalization strategy that integrates the advantages of both Pre-Norm and Post-Norm. Specifically, HybridNorm employs QKV normalization within the attention mechanism and Post-Norm in the feed-forward network (FFN) of each transformer block. We provide both theoretical insights and empirical evidence to demonstrate that HybridNorm improves the gradient flow and the model robustness. Extensive experiments on large-scale transformer models, including both dense and sparse variants, show that HybridNorm consistently outperforms both Pre-Norm and Post-Norm approaches across multiple benchmarks. These findings highlight the potential of HybridNorm as a more stable and effective technique for improving the training and performance of deep transformer models. Code is available at https://github.com/BryceZhuo/HybridNorm.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nHybridNorm applies QKV-Norm in the attention mechanism and Post-Norm in the feed-forward network. The paper provides theoretical analysis of gradient flow benefits and validates the approach through experiments on both dense and MoE models.\n\nSTRENGTHS AND WEAKNESSES:\n- Theoretical analysis and empirical results offer clear insights into gradient stabilization.  \n- The paper lacks measurements of training and inference throughput, as well as memory overhead, associated with the added normalization steps. This makes it difficult to assess the practical efficiency of the proposed method.\n- Experimental comparison limited to Pre-Norm, Post-Norm, and Mix-LN, advanced schemes like DeepNorm and Sandwich-LN are mentioned but not evaluated.\n\nQUESTIONS:\n- The authors claim that HybridNorm improves the stability and performance of deep transformers. However, the evaluation is limited to models with up to 16 layers, which are relatively shallow compared to mainstream LLMs—such as the 28-layer 1.5B DeepSeek R1. Can the authors provide evidence that the observed stability gains generalize to deeper architectures commonly used in practice?\n- What is the computational overhead introduced by HybridNorm in terms of latency and memory consumption? How does this overhead scale with model size?\n- Table 3 shows Pre-Norm and HybridNorm have different sensitivities to initialization methods. What drives this divergence, and can the author provide an intuitive explanation?\n\nLIMITATIONS:\nWhile HybridNorm demonstrates improvements over Pre-Norm and Post-Norm under controlled experimental settings, the lack of direct comparisons with widely recognized foundation models (e.g., LLaMA 3.2, to which the authors’ tested architecture is closely related) limits the reader’s ability to contextualize the magnitude of these gains.\n  \nFor example, on the ARC-C benchmark, the 1.2B dense model with HybridNorm achieves a score of 37.06, which is lower than the Pre-Norm baseline at 39.2. Although the average score across benchmarks increases modestly from 62.99 (Pre-Norm) to 64.15 (HybridNorm∗), the relative improvements remain small. In contrast, mainstream foundation models such as Qwen2-1.5B and its enhanced variant Qwen2.5-1.5B achieve 43.7 and 54.7 on ARC-C, respectively—with the latter improvement largely attributed to higher-quality training data rather than architectural changes.\n\nCompared to such data-driven advancements, the performance gains offered by HybridNorm appear marginal. Without contextual baselines and further explanation, it remains difficult to assess whether the improvements are meaningful or competitive relative to more influential factors such as data quality or model scale.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper introduces a novel normalization technique that integrates post-normalization with QKV normalization to enhance the stability of transformer training. The authors demonstrate consistent performance improvements on dense LLMs (ranging from 150M to 1B parameters) and MoE models. Furthermore, the paper provides a theoretical analysis comparing the proposed solution with existing PreNorm and PostNorm approaches.

STRENGTHS AND WEAKNESSES:
Strengths:
* The development of improved normalization techniques is crucial for scaling base models, addressing a significant concern regarding training stability.
* The proposed solution is not only empirically validated but also theoretically supported.
* The experimental setup is extensive, although an evaluation on larger model scales (e.g., > 7B) would be beneficial.

Weaknesses:
* The method, primarily an exploration of normalization placement, has limited novelty.
* Evaluating the solution solely on models under 1B parameters may not be sufficient to assess its scalability and effectiveness in real-world LLM training scenarios.

QUESTIONS:
Questions:
* No questions regarding the technical aspects, as they are straightforward.
* Have the authors explored scaling the model beyond 7B parameters?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the HybridNorm method by combining the advantages of Pre-Norm and Post-Norm, it innovatively designs the normalization strategies in Transformer training, especially normalizing QKV in attention mechanism, and adopting Post-Norm in FFN, which in theory helps improve training stability and model performance. The author not only gave theoretical analysis, but also proved the effectiveness of HybridNorm through a large number of experiments, showing the application potential of this method in actual large-scale model training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe author conducted research in the normalization process of Transformer and proposed a training method that can efficiently transformer architecture. It systematically compared with existing mainstream normalization methods such as Pre-Norm, Post-Norm, QK-Norm, etc. The experimental part covers a variety of model structures and data sets, has a relatively comprehensive index, and is meticulous in the ablation experiment, which can fully verify the effectiveness of the method and the role of each component. It also displays a complete theoretical derivation process in the supplementary material, proving the effectiveness of the proposed method.\n\nWeaknesses:\n1.From the overall perspective of the article, the paper mainly focuses on performance improvement and training stability, but there is less interpretability analysis of the internal mechanism of HybridNorm, which can supplement some feature maps of different layers and experiments with heat maps to further verify the effectiveness of the method.\n2.The method proposed in this paper is carried out from the model with 151M to 7B parameters, but the model with a larger parameter quantity has not been fully experimentally verified. I hope that additional experimental verification can be carried out here in the future.\n\nQUESTIONS:\n1.From the overall perspective of the paper, the paper mainly focuses on performance improvement and training stability, but there is less interpretability analysis of the internal mechanism of HybridNorm. Why not you use some feature maps of different layers and experiments with heat maps to further verify the effectiveness of the method?\n2.The method proposed in this paper is carried out from the model with 151M to 7B parameters, but the model with a larger parameter quantity has not been fully experimentally verified. Why?\n\nLIMITATIONS:\nSee the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces HybridNorm, a normalization strategy for transformers that aims to combine the benefits of both Pre-Norm and Post-Norm approaches to address challenges in training deep transformer networks. HybridNorm specifically applies QKV normlization within the attention mechanism and Post-Norm in FFN of each block. The authors provide theoretical analysis and empirical evidence to demonstrate that HybridNorm improves gradient flow and model robustness.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. HybridNorm outperforms both Pre-Norm and Post-Norm across dense and Mixture of Experts models onmultiple benchmarks, demonstrating improved training stability and better downstream performance. \n2. This paper provides theoretical insights and gradient analysis to explain why HybridNorm improves gradient flow stability and model robustness.\n \nWeakness:\n1. The paper mentions that their theoretical analysis withRMSNorm. While RMSNorm and LayerNorm are common, the effectiveness of HybridNorm might be sensitive to the specific normalization function chosen, and the theoretical guarantees might not directly translate if other normalization techniques were considered\n\n2. The related work section mentions DeepNorm, Sandwich-LN, and OLMo-2 as normalization strategies. While the paper generally compares against Pre-Norm and Post-Norm, a more direct and in-depth empirical comparison with these advanced normalization methods could further solidify the superior performance of HybridNorm.\n\n3. This paper provides limited technical contribution. The method is very similar to Figure 6 in [1], merely replacing pre-norm with post-norm.\n\nQUESTIONS:\n1. Could the authors elaborate on the specific differences in how RMSNorm and LayerNorm  might influence the theoretical gradient analysis and the empirical performance of HybridNorm? \n2. Please empirically validate HybridNorm on 7B or larger models \n3. Fig. 6 in [1] uses QKV-Norm to stably train a Transformer. It seems that you only changed pre-norm to post-norm. Both of the pre- and post-norm have advantages, they have their own shortcomings, too. You used post-norm, which introduces the Gradient Vanish problem mentioned in [2]\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}