NeurIPS 2025

HybridNorm: Towards Stable and Efficient
Transformer Training via Hybrid Normalization
Zhijian Zhuo1, 2
Yutao Zeng2, †
Ya Wang2, †
Sijun Zhang2
Jian Yang3
Xiaoqing Li4
Xun Zhou2
Jinwen Ma1
1 School of Mathematical Sciences, Peking University
2 ByteDance Seed
3 Beihang University
4 Capital University of Economics and Business
Abstract
Transformers have become the de facto architecture for a wide range of machine
learning tasks, particularly in large language models (LLMs). Despite their remark-
able performance, many challenges remain in training deep transformer networks,
especially regarding the position of the layer normalization. While Pre-Norm
structures facilitate more stable training owing to their stronger identity path, they
often lead to suboptimal performance compared to Post-Norm. In this paper,
we propose HybridNorm, a simple yet effective hybrid normalization strategy
that integrates the advantages of both Pre-Norm and Post-Norm. Specifically,
HybridNorm employs QKV normalization within the attention mechanism and
Post-Norm in the feed-forward network (FFN) of each transformer block. We
provide both theoretical insights and empirical evidence to demonstrate that Hy-
bridNorm improves the gradient flow and the model robustness. Extensive ex-
periments on large-scale transformer models, including both dense and sparse
variants, show that HybridNorm consistently outperforms both Pre-Norm and
Post-Norm approaches across multiple benchmarks. These findings highlight the
potential of HybridNorm as a more stable and effective technique for improving
the training and performance of deep transformer models. Code is available at
https://github.com/BryceZhuo/HybridNorm.
1
Introduction
Transformers have become the backbone of large language models (LLMs) and a wide range of
machine learning applications. These architectures are capable of modeling long-range dependencies
through self-attention mechanisms, which have made them the preferred choice for a variety of
t...