GPAS: Accelerating Convergence of LLM Pretraining via
  Gradient-Preserving Activation Scaling
Authors: name: Tianhao Chen; name: Xin Xu; name: Zijing Liu; name: Pengxiang Li; name: Xinyuan Song; name: Ajay Kumar Jaiswal; name: Fan Zhang; name: Jishan Hu; name: Yang Wang; name: Hao Chen; name: Shizhe Diao; name: Shiwei Liu; name: Yu Li; name: Yin Lu; name: Can Yang
Published: 2025-06-27 09:45:15

Abstract: Modern Large Language Models, such as the LLaMA, Qwen and DeepSeek series,
predominantly adopt the Pre-LayerNorm (Pre-LN) Transformer architecture. While
being stable during pretraining and scalable to large model sizes, Pre-LN
suffers from an exponential growth in activation variance across layers,
causing the residual path to dominate over sub-layer outputs and limiting the
learning capacity of deeper layers. To mitigate this issue, we propose
Gradient-Preserving Activation Scaling (GPAS), a simple technique that can be
used in combination with existing approaches. GPAS works by scaling down the
intermediate activations while keeping their gradients unchanged. This leaves
information in the activations intact, and avoids the gradient vanishing
problem associated with gradient downscaling. Extensive experiments across
various model sizes from 71M to 1B show that GPAS achieves consistent
performance gains. Beyond enhancing Pre-LN Transformers, GPAS also shows
promise in improving alternative architectures such as Sandwich-LN and
DeepNorm, demonstrating its versatility and potential for improving training
dynamics in a wide range of settings.