Hidden Dynamics of Massive Activations in Transformer Training
Authors: name: Jorge Gallego-Feliciano; name: S. Aaron McClendon; name: Juan Morinelli; name: Stavros Zervoudakis; name: Antonios Saravanos
Published: 2025-08-05 16:29:51

Abstract: Massive activations are scalar values in transformer hidden states that
achieve values orders of magnitude larger than typical activations and have
been shown to be critical for model functionality. While prior work has
characterized these phenomena in fully trained models, the temporal dynamics of
their emergence during training remain poorly understood. We present the first
comprehensive analysis of massive activation development throughout transformer
training, using the Pythia model family as our testbed. Through systematic
analysis of various model sizes across multiple training checkpoints, we
demonstrate that massive activation emergence follows predictable mathematical
patterns that can be accurately modeled using an exponentially-modulated
logarithmic function with five key parameters. We develop a machine learning
framework to predict these mathematical parameters from architectural
specifications alone, achieving high accuracy for steady-state behavior and
moderate accuracy for emergence timing and magnitude. These findings enable
architects to predict and potentially control key aspects of massive activation
emergence through design choices, with significant implications for model
stability, training cycle length, interpretability, and optimization. Our
findings demonstrate that the emergence of massive activations is governed by
model design and can be anticipated, and potentially controlled, before
training begins.