Title: Learning in Compact Spaces with Approximately Normalized Transformer
Venue: NeurIPS 2025
Accepted: yes
Keywords: Deep Learning

Transformer

GPT

Normalization

Regularization
Abstract: The successful training of deep neural networks requires addressing challenges such as overfitting, numerical instabilities leading to divergence, and increasing variance in the residual stream. A common solution is to apply regularization and normalization techniques that usually require tuning additional hyperparameters. An alternative is to force all parameters and representations to lie on a hypersphere. This removes the need for regularization and increases convergence speed, but comes with additional costs. In this work, we propose a more holistic, approximate normalization via simple scalar multiplications motivated by the tight concentration of the norms of high-dimensional random vectors. Additionally, instead of applying strict normalization for the parameters, we constrain their norms. These modifications remove the need for weight decay and learning rate warm-up as well, but do not increase the total number of normalization layers. Our experiments with transformer architectures show up to 40% faster convergence compared to GPT models with QK normalization, with only 3% additional runtime cost. When deriving scaling laws, we found that our method enables training with larger batch sizes while preserving the favorable scaling characteristics of classic GPT architectures.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nNormalization is crucial for the training stability of deep neural networks. Previous work nGPT [1], extensively use normalization layers to update all latent representations on a hypersphere, which eliminates the need for pre-training techniques such as warmup schedules and weight decay. However, this extensive normalization introduces significant computational overhead. Based on concentration measure in high-dimensional spaces, this paper proposes a more efficient approach: rescale each feature dimension with precomputed normalization factors, thereby avoiding weight normalization, which makes the entire transformer model \"approximately normalized\".\n\n[1] [nGPT: Normalized Transformer with Representation Learning on the Hypersphere](https://arxiv.org/abs/2410.01131)\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The motivation and core idea of this paper are clear: nGPT accelerates LLM pre-training and eliminates the need for hyperparameters related to learning rate schedules and weight decay. Given that feature vector norms concentrate in high-dimensional spaces, pre-estimating these norms allows normalization steps to be avoided.\n2. The empirical study of the paper is comprehensive, featuring clear scaling trends and insightful ablation studies.\n\n**Weaknesses**\n\n1. The example regarding fast contraction in Section 3.1 is not clearly connected to the authors' arguments about global learning rate selection. It would be beneficial to either find a more directly relevant example or to clarify the existing connection.\n\n2. The paper lacks some important details: (a) What are the exact procedures used to derive each of the normalization factors? (a) In Section 3.2, why $||h_{\\leq L} || = \\sum_{l=1}^L ||h_l || $? (c) How precisely are the \"effective learning rates\" applied? Although readers might guess, explicit clarification from the authors would significantly improve this paper.\n\n3. While assumptions such as the i.i.d. condition clearly hold at initialization, the paper does not sufficiently explain why these assumptions remain valid throughout training.\n\nQUESTIONS:\n1. How do you ensure that core assumptions, such as the i.i.d. condition or features being uniformly distributed on the sphere, continue to hold after training?\n\n2. Could you address the additional comments in the weaknesses section?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the approximately normalized Transformer (anGPT), a variant of the Transformer architecture that replaces standard normalization layers (e.g., LayerNorm) with fixed, pre-computed scaling factors. The central premise is that in high-dimensional spaces, the norms of random vectors tend to concentrate tightly around their expected values, a phenomenon known as concentration of measure. Leveraging this property, anGPT applies fixed scalar normalization factors throughout the network to approximate norm preservation, while also constraining the weight matrices to lie within a compact (bounded-norm) parameter space. This design eliminates the need for learnable normalization parameters, weight decay, and learning rate warm-up, resulting in faster convergence with minimal computational overhead.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n1) By replacing LayerNorm with fixed scalar normalization factors, anGPT reduces the computational and memory overhead typically associated with normalization, especially during inference, without sacrificing stability.\n\n2) anGPT achieves up to 40% faster convergence compared to GPT models with QK normalization, while removing the need for learning rate warm-up and weight decay, thus simplifying hyperparameter tuning.\n# Weaknesses\n\nPlease refer to the detailed questions I provided below, which highlight potential concerns around:\n\n* The validity of norm concentration under non-uniform, evolving training distributions.\n* The bias introduced by using fixed normalization factors.\n* The effect of LERP residuals on gradient flow.\n* Unclear robustness to domain shifts and out-of-distribution data.\n\nQUESTIONS:\nI appreciate the effort to simplify normalization in large Transformer models through a theoretically motivated framework. However, several mathematical and empirical claims in the paper warrant deeper scrutiny. Below are the specific questions intended to clarify the assumptions and implications of the proposed method.\n\n---\n1) The core approximation in anGPT assumes that for high-dimensional inputs \\( x \\), the output norm of a function \\( g(x) \\) concentrates:\n\\[\n\\|g(x)\\|_2 \\approx \\mathbb{E}[\\|g(x)\\|_2]\n\\]\nThis is motivated by classical concentration results (e.g., Vershynin's theorem for \\( x \\sim \\text{Unif}(\\mathbb{S}^{d-1}) \\)).\n\n- How valid is this assumption during training, where the input distribution evolves due to gradient updates and may be far from uniform or isotropic?\n- Can you provide empirical evidence of concentration in intermediate activations (e.g., histograms of \\( \\|g(x)\\|_2 \\))?\n---\n2) anGPT replaces exact normalization with a scalar factor \\( \\nu = 1 / \\sqrt{\\mathbb{E}[\\|x\\|^2]} \\), which is a biased approximation of \\( \\mathbb{E}[1/\\|x\\|] \\).\n\n- What is the theoretical or empirical impact of this bias on gradient magnitudes or effective learning rates?\n- Does this approximation systematically distort the optimization trajectory or loss surface?\n- Can you quantify the tradeoff between computational efficiency and scaling accuracy introduced by this simplification?\n\n---\n3) anGPT modifies the residual connection using a learnable interpolation:\n\\[\nh \\leftarrow (1 - \\alpha) h + \\alpha x\n\\]\nwith an additional normalization factor based on \\( \\nu(\\alpha) \\).\n\n- This changes the gradient propagation structure. Have you analyzed the Jacobians \\( \\partial h / \\partial x \\) and \\( \\partial h / \\partial h_{\\text{prev}} \\) to assess gradient flow?\n- For small \\( \\alpha \\), does the network risk under-utilizing new features, leading to a form of representational inertia?\n---\n\n4) The effectiveness of fixed normalization factors implicitly assumes consistent norm statistics across training and test distributions.\n\n- How does anGPT perform under domain shift, few-shot adaptation, or continual learning settings where norm distributions may vary?\n- Have you tested whether norm concentration persists in out-of-distribution scenarios?\n- Could fixed normalization factors cause feature collapse or instability in low-data or transfer regimes?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the Approximately Normalized Transformer (anGPT), a modification of the GPT architecture designed to stabilize training and accelerate convergence without relying on traditional normalization layers or extensive hyperparameter tuning. By using approximate normalization through input-independent scaling factors, anGPT mitigates the \"Curse of Depth,\" where deeper layers in vanilla GPT models experience growing input norms, leading to training instability. The approach eliminates the need for weight decay and learning rate warm-up, reducing hyperparameter complexity. Experimental results demonstrate that anGPT achieves up to a 40% convergence speedup compared to the vanilla GPT, with only a minimal (~3%) increase in training step runtime and potentially reduced inference times. The authors validate their approach on model sizes from 32M to 1B parameters and provide detailed hyperparameter scaling trends and compute-dependent scaling laws. \n\nContributions:\n\n1- Proposes the anGPT architecture, which uses approximate normalization to stabilize training and improve convergence speed.\n\n2- Eliminates the need for weight decay and learning rate warm-up, simplifying hyperparameter tuning.\n\n3- Demonstrates up to 40% faster convergence compared to vanilla GPT, with minimal runtime overhead.\n\n4- Provides comprehensive empirical analysis, including hyperparameter scaling trends and compute-optimal scaling laws.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Novel Approach to Normalization: The paper introduces a principled and lightweight alternative to full normalization layers using approximate scalar normalization factors derived from concentration of measure, which is both theoretically motivated and practically effective.\n\n- Improved Efficiency: anGPT achieves significant convergence speedups (up to 40%) with only ~3% added training cost and no increase in inference time, demonstrating clear practical benefits.\n\n- Simplified Training Pipeline: By eliminating the need for weight decay and learning rate warm-up, the method reduces hyperparameter tuning complexity, which is valuable in large-scale pretraining.\n\n- Strong Empirical Validation: The authors conduct thorough experiments across multiple model sizes and data scales, including compute-optimal scaling laws, showing consistent performance improvements.\n\n- Insightful Analysis: The work provides a clear theoretical explanation of the issues with norm growth in deep transformers (“Curse of Depth”) and demonstrates how approximate normalization addresses them.\n\nWeaknesses:\n\n- Limited Scope of Evaluation: The experiments are confined to GPT-style language models and two datasets. It remains unclear how the approach generalizes to other architectures (e.g., encoders) or modalities (e.g., vision, audio).\n\n- Approximation Validity Not Fully Explored: While the use of norm concentration is well-motivated, the paper does not deeply investigate cases where this assumption may break down (e.g., during early training or low-dimensional representations).\n\n- Complexity of Implementation: Despite simplifying training, the architectural modifications (LERP, ν-scaling factors) and need to derive dimension-specific constants may increase implementation burden for practitioners unfamiliar with the theory.\n\nQUESTIONS:\nsee Strengths And Weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the approximately normalized Transformer (anTransformer), a novel architecture that introduces approximate normalization through fixed scalar multipliers, motivated by the concentration of measure in high dimensions. Instead of exact normalization, it constrains the norm of parameters and representations to reduce the need for components like weight decay and learning rate warm-up. Applied to a GPT-style model (anGPT), the proposed method achieves up to 40% faster convergence than GPT+ (with QK norm) and matches or outperforms nGPT, while incurring only 3% extra training cost and negligible inference overhead.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well-written and presents a compelling motivation for approximate normalization based on theoretical insights and empirical observations. It introduces a clean and practical design that removes the need for additional normalization layers while retaining stability and training efficiency. Experiments are thorough, covering scaling laws, ablations, and comparisons across multiple model sizes and compute budgets. The results show consistent convergence gains and demonstrate that anGPT can match or outperform both GPT+ and nGPT.\n\nHowever, some concerns remain. First, the assumptions behind norm concentration and fixed normalization factors are only partially validated in practice; more ablations or analysis of failure modes would strengthen the argument. Second, while convergence is improved, the validation loss differences in the scaling law experiments are relatively modest. Lastly, downstream task evaluations are missing, which makes it unclear whether the improvements transfer beyond pretraining loss.\n\nQUESTIONS:\n- How sensitive is the method to the choice of fixed normalization factors? Would small misestimations hurt performance significantly?\n- Did the authors evaluate the method on any downstream tasks or instruction tuning settings?\n- Could the proposed approximate normalization be combined with other efficiency improvements, such as low-rank or quantized attention layers?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}