Learning in Compact Spaces with Approximately Normalized Transformer
Authors: Jörg K.H. Franke
Urs Spiegelhalter
Marianna Nezhurina
Jenia Jitsev
Frank Hutter
Michael Hefenbrock
Venue: NeurIPS 2025 poster ()
Keywords: Deep Learning
Transformer
GPT
Normalization
Regularization
Primary area: deep_learning
TL;DR: This paper presents a holistic and approximate normalization approach that accelerates GPT training by up to 40% while eliminating the need for weight decay and learning rate warm-up.

Abstract: The successful training of deep neural networks requires addressing challenges such as overfitting, numerical instabilities leading to divergence, and increasing variance in the residual stream. A common solution is to apply regularization and normalization techniques that usually require tuning additional hyperparameters. An alternative is to force all parameters and representations to lie on a hypersphere. This removes the need for regularization and increases convergence speed, but comes with additional costs. In this work, we propose a more holistic, approximate normalization via simple scalar multiplications motivated by the tight concentration of the norms of high-dimensional random vectors. Additionally, instead of applying strict normalization for the parameters, we constrain their norms. These modifications remove the need for weight decay and learning rate warm-up as well, but do not increase the total number of normalization layers. Our experiments with transformer architectures show up to 40% faster convergence compared to GPT models with QK normalization, with only 3% additional runtime cost. When deriving scaling laws, we found that our method enables training with larger batch sizes while preserving the favorable scaling characteristics of classic GPT architectures.

Learning in Compact Spaces with
Approximately Normalized Transformer
Jörg K.H. Franke 1233,44
Urs Spiegelhalter'
Marianna Nezhurina 3.4.5
Jenia 31t
Frank Hutter
Michael Hefenbrock?
Uniiversity of Freiburg
"ELLIS Institute Tübingen
SOpen-Sii Collective
*LAION SJülich Supercomputing Centre (JSC)
"rrior Labs
PPerspix.ai
Abstract
The successful training of deep neural networks requires addressing challenges such as overfitting, numerical instabilities leading to divergence, and increasing variance in the residual stream. A common solution apply regularization and normalization techniques that usually require tuning additional hyperparameters. An alternative force all parameters and representations to lie on hypersphere. This removes the need for regularization and increases convergence speed, but comes with additional costs. In this work, we propose more holistic, approximate normalization via simple scalar multiplications motivated by the tight concentra- tion the norms of high-dimensional random vectors. Additionally, instead applying strict normalization for the parameters, we constrain their norms. These modifications remove the need for weight decay and learning rate warm-up as well, but do not increase the total number of normalization layers. Our experiments with transformer architectures show up to 40% faster convergence compared to GPT models with QK normalization, with only 3% additional runtime cost. When deriv- ing scaling laws, we found that our method enables training with larger batch sizes while preserving the favorable scaling characteristics of classic GPT architectures.
Introduction
Normalization techniques, such as LayerNorm, are fundamental for stable and efficient Transformer training (144]. Loshchilov et al. (5) extends the concept and proposes the normalized Transformer (nGPT), where all latent residual representations and all parameters in the direction of the residual normalized to lie hypersphere. We argue that the benefits of normalization come from two effects. Normalization prevents the representations on the residual stream from blowing up and requiring deeper layers to significantly amplify their output magnitudes. An effect observed by Sun et al. [6] and termed the "Curse of Depth" (see Figure D. Additionally, normalization ensures consistent input scale, which allows for the selection of more suitable (global) learning rate.
These benefits drive us toward architectures that consistently apply normalization throughout the net- work, potentially normalizing all representations, Unfortunately, such excessive of normalization increases training and, more importantly, inference times. To combat this problem, we introduce an approximate normalization technique for normalizing vector via normalizing factors satisfying
(||=||2))- that
||| 2:12 1.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

GPT+
anGPT
nGPT
Layer inpat: norm
163
163
163
1 Loyy index
1e2
262
1e2
2:2
15
Bel
201
10
lol
lo1
1=1
1-
20k
10%
10k
Training steps
Training steps
Training steps
Figure 1: The input norm log scale for each layer as function of training 0.5B model on 10B tokens. Deeper layers obtain higher imput norm in the classical GPT. While nGPT completely eliminates this "Curse of Depth", anGPT effectively mitigates it.
These normalizing factors are enabled by the concentration of measure phenomenon in high- dimensional spaces, which may be perceived as a "blessing of dimensionality". 'hee augmentation of each operation with (approximate) normalizations, along with other modifications like bounding the norm of the input dimension of each linear map, forms the basis of proposed approximately normalized Transformer (anTrasffommer) When applied to pretraining large language models, we approximately normalized GPT (anGPT) architecture without the
adapt the GPT architecture need for additional normalization layers. Our approach does not require weight decay nor learning rate warm-up, effectively reducing the number of hyperparameters in training. Compared to vanilla GPT architecture, anGPT achieves up to 40% convergence speedup. Measurements show less than 3% larger training step runtime, while expecting further reduction for inference times by subsuming normalization factors into the model parameters.
Our core contributions be summarized follows:
We motivate the benefits of normalization Section 3
The proposed approximately normalized transformer Tannrassfrrmer) in Section + displays faster convergence and fewer hyperparameters at minor increase in training time.
Section presents extensive experimental evaluations:
Hyperparameter scaling trends are derived across multiple model sizes.
Results demonstrate over 40% convergence speedup compared to GPT models with QK normalization and outperform or perform on par with nGPT.
Compute-dependent scaling laws reveal scaling behavior matching GPT.
Related Work
Normalization techniques for deep neural networks have evolved significantly, from BatchNorm which standardizes across batch dimensions but struggled with sequential data, to LayerNorm []] which computes statistics across feature dimensions independently for each sample, making it ideal for transformers. The original Transformer 2 uses Post-Norm configuration (LayerNorm after residual connections). requiring careful learning rate warm-up to prevent gradient explosion, while Pre-Norm architectures (normalization before operations) offer more stable training dynamics with higher potential learning rates [3] 4). RMSNorm [8] further refined LayerNorm by climinating the mean-centering step, delivering runtime improvements while maintaining performance in state-of- the-art LLMs 99 I0]. The normalized Transformer (nGPT) uses the normalized representation and parameter space to ensure that input tokens "travel" on the surface of hypersphere, with each layer contributing a displacement towards target output predictions. These modifications render weight decay and learning rate warm-up unnecessary as vector magnitudes are explicitly controlled [5]. Empirically, nGPT demonstrates convergence speedups and reduces the number of training steps required achieve equivalent accuracy. However, these improvements come with computational overhead due to additional normalization layers. Further related work is discussed in Appendix A

---

Preliminaries
Normalized representations are known stabilize training and accelerate convergence [35]. We hypothesize that this stems primarily from two effects. First, by ensuring each layer receives normalized input, the imput scale becomes consistent across the network. This uniformity can streamline learning and simplify the selection of global learning rate. Second, as contributions to the residual stream accumulate in vanilla Transformer architectures, subsequent layers need to amplify their output to remain influential. This leads to growing norm of the representation on the residual stream (see Figure D. which may destabilize training. Both effects are elaborated below using toy examples.
3.1
Why does normalization influence optimization?
To better understand the role of the input scale in gradient descent on the learning rate, consider the problem ming e Aæ with A = diag(A, (A,,((A...... Ad) and X > O. Using learning rate of a > 0, the gradient descent update is given by
- x aAT (I aA)æ.
Each entry 2; of z converges with rate determined by the contraction factor p=(a) = a&g. To ensure convergence, we require (a))) < 1 with smaller values indicating faster convergence. Hence, the goal is to set to minimize the maximum contraction factor max; pi(ar). The max; (a is
either achieved for the smallest Amin or the largest Amax. To achieve the same convergence speed for both, we require the optimal learning rate a* to satisfy
K Amax Amin and max pi
11 aaAmax = |1 admin
K K+1
where Amax/ Amin is the condition number of A. One can see that the fastest contraction is realized if k = 1, or equivalently, if Amax = Amin. Additionally, to ensure convergence, a must satisfy a 2/Amax. If Amax >> Amin. we expect a slow convergence rate for the entry relating to Amin as its contraction factor is bounded by the largest learning rate that Amax allows.
Now, consider case where each column of A is normalized by multiplication of diagonal preconditioner P diag(pı,... Pa), with P; = (112)12)) where X3 is the j-th column of A. In this case, all X; and learning rate be picked that leads the same convergence speed for all coordinates. This fact makes good learning rates more effective, regardless of how they are found (e.g. manual tuning, grid-search, some sophisticated optimization). Consequently, normalization can serve as a (diagonal) preconditioner and improve the convergence speed of gradient-basee learning methods.
While the example problem simple, may still provide some intuition about how normalization can help learning, namely, by allowing the selection of well-wort well-working learning rate for all parameters. Even together with methods like Adam [T], normalization may provide benefits by improving the conditioning of the optimization landscape and yielding more stable gradient statistics for moment estimation. This can be seen by the gradual increase of the variance of the first moment, see Figure B.1
3.2
Why does the norm of the residual connection increase?
Assume independent random vectors h, with [hrr = 0 and | |l = 1, representing the contribution of each layer on the residual connection. For the (L 1)-th layer to have effective contribution the residual state hgg = Σ hú, such the ability to overwrite has have magnitude similar to that of h Specifically.
L
L E
h R
= E
h
h
+
Ehh E[hrr] = - 1yp
Consequently, || ||2 VT. is expected. Such effects lead to growing outputs on the hidden state that can also observed experimentally, see Figure Since the weights are subject to regularization, the large output scales are likely produced by the scaling factor 7 in the input norm of each block of
3

---

Growing norms on the residual stream were also described in [6] and coined "Curse of Depth". As fix, they proposed to scale the LayerNorm output of layer / by 1 1/vl. Alternatively, this growth can also be addressed by employing Post-Norms and keeping the residual connection normalized as in Loshchilov et al. 5
the transformer, see Figure C.1
4
Approximately Normalized Transformer
Due to the potential benefits of normalization, it is tempting to normalize the inputs for each primitive, namely, linear maps, activation functions, and residual updates. Unfortunately, such excessive use of normalizations might significantly influence the runtime [5]. However, to still keep the benefits of consistent normalization at lower cost, this work explores an approach to replace normalization operations with cheaper, approximate computations.
4.1 Approximate Normalization
To reduce the overhead introduced by excessive normalization (in particular at inference time), the proposed method attempts to approximately normalize the representations in the architecture through input independent normalization factors v. Concretely, for some vector æ, there exists some v, with
~ (1|=112))- so that || x||2 2m
we may use in place of || for normalizing æ. If such normalizing factors v can be found for all operation primitives, approximately normalized representations should be achievable without the need for exact normalization operations. It is clear that for such normalizing factors to exist, the norms have to concentrate closely around some value w-1 Fortunately, under certain conditions, such behavior can indeed be observed.
Theorem (Concentration of Lipschitz functions on the sphere).
12
pp. 106-109)
Let U(Su-11 be a random vector uniformly distributed the Euclidean unit sphere Sd =(x Rdd ||0||2 1} and let f S l. be Lipschitz function. Then, for every
t t2 0,
cdl2
P(f((=) E(](=) 2exp
where e > and Il denotes the Lipschitz norm (smallest Lipschitz constant) of f.
In words, Theorem tells us that the deviation of f(x) from its expected value decays exponentially in the dimension d, assuming f is Lipschitz and æ ~ Un This implies that in high dimensions, f(x) is likely to be close [=]). phenomenon often referred to concentration of measure.
In our setting, we consider functions of the form f(xx) [g(())]2, where g(x)) denotes the output of network component (e.g., feedforward layer) given input æ. Assuming 9(-) is Lipschitz and inputs a are normalized and approximately uniformly distributed on the sphere (e.g., by sampling from a Gaussian and renormalizing [12] p. 52]), the conditions of Theorem I are approximately satisfied. Consequently, the output norm [ (z) ||z may concentrate around its expected value.
While these assumptions do not strictly hold during training, we may still observe concentration empirically, particularly due the high dimensionality of representations. This "blessing of di- mensionality" can justify approximating [[g(2))|2 by its expected value -ー [[]])(())]])) for the purpose of normalization.
4.2 Derivation the normalizing factors
Motivated by concentration effects in high dimensions, derive normalizing factors for squared norms / /E[[|a>] ). and then use wー / TE| [3 Computing this way should generally lead to an overestimation due to Jensen's 's inequality, as |a > | = E [2]] However, for sufficiently high-dimensional derivation for the network components used for Transfformer is described below. normalization
the bound tends to an equality due to the concentration effect. The
of the attention matrix is described in Appendix D since it is not part of the architecture.

---

GPT+
nGPT
anGPT (ours)
Embed
x Win
h Win
Wa
ha 5 rm)( TYa
q, k, v + W. Wyku
k; + Wykko
q, k, v + Wykeft Mykko norm(k)
norm
k norm(A) (k)
MHA
7 norm (g)
9 norm(q) q) 8g
q norm
A - softmax (gl
A < softmax (gk" ddd)
A t softmax(gk"" :g)
ha < (p(Aw)
he < Wp(An)
ha + (v(Av) V/p
he norm(h.) ha)
h. norm(ho)
Residual
4---11 h + h
h < norm + ac (h.. h))
+ (h + ana ana h)) M(
hm +- mms(h)- mm
u <- Wuch
Wush u u Su.
u, +- Wush Vux
MLP
85 Vil hm Si).U(z)
hm t- (z)
hm Sii.U(z) Macf
hm t Wahm
hm f Walm hm + norm hm
hm t Wahm- V/d
hm t norm (hm)
Residual
h + h hmm
h norm + 0.m (mm h))
h f (h a.m (hm h.) vl
f- (tt) 7
Head
logits +- Wi.h
logits <- az (Wah)
logits +- az (Wih))
Table Comparison between GPT implementation with SwiGLU, RMSnorm, and QK-norm (GPT+), the normalized Transformer (nGPT), and our approximated normalized GPT (anGPT), We define
rms h) h/ I hă and norm(h) h/|lblla and colored learnable parameters green, constant scaling factors blue, and normalization factors purple.
Linear map Wæ Assume E Rd and W E Rexd with entries T4, waj ~ ~N00 1) and normalized afterwards such that ||>|| = 1 and l|w,;12 where un denote the rows of W. Then,
B W 2
5 E
w x)²
Σ
B[ x) =
1 d
d
Note that specifically the assumption that W is normalized along its input dimension, i.e, the "weights of each neuron", has to be reflected in training in the form of constraints.
Residual update Assume x; E IR with independent entries Is, h4 ~ No,, 1) and normalized afterwards such that ||>||| = 1 and ||h]|2 = 1 For the classic residual update, this yields
B + xc [/ Bih
2h Th x
0+10+ -=
Loshchilov et al. 5 proposed to replace the classic residual update by a linear interpolation (LERP) + + æ with < + a. h) and a > 0 which leads to the benefit of explicitly learning the impact of a layer.
E| h + a(x
h)] [B[[(1 a:) + amr
= (1 - a) E T + 2a(11 - [F ax] + R/xx
= -1 - 2a + a² + + a² = 1- 2a + 2a
The same derivation also holds for element-wise multiplication with a vector a instead scalar.
Activation function Due to the nonlinearity of activation functions, we resort to numerical compu- tation of the (squared) norm of the activations. For the calculations, we assume uniform distribution on sphere 5d-1, Specifically, inputs 2; ~N(0, 1) and normalize afterwards such that || 1. For computing the expected value, quadrature methods or Monte Carlo estimation can be used.
Constraining Parameters As mentioned in the derivation of the linear map, we require our weights be normalized along the input dimension. However, this does not allow to express zero vectors (the zero vector does not lie on the hypersphere). We thus only employ bound ||w]|| < 1 similar to [3] which allows the weights to adapt more freely. Consequently, our representations are not necessarily normalized but bounded, and tending towards the boundary, see Figure P.I These representations
5

---

are therefore termed compact. Bounding the parameters also eliminates the need for regularization, such as weight decay. In line with Loshchilov al. [5], we found preliminary experiments that we do not require learning rate warm-up when initializing the parameters already normalized along the input dimension. This may be due to the more stable behavior of optimization, see e.g., Figure B. for the variance of the first moment in Adam.
4.3 Approximal normalized GPT (anGPT)
In the following, describe our approximal normalized GPT model, anGPT, inspired by [5] In contrast to nGPT, we perform normalization consistent with the assumption above, while nGPT nor- malizes along the residual dimension (to stay on hypersphere). This leads different normalization dimensions for linear maps in the input and output of an attention or feed-forward layer. Let dm. d,. d denote the model, attention head, and up-scaled dimension, respectively. In the feed-forward MLP, I as the number of layers, and du as the vocabulary size. An overview of our architectural modifications, including nGPT, can be found in Table T
Replace norm and residual update First of all, we remove all classical normalization layers and replace them with post-L--normalization norm(z) z/ | Removing normalization completely leads to unstable training. The architecture replaces the classic residual update with LERP and replaces the learnable per-element affine parameter from the pre-norm layer with a learnable interpolation parameter a.
Add normalization factors We add constant normalization factors in the attention layer for the query, key, and value map, but due to the reshape into the head dimension, we consider the head dimension as the output dimension Vyke / Hom/dn- Similarly, we add normalization factor for the output map U/- ddn/dm but with dn input dimension. In the feed forward layer we add three normalization factors Vuu = ddm/ Va = (4ddm)/dm. and ss 3.74 (estimated via Monte Carlo) for upscaling. downscaling and the activation function, respectively. The normalization factor for the residual LERP update is a function of a and calculated at each step through v v(()) 1 2a + 2a². A list of normalization factors for each primitive can be found in Appendix E During inference, we can subsume constant normalization factors and the logits scaling into the model parameters. Since we need expect the same inference time.
normalization layers than the GPT model,
Logits scaling anGPT removes the RMSnorm before the head linear since the representation is already normalized. However, to scale the logits, a learnable scaling vector 8 E Rd is added, similar to nGPT. Scaling before the head linear was also tested but reduced performance.
Parameter Reparameterization for Uniform Optimization Following nGPT [5], we employ a reparameterization scheme to ensure uniform optimization dynamics across parameter types. For any trainable scaling parameter Sa (e.g., a,, aMM 82), we optimize a surrogate parameter &a and compute:
iaiini
84
80
sa,ccae
The surrogate is initialized as =00)) Sa,caae. ensuring 8a
(0) = 8ajinit This reparameterization ensures all stored parameters have comparable magnitude 8a,scales enabling Adam's 's adaptive learning rate mechanism to work uniformly across the network. The effective learning rate for updates to sa
becomes Qut = a (So. S s..ee)
5 Experiments
In the following, we describe multiple LLM pretraining experiments comparing anGPT to GPT and nGPT. The experiments use SlimPajama [14] with -627B tokens and train for < 1 epoch. All experiments employ the GPT-NeoX tokenizer with a 50k vocabulary size [15] and a context window of 2048 tokens. As baseline, we extend the vanilla GPT architecture with SwiGLU activation function [66), rotary position embedding [17], RMS normalization [8) and QK normalization [18] and dub the resulting model GPT+. All models are trained with Adam ||| and, in the case of GPT+,

---

Batch
Learnnng
anGPT and poizts
1024
1e-5/
GPT+ points
pnCPT interpolated points
16:2
GPT; Interplated points
250
1n-7/3
mGGPT line
GPT+ fit line
In-8/
Fits:
64
anCPT:
BS - 1.1e 0cypAss (R"-0004)
0.0652V-6 cam (PS-0.630)
GPT+:
10
1e/10/0
BS = Se 15,y (tP000 (RP-0783)
32M
62M
12/M
20/34
503M
323
62M
12/M 24YM
305M
N
NV
Figure 2: Scaling trend fits for optimal batch size and learning rate as functions of model size N. Grid point markers shaded by excess loss relative all configurations for this parameter. Diamond markers show the two-stage interpolation- based estimates of optimal hyperparameters. Dashed lines represent fitted power laws using the estimated optimal hyperparameters.
with AdamW [19]. For learning rate scheduling, cosine learning rate annealing (20 is employed. If learning rate warm-up is used, 10% the total training steps are dedicated to warm-up. For adapting the effective learning rate scaling parameters, we set ini = 0.01 for anGPT and &a,init = (d) for nGPT. The following experiments compare anGPT to nGPT and GPT+. Hyperparameters reported in Appendix and comparisons with related work in Appendix A
5.1 Hyperparameter Scaling Trends
To determine optimal learning rates and batch sizes, hyperparameter scaling trends were derived across model sizes from N 32M N 0.5B parameters (Configuration in Table F.Ip. The methodology follows Porian et al. [21] and DeepSeek-Al et al. [10]. We performed a grid search each model size over the batch size and learning rate with a Chinchilla optimal token budget (D 20N, )22). Optimal batch size and learning rate for each model size were obtained via two-stage interpolation process adapted from Porian et [2 The estimates for the optimal values are shown in Figure The process is described in detail in Appendix G with full hyperparameter sweep results in Figure G.I When fitting hyperparameter scaling trends for the optimal parameter selection, we found that anGPT can use larger batch sizes for larger model sizes, which could be beneficial for scaling the pretraining to a large number workers for achieving speedup. For the scaling behavior of the learning rate, we found small negative exponent for anGPT, so the learning rate decreases gradually as models get larger, which is expected behavior in line with previous works
23 10]
5.2
Performance comparison
To compare our approach to GPT+ and nGPT, we trained a 0.5B model of each architecture on token budgets from 5B to 70B (0.5x to 7x Chinchilla optimal). For GPT+ and anGPT, we used the optimal hyperparameters from the previous grid search and performed additional grid search for nGPT. Figure reports the results and convergence speedup against GPT+, Results show that anGPT and nGPT outperform GPT+ across all token budgets, with nGPT performing better at smaller token budgets and reaching comparable performance at larger ones. Convergence measurements reveal an average speedup factor 1.4x for anGPT and 1.29x for nGPT compared to GPT+ Similar to Loshchilov et al. (5), convergence speedup measurements against GPT+ without QK normalization yield speedup factors of 2.0x for anGPT and 1.8x for nGPT, see Figure H.2 We discuss the discrep- ancy to [5] reported speedups in Appendix H In addition, we performed performance comparisons on different token budgets for the 250M and 1.0B model sizes and find the 40% convergence speedup observed for the 0.5B model consistent. Convergence plots for both models are provided in Appendix[!
Table 2 reports the average runtime per training step for all three architectures. anGPT shows an increase of ~3% and nGPT shows ~9% increase. The additional runtime attributed to the additional scaling factors, the norm implementation, and the additional norm for nGPT. During inference, similar
7

---

Model size 0.5B trained 5-70B Slim Sijmaaama tokens
2.8
Final Validation Loss
GPT+
nGPT
mnGPT
26
10
20
30
40
50
60
70
Convergencee
Speeduup
Reference Data
066
10
20
30
40
60
70
Tokens Budget (B)
Figure 3: Training the 0.5B model up to 7x Chinchilla optimal token budget. Each point is the final validation loss of full training with the training budget noted the abscissa. Below, measure the convergence speed-up against the GPT+ model with QK normalization.
runtimes were observed for GPT+ and anGPT but -3% increase for nGPT, due the additional normalization operations.
5.3 Comparing performance via compute-dependent scaling law
We further compare GPT+ and anGPT by training model sizes from 32.M to 1B parameters on to- ken budgets from 0.5x to Chinchilla optimal. This enables investigation of the scaling behavior of the new anGPT architecture. ssows the results and scaling fits. Given the same compute budget, anGPT outperforms GPT+ on any model size and compute budget. These results were used to derive scaling laws using the approach of Hoff- mann al. [22). described in Appendix []] Both GPT+ and anGPT exhibit nearly identical esti- mated scaling law exponents, implying that the improvement of validation loss not significantly different across these architectures.
Table 2: Runtime Comparison with 0.5B pa- rameter model on a GPU node with 4 A100 (40GB) GPUs with sequence length of 2048 and a batch size of 8. The experiments use torch.compile with default settings.
Avg. Runtime per Step
Rel. Increase (%)
Model
GPT+ anGPT nGPT
0.1416
0.1455
2.75
0.1552
9.60
5.4 Downstream Evaluation
To verify that pretraining improvements transfer beyond perplexity metrics, we evaluate our models on standard benchmarks. The IB anGPT model consistently outperforms GPT+ across six benchmarks and three training budgets, with improvements ranging from 3% to 22% depending on the task. Detailed downstream evaluation results are presented in Appendix K
5.5 Ablation studies
To investigate the modifications added to vanilla GPT architecture, small experiments were con- ducted on a 0.5B GPT model trained on 10B tokens from OpenWebText [24]. When adding QK norm to the baseline, it leads to performance gain of 2.1% as visualized in Figure 5) Additional benefits emerge from using nGPT, with further improvements from using weight bounds instead of weight normalizations. The replacement of the normalization layer after the LERP update by normalization factor does not reduce the performance. For anGPT (additional normalization factors, different normalization dimension, removing scaling vectors),
additional gain of 0.4%. Ex-

---

Scaling Curves: Loss
Compute Budget.
GPT+, 62M
Final validation loss
GPT+, N 127M
34
GPT+, 247M
GPT+, N = 505M
GPT+, 1072M
30e
anGPT: 28.4 x
GPT+: 32.6 32.6*x-0.060
10
anGPT, N = 62M
anGPT, 127M
2.6 10
anGPT, N = 247M
anGPT, 505M
x 10"
anGPT, N 1072M
10
1019
Compute C [FLOPs]
Figure 4: Training different model sizes on different token budgets. Each point represents a full training with the training budget noted on the abscissa. The scaling law is fitted for both architectures as described in Appendix [ and indicated by the dashed line.
tensive ablation studies further assess the sensitivity of anGPT to variations in normalization factors. Our analysis shows that while the method robust to small estimation errors (below 2%), the residual normalization factors are critical for training stability. Complete ablation results, including sensitivity to factor scaling and comparisons with different configurations, are provided in Appendix
Model
Style
Improv.
Validation Loss
3.0
GPT+ w/o QK norm GPT+
O
2.1%
nGPT
3.0%
2.8
nGPT with LERP correction
3.0%
nGPT with param. bounding anGPT
A
3,4%
3.8%
5000
10000
15000
20000
Training Steps
Figure We ablation experiments with a 0.5B parameter model using 10B tokens from Open WebText. Adding QK norm shows performance gain. We modify nGPT by replacing the normalization of the LERP update with a normalization factor and, addition, by bounding weights instead of normalizing them. The anGPT mainly replaces scaling vectors by normalization factors.
5.6 Analysis of anGPT
First of all, we analyze the learnable interpolation parameters a and find adaptive feature utiliza- tion throughout training, with values increasing from 0.05 to 0.12 1.25.. Details are provided in Appendix M Further, we empirically verify that approximate normalization maintains stable norms throughout the network. Analysis shows that anGPT maintains near-unity norm ratios (0.86-1.86) across layers while GPT+ exhibits significant norm growth (up to 5.83 deeper layers). Detailed measurements are provided in Appendix N Finally, we evaluate robustness to distribution shifts using pathological inputs. anGPT maintains better norm stability (1.5x change) compared to GPT+ (3.4x change), with catastrophic failures. See Appendix O for detailed analysis.

---

6 Limitations
Despite the extensive experimental evaluation, we do not perform experiments with more than 7x Chinchilla optimal token budgets; our approach could perform worse than nGPT or GPT+ training with larger budgets. We also did not perform multiple experiments with different random seeds to generate error bars due to the high cost of GPT pretraining. We do not evaluate on the downstream task and assume that the validation loss correlates with the downstream performance. It is also hard to get meaningful downstream signal from small and short-trained LLMs. Lastly, we perform the majority of the experiments on only one dataset and only with the GPT architecture; the performance could diverge with different datasets and data modalities.
7 Discussion
In our preliminaries, we hypothesized that the benefits of normalization stem from two effects: stabilizing input scales across layers and preventing representation norm escalation. Our experimental results provide strong evidence supporting both hypotheses. First, we hypothesized that normalization enhances optimization, as confirmed by Figure [B1 This shows that anGPT exhibits significantly re- duced variance in Adam's is first moments compared to GPT+, without requiring learning rate warm-up. Second, our hypothesis addressed the "Curse of Depth." As seen in Figure [ GPT+ shows input norms growing exponentially with depth, while anGPT successfully constrains this growth. Traditional normalization layers employ learnable parameters y that serve two purposes: stabilizing network acti- vations while simultaneously participating in loss minimization. This conflation of roles complicates optimization dynamics. In anGPT, we deliberately decouple these concerns- normalizaionnnnn factors handle stabilization exclusively, while the remaining parameters focus solely on minimizing the loss. This decoupling eliminates the need for warm-up while preserving favorable scaling properties. Since weight decay is also unnecessary, the practical impact is substantial: hyperparameter tuning becomes simpler and scaling law derivation more efficient, as fewer parameters need to be tuned.
Interestingly, the theoretical assumptions motivating our normalization factors do not strictly hold during training and are even violated by design. This occurs because weights are bounded rather than normalized, leading to compact rather than perfectly normalized representations. Nevertheless, we still observe concentration empirically (see Figure P.I.. particularly due to the high dimensionality of representations. This robustness to assumption violations actually strengthens our approach, demon- strating that exact normalization is unnecessary to achieve the benefits traditionally associated with normalization layers. The key insight is that decoupling stabilization from loss minimization enables 1.4x faster convergence compared GPT with QK norm, while adding only minor computational overhead (see Figure 3
Looking forward, several directions warrant further investigation. The compact representation space maintained by anGPT makes it particularly amenable to reduced-precision training. The bounded nature of activations and weights could enable efficient FPS training without the numerical instabilities typically associated with low- low-precision arithmetic in unbounded architectures. Additionally, extending the approximate normalization framework other architectures such as vision transformers and diffusion models could yield similar efficiency gains. Another promising avenue is exploring whether alternative optimizers might be more suitable for normalized architectures than Adam, as the bounded parameter space and stable gradient flow could benefit from optimization algorithms specifically designed for compact spaces.
8 Conclusion
We presented anGPT, approximately normalized transformer that achieves faster convergence through scalar multiplication- based normalization. By leveraging the concentration of norms in high-dimensional spaces and decoupling stabilization from loss minimization, our method eliminates the need for weight decay and learning rate warm-up while maintaining only 3% computational overhead. Overall, an Transformer demonstrates that the benefits of consistent normalization, such as convergence speedup and fewer hyperparameters, can be achieved with minimal computational overhead, and provides promising approach to training large language models with predictable scaling behavior.
10

---

Acknowledgements
This research was funded by the Deutsche Forschungseemeinschati (DFG, German Research Foun- dation) under grant number 417962828. We acknowledge funding by the European Union (via ERC Consolidator Grant DeepLearnnng 2.0, grant no. 101045765). Views and opinions expressed are. however, those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
Funded by the European Union
Further, we acknowledge funding by the Federal Ministry of Education and Research of Ger- many (BMBF) under grant no. 011S22094B (WestAI AI Service Center West), under grant no. 011$24085C (OPENHAFM) and under the grant 16HPC117K (MINERVA), as well as co-funding by EU from EuroHPC Joint Undertaking program under grant no. 101 101787737 (MINERVA) and from the Digital Europe Programme under grant no. 195233 (openFurol.LM).
We gratefully acknowledge the Gauss Centre for Supercomputing e.V. for funding this work by providing computing time through the John von Neumann Institute for Computing (NIC) on the super- computer JUWELS Booster at Jülich Supercomputing Centre (JSC), EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer LEONARDO, hosted by CINECA (Italy) and the LEONARDO consortium through an EuroHPC Extreme Access grant EHPC-EXT-2023E02- 068, storage resources on JUST granted and operated by JSC and supported by Helmholtz Data Federation (HDF), compute resources provided via WestAl compute grant "Measuring and enhancing advanced reasoning capabilities of foundation models via local model deployment" (westai0007, westai0066) at JSC and compute resources provided via JSC JURECA
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.
[2] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Atten- tion is all you need. In I. Guyon, U. von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Proceedings of the 30th International Conference on Advances in Neural Information Processing Systems (NeurIPS 17). Curran Associates, Inc., 2017.
[3] Toan Q. Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self- attention. In Jan Niehues, Rolando Cattoni, Sebastian Stüker, Matteo Negri, Marco Turchi, Thanh-Le Ha, Elizabeth Salesky, Ramon Sanabria, Loic Barrault, Lucia Specia, and Marcello Federico, editors, Proceedings of the 16th International Conference on Spoken Language Translation, Hong Kong. November 2-3 2019. Association for Computational Linguistics.
[4] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization the transformer architecture. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10524-10533. PMLR, 13-18 Jul 2020.
[5] Ila Loshchilov, Cheng Cheng-Ping Hsieh, Simeng Sun, and Boris Ginsburg. nGPT: Normalized transformer with representation learning on the hypersphere. In The Thirteenth International Conference on Learning Representations, 2025.
[6] Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, and Shiwei Liu. The curse of depth in large language models, 2025.
[7] S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning (ICML' 15), volume 37. Omnipress, 2015.
[8] Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32. 2019.
[9] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux. Timothée Lacroix, Baptiste Rozière. Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
11

---

[10] DeepSeek-AL, Xiao Bi. Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng. Honghui Ding, Kai Dong. Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, panpan Huang. Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, A. X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nic, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Junxiao Song. Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Minghui Tang. Bingxuan Wang. Peiyi Wang, Shiyu Wang. Yaohui Wang. Yongji Wang, Tong Wu, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yiliang Xiong, Hanwei Xu, R. X. Xu, Yanhong Xu, Dejian Yang. Yuxiang You, Shuiping Yu, Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang. Minghua Zhang, Wentao Zhang. Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou. Deepseek Ilm: Scaling open-source language models with longtermism. 2024.
[11] D. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR 15), 2015. Published online: iclr.
[12] Roman Vershynin. Hiig--Dimensionll Probability: An Introduction with Applications in Data Science. Cambridge University Press. edition, 2018.
httpa://wuw math. uci edu/ eversshn/pappes
HDP-book/Hi Boook//HDP-BRE book.
[13] Jörg K.H. Franke, Michael Hefenbrock. Gregor Koehler, and Frank Hutter. Improving deep learning optimization through constrained parameter regularization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[14] Daria Soboleva, Faisal Al-Khatech, Robert Myers, Jacob Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. /tttpa://cerebaat a version- of redpaj ama 2023.
/log/slimaajamaaeae 627b- token- cleaned- www.dddddddddaaaeeeeddd (ttLhhts:://// http::::///gggginngggggeg ace. co/ wwwaaattssttseeeeebaaeeeeee S1 imliimmaa. immaajamaaaa..........aaaaaaaa -mm26627
[15] Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leaby, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang. and Samuel Weinbach. GPT-NeoX- 20B: An open-source autoregressive language model. In Proceedings of the ACI. Workshop on Challenges & Perspectives in Creating large Language Models, 2022.
[16] Noam Shazeer. Glu variants improve transformer, 2020.
[17] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021.
[18] Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query- key normal- ization for transformers. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4246-42533, Online, November 2020. Association for Computational Linguisties. doi: 10. 1653/v112020 10000////0222...mnnnnnmmmmmmmmmm findings-emnll.799
[19] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Representations (ICLR') 19), 2019.
[20] 1. Loshehilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In Proceedings of the International Conference on Learning Representationss (ICLR' 17), 2017. Published online: iclr..c
[21] Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrep- ancies in compute-optimal scaling of language models. In 2nd Workshop on Advancing Neural Network Training: Compitational Ffficiency, Scalability, and Resource Optimization ( (WANT@ ICMI. 2024), 2024.
[22] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy. Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. Training compute-optimal large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 30016-30030. Curran Associates, Inc., 2022.
[23] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji. Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma. Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu. Peng Wang. Shijie Wang. Wei Wang. Shengguang Wu, Benfeng Xu. Jin Xu, An Yang. Hao Yang. Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang,
12

---

Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023.
[24] Aaron Gokaslan and Vanya Cohen. Openwehtext corpus.
http://Skylionoo0 . github. io/
OpenWebTextCorpus 2019.
[25] Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. Rezero is all you need: fast convergence at large depth. In Cassio de Campos and Marloes H. Maathuis, editors, Proceedings the
Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 1352 1361. PMLR, 27-30 Jul 2021.
[26] Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Residual learning without normalization via better initialization. In International Conference on Learning Reprexentations, 2019.
[27] Liyuan Liu. Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. In Bonnie Webber, Trevor Coha, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5747-5763, Online, November 2020. Association for Computational Linguisties.
[28] Andrew Brock, Soham De, and Samuel L Smith. Characterizing signal propagation to close the performance gap in unnormalized resnets. In International Conference Learning Representations, 2021.
[29] Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In International conference machine learning, pages 1059-1071. PMLR, 2021.
[30] Bobby He and Thomas Hofmann. Simplifying transformer blocks. In The Twelfth International Conference on Learning Representations, 2024.
[31] Stefan Heimersheim. You remove GPT2's layernorm by fine-tuning. In Second NeurlPS Workshop on Attributing Model Behavior at Scale, 2025.
[32] Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. Transformers without normalization. arXiv preprint arXiv:2503.1002: 2025.
[33] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory- efficient exact attention with 1O-awareness. Advances in Neural Information Processing Systems. 2022.
[34] A. Paszke. Gross, F. Massa, A. Lerer, el PyTorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alche Buc, E. Fox, and R. Garnett, editors, Proceedings of the 32nd International Conference on Advances in Neural Information Processing Systems (NeurlPS' 19), pages 8024 2019, 2019.
[35] J. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 93))90-95, 2007.
[36] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
[37] Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katic E Everett, Alexander A Alemi, Ben Adlam, John D Co- Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Dock-Dicctt,in, Kelvin Xu, Jachoon Lee, Justin Gilmer, and Simon Korbliihh. Small-scale proxies for large-scal transformer training instabilities. In The Twelfth International Conference on Learning Representations, 2024.
[38] Ilya Loshchilov. Github repository "ngpt: Normalized transformer with representation learning on the hypersphere", 2025. URLptttee :tttppa:/// :/gi thub. com/t Accessed: 2025-05-14.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction state that the paper's contribution is a "Approxi- mately Normalized Transformer" with faster convergence and fewer hyperparameters at a minor increase in training time. These claims are consistent with the experimental results presented in the paper. Further, the abstract and introduction claim to motivate normalization, which is consistent with the Preliminaries Section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations and provide critical reflection of our experimental
setup in Section 6 Regarding the computational efficiency, we report the runtime of our approach in Table
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattio, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
14

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Theorem 1 about the concentration of Lipschitz functions on sphere is not our result but was taken from 12], we only use it to justify our approach. All assumptions used to derive the normalizing factors stated in Section 4.2
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification:
The paper describes the architecture Section 4.3 and the experimental
setup in Section[5] It provides detailed hyperparameters in Appendix [F] including model architectures, optimization settings, and training configurations. The used datasets are publicly available. The supplemental material contains the source code of all described architectures.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The used datasets are publicly available. An open-source implementation of anGPT is available at https: https:///ithbbb coom/automm/antaa
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
public//uiide/CodeSuteee www.aabmissoooooooooooo onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips. chajps.cc/pullic/uiiiee gwwp///ieee//eee/ Codeeubmii ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The paper describes the architecture Section 433 and the experimental setup in Section[5] It provides detailed hyperparameters in Appendix including model architectures, optimization settings, and training configurations. The used datasets are publicly available (c.g.. Huggingface). The supplemental material contains the source code of all described architectures, including the training pipeline.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
16

---

Justification:
The experimental results are presented without error bars or statistical sig- nificance tests. The performance comparisons show single runs rather than averages over multiple seeds. This is also mentioned in the limitations, Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: We provide details about the used hardware and the total compute budget in Appendix F but not for each single experiment. Since we derived scaling laws, we performed a huge variety of different-siin sizeee experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neuurpp::
Answer: [Yes]
Justification: The research focuses on technical improvements to transformer architectures without raising ethical concerns. It doesn't involve sensitive data, harmful applications, or privacy issues.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
17

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
This work is foundational research on transformer architecture optimization, not tied to specific applications. Our contribution is a technical improvement to training efficiency rather than introducing new capabilities with direct societal implications.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairesss considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper focuses on a training methodology rather than releasing high-risk models datasets, so safeguards are not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
18

---

Answer: [Yes]
Justification: The paper properly references existing datasets and software in Appendix with appropriate citations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper introduces new model architecture rather than datasets other assets, so this question isn't applicable.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or human subjects research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
19

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve human subjects research, IRB approval is not applicable.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
The paper develops a training methodology for transformers but does not use LLMs component in the research process itself.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non sta-standard components.
Please refer to our LLM policy (https http::://neurpppet cc/Conferrecess/20000 /LLM) for what should or should not be described.
20

---

Appendix
A
Related Work
Previous approaches have addressed the "Curse of Depth" [6] through depth-specific scaling [25]. initialization strategies [26]. hybrid normalization schemes [27]. Sun et al. introduced constant scaling factor depending on the layer number after the pre-normalization layer. This reduces the effect of increasing variance on the residual, but in contrast to our approach, does not aim to eliminate the effect. In this work, we propose comprehensive solution that normalizes the entire representation at each layer, ensuring all network components contribute effectively regardless of depth.
Franke et al. [3] introduces Constrained Parameter Regularization which bounds statistical measure, like the L2-norm, of learnable parameters using an augmented Lagrangian optimization instead of applying weight decay. Therefore, they introduce multiple initialization methods to find the right norm value. In contrast, our work bound all parameter matrices to one, and we also approximately normalized the representation space.
Multiple works proposed methods or architecture changes to remove or replace the normalization [28 31. Most recently, Zhu et al. (32) proposed replace the normalization layer with Dynamic Tanh layer (DyT) based on the observation that normalization produces S-shaped input-output mappings. DyT consists of tanh function with input scaling scalar and output scaling vector. In contrast our approach, DyT' does not change the representation or weight space and only aims for an improved runtime.
Comparison related work
We compare our anGPT approach to the constant scaling factor (6] and the dynamic tanh (DyT) replacement of the layer normalization (3). We train both 0.5B GPT setting 10B SlimPajama tokens (Chinchilla optimal) using the same configuration as for GPT+. However, we tune the learning rate for both approaches. We apply the additional learnable scaling factor for DyT as proposed and Og initialization of 1.0 according to Table 12 and show the results Figure A.I
32
We include in the comparison nGPT and GPT+
We see a strong performance drop using DYT. When using LN scaling, we find the results on par with GPT+ without scaling, nGPt outperforms GPT+ slightly, and anGPT shows the best performance in this setting.
Model
Style Improv.
Validation Loss
3.0
GPT+
GPT+ without QK norm
-2.5%
GPT+ with LN scaling
0.0%
2.8
GPT+ with DyT
-6.5%
nGPT
0.2%
anGPT
1.3%
5.0
Training Tokens (B)
Figure A.1: Comparison of GPT+, nGPT. and anGPT to the LN scaling 6 and DyT normalization B2 in 0.5B GPT training on 10B SlimPajama token budget. We use the same configuration as for GPT+ and tune the learning rate.
21

---

B Variance on the Adam momentum
To understand the effects of our normalization on training with Adam, we analyze the (relative) variance of the momentum vector m. The relative variance was computed by dividing each step of the timeseries ((m)) by Ei=1 ((m))
We can see that for GPT+, warm-up allows the (relative) variances of the momentum terms become small before the main training starts at 2000 steps, see Figure B. Without warm-up. see Figure B.2 we observe stronger peaks in the relative variance of the momentum vector, which likely destabilizes training. It can also be seen that the (relative) variance of the momentum vector for anGPT starts mostly at zero for all parameters and develops gradually with little deviation between parameters.
W gkv relative momentum variance
W relative momentum variance
0.15
0.15
End
up
GPT-
End of warm up
GPT-
Relative variance
anGPT
Relative variance
anGPT
0.10
0.10
0.05
0.00
0.00
5000
10000
15000
20000
5000
10000
15000
20000
Training step
Training step
Was relative momentum variance
Wa relative momentum variance
0.15
0.15
End warm up
End of warm up
GPT-
GPT-
Relative variance
anGPT
Relative variance
anGPT
0.10
0.10
0.05
0.00
0.00
5000
10000
15000
20000
5000
10000
15000
20000
Training step
Training step
Figure B.1: The Variance of the Adam momentum for the Transformer parameter groups during training. The traces correspond to individual layers. Each subplot shows the per-step variance of Adam's first moment estimates for one weight matrix, relative to each layer's total variance over all training steps. Despite learning-rate warm-up, GPT+ shows variance spike in all four parameter groups at the start the training.
22

---

W Wyko relative momentum variance
W relative momentum variance
0.15
0.15
GPT-
GPT-
Relative variance
anGPT
Relative variance
anGPT
0.10
0.10
0.05
0.05
5000
10000
15000
20000
0
5000
10000
15000
20000
Training step
Training step
W relative momentum variance
Wa relative momentum variance
0.15
0.15
GPT-
GPT-
Relative variance
anGPT
Relative variance
anGPT
0.10
0.10
0.05
0.05
5000
10000
15000
20000
0
5000
10000
15000
20000
Training step
Training step
Figure B.2: The Variance of the Adam momentum for the Transformer parameter groups during training. The traces correspond to individual layers. Each subplot shows the per-step variance of Adam's first moment estimates for one weight matrix, relative each layer's total variance over all training steps. GPT+ is trained without a learning rate warm-up.
23

---

C RMSNorm y-norms and block-input norms in GPT+
GPT+ RMSNorm ? parameter
50
Attn RMSNorm
MLP RMSNoru ?
40
L.2 Norm
30
20
GPT+ layer input representation
800
600
1,2 Norm
400
200
Layer index
The RMSNorm y norm and layer input norm for each layer in GPT+ after the final training. We see the growing norm on the residual and a slightly correlated growth of the norm in the ? parameter.
Figure C.1:
24

---

D Normalizing the Attention Matrix
To apply the normalization for the linear map to Av, we require the rows a, of A be normalized. Assuming the entries aij of A E R*x* are identically distributed, and the softmax computation is applied along the rows A, know that Σ, aij 1. By symmetry (identical distribution), we know that E(ay] = 1/8. For the rows a, to be normalized, is required that |a,ll =3
i.e. E[asy] 1/v5. We therefore scale the attention matrix by 8/v5 vs. If mask for causal attention is used, we know the number of zero entries in each row. Thus, the expected value of the nonzero entries in the 7-th row is E(a,)] 1/r, so the r-th row is scaled by r/vr= VF.
Note that the normalization factors are likely less useful for attention as the expected values effectively model dense attention scores, while sparse attention is usually more realistic. Hence, calculating useful normalization factors for the attention matrix likely requires information about the number of effectively nonzero attention scores. Unfortunately, this information is often not directly acces- sible due the use of Flash normalization may be seen as directions for future work.
33 Approaches to leverage this information for effective
E Explicit Normalization Factors
We derive the normalization factors using Vg 1/ / ( for linear maps and Monte Carlo estimation for activation functions. The specific factors are:
Query, Key, Value projections: Vyku /d/h for akk gaxd
Output projection: Up = 1 for W, RRd
Up projection: Vus ddf for Wus R/xd
Activation function: Veef 3.74 (via Monte Carlo with 10 samples)
Down projection: Vaa = / f/d for Wa E Ødx
Residual interpolation: v(a) = 1/ / + az)
where d is the model dimension, the head dimension (64 our experiments), and f is the feed-forward dimension (typically 4d).
25

---

Training Details
In our experiments. we use model sizes from 32M up 1B parameters and list the architecture hyperparameters in Table F.T For the optimization, we use Adam and Adam W and list the corre- sponding training hyperparameters
Table F.2 We used two datasets in this paper, SlimPajama (Apache 2.0 license) [14] and OpenWebText (Creative Commons Zero v1.0) [24 SlimPajama provides validation set, and for Teb we used 10k randomly selected documents as validation set.
We implemented our experiments in PyTorch 2.6 34 and used Flash Attention 0.7.3 33
All plots
are generated with Matplotlib [35]. We performed all experiments on research cluster with 4x A100 40GB GPU nodes and used total about 30k GPU hours. The smallest experiments are around GPU hour, and the largest are up to 750 GPU hours, An open-source implementation of anGPT is available at https::/ //gh::////ttubb..../ com/ automm/ltaaaa anGPT
Table F.I: The different GPT-style language model architectures used in this work and the total parameter counts. All models use vocabulary size of 50,304 tokens. The difference between GPT+ and anGPT accrues due to the difference in head scaling size (vocabulary size instead of model dimension). The difference between GPT+ and nGPT comes from additional scaling vectors in each MLP (2 x dMLP) and MHA (2 x dmodsl) layer.
Model
32M
62M
125M
250M
0.5B
1B
Model Dimension (dmodel)
256
384
512
768
1024
1280
Number of Layers (Mayers)
10
18
18
24
36
Number of Attn. Heads (Theads)
8
12
16
20
Head Dim. (da- dmadee Theads)
64
64
64
64
64
64
MLP Dim. (dMLP- = x dmodet
1024
1536
2048
3072
4096
5120
Parameters in GPT+
32.05M
62.24M
127.03M
247. 147.17M
505.73M
1.073B
Parameters nGPT
32.11M
62.33M
127.16M
247.34M
506.00M
1.073B
Parameters in anGPT
32.10M
62.28M
127.08M
247.21M
505.78M
1.073B
Table F.2: not other specified, we used the following hyperparameters of the GPT+, nGPT, and anGPT training runs in the experiment section.
Parameter
GPT+
nGPT/anGPT
Gradient Clip Val
1.0 bf16-mixed
Precision
Optimizer
AdamW
Adam
Betal
0.9
Beta2
0.95
Eps
1.0 x 10-9
Weigth decay
0.1 20%
Lr Num warm-up Steps
0
Lr Decay Factor
0.01
Lr Schedule
Cosine
Param. Scale Init
1/v.m / 0.001
Dropout
Rotary Pos Embed
True 0.5
Rotary Emb Fraction
Use Bias
False True
Flash Attention
Torch Compile
True
Context size
2048
'Both datasets are accessible Huggingface: http:://wuuginggeooo.//atttsssss / SlimPajann amaa-aa-ee 627B and http://///hhhago..oooooo///aaaaa http://hwwggggaoooooo/////oottaa
26

---

G Fitting hyperparameter scaling trends
To investigate the optimal hyperparameters and scaling trends of the new anGPT architecture, we performed grid search on different scales and used the extrapolated optimal configuration to train multiple models on different token budgets. We orient our procedure on Porian et al. [21] work investigating the discrepancies in compute-optimal scaling of language models between Kaplan et al.
36) and Hoffmann et [22].
For the grid search, we performed training runs with GPT+ and anGPT on the SlimPajama dataset with different model scales from 32M to 0.5 parameters. We train Chinchilla optimal token budget of x0x the number of training parameters [36]. Similar to Porian et al. [21]. we used an Adam 32 parameter of 0.99 for experiments below 100M parameters and 0.95 above. We performed least three experiments per scale and batch size with different learning rates, so that the best configuration always in the middle of the parameter grid. Our raw results of the hyperparameter sweeps can be found in Figure G.1
Estimating the optimal batch size and learning rate via interpolation
Our interpolation method for estimating the optimal batch size and learning rate closely follows the two-stage procedure proposed by Porian et al. [21]. In their method in the first stage, for each model size and batch size, the optimal learning rate was identified by performing Akima interpolation (in log-space) on the loss as a function of the learning rate, taking the lowest loss among three tested values of the hyperparameter 32, and subsequently identifying the minimizing argument. In the second stage, they applied interpolation again, this time over batch sizes, using the previously interpolated minimal losses to pinpoint an optimal batch size. The final optimal learning rate for the identified batch size was obtained by interpolating the sequence of (batch size, minimizing learning rate) pairs and evaluating this interpolant at the determined optimal batch size.
Our approach mirrors this two-stage interpolation methodology but differs by utilizing pre-selected and fixed 32 values, thus avoiding the need for optimization over B2. As discussed in the previous section, we use an Adam 32 parameter of 0.99 for experiments below 100M parameters and 0.95 above.
27

---

Batch size
16
Batch size
32
Batch size 6d
Batch size 128
4.25
4.25
4.25
4.25
Finel Validation Loss
4.00
4.00
4.00
4,00
3.75
3.75
3.75
3.75
3.50
3.25
3.25
3.25
3.25
3.00
3.00
3,00
3.00
2.50
2,50
2,50
2.50
1.0e403
2.3e-00
4.6c-10
1.ss00
2.2x-22
2.3--1
4.6e-00
100k.22
2.20-122
1.0s00
2.be00
4.6-01
Like42
2.30402
1.te-00
2.36-00
4.64-33
1.0e-22
2.0e-f2
Batch size 236
Batch size = 512
Batch size 1021
4.25
4.25
4.25
Model size
N 32M
Final Validation Loss
4.00
4.00
4,00
N 62M
3.70
3.75
3.76
127M
N 247M
3.50
3.50
3.50
N 506M
3.25
3.25
3.25
Model type
3.00
3.00
3,00
anGPT
2.75
2.75
GPT+
2,50
2,50
2.50
1.0e411
2.2e-00
1.0e-22
2.202
10e.gg
2:2e41
1.6e4g
02
4 Aelig
Learning rate
Learning rate
Learning rate
Figure G.1: Hyperparameter sweep results: The validation loss after 20 N training steps as function of the learning rate for different model sizes N and batch sizes. Plot design inspired by Wortsman al. (37. The stars with a black outline indicate the interpolated minimum learning rate as used in the first stage of the estimation of the optimal batch size and learning rate.
28

---

H
Comparison to nGPT
Since we compare our work with nGPT [5] we performed sanity check with our code base and compared the results of the official nGPT implementation [38]. We reconstructed the values of the reported model performance across different learning rates from the repository in Figure H.T and added our results. We find the final validation loss values for nGPT and GPT without QK norm match the reported numbers. We also found anGPT has slightly lower final validation loss (0.6%) than nGPT, but, maybe more importantly, we also found that GPT with QK norm increases the GPT baseline performance substantially (2.6% lower final validation loss).
0.5B model with 1K context on Web
3.00
rep. GPT
rep. TGPT
2.95
our GPT+
GPT+ noQK our nGPT anGPT
Final validation loss
2.85
2.80
2.75
2.70
2.60
2 i0-3
3 x io-3 x io-3
x io-s'
10
10
Initial learning rate
Figure H.1: We reconstructed the Figure from the nGPT Github repository and compared the results of our implementation with the official reported loss values (gray) from experiments with 0.5B parameter model, 512 batch size, Ik context size, and on 5B OpenWebText tokens / 10k training steps.
This finding is in line with our experiment on SlimPajama, shown in Figure H.2
We find higher convergence speedup when compared to GPT+ without QK norm. However, there is still gap to the reported speedup factors of xx up to 20x by Loshchilov al. (5). We hypothesize this could due to different training data, training budget, hyperparameters, and/or the codebase.
In contrast to nGPT, we use a larger training corpus perform LLM pretraining experiments without training multiple epochs. While Loshchilov et al. [5] used WebTexx (24] with ~9/ tokens (32k tokenizer) and trained for multiple epochs (up 50 epochs), we use SlimPajama [E with -627B tokens (50k tokenizer) and train for < 1 epoch. Also, we trained only up to x7 Chinchilla optimal [22] token budgets while nGPT was trained on up to x Chinchilla optimal token budgets. Furthermore, we tuned the batch size and found smaller batch size for GPT+ slightly better than for anGPT. We trained only on a context length of 2k tokens (since the average document length in SlimPajama is only 1K tokens) while nGPT was trained up to 8k tokens context size. Lastly, the publicly available codebase on GitHub is different from the one used in the paper, as the author explains in GitHub Issue 6 (38 Nevertheless, the nGPT codebase was very helpful for our work.
29

---

Model size 0.5B trained on 5-70B Slim SlimPajama tokens
Final Validation Loss
2.8
GPTT w/o QK
2.7
mGPPT
2.6
2.5
20
30
80
70
Convergonce Speelup
GPT+ w/o QK norm
150% 100%
No Refurence Data
0%
10
20
90
50
60
70
Tokens Budget (B)
Figure H.2: Training the 0.5B model up to 7x Chinchilla optimal token budget. Each point is a full training with the training budget noted on the abscissa. The plot below shows the convergence speed up against the GPT+ model without QK normalization.
30

---

I Convergence Analysis Across Model Scales
Figure [.I shows convergence comparisons for 250M and IB parameter models, demonstrating that the 40% speedup observed for 0.5B models (Figure 3 in main paper) is consistent across scales.
Model size 250M trained on 2-27B SlimPajaaaa tokens
3.2
Final Valiclation Loss
GPT+
inGPT
3.0
10
25
Convergencee
Speeduu
8096
60%C
40%6
Tokens Budget (B)
(a)
Model size 1.0B trained 10-42B SlimPaajun tokens
Final Validlaion Loss
GPT+
2.6
anGPT
2.5
2.4
20
25
35
40
Convergence
Speedup
50%
300
0%
10
15
20
25
30
35
40
Tokens Budgel (B)
(b)
Figure I.1: Convergence comparison for (a) 250M and (b) IB parameter models. Similar the 0.5B results, anGPT achieves the same loss with significantly fewer iterations, demonstrating consistent 40% speedup across model scales.
31

---

J Derive scaling laws
For each combination of compute scale C and architecture, select point with the minimal validation loss. We use the approach described in Hoffmann et al. [22] we bin compute into 1500 FLOPs logarithmically spaced intervals and for each bin we obtain point with the minimal loss. We obtain a mapping from each combination of parameters and number of tokens to the compute C.
Following Hoffmann et al. [22], (36] we assume that the validation loss L(C) proportional to C-a
(a 0) up to some positive constant Ao-
From the Figure 4 we see that GPT+ has slightly higher coefficient Ag. which means starts off with higher loss than anGPT. Both GPT+ and anGPT have nearly identical estimated scaling law parameters, which implies that improvement of validation loss comparable across these architectures.
We follow [22 and model compute optimal number of tokens Nopt (C) and Dopt (C) power laws. From the set of point {N, D} we select such D and N that correspond to minimal validation loss:
Dopt Nopt == arg min L(C)
To estimate 95% confidence intervals for model predictions, we propagate the uncertainty from the fitted parameters through the model. This involves computing the Jacobian matrix J of the model output with respect the parameters, evaluated at the extrapolated inputs. The variance of the predicted values is then approximated using the delta method as
o² J Cov J.
The resulting confidence intervals are given by
① + ///2.
where 1a/2, the critical value from the Student's I-distribution at a significance level of 0.05. We fit power functions through the obtained points estimate parameters for (C) and Nope (C). The obtained power law fits are shown in Figure J.1 and the estimated parameters are Table [.1
Model
Do
aD
No
GPT+
13.408079
0.466271
0.336617
0.465600
anGPT 9.727337
0.473190
0.650506
0.451378
Table J.1: Estimated scaling law parameters for compute-optimal dataset size for (C) and compute-optimal model size (C). The table reports fitted values for both anGPT and GPT+ We observe that both models exhibit very similar exponents, which suggests comparable trends in how optimal dataset and model size should be selected under certain compute.
32

---

IPT 0.3-x4465
GPT 13.4
Number of trainiing tokeos
Number of paramelers
Compute (FLOPs]
Compute C (FLOPS]
(a) Compute optimal number parameters (C) (b) Compute optimal dataset Size Dopc(C)
Figure J.1: Scaling laws for compute-optimal model size and dataset size. a) shows the estimation for compute optimal number of parameters Nopr (C) and b) shows compute-optimal dataset size Dopt(C), both as function of compute C. We observe that both anGPT and GPT+ have similar scaling trends for compute-optimal allocations. Notably, both fits exhibit high uncertainty, which is indicated by the wide confidence intervals surrounding the curves.
33

---

K Downstream Task Evaluation
We evaluate our models on standard benchmarks verify that pretraining improvements transfer to downstream tasks. Table K.1 shows results for IB models across different training budgets.
Table K.I: Downstream evaluation on 1B models. Higher is better for all metrics except perplexity (PPL).
Training Budget 21B Tokens
Metric
Model
10B Tokens
42B Tokens
GPT+
0.669
0.674
0.700
PIQA (Acc)
anGPT
0.680
0.702
0.718
GPT+
0.508
0.532
0.561
ARC-Easy (Acc)
anGPT
0.539
0.561
0.596
GPT+
0.338
0.363
0,400
HellaSwag (Acc)
anGPT
0.354
0.386
0.413
GPT+
36.915
24.932
16.726
LAMBADA (PPL)
anGPT
28.780
17.577
14.679
GPT+
22.648
21.661
17.621
WikiText (PPL)
anGPT
20,518
17.734
16.175
GPT+
0,522
0.536
0.554
WinoGrande (Acc)
anGPT
0.530
0,559
0,563
GPT+
0.232
0.239
0.243
MMLU (Acc)
anGPT
0.232
0.241
0.256
anGPT consistently outperforms GPT+ across benchmarks, with particularly strong improvements in crplexity-based metrics (LAMBADA, WikiText) and reasoning tasks (ARC-Easy, HellaSwag). The performance gains correlate well with the validation loss improvements observed during pretraining.
34

---

L
Ablation Studies
We conduct comprehensive ablation studies investigate the sensitivity of anGPT variations in normalization factors. Table [._ shows the impact of different modifications to the normalization scheme on 0.5B model trained on 10B tokens.
Table L.I: Ablation study normalization factors. We report relative change to anGPT in parentheses (%).
Configuration
Validation Loss
PPL
GPT+
2.677 (+1.46%)
14.541 (+3.94%)
anGPT (baseline)
2.638
13.990
anGPT (const. norm factor x0.5)
2.647 (+0.34%)
14.116 (+0.90%%
anGPT (const. norm factor x2.0)
2.642 12%)
14.033 (+0.31 %)
anGPT (all norm factors x0.5)
7.935 (+200%)
2792.69 (+19k%)
anGPT (all norm factors x2.0)
NaN
NaN
anGPT (no const. norm factors)
2.667 (+1.09%)
14.400 (+2.93%)
anGPT (no residual norm factor)
2.719 (43.04%)
15.156 (+8.33%)
anGPT (no norm factors)
2.718 (+3.02%)
15.148 (+8.28%)
anGPT (no LERP)
2.688 (+1.89%)
14.700 (+5.08%)
anGPT (half token budget)
2.755 (+4.42%)
15.718 (+12.36%)
anGPT (double token budget)
2.565 (-2.78%)
13,000 (-7.07%)
These results demonstrate that while the normalization factors are sensitive to large perturbations (50% or 200% scaling), the method remains robust to smaller estimation errors. The residual normalization factors are most critical, as their manipulation can cause training collapse. The 1.09% performance drop from removing constant normalization factors is significant when compared the performance gap between anGPT and GPT+ (1.46%).
35

---

M Evolution of Learnable Parameters
We track the evolution of learnable interpolation parameters a (attention) and av (MLP) throughout training. Figure M.1 M.1 shows these parameters across different layers and model sizes. The increase in a values demonstrates that the model learns to increasingly incorporate new features from each block.
Atticttonn
Alpha
Sape
0.20
0.00
Save
0.25
Lape
0.25
0.00
Lape
0.00
0.25
0.00
Laps
0.20
0.00
Taper
0.55
Tapee
0.50
0.00
Tape
Aljho Moon Beadel
Leper
Lxper 14
0.25
0.00
0.50
Taye 17
0.50
0.00
Taye
0.55
0.00
Lepes
0.26
0.00
Lupe 21
0.25
0.00
0.00
0.50
2000
1003
7700
10:00
19500
15000
17500
20000
2500
3000
7500
11003
1950
15000
17500
30000
Trining Stage
Tistting Stopo
Figure M.1: Evolution of interpolation parameters during training of a 0.5B parameter model.
36

---

Empirical Validation of Norm Concentration
To verify that approximate normalization maintains stable norms throughout the network, measure the ratio of mean input norm to mean output norm for each block. Table N.1 shows these ratios at different layers.
Table N. Ratio of mean output norm to mean input norm for transformer blocks. Values close to 1.0 indicate stable norm propagation.
Layer
anGPT
GPT+
First attention
1.07
0.01
First MIP
1.86
1.60
Middle attention
0.86
2.47
Middle MLP
0.97
2.60
Last attention
0.92
4.62
Last MLP
1.22
5.83
anGPT maintains near-unity ratios (0.86-1.88 1.86) while GPT+ exhibits significant norm growth in deeper layers (up to 5., demonstrating the effectiveness of upproximate normalization in stabilizing
gradient flow. We measure the error introduced by using = 1/ E | as approximation:
Table N.2: Estimation error normalization factors
Factor
Relative Error
Vake 1.16%
Up
0.07%
Vuz
0.02%
Va
0.08%
Voef 0.41%
v(a))
0.00%
All errors below 2%, well within the robustness margins demonstrated in our sensitivity analysis.
37

---

Robustness to Distribution Shift
We evaluate norm stability under out-of-distribution inputs by comparing normal samples against pathological inputs (512 repetitions of the same token). This tests whether fixed normalization factors remain effective under extreme distribution shifts.
Table O.1: Mean residual norms under distribution shift (0.5B model)
Text Input
Token Same
Model
Mean
Std
Mean
Std
GPT+
1831.42
353.93
545.68
78.80
anGPT
2.48
0.18
1.65
0.15
While both architectures exhibit norm changes under distribution shift, anGPT maintains better stability with only 1.5x change compared to GPT+'s 3.4x change. The moderate norms and absence of catastrophic failure indicate robustness distribution shifts, supporting the generalizability of approach beyond standard training distributions.
38

---

P Norms of the parameter groups in anGPT
Each line represents the average norm of the input direction, i.c., rows in Wa æ.
anGPT W gku
anGPT W
1.1
Avg. Norm (Input Dim)
Avg. Norm (Input Dim)
1.0
0.9
0.9
0.8
0.8
5000
10000
15000
20000
5000
10000
15000
20000
Training step
Training step
anGPT Was
anGPT W₂
1.1
Avg. Norm (Input Dim)
Avg. Norm (Input Dim)
1.0
0.9
0.9
0.8
0.8
5000
10000
15000
20000
5000
10000
15000
20000
Training step
Training step
Figure P.I: The mean norm of dimension (input dimension) of anGPT parameter groups during training. The traces correspond to individual layers. We see that despite the bound, the norm stays close to one.
39


Reviews:
[{"review_id": "cfNsDy4h3y", "reviewer": "Reviewer_SY5g", "summary": "This paper proposes the approximately normalized Transformer (anTransformer), a novel architecture that introduces approximate normalization through fixed scalar multipliers, motivated by the concentration of measure in high dimensions. Instead of exact normalization, it constrains the norm of parameters and representations to reduce the need for components like weight decay and learning rate warm-up. Applied to a GPT-style model (anGPT), the proposed method achieves up to 40% faster convergence than GPT+ (with QK norm) and matches or outperforms nGPT, while incurring only 3% extra training cost and negligible inference overhead.", "questions": "- How sensitive is the method to the choice of fixed normalization factors? Would small misestimations hurt performance significantly?\n- Did the authors evaluate the method on any downstream tasks or instruction tuning settings?\n- Could the proposed approximate normalization be combined with other efficiency improvements, such as low-rank or quantized attention layers?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well-written and presents a compelling motivation for approximate normalization based on theoretical insights and empirical observations. It introduces a clean and practical design that removes the need for additional normalization layers while retaining stability and training efficiency. Experiments are thorough, covering scaling laws, ablations, and comparisons across multiple model sizes and compute budgets. The results show consistent convergence gains and demonstrate that anGPT can match or outperform both GPT+ and nGPT.\n\nHowever, some concerns remain. First, the assumptions behind norm concentration and fixed normalization factors are only partially validated in practice; more ablations or analysis of failure modes would strengthen the argument. Second, while convergence is improved, the validation loss differences in the scaling law experiments are relatively modest. Lastly, downstream task evaluations are missing, which makes it unclear whether the improvements transfer beyond pretraining loss.", "was_revised": true, "final_justification": "- Authors provided new downstream evaluations, which show consistent improvements over baselines.\n- Ablations demonstrate moderate robustness to normalization factor variation.\n- Theoretical motivations remain partially validated, and failure mode analysis is limited.\n- Evaluations focus on a single model size and selected benchmarks.\n- I maintain my borderline accept score, recognizing the practical promise while noting open questions around generalization and theoretical grounding.", "rebuttal": "Dear Reviewer SY5g,\n\nThank you for your review and feedback on our paper. We appreciate the positive assessment and the constructive feedback. We address our named weaknesses and answer your questions in the follow:\n\n> First, the assumptions behind norm concentration and fixed normalization factors are only partially validated in practice\n\nThe theory only serves as motivation to investigate the application of the norm concentration phenomenon empirically. \nAs we state this in Section 4.1 line 140: “While these assumptions may not hold during training, we may still observe concentration empirically, particularly due to the high dimensionality of representations”. This violation is partially by design,  as the weights in our case are not normalized but only bounded, see Sec. 4.2  Constraining Parameters. Bounding the norm of the parameters instead of normalizing led to better results empirically, see Fig. 5. \nAdditionally, one can see from Figure 1, that the intermediate representations are not truly normalized (but only approximately) reaching norms up to ~2.9. Nevertheless, we observe many of the beneficial properties associated with normalization such as a very moderate growth of the norm on the residual connection. \n\n> more ablations or analysis of failure modes would strengthen the argument. / How sensitive is the method to the choice of fixed normalization factors? Would small misestimations hurt performance significantly?\n\nWe agree with the reviewer that additional ablations and analysis would be beneficial. Therefore, we run multiple additional experiments, please find all results in Table 1 below.\n\nWhen multiplying the const. norm factors with 0.5x and 2x we find a validation loss increase of 0.34% and 0.12% respectively. If we apply the same multiplier to the residual normalization factor ($\\nu(\\alpha)$ the training collapses. This is caused by exploding or vanishing latent representation on the residual $h$. However, when we remove the constant normalization factors, we face a validation loss increase of 1.09%, if we remove all normalization factors (including on the residual) we see an increase of 3%, similar to removing only the normalization factor on the residual connection. \n\nTo put the validation loss improvement of 1.09% into perspective, by using GPT+ instead of anGPT the validation loss increases by 1.46%. When using a half / double token budget training anGPT it leads to an 4.42% increase / 2.78% decrease of the validation loss. So 1.09% display already a significant performance drop.\n\nSo the normalization factors are sensitive but even with a 0.5x or 2x it is better than a GPT+ model. \nWe expect small errors in the estimation of the normalization factors Jensen’s inequality (see response to Reviewer skhp to question 2.3) below 2% which is way below the tested 50% or 200 % and therefore we do not expect a significant negative impact of this error. \n\n\nIf we manipulate the residual, we collapse the training and the normalization factors on the residual are most important. Using our calculated normalization factors on all proposed positions leads to the best performance. \n\nRegarding the failure modes, we would like to note that without any assumptions, anGPT would simply be a transformer with post norm, rescaled activations, and a missing normalization before the head layer that is trained with bounded weights. The missing normalization layer before the head could be added as a failsafe at minimal cost. On the other hand, bounding the weights can be seen as an alternative to weight decay, see [2].\nConsequently, none of this would lead us to believe that catastrophic failures may occur.\n\nWe will add this clarification, the additional ablation experiments and the error analysis into the paper.\n\n\n> Second, while convergence is improved, the validation loss differences in the scaling law experiments are relatively modest. \n\nWhile in the submitted paper we only plot the convergence speedup for the 0.5B model we do have a similar convergence speedup (~40%) for the other model size including our largest trained model,with 1B parameters. We will add plots in the same style as Figure 3 for 1B and 247M into the appendix in the CRC. This consistent benefit is also reflected in the derived scaling laws of GPT+ and anGPT with a significantly smaller scale for anGPT, see Figure 4.  \n\nIn addition to the convergence speedup, we also eliminate the need for weight decay and learning rate warmup, and consequently the associated hyperparameter tuning effort.\n\n> Downstream task evaluations are missing, which makes it unclear whether the improvements transfer beyond pretraining loss. / Did the authors evaluate the method on any downstream tasks or instruction tuning settings?\n\nUpon your request, we performed downstream evaluations on all our largest trained models (1B) and find a performance improvement similar to the reduced validation loss (see Table 2 below). We outperform on six benchmarks and on all token budgets GPT+. Regarding the selection of the benchmarks, for some other benchmarks the models are around random guess or only slightly above (see MMLU), so we picked downstream tasks which give a signal at all. The weakness of the model is also the reason why we did not perform instruction tuning experiments.  \n\n>Could the proposed approximate normalization be combined with other efficiency improvements, such as low-rank or quantized attention layers?\n\nYes. The approximate normalization approach is orthogonal to low-rank or quantization attention. It could also be combined with other modifications such as linear attention or in hybrid architectures. We see the benefit of anGPT is the reduced computational overhead, which matters not only during training but also at inference of foundation models.\n\n\nThank you again for the thorough and constructive review. The feedback will help us improve both the current work and guide future research directions. We added more ablation studies, an analysis to the expected error of the normalization factor and down stream evaluations. \nPlease do not hesitate to ask further questions. In case we addressed your concerns, we would appreciate a reconsideration of your score. \n\nBest regards, the authors. \n\n____\n**Table 1 Additional Ablation**\n\nThis table contains additional ablation experiments on the normalization factor using a 0.5B model on 10B tokens. * We report the relative change to anGPT in (*%). \n\n|  | validation loss | PPL |\n| :---- |-----: | ----: |\n| GPT+       |   2.677 (1.46%)   | 14.541 (3.94%) |\n| anGPT*           |   2.638    | 13.990  |\n| anGPT (const. norm factor x0.5)   |  2.647 (0.34%)   | 14.116 (0.90%) |\n| anGPT (const. norm factor x2.0) |  2.642 (0.12%)   | 14.033 (0.31%) |\n| anGPT (all norm factor x0.5)   |  7.935 (200%)   | 2792.69 (19k %) |\n| anGPT (all norm factor x2.0) |  NaN   | NaN |\n| anGPT (no const. norm factor) |  2.667 (1.09%)   | 14.400 (2.93%) |\n| anGPT (no residual norm factor) |  2.719 (3.04%)   | 15.156 (8.33%) |\n| anGPT (no norm factor at all) |  2.718 (3.02%)   | 15.148 (8.28%) |\n| anGPT (no LERP) |  2.688 (1.89%)   | 14.700 (5.08%) |\n\nFor reference, we include the results on a half/double token budget. \n\n|  | validation loss | PPL |\t\n| :---- |-----: |----: |\n| GPT+ (half token budget) |  2.787\t(4.11%)   | 16.231 (11.62%) |\n| anGPT (half token budget) |  2.755 (4.42%)   | 15.718 (12.36%) |\n| GPT+ (double token budget) |  2.593 (-3.13%)   | 13.371 (-8.05%) |\n| anGPT (double token budget) |  2.565 (-2.78%)   | 13.000 (-7.07%) |\n\n\n____\n\n**Table 2 Downstream Evaluation**\n\nWe perform downstream evaluation on the 1B model size trained on token budgets from 10B to 42B.\n\n| PIQA Acc   |   10 B |   21 B |   42 B |\n|:-------|-----:|-----:|----:|\n| GPT+  |  0.669 |  0.674 |  0.700 |\n| anGPT    |  0.680 |  0.702 |  0.718 |\n\n\n\n| ARC‑Easy Acc   |   10 B |   21 B |   42 B |\n|:-------|-----:|-----:|----:|\n| GPT+   |  0.508 |  0.532 |  0.561 |\n| anGPT   |  0.539 |  0.561 |  0.596 |\n\n\n| HellaSwag Acc  |   10 B |   21 B |   42 B |\n|:-------|-----:|-----:|----:|\n| GPT+   |  0.338 |  0.363 |  0.400 |\n| anGPT    |  0.354 |  0.386 |  0.413 |\n\n\n\n| LAMBADA PPL   |   10 B |   21 B |   42 B |\n|:-------|-----:|-----:|----:|\n| GPT+        | 36.915 | 24.932 | 16.726 |\n| anGPT   | 28.780 | 17.577 | 14.679 |\n\n\n\n| WikiText word PPL   |   10 B |   21 B |   42 B |\n|:-------|-----:|-----:|----:|\n| GPT+    | 22.648 | 21.661 | 17.621 |\n| anGPT      | 20.518 | 17.734 | 16.175 |\n\n\n| WinoGrande Acc   |   10 B |   21 B |   42 B |\n|:-------|-----:|-----:|----:|\n| GPT+     |  0.522 |  0.536 |  0.554 |\n| anGPT    |  0.530 |  0.559 |  0.563 |\n\n\n| MMLU Acc   |   10 B |   21 B |   42 B |\n|:-------|-----:|-----:|----:|\n| GPT+                  |  0.232 |  0.239 |  0.243 |\n| anGPT                 |  0.232 |  0.241 |  0.256 |\n\n\n\n____\n\n[1] Loshchilov, Ilya, et al. \"ngpt: Normalized transformer with representation learning on the hypersphere.\" 13th International Conference on Learning Representations (ICLR 2025) .\n\n[2] Franke, Jörg, et al. \"Improving deep learning optimization through constrained parameter regularization.\" Advances in Neural Information Processing Systems 37 (NeurIPS 2024): 8984-9025."}, {"review_id": "chjXOcE5TH", "reviewer": "Reviewer_7NrH", "summary": "The paper introduces the Approximately Normalized Transformer (anGPT), a modification of the GPT architecture designed to stabilize training and accelerate convergence without relying on traditional normalization layers or extensive hyperparameter tuning. By using approximate normalization through input-independent scaling factors, anGPT mitigates the \"Curse of Depth,\" where deeper layers in vanilla GPT models experience growing input norms, leading to training instability. The approach eliminates the need for weight decay and learning rate warm-up, reducing hyperparameter complexity. Experimental results demonstrate that anGPT achieves up to a 40% convergence speedup compared to the vanilla GPT, with only a minimal (~3%) increase in training step runtime and potentially reduced inference times. The authors validate their approach on model sizes from 32M to 1B parameters and provide detailed hyperparameter scaling trends and compute-dependent scaling laws. \n\nContributions:\n\n1- Proposes the anGPT architecture, which uses approximate normalization to stabilize training and improve convergence speed.\n\n2- Eliminates the need for weight decay and learning rate warm-up, simplifying hyperparameter tuning.\n\n3- Demonstrates up to 40% faster convergence compared to vanilla GPT, with minimal runtime overhead.\n\n4- Provides comprehensive empirical analysis, including hyperparameter scaling trends and compute-optimal scaling laws.", "questions": "see Strengths And Weaknesses", "limitations": "Yes", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- Novel Approach to Normalization: The paper introduces a principled and lightweight alternative to full normalization layers using approximate scalar normalization factors derived from concentration of measure, which is both theoretically motivated and practically effective.\n\n- Improved Efficiency: anGPT achieves significant convergence speedups (up to 40%) with only ~3% added training cost and no increase in inference time, demonstrating clear practical benefits.\n\n- Simplified Training Pipeline: By eliminating the need for weight decay and learning rate warm-up, the method reduces hyperparameter tuning complexity, which is valuable in large-scale pretraining.\n\n- Strong Empirical Validation: The authors conduct thorough experiments across multiple model sizes and data scales, including compute-optimal scaling laws, showing consistent performance improvements.\n\n- Insightful Analysis: The work provides a clear theoretical explanation of the issues with norm growth in deep transformers (“Curse of Depth”) and demonstrates how approximate normalization addresses them.\n\nWeaknesses:\n\n- Limited Scope of Evaluation: The experiments are confined to GPT-style language models and two datasets. It remains unclear how the approach generalizes to other architectures (e.g., encoders) or modalities (e.g., vision, audio).\n\n- Approximation Validity Not Fully Explored: While the use of norm concentration is well-motivated, the paper does not deeply investigate cases where this assumption may break down (e.g., during early training or low-dimensional representations).\n\n- Complexity of Implementation: Despite simplifying training, the architectural modifications (LERP, ν-scaling factors) and need to derive dimension-specific constants may increase implementation burden for practitioners unfamiliar with the theory.", "was_revised": true, "final_justification": "Authors addressed my comments, I am keeping my score at 5", "rebuttal": "Dear eviewer 7NrH,\n\nThank you for your review and feedback on our paper. We deeply appreciate the positive assessment and address our named weaknesses in the follow:\n\n> Limited Scope of Evaluation: The experiments are confined to GPT-style language models and two datasets. It remains unclear how the approach generalizes to other architectures (e.g., encoders) or modalities (e.g., vision, audio).\n\nWe are aware of this limitation (see Section 6) and see this as part of future work. In this work we focused on proper evaluation for LLM pretraining and on deriving scaling laws to provide empirical evidence for the benefit of approximate normalization.\n\n> Approximation Validity Not Fully Explored: While the use of norm concentration is well-motivated, the paper does not deeply investigate cases where this assumption may break down (e.g., during early training or low-dimensional representations).\n\nThe assumptions break already in the early training, see Figure 1 in the middle, and are even violated by design since our weights are only bounded instead of normalized. \nWe state this in Section 4.1 line 140: “While these assumptions may not hold during training, we may still observe concentration empirically, particularly due to the high dimensionality of representations”.\n\nBut due to the post-norm operation and our approx. normalization factors, we observe many of the beneficial properties associated with normalization such as a very moderate growth of the norm on the residual connection (Fig. 1). We found an increased convergence speed of ~40% across all model dimensions from 256 to 1280. However we only plotted it for 0.5B since we trained this model size for longer. We will clarify that and add more plots like Figure 3 on different model scales to the appendix. \n\n\n> Complexity of Implementation: Despite simplifying training, the architectural modifications (LERP, ν-scaling factors) and need to derive dimension-specific constants may increase implementation burden for practitioners unfamiliar with the theory.\n\nWe acknowledge this tension between architectural modifications and ease of use. Our work focuses on large scale training where a training run could cost thousands of dollars. The anGPT approach could save significant cost and energy which may justify the additional implementation overhead. \n\nRegarding your confidence rating of 1 (\"educated guess\"), we would like to clarify any technical details that might help increase confidence in the assessment. The core mathematical foundation (concentration of measure) is well-established, and our empirical validation follows rigorous scaling law methodologies from recent literature.\n\nThank you again for the thorough and constructive review. Please mind, we also performed downstream evaluations and additional ablation experiments, see our response to Reviewer SY5g. Your feedback will help us improve both the current work and guide future research directions.\n\nBest regard, the authors."}, {"review_id": "2xPA6OdwRR", "reviewer": "Reviewer_skhp", "summary": "The paper introduces the approximately normalized Transformer (anGPT), a variant of the Transformer architecture that replaces standard normalization layers (e.g., LayerNorm) with fixed, pre-computed scaling factors. The central premise is that in high-dimensional spaces, the norms of random vectors tend to concentrate tightly around their expected values, a phenomenon known as concentration of measure. Leveraging this property, anGPT applies fixed scalar normalization factors throughout the network to approximate norm preservation, while also constraining the weight matrices to lie within a compact (bounded-norm) parameter space. This design eliminates the need for learnable normalization parameters, weight decay, and learning rate warm-up, resulting in faster convergence with minimal computational overhead.", "questions": "I appreciate the effort to simplify normalization in large Transformer models through a theoretically motivated framework. However, several mathematical and empirical claims in the paper warrant deeper scrutiny. Below are the specific questions intended to clarify the assumptions and implications of the proposed method.\n\n---\n1) The core approximation in anGPT assumes that for high-dimensional inputs \\( x \\), the output norm of a function \\( g(x) \\) concentrates:\n\\[\n\\|g(x)\\|_2 \\approx \\mathbb{E}[\\|g(x)\\|_2]\n\\]\nThis is motivated by classical concentration results (e.g., Vershynin's theorem for \\( x \\sim \\text{Unif}(\\mathbb{S}^{d-1}) \\)).\n\n- How valid is this assumption during training, where the input distribution evolves due to gradient updates and may be far from uniform or isotropic?\n- Can you provide empirical evidence of concentration in intermediate activations (e.g., histograms of \\( \\|g(x)\\|_2 \\))?\n---\n2) anGPT replaces exact normalization with a scalar factor \\( \\nu = 1 / \\sqrt{\\mathbb{E}[\\|x\\|^2]} \\), which is a biased approximation of \\( \\mathbb{E}[1/\\|x\\|] \\).\n\n- What is the theoretical or empirical impact of this bias on gradient magnitudes or effective learning rates?\n- Does this approximation systematically distort the optimization trajectory or loss surface?\n- Can you quantify the tradeoff between computational efficiency and scaling accuracy introduced by this simplification?\n\n---\n3) anGPT modifies the residual connection using a learnable interpolation:\n\\[\nh \\leftarrow (1 - \\alpha) h + \\alpha x\n\\]\nwith an additional normalization factor based on \\( \\nu(\\alpha) \\).\n\n- This changes the gradient propagation structure. Have you analyzed the Jacobians \\( \\partial h / \\partial x \\) and \\( \\partial h / \\partial h_{\\text{prev}} \\) to assess gradient flow?\n- For small \\( \\alpha \\), does the network risk under-utilizing new features, leading to a form of representational inertia?\n---\n\n4) The effectiveness of fixed normalization factors implicitly assumes consistent norm statistics across training and test distributions.\n\n- How does anGPT perform under domain shift, few-shot adaptation, or continual learning settings where norm distributions may vary?\n- Have you tested whether norm concentration persists in out-of-distribution scenarios?\n- Could fixed normalization factors cause feature collapse or instability in low-data or transfer regimes?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n1) By replacing LayerNorm with fixed scalar normalization factors, anGPT reduces the computational and memory overhead typically associated with normalization, especially during inference, without sacrificing stability.\n\n2) anGPT achieves up to 40% faster convergence compared to GPT models with QK normalization, while removing the need for learning rate warm-up and weight decay, thus simplifying hyperparameter tuning.\n# Weaknesses\n\nPlease refer to the detailed questions I provided below, which highlight potential concerns around:\n\n* The validity of norm concentration under non-uniform, evolving training distributions.\n* The bias introduced by using fixed normalization factors.\n* The effect of LERP residuals on gradient flow.\n* Unclear robustness to domain shifts and out-of-distribution data.", "was_revised": true, "final_justification": "Initially, I was concerned about the concentration of measure phenomena during training and the method’s robustness to distribution shifts. However, the authors have addressed my questions thoroughly. I now believe this is a valuable piece of research that contributes to accelerating the training and convergence of large language models.", "rebuttal": "Dear Reviewer skhp,\n\nThank you for your extensive review of our paper. We appreciate the good rating for quality, clarity, and originality. In the following we will respond to your detailed questions. \n\nBut first of all, we would like to comment on your summary by pointing out that the theory only serves to motivate the empirical investigation of the phenomenon, and, if it can help to gain some of the benefits that have been empirically observed when using normalizations, see nGPT[1]. Our goal is ultimately to improve transformers for language modelling similar to nGPT but using cheap computations. For this, we show extensive pretraining results and derive scaling rules for our proposed architecture. \nRegarding your named strength,\n\n> …anGPT reduces the computational and memory overhead typically associated with normalization…\n\nwe would like to point out that our method does not reduce memory overhead nor computational overhead in contrast to standard GPT architecture, but we do reduce overhead in contrast to nGPT [1], see Table 2. \n\n> 1.1 How valid is this assumption during training, where the input distribution evolves due to gradient updates and may be far from uniform or isotropic?\n\nAs stated in Section 4.1 line 140: “While these assumptions may not hold during training, we may still observe concentration empirically, particularly due to the high dimensionality of representations”. This can also be seen from Figure 1, where the norms of the residual state vectors do not display a norm of one.\n We even violate the assumptions directly by using bounded instead of normalized weights since we observed it to produce better results empirically (see Ablation study in Fig. 5 “nGPT with param bounding”). \n\n> 1.2 Can you provide empirical evidence of concentration in intermediate activations (e.g., histograms of ( |g(x)|_2 ))?\n\nThanks for this question, we missed to show this in our submission. Figure 1 shows only the residual norm including the post normalization. In addition, we can measure the norm change of a block (g(x) = MLP() or Attention() before the post norm). Since we can’t add diagrams to the rebuttal we report the ratio of the mean input norm to a block to the mean output norm of a block (for anGPT before the post norm and for GPT+ after the RMS Norm) on different layer (first, middle and last). \n\n| | anGPT | GPT+ |\n|:---|---:|---:|\n| first attn | 1.07 | 0.01 |\n| first mlp | 1.86 | 1.60 |\n| mid attn | 0.86 | 2.47 |\n| mid mlp | 0.97 | 2.60 |\n| last attn | 0.92 | 4.62 |\n| last mlp | 1.22 | 5.83 |\n\nWe see that the approximate normalization factors stabilize the norm of a block without explicit normalization by keeping the ratio close to 1. We will add this analysis including a histograms plot to the paper. \n\n> 2.1 What is the theoretical or empirical impact of this bias on gradient magnitudes or effective learning rates?\n\n\nSince we use the square root of estimates for the squared norm as scaling factors (for the linear maps and the interpolation on the residual connection) we can expect an overestimation of the norm due to Jensen's inequality (as stated in Sec. 4.2), as \n$\\sqrt{\\mathbb{E}[ \\Vert x \\Vert^2]} \\ge \\mathbb{E}[ \\Vert x \\Vert]$ \ndue to the concavity of the $\\sqrt{\\cdot}$. Consequently, we expect and underestimation of the inverse of $ 1/ \\mathbb{E}[ \\Vert x \\Vert ] $, which is again an underestimation of $\\mathbb{E}[1 / \\Vert x \\Vert]$ due the the convexity of $1/x$. \nOverall, we may expect an underestimation of the inverse norm which should result in norms larger than 1. We also added this to Section 4.2 for the derivation of the scaling rules. \nSince these scaling factors scale the weight matrix/activations, they also scale gradients in the same way. Hence, gradients may be scaled up based on the degree of underestimation (due to Jensen's inequality). \n\nWe also measured the empirical impact by performing additional ablation experiments, please find all results in Table 1 in the response to Reviewers SY5g. \nWhen we remove the constant normalization factors, we face a validation loss increase of 1.09%, if we remove all normalization factors (including on the residual) we see an increase of 3%, similar to removing only the normalization factor on the residual connection. If we replace all normalization factors by exact normalization ($1/||x||_2$) we get a slightly worse validation loss of +0.4% and an increased runtime of 30%. \n\nWe hypothesize that this slight loss increase is caused by having normalized, instead of bounded representations, in line to the observation in Fig. 5, where using bounded weights \ninstead of normalized weights improve performance. We add this analysis to the paper. \n\n\n\n\n> 2.2 Does this approximation systematically distort the optimization trajectory or loss surface?\n\nAs outlined in the quadratic toy example, normalization may distort the loss landscape for problems with axis-aligned distortions, in other words, if some parameters are more sensitive than others.\n\n> 2.3 Can you quantify the tradeoff between computational efficiency and scaling accuracy introduced by this simplification?\n\nWe calculated the error of computing $\\nu = 1 / \\sqrt{\\mathbb{E}[\\Vert x \\Vert^2]}$ (using MC estimation with 10^5 samples), which is a biased approximation of ($ \\mathbb{E}[1/\\Vert x\\Vert ]$ ): \n| | $abs(\\nu - \\nu_{MC})$ |\n|:--|--:|\n| $\\Delta \\nu_{qkv}$  | 0.047 (1.16%) |\n| $\\Delta \\nu_{p}$  | 0.001 (0.07%) |\n| $\\Delta \\nu_{u}$  | 0.000 (0.02%) |\n| $\\Delta \\nu_{d}$  | 0.002 (0.08%) |\n| $\\Delta \\nu_{acf}$  | 0.015 (0.41%) |\n| $\\Delta \\nu_{alpha}$  | 0.000 (0.00%) |\n\nBased on our sensitivity analysis in response to Reviewers SY5g, we do not expect any negative impact caused by these low error values.   \n\n> 3.1 This changes the gradient propagation structure. Have you analyzed the Jacobians to assess gradient flow?\n\nWe did not perform this analysis but adapted linear interpolation on the residual from nGPT [1]. While we think the idea is interesting, we are not exactly sure what we could learn from analyzing the Jacobians (assessing better from worse).\n\n> 3.2 For small ($\\alpha$ ), does the network risk under-utilizing new features, leading to a form of representational inertia?\n\nThe factors $\\alpha$ are learnable and were adopted from nGPT [1]. In general, our architecture can also be adapted to using standard residuals updates (See Section 4.2 for the respective scaling factors), but using LEARP performed better in our experiments.\nWe performed an ablation where we removed the LERP residual connection and find a validation loss increase of 1.89% (for reference, using GPT+ instead of anGPT increase validation loss of 1.46%). Please find the numbers in Table 1 in Reviewers SY5g rebuttal.  \n\nAdditionally, we logged the $\\alpha$ during training and see an increase from an initial value of 0.05 to 0.12 -0.25 till the end of the training across all layers and multiple model sizes. This could indicate that the model learns to increase $\\alpha$ since the block output is beneficial for the model performance. We argue that we do not face an under-utilizing or representational inertia due to the increased performance in contrast to GPT+. \nWe will add this experiment and analysis to the paper. \n\n> 4.1 How does anGPT perform under domain shift, few-shot adaptation, or continual learning settings where norm distributions may vary?\n\nThis work focuses on pretraining and the introduction of architecture and deriving scaling laws. The empirical investigation of cases like few-shot adaptation or continual learning is out of the scope of this work.\nGenerally, it is important to note that the distribution assumptions, such as uniform on a sphere, are required for the distribution of the *entries in the representation vector or weight matrixes, not between tokens*. The distributions of the tokens do not directly influence the normalization factors. We thus do not expect any out of the ordinary problems when faced with domain shift or continual learning.\n\n> 4.2 Have you tested whether norm concentration persists in out-of-distribution scenarios?\n\nSince the normalization factors do not use the data distribution in their derivation and are also not adapted throughout training, one may assume that there is no out of the ordinary impact when faced with out-of-distribution scenarios. \nNevertheless, we performed an experiment where we forward passed with a 0.5B model 1000 text samples(wikitext-2) in contrast to 1000 samples containing 512 times the same token and analyzed the mean norm of the residual connection. \n\n| | mean | std | \n|:---|---:|---:|\n| text input GPT+  | 1831.42 | 353.93 |\n| same token GPT+  | 545.68 | 78.80 |\n| text input anGPT | 2.48 | 0.18 |\n| same token anGPT | 1.65 | 0.15 |\n\nWe observe a clear change in the norm using an out-of-distribution input, however the change is for anGPT ($\\times 1.5$) smaller as for GPT+ ($\\times 3.4$). Notably, the residuals norms still stay moderate and there is no indication of a severe breakdown.\n\n> 4.3 Could fixed normalization factors cause feature collapse or instability in low-data or transfer regimes?\n\nSince the derivation of the normalization factors did not consider the data distribution, we do not expect any out of the ordinary impact of data-dependent artifacts coming from low-data distributions or transfer regimes.\n\nThank you again for the thorough and constructive review. Your questions and  feedback helped us improve our work. Also, we added more ablation experiments, an analysis to the expected error of the normalization factor and down stream evaluations (see response to Reviewer SY5g). Please do not hesitate to ask further questions. In case we addressed your concerns, we would appreciate a reconsideration of your score. \n\nBest regard, the authors. \n\n___\n\n\n[1] Loshchilov, Ilya, et al. \"ngpt: Normalized transformer with representation learning on the hypersphere.\" 13th International Conference on Learning Representations (ICLR 2025) ."}, {"review_id": "aNVVxnbzEa", "reviewer": "Reviewer_p2iW", "summary": "Normalization is crucial for the training stability of deep neural networks. Previous work nGPT [1], extensively use normalization layers to update all latent representations on a hypersphere, which eliminates the need for pre-training techniques such as warmup schedules and weight decay. However, this extensive normalization introduces significant computational overhead. Based on concentration measure in high-dimensional spaces, this paper proposes a more efficient approach: rescale each feature dimension with precomputed normalization factors, thereby avoiding weight normalization, which makes the entire transformer model \"approximately normalized\".\n\n[1] [nGPT: Normalized Transformer with Representation Learning on the Hypersphere](https://arxiv.org/abs/2410.01131)", "questions": "1. How do you ensure that core assumptions, such as the i.i.d. condition or features being uniformly distributed on the sphere, continue to hold after training?\n\n2. Could you address the additional comments in the weaknesses section?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The motivation and core idea of this paper are clear: nGPT accelerates LLM pre-training and eliminates the need for hyperparameters related to learning rate schedules and weight decay. Given that feature vector norms concentrate in high-dimensional spaces, pre-estimating these norms allows normalization steps to be avoided.\n2. The empirical study of the paper is comprehensive, featuring clear scaling trends and insightful ablation studies.\n\n**Weaknesses**\n\n1. The example regarding fast contraction in Section 3.1 is not clearly connected to the authors' arguments about global learning rate selection. It would be beneficial to either find a more directly relevant example or to clarify the existing connection.\n\n2. The paper lacks some important details: (a) What are the exact procedures used to derive each of the normalization factors? (a) In Section 3.2, why $||h_{\\leq L} || = \\sum_{l=1}^L ||h_l || $? (c) How precisely are the \"effective learning rates\" applied? Although readers might guess, explicit clarification from the authors would significantly improve this paper.\n\n3. While assumptions such as the i.i.d. condition clearly hold at initialization, the paper does not sufficiently explain why these assumptions remain valid throughout training.", "was_revised": true, "final_justification": "I would like to thank the authors for their detailed explanations. This work is valuable, and its main shortcomings pertain primarily to presentation, which can be easily addressed. I recommend that the authors present the derived normalizing factors in a dedicated table. Additionally, providing the exact effective learning rates explicitly in the appendix would enhance accessibility. Lastly, the paper would become even more convincing if framed as an strong empirical study motivated by loose theoretical insights rather than theory-driven with empirical support. While the theoretical rigor does not meet high standard, I agree that the proposed approach is practically useful.", "rebuttal": "Dear Reviewer p2iW,\n\nThank you for your review and feedback on our paper. We appreciate the positive assessment and the constructive feedback. We address our named weaknesses and answer your questions in the follow:\n\n> 1 The example regarding fast contraction in Section 3.1 is not clearly connected to the authors' arguments about global learning rate selection. \n\nFor our example problem $\\min_{x} \\frac{1}{2} x^\\top \\Lambda x$ with $\\Lambda = \\text{diag}(\\lambda_1, \\dots, \\lambda_d)$ and $\\lambda_i > 0$, the contraction rate for the $i$th entry is given by $\\rho_i(\\alpha) = | 1 \\alpha \\lambda_i|$.\nWe can make any specific entry $\\lambda_i$ converge fast (in one step), if we set $\\alpha = (\\lambda_i)^{-1}$. However, other entries $\\lambda_{j\\neq i}$ may either converge slowly or diverge. The best (global) learning rate $\\alpha^\\star$ that achieves the fastest overall contraction while ensuring convergence for all entries is given by\n$\\alpha^\\star = \\frac{2}{\\lambda_{\\max} + \\lambda_{\\min}}$, which achieves a contraction rate of\n$$\\rho_{\\max}(\\alpha^\\star)= \\frac{\\kappa - 1}{\\kappa + 1} $$\nwith $\\kappa = \\lambda_{\\max} / \\lambda_{\\min}$. Hence, the speed of contraction is problem dependent and may be slow even with the knowledge of $\\alpha^\\star$ (best possible fixed learning rate).\nHowever, if we apply normalization before running gradient descent on this problem, we always have $\\lambda_{\\max} = \\lambda_{\\min}$ and thus fast convergence (since  $\\kappa=1$, convergence in one step).\nConsequently, we always achieve the fastest possible convergence for $\\alpha^\\star$. Note that it does not matter how we found the optimal learning rate, *via manual tuning, grid search or sophisticated hyperparameter tuning*, but we argue that it is the most effective in a setting where $\\lambda_{\\max} = \\lambda_{\\min}$ (which normalization can help to achieve) as all variables may converge at similar rates.\nWe will clarify this point in the paper.\n\n> 2a) What are the exact procedures used to derive each of the normalization factors? \n\nThe procedure to derive the normalizing factors is decribed in Section 4.2: \"Derivation of the normalizing factors\". We use \n\n$\\eta^{-1}_{g(x)} = \\sqrt{\\mathbb{E}[\\Vert g(x) \\Vert_2^2]}$ \n\nfor analytical computations in case of linear maps and the residual update. For the activation function, we use numerical computation using Monte Carlo estimation for the activation function to compute \n\n$\\eta^{-1}_{acf} = \\mathbb{E}[\\Vert \\text{acf}(x) \\Vert_2]$.\n\nThe respective normalizing factors are then given by $\\nu = 1/\\eta^{-1}$. We list the specific normalization factors we used below with model dimension $d$, head dimension $h$ ($64$ in our case) and feed-forward dimension $f$ (typically $4 \\times d$) :\n\n$\\nu_{qkv} = 1 / \\sqrt{\\frac{h}{d}} $ for $W_{qkv} \\in \\mathbb{R}^{h \\times d} $\n\n$\\nu_{p} = 1 $ for $ W_p \\in \\mathbb{R}^{d \\times d} $ \n\n$\\nu_{uz} = 1 / \\sqrt{\\frac{f}{d}}$ for $ W_{uz} \\in \\mathbb{R}^{f \\times d} $\n\n$\\nu_{acf} = 1 / \\text{Monte Carlo estimation}$ \n\n$\\nu_{d} = 1 / \\sqrt{\\frac{d}{f}}$ for $ W_{uz} \\in \\mathbb{R}^{f \\times d} $\n \nWe are sorry that this is not clear, we will clarify this in the paper accordingly. \n\n> 2b) In Section 3.2, why $\\Vert h_{\\le L} \\Vert = \\sum^{L}_{l=1} \\Vert h_l \\Vert $ ? \n\nWe define $h_{\\le L} := \\sum_l^L h_l$, i.e., the residual state of the classic transformer after summing the contribution of $L$ blocks with, for simplicity, normalized outputs (post-norm). Estimating its norm should give us an idea about the growth of the hidden state in this case.\n\nNote that we analyze $ \\mathbb{E}[\\Vert h_{\\le L} \\Vert_2^2] $, not $\\Vert h_{\\le L} \\Vert_2 = \\sqrt{\\Vert h_{\\le L} \\Vert_2}$ (or $\\Vert h_{\\le L} \\Vert_2)$. For $\\Vert h_{\\le L} \\Vert_2 $ (not squared) we cannot pull the expected value inside the $\\sqrt{\\cdot}$ operation. Consequently, $ \\mathbb{E}[\\Vert h_{\\le L} \\Vert_2] = \\sum_{l=1}^L \\mathbb{E}\\Vert h_l \\Vert $ would not be correct. However, we unfortunately only stated $\\Vert h_l \\Vert = 1 $ but forgot to add the assumption $\\mathbb{E}[h_l] = 0$. Using these assumptions, the derivation of $\\mathbb{E}[\\Vert h_{\\le L} \\Vert_2^2]$ is as follows:\n\\begin{align*}\n\\Vert h_{\\le L} \\Vert_2^2 = \\left( \\sqrt{ h_{\\le L}^\\top h_{\\le L} } \\right)^2\n= h_{\\le L}^\\top h_{\\le L}\n= \\left(\\sum_l h_l \\right)^\\top \\left(\\sum_l h_l \\right) = \\sum_l \\sum_{l'} h_l^\\top h_{l'}\n\\end{align*}\nNow we analyze the expected value of this under the assumption that $h_l$ are independently distributed and $\\mathbb{E}[h_l] = 0$. From the assumptions follows, that for $l\\neq l'$, $ \\mathbb{E} [h_l^\\top h_{l'}] = 0$.\nConsequently,\n\\begin{align*}\n\\mathbb{E}[ \\Vert h_{\\le L} \\Vert_2^2 ] &= \\mathbb{E} \\left[ \\sum_l \\sum_{l'} h_l^\\top h_{l'} \\right] \n&= \\sum_l \\sum_{l'} \\mathbb{E} [ h_l^\\top h_{l'} ] = \\sum_l \\mathbb{E} [ h_l^\\top h_{l} ] \n&= \\sum_l \\mathbb{E}[ \\Vert h_l \\Vert_2^2] = L\n&& \\text{since for $l \\neq l'$, $ \\mathbb{E} [h_l^\\top h_{l'} ] = 0$ }  \\text{and $\\Vert h_l \\Vert = 1$ by assumption}\n\\end{align*}\n\n> 2c) How precisely are the \"effective learning rates\" applied? Although readers might guess, explicit clarification from the authors would significantly improve this paper.\n\nWe adopted the effective learning rates from the nGPT [1] paper but in our work crucial implementation detail deserves explicit clarification. \nFollowing nGPT's approach, for any trainable scaling parameter $ s_a$ (such as $\\alpha_A$, $ \\alpha_M$, $s_z$, etc.), we employ a reparameterization scheme using two scalars: $ s_{a,\\text{init}}$ and $s_{a,\\text{scale}}$. Instead of directly optimizing $s_a$, we optimize a surrogate parameter $ \\hat{s}_a$ and compute: \n\n$$ s_a = \\frac{s_{a,\\text{init}}}{s_{a,\\text{scale}}} \\cdot \\hat{s}_a $$ \n\nThe surrogate parameter is initialized as: \n\n$$ \\hat{s_a}^{(0)} = s_{a,\\text{scale}}$$ \n\nwhich ensures the initial value is: \n\n$$ s_a^{(0)} = \\frac{s_{a,\\text{init}}}{s_{a,\\text{scale}}} \\cdot s_{a,\\text{scale}} = s_{a,\\text{init}}$$\n\nFollowing the standard Adam update rule: \n\n$$ \\hat{s}_a^{(t+1)} = \\hat{s}_a^{(t)} - \\alpha \\cdot \\frac{m^{(t)}}{\\sqrt{v^{(t)}} + \\epsilon}  $$\n\nwhere $ m^{(t)} $ and $ v^{(t)} $ are the momentum terms computed using the scaled gradient. The effective learning rate for the actual parameter $s_a$ becomes: \n\n$$ \\alpha_{\\text{eff}} = \\alpha \\cdot \\frac{s_{a,\\text{init}}}{s_{a,\\text{scale}}}$$ \n\nThis allows us to control the learning dynamics of different parameter types while maintaining a single global learning rate $\\alpha$. For the interpolation parameters $\\alpha_A$ and $\\alpha_M$: \n\n$ s_{\\alpha,\\text{init}} = 0.01, \\quad s_{\\alpha,\\text{scale}} = \\frac{1}{\\sqrt{d_{\\text{model}}}}$ \n\nThis yields an effective learning rate: \n\n$ \\alpha_{\\text{eff}}(\\alpha_A, \\alpha_M) = \\alpha \\cdot 0.01 \\cdot \\sqrt{d_{\\text{model}}}$ \n\nFor scaling parameters like $s_z$: \n\n$ s_{z,\\text{init}} = 1, \\quad s_{z,\\text{scale}} = \\frac{1}{\\sqrt{d_{\\text{model}}}}$ \n\nyielding: $ \\alpha_{\\text{eff}}(s_z) = \\alpha \\cdot \\sqrt{d_{\\text{model}}}$ \n\nThis mechanism addresses the fundamental challenge that different parameter types require different update magnitudes. Our reparameterization ensures that parameters with magnitude $ \\mathcal{O}(1/\\sqrt{d_{\\text{model}}})$ receive appropriate learning rates, all parameters can be optimized with a single global learning rate, and training remains stable by avoiding the need for manual learning rate scheduling across parameter types. This approach ensures that scaling parameters like $s_z$ are updated with effective learning rates comparable to other normalized parameters in the network, eliminating the need for separate learning rate schedules. We will add more details to Section 4.3. \n\n> 3 While assumptions such as the i.i.d. condition clearly hold at initialization, the paper does not sufficiently explain why these assumptions remain valid throughout training. / How do you ensure that core assumptions, …, continue to hold after training?\n\nThe theory only serves as motivation to investigate the application of the norm concentration phenomenon empirically. The assumptions do not hold in training, partially by design. As stated in Section 4.1 line 140: “While these assumptions may not hold during training, we may still observe concentration empirically, particularly due to the high dimensionality of representations”. Also, it does not hold since the weights in our case are not normalized but only bounded, see Sec. 4.2  Constraining Parameters. Bounding the norm of the parameters instead of normalizing led to better results empirically, see Fig. 5. Additionally, one can see from Figure 1, that the intermediate representations are not truly normalized (but only approximately) reaching norms up to ~2.9. \n\nDue to the post-norm operation and our approx. normalization factors, we observe many of the beneficial properties associated with normalization such as a very moderate growth of the norm on the residual connection (Fig. 1). \n\nExperimentally, we can see that the blocks in anGPT keep the normalization concentration of norm due to the norm factors. The ratio of the mean input norm to the mean output norm of a block (for anGPT before the post norm and for GPT+ after the RMS Norm) on different layer (first, middle and last) does deviate less in anGPT than in GPT+ Please find the results of this experiment in the Table in Section 1.2 in the response to Reviewers skhp. \n\nThank you again for the thorough and constructive review. Your feedback will help us improve our work. Please mind, we also performed downstream evaluations and additional ablation experiments, see response to Reviewer SY5g. Please do not hesitate to ask further questions. In case we addressed your concerns, we would appreciate a reconsideration of your score. \n\n\nBest regard, the authors. \n\n___\n[1] Loshchilov, Ilya, et al. \"ngpt: Normalized transformer with representation learning on the hypersphere.\" 13th International Conference on Learning Representations (ICLR 2025) ."}]

Author rebuttal:

Meta-review: This paper presents an approximately normalized Transformer (anGPT) that uses fixed scalar multipliers, motivated by the concentration of measure phenomenon, to accelerate convergence by up to 40% and eliminate the need for weight decay and learning rate warm-up. The primary strengths identified by reviewers are the significant, empirically validated training speedup at small scales and the simplification of the training pipeline. Initial weaknesses centered on the limited scope of evaluation beyond pretraining loss and the partial validation of the underlying theoretical assumptions during training. During rebuttal, the authors provided extensive downstream evaluations and sensitivity analyses that demonstrated the method's practical benefits, addressing the most critical concerns. Although questions about the theoretical grounding remain, the strong empirical results and the prospect of practical utility support accepting the paper.
Decision: Accept (poster)