Normalization in Attention Dynamics
Authors: Nikita Karagodin
Shu Ge
Yury Polyanskiy
Philippe Rigollet
Venue: NeurIPS 2025 poster ()
Keywords: Transformers
Self-Attention
normalization
continuous-time interacting particle systems
clustering
representation collapse
Primary area: theory
TL;DR: Theoretical study of impact of normalization layers in evolution of tokens representations as they propagate through layers of a transformer.

Abstract: We study the effect of normalization schemes on token representations in deep transformers. Modeling their evolution as interacting particles on the sphere, we show that normalization acts as a form of speed regulation. This perspective enables a unified analysis of several schemes---including **Post-LN**, **Pre-LN**, **Mix-LN**, **Peri-LN**, **nGPT**---revealing how they influence clustering dynamics and representation collapse. Our framework clarifies how different schemes shape token representations across layers and provides a principled basis for comparing them, identifying **Peri-LN** as a particularly effective choice.

Normalization in Attention Dynamics
Nikita Karagodin'
Shu Ge
Yury Polyanskiy!
Philippe Rigollet
Abstract
We study the effect of normalization schemes on token representations in deep transformers. Modeling their evolution as interacting particles the sphere, we show that normalization acts as form of speed regulation. This perspective enables unified analysis of several schemes-including Post-LN, Pre-LN, Mix- LN, Peri-LN, nGPT-revealing how they influence clustering dynamics and representation collapse. Our framework clarifies how different schemes shape token representations across layers and provides a principled basis for comparing them, identifying Peri-LN particularly effective choice.
1 Introduction
Transformer architectures have revolutionized natural language processing and beyond, demonstrating unprecedented performance across diverse tasks-froe machine translation and text generation to reasoning and protein folding. The remarkable capabilities of transformers, including their emerging reasoning abilities, are enabled by the attention mechanism introduced in Bahdanau et al. (2015): Vaswani et (2017).
recent line of theoretical work, initiated in Geshkovski et al. (2023). studies information processing across deep transformer layers by reframing them interacting particle systems, building the original setup of Sander et al. (2022). Following this initial work, layer normalization (I (LayerNorm) emerged critical component significantly influencing the long-term dynamics of these systems. Geshkovski et al. 2025) proposed a model in which particles are constrained to evolve on a sphere, corresponding the so-called Post-layer norm (Post-LN) scheme. This model has since become the standard paradigm for transformer analysis in subsequent research (Karagodin et al. 2024
Geshkovski al. 2024a Bruno 2025a Criscitiello et al. 2024).
Several alternatives to Post-LayerNorm (Post-LN) have emerged recent years to improve training performance, each subtly altering transformers' long-term clustering behavior. Most notably, Pre- LayerNorm (Pre-IN) has become the default choice for leading large language models including GPT (Radford al. 2019) and L.LaMA (Touvron, H. et all 2023). This approach originally introduced in ResNet-v2 He et al.] (2016) before being adapted for Transformer architectures. enables more stable training of deeper networks while reducing sensitivity to hyperparameters such as learning rates (Xiong et al. 2020).
Understanding normalization schemes is essential for advancing the design and performance of transformer architectures. In particular, [Sun et al. (2025) and Gromov et al. (2025) identify a phenomenon known the curse depth, in which deep layers of large language models (LLMs) degenerate into mear-identity transformations. This effect is so pronounced that it enables pruming of deeper layers with minimal impact on performance (Muralidharan al. 2024 (iddiqui et al. 2024). On the other hand, the well-known issue representation collapse presents a significant challenge to increasing the depth of L.Mss.
Department of EECS, MIT, Cambridge, MA, USA 12 Department Mathematics. MIT, Cambridge. MA, USA
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To mitigate this issue, Li ct al. LN normalization the early layers and reserves Pre-LN normalization for the deeper layers. This strategy was further refined in the development of Peri-LN (Kim, B., Johnson, M. et al. 2025 which has been reported to used the Gemma-3 model (Gemma Team et al. (025). Alternatively, Noci et al. (2022) suggest simpler fix: rescaling residuals by the square root of the depth (a method we will call sgrt-scaling). This method with Pre-LN architecture was then studied in-depth by [Sun] et al. 2025), Additionally, Loshchilov al. (2025) show that with careful architectural design, as in nGP1 normalizing tokens to lie on the unit sphere can further streamline the normalization process.
(2025 propose a hybrid normalization scheme that applies Post-
Given the diversity of these approaches, we are motivated to explore the following question:
How do normalization schemes influence deep representations in transformers?
To answer this question, we revisit both classical and novel LayerNorm schemes through the lens of the simplified interacting particle dynamics introduced in Geshkovski et al. (2025) to bring a theoretical understanding of these various design choices. Since the final decoding layer of transformer is typically preceded by a normalization step, we focus on the direction of token representations. Regardless of the specific normalization used, these directions naturally form an interacting particle system on the sphere. This shared geometric setting enables a direct, side-by-side comparison of various normalization schemes, all of which we reinterpret as forms of speed regulation. Despite its simplicity, our model captures complex behaviors observed in practice, including curse of depth and representation collapse.
Related Work. A growing body of work has examined normalization in Transformers, with a primary focus its empirical and theoretical implications for gradient stability. Notably, Xiong et al. (2020) and Sun et al. (2025) provide experimental evidence that improper placement of normalization layers lead to exploding vanishing gradients in deep models. These findings often supported by variance- based analyses that track the propagation of activations and gradients through the network, such as (Noci et al. 2022) and (Kedia al. 2024). WWotsman et al. (2024) further identify normalization-relate
training instabilities that emerge at scale. Building on this foundation, [i al. (2025) and Kim, B., Johnson, M. et al. (2025) explore hybrid normalization strategies in large-scale settings, using both theoretical approximations and empirical diagnostics to study gradient flow and the stability of learned representations.
In contrast to prior work that primarily investigates gradient dynamics, our study focuses on the forward evolution of token representations through the network. This perspective complements the analysis of gradient flow by shifting the emphasis from the ability to train (via backpropagation) to the expressiveness and structure of the learned representations. While both viewpoints offer valuable insights, we focus on the latter in the present work. A companion paper dedicated to the analysis of gradients is currently in preparation.
Our contributions. We provide different perspectives on normalization architecture, by casting differently normalized Transformers variations of a common interacting-particle ODE, where the normalization method determines speed factor, which can amplify initial velocity and dampen representation collapse in deep layers. Within this unified framework, we extend the framework of Geshkovski et al. )2025) for Post-LN and in particular, establish asymptotic clustering under general conditions on the speed regulation mechanism. To differentiate various normalization schemes we further study the initial and final velocity of tokens corresponding to first and deep layers respectively. In particular, we recover the representation collapse phenomenon that plagues Post-LN. Our theoretical framework identifies Peri-LN as particularly effective scheme that makes good use of both early and deep layers.
Normalized Attention Dynamies
A sequence of tokens is represented by their column- embeddings X Rddx.. In the rest of this section, functions f Rd + Rd applied to such a matrix are understood column-wiser f(X)= = [f()), (xx,)) For each token embedding Fk, we define its direction 02 x | gd-1 and its magnitude r = ||<]|| 0, so that
(Z1,..., 2n)
Ib Tk 0x.

---

As the sequence of token embeddings is processed through the layers of transformer, it gets updated from X' to x layer In the rest of this section derive the updates obtained by different normalization rules and recast them speed regulation mechanisms for token directions.
For simplicity and convenience of exposition, we omit FFN layers and focus pure attention. The approach could be extended to more general architecture, but this would introduce additional technical complexities beyond the scope of this paper.
2.1 Attention
At layer I, an attention head is characterized by three matrix parameters Q', K* VL, called Query, Key, and Value respectively. These matrices used to create the attention matrix, which is of pairwise interactions between tokens with entries given by
matrix W
8IQ' k()
E eBQ KIa
where added a redundant temperature parameter usually taken equal 1 but that will be convenient in our simplifications below. The attention function is the linear operator A Rdxx + Rd defined as (X) (X). ..... A (X)] where each column is given by
AJ(X) - Σ
j j I..,...,
Throughout this paper, we focus on the simplified setting of Geshkovski et (2025) where Q' Kt = V La for all f 2 0.
2.2
Normalization.
The Root Mean Squared (RMS) norm Norm(z) >/||>=|| of token Zhang and Titov 2019 is critical ingredient all normalization schemes considered here.
Layer Norm
Layer
Layer Norm
Layer Norm
FFN
FFN
FFN
Layer Norm
FFN
Layer
Layer Norm
Layer Norm
Layer Norm
Attention
Attention
Layer
Attention
Layer Norm
Layer Norm
Attention
X
X
Post-LN
(b) Pre-LN
(c) Peri-LN
(d) nGPT
Figure 1: Normalization layer placements in various architectures.
In this paper, we study six major schemes: Post-LN (Vaswani et al. 2017). Pre-LN (Xiong et al.
2020), Mix-LN (Li et al. 2025). Peri-LN (Kim, B., Johnson, M. al 2025), nGPT Loshchilov

---

et al. 2025), and sqrt-scaling (Sun et al. for S and Pre-LN for > T while sqrt-sealingis deterministic rescaling of Post-LN. The four remaining schemes are presented in Figure Such explicitly layer-normalization rules are not the only strategies employed in practice. Other attempts to improve normalization suggest better initializations Qo, K v (Kedia et al. 2024) and explicit scaling of the updates, similarly to o that is trainable in nGPT.
2025). Note that Mix-LN is combination of Post-LN
Thanks to the residual connections, each layer-update can be seen as forward Euler discretization of a continuous- time ODE that captures the dynamics of tokens while enabling the deployment of useful calculus tools. In this context, it is convenient to write X (t) as function of time and replace X X with (t).. For any two matrices X, Y E Iedx where X has unit-norm columns, define the projection operator PxY Y to be the column- wise projection on to the tangent space of the sphere S
PxY= = [P, V1, Pa..Un)
where for any 8 y E D = y - (y, x) is the projection of y onto the tangent space of Sa-1 at
The dynamics described by each normalization schemes are presented in Table !
Table 1: Normalization Schemes in Discrete and Continuous Time Domains. In nGPT, oz E Ris layer-dependent learnable parameter.
Scheme
Discrete Time Update
Continuous Time Update
Post-LN
X++ Norm (X' + A*(())
X(t)= P A((
Pre-LN
x = X + A' (Norm(XX))
X(t) = A' (Norm(X
Mix-LN
X = [Norm (x* + (x) Hgr
(t) = P A' (x))) Hirr
+ [X' + A' (Norm [)] 1ir
+ AA (Norm (x [+)
Peri-LN
X X + Norm (A' (Norm (orm(x')))
(() Norm (Norrm (X(())))
nGPT
X = Norm (X' + az Norm (Ar (A (Xx)))
X() P x (X))
sqrt-scaling
X++1 = Norm (X' + V (x))))
((t) = A P Ax())
2.3 Speed regulation formulation
In Post-LN, nGPT, and sqrt-scaling. tokens are constrained to the the sphere Sdd with sqrt-scaling simply adjusting the speed of the particles function of t compared to Post-LN. For the other rules where tokens may have varying magnitude, one final projection is typically applied before the final decoding layer in practice. In particular, this means that decoding depends on directions (1) = Norm(a,, (4)), for j =
Interestingly, when tracking only the directional components (t),, (t) E Sd-1, all normaliza- tion rules give rise to interacting particle systems evolving on the sphere, governed by a common velocity field but subject to distinct, rule-dependent speed-regulation mechanisms. Note that this does not imply the particles follow the same trajectories at different speeds; indeed the speed parameter has a significant impact on the trajectories. More specifically, directions 01....,6, 0n St undergo the normalized attention dynamics given by
0,(1) 1 Po,(o ()()()) 81
(NA)
where (t)) = (t)..). .0.))...,,,,,, (t)) and we recall that Ps = Ia 00 is the projection from Rd to the tangent space of the sphere 0. Using the following identities
(t) = (t),
(,()) -100
1 =11
T (2)(1) (t) =T((t) (t),
readily get:

---

Table 2: Speed regulation factors
8y(t)
tt(t)
Post-LN
0
Pre-LN
T2(()
(0;(),
Mix-LN
t + +r,(1)
(0,0),4 T,>
Peri-LN
(0()))
(100)410000/14400000
nGPT
04 (1())
0
sqrt-scaling
v++1
0
Asymptotic clustering
Since the work of Geshkovski et al. 2223 2025),
theoretical analyses of attention dynamics have
primarily focused on establishing asymptotic clustering under the Post-LN scheme, namely (t(t)
8" as - oC for all j = 1, 1,..., n, under generic initialization; see also Criscitiello et al. [024])
Chen et al. (205).. However, empirical studies have revealed that in practice, tokens often remain trapped in metastable states for extended periods before clustering emerges (Geshkovski al. 2024a
Bruno et al. 2025a). Despite this, the clustering phenomenon appears to occur at multiple local scales, and the simplified setting considered in prior work continues to offer valuable insights, will demonstrate in the next section. In this section, we extend the analysis and show that asymptotic clustering persists beyond the original Post-LN framework other normalization schemes.
Recall that we study the normalized attention dynamics
NA defined by
(1)0 (9)
0,01 st( Po, A7(0(())
sT(1) P0.
Σ VO
j == ,,
Ei=1" (10,0))
A=1
where the speed regulation factor sy (t) is given in Table and Peri-LN are not directly regulated by an explicit mechanism but rather by the magnitude. In particular, this mechanism dampens the speed of each token individually according to their magnitude.
2 It is interesting to note that both Pre-LN
The main observation of Geshkovski et al. (0255) that when KQ QKT =V the Post-LN
system is a gradient flow for the energy function
E(O) =
Σ
3(0%,
J,k=1
where we recall that O = 10,,..., 0n).
For NA), we have
0,(1) 8, (1) Vo, (O)),,, where 2,(1) = (10
I=1
and V denotes the spherical (Riemannian) gradient.
The above dynamics can be seen as modulated gradient flow, albeit with complicated modulator that depends on time and space. For vanilla gradient flows, that is for (t) const., celebrated result of łojasiewicz guarantees convergence of this gradient flow to a critical point of the energy. Following the same steps, we show in the Appendix D.1 that this result extends to the present framework, guaranteeing convergence of any trajectory. From there, we establish the following clustering result.
Consider the normalized attention dynamics (NA) with = - la. Then for uniformly sampled initializations ((0) E (gd-1)e Post-LN, nGPT, sqrt-scaling cluster asymptoti- cally
Theorem 3.1.
P(otokkes synchronize cluster)] =1 whereas for standard Gaussian sample of (()) (()) with ((0) E (8n-1)8n for Pre-LN, Mix-IN, Peri-LN one has
-(()) e R
Plttokens 0, synchronize to cluster) min inf (t) = 0}} = 1. JE[m] 1-520
5

---

In fact, this result holds not only for - but more generally for Q' = Q and V = Q K = K' as in Sander et al. (2022). The second condition on the magnitude growth can be traced with Table deriniiinn to work with further. For example, we immediately get the following.
Corollary 1. For Pre-LN, Peri-LN with S e we have unconditional synchronization.
This statement follows from simple lower bound on i.. We write it for Pre-LN, and Peri-LN can be done similarly.
ry (0.. A,())) Z
(c (0).0)) +
Σ ((0 (0z, kng
nea (e (n-1)) ne
where we used the fact that any negative term in the second sum is at most 1, e 2 n and a trivial bound on Zz.
4 Initial and terminal token velocities
The previous section established an asymptotic result but did address the rate at which tokens cluster, aspect that is crucial for understanding how representations evolve. This question is important because the velocity at time determines the influence of the tth layer in shaping the final token representation.
Before analyzing the propagation speed of tokens in our attention dynamics model, we first discuss a benchmark for desirable behavior. In an efficient architecture, each layer should meaningfully transform token representations, causing substantial displacement in representation space. If tokens remain nearly stationary across many layers, the architecture risks representation collapse. Equally important, however, is ensuring that early layers contribute until later stages can limit the expressive power of the network.
delaying transformation
4.1 Prelude: Symmetric initialization
Following Geshkovski al. (0225); Cowsik et al.
2024), we begin with so-called orthogonal symmetric initialization where (6,(0) (0). 9 (0)) = 0 for + A and (0)
for all j. This config- uration approximately matches that of randomly initialized tokens in high dimension. Due to the symmetry, the cosine similarity (t) = (t), (t)) does not depend on j * and the entire token dynamics reduces to the evolution of two scalar quantities: 7() and r()). In the Appendix, we derive a simple ODE for (L), r(t) following Geshkovski et al. 2025 Theorem 6.8). We plot ODF-bbased 256. Despite its simplicity, this setup already provides striking insight into the effects different normalization schemes. The importance tracks in how cosine similarity evolution alike the theoretical formula plotted in Figure 2 and the experimental setup with random weights modeled in Figure
evolution of (t) in Figure 2 with parameters 3 5,
La initialized at symmetric orthogonal configuration, i.e. (0)(0), (x) 6jk and r (0) = To for all j. Then, for all the cosine similarity 7t)) (t), Oz)) remains constant across all pairs j + k and
Theorem 4.1. Consider the normalized attention dynamics NA
with Q K
(t) for t + 0 and t + oo is given by
t+ 0
t- 0C
2
Post-LN
Ce
@ +n +n-1
Pre-LN
C/t3
ro(e@ + 1)
2
Mix-LN
C/t3
@ + n2 2
Peri-LN
C/t3
Tov
22 + n
nGPT
200
Ca.e 256 a,ds
Ve +
ce-iv
sqrt-scaling
e +
Vt

---

where C' > 0 may change from line to line.
A few remarks are in order. First, the initial velocities are comparable across models, up to the effects of the tuning parameters op and ro. Notably, the temperature parameter & exponentially damps the initial velocity, suggesting that initializing Q and K with smaller magnitudes in the early layers may be beneficial. More striking the effect of speed regulation at terminal velocity: Pre-LN, Mix-LN, and nGPT (with constant ox)) exhibit polynomial slowdown, in contrast to other normalization schemes. While sqrt-sealing converges more slowly than exponential, it still outpaces the polynomial decay. This implies that Pre-LN, Mix-LN,, and nGPT cluster their counterparts indicating a more effective use of intermediate layers and a stronger resistance to representation collapse. Finally, note that the trainable parameter a, in nGPT can have drastic impact on both initial and terminal velocity. See Figure 2 for a visual representtion of cosine similarity and evolution of relative time and position. See Figure for comparison between different 0₁ nGPT.
gradually than
3.01
0.01
Prrs
04-
13433
0.00
0.04
0.64
TerLS
MM
0.09
(a) Cosine sim. 7()) vs. L
(b) Speed (()) vs.
(c) Phase plot (1) vs. 7(1)
Figure 2: (a) Evolution of cosine similarity (t), (b) its speed +tt) over time, (c) phase-plot of (t) vs. (t), for introduced normalization strategies. Here nGPT has a 1, to showcase the significance of that parameter. Pre-LN and Peri-LN are the last to converge, mitigating representation collapse. On the other hand, Post-LN, nGPT and Peri-LN move faster in early layers, effectively utilizing them. In the phase-plot (c) how at the same position the speed is defined by known speed control parameter, ranking different methods.
65
8
0.2
50
12.0
315 201
24
130
235 201
(a) Cosine sim. (() vs.
(b) Speed ((t) vs.
(c) Phase plot +(1) vs. 7(1)
Figure 3: Convergence nGPT from orthogonal initialization for different choices of az - constant, root, linear, combination of linear and constant with weights sin (4t) and cos(4t).
4.2 Initial velocity
The symmetric evolution described above is too coarse to properly discriminate between normalization schemes at initialization. Here show that early Peri-LN/nGPT layers move tokens order-one distances on the hypersphere, while Post-LN and Pre-LN advance more slowly, with step sizes on the order of Olog n/ n/d).

---

Cosine similarity with confidence intervals
1.0
005 5m
0.4
Post-UN
0.2
Peri-LN
MixiN
supt-soaling
10
Figure 4: Evolution average cosine similarity (t) with 90% confidence interval with randomly initialized weights (Kaiming init), d = 512, Theads = 1, 3 Vd, d > n and random initial X. We set = 1 for nGPT. We see that Peri-LN and nGPT initially move faster, and that Post-LN and nGPT eventually collapse tokens faster than Pre-LN and Peri-LN. Appendix for more studies, including multi-head, untied weights and more.
Theorem 4.2. Let K,K, V E Idxad satisfy max K Koop [|V S 1, B = 1. Let the initial directions ()) lid. Unif(gd-1) 1) and set the attention vector
A,(0) Zj
Zj (00))
Then there are absolute constants c, C 1 - n simultaneously for all E [n]
such that for vd 2 n log 2 d. with probability
log n
log Ti
||4,(0)l S
d
To interpret the significance of Theorem 4.2, recall from Table that the initial velocity of direction 0, is dampened by a factor proportional to A, (0)) || for both Peri-LN and nGPT. Consequently the first-layer angular displacement of Peri-LN and nGPT exceeds that of Post-LN, Pre-LN, Mix-LN, and sqrt-scaling, by factor (mmin(d/ log n, v vn/ logn)).
4.3 Terminal velocity
The idealized setup of Section 4.1 sheds light on a qualitative difference between Post-LN and Pre- LN: Post-LN clusters tokens much more aggressively than Pre-LN in the late stages of clustering, In retrospect, the intuition behind this phenomenon rather clear: under Pre-LN, the angular velocity (1) of token is divided by growing radial factor ry ((), which increasingly dampens the rate at which tokens collapse toward one another. In contrast, Post-LN normalizes this growth away, allowing tokens to continue clustering higher rate.
In this section, we go beyond the symmetric case Section 4.1 and analyze a simplified setting in which tokens are pre-clustered, in the sense that they all lie within narrow cone. This configuration captures the behavior of a single well-formed cluster and isolates the dynamics from interference by other clusters. The results below confirm findings of Section 4.1 indicating that this idealized setup is already informative.
Radial Growth under Pre-LN. Our first goal is estimate the rate of growth of r()),. the norm of token j's representation, under Pre-LN normalization. Empirically, the growth of hidden states in transformers has been well-documented. For instance, studies such (Xiong et al.) 2020 Kedia

---

et al. 2024) observe that in randomly initialized transformers, rt(t) ~ vł, reflecting the diffusive nature random walk induced by randomly sampled projections V.
However, in an aligned regime where all tokens are directionally coherent, the dynamics reinforce alignment and exhibit linear radial growth: ry (t) as in Section This linear scaling significantly alters the clustering behavior. Because the angular update is effectively scaled by (/r,(t), linear growth ry (t) slows the clustering rate from exponential to polynomial.
Speed cluster collapse. To quantify the normalization induced slowdown, we introduce the Var(!) proxy for intra-cluster variance. Specifically, given token directions, (t),,.... On(() let
1 T1
Var(t) == [[0((t) 8(t)1 where = Σ 03.
Theorem 4.3. Consider the normalized attention dynamics with V La and arbitrary Q, K s.s. ||Q K|| & initialized at 01
in local cone, namely (0, (0), 8x 2 1 for < 1/(100n²33) Let the cluster center be defined as 0 = Ug. Then the following properties hold
(i) Radial growth For all k, the radial component satisfies
Tx(1)) 2 -8), for both Pre-LN and Peri-LN
(ii) Speed of clustering. It holds
-((ar(t)),
for Post-LN
θ (Var(t)/t)
for Pre-LN for Peri-LN
d
θ (Var(t)/t)
Var(t)
dl
θ (Var(t)/oa)
for nGPT (Var(t)/t)
for Mix-LN for sqrt-scaling
(Var(t)/VB)
Again this result corroborates the findings of Section 4.1 Post-LN and sqrt-scaling cluster token directions at an exponential rate, while Pre-LN, Peri-LN and Mix-IN slow down to a polynomial (~ 1/t0) decay. Moreover, nGPT has the ability control rate of clustering through a, This confirms that Pre-LN makes better use of depth, as tokens continue evolve meaningfully across many layers, rather than collapsing too quickly. In particular, this difference explains why Pre-LN less prone representation collapse in very deep models compared to Post-LN.
Theorems 4.2 4.3 together give concrete guidelines select a normalization scheme with large initial and terminal velocities so as to ensure adequate progress of token representations across both first and deep layers. clear winner here is Peri-LN that manages to do both automatically, and nGPT that has ability to control the behavior via Ot.
5 Limitations
Our study offers a unifying dynamical-systemss view of several normalization schemes, yet some theoretical and practical caveats temper its scope.
Theoretical limitations Theorem 3.1 proves that every trajectory of the normalized-attento ODE converges. Our approach relies on transforming the system compact autonomous frame, therefore it gives some convergence rate(that roughly aligns with Theorem4.1 but it furnishes neither an explicit rate nor any metastability guarantees.
We bound only the initial and terminal speeds; the intermediate regime remains uncharacterized In particular, one flow could enter region where it moves faster than another-ever though its speed-control factor is larger. Comparing two flows in the general case, even when one enjoys a higher speed-control factor, remains an open problem.
Because of the strict assumptions on the weight matrices, the analysis does not capture the full behavior observed in both theory and practice. For example, in this work representation norms in

---

Pre-LN are predicted to grow linearly (when matrices Q, K. V are tied), whereas empirical work reports a vł trend at initialization (when weights are random). Reconciling these gaps calls for stochastic analysis of the problem.
Our theory also leaves unexplained several optimization such as exploding updates in Ma-LNN-becaure it omits working with the gradient propagation. A companion gradient-flow analysis is required for complete picture and is the subject of ongoing work.
Practical limitations From a practical perspective, make two key simplifications. (i) MLP layers are omitted to focus purely on attention; and (ii) the query, key, and value matrices obey restrictive assumptions. Although the intuition gained from these toy settings is instructive, the proofs rely heavily on the simplifying hypotheses. Finally, in this work, we do not give any specific model architecture to train and validate, which currently limits direct architectural recommendations could offer.
Addressing these limitations- tight metastability bounds, inclusion of MLP layers, gradient-flow analysis, and empirical verification consaittus- fertile ground for future research.
Acknowledgments
PR is supported by NSF grants DMSS2022444 and CCF-2106377.
References
Bahdanau, D., Cho, K., and Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. ICLR.
Bruno, G., Pasqualotto, F., and Agazzi, A. (2025a). Emergence of meta-stable clustering in mean-field transformer models. In ICLR.
Bruno, G., Pasqualotto, F., and Agazzi, A. (2025b). A multiscale analysis of mean-field transformers in the moderate interaction regime. arXiv preprint arXiv:2509.25040.
Chen, S., Lin, Z., Polyanskiy, Y., and Rigollet, P. (2025). Quantitative clustering in mean-field transformer models. arXiv.
Cowsik, A., Nebabu, T., Qi, X.-I., and Ganguli, S. (2024). Geometric dynamics of signal propagation predict trainability of transformers. arXiv.
Criscitiello, C., Rebjock, Q., McRae, A. D., and Boumal, N. (2024). Synchronization on circles and spheres with nonlinear interactions. arXiv.
Gemma Team et al. (2025). Gemma 3 technical report. arXiv.
Geshkovski, B., Koubbi, H., Polyanskiy, Y., and Rigollet, P. (2024a). Dynamic metastability in the self-attention model. arXiv.
Geshkovski, B., Letrouit, C., Polyanskiy, Y., and Rigollet, P. (2023). The emergence of clusters in self-attention dynamics. NeurlPS.
Geshkovski, B., Letrouit, C., Polyanskiy, and Rigollet, (2025). A mathematical perspective on transformers. Bull. Amer: Math. Soc.
Geshkovski, B., Rigollet, P., and Ruiz-Balet, D. (2024b). Measure-to-measure interpolation using transformers. arXiv.
Gromov, A., Tirumala, K., Shapourian, H., Glorioso, P., and Roberts, (2025). The unreasonable ineffectiveness of the deeper layers. In ICLR.
Haraux, A. (2012). Some applications the łojasiewiez gradient inequality. Communications on Pure and Applied Analysis, 116):2417-24277
10

---

He, K., Zhang, X., Ren, S., and Sun. J. (2016). Identity mappings in deep residual networks. In Computer AitionECCVV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 650-645.55 Springer.
Karagodin, N., Polyanskiy, Y., and Rigollet, P. (2024). Clustering in causal attention masking. In NeulPS.
Kedia, A., Zaidi, M. A., Khyalia, S., Jung. J., Goka, H., and Lee, H. (2024). Transformers get stable: An end-to-end signal propagation theory for language models. arXiv.
Kim, B., Johnson, M. et al. (2025). Peri-LN: A hybrid normalization scheme for deeper transformer models. arXiv.
Li, P., Yin. L., and Liu, S. (2025). Mix-In: Unleashing the power of deeper layers by combining pre-In and post-In. In ICI.R.
Loshchilov, I., Hsieh, C.-P., Sun, S., and Ginsburg, B. (2025). nGPT: Normalized transformer with representation learning the hypersphere. In ICLR.
Muralidharan, S., Sreenivas, S. T., Joshi, R. B., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro,
B., Kautz, J., and Molchanov, (2024). Compact language models via pruning and knowledge distillation. In NeurIPS.
Noci, L., Anagnostidis, Biggio, L.. Orvieto, A., Singh, S. P., and Lucchi, A. (2022). Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. In NeurlPS.
Polyanskiy, Y., Rigollet, P., and Yao, A. (2025). Synchronization of self-attention dynamics on the circle. arXiv.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAl Technical Report.
Sander, M. E., Ablin, P., Blondel, M., and Peyré, G. (2022). Sinkformers: Transformers with doubly stochastic attention. In AISTATS.
Siddiqui, S. A., Dong, X., Heinrich, G., Breuel, T., Kautz, J., Krueger, D., and Molchanov, P. (2024).
deeper look depth pruning LLMs. In ICML 2024 Workshop Theoretical Foundations of Foundation Models.
Sun, W., Song, X., Li, P., Yin, L., Zheng, Y., and Liu, S. (2025). The curse of depth in large language models. arXiv.
Touvron, H. et al (2023). LLaMA: Open and efficient foundation language models. arXiv.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.
Wortsman, M., Liu, J., Xiao, L., Everett, K., Alemi, A., Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., Pennington, J., Sohl-dickstein, J., Xu, K., Lee, J., Gilmer, J., and Kornblith, (2024). Small-scale proxies for large-scale transformer training instabilities. In ICLR.
Xiong, R., Yang. Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L.,, and Liu, T.-Y. (2020). On layer normalization in the transformer architecture. arXiv.
Zhang, B. and Titov, (2019). Root mean square layer normalization. arXiv.

---

NeurIPS Paper Checklist
Claims
(a) Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
(b) Answer: [Yes]
(c) Justification: Our abstract and introduction clearly articulate the two core contribu- tions unified ODE model covering six normalization variants and the analysis of said model in some specific regimes, All subsequent sections deliver matching formal statements and proofs, so the scope advertised up front aligns with the body of the paper. No claims exceeding the proven results are made.
(d) Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings, It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
(a) Question: Does the paper discuss the limitations of the work performed by the authors?
(b) Answer: [Yes]
(c) Justification: The limitations of the work are described in the Limitations Section
from different perspectives.
(d) Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-texx system might not used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
12

---

(a) Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
(b) Answer: [Yes]
(c) Justification: The settings we are studying are always stated in details, Proofs are suspended to supplementary materials.
(d) Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
(a) Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
(b) Answer: [NA]
(c) Justification: The paper has no experiments to reproduce
(d) Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, may necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
novel architecture, describing the
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility. which may depend on the nature of the contribution. For example
i. If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
ii If the contribution primarily a new model architecture, the paper should describe the architecture clearly and fully.
iii. If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how construct the dataset).
iv. We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibil- ity. In the case of closed-soure limited some way (c.g., to registered users), but should be possible for other researchers to have some path to reproducing or verifying the results.
models, it may be that access to the model is
13

---

5. Open access to data and code
(a) Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
(b) Answer: [NA]
(c)
Justification: Paper does not include experiments requiring code.
(d) Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guide- lines (https: ftttps://nippa httppppp..ppblllc.ccccc http://pppbicciiiiee//////ataaaa details.
Ctt////oodeeeeeeeeeeeeeeeee ssionnolicy) for more
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs data and code is permitted.
6. Experimental Setting/Details
(a) Question:
Does the paper specify all the training and test details (e.g., data splits, hy- perparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
(b) Answer: [NA]
(c) Justification: The paper does not include experiments
(d) Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
(a) Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
(b) Answer: [NA]
(c) Justification: There are experiments
(d) Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap. etc.)
14

---

The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
(a) Question:
For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
(b) Answer: [NA]
(c) Justification: There are experiments
(d) Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal
cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
(a) Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttt::///ttuiiit http:///wwwaaaccooooo//oooctttta
(b) Answer: [Yes]
(c) Justification:
(d) Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make consideration due to laws or regulations their jurisdiction).
to preserve anonymity (e.g., if there is special
10. Broader Impacts
(a) Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
(b) Answer: [NA]
(c) Justification: The paper is theoretical, thus its impact is limited to scientific advances and has no immediate societal impact otherwise.
(d) Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness consid- erations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
15

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement the quality of generative models could be used to generate deepfakes for disinformation, On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
(a) Question: Does the paper describe safeguards that have been put place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or seraped datasets)?
(b) Answer: [Yes]
(c)
Justification: We do not have experiments
(d) Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
(a) Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms use explicitly mentioned and properly respected?
(b) Answer: [NA]
(c)
Justification: Does not use existing assets
(d) Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, haperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that of the derived asset (if it has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's creators.
re-packaged, both the original license and the license
16

---

13. New Assets
(a) Question: Are new assets introduced in the paper well documented and is the docu- mentation provided alongside the assets?
(b) Answer: [NA]
(c)
Justification: No new assets
(d) Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
(a) Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, well details about compensation (if any)?
(b) Answer: [NA]
(c)
Justification: No crowdsoureing
(d) Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
(a) Question: Does the paper describe potential risks incurred by study participants. whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
(b) Answer: [NA]
(c) Justification: Out of scope
(d) Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equiv- alent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethies and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
(a) Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration not required.
17

---

(b) Answer: [NA]
(c)
Justification: There are such parts in our paper
(d) Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy
LLM) for what should should not be described.
18

---

A Symmetric initialization
This section supplements the results of Section 4.1 by establishing the ODE governing the evolution of cosine similarity (t) and the magnitude r(t) for each normalization scheme. While Theorem (3.1 guarantees convergence to point mass from almost all initial configurations, we need ensure that (()) + 1 from a symmetric initialization as it approximates random initial configuration when the embedding dimension d is large. Below, the ODEs governing the evolution of 7(1), that the form of (t) = 2(0x (t)) can be derived using basic substitutions and we omit these details. Moreover, since, Mix-LN is simply combination of Post-LN and Pre-LN, the initial and terminal velocity in this case follow directly.
Post-LN. The ODE governing the evolution of the cosine similarity 7t) was already derived (Geshkovski et al. 2025 Theorem 6.8). It is given by
{(1) 200r(0) (1 (())((n - ()((1)+1) +1)
((n 1)e8hlt) + e
At t = 0, (t) = 0 and it is known from the aforementioned theorem that (t) + 1 as - cc. In fact we readily see from the ODE that 71) monotonically increasing. Writing e(l) 1 -1()), get é() ~ (1). It yields
2
(t) ~-+0
((1) ~- 22
の + n
Pre-LN. The ODEs governing r() and (()) are given by
+(t) (n 1) ()e + e (n 1)00((() eB
and
1(1) 2007 (1 (1))((( ())(((11)7(+1) r(t)((n- - 1)e + e²)
(1)
Note that y increasing so 7(t) 2 (0) for all positive Hence,
2e
+(1) 2 r(t)ne (1 7()).
By Grönwall's inequality, we get
1- s exp (-
n (- ds r(s)
2e
r(8) x: l 0o. Hence 7() + 1 and, in turn, A(t) + so tha rr()) ~ t as - xc by I'Hôpital's 's rule.
But since F S 1, we have r()) S l 1 r00) and 10 ds
Writing e() ((), we get E(l) ~ -2((()/((() ~ -2((1)/L It yields that
it) ~ r 8 + n 1))
7(1) ~1000 73'
Peri-LN. The ODEs governing r(t) and 7(t) are given by
(n 1)) + e 8
r(t) Ve20 + 2(n 1)e + (n- 1)) (1))))())))
and
2e (1)))))))))))(( 1)((t) +1)
(t) r(t) r(t) 22 + 2(n 1)08(4+7(0))) + 1nn11)e28000 ()c20x()) (n 2))(1)
For two function a(t) and b(t) and 1' (0, oo). write a(t) ~e-T b(t) if (t(()//(( - 1 a.
19

---

The argument follows the same lines as for Pre-LN. Indeed, we have
2e
2e
+(1) 2 r(1)eB v 1 (n 1)2
(1 -x(() 2 r(t) (1 >()),
and hence
2e ds
- 7t) S exp
ne
r(s)
To show that i < in this case too, we employ a coarser approximation that is sufficient for our purpose:
ne
((t)
n. + n2
Vea
It readily yields that (t) + as t + co and in turn that r(t) ~10000 t. Hence,
C *(t) ~1+20 $3*
{(t) ~1+0 r(0) 20 +n
nGPT. The ODE governing 7(t) is given by
87(0 (1 20
(()) (m) +1)
7(t)
Ve28 + 2(n 81+ +700))))))) + (n - (0) (n ())(
This is the same formula Peri-LN where r(t) is replaced with oj Hence,
(-22)
2e
/ a,ds
1 - -t) exp
(=ne
neB
Assuming that or is chosen such that the above integral diverges as + co, we get that 7(1) + 1 as 1 It yields
H(t) ~--+
200
i(t) ~10000 Cafe 25 a,ds
Vc28 + n 1
sqrt-scaling. The ODE governing (() is given by
(t) 2e (1 ((t)((( - (t)+++1))))) Vt + 1((n - 1)e + e²)
Observe that the cosine similarity evolves precisely ① but with predetermined magnitude r() vt + 1. In particular, get that (tt) - 1 as
We readily get
e-vv AVT
{()) 000 Φ + n
*(1) ~0100 C
V
B Proof of Theorem 4.2
Here we prove an upper bound on the initial attention vector. Assume 3 log n 2 d d llog n. 1, i.i.d. random uniform 0y. Then
lIQ K|lop, ||V]
Vj e [ (m) S C log
2 21-n
Proof. Throughout this proof, C' > denotes universal constant that may change from line to line.
Fix token j without loss generality, j n and work conditionally on f Define the random variables:
X 0 Q Kθ., k 1,...,n.
Our goal is to control the norm of the vector
A, V Ei- ex
Σ Xx
20

---

Since we assumne that | I 1, we may assume without loss of generality that V = Ia.
Let w denote the probability vector given by wk x e and observe that
wkOk - シ 0x XmOx
X )0
[wy
(2)
k=
Since the O&S are i..d. centered and subGaussian with variance proxy C/d, we get that with probability least n -C
1
Σ
0ell - C.
logn
(3)
n
Moreover, observe that for any l: S n, we have
C X20A|| -
d
Hence, by vector Hoeffding, with probability least 1 n also have
Σ Jam
XxOx|1
C + C n S d
logn
because we assumed n log 2
We now control the third and last term in the right-hand side of 2 and observe that X,, is deterministic and with norm most 1. For k S n 1, the random variables X i.i.d centered and subGaussian with variance proxy C/d. Hence there exists an event E, with probability at least 1 n C, on which
log
max [&&l S CV ASn-1
Since eva E, it holds for all A 1,
log
le 1 1- Xa 1
e- & Ne S c. Together, these bounds yield that Moreover, we have that [x,] < so that
1 1 X
C log
1 + Xx + + log
S wi
n 1 k
n 1+e
that
X log n nd
Wk
C
where we used the fact that n 2 Vd. Moreover, using similar arguments, we also have
X
[wk
2
n
n
Put together, the last two displays yield
X X
log
wk
c
m
Combined together we get the claimed estimate.
C Proof of Theorem 4.3
Denote average 0 = Eh= Oz- Consider variance of tokens
1 n 71 10
V(t)
012
1
1=1
21

---

Then
V( =
2 1
Σ 10k 0, 0x
1
Σ
n
0,)
71
j=1
We immediately have
1 n
Σ
10
0, 7
Σ
0,) Σ
02 nd,
Σ
0,) 0.
11 j=1
j=1
Thus
(t) 2 Σ
2
n Σ
(0x 0.,00
10% 0, Ph Pk)
n k=1
Sk
k=1
Let's decompose 6% A 02 to get
n 10
2
1
(1())
P&0)
+
10 0, P&Ox) = I1 + 12.
8k
11 k=1
8k
k=1
For the first term we write
11= Sk (Op - PxO)
Σ
(-0,
0
(0k,
0)0k)
Σ
1
(10x, 0)²- 11011)).
Sk
8k
k=
k=
Each term in the sum is non-positive, thus we can bound
n
(00.0)=-1100)
2 2 minl 8k
1
n
(10x, -000-10113))
IIXX SR
The sum itself can be written as
Σ C10R, ( 11] = Σ (0x ()²,0)² m|1011 = > 1(x,, 0)² -(0z, 0)) k=1
k=1
R=1
since En k= (0z, 12 n (1). With fixed sum of (Ox. 0). the min/max sum of squares (Os, 0)2 is achieved when they equal/spread out, which gives
max (-2V + 2n\²) 2 1 ming Sk
+ 222²+
For the second term, we first upper bound the length of Psog. To this aim, consider
(Q0x, K0,) (Q0k, K0,) (0k, Q K(B,- - 0,)) - | Kllop 11, 0 s 110,-0,) S v28. Consequently,
1
((QO
ne 28
35
Which implies
Wki
1
BtQ0,, K02)
1
<
=
1
1600220 -1.
71
n
Therefore,
> (ukk |Po0Bl n
71
1 (c6V288 -1) Σ |Pr0 S
(c8V20 -1)/n
Σ PP
Finally, one has
12
n
1 n
1100,12 - (1 (1- (0,, S n
- > 0 02))
71 j=1
j=1
< n(1 - (1 - 2)) < 2n
22

---

Combined, we obtain an upper bound
T2
n
0
P
2
1 1
(e 1) 2n
n
1 81
71
8k
n ming Sk n
2
(6
1)/2v
1 01 =2 ming 8k
V2n(e 1)
ming 8h
Thus, obtain upper and lower bounds on (1( Ii 12 in terms of V.
-2V + 2n1² + 2V2n(e8v28 -1) V (t( 22 + +22 - 22/2(e
V. (4)
maxe 8% Let us also mention that
ming 8K
mink 8%
28 = max kjj
|0 0 < 4 max 10 011 4nV,
whereas
1 y= (0
8, i.c. V VS 8.
Therefore, the true local rate of clustering that we get from bounds (4) is defined by the main terms on both sides 22V/ 8k and -2V/ ming $Ą- Moreover, V - 0, min Sk max 8g, obtain a tight rate of convergence. To establish the result we claimed, notice that for < 10 one has
V2 3/
2 2n 3V26 1))
2n S 22nVV,
giving us
-2 2nd v2/(3//)
-2 v2/(3/n) V.
v V
(5)
max; Sk.
ming
Finally, we finish the proof with trivial estimates on sg. that follow from the fact that all products (0x, 0,) 2 - and definitions.
For Post-LN 8% 1. for Pre-LN 2 2 (1 8)t.
for Peri-LN t 2 2 (1 8)3/2t
for nGPT 2 2 (1 8)
for Mix-LN t 2 $; 2 (1 () -
for sqrt-scaling 8 = VE
Substituted into the estimate 5). we obtain the claimed rates.
Remark 1. The true local rate of convergence of as I - oo that we get from equation is
V e 20(1+((1)) for Post-LN,
V ==2logg((+0(()) for Pre-LN,
V e (logg((+o())) for Peri-LN,
V 2 a.do(1+)(1)) for nGPT,
V e 2log ((1+)(1)) for Mix-LN,
V --Vv(((+o(1) for sqrt-scaling.
D Final convergence
In this section we prove Theorem from the main text, that claims that under some assumptions, for almost any initial configuration of particles, any normalized attention dynamics that we study (that is Post-LN, Pre-LN, Peri-LN, nGPT, Mix-LN and sqrt-scaling) converges to a single cluster. First, let outline the core of the proof.
23

---

D.1 Proof Outline for Token Synchronization in Pre-LN
conventional proof that all tokens converge single state consists of two stages. Showing that there is some limiting configuration of tokens, and then verifying that the only possible limiting configuration is the consensual one. We follow the same approach, but at each step we introduce novel technical details due to our general point of view. For simplicity of exposition, in the outline we follow Pre-LN case.
Existence of a Limit Point First, we demonstrate that the token dynamics indeed converge to a limiting configuration. This step heavily depends on the system. Common approach leverages the Lojasiewiez inequality, as seen in Geshkovski et al. (2025). It can be adopted to our setting, as we will show later. Moreover, our proof extends the gradient case Q K to more general case, extending the synchronization results by Geshkovski et al. (2025 Criscitiello et al. (2024). even in Post-LN case.
Local behavior at the limiting point. Second, we must prove that any such limit point corresponds to the synchronized state where all tokens are identical. The classical argument involves local stability analysis around the system's ' critical points. One can typically show that any non-synchronized critical points are unstable and that their basin of attraction has measure zero, making them insignificant as final states. A comprehensive linearization analysis can be found in Crisccitiello et al. (2024) that, in particular, covers Post-LN dynamics with d 2 3. Together with recent proof of synchronization for d = 2 Pollynnkky et al. (2025), the stability of Post-LN system Jacobian is well-studied. We also rely on this method, but first we need resolve the fact that Pre-LN system is non-compact.
Transformation to compact state space. The Pre-LN state-space is non- non-compact, because both empirically and theoretically tokens' magnitude ry grows to infinity with t. This restricts the direct study of the limiting point in that space. We can transform it to a compact state space by the following trick, however, Consider logarithmic time scale = In and modified scale variables qj = 8j/t. Applying the chain rule, we find the transformed dynamics:
d0, - P)),((0)
dr 1j
dq; (0,, A,(0)) 93-
dr
This formulation is interesting in its own right. reveals that the Pre-LN system evolves on a logarithmic time scale, which may explain its observed stability advantages over Pust-LN variants in deep architectures. Furthermore, the dynamics are scaled by qj. which are driven toward (0,, (9)),, the alignment between a token and its attention vector.
Crucially for our proof, this transformed system is still autonomous. This allows us to proceed with the final step: rigorous linearization analysis of its critical points. By showing that all critical points corresponding to non-consensual states are unstable the (0, g, T) frame, we can conclude that the system must converge to the state where all tokens are identical.
In what follows are going to cover all the proof steps in detail.
D.2 Generalized gradient descent convergence
First, we need to refine an important result of Lojasiewicz on convergence of gradient descent, so that it fits our problem setting. We follow an approach similar to the one presented in Haraux (2012).
Lemma 1. For any 0, let M(I) be a symmetric real matrix CX(()T M(I) X()) with
A(t) > O, fo A(t)dt oo, and some constant C. Let energy function E(x) be analytic in open set U c R. Consider a compact path x(() C U that satisfies the following modified gradient
descent equation
M (ttV, Then, x(t) converges a critical point of the energy function x(t) - x- such that VE(x*) 0.
Proof. Step I. Change of time. First, define a new time variable r(t) = So A(s)ds. By assumption monotonically grows to infinity as t + Moreover,
dx. dx/dl M F(x)
dr d
A(t)
24

---

Take M(r) = M(((T))/X(((r)) Then
dr dr = -
with CT > M()) > T. This change of time proves that it is sufficient to prove the Lemma in its initial form under the assumption CI (t) > whereas x(t) corresponds time change.
Step 2. Now that we have CI (() > I, let us follow a known approach to the proof of gradient descent convergence. Consider the energy along the trajectory. i.e.
f(t) E(x(t)).
Then
('(1) - V = (2) -
In particular, ft(t O, the energy is decreasing along the trajectory. Since E is bounded compact trajectory, we get that (t) E [10,, oo)). Because
| < (t))
we get that & E L2( oo)). This implies that i - 0, because à is an absolutely continuous function (10, oo)).
Therefore, since (1) > cT, we get that V. F(x) - 0. For convergence to a point this is not enough, but it already shows that dist(ar, E) where & = {a V 0}. Then, because the limit set T compact trajectory æ=t)) is compact and connected, we can use uniform ł.ojasiewicz. inequality.
To get + x² need to sharpen the estimate á. This is where the Lojasiewicz inequality is used. It says that in some neighbourhood 12 of and some constants V, a one has
[F()) VI < [[V
We can assume V by shifting the energy function. particular, it means that (t) decreases to as + co. Moreover, because zz(t) approaches I as t - o0, we know that as t - 00 it is true that
[E(x(t)) - [[VE(((t)]
Therefore, from our assumption M())= > we get
('(1) - (V.bll))))))) -VV,El=() ( 2l) S --I1 --100))
Then
(1-20(t)) = (1 - 2a) 21 2a 2 (2a - 1).
Consequently, for B = 1/(2a 1) one has
f(t) S Kt-D
We know that
1 clf((t) -Cf(tt).
Then
28
| ds < C(f(t) - f(28)) & C
From this inequality and Cauchy- Seech--Schwazz we get
Ldds CK((1-8/2
Finally, this estimate shows convergence of the path æxt) to some limiting point, because
j < CK
21(1-8)/2 < xc.
n=U
25

---

D.3 Proof of Theorem 3.1
In this appendix we provide complete proof of Theorem 3.1 The argument follows the roadmap outlined in Section[D, D.I] with minor adjustments for each normalization scheme. For simplicity of exposition, we first give full analysis of Pre-LN. Then, we provide remarks on how adapt the proof for each normalization scheme. Thanks to our unified formulation of normalization in (NA). the core proof applies verbatim across schemes- only variation lies in some technical details. A forthcoming work will pursue that broader unification and extend the analysis beyond purely architectural speed regulators.
For convenience, let us recall the object of study. We consider the evolution of particles 03 on a unit sphere gd-1 governed by the ODE
1 0, - P
((O)
VO
B(Q8, Ko,)
8j
DE=
k=1
with normalization factor sn evolving according the following table.
Table 3: Speed regulation factors
st(t)
(t
Post-LN
1
Pre-LN
Ty(t)
()(())
Mix-LN
Hisr + ry (()
(0()
Peri-LN
r1(0)lA)(((00)
0110,,(0000))))1000000
nGPT
of (14 v +1
sqrt-scaling
Proposition 1. Consider monotonically growing to infinity time change Tt)). Then, normalized attention dynamics with speed regulation factors sy(t) equivalent normalized attention dynamics with speed regulation factors (T(T) = s;(t(T))/tt ((((T)))((((r))) in time
Proof. This immediately follows from the definition
d0; d0 dr ('(7)= s,(t(T))/U(T) POA,
This proposition shows that in the normalized attention dynamics can divide sj by the same factor, as long as it's positive and its inverse integrates to infinity. This notion helps us reduce time dependence in normalization dynamics.
Proof. Step Time change
Consider evolution starting at time / = and time change == In so that di/dr L. Moreover, set (j = (1)/t. Then, we rewrite Pre-LN time T as
0,(7) = (T( (O()))
( = (7) - (T)
The function (0) A, is continuous and thus bounded on the compact. Then, all gj upper bounded from the equation, and thus evolve on a segment [0, Q]. This frame change is important, as it allows study autonomous system usually has r - oo. Moreover, the condition
compact, whereas in the original coordinates one
inf inm inf fy > 0
1-000
implies that all magnitudes ry are lower bounded by some linear function at - co, which translates into
inf inf (T( > 0.
26

---

Step 2. Gradient- structure. We consider the event {infy inf, 9j > 0). It is enough to show synchronization under this assumption to prove the result. First, to show the convergence of the system to some limiting configuration of angles @ we use Lemma] 1 For any trajectory ((r) we can write
0, 1 Ve, E(O)
2
with spherical gradient of the following energy function
E(O)
Σ
310
23
j.k
To get convergence of a specific trajectory (7)) to some critical point O" we need verify that the time-dependent matrix M() with diagonal blocks 2;2; satisfies the assumptions of Lemmal This is true, because the blocks are uniformly bounded. Indeed, the function Zj is uniformly bounded as continuous functions on a compact. Whereas yj are uniformly bounded on any trajectory we consider, with {infy inf, Qj > 0}.
Step 3. Local behavior We consider the event inf, inf, qj > 0 and ((T) O Our goal is to show that when the limiting point is not Of = =... = 0n this event has probability zero. We can split the event into countable union with assumptions (T) 2 1.
(inf inf oj 0} C
U (Ve mER30
0 1
As we already mentioned, dj are bounded from above. This means that under the restriction qj 2 1 the combined state space of (O,g) q) compact manifold. Our goal is show that the event
(Aj
[m] Vt 0 gj
Ju E [n] 0, - 03 O not synchronized] m
1
has probability zero.
When we get an autonomous dynamical system on a compact manifold, and we study its convergence to limiting point, need study the Jacobian at that limiting point. Specifically, well- known stability argument that was already written down several times (see Criscitiello et al. (2024). (Geshkovski et al. 2025 Lemma A.))), employs central manifold theorem to show that basin of attraction of unstable critical points has measure zero.
This argument applies to case. Therefore, move studying stability of critical points in the next part.
Step 4. Unstable direction of the 0 part Consider the dynamics in the form
8; = ()(T)g;(() Vo, E(O),
4 f) 9j-
where 93 Z, and fy (0, (9)) for Pre-LN. In order to show that all limiting points (9", ,²),") > O that are not fully synchronized (i.e. not all 0g are equal) have measure zero basin of attraction, we only need to check that they are all unstable. More specifically, that the Jacobian matrix at any such point O", 1² has an eigenvalue with positive real part. Because of the specific form of our system, the Jacobian has a convenient block form
Jag
Jgp Joo
J=
(Jo Jeq
where Jgg = -,, and Jog 0 because at the critical point V E(e") 0. Therefore, it is enough to show that Joe has a positive eigenvalue. Because of the gradient-like structure, Jno is the product of two matrices diag( (1(00)91(('))."
J.. F. (er)g.((r)) and symmetric Hessian of the energy function E. The Hessian itself is unstable, this is established result due to Criscitiello al. (024) (for d 2 3) and Polyanskiy et al. (2025) (for d = 2) that together closed synchronization for Post-LN.
Surprisingly, this is enough for our cause, because of the following matrix property, that shows the product of the diagonal matrix and unstable Hessian is again unstable. Note that the lemma not true without the symmetry assumption on A.
27

---

Lemma 2. For symmetric unstable matrix A and symmetric positive-defintte D, the product DA also unstable.
Proof. First, because D is symmetric positive-definite there is symmetric positive-defintte square root P. ie. p2 = D. Consider symmetric matrix B PAP. Notice that
P-P PAP B,
thus matrices DA and are similar, i.e. they share eigenvalues. On the other hand, by Sylvester's law of inertia, B and A have the same inertia, and in particular the number of positive eigenvalues. Therefore, B has a positive eigenvalue, and so does DA
Step 5. Basin of attraction of unstable critical points.
well-known that the set unstable critical points of dynamical system on a compact has measure-zero basin of attraction (see for example (Geshkovski et al. 2025 Lemma A.1) for proof outline). Thus, we obtain that the event
(Vj [n] 1A 0g > 1 Ju [n] -03 10"s not synchronized}
has measure zero.
For completeness, we include the proof here. Let o, (zo) be the flow for the system à = F(z), where (θ,g)- The vector field F(x) smooth on the open domain where all qj > For any fixed
m > 0, we consider the dynamics on the compact manifold
Mm == (S [1/m, Q)",
on which the flow is smooth. Let Km C. Mm be the compact, forward-invariant set of initial conditions whose trajectories remain in Mm.
Let Sns C Km be the set of non-synchronized critical points. By Step 4, every point xf Sns is unstable. Let Am, C Km be the basin of attraction for Sns. i.e, the set of zo E Km such that
lim 4, (1o) E Sns-
For any x² Sus. which lies in the interior of Mn, the Center-Stabll Manifold Theorem applies. It guarantees the existence of a local center-stable manifold Wine Wloe(*", (z*). The instability x implies that
dim (Wlorr (x²)) di.(M,..) 1,
so (x"" has measure zero. From the Center-Stable Manifold Theorem, there a neighborhood of x² such that any trajectory staying in this neighborhood has to enter and remain on (x*). By choosing finite covering of the compact set Snx with respective neighborhoods of x², we get that any initial condition Zo E Am,us has a trajectory (xoo) that must eventually enter and remain on some Whee (zt), with a finite number of r k < K chosen from the covering. Thus, for some N 24, k < K, have
Tg = t n (2x(2z)) where (xo) Wlle (x())
Since I_N is a local diffeomorphism. it preserves the dimensionality. Manifold W has positive co-dimension, and thus its pre-image too, which implies that has measure zero in Mm Consequently, measure of Am,ns is also zero, as a countable union of measure zero sets. Finally, to completely finish, we need to map the set to t = O, because 7 = 0 corresponds to 1. This is again smooth backward flow that preserves measure initial coordinates (9()), r(0),, because they are distributed with standard Gaussian (9)).
set. We arrive at measure zero set in
Remark 2. Here we describe modifications of the proof for each scheme.
Post-LN No time change required. The system is already autonomous and compact. Step 2 works with modified gradient descent from 7 because the modification matrix (t) is diagonal with blocks Z, that are uniformly bounded. As such, we get convergence to some critical point. Finally, we use existing analysis of the stability of the energy functional together with 2 to establish synchronization.
28

---

Peri-LN For the Step I we use time change 7 := In t and also consider 9y(t) == r. (t)//tt It leads to the dynamics of the form
1 4P () (O), i [A)())]I 4j-
(0),A(())
The rest of the proof remains the same Pre-LN, because this system satisfies gradient-like structure of Step 2, we also use the assumption to separate qj from 0, and then show that all critical points that not synchronized have unstable direction in Step 4. The form of the system and the Jacobian in Step 4 is written generally, to accommodate this case too.
Mix-LN At infinity Mix-IN follows exactly Pre-LN, and the argument follows from the proof of Pre-LN.
nGPT Time T T
from Step 1 simplifies nGPT to the case 0q = This makes the original dynamics autonomous on a compact manifold. As such, it requires no frame change, and immediately move that system. Modified tojasiewicz from Step and analysis of the unstable direction of the Jacobian from Step 4 follow similar steps. For the Jacobian, the only component is Joo, and it is unstable from the same Lemma. The only complication, for the system are points with Aj 0. They however, break the original dynamics too, and can be excluded with careful analysis.
studying convergence and local behavior of
sqrt-scaling Time change from Step with T = 2 +1 reduces sqrt-scaling to Post-LN.
E
Simulation results with random weight matrices
Attention Update Formulation. To align our simulations with practical transformer architectures. we now explicitly include the output projection matrix, W E Radd in the attention update. For multi-head configuration, the output of each head h first computed and then concatenated, after which the final projection is applied: O₂ softmax K7 X
1,.... Thead X = Concat(O) (OnccttOO,.......... O mes) )W where Qn Kn. Vn, E Rdxdde The inclusion of the matrix W is a linear transformation applied after the core softmax-
interaction. While this is crucial for model capacity in practice, it does not impact the theoretical dynamics description, which is why it was omitted from the preceding theoretical analysis for notational simplicity.
Experimental Settings. We present simulation results illustrating the evolution of average token cosine similarity. All plots show the mean trajectory averaged over 10 independent runs, with shaded regions indicating the 90% confidence interval. Each run begins with a fresh draw of initial token positions X from an isotropic distribution and random weight matrices. All simulations use context of n = 128 tokens. For the normalization methods Mix-LN and nGPT, we use parameters respectively.
0.25T and a
Our plots vary several factors. The majority of our experiments use Kaiming initialization. In this setting, we fix the number of heads to Thends 1 (so dhead d) isolate the core dynamics. We systematically vary the following parameters:
Dimension (d): small (16), medium (128), and large (512).
Temperature (3): low (B 1), medium (B Vd), and high (3 (3= 4v
Weight Sampling: static single draw of Q, K, V, W fixed for all time steps) vs. re- sampled (new matrices are drawn at each time step At).
GPT-style Initialization: We conduct one experiment that mirrors the configuration a small GPT-2 style model.
It uses d = 768, Theads = 12 (implying duewd = 64), and a temperature of B v /dheal-
Weights are drawn from a Gaussian distribution with variance 2 0.02 and are held static.
Figures arranged to facilitate comparison, with each caption specifying the experimental signature (d, Thecads 3, weights, init).
29

---

Do sendarty with corsdence rtaraa
Cosiee sirilriy with corficanee Hoervdd
CEIT
(a) d = 512, Theads = 1, vd (medium), static
(b) d = 512, Naeods 1, 3 4vd (high), static
Kaiming weights. Case where d
Kaiming weights, Case where d >
sirillatty with carrience sterves
Costnn similerit conficarca Iitersll
(GFT
0.2
170
65a
(c) 512, Tibeads 1.3 vd (medium),
(d) d = 16, Thiads = 1,8 = 1, static Kaiming
sampled Kaiming weights at each At
weights. Case where d < n.
33
1.0
02-
0.
3.
20.2
(e) d 128, Thheads = 1, v (medium), static Kaiming weights. Case where d n..
(f) 128, Theals = 1, = 1yd (high), static Kaiming weights. Case where d = n.
Coalnn sim larty with confiencc interaaa
06
(g) NanoGPT-style::
768, Theads
12
(chesd = 64), 3 = / /head static Gaussian weights
with o = 0.02.
Figure Evolution average cosine similarity for tokens under the pure attention update.
30


Reviews:
[{"review_id": "P7Vb1tXPux", "reviewer": "Reviewer_cqGC", "summary": "They conduct a theoretical analysis of how the radial component of token representations evolves with transformer depth and as a function of the layer normalization scheme used. They compare several different layer norms: pre-LN, post-LN, Peri-LN, nGPT, Mix-LN, and LN-Scaling. They find that Pre-LN makes better use of depth, as tokens continue to evolve across many layers. Moreover, nGPT has the advantage of being able to adapt the rate of representation evolution / clustering.", "questions": "Is it reasonable for future work to conduct similar analysis without as many simplifying assumptions on the architecture e.g. no MLP / restricting the attention?\n\nIt seems that a lot of your findings match empirical beliefs in the field, are there any results where there is a mismatch from empirically derived preferences practitioners have for LN schemes?", "limitations": "One thing that seems to be missing is a prediction or improvement to existing methods that would result from this analysis. Otherwise the authors seem to do a good job of listing out limitations in the paper.", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper is a bit outside the area of research I have worked in for the last several years, so this one is pretty difficult for me to evaluate properly. I don't have the ability to rigorously evaluate the theory, which is probably the most important piece here. And as for the validity of their simplifying assumptions, I don't have a strong sense here either (e.g. removing MLP and restricting the attention KQV).\n\nI will say that the paper seems to be well written and directionally I think it is important to have a stronger understanding of how these decisions shape the NN dynamics.\n\nThe takeaways from the paper seem to match empirical findings (e.g. Pre-LN is preferable to Post-LN), which is a good sign.\n\nOne thing that seems to be missing is potentially a prediction or improvement to existing methods that would result from this analysis.", "was_revised": true, "final_justification": "I appreciate the author's response and it seems like solid work, but my ability to confidently asses the work is low as it is outside my main area of research.", "rebuttal": "We sincerely thank the reviewer for the strong support and for recognizing the importance of understanding how normalization decisions shape neural network dynamics.\n\n**On assumptions:**\nOur assumptions follow field conventions (Geshkovski et al.) but are actually quite flexible:\n- **Theorem 4.2**: Handles **arbitrary Q,K,V** \n- **Theorem 4.3**: Arbitrary Q,K with V=I_d\n- **Theorem 3.1**: Presented with Q=K=V=I_d for clarity, but extends to Q^TK=Diag(λ_i)V\n\nIn the revision we updated Figure 4 to validate predictions with Kaiming-initialized random matrices, multiple heads, and varying parameters - showing our insights generalize beyond the theoretical assumptions.\n\nRegarding MLPs: Following standard practice, we focus on attention-driven dynamics. MLPs add drift terms that don't fundamentally change normalization comparisons.\n\n**On predictions and improvements:**\nOur analysis yields concrete insights:\n1. **Peri-LN is optimal** - balancing early/late layer dynamics (used by recent Gemma 3 and (with modification) OLMo 2)\n2. **Design principle** - Future architectures could use custom speed regulation s_j(t), or employ the framework to help with the design. For example, current nGPT paper uses trainable $\\alpha_t$, with initial value empirically picked. However, their initial value would work poorly at scale, whereas from introduced analysis they could tailor what $\\alpha_t$ to use at scale.\n\n**On empirical alignment:**\nOur findings confirm Pre-LN > Post-LN (well-known empiricaly wisdom) but also suggest Peri-LN > Pre-LN - a prediction validated by several recent production models, but still not widely adopted.\n\n**Looking forward:**\nWe hope our framework would help to bridge the gap between the theory of token propagation in transformers and practically employed models. As noted, introducing the term induced by MLP layers is a natural step for further research(understanding its exact impact on the clustering behaviour).\n\nThank you for your support and thoughtful questions!"}, {"review_id": "YiVJGT9zwY", "reviewer": "Reviewer_KEPZ", "summary": "This paper presents a theoretical analysis of different layer normalization schemes in transformers by modeling the evolution of token representations across layers as a continuous-time interacting particle system. Within this framework, various normalization methods are characterized by a “speed regulation” factor that modulates the dynamics and affects the phenomenon of representation collapse. By comparing the speeds at early and later layers, the authors identify Peri-LN as a favorable compromise.", "questions": "- Cited works (e.g., Geshkovski et al., 2025, Figure 2) exclude the d=2 and intermediate beta case from their clustering theorems. Does Theorem 3.1 guarantee clustering even in these regimes? Furthermore, to improve clarity for the reader, it would be beneficial to add a reference in the main text to the discussion in the appendix regarding the assumption s_j' ≥ c > 0, as it appears to be a key condition for the theorem.\n- Generalizing the results to arbitrary $Q, K, V$ matrices is clearly challenging. However, considering the simple case where $Q=K=I_d$ and $V=-I_d$ could provide valuable insight. In this setting, the dynamics become repulsive, encouraging tokens to spread apart rather than cluster. Do the paper's conclusions about the effectiveness of different normalization schemes, particularly the advantages of Peri-LN, still hold in this repulsive regime?\n- The inclusion of an MLP layer would introduce a drift term into the dynamics, independent of the attention mechanism, which would likely have a strong effect on $r_j(t)$. What is the authors' intuition on how this MLP-induced drift would interact with the different normalization schemes?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n- The paper's main strength is its unification of various normalization schemes into a single dynamical systems framework. By modeling different schemes as \"speed regulations\", it provides an intuitive basis for comparison.\n\n- This framework explains several empirically observed phenomena, such as the superior stability of Pre-LN over Post-LN in deep networks. The analysis of initial and terminal velocities offers insights into how different schemes utilize early versus late layers, leading to actionable conclusions about architectural design.\n\n- The work rigorously extends a recent line of theoretical analysis (Geshkovski et al.). While based on a simplified model, the paper's conclusions are theoretically sound and provide an interesting starting point for future research to build upon, potentially by relaxing assumptions or incorporating more architectural components.\n\n**Weaknesses:**\n- The model is still quite simplified (see questions 1 and 2 below). I understand that it is mathematically very challenging to extend the results, but some more discussion about different values for Q, K, V and the inclusion of the MLP could help the reader.\n- (Minor) The presentation is clear but there are several unclear phrases or typos, for example:\n  - Line 91: At(X) = [X11(X), . . . , Atn(X)]\n  - Line 122: \"we readily get: Note that\"\n  - Line 126: \"(a.k.a)\"\n  - Line 136: \"I particular\"\n  - Line 214: \"which tokens a pre-clustered\"", "was_revised": true, "final_justification": "Assuming that the additional experiments discussed by the authors in the rebuttal are consistent with their claims, and that the final version will strengthen the connection to relevant results in the literature, particularly in the supplementary material, I believe this paper could help increase the community’s interest in an emerging mathematical framework for studying transformers, while also demonstrating a concrete application. This is despite the existing limitations, which I recommended the authors clearly state in the paper. For these reasons, my final score is an accept.", "rebuttal": "We sincerely thank the reviewer for their careful reading and insightful questions. Your technical understanding of our framework is much appreciated.\n\n**On typos and presentation:**\nThank you for identifying these errors. We will fix all of them in the revision.\n\n**On clustering in $d=2$ and intermediate $\\beta$ (Question 1):**\nGreat question! Yes, Theorem 3.1 works in these regimes, because it uses a very recent result, that completed the proof for $d=2$. We clearly state that in the revised appendix. \n\nThank you for the suggestion, we are going to add a discussion about that condition. It is mainly technical, and we will present it in the way easiest to analyze. Actually, the condition can be omited for PostLN, nGPT, LN-Scaling, and modified to $\\dot r_j ≥ c > 0$ for PreLN and PeriLN, which is easier to understand given the Table 2 definition, we'll modify it for clarity.\n\n**On assumptions - less restrictive than they appear:**\n\nOur theoretical framework follows established conventions (Geshkovski et al., Sander et al.), but in that scope it is actually quite general:\n\n- **Theorem 4.2**: Handles **arbitrary $Q,K,V$** (only requiring $||Q^TK||, ||V|| ≤ 1$)\n- **Theorem 4.3**: Arbitrary attention with $V = I_d$\n- **Theorem 3.1**: Can be immediately extended to $Q^\\top K = Diag(\\lambda_i)V, \\lambda>0$, generalizing even prior work (Geshkovski et. al. assume $Q^\\top K = V$). The case $Q=K=V=I_d$ is presented to simplify exposition of an already complicated proof. \n\nThe **Theorem 4.1** with Q=K=V=Id case serves as an analytical baseline, later modeled in Figure 2a. We updated Figure 4 to model the dynamics with:\n- Kaiming-initialized random matrices\n- Multiple attention heads\n- Time-varying matrices\n- Various temperature parameters β\n\nIn all those cases the qualitative behaviour is the same as in Figure 2a (current Figure 4 can be misleading, because it uses $d=2$).\n\nRegarding MLPs: These add token-wise transformations that don't affect relative comparisons between schemes too much. Including them would be an interesting line of research, but it also might obscure the key insights about how normalization specifically impacts token mixing. We think this makes it a topic for another work.\n\n\n**On repulsive dynamics with V=-I (Question 2):**\nFascinating question! With V=-I, the dynamics is governed with repulsive interactions between tokens, instead of attracting ones. Let us quickly note, that empirically the heads with negative $V$ matrices are rare (though exist), arguably their behaviour is to mitigate representation collapse, and in this scope they should be studied as a part of the system. Analyzing pure $V = -I_d$ case is possible in our framework, but it does not directly fit the clustering perspective (Theorem 3.1 and Theorem 4.3), and changes the system in more ways than one, because token magnitudes $r_j$ would evolve differently. As such, it is not clear if in this case PeriLN remains the best choice, but practically it could appear as only one head out of many, not having that drastic impact (thus PeriLN still would be best).\n\n**On MLP-induced drift (Question 3):**\nAs correctly noted, MLPs add a drift term to the dynamics. Indeed, it has a different impact on the dynamics, introducing additional attraction in specific directions. This term does not impact the evolution of magnitudes $r_j$ drastically, the order of growth remains the same. As such, the core intuition behind comparing all schemes persists.\n\n**Final words**\nAll in all, we want to highlight that our assumptions might appear more restrictive than they are, especially when compared to common practice in the field. We use $Q=K=V=I_d$ in Theorem 3.1 for convenience, while the proof is correct in a more general case. The same assumption is made in Theorem 4.1 because it serves as a complete analytical baseline, while Theorems 4.2-4.3 are much more general.\n\nOther than that, thank you for all the questions and highlighted points for improvement, we hope we answered the questions and we are going to use all suggestions in the revision."}, {"review_id": "V968sBQcdg", "reviewer": "Reviewer_rFjq", "summary": "This paper provides a theoretical analysis of various normalization schemes in transformer architectures by modeling token representations as interacting particles on a sphere. The authors reframe different normalization methods (Post-LN, Pre-LN, Mix-LN, Peri-LN, nGPT, and LN-Scaling) as variations of a unified interacting particle system where normalization acts as speed regulation. Through this framework, they analyze how different schemes affect clustering dynamics and representation collapse, establishing asymptotic clustering results and characterizing initial and terminal token velocities. The work identifies Peri-LN as particularly effective for maintaining meaningful token evolution across both early and deep layers.", "questions": "1. How do the theoretical predictions about different normalization schemes translate to measurable differences in LLM capabilities? Could you provide experiments showing how Post-LN vs Pre-LN vs Peri-LN affect reasoning ability, language understanding, or other downstream tasks? This connection between your mathematical framework and real model performance would significantly strengthen the practical relevance of your work.\n2. Given the substantial assumptions (Q = K = V = I_d, no MLP layers), what is your roadmap for extending this analysis to realistic transformer architectures? Can you provide preliminary results or concrete next steps that would make your theoretical framework applicable to actual LLMs used in practice?", "limitations": "Yes", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper provides an elegant unification of diverse normalization schemes under a single dynamical systems perspective. By casting all methods as speed regulation mechanisms for particles on a sphere, the authors enable direct theoretical comparison of fundamentally different approaches.\n2. The theoretical results are well-developed, particularly Theorems 3.1-4.3, which establish asymptotic clustering guarantees and characterize velocity profiles. The extension of gradient flow analysis to general normalization schemes beyond Post-LN is a significant theoretical contribution.\nWeaknesses\n1. The analysis relies on highly restrictive assumptions (Q = K = V = I_d, omission of MLP layers, simplified attention) that significantly limit its applicability to real transformer architectures. The gap between theory and practice is substantial, making it difficult to extract actionable insights for actual model design.\n2. The paper fails to demonstrate how their theoretical insights about clustering dynamics and representation collapse translate to actual LLM performance on reasoning tasks, language understanding, or other practical capabilities. Without showing the impact on real-world model behavior, the theoretical contributions remain disconnected from what practitioners care about most.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our \"elegant unification of normalization schemes\" and \"well-developed theoretical results.\" We're happy to clarify how our work bridges theory and practice.\n\n**On assumptions - less restrictive than they appear:**\nOur theoretical framework follows established conventions (Geshkovski et al., Sander et al.) while actually being quite general:\n\n- **Theorem 4.2**: Handles **arbitrary $Q,K,V$** (only requiring $||Q^TK||, ||V|| ≤ 1$)\n- **Theorem 4.3**: Has **arbitrary $Q, K$** with the only assumption $V = I_d$, to remove the drift and focus on clustering\n- **Theorem 3.1**: Can be extended to **$Q^TK = Diag(\\lambda_i)V$**, generalizing all prior work\n\nWe assume $Q=K=V=I_d$ in Theorem 3.1 for simplicity of exposition, so that the presented proof is not too overwhelming.\n\nThe **Theorem 4.1** with $Q=K=V=I_d$ case serves as an analytical baseline. We have updated Figure 4 to further show that our predictions (modeled in Figure 2a) can be observed with:\n- Kaiming-initialized random matrices\n- Multiple attention heads\n- Time-varying matrices\n- Various $\\beta$\n\nRegarding MLPs: These layers add token-wise transformations that don't affect relative comparisons between schemes that significantly. Including them would obscure the key insights about how normalization specifically impacts token clustering, guided by attention mechanism. This simplification is a deliberate choice following previous established theoretical analysis (Geshkovski et al., 2023, 2025).\n\nAnswering **Question 1**, the assumptions on matrices $(Q, K, V)$ are already not that restrictive, with the figures supporting that even more general changes (varying common architecture parameters) do not change the presented qualitative results. The introduction of MLP layers is an interesting avenue for further work (currently there are no proved clustering results with MLP layers), but it is not that significant for the work presented. MLP layers add a drift term to the dynamics, and one can still compare different normalization schemes with that term, but the impact on the **clustering** would be not as easy to see. \n\n**On practical relevance and empirical experiments:**\n\nThe reviewer asked about experiments comparing normalization schemes. We build upon extensive existing empirical work -- each studied scheme was introduced/analyzed with an empirical paper justifying its performance, for example:\n- Xiong et al. (2020): Pre-LN outperforms Post-LN\n- Kim et al. (2025): Peri-LN's effectiveness, the scheme adopted in Gemma 3\n\n**Our unique contribution for practitioners**: We explain *why* some of these empirical phenomena occur:\n- Post-LN's exponential clustering → representation collapse\n- Pre-LN's polynomial clustering → better depth utilization  \n- Peri-LN's balances initial and late velocities → optimal performance\n\nThis isn't just retrospective analysis, as originally our work was motivated by bridging recent Self-Attention dynamics research with a widely adopted practical usage of PreLN. However, the analysis presented showed clear advantages of PeriLN architecture. It was later we learned about its adoption in production models like Gemma-3 and (with modifications) OLMo-2.\n\n**On bridging theory and practice:**\n\nOur work does not describe full Transformer in all details, but it does not make it completely \"disconnected from practitioners\". We aim to connect the evolving theoretical field of transformers (started with works by Geshkovski et.al.) with the components important in practice. However, already the simplified analysis introduced, focusing on a specific architecture component, allowed to explain phenomena practitioners observe (representation collapse), and predict efficiency of PreLN compared to PostLN, and PeriLN in general, in a one unified approach.  \n\n**On theoretical contributions in ML:**\n\nWe respectfully disagree that theoretical work requires new empirical validation to be valuable. The transformer's core architecture (attention, residuals, normalization) survived the test of time in a rapidly progressing practical field. Understanding *why* it works is important in itself, and it could enable principled design improvements rather than trial-and-error in any future research. \n\nHowever, experimenting with **novel** design choices based on our analysis asks for pretraining experiments, which is hard and computationally requiring. This makes it a topic for a separate paper. Existing methods already have supporting experiments, and our study shows another argument to choose between them.\n\n**Looking forward:**\n\nThis work opens exciting avenues:\n- Designing custom normalization\n- Choosing appropriate normalization for different model depths\n- Extending existing theoretical analysis to include the impact of normalization schemes.\n\nWe hope this clarifies that our work aims to create a clear and unified theoretical perspective on the impact of normalization layers onto geometry of tokens, helping both bridge an evolving theoretical field with practice and empower more informed decisions for practicioners."}, {"review_id": "SH0bxgAvmq", "reviewer": "Reviewer_bGjp", "summary": "This paper presents a very interesting analysis on the normalziation layers in transformers. The authors model the evolution of tokens as interacting particles on a sphere and show that normalization acts as speed regularizaiton. With this, they analyze several normalization schemes and provide recommendations along different axes.", "questions": "1. While the paper analyses only on initial and terminal speeds, what can the authors comment about the intermediate region? The authors ackowledge this limitation. But, I would like to hear the thoughts that authiors have on how this can be analyzed.\n2.", "limitations": "I am not an expert in theory, so it is a bit hard for me to fully address the limitations of this work. Looking at a high level, the analysis looked interesting.", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The theoretical study is quite interesting. I like the idea of analyzing the impact of normalization layers. The field of transformers is quite empirical and such analysis is quite useful to the community.\n2. I like how the authors provide recommendations after each theorem.\n- In 4.1, they suggest that Pre-LN, Mix-LN, and nGPT cluster more gradually than their counterparts, indicating a more effective use of intermediate layers.\n- In 4.2, they show that Peri-LN/nGPT layers move tokens order-one distances on the hypersphere, while Post-LN and Pre-LN advance more slowly.\n- In 4.3 they say that  Post-LN and LN-Scaling cluster toke directions at an exponential rate, while Pre-LN, Peri-LN and Mix-LN slow down to a polynomial decay.\nWith these they provide recommendation that  Peri-LN and  nGPT seems to be the better choice. Such kind of recommendation is quite interesting.\n\nWeakness:\n1. It would have been nice if the theoretical explanations and observations were backed by some expeirments. If the authors showed that some LLM training on small dataset indeed produces these observations, that would have backed their claims even strongly.\n2. The assumption that the Q, K and V matrices are identity is quite strong.", "was_revised": true, "final_justification": "I am not a theory person, so I am not so confident about the theory results. But, the results look interesting, hence I stay with my initial rating.", "rebuttal": "We thank the reviewer for recognizing our theoretical contributions and for appreciating the practical recommendations we derive from our analysis. We're glad you found the insights about Pre-LN, Peri-LN, and nGPT's relative advantages well-structured.\n\n**On experimental validation:**\nWe understand the desire for empirical backing. We'd like to clarify that the normalization schemes we analyze (Post-LN, Pre-LN, Peri-LN, etc.) have already been extensively validated empirically:\n- **Pre-LN vs Post-LN**: Xiong et al. (2020) demonstrated Pre-LN's superior training stability\n- **Peri-LN**: Kim et al. (2025) showed its effectiveness. This approach was recently adopted by Gemma 3 and a slightly modified version by OLMo 2. As we establish its effectiveness theoretically, it is nice to see how relevant it becomes in practice. \n- **nGPT**: Loshchilov et al. (2025) validated its performance with learnable α_t\n\nOur contribution is providing theoretical understanding of *why* these empirically successful methods work. Rather than proposing untested schemes, we explain the mechanisms behind proven approaches. It would be extremely interesting to experiment with custom normalizators based on our study, but it is a complex practical contribution to devise a novel normalization approach. As such, we think it is out of scope of this primarily theoretical paper, but would be a great avenue for future research.\n\n**On the Q=K=V=I assumption:**\nWe appreciate this concern and want to clarify:\n- **Theorem 4.2** actually handles **arbitrary $Q,K,V$** (only requiring $||Q^TK||, ||V|| ≤ 1$)\n- **Theorem 4.3** uses arbitrary $Q,K$ matrices, and $V = I$. Moreover, this is done to focus on the clustering introduced by attention, whereas adding $V$ would add some drift into the dynamics. In other words, tokens would simultaneously cluster and move, and the intuition derived from this result remains, but it would not be as clean.\n- **Theorem 3.1** uses $Q = K = V=I$ only for convenience of exposition (because the proof is quite complicated), but the proof works for a more general $Q^\\top K = Diag(\\lambda_i) V, \\lambda_i > 0$ case. This generalises even prior works that study pure postln, because they assume $Q^\\top K = V$ (see Geshkovski et. al.)\n- Only Theorem 4.1 uses $Q=K=V=I$, but it serves as an analytical baseline to establish what to expect.\n\n**On limitations**\nAdditionally, we have updated Figure 4 to shows that qualitatevely the dynamics of Figure 2a, modeled by Theorem 4.1 is the same in much more general case: with Kaiming-initialized random matrices, multiple attention heads, and time-varying parameters. This, effectively, allows for exact parameters employed in real transformers. This backs up our claim that introduced assumptions on $Q, K, V$ are not that restrictive.\n\nRegarding MLPs: These add token-wise transformations that don't influence the mixing component, each token additionally has its own drift. This effect does not break our relative comparisons between schemes, but it complicates overall study. We leave it for future research.\n\n**On intermediate region analysis (Question 1):**\nGreat question! Analyzing the intermediate regime is the most complicated topic in this field. Relative to our work, it is not clear how speed regulation affects the space component where tokens end up. And starting from different parts of the state space the dynamics can be different. However, there are prominent works (for example Bruno et. al.) that study initial and intermediate regimes, most notably focusing to dimension reduction in the initial regime and meta-clustering in the intermediate regime. In their analysis they use PostLN. We think that our framework can be combined with those approaches, to derive similar results(with say different rates of convergence). This is a great topic for future research. \n\nThank you for your supportive review and for highlighting areas where our exposition could be clearer."}]

Author rebuttal:

Meta-review: This paper studies 
how different normalization schemes influence clustering dynamics and token representations
within a unified framework of interacting particle dynamics on the sphere and speed regulation formulation.


The reviewers appreciate the **strengths** of the paper: 
(cqGC) well-written, takeaways seem to match empirical findings,
(KEPZ) unification of various normalization schemes into a single dynamical system framework, intuitive basis (speed regulations) for comparison,
Explains several empirically observed phenomena, rigorously extends a recent line of theoretical analysis,
(rFjq) elegant unification of diverse normalization schemes under a single dynamical systems perspective, theoretical comparison with speed regulation mechanisms, well-developed, significant theoretical contributions,
(bGjp) useful to the community, recommendations after each theorem.


The reviewers also find the **weaknesses** of the paper: 
(cqGC) missing a prediction or improvement to existing methods,
(KEPZ) still quite simplified model, several unclear phrases,
(rFjq) highly restrictive assumption, fails to demonstrate how their theoretical insights translate to practical capabilities,
(bGjp) lack of experiments to support the theory, strong assumptions.

**After the discussion**, many reviewers were not very confident in their score, but they express a similar opinion: elegant unification and intuitive basis, but highly restrictive assumptions.
I recommend to accept this paper as it has significant merits despite the existing limitations (e.g., strong assumptions).
Decision: Accept (poster)