Title: Normalization in Attention Dynamics
Venue: NeurIPS 2025
Accepted: yes
Keywords: Transformers

Self-Attention

normalization

continuous-time interacting particle systems

clustering

representation collapse
Abstract: We study the effect of normalization schemes on token representations in deep transformers. Modeling their evolution as interacting particles on the sphere, we show that normalization acts as a form of speed regulation. This perspective enables a unified analysis of several schemes---including **Post-LN**, **Pre-LN**, **Mix-LN**, **Peri-LN**, **nGPT**---revealing how they influence clustering dynamics and representation collapse. Our framework clarifies how different schemes shape token representations across layers and provides a principled basis for comparing them, identifying **Peri-LN** as a particularly effective choice.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a very interesting analysis on the normalziation layers in transformers. The authors model the evolution of tokens as interacting particles on a sphere and show that normalization acts as speed regularizaiton. With this, they analyze several normalization schemes and provide recommendations along different axes.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The theoretical study is quite interesting. I like the idea of analyzing the impact of normalization layers. The field of transformers is quite empirical and such analysis is quite useful to the community.\n2. I like how the authors provide recommendations after each theorem.\n- In 4.1, they suggest that Pre-LN, Mix-LN, and nGPT cluster more gradually than their counterparts, indicating a more effective use of intermediate layers.\n- In 4.2, they show that Peri-LN/nGPT layers move tokens order-one distances on the hypersphere, while Post-LN and Pre-LN advance more slowly.\n- In 4.3 they say that  Post-LN and LN-Scaling cluster toke directions at an exponential rate, while Pre-LN, Peri-LN and Mix-LN slow down to a polynomial decay.\nWith these they provide recommendation that  Peri-LN and  nGPT seems to be the better choice. Such kind of recommendation is quite interesting.\n\nWeakness:\n1. It would have been nice if the theoretical explanations and observations were backed by some expeirments. If the authors showed that some LLM training on small dataset indeed produces these observations, that would have backed their claims even strongly.\n2. The assumption that the Q, K and V matrices are identity is quite strong.\n\nQUESTIONS:\n1. While the paper analyses only on initial and terminal speeds, what can the authors comment about the intermediate region? The authors ackowledge this limitation. But, I would like to hear the thoughts that authiors have on how this can be analyzed.\n2.\n\nLIMITATIONS:\nI am not an expert in theory, so it is a bit hard for me to fully address the limitations of this work. Looking at a high level, the analysis looked interesting.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical analysis of various normalization schemes in transformer architectures by modeling token representations as interacting particles on a sphere. The authors reframe different normalization methods (Post-LN, Pre-LN, Mix-LN, Peri-LN, nGPT, and LN-Scaling) as variations of a unified interacting particle system where normalization acts as speed regulation. Through this framework, they analyze how different schemes affect clustering dynamics and representation collapse, establishing asymptotic clustering results and characterizing initial and terminal token velocities. The work identifies Peri-LN as particularly effective for maintaining meaningful token evolution across both early and deep layers.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper provides an elegant unification of diverse normalization schemes under a single dynamical systems perspective. By casting all methods as speed regulation mechanisms for particles on a sphere, the authors enable direct theoretical comparison of fundamentally different approaches.\n2. The theoretical results are well-developed, particularly Theorems 3.1-4.3, which establish asymptotic clustering guarantees and characterize velocity profiles. The extension of gradient flow analysis to general normalization schemes beyond Post-LN is a significant theoretical contribution.\nWeaknesses\n1. The analysis relies on highly restrictive assumptions (Q = K = V = I_d, omission of MLP layers, simplified attention) that significantly limit its applicability to real transformer architectures. The gap between theory and practice is substantial, making it difficult to extract actionable insights for actual model design.\n2. The paper fails to demonstrate how their theoretical insights about clustering dynamics and representation collapse translate to actual LLM performance on reasoning tasks, language understanding, or other practical capabilities. Without showing the impact on real-world model behavior, the theoretical contributions remain disconnected from what practitioners care about most.\n\nQUESTIONS:\n1. How do the theoretical predictions about different normalization schemes translate to measurable differences in LLM capabilities? Could you provide experiments showing how Post-LN vs Pre-LN vs Peri-LN affect reasoning ability, language understanding, or other downstream tasks? This connection between your mathematical framework and real model performance would significantly strengthen the practical relevance of your work.\n2. Given the substantial assumptions (Q = K = V = I_d, no MLP layers), what is your roadmap for extending this analysis to realistic transformer architectures? Can you provide preliminary results or concrete next steps that would make your theoretical framework applicable to actual LLMs used in practice?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical analysis of different layer normalization schemes in transformers by modeling the evolution of token representations across layers as a continuous-time interacting particle system. Within this framework, various normalization methods are characterized by a “speed regulation” factor that modulates the dynamics and affects the phenomenon of representation collapse. By comparing the speeds at early and later layers, the authors identify Peri-LN as a favorable compromise.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The paper's main strength is its unification of various normalization schemes into a single dynamical systems framework. By modeling different schemes as \"speed regulations\", it provides an intuitive basis for comparison.\n\n- This framework explains several empirically observed phenomena, such as the superior stability of Pre-LN over Post-LN in deep networks. The analysis of initial and terminal velocities offers insights into how different schemes utilize early versus late layers, leading to actionable conclusions about architectural design.\n\n- The work rigorously extends a recent line of theoretical analysis (Geshkovski et al.). While based on a simplified model, the paper's conclusions are theoretically sound and provide an interesting starting point for future research to build upon, potentially by relaxing assumptions or incorporating more architectural components.\n\n**Weaknesses:**\n- The model is still quite simplified (see questions 1 and 2 below). I understand that it is mathematically very challenging to extend the results, but some more discussion about different values for Q, K, V and the inclusion of the MLP could help the reader.\n- (Minor) The presentation is clear but there are several unclear phrases or typos, for example:\n  - Line 91: At(X) = [X11(X), . . . , Atn(X)]\n  - Line 122: \"we readily get: Note that\"\n  - Line 126: \"(a.k.a)\"\n  - Line 136: \"I particular\"\n  - Line 214: \"which tokens a pre-clustered\"\n\nQUESTIONS:\n- Cited works (e.g., Geshkovski et al., 2025, Figure 2) exclude the d=2 and intermediate beta case from their clustering theorems. Does Theorem 3.1 guarantee clustering even in these regimes? Furthermore, to improve clarity for the reader, it would be beneficial to add a reference in the main text to the discussion in the appendix regarding the assumption s_j' ≥ c > 0, as it appears to be a key condition for the theorem.\n- Generalizing the results to arbitrary $Q, K, V$ matrices is clearly challenging. However, considering the simple case where $Q=K=I_d$ and $V=-I_d$ could provide valuable insight. In this setting, the dynamics become repulsive, encouraging tokens to spread apart rather than cluster. Do the paper's conclusions about the effectiveness of different normalization schemes, particularly the advantages of Peri-LN, still hold in this repulsive regime?\n- The inclusion of an MLP layer would introduce a drift term into the dynamics, independent of the attention mechanism, which would likely have a strong effect on $r_j(t)$. What is the authors' intuition on how this MLP-induced drift would interact with the different normalization schemes?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThey conduct a theoretical analysis of how the radial component of token representations evolves with transformer depth and as a function of the layer normalization scheme used. They compare several different layer norms: pre-LN, post-LN, Peri-LN, nGPT, Mix-LN, and LN-Scaling. They find that Pre-LN makes better use of depth, as tokens continue to evolve across many layers. Moreover, nGPT has the advantage of being able to adapt the rate of representation evolution / clustering.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper is a bit outside the area of research I have worked in for the last several years, so this one is pretty difficult for me to evaluate properly. I don't have the ability to rigorously evaluate the theory, which is probably the most important piece here. And as for the validity of their simplifying assumptions, I don't have a strong sense here either (e.g. removing MLP and restricting the attention KQV).\n\nI will say that the paper seems to be well written and directionally I think it is important to have a stronger understanding of how these decisions shape the NN dynamics.\n\nThe takeaways from the paper seem to match empirical findings (e.g. Pre-LN is preferable to Post-LN), which is a good sign.\n\nOne thing that seems to be missing is potentially a prediction or improvement to existing methods that would result from this analysis.\n\nQUESTIONS:\nIs it reasonable for future work to conduct similar analysis without as many simplifying assumptions on the architecture e.g. no MLP / restricting the attention?\n\nIt seems that a lot of your findings match empirical beliefs in the field, are there any results where there is a mismatch from empirically derived preferences practitioners have for LN schemes?\n\nLIMITATIONS:\nOne thing that seems to be missing is a prediction or improvement to existing methods that would result from this analysis. Otherwise the authors seem to do a good job of listing out limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}