Title: Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
Venue: NeurIPS 2025
Accepted: yes
Keywords: dynamical property of tokens

deep neural network

attention

attention with positional encoding
Abstract: This paper investigates the dynamical properties of tokens in pre-trained transformer models and explores their application to improving Transformers. To this end, we analyze the dynamical system governing the continuous-time limit of the pre-trained model and characterize the asymptotic behavior of its solutions.  Specifically, we characterize when tokens move closer to or farther from one another over time, depending on the model parameters. We provide sufficient conditions, based on these parameters, to identify scenarios where tokens either converge to zero or diverge to infinity.  Unlike prior works, our conditions are broader in scope and more applicable to real-world models. Furthermore, we investigate how different forms of positional encoding - specifically absolute and rotary - affect these dynamical regimes. Empirical evidence reveals that the convergence scenario adversely impacts model performance. Motivated by these insights, we propose simple refinements to Transformer architectures that mitigate convergence behavior in models with absolute or rotary positional encoding. These findings support theoretical foundations and design principles for improving Transformer models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the dynamical behavior of tokens in Transformer models using a Neural-ODE formulation of attention. It examines whether, under certain conditions, tokens tend to converge or diverge as they progress through layers, depending on model parameters. The authors also analyze the role of different positional encodings, specifically absolute and rotary, in shaping these dynamics, and try to support their claims through empirical evaluation.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper explores an interesting and relevant topic: how token representations evolve as they propagate through the layers of a Transformer, with a focus on whether tokens converge or remain separated.\n\n2. The authors conduct an extensive set of experiments to provide empirical support for their theoretical analysis.\n\n**Weaknesses:**\n\n1. I think the related works section could be improved. It would be helpful if the authors contextualized their contributions more clearly in relation to previous work. Specifically, in lines 37 to 40, the authors state that prior works rely on “unrealistic and impractical assumptions on model parameters... to support the development of richer theoretical results...” and “have not yet offered practical applications or insights...” However, it is not clear what specifically makes those assumptions unrealistic or impractical, nor what assumptions this paper adopts instead or why they are more realistic. Clarifying this distinction would help establish the novelty and practical relevance of the contribution. At present, the difference between this work and earlier studies remains vague, and I could not find any further discussion or elaboration in the appendix.\n\n2. I believe some of the mathematical derivations may contain inaccuracies - please see the questions. \n\n3. I think some of the experiments are constructed in a way that closely reflects the conditions assumed in the theoretical analysis, which may make the results somewhat self-reinforcing. However, it still remains to be seen whether such behaviors occur in real-world settings. Please see questions for a follow-up. \n\n4. The results lack standard deviations.\n\n5. I find the setting to be overly simplistic. While this is not necessarily a flaw, it raises concerns about the practical relevance of the analysis. The Neural ODE model is a coarse approximation of how information propagates through multiple layers of a Transformer. For the analysis, the authors assume that the attention matrix $QK^T$ can be replaced by a fixed matrix $W$, where $Q$ and $K$ are the query and key matrices used in the self-attention mechanism. They further assume for their analysis that $W$ is either positive semidefinite (PSD) or negative semidefinite (NSD). However, in causal attention, $QK^T$ is not symmetric, so $W$ is unlikely to satisfy these conditions. These assumptions simplify the analysis but may significantly limit its applicability to real-world models.\n\nQUESTIONS:\n**Questions:**\n\n1. I have several questions and suggestions regarding the experimental validation of the paper’s claims:  \n\n    a) Is it possible to empirically validate whether tokens in a pretrained model converge or diverge as they propagate through layers? For instance, could this be tested by analyzing distances between token embeddings across layers in a realistic model? Appendix D.5.1 touches on this, but I was hoping for a more thorough empirical investigation using standard pretrained architectures.  \n\n    b) Can one analyze the structure of the attention matrices \\$QK^\\top\\$ in real models to determine whether they are PSD, NSD, or neither? In decoder-only models using causal self-attention, these matrices are typically not symmetric, even when assuming \\$Q = K\\$, so they are unlikely to satisfy PSD or NSD conditions. How would this affect the applicability of the theoretical analysis?  \n\n    c) Is it possible to evaluate the core Neural ODE assumption that the model can be approximated by an Euler discretization due to small weight changes between layers? In practice, the \\$Q\\$, \\$K\\$, and \\$V\\$ matrices vary significantly across layers in standard Transformers, which may invalidate this assumption.  \n\n    d) Could the experiments in Section 3.4 be rerun without explicitly imposing the constraints used in the theoretical setup? This would help evaluate whether the convergence or divergence behavior still appears under more realistic conditions.  \n\n    e) Is it possible to study how convergence or divergence behavior is affected by other architectural components, such as MLP blocks or layer normalization?  \n\n    f) Finally, could the analysis be extended to investigate the effect of varying the input sequence length, at least empirically?  \n\n2. For what set of models is this analysis even useful? For causal self attention I think a lot of the analysis would probably not hold. Could you please clarify what would change ? \n\n3. In the caption of Figure-4 what does it mean by similar shapes?\n\n4. I think I probably missed this in the paper, but how do you obtain figure 4 and figure 5? Is it via the PCA of the tokens or is there something else ?\n\n5. For Table - 1 and Table -2 is it possible to add any standard deviation \n\n6. In the solution of Equation (11) (see lines 575–578), could the authors clarify the initial conditions used for the token representations $x_\\ell(0)$? This question arises because if the quantity $e^{c||x_\\ell(0)||^2} - L$ is negative, then the expression may decrease as $t$ increases and eventually become undefined. In such a case, the conclusion that $\\lim{t \\to +\\infty} ||x_\\ell(t)||= +\\infty$ would no longer hold. This also suggests that the behavior of the system would depend on $t$, meaning that the model's depth (or number of layers) must lie within a bounded domain to ensure validity. If this interpretation is correct, some of the paper's conclusions may need to be reconsidered under such conditions. Could the authors elaborate on what initial values are assumed in these equations, and whether such conditions are expected to occur in practice? Furthermore, for very long input sequences, it appears that $L$ increases with sequence length. In this case, under what conditions on $A$ and $x\\ell(0)$ does the term $e^{c||x_\\ell(0)||_A^2} - L$ remain positive to ensure the validity of the derivation? That is, how must the matrix $A$ and the initialization be chosen to guarantee that the analysis holds for long sequences?\n\n7. I would like to request clarification regarding the setup used to produce Figure 6. Specifically, is the figure generated by taking a pretrained model and subsequently removing the layer normalization and MLP components, or is it based on a model trained from scratch without using layer norms and MLPs? Additionally, can experiments be conducted where the constraints described in lines 263–272 are removed, in order to observe what happens when these artificial constraints are relaxed?\n\n8. Could the authors elaborate on the intuition behind why divergence leads to improved performance, as suggested in lines 275–279? Is the idea that token representations become sufficiently separated in space, making them easier to distinguish and thus improving model expressivity? This seems somewhat counterintuitive from a language modeling perspective, where one might expect that semantically similar tokens should cluster together, while only dissimilar tokens diverge. Otherwise, it could hinder the model’s ability to learn meaningful attention maps, especially if all tokens diverge indiscriminately. Could the authors provide a more detailed intuition for why divergence is beneficial in this context, and how this compares to what is observed in standard pretrained models?\n\n9. What is the practical benefit of Theorems and Propositions 3.1–3.6? All of these results appear to focus on asymptotic behavior with respect to model depth. However, in practice, the depth is always finite for example, DeepSeek-V3 [A] has approximately 61 layers. Given this, how should we interpret the relevance of these asymptotic results? Do they meaningfully inform us about the behavior of realistic models? More specifically, is there a way to understand how the described properties evolve layer by layer, rather than only in the limit as depth approaches infinity?\n\n**Minor Typos:**\n\n1. Line 179: “We observe from randomly ~~chosen of~~ model parameters”, \"chosen\" -> \"initialized\"? \n\n\n**Note:** I find the analysis in this work to be interesting. However, my main concern is regarding the applicability of the results. Much of the analysis appears to depend on very simplified settings, and the assumptions made are quite strong and seem to diverge significantly from practical scenarios. Based on these concerns, I am currently assigning a lower score. That said, I am open to revisiting my evaluation during the rebuttal phase, depending on the authors' clarifications and the views expressed by other reviewers.\n\n---\n\n**References:**\n\n[A] Liu, Aixin, et al. \"Deepseek-v3 technical report.\" arXiv preprint arXiv:2412.19437 (2024).\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript takes a dynamical-systems viewpoint—modeling layers of an attention-only deep neural network as continuous—with the motivation of exploring the evolution of token embeddings through the layers of a transformer during pretraining.  The authors consider a regime where weight matrices are tied (i.e., identical) across layers.  In this regime, they find that activations from different inputs either converge or diverge through the layers, depending on whether the symmetric components of certain matrices ($W = Q K^\\top$ and $A =  Q K^\\top (V^{-1})^\\top$) are positive definite (i.e., have only positive eigenvalues) or negative definite (i.e., have only negative eigenvalues).  Effects of positional embeddings are considered.  The theoretical predictions are borne out by transformers that are pretrained with weight matrices tied across layers, even when the transformers include FFN layers in addition to the attention layers.  In the end, the paper suggests ways to promote divergent embeddings.  Appendices provide derivations of all claims.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper does a good job of stating the equations and assumptions for their modeling framework.  Once this framework is set up, the ensuing math and dynamical-systems analysis all seems valid.  (Good Clarity and fine Quality for the topic of study.)\n\nThe primary weakness of this paper is that the modeling assumptions seem to create a serious disconnect from real transformers, such that the predictions seem irrelevant.  The form of attention equations looks similar, but the continuous-time limit has no obvious motivation other than it allows for a dynamical system that can be analyzed.  Tying the attention weight matrices across layers of a transformer is a strong restriction, which mirrors the ALBERT architecture but does not reflect standard architectures of interest.  As a further significant restriction, this paper does not incorporate any analog of the FFN layers of transformers in the theoretical analysis.  I recognize that other papers like Ref. [17] have been published before using a similar modeling framework, but I don’t believe this precedent provides sufficient justification for the importance of the current analysis for understanding real neural nets which would be an expected standard for a conference like NeurIPS.  Rather, the existence of several other papers that make very similar assumptions reduces the novelty of this approach and makes the current work feel quite iterative.  (Low Originality and Significance.)  I believe this paper would be better suited for a dynamical systems audience looking for neat dynamics, rather than an audience trying to understand transformers.  \n\nThe authors say that “Our results are also validated against pre-trained Transformer models, confirming that\nthe theoretical insights carry over to real-world settings.”  The authors indeed show some relevance of their theoretical predictions for the transformers they train, including those with FFN layers in addition to attention layers, but my understanding is that these transformers have their weight matrices tied across layers, which is not much of a real-world setting (besides the ALBERT architecture).\n\nSome standard ML terms seem to be used unusually in this paper. In particular, the word “token” is used throughout the paper to rather mean something like “token embedding” or “residual stream”.  The stated study of how tokens evolve is really about how the activations in the residual stream evolve through the layers.  For example, “when tokens move closer” would be more precise if it was changed to say “when embeddings move closer in deeper layers of the network”.  It would be much more conventional for “tokens” to refer to the symbols from a fixed discrete set, rather than real values.\n\nQUESTIONS:\nThe authors assert that their first contribution is to “provide conditions that predict whether tokens in self-attention converge to zero or diverge to infinity”.  By standard use of these terms, this claim is inscrutable.  I am guessing the authors mean that the magnitude of token embeddings in a transformer are converging or diverging?  Can the authors provide citation(s) of this issue in real transformers?\n\nRegarding the framework laid out in Sec. 2.2, the authors may want to consider causal attention masking in future work, since this would be most relevant for autoregressive transformers.\n\nIt is not clear how the eigenvalues of $A$ correspond to the eigenvalues of its symmetrization $A_\\text{sym}$.  In the general case, I’m not aware of any easy correspondence.  Accordingly it doesn’t seem to be the case that, e.g., the positive-definiteness of one implies the positive-definiteness of the other.  In the paper, sometimes positive-definiteness is talked about in terms of $A$ and sometimes in terms of $A_\\text{sym}$, and sometimes in terms of $A$ after assuming it is symmetric such that $A = A_\\text{sym}$.  How should the reader think about these relationships?  In particular, the claims starting on Line 128 involving the positive definiteness of $A$ don’t appear to assume that $A$ is symmetric, but then Thm. 1 assumes that $A$ is a symmetric matrix and Remark 3.2 is about the positive definiteness of the symmetrization of $A$.\n\n\nMinor suggestions:\n\nLine 41 and Section A:\n“Only a limited number of studies in the literature provide a theoretical understanding of the internal structure of learned representations in pre-trained Transformer models (see Section A for a comprehensive list of relevant works).” \nThis field moves fast, so it seems overly confident to suggest a “comprehensive” list of relevant works that provide understanding of the internal structure of learned representations.  I can certainly think of many more.  Perhaps remove the word “comprehensive”.\n\nLine 48:\nIt seems that “fourth-fold” should be changed to “fourfold”.\n\nLine 118:\nConsider explicitly stating the intended meaning/definition of “equivariant” in the statement that “it is equivariant to the standard Euclidean norm”\n\nLine 157:\nTypo:  Change “randomly chosen of model parameters” to “randomly chosen model parameters”\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
This is a very good paper. It extends the existing line of work on studying signal propagation in Transformer models, where the parameters are fixed (pre-trained model) and the evolution of tokens across layers is analyzed. The authors take a « neural ODE » point of view in which time plays the role of a layer, and this perspective is well-accepted in the community as being particularly helpful in doing theory. This line of research is clearly fundamental and critical for improving the interpretability and understanding of how large language models work.

STRENGTHS AND WEAKNESSES:
In particular, the authors address the effect of positional encoding on the dynamics of tokens within the Transformer architecture, which has not been seriously considered before. The novelty is clear. There are also clear extensions on the existing results of Transformer dynamics to more general classes of parameter matrices (key-query and value matrices, to be more precise).

QUESTIONS:
1. I have a question regarding the addition of positional encoding in the equation. I thought positional encoding is added to the initial condition, and that’s it. Why does it appear in the ODE in equation (5)? I am not totally sure I understand this point.

2. The authors are also very welcome to comment on the possible impact that an MLP layer (or a normalisation layer) may have on the divergence, or the avoidance thereof. 

3. Moreover, there is a paper by Koubbi, Boussard and Hernandez, where they also generalise the results of Geshkovski et al. to somewhat more general parameters, but it’s not cited herein. Perhaps the authors can discuss the differences with that paper.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the dynamics of tokens in transformers (removing all blocks other than the self-attention) when all the parameters are trained and fixed. By viewing the forward of tokens as a continuous (partial) differential equation, the authors are able to reveal the divergence ($||x|| \\to \\infty$) and convergence ($x \\to 0$) phases of tokens that depend explicitly on $W$ (by merging the query matrix $Q$ and key matrix $K$) and the value matrix $V$. As an application, by exploring how different positional encodings can change the dynamics, the authors propose a new simple technique to encourage token divergence to improve the performance of transformers.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. This paper is overall well written. In particular, the authors provide a clear motivation for the problem being studied and organize the flow of the paper very well, hence it is very easy to understand the main results, i.e., the main results for the dynamics of tokens are clearly summarized as three parts: the distance between tokens, the convergence of tokens, and the divergence of tokens.\n\n2. The conditions for the divergence and convergence phases of tokens are surprisingly simple: by examining whether $W$ and $A$ are positive definite or not. Thus the conditions might be easily checked in practice by examining the eigenvalues of these matrices. \n\n## Weaknesses\n\n1. Please correct me if I understand the setting in a wrong way: the parameters are indeed trained and fixed, however, the time variable $t$ refers to the depth of the transformers. Hence the fact that $W$ and $A$ do not depend on $t$ means that the merged weight $W$ of attention and the value matrix $V$ are also the same for all layers. This does not align with the actual case. As the cluster of tokens has already been studied by prior works, I think it is better to take a step further to remove the limitation by considering a case where $W$ can depend on $t$. \n\n2. One aspect that has not been touched by this paper is the statistics of the conditions for $W$ and $A$ of real-world pre-trained models (not manually designed by some re-parameterization as in Appendix D.2). Specifically, the condition that the value matrix $V$ is invertible cannot necessarily be guaranteed in practice. Thus it is unclear whether these pre-trained models indeed exhibit such divergence and convergence of tokens that can match the conclusion of this paper. In a word, it is not clear whether these conditions and the corresponding divergence/convergence of tokens only exist in the special setting of the paper or are more general across a wide range of settings.\n\nQUESTIONS:\n1. Could the authors explain what the conclusions of the current manuscript will be if all the weights depend on the variable $t$?\n\n2. Could the conclusions/prediction be applied to real-world pre-trained models? And if so, what will the comparison between a randomly initialized model and the pre-trained model be regarding the conditions given by this paper?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}