Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
Authors: Duy-Tung Pham
An Nguyen The
Viet-Hoang Tran
Nhan-Phu Chung
Xin T. Tong
Tan Minh Nguyen
Thieu Vo
Venue: NeurIPS 2025 poster ()
Keywords: dynamical property of tokens
deep neural network
attention
attention with positional encoding
Primary area: theory
TL;DR: We analyze the flow of tokens across attention layers and use these insights to enhance performance of Transformers.

Abstract: This paper investigates the dynamical properties of tokens in pre-trained transformer models and explores their application to improving Transformers. To this end, we analyze the dynamical system governing the continuous-time limit of the pre-trained model and characterize the asymptotic behavior of its solutions.  Specifically, we characterize when tokens move closer to or farther from one another over time, depending on the model parameters. We provide sufficient conditions, based on these parameters, to identify scenarios where tokens either converge to zero or diverge to infinity.  Unlike prior works, our conditions are broader in scope and more applicable to real-world models. Furthermore, we investigate how different forms of positional encoding - specifically absolute and rotary - affect these dynamical regimes. Empirical evidence reveals that the convergence scenario adversely impacts model performance. Motivated by these insights, we propose simple refinements to Transformer architectures that mitigate convergence behavior in models with absolute or rotary positional encoding. These findings support theoretical foundations and design principles for improving Transformer models.

Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding
Duy-Tung Pham" FPT Software Al Center
An Nguyen The FPT Software Al Center Hanoi, Vietnam annt68@f com
Viet-Hoang Tran
Department of Mathematics National University of Singapore hoang. tranvi nus edu
Hanoi, Vietnam tungpd108fpt.com
Nhan-Phu Chung
Xin T. Tong
University of Economics Ho Chi Minh City Ho Chi Minh City, Vietnam phucndueh. edu. vn
Department of Mathematics National University of Singapore mattxin@nus edu. sg
Tan M. Nguyen
Thien N. Vo Department of Mathematics National University of Singapore thieuvo@nus. edu. sg
Department of Mathematics National University of Singapore tanmnonus. edu. ag
Abstract
This paper investigates the dynamical properties of tokens in pre-trained Trans- former models and explores their application to improving Transformers. To this end, we analyze the dynamical system governing the continuous-time limit of the pre-trained model and characterize the asymptotic behavior of its solutions. Specif- ically, we characterize when tokens move closer or farther from one another over time, depending on the model parameters, We provide sufficient conditions, based on these parameters, identify scenarios where tokens either converge to zero or diverge to infinity. Unlike prior works, our conditions are broader in scope and more applicable to real-worke models. Furthermore, we investigate how different forms of positional encoding specifically absolute and rotary affect these dynamical regimes. Empirical evidence reveals that the convergence scenario adversely impacts model performance. Motivated by these insights, we propose simple refinements to Transformer architectures that mitigate convergence behavior in models with absolute rotary positional encoding. These findings support theoretical foundations and design principles for improving Transformer models.
Introduction
Transformers [51] have revolutionized multiple domains, demonstrating remarkable success in natural language processing [10, 8, 2, 4, 41], computer vision [13, 43, 38, 32], machine learning [48, 23, 29], and reinforcement learning [22, 5]. Their widespread adoption largely driven by their ability to leverage large-scale pretraining, enabling efficient knowledge transfer to downstream tasks [40, 12, 39]. Unlike traditional architectures that rely on recurrence or convolution, Transformers are built upon a self-attention mechanism. This mechanism dynamically computes relationships among all tokens in an input sequence, assigning importance scores that dictate how strongly each token influences the others. As a result, self-attention effectively captures intricate dependencies across long
"Co-first authors.
CCo-last authors. Please correspond tanmnonus. edu.sg sg and thieuvo@nus.c edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

sequences, making it highly proficient in contextual representation learning [30, 53, 49, 37]. By modeling global interactions across an input, Transformers have surpassed earlier deep learning architectures, solidifying their position as the dominant paradigm in modern artificial intelligence.
Only limited number of studies in the literature provide theoretical understanding of the internal structure of learned representations in pre-trained Transformer models (see Section A for com- prehensive list of relevant works). Notably, in the seminal papers [17, 16], the authors modeled Transformers as interacting particle systems and demonstrated that tokens tend to cluster around specific limiting objects determined by the initial tokens, thereby confirming the context-awarent of representations learned by Transformers. It was also observed in [15] that, in the self eelf-attention dynamic, although tokens collapse configuration of several clusters for an exponentially long period of time. The authors in [3] analyzed a pure-attention hardmax Transformer model in a similar manner under a discrete framework. A key limitation of these works lies in their reliance on unrealistic and impractical assumptions on model parameters, primarily introduced to support the development of richer theoretical results and technical proofs. Moreover, by focusing exclusively on theoretical analysis, these studies have not yet offered practical applications or insights for improving model performance.
single cluster in infinite time, they remain trapped near a
A comprehensive list of related works can be found in Appendix A.
1.1 Our Contribution
In this paper, we investigate the internal dynamics of tokens in self-attention mechanisms, together with the effects of absolute and rotary positional encoding, under more realistic and practical assump- tions on model parameters, extending prior theoretical studies that often rely on restrictive conditions. We focus on the continuous-time limit of pre-trained Transformer models and systematically analyze the asymptotic behavior of token trajectories such as convergence, divergence, and token distances over time. Our contribution is fourth-fold:
1. We provide conditions that predict whether tokens in self-attention converge to zero or diverge infinity. The conditions more general than those considered in prior works (e.g.. [17, 16)) and the theoretical results are validated on pre-trained Transformer models.
2. We show that absolute positional encoding has little effect on token dynamics, while rotary encoding significantly alters them to promote the divergence of tokens.
3. We empirically verify that the convergence of tokens to zero negatively impacts model perfor- mance. In contrast, in divergence scenarios, tokens organize into a small number of groups, with tokens within the same group diverging to infinity in a consistent direction. This is often beneficial for model performance.
4. Building on these findings, we propose simple yet effective improvements to mitigate conver- gence scenarios in Transformers with absolute or rotary positional encodings.
To verify our findings, we conducted language modeling experiments on WikiText- [34] and EnWik8 [20], and object recognition ImageNet- IK [11]. The results support our thcoretical claim and provide insights to improve Transformer models.
Organization. The paper is organized as follows. Section 2 introduces the continuous-time dynamical system representing a pre-trained Transformer. Section 3 presents our main theoretical and empirical results token dynamics in self-attention. Section 4 extends the analysis to absolute and rotary positional encodings. Section provides empirical validation and proposes improvements to positional encoding schemes. Proofs and additional experiments provided in the Appendix.
Notations. Through this paper, ||- || denotes the Euclidean norm of a vector. For each subset A C R.P. denote by (A)v(A) the convex hull of A, which is the smallest convex set containing A in R For a square matrix B, we denote by q the quadratic form associated to B and by Boym = (B + BT the symmetric part of B. When B is positive definite, we write B > 0. In case B is negative definite, we write B < O.
2 Background:
continuous-time Limit of Attention
In this section, we present the dynamical system that represents the continuous time counterpart of pre-trained Transformer model.

---

2.1
Continuous-time Limit of Deep Neural Network
We build on prior work in the literature that explores the dynamical systems underlying the continuous- time limits of deep neural networks (DNNs). prominent example of DNNs residual neural network (ResNet). Each layer in ResNet is residual block that transforms input vector I: R into an output vector 2 = I + y(z 0) E R2, where y = y(z, 0) is two-layer feed-forward neural network parameterized by 0. The continuous-time counterpart of ResNet is represented flow map that takes an input vector z(0) E IR_ and produces an output vector x(T') E RR, governed by the associated dynamical system:
(t) = v(x(t,
E (O,
There a substantial body of research examining the interpolation, approximation, and controllability properties of such DNN architectures [28, 54, 27, 47, 44, 6].
2.2 Self-Attention Dynamics
In contrast to ResNet, Transformer operates on a sequence of D-dimensional tokens rather than solely on individual inputs. Central to the Transformer architecture is the self-attention map. Let
X [1 æt] RLXD be an input sequence of L tokens with D features. Each token (column) vector 2, IR.. The self-attention map transforms X into the output sequence
= y1 yr] E RL defined as
( (IX (X....XX..0 (K..KK) ) X V.
Y = softmax
(I)
vDx
The matrices Q, K E RDDD= and V E RDD. are learnable parameters, and they are called the query, key and value matrices. our context, we will always assume that D, D, thus is square matrix. We can rewrite equation (1) as
W-w;
y1 = V
2==1 e W-
e
for 1 = 1, ...., I., where W -/ KT, which is matrix in RDD.
To better understand the internal learning representations of pre-trained Transformer model, consider the differential system that governs the continuous-time dynamics of self-attention:
dt (t) VT -T
(xt()) T (W),(()
xt)(),
(2)
Σ (0)( W W.n;(t)
for 1 = 1, ,,...,,,,, L, with the initial conditions (æ) dynamical system for the self-attention with positional encoding will be discussed later in Section
(0) = (#10 LLo) E (RP)L, The
Building on the methodologies introduced in [17, 3]. we focus our analysis specifically on the self-attention mechanism core component of Transformer architectures and the role of skip connections within the associated dynamical system. To facilitate tractable analysis, we omit other token-wise operations such layer normalization and feed-forward networks, and assume time- invariant model parameters Q, K, and V. While this assumption is primarily for analytical simplicity, it also aligns with parameter-sharing strategies used in models like ALBERT [26] to reduce training costs.
Remark 2.1 (Scope and Generality). Although our theoretical setup adopts simplifying assumptions, this is standard practice in theoretical work to make the analysis feasible while preserving the ability to capture core empirical behaviors. Several influential studies follow similar simplifications [17, 16, 15, 3). Notably, our framework generalizes key aspects of [17, 3] by relaxing certain assumptions. Our results are also validated against pre-trained Transformer models, confirming that the theoretical insights carry
real-world settings.
3 Dynamical Properties of Tokens in Self-Attention
We present our main theoretical and empirical results on the dynamical properties of tokens in self-attention dynamics in this section.
3

---

Quadratic Space. To analyze the dynamical properties of tokens in more general context, we conduct our study within the framework of quadratic spaces. Each matrix B IR Dx D is associated with a quadratic form q RD + IR, defined as qn(u) = u B for each u E RD.. The pair (RR QB) ) is referred to as quadratic space. The quadratic form q is uniquely determined by the symmetric component Baym } B) of B. This means that, for arbitrary matrices B and B', B
have a qn if and only if B Bxym B'aym In the special case where q is positive definite (i.e., Bxym > 0), qa corresponds to the square of a norm. Conversely, when qn is negative definite (i.e., Bsym 0), qu corresponds the negative of squared norm. In particular, in the Baym is a definite matrix, the map
V.B().
sym 0,
|1- (V-qB(-), if Bsy < 0,
actually a norm on RD, and it is equivariant to the standard Euclidean norm || Therefore, the dynamical properties of tokens as elements of the quadratic space (R²,qs) .q)) faithfully reflect their dynamical behavior in the standard Euclidean space (IR ? II).
To start the analysis of the dynamical properties tokens, we assume that (tr(t)..... .I......... (t)) C(OO, +oo)) " is the unique solution of the dynamical system (2). The existence and uniqueness as well as the well-posedness of this solution was proved in [17,
Dynamical Properties of Tokens.
Proposition 6.2]. We consider the function f: IR RRD + R defined by
f(t. u) = =log Σ c" (w;
M E D.
(3)
Then the dynamical system (2) can be written as
d A
mi(1) di
Du 11,
(4)
where A W (VTT) provided that V invertible. Based on this observation, we characterize the dynamical properties of tokens via A and W. In particular, we will show that:
1. Distances between Tokens. If A < O, then all tokens will move closer each other time approaches infinity. In contrast, move farther away from each other time / approaches infinity (see Theorem 3.1).
> O, the tokens will either maintain constant distances
2. Convergence Scenario. If A < 0 and W > 0, then all tokens will tend to zero as the time t tends to infinity (see Theorem 3.4). Furthermore, our simulations with randomly selected model parameters suggest that the convergence scenario occurs whenever Asym < 0 and W Wyym= > 0.
3. Divergence Scenario. Our simulations with randomly selected model parameters suggest that the divergence scenario occurs whenever the condition "Asym < 0 and W sym > 0" is violated. In the special case when V has least one positive eigenvalue and W arbitrary, we prove that all tokens will diverge to infinity under certain assumption on the initial data (see Theorem 3.6).
We will theoretically prove and empirically validate these observations in the subsequent subsections. The choice of parameters used in the simulation is described in Appendix D.I.
3.1 Distances between Tokens
We characterize the dynamical properties of the distances between tokens using the quadratic forms qA associated the matrix A W. (VT) In particular, we prove that:
Theorem 3.1 (Distances between Tokens). If symmetric, then the map + qu 22) is non-decreasing on [0, +oo). As consequence,
(a) if A > 0, then (t) x (t)llA A is non-decreasing on [0, +oo);
if A 0, then [tx Ir (t) A non-increasing [0, +00).

---

30
30
[|0000-40000
[1000-
1bs00-xx Ios00-x/tlla
Figure Distances between tokens over time. Left: When Agm O. distances between tokens not decrease. Right: Conversely, when Asym < 0, the distances between tokens do not increase.
Token trajectories under configurations. Left: When Asm negative definite and W is positive definite, tokens converge zero time f oo. Right: When 2W, tokens diverge to infinity over time, forming a few distinct groups that move in aligned directions.
Figure 2:
Intuitively, Theorem 3.1 states that the tokens will move closer to each other when A negative definite, while the tokens will either maintain constant distances or move farther away from each other when is positive definite. This result can be intuitively understood from equation (4). Indeed, one can verify that ff is convex function in u. Therefore, the subtraction 2;(t) - =z(() will have the same 8 orientation as Tū [t,, (t) Su f xz (t)), which is equal to the derivative #A A (x;(t) xy (t)).
As consequence, the quantity 9.1 (xx;1)) ar (4)) is always non-negative. The formal proof of this theorem can be found in Appendix B.1.
Remark 3.2 (Distances between Tokens). In case is not necessarily symmetric, we observe from randomly chosen of model parameters that the distance [1x:(1) (t)(l] is non-decreasing (respectively, non-increasing) whenever Asmm > 0 (respectively, Asym < 0).
Figure illustrates the distances between tokens when Aym positive (on the left side) and when Agm is negative (on the right side). As shown in Figure 1, token distances remain constant or increase when Asym O, and remain constant or decrease when Asym O, confirming Theorem 3.I.
3.2 Convergence Scenario
Our simulations with randomly selected model parameters suggest that the convergence scenario occurs whenever Asmm < and Wsym > 0. In this section, we actually prove that when A x 0 and W > 0, then all tokens will tend to zero t approaches infinity, First, established relation between the quadratic forms qı and qw in general setting below:
Proposition 3.3. Assume that V an invertible matrix and W (VT)-1 symmetric. Then,
d (x(((())
2L
caw (a1
for all =1=1,... and E [0, +00). As a consequence, if and Wym 0, then all tokens are bounded, i.e there exists e > 0 such that [|](()]| < cfoor all t E [0, +00 and I = 1,..., L.
In the case x and W > 0. we prove a stronger result as stated in the following theorem: Theorem 3.4 (Convergence Scenario). If A and > 0, then all tokens converge to zero, i.e. lim zr(t) foror all | I,...,..
5

---

2032
0.3
Token nom
8
0.0
Token
Second
2
-1.0
5
10
15
20
Dopth
Depth
First dimension
(a) Token norms decrease der the convergence scenario (Asm < and Wym > 0).
(b) Token norms diverge under the divergence scenario (Aym >
(c) Tokens form distinct groups with similar trajectory shapes in diver- gence scenario.
0 Wsym 0).
Figure 3: Token dynamic in pre-trained model when omitting LayerNorm and feed-forward network
The special case where A = I and W Io was already proved in [17, Section 8.2]. We generalize their results to parameters. The proofs of Proposition 3.3 and Theorem 3.4 can be found Appendix B.2.
practical setting which requires weaker assumptions the model
Remark (Convergence Scenario). We observe from randomly chosen of model parameters that when Asm < 0 and sym e > O, all tokens will converge to zero. Otherwise, the tokens are likely to divergent to infinity.
Figure 2a depicts the trajectories of tokens for the scenario analyzed in Theorem 3.4. Under these conditions, all tokens converge to zero as + oo, thus confirming Theorem 3.4.
3.3 Divergence Scenario
Our simulation on random selections of model parameters suggests that the divergence scenario is likely to occur whenever the assumption "Asym < O and Wsym > 0" in Remark 3.5 is violated. In this subsection, we prove that in the special case where the value matrix V has at least one positive eigenvalue, the divergence scenario indeed occurs under certain assumptions on the initial data.
For each nonzero vector R2, denote by Hn the closed half-space of R with normal vector n such that zero belongs to its boundary 0H,- In this case, 0H, represents the hyperplane containing with normal vector In the following, project the tokens onto the affine line along the eigenvector corresponding to positive eigenvalue of V to see when token tend to infinity.
Theorem 3.6 (Divergence Scenario). Assume that the value matrix has at least one positive eigenvalue. Let n be eigenvector of V corresponding
positive eigenvalue. Then
min (n' xx) S -e- x1(1) max (n" (nTxx xxa) 15$5
15iSL
for all t E (00+ and 1...,,
As a consequence, the initial points ;10,...,, are all in one side of the hyperplane OH,, and if V has only positive eigenvalues, then lim [()[(()] for all I.
The proofs of Theorem 3.6 can be found in Appendix B.3. Figure illustrates the divergence scenario with A 2W (thus V D). We further empirically verify that the divergence-scenarro constraints arise practical pretrained models; see Appendix D.6.2.
3.4 Tokens' Dynamic in Practical Transformers Architecture
This section presents empirical study of token dynamics in practical Transformer architecture. We analyze 24-layer model pre-trained on WikiText-103 [34], enforcing our theoretical framework by constraining the symmetric matrices W sym and Asym to be either positive definite or negative definite (see Appendix D.2 for details).
We begin by empirically verifying the convergence that omit feed-forward networks and LayerNorms, directly validating our theoretical results. When Asym < 0 and W Wsym 0, token norms contract monotonically across layers, potentially collapsing toward zero number of layers increased (Figure 3a). By contrast, under Asym > and W Wsym > norms grow without bound (Figure 3b), consistent with Proposition 3.3. Replicating these experiments
convereence-divergnncee divergence behavior of simplified model variants

---

Socord dimongion
Second dimension
1.0
0.5
-2.5
-2.0
1.5
1.0
First dimension
Figure 4: Tokens' trajectory in the later layers of Transformer forms clusters with similar shapes in model (left) with LayerNorm, no feodforward, and (right) with both LayerNorm and feedforward.
on the GPT-2 pre-traie pre-trained model [40]-with LayerNorms and feed-forward networks -xhiibits the same divergent behavior (see Appendix D.6.3).
then demonstrate clustering behavior both simplified and full models. With Asym > 0 and W Wsym > 0. projecting token trajectories into two dimensions reveals groups with aligned directional patterns (Figure 3c). Even in the full configuration with FFNs and LayerNorms, the model maintains persistent directional clustering in deeper layers (Figure 4). Finally, observe similar clustering behavior on GPT-2 without enforcing any constraints (see Appendix D.5.1).
Beyond Self-Attention Dynamic: Effect of Positional Encoding
In practice. Transformer models typically incorporate positional encodings into self-attention mecha- nisms to enhance expressivity and computational efficiency. In this section, we analyze how positional encodings influence token dynamics. Among common approaches, absolute and rotary positional encodings are predominant. We demonstrate that, under rotary positional encoding, tokens are more likely to exhibit divergent behavior rather than convergent dynamics comparison with the self-attention with/without absolute positional encoding.
4.1 Absolute Positional Encoding.
The self-atte self-attention map with absolute positional encoding [51, 50) transforms the input sequence X into the output sequence Y
E tp (oyty)))) ) (I; + p.),
e( T (we+t))))
y1 = VT.
1=1
for / 1,..., L. where P. [pr....... .P.DD T R., A common choice is either learn the positional embeddings ps jointly with the model parameters, or to fix them using sinusoidal scheme:
sin(i (i 10000 b). Ptj
if j is even,
cos(i 10000 S if j is odd.
The differential system that governs the continuous-time dynamics of the self-attention with absolute positional encoding is given by
((i(())p)) (W (mo())))))))
(t,
(zx(() + +P),
(5)
for = 1,
Remark 4.1 (Absolute positional encoding has minimal impact). The dynamical properties of the self-attention with and without absolute are similar, since the above differential system can be transformed into the original self-attention dynamic (2) by the transition ト + In particular, follows from Theorem 3.4 that: when < 0 and W > 0, the solution X(1) equation (5) will converge to the sequence P (-P1...., -Pr). In addition, it follows from Theorem 3.6 that, in case V has least one positive eigenvalue, tokens will diverge to infinite under certain assumptions on the initial conditions.

---

-2
-10
-7.5
-5.0
-2.5
0.0
Figure 5: By introducing the additional AWR. the system's hehavior shifts from convergent regime (left) a divergent regime (right).
4.2 Rotary Positional Encoding.
In contrast to absolute positional encoding, for even dimension D, the rotary positional encoding [31, 45] maps the input X into the output Y as
L
c Wa
y1 VT
I2:
(6)
eai
Wiy
====
forl L, where
WR vDk (Q.KTT
K +Q-
RD Ro K
DxD
(7)
with Q, K are two additional learnable matrices and
m0; ((COs sin m01
sin m8] cos m01
) ?m.
cos cosmmDp> sin mimm0p/2
-sin m0 simmppp) cos ml D/2
)
RB,m = blockdiag
where Q (0, 10000 10000-20-1))) 2
1//
The continuous-time dynamics of the self-attention with rotary positional encoding can be described via the differential system
(zy VT
1.
(t) (1)
Σ
(t).))
(8)
dt
Σ (1)) (,)
for l = 1,...,,. L. Rotary positional encoding is an essential component of latent attention which is at the core of Deepseek [31].
Remark 4.2 (Rotary positional encoding encourages token divergence). Unlike absolute positional encoding, the rotary positional encoding exhibits markedly different behavior as its encoding en-
courages token divergence. Indeed, the additional term AWi = Q RP_--- K in the query-key interaction matrix Wi, defined equation (7), can inhibit the system from entering the conver- gence regime, shown in Figure 5. As result, in self-attention mechanisms with rotary positional encoding, the divergence scenario tends to occur more frequently compared to those employing absolute no positional encoding.
To support our findings, we visualize the evolution of token norms and pairwise L2 distances in a pre-trained Transformer model without LayerNorm and feed-forward layers, comparing Rotary Positional Encodings (RoPE) with sinusoidal positional encodings. Figure 6 shows that both token norms and token distances diverge faster with RoPE than with sinusoidal encodings, suggesting that RoPE mitigates the convergence aspect in self-attention.

---

RaFE
RoPE
10a
Mean L2 distance
107
Mean
10a
10
12
14
8
10
12
16
Depth
Depth
Figure Token norm and distance throughout layers of pre- pre-trained model with RoPE and Sinusoidal positional encoding.
Table I: Performance of Transformers with sinusoidal positional encoding across scenarios on language mod- cling task.
Table 2: Top and Top Validation Accuracy of
DeiT (Icarable positional encoding) across differ- ent scenarios on ImageNetIK.
EnWik8 BPC (J)
WikiText- 103 PPL (1)
Scenario
Scenario
Top Acc (1)
Top 5 Acc (T)
Valid
Test
Baseline
1.331
31.63
32.37
DeiT Baseline
71.80
91.01
Convergence
1.350
32.24
33.05
Convergence
71.34
90.64
Intermediate
1.345
31.91
32.78
Intermediate
71.60
90.91
Divergence
1.324
31.12
32.07
Divergence
71.96
91.05
Experiments
In this section, provide empirical validation of our theoretical findings and introduce enhancements to absolute positional encoding and Rotary Positional Embedding (RoPE) in practical Transformer models. We conduct experiments three benchmark tasks: language modeling WikiText- 103 [34] and EnWik8 [20], and image classification on ImageNet- 1K [11]. Our goals are twofold: (1) to show that convergence behavior adversely affects the performance of Transformers and should thus be mitigated; and (2) to demonstrate that encouraging divergence in Transformers improves performance. Throughout the experiments, we compare
modified Transformers with the baseline Transformers of the same configuration. Our results are averaged over runs. Detailed information about the model architecture, hyperparameters, and training procedures provided in Appendix D.
5.1 Negative Effeets of the Convergence Scenario
We empirically examine the impact of convergence and divergence dynamics predicted by our analysis across different Transformer architectures. Specifically, we evaluate models on the WikiText-103 and EnWik8 language modeling tasks using sinusoidal positional encoding, and on ImageNet- KK using Vision Transformer variant DeiT [50] which uses learnable positional encoding. The parameters attention layers are explicitly constrained to conform these distinct scenarios. In the convergence scenario, we impose the conditions Wsym > 0 and Asym O. Conversely, in the divergence scenario, we require WT Wsym and Asym contains an equal number of positive and negative cigenvalues. Further details on the benchmark setup and the implementation of these constraints provided in Appendix D.3 and Appendix D.2, respectively.
0 and Asym > O. We additionally examine an intermediate scenario in which
Table reports model performance across the baseline and three constrained settings. As the con- straints shift toward convergence, performance consistently deteriorates. In contrast, the divergence setting achieves the best results, yielding the lowest BPC and perplexity. On WikiText- 103, the divergence case outperforms the convergence and baseline settings by reducing validation perplexity by 1.12 and 0.51, respectively. On EnWik8, it lowers test BPC to 1.324, compared to 1.350 under convergence and 1.331 under the baseline. These results suggest that divergence constraints improve model performance, while convergence constraints have detrimental effect.

---

Table 3: Bits Per Characters (BPC) and Perplexity (PPL) of Transformers with Rotary Positional Encoding across different scenarios on EnWik8 WikiText-103 language modeling.
EnWik8 Pretrain
WikiText- 103 Pretrain
Scenario
Test BPC (L)
Valid PPL (I)
Test PPL (4)
Transformer + RoPE
1.295
31.37
32.35
Transformer + RoPE Alp
1.288
31.10
32.09
Transformer + RoPE + XA
1.281
31,06
32.04
Divergence
Divergence
Baseline
Beseline
Mean L.2 distance
Internediate
Intermediate
10a
Convergence
10
Convergence
Mean taken
10a
10a
10
6
10
14
16
10
12
14
Depth
Depth
Figure 7: Mean token distance (left) and mean token norm (right) across Transformer layers a pre-trained
model without LayerNorm or feedforward network.
To analyze the impact of convergence and divergence constraints, we plot the mean token norm and mean pairwise L2 distance across layers (without LayerNorm or feed-forward), As shown in Figure 7, both metrics decrease under the convergence constraint, whereas they rise under the intermediate and divergence regimes, with divergence exhibiting the steepest increase. These empirical trends validate our theoretical analysis in Section and suggest that the intermediate setting combines aspects of both convergence and divergence regimes.
5.2 Promoting Divergence in Rotary Positional Embedding
Motivated by the influence the additional term AW on the query-key interaction matrix Wi in promoting divergent token dynamics under RoPE, we propose simple yet effective modification to further mitigate convergence an undesirable regime shown to negatively affect model performance. Specifically, we add learnable regularization term XI or XA E D to Wi. where À is learnable negative scalar, ID is the identity matrix, and is a learnable diagonal matrix with strictly positive entries. This modification effectively learns to subtract positive quantity from the diagonal of Wi., promoting negative eigenvalues, thus discouraging convergence scenario in attention layers.
We provide empirical evidence on Wiki'eext 103 and En Wik8 demonstrating our findings' practical relevance. As shown in Table 3, even minimal intervention yields measurable gains. On En Wik88 adding ATD reduces Test BPC from 1.295 to 1.288, and XA further reduces it to 1.281. On WikiText- 103, both variants outperform ROPE, with XA achieving a 0.3-point improvement Validation and Test perplexity. These results, while modest, are consistent with our theoretical predictions and validate the utility of encouraging divergence self-attention dynamics.
Conclusion
analyzed token dynamics in self-attention and leveraged these insights to improve Transformer performance. Through theoretical investigation, we derived conditions under which tokens converge or diverge during self-attention and explored how positional encodings such as absolute and rotary affect these dynamics. Empirical evaluations on pre-trained Transformers supported our findings, highlighting the negative impact token convergence on performance. To address this, we proposed simple enhancements to self-attention that yielded measurable improvements. A limitation of our analysis its focus on self-attention, omitting components like layer normalization and forward networks. However, empirical validation confirms our conclusions remain relevant in practice. Extending our framework to the full Transformer architecture is a promising direction for future research.
10

---

Acknowledgments and Disclosure of Funding
This research/ / project is supported by the National Research Foundation Singapore under the AI Singapore Programme (AISG Award No: AISG2-TC-22023-112-511
This research / project is supported by the Ministry of Education, Singapore, under the Academic Research Fund Tier (FY2023) (A-8002040-00-00, A-8002039--0-000) This research / project is also supported by the NUS Presidential Young Professorship Award (A-0009807-01-00 and the NUS Artificial Intelligence Institute-Sect Funding (A-8003062-00-00)
References
[1] Juan A Acebrón, Luis L Bonilla, Conrad J Pérez Vicente, Félix Ritort, and Renato Spigler. The kuramoto model: A simple paradigm for synchronization phenomena. Reviews of modern physics, 17(1):137-185, 2005.
[2] Rami AI-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3159-3166. 2019.
[3] Albert Alcalde, Giovanni Fantuzzi, and Enrique Zuazua. Clustering in pure-attention hardmax transformers and its role in sentiment analysis. arXiv preprint wrXiv:2407.0102, 2024.
[4] Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In International Conference on Learning Representations, 2019.
[5] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
[6] Jingpu Cheng, Qianxiao Li, Ting Lin, and Zuowei Shen. Interpolation, approximation and controllability of deep neural networks. arXiv preprint arXiv:2309.0601 2023.
[7] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-173 1734, Doha, Qatar, October 2014. Association for Computational Linguisties.
[8] Zihang Dai, Zhilin Yang. Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-XI. Attentive language models beyond a fixed-length context. In Anna Korho- nen, David Traum, and Lluís Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1978-2988, Florence, Italy, July 2019. Association for Computational Linguistics.
[9] Zihang Dai, Zhilin Yang, Yiming Yang. Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhut- dinov. Transformer-ye xl: Attentive language models beyond a Exxe-llenghh context. In Anna Korhonen, David R. Traum, and Lluís Märquez, editors, Proceedings the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2019, Volume 1: Long Papers, pages 2019.
Association for Computational Linguistics,
[10] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Uni- versal transformers. In International Conference on Learning Representations, 2019.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image datahase. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee, 2009.
[12] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter
11

---

of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[14] Subhabrata Dutta, Tanya Gautam, Soumen Chakrabarti, and Tanmoy Chakraborty. Redesigning the transformer architecture with insights from multi-particle dynamical systems. Advances in Neural Information Processing Systems, 34:5531-555 2021.
[15] Borjan Geshkovski, Hugo Koubbi, Yury Polyanskiy, and Philippe Rigollet. Dynamic metasta- bility in the self-attention model. arXiv preprint arXiv:2410.06833 2024.
[16] Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. mathematical perspective on transformers. arXiv preprint arXiv:2312.10794 2023.
[17] Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. The emergence of clusters in self-attention dynamics. Advances in Neural Information Processing Systems, 36, 2024.
[18] Gene H. Golub and Charles Van Loan, Matrix Computations. The Johns Hopkins University Press, Baltimore, Maryland, 4th edition, 2013.
[19] Seung- Yeal Ha, Jeongho Kim, Jinyeong Park, and Xiongtao Zhang. Complete cluster pre- dictability of the cucker-smale flocking model on the real line. Archive for Rational Mechanics and Analysis, 231:319-365 2019.
[20] Marcus Hutter. The human knowledge compression contest. 2006.
[21] Pimrre-Emmanuel Jabin and Sebastien Motsch. Clustering and asymptotic behavior in opinion formation. Journal of Differential Equations, 257(11):4165 4187, 2014.
[22] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances neural information processing systems, 34:1273- 1286, 2021.
[23] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s):1 41, 2022.
[24] Uirich Krause et al. A discrete nonlinear and non-autonomous model of consensus formation. Communications in difference equations, 2000:227-22 2000.
[25] Yoshiki Kuramoto. Self-entrainment of population of coupled non-linear oscillators. In International Symposium Mathematical Problems in Theoretical Physics: January 23-29, 1975, Kyoto University, Kyoto/Japan, pages 420-422. Springer, 1975.
[26) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Machine Learning. PMLR, 2020.
[27] Qianxiao Li, Ting Lin, and Zuowei Shen. Deep learning via dynamical systems: An ap- proximation perspective. Journal of the European Mathematical Society, 25()):17711179,, 2022.
(28) Hongzhou Lin and Stefanie Jegelka. Resnet with one-neuron hidden layers a universal approximator. Advances in neural information processing systems, 31, 2018.
[29] Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. Al open, 3:1 1-112, 2022.
12

---

[30] Zhouhan Lin, Minwei Feng. Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self- atte-tttetive sentence embedding. In International Conference on Learning Representations, 2017.
[31] Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405. 04434, 2024.
[32] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
[33] Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie- Tin Liu. Understanding and improving transformer from a multi-particle dynamic system point of view. arXiv preprint 1x0vv1900..2200
2019.
[34] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
[35] Sebastien Motsch and Eitan Tadmor. Heterophilious dynamics enhances consensus. SIAM review, 56(4):577-6211 2014.
[36] Tan Minh Nguyen, Tam Minh Nguyen, Hai Ngoc Do, Khai Nguyen, Vishwanath Saragadam, Minh Pham, Nguyen Duy Khuong, Nhat Ho, and Stanley Osher. Improving transformer with an admixture of attention heads. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[37] Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model for natural language inference. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2249-2255, Austin, Texas, November 2016. Association for Computational Linguisties.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages
PMLR, 2021.
[39] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre- pre-training. 2018.
[40] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog. 188)99 2019.
[41] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1667, 2020.
[42] Hegselmann Rainer and Ulrich Krause. Opinion dynamics and bounded confidence: models, analysis and simulation. 2002.
[43] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning. pages 8821-8831 Pmlr, 2021.
[44] Domenec Ruiz-Balet and Enrique Zuazua. Neural ode control for classification, approximation, and transport. SIAM Review, 65(3):735-773, 2023.
[45] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 166::27763, 2024.
[46] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, Neurocomputing. 127063, 2024.
13

---

[47] Paulo Tabuada and Bahman Gharesifard. Universal approximation power of deep residual neural networks through the lens of control. IEEE Transactions on Automatic Control, 68(5))77 2728, 2022.
[48] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1-28, 2022.
[49] Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Anna Korhonen, David Traum, and Lluís Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy, July 2019, Association for Computational Linguistics.
[50] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data- -itta-fffiiintt image transformers & distillation through attention. In International conference on machine learning. pages 10347-10357. PMLR, 2021.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[52] Tamás Vicsek, András Czirók, Eshel Ben-Jacob, Inon Cohen, and Ofer Shochet. Novel type of phase transition in system of self-driven particles. Physical review letters, 75(6):1226, 1995.
[53] Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in transformer language model. In Tal Linzen, Grzegorz Chrupata, Yonatan Belinkov, and Dieuwke Hupkes, editors, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63-76, Florence, Italy, August 2019. Association for Computational Linguistics.
[54] Han Zhang, Xi Gao, Jacob Unterman, and Tom Arodz. Approximation capabilities of neural odes and invertible residual networks. In International Conference on Machine Learning, pages 11086-1 1095. PMLR, 2020.
14

---

Supplement to "Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding"
Contents
Related Works
16
Dynamical Properties of Tokens in Self-Attention
16
B.1
Distances Between Tokens Over Time
16
B.2 Convergence Scenario
17
B.3 Divergence Scenario
20
Effects of Absolute and Rotary Positional Encodings
22
C.1 Absolute Positional Encoding
23
C.2 Rotary Positional Encoding
23
Additional Experimental Details
24
D.1 Parameter settings used in the simulations
24
D.1.1 Figure 1: Distances between tokens time
24
D.1.2 Figure Token trajectories under convergence and divergence scenarios
24
D.1.3 Figure 5: Token trajectories shift to divergence regime in RoPE
25
D.1.4 Figure 9; Token trajectories in convergence and divergence regime
25
D.2
Implementation of Convergence, Divergence, and Intermediate scenarios.
25
D.3
Details on Language Modeling experiments
26
D.3.1
Dataset
26
D.3.2
Wikitext103 Model and Training Configurations
27
D.3.3
EnWik8 Model and Training Configurations
27
D.4
Details on ImageNet- IK object recognition task
27
D.4.1
Dataset
27
D.4.2
Model and Training Configurations
D.5
Additional visualizations
28
D.5.1 Tokens' trajectory in pre-trained Transformers
28
D.5.2
Token trajectories with RoPE
28
D.6
Additional Results... Results......
28
D.6.1
Full Evaluation Results
28
D.6.2
Condition verification pretrained Transformers
28
D.6.3
Layerwise Token Norms and Pairwise Distances in GPT-2
E Broader Impact
30
15

---

A Related Works
Transformers as Interacting Particle Systems. Particle systems offer a novel perspective to understand the dynamics of Transformer models and inspire architectural innovations. In [33], the Transformer architecture is mathematically framed as numerical ODE solver for a convection- diffusion equation in muulti-prtticl dynamic system. Similarly, leveraging insights from numerical ODE solvers, [14] introduces TransEvolve, a temporal evolution scheme inspired by interacting particle dynamics. Furthermore, the ODE governing Transformer dynamics is closely connected to the extensive literature on nonlinear systems, including flocking phenomena [19], the Kuramoto model [25, 1], consensus formation [24, 35), opinion formation [21, 42), and systems of self-driven particles [52].
Clustering Effects of Transformers. Research interacting particle systems has shown that tokens in Transformer models exhibit a long-time clustering phenomenon. Geshkovski et proved in [17] that tokens cluster around limiting objects determined by their initial values, highlighting their context awareness. This was extended to the study of metastability of self-attention with layer normalization in [16, 15]. Additionally, [3] proved that tokens pure-attention hardmax Transformer models asymptotically converge to clustered equilibria.
B Dynamical Properties of Tokens in Self-Attention
Recall that the dynamical system governing the continuous-time dynamic of the self-attention is given by:
(z)
L dt = VT. Σ
e=1(0 (T-0)-Www.ol (W)
-x;(1), 1 == I,...,L, 2-=1 e0i(t) T Wia)(()
(9)
with the initial condition (an (0)).....z(())) (0) = .., (RD))4 In this system, W Q KT The matrices Q, K, are learnable matrices and they are called queries, keys and values matrices, respectively. In our setting, the parameters W and are assumed to be time-independent.
Set A = W (VT)-! We will prove the following observations in the subsequent subsections.
Distances Between Tokens. If A < O. then all tokens will tend to move closer and closer to each other as time approaches infinity. In contrast, if 0, the tokens will either tend maintain constant distances move farther away from each other as time approaches infinity (see Theorem B.2).
Convergence Scenario. If A < 0 and W > 0, then all tokens will tend to zero as the time t tends to infinity (see Theorem B.6).
Divergence Scenario. If V has at least one positive eigenvalue and W is arbitrary, then all tokens will diverge to infinity under certain assumption on the initial data (see Theorem 3.6).
B.1 Distances Between Tokens Over Time
In this subsection, we analyze the dynamical properties of the distances between tokens as the time / increases. We start with the following lemma, which can be seen refinement of [17, Lemma 7.1].
Lemma B.1. Let W be an arbitrary matrix in R DxD, For each æ1...., .....,,LL EL in RD, the function f: R + R defined by
f(u) log
Σ "
-Wazy
E R
is convex function.
16

---

Proof. For arbitrary E RR,
we have
1
2
(()))((()) -22 =
e Wizi Σ e Too--wes -W.zs
Σ
c
(22) Wo
-
(((ttt)) T-W-as + elutt))) (oottt))wwi)))
W w. W
2
1
WeP: - W.)) 2
2
w
2 0.
Therefore, [() + [(u) 2 2/ (*=2) Hence, is convex.
The function defined the above lemma is not strictly convex as the equality happens when V =0 0.
Theorem B.2. Lef (z1(4),..... x.) C ((0, +x)))) be solution of the dynamical system (9). Assume that is invertible and A = W (VVT)- is symmetric. Then the map t + (x, - 2z (t) non- nec-eaccgg on [0, +oc).
As a consequence.
(a) A > then (t, x (t)lla a is non-decreasing on [0, +o0);
(b) if A then (() xy (t)lla is non-increasing on [0, +00).
Proof. Fix an arbitrary time E (0, +oc). Let f be the function defined in Lemma B.1 (with zy there is replaced by (s)) in this proof). Then we see that
af Ou
W Wor(())
(u) = W.
(8).
WW2,(()
1=1 ==
Therefore, from the dynamical system (9), we have
0f (I](s)) A
1= I,...,, L. (s)),
(10)
Ou
ds
Since is convex, have
0f
of
(1:(s) (1,5))
(xx(5))
(zz(s)) 2 0.
Du
Du
From equation (10), we have
(x) (s))) A
- 2,,)) 2
d
ds
ds
Since symmetric, follows from the above inequality that:
d (z:((s))) 2s)s)) 2 0.
ds
This shows that the map (s) x, 2s)s)) is non-decreasing [0, +00). The items (a) and (b) are obtained due to the definition of ||A-
B.2 Convergence Scenario
In this section, we will prove that when A 0 and W > 0, the solution of dynamical system (9) tends to zero as L. approaches infinity. The special case where A = -Tn and W = Tp was already proved in [17, Section 8.2]. We borrow the proof technique from [17, Section 8.2] and carefully refine it so that it applics to a broader generalization.
17

---

Proposition B.3. Let (x(())....,z((()) E 000 + be solution of the dynamical sys- Tem (9). Assume that V is invertible and A = W. (VT)-1 is symmetric. Then, we have dt 9A(ai(t)) 2 2- for all I 1....,, and I. (0, ))). As consequence, if A < 0 and Wam > 0, then [|r2(1)]aa is bounded for all I,...,,.. L.
d
2L
eaw (t)
Proof. Since A is symmetric, we have
qa (z)(t)) = 21(() T, W. W.(V (.VVV)- ( d dt
(t
2 dt
Σl= emi(t) W.x((t) 2,) W 2t())
Σj=1 TW
D (a1()) .W.x.()) -
ΣE=1 (ar(t) (
L
Σラ= (0) (t)
In the above estimation, to obtain the inequality in the third line, used the fact that e ed for all À E R. As a consequence, we have
d
2L
91 (a()((t)) 2 2
(11)
eaw (xx
as claimed.
Next, assume that A < 0 and W sym follows from equation (11) that
> 0. There is constant e > 0 such that -1 2 ely. Then
2L
[t)(1 2 2
dt
(ella((()11
or equivalently,
d
2L
(t) < -2 e (+))1)
Hence, [2x log (e-2et e (e)))=(0)) L.) + l.). which is bounded.
The following lemma studies the limitation of the derivative of tokens in case A < 0 and W > 0. Lemma B.4. Let (x) Zz. E ( +c be the unique solution of the dynamical system (9), Set A = W (V ) If A x and W > 0, then
d
2
2z (s)
ds < +00,
ds
for all I. In particular, we have lim, +000 बe (t)
0 for all I.
Proof. Consider the function h: [0. +00) + R defined by
h(t) =
Then a positive function. Since W symmetric, the derivative of h is
M(11) 2
(1) W. (1)
dt
dt
1=1
Σ d dt 21(1)
21(1) T. TW (1) mz(()
.W.
2 Σ d
di (t)
Σ (T) w-2z(0) rt)
dt
18

---

Since A < 0, we have =z,(t) A zz;(t)
21)1 Therefore, proceed the above
expression as
d _ h||)= - Σ
T Way
(12)
i=1
which is nonpositive. As a consequence, h(t) is non-increasing. Thus, lim h(t) exists and finite.
Next, since A < and W > 0, follows from Proposition B.3 that (t) are bounded for all I == 1.....,, L. Therefore, there exists é > such that
- W-x 26.
for all E [0,+ +00) and I,...
, L.
follows from equation (12) that
d
d dt
|
h(t) - -2
21(1)
dt
By taking the integral both sides, that
d
1 (M(0)
ds mis)
ds S
lim h(s) < to0.
20
The lemma is then proved.
We will also require the following lemma, which holds for any matrix W whose symmetric component W Waym is either positive definite or negative definite. The special case where W = /D was established in [17, Lemma 8.8]. Our proof, however, is simpler and extends more general matrices W.
Lemma B.5. Assume that W sym is (either positive or negative) definite. Let xi...., 2t, be point in RD such that
Σ ==1
W.jj. 2. = 0, vI == 1,.....,, L. [o]) To
Then oi = =... = =0
Proof. Consider the function g: RDD + defined by
g(u)
uu W- Vu R.P.
Then for arbitrary u, v E RDD, we have
+
edu [e W _ wwai) w
s(u) g(v) 2g
2 20
1=1
Therefore g is convex. From the hypothesis, we have
Vg(zi)= =...-
This means that -i... # are global minimum of y and the values of y at these points are all equal. Since is convex, g achieves the global minimal value on the convex hull conv({2;)[ As consequence, have
x + x
g(x;) =(x;)) =
for all i.j. Therefore,
0- 0=g(z?) + 9(x;)) 2g
x
+
+x
Lo W
-et
2
19

---

This happens only when
1
2
W 27 " WW.j., #i,
or equivalently.
(a-- ( m3) W. 1. = 0,
for all 1,.... L. In particular, have
qw (x) - 23)) = (2) =5)² W. x.. (xi xzy]² W. 1. = 0.
Since W sym is definite, qw is nondegenerate and x = 23. Thus of _... x The only possibility is γ -... x 0.
We are ready to prove the main result of the convergence scenario. Theorem B.6. Let (t)(t)...... (t)) E C ((O, +cc)) be solution of the dynamical system (9). If A < O and W > 0, then lim (t) = for all :....:L. L.
Proof. Set (t) (21 (t)) (t). We need to prove that lim X(t) 0. Assume that this is not the case. According to item (a) of Proposition B.3, X(t) lies a compact subspace of (R. Therefore, there exists sequence (t& in [0, +00) such that limg-++od t42 too and lime ((x) = X for some X = (ii...., E (RD)² and X * 0. As conse- quence, have + mn (4x) sं for each 1,..., L. While, according to Lemma B.4, lim d =t(a)) 0. Therefore, from the dynamical system (9), we obtain
Tx
2 = == L. Dj=1 (af) Waa
(13)
Then it follows from Lemma B.5 that oj o = 0. However, this contradict to the fact that X" # O. Hence, lim;-++ee X(t) and the theorem proved.
B.3 Divergence Scenario
To simplify the technical details, we will consider the case where A = XW, i.e. V = D.. . for some positive real number A. The case where A and W have the same signs but * >W may require certain adaptations. In particular, we will prove that when V = AlD and W is arbitrary, all tokens tend to infinity at an exponential rate (under certain assumptions the initial conditions). The case where V = W Ip was already solved in [17, Section 8J. We borrow the technique from there and modify ensure works for all V AlD and arbitrary W.
In the following, for each subset H C IR,", we denote by conv (H) the convex hull of H, which is the smallest convex set containing H in R?, For point u, the notation d(u, H) represents the Euclidean distance from to H, which is defined as
d(u, H) = inf ||u w/|-
vEH
If H is a closed half-space of D with an outer normal vector n and u & H. then
d(u, H) = n T (u- (w))
where proju (u) is the projection of onto H. We begin with the following lemma, whose proof can be found in [17].
Lemma B.7. Let H be a closed half-space of ED with an outer wnit normal vector n. Let
U1... uL: [0, Hoo) + RD be an arbitrary sequence of differentiable functions. For each
t [0, (00), set
Min(t) = (1 < L H) = min d(uu(t), H)}.
1SiSL
Then we have
d min d(u,(t), H) 18:L
nnT
dt mn(t)
min i (M)i
dt
Proof. This lemma is already proved in the proof of [17, Proposition 8.2].
20

---

The following proposition refines [17, Proposition 8.2], where the condition V = W ID was assumed. Here, we extend the proof by relaxing this condition and demonstrating that the result remains valid for matrices of the form V = AT with À > O. without imposing any constraints on W. Another proof of this proposition can also be found [21, Proposition 2.1].
Proposition B.8. Assume that V AlD for some real number X 0 and W is arbitrary matrix in TR. D Let (I1 (0h...,,,((()) E -* ([0, +o))) be the unique solution of the dynamical system (9). Then
(t)) E conv ((I:0)==)),
for all 1,.... and E [0, +00).
Proof. Let 21) e-xx) Then have dt (t) = -Ac
ai(() - XxI(1)
dt
a W.iz
e At
1
22(())) (1))
Cechtej W.e.
1=1
==
(=(1)-
E===1 CnRAN Wiz:
Therefore, the function (z,,
(t)) satisties the dynamical system:
dz((1) (t) (z)) (z((1) (t),,, 1= L....,,4
(14)
dl
with the initial conditions (0) = Iı0, where
Go T. WW
Pt. 21, =1)
X.
1==1 C -W-s
claim that, for every closed half-space H DR such that conv((2 a: [0, +x)) + IR defined by
0, the map
a(t) min d (12 H) isitt
non-decreasing. Indeed, using item (a) of Lemma B.7, we have
a(t
min iEMin(!)
n dt
2t(t)
dt
min sEMin(1)
Σ
n. (2y(t) =2(1))
On the right hand side, we have
n (z) (t)) = n T (2)(t) - (r))) -n (z))
+n (
= dd); H) - d(z(().
which nonnegative since i Min((). Therefore, a 2 and non-decreasing. As consequence, we have
d(=x(t), H!) > min d(am H1 0, 1S1SL
for all L....,L and E [0, +oo) This means that, (t) outside H as long as и n conv (1 = 0. Hence,
(t) E
H
H' = conv ((ralte)).
H closed half-space co )=
H' open half-space
H (coo)(()
The proposition is then proved.
21

---

In the following, for each nonzero vector E R,,, we denote by Hn the closed half- of IR' with normal vector n such that zero belongs to its boundary äHn, In this case, HH, represents the hyperplane containing zero with normal vector n.
Theorem B.9. Assume that the value matrix V has at least one positive eigenvalue. Let n be an positive eigenvalue. Then
eigenvector of V corresponding
min (n T 240) Sn T TV 21(t) S max 1SSS
(n 220),
isish
for all (50) and 1...., L.
As a consequence, if the initial points x. Z10, 30 are all on one side of the hyperplane HHn and if V has only positive eigenvalues, then
lim |لt(t)]| = +o0, for all L.
Proof. Let 21) e-V (t). Then the function (z1(t),... zt)()) satisfies the dynamical system:
L (t) Σ Pr,((, 21 (E)...)(())......)))))) V (z,(1) - 21(1),, 1= l.... L,
(15)
with the initial conditions 20) = 3i0, where
Σ/1 etl et W.W.Weel
Let be the eigenvalue associated n. By multiplying n² into both sides of equation (15), and set (t) n 21 (t), we obtain
(t)(t) dt
L Σ Pr.ltt (2t1)) A (m)(t) (t)), == 1= L,
(16)
with the initial conditions (0) = n æ10 E R. By using the same argument in Proposition B.8,
with 2 is replaced by y here, we see that
(((t) E conv (1HO))=1)
and hence,
min yio S (t) S ISISL
max yio. IsiL
for all [0, +00) and / I,.., L. Therefore.
min ISEL
(n' T r20) < In -- -VT (t()) < max (n" T r23)
1S1T.
claimed.
In case the initial points E10...., FLO are all one side of the hyperplanc ÖH,, then either ming<ish (n" =30) > 0 or maxigiet (n" z:0) < O. In both case, there is > 0 such that
-v mi(1)
for all E [0, +00) and I. Hence, when V has only positive eigenvalues, must have
+oo [x((1)] = +oc, for all I.
lim
The theorem is then proved.
C Effects of Absolute and Rotary Positional Encodings
In this section, we analyze the influence of absolute and rotary positional encodings on the dynamical behavior of tokens within self-attention mechanisms.
22

---

C.1 Absolute Positional Encoding
Recall that the dynamical system governing the continuous-time limit of the self-attention with absolute positional encoding is given by
drr(() dt
c(a10)mm) (w00000)))
) (zi(t) + p.).
= =V
(17)
for 1 = =1 .....,,44 L. with the initial conditions
(x) (z)
(0), ................................ xr)()) (æ100 (2 Fro) (R.)".
Here P: P........ PI E RR. A common choice is to either learn the positional embeddings ps jointly with the model parameters, or to fix them using a sinusoidal scheme: with
sin(i i 10000 b),
if is even,
PiJ
cos(i - 100000000000 - if j is odd.
This differential system be easily transformed into equation (2) by the transition æn 22 pt- Therefore, the dynamical properties of the self-attention with and without absolute are the similar as we will see in the following corollaries.
Corollary C.1. Let (x1 (4)....,4:) x1)( E ( +00)) be solution of the dynamical system (17). < 0 and W O, then lim,- (1) -p for all 1...., T..
Corollary C.2. Let (() = (x1(t)...., 2L (t) E C (10, ((0)+))))' system (17). Assume that the value matrix V has at least one positive eigenvalue. Let be
a solution of the differential
eigenvector of V corresponding to posilive eigenvalue. Then
min 18iS
(n (x=o Pi) etv pi) Tnee- 21(1) S max (n (2:0 + p:)) -tV
Pt.
18iS
for all t E [0, 500) and = L.
As consequence, if the points 010 II. PL all side of the hyperplane OHn + evV Pt, and V has only positive eigenvalues, then lim |[x(())| = +00 for all I. t10000
C.2 Rotary Positional Encoding
In this section, we assume that the token dimension is an even number. In contrast to absolute po- sitional encoding, the continuous-time dynamics of the self-attention with rotary positional encoding can be described via the differential system [46]
dzy(t) VT
L
(A) (1,
xt),),
(18)
dt
L
(1) (1( 21)
for l ..., L. where
(Q-KT R..-1K
WB DD
(19)
with Q, K, Q. K E DXD are two additional learnable matrices and
coos m0
sin m01
0
0
0
sin m0,
m0
0
0
0
0
COS m0
sin m0 cos md,
sin mfl
R
0
0
0
cos m0
sin sinm0pp sin mf D/2 cos m20 D/2
0
and θ = (0, = 10000-2(6-1/D, E
Rotary positional encoding is an essential component of latent attention, which lies at the core of DeepSeek [31]. Unlike absolute positional encoding, the differential system governing rotary positional encoding exhibits markedly different behavior. In cases where token trajectories diverge to infinity under absolute positional encoding, we observe similar divergence scenario the presence of rotary positional encoding, demonstrated below:
23

---

Corollary C.3. Let (t) (..(t)..... E (1,, - be solution of the differential system (18). Assume that the value matrix V has at least one positive eigenvalue. Let n be an eigenvector of V corresponding to positive eigenvalue. Then
min ra)) I n Te 15:5L
evv 21(1) S max (n T:n),
ISiSL
for all t E +00) and 1, 1,...,,. L.
As a consequence, if the points ........ To are all on one side of the hyperplane TH and if V has only positive eigenvalues, then
lim 2++00
||x((00]| = +00, for all l.
Proof. Apply the same argument as the proof of Theorem B.9.
Remark C.4. In contrast to absolute positional encoding, rotary positional encoding induces notably different dynamical behavior by promoting token divergence (even for the cases when tokens converge to finite point in the absolute positional encoding). Specifically, the presence of the additional term
Q R- K in the query-key interaction matrix Wt, as defined in equation (19), can hinder the system from transitioning into a convergence regime. Consequently, self-attention equipped with rotary positional encoding tends to exhibit divergence behavior more frequently than those using absolute encoding or no positional encoding at all.
D Additional Experimental Details
D.1 Parameter settings used in the simulations
D.1.1 Figure 1: Distances between tokens over time
On the left side of Figure 1, choose
A 1.72628
3.795922
whose symmetric component Asym has positive eigenval-
0.914069
3.49779
ues 5. 2809 and 0.0959758;
0.534636
-0.798866
W
and
-1.17152
-1.92153
the initial values T10 (- 1-117552 1.99834), T20 (-0.0231564, 0.5916677, 250
(-0.94811, 1.37999)) x4 = (1.00246, 1.69333).
On the right side of Figure 1, we choose
1.43778
-1.10989
A =
whose symmetric component Asym has negative eigenval-
0.563455
-0.401696
ues
1.50541 and
-0.334061;
0.433083
-0.0371911
W =
and
0.715343
-1.53568
the initial values T10 = (0.123688, 0,20691,, (æ20 (0.53086, 1.47281), 030 =
(-0.78388, 1 Zao (1.63476, 0.321809.
D.1.2 Figure 2: Token trajectories under convergence and divergence scenarios In Figure 2a, we choose the following parameters:
2.940558
22.2277
1 =
thus the symmetric component Asm has negative eigenval-
5..14998
-4.58104
ues
7.48513 and
-0.0364942;
W = (0.902496
-2.37879)
thus the symmetric component W sym has positive eigen-
4.36478
3.84768
values 4.15119 and 0.598979.
Figure 2b illustrates the divergence scenario with A 2W (thus = 2/) and
0.404478
0.982735
W = 0.567909
0.600242
24

---

D.1.3 Figure 5: Token trajectories shift to divergence regime in RoPE
On the left of Figure 5, choose
0.07331133 137 0.32738218
0.17647239
0.43457359
- 2. 54009796
1829916622
thus Wsym have positive eigenvalues 0.03766541
0..0..95 05.55686677
0.60349328)
and 0.15005164
-1.51
On the right of Figure 5, we choose the additional parameters
-3001517413
2.4430872
Q
2.11630464
1.40111342
5.03454859 4.58643881
-3.12492845
K
-2.00780098
D.1.4 Figure 9: Token trajectories in convergence and divergence regime
On Figure 9a (convergence case), we choose the following parameters:
1. 187765551
0.8975229 0. 0.79105257
Q = 0. 7793589
1. 1.75520662
-1.98198651
K
2.24167927
2.93460903
1.04027991
0.12991073]
Q
L.32542484
1.08074871
1.00477795
0.48804888
0.42151108
0.02556926
On Figure 9b (divergence case), we choose the following parameters:
2.068739
1.83750201 0.4784381
Q
0.75622145
1. 1.12583337
-1.40120114
K =
2.79629618
-3.24668939
0.67880222
1.21234986
Q
0.67132474
0.90406252
0.57406142
2.88899216
K
-1.10421806
O. 0.75603913
V 1.5l
D.2
Implementation of Convergence, Divergence, and Intermediate scenarios.
We propose strategy to guarantee that the model falls into the three scenarios described in Section 5, and Section 3.4. In particular, we introduce reparameterizzaiin of the matrices Q, K, and that enforces the positive negative definiteness of the matrices
1 (W + WT), Asm (A+ 2
W sym
(20)
where W = QKT and A W(VT)-1 We first describe the approach to guarantee the positive definiteness of Asm and W Wym and subsequently extend the framework to accommodate negative definiteness and intermediate cases.
25

---

Ensuring the Positive Definiteness of Asym- To enforee the positive definiteness of Asm- we leverage the LDDIT decomposition [18]. Specifically, we parametrize:
Asm L. LaLa
(21)
where Le is a lower triangular matrix with ones on the diagonal, and Da is a diagonal matrix with strictly positive elements. The positivity of Da is ensured by applying the Softplus function:
De diag (Softplus(da)).
(22)
Parametrization of and W. Given Asm. from Eq. 20 we obtain
Asym + Xaz
(23)
where X₂ is antisymmetric matrix satisfying Xa -X A natural parametrization for such matrix is:
X, Ta-
(24)
Consequently, the final parametrization of ensuring positive definiteness of Asym is
A = La disg((Sftpluus(dd)) To - -TT--
(25)
A similar parametrization applies to W to enforce the positive definiteness of W Wsm:
W = Lu diag(Softpluus(44))4... + Tw TT.
(26)
The above formulations ensures that both W sym and Asym maintain the desired definiteness properties while allowing for flexible parametrization of W and A.
Parametrization of Q, K, and V. In terms of Q, K, and V, the aforementioned formulations equivalent to:
Q-KTT (E.KT(VTT+
L. diag(Softplus (d + TT, -T La diag(Softplus(dd)) To -TT
(27)
We designate Q, Lews dwu, Las da Two and Ta as learnable parameters. The key and value projection matrices for self-attention then computed as:
K = [Q-1 Lu diag(Softplus(d..) + TT - -T (Tn diag (ioi + +Ta T (Tw diag((Sfttllus(dd))) + + -TT
(28)
This parametrization guarantees that Asym and W sym positive definite.
A Adjustments for Negative Definite and Intermediate Cases. The definiteness of Asym and W Wsym is determined by the sign of the elements in D. and Dw.. respectively. In the positive definite case, these elements are constrained to be positive using the Softplus function. To adapt to the negative definite and intermediate cases, the output of Softplus with a vector containing only -1 1 for the negative definite case or a vector containing equal number of and -1 for the intermediate case.
control the sign of the elements in D, and D by multiplying
This approach ensures that our parametrization flexible enough to accommodate different definite- ness requirements while maintaining learnability and numerical stability.
D.3 Details Language Modeling experiments
D.3.1 Dataset
WikiText-103, [34] The WikiText- 103 dataset comprises approximately 268,000 unique words. Its training set includes around 28,000 articles, totaling 103 million tokens. On average, this corresponds to text blocks of about 3,600 words per article. The validation and test sets each consist of 60 articles, containing 218,000 and 246,000 tokens, respectively.
Wnkik8. [20] The Enwik8 dataset is a byte-level corpus comprising 100 million bytes extracted from Wikipedia. In addition to standard English text, it includes markup, special characters, and content in multiple languages. The standard split provides million bytes for training and million for testing.
26

---

D.3.2 Wikitext103 Model and Training Configurations
Model. We utilize the Transformer-XI. [9] ture for word-fevel language modeling on the Wiki'Tex-- 103 dataset. The model comprises 16 layers, each with hidden size of 410. Multi-head attention is implemented with 10 heads, each having dimensionality of 41. The position-wise feedforward networks have an inner dimension of 2100, Regularization is applied via dropout rate of 0.05 on residual connections. For Rotary positional embedding, the rotational dimension used is 16.
architec-
Training Configurations. Training is conducted using the Adam optimizer with learning rate of 0.00025. A linear warmup is applied for the first 1,000 steps, followed by a cosine annealing schedule over total of 200,000 training steps. The model is trained with target sequence length of 150 tokens and no memory length, effectively disabling the segment-level recurrence mechanism. Evaluation is performed with slightly longer target length of 156 tokens. Training utilizes 2 NVIDIA A100 SXM4 80GB GPUs with a total batch size of 60.
D.3.3 EnWik8 Model and Training Configurations
Model. We trained an autoregressive Transformer model on the Enwik8 dataset using the transformers (httss//github.com//i architecture, comprises 6-layer Transformer decoder. Each layer uses attention heads and a model dimension of 512. For tokenization, byte-level encoding was used, resulting in vocabulary size of 256 unique tokens. Both training and generation sequences were fixed at 1024 tokens.
library. The model follows a GPT--ttll
Training Configurations. Data was sampled into overlapping sequences of length 1025 (1024 input tokens plus target token). We used the Adam optimizer with learning rate of x x10-1, and applied gradient clipping with a maximum norm of 0.5, Gradients were accumulated over 4 steps to simulate effective batch
larger batch sizes. A batch size of 4 was used, with gradient accumulation yielding size of 16. The model was trained for 100,000 iterations. Validation was performed every 100 steps, and text samples were generated every 500 steps with generation length of 1024 tokens. Training was performed on a NVIDIA A100 SXM4 80GB GPU.
D.4
Details ImageNet-1K -K object recognition task
D.4.1
Dataset
ImageNet-IK. [11] This dataset spans 1000 object classes and contains 1,281,167 training images, 50,000 validation images. The model learns predict the class of the input image among 1000 categories. We report the top-1 and top-5 accuracy on all experiments.
D.4.2 Model and Training Configurations
architecture, lightweight Vision Transformer (ViT) variant designed for efficient image classification. The model used is Tiny version, which divides each 224x224 input image into non-overlapping 16x16 patches, resulting in sequence of 196 tokens. Each patch is linearly projected into a 192-dimensional embed- ding space. The Transformer encoder consists of 12 layers, each employing multi-head self-attention with 3 heads, and an MLP block with hidden dimension four times the embedding size (i.e., 768), Biases are included in the query, key, and value projections, and layer normalization applied with epsilon value of le-6. The model includes a learnable [CLS] token and absolute positional embeddings. No distillation token or teacher model was used, and no architectural modifications (e.g., convolutional stems hybrids) were introduced.
Models. We adopted the DeiT [50]
Training Configurationss The model was trained on the full ImageNet- Ik training set for 300 epochs using the AdamW optimizer with a base learning rate of 5 x 10-1, and weight decay of 0.05. The learning rate followed a cosine decay schedule, with a linear warmup phase over the first 5 epochs, and minimum learning rate of 10 10-5. A stochastic depth rate (drop path) of 0.1 were used for regularization. Data augmentation included RandAugment, Mixup with a 0.8, CutMix with 1.0, label smoothing with 0.1, and random erasing with a probability of 0.25. The model was trained with a batch size of 64 across 4 NVIDIA A A00 SXM4 80GB GPUs using mixed precision. Exponential Moving Average (EMA) of model weights was maintained with decay factor of 0.99996.
27

---

Second dimension
Second dimensioon
-22
-100
-0.
-25
First dimension
First dimension
With NayerNorm, without feedforward.
With both LayerNora and fecdforward.
Figure 8: Tokens' trajectory in the later layers of a pre-trained GPT-2 model forms clusters with similar shapes.
D.5
Additional visualizations
D.5.1
Tokens' trajectory in pre-trained Transformers
We provide additional visualization pre-trained GPT-2 model observe the trajectory of tokens. As illustrated in Figure 8, tokens trajectories later layers still forms clusters with similar shapes.
D.5.2
Token trajectories with RoPE
We provide additional simulations of token trajectories the system of Equation for the convergence (Figure 9a) and divergence case (Figure 9b).
X3
0.5
10.0
7.5
0.5
-1.0
2.5
0.0
-1.5
-2.5
-2.0
-2.5
0.5
0.0
0.5
1.0
1.5
10
20
(a) Convergence case of Self-Attention RoPE. To- kens tend converge zero when co.
(b) Divergence of Self-Attention RoPE. Tokens tend to diverge to oo when - 3o.
Figure 9: Self-Attention with RoPE under (a) convergence and (b) divergence settings.
D.6
Additional Results
D.6.1 Full Evaluation Results
In this section, we demonstrate the means and standard deviations the experiments we executed. Table 4, 5, show the means and standard deviations of our experiments in Section 5.2, Section 5.1.
D.6.2 Condition verification pretrained Transformers
performed an empirical analysis of real-world pretrained Transformer models to verify the conditions in our theory. Specifically, for each model, measured the mean percentage across
28

---

Table 4: Bits Per Characters (BPC) and Perplexity (PPL) of Transformers with Rotary Positional Encoding across different scenarios on EnWik8 and WikiText-103 language modeling.
EnWik8 Pretrain
WikiText-109 103 Pretrain
Scenario
Test BPC ())
Valid PPL (4)
Test PPL (L)
Transformer + RoPE
1.295 + 0.003
31.37 + 0.14
32.35 + 0.18
Transformer RoPE AlD
1.288 + 0.002
31.10 + 0.16
32.09 + 0.17
Transformer + RoPE AA
1.281 + 0.002
31.06 + 0.11
32.04 + 0.14
Table Bits Per Characters (BPC) and Perplexity (PPL) Transformers with sinusoidal positional encoding across scenarios on En Wik8 and Wiki 103 language modeling.
Wnki Pretrain
WikiText- 103 Pretrain
Scenario
Test BPC (L)
Valid PPL (1)
Test PPI. (L)
Baseline
1.331 + 0.002
31.63 + 0.12
32.37 = 0.10
Convergence
1.350 + 0.003
32.24 + 0.15
33.05 0.13
Intermediate
1.345 + 0.002
31.91 0.11
32.78 + 0.12
Divergence
1.324 + 0.002
81.12
32.07 0.13
Table 6: Top and 5op Validation Accuracy of DeiT (learnale positional encoding) across different scenarios ImageNetIK.
Top-1 Acc (1)
Top-5 Acc (t)
DeiT baseline
71.79 + 0.02
90.99 0.02
+ convergence
71.37 0.02
90.65 0.01
+ intermediate
71.62 + 0.03
90.94 0.02
+ divergence
71.93 = 0.02
91.03 = 0.03
all layers of (i) "near-zero" eigenvalues of the value matrix V (using threshold 10-3), and (ii) positive eigenvalues of the symmetrized matrices W Weym and Asym.
Table 1 reports the mean + standard deviation of these percentages for GPT-2 XL. DistilGPT2, and LLaMA-2 13B. We observe that: (i) all cases, exhibited 0% near-zero eigenvalues, indicating that FGPTV is invertible in practice. (ii) The proportion of positive eigenvalues in both W Waym and Asym is approximately 50%, which matches the divergence regime predicted by Remark 3.5 omitting FFNs and LayerNorms. The divergence behaviour of pretrained model is also illustrated in 'bascline' case of Figure 6 in Section 5.1.
These findings align with theoretical expectations: (i) the set of singular matrices has measure zero in continuous parameter spaces, and (ii) unconstrained weight matrices, whether randomly initialized or learned, tend to have eigenvalue distributions symmetric about zero. We also repeated this analysis on a randomly initialized model (GPT-2 XL reinit) and obtained similar results (approximately 50% positive eigenvalues in W sym Asyms and near-zero eigenvalues in V).
D.6.3 Layerwise Token Norms and Pairwise Distances in GPT-2
We measured the distance between tokens and token norms across layers using test sequences consist of 100 tokens from WikiText- 1033. Two variants considered: simplified (GPT-2 with no FFNs or LayerNorms) and full GPT-2 models.
In the simplified model (Figures 10a and 10b), both the mean distance between tokens and the mean token norm grow exponentially across layers, consistent with the behavior of the divergence scenario.
In contrast, for the full GPT-2 model (which includes FFNs and LayerNorms), we observe that both token norms and distances increase the early layers but plateau or decline in the later layers (see Figures 10c and 10d). This behavior deviates from our theoretical predictions, indicating that our result does not fully hold in the presence of LayerNorms and FFNs. A comprehensive theoretical and empirical analysis of these effects is promising direction for future work.
29

---

Table 7: Mean + std. dev. of eigenvalue statistics (in %).
Model
% positive cigval of W % positive eigval of Asym
% near-zero-eig of V
GPT2-x1
50.20 = 3.76
49.99 + 0.06
0.00
DistilGPT2
46.01 = 4.36
49.96 + 0.16
0.00 0.00
Llama2 13B
52.35 = 2.84
50.00 + 0.02
0.00
50.20 = 2.85 GPT2-xI reinit
49.99 = 0.06
0.00
Mean distance between tokens
10!
Mean
Muan
10
10ts
10
norm
10tt
20
10a
10a
10'
10²
10a
101
101
10
10
2
3
5
11
12
Layer
Layer
(a) Mean distance (w/o FFNs & LayerNorms)
(b) Mean token norm (w/o FFNs & LayerNorms)
Mcan distance between takens
250
Mean
250
Mean
=1 std
200
200
150
100
50
Layer
Layer
(c) Mean distance (full GPT-2)
(d) Mean token norm (full GPT-2)
Figure 10: Layerwise token 100-token sequence. Top: GPT-2 without FFNs/LayerNorms; Bottom: full
GPT-2. Left: mean distance between tokens; Right: mean token norm.
Additionally, present extended experimental results using broader range of sequence lengths and a larger set of test sequences to provide a more comprehensive analysis of token behavior. In Figures 1lb and lla, report the mean token norm and the mean distance between tokens, respectively, for sequence lengths of 16, 64, 100, 128, and 256 tokens. For each length, the reported values averaged over 100 randomly selected sequences from the test of WikiText-103..
E Broader Impact
This work advances the theoretical understanding of token dynamics in Transformer models, providing insights that could enhance model stability and performance across various applications. While primarily theoretical, these findings may inform the development of more robust Al systems. However, as with any advancement in AI, there is potential for misuse applications. We encourage the community consider these implications and promote responsible of such technologies.
30

---

1047
10fr
Mean diettnnc bitwecen tokens
16 tolens
16 tobens
10
64 takens
64 tokens
1044
1044
128 tokens
128 tokens
2011
2011
+ 256 takens
256 tokens
Mosn
307
103
10
10a
6
7
10
5
6
12
Layer
Layer
(a) Mean distance vs. layer.
(b) Mean token norm vs. layer.
Figure 11: Sequence-length sensitivity GPT-2 without FYNs/lay (100 sequences per length).
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have mentioned the Limitations in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
31

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We have provided the full set of assumptions and complete proof for the theoretical results in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided all experimental setup in Appendix D. Besides, we also provide the code reproduce the results in the paper, which can be found in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
32

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have provided the code in the supplemental material, with sufficient instructions to reproduce the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
Baliii/ggiides/odeeuuiie oonpooooiiiiooooioocooooooo/o for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/nips. wttpcccccpcpcppeeececceccccccctt
ssi cesiiniylcy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We have specified all the training and test details, which can be found in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
33

---

7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have provided error bars for an experiment in Section D.6.1. Besides, all the experiments are averaged over runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етгог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided information about computing resources needed in Ap- pendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
34

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impact of the work performed Appendix E.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymentt of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
35

---

Answer: [Yes]
Justification: The paper has credited all code and data that has been used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi tacoieccoo.... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The source code of the paper is based on existing source (which has been credited properly). We also provide README file to run the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
36

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //tt://ne....peeeee cc//onnerrencee22255/////// for what should or should not be described.
37


Reviews:
[{"review_id": "iD1Wct7AFg", "reviewer": "Reviewer_waqm", "summary": "This paper studies the dynamics of tokens in transformers (removing all blocks other than the self-attention) when all the parameters are trained and fixed. By viewing the forward of tokens as a continuous (partial) differential equation, the authors are able to reveal the divergence ($||x|| \\to \\infty$) and convergence ($x \\to 0$) phases of tokens that depend explicitly on $W$ (by merging the query matrix $Q$ and key matrix $K$) and the value matrix $V$. As an application, by exploring how different positional encodings can change the dynamics, the authors propose a new simple technique to encourage token divergence to improve the performance of transformers.", "questions": "1. Could the authors explain what the conclusions of the current manuscript will be if all the weights depend on the variable $t$?\n\n2. Could the conclusions/prediction be applied to real-world pre-trained models? And if so, what will the comparison between a randomly initialized model and the pre-trained model be regarding the conditions given by this paper?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. This paper is overall well written. In particular, the authors provide a clear motivation for the problem being studied and organize the flow of the paper very well, hence it is very easy to understand the main results, i.e., the main results for the dynamics of tokens are clearly summarized as three parts: the distance between tokens, the convergence of tokens, and the divergence of tokens.\n\n2. The conditions for the divergence and convergence phases of tokens are surprisingly simple: by examining whether $W$ and $A$ are positive definite or not. Thus the conditions might be easily checked in practice by examining the eigenvalues of these matrices. \n\n## Weaknesses\n\n1. Please correct me if I understand the setting in a wrong way: the parameters are indeed trained and fixed, however, the time variable $t$ refers to the depth of the transformers. Hence the fact that $W$ and $A$ do not depend on $t$ means that the merged weight $W$ of attention and the value matrix $V$ are also the same for all layers. This does not align with the actual case. As the cluster of tokens has already been studied by prior works, I think it is better to take a step further to remove the limitation by considering a case where $W$ can depend on $t$. \n\n2. One aspect that has not been touched by this paper is the statistics of the conditions for $W$ and $A$ of real-world pre-trained models (not manually designed by some re-parameterization as in Appendix D.2). Specifically, the condition that the value matrix $V$ is invertible cannot necessarily be guaranteed in practice. Thus it is unclear whether these pre-trained models indeed exhibit such divergence and convergence of tokens that can match the conclusion of this paper. In a word, it is not clear whether these conditions and the corresponding divergence/convergence of tokens only exist in the special setting of the paper or are more general across a wide range of settings.", "was_revised": true, "final_justification": "The authors have addressed my concerns regarding the conditions of the initialized model paramters, while the setting regarding time-dependent parameters cannot be solved in the current manuscript. As this is a follow-up work, I maintain my score of 4.", "rebuttal": "We thank the Reviewer for the constructive feedback. Below we address your concerns.\n\n-----\n\n**W1. Please correct me if I understand the setting in a wrong way: the parameters are indeed trained and fixed, however, the time variable $t$ refers to the depth of the transformers. Hence the fact that $W$ and $A$ do not depend on $t$ means that the merged weight $W$ of attention and the value matrix $V$ are also the same for all layers. This does not align with the actual case. As the cluster of tokens has already been studied by prior works, I think it is better to take a step further to remove the limitation by considering a case where $W$ can depend on $t$.**\n\n**A1.** Yes, you are correct. In our theorems, we assume that $W$ and $A$ are independent of $t$. We empirically verify our theoretical findings in real-world pretrained transformers in Section 3.4.\n\nWhile your concern regarding the limited scope of practical scenarios under these assumptions is valid, it is common for theorists to adopt simplified settings of the model to enable feasible theoretical analysis, while still retaining sufficient capacity to capture key patterns observed in practical scenarios. Several high-standard related works by notable researchers also operate within such simplified settings, for example:\n\n[A] Geshkovski et al. The Emergence of Clusters in Self-Attention Dynamics. NeurIPS 2024. (The assumption $Q=K=V=Id$ was used.)\n\n[B] Geshkovski et al. A Mathematical Perspective on Transformers. Bulletin of the AMS 2024. (The assumption $Q=K=V=Id$ was used.)\n\n[C] Vo et al. Demystifying the Token Dynamics of Deep Selective State Space Models. ICLR 2025 (spotlight). (The assumptions dim $=1$ and time-invariant parameters were used.)\n\nOur results generalize certain findings in [A] by employing more general assumptions on $Q$, $K$, and $V$, and even analyze the effect of positional encodings. We believe that this generalization is already nontrivial.\n\nThe case where $W$ and $A$ depend on $t$ remains highly challenging, and to the best of our knowledge, no existing work has achieved this result so far. We leave this interesting problem for future work.\n \n\n \n**W2-Q2. One aspect that has not been touched by this paper is the statistics of the conditions for $W$ and $A$ of real-world pre-trained models (not manually designed by some re-parameterization as in Appendix D.2). Specifically, the condition that the value matrix $V$ is invertible cannot necessarily be guaranteed in practice. Thus it is unclear whether these pre-trained models indeed exhibit such divergence and convergence of tokens that can match the conclusion of this paper. In a word, it is not clear whether these conditions and the corresponding divergence/convergence of tokens only exist in the special setting of the paper or are more general across a wide range of settings (W2). Could the conclusions/prediction be applied to real-world pre-trained models? And if so, what will the comparison between a randomly initialized model and the pre-trained model be regarding the conditions given by this paper? (Q2)**\n\n**A2.** We performed an empirical analysis of real-world pretrained Transformer models to verify the conditions in our theory. Specifically, for each model, we measured the mean percentage across all layers of (i) “near-zero” eigenvalues of the value matrix $V$ (using threshold $\\epsilon = 10^{-3}$), and (ii) positive eigenvalues of the symmetrized matrices $W_{\\mathrm{sym}}$ and $A_{\\mathrm{sym}}$.\n\nTable 1 reports the mean ± standard deviation of these percentages for GPT-2 XL, DistilGPT2, and LLaMA-2 13B. We observe that: (i) In all cases, $V$ exhibited 0% near-zero eigenvalues, indicating that $V$ is invertible in practice. (ii) The proportion of positive eigenvalues in both $W_{\\mathrm{sym}}$ and $A_{\\mathrm{sym}}$ is approximately 50%, which matches the divergence regime predicted by Remark 3.5 if omitting FFNs and LayerNorms. The divergence behaviour of an unconstrained/real-world pretrained model is also illustrated as 'baseline' case of Figure 6 in Section 5.1.\n\nThese findings align with theoretical expectations: (i) the set of singular matrices has measure zero in continuous parameter spaces, and (ii) unconstrained weight matrices, whether randomly initialized or learned, tend to have eigenvalue distributions symmetric about zero. We also repeated this analysis on a randomly initialized model (GPT-2 XL reinitialized) and obtained similar results (approximately 50% positive eigenvalues in $W_{\\mathrm{sym}}$, $A_{\\mathrm{sym}}$, and no near-zero eigenvalues in $V$).\n\n\n*Table 1: Mean ± std. dev. of eigenvalue statistics (in %) across layers.*\n|Model|% positive eigenvalue of $W_{sym}$|% positive eigenvalue of $A_{sym}$|% near-zero-eig of $V$|\n|-|-|-|-|\n|GPT2-xl|$50.20\\pm3.76$|$49.99\\pm0.06$|$0.00\\pm0.00$|\n|DistilGPT2|$46.01\\pm4.36$|$49.96\\pm0.16$|$0.00\\pm0.00$|\n|Llama2 13B|$52.35\\pm2.84$|$50.00\\pm0.02$|$0.00\\pm0.00$|\n|GPT2-xl random reinitialized|$50.20\\pm2.85$|$49.99\\pm0.06$|$0.00\\pm0.00$|\n\n**Q1. Could the authors explain what the conclusions of the current manuscript will be if all the weights depend on the variable $t$?**\n\n**A3.** Unfortunately, the theorems may no longer hold if the weights depend on the variable $t$. However, our simulations using randomly selected time-dependent weights suggest that the theorems may still hold under analogous assumptions. For example: Theorem 3.1 appears to remain valid if $A$ is definite and maintains the same sign for all $t > 0$; Theorem 3.4 still holds if $A < 0$ and $W > 0$ for all $t > 0$; and Theorem 3.5 continues to hold if $V$ possesses a fixed positive eigenvalue for all $t > 0$. We will include this discussion in the revised version of the paper.\n\n-----\n\nWe will incorporate the discussion in the revised version. If our responses adequately address the concerns, we kindly hope the evaluation may be reconsidered accordingly. We remain open to further discussion in the next stage of discussion."}, {"review_id": "h7rXkBfNIF", "reviewer": "Reviewer_1jLS", "summary": "This is a very good paper. It extends the existing line of work on studying signal propagation in Transformer models, where the parameters are fixed (pre-trained model) and the evolution of tokens across layers is analyzed. The authors take a « neural ODE » point of view in which time plays the role of a layer, and this perspective is well-accepted in the community as being particularly helpful in doing theory. This line of research is clearly fundamental and critical for improving the interpretability and understanding of how large language models work.", "questions": "1. I have a question regarding the addition of positional encoding in the equation. I thought positional encoding is added to the initial condition, and that’s it. Why does it appear in the ODE in equation (5)? I am not totally sure I understand this point.\n\n2. The authors are also very welcome to comment on the possible impact that an MLP layer (or a normalisation layer) may have on the divergence, or the avoidance thereof. \n\n3. Moreover, there is a paper by Koubbi, Boussard and Hernandez, where they also generalise the results of Geshkovski et al. to somewhat more general parameters, but it’s not cited herein. Perhaps the authors can discuss the differences with that paper.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "In particular, the authors address the effect of positional encoding on the dynamics of tokens within the Transformer architecture, which has not been seriously considered before. The novelty is clear. There are also clear extensions on the existing results of Transformer dynamics to more general classes of parameter matrices (key-query and value matrices, to be more precise).", "was_revised": true, "final_justification": "I am satisfied by the answers of the authors, and my issues have been resolved.", "rebuttal": "We thank the Reviewer for the constructive feedback. Below we address your concerns.\n\n-----\n\n\n**Q1. I have a question regarding the addition of positional encoding in the equation. I thought positional encoding is added to the initial condition, and that’s it. Why does it appear in the ODE in equation (5)? I am not totally sure I understand this point.**\n\n**A1.** Indeed, adding the positions $p_i$ to the initial conditions is equivalent to fixing the initial condition and adding $p_i$ to $x_i$, as in Equation (5).\n\n**Q2. The authors are also very welcome to comment on the possible impact that an MLP layer (or a normalisation layer) may have on the divergence, or the avoidance thereof.**\n\n**A2.** Unfortunately, not all theoretical results continue to hold when an MLP or LayerNorm is introduced. However, under the divergence scenario where MLP/LayerNorm are present, Figure 4 of Section 3.4 and Figure 7 of Appendix D.5.1 in our manuscript show that tokens still form distinct clusters, with each group following a similar trajectory pattern. This suggests that our observation on directional token clustering persists even in a fully structured Transformer model with MLP/LayerNorm. A theoretical analysis for these cases requires further adaptation, which we leave for future research.\n\n**Q3. Moreover, there is a paper by Koubbi, Boussard and Hernandez, where they also generalise the results of Geshkovski et al. to somewhat more general parameters, but it’s not cited herein. Perhaps the authors can discuss the differences with that paper.**\n\n**A3.**  Thank you for pointing out the interesting work in [A]. As far as we are aware, the authors of [A] assume that the triple $(Q, K, V)$ already leads to a clustering phenomenon (at infinity) of tokens and investigate how such clustering is affected by small perturbations of the triple through the addition of low-rank components. In contrast to this approach, we directly characterize the dynamical properties of tokens for the triple $(Q, K, V)$ based on an analysis of their eigenvalues. Furthermore, we analyze the effect of positional encoding on the dynamics. Beyond that, we also propose model modifications to improve performance across different tasks. We will add reference [A] and include this discussion in the revised version of the paper.\n\n\n[A] Koubbi, Boussard and Hernandez et al., The Impact of LoRA on the Emergence of Clusters in Transformers, arxiv.\n\n-----\n\nWe will incorporate the discussion in the revised version. We remain open to further discussion in the next stage of discussion."}, {"review_id": "M4SxsgLCjY", "reviewer": "Reviewer_p6VJ", "summary": "This manuscript takes a dynamical-systems viewpoint—modeling layers of an attention-only deep neural network as continuous—with the motivation of exploring the evolution of token embeddings through the layers of a transformer during pretraining.  The authors consider a regime where weight matrices are tied (i.e., identical) across layers.  In this regime, they find that activations from different inputs either converge or diverge through the layers, depending on whether the symmetric components of certain matrices ($W = Q K^\\top$ and $A =  Q K^\\top (V^{-1})^\\top$) are positive definite (i.e., have only positive eigenvalues) or negative definite (i.e., have only negative eigenvalues).  Effects of positional embeddings are considered.  The theoretical predictions are borne out by transformers that are pretrained with weight matrices tied across layers, even when the transformers include FFN layers in addition to the attention layers.  In the end, the paper suggests ways to promote divergent embeddings.  Appendices provide derivations of all claims.", "questions": "The authors assert that their first contribution is to “provide conditions that predict whether tokens in self-attention converge to zero or diverge to infinity”.  By standard use of these terms, this claim is inscrutable.  I am guessing the authors mean that the magnitude of token embeddings in a transformer are converging or diverging?  Can the authors provide citation(s) of this issue in real transformers?\n\nRegarding the framework laid out in Sec. 2.2, the authors may want to consider causal attention masking in future work, since this would be most relevant for autoregressive transformers.\n\nIt is not clear how the eigenvalues of $A$ correspond to the eigenvalues of its symmetrization $A_\\text{sym}$.  In the general case, I’m not aware of any easy correspondence.  Accordingly it doesn’t seem to be the case that, e.g., the positive-definiteness of one implies the positive-definiteness of the other.  In the paper, sometimes positive-definiteness is talked about in terms of $A$ and sometimes in terms of $A_\\text{sym}$, and sometimes in terms of $A$ after assuming it is symmetric such that $A = A_\\text{sym}$.  How should the reader think about these relationships?  In particular, the claims starting on Line 128 involving the positive definiteness of $A$ don’t appear to assume that $A$ is symmetric, but then Thm. 1 assumes that $A$ is a symmetric matrix and Remark 3.2 is about the positive definiteness of the symmetrization of $A$.\n\n\nMinor suggestions:\n\nLine 41 and Section A:\n“Only a limited number of studies in the literature provide a theoretical understanding of the internal structure of learned representations in pre-trained Transformer models (see Section A for a comprehensive list of relevant works).” \nThis field moves fast, so it seems overly confident to suggest a “comprehensive” list of relevant works that provide understanding of the internal structure of learned representations.  I can certainly think of many more.  Perhaps remove the word “comprehensive”.\n\nLine 48:\nIt seems that “fourth-fold” should be changed to “fourfold”.\n\nLine 118:\nConsider explicitly stating the intended meaning/definition of “equivariant” in the statement that “it is equivariant to the standard Euclidean norm”\n\nLine 157:\nTypo:  Change “randomly chosen of model parameters” to “randomly chosen model parameters”", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "This paper does a good job of stating the equations and assumptions for their modeling framework.  Once this framework is set up, the ensuing math and dynamical-systems analysis all seems valid.  (Good Clarity and fine Quality for the topic of study.)\n\nThe primary weakness of this paper is that the modeling assumptions seem to create a serious disconnect from real transformers, such that the predictions seem irrelevant.  The form of attention equations looks similar, but the continuous-time limit has no obvious motivation other than it allows for a dynamical system that can be analyzed.  Tying the attention weight matrices across layers of a transformer is a strong restriction, which mirrors the ALBERT architecture but does not reflect standard architectures of interest.  As a further significant restriction, this paper does not incorporate any analog of the FFN layers of transformers in the theoretical analysis.  I recognize that other papers like Ref. [17] have been published before using a similar modeling framework, but I don’t believe this precedent provides sufficient justification for the importance of the current analysis for understanding real neural nets which would be an expected standard for a conference like NeurIPS.  Rather, the existence of several other papers that make very similar assumptions reduces the novelty of this approach and makes the current work feel quite iterative.  (Low Originality and Significance.)  I believe this paper would be better suited for a dynamical systems audience looking for neat dynamics, rather than an audience trying to understand transformers.  \n\nThe authors say that “Our results are also validated against pre-trained Transformer models, confirming that\nthe theoretical insights carry over to real-world settings.”  The authors indeed show some relevance of their theoretical predictions for the transformers they train, including those with FFN layers in addition to attention layers, but my understanding is that these transformers have their weight matrices tied across layers, which is not much of a real-world setting (besides the ALBERT architecture).\n\nSome standard ML terms seem to be used unusually in this paper. In particular, the word “token” is used throughout the paper to rather mean something like “token embedding” or “residual stream”.  The stated study of how tokens evolve is really about how the activations in the residual stream evolve through the layers.  For example, “when tokens move closer” would be more precise if it was changed to say “when embeddings move closer in deeper layers of the network”.  It would be much more conventional for “tokens” to refer to the symbols from a fixed discrete set, rather than real values.", "was_revised": true, "final_justification": "I have slightly upped my recommendation since the authors have been able to show relevance of their work when the (otherwise quite restrictive) weight-tying assumption is at least slightly relaxed.", "rebuttal": "We thank the Reviewer for the constructive feedback. Below we address your concerns.\n\n-----\n\n**W1. Concerns include the reliance on a continuous‐time limit and simplifying assumptions, specifically tying attention weights across layers and omitting feed‐forward and layer‐normalization modules, which undermine both the work’s relevance to real Transformer architectures and its originality.** \n\n**A1.** *Regarding the use of continuous-time limit*: We respectfully disagree with the reviewer about the comment \"continuous-time limit has no obvious motivation other than it allows for a dynamical system that can be analyzed\". Modern deep networks often have a large number of layers, making them well-approximated by continuous-time models. For example, ResNets can be seen as discretizations of ODEs, and this perspective becomes increasingly accurate as depth increases. Moreover, the continuous-time limit is a well-established technique in both the machine learning and mathematics communities. It has led to important insights into training dynamics, stability, and generalization, and underpins models like Neural ODEs, State Space Models, and Liquid Neural Networks. Thus, it is both theoretically justified and practically relevant.\n\n*Regarding your concern about our simplifying assumptions, namely, tying the attention weight matrices across layers and excluding the FFN/LayerNorm*: While your concern is valid, it is common for theorists to adopt simplified settings of the model to enable feasible theoretical analysis, while still retaining sufficient capacity to capture key patterns observed in practical scenarios. Several high-standard related works by notable researchers also operate within such simplified settings, for example:\n\n[A] Geshkovski et al. The emergence of clusters in self-attention dynamics. NeurIPS 2024. (The assumption $Q=K=V=Id$ was used.)\n\n[B] Geshkovski et al. A mathematical perspective on Transformers. Bulletin of the AMS 2024. (The assumption $Q=K=V=Id$ was used.)\n\n[C] Vo et al. Demystifying the Token Dynamics of Deep Selective State Space Models. ICLR 2025 (spotlight). (The assumption dim $=1$ and time-invariant parameters were used.)\n\nOur results generalize certain findings in [A] by employing more general assumptions on $Q, K, V$, and by additionally analyzing the effect of positional encodings. We believe that this generalization is already nontrivial.\n\nThe general case involving a full and practical Transformer model remains highly challenging, and to the best of our knowledge, no existing work has achieved this result to date.\n\n\n\n\n\n**W2. The authors say that “Our results are also validated against pre-trained Transformer models, confirming that the theoretical insights carry over to real-world settings.” The authors indeed show some relevance of their theoretical predictions for the transformers they train, including those with FFN layers in addition to attention layers, but my understanding is that these transformers have their weight matrices tied across layers, which is not much of a real-world setting (besides the ALBERT architecture).**\n\n\n**A2.**  To clarify, we did verify our theoretical results using different settings of practical Transformer architectures in Sections 3.4 and 5.1, none of which restrict weights from being shared across layers. In particular:\n(i) Figure 3 demonstrates that our theoretical findings hold in the absence of the weight-tying assumption,\n(ii) Figure 4 illustrates the clustering behavior of models with FFNs and LNs, but constrained to the divergence scenario, and\n(iii) Figures 6 and 7 demonstrate the behavior of the models when lifting the constraints on  $W_{sym}$ or $A_{sym}$, generalizing to the practical pre-trained models.\n\n\n**W3-Q1. Some standard ML terms seem to be used unusually in this paper. In particular, the word “token” is used throughout the paper to rather mean something like “token embedding” or “residual stream”. The stated study of how tokens evolve is really about how the activations in the residual stream evolve through the layers. For example, “when tokens move closer” would be more precise if it was changed to say “when embeddings move closer in deeper layers of the network”. It would be much more conventional for “tokens” to refer to the symbols from a fixed discrete set, rather than real values (W3). The authors assert that their first contribution is to “provide conditions that predict whether tokens in self-attention converge to zero or diverge to infinity”. By standard use of these terms, this claim is inscrutable. I am guessing the authors mean that the magnitude of token embeddings in a transformer are converging or diverging? Can the authors provide citation(s) of this issue in real transformers? (Q1)**\n\n\n**A3.**  The way we use the term \"token\" in this paper is consistent with its usage in many previous works in this line of research. For example, it appears in high-standard publications [A, B, C, D]. In particular, the authors of [A] characterized the convergence/divergence properties of tokens when $Q=K=V=Id$. Our results generalize the findings in [A] by employing more general assumptions on $Q,K,V$, and by also analyzing the effect of positional encodings.\n\nFollowing your suggestion, we will add a remark to explain alternative terms for \"token\" used in other areas, such as \"token embedding\" or \"residual stream.\"\n\n\n\n\n**Q2: Regarding the framework laid out in Sec. 2.2, the authors may want to consider causal attention masking in future work, since this would be most relevant for autoregressive transformers.**\n\n**A4.**  The dynamical properties of tokens under causal attention masking represent an interesting research direction. An initial theoretical result in this area was presented in [D]. Generalizing this work would certainly be a valuable avenue for future research. We will add this to the Future Work section of our paper.\n\n[D] Karagodin et al. Clustering in Causal Attention Masking. NeurIPS 2024.\n\n**Q3. It is not clear how the eigenvalues of $A$ correspond to the eigenvalues of its symmetrization $A_{sym}$. In the general case, I’m not aware of any easy correspondence. Accordingly it doesn’t seem to be the case that, e.g., the positive-definiteness of one implies the positive-definiteness of the other. In the paper, sometimes positive-definiteness is talked about in terms of $A$ and sometimes in terms of $A_{sym}$, and sometimes in terms of $A$ after assuming it is symmetric such that $A=A_{sym}$. How should the reader think about these relationships? In particular, the claims starting on Line 128 involving the positive definiteness of $A$ don’t appear to assume that $A$ is symmetric, but then Thm. 1 assumes that $A$ is a symmetric matrix and Remark 3.2 is about the positive definiteness of the symmetrization of $A$.**\n\n\n**A5.** We did not claim any correspondence between the eigenvalues of $A$ and those of $A_{sym}$. Instead, we used the following two distinct assumptions:\n\n(a) $A$ is positive definite;\n\n(b) $A_{sym}$ is positive definite.\n\nWhen we use (a), this means that $A = A_{sym}$ and is positive definite. However, when we use (b), $A$ is not necessarily symmetric, but its symmetric part $A_{sym}$ must be positive definite. To clarify, (a) implies (b), but (b) does not imply (a).\n\n\n\n**MS1. Line 41 and Section A: “Only a limited number of studies in the literature provide a theoretical understanding of the internal structure of learned representations in pre-trained Transformer models (see Section A for a comprehensive list of relevant works).” This field moves fast, so it seems overly confident to suggest a “comprehensive” list of relevant works that provide understanding of the internal structure of learned representations. I can certainly think of many more. Perhaps remove the word “comprehensive”.**\n\n**A6.** We agree to remove the word \"comprehensive\" in the revised version. In addition, we will add further relevant works to the related works section.\n\n**MS2. Line 48: It seems that “fourth-fold” should be changed to “fourfold”.**\n\n**A7.** We will change “fourth-fold” to “fourfold” in the revised version.\n\n\n**MS3. Line 118: Consider explicitly stating the intended meaning/definition of “equivariant” in the statement that “it is equivariant to the standard Euclidean norm”**\n\n**A8.** The correct statement is: “it is equivalent to the standard Euclidean norm.” Two norms on $\\mathbb{R}^D$ are equivalent if and only if they induce the same topology. It is well known that all norms on $\\mathbb{R}^D$ are equivalent. We will update this in the revised version.\n\n\n**MS4. Line 157: Typo: Change “randomly chosen of model parameters” to “randomly chosen model parameters”**\n\n**A9.** We will update this in the revised version.\n\n-----\n\nWe will incorporate the discussion in the revised version. If our responses adequately address the concerns, we kindly hope the evaluation may be reconsidered accordingly. We remain open to further discussion in the next stage of discussion."}, {"review_id": "ku7l3OMr5R", "reviewer": "Reviewer_m1Uq", "summary": "The paper investigates the dynamical behavior of tokens in Transformer models using a Neural-ODE formulation of attention. It examines whether, under certain conditions, tokens tend to converge or diverge as they progress through layers, depending on model parameters. The authors also analyze the role of different positional encodings, specifically absolute and rotary, in shaping these dynamics, and try to support their claims through empirical evaluation.", "questions": "**Questions:**\n\n1. I have several questions and suggestions regarding the experimental validation of the paper’s claims:  \n\n    a) Is it possible to empirically validate whether tokens in a pretrained model converge or diverge as they propagate through layers? For instance, could this be tested by analyzing distances between token embeddings across layers in a realistic model? Appendix D.5.1 touches on this, but I was hoping for a more thorough empirical investigation using standard pretrained architectures.  \n\n    b) Can one analyze the structure of the attention matrices \\$QK^\\top\\$ in real models to determine whether they are PSD, NSD, or neither? In decoder-only models using causal self-attention, these matrices are typically not symmetric, even when assuming \\$Q = K\\$, so they are unlikely to satisfy PSD or NSD conditions. How would this affect the applicability of the theoretical analysis?  \n\n    c) Is it possible to evaluate the core Neural ODE assumption that the model can be approximated by an Euler discretization due to small weight changes between layers? In practice, the \\$Q\\$, \\$K\\$, and \\$V\\$ matrices vary significantly across layers in standard Transformers, which may invalidate this assumption.  \n\n    d) Could the experiments in Section 3.4 be rerun without explicitly imposing the constraints used in the theoretical setup? This would help evaluate whether the convergence or divergence behavior still appears under more realistic conditions.  \n\n    e) Is it possible to study how convergence or divergence behavior is affected by other architectural components, such as MLP blocks or layer normalization?  \n\n    f) Finally, could the analysis be extended to investigate the effect of varying the input sequence length, at least empirically?  \n\n2. For what set of models is this analysis even useful? For causal self attention I think a lot of the analysis would probably not hold. Could you please clarify what would change ? \n\n3. In the caption of Figure-4 what does it mean by similar shapes?\n\n4. I think I probably missed this in the paper, but how do you obtain figure 4 and figure 5? Is it via the PCA of the tokens or is there something else ?\n\n5. For Table - 1 and Table -2 is it possible to add any standard deviation \n\n6. In the solution of Equation (11) (see lines 575–578), could the authors clarify the initial conditions used for the token representations $x_\\ell(0)$? This question arises because if the quantity $e^{c||x_\\ell(0)||^2} - L$ is negative, then the expression may decrease as $t$ increases and eventually become undefined. In such a case, the conclusion that $\\lim{t \\to +\\infty} ||x_\\ell(t)||= +\\infty$ would no longer hold. This also suggests that the behavior of the system would depend on $t$, meaning that the model's depth (or number of layers) must lie within a bounded domain to ensure validity. If this interpretation is correct, some of the paper's conclusions may need to be reconsidered under such conditions. Could the authors elaborate on what initial values are assumed in these equations, and whether such conditions are expected to occur in practice? Furthermore, for very long input sequences, it appears that $L$ increases with sequence length. In this case, under what conditions on $A$ and $x\\ell(0)$ does the term $e^{c||x_\\ell(0)||_A^2} - L$ remain positive to ensure the validity of the derivation? That is, how must the matrix $A$ and the initialization be chosen to guarantee that the analysis holds for long sequences?\n\n7. I would like to request clarification regarding the setup used to produce Figure 6. Specifically, is the figure generated by taking a pretrained model and subsequently removing the layer normalization and MLP components, or is it based on a model trained from scratch without using layer norms and MLPs? Additionally, can experiments be conducted where the constraints described in lines 263–272 are removed, in order to observe what happens when these artificial constraints are relaxed?\n\n8. Could the authors elaborate on the intuition behind why divergence leads to improved performance, as suggested in lines 275–279? Is the idea that token representations become sufficiently separated in space, making them easier to distinguish and thus improving model expressivity? This seems somewhat counterintuitive from a language modeling perspective, where one might expect that semantically similar tokens should cluster together, while only dissimilar tokens diverge. Otherwise, it could hinder the model’s ability to learn meaningful attention maps, especially if all tokens diverge indiscriminately. Could the authors provide a more detailed intuition for why divergence is beneficial in this context, and how this compares to what is observed in standard pretrained models?\n\n9. What is the practical benefit of Theorems and Propositions 3.1–3.6? All of these results appear to focus on asymptotic behavior with respect to model depth. However, in practice, the depth is always finite for example, DeepSeek-V3 [A] has approximately 61 layers. Given this, how should we interpret the relevance of these asymptotic results? Do they meaningfully inform us about the behavior of realistic models? More specifically, is there a way to understand how the described properties evolve layer by layer, rather than only in the limit as depth approaches infinity?\n\n**Minor Typos:**\n\n1. Line 179: “We observe from randomly ~~chosen of~~ model parameters”, \"chosen\" -> \"initialized\"? \n\n\n**Note:** I find the analysis in this work to be interesting. However, my main concern is regarding the applicability of the results. Much of the analysis appears to depend on very simplified settings, and the assumptions made are quite strong and seem to diverge significantly from practical scenarios. Based on these concerns, I am currently assigning a lower score. That said, I am open to revisiting my evaluation during the rebuttal phase, depending on the authors' clarifications and the views expressed by other reviewers.\n\n---\n\n**References:**\n\n[A] Liu, Aixin, et al. \"Deepseek-v3 technical report.\" arXiv preprint arXiv:2412.19437 (2024).", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper explores an interesting and relevant topic: how token representations evolve as they propagate through the layers of a Transformer, with a focus on whether tokens converge or remain separated.\n\n2. The authors conduct an extensive set of experiments to provide empirical support for their theoretical analysis.\n\n**Weaknesses:**\n\n1. I think the related works section could be improved. It would be helpful if the authors contextualized their contributions more clearly in relation to previous work. Specifically, in lines 37 to 40, the authors state that prior works rely on “unrealistic and impractical assumptions on model parameters... to support the development of richer theoretical results...” and “have not yet offered practical applications or insights...” However, it is not clear what specifically makes those assumptions unrealistic or impractical, nor what assumptions this paper adopts instead or why they are more realistic. Clarifying this distinction would help establish the novelty and practical relevance of the contribution. At present, the difference between this work and earlier studies remains vague, and I could not find any further discussion or elaboration in the appendix.\n\n2. I believe some of the mathematical derivations may contain inaccuracies - please see the questions. \n\n3. I think some of the experiments are constructed in a way that closely reflects the conditions assumed in the theoretical analysis, which may make the results somewhat self-reinforcing. However, it still remains to be seen whether such behaviors occur in real-world settings. Please see questions for a follow-up. \n\n4. The results lack standard deviations.\n\n5. I find the setting to be overly simplistic. While this is not necessarily a flaw, it raises concerns about the practical relevance of the analysis. The Neural ODE model is a coarse approximation of how information propagates through multiple layers of a Transformer. For the analysis, the authors assume that the attention matrix $QK^T$ can be replaced by a fixed matrix $W$, where $Q$ and $K$ are the query and key matrices used in the self-attention mechanism. They further assume for their analysis that $W$ is either positive semidefinite (PSD) or negative semidefinite (NSD). However, in causal attention, $QK^T$ is not symmetric, so $W$ is unlikely to satisfy these conditions. These assumptions simplify the analysis but may significantly limit its applicability to real-world models.", "was_revised": true, "final_justification": "I thank the authors for their comments and efforts throughout the rebuttal process. I was on the fence about whether to increase the score for the paper. After weighing the pros and cons, I have decided to raise the score from **2** to **4**.\n\n**Pros:**  \n1. The authors made substantial efforts during the rebuttal, including running additional experiments and clarifying important points.  \n2. One of the experiments (the GPT-2 experiments) appears to be consistent with the theoretical findings.\n\n**Cons:**  \n1. During the discussion phase, we identified a flaw in one of the proofs. While this is not unusual (the rebuttal process often serves to uncover such issues), it will require rewriting portions of the paper. This makes it challenging to judge the quality of the future, revised version.  \n2. The setting still feels overly simplistic, and many assumptions remain invalidated. For example, is NeuralODE truly the appropriate framework for analyzing such systems? Although prior work has made similar assumptions, to my knowledge these have not been explicitly validated, but rather used as an analytical convenience.  \n3. Much of the analysis is limited to only two settings, PSD and NSD, which are quite restrictive.  \n4. It is unclear whether the authors attempted an experiment retraining GPT-2 without layer norms and MLP, and without constraints. While I realize it is too late to request this during the rebuttal, I suspect such a setting might not exhibit the same divergence (or convergence) behavior.\n\n**Overall Assessment:**  \n\nIn my view, making progress in theoretical ML is extremely challenging, and this work contains elements that could help us better understand the behavior of large language models. Moreover, the authors engaged constructively and thoroughly during the rebuttal, which I greatly appreciate.  Hence, I recommend acceptance.\n\nI recommend that the authors:  \n1. Revise the paper to address the incorrect proposition identified during the discussion.  \n2. Consider conducting the experiment described in point 4 above for the revised version.", "rebuttal": "We thank the Reviewer for the constructive feedback. Below we address your concerns.\n\n---\n\n**W1. Difference between this work and earlier studies remains vague.**\n\n**A1.** Let us take this opportunity to clarify the distinction between our results and those in relevant previous works, in order to highlight the novelty and practical relevance of our contributions. Our results directly generalize certain findings from [A] regarding the dynamical properties of tokens in self-attention. In particular, the authors of [A] characterized the convergence, divergence, and clustering behaviors of tokens under the assumption $Q = K = Id$ and $V=Id,-Id$, or parametric. We theoretically generalize these findings in [A] to the case of arbitrary matrices $Q$, $K$, and a matrix $V$ with at least one positive eigenvalue (see Section 3 and Appendix B). We further extend the analysis by investigating the effect of positional encodings in addition to self-attention (see Section 4 and Appendix C). These theoretical results are then empirically verified in practical Transformer settings (see Section 3.4). Moreover, unlike previous works in this line of research, we go beyond theoretical analysis and verification by demonstrating model performance improvements on several tasks (see Section 5 and Appendix D).\n\nWe hope that this response clarifies the novelty and originality of our contribution.\n\n**W2-Q6-Q1f. Some of the mathematical derivations may contain inaccuracies regarding the constraint on initial condition. Could the empirical analysis be extended to examine the impact of this constraint?**\n\n\n**A2.** Thank you for pointing out this error. Indeed, the inequality $e^{c\\Vert x_\\ell(0)\\Vert_A^2}-L>0$ is needed to ensure that $\\lim_{t \\to+\\infty}\\Vert x_\\ell(t)\\Vert_A=+\\infty$ in the proof of Proposition B3(b). This inequality implies that the initialization and the matrix $A$ must be chosen such that\n$$\\Vert x_\\ell(0)\\Vert_A>\\sqrt{\\frac{\\log(L)}{\\max\\limits_{u\\in\\mathbb{R}^D,\\Vert u\\Vert_A=1}\\frac{\\Vert u\\Vert_W}{\\Vert u\\Vert_A}}}=O\\left(\\sqrt{\\log(L)}\\right).$$\n\nIntuitively speaking, for Proposition B3(b) to hold, the $A$-norm of the initial point must be at least on the order of the square root of the logarithm of the sequence length $L$.\n\nFortunately, this proposition does not affect any subsequent lemmas or theorems, as Proposition B3(b) is not used later. (Only Proposition B3(a) is referenced.) Our main results (including Theorems 3.1, 3.4 and 3.6) still hold for arbitrary sequence length $L>0$.\n\n**W3. Some of the experiments are constructed in a way that closely reflects the conditions assumed in the theoretical analysis, which may make the results somewhat self-reinforcing**\n\n**A3.** While we include experiments that impose the theoretical constraints described in lines 263–272, none of our empirical validations rely on the NeuralODE assumption of shared weights across layers. Instead, each experiment uses distinct weight matrices per layer to ensure our results hold even when layer parameters vary. Specifically:\n\n(i) Figure 3 shows that our theoretical findings hold without the NeuralODE assumption,\n\n(ii) Figure 4 illustrates clustering behavior in models with FFNs and LNs under the divergence scenario, and\n\n(iii) Figures 6 and 7 show model behavior when relaxing the constraints on $W_{sym}$ or $A_{sym}$, generalizing to the practical pretrained models.\n\n**Q7-Q1ad. Clarification regarding the setup used to produce Figure 6 and extension of experiments in Figure 6 and Section 3.4 to unconstrained settings of pretrained models**\n\n**A4.** To clarify, Figure 6 was generated by removing all LayerNorm and MLP (FFN) modules from a full-architecture pretrained Transformer to isolate pure attention dynamics. We measured token norms and mean pairwise L2 distances across layers under both constrained (divergence, intermediate, convergence) and unconstrained (baseline, similar to real-world Transformers) scenarios.\n\nWe visualized relaxed (intermediate/baseline) versus strict (divergence/convergence) constraint regimes. Under intermediate and unconstrained settings, token norms and pairwise L2 distances still diverge, verifying Remarks 3.5, albeit more slowly than in the pure divergence regime. Moreover, Appendix D.5.1’s Figure 7 further shows that, without any constraints on $W_{sym}, A_{sym}$, pretrained GPT-2 token trajectories still exhibit clustering.\n\n**Q1c. Is it possible to evaluate the core Neural ODE assumption?**\n\n**A5.** While the Neural ODE assumption may be violated in real-world settings, our experiments in Section 3.4, our analysis in Section 5.1, and our visualizations in Appendix D.5.1 were conducted without this assumption, thus empirically confirming our theoretical findings in more realistic settings.\n\n**W5-Q1b-Q2. Concerns about simplified setting and practical relevance to causal self-attention. Can one analyze the structure of the matrices $W$ in real models, especially for causal self-attention?**\n\n**A6.** *Regarding the causal attention:* We aim to work with the normal attention, not the causal attention at the moment. The dynamical properties of tokens under causal attention masking represent a different interesting research direction. An initial theoretical result in this area has been presented in [D]. Generalizing this work would certainly be a valuable avenue for future research.\n\n*Regarding the simplified assumptions on the theorem:* Please refer to our answer **A1** to Reviewer p6VJ.\n\n*Regarding the structure of the attention matrices $W = QK^\\top$ in real models:* In pretrained practical Transformers, $W$ is not necessarily to be positive or negative definite. However, our analysis is not just restricted to definite matrices. For example, Theorems 3.1 and 3.6 hold for arbitrary $W$, and there are no restrictions on $Q$, $K$, or $V$ in Remark 3.5.\n\n**Q1e. Impact of other architectural components on converge/diverge property**\n\n**A7.** Please refer to our answer **A2** to Reviewer 1jLS.\n\n**W4-Q5. The results lack standard deviations.**\n\n**A8.** We did provide full results (with standard deviation) of Table 1 and 3 in Appendix D.6 (Table 4 and 5, respectively). For Table 2, we provide the full results averaged after 3 runs below:\n\n||Top-1|Top-5|\n|-|-|-|\n|DeiT baseline|$71.79\\pm0.02$|$90.99\\pm0.02$|\n|+ convergence|$71.37\\pm0.02$|$90.65\\pm0.01$|\n|+ intermediate|$71.62\\pm0.03$|$90.94\\pm0.02$|\n|+ divergence|$71.93\\pm0.02$|$91.03\\pm0.03$|\n\n**Q3. In the caption of Figure-4 what does it mean by similar shapes?**\n\n**A9.** To clarify, Fig. 4 shows tokens clustered into groups, and tokens within each group propagate through layers in a similar direction.\n\n**Q4. How do you obtain figure 4 and figure 5?**\n\n**A10.** Figure 4 was plotted by projecting onto a manually selected 2D plane (without PCA) to highlight clustering properties. Figure 5 uses tokens that are inherently 2D in our setup, so no dimensionality reduction was needed.\n\n**Q8. Intuition for why divergence leads to improved performance**\n\n**A11.** We claimed in the paper that convergence may negatively affect model performance. Therefore, encouraging the model to avoid the convergence scenario can lead to better practical results. One simple way to avoid convergence is to push the model toward the divergence scenario. The reason convergence may hinder model improvement is that, in this case, token representations tend to shrink toward zero as time $t$ increases. This leads to information loss during training updates, which is not preferable.\n\n**Q9. Practical benefit of Theorems and Propositions 3.1–3.6**\n\n**A12.** The practical benefit of Theorems and Propositions 3.1–3.6 relies on the well-known fact that the continuous-time limit can be used to efficiently approximate modern deep networks. These networks often have a large number of layers, making them well-approximated by continuous-time models. For example, DeepSeek-v3 has approximately 61 layers and can thus be viewed as a discretization of an ODE system. Moreover, the continuous-time limit is a well-established technique in both the machine learning and mathematics communities. It has provided meaningful insights into important behaviors of realistic models, including training dynamics, stability, and generalization.\n\nAn alternative way to understand how the described properties evolve layer by layer is through the use of difference equations (instead of differential equations). Exploring this direction is an interesting avenue for future research.\n\n**Answer for Note.** Thank you for the opportunity to address your concerns in detail. Regarding the simplified setting used in our theorems, we adopt a common theoretical approach in which simplified models allow for tractable and rigorous analysis while retaining enough expressive power to capture key patterns seen in practice. Rather than analyzing the exact, highly intricate architectures used in real-world models, theoretical work typically focuses on simplified yet sufficiently general approximations. This methodology aligns with several high-quality works by leading researchers in the field [A, B, C, D]. Fully addressing the complexity of practical Transformer models remains a significant challenge and is currently beyond the scope of comprehensive theoretical analysis.\n\nFor a more detailed explanation, please refer to **A6**.\n\n**References**\n\n[A] Geshkovski et al. The emergence of clusters in self-attention dynamics. NeurIPS 2024.\n\n[B] Geshkovski et al. A mathematical perspective on Transformers. Bulletin of the AMS 2024.\n\n[C] Vo et al. Demystifying the Token Dynamics of Deep Selective State Space Models. ICLR 2025 (spotlight).\n\n[D] Karagodin et al. Clustering in Causal Attention Masking. NeurIPS 2024.\n\n-----\n\nWe will incorporate the discussion in the revised version. If our responses adequately address the concerns, we kindly hope the evaluation may be reconsidered accordingly. We remain open to further discussion in the next stage of discussion."}]

Author rebuttal:

Meta-review: This paper investigates the impact of different forms of positional encoding upon token representations in Transformers from a theoretical perspective in the continuous-time limit. It shows that the behavior as time goes to infinity is ultimately determined by the positive- or negative-definiteness of two matrices formed from Q, K, and V, assuming that those matrices are symmetric. It further proposes a strategy to ensure divergent behavior rather than convergent behavior.

Reviewers at first found the disconnect between the setting studied and real Transformers to be implausible, especially the weight-tying assumption. In response, the authors pointed to other papers that make similar assumptions and papers that make more restrictive assumptions, and, perhaps more importantly, extended Theorem 3.6 to untied weights. They also pointed to results showing that aspects of the theory seem to hold for real networks, e.g. when LayerNorm and the FFN are present token trajectories still cluster, and the eigenvalue distributions of symmetrized matrices from real models are consistent with the theory.

Ultimately, the reviewers were satisfied, and all reviewers now recommend either borderline acceptance or acceptance. I concur with their assessment.
Decision: Accept (poster)