Clustering in Causal Attention Masking
Authors: Nikita Karagodin
Yury Polyanskiy
Philippe Rigollet
Venue: NeurIPS 2024 poster (2024)
Keywords: Transformers
causal attention
continuous-time interacting particle systems
clustering
Primary area: learning_theory
TL;DR: Theoretical study of clustering phenomena in evolution of tokens representations as they propagate through layers of a causal transformer.

Abstract: This work presents a modification of the self-attention dynamics proposed in Geshkovski et al to better reflect the practically relevant, causally masked attention used in transformer architectures for generative AI. This modification translates into an interacting particle system that cannot be interpreted as a mean-field gradient flow. Despite this loss of structure, we significantly strengthen the results of Geshkovski et al in this context: While previous rigorous results focused on cases where all three matrices (key, query, and value) were scaled identities, we prove asymptotic convergence to a single cluster for arbitrary key-query matrices and value matrix equal to the identity.
Additionally, we establish a connection to the classical R\'enyi parking problem from combinatorial geometry to make initial theoretical steps towards demonstrating the existence of meta-stable states.

Clustering in Causal Attention Masking
Nikita Karagodin'
Yury Polyanskiy
Philippe Rigollet
Abstract
This work presents a modification of the self-attention dynamics proposed by Geshkovski et al. (202b)) better reflect the practically relevant, causally masked attention used in transformer architectures for generative AI. This modifi- cation translates into an interacting particle system that cannot be interpreted as a mean-field gradient flow. Despite this loss of structure, we significantly strengthen the results of Geshkovski et al. (2023b) this context: While previous rigorous results focused on cases where all three matrices (Key, Query, and Value) were scaled identities, we prove asymptotic convergence to a single cluster for arbitrary key-query matrices and value matrix equal to the identity. Additionally, we establish a connection to the classical Rényi parking problem from combinatorial geometry to make initial theoretical steps towards demonstrating the existence of meta-stable states.
Introduction
The introduction of the Transformer architecture Vaswani et al. (2017) has markedly impacted the landscape of natural language processing (NLP), signaling the advent of large language models. Central to the Transformer architecture is the self-attention mechanism, a special kind of layer that distinguishes it from preceding models such as ResNets. This innovation has yielded unprecedented performance not only in machine translation and text summarization but also in areas beyond NLP, including computer vision, speech recognition, and robotics. The flexibility and efficiency of Transformers underscore their integral role in the progression of artificial intelligence. Despite their widespread use, the theoretical foundations underlying their success remain underexplored.
Following Sander et al. (2022), recent studies by Geshkovski et al. (023a) and Geshkovski et al. (2023b) have proposed a mathematical framework analyze Transformers interacting particle systems, demonstrating that tokens, when modeled as particles, exhibit clustering under certain conditions on the Key, Query, and Value matrices. These works primarily focus on full (mean-field) attention mechanisms, where each token can interact with every other token. Building upon this foundation, our research extends the analysis causal attention mechanisms, wherein each token is restricted to interact only with preceding tokens. This distinction is crucial, as causal attention is prevalent Transformer models employed in generative AI and known decoder architectures.
Causal attention is crucial for sequence generation tasks, ensuring that each token only attends to previous tokens and not future ones, thereby preserving the correct temporal order. This mechanism, also known as autoregressive attention, masks future tokens during attention computation to prevent the model from accessing information hasn't generated yet. At inference time, causal attention allows the model to generate text one token at time, using previously generated tokens to inform the next, ensuring coherent and contextually accurate sequences. This step-by-step generation process is computationally efficient, as each token is produced in a forward pass without needing to revisit previous steps. contrast to full attention, which considers all tokens simultaneously and suitable
"Laboratory for Information and Decision Systems, MIT, Cambridge, MA, USA I Laboratory for Information and Decision Systems, MIT, Cambridge, MA, USA I Department of Mathematics, MIT, Cambridge, MA, USA
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table 1: Possible Final Configurations of Particles
Largest Eigenvalue
Multiplicity
Final Configuration
Figure
mmx >0 0
d
First particle zig
la
Amax > 0
2 2
One point in L Two points and -
16
Amax > 0
1
Ic
Amax < 0
2 2
Point cloud around L Two point clouds around & and -
1d
Amax < 0
1
Te
for tasks like machine translation where the entire sequence is known, causal attention is essential for tasks requiring real-time, sequential output. This computational advantage explains the pervasiveness of causal attention not only in natural language processing but also in image generation with tools like DALL-E (Ramesh et al., (02I). VQGAN (Esser et al. 2021), or Parti (Yu et al., 2022) and multimodal foundation models, notably Chameleon (Team) 2024). More generally, the use of masked attention where tokens pay attention to a subset of other tokens has been driving recent scaling efforts and has led to ofateooftthe-ar models such as MUSE (Chang et al. 2023) or Alphafold (Abramson interacting particle system but it requires
et al., 2024). Causal attention also be recast different analytical techniques. This is the goal of our paper.
Our contributions. Our main theoretical result establishes asymptotic clustering of tokens for causal self-attention transformer modeled as an interacting particles system on the sphere (Theorem 441). While mathematically accurate, this asymptotic collapse to a single cluster is seldom observed numerically. Instead, particles collapse to multiple clusters and stay in this configuration for a very long time (see Fig. 2 for representative example) - such meta-slable states were already alluded to in Geshkovski et al. (2023b) and their study was recently initiated in Koubbi et al. (2024). In Section 5 we describe such meta-stable states using analogy with the Rényi parking process (Lemma Theorem (5.I) Additionally, Theorem 5. covers asymptotic clustering of tokens for causal self-attention with additional cross-attention component. Moreover, we predict that, akin to lincar dynamical systems, the most important factors that qualitatively describe final particles configuration both in causal and full-attention cases the eigenvalue of the Value matrix V with the largest real part Amax and its eigenspace L, while Query and Key matrices Q K and temperature parameter & do not matter. Our conjectured atlas of possible meta-stable configurations is listed in Table D We prove the result stated as the first line of this table, namely that particles eventually collapse into a point when V Ia in Theorem 4. We remark that assumptions of Theorem 4.] are much weaker than for the similar results in the full-attention case Geshkovski et al. (2023b), in particular we put no constraints on K, Q B. This work is a combination of rigorous mathematical results and non-trivial predictions based on analytical insights and numerical simulations. We summarize all limitations in 6ection
Related work. Our work builds upon the framework of Geshkovski et al. (203b a where clus- tering properties of transformers are analyzed systems of particles interacting the sphere. Specifically, Geshkovski et al. (2023b) proved that encoder-only (i.e. unmasked) self-attention with (post) LayerNorm leads to tokens clustering to single point, in the limit of number of layers going to infinity. This phenomenon is also known as consensus in the related literature of multi-agent systems Markdahl et al. (2077) Criseitiello al. (2024) and Kuramoto oscillators Strogatz (2000); Abdalla et al. (2022). Work Geshkovski er al. (2023b) in turn expands on the original perspective brought forward by Sander al. (2022) that identify the self-attention layer as measure-tor measure-to-mmasure measure map, see also Vuckovic et al. (2021). J. More recently, Casttin et al. (2024) studied the smoothness of this map in a framework that also covers causal attention. This work introduces a clever reparametriza- tion that allows them to recast causal attention mean-field dynamics, akin to their full attention counterpart. Using various approximations, Cowssik et al. (2024) were able to study a more realistic architecture that also includes MLP layers and produce accurate predictions for the final configuration of particles. This setup was further investigated by Agrachev and Letrouit (2024) from a geometric control perspective. We note also that clustering in the absence of residual connection (replace ir (1) with (t 1) (SA)) was established in Wu et al. (2023). Additional effects of the residual connection are studied in Dong al. (2021) and Zhang al. (2024).

---

(a) V= diag(1, 1, 1)
(b) V diag(1, 0)
(c) V = diag(1, 0, 0)
(d) V diag(-1. 1, -1,
(e) V = diag(- -3,
Figure I: Particle trajectories for different Value matrices. In all cases we take simple Query and Key and final time T 5000 for n 32 particles initialized uniformly at random on the sphere. Positions of particles at time are indicated by red dot.
matrices K Q Ia. temperature
2 Causal attention
Before describing our model of causal attention dynamics, we review the idea of Geshkovski et al. (0233b) for modeling the full attention dynamics. that work, the evolution of representations of tokens through the layers is modeled as system of coupled Ordinary Differential Equations (ODEs) describing dynamics of a system of particles z1 (t).... Iy (t). A brief part of their derivation of the dynamics from the transformers architecture is written in Section A. The particle position ny (1) corresponds to representation of the k--th token at layer (where for convenience, & allowed take non-integer values) and due RMSNorm the particles are forced live on unit sphere $d-1 (RMSNorm layer usually also includes multiplication by a trainable diagonal matrix D, but the effect of this step can be equivalently achieved by multiplying K, Q, V matrices by D.) These ODEs are parametrized by three matrices, known as the query Q, the key K and the value V, respectively, and that assumed be square d matrices. More specifically, token & evolves according
(t) = (z)(1)
(l0m (1)
(SA)
(t(t)
where Pzy y
fa.y]
is the projection onto the tangent space of Sd-1 at and 1x
" Zx(t)= Σ e (1)
a normalizing factor. Note that the dynamics of the k-th token depend on the positions of all tokens j E [m]. which is landmark characteristic of full attention leading to the so-called mean-field dynamics studied in Geshkovvki et al. (2023b); see also Geshkovski et al. (2023)): Castin et al. (2024): Paul and Trélat (2024).
this work we focus on causal attention, where the dynamics of token & depend only on the position of tokens j < k. As described in the introduction, this modification is by now the dominant type of transformer architecture
generative AI. To reflect causal masking, we modify the ODE governing the dynamics of token k as follows:
d (t) Pr.(1)
(CSA)
Zx(1)
3

---

where the normalizing factor (t) is naturally updated to
(t) = Σ (00x200)
(x(t)=
3 Single token dynamics
Note that in (CSA) dynamics, the first token is evolving fully autonomously without the influence of others. Thus, we start from the description of its evolution. will also guide our understanding the dynamics of subsequent tokens. The first token moves according to the equation
f(t) = Px 2()) (V (Vr(t)).
To state its behavior for any matrix V. need few definitions. Denote Amax the largest real part of all the eigenvalues of V. Let be the span of all generalized eigenvectors to eigenvaluesxpotennillly different) with their real part equal to Amax. Let L C L' be the subspace generated by only the eigenvectors in L' with the largest corresponding Jordan block (the vectors might correspond to different blocks and even to different eigenvalues).
V associated
Lemma 1. Let r(t) be solution of an ODE i(() = P2 P (Vr(t)) defined on the unit sphere S Then, for almost every initial value x00) $ there exists C, O such that the following convergence rates for the geodesic distance dist hold:
(i) Exponential convergence IA: dist( n nn-nngg------ Ce-st, and
(ii) linear convergence L: dist(x(()), Ln Lng cl
This result can be derived from standard results the theory of linear ODEs (proof in Section B.1). We note that this result is important for other tokens as well. Indeed, for every token Ty. the contribution to d3 in (CSA) from the term with j k often has the biggest weight, effect amplified by large B.
In general, eigenvectors corresponding to real eigenvalue À = Amax create fixed set in L 8d-1 while the complex eigenvalues with the largest real part produce limit torus in I. gd-1, In what follows, only consider the case where the eigenvalue with the largest real part is real itself and it only has Jordan blocks of size Then, = L' and convergence to exponentially fast. Note also that when dim L = 1, we have L n { for some unit vector &. In this case, x(() - : as + co, again with exponential speed. These observations will be important for the next section, when we describe asymptotic configurations of tokens.
4 Final Configuration
The system of n tokens that we are studying is far more complicated than for single token. Even establishing convergence some point as / oo is challenging. In Geshkovski et al. (2023b),
similar models were analyzed analytically by noticing that the dynamical system has the structure of the gradient flow of some potential function:
+(() = VH()).
1984) (see Haraux (012 for For such systems, groundbreaking results of Eojasiewizz 1962 1965
self-contained overview) guarantee convergence critical point of H assuming real-analytic.
However, our system (CSA) does not have a gradient- flow structure and thus techniques of Lo- jasiewicz are not applicable. On the other hand, we have significant advantage in the hierarchical structure of our system, allowing us to study tokens sequentially.
We have already understood the evolution of the first token. In this section, we do two things. First, we describe, based on our analytical and numerical insights, conjectures about the asymptotic configuration sr()) for 1. oo. The surprising result here is that only the spectral properties of V (and not K Q) affect asymptoties. Second, we rigorously prove convergence to a single point for the special case of V La. We note that unlike the proof in Geshkovski et al.) (2023b) (see also Markdahl

---

et al. (2017) C Crscittielo et al. 2024)), our result works for all K and Q matrices. while the proof Geshkovski al. (2023b) works only for Q K V and Markdahl et al. (2117): Criscitiello et al.
):
(024)) is restricted to Q @@" la-
major forces that drive each token: its internal force which is described by Lemma [ and the external force induced by all the particles preceding it, which is either attractive or repulsive depending on the sign of the top eigenvalue(s) of V. The balance between the two forces is defined via attention.
Our main insight is that there
To get a better grasp of how the external foree works, consider the case where the first (internal) force vanishes, that is, V = Ia- In this case, the tokens collapse asymptotically to a single point.
Theorem 4.1. Let V = La and Q, K be arbitrary matrices. Then, for almost any starting point (an (O..... ...., F. (0)) with respect to the volume measure on (Su-1)², the causal transformer dynam- ics CSA converge to single cluster:
Vk [n), lim 2x((t) (0)
We prove this result in Section B.2 In the proof, weight functions only required to be positive and continuously differentiable (C") This ambiguity suggests that incorporating iimedependence of Q and K might not alter the theorem's 's validity, but it significantly adds complexity to the proof in dealing with non-autonomous systems.
Steps similar to our proof of Theorem 4.1 can be followed to study the more general case of the matrix V * la. Unfortunately, one runs into multiple technical issues with application of the stable-manifold theorem from dynamical systems due to the emergence of critical manifolds (as opposed to critical points in the V = Lạ) case. Thus, we leave the general case at the status of conjectures, which we describe next.
In what follows, we denote the eigenvalue of V with the largest real part Amax and assume that is real. If it is not, the limiting configuration is additionally rotating with a constant speed, which complicates the discussion and is omitted.
Let L. denote the eigenspace of Amax- If Amax has multiplicity 1, then we denote the corresponding unit eigenvectors as + For simplicity we assume that Amax has all of the corresponding Jordan blocks of size 1.
First of all, if dim L 1 then, according to Lemma I every token is driven towards & or - by their own force. Moreover, for Amax > the force of other tokens is attractive, while for Amax < 0 it is repulsive.
Thus, for Amax 0 all the particles collapse into & and -E, whereas for Amax < the repulsion force prevents the particles from going all the way to + and instead the particles stabilize at two clouds around & and - This behavior captured in Figures In [c For the case Amax > 0 we formally express it as:
Conjecture 1. Let Q, K be arbitrary matrices and V be diagonalizable with d different positive real eigenvalues. Denote the largest eigenvalue as Xmax and unit vector & V Amax Then, for almost any starting point causal transformer dynamics (CSA) converge two clusters
with respect to the volume measure on (gd-1)", the
A k E [n], lim (x (5.-E)
If Amax has multiplicity at least 2, then from Lemma I each token internally gets attracted by the eigenspace L. When tokens are close to L. the action of V becomes close to Amax which for Amax 0 according to Theorem 4.1 forces tokens to collapse to singleton, while for Amax 0 other tokens exude a repelling force, causing particles to spread out around L. This behavior is captured in Figures and [d] For the case Amax O we formalize it as follows:
Conjecture 2. Let Q, K be arbitrary matrices and V be any matrix such that its largest eigenvalue Amax 0 is real and has an eigenspace of dimension dim L 2 2, while for any 2 E L one
"The spread of the clouds depends on the relative importance of each token's 's own attention, that differs with various K. Q. There are choices of K and Q that result in complex interactions without structure. For example, when K [[0, 1, 0,0, [1. O, 0), [0, 1]], V = -13
5

---

170
370
4.5
-.5
-00
(a) Time
(b) Time 5
(c) Time 10
10
12
435
4.
4.5
4.5
(d) Time 75
(e) Time 150
(l) Time 500
with n 200, d 2, B 64, strong Rényi centers (red) and Rényi centers (black) with 5 4,9-1/2, Note that strong Rényi centers are visually stationary (as per Lemma but do not explain all clusters. In turn Rényi centers are moving and merging (one disappears between 75 and 150). but capture more meta-stable clusters.
Figure 2: Evolution of the system (CSA with K Q
(O)....,,,...... F....,,,(()) (O))),, the causal attention dynamics (CSA) converge to one cluster. More specifically, if we define & as the normalized L-component of x i.e., for y1 := I.. (x) (0)), E /1/lyi,, then
has Vz E L with (V2,2) <mmxxlz|3 Then, for almost any initialization (x: (z) (0),.... ....,
k E [n]. lim (t) c.
1-000
(Note that & undefined when xi (0) T., but this happens with probability zero.)
An important practical observation is that these conjectures explain that V performs dimensionality reduction in the following way. Tokens converge L n gd and, in that space, they move as if acted upon by the Ala matrix on sphere 1-1 For the pre-trained Lan et al. (2020) the spectra of value matrices is depicted in Figure Interestingly, there are heads with negative Xmaxx Future work will be concerned with studying real-world matrices V and connecting their top eigenspaces to semantic meaning of layers and tokens.
5 Meta-stable clustering
As we discussed earlier, perhaps the most fascinating discovery of Geshkovski et al. (2023b) the existence of meta-stable clusters in the full-attention dynamics. turns out that the same phenomenon persists in the causal-attention dynamics that we study here.
The dynamical evolution of the system is illustrated in Fig. At / 150, the initially uniform distribution of 200 particles consolidates into seven distinct clusters. While Theorem 4.1 establishes the eventual collapse into single cluster, these intermediate clusters exhibit remarkable metastability, persisting with negligible movement over extended time periods-an least until t = 500 according to
Fig. 2 before sequential merging occurs. We define these meta-stable configurations meta-stable
clusters, with dimee-iimesional analogues shown Fig. la and lb
Given that the time parameter in our dynamics corresponds to network depth in transformer archi- tectures, the meta-stable configurations, rather than final states (achieved at = exp(P((3))), hold greater practical significance. The emergence of meta-stable clustering and its associated dimension- ality reduction may provide fundamental insights into transformers' capacity for generating efficient context-dependent representations.

---

From a theoretical perspective, understanding meta-stable clustering presents significant challenges, as traditional techniques for asymptotic analysis such as those used our Theorem 4.1 prove insufficient. Recent work on full attention transformers has made partial progress in this direction. Koubbi et al. (2024) demonstrated that when self-attention dynamies approach a nearly clustered state, they will converge a tightly clustered configuration and remain stable for an exponential time period. Complementing this, Bruno et al. (024) proved that tokens initialized near a uniform distribution on the sphere will spontaneously organize into a loosely clustered state. However, the bounds on the clustering tightness in this second line work are not sufficient to trigger the convergence conditions required by Koubbi al. (2024)* theorem.
This Section presents a fundamental discovery regarding the identification of cluster centers in causal- -attention dynamics. We establish three key claims: First, we demonstrate that initialization irregularities generate distinctive tokens, termed Rényi parking centers, which evolve into mttaestable cluster nuclei. While this phenomenon is primarily supported by numerical evidence (Fig. [3. it provides crucial insight into the clustering mechanism. Second, we prove that a subset of these special tokens, called strong Rényi centers, maintains near-stationarity over extended time periods (Lemma 2). Both Rényi and strong Rényi centers occur with frequency O "=1 scaling predicted for d = by Geshkovski et al. (2023b); see also Koubbi et al. (2024); Bruno et al. (2024). Third, we establish in Theorem 5.1 that as t + oc, all remaining tokens will converge the vicinity of one of these stationary tokens, completing the meta-stable clustering process.
confirming the vB
This section restricts analysis to the case where V Ia. For general matrices V, our empirical observations suggest that particles rapidly converge to a pover-dimensional subspace spanned by di < principal eigenvectors. Consequently, we conjecture that the number of meta-stable clusters should rather be 3 - where the ambient dimension d replaced by the effective dimension d1- While rigorous proof of this cumension-reduction remains an open problem for future investigation, this phenomenon motivates our focus on low-dimensional cases (specifically d = 2) throughout this section.
For convenience, we also fix = K = Id. though this condition could be easily relaxed (c.g. to Q K KTQ Q V). Under these assumptions, the system can be rewritten in polar coordinates Ik = [cos((pk), sin({px)] as
1 h
k Σ
e (ccos(ps
sin(yo - (p.)
h(pp
spx))
(CSA-2d)
7xk
7k
j=1
with interaction potential given by
k Σ
(x) = ((con(o)) sin and Zk.
Mennl -Pp)))1))))))
(IntPot)
j=1
5.1 Rényi Parking
The prediction meta-stable clustering center locations exhibits a notable connection to the Rényi parking problem.
Consider sequence tokens (I,) on the sphere Sd-1 equipped with geodesic distance dist. For fixed separation parameter >> O, we define:
Rényi centers as the subsequence (2,,)/21 where (sy)121 is strictly increasing and satisfies:
dist dis ) >> 8 for all i <
Strong Rényi centers as the subsequence (Ie, )22 satisfying: dist( x:) for all 8j
By construction, the set of Strong Rényi centers forms a subset of Rényi centers.
As demonstrated in Section A.2, particles in system exert maximal attractive force distances of order at most 812/2 with rapid decay beyond this scale. For strong Rényi centers defined
7

---

Parteniage particiss consumed by prodiction
140
50
Bote
Strong Berry
20
50
75
100
:25
150
375
202
Figure 3: Total percentage of particles consumed by Rényi and strong Rényi centers time. Here we have plotted average, 0.1 and 0.9 quantiles over 5000 experiments with n = 200, = 2, B = 64, = 43 4/21/2 1/2.
with separation parameter á == c3-1/2 (where e is sufficiently large), this decay ensures negligible influence from preceding particles and thus remain stable for long time phenomenon formally established in Lemma 2 This metastability, coupled with rapid particle aggregation tokens, indicates that strong Rényi centers serve as primary attractors for subsequent tokens.
Rényi centers are unaffected by previous particles but only by previous Rényi centers, thereby generating new clustering centers. For fixed there are more Rényi centers than strong Rényi centers (see Section C.4 for exact cardinality analysis). While Rényi centers better capture the meta-stable clustering effect, as illustrated in Figures and 2 they lack positional stability and may converge to other centers over time. Although Rényi centers rapidly aggregate large fraction of particles, some of these particles continue to migrate and eventually converge to strong Rényi centers.
The next result shows that strong Rényi centers remain nearly fixed for a long time.
Lemma 2. Let 2 and Q K V 1z- Consider subsequence of strong Rényi centers Esy: Dsm. satisfying the separation condition with constants E, > 0
min E2y x > c(1 2=)2-1/2
Assume that
c >> 81/2 arccos((-1+ 4.2 + (23))
(1)
Then for any time T3 such that
Tys,h(c3-112) 00:3-145 1/2
(2)
where the interaction potential h is defined in (IntPo the displacement of each center is bounded by
12 (t) Fn, (0)) 1/2
max kE(..T)
The key observation driving this result that strong Rényi centers weakly affected by all previous particles. However, though this is correct on short time scales, it should be checked for all times in [0, T). complete proof can be found in Section C.I
Remark 1. Using the properties of h derived in Section A.2 it can be shown that
For 3 > sufficient condition for ( to hold is simply
sufficient condition for ② hold is
Tys c 12-22 11288
Moreover, is easy to prove that indexes 8j are mostly small. Thus, we see that early strong Rényi centers almost stationary for a time that is exponential with the square of separation magnitude.

---

Rényi centers and strong Rényi centers play a fundamental role in meta-stable clustering, warranting analysis of their properties extreme points in a sequence. While defined here using geodesic distance on sphere, the definition extends naturally to distances induced by (Qx, K Ky) under appropriate conditions. This generalization aligns with our observation that meta-stable clustering occurs in the subspace I. where V sends tokens and acts as the identity on L.
The distribution of these centers under various initialization schemes presents key analytical challenge. Section C.4 addresses the uniform i.i.d. case, where Rényi's classical result characterizes the expected number of centers. Extensions general distributions and Markov processes more relevant to language processing complexity. particularly in higher dimensions > 2). In contrast, strong Rényi centers are much easier to handle: even our computation of the average number of centers in Section C.4 works for any distribution regardless of the dimension.
-etiiattion--mmmlll open for Rényi centers due to their structural
5.2 Fixed Meta-stable Clustering Centers
Having established the existence of O(V3) quasi-stationary tokens for d 2 and n >> 1, we next examine their role cluster centers. While Figures and [ provide substantial numerical evidence that these tokens attract and aggregate nearby partieles, a rigorous proof remains elusive. We establish instead weaker result: when quasi-stationary tokens artificially frozen (analogous to cross-attention in encoder-decoder architectures), all other tokens converge to these frozen centers. This simplified model, while instructive, differs from true meta-stable clustering in important aspects detailed in Section 6
We only state result for d = 2 and identity parameter matrices in CSA-2d)
Theorem 5.1 (Clustering to frozen tokens for K Q V I2). Let 01,... 0m be fixed tokens
that are well-separated, namely 00 0gl c3 -1/2, Let μr be an absolutely continuous probability measure on ( and let şi (0)) (O)..... ...., (n( po- Consider causal attention dynamics (CSA-2d)
with additional influence from the fixed tokens 0j, which enter evolution with additional weights
aj 1. Specifically, we have
((coo((1- sin((p - 4pt) Σ
a a @ccoolpp -0,)) -0,)-1) sin(0; - xpk)
$x = Z
1-1
with
m
Zz
Blcoa(ee- -5))-1
age Bloos ((x))))(x=-0)----00 11))
j=1
j=1
Define N + E aj and N' where the interaction potential of [InPPo
N, 8, 0, and <> 2 2E satisfy:
Nh((c - 1- 26) < hee881122
Ng((c- 28)8 1/2) < 1/28-112).
then with probability one, pf() converges to asymptotically stable critical point spt (8²) satisfying:
Vk [m]. 3j E [m] 15t - - 105-01<<E5 <
=B-112
Since our dynamical system is not a gradient flow, the classical Lojasiewicz convergence theorem does not apply. Instead, establish convergence by observing that the causal dynamics (both with and without frozen tokens) is, in fact, a sequential gradient flow, where each particle minimizes a slightly different energy potential. For such systems on S' demonstrate convergence through an alternative approach that circumvents the Fojasiewicz framework.
Lemma 3. Consider system of particles on S! with angular coordinates (P1....,,PP Pn 2TR/Z evolving according to
0 Ea Ea
Gk Z spx) 0er
where Fn...., F'n are C2 energy functions and 7x are C normalization factors bounded by 0 < < Zx(Ø)) < C. Assume:

---

1. Each Eş, has isolated critical points in Pp for fixed 93. + k (satisfied by analyticity),
2. For any & E [n], critical points restricted to the first k particles are either strongly stable (the Jacobian has only eigenvalues with strictly negative real parts) or strongly unstable (there is an eigenvalue with a strictly positive real part).
Then for almost every initial condition sp(0) with respect to Lebesgue measure, (p(t) converges to strongly stable critical point s²"
The proof is deferred to Section
Remark 2. The conditions of Theorem
5.1 are satisfied under the the following explicit bounds:
8 0.1, C 5., B (c- 2z)² N
C exp(3(c -2))²/8)..
Note that this result requires only 3 2 log N. For example, taking & = 0.1 and c = 6.5 yields ,3 2 14
and N < 700 is sufficient. See Lemma for the proof.
6 Limitations
Our analysis presents both theoretical and practical limitations. From a theoretical perspective, we establish two key results: (1) strong Rényi centers maintain quasi-stationarity for time scales of order exp(c ( c22 per Lemma and (2) exactly stationary centers attract all remaining particles (Theorem (5T)). However, this falls short of proving meta-stable clustering, as Theorem [5. provides no bounds on the convergence time. Consequently, we cannot guarantee that strong Rényi centers remain sufficiently stationary during particle aggregation. complete meta-stability theory would require demonstrating that each Rényi center captures M(n) particles in O(1) time Currently, even the weaker claim of capturing w(1) particles remains unproven, presenting a crucial direction for future research.
o0.
The practical limitations stem from two model simplifications: the use of tied weights across layers (though supported by successful implementations, see Lan et al. (2020)), and the omission of the MLP layer central Transformer architectures. Incorporating the MLP dynamics into our theoretical framework remains a significant open challenge.
Acknowledgments
The work of NK and YP was partially supported by the MIT-IBM Watson AI Lab and the National Science Foundation under Grant No CCF-2131115. PR is supported by NSF grants and CCF-2106377, and a gift from Apple.
References
Abdalla, P., Bandeira, A. S., Kassabov, M., Souza, V., Strogatz, S. H., and Townsend, A. (2022).
Expander graphs are globally synchronising. arXiv preprint arXiv:2210. 127888
Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J., Bambrick, J., Bodenstein, S. W., Evans, D. A., Hung, C.-C., O' Neill,, M., Reiman,
D., Tunyasuvunakool, K., Wu, Z., Zemgulyté, A., Arvaniti, E., Beattie, C., Bertolli, O., Bridgland, A., Cherepanov, A., Congreve, M., Cowen-Rivers, A. I., Cowie, A., Figurnov, M., Fuchs, F. B., Gladman, H., Jain, R., Khan, Y. A., Low, C. M. R., Perlin, K., Potapenko, A., Savy, P., Singh, S., Stecula, A., Thillaisundaram, A., Tong, C., Yakneen, S., Zhong, E. D., Zielinski, M., Zidek, A., Bapst, V., Kohli, P., Jaderberg, M., Hassabis, D., and Jumper, M. (2024). Accurate structure prediction of biomolecular interactions with alphafold 3. Nature.
Agrachev, A. and Letrouit, C. (2024). Generic controllability of equivariant systems and applications to particle systems and neural networks. arXiv preprint arXiv:2404.0 08289.
Bruno, G., Pasqualotto, F,, and Agazzo, (2024). Emergence of meta-stable clustering in mean-field transformer models.
10

---

Castin, V., Ablin, P., and Peyré, G. (2024). How smooth is attention? In International Conference on Machine Learning.
Chang, H., Zhang, H., Barber, Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K. P., Freeman, W. T., Rubinstein, M., Li, Y., and Krishnan, D. (2023). Muse: Text-to-image generation via masked generative transformers. In Krause, A., Brunskill, E.. Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J., editors, Proceedings of the 40th International Conference Machine Leurning, volume 202 of Proceedings of Machine Learning Research, pages 4055-4075, PMLR.
Cowsik, A., Nebabu, T., Qi, X.-L., and Ganguli, (2024). Geometric dynamics of signal propagation predict trainability of transformers. arXiv preprint arXiv:2403.02579.
Criscitiello, C., McRae, A., Rebjock, Q., and Boumal, N. (2024). Synchronization on circles and spheres with nonlinear interactions. arXiv preprint arxiv:2405./8273
Dong, Y., Cordonnier, J.-B., and Loukas, A. (2021). Attention is not you need: pure attention loses rank doubly exponentially with depth. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2793-2803, PMLR.
Dvoretzky, A. and Robbins, H. (1964). On the parking problem. Publ. Math. Inst. Hung. Acad. Sci., 9:209-224.
Esser, P., Rombach, R., and Ommer, B. (2021). Taming transformers for high-resolution image synthesis. In 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 12868-12878, Los Alamitos, CA, USA. IEEE Computer Society.
Geshkovski, B., Letrouit, C., Polyanskiy, Y., and Rigollet, P. (2023a). The emergence of clusters in self-attention dynamics, Advances in Neural Information Processing Systems, 36:57026-57037.
Geshkovski, B., Letrouit, C., Polyanskiy, Y., and Rigollet, P. (2023b). A mathematical perspective on transformers. arXiv preprini 12 10794.
Haraux, A. (2012). Some applications the Lojasiewicz gradient inequality. Communications on Pure and Applied Analysis, 1166):241722477.
Koubbi, H., Geshkovski, B., Polyanskiy, Y., and Rigollet, P. (2024). Dynamic metastability in the self-attention model. arXiv preprint arX1v:2410.06833.
Lan, Z.., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2020). ALBERT: Lite BERT for Self-supervised Learning of Language Representations. In International Conference on Learning Representations.
Eojasiewicz, S. (1962). Une propriété topologique des sous-ensembles analytiques réels. Colloques internationaux du C.N.R.S.: Les équations aux dérivées purtielles, pages 87-89, Paris. Editions du C.N.R.S.
Lojasiewicz, S. (1965), Ensembles semi-analytiques. Preprint, I.H..S.,, Bures-sur- Yvette.
tojasiewicz, (1984). Sur trajectoires gradient d' u'e fonction analytique. In Geometry seminars, 1982-1983 (Bologna, 1982-1983), pages 115-117, Univ. Stud. Bologna, Bologna.
Markdahl, J., Thunberg, J., and Gonçalves, J. (2017). Almost global consensus on the nesphere. IEEE Transactions on Automatic Control, 63(6); 164-16666
Paul, and Trélat, E. (2024). From microscopic to macroscopic scale dynamics: mean field, hydrodynamic and graph limits.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021). Zero-shot text-to-image generation. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8821-8831. PMLR.
Renyi, A. (1958). On a one-dimensional problem concerning space-filling. Publ. Math. Inst. Hungar. Acad. Sci., 3:109-127.

---

Sander, M. E., Ablin, P., Blondel, M., and Peyré, G. (2022). Sinkformers: Transformers with doubly stochastic attention. In International Conference on Artificial Intelligence and Statistics, pages 3515-3530. PMLR.
Shub, M. (2013). Global stability of dynamical systems. Springer Science Business Media.
Strogatz, S. H. (2000). From Kuramoto Crawford: exploring the onset of synchronization in populations of coupled oscillators. Physica D: Nonlinear Phenomena, 143(1-4):1-2)
Team, C. (2024). Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09888. 09888.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez. A. N., Kaiser, L. u., and Polosukhin, I. (2017). Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Vuckovic, J., Baratin, A., and des Combes, R. T. (2021). On the regularity of attention. arXiv preprint 2rX1vv2102..5628.
Wu, X., Ajorlou, A., Wu, Z., and Jadbabaie, A. (2023). Demystifying oversmoothing in attention- based graph neural networks. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S., editors, Advances in Neural Information Processing Systems, volume 36, pages 35084-35106. Curran Associates, Inc.
Yu, J., Xu, Koh, Y., Luong, T., Baid, G., Wang. Z., Vasudevan, Ku, A., Yang, Y., Ayan, B. K., Hutchinson, B., Han, W., Parekh, Z., Li, X., Zhang, H., Baldridge, J., and Wu, Y. (2022). Scaling autoregressive models for content-rich lext-to-image generation. Transactions on Machine Learning Research. Featured Certification.
Zhang, B. and Sennrich, R. (2019). Root mean square layer normalization. Proceedings of the 33rd International Conference on Neural Information Processing Systems.
Zhang, X., Jiang, R., Gao, W., Willett, R., and Maire, M. (2024). Residual connections harm self-super abstract feature learning. arXiv preprint arXiv:2404.
12

---

A Supplementary Material
A.1 From Transformers to ODEs
The derivation of the equation SA was thoroughly described in Geshkovski et al. (2023b). but for completeness, briefly repeat it here to explain how the problem arises.
In general, typical Transformer architecture consists of repeated layers of multi-layer perceptrons (MLP), normalization, and residual connections Vaswani al. (2017). In this work, we simplify this setting by focusing only on the geometric behavior of a single-head attention layer with normalization and residual connections, omitting the MLP for brevity.
attention,
One head of standard attention layer is defined as follows. Given an input sequence represented by the token embeddings X E R= where is the number of tokens and d is the dimension of the embedding space, and matrices Wo. K. Wy compute queries, keys, and values, the attention mechanism computes a weighted sum of values based on their relevance to query in the following form
X
Attention (X) = softmax
XW
vd
By adding RMS normalization from Zhang and Sennrich (2019) and residual connection, the transformation from layer t to layer + is given by:
x = X AMSNorm(AAtenntonn)
(3)
Here, different tokens are represented as rows of the matrix X for computational reasons. For consistency with the convention that vectors are represented columns, transpose everything and denote a sequence of tokens encoded as particles the d-dimensional embedding space IR as (21, ................................ J, corresponding the columns of X Additionally, to simplify the notation we denote : W v. Q W and K := W K' and introduce arbitrary temperature parameter /3 instead of the fixed scaling factor 1 1/vd. With these notational adjustments, one term of attention added to the k-th token be written explicitly as:
attn
Fnnk
c K(
Zx j=1
where
Zk [
The equation (3) can be interpreted as discrete derivative, with X X* representing the difference between layers. Therefore, the trajectory X can be viewed as discretization of continuous flow. RMS normalization ensures that tokens remain on the scaled unit sphere, but from properly rescaling Q, K, V we can assume that they stay on the standard unit sphere $d-1 Combining all these observations, the dynamics of token propagation through layers can be expressed as:
1
Σ (100000 20000000000000000000000000000000
(1)
(1)
with
Zx(1) = Σ ((0000=0))
and the projector Pz(y) = (x, y) ensuring that øs remains on the sphere. This leads to the equation (SA), and applying a causal constraint, where each token attends only to the previous ones, transforms into the causal attention equation (CSA) studied in this work.
A.2 Interaction Potential
For completeness, here we describe the key properties of the interaction potential M(x) = ((cos(())1)) sin(x) from (IntPot), which defines the interactions between particles, and derivative g(z)= h1(x).
13

---

Lemma 4 (Properties of Interaction Functions). Ler h(x) = ( sin and g(x) h()x).
Then:
1. h(x) is odd and g(x) is even.
2. h(x) is increasing on [0, r3) and decreasing on 17 20),
where
-1+
4,3² +
cOs T
2
and for ( 2
(821 < T3 < <g-1/2
3. For x >> O, h(x) is bounded by
e eBa (x x2- x²/ < h(x) < 12+
4. For g(z), the following bounds hold:
g(x) > B2² 12 2² 3xx²)
for 0 < (8 182
g(z) > - - Bo² /2+ / 3x?
for x > 0
Proof. The oddness of h and evenness of g follow directly from their definitions. Computing g(z) explicitly:
g(x) = The sign of ((x) changes at r3. where
+ cos(x)) e((o00())-11 (B)o(zz) + cos(2) 3)
4.2 +
cos 7
establishing that hxx) increases [0, r/l and decreases on [T 00).
For the the lower bound on Tă: For (5 + 1/2) 1/2, we have
x²² 22/22 Bx² Bsin
implying g(x) 0 in this region.
For the upper bound on Tg. fix z = 1/2 and observe that it suffices to show
T3- -1 4.3 +1
cos 8-1/2 cos
23
Using cos <1 22/2 24 /24, this reduces verifying
-22/2+ + /
/ 24/24
which holds for = < 3.13, satisfied when 8 22
3. and 4. The bounds on / and follow from the standard inequalities
I ーー 2 /6 < sin x < and - -22/2 cos -1²//2 /24
combined with our characterization of g's sign via rÿ.
We now turn to the proof of Remark Lemma 5. Let N, and 3 satisfy:
c2 5.5
B (c - 26)3/2
N < 3c-1--25) 8
C-1 1
Then:
Nh((c- 1- -2=)3-11122 < 1((8-1122 and -g((c 25)8-1/2 1112
14

---

Proof. L= c- 1 2s. From the assumptions, we have r 2 4.5, B 2 /2/2> > 10, and & < 0.1. We must verify:
N h(E3-1/22
h(r3-1/2)
9(55-1123
N -g((r + 1188-122)
Using the bounds from Lemma 4 these inequalities reduce to:
N cx0(==22)a311((11=/000
exp(=r7/24rr//248))88-11
and
N exp(=="/2(1-3/((23--
+ 101--+=--11//2((++)))))))33 1)2
Given < exp(3r2/8)e/ suffices to verify:
1. First inequality: Taking logarithms and using < 0.1, 3 > 10, we need:
1
8²
8
243
2
200
Since 3 2 r2/ this follows from:
2
8
12
50
which holds for r 2 4.5.
2. Second inequality: After simplification using B 2
/2,
10, E 0.1, need:
3r"
(r 11²
(r
1)a
1
f(r)
+ 21n(r
1) In <
2
12r2
200
The derivative
1
6
2
1
(r)
14
3
r 1r 1) 1)
r
for 4.5, as and 2r > Therefore, it suffices to verify f(4.5) = -4.14 <0.
B Final configuration
B.1 Proof of Lemma D
Let us show that trajectories æ<t) of our system can be characterized as normalized solutions of linear homogeneous ODE in Rd. Consider solution ytt of:
ú(t) Vy(t),
v(0) = (0)
(4)
For stt) y(t) ((t)/Il((t]]), we derive:
i(t)
v(t)
y(t)
3(t)
[[v(()] i(()) Vs(() (s(t), Vs(t))s(t) Pa(o (V)(t)
Thus v() >(1) v())(((l)) [l]]]
The solution to 4 has the following explicit form. Let {Jk} denote the Jordan blocks V with: sizes ne. eigenvalues Ag. and generalized eigenvectors {ch ...., ck }.
Then:
v(t)=
Σ k
e
dkt
(5)
15

---

where the coefficients {ef) satisfy:
A1 5 x00)
j=1
For almost all initial conditions z0) with respect to the surface measure on the sphere, all coefficients oj are non-zero. For complex eigenvalues A we combine conjugate terms to obtain real-valued solutions involving trigonometric functions.
The asymptotic behavior follows from two observations: (i) Terms with largest R(A))) dominate as t + oo, corresponding to convergence L' 5 at exponential rate, and (ii) Among these terms, those with highest power of t (i.e., t -f terms) determine the slower convergence to L n Sd-1
B.2 Proof of Theorem 4.1
We begin with simple geometric lemma.
Lemma 6, Let I, U; 2 E IRd with ||>||| = |ly|| = 1. (y, 2) S (x, z). then:
(x, 2 (ar, v) (y, z)
with equality if and only if either: (i) (z. 2) = 0, or (ii) Ky 2)ا = (x,, z) and (c,, v) = sign(( z)).
Proof. By the Cauchy-Schwarzzinequuee inequallty and the hypothesis:
(5.u) (y, z) s (1r, y) [,, =)1 s (a, u) (ar, & (ar,
where the last inequality follows since (xx, w) < for unit vectors.
The equality conditions follow from examining when each inequality becomes equality the chain above.
We continue with the proof of Theorem
Proof. The system of particles is governed by equations
1
Kos), (xz (x)))))
ZR
where we omit t from the notation for simplicity.
This system autonomous,
first explore its critical points and their stability. For autonomous systems with established convergence, is well-known that for any absolutely continuous initializa- tion, the limiting point is strongly unstable with probability zero (see (Shub 2013 Thm. III.7, Ex. III.3) and (Geshkovski et al., 2023b)] Lemma B.1)). Note that the proof in Geshkovski et al. (202bb) stated for gradient ascent dynamics but it readily extends to any smooth autonomous dynamics on a compact Riemannian manifold.
Define:
fa(x) = Σ/_ Kar,
giQus Kuja (I), xx)
1=1
We aim to (i) find stationary points i where all fa(x) 0 and (ii) analyze eigenvalues of the Jacobian 0f. at said stationary points. Gay
Any critical point must satisfy one of the following:
for some & Sd-11
* =
There exists ( n) such that #1
2z-1 = and 2g -
16

---

Indeed, if the first condition fails, consider the first token where 2e * 21. Then fa(2) implies Is (z E)E O, forcing I, = = so that 2, - since we required I, # Z1.
Our goal to show that stationary points of the second kind are limiting points with probability zero with respect the initialization distribution. Observe that since the system formed by the first particles is independent of subsequent ones, it suffices to show:
-5))
Since (t) 21 (0), this reduces to:
P(11 = &. 1o + + (f.......-EE). =
By the law of total probability, it suffices to show that for almost all & E 8d-1;
F Pag..... (a (2222 Io-1, xz) + ( =
We are left to the function fa around (II,..., 1z- v.) (E..... -E) Observe that E. = w(t 2. (5 - (5, xz)
where
(s 1)e (Qu,..K))) wrE,, x.) (s 1)e + eB(Qu,,Kur)) > 0.
a/x is block lower triangular, with blocks given by 8fc We show below Observe that the Jacobian (ax,)
that 0fa has eigenvalue with positive real part, which is sufficient establish strong instability, Dn,
At 22 Ie-1 and 2 -E:
f E,,,) = w(E,xx)( xx)(E - (5. v.)x
where
(s - 1) w(E, x,) (s (Q + cf(Qus,Kas) 0.
The classical Jacobian in Rd is:
Oz. f.(E. f..E...................... [. = (0_ u(s (5 (E. xo) + w(5, xz) dr.
(E (5. x2)
dr.
Hence
0 (.ls....,,,,,,)).
w() ② (5 (5,2x))))
0x,
Ox.
The spherical Jacobian is obtained by projecting onto ++ and given by
(I - EE Ox. fa(S...., E,.,), = w(g. -E) -II-I- (I-
This linear operator acts on +d and has eigenvalues w(S, -E) with multiplicity d - 1, which are all real positive, confirming strong instability.
By the center-stable manifold theorem (Shub) 2013 Thm. 111.7, Ex. III.),, if a point has at least one eigenvalue with positive real part, then: (i) The center-stable manifold Wres has positive co-dimension and (ii) Points converging to this equilibrium must enter Wes at some finite time and (iii) the set of such points has measure zero.
More precisely, a trajectory (Z2..... v.) converges to E, -E), then:
E Zzo (x2(m). ..... (m)) E toe-
Since our flow is a diffeomorphism, the pre-image of Wes is also manifold of positive codimension.
Therefore, the set of initial conditions leading to convergence to (E....: -E) contained countable union of measure-zero sets, making measure-zero set itself.
17

---

We continue with an induction on the number of particles to show that with probability one, 22...., all converge to Ç.
For the base case k = 2, observe that æ2 converges to & except when initialized unstable equilibrium (0) = -.
Assume next that 2x-1 + Ç with probability one so that for any > 0, there exists a time To after which
min( > - a.s. jek
Consider:
d(xx(1),
(22(())
2lQ02 100300 :5)
dt
(t)
1
(00mmt)) ((,,))
(t)
(z,(1), From Lemma 6 we get that (z) (1) (0) (Iz,E)) k). In particular, we get that the time derivative above is positive after time To whenever E after that time. But from the center-stable theorem argument, we know that (æk, E) does not converge to - so there exists time 7 To at which (I <) -1+ E. After this time, either F; gets closer to & (positive derivative) until (I., { =
0 (z,, > 0 and (Tx, <
-1+ < (#k.
since we are guaranteed that Vj k, (.,.
after which time, (KA, E) 2 1 forever. Since this argument is valid, for all O, we get that 2 +> &.
By induction, all points converge to & with probability one.
B.3
Spectra of V
Here we include figure showing the spectra of value matrices of different heads pre-trained transformer. Notice that most of them have real eigenvalue corresponding to Amax. justifying our focus on the real case. Interestingly, there are heads with negative Amax, even though starting spectrum of a Gaussian matrix is far from the left half-plane. This could indicate that some properties of Amax < are desirable practice, suggesting to consider other ways initialise V for training.
head 2
head
1
0225
025
-1
-0225
-0.5
head
head
head
heead
0.5
0.5
0.0
0.00
0.0
-0225
9.5
head 15
headin
head 10
head
0.5
1
0
0.0
0.0
0.5
hear
0.5
head 1
head 1d
0.5
0.0
-0.5
5
05
Figure 4: Eigenvalues of different heads of pre-trained albert-xlarge-v2 model in the complex plane.
18

---

C Meta-stable clustering
C.1 Proof of Lemma
We prove the lemma for d = 2.
Proof. For d = 2, written in polar coordinates z exp(i<x); the system is
1
Gk
(c))(pp=-;)--1) sin(ys ppx)
(6)
Zk
j=l
with
Zk ((00((p=-))-11)
/=1
Because of the j = k term one has Za 2 1, which implies
1 S Σ e Alcastes -1))-) sin(ssnn (Po))== = Σ M1ps Po.l).
Consider (o,, to be the closest particle fs. among previous ones, here r depends on 8 and Wlog, let us assume that spr. [Pss, f9,, + T]] and denote A = ffr Pss. We are going to show that A cannot decrease fast. At any time one has
1
Σ
h(pp Pol)
Σ h(pp - D.-)).
Zou
Z,
Let bound both terms separately. Due the definition of A, and as long as A T (from Lemma 4 can bound the first term
1 S sgh(A).
For the second one, we only leave the terms with 9o E lipr - pr-), because the other ones are negative
Pr- Σ j== [h(p; - (op---p--)1 91
In other words, the only particles that drive so.- towards ys, are the ones that are in the same direction. However, there are no particles closer to Paa than yor.. Therefore, being in the same direction implies being at least 2A far away from ,
If h(2A) O, there no particles S2 (,,, ग,, pr.] and the sum is zero, otherwise we Lemma to upper bound each term with (22)). Thus,
b, S 8y max (anx(h(2A)) (2(2)), < sah(A).
The last inequality follows from the fact that h())) > and monotonicity of when A > TBt Combining the estimates, we obtain that for A > T one has
< 2sch(A).
To prove that for e [b, we have (t) > c3 that initially
1/2 1/2 we just need verify that 1 rg- and
A()) c 1/2 28 Ti
which is true by definition of Rényi centers, and bounds ① and TІ (2). Then, from 7 one has
max 1p (t) Fo, (0) sxThh(c8-1122 1/2-
3 [E T
19

---

C.2 Proof of Lemma 3
Proof. We prove that the particles converge exponentially fast some strongly stable critical point "p" via induction on their number.
Induction base. In this proof the induction base for 1 follows from the induction step proof applied to the first particle 991-
Induction step. Consider 4p P.-1((t))))))).)... They do not depend on yn and from induction hypoth- esis converge exponentially fast to some asymptotically stable point .i...., Ph- 1. In particular, one has *ēx E ([0, oo)), & E [n 1/.
Consider full derivative
d.Fe((((t)) di
Σ 0e, OF (t) +
O (1)
n-1 Σ
dP di, 2,(1)
(.)0 -.,000000000
Ppn
5=1
Since partial derivatives e.,/ /ap; are bounded on the compact manifold and derivatives (..j < n are in ([0, oo)), one has
s(t) = Σ FF (t( E ()(),,00)..
dØ; 3=1
For the base case, i.e. n = 1, one simply has g(t) = 0. Let us show that f(t) == L ((0, oo)). Notice how ft) has finite integral on any interval
(p((t) di
is also in
Jh f(t)dt = =E((p(a)) (n(((()) <0<00
(8)
and is upper bounded
S(1) S a(1) E (10, oc)).
Consider f+ f(t)1 (()))) f f+ - f-. Clearly f+ E Li oc)) as it is upper bounded by lgl E L, o.). Moreover, the bound (8) shows that for all T > 0 one has
( [=)dss < ( f+()d + C || ((0, + C.
From this we get that f- E LI(( oo)). Consequently, f (10, o2)). Since
(n)+))l,., g(t) 10- f(t) (10 oo));
and (1) has a uniform lower bound, we obtain cx, E (2(
Let show that i.. is absolutely continuous. The trajectory of all particles cp(t) is absolutely continuous, because (tt) is bounded as continuous vector field on a compact manifold. Then, (p, is absolutely continuous, because it is composition of absolutely continuous vector field and an absolute continuous trajectory (tt).
Because & E L2([0, oo)) and &n. is absolutely continuous, it satisfies
lim sup |4hn (t)
t-
In other words, because of the upper bound Z, (t) < C, one has
2
lim t-
E, (pp (1), (p,(()) =
Finally, we are going to prove that en converges to some critical point 9 with
Ppn En (2,)) O.
Consider the set E = {bb ap..
E(Pi. ..... - 1: w) 0}. By assumption, the energy function
E, (y. - 1)) (6) has isolated critical points w.r.t. 1b, thus the of zeroes E finite collection of points E = {111,..., }.
20

---

When P1. , Px-1 converge to pi..... ........... Pn-1 the set
Pn
B(p ................................ (nn) <e]
Oep
inside a collection of distinct intervals around d/ .i...... Um for small enough e. Therefore, due to (9), from some moment y (t) stays only in one of those intervals. They collapse into points as we take ε + 0 and + oo, proving that 9n, converges to some s E E. remains to prove that observed convergence is to strongly stable point.
know that the limiting point is critical and for almost any initialisation not strongly unstable. This is well-known fact that for autonomous systems convergence to strongly unstable point happens with probability zero, that we used in Section B.2 One could find in (Geskkossii et al. 2023b Lemma B. based the center manifold theorem (Shub) 2013 Thm. III.7, Ex. III.).. Therefore, from the assumption on critical points, the limiting point is strongly stable. Then, the convergence happens exponentially fast, because locally the whole neighbourhood of strongly stable point is its stable manifold W loc: see (Shub 2013 Thm. III.7, Ex. III.3).
C.3 Proof of Theorem
Proof. Our prove consists of two parts. In order to obtain convergence we are going to apply Lemma The system has exactly gradient-like form for energy functionals
(ccoo(p)
a ((co)(pL-0)))11
and bounded normalization factors
Zu(PI,...... ..... pp)) (c00((1 +
Σ age Blcos(
Heoossl -8))
Therefore, if conditions of Lemma 3 are satisfied, have convergence a strongly stable critical point. In that case, it is sufficient to prove that for any strongly stable critical point yo", all particles si are 2f8-1/2-cose to one the centers 0j. We show this via induction on k, because new particles do not affect the movement of the previous ones.
To justify our use of Lemma 3 we need to check that all critical points are either strongly stable or strongly unstable, i.e. that the Jacobian at any critical point with only non non-positive eigenvalues actually has no zero eigenvalues. Moreover, for our conclusion, we need to check that if all the eigenvalues are negative, then the critical point is clustered around 0. Let us start with that.
In what follows we repeatedly use properties of and g from Lemma
The system is of the form
1
h(pp (01)) aayhisp 0,) fe(p,)))
m Σ 1=1
Z
Since fü does not depend on pb+1...., Pa. the Jacobian ( (ofz) lower-triangular, Therefore, its 8fc Let us assume that all the eigenvalues are non-positive. Since at a critical eigenvalues are dip. point fx(p"" 0) itself is zero, one has
0. (6 0) Dec
Σ
40)) + Σ agg(e 0,) < O.
7
This implies that one of the terms g(vp - 19)) or glvi - 0,) is positive. From properties of g in Section A.2 we obtain that vtt is r---close to either one of the centers 0; or one of the previous
particles vp. By induction hypothesis, all previous particles are - to the centers, i.c. sp 10-11/2-ncee neighbouroo of some center, wlog 0,.
in 73 + 1-22
21

---

Let us assume that &X is EB ffa--affarr from 01. From criticality of the point, it is true that
Σ h(pp - 03) + Σ az ayhlip 0,)
By induction hypothesis, all previous particles are a, 1/2-close some center. Denote the set of particles that are close to 01 as S. Then the criticality can be written as
m
aph(pi 01) + Σ h(pt 49) =
Σ hipt 93) Σ JES
ag aahlpi 8y).
jES
j=2
All of the terms on the left hand side have the same sign, because vj J are e3 10/011122000 1/2-cl to 0 and P is (73 + + to 0j. Any other particle/center is at least (c 2 1/2 7 far from Pk because centers are cf-1/2 separated. The absolute value of the l.h.s. can be lower bounded
(ath(pp)) +
EE(et - -251 2 ((28-1/2)
jES
The absolute value of the r.h.s. can be upper bounded
Σ h(pp 03) + E ath(pt 0,)1 s Nh((c- -000--22)3-22 T/) < Nhilc- 22)8-1/2). JES
where in the last part we used the fact from Section A.2 that TB < <B-1/2 Therefore, one has
Nh((c- 22) 1/2) 2 (01))
which contradicts our assumption on system parameters.
remains to check that when the eigenvalues of the Jacobian are not positive, they are actually strictly negative. We already know that in that case all the particles clustered around centers 0j. Let us assume that
0
(v' 8) = 0.
0.p
We know that st is 1/- close to some center, wlog 0, Therefore,
q 01) 2 g(EB-1/2).
On the other hand, the only negative terms in the
k-1 Σ
g(ph
99))
Σ agglpp 0,))
from particles that are in the neighbourhood fj, they are least (c- 2- 1/2-112-faa from Pk. Therefore, for the negative terms to balance the positive g 01), it should be true that
g(E 1/3) -Ng((c - -1/2).
This contradicts the parameters choice.
C.4
Rényi centers strong Rényí centers
In this section we estimate the number of clusters our approach predicts. As a reminder, for sequence of particles X, on a unit sphere, and geodesic distance dist, we consider two subsequences
Rényi centers is X.,,J . 2 such that Vj min:ey dist(X., Xx,) > 8,
strong Rényi centers is Xey.j such that Vj minsca, dist(Xe,, 8.
22

---

Estimating the number of elements in the first sub-sequence is well-known as famous Rényi parking problem Renyi (1958). In the case d 2, the result from Dvoretzky and Robbins (1964) implies that as 6 + 0, the average number of elements in the sequence approaches c2'r / superexponentially fast, where e 0.75 is the Rényi constant. However, this problem becomes significantly harder in higher dimensions. In contrast, it is easy to compute the average number of elements in the second sequence in any dimension and even for a wider class of distributions.
Lemma 7. In an infinitely long sequence X; the average number of variables Xs, chosen by strong Rényi parking is the inverse spherical cap surface area 1/o)->(Bs). In particular, grows as 1 1/8d-1 with dimension and can be computed directly in lower dimensions
1/00-1(B3 (Bs) = [Tō-1, ,d=
,d
Proof. Consider sequence of i.i.d. points X; being sampled on d-dimensional sphere gd-1 according to some distribution μr. Let us find the probability that k-th particle X is chosen by strong Rényi parking with distance 8. This event can be written
Pintzl Xx, Xj) > 63) /
Xj) > 8))dp(z)
where we used total probability formula. Then, since Xj are i.i.d,, we can write as
[ Jse-1 Pdiist(x, (o,ist(x,,))) X1) 8)* dp(x)) = J3e-1 (1 -1- ((B((()))*
From this obtain that the average number of chosen points is equal to
gd-1 P(dist(z, Xi) > (y--dd(()) = ( ((Bs(x)) du(xz) od=I(Bs)
where the last equality is correct for any spherically harmonic distribution μ,, in particular for uniform measure.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The work is dedicated to causal self-attention from interacting particles point view. Asymptotic convergence for arbitrary key-query matrices is proven in Theorem 4.1 final configuration for non-identity Value matrices is presented in Table I and Figure Connection with Rényi parking is shown in Section 3aand and properties of meta-stable clusters are proven in Lemma and Theorem 5.1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Significant limitations are discussed in Section 6 As mentioned in Section 4 provided approach to asymptotical convergence is generally close, but insufficient to prove all results listed in Table T that is why those results are only conjectured.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The proofs of all claimed results can be found in appendix. When stated, every result refers the section that is dedicated to its proof. All proofs well-structured, rely on known results and have all the assumptions clearly stated.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The only experiments are numerical models of considered dynamics presented in Figures and 2 Their main purpose is to support what is claimed visually. These calculations can be easily reproduced, as all the parameters used are listed and the system is not computationally complicated.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
25

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification: The only experiments numerical modelings of the system, that used to support conjectures in Section 4 and show performance of predicted meta-stable clustering centers in Figure They not claimed as main results of the paper and if needed the code for creating the pictures easy to reproduce.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
public/guides htt////oodeeiiiiioooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips wttpcccccpppcuauoaeeeeece guides/ www.//oooiisiiiioooo ssiiisolicyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification:
The paper does not include experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification: The paper does not include experiments.
Guidelines:
26

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: The paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://ppp....a......o/oooooo//a
Answer: [Yes]
Justification: Conducted research conforms with the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction)
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
27

---

Justification: The paper is focused on theoretical research and does not provide any direct improvement of existing models, thus it has no way of deployment that could somehow have harmful societal impact. In general, this work aims to improve our theoretical understanding of Transformers, that are prevalent in modern state-of-the-arr Al models. In this regard, could lead to improved long-term development of the field, that we see to be beneficial for the society.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper is theoretical and does not have any model to release.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
The paper does not utilize any existing code. Every existing idea or intellectual result is properly referenced upon its introduction.
Justification:
28

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicablle as well as details about compensation (if any)?
Answer: [NA]
Justification: The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
29

---

Answer: [NA]
Justification: The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as institution conducting the review.
30


Reviews:
[{"review_id": "4NOVjJlvvM", "reviewer": "Reviewer_6isq", "summary": "This paper presents a theoretical framework where causal attention masking can be recast into an interacting particle system. The authors start by introducing the dynamics of the first token and extend it to $n$ tokens. They then discuss the token configurations as $t \\rightarrow \\infty$ (i.e., infinite number of attention layers) when $V = I_d$ and make conjectures for more general cases where $V \\neq I_d$. Lastly, the authors discuss the discovery of meta-clustering in Geshkovski et al. and adapt causal attention to this framework using the definition of Rényi centers.", "questions": "It is well-known that transformer-based language models produce more stochastic outputs at higher temperatures $\\beta$. What additional insights does meta-clustering provide beyond this established understanding? Does clustering of the tokens indicate similar outputs?", "limitations": "The authors have provided a relatively comprehensive discussion of limitations in Section 6.", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "•\tExploring the theoretical aspects behind the full attention and causal attention mechanisms is a very important topic in our understanding of how Transformers and modern LLM/LMMs work.\n\n•\tThe authors give a comprehensive overview of the background and make a smooth transition to the token dynamics in causal attention.\n\n•\tThe authors clearly explain the dynamics and the final states of the tokens with visualizations.\n\n•\tDespite the paper’s extensive theoretical and mathematical details, its main narrative is clear and easy to understand.", "weaknesses": "•\tAlthough the authors argue about the complexity of the problem, $V = I_d$ might be too limited for real-world use cases.\n\n•\tThe probability measure $\\mu_0$ is first mentioned in Conjecture 1, but it is not clearly defined until Theorem 5.1. Same for the geodesic distance $dist$, which is first introduced in Lemma 1 but is not defined until Section 5.1.\n\n•\tSmall typo: Line 140, To get a better grasp of the effects of “how” the external force works, … Line 186, in the full-attention dynamics, …", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback. We are glad they think that **exploring the theoretical aspects behind the full attention and causal attention mechanisms is a very important topic in our understanding of how Transformers and modern LLMs/LMMs work** and that they enjoyed our presentation.\n\nTo address the biggest mentioned weakness, **although the authors argue about the complexity of the problem, it might be too limited for real-world use cases**, we note that even though on paper $V = I_d$ is a significant limitation, it is not irrelevant for practice. This is largely because of the dimensionality reduction effect, which is justified by numerical experiments and can be seen as an effect of the matrix $V$, as discussed in Section 3 and mentioned in line 219. We expect that for the general case of $V$, all particles quickly converge to the subspace spanned by a few top eigenvectors, where the matrix action is close to the identity (see Fig. 1b for an example). Thus, understanding the behaviour of the system for $V = I_d$ (and even in low-dimensional cases as Theorem 5.1 does) is important for understanding the general picture as well.\n\nWe will add earlier definitions of $\\mu_0$ and $\\textrm{dist}$ and correct the mentioned typos. Thank you for pointing them out.\n\n**It is well-known that transformer-based language models produce more stochastic outputs at higher temperatures. What additional insights does meta-clustering provide beyond this established understanding? Does clustering of the tokens indicate similar outputs?**\n\nIn addition to the mentioned knowledge, our observation might lead to several additional insights. For example, the observed connection to the R\\'enyi parking problem allows us to connect the number of meta-clusters with temperature and effective dimension. Moreover, there is an empirical study, that suggests that clustering corresponds to transformers learning specific tasks/concepts and assigning them positions in the representation space. Then, being able to control the appearance of these meta-clusters through parameters and predict their appearance might help construct better models for specific tasks. To our best understanding, in practical transformers tokens exhibit clustering, but do not converge to the same point completely (as one might expect from Theorem 4.1). This is an effect of different weights/MLP layers, but it also can be seen in our Theorem 5.1 -- with some tokens fixed, convergence happens not to the centers themselves, but somewhere close to them. Then, the outputs are not going to be completely the same, but arguably correspond to the same topic/task depending on the field of use. This is a really interesting question, how clustering corresponds to produced outputs, that requires further research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f5ZArgSagC", "reviewer": "Reviewer_Bsfq", "summary": "This work extends the work by Geshkovski et al. 23c, which analyzes the mean-field gradient flow of Transformer models and shows the emergence of clusters with full self-attention, to the ones with causal self-attention.\nTransformer with causal self-attention is modeled as an interacting-particle system on the sphere, as tokens are normalized by Layer Normalization.\nThe authors conjecture that the largest eigenvalues of the value matrices alone governs the final states of the token particles.\nFinally, the problem is connected with the Rényi parking process to show that the particles of causal self-attention Transformer reach metastable clusters under certain conditions.", "questions": "- What are the practical implications of the results?", "limitations": "The authors discuss the limitations of the work (Section 6).\n\n# Suggestions\n\n- The references are not well maintained. Geshkovski et al. 2023a and Geshkovski et al. 2023b are identical, and it is accepted at NeurIPS 2023.\n- $\\textsf{dist}$ in L 109 seems to be defined in L 228.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors extend the analysis of full self-attention Transformers as interacting-particle systems to causal self-attention Transformers.\nAs the current success of Transformers is mainly attributed to the autoregressive ones, the analysis of such models is relevant to the community.\n- They connect the problem with the Rényi parking process and then show that, when the weight matrices are identity in 2D, the particles reach metastable clusters (Theorem 5.1). \n- I appreciate that they discuss limitations using a single independent section.\n- They presented many figures from numerical experiments, which helped me understand the work.", "weaknesses": "- Although the results are interesting, the most exciting parts are conjectures, and the proven results are under strict conditions (e.g., Theorem 4.1 and Theorem 5.1).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback. We are encouraged that they appreciate the relevance of our work's subject: **The authors extend the analysis of full self-attention Transformers as interacting-particle systems to causal self-attention Transformers. As the current success of Transformers is mainly attributed to the autoregressive ones, the analysis of such models is relevant to the community.**\n\nLet us address the weakness **Although the results are interesting, the most exciting parts are conjectures, and the proven results are under strict conditions (e.g., Theorem 4.1 and Theorem 5.1)** by discussing the conditions. Theorem 4.1 is limited to the case $V=I_d$. Firstly, it is already a generalization of the results by [Geshkovski et al. (2023)](https://arxiv.org/abs/2312.10794) and [Criscitiello et al. (2024)](https://arxiv.org/abs/2405.18273) for non-causal models. This is because they utilize a gradient descent approach to convergence, which is limited to the case $QK^T = V$. In this regard, our result shows that matrices $Q$ and $K$ do not affect final convergence. Secondly, the provided proof is more general and could be followed to argue that the conjectured results for $V$ are true. The problem of the appearance of unstable critical curves that arises there is mostly technical, not ideological, but it requires further thorough research. Theorem 5.1 is limited to $V = I_d$ for a reason. There is numerical evidence and an argument in Section 3, mentioned on line 218, that we expect that tokens quickly project to the subspace of a few top eigenvectors. This is why, from that point of view, $V = I_d$ in low dimension is a foundational case and an important part of any system. The biggest obstacle in meta-clustering is its non-asymptotic nature, which makes it hard to study. That is why we assume the simplest $Q=K=V=I_2$ setting for two reasons: this assumption allows us to focus on the hardest part (meta-clustering itself), and it is still an important step for general dynamics because of the heuristic dimension reduction that is yet to be proven.\n\n**What are the practical implications of the results?**\n\nRelevant practical implications are yet to be tested, but there are several promising avenues. Clustering has been observed numerically, but it is poorly understood what mechanisms are underlying its occurrence in transformers. There is an empirical study that suggests that clustering might correspond to transformers learning specific tasks/concepts and assigning them positions in the representation space. Then, being able to control the appearance of these meta-clusters through parameters and predict their appearance might help construct better models for specific tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sXQkg7EM0B", "reviewer": "Reviewer_9pe5", "summary": "This paper strengthens the theoretical results from prior work by presenting causally masked attention used in AIGC.\n\nThe authors prove asymptotic convergence to a single cluster for arbitrary key-query matrices and an identity value matrix under causal self-attention. This significantly extends the results of previous studies.", "questions": "This work largely builds upon the framework of Geshkovski et al. (2023c). Could the author give a more concise statement to emphasize the differences and contributions in this paper?\n\nCould the author provide more practical examples or evidence to prove the superiority of this theory?", "limitations": "This paper has introduces the limitations.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper provide novel insight in understanding causal attention mechanisms, proposing new mathematical models and proof techniques that extend existing knowledge. \n\nBy linking the study to the Rényi parking problem, the authors provide a unique perspective on clustering phenomena in self-attention mechanisms.", "weaknesses": "This paper is really hard to follow, Geshkovski et al. (2023c) is referenced numerous times throughout the article, even including in the abstract. The authors should clearly summarize the previous work, and then emphasize their own contributions building on that foundation.\n\nWhile the paper extends the understanding of causal attention mechanisms, more empirical evidence is needed to validate the results across a wider range of scenarios and applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We are glad that they noted our perspective on the clustering phenomena in transformers **By linking the study to the Rényi parking problem, the authors provide a unique perspective on clustering phenomena in self-attention mechanisms** and the theoretical novelty of the paper **This paper provides novel insights in understanding causal attention mechanisms, proposing new mathematical models and proof techniques that extend existing knowledge.**\n\n**This paper is really hard to follow, Geshkovski et al. (2023) is referenced numerous times throughout the article, even including in the abstract. The authors should clearly summarize the previous work, and then emphasize their own contributions building on that foundation.** This is a great point that we are going to address by providing a supplementary paragraph on how the dynamical system is obtained from the causal transformer architecture for a self-contained introduction to the topic.\n\n**This work largely builds upon the framework of Geshkovski et al. (2023). Could the authors give a more concise statement to emphasize the differences and contributions in this paper?** \nTo summarize the key differences: in Geshkovski et al. (2023), the authors analyze non-causal attention by utilizing its gradient descent structure when $QK^T = V$. The part of their work that is connected to ours is the clustering part, where they prove asymptotic convergence to a single cluster under additional dimension/temperature restrictions. These restrictions were significantly lifted in [Criscitiello et al. (2024)](https://arxiv.org/abs/2405.18273) to $d \\geq 3$ and $\\beta \\leq 1$, but they are still concerned with non-causal dynamics (and utilize gradient descent structure). In our work, we study causal attention in the same framework, but there is no gradient descent structure, which leads to different analysis techniques. In particular, that is why we are able to show asymptotic convergence to a single cluster for arbitrary matrices $Q$, $K$, and $V = I_d$, and conjecture final configurations for different choices of $V$ (ideologically, those conjectures follow from our proof of Theorem 4.1, but there are significant technicalities that are yet to be resolved). Moreover, to the best of our knowledge there is no theoretical approach to meta-clustering, in [Geshkovski et al.] it is only conjectured that the number of meta-clusters should be of order $\\sqrt{\\beta}$ in the 2-dimensional case. In our work, we introduced a unique perspective on the appearance of meta-clusters in causal attention through Rényi parking, which predicts the conjectured $\\sqrt{\\beta}$ clusters, and showed relevant clustering results (mainly Theorem 5.1). \nWe are going to add a similar comparison in the paper.\n\n**While the paper extends the understanding of causal attention mechanisms, more empirical evidence is needed to validate the results across a wider range of scenarios and applications.**\nExtensive experiments with real-life models are very interesting and our theory suggests several insights to test. However, as the paper is already heavily theoretical, we believe that any proper experiments are out of scope of this work and a prospect for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pe8lW6qmKU", "reviewer": "Reviewer_Apon", "summary": "This paper studies the representations or tokens generated by a sequence of causal attention layers. To this end, and following the example of prior works, the authors model such a sequence as a discretization of a system of ODEs. Each token in the input sequence is modeled as a particle and the evolution of each token with depth is modeled as an interacting particle system. Layer normalization is used to ensure each token lies on he unit sphere. A number of theoretical results are derived and conjectures made in this setting (referred to as CSA dynamics).\n- Lemma 1 roughly characterizes rate at which tokens approache the leading eigenspace of the value matrix.\n\n- Theorem 4.1 characterizes a simple setting in which tokens collapse to a single point asymptotically.\n\n- A number of conjectures are made based on the dynamics as well as experiments describing situations in which tokens or particles spread out or collapse onto a few discrete points.\n\n- Meta stable clusters (which persist for a significant time but disappear eventually) are also studied in a simplified 2 dimensional setting and an interesting connection to the Renyi parking problem is made.  \n\nPerhaps the key takeaway is that each token (or particle) is driven by an internal force as well as an external force, which is either attractive or repulsive according to the sign of the largest eigenvalue of the value matrix. This in turn controls the diversity of token representations asymptotically.", "questions": "The main thing I think it would be nice to see more discussion of is how the restriction to causal attention impacts the dynamics: can you highlight any important differences between the dynamics of the tokens of causal versus standard self-attention?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality: not aware of other similar analyses for causal attention, the modified dynamics due to the causal masking also make the analysis challenging.\n\n- Quality and clarity: paper is well written, motivated and clear.\n\n- Significance: there are some interesting takeaways, e.g., the role of the largest eigenvalue in driving particles to be diverse versus collapsing onto a few points as well as the implications for dimensionality reduction.", "weaknesses": "- Many results are either asymptotic in nature, this could limit their relevance for explaining phenomena observed in practice.\n\n- The study of meta clustering is restricted to the 2D setting. \n\n- One might argue that there are a number of other aspects which hinder drawing practical takeaways, for instance weights are tied across different layers, no MLP layers between attention layers etc.\n\n- I don't see anywhere a discussion of the differences in the token dynamics + asymptotics of causal attention versus standard self attention, which would seem a natural and useful thing to include.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewers for their positive feedback. We appreciate that they acknowledged the novelty and significance of our setting and are glad that they enjoyed our exposition **Quality and clarity: the paper is well-written, motivated, and clear**.\n\n**Many results are either asymptotic in nature, this could limit their relevance for explaining phenomena observed in practice.**\nThis is true for final configurations and most of the results, but the meta-clustering section is devoted to the study of a phenomenon that is significant for practice. As mentioned in Section 6, it lacks a complete non-asymptotic proof, but it is a step forward toward better understanding what is happening in intermediate times that are practically relevant (see Fig. 2).\n\n**The study of meta-clustering is restricted to the 2D setting.** Although the restriction to the 2D setting is made, this is an important foundational step for two reasons. First, the interaction force is the same in any dimension, and so we believe that it is possible to lift the $d = 2$ assumption in the future. Secondly, because of the dimension reduction that matrix $V$ introduces, the lower-dimensional case $d = 2$ is actually relevant for the general case too.\n\n**The main thing I think it would be nice to see more discussion of is how the restriction to causal attention impacts the dynamics: can you highlight any important differences between the dynamics of the tokens of causal versus standard self-attention?**\nThank you for this question. We are going to include an overview of key differences and compare our results with what was already known for the non-causal case. To summarize the key differences: in Geshkovski et al. (2023), the authors analyze non-causal attention by utilizing its gradient descent structure when $QK^T = V$. The part of their work that is connected to ours is the clustering part, where they prove asymptotic convergence to a single cluster under additional dimension/temperature restrictions. These restrictions were significantly lifted in [Criscitiello et al. (2024)](https://arxiv.org/abs/2405.18273) to $d \\geq 3$ and $\\beta \\leq 1$, but they are still concerned with non-causal dynamics (and utilize gradient descent structure). In our work, we study causal attention in the same framework, but there is no gradient descent structure, which leads to different analysis techniques and different restrictions. In particular, that is why we are able to show asymptotic convergence to a single cluster for arbitrary matrices $Q$, $K$, and $V = I_d$, and conjecture final configurations for different choices of $V$ (ideologically, those conjectures follow from our proof of Theorem 4.1, but there are significant technicalities that are yet to be resolved). From our understanding, the final configurations in causal and non-causal cases are similar. The key difference is in the more practically relevant meta-clustering phenomenon. To the best of our knowledge, there is no successful theoretical approach to meta-clustering; in [Geshkovski et al.] it is noticed for original non-causal dynamics and it is conjectured that the number of meta-clusters should be of order $\\sqrt{\\beta}$ in the 2-dimensional case. In our work, we introduced a unique perspective on the appearance of meta-clusters in causal attention through Rényi parking, which predicts the conjectured $\\sqrt{\\beta}$ clusters, and showed relevant clustering results (mainly Theorem 5.1). Here, the difference is that in the causal case we are able to predict where meta-clustering is going to occur, while for standard non-causal dynamics it seems to be more complicated and it is yet to be done.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper studies clustering dynamics in causal transformers, following up on previous work that considered non-causal transformers. The reviewers found the theory interesting and insightful, though with unclear connections to practice. Overall, the consensus was positive as the proposed theory seems relevant for the community, and the paper is well written. We therefore recommend acceptance as a poster.
Decision: Accept (poster)