Consensus Is All You Get: The Role of Attention in Transformers
Authors: Álvaro Rodríguez Abella
João Pedro Silvestre
Paulo Tabuada
Venue: ICML 2025 poster (2025)
Keywords: Attention
Transformers
Consensus
Primary area: theory->deep_learning
TL;DR: We provide a rigorous, mathematical analysis of the asymptotic properties of attention in transformers, and also empirical results, showing that under certain assumptions  all tokens asymptotically converge to one cluster.

Abstract: A key component of transformers is the attention mechanism orchestrating how each token influences the propagation of every other token along the layers of a  transformer. In this paper we provide a rigorous, mathematical analysis of the asymptotic properties of attention in transformers. Although we present several results based on different assumptions, all of them point to the same conclusion, all tokens asymptotically converge to each other, a phenomenon that has been empirically reported in the literature. Our findings are carefully compared with existing theoretical results and illustrated by simulations and experimental studies using the GPT-2 model.

Consensus is all you get: the role of attention in transformers
Álvaro Rodríguez. Abella"
João Pedro Silvestre Paulo Tabuada
1
Abstract
plexity, making it challenging to comprehensively explain their behavior (Peng et al., 2024). However, recent studies have shown the emergence of clusters of tokens empiri- cally and theoretically (Dong et al., 2021; Feng et al., 2022; Geshkovski al., 2023b; 2024). These findings suggest that without proper care, large transformers may collapse, a phenomenon where the tokens cluster, limiting the model's ability to produce different outputs.
A key component of transformers is the atten- tion mechanism orchestrating how each token influences the propagation of every other token along the layers of a transformer. In this paper we provide a rigorous, mathematical analysis the asymptotic properties of attention transformers. Although we present several results based on dif- ferent assumptions, all of them point to the same conclusion, all tokens asymptotically converge to each other, a phenomenon that has been em- pirically reported in the literature. Our findings are carefully compared with existing theoretical results and illustrated by simulations and experi- mental studies using the GPT-2 and the GPT-Neo models.
Our work was motivated by the paper (Geshkovski et al., 2023a) where mathematical model for attention was pro- posed, based prior work similar models (Lu et al., 2019; Dutta et al., 2021), and investigated. The authors share the vision outlined in (Geshkovski et al., 2023a), a better understanding of the role and importance of attention mechanisms can be achieved through the study of math- ematical models. Our contribution lies in bringing ideas developed by the control community, where the study of asymptotic properties of dynamical and control systems is central preoccupation, to bear on this problem. While deferring to the next section a more detailed comparison between our results and those available in the literature, we emphasize here that, in contrast with (Geshkovski et al., 2023a;b; 2024), we do not rely on stochastic and/or mean- field techniques and rather adopt a geometric perspective drawing from control theory, e.g., from consensus dynamics manifolds (Sarlette & Sepulchre, 2009) such spheres (Markdahl et al., 2017; Thunberg et al., 2018), and from Input-to-State Stability (Sontag, 1989; 2001; 2008).
1. Introduction
The incorporation of attention (Chorowski et al., 2015) in natural language processing was significant breakthrough, particularly in the context of sequence-to-seqqenee models, enabling the creation of transformers (Vaswani et al., 2017) which revolutionized the field. Even initial transformer models such as GPT (Radford et al., 2018) or Bert (De- vlin et al., 2018) showed drastic improvements over pre- vious approaches such as the Long Short-Term Memory model (Sainath et al., 2015).
As practical applications of deep neural networks, such as image recognition (O' shea & Nash, 2015), natural lan- guage processing (Torfi et al., 2020), and autonomous driving (Grigorescu et al., 2020), continue to advance, our understanding of these networks is struggling to keep pace (Van Dijk et al., 2023). This underscores the criti- cal importance of our study, which aims to delve deeper into transformers and their dynamics. Our understanding of transformers is currently limited by their inherent com-
Contributions of the paper
The main contribution of this work is to provide a number of results', for differential equation model of attention showing that all tokens converge to a single cluster thereby leading to a collapse of the model. We use the term con- sensus equilibria to refer to such clusters as is done in the consensus literature (Ren et al., 2005; Cao et al., 2013). These results hold under different assumptions rameters of the model -nmeely, the query (Q), key (K) and value matrices (U), as well as the number of heads (h)
the pa-
Equal contribution 'Department of Electrical and Computer Engineering, University California, Los Angeles, USA. Corre- spondence to: João Pedro Silvestre <joaosilvestre @g uula..euu>.
'A detailed proof for each theorem stated in this paper can be found in (Rodríguez Abella al., 2024).
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
"Since we focus on the attention mechanism, this model does not describe the effect of feed forward layers.

---

Consensus is all you get
Table Summary the results presented in this work for several particular cases of continuous model (6), where Q(t), (t), and (t) denote the query, key and value matrices, respectively.
FULL ATTENTION
CAUSAL ATTENTION (AUTO-REGRESSIVE
SECTION
$3
$4.1 h 2
$4.2
# OF HEADS
h
TIME VARYING, UNIFORMLY CONTINUOUS, BOUNDED
P(t) = Q(t) K(t)
TIME VARYING, BOUNDED
TIME VARYING, BOUNDED
TIME INVARIANT,
(())
IDENTITY
IDENTITY
SYMMETRIC
RESULT
THEOREM 3.2 CONVERGENCE CONSENSUS
THEOREM 4.2 ASYMPT. STABILITY OF CONSENSUS
THEOREM 4.5
АSYMPT. STABILITY CONSENSUS
STATEMENT
DOMAIN OF ATTRACTION
CONULL (COMPLEMENT OF ZERO MEASURE)
SOME HEMISPHERE
FIXED HEMISPHERE
that are summarized Table 1.
et al., 2024).
In particular, Theorem 3.2 states that tokens converge a consensus equilibrium whenever their starting positions lie in the interior of some hemisphere of the ellipsoid. This result holds for any number of heads and time varying ma- trix P = Q" K provided that U is the identity and P is bounded and uniformly continuous as function of the time. similar result is reported in (Geshkovski et al., 2023a) under Lemma 4.2. However, its conclusions hold under the stronger assumptions that both U and = Q K are the identity matrix and there is single attention head. The- the attention matrix other those induced by the assumptions on P. In contrast, Theorems 4.2 and 4.5 focus on the auto-regressive case, also known as causal attention, where the self-attention ma- trix is lower triangular. Theorem 4.2 states that when U is the identity, the first token fixed and all the other tokens converge the position of the first one for almost every initial position of the tokens. In fact, we have asymptotic stability of this consensus equilibrium. This holds for any number of heads and any time varying P matrix provided it is bounded. Similar conclusions are reported under The- orem 4.1 in (Karagodin et al., 2024) by imposing stronger assumptions: time invariance of Q K and existence of single attention head. Theorem 4.5 extends these result to the where U is time invariant symmetric matrix and the multiplicity of its largest eigenvalue is one. In this case all the tokens will converge to consensus equilibrium (moreover, that equilibrium is asymptotically stable) if they start in one of the two hemispheres defined by the eigenvec- tor associated with the largest eigenvalue of U. We were only able to establish this result for the single-head case although we believe it holds in greater generality. To the best of the author's knowledge there is result available in the literature for the case where U is not the identity matrix although this is conjectured, but not proved, in (Karagodin
Our theoretical findings are validated by simulations of the mathematical model for attention. Moreover, experiments with the GPT-2 and the GPT-Neo models provide empirical evidence for convergence consensus equilibria in more general situations than those captured by our theoretical results, thus providing additional confirmation for model collapse.
Notations
We use the letters n, e, r, and 8 to denote elements of N = (1, The space of r x real matrices is de- noted by (IR) particular, 4, E &Myxr(B8 denotes the identity matrix. The Frobenius norm of square ma- trix A Mrxx r (R) is denoted by || The elements of R=+1 are denoted by x, and tuples of & elements are denoted by x (xi...., xx) (Ra+tye (note the different font), The tangent space of smooth manifold M at E M and its elements are denoted by T,M and X, E T,.M, respec- tively. Given another smooth manifold N and a smooth map M + N, i... C (M, N), ), its tangent map denoted by Té TM + TN.
orem 3.2 makes assumptions
2. Dynamies of transformers
2.1. Configuration space
Let in E N. A symmetric, positive-definite matrix W E M ((+)))((++))) defines an inner product on R=+11
Y2, X WYx,
for each Xx, Yx E T=++1 and x E 1=+ where the super- script T denotes the transpose. The corresponding norm is denoted by [X&lw = (x) W X/) The points of g=+1 of unit norm define an n-dimensional ellipsoid, which
2

---

Consensus is all you get
yi(k + 1)= Tw
(v(()))
Σ
W (k) (y)() exp (y)(k) (a) (n(())))()))ss()
(1)
1=
denoted by:
* E (R) of the self-attention mechanism is given by:
Ei -x fx R=+1
1 x T Wx = 1}.
=(k) ((K))())D(()) exp (<(k)T((()) (((()TT((((((()))))))))
In this work, we consider transformer consisting of tokens of dimension n + constrained to evolve on an ellipsoid. As we have tokens, the resulting state space is the Cartesian product of & copies of the ellipsoid, i.e.:
(4)
where exp(-) denotes the entry-wise exponential (i.e., [exp(R)]is = e R.,, and ((k) E (T) is defined as:
(EW)
=& en X X &W
D(k),s
Vn++1
Σ t=
exp(x((1) K(K) Q(1)xx(1))
É-times
i = j. and D(k),s = 0 otherwise.
Similarly, we consider the following projection:
Practical transformer applications often distribute the com- putations of the self-attention mechanism through several parallel heads, leading to what commonly known multi- headed self-attention. To make explicit the dependence on the head, we write (4) as:
TW: 180t111 - & - nW (x) x W
(2)
where Rn-1_ Ra+1_ {0), whose tangent map at each x E +
Taw:: T2R T x (x) &x)
(k) = (k) x(k) (kk) exp (x(R)T (k) (x))))()))
given by:
for each 1 S 1 S h.
The outputs from all attention heads are added af- ter being multiplied by certain weight matrices W,, (R),
TW Xx = 1xx W (In+1 - xx W ( Xx,,
(3)
S " S l. Then, the result- ing sum is added to the input of the layer x(k), using what is often called a skip connection. Lastly, a nor- malization function is applied to ensure that the output is bounded. In this work, we consider functions that normalize each token of the transformer separately, which is known as layer normalization and was first proposed in (Ba et al., 2016). Hence, the normalization function N M(n+1)xe(R)) - M(n+))xe(B)) is of the form:
for each Xx E T In particular, for y Eij have yy TyTW- Xy y W) -Xy.
Remark 2.1 (Evolution on the sphere). There are a number of models in which the tokens evolve on the n-sphere, i.e., S" ET, 11 For brevity, in that case we will drop the subscripts standing for the matrix W - For instance, we will write |= I 1 Tl.41 etc.
2.2. Discrete-time attention model
== (xi,...,,x)) Xx) + N(x) = (N(x),..., (xe)).
In this section we present the mathematical model for transformer. Similarly to (Geshkovski et al., 2023a), the model encompasses the self-attention mechanism, the skip connection, and the normalization layer, but excludes the feedforward layer.
for some N Rn+1 + 1 Similarly to (Geshkovski et al., 2023a), the following we consider the normalization function N Tw given in (2), which projects each token to the ellipsoid Eq. In practice, this projection has been explicitly used in some models such as (Jiang et al., 2023). For clarity, we utilize the symbol y = (yi,,.... ye) for the tokens evolving on the ellipsoid (after this explicit choice of normalization). The resulting discrete-time dynamical system shown in (1) where < i < L indexes each token. Remark 2.2 (Standard layer normalization). The standard layer normalization utilized in most transformers is given by:
Let w E N be design parameter. The weight matri- ces at the k-th layer of the transformer, E N. are de- noted by Q(k) E (()) (Kk) E Mwxx(x+1)RR. and (k) E MMx(nn1))1BR) and are typically known as the Query. Key, and Value" matrices, respec- tively. The input to the k-th layer is denoted by x (x,,...,,,x) E M(n=I)xe((R) and the output
'In the introduction we used U refer the value matrix; this difference is resolved in this section.
1 xx o(x)
N()
(x (x- x(x)1) + B,
3

---

Consensus is all you get
81 = Ty; Tw TW
Wo(() P,(()a exp exp k2(1) 2n(0))))s)
(5)
for each x = (x',..., 1) E Ra+1, In the previous ex- pression, denotes the vector (I,...,1) E IR++1 * denotes the element-wise product of vectors, and 7, 8 E R=+1 are the learned scale and shift, respectively. Similarly, (xxx) and a (x) denote the mean and standard deviation of x, respec- tively. Under this normalization, we have:
where fx : (E - R=+1 is defined as:
h E
fx(y) Σ ==
(k) V2(K)) D,(k),
exp (y] (k) (k) yz) Nj.
[N(x) - 3
(1tt m(x))² (7t1)
for each y E (EW)). For each 1 S i S E, the best linear approximation in given by:
(x)2
n+1
n + 1
(xt - -xxx)) (7
d y: = dr r=0
Tw + + fa(y)) = Tw Tw - fy.)...
Ep (I# u(x))²
1=1
where we used the notation 7 (71 ,.... , 1. is clear that, if y =o1 for some To 0, then the tokens lie the n-sphere of center 3 and radius (n + 1)g Therefore, the results in this paper also apply to this type of normal- ization. We conjecture that, for arbitrary scale parameters, the tokens will also lie on a certain hypersurface of 1=++1 diffeomorphic to the n-sphere, but a more careful analysis has to be carried out to understand the geometry of such hypersurface
Therefore, the continuous-time model is given by (5), with < i S e, y (yi,...,Yel) .e) E (E and / 2 as the differential equation whose solution provides the best first order approximation of (1).
To simplify notation we introduce the following (time- dependent) auxiliary matrices:
(2(() (t) Vi(t) E M(+1)x(411(BB)) (t((t) (t)(t) (t(t) E Maat)x(x+nn(B))
2.3. Continuous-timm attention model
for each 1 S r S and 2 0. We still refer to the matrix (l) as the value matrix since it plays a similar role. Simi- larly, we define the functions a. 2," E x (EN))² E by:
In this section we introduce additional notation that only used to derive the continuous time model. Readers not interested in the model's derivation can skip the next two paragraphs and start reading the paragraph commencing with "To simplify notation".
o22(4. y,(,.y) exply? P,(O)s)
Let Y E x((E))) )²) be a vector field and denote its flow by Y (EW)) (E Note that Y is complete, i.e., its flow is defined for each T R. since (EW)) is compact. Given map g (En-) ) x R B use the notation g(y,T) T) Oy(r") to denote the existence of constant T E I and a function o (E
(t,t,)) = =Dx(t)) = vn Σ exp(y? P,(t)
B such that, for each T E [o, T] and y E (E ), we have g(y,t) I a(y)+?. A map Φ : (EW)) )' R > ) is first order approximation to the flow Y d(Y" ((y,)) = (r²) where d
respectively, for each ki,jst, y (yI,.... Ve) E
h, 0 and
(E The matrix having a" as its i-th row and j-th column entry is usually called the attention matrix of head 77.
denotes the distance on (En-)* induced by the Euclidean distance (R3+1).
With the notation just introduced, the dynamical system that describes the evolution of a transformer with h heads and & tokens evolving on the ellipsoid Eny is given by:
Using the concepts introduced in the previous paragraph, our objective is to construct vector field Y such that the map defined by the right-hand side of (1) is the best first order approximation of Yr. To that end, we write (k) as (k) = (k)(k) for each S 1) I h, with < < being a small parameter. Hence, (1) may be rewritten as:
É ΣΣ 1=1==
y, TW
(6)
(k + 1) = nw (k)( fk(y())),
isise,
for each 1 i 0 and y = (y1,...,ye) E (8

---

Consensus is all you get
3. Full self-attention matrix
overcome this, we compute the upper Dini derivative instead of the standard one (which is not defined the discontinu- ities) using Danskin's theorem. This gives an upper bound on the change of V along the solution that allows for a yappunov-lke theorem.
In this section we consider a particular case of the model (6) described by the following assumptions.
Assumption 3.1. For each head 1 $ 1 $ h, we have:
Remark 3.3 (Closest result available in the literature). Sim- ilar conclusions appear in (Geshkovski et al., 2023a) (see Lemma 4.2) under the stronger assumptions of single at- tention head and that both U and P = Q' K are the identity matrix.
1. (11()) In+
2. (c(()) is bounded, i.e., sup:20 (1)) < oo, and
3. Pt)(t) is uniformly continuous on oof.
Remark 3.4 (Higher dimensions). Let restrict ourselves to the case where we have normalization to the sphere, i.e., W = In- Wendel's theorem (cf. Eq. (1) of (Wendel, 1962)) gives the probability that & tokens lie on the same hemisphere when distributed uniformly at random; namely:
Recall that a set c (ETy ) is said to be attractive for (6) with domain of attraction H. c (Ei-)) o if the following conditions hold:
1. C is nerward-ivvrian i.e., each solution of (6) starting in C remains in C for all positive times.
R-1
1
Pr, =
2
n=0
2. lim (ys), C)
0 for each solution y R + ( of (6) with y(0) E H. where dist(y(t),C) = inf dly(t), 2) and d denotes the distance on (EW-)* ) induced by the Euclidean distance on (RØ+1).
In particular, Pr.. whenever n 2 E. As a result, if the starting position of the tokens is chosen from a uniformly random distribution and n 2 é, then they will lie on the same hemisphere almost surely. The previous result thus deals with the most general situation for higher dimensions.
The next result claims attractivity of the consensus set pro- vided that the initial position of the tokens is in some open hemisphere of the ellipsoid.
4. Auto-regressive self-attention matrix
This section addresses the auto-regressive (also known as causal) case, that is, the case where the dynamics of each to- ken only depends on itself and the previous tokens. This cor- responds to the model (6) with the so-called auto-regressive self-attention matrix, i.e.:
Theorem 3.2. Let v E E and consider the open hemi- sphere
(t = (y E &W u y 0}.
If Assumption 3.1 holds, then the consensus set CJ (v) in the product of hemispheres H (v)², given by:
(, exply. P,(0),, 12 0,
t (v) = (y (y, (y..... E (EW)) y yE
i<
attractive for (6) with domain of attraction (u)².
(, = vn+1 Σ exply. P,(1)s))
The idea of the proof is to first check that H (v) is forward- invariant and then define a Lyapunov function as follows:
4.1. Identity value matrix
yo V(y) = max Vi(y), V (0) R,
(7)
ISiSE
Let us consider the case where W = In+1, i.e., the tokens evolve on the sphere.
where Vy()) 11 - y.. In other words, V describes how far is each token y, from being aligned with e by represent- ing the largest misalignment, i.e., the worst case over all tokens. We show that V is strictly decreasing along solu- tions of (6) that start in H (u),, except when the solution lies in Ct (v), i.e., when the tokens reach consensus. It fol- lows from the Lyapunov stability theory that, under these conditions, c+ (v) is attractive. We only obtain attractiv- ity instead of asymptotic stability due to the system being time- vareinarying It should be pointed out that V is defined by a maximum and, thus, it may not be differentiable. To
Assumption 4.1. The model (6) is auto-regressive,
W In+1 and, for each head 1 S 1 < h, we have:
1. U, -000)=00 In+1, -I,1))=++..00 and
2. (t) bounded, i.e., sup (PP)(t)] < cc.
It is straightforward that, under the previous conditions, ý1 = 0 and, thus, the first token remains fixed: y1()) = yf for each 2 O. For each other token y.. 2 S i S E, the inner
5

---

Consensus is all you get
product between vf and y;, i.e., the (cosine of the) angle between them, provides a projection of the dynamies onto the real line. In other words, using this angle we construct scalar differential equation governing the evolution of the projection of the token on the real line. It them becomes simple to construct a Lyapunov function for the second token, ensuring convergence to yi for every initial condition except for y2(() -y-. For the remaining tokens, an input-to-state st-tt stability argument coupled with the triangular nature of the dynamics leads to the asymptotic stability of the consensus set for almost all initial conditions.
treating the distance of the previous tokens to the equilib- rium error (as it converges zero within time), an inpu:tto-saae-stbbiiit-yyyy
function for the projected dynamics of y, can be found, yielding the result.
Theorem 4.5. Suppose that Assumption 4.4 holds and dim (U)) 1, where À = max A(U). The elements of LA (U) n S" are denoted by {-v, If X > 0, then y* = (u...., 2) (resp. y = - (D..... ................................ vj) is an asymptoti- cally stable equilibrium of (6) and its domain of attraction contains the set:
('()) - (11....................... ve) (8")² l y1 0, isish,
Theorem 4.2. If Assumption 4.1 holds, then the consensus set:
(resp. D (-u)).
C. (y .y. (s")",,
Remark 4.6 (Closest results available in the literature). The authors were not able to find results in the literature address- ing the case where U is not the identity matrix although two conjectures are proposed, but not proved, in (Karagodin et al., 2024).
is asymptotically stable for the system (6) and the domain of attraction contains the following set:
D = (yy......y.)) ye) E (f ly, # -1. 2 s..
Remark 4.3 (Closest result available in the literature). Similar conclusions are reported under Theorem 4.1 in (Karagodin et al., 2024) by imposing stronger assump- tions, time invariance of P = Q K and existence of a single attention head, although the authors state that time- invariance is not explicitly used.
Remark 4.7 (Time-varying value matrix). In this remark we discuss a possible relaxation of Assumption 4.4 to a time-varying matrix (t). Once (t)) becomes time- varying, so will X(), v()), and the consensus equilibrium y (t) (u(t),... ..... u(t)). If v(t) varies slowly, we expect convergence to ball centered at (t). The scalar differen- tial equation, upon which the proof of Theorem 4.5 is built, becomes:
4.2. Symmetric value matrix
Now we extend the results of the previous section more general value matrices. As above, the tokens evolve the sphere, i.c., W = I1x+1-
= a - U(0))++ "
12 0.
(8)
Let us denote the extra term as e = oT y. When e = 0, an asymptotically stable equilibrium with domain
b' = 1 of attraction Jo, 1] (resp. 1" -1 with domain [ Hence, an input-to-state stability argument would show con- vergence to ball centered at b* = 1 (resp. b" - 11) whose radius is an increasing function of suPr20 le(()).
Assumption 4.4. The model (6) is auto-regressive,
W In+,, and we have:
There is only one head, i.e., / = 1,
(1) = U with UT= = U. and
5. Simulations and empirical validation
(t) = P(t) is bounded, i.c., sup1zo I [(()]] < cc.
In this section we illustrate the theoretical results and show that their conclusions appear to hold even when our assump- tions are violated. We start by simulating the continuous transformer model and illustrating our theoretical results. In addition to simulations, we provide empirical evidence using the GPT-2 XL and the GPT-Neo 2.7B to show how token consensus seems to occur even if the assumptions in our theoretical results are not satisfied.
We denote the spectrum U by A(U). Note that A(U) TR as U is symmetric. Given À E X(U), the corresponding eigenspace is denoted by (U) C R=+1
Unlike the case U- = In-1 considered in the previous sec- tion, the first token is no longer fixed. However, it can be shown that it converges to a fixed position provided the multiplicity of the largest eigenvalue is one. This allows for establishing the asymptotic stability of two specific consen- sus points induced by the matrix U using the same technique as in Theorem 4.2. Namely, the dynamics of each other to- ken y., 2 < < E, is projected to the real line using its inner product with the corresponding asymptotically stable equilibrium of yr. By following an induction argument and
5.1. Numerical simulations
5. 1.1.1. ILLUSTRATTIO OF THEOREM 3.2.
We simulate the motion of 10 tokens, each of them randomly placed on the sphere $ C R3, according to the dynam- ics (6) with h = 2. All matrices, except for Pi (t)) and Pz
6

---

Consensus is all you get
04
0.1
Figure 1. Frobenius norm of the matrices (t( and Pa(t).
Figure 3. Evolution of the Lyapunov function (7) used in the proof of Theorem 3.2.
were randomly chosen, and each element was drawn from uniform distribution in the interval [-0.5, 0.5]. The ma- trices (t) and (t) were computed (t) Di (t)
the tokens start and remain in an hemisphere and that they converge to a consensus equilibrium.
P2(t) P2 P/2 with P| and Pá randomly generated:
The proof of Theorem 3.2 is based on the Lyapunov func- tion (7), whose time-evolution displayed in Figure 3 for the case where (1, 0,...
0.08
0.19
0.20
P-
0..3
0.31
-0.23
0.18
-0.17
-0.16
0.31
0.03
0.11
5.1.2. I.LUSTRATION OF THEOREM 4.2
P2
0.06
0..6
0.13
We now consider the auto-regressive model with 50 tokens on $499 C 50000 The number and dimension of the to- kens were chosen to make them comparable to the GPT- model. We use two heads (h 2) with the matrices = (t) and P3 = (1) P3 obtained by randomly generating PL and Py and taking Di (t) and (2 to be diagonal with entries (D,);s
0.14
0.11
0.10
The matrices (i( and D2(1) were given by (i( = 2 diag cos((1rt), sin(10rt), cos(6rt)) and D2(1) 2 diag cos((6Tt), sin(Grt), cos cos(4)))), where diag: R (I))(RR) denotes the function that maps a vector to the diagonal matrix with its components on the diagonal.
sin us 6) for 77 1,2, = 1=....,,500 500, w drawn from the uniform distribution on J0, 1[ and drawn from the uniform distribution on j0, 2 To measure the error between tokens we use the cosine similarity, E (EW + R defined as:
To better appreciate the time-varying nature the matrices Pi and P2. Figure we shown their Frobenius norm.
In Figure 2 we show the motion of the tokens in blue with their initial position represented by white circle and fi- nal position by gray circle. We can appreciate that all
Yi
E
Iya
which becomes zero when all the tokens belong the con- sensus set.
In Figure 4 we display the evolution of the function along 100 trajectories of (6) for random initial conditions drawn from an element-wise uniform distribution on 0.5, 0.5[. and then projected the sphere. We can appreciate in Figure 4 how the function F. converges trajectories.
along all the
1.1.3. ILLUSTRATION OF THEOREM 4.5
In the final case we use the auto-regressive model with 10 tokens on $ with randomly assigned initial positions. As for the previous cases, we choose P(t) = D(t) with
Figure Convergence consensus cquilibrium the sphere S², All the tokens start and remain in an hemisphere.
7

---

Consensus is all you get
assumptions. Since our results asymptotic, we need increase the depth of both models. We do by running the same set of tokens through the model multiple times. In other words, we extract the tokens at the end of the model, after the final normalization, and feed them to the model for another pass thereby simulating a model of increased length, the code used for our experiments available at (Silvestre, 2025).
677
0.6
0.5
0.4
For all our experiments, used the same set of 100 random prompts, each generated by uniformly sampling 200 tokens from the GPT-2 tokenizer's vocabulary. In each experiment, we plot the average of E across all the prompts. As an example, the first 10 tokens of the first sampled prompt are:
0.1
100
Figure 4. Illustration of Theorem 4.2; evolution of the function defined (9) along 100 solutions (6) with random initial conditions drawn from element wise uniform distribution | 0.5, 0.5 and then projected to the sphere.
divest anxYou coasts Oz Vi Happy appreciate tcp
In the first experiments, we removed the feedfoward layers of each model, make them closer to the structure we assume in our theoretical work. The experiments were then repeated without removing the feedforward layers, showing that in both cases convergence to consensus occurs.
randomly generated Pr and U given by:
/0.36
0.42
0.13
0.26
0.50
0.56
P'
0.10
0.07
0020
U
0.50
0.72
-0.50
0.115
-0.21
0.12
0.56
-050
-0.02/
The experiments were conducted on the standard configura- tion of the GPT-2 XL model and the GPT-Neo 2.7B model, using the pre-trained weights provided by the Hugging Face library (Wolf et al., 2020). The multiple passes through the model results in matrices P and U that are time-varying but periodic with period corresponding to the depth each model: 48 layers for the GPT-2 XL and 32 for the GPT-Neo 2.7B. To measure how far the tokens are from each other we used the function (9) whose evaluation after each layer is depicted in Figures 7 and 8.
and (tt) diag ( (cos(10rt), sin(10rt), c6s(t)))).
In Figure 5 we can observe convergence of the tokens consensus equilibrium point whereas in Figure 6 we have the time evolution of V = 1 1-yll I and V22 = 1-y2 u where V E R is the eigenvector of U corresponding to its largest eigenvalue. Note that V/2 is not a Lyapunov function, and therefore may increase, although the proof of Theorem 4.5, establishes that it will eventually converge to zero.
5.2. GPT-2 and GPT-Neo Experiments
We can observe that in both models the average of the func- tion E over all the prompts converge asymptotically to 0, thus implying the tokens converge to a consensus equilib- rium. We recall that our theoretical results predict this
In this section we report on experiments conducted on the GPT-2 XL model and the GPT-Neo 2.7B model suggest- ing that our theoretical findings hold under more general
37
15
20
Figure 5. Convergence consensus cquilibrium on the sphere S², All the tokens start and remain in the hemisphere defined by v.
Figure 6. Evolution of the functions Vi and V/2 used in the proof of Theorem 4.5.
8

---

Consensus is all you get
observation only when feedforward layers are absent, i.e., be seen
by removing the feedforward layers (including the associ- ated normalization function and skip connection) to better understand the impact of these token consensus. The results are reported in Figures and 10, where we can see that convergence towards consensus still occurs across all experiments. Furthermore, Figures 9 and 10 suggest that the feedfoward layer may decrease the convergence rate.
the depicted in Figure 7. However, in Figure 8, even when the feedfoward layers are present, convergence still occurs. The rate of convergence appears to be dependent on the weights of the feedfoward layers as their presence increases the convergence rate in the GPT-2 model, but decreases in GPT-Neo model. In both cases, these findings suggest that feedfoward layers may not be sufficient to preclude consensus.
Our experiments suggest that the convergence phenomenon is a product of the structure of the transformers and not of the choice of weights. We observe convergence with trained, random periodic, and random aperiodic matrices P and U. In terms of rate of convergence, the choice of weight matrices appears to have an impact with faster convergence being observed when pretrained matrices were used.
Although the previous experimental results suggest that con- sensus occurs even in the presence of feedfoward layers, it does not address the question of consensus being a structural property of the the transformer architecture or of the choice of weight matrices. To address this question we repeated the experiments by using random matrices in GPT-2 and GPT-Neo. Since this results in time-varying matrices, we further repeated the experiments by randomly selecting new weight matrices before each model pass. Moreover, we conducted these experiments with the full model and also
0.5
60
422
1500
2500
Soner Teae
0.5
0.2
1000
1730
1430
15.00
15.00
2000
0.
Figure 9, Comparison between the GPT-2 XL and GPT-Neo 2.7B architectures with fixed and randomly chosen weight matrices. Each model evaluated with and without feedfoward layers using the average of (9) across all the random prompts.
000
Layer
Figure 7. Comparison between the GPT-2 XL and GPT-Nco 2.7B architectures with feedforward layers removed; evaluation of the average of (9) across all the random prompts.
0.5
02-
1200
1000
fum athenoe)
0.5
0.8
Lepe
kayer
Figure 10. Comparison between the GPT-2 XL. and GPT-Neo 2.7B architectures with random weight matrices chosen before each model pass. Each model was evaluated with and without feed- foward layers using the average of (9) across all the random prompts.
Figure 8. Comparison between the GPT-2 XL and GPT-Nco 2.7B architectures with the full model; evaluation of the average of (9) across all the random prompts.

---

Consensus is all you get
Acknowledgments
Geshkovski, B., Koubbi, H., Polyanskiy, Y., and Rigollet, P. Dynamic metastability in the self-attention model. arXiv 2024.
This work was partially supported by the NSF award 2211146. The work of João Pedro Silvestre was partially supported by the PhD fellowship 2023.01843.BD from the Fundação para Ciência Tecnologia (FCT), Portugal.
preprint arXiv:2410.0000
Grigorescu, S., Trasnea, B., Cocias, T., and Macesanu, G. A survey of deep learning techniques for autonomous driving. Journal of field robotics, 37(3):362-386, 2020.
Impact Statement
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F,, Lengyel, G., Lample, G., Saulnier, L.,, et al. Mistral 7b, arXiv preprint arXiv:2310.06825, 2023.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Karagodin, N., Polyanskiy, Y., and Rigollet, P. Clus- tering in causal attention masking. arXiv preprint 2024.
References
Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. 2016.
Lu, Y., Li, Z.,, He, D., Sun, Z., Dong, B., Qin, T., Wang, L., and Liu, T.-Y. Understanding and improving transformer from multi-partichl dynamic system point of view. arXiv preprint arXiv: 02762, 2019.
arXiv preprint
Cao, Y., Yu, W., Ren, W. and Chen, G. An overview of recent progress in the study of distributed multi-agen agent ordination. IEEE Transactions 9(1):427-438, 2013. doi: (0.1109/11.21122222
Industrial Informatics,
Markdahl, J., Thunberg, J., and Gonçalves, J. Almost global consensus on the n-Sphere. IEEE Transactions on Auto- matic Control, 63(6): 1664)11664-6
Chorowski. J. K., Bahdanau, D., Serdyuk, D., Cho, K., and Bengio, Y. Attention- eitention-based models for speech recogni- tion. Advances in neural information processing systems, 28, 2015.
O'shea, K. and Nash, R. An introduction convolutional neural networks. arXiv preprint arXiv:1511.088 2015,
Peng, B., Narayanan, S., and Papadimitriou, C. On lim- itations of the transformer architecture. arXiv preprint arXiv:2402.08164 2024.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. arXiv preprint arX.v:1810.04805, 2018.
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. OpenAl, 2018.
Dong, Y., Cordonnier, J.-B., and Loukas, Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pp. 7793-2803, PMLR, 2021.
Ren, W., Beard, R., and Atkins, E. A survey of consensus problems in multi-agent coordination. In Proceedings of the 2005, American Control Conference, 2005., pp. 1859 1864 vol. 3, 2005. doi: 10. 10.1109/ACC.220051477033
Dutta, S., Gautam, T., Chakrabarti, S., and Chakraborty, T. Redesigning the transformer architecture with insights from purticp dynamical systems. arXiv preprint arXiv:2109.15144 2021.
Rodríguez Abella, Á., Silvestre, J. P., and Tabuada, P. The asymptotic behavior of attention transformers. arXiv preprint arXiv:2412.02662. 2024.
Feng, R., Zheng, K., Huang, Y., Zhao, D., Jordan, M., and Zha, Z.-J. Rank diminishing in deep neural networks. Advances in Neural Information Processing Systems, 35: 33054-33065, 2022.
Sainath, T. Vinyals, O., Senior, A., and Sak, H. Convo- lutional, long short-term memory, fully connected deep neural networks. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4580-4584, Icee, 2015.
Geshkovski, B., Letrouit, C., Polyanskiy, Y., and Rigollet, P. A mathematical perspective on transformers. arXiv preprint XI1:22312.10794 2023a.
Sarlette, A. and Sepulchre, R. Consensus optimization manifolds. SIAM journal on Control and Optimization, 48(1):56-76, 2009.
Geshkovski, B., Letrouit, C., Polyanskiy, Y., and Rigollet, P. The emergence of clusters in self-attention dynamics. arXiv preprint arXiv:2305. 05465, 2023b.
Silvestre, P. GPT-Consensus, https://github: com/cyppylabb/ceo wttpaa////cccCcccuoosooo 2025.
10

---

Consensus is all you get
Sontag, E. Smooth stabilization implies coprime factoriza- tion. IEEE Transactions Automatic Control, 34(4): 435 443, 1989. doi: 10.110999.28018.
Sontag, E. D. The ISS philosophy as unifying frame- work for stability-like behavior. In Isidori, A., Lamnabhi- Lagarrigue, F., and Respondek, W. (eds.), Nonlinear con- trol in the year 2000 volume 2, pp. 443-467, London, 2001. Springer London. ISBN 11881-84628-569-1
Sontag, E. D. Input to State Stability: Basic Concepts and Results, pp. 163-220. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008. ISBN 678-3-540-77653-6 doi: 10. 7211007997883-544--776666- URL httpp:://// //doi. /doi. 540 - 77 77653-66 6_3.
Thunberg, Markdahl, J., Bernard, F., and Goncalves, J. A lifting method for analyzing distributed synchronization on the unit sphere. Automatica, 96:253-258, 2018
Torfi, A., Shirvani, R. A., Keneshloo, Y., Tavaf, N., and Fox, E. Natural language processing advancements by deep learning: A survey. arXiv preprint arXiv:2003.01200, 2020.
Van Dijk, B., Kouwenhoven, Spruit, M. R., and van Duijn, M. Large language models: The need for nu- ance in current debates and pragmatic perspective understanding. arXiv preprint urXiv:2310. 19671,, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Wendel, J. G. problem in geometric probability. Mathe- matica Scandinavica, 11(1):109-111, 1962. doi: https: //doi.org/10.1 146/mtth..candd 10655.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 conference em- pirical methods in natural language processing: system demonstrations, pp. 38-45 45, 2020.


Reviews:
[{"review_id": "denvUL7zo1", "reviewer": "Reviewer_aNPn", "summary": "This paper theoretically investigates the phenomenon of token representation collapse in transformers as the number of layers blow up. The authors analyze a continuous-time differential equation of the attention model and show that all tokens converge asymptotically. They present results under different assumptions on the key, query, and value matrices, improving upon previous work that required stricter conditions on the composed key-query matrix, the value matrix, and the number of attention heads. Finally, they provide small-scale experiments to validate their theory.\n\n**Update after rebuttal**\n\nI thank the authors for engaging during the rebuttal period. They addressed most of my key questions and agreed to make appropriate additions to the paper including some additional experiments. Overall, I believe that it is a good paper that improves upon the previous theory on representation collapse, and I keep my positive rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time devoted to reading our paper and providing feedback. \n\nWe were not aware of the paper Wu et al. (2024). Upon reading it we found that it confirms the results in our paper since it provides several scenarios of rank degeneration, i.e., consensus. It also provides an example where consensus does not occur but this example requires the query and key matrices to be constant and equal to zero which does not happen in practice.\n\nWe will be happy to add a conclusion/discussion section as suggested.\n\n**Layer normalization** \n\nThank you for this question, we should have included a discussion about this in the paper. At an intuitive level, any normalization technique has the objective of restricting the tokens to a compact space, hence the results should not depend on how the normalization is done. At the technical level, we can show that layer normalization (removing the mean, dividing by standard deviation, multiplying by a learned scale, and adding a shift) corresponds to projecting on a suitably translated sphere when the learned scale parameters are all equal and non-zero. Hence, our results apply to this case. We suspect the same holds for arbitrary scale parameters but we need to perform a more careful analysis to ensure the topology/geometry of the resulting compact space is homeomorphic/diffeomorphic to that of a sphere.\n\n**Matrix U time-varying**\n\nOur proof technique is based on projecting the dynamics along the eigenvector of U corresponding to the largest eigenvalue to obtain a scalar differential equation that is easier to analyze. When U is time varying, the eigenvectors of U are time varying. This has two consequences: 1) the consensus point is now time-varying; 2) the scalar ODE has one additional term coming from the time derivative of the eigenvector (this derivative is zero in the time-invariant case). Provided the eigenvector changes slowly, all the tokens will asymptotically converge to the time-varying eigenvector and consensus is still reached. We are happy to discuss this extension in the final version if the reviewer finds it useful.\n\n**Theorem 3.2, tokens starting outside an hemisphere** \n\nThe conclusions of Theorem 3.2 still hold when the tokens don't start in an hemisphere but enter an hemisphere at some future time. We proved that hemispheres are invariant (once the tokens enter one, they cannot leave) and thus the conclusion follows. In our experiments we always observed consensus, independently of the initial condition.\n\n**Experiments**\n\nDefining the function E in the appendix was an oversight that we will correct. We will also be happy to average the results and provide confidence intervals in the final version.\n\n**Collapse is less prominent for random weights/Removing periodicity seems to reduce collapse.**\n  \nThis is an interesting question. Our educated guess is that convergence is exponential when the matrices are constant (this is supported, under additional assumptions, by Theorem 6.3 in version 4 of https://arxiv.org/abs/2312.10794). One may speculate that the rate of convergence is related to the time derivative of the matrices. Constant matrices, zero derivative, faster convergence, random matrices, highest derivative, slowest convergence. The periodic case seems to lie in between the constant and random cases but we do not have a good explanation for this.", "claims_and_evidence": "All claims and evidence seem to be well-supported. The key contribution—demonstrating that the continuous-time differential equation model collapses asymptotically under different sets of assumptions on the model parameters—is backed by theoretical results. While limited, the toy experiments provided support for these claims.", "methods_and_evaluation_criteria": "It’s mainly a theory paper. I don’t think this question is very valid, but whatever toy experiments they have make sense.", "theoretical_claims": "No, I did not read the proofs of the theorems.", "experimental_designs_or_analyses": "Since this is a theory paper, the experiments are primarily toy examples designed to validate the theory. They involve prompting a pre-trained GPT-2 XL and analyzing representation collapse, which the results confirm and align with previous findings. From this perspective, the experiments appear sound, though I have some questions and suggestions (see the questions section).", "relation_to_broader_scientific_literature": "The paper theoretically examines representation collapse in tokens as the number of layers in transformers increases. While previous work has studied this using mean-field techniques, this paper appears to apply ideas from control theory to analyze the asymptotic behavior. Additionally, the theoretical insights developed here could potentially be used in the future to identify architectural components that contribute to representation collapse.", "essential_references_not_discussed": "I believe the recent work by Wu et al. (2024) is highly relevant and missing from the discussion. They also investigate representation collapse, focusing on the role of layer normalization and attention masks. While their approach differs, it should be addressed in this paper, along with a comparison of their theoretical results. For instance, Wu et al.'s results appear to be finite-time, whereas this paper's findings are asymptotic. A comparison of their respective assumptions would also be valuable.\n\n\nWu et al. (2024) On the role of attention masks and layer norm in transformers.", "other_strengths_and_weaknesses": "The paper studies an important problem from a theoretical point of view, well-written in general, and interesting to read.", "other_comments_or_suggestions": "Please add a small conclusion/discussion section with open questions and drawbacks of the results presented in the paper.", "questions_for_authors": "1. The theory is developed for the ellipsoid projection which is playing the role of layer-normalization in a standard transformer. Do you think it's possible to extend these results to the standard layer-norm? \n\n\n2. General question for all results: the value-matrix $U$ is time-independent (which also appears to be the same in previous works). I understand the assumptions are less strict from previous works, but it seems a little strange. Do you believe the results still hold if $U$ is time-dependent, similar to the composed key-query matrix $P$? \n\n\n3.  A few questions on the assumptions: For Thm 3.2, what happens if the initial position of the tokens do not like in some hemisphere? Related to this, for the experiment in figure 2, do you still see consensus if the tokens do not start in a hemisphere? \n\n\n4. Re the experiments: The evaluation metric used in all plots should be discussed in the main body, it is just referred to as some equation in the appendix. I like the toy experiments in general, but I think the results should be averaged over multiple prompts (for example figure 5 with random prompts), and an average with some confidence intervals should be reported. This is especially important given the variance we see for different prompts in figure 7.  \n\n\n5a.  Why do you think collapse is less prominent for random weights in almost all the figures? The theory does not seem to distinguish between different sets of weights.\n\n\n5b. Comparing Figure 6 to Figure 5, removing periodicity seems to reduce collapse. Why do you think this happens? Again, the theory does not appear to separate these cases.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TX1DD8lREy", "reviewer": "Reviewer_Ek83", "summary": "This paper try to theoretically analyze the phenomenon that using auto-regressive attention model, all tokens will asymptotically converge to the \"consensus set\". Authors construct discrete/continues-time attention model and show that under full/auto-regressive attention cases, tokens will converge to some special cases. Authors also support their claim with some experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time devoted to reading our paper and providing feedback. \n\nWe were not aware of the referenced papers. All but the second paper discuss training dynamics whereas we study already trained transformers. Hence, they do not seem relevant. The second paper shows that the evolution of tokens along a transformer can be interpreted as an instantiation of gradient descent. We cannot see any direct connection to our work but we will be studying this paper out of intellectual curiosity.\n\nThe reviewer states that the contribution is \"somewhat too incremental\" and supports this opinion with \"similar claims have been shown in previous paper\" and \"still requires some strong condition (...) like U still need to be identity\". \nThe latter statement is factually incorrect. Section 4.2 discusses the case where the matrix U is not the identity. The reviewer can find in Assumption 4.4 that we only require U to be symmetric and time-invariant. See also our reply to reviewer aNPn where we explain that the time-invariance assumption can be relaxed.\n\nWe would also like to politely disagree with the incremental characterization of our work and with \"similar claims have been shown in previous paper\". Theorems are written as implications, say a implies b, where a is called the antecedent and b the consequent. Two theorems are not similar because they have the same consequent. The strength of two theorems offering the same consequent is measured by how weak the antecedent is. A weaker antecedent means the implication applies to a larger class of systems and is thus a stronger result. Here is one example, before people could travel by aircraft, the US could be reached from Europe by boat in several weeks. Once air travel began, the US could be reached from Europe in several hours. In both cases the consequent is the same, the US can be reached from Europe, but these scenarios cannot be characterized as being \"similar\".\n\nThe reviewer writes \"The authors haven't detailly compared the techniques between them and previous literature to show why they can achieve relaxed conditions.\" It is not clear to the authors what is meant by a detailed comparison of techniques. An intuitive comparison is provided in the 3rd paragraph of the introduction. Intuitively speaking, some prior work used a mean-field model that resulted in a partial differential equation whose solution was interpreted as a distribution. We used ordinary differential equations that enabled us to use control theoretic techniques such as Input-to-State Stability as well as existing results on consensus on spheres. Existing work that also used ordinary differential equations did not use control theoretic techniques. We would be happy to expand this intuitive comparison in the final version of the paper and, in particular, highlight which control theoretic techniques were used in each proof, if this is what the reviewer has in mind.\n\nWe are more than happy to change the paper's title.\n\nWe agree with the criticism related to the need to use multiple prompts. We will provide such experiments in the final version.\n\nThe matrix V_\\eta is defined in the second paragraph of Section 2.2 as the value matrix. Please let us know if additional explanations regarding this matrix are needed.\n\n**We could not understand the remark «Seems all the \"[0, \\infty]\" are \"[0, \\infty[\"».** \n\nWe should have made this clear in the paper. We follow Bourbaki's notation (introduced in Éléments de mathématique) and write [a,b[ to denote the interval including the point a and excluding the point b. This is done so that there is no confusion between the ordered pair (a,b) and the interval ]a,b[ that excludes the points a and b. We did find a few instances of sets in the appendix using the notation (a,b) and we will correct them.\n\nNot including the definition of E in the main part of the paper was an oversight of our part. It will be rectified.\n\nNot defining attractivity was an oversight, it will be corrected.\n\nThe reviewer asks about the \"Key difference in technique\". This is the first paper that uses control theoretic techniques such as Input-to-State Stability or draws inspiration from results available in the control literature on consensus on spheres. As we mention in the reply to the last reviewer, we propose to comment on which control theoretic techniques are used in each proof so as to provide deeper insights into our approach. We hope this change addresses the concerns of both reviewers.", "claims_and_evidence": "I think the claims are supported by both previous literature and the toy experiments in Section 5", "methods_and_evaluation_criteria": "there is no criteria for the experimental parts in this paper. For the proposed theoretical model, I think in general it makes sense for the problem and may follow some framework from previous theoretical paper", "theoretical_claims": "I haven't checked the proof of main theorem in detail, but I haven't seen critical problems in the claim and lemmas (like App. B, or Lemma C.2) in appendix.", "experimental_designs_or_analyses": "I check the validity of experimental designs. I think they are toy experiments and it's acceptable for a theory paper. But in GPT2-XL experiment, authors should not just try one well-designed prompt, but need to try more diverse prompts randomly sampled from natural language dataset and see some average performance.", "relation_to_broader_scientific_literature": "I think the main contributions of the paper is to prove the \"consesus convergence\" phenomenon based on more relaxed conditions (like no requiring Q^TK to be identity). Previous work like Karagodin et al. 2024 (L230) or Geshkovski et al. 2023a (L172) have shown similar conclusion, but require stronger conditions (like  P = Q^TK need to be time invariant or identity). It's nice that authors clearly discuss their contribution in the paper.", "essential_references_not_discussed": "I think the main techniques of this paper is about analyzing the training dynamics of transformer and see the property of tokens, but there are a lot of transformer-dynamics related works recently that may be worth to be cited/discussed, although they may have different kinds of modeling [1,2,3,4,5].\n\n[1] Nichani E, Damian A, Lee J D. How transformers learn causal structure with gradient descent[J]. arXiv preprint arXiv:2402.14735, 2024.\n\n[2] Cheng X, Chen Y, Sra S. Transformers implement functional gradient descent to learn non-linear functions in context[J]. arXiv preprint arXiv:2312.06528, 2023.\n\n[3] Huang Y, Cheng Y, Liang Y. In-context convergence of transformers[J]. arXiv preprint arXiv:2310.05249, 2023.\n\n[4] Tian Y, Wang Y, Chen B, et al. Scan and snap: Understanding training dynamics and token composition in 1-layer transformer[J]. Advances in neural information processing systems, 2023, 36: 71911-71947.\n\n[5] Li Y, Li Y, Risteski A. How do transformers learn topic structure: Towards a mechanistic understanding[C]//International Conference on Machine Learning. PMLR, 2023: 19689-19729.", "other_strengths_and_weaknesses": "Strengths:\n\n+: the paper is in general easy to follow\n\n+: toy experiments are interesting and support the theoretical findings.\n\nWeakness:\n\n-: I think one key problem is that the contribution is somewhat too incremental. As shown in \"Relation To Broader Scientific Literature*\" part, because similar claims have been shown in previous paper, although with kindly stronger condition. However, this paper still requires some strong condition that is far aways from real-world cases (like U still need to be identity, but this is related to value matrix, and but in real-world we indeed need to update the value matrix). And there seems no additional new theoretical findings in this paper (for example, convergence rate for the consenus phenomenon). \n\n-: The authors haven't detailly compare the techniques between them and previous literature to show why they can achieve relaxed conditions. These can be down by adding some discussions or proof scratch\n\n-: Title seems confusing to me, as consensus is an asymptotically phenomenon that in practice don't happen a lot for SOTA LLM, it can not be 'all you get', and the role of attention can be much more complex (as previous works mentioned in additional reference above)\n\n-: in GPT2-XL experiment, authors should not just try one well-designed prompt, but need to try more diverse prompts randomly sampled from natural language dataset and see some average performance.", "other_comments_or_suggestions": "1. I think if author don't explain V_\\eta(t) in the main paper in detail, (5) better not use it (like using V_\\eta(t))  (L166)\n2. Seems all the \"[0, \\infty]\" are \"[0, \\infty[\"\n3. I think it's better to include the metrics (like the definition of 'E' in Figure3/5, i.e., (18)) \n4. I think it's necessary to define 'attractive' or 'attractivity' in Theorem 3.2 in the main paper.", "questions_for_authors": "What's the key difference in technique between your work and previous works such that you only need relaxed conditions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2v1nbBlibf", "reviewer": "Reviewer_tPgY", "summary": "This paper theoretically demonstrates that a large language model using Transformers may collapse. The model collapse is analyzed through a mathematical analysis of the asymptotic properties of attention in Transformers. The authors claim that all tokens asymptotically converge to each other. This claim is supported by a simulation experiment, where GPT-2 XL is used iteratively by feeding its output back as the input.\n\n\n## Update after rebuttal\nThank you for the clarification. I revised the final score to 3. This reviewer encourages the authors to include experimental support using larger language models to back up the theory, and believes acceptance should be considered based on the to-be-conducted experiments and some writing clarifications (including the reviewers’ suggestions and the authors’ own suggested to-do list) in the next revision.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reading our paper and providing feedback. \n\nWhile there is no section titled \"related work\", after each theorem there is a remark titled \"closest result available in the literature\" where we provide a detailed comparison with the most relevant work. We would be happy to consider any paper the reviewer finds relevant.\n\nThe reviewer wrote \"The theoretical approaches in this paper give a sense of direction, but the exact goal is not clearly presented. From my perspective, the theories are limited by the provided assumptions and seem to hold only under specific conditions. As a result, this reviewer feels that the experimental support is limited and not clearly aligned with the theory.\" \n\nIn the subsection \"contributions of the paper\" we wrote \"The main contribution of this work is to provide a number of results (...) showing that all tokens converge to a single cluster thereby leading to a collapse of the model.\" This is the goal of the paper. We are happy to expand the goal's description.\n\nWe interpret the second sentence as: the stated assumptions are too strong. Could the reviewer tell us which assumptions are too strong so that we can address them? Please see the reply to reviewer aNPn where we explain how specific assumptions can be relaxed.\n\nAll the experiments support the conclusions of the theorems, i.e., they show evidence of consensus. The experiments either satisfy the theory's assumptions or do not. When the assumptions are satisfied, there is perfect agreement with the theory. When they are not, the experiments show that consensus still holds. This does not mean the results are wrong, it means that consensus can be proved under weaker assumptions\n\n Having the definition of E in the appendix was an oversight that will be corrected.\n\nThe reviewer wrote \"The experiment is designed as a simulation (...) However, this setup does not accurately simulate the behavior of a large language model. It is unclear why the authors did not use larger models (...)\"\n\nThe experiments in Figure 3, and Figures 5-7 were performed on the GP2-XL model and are not simulations. Since this is a theoretical paper, experiments do not require large models as we do not seek to compare performance metrics. The experiments serve two purposes: 1) illustrate the theorems proved in the paper; 2) show that conclusions hold under weaker assumptions.\n\nThe reviewer wrote \"This reviewer speculates that the outdated GPT-2 XL, trained on a limited corpus with fixed token lengths, may easily generate meaningless tokens when iteratively fed its own outputs. This may not accurately reflect the model collapse described in the theory.\"\n\nThe experiments do not show GPT-2 XL providing meaningless tokens, they show that all the tokens converge to the same token as predicted by the theory. The function E is only zero when all the tokens are the same. In Figures 3, 5, 6, 7, and 8 we see the function E converging to zero which indicates that all the tokens converge to consensus, i.e., perfect agreement between experiments and theory. \n\nThe reviewer wrote \"This reviewer believes this paper can be improved by presenting its motivation and goals better. Additionally, scaling up and modernizing the experiments instead of just using GPT2-XL would support the claims effectively (...) If the authors address the concerns, I am willing to raise my score.\"\n\nWe will revise the paper to provide more intuitive explanations for the theoretical results and better describe the objectives. We hope to have convinced the reviewer that: 1) theoretical claims are proved theoretically, not experimentally; 2) there is perfect alignment of theory with experiments. We will also provide more experiments to address the concerns raised by multiple reviewers and would be delighted by a raised score.\n\n**Why the continuous concept is necessary for deriving the equation from (4) to (5).**\nWe can interpret (4) as a discrete-time dynamical system. Equation (5) is a differential equation, i.e., a continuous-time dynamical system. Continuous-time models enable the use of tools developed in physics, mathematics, and control theory. Corresponding tools for discrete-time either do not exist or are much more difficult to use.\n\n**Why do the trained weights converge faster than the random ones?**\nReviewer aNPn had a similar question, please see the answer we provided.\n\n**A traditional element layer normalization in Transformers does not follow the norm-scaling formula described in eq.(2).**\nEq. (2) has been used in concrete transformers, see the citation in the paper. See also the reply to reviewer aNPn who had a similar question.\n\n**Lemma 4.2 does not exist in the reference Geshkovski et al., 2023a.**\nLemma 4.2 appears in versions 1-3 of the arXiv preprint, but in the latest version is now  Lemma 6.4. This will be corrected.\n\n**$y_1$ in Eq.(18) should be $y_0$?**\nWe will rectify this typo on the final version.", "claims_and_evidence": "The claim appears reasonable under certain assumptions; however, the practical evidence supporting it may be insufficient.", "methods_and_evaluation_criteria": "The paper presents a theoretical analysis suggesting that the model could eventually collapse, which could possibly occur under certain conditions.", "theoretical_claims": "Since my focus was primarily on the practical aspects of this paper, the theorems appear mostly correct under given assumptions.", "experimental_designs_or_analyses": "Experiments based on simulations are provided, but some aspects are not practical and problematic.", "relation_to_broader_scientific_literature": "The theoretical backup related to model collapse provided in this paper is an important contribution.", "essential_references_not_discussed": "There is no related work section, despite the existence of many continuous Transformer-based papers that could be cited, none of which are referenced. While there may be differences in the terminology of \"continuous\" used in this paper compared to others, the authors should explicitly discuss how this work differs from previous studies.", "other_strengths_and_weaknesses": "- The theoretical approaches in this paper give a sense of direction, but the exact goal is not clearly presented. From my perspective, the theories are limited by the provided assumptions and seem to hold only under specific conditions. As a result, this reviewer feels that the experimental support is limited and not clearly aligned with the theory.\n- In that regard, the presentation of this paper should be more refined to make it easier to grasp the authors' claims. Furthermore, all materials should be self-contained; for instance, it took multiple times to find that E (appears at the y-axis in the graphs such as  Figures 6 and 7) was in the Appendix.\n- The experiment is designed as a simulation, as the authors stated, by using generated tokens as input for the next iteration of the generation to mimic model collapse. However, this setup does not accurately simulate the behavior of a large language model. It is unclear why the authors did not use larger models while given the availability of many such models up to 70B-scale, for example.\n- This reviewer speculates that the outdated GPT-2 XL, trained on a limited corpus with fixed token lengths, may easily generate meaningless tokens when iteratively fed its own outputs. This may not accurately reflect the model collapse described in the theory. Looking forward to seeing similar results with larger models, up to 32B (which I believe would be sufficient for testing).", "other_comments_or_suggestions": "This paper should be further refined to use more precise terminology and notations. This reviewer believes this paper can be improved by presenting its motivation and goals better. Additionally, scaling up and modernizing the experiments instead of just using GPT2-XL would support the claims effectively. Although this reviewer is not a theorist and partially follows the derivations, the attempt to address model collapse theoretically is valuable. If the authors address the concerns, I am willing to raise my score.", "questions_for_authors": "- This reviewer could not get exactly why the continuous concept is necessary for deriving the equation from (4) to (5). Could the authors clarify this reasoning?\n- Why do the trained weights converge faster than the random ones?\n- A traditional element layer normalization in Transformers does not follow the norm-scaling formula described in eq.(2). \n- Lemma 4.2 does not exist in the reference Geshkovski et al., 2023a.\n- $y_1$ in Eq.(18) should be $y_0$?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kGlDOJ7XJO", "reviewer": "Reviewer_3qpp", "summary": "In this paper, the authors the asymptotic properties of the attention mechanism of transformers. In order to do so, the authors introduce a continuous differential equation emulating the evolution of tokens across an increasing number of layers, and they show that asymptotically -- i.e., as the number of layers involved goes to infinity -- all the tokens tend to collapse to a single point under some assumptions on the initial configuration of the tokens and on the value of the Q, K and V matrices across layers.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time devoted to reading our paper and providing feedback. \n\n**Why are the results important for the community? Is there any implication for real-world applications?**\n\nThe paper answers the following question, what is the role of attention? The role is to bring the tokens together, i.e., to achieve consensus. In practice, our results show that transformers cannot be too deep since the information contained in the tokens disappears as they converge to a location that is independent of where the tokens start. Our results also show that a transformer has to be deep enough since, otherwise, the last token (used to predict the next token) is not sufficiently influenced by the tokens that precede it which would lead to a next token prediction that is independent from the query/context. Our results then raise the following question, what is the optimal depth of a transformer? This is a question we are trying to answer.\n\n**While experiments are important, I believe that the paper wastes too much space on the experimental side.**\n\nAs can be seen by the different reviews and our replies, there are different opinions about experiments. We hope to reach a compromise by reducing the space devoted to experiments while providing more informative experiments (e.g., randomization over multiple prompts as suggested by other reviewers).\n\n**This novelty is completely lost in the paper, as there is no mention of it in the main paper after the introduction.**\n\nWe will address the reviewer's concern by using the space freed by reducing the number of experimental plots to succinctly describe the techniques used in each proof and highlight their control theoretic origin.", "claims_and_evidence": "The proofs of the theorems seem solid to me, and the experimental results provided are satisfactory and seem to back the theoretical results.", "methods_and_evaluation_criteria": "The proof techniques employed are reasonable and the experimental setup is sensible.", "theoretical_claims": "I checked the proofs of Theorems 3.2, 4.2 and 4.3 without going into the details, and I could not spot any serious mistake.", "experimental_designs_or_analyses": "The experimental setup used for Section 5 is sensible. However, the code for the experiments was not provided, so that it is not possible to independently verify the experimental results presented in the paper.", "relation_to_broader_scientific_literature": "The main contribution of the paper is to generalize a series of results in the same setting (asymptotic evolution of tokens across attention layers) to broader configurations of the Q, K and V matrices and to multiple heads.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "The paper is generally well written, and the theory is complemented with many experiments. I have a couple of concerns, though:\n1. I think the paper lacks motivation: why are the results important for the community? Is there any implication for real-world applications?\n2. While experiments are important, I believe that the paper wastes too much space on the experimental side, when the main contributions seem to be on the theoretical side. In particular, it seems that the main novelty of the paper is to use different techniques (borrowed from control theory) to prove their results, compared to previous works in the same area. This novelty is completely lost in the paper, as there is no mention of it in the main paper after the introduction. I would have rather used more space to explain the intuition behind the proofs instead of including the random weights (e.g., first column of p. 6) or the random inputs (e.g., Table 2 or second column of p. 7) used in the experiments, which could be moved to the appendix.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper focuses on developing rigorous results for token representation collapse in Transformer networks as one scales the number of layers. Unlike the existing works that rely on mean-field and/or stochastic techniques, this work leverages control theoretic tools to establish token representation collapse for both full attention and causal attention. Interestingly, the results in the paper also relax various assumptions considered in the prior works.

Overall, the reviewers acknowledged that the submission made important contributions by expanding the current understanding of the representation collapse in the most widely popular architecture. The reviewers raised some valid questions/concerns about deviations from the real-life transformer, e.g., identity or time-invariant value matrices and layer normalization. The authors are encouraged to incorporate promised discussions on extending their analysis to more realistic settings or highlight the main challenges preventing the realization of such desirable extensions.

The submission also presents empirical results that corroborate the theoretical analysis and show asymptotic representation collapse in synthetic settings. That said, the AC and one of the reviewers feel that the experimental section can be significantly compressed to include proof sketches and a slightly more self-contained theoretical treatment of the underlying problem. For instance, is the empirical demonstration of token collapse under the weaker assumption (i.e., for realistic transformers) a novel contribution of this submission? If not, much of these results can be compressed in the main text and relegated to the appendix. The authors may also want to cite the work on looped transformers as the main experimental setup in Section 5.2 is closer to looped transformers. 

Furthermore, the authors may want to highlight the exact architecture they study early in the paper, e.g., in the Introduction. Currently, the authors briefly mention that, similar to Greshkovski et al. 2023a, they don't consider the feed-forward layer (in lines 136-140) and standard layer normalization (in lines 137R-139R). It would benefit the readers if these assumptions/restrictions were mentioned much earlier in the paper along with the fact that prior theoretical studies have also made such assumptions.
Decision: Accept (poster)