SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that the case  $V = -I_{d}$  may appear equally natural. For such a choice of  $V$ , we show in Appendix C.2 that the dynamics converge to a single cluster located at the origin. Multiplicative constants preserving the sign, i.e.,  $V = \pm c I_{d}, c > 0$  trivially yield the same conclusions.
attracting all tokens as  $t$  grows. It has been experimentally observed (first in [VSP+17]) that in trained Transformers, tokens focus their attention on local leaders in a way that seems to reproduce the syntactic and semantic structure of sentences.
The proof of Theorem 3.1 is postponed to Appendix C, and amounts to a couple of effects entailed by the dynamics. First of all, the convex hull of the particles is shrinking over time (Proposition C.2). This is due to the fact that the distance of the particle nearest to any half-space (not containing the particles) increases with time. On the other hand, the convex hull ought not collapse since particles which have not concentrated near the boundary of the limiting polytope will continue to increase in magnitude until they themselves reach this boundary (Step 2 in the proof). This occurs due to the time-rescaling.
Remark 2. Assuming  $Q^{\top}K > 0$  does not seem to be essential for our conclusions; instead, it guides the direction of the proof. To emphasize the broader validity of our conclusion beyond this specific assumption, we conducted additional experiments (refer to Section F.6) which suggest that Theorem 3.1 (as well as Theorems 4.1 and 5.1 stated below) holds in more generality.
Remark 3 (Rate of convergence). Although Theorem 3.1 (as well as Theorems 4.1 and 5.1 stated below) does not specify a rate of convergence toward  $\partial \mathcal{K}$ , we expect (and observe through numerics) that convergence happens very quickly—after few layers, most tokens are already clustered. What "few layers" means here necessarily depends on the typical modulus of the initial tokens, since the dynamics (1) is not invariant under multiplication of all initial conditions by a fixed real number.
Remark 4 (Discrete time). As alluded to in Remark 1, all our results extend to the discrete-time Transformers (3). Indeed, just as in the continuous-time case, there is a natural rescaled dynamics, which is the discrete analogue of (4): if we set  $R = I_d + V\Delta t$ , and assume that  $R$  is invertible (which is the case for sufficiently small  $\Delta t$ ), then  $z_i(k\Delta t) = R^{-k}x_i(k\Delta t) \coloneqq z_i^{[k]}$  satisfies
$$
z _ {i} ^ {[ k + 1 ]} = z _ {i} ^ {[ k ]} + \Delta t \sum_ {j = 1} ^ {n} \left(\frac {e ^ {\langle Q R ^ {k} z _ {i} ^ {[ k ]} , K R ^ {k} z _ {j} ^ {[ k ]} \rangle}}{\sum_ {\ell = 1} ^ {n} e ^ {\langle Q R ^ {k} z _ {i} ^ {[ k ]} , K R ^ {k} z _ {\ell} ^ {[ k ]} \rangle}}\right) R ^ {- 1} V \left(z _ {j} ^ {[ k ]} - z _ {i} ^ {[ k ]}\right), \qquad k \in \mathbb {N}.
$$
The proofs of Theorems 2.1, C.5, 3.1, 4.1, and 5.1 carry through with straightforward modifications.
Let us provide some comments on the proof of Theorem 3.1 in the discrete-time setting, for the sake of completeness. First of all, Proposition C.2 holds intuitively because for all integers  $i \in [n]$  and  $k \geqslant 1$ ,
$$
z _ {i} ^ {[ k + 1 ]} = \frac {1}{1 + \Delta t} \left(z _ {i} ^ {[ k ]} + \Delta t \sum_ {j = 1} ^ {n} P _ {i j} ^ {[ k ]} z _ {j} ^ {[ k ]}\right) \in \operatorname {c o n v} \left(\left\{z _ {j} ^ {[ k ]} \right\} _ {j \in [ n ]}\right).
$$
We then define the candidate set of limit points as in (37), and Claim 1 holds without any change in the statement or in the proof. Then, just as in Steps 2 and 3 in the proof of C.1, we can first show that if  $z_{i}^{[k]}$  is not already near some point in the candidate limit set, it will keep moving toward the boundary of the convex polytope. Finally, we can prove that tokens cannot circulate indefinitely between different points on the boundary. The combination of these arguments would establish the convergence of each token toward some point in the set given by (37).
4 Clustering toward hyperplanes
While being a natural example to consider, value matrices found empirically are much more general than  $V = I_{d}$ , which we considered in the previous section. We now turn our attention to a significantly more general setting of value matrices, which we formalize as follows.
Definition 1. We call  $(Q, K, V)$  a good triple if the two following conditions are satisfied:
- the eigenvalue of  $V$  with largest modulus is real, positive, and simple; namely,
$$
\lambda_ {1} (V) > | \lambda_ {2} (V) | \geqslant \dots \geqslant | \lambda_ {d} (V) |.
$$
-  $\langle Q\varphi_1,K\varphi_1\rangle >0$  for any  $\varphi_{1}\in \mathbb{R}^{d}$  lying on the line  $\ker (V - \lambda_1(V)I_d)$ .
The second condition simply states that the quadratic form  $\langle Q\cdot ,K\cdot \rangle$  is positive definite along the eigenspace associated to the leading eigenvalue of  $V$ . Note also that if all entries of  $V$  are positive,
the first condition is automatically satisfied by virtue of the Perron-Frobenius theorem. In fact, this assumption is generic. On the one hand, it is satisfied by some pre-trained value matrices for ALBERT (Figure 10). On the other hand, numerical experiments indicate that a constant fraction (about  $14\%$ ) of matrices from the real Ginibre ensemble in dimension  $d = 128$ —this proportion is known to vanish as  $d \to \infty$ , albeit very slowly [RS14].
Our clustering result in the setting of good triples can be summarized as follows: the coordinate  $\langle z_i(t),\frac{\varphi_1}{\|\varphi_1\|}\rangle$  of any token  $z_{i}(t)$  along the eigenspace spanned by  $\varphi_{1}$  converges, as  $t\to +\infty$ , toward one among possibly 3 real scalars. Consequently, all the tokens  $z_{i}(t)$  converge toward one among at most three parallel hyperplanes; see Fig. 6 for an illustration.
Theorem 4.1 (Convergence toward  $\leqslant 3$  hyperplanes). Assume that  $(Q, K, V)$  is a good triple in the sense of Definition 1. Then, for any initial sequence of tokens  $\{z_i(0)\}_{i\in [n]} \subset \mathbb{R}^d$ , there exist at most three parallel hyperplanes in  $\mathbb{R}^d$  such that for any  $i \in [n]$ , the distance of the solution  $z_i(t)$  to (4) to one of these hyperplanes converges to 0 as  $t \to +\infty$ .
Figure 6: Illustrating Theorem 4.1 with  $n = 40$  tokens in  $\mathbb{R}^2$ . Here  $Q = K = I_2$ ,  $V$  is a random symmetric matrix with eigenvalues  $\{1.35, -0.07\}$ , and  $\varphi_1 = (0.76, 0.65)$ . The components of the tokens in the direction of  $\varphi_1$  (orange arrow) cluster over time. (See Figures 13-14 for examples in  $\mathbb{R}^3$ .) We also observe that tokens typically cluster toward only two hyperplanes—a third one (passing through the origin) may appear for non-generic initial sequences. The hyperplanes are perpendicular to  $\varphi_1$  since  $V$  is diagonalizable.
The proof may be found in Appendix D. The important role played by  $\lambda_1(V)$  in the dynamics may be seen in (4): the component of  $z_{i}(t)$  along  $\varphi_{1}$  determines the size of  $e^{tV}z_i(t)$  in the exponent appearing in (4). The tokens  $z_{j}(t)$  attracting other tokens  $z_{i}(t)$  are those for which this component along  $\varphi_{1}$  is largest in modulus. This attraction process forms the clusters. These leaders, as in all our results, have been empirically observed to be the ones carrying the largest amount of information in the sentence (see Supplementary material in [VSP+17]).
Furthermore, Theorem 4.1 can also be interpreted in more classical machine learning terms. On the one hand, it can be seen as an instance of  $K$ -flats clustering [BM00, Vid11]—points in the input sequence are clustered, based on their intrinsic similarity, to at most 3 "flats" of dimension  $d - 1$ . On the other hand, it ensures that for a good triple  $(Q, K, V)$ , (4) generates a linearly separable representation of tokens.
Beyond a single direction?
Numerical experiments (e.g., Fig. 7) indicate that a similar phenomenon emerges for more complex  $V$ . We formulate following conjecture which is a natural generalization of Theorem 4.1.
Conjecture 4.2 (Codimension conjecture). Let  $k \geqslant 1$  be the number of eigenvalues of  $V$  with positive real part. Then there exist at most three parallel Euclidean subspaces of  $\mathbb{R}^d$  of codimension  $k$  such that for any  $i \in [n]$ , the distance of  $z_i(t)$  to one of these subspaces converges to 0 as  $t \to +\infty$ .
5 A mix of hyperplanes and polytopes
We now turn our attention to an even more general version of Theorem 4.1, which does not require the leading eigenvalue of  $V$  to be simple. The resulting theorem can be viewed as a combination of
$t = 10.0$
$t = 0.0$
$t = 5.0$
$t = 15.0$
(a) Conjecture 4.2: low-dimensional case.
Positive limits for clustered coordinates
Negative limits for clustered coordinates
(b) Conjecture 4.2: high-dimensional case.
Figure 7: (a)  $n = 40$ ,  $d = 3$  and  $Q = K = I_3$  with  $V$  a random matrix with eigenvalues  $\{1.96, -0.22, 0.25\}$ . The  $k = 2$  positive eigenvalues of  $V$  generate attraction between the tokens and even convergence in the corresponding eigenspaces—this explains the codimension  $k$  statement. The negative eigenvalue generates a repulsive effect between the tokens, and we see a divergence along two lines (note the different scales between the four figures). (b)  $n = 256$ ,  $d = 128$ , with  $(Q, K, V)$  fixed random matrices and  $V$  symmetric. For each coordinate  $j$  corresponding to a positive eigenvalue, the variance of the set  $\{\varphi_j^*(z_i(t)) : i \in [n]\}$  (shaded area) tends to 0 with  $t$ , while the mean (solid lines) converges to one among two real scalars: one positive (top figure), one negative (bottom) figure. Coordinates corresponding to negative eigenvalues diverge (Fig. 15).
Theorem 4.1 and Theorem 3.1. Specifically, we assume that  $V$  behaves as the identity when acting on the eigenspace of the leading eigenvalue. This property is automatically satisfied if  $V$  is normal—so that its eigenvectors form an orthonormal basis—so we call such a  $V$  paranormal.
Definition 2. We call  $(Q,K,V)$  a good triple with multiplicity if the following conditions hold:
(i)  $Q^{\top}K$  is positive definite:  $Q^{\top}K > 0$ ;
(ii)  $V$  is paranormal: there exist two linear subspaces  $\mathcal{F},\mathcal{G}\subset \mathbb{R}^d$  which are invariant under  $V$  and such that  $\mathcal{F}\oplus \mathcal{G} = \mathbb{R}^d$ ,  $V_{|\mathcal{F}} = \lambda \mathrm{Id}$  for  $\lambda >0$ , and  $\rho (V_{|\mathcal{G}}) < \lambda$ , where  $\rho (\cdot)$  denotes the spectral radius (the maximal modulus of eigenvalues).
An example of such a  $V$  is used for Fig. 8. We may now state our main result in the setting of good triples with multiplicity. The proof may be found in Appendix E.
Theorem 5.1 (Clustering for  $\lambda_{1}$  with multiplicity). Suppose that  $(Q, K, V)$  is a good triple with multiplicity in the sense of Definition 2. Then, for any initial sequence  $\{z_{i}(0)\}_{i \in [n]} \subset \mathbb{R}^{d}$ , there exists a bounded convex polytope  $\mathcal{K} \subset \mathcal{F}$  such that setting  $\mathcal{H} := (\partial \mathcal{K} \cup \{0\}) \times \mathcal{G}$ , for any  $i \in [n]$ , we have  $\mathrm{dist}(z_{i}(t), \mathcal{H}) \to 0$  as  $t \to +\infty$ .
6 Outlook
Several important directions regarding the mathematical theory of Transformers remain unexplored. An important extension of our work would amount to studying multi-headed Transformers—borrowing the notation from Remark 4, they amount to:
$$
x _ {i} ^ {[ k + 1 ]} = x _ {i} ^ {[ k ]} + \Delta t \sum_ {h = 1} ^ {H} \sum_ {j = 1} ^ {n} \left(\frac {e ^ {\langle Q _ {h} x _ {i} ^ {[ k ]} , K _ {h} x _ {j} ^ {[ k ]} \rangle}}{\sum_ {\ell = 1} ^ {n} e ^ {\langle Q _ {h} x _ {i} ^ {[ k ]} , K _ {h} x _ {\ell} (k) \rangle}}\right) V _ {h} x _ {j} ^ {[ k ]}, \qquad k \in \mathbb {N}.
$$
Figure 8: Illustrating Theorem 5.1 with  $n = 40$  tokens in  $\mathbb{R}^3$ . As before,  $Q = K = I_d$ , and we take  $V = \mathrm{diag}(1,1, - \frac{1}{2})$ . A convex polytope  $\kappa$  emerges before time 5, toward which two coordinates of the tokens cluster, and persists throughout the evolution, while the tokens diverge along the coordinate corresponding to the eigenvalue  $-\frac{1}{2}$  (note the different scales between the four figures).
$t = 0.0$
$t = 5.0$
$t = 10.0$
$t = 15.0$
For each  $h \in [H]$  (corresponding to a different head), the weight matrices  $Q_h, K_h, V_h$  are constant. Proofs regarding clustering or convergence of the self-attention matrix for such dynamics is an open problem. Preliminary numerical investigations seem to indicate that interesting clustering phenomena also occur in this context. A characterization or properties of optimal weights by invoking the optimal control correspondence in the spirit of [Wei17] is also an interesting avenue for future research.
We thank Pierre Ablin, Léonard Boussioux, Enric Boix-Adsera, Gabriel Peyre, Yair Shenfeld and Emmanuel Trélat for helpful discussions. C.L. was supported by the Simons Foundation Grant 601948, DJ. P.R. is supported by NSF grants IIS-1838071, DMS-2022448, and CCF-2106377. Y.P. is supported in part by the MIT-IBM Watson AI Lab.
[ABV+05] Juan A Acebrón, Luis L Bonilla, Conrad J Pérez Vicente, Félix Ritort, and Renato Spigler. The Kuramoto model: A simple paradigm for synchronization phenomena. Reviews of modern physics, 77(1):137, 2005.
[BM00] Paul S Bradley and Olvi L Mangasarian. K-plane clustering. Journal of Global optimization, 16:23-32, 2000.
[Che95] Yizong Cheng. Mean shift, mode seeking, and clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(8):790-799, 1995.
$\left[\mathrm{CHH}^{+}16\right]$  Junghee Cho, Seung-Yeal Ha, Feimin Huang, Chunyin Jin, and Dongnam Ko. Emergence of bi-cluster flocking for the Cucker-Smale model. Mathematical Models and Methods in Applied Sciences, 26(06):1191-1218, 2016.
[CRBD18] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 31, 2018.
[CS07] Felipe Cucker and Steve Smale. Emergent behavior in flocks. IEEE Transactions on Automatic Control, 52(5):852-862, 2007.
[DCL21] Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pages 2793-2803. PMLR, 2021.
[Dob79] Roland L'vovich Dobrushin. Vlasov equations. Funktional'nyi Analiz i ego Prilozheniya, 13(2):48-58, 1979.
[Gol13] François Golse. Mean field kinetic equations. Course of Polytechnique, 2013.
[PRT15] Benedetto Piccoli, Francesco Rossi, and Emmanuel Trélat. Control to flocking of the kinetic Cucker-Smale model. SIAM Journal on Mathematical Analysis, 47(6):4685-4719, 2015.
[RS14] Brian Rider and Christopher D. Sinclair. Extremal laws for the real Ginibre ensemble. The Annals of Applied Probability, 24(4):1621 - 1651, 2014.
[SABP22] Michael E Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Sinkformers: Transformers with doubly stochastic attention. In International Conference on Artificial Intelligence and Statistics, pages 3515–3530. PMLR, 2022.
$\left[\mathrm{VCBJ}^{+}95\right]$  Tamás Vicsek, András Czirók, Eshel Ben-Jacob, Inon Cohen, and Ofer Shochet. Novel type of phase transition in a system of self-driven particles. Physical Review Letters, 75(6):1226, 1995.
[VID11] René Vidal. Subspace clustering. IEEE Signal Processing Magazine, 28(2):52-68, 2011.
[VKF20] Apoorv Vyas, Angelos Katharopoulos, and François Fleuret. Fast transformers with clustered attention. Advances in Neural Information Processing Systems, 33:21665-21674, 2020.
$\left[\mathrm{VSP}^{+}17\right]$  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[Wei17] E Weinan. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 1(5):1-11, 2017.
[WHL19] E Weinan, Jiequn Han, and Qianxiao Li. A mean-field optimal control formulation of deep learning. Research in Mathematical Sciences, 6(1):10, 2019.
[WLK+20] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020.
[YBR+20] Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? In International Conference on Learning Representations, 2020.