# The emergence of clusters in self-attention dynamics

Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, Philippe Rigollet

► To cite this version:

Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, Philippe Rigollet. The emergence of clusters in self-attention dynamics. 2024. hal-04092937v2

HAL Id: hal-04092937

https://hal.science/hal-04092937v2

Preprint submitted on 20 Feb 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
<figure><img src="image_1.png" /></figure>

HAL
open science

# The emergence of clusters in self-attention dynamics

Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, Philippe Rigollet

► To cite this version:

Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, Philippe Rigollet. The emergence of clusters in self-attention dynamics. 2024. hal-04092937v2

HAL Id: hal-04092937

https://hal.science/hal-04092937v2

Preprint submitted on 20 Feb 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Open licence - etalab
# THE EMERGENCE OF CLUSTERS IN SELF-ATTENTION
DYNAMICS

BORJAN GESHKOVSKI, CYRIL LETROUIT, YURY POLYANSKIY,
AND PHILIPPE RIGOLLET

ABSTRACT. Viewing Transformers as interacting particle systems, we describe the geometry of learned representations when the weights are not time dependent. We show that particles, representing tokens, tend to cluster toward particular limiting objects as time tends to infinity. Cluster locations are determined by the initial tokens, confirming context-awareness of representations learned by Transformers. Using techniques from dynamical systems and partial differential equations, we show that the type of limiting object that emerges depends on the spectrum of the value matrix. Additionally, in the one-dimensional case we prove that the self-attention matrix converges to a low-rank Boolean matrix. The combination of these results mathematically confirms the empirical observation made by Vaswani et al. [VSP+17] that leaders appear in a sequence of tokens when processed by Transformers.

## CONTENTS

<table><tr><td><strong>Part 1. Introduction and main results</strong></td><td>1</td></tr><tr><td>1. Introduction</td><td>1</td></tr><tr><td>2. Asymptotic low-rankness of the self-attention matrix</td><td>6</td></tr><tr><td>3. Clustering toward vertices of convex polytopes</td><td>8</td></tr><tr><td>4. Clustering toward hyperplanes</td><td>10</td></tr><tr><td>5. A mix of hyperplanes and polytopes</td><td>13</td></tr><tr><td><strong>Part 2. Proofs</strong></td><td>13</td></tr><tr><td>6. Well-posedness</td><td>13</td></tr><tr><td>7. Proof of Theorem 2.1</td><td>20</td></tr><tr><td>8. Proofs of Theorems 3.1 and 8.5</td><td>28</td></tr><tr><td>9. Proof of Theorem 4.2</td><td>39</td></tr><tr><td>10. Proof of Theorem 5.2</td><td>44</td></tr><tr><td>11. Numerical experiments</td><td>46</td></tr><tr><td><strong>Part 3. Discussion and open questions</strong></td><td>50</td></tr><tr><td>12. Outlook</td><td>50</td></tr><tr><td>References</td><td>53</td></tr></table>

## Part 1. Introduction and main results

### 1. INTRODUCTION

The introduction of Transformers in 2017 [VSP+17] marked a turning point in the AI revolution, powering breakthroughs in natural language modeling and computer vision. With remarkable empirical success, Transformers enable large language models to compute very powerful representations using the self-attention
# THE EMERGENCE OF CLUSTERS IN SELF-ATTENTION DYNAMICS

BORJAN GESHKOVSKI, CYRIL LETROUIT, YURY POLYANSKIY, AND PHILIPPE RIGOLLET

**ABSTRACT.** Viewing Transformers as interacting particle systems, we describe the geometry of learned representations when the weights are not time dependent. We show that particles, representing tokens, tend to cluster toward particular limiting objects as time tends to infinity. Cluster locations are determined by the initial tokens, confirming context-awareness of representations learned by Transformers. Using techniques from dynamical systems and partial differential equations, we show that the type of limiting object that emerges depends on the spectrum of the value matrix. Additionally, in the one-dimensional case we prove that the self-attention matrix converges to a low-rank Boolean matrix. The combination of these results mathematically confirms the empirical observation made by Vaswani et al. [VSP+17] that leaders appear in a sequence of tokens when processed by Transformers.

## CONTENTS

<table><tr><td><strong>Part 1. Introduction and main results</strong></td><td>1</td></tr><tr><td>1. Introduction</td><td>1</td></tr><tr><td>2. Asymptotic low-rankness of the self-attention matrix</td><td>6</td></tr><tr><td>3. Clustering toward vertices of convex polytopes</td><td>8</td></tr><tr><td>4. Clustering toward hyperplanes</td><td>10</td></tr><tr><td>5. A mix of hyperplanes and polytopes</td><td>13</td></tr><tr><td><strong>Part 2. Proofs</strong></td><td>13</td></tr><tr><td>6. Well-posedness</td><td>13</td></tr><tr><td>7. Proof of Theorem 2.1</td><td>20</td></tr><tr><td>8. Proofs of Theorems 3.1 and 8.5</td><td>28</td></tr><tr><td>9. Proof of Theorem 4.2</td><td>39</td></tr><tr><td>10. Proof of Theorem 5.2</td><td>44</td></tr><tr><td>11. Numerical experiments</td><td>46</td></tr><tr><td><strong>Part 3. Discussion and open questions</strong></td><td>50</td></tr><tr><td>12. Outlook</td><td>50</td></tr><tr><td>References</td><td>53</td></tr></table>

## Part 1. Introduction and main results

### 1. INTRODUCTION

The introduction of Transformers in 2017 [VSP+17] marked a turning point in the AI revolution, powering breakthroughs in natural language modeling and computer vision. With remarkable empirical success, Transformers enable large language models to compute very powerful representations using the self-attention
mechanism. Yet, little is known about the geometric structure of these representations. As the size of these models grows at an astonishing rate, the need to understand their inner workings is becoming a pressing scientific challenge. In this work, we make a first step in this direction by describing the geometry of learned representations.

To provide a transparent presentation of our findings, we take a leaf out of the literature on continuous-time dynamics such as neural ordinary differential equations (ODEs) [CRBD18, Wei17, HR17]. By viewing layers as a time variable, this formalism has emerged as a flexible mathematical framework to implement and study ResNets [HZRS16a] as particular discrete-time versions of a parametrized dynamics of the form

$$ \dot{x}(t) = f_{\theta}(x(t)), \quad t \in [0, T]. $$

Here $\theta$ is the trained parameter of a neural network and $f_{\theta}$ is characterized by the precise architecture of the ResNet<fnref n="1" />. In turn, an input (e.g., an image) $x(0) \in \mathbb{R}^d$ is mapped to its representation $x(T)$.

Unlike neural ODEs and ResNets, the representation map of Transformers is not solely a function of an individual input $x(0) \in \mathbb{R}^d$ but rather of a set/sequence $(x_1(0), \dots, x_n(0))$ of $n \ge 1$ $d$-dimensional tokens. These tokens then evolve in time by interacting with each other per the self-attention mechanism. Namely, following [SABP22], we view tokens as particles, and the transformer dynamics as an interacting particle system of the form

$$ \dot{x}_i(t) = \sum_{j=1}^{n} P_{ij}(t) V x_j(t), \quad t \in [0, +\infty), \qquad (1.1) $$

for any $i \in [n]$, where $P_{ij}(t)$ are the entries of a $n \times n$ stochastic matrix $P(t)$, given by

$$ P_{ij}(t) := \frac{e^{\langle Qx_i(t), Kx_j(t) \rangle}}{\sum_{\ell=1}^{n} e^{\langle Qx_i(t), Kx_{\ell}(t) \rangle}}, \quad (i, j) \in [n]^2. \qquad (1.2) $$

Here the matrices $Q$ (Query), $K$ (Key), and $V$ (Value) are learned from data. Note that $Q, K$ need not be square. The $n \times n$ matrix $P(t)$ is called self-attention matrix. The wording attention stems precisely from the fact that $P_{ij}(t)$ captures the attention given by token $i$ to token $j$ relatively to all tokens $\ell \in [n]$. The matrices $Q$ and $K$ in (1.2) warp the geometry of the input tokens, so that a trained attention matrix contains weights which indicate semantic relations between words. Such conclusions have been drawn in the context of language processing tasks in [VSP+17, Figures 3-5].

Our goal is to showcase the fact that self-attention, which itself is the core novelty of Transformers, entails a clustering effect. To that end, we focus on the pure self-attention dynamics described in (1.1). In particular, we do not model variations such as multiple heads, feed-forward layers, and layer normalization that are typically adjoined to self-attention dynamics of (1.1). However, on this last point, we note that our theoretical findings indicate that without any normalization, the dynamics (1.1) can diverge in some (or even all) directions over time. We leave these additional questions for future research; see Section 12.

<footnote n="1">¹A classical choice is $\theta = (W, A, b) \in \mathbb{R}^{d \times d} \times \mathbb{R}^{d \times d} \times \mathbb{R}^d$ and $f_{\theta}(x) = W\sigma(Ax + b)$ where $\sigma$ is an elementwise nonlinearity such as the ReLU ($\mathrm{ReLU}$ ($\mathrm{HZRS16b}$)).</footnote>
mechanism. Yet, little is known about the geometric structure of these representations. As the size of these models grows at an astonishing rate, the need to understand their inner workings is becoming a pressing scientific challenge. In this work, we make a first step in this direction by describing the geometry of learned representations.

To provide a transparent presentation of our findings, we take a leaf out of the literature on continuous-time dynamics such as neural ordinary differential equations (ODEs) [CRBD18, Wei17, HR17]. By viewing layers as a time variable, this formalism has emerged as a flexible mathematical framework to implement and study ResNets [HZRS16a] as particular discrete-time versions of a parametrized dynamics of the form

$$ \dot{x}(t) = f_{\theta}(x(t)), \quad t \in [0, T]. $$

Here $\theta$ is the trained parameter of a neural network and $f_{\theta}$ is characterized by the precise architecture of the ResNet<fnref n="1" />. In turn, an input (e.g., an image) $x(0) \in \mathbb{R}^d$ is mapped to its representation $x(T)$.

Unlike neural ODEs and ResNets, the representation map of Transformers is not solely a function of an individual input $x(0) \in \mathbb{R}^d$ but rather of a set/sequence $(x_1(0), \dots, x_n(0))$ of $n \ge 1$ d-dimensional tokens. These tokens then evolve in time by interacting with each other per the self-attention mechanism. Namely, following [SABP22], we view tokens as particles, and the transformer dynamics as an interacting particle system of the form

$$ \dot{x}_i(t) = \sum_{j=1}^{n} P_{ij}(t) V x_j(t), \quad t \in [0, +\infty), \qquad (1.1) $$

for any $i \in [n]$, where $P_{ij}(t)$ are the entries of a $n \times n$ stochastic matrix $P(t)$, given by

$$ P_{ij}(t) := \frac{e^{\langle Qx_i(t), Kx_j(t) \rangle}}{\sum_{\ell=1}^{n} e^{\langle Qx_i(t), Kx_{\ell}(t) \rangle}}, \quad (i, j) \in [n]^2. \qquad (1.2) $$

Here the matrices $Q$ (Query), $K$ (Key), and $V$ (Value) are learned from data. Note that $Q, K$ need not be square. The $n \times n$ matrix $P(t)$ is called self-attention matrix. The wording attention stems precisely from the fact that $P_{ij}(t)$ captures the attention given by token $i$ to token $j$ relatively to all tokens $\ell \in [n]$. The matrices $Q$ and $K$ in (1.2) warp the geometry of the input tokens, so that a trained attention matrix contains weights which indicate semantic relations between words. Such conclusions have been drawn in the context of language processing tasks in [VSP+17, Figures 3-5].

Our goal is to showcase the fact that self-attention, which itself is the core novelty of Transformers, entails a clustering effect. To that end, we focus on the pure self-attention dynamics described in (1.1). In particular, we do not model variations such as multiple heads, feed-forward layers, and layer normalization that are typically adjoined to self-attention dynamics of (1.1). However, on this last point, we note that our theoretical findings indicate that without any normalization, the dynamics (1.1) can diverge in some (or even all) directions over time. We leave these additional questions for future research; see Section 12.

<footnote n="1">¹A classical choice is $\theta = (W, A, b) \in \mathbb{R}^{d \times d} \times \mathbb{R}^{d \times d} \times \mathbb{R}^d$ and $f_{\theta}(x) = W\sigma(Ax + b)$ where $\sigma$ is an elementwise nonlinearity such as the ReLU ($[HZRS16b]$).</footnote>
**1.1. Organization of the paper and summary of contributions.** The goal of this paper is to characterize clustered representations of a *trained* Transformer by studying the asymptotic behavior of a sequence of tokens $(x_1(t), \dots, x_n(t))$ as they evolve through the layers of a transformer architecture using the dynamics (1.1). In this setup, a Transformer is completely described by the weight matrices $(Q, K, V)$ obtained during training. Note that we assume that these three matrices are *time-independent*. While this assumption is motivated

by mathematical convenience, it is worth noting that such weight-sharing scenarios are in fact used in practice—see, e.g., ALBERT [LCG$^{+20}$]—as they drastically reduce the number of parameters of a network.

With parameters $(Q, K, V)$ fixed, tokens are subject to collective dynamics that we call *transformer dynamics*. While these dynamics are reminiscent of existing models for opinion dynamics and flocking, they present their own mathematical challenges requiring ad-hoc tools to study their asymptotic behavior.

The main conclusion of our analysis is that the set of tokens $\{x_1(t), \dots, x_n(t)\}$, appropriately rescaled, tends to a *clustered configuration* as $t \to \infty$. Our theoretical findings justify the empirical observation made in [VSP$^{+17}$] that *leaders* appear in a sequence of tokens when processed by Transformers. We now list our main contributions.

(i) As a warm-up to the geometric characterization of the limits of sequences of tokens, we show in **Section 2** that when $d=1$ and $V > 0$, the self-attention matrix $P(t)$ converges to a low-rank matrix with entries 0 and 1 as $t \to +\infty$ thus revealing the emergence of a small number of leaders that drive the transformer dynamics. The restriction $d=1$ follows from technical considerations, and some pathological phenomena may occur in higher dimensions (see Remark 7.9). The proof may be found in **Section 7**. But numerical experiments (as well as past empirical work) indicate that the result may extend to higher dimensions for almost all initial sequences of tokens.

(ii) In **Section 3** we first focus on the case $V = I_d$ as a natural canonical choice that enables us to establish some of the main tools of the paper. We introduce a time re-scaling reminiscent of the layer normalization heuristics to alleviate the possible divergence of tokens. We show that along this scale the tokens converge to the boundary of a convex polytope. For almost all initial sequences they even converge to the vertices of the polytope, the number of which is significantly smaller than $n$. This elucidates the clustering phenomenon. (See Fig. 1.) When $V = -I_d$, all tokens following the dynamics (1.1) collapse to 0. The proofs are given in **Section 8**.

(iii) We build on these results and in **Section 4** consider the case wherein $V$ is only assumed to have a simple and positive leading eigenvalue. This setting is much closer to reality and corresponds to actual learned matrices $V$ (see Figure 10). We show that along the particular timescale, tokens cluster toward one of at most three

<figure><img src="image_2.png" /><figcaption>**Figure 1.** For $V = I_3$ tokens cluster toward the vertices of a convex polytope (Theorem 3.1).</figcaption></figure>
**1.1. Organization of the paper and summary of contributions.** The goal of this paper is to characterize clustered representations of a *trained* Transformer by studying the asymptotic behavior of a sequence of tokens $(x_1(t), \dots, x_n(t))$ as they evolve through the layers of a transformer architecture using the dynamics (1.1). In this setup, a Transformer is completely described by the weight matrices $(Q, K, V)$ obtained during training. Note that we assume that these three matrices are *time-independent*. While this assumption is motivated

<figure><img src="image_3.png" /><figcaption>**Figure 1.** For $V = I_3$ tokens cluster toward the vertices of a convex polytope (Theorem 3.1).</figcaption></figure>

by mathematical convenience, it is worth noting that such weight-sharing scenarios are in fact used in practice—see, e.g., ALBERT [LCG$^{+20}$]—as they drastically reduce the number of parameters of a network.

With parameters $(Q, K, V)$ fixed, tokens are subject to collective dynamics that we call *transformer dynamics*. While these dynamics are reminiscent of existing models for opinion dynamics and flocking, they present their own mathematical challenges requiring ad-hoc tools to study their asymptotic behavior.

The main conclusion of our analysis is that the set of tokens $\{x_1(t), \dots, x_n(t)\}$, appropriately rescaled, tends to a *clustered configuration* as $t \to \infty$. Our theoretical findings justify the empirical observation made in [VSP$^{+17}$] that *leaders* appear in a sequence of tokens when processed by Transformers. We now list our main contributions.

(i) As a warm-up to the geometric characterization of the limits of sequences of tokens, we show in **Section 2** that when $d = 1$ and $V > 0$, the self-attention matrix $P(t)$ converges to a low-rank matrix with entries 0 and 1 as $t \to +\infty$ thus revealing the emergence of a small number of leaders that drive the transformer dynamics. The restriction $d = 1$ follows from technical considerations, and some pathological phenomena may occur in higher dimensions (see Remark 7.9). The proof may be found in **Section 7**. But numerical experiments (as well as past empirical work) indicate that the result may extend to higher dimensions for almost all initial sequences of tokens.

(ii) In **Section 3** we first focus on the case $V = I_d$ as a natural canonical choice that enables us to establish some of the main tools of the paper. We introduce a time re-scaling reminiscent of the layer normalization heuristics to alleviate the possible divergence of tokens. We show that along this scale the tokens converge to the boundary of a convex polytope. For almost all initial sequences they even converge to the vertices of the polytope, the number of which is significantly smaller than $n$. This elucidates the clustering phenomenon. (See Fig. 1.) When $V = -I_d$, all tokens following the dynamics (1.1) collapse to 0. The proofs are given in **Section 8**.

(iii) We build on these results and in **Section 4** consider the case wherein $V$ is only assumed to have a simple and positive leading eigenvalue. This setting is much closer to reality and corresponds to actual learned matrices $V$ (see Figure 10). We show that along the particular timescale, tokens cluster toward one of at most three
hyperplanes which are determined by the corresponding eigenvector. The proof is given in **Section 9**.

(iv) In **Section 5** we complete the results of Sections 3 and 4 by addressing the case where the leading eigenvalue has multiplicity. This results in clustering toward the vertices of a convex polytope in some directions, and a linear subspace in the others. The proof is provided in **Section 10**.

(v) We also prove the global existence and uniqueness of solutions of all dynamics considered in this work (including the mean field limit). We refer the reader to **Section 6** for more details.

We also observed numerically that our conclusions extend to more compound architectures (see Conjecture 4.3, **Section 12** and **Section 11**).

<table><caption>**Table 1.** Summary of the clustering results of this work. *All results except for the case $V = -I_d$ hold for the time-scaled dynamics (3.1).</caption><thead><tr><th>Value</th><th>Key and Query</th><th>Limit geometry</th><th>Reference</th></tr></thead><tbody><tr><td>V = I<sub>d</sub></td><td>Q<sup>T</sup>K > 0</td><td>vertices of convex polytope</td><td>Theorem 3.1</td></tr><tr><td>λ<sub>1</sub>(V) > 0, simple</td><td>&lt;Qϕ<sub>1</sub>, Kϕ<sub>1</sub>&gt; > 0</td><td>union of 3 parallel hyperplanes</td><td>Theorem 4.2</td></tr><tr><td>V paranormal</td><td>Q<sup>T</sup>K > 0</td><td>polytope × subspaces</td><td>Theorem 5.2</td></tr><tr><td>V = -I<sub>d</sub></td><td>Q<sup>T</sup>K = I<sub>d</sub></td><td>single cluster at origin*</td><td>Theorem 8.5</td></tr></tbody></table>

**Remark 1.1** (Discrete time). While we focus on the idealized setting of self-attention dynamics in continuous-time, this is solely done for convenience and all of our methods are straightforwardly applicable to the discrete-time setting. (See also Remark 3.4.) The discrete-time analog of (1.1) with time-step $\Delta t > 0$ (equal to 1 in practice) is simply the forward Euler iteration

$$x_i((k+1)\Delta t) = x_i(k\Delta t) + \Delta t \sum_{j=1}^{n} \left( \frac{e^{\langle Qx_i(k\Delta t), Kx_j(k\Delta t) \rangle}}{\sum_{\ell=1}^{n} e^{\langle Qx_i(k\Delta t), Kx_\ell(k\Delta t) \rangle}} \right) V x_j(k\Delta t), \quad (1.3)$$

for $k \in \mathbb{N}$.

1.2. **Notation.** We denote by $\langle \cdot, \cdot \rangle$ and $\| \cdot \|$ the Euclidean dot product and norm respectively, and we use the shorthand $[n] := \{1, \dots, n\}$. For any matrix $M \in \mathbb{R}^{d \times d}$, we order its eigenvalues (repeated according to multiplicity) by decreasing order of modulus: $|\lambda_1(M)| \ge \dots \ge |\lambda_d(M)|$. We denote by $\|M\|_{\text{op}}$ the $\ell^2$-operator norm of the matrix $M$, equal to the largest singular value of $M$. Given a set $S \subset \mathbb{R}^d$, we define the distance of a point $x \in \mathbb{R}^d$ to $S$ as $\text{dist}(x, S) := \inf_{s \in S} \|x - s\|$, and by $\text{conv}(S)$ the convex hull of $S$.

1.3. **Related work.** Our study and results build on several different lines of work, and we draw some parallels in what follows.

1.3.1. *Analysis of attention-based models.* Given the widespread use of Transformers in natural language processing, there has been a surge of interest in understanding the function and significance of attention layers within these models. In [YBR+20], the authors show that when treated as discrete-time systems with additional dense layers and multiple heads appended to the core attention mechanism,
hyperplanes which are determined by the corresponding eigenvector. The proof is given in **Section 9**.

(iv) In **Section 5** we complete the results of Sections 3 and 4 by addressing the case where the leading eigenvalue has multiplicity. This results in clustering toward the vertices of a convex polytope in some directions, and a linear subspace in the others. The proof is provided in **Section 10**.

(v) We also prove the global existence and uniqueness of solutions of all dynamics considered in this work (including the mean field limit). We refer the reader to **Section 6** for more details.

We also observed numerically that our conclusions extend to more compound architectures (see Conjecture 4.3, **Section 12** and **Section 11**).

<table><caption>**Table 1.** Summary of the clustering results of this work. *All results except for the case $V = -I_d$ hold for the time-scaled dynamics (3.1).</caption><thead><tr><th>Value</th><th>Key and Query</th><th>Limit geometry</th><th>Reference</th></tr></thead><tbody><tr><td>V = I<sub>d</sub></td><td>Q<sup>T</sup>K > 0</td><td>vertices of convex polytope</td><td>Theorem 3.1</td></tr><tr><td>λ<sub>1</sub>(V) > 0, simple</td><td>&lt;Qϕ<sub>1</sub>, Kϕ<sub>1</sub>&gt; > 0</td><td>union of 3 parallel hyperplanes</td><td>Theorem 4.2</td></tr><tr><td>V paranormal</td><td>Q<sup>T</sup>K > 0</td><td>polytope × subspaces</td><td>Theorem 5.2</td></tr><tr><td>V = -I<sub>d</sub></td><td>Q<sup>T</sup>K = I<sub>d</sub></td><td>single cluster at origin*</td><td>Theorem 8.5</td></tr></tbody></table>

**Remark 1.1** (Discrete time). While we focus on the idealized setting of self-attention dynamics in continuous-time, this is solely done for convenience and all of our methods are straightforwardly applicable to the discrete-time setting. (See also Remark 3.4.) The discrete-time analog of (1.1) with time-step $\Delta t > 0$ (equal to 1 in practice) is simply the forward Euler iteration

$$x_i((k+1)\Delta t) = x_i(k\Delta t) + \Delta t \sum_{j=1}^{n} \left( \frac{e^{\langle Qx_i(k\Delta t), Kx_j(k\Delta t) \rangle}}{\sum_{\ell=1}^{n} e^{\langle Qx_i(k\Delta t), Kx_\ell(k\Delta t) \rangle}} \right) V x_j(k\Delta t), \quad (1.3)$$

for $k \in \mathbb{N}$.

1.2. **Notation.** We denote by $\langle \cdot, \cdot \rangle$ and $\| \cdot \|$ the Euclidean dot product and norm respectively, and we use the shorthand $[n] := \{1, \dots, n\}$. For any matrix $M \in \mathbb{R}^{d \times d}$, we order its eigenvalues (repeated according to multiplicity) by decreasing order of modulus: $|\lambda_1(M)| \ge \dots \ge |\lambda_d(M)|$. We denote by $\|M\|_{\text{op}}$ the $\ell^2$-operator norm of the matrix $M$, equal to the largest singular value of $M$. Given a set $S \subset \mathbb{R}^d$, we define the distance of a point $x \in \mathbb{R}^d$ to $S$ as $\text{dist}(x, S) := \inf_{s \in S} \|x - s\|$, and by $\text{conv}(S)$ the convex hull of $S$.

1.3. **Related work.** Our study and results build on several different lines of work, and we draw some parallels in what follows.

1.3.1. **Analysis of attention-based models.** Given the widespread use of Transformers in natural language processing, there has been a surge of interest in understanding the function and significance of attention layers within these models. In [YBR+20], the authors show that when treated as discrete-time systems with additional dense layers and multiple heads appended to the core attention mechanism,
Transformers exhibit the universal approximation property. In [LLH$^{+20}$], the authors present, to the best of our knowledge, the first interacting particle systems perspective on Transformers. They then leverage the similarities between Transformers (with an additional feed-forward layer compared to (1.1)) and convection-diffusion equations to slightly improve the performance of Transformers by employing a Strang-Marchuk splitting scheme for time discretization. In [SABP22], the authors interpret system (1.1) as the characteristics of a continuity equation. Drawing on the similarities between (1.1) and Sinkhorn iterations, they propose a novel architecture dubbed *Sinkformer*, which possesses the desirable property of being a Wasserstein gradient flow.

### 1.3.2. *Quadratic complexity of Transformers.*
The major computational challenge of Transformers is their high computational complexity, particularly when processing long sequences. Transformers require quadratic time and space complexity to process sequences, because each self-attention layer contains $n^2$ products of the form $\langle Qx_i, Kx_j \rangle$ (for $i, j \in [n]$). The empirical observation that the self-attention matrix $P$ is close to a low rank matrix—see [LWLQ22, Section 4.4] for references—is cited as the inspiration behind *Linformers* [WLK$^{+20}$] and the fine-tuning algorithm LoRA [HysW$^{+22}$. For both approaches, the low-rank structure is imposed rather than extracted from $P$ itself. Other methods called *sparse attention* and *block attention* have been proposed to reduce the quadratic complexity—see [WLK$^{+20}$, Section 2.2] for references. In the spirit of these works, a foreshadowing of the clustering mechanism was invoked in [VKF20], where queries are clustered into groups, again in view of reducing the quadratic complexity of self-attention. We point out that [DCL21] previously demonstrated that without skip connections, the dynamics trivializes and all tokens quickly lump together into a single tight cluster. Our work, in contrast, shows that in the presence of skip connections a rich cluster structure emerges.

Compared to the usual BERT, ALBERT [LCG$^{+20}$] uses parameter-sharing across layers, meaning that the weight matrices $Q, K, V$ in (1.1)-(1.2) do not depend on time, as in the present paper. This does not reduce the theoretical $O(n^2)$ complexity of the original Transformer, but, quoting [LCG$^{+20}$], it "significantly reduce[s] the number of parameters for BERT without seriously hurting performance, thus improving parameter-efficiency. An ALBERT configuration similar to BERT-large has 18x fewer parameters and can be trained about 1.7x faster. The parameter reduction techniques also act as a form of regularization that stabilizes the training and helps with generalization".

### 1.3.3. *Neural collapse.*
Our results and conclusions bear a resemblance to some geometric aspects of neural collapse for classification tasks [PHD20]. A key geometric aspect of neural collapse is the observation that, during the training of deep neural networks, the representation of different classes in the later layers of the network tends to form a tight cluster around the vertices of a simplex. The emergence of a simplex structure in the representation space provides insights into how the neural network organizes and separates the different classes.

### 1.3.4. *Clustering in interacting particle systems.*
The transformer dynamics (1.1) have a strong connection to the vast literature on nonlinear systems arising in the modeling of opinion dynamics and flocking phenomena. In addition to the classical Kuramoto model describing synchronization/clustering of oscillators [Kur75,
Transformers exhibit the universal approximation property. In [LLH$^{+20}$], the authors present, to the best of our knowledge, the first interacting particle systems perspective on Transformers. They then leverage the similarities between Transformers (with an additional feed-forward layer compared to (1.1)) and convection-diffusion equations to slightly improve the performance of Transformers by employing a Strang-Marchuk splitting scheme for time discretization. In [SABP22], the authors interpret system (1.1) as the characteristics of a continuity equation. Drawing on the similarities between (1.1) and Sinkhorn iterations, they propose a novel architecture dubbed *Sinkformer*, which possesses the desirable property of being a Wasserstein gradient flow.

### 1.3.2. *Quadratic complexity of Transformers.*
The major computational challenge of Transformers is their high computational complexity, particularly when processing long sequences. Transformers require quadratic time and space complexity to process sequences, because each self-attention layer contains $n^2$ products of the form $\langle Qx_i, Kx_j \rangle$ (for $i, j \in [n]$). The empirical observation that the self-attention matrix $P$ is close to a low rank matrix—see [LWLQ22, Section 4.4] for references—is cited as the inspiration behind *Linformers* [WLK$^{+20}$] and the fine-tuning algorithm LoRA [HysW$^{+22}$. For both approaches, the low-rank structure is imposed rather than extracted from $P$ itself. Other methods called *sparse attention* and *block attention* have been proposed to reduce the quadratic complexity—see [WLK$^{+20}$, Section 2.2] for references. In the spirit of these works, a foreshadowing of the clustering mechanism was invoked in [VKF20], where queries are clustered into groups, again in view of reducing the quadratic complexity of self-attention. We point out that [DCL21] previously demonstrated that without skip connections, the dynamics trivializes and all tokens quickly lump together into a single tight cluster. Our work, in contrast, shows that in the presence of skip connections a rich cluster structure emerges.

Compared to the usual BERT, ALBERT [LCG$^{+20}$] uses parameter-sharing across layers, meaning that the weight matrices $Q, K, V$ in (1.1)-(1.2) do not depend on time, as in the present paper. This does not reduce the theoretical $O(n^2)$ complexity of the original Transformer, but, quoting [LCG$^{+20}$], it "significantly reduce[s] the number of parameters for BERT without seriously hurting performance, thus improving parameter-efficiency. An ALBERT configuration similar to BERT-large has 18x fewer parameters and can be trained about 1.7x faster. The parameter reduction techniques also act as a form of regularization that stabilizes the training and helps with generalization".

### 1.3.3. *Neural collapse.*
Our results and conclusions bear a resemblance to some geometric aspects of neural collapse for classification tasks [PHD20]. A key geometric aspect of neural collapse is the observation that, during the training of deep neural networks, the representation of different classes in the later layers of the network tends to form a tight cluster around the vertices of a simplex. The emergence of a simplex structure in the representation space provides insights into how the neural network organizes and separates the different classes.

### 1.3.4. *Clustering in interacting particle systems.*
The transformer dynamics (1.1) have a strong connection to the vast literature on nonlinear systems arising in the modeling of opinion dynamics and flocking phenomena. In addition to the classical Kuramoto model describing synchronization/clustering of oscillators [Kur75,
ABV$^{+05}$], the model which is most similar to (1.1) is the Krause model [Kra00]

$$ \dot{x}_i(t) = \sum_{j=1}^{n} a_{ij}(x_j(t) - x_i(t)), \qquad a_{ij} = \frac{\phi(\|x_i - x_j\|^2)}{\sum_{k=1}^{n} \phi(\|x_i - x_k\|^2)}. $$

which is non-symmetric in general ($a_{ij} \neq a_{ji}$), much like (1.1). When $\phi$ is compactly supported, it has been shown in [JM14] that the particles $x_i(t)$ assemble in several clusters as $t \to +\infty$. Other models of opinion dynamics and flocking have been proposed and studied, among which the Vicsek model [VCBJ$^{+95}$], the Hegselmann-Krause model [HK02] and the Cucker-Smale model [CS07]. These models may also exhibit a clustering behavior under various assumptions (see [MT14, CHH$^{+16}$, HKPZ19] and the references therein). The transformer dynamics are also closely related to the dynamics employed in mean-shift clustering [Che95], and this work indirectly sheds some light on its theoretical properties.

The analysis of transformer dynamics presents unique mathematical challenges that cannot be addressed using the tools developed for these more primitive models. In particular, our work demonstrates how different choices for the parameters lead to remarkably diverse clustering patterns. Much more remains to be discovered and this work is a first attempt a rigorous mathematical analysis of these synthetic dynamics.

**Acknowledgments.** We thank Pierre Ablin, Léonard Boussioux, Enric Boix Adsera, Gabriel Peyré, Yair Shenfeld and Emmanuel Trélat for helpful discussions. C.L. was supported by the Simons Foundation Grant 601948, DJ. P.R. is supported by NSF grants IIS-1838071, DMS-2022448, and CCF-2106377. Y.P. is supported in part by the MIT-IBM Watson AI Lab.

## 2. ASYMPTOTIC LOW-RANKNESS OF THE SELF-ATTENTION MATRIX

As mentioned in Section 1.3, numerical experiments in [WLK$^{+20}$] show that the self-attention matrix $P$, defined in (1.2), has an almost low-rank structure. This observation has then been leveraged to reduce the quadratic complexity in the sequence length $n$ which is inherent to Transformers, resulting in a non-negligible decrease in the cost of training.

As a warm-up to deriving complete geometric representations of the dynamics, our first result shows, in the simple 1d case that $P(t)$ indeed converges exponentially fast toward a matrix which is typically both Boolean and low-rank (see Fig. 3). Although there are clear obstructions to a rigorous extension of this result to higher dimensions (Remark 7.9), numerical experiments appear to show that this result holds in greater generality, for almost all initial sequences (Section 11).

To set this up, we introduce the set $\mathcal{P}$ of $n \times n$ matrices having the form illustrated in Fig. 2, where the asterisks denote arbitrary non-negative real numbers which add up to 1. The row of asterisks may actually be any row between the first and the last one.

**Theorem 2.1** (Self-attention matrix converges to a low-rank Boolean matrix). Let $d = 1$. Suppose that the scalars $(Q, K, V)$ satisfy $V > 0$ and $QK > 0$. For any initial sequence of pairwise distinct tokens $(x_1(0), \dots, x_n(0)) \in \mathbb{R}^n$, there exists some $P^* \in \mathcal{P}$ such that the self-attention matrix $P(t)$ defined in (1.2) converges to $P^*$ as $t \to +\infty$.
ABV$^{+05}$], the model which is most similar to (1.1) is the Krause model [Kra00]

$$ \dot{x}_i(t) = \sum_{j=1}^{n} a_{ij}(x_j(t) - x_i(t)), \qquad a_{ij} = \frac{\phi(\|x_i - x_j\|^2)}{\sum_{k=1}^{n} \phi(\|x_i - x_k\|^2)}. $$

which is non-symmetric in general ($a_{ij} \neq a_{ji}$), much like (1.1). When $\phi$ is compactly supported, it has been shown in [JM14] that the particles $x_i(t)$ assemble in several clusters as $t \to +\infty$. Other models of opinion dynamics and flocking have been proposed and studied, among which the Vicsek model [VCBJ$^{+95}$], the Hegselmann-Krause model [HK02] and the Cucker-Smale model [CS07]. These models may also exhibit a clustering behavior under various assumptions (see [MT14, CHH$^{+16}$, HKPZ19] and the references therein). The transformer dynamics are also closely related to the dynamics employed in mean-shift clustering [Che95], and this work indirectly sheds some light on its theoretical properties.

The analysis of transformer dynamics presents unique mathematical challenges that cannot be addressed using the tools developed for these more primitive models. In particular, our work demonstrates how different choices for the parameters lead to remarkably diverse clustering patterns. Much more remains to be discovered and this work is a first attempt a rigorous mathematical analysis of these synthetic dynamics.

**Acknowledgments.** We thank Pierre Ablin, Léonard Boussioux, Enric Boix Adsera, Gabriel Peyré, Yair Shenfeld and Emmanuel Trélat for helpful discussions. C.L. was supported by the Simons Foundation Grant 601948, DJ. P.R. is supported by NSF grants IIS-1838071, DMS-2022448, and CCF-2106377. Y.P. is supported in part by the MIT-IBM Watson AI Lab.

## 2. ASYMPTOTIC LOW-RANKNESS OF THE SELF-ATTENTION MATRIX

As mentioned in Section 1.3, numerical experiments in [WLK$^{+20}$] show that the self-attention matrix $P$, defined in (1.2), has an almost low-rank structure. This observation has then been leveraged to reduce the quadratic complexity in the sequence length $n$ which is inherent to Transformers, resulting in a non-negligible decrease in the cost of training.

As a warm-up to deriving complete geometric representations of the dynamics, our first result shows, in the simple 1d case that $P(t)$ indeed converges exponentially fast toward a matrix which is typically both Boolean and low-rank (see Fig. 3). Although there are clear obstructions to a rigorous extension of this result to higher dimensions (Remark 7.9), numerical experiments appear to show that this result holds in greater generality, for almost all initial sequences (Section 11).

To set this up, we introduce the set $\mathcal{P}$ of $n \times n$ matrices having the form illustrated in Fig. 2, where the asterisks denote arbitrary non-negative real numbers which add up to 1. The row of asterisks may actually be any row between the first and the last one.

**Theorem 2.1** (Self-attention matrix converges to a low-rank Boolean matrix). Let $d = 1$. Suppose that the scalars $(Q, K, V)$ satisfy $V > 0$ and $QK > 0$. For any initial sequence of pairwise distinct tokens $(x_1(0), \dots, x_n(0)) \in \mathbb{R}^n$, there exists some $P^* \in \mathcal{P}$ such that the self-attention matrix $P(t)$ defined in (1.2) converges to $P^*$ as $t \to +\infty$.
The proof may be found in Section 7. The rate of convergence toward $P^*$ is in fact doubly exponential in $t$ for coefficients outside the row of asterisks in Fig. 2. The proof the theorem also reveals that for almost all initial sequences of pairwise distinct tokens, $P^*$ is actually of rank 1 or 2, i.e., the row of asterisks is equal to either $e_1 = (1, 0, \dots, 0)$ or $e_n = (0, \dots, 0, 1)$.

The interpretation of Theorem 2.1 is that in the 1d case, at most three tokens capture the attention of all tokens except at most one. Typically, these leading tokens are those carrying the largest amount of information. This is also illustrated in Fig. 4. Since the tokens $x_i$ here evolve on $\mathbb{R}$, the right-most and left-most ones (which typically tend toward $\pm\infty$) capture the attention of all the others.

$$P_{\sigma} = \begin{bmatrix} 1 & 0 & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 1 & 0 & \dots & 0 \\ * & * & \dots & * \\ 0 & \dots & 0 & 1 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & \dots & 0 & 1 \end{bmatrix}$$

<figure><img src="image_3.png" /><figcaption>**Figure 3.** An illustration of the asymptotics of $P(t)$ entailed by Theorem 2.1 for $n = 40$ tokens, with $Q = K = 1$ and $V = 1$. (See Section 11 for details on computing.) Increasing $n$ has no effect on this behavior of $P(t)$—see Fig. 11.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>**Figure 4.** The clouds $\{Kx_i(t)\}_{i \in [20]}$ (green) and $\{Qx_j(t)\}_{j \in [20]}$ (purple) for $d = 2$ where pairwise points of clouds are connected by a line of width equal to $P_{ij}(t)$. Here $V > 0$ and $Q > 0$ are random matrices and $K = I_2$. The creation of clusters is reflected by the rank $\le 2$ structure of the self-attention matrix $P(t)$. This interaction echoes findings illustrated in the original paper [VSP$^{+}$17]—for instance, Figures 3-5 therein.</figcaption></figure>
The proof may be found in Section 7. The rate of convergence toward $P^*$ is in fact doubly exponential in $t$ for coefficients outside the row of asterisks in Fig. 2. The proof the theorem also reveals that for almost all initial sequences of pairwise distinct tokens, $P^*$ is actually of rank 1 or 2, i.e., the row of asterisks is equal to either $e_1 = (1, 0, \dots, 0)$ or $e_n = (0, \dots, 0, 1)$.

The interpretation of Theorem 2.1 is that in the 1d case, at most three tokens capture the attention of all tokens except at most one. Typically, these leading tokens are those carrying the largest amount of information. This is also illustrated in Fig. 4. Since the tokens $x_i$ here evolve on $\mathbb{R}$, the right-most and left-most ones (which typically tend toward $\pm\infty$) capture the attention of all the others.

$$ P_{\sigma} = \begin{bmatrix} 1 & 0 & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 1 & 0 & \dots & 0 \\ * & * & \dots & * \\ 0 & \dots & 0 & 1 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & \dots & 0 & 1 \end{bmatrix} $$

<figure><img src="image_4.png" /><figcaption>**Figure 3.** An illustration of the asymptotics of $P(t)$ entailed by Theorem 2.1 for $n = 40$ tokens, with $Q = K = 1$ and $V = 1$. (See Section 11 for details on computing.) Increasing $n$ has no effect on this behavior of $P(t)$—see Fig. 11.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Figure 4.** The clouds $\{Kx_i(t)\}_{i \in [20]}$ (green) and $\{Qx_j(t)\}_{j \in [20]}$ (purple) for $d = 2$ where pairwise points of clouds are connected by a line of width equal to $P_{ij}(t)$. Here $V > 0$ and $Q > 0$ are random matrices and $K = I_2$. The creation of clusters is reflected by the rank $\le 2$ structure of the self-attention matrix $P(t)$. This interaction echoes findings illustrated in the original paper [VSP$^{+}$17]—for instance, Figures 3-5 therein.</figcaption></figure>
### 3. CLUSTERING TOWARD VERTICES OF CONVEX POLYTOPES

In the rest of the paper, we seek to taxonomize various clustering results for the solutions to (3.1) when $t \to +\infty$, depending on the sign and the multiplicity of the eigenvalues of $V$. We begin by focusing on what may appear to be the most natural<fnref n="2" /> case $V = I_d$, as is also done in [SABP22]. In fact, we demonstrate (theoretically and numerically) later on, clustering is a generic phenomenon which holds under much less restrictive assumptions.

The transformer dynamics considered in (1.1) does not contain a layer normalization mechanism typically encountered in practice [VSP+17]. In absence of such a device, tokens may diverge to infinity as in Theorem 2.1. In fact, the norm of the tokens $x_i(t)$ typically diverges exponentially toward $+\infty$ for any $d$: this is expected, by analogy with the non-trivial solutions to $\dot{y}(t) = y(t)$.

To remedy this situation, we take inspiration from the solution $y(t) = e^{tV}y(0)$ to $\dot{y}(t) = Vy(t)$. Namely, for any $i \in [n]$ we consider the rescaled tokens

$$z_i(t) := e^{-tV}x_i(t),$$

which solve

$$\dot{z}_i(t) = \sum_{j=1}^{n} \left( \frac{e^{\langle Qe^{tV}z_i(t), Ke^{tV}z_j(t) \rangle}}{\sum_{k=1}^{n} e^{\langle Qe^{tV}z_i(t), Ke^{tV}z_k(t) \rangle}} \right) V(z_j(t) - z_i(t)), \quad t \in [0, +\infty). \quad (3.1)$$

The initial condition remains the same: $x_i(0) = z_i(0)$ for any $i \in [n]$. More importantly, the coefficients of the self-attention matrix for the rescaled tokens $z_i(t)$ are the same as those for the original tokens $x_i(t)$. Whence, the conclusion of Theorem 2.1 also applies to the dynamics (3.1). We see this rescaling of tokens as a mathematically justified surrogate for the layer normalization.

The appearance of the exponential factor within the self-attention kernel facilitates the analysis of (3.1) compared to (1.1), and it is in fact instrumental in the proofs of all results that follow. Each result on the rescaled tokens $z_i(t)$ then gives information on the dynamics of the original tokens $x_i(t)$ by virtue of the relation $x_i(t) = e^{tV}z_i(t)$.

We are now able to state the main result of this section on the case $V = I_d$. The following theorem shows that the tokens $z_i(t)$ evolving per dynamics (3.1) converge to the boundary of a convex polytope as $t \to +\infty$. We present here a simplified but weaker version of our result for convenience, and refer the reader to Theorem 8.1 for a complete statement.

**Theorem 3.1** (Convergence to points on the boundary of a convex polytope). Suppose $V = I_d$ and $Q^\top K > 0$. Then, for any initial sequence of tokens $\{z_i(0)\}_{i \in [n]} \subset \mathbb{R}^d$, there exists a convex polytope $\mathcal{K} \subset \mathbb{R}^d$ such that for any $i \in [n]$, $z_i(t)$ converges either to 0 or to some point on $\partial\mathcal{K}$ as $t \to +\infty$.

The convex polytope $\mathcal{K}$ is completely determined by the initial sequence of tokens, and $Q^\top K$ (refer to Claim 1). Numerical experiments (e.g. Fig. 5) also lead us to claim that for almost all initial sequences of tokens, one should expect convergence of $z_i(t)$ ($i \in [n]$) toward some vertex of $\mathcal{K}$. (Furthermore, the number of vertices of $\mathcal{K}$ is often found to be significantly smaller than $n$.) It may however

<footnote n="2">²Note that the case $V = -I_d$ may appear equally natural. For such a choice of $V$, we show in Section 8.2 that the dynamics converge to a single cluster located at the origin. Multiplicative constants preserving the sign, i.e., $V = \pm cI_d, c > 0$ trivially yield the same conclusions.</footnote>
### 3. CLUSTERING TOWARD VERTICES OF CONVEX POLYTOPES

In the rest of the paper, we seek to taxonomize various clustering results for the solutions to (3.1) when $t \to +\infty$, depending on the sign and the multiplicity of the eigenvalues of $V$. We begin by focusing on what may appear to be the most natural<fnref n="2" /> case $V = I_d$, as is also done in [SABP22]. In fact, we demonstrate (theoretically and numerically) later on, clustering is a generic phenomenon which holds under much less restrictive assumptions.

The transformer dynamics considered in (1.1) does not contain a layer normalization mechanism typically encountered in practice [VSP+17]. In absence of such a device, tokens may diverge to infinity as in Theorem 2.1. In fact, the norm of the tokens $x_i(t)$ typically diverges exponentially toward $+\infty$ for any $d$: this is expected, by analogy with the non-trivial solutions to $\dot{y}(t) = y(t)$.

To remedy this situation, we take inspiration from the solution $y(t) = e^{tV}y(0)$ to $\dot{y}(t) = Vy(t)$. Namely, for any $i \in [n]$ we consider the rescaled tokens

$$z_i(t) := e^{-tV}x_i(t),$$

which solve

$$\dot{z}_i(t) = \sum_{j=1}^{n} \left( \frac{e^{\langle Q e^{tV} z_i(t), K e^{tV} z_j(t) \rangle}}{\sum_{k=1}^{n} e^{\langle Q e^{tV} z_i(t), K e^{tV} z_k(t) \rangle}} \right) V(z_j(t) - z_i(t)), \quad t \in [0, +\infty). \quad (3.1)$$

The initial condition remains the same: $x_i(0) = z_i(0)$ for any $i \in [n]$. More importantly, the coefficients of the self-attention matrix for the rescaled tokens $z_i(t)$ are the same as those for the original tokens $x_i(t)$. Whence, the conclusion of Theorem 2.1 also applies to the dynamics (3.1). We see this rescaling of tokens as a mathematically justified surrogate for the layer normalization.

The appearance of the exponential factor within the self-attention kernel facilitates the analysis of (3.1) compared to (1.1), and it is in fact instrumental in the proofs of all results that follow. Each result on the rescaled tokens $z_i(t)$ then gives information on the dynamics of the original tokens $x_i(t)$ by virtue of the relation $x_i(t) = e^{tV}z_i(t)$.

We are now able to state the main result of this section on the case $V = I_d$. The following theorem shows that the tokens $z_i(t)$ evolving per dynamics (3.1) converge to the boundary of a convex polytope as $t \to +\infty$. We present here a simplified but weaker version of our result for convenience, and refer the reader to Theorem 8.1 for a complete statement.

**Theorem 3.1** (Convergence to points on the boundary of a convex polytope). Suppose $V = I_d$ and $Q^\top K > 0$. Then, for any initial sequence of tokens $\{z_i(0)\}_{i \in [n]} \subset \mathbb{R}^d$, there exists a convex polytope $\mathcal{K} \subset \mathbb{R}^d$ such that for any $i \in [n]$, $z_i(t)$ converges either to 0 or to some point on $\partial\mathcal{K}$ as $t \to +\infty$.

The convex polytope $\mathcal{K}$ is completely determined by the initial sequence of tokens, and $Q^\top K$ (refer to Claim 1). Numerical experiments (e.g. Fig. 5) also lead us to claim that for almost all initial sequences of tokens, one should expect convergence of $z_i(t)$ ($i \in [n]$) toward some vertex of $\mathcal{K}$. (Furthermore, the number of vertices of $\mathcal{K}$ is often found to be significantly smaller than $n$.) It may however

<footnote n="2">²Note that the case $V = -I_d$ may appear equally natural. For such a choice of $V$, we show in Section 8.2 that the dynamics converge to a single cluster located at the origin. Multiplicative constants preserving the sign, i.e., $V = \pm cI_d, c > 0$ trivially yield the same conclusions.</footnote>
happen that for initial sequences taken in some null set (not seen when tokens are drawn at random) some tokens converge to other points of the boundary $\partial K$, namely in the interior of facets. On the other hand, for generic choices of initial sequences, we do not see a way to predict $K$ explicitly besides running the full dynamics.

$t = 0.0$

$t = 1.0$

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

$t = 2.0$

$t = 5.0$

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /><figcaption>**Figure 5.** A toy example illustrating Theorem 3.1 with $n = 40$ tokens in $\mathbb{R}^3$. Here $Q = K = I_3$. The tokens converge to one of the vertices (leaders) of the limiting convex polytope.</figcaption></figure>

Recall that the points $x_i(t) = e^t z_i(t)$ when $V = I_d$ follow the original dynamics (1.1). Akin to Theorem 2.1, this result also shows the emergence of a set of leaders (given by the vertices of $K$) attracting all tokens as $t$ grows. It has been experimentally observed (first in [VSP+17]) that in trained Transformers, tokens focus their attention on local leaders in a way that seems to reproduce the syntactic and semantic structure of sentences.

The proof of Theorem 3.1 is postponed to Section 8, and amounts to a couple of effects entailed by the dynamics. First of all, the convex hull of the particles is shrinking over time (Proposition 8.2). This is due to the fact that the distance of the particle nearest to any half-space (not containing the particles) increases with
happen that for initial sequences taken in some null set (not seen when tokens are drawn at random) some tokens converge to other points of the boundary $\partial K$, namely in the interior of facets. On the other hand, for generic choices of initial sequences, we do not see a way to predict $K$ explicitly besides running the full dynamics.

$t = 0.0$

$t = 1.0$

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

$t = 2.0$

$t = 5.0$

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /><figcaption>**Figure 5.** A toy example illustrating Theorem 3.1 with $n = 40$ tokens in $\mathbb{R}^3$. Here $Q = K = I_3$. The tokens converge to one of the vertices (leaders) of the limiting convex polytope.</figcaption></figure>

Recall that the points $x_i(t) = e^t z_i(t)$ when $V = I_d$ follow the original dynamics (1.1). Akin to Theorem 2.1, this result also shows the emergence of a set of leaders (given by the vertices of $K$) attracting all tokens as $t$ grows. It has been experimentally observed (first in [VSP+17]) that in trained Transformers, tokens focus their attention on local leaders in a way that seems to reproduce the syntactic and semantic structure of sentences.

The proof of Theorem 3.1 is postponed to Section 8, and amounts to a couple of effects entailed by the dynamics. First of all, the convex hull of the particles is shrinking over time (Proposition 8.2). This is due to the fact that the distance of the particle nearest to any half-space (not containing the particles) increases with
time. On the other hand, the convex hull ought not collapse since particles which have not concentrated near the boundary of the limiting polytope will continue to increase in magnitude until they themselves reach this boundary (Step 2 in the proof). This occurs due to the time-rescaling.

**Remark 3.2.** Assuming $Q^\top K > 0$ does not seem to be essential for our conclusions; instead, it guides the direction of the proof. To emphasize the broader validity of our conclusion beyond this specific assumption, we conducted additional experiments (refer to Section 12.1) which suggest that Theorem 3.1 (as well as Theorems 4.2 and 5.2 stated below) holds in more generality.

**Remark 3.3 (Rate of convergence).** Although Theorem 3.1 (as well as Theorems 4.2 and 5.2 stated below) does not specify a rate of convergence toward $\partial K$, we expect (and observe through numerics) that convergence happens very quickly—after few layers, most tokens are already clustered. What "few layers" means here necessarily depends on the typical modulus of the initial tokens, since the dynamics (1.1) is not invariant under multiplication of all initial conditions by a fixed real number.

**Remark 3.4 (Discrete time).** As alluded to in Remark 1.1, all our results extend to the discrete-time Transformers (1.3). Indeed, just as in the continuous-time case, there is a natural rescaled dynamics, which is the discrete analogue of (3.1): if we set $R = I_d + V\Delta t$, and assume that $R$ is invertible (which is the case for sufficiently small $\Delta t$), then $z_i(k\Delta t) = R^{-k}x_i(k\Delta t) := z_i^{[k]}$ satisfies

$$z_i^{[k+1]} = z_i^{[k]} + \Delta t \sum_{j=1}^{n} \left( \frac{e^{\langle QR^k z_i^{[k]}, KR^k z_j^{[k]} \rangle}}{\sum_{\ell=1}^{n} e^{\langle QR^k z_i^{[k]}, KR^k z_\ell^{[k]} \rangle}} \right) R^{-1} V (z_j^{[k]} - z_i^{[k]}), \quad k \in \mathbb{N}.$$

The proofs of Theorems 2.1, 8.5, 3.1, 4.2, and 5.2 carry through with straightforward modifications.

Let us provide some comments on the proof of Theorem 3.1 in the discrete-time setting, for the sake of completeness. First of all, Proposition 8.2 holds intuitively because for all integers $i \in [n]$ and $k \ge 1$,

$$z_i^{[k+1]} = \frac{1}{1 + \Delta t} \left( z_i^{[k]} + \Delta t \sum_{j=1}^{n} P_{ij}^{[k]} z_j^{[k]} \right) \in \text{conv} \left( \left\{ z_j^{[k]} \right\}_{j \in [n]} \right).$$

We then define the candidate set of limit points as in (8.6), and Claim 1 holds without any change in the statement or in the proof. Then, just as in Steps 2 and 3 in the proof of 8.1, we can first show that if $z_i^{[k]}$ is not already near some point in the candidate limit set, it will keep moving toward the boundary of the convex polytope. Finally, we can prove that tokens cannot circulate indefinitely between different points on the boundary. The combination of these arguments would establish the convergence of each token toward some point in the set given by (8.6).

## 4. CLUSTERING TOWARD HYPERPLANES

While being a natural example to consider, value matrices found empirically are much more general than $V = I_d$, which we considered in the previous section. We now turn our attention to a significantly more general setting of value matrices, which we formalize as follows.

**Definition 4.1.** We call $(Q, K, V)$ a good triple if the two following conditions are satisfied:
time. On the other hand, the convex hull ought not collapse since particles which have not concentrated near the boundary of the limiting polytope will continue to increase in magnitude until they themselves reach this boundary (Step 2 in the proof). This occurs due to the time-rescaling.

**Remark 3.2.** Assuming $Q^\top K > 0$ does not seem to be essential for our conclusions; instead, it guides the direction of the proof. To emphasize the broader validity of our conclusion beyond this specific assumption, we conducted additional experiments (refer to Section 12.1) which suggest that Theorem 3.1 (as well as Theorems 4.2 and 5.2 stated below) holds in more generality.

**Remark 3.3 (Rate of convergence).** Although Theorem 3.1 (as well as Theorems 4.2 and 5.2 stated below) does not specify a rate of convergence toward $\partial K$, we expect (and observe through numerics) that convergence happens very quickly—after few layers, most tokens are already clustered. What "few layers" means here necessarily depends on the typical modulus of the initial tokens, since the dynamics (1.1) is not invariant under multiplication of all initial conditions by a fixed real number.

**Remark 3.4 (Discrete time).** As alluded to in Remark 1.1, all our results extend to the discrete-time Transformers (1.3). Indeed, just as in the continuous-time case, there is a natural rescaled dynamics, which is the discrete analogue of (3.1): if we set $R = I_d + V\Delta t$, and assume that $R$ is invertible (which is the case for sufficiently small $\Delta t$), then $z_i(k\Delta t) = R^{-k}x_i(k\Delta t) := z_i^{[k]}$ satisfies

$$z_i^{[k+1]} = z_i^{[k]} + \Delta t \sum_{j=1}^{n} \left( \frac{e^{\langle QR^k z_i^{[k]}, KR^k z_j^{[k]} \rangle}}{\sum_{\ell=1}^{n} e^{\langle QR^k z_i^{[k]}, KR^k z_\ell^{[k]} \rangle}} \right) R^{-1} V (z_j^{[k]} - z_i^{[k]}), \quad k \in \mathbb{N}.$$

The proofs of Theorems 2.1, 8.5, 3.1, 4.2, and 5.2 carry through with straightforward modifications.

Let us provide some comments on the proof of Theorem 3.1 in the discrete-time setting, for the sake of completeness. First of all, Proposition 8.2 holds intuitively because for all integers $i \in [n]$ and $k \ge 1$,

$$z_i^{[k+1]} = \frac{1}{1 + \Delta t} \left( z_i^{[k]} + \Delta t \sum_{j=1}^{n} P_{ij}^{[k]} z_j^{[k]} \right) \in \text{conv} \left( \left\{ z_j^{[k]} \right\}_{j \in [n]} \right).$$

We then define the candidate set of limit points as in (8.6), and Claim 1 holds without any change in the statement or in the proof. Then, just as in Steps 2 and 3 in the proof of 8.1, we can first show that if $z_i^{[k]}$ is not already near some point in the candidate limit set, it will keep moving toward the boundary of the convex polytope. Finally, we can prove that tokens cannot circulate indefinitely between different points on the boundary. The combination of these arguments would establish the convergence of each token toward some point in the set given by (8.6).

## 4. CLUSTERING TOWARD HYPERPLANES

While being a natural example to consider, value matrices found empirically are much more general than $V = I_d$, which we considered in the previous section. We now turn our attention to a significantly more general setting of value matrices, which we formalize as follows.

**Definition 4.1.** We call $(Q, K, V)$ a good triple if the two following conditions are satisfied:
* the eigenvalue of V with largest modulus is real, positive, and simple; namely,
  $\lambda_1(V) > |\lambda_2(V)| \ge \dots \ge |\lambda_d(V)|$.

* $\langle Q\varphi_1, K\varphi_1 \rangle > 0$ for any $\varphi_1 \in \mathbb{R}^d$ lying on the line $\ker(V - \lambda_1(V)\mathrm{Id})$.

The second condition simply states that the quadratic form $\langle Q, K \rangle$ is positive definite along the eigenspace associated to the leading eigenvalue of $V$. Note also that if all entries of $V$ are positive, the first condition is automatically satisfied by virtue of the Perron-Frobenius theorem. In fact, this assumption is generic. On the one hand, it is satisfied by some pre-trained value matrices for ALBERT (Figure 10). On the other hand, numerical experiments indicate that a constant fraction (about 14%) of matrices from the real Ginibre ensemble in dimension $d = 128$—this proportion is known to vanish as $d \to \infty$, albeit very slowly [RS14].

Our clustering result in the setting of good triples can be summarized as follows: the coordinate $\langle z_i(t), \frac{\varphi_1}{\|\varphi_1\|} \rangle$ of any token $z_i(t)$ along the eigenspace spanned by $\varphi_1$ converges, as $t \to +\infty$, toward one among possibly 3 real scalars. Consequently, all the tokens $z_i(t)$ converge toward one among at most three parallel hyperplanes; see Fig. 6 for an illustration.

**Theorem 4.2** (Convergence toward $\le 3$ hyperplanes). Assume that $(Q, K, V)$ is a good triple in the sense of Definition 4.1. Then, for any initial sequence of tokens $\{z_i(0)\}_{i \in [n]} \subset \mathbb{R}^d$, there exist at most three parallel hyperplanes in $\mathbb{R}^d$ such that for any $i \in [n]$, the distance of the solution $z_i(t)$ to (3.1) to one of these hyperplanes converges to 0 as $t \to +\infty$.

<figure><img src="image_9.png" /><figcaption>**Figure 6.** Illustrating Theorem 4.2 with $n = 40$ tokens in $\mathbb{R}^2$. Here $Q = K = I_2$, $V$ is a random symmetric matrix with eigenvalues $\{1.35, -0.07\}$, and $\varphi_1 = (0.76, 0.65)$. The components of the tokens in the direction of $\varphi_1$ (orange arrow) cluster over time. (See Figures 13–14 for examples in $\mathbb{R}^3$.) We also observe that tokens typically cluster toward only two hyperplanes—a third one (passing through the origin) may appear for non-generic initial sequences. The hyperplanes are perpendicular to $\varphi_1$ since $V$ is diagonalizable.</figcaption></figure>

The proof may be found in Section 9. The important role played by $\lambda_1(V)$ in the dynamics may be seen in (3.1): the component of $z_i(t)$ along $\varphi_1$ determines the size of $e^{tV}z_i(t)$ in the exponent appearing in (3.1). The tokens $z_j(t)$ attracting other tokens $z_i(t)$ are those for which this component along $\varphi_1$ is largest in modulus. This attraction process forms the clusters. These leaders, as in all our results, have been empirically observed to be the ones carrying the largest amount of information in the sentence (see Supplementary material in [VSP+17]).
* the eigenvalue of V with largest modulus is real, positive, and simple; namely, $\lambda_1(V) > |\lambda_2(V)| \ge \dots \ge |\lambda_d(V)|$.

* $\langle Q\varphi_1, K\varphi_1 \rangle > 0$ for any $\varphi_1 \in \mathbb{R}^d$ lying on the line $\ker(V - \lambda_1(V)\mathrm{Id})$.

The second condition simply states that the quadratic form $\langle Q, K \rangle$ is positive definite along the eigenspace associated to the leading eigenvalue of V. Note also that if all entries of V are positive, the first condition is automatically satisfied by virtue of the Perron-Frobenius theorem. In fact, this assumption is generic. On the one hand, it is satisfied by some pre-trained value matrices for ALBERT (Figure 10). On the other hand, numerical experiments indicate that a constant fraction (about 14%) of matrices from the real Ginibre ensemble in dimension $d = 128$—this proportion is known to vanish as $d \to \infty$, albeit very slowly [RS14].

Our clustering result in the setting of good triples can be summarized as follows: the coordinate $\langle z_i(t), \frac{\varphi_1}{\|\varphi_1\|} \rangle$ of any token $z_i(t)$ along the eigenspace spanned by $\varphi_1$ converges, as $t \to +\infty$, toward one among possibly 3 real scalars. Consequently, all the tokens $z_i(t)$ converge toward one among at most three parallel hyperplanes; see Fig. 6 for an illustration.

**Theorem 4.2** (Convergence toward $\le 3$ hyperplanes). Assume that $(Q, K, V)$ is a good triple in the sense of Definition 4.1. Then, for any initial sequence of tokens $\{z_i(0)\}_{i \in [n]} \subset \mathbb{R}^d$, there exist at most three parallel hyperplanes in $\mathbb{R}^d$ such that for any $i \in [n]$, the distance of the solution $z_i(t)$ to (3.1) to one of these hyperplanes converges to 0 as $t \to +\infty$.

<figure><img src="image_10.png" /><figcaption>**Figure 6.** Illustrating Theorem 4.2 with $n = 40$ tokens in $\mathbb{R}^2$. Here $Q = K = I_2$, $V$ is a random symmetric matrix with eigenvalues $\{1.35, -0.07\}$, and $\varphi_1 = (0.76, 0.65)$. The components of the tokens in the direction of $\varphi_1$ (orange arrow) cluster over time. (See Figures 13–14 for examples in $\mathbb{R}^3$.) We also observe that tokens typically cluster toward only two hyperplanes—a third one (passing through the origin) may appear for non-generic initial sequences. The hyperplanes are perpendicular to $\varphi_1$ since $V$ is diagonalizable.</figcaption></figure>

The proof may be found in Section 9. The important role played by $\lambda_1(V)$ in the dynamics may be seen in (3.1): the component of $z_i(t)$ along $\varphi_1$ determines the size of $e^{tV}z_i(t)$ in the exponent appearing in (3.1). The tokens $z_j(t)$ attracting other tokens $z_i(t)$ are those for which this component along $\varphi_1$ is largest in modulus. This attraction process forms the clusters. These leaders, as in all our results, have been empirically observed to be the ones carrying the largest amount of information in the sentence (see Supplementary material in [VSP+17]).
Furthermore, Theorem 4.2 can also be interpreted in more classical machine
learning terms. On the one hand, it can be seen as an instance of *K*-flats clustering [BM00, Vid11]—points in the input sequence are clustered, based on their
intrinsic similarity, to at most 3 "flats" of dimension *d* − 1. On the other hand, it
ensures that for a good triple (*Q*, *K*, *V*), (3.1) generates a *linearly separable* representation of tokens.

**Beyond a single direction?** Numerical experiments (e.g., Fig. 7) indicate that a similar phenomenon emerges for more complex *V*. We formulate following conjecture which is a natural generalization of Theorem 4.2.

**Conjecture 4.3 (Codimension conjecture).** Let $k \ge 1$ be the number of eigenvalues of $V$ with positive real part. Then there exist at most three parallel Euclidean subspaces of $\mathbb{R}^d$ of codimension $k$ such that for any $i \in [n]$, the distance of $z_i(t)$ to one of these subspaces converges to 0 as $t \to +\infty$.

<figure><img src="image_10.png" /><figcaption>(a) Conjecture 4.3: low-dimensional case.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>(b) Conjecture 4.3: high-dimensional case.</figcaption></figure>

**Figure 7.** (a) $n = 40$, $d = 3$ and $Q = K = I_3$ with $V$ a random matrix with eigenvalues $\{1.96, -0.22, 0.25\}$. The $k = 2$ positive eigenvalues of $V$ generate attraction between the tokens and even convergence in the corresponding eigenspaces–this explains the codimension $k$ statement. The negative eigenvalue generates a repulsive effect between the tokens, and we see a divergence along two lines (note the different scales between the four figures). (b) $n = 256$, $d = 128$, with $(Q, K, V)$ fixed random matrices and $V$ symmetric. For each coordinate $j$ corresponding to a positive eigenvalue, the variance of the set $\{\varphi_j^*(z_i(t)): i \in [n]\}$ (shaded area) tends to 0 with $t$, while the mean (solid lines) converges to one among two real scalars: one positive (top figure), one negative (bottom) figure. Coordinates corresponding to negative eigenvalues diverge (Fig. 15).
Furthermore, Theorem 4.2 can also be interpreted in more classical machine
learning terms. On the one hand, it can be seen as an instance of *K*-flats clustering [BM00, Vid11]—points in the input sequence are clustered, based on their
intrinsic similarity, to at most 3 "flats" of dimension *d* − 1. On the other hand, it
ensures that for a good triple (*Q*, *K*, *V*), (3.1) generates a *linearly separable* representation of tokens.

**Beyond a single direction?** Numerical experiments (e.g., Fig. 7) indicate that a similar phenomenon emerges for more complex *V*. We formulate following conjecture which is a natural generalization of Theorem 4.2.

**Conjecture 4.3 (Codimension conjecture).** Let $k \ge 1$ be the number of eigenvalues of $V$ with positive real part. Then there exist at most three parallel Euclidean subspaces of $\mathbb{R}^d$ of codimension $k$ such that for any $i \in [n]$, the distance of $z_i(t)$ to one of these subspaces converges to 0 as $t \to +\infty$.

<figure><img src="image_11.png" /><figcaption>(a) Conjecture 4.3: low-dimensional case.</figcaption></figure>

<caption>(b) Conjecture 4.3: high-dimensional case.</caption>

**Figure 7.** (a) $n = 40$, $d = 3$ and $Q = K = I_3$ with $V$ a random matrix with eigenvalues $\{1.96, -0.22, 0.25\}$. The $k = 2$ positive eigenvalues of $V$ generate attraction between the tokens and even convergence in the corresponding eigenspaces–this explains the codimension $k$ statement. The negative eigenvalue generates a repulsive effect between the tokens, and we see a divergence along two lines (note the different scales between the four figures). (b) $n = 256$, $d = 128$, with $(Q, K, V)$ fixed random matrices and $V$ symmetric. For each coordinate $j$ corresponding to a positive eigenvalue, the variance of the set $\{\varphi_j^*(z_i(t)): i \in [n]\}$ (shaded area) tends to 0 with $t$, while the mean (solid lines) converges to one among two real scalars: one positive (top figure), one negative (bottom) figure. Coordinates corresponding to negative eigenvalues diverge (Fig. 15).
5. A MIX OF HYPERPLANES AND POLYTOPES

We now turn our attention to an even more general version of Theorem 4.2, which
does not require the leading eigenvalue of $V$ to be simple. The resulting theorem
can be viewed as a combination of Theorem 4.2 and Theorem 3.1. Specifically,
we assume that $V$ behaves as the identity when acting on the eigenspace of the
leading eigenvalue. This property is automatically satisfied if $V$ is normal—so that
its eigenvectors form an orthonormal basis—so we call such a $V$ *paranormal*.

**Definition 5.1.** We call $(Q, K, V)$ a good triple with multiplicity if the following conditions hold:

(i) $Q^\top K$ is positive definite: $Q^\top K > 0$;

(ii) $V$ is paranormal: there exist two linear subspaces $\mathcal{F}, \mathcal{G} \subset \mathbb{R}^d$ which are invariant under $V$, and such that $\mathcal{F} \oplus \mathcal{G} = \mathbb{R}^d$, $V_{\mathcal{F}} = \lambda \mathrm{Id}$ for $\lambda > 0$, and $\rho(V_{\mathcal{G}}) < \lambda$, where $\rho(\cdot)$ denotes the spectral radius (the maximal modulus of eigenvalues).

An example of such a $V$ is used for Fig. 8. We may now state our main result in
the setting of good triples with multiplicity. The proof may be found in Section 10.

**Theorem 5.2** (Clustering for $\lambda_1$ with multiplicity). Suppose that $(Q, K, V)$ is a good triple with multiplicity in the sense of Definition 5.1. Then, for any initial sequence $\{z_i(0)\}_{i \in [n]} \subset \mathbb{R}^d$, there exists a bounded convex polytope $\mathcal{K} \subset \mathcal{F}$ such that setting $\mathcal{H} := (\partial\mathcal{K} \cup \{0\}) \times \mathcal{G}$, for any $i \in [n]$, we have $\mathrm{dist}(z_i(t), \mathcal{H}) \to 0$ as $t \to +\infty$.

**Part 2. Proofs**

6. WELL-POSEDNESS

We collect several facts regarding the global-in-time existence and uniqueness of solutions to all systems under consideration. Throughout the remainder of the paper, we use the terminology "tokens" and "particles" interchangeably.

To prove these results, we leverage the underlying continuity equation (see (6.1)).
For the sake of future use, we prove a more general well-posedness result for the
continuity equation than what is needed in this paper.

6.1. **Notation.** We denote by $\mathcal{P}_c(\mathbb{R}^d)$ the set of compactly supported probability measures on $\mathbb{R}^d$, and by $\mathcal{P}_2(\mathbb{R}^d)$ the set of probability measures $\mu$ on $\mathbb{R}^d$ having finite second moment: $\int_{\mathbb{R}^d} \|x\|^2 d\mu(x) < +\infty$. Let $C^0(\mathbb{R}; \mathcal{P}_c(\mathbb{R}^d))$ denote the Banach space of continuous curves $\mathbb{R} \ni t \mapsto \mu(t) \in \mathcal{P}_c(\mathbb{R}^d)$. Here $\mathcal{P}_c(\mathbb{R}^d)$ is endowed with the weak topology, which coincides with the topology induced by the Wasserstein distance $W_p$ for any $p \in [1, +\infty)$.

As seen below, for compactness purposes regarding solutions to the continuity
equation, we consider an additional property on the support of such curves, sum-
marized by the following definition.

**Definition 6.1** (Equi-compactly supported curves). The set $C_{\mathrm{co}}^{0}(\mathbb{R}; \mathcal{P}_{c}(\mathbb{R}^{d}))$ consists of all elements $\mu \in C^{0}(\mathbb{R}; \mathcal{P}_{c}(\mathbb{R}^{d}))$ such that for any $t_0, t_1 \in \mathbb{R}$, there exists a compact subset $\mathcal{K} \subset \mathbb{R}^{d}$ such that $\mathrm{supp}(\mu(t)) \subset \mathcal{K}$ for any $t \in [t_0, t_1]$.

We emphasise that there exist elements in $C^0(\mathbb{R}; \mathcal{P}_c(\mathbb{R}^d))$ which do not satisfy
this property with regard to their support—e.g., $\mu(t) = (1 - e^{-\frac{t}{2}})\delta_0 + e^{-\frac{t}{2}}\delta_{\frac{1}{t}}$.
5. A MIX OF HYPERPLANES AND POLYTOPES

We now turn our attention to an even more general version of Theorem 4.2, which
does not require the leading eigenvalue of $V$ to be simple. The resulting theorem
can be viewed as a combination of Theorem 4.2 and Theorem 3.1. Specifically,
we assume that $V$ behaves as the identity when acting on the eigenspace of the
leading eigenvalue. This property is automatically satisfied if $V$ is normal—so that
its eigenvectors form an orthonormal basis—so we call such a $V$ *paranormal*.

**Definition 5.1.** We call $(Q, K, V)$ a good triple with multiplicity if the following conditions hold:

(i) $Q^\top K$ is positive definite: $Q^\top K > 0$;

(ii) $V$ is paranormal: there exist two linear subspaces $\mathcal{F}, \mathcal{G} \subset \mathbb{R}^d$ which are invariant under $V$, and such that $\mathcal{F} \oplus \mathcal{G} = \mathbb{R}^d$, $V_{\mathcal{F}} = \lambda \mathrm{Id}$ for $\lambda > 0$, and $\rho(V_{\mathcal{G}}) < \lambda$, where $\rho(\cdot)$ denotes the spectral radius (the maximal modulus of eigenvalues).

An example of such a $V$ is used for Fig. 8. We may now state our main result in
the setting of good triples with multiplicity. The proof may be found in Section 10.

**Theorem 5.2** (Clustering for $\lambda_1$ with multiplicity). Suppose that $(Q, K, V)$ is a good triple with multiplicity in the sense of Definition 5.1. Then, for any initial sequence $\{z_i(0)\}_{i \in [n]} \subset \mathbb{R}^d$, there exists a bounded convex polytope $\mathcal{K} \subset \mathcal{F}$ such that setting $\mathcal{H} := (\partial\mathcal{K} \cup \{0\}) \times \mathcal{G}$, for any $i \in [n]$, we have $\mathrm{dist}(z_i(t), \mathcal{H}) \to 0$ as $t \to +\infty$.

Part 2. Proofs

6. WELL-POSEDNESS

We collect several facts regarding the global-in-time existence and uniqueness of solutions to all systems under consideration. Throughout the remainder of the paper, we use the terminology "tokens" and "particles" interchangeably.

To prove these results, we leverage the underlying continuity equation (see (6.1)).
For the sake of future use, we prove a more general well-posedness result for the
continuity equation than what is needed in this paper.

6.1. **Notation.** We denote by $\mathcal{P}_c(\mathbb{R}^d)$ the set of compactly supported probability measures on $\mathbb{R}^d$, and by $\mathcal{P}_2(\mathbb{R}^d)$ the set of probability measures $\mu$ on $\mathbb{R}^d$ having finite second moment: $\int_{\mathbb{R}^d} \|x\|^2 d\mu(x) < +\infty$. Let $C^0(\mathbb{R}; \mathcal{P}_c(\mathbb{R}^d))$ denote the Banach space of continuous curves $\mathbb{R} \ni t \mapsto \mu(t) \in \mathcal{P}_c(\mathbb{R}^d)$. Here $\mathcal{P}_c(\mathbb{R}^d)$ is endowed with the weak topology, which coincides with the topology induced by the Wasserstein distance $W_p$ for any $p \in [1, +\infty)$.

As seen below, for compactness purposes regarding solutions to the continuity
equation, we consider an additional property on the support of such curves, sum-
marized by the following definition.

**Definition 6.1** (Equi-compactly supported curves). The set $C_{\mathrm{co}}^{0}(\mathbb{R}; \mathcal{P}_{c}(\mathbb{R}^{d}))$ consists of all elements $\mu \in C^{0}(\mathbb{R}; \mathcal{P}_{c}(\mathbb{R}^{d}))$ such that for any $t_0, t_1 \in \mathbb{R}$, there exists a compact subset $\mathcal{K} \subset \mathbb{R}^{d}$ such that $\mathrm{supp}(\mu(t)) \subset \mathcal{K}$ for any $t \in [t_0, t_1]$.

We emphasise that there exist elements in $C^0(\mathbb{R}; \mathcal{P}_c(\mathbb{R}^d))$ which do not satisfy this property with regard to their support—e.g., $\mu(t) = (1 - e^{-t/2})\delta_0 + e^{-t/2}\delta_{1/2}$.
$t = 0.0$

$t = 5.0$

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

$t = 10.0$

$t = 15.0$

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /><figcaption>**Figure 8.** Illustrating Theorem 5.2 with $n = 40$ tokens in $\mathbb{R}^3$. As before, $Q = K = I_d$, and we take $V = \text{diag}(1, 1, -\frac{1}{2})$. A convex polytope $\mathcal{K}$ emerges before time 5, toward which two coordinates of the tokens cluster, and persists throughout the evolution, while the tokens diverge along the coordinate corresponding to the eigenvalue $-\frac{1}{2}$ (note the different scales between the four figures).</figcaption></figure>

**6.2. Well-posedness of the ODEs.** For any initial datum, i.e. a sequence of $n$ points in $\mathbb{R}^d$, the dynamics (1.1) is well-posed, in the sense that it admits a unique solution defined for all times.

**Proposition 6.2.** For any initial datum $\mathbf{X}_0 = (x_1^0, \dots, x_n^0) \in (\mathbb{R}^d)^n$, there exists a unique Lipschitz continuous function $\mathbb{R} \ni t \mapsto \mathbf{X}(t) = (x_1(t), \dots, x_n(t))$ such that $x_i(\cdot)$ solves (1.1) and satisfies $x_i(0) = x_i^0$ for any $i \in [n]$.

We postpone the proof which is seen as a corollary of the well-posedness for the corresponding continuity equation. It follows that the equation (3.1) is also well-posed:

**Proposition 6.3.** For any initial datum $\mathbf{Z}_0 = (z_1^0, \dots, z_n^0) \in (\mathbb{R}^d)^n$, there exists a unique Lipschitz continuous function $\mathbb{R} \ni t \mapsto \mathbf{Z}(t) = (z_1(t), \dots, z_n(t))$ such that $z_i(\cdot)$ solves (3.1) and satisfies $z_i(0) = z_i^0$ for any $i \in [n]$.
$t = 0.0$

$t = 5.0$

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

$t = 10.0$

$t = 15.0$

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /><figcaption>**Figure 8.** Illustrating Theorem 5.2 with $n = 40$ tokens in $\mathbb{R}^3$. As before, $Q = K = I_d$, and we take $V = \text{diag}(1, 1, -\frac{1}{2})$. A convex polytope $\mathcal{K}$ emerges before time 5, toward which two coordinates of the tokens cluster, and persists throughout the evolution, while the tokens diverge along the coordinate corresponding to the eigenvalue $-\frac{1}{2}$ (note the different scales between the four figures).</figcaption></figure>

**6.2. Well-posedness of the ODEs.** For any initial datum, i.e. a sequence of $n$ points in $\mathbb{R}^d$, the dynamics (1.1) is well-posed, in the sense that it admits a unique solution defined for all times.

**Proposition 6.2.** For any initial datum $\mathbf{X}_0 = (x_1^0, \dots, x_n^0) \in (\mathbb{R}^d)^n$, there exists a unique Lipschitz continuous function $\mathbb{R} \ni t \mapsto \mathbf{X}(t) = (x_1(t), \dots, x_n(t))$ such that $x_i(\cdot)$ solves (1.1) and satisfies $x_i(0) = x_i^0$ for any $i \in [n]$.

We postpone the proof which is seen as a corollary of the well-posedness for the corresponding continuity equation. It follows that the equation (3.1) is also well-posed:

**Proposition 6.3.** For any initial datum $\mathbf{Z}_0 = (z_1^0, \dots, z_n^0) \in (\mathbb{R}^d)^n$, there exists a unique Lipschitz continuous function $\mathbb{R} \ni t \mapsto \mathbf{Z}(t) = (z_1(t), \dots, z_n(t))$ such that $z_i(\cdot)$ solves (3.1) and satisfies $z_i(0) = z_i^0$ for any $i \in [n]$.
*Proof of Proposition 6.3.* Since the equations (1.1) and (3.1) are related by the change of variables $x_i(t) = e^{tV}z_i(t)$, Proposition 6.3 is an immediate consequence of Proposition 6.2. □

**6.3. The continuity equation.** To prove Proposition 6.2, we show a more general result concerning global existence and uniqueness of solutions to the corresponding continuity equation<fnref n="3" />

$$ \begin{cases} \partial_t \mu + \operatorname{div}(\mathcal{X}[\mu]\mu) = 0 & \text{in } (0, +\infty) \times \mathbb{R}^d \\ \mu|_{t=0} = \mu_0 & \text{in } \mathbb{R}^d, \end{cases} \qquad (6.1) $$

when $\mathcal{X}[\mu]$ is the attention kernel

$$ \mathcal{X}[\mu](x) := \frac{\int_{\mathbb{R}^d} e^{\langle Qx, Ky \rangle} Vy \, d\mu(y)}{\int_{\mathbb{R}^d} e^{\langle Qx, Ky \rangle} \, d\mu(y)}. \qquad (6.2) $$

We will make use of the following notion of solution.

**Definition 6.4.** Fix $\mu_0 \in \mathcal{P}_c(\mathbb{R}^d)$. We say that $t \mapsto \mu(t) =: \mu_t$ is a solution to the Cauchy problem (6.1) if $\mu \in C_{\text{co}}^0(\mathbb{R}, \mathcal{P}_c(\mathbb{R}^d))$, the function

$$ \mathbb{R} \ni t \mapsto \int_{\mathbb{R}^d} g(x) \, d\mu_t(x) $$

is absolutely continuous for every $g \in C_c^\infty(\mathbb{R}^d)$, and

$$ \int_{\mathbb{R}^d} g(x) \, d\mu_t(x) = \int_{\mathbb{R}^d} g(x) \, d\mu_0(x) + \int_0^t \int_{\mathbb{R}^d} \langle \nabla g(x), \mathcal{X}[\mu_t](x) \rangle \, d\mu_s(x) \, ds $$

holds for almost every $t \in \mathbb{R}$.

We will make use of the following lemma regarding (6.2).

**Lemma 6.5.** For any $R > 0$ there exists a constant $C_1(R) > 0$ such that for any $\mu, \nu \in \mathcal{P}_c(\mathbb{R}^d)$ with support in $B(0, R)$,

$$ \|\mathcal{X}[\mu]\|_{L^\infty(\mathbb{R}^d; \mathbb{R}^d)} \le \|V\|_{\text{op}} R, \qquad (6.3) $$

$$ \|\nabla_x \mathcal{X}[\mu]\|_{L^\infty(\mathbb{R}^d; \mathbb{R}^d \times \mathbb{R}^d)} \le 2 \|Q^\top K\|_{\text{op}} \|V\|_{\text{op}} R^2 \qquad (6.4) $$

$$ \|\mathcal{X}[\mu](\cdot) - \mathcal{X}[\nu](\cdot)\|_{L^\infty(B(0,R); \mathbb{R}^d)} \le C_1(R)W_2(\mu, \nu). \qquad (6.5) $$

*Proof.* We henceforth set $G(x, y) := e^{\langle Qx, Ky \rangle}$. To show (6.3), since $G > 0$ we see that for any $x \in \mathbb{R}^d$,

$$ \|\mathcal{X}[\mu](x)\| \le \|V\|_{\text{op}} \frac{\int_{B(0,R)} G(x, y) \|y\| \, d\mu(y)}{\int_{B(0,R)} G(x, y) \, d\mu(y)} \le \|V\|_{\text{op}} R. $$

<footnote n="3">³which can be seen as a mean-field limit, and is sometimes also referred to as a Vlasov equation.</footnote>
*Proof of Proposition 6.3.* Since the equations (1.1) and (3.1) are related by the change of variables $x_i(t) = e^{tV}z_i(t)$, Proposition 6.3 is an immediate consequence of Proposition 6.2. □

**6.3. The continuity equation.** To prove Proposition 6.2, we show a more general result concerning global existence and uniqueness of solutions to the corresponding continuity equation<fnref n="3" />

$$ \begin{cases} \partial_t \mu + \operatorname{div}(\mathcal{X}[\mu]\mu) = 0 & \text{in } (0, +\infty) \times \mathbb{R}^d \\ \mu|_{t=0} = \mu_0 & \text{in } \mathbb{R}^d, \end{cases} \qquad (6.1) $$

when $\mathcal{X}[\mu]$ is the attention kernel

$$ \mathcal{X}[\mu](x) := \frac{\int_{\mathbb{R}^d} e^{\langle Qx, Ky \rangle} Vy \, d\mu(y)}{\int_{\mathbb{R}^d} e^{\langle Qx, Ky \rangle} \, d\mu(y)}. \qquad (6.2) $$

We will make use of the following notion of solution.

**Definition 6.4.** Fix $\mu_0 \in \mathcal{P}_c(\mathbb{R}^d)$. We say that $t \mapsto \mu(t) =: \mu_t$ is a solution to the Cauchy problem (6.1) if $\mu \in C_{\text{co}}^0(\mathbb{R}, \mathcal{P}_c(\mathbb{R}^d))$, the function

$$ \mathbb{R} \ni t \mapsto \int_{\mathbb{R}^d} g(x) \, d\mu_t(x) $$

is absolutely continuous for every $g \in C_c^\infty(\mathbb{R}^d)$, and

$$ \int_{\mathbb{R}^d} g(x) \, d\mu_t(x) = \int_{\mathbb{R}^d} g(x) \, d\mu_0(x) + \int_0^t \int_{\mathbb{R}^d} \langle \nabla g(x), \mathcal{X}[\mu_t](x) \rangle \, d\mu_s(x) \, ds $$

holds for almost every $t \in \mathbb{R}$.

We will make use of the following lemma regarding (6.2).

**Lemma 6.5.** For any $R > 0$ there exists a constant $C_1(R) > 0$ such that for any $\mu, \nu \in \mathcal{P}_c(\mathbb{R}^d)$ with support in $B(0, R)$,

$$ \|\mathcal{X}[\mu]\|_{L^\infty(\mathbb{R}^d; \mathbb{R}^d)} \le \|V\|_{\text{op}} R, \qquad (6.3) $$

$$ \|\nabla_x \mathcal{X}[\mu]\|_{L^\infty(\mathbb{R}^d; \mathbb{R}^d \times \mathbb{R}^d)} \le 2 \|Q^\top K\|_{\text{op}} \|V\|_{\text{op}} R^2 \qquad (6.4) $$

$$ \|\mathcal{X}[\mu](\cdot) - \mathcal{X}[\nu](\cdot)\|_{L^\infty(B(0,R); \mathbb{R}^d)} \le C_1(R)W_2(\mu, \nu). \qquad (6.5) $$

*Proof.* We henceforth set $G(x, y) := e^{\langle Qx, Ky \rangle}$. To show (6.3), since $G > 0$ we see that for any $x \in \mathbb{R}^d$,

$$ \|\mathcal{X}[\mu](x)\| \le \|V\|_{\text{op}} \frac{\int_{B(0,R)} G(x, y) \|y\| \, d\mu(y)}{\int_{B(0,R)} G(x, y) \, d\mu(y)} \le \|V\|_{\text{op}} R. $$

<footnote n="3">³which can be seen as a mean-field limit, and is sometimes also referred to as a Vlasov equation.</footnote>
We now show (6.4). Note that $\nabla_x \mathcal{G}(x, y) = Q^\top K y \mathcal{G}(x, y)$, thus, arguing as above, we find

$$ \begin{aligned} \|\nabla_x \mathcal{X}[\mu](x)\| &\le \frac{\int_{B(0,R)} \|\nabla_x \mathcal{G}(x, y)\| \|V y\| \mathrm{d}\mu(y)}{\int_{B(0,R)} \mathcal{G}(x, y) \mathrm{d}\mu(y)} \\ &\quad + \|V\|_{\mathrm{op}} \frac{\int_{B(0,R)} \mathcal{G}(x, y) \|y\| \mathrm{d}\mu(y)}{\int_{B(0,R)} \mathcal{G}(x, y) \mathrm{d}\mu(y)} \frac{\int_{B(0,R)} \|\nabla_x \mathcal{G}(x, y)\| \mathrm{d}\mu(y)}{\int_{B(0,R)} \mathcal{G}(x, y) \mathrm{d}\mu(y)} \\ &\le 2 \|Q^\top K\|_{\mathrm{op}} \|V\|_{\mathrm{op}} R^2. \end{aligned} $$

We finally prove (6.5). Using the fact that

$$ \int_{\mathbb{R}^d} \mathcal{G}(x, y) \mathrm{d}\mu(y) \ge \left( \inf_{(x,y) \in B(0,R)^2} \mathcal{G}(x, y) \right) \mu(B(0, R)), $$

with an analogous bound for $\nu$, we see that it suffices to bound

$$ \left| \int_{\mathbb{R}^d} \mathcal{G}(x, y) V y \mathrm{d}\mu(y) \int_{\mathbb{R}^d} \mathcal{G}(x, y) \mathrm{d}\nu(y) - \int_{\mathbb{R}^d} \mathcal{G}(x, y) V y \mathrm{d}\nu(y) \int_{\mathbb{R}^d} \mathcal{G}(x, y) \mathrm{d}\mu(y) \right| $$

from above. We rewrite this difference by making $\mu - \nu$ appear artificially, and we then use the triangle inequality along with the fact that both $\int_{\mathbb{R}^d} \mathcal{G}(x, y) V y \mathrm{d}\mu(y)$ and $\int_{\mathbb{R}^d} \mathcal{G}(x, y) \mathrm{d}\mu(y)$ are bounded from above (by $e^{\|Q^\top K\|_{\mathrm{op}} R^2} \max(1, \|V\|_{\mathrm{op}} R)$). We thus end up with the task of bounding from above the absolute values of

$$ \int_{\mathbb{R}^d} \mathcal{G}(x, y)(\mathrm{d}\nu - \mathrm{d}\mu)(y) \quad \text{and} \quad \int_{\mathbb{R}^d} \mathcal{G}(x, y)Vy(\mathrm{d}\nu - \mathrm{d}\mu)(y). \quad (6.6) $$

For the first integral, from the Kantorovich-Rubinstein duality we deduce

$$ \left| \int_{\mathbb{R}^d} \mathcal{G}(x, y)(\mathrm{d}\nu - \mathrm{d}\mu)(y) \right| \le \|\mathcal{G}(x, \cdot)\|_{C^{0,1}(B(0,R))} W_1(\mu, \nu). \quad (6.7) $$

We now recall the following inequality relating Wasserstein distances of different orders: for any $p \ge 1$ and any bounded set $B$, for all Radon measures $\mu, \nu$ supported in $B$,

$$ W_1(\mu, \nu) \le W_p(\mu, \nu) \le \mathrm{diam}(B)^{1-\frac{1}{p}} W_1(\mu, \nu)^{1/p}. \quad (6.8) $$

Using (6.8) and the fact that the Lipschitz constant $\|\mathcal{G}(x, \cdot)\|_{C^{0,1}(B(0,R))}$ is uniformly bounded for $\|x\| \le R$ by some $C_R > 0$ in (6.7), we end up with

$$ \left| \int_{\mathbb{R}^d} \mathcal{G}(x, y)(\mathrm{d}\nu - \mathrm{d}\mu)(y) \right| \le C_R W_2(\mu, \nu). $$

The same chain of inequalities applies to the second integral in (6.6) (with the additional multiplier $\|V\|_{\mathrm{op}} R$), which finally leads us to (6.5). $\square$

The following existence and uniqueness result is adapted from [PRT15, Theorem 2.3]. In fact, the result holds true for any vector field $\mathcal{X}[\mu]$ on $\mathbb{R}^d$ satisfying conditions analog to those entailed by Lemma 6.5.
We now show (6.4). Note that $\nabla_x \mathcal{G}(x, y) = Q^\top K y \mathcal{G}(x, y)$, thus, arguing as above, we find

$$ \begin{aligned} \|\nabla_x \mathcal{X}[\mu](x)\| &\le \frac{\int_{B(0,R)} \|\nabla_x \mathcal{G}(x, y)\| \|Vy\| \mathrm{d}\mu(y)}{\int_{B(0,R)} \mathcal{G}(x, y) \mathrm{d}\mu(y)} \\ &\quad + \|V\|_{\mathrm{op}} \frac{\int_{B(0,R)} \mathcal{G}(x, y) \|y\| \mathrm{d}\mu(y)}{\int_{B(0,R)} \mathcal{G}(x, y) \mathrm{d}\mu(y)} \frac{\int_{B(0,R)} \|\nabla_x \mathcal{G}(x, y)\| \mathrm{d}\mu(y)}{\int_{B(0,R)} \mathcal{G}(x, y) \mathrm{d}\mu(y)} \\ &\le 2 \|Q^\top K\|_{\mathrm{op}} \|V\|_{\mathrm{op}} R^2. \end{aligned} $$

We finally prove (6.5). Using the fact that

$$ \int_{\mathbb{R}^d} \mathcal{G}(x, y) \mathrm{d}\mu(y) \ge \left( \inf_{(x,y) \in B(0,R)^2} \mathcal{G}(x, y) \right) \mu(B(0,R)), $$

with an analogous bound for $\nu$, we see that it suffices to bound

$$ \left| \int_{\mathbb{R}^d} \mathcal{G}(x, y) Vy \mathrm{d}\mu(y) \int_{\mathbb{R}^d} \mathcal{G}(x, y) d\nu(y) - \int_{\mathbb{R}^d} \mathcal{G}(x, y) Vy \mathrm{d}\nu(y) \int_{\mathbb{R}^d} \mathcal{G}(x, y) d\mu(y) \right| $$

from above. We rewrite this difference by making $\mu - \nu$ appear artificially, and we then use the triangle inequality along with the fact that both $\int_{\mathbb{R}^d} \mathcal{G}(x, y) Vy \mathrm{d}\mu(y)$ and $\int_{\mathbb{R}^d} \mathcal{G}(x, y) d\mu(y)$ are bounded from above (by $e^{\|Q^\top K\|_{\mathrm{op}} R^2} \max(1, \|V\|_{\mathrm{op}} R)$). We thus end up with the task of bounding from above the absolute values of

$$ \int_{\mathbb{R}^d} \mathcal{G}(x, y)(d\nu - d\mu)(y) \quad \text{and} \quad \int_{\mathbb{R}^d} \mathcal{G}(x, y)Vy(d\nu - d\mu)(y). \quad (6.6) $$

For the first integral, from the Kantorovich-Rubinstein duality we deduce

$$ \left| \int_{\mathbb{R}^d} \mathcal{G}(x, y)(d\nu - d\mu)(y) \right| \le \|\mathcal{G}(x, \cdot)\|_{C^{0,1}(B(0,R))} W_1(\mu, \nu). \quad (6.7) $$

We now recall the following inequality relating Wasserstein distances of different orders: for any $p \ge 1$ and any bounded set $B$, for all Radon measures $\mu, \nu$ supported in $B$,

$$ W_1(\mu, \nu) \le W_p(\mu, \nu) \le \mathrm{diam}(B)^{1-\frac{1}{p}} W_1(\mu, \nu)^{1/p}. \quad (6.8) $$

Using (6.8) and the fact that the Lipschitz constant $\|\mathcal{G}(x, \cdot)\|_{C^{0,1}(B(0,R))}$ is uniformly bounded for $\|x\| \le R$ by some $C_R > 0$ in (6.7), we end up with

$$ \left| \int_{\mathbb{R}^d} \mathcal{G}(x, y)(d\nu - d\mu)(y) \right| \le C_R W_2(\mu, \nu). $$

The same chain of inequalities applies to the second integral in (6.6) (with the additional multiplier $\|V\|_{\mathrm{op}} R$), which finally leads us to (6.5). $\square$

The following existence and uniqueness result is adapted from [PRT15, Theorem 2.3]. In fact, the result holds true for any vector field $\mathcal{X}[\mu]$ on $\mathbb{R}^d$ satisfying conditions analog to those entailed by Lemma 6.5.
**Proposition 6.6.** For any initial condition $\mu_0 \in \mathcal{P}_c(\mathbb{R}^d)$, the Cauchy problem (6.1) admits a unique solution $\mu \in C_{\text{co}}^0(\mathbb{R}; \mathcal{P}_c(\mathbb{R}^d))$ in the sense of Definition 6.4.

Furthermore, we have the following stability estimate for solutions: for any $R > 0$ and $T > 0$, there exists a constant $C(T, R) > 0$ such that for any $\mu_0, \nu_0 \in \mathcal{P}_c(\mathbb{R}^d)$ with support in $B(0, R)$,

$$W_2(\mu(t), \nu(t)) \le e^{C(T,R)t} W_2(\mu_0, \nu_0) \quad (6.9)$$

for any $t \in [0, T]$, where $\mu(t)$ and $\nu(t)$ solve (6.1) with initial conditions $\mu_0$ and $\nu_0$ respectively.

Results of this nature can be found in the literature—see for instance [PRT15]. They are however not sufficient for our purposes. We wrote Proposition 6.6 in the $W_2$ setting instead of the usual $W_1$ setting (used for instance for the classical Do-brushin estimate [Dob79, Goll13]) because it allows to extend the results of [WHL19] without difficulty from classical ResNets to self-attention dynamics. We recall that the goal of [WHL19] is to import classical (mean-field) optimal control tools such as the Pontryagin maximum principle and the analysis of Hamilton-Jacobi-Bellman equations to deep learning, and relies heavily on $W_2$ estimates (e.g., in [WHL19, Section 4]).

*Proof of Proposition 6.6.* To ease reading, we split the proof in three parts.

**Part 1: Existence.** Fix an arbitrary $T > 0$. For $k \ge 1$, set

$$\tau_k := \frac{T}{2^k}.$$

We define a sequence of curves $\mu^k : [0, T] \to \mathcal{P}_c(\mathbb{R}^d)$ by the following scheme$^4$:

(i) $\mu^k(0) := \mu_0$;

(ii) $\mu^k(\ell\tau_k + t) := (\Phi_{\mathcal{X}[\mu^k(\ell\tau_k)]}^t)_{\#} \mu^k(\ell\tau_k)$ for $\ell \in \{0, \dots, 2^k - 1\}$ and $t \in (0, \tau_k]$,

where for any $x \in \mathbb{R}^d$, $\Phi_{\mathcal{X}[\mu^k(\ell\tau_k)]}^t(x)$ is the unique solution to the Cauchy problem

$$\begin{cases} \dot{y}(t) = \mathcal{X}[\mu^k(\ell\tau_k)](y(t)) & \text{on } [0, \tau_k] \\ y(0) = x. \end{cases}$$

(The above problem indeed has a unique solution for any $x \in \mathbb{R}^d$ by virtue of the Cauchy-Lipschitz theorem, using (6.4).) By construction, $\mu^k \in C^0([0, T]; \mathcal{P}_c(\mathbb{R}^d))$ for any $k \ge 1$.

We begin by showing that there exists a radius $R = R(T) > 0$ independent of $k$ such that $\text{supp}(\mu^k(t)) \subset B(0, R)$ for any $k \ge 1$ and $t \in [0, T]$. To this end, for any $t \in [0, T]$ and $k \ge 1$, let $R_k(t) > 0$ denote the smallest positive radius$^5$ such that $\text{supp}(\mu^k(t)) \subset B(0, R_k(t))$. We will first look to show that

$$\text{supp}(\mu^k(\ell\tau_k + t)) \subset B(0, R_k(\ell\tau_k) + t\|V\|_{\text{op}}R_k(\ell\tau_k)). \quad (6.10)$$

Let $x \in \text{supp}(\mu^k(\ell\tau_k + t))$, thus $\mu^k(\ell\tau_k + t)(B(x, \varepsilon)) > 0$ for any $\varepsilon > 0$. By the change of variables formula, we find that

$$\int_{(\Phi_{\mathcal{X}[\mu^k(\ell\tau_k)]}^t)^{-1}(B(x, \varepsilon))} d\mu^k(\ell\tau_k)(z) > 0.$$

<footnote>$^4$In other words we "freeze" the vector field $\mathcal{X}$ on each interval of the form $[\ell\tau_k, (\ell+1)\tau_k)$, and during this time interval, we follow the flow generated by this vector field starting from $\mu^k(\ell\tau_k)$.</footnote>

<footnote>$^5$This radius always exists, since $\mu^k(t)$ is compactly supported.</footnote>
**Proposition 6.6.** For any initial condition $\mu_0 \in \mathcal{P}_c(\mathbb{R}^d)$, the Cauchy problem (6.1) admits a unique solution $\mu \in C_{\text{co}}^0(\mathbb{R}; \mathcal{P}_c(\mathbb{R}^d))$ in the sense of Definition 6.4.

Furthermore, we have the following stability estimate for solutions: for any $R > 0$ and $T > 0$, there exists a constant $C(T, R) > 0$ such that for any $\mu_0, \nu_0 \in \mathcal{P}_c(\mathbb{R}^d)$ with support in $B(0, R)$,

$$W_2(\mu(t), \nu(t)) \le e^{C(T,R)t} W_2(\mu_0, \nu_0) \quad (6.9)$$

for any $t \in [0, T]$, where $\mu(t)$ and $\nu(t)$ solve (6.1) with initial conditions $\mu_0$ and $\nu_0$ respectively.

Results of this nature can be found in the literature—see for instance [PRT15]. They are however not sufficient for our purposes. We wrote Proposition 6.6 in the $W_2$ setting instead of the usual $W_1$ setting (used for instance for the classical Do-brushin estimate [Dob79, Goll13]) because it allows to extend the results of [WHL19] without difficulty from classical ResNets to self-attention dynamics. We recall that the goal of [WHL19] is to import classical (mean-field) optimal control tools such as the Pontryagin maximum principle and the analysis of Hamilton-Jacobi-Bellman equations to deep learning, and relies heavily on $W_2$ estimates (e.g., in [WHL19, Section 4]).

*Proof of Proposition 6.6.* To ease reading, we split the proof in three parts.

**Part 1: Existence.** Fix an arbitrary $T > 0$. For $k \ge 1$, set

$$\tau_k := \frac{T}{2^k}.$$

We define a sequence of curves $\mu^k : [0, T] \to \mathcal{P}_c(\mathbb{R}^d)$ by the following scheme$^4$:

(i) $\mu^k(0) := \mu_0$;

(ii) $\mu^k(\ell\tau_k + t) := (\Phi_{\mathcal{X}[\mu^k(\ell\tau_k)]}^t)_{\#} \mu^k(\ell\tau_k)$ for $\ell \in \{0, \dots, 2^k - 1\}$ and $t \in (0, \tau_k]$,

where for any $x \in \mathbb{R}^d$, $\Phi_{\mathcal{X}[\mu^k(\ell\tau_k)]}^t(x)$ is the unique solution to the Cauchy problem

$$\begin{cases} \dot{y}(t) = \mathcal{X}[\mu^k(\ell\tau_k)](y(t)) & \text{on } [0, \tau_k] \\ y(0) = x. \end{cases}$$

(The above problem indeed has a unique solution for any $x \in \mathbb{R}^d$ by virtue of the Cauchy-Lipschitz theorem, using (6.4).) By construction, $\mu^k \in C^0([0, T]; \mathcal{P}_c(\mathbb{R}^d))$ for any $k \ge 1$.

We begin by showing that there exists a radius $R = R(T) > 0$ independent of $k$ such that $\text{supp}(\mu^k(t)) \subset B(0, R)$ for any $k \ge 1$ and $t \in [0, T]$. To this end, for any $t \in [0, T]$ and $k \ge 1$, let $R_k(t) > 0$ denote the smallest positive radius$^5$ such that $\text{supp}(\mu^k(t)) \subset B(0, R_k(t))$. We will first look to show that

$$\text{supp}(\mu^k(\ell\tau_k + t)) \subset B(0, R_k(\ell\tau_k) + t\|V\|_{\text{op}}R_k(\ell\tau_k)). \quad (6.10)$$

Let $x \in \text{supp}(\mu^k(\ell\tau_k + t))$, thus $\mu^k(\ell\tau_k + t)(B(x, \varepsilon)) > 0$ for any $\varepsilon > 0$. By the change of variables formula, we find that

$$\int_{(\Phi_{\mathcal{X}[\mu^k(\ell\tau_k)]}^t)^{-1}(B(x, \varepsilon))} d\mu^k(\ell\tau_k)(z) > 0.$$

<footnote>$^4$In other words we "freeze" the vector field $\mathcal{X}$ on each interval of the form $[\ell\tau_k, (\ell + 1)\tau_k)$, and during this time interval, we follow the flow generated by this vector field starting from $\mu^k(\ell\tau_k)$.</footnote>

<footnote>$^5$This radius always exists, since $\mu^k(t)$ is compactly supported.</footnote>
Consequently $(\Phi_{\mathcal{X}[\mu^k(\ell\tau_k)]}^t)^{-1}(B(x, \varepsilon)) \cap \text{supp}(\mu^k(\ell\tau_k)) \neq \emptyset$, and let $z$ be an element lying in this set. From the Duhamel formula, we gather that

$$ \Phi_{\mathcal{X}[\mu^k(\ell\tau_k)]}^t(z) := y(t) = z + \int_0^t \mathcal{X}[\mu^k(\ell\tau_k)](y(s)) \, ds. $$

Since $z \in (\Phi_{\mathcal{X}[\mu^k(\ell\tau_k)]}^t)^{-1}(B(x, \varepsilon))$, we find that

$$ \left\| z + \int_0^t \mathcal{X}[\mu^k(\ell\tau_k)](y(s)) \, ds - x \right\| \le \varepsilon. $$

Using the triangle inequality, (6.3), and since $z \in \text{supp}(\mu^k(\ell\tau_k))$ implies $z \in B(0, R_k(\ell\tau_k))$, we deduce that

$$ \|x\| \le \varepsilon + t \|V\|_{\text{op}} R_k(\ell\tau_k) + R_k(\ell\tau_k). $$

Since $\varepsilon > 0$ is arbitrary, this inequality yields (6.10). We now use (6.10) to prove the original claim. Using the definition of the radius $R_k(t)$, we evaluate (6.10) at $t = \tau_k$ and find

$$ R_k((\ell + 1)\tau_k) \le (1 + \|V\|_{\text{op}}\tau_k)R_k(\ell\tau_k). $$

By induction, we deduce that

$$ R_k(\ell\tau_k) \le (1 + \|V\|_{\text{op}}\tau_k)^\ell R_k(0), $$

whence

$$ R_k(\ell\tau_k) \le \left( 1 + \|V\|_{\text{op}} \frac{T}{2k} \right)^{2k} R_k(0) < e^{\|V\|_{\text{op}} T} R_0, $$

where $R_0 > 0$ denotes the smallest positive radius such that $\text{supp}(\mu_0) \subset B(0, R_0)$. Since the above bound is independent of $k$, the claim follows, yielding the desired radius $R = R(T) > 0$ bounding the support of every element in the sequence. In turn, we also deduce that $\mu^k \in C_{\text{co}}^0(\mathbb{R}; \mathcal{P}_c(\mathbb{R}^d))$ for any $k \ge 1$.

Using the above fact, along with (6.3) and the definition of $\mu^k(\ell\tau_k + t)$, we find that

$$ W_2(\mu^k(\ell\tau_k + t), \mu^k(\ell\tau_k)) \le \|V\|_{\text{op}}Rt $$

for any $\ell \in \{0, \dots, 2^k - 1\}$, $t \in (0, \tau_k]$ and $k \ge 1$. Gluing these inequalities (for different $\ell$ and $t$) with the triangle inequality yields

$$ W_2(\mu^k(t), \mu^k(s)) \le \|V\|_{\text{op}}R|t - s| $$

for any $t \in [0, T]$. Since $\mu^k(0) = \mu_0$ for any $k \ge 1$, and since $\mathcal{P}_2(\mathbb{R}^d)$ is the completion of $\mathcal{P}_c$ for the Wasserstein distance $W_2$, the Arzelà-Ascoli theorem implies the existence of a subsequence uniformly converging to some $\mu^* \in C^0([0, T]; \mathcal{P}_2(\mathbb{R}^d))$. Since for any $t \in [0, T]$ the curves $\mu^k(t)$ have their support enclosed in $B(0, R)$ for any $k \ge 1$, we even deduce that $\mu^* \in C_{\text{co}}^0(\mathbb{R}, \mathcal{P}_c(\mathbb{R}^d))$. Note moreover that $\mu^*(0) = \mu_0$ and that

$$ W_2(\mu^*(t), \mu^*(s)) \le \|V\|_{\text{op}}R|t - s| $$

for any $t, s \in [0, T]$.

The fact that $\mu^*$ is a solution of (6.1) follows exactly from the same computations as in [PRT15, p. 4711-4712], starting from (A.2) therein. We do not reproduce here this argument since the computations are the same word for word. The fact that for any $T > 0$ we have $\sup_{t \in [0, T]} W_1(\mu^*(t), \mu^k(t)) \to 0$ as $k \to +\infty$, which is instrumental in [PRT15, p. 4711-4712], follows in our case from the left-hand-side of (6.8).
Consequently $(\Phi_{\mathcal{X}[\mu^k(\ell\tau_k)]}^t)^{-1}(B(x, \varepsilon)) \cap \text{supp}(\mu^k(\ell\tau_k)) \neq \emptyset$, and let $z$ be an element lying in this set. From the Duhamel formula, we gather that

$$ \Phi_{\mathcal{X}[\mu^k(\ell\tau_k)]}^t(z) := y(t) = z + \int_0^t \mathcal{X}[\mu^k(\ell\tau_k)](y(s)) \, ds. $$

Since $z \in (\Phi_{\mathcal{X}[\mu^k(\ell\tau_k)]}^t)^{-1}(B(x, \varepsilon))$, we find that

$$ \left\| z + \int_0^t \mathcal{X}[\mu^k(\ell\tau_k)](y(s)) \, ds - x \right\| \le \varepsilon. $$

Using the triangle inequality, (6.3), and since $z \in \text{supp}(\mu^k(\ell\tau_k))$ implies $z \in B(0, R_k(\ell\tau_k))$, we deduce that

$$ \|x\| \le \varepsilon + t \|V\|_{\text{op}} R_k(\ell\tau_k) + R_k(\ell\tau_k). $$

Since $\varepsilon > 0$ is arbitrary, this inequality yields (6.10). We now use (6.10) to prove the original claim. Using the definition of the radius $R_k(t)$, we evaluate (6.10) at $t = \tau_k$ and find

$$ R_k((\ell + 1)\tau_k) \le (1 + \|V\|_{\text{op}}\tau_k)R_k(\ell\tau_k). $$

By induction, we deduce that

$$ R_k(\ell\tau_k) \le (1 + \|V\|_{\text{op}}\tau_k)^\ell R_k(0), $$

whence

$$ R_k(\ell\tau_k) \le \left( 1 + \|V\|_{\text{op}} \frac{T}{2k} \right)^{2k} R_k(0) < e^{\|V\|_{\text{op}} T} R_0, $$

where $R_0 > 0$ denotes the smallest positive radius such that $\text{supp}(\mu_0) \subset B(0, R_0)$. Since the above bound is independent of $k$, the claim follows, yielding the desired radius $R = R(T) > 0$ bounding the support of every element in the sequence. In turn, we also deduce that $\mu^k \in C_{\text{co}}^0(\mathbb{R}; \mathcal{P}_c(\mathbb{R}^d))$ for any $k \ge 1$.

Using the above fact, along with (6.3) and the definition of $\mu^k(\ell\tau_k + t)$, we find that

$$ W_2(\mu^k(\ell\tau_k + t), \mu^k(\ell\tau_k)) \le \|V\|_{\text{op}}Rt $$

for any $\ell \in \{0, \dots, 2^k - 1\}$, $t \in (0, \tau_k]$ and $k \ge 1$. Gluing these inequalities (for different $\ell$ and $t$) with the triangle inequality yields

$$ W_2(\mu^k(t), \mu^k(s)) \le \|V\|_{\text{op}}R|t - s| $$

for any $t \in [0, T]$. Since $\mu^k(0) = \mu_0$ for any $k \ge 1$, and since $\mathcal{P}_2(\mathbb{R}^d)$ is the completion of $\mathcal{P}_c$ for the Wasserstein distance $W_2$, the Arzelà-Ascoli theorem implies the existence of a subsequence uniformly converging to some $\mu^* \in C^0([0, T]; \mathcal{P}_2(\mathbb{R}^d))$. Since for any $t \in [0, T]$ the curves $\mu^k(t)$ have their support enclosed in $B(0, R)$ for any $k \ge 1$, we even deduce that $\mu^* \in C_{\text{co}}^0(\mathbb{R}, \mathcal{P}_c(\mathbb{R}^d))$. Note moreover that $\mu^*(0) = \mu_0$ and that

$$ W_2(\mu^*(t), \mu^*(s)) \le \|V\|_{\text{op}}R|t - s| $$

for any $t, s \in [0, T]$.

The fact that $\mu^*$ is a solution of (6.1) follows exactly from the same computations as in [PRT15, p. 4711-4712], starting from (A.2) therein. We do not reproduce here this argument since the computations are the same word for word. The fact that for any $T > 0$ we have $\sup_{t \in [0, T]} W_1(\mu^*(t), \mu^k(t)) \to 0$ as $k \to +\infty$, which is instrumental in [PRT15, p. 4711-4712], follows in our case from the left-hand-side of (6.8).
**Part 2: Uniqueness.** Regarding uniqueness, we proceed as follows. We first recall the following estimate from [PR16, Proposition 4]. Let $p \ge 1$, let $t \ge 0$, let $v, w \in C^{0,1} \cap L^\infty([0,t] \times \mathbb{R}^d; \mathbb{R}^d)$ (both with Lipschitz constant $L > 0$, say), and let $\mu, \nu \in \mathcal{P}_c(\mathbb{R}^d)$. Then

$$ W_p((\Phi_v^t)_{\#}\mu, (\Phi_w^t)_{\#}\nu) \le e^{\frac{p+1}{p}Lt} W_p(\mu, \nu) + \frac{e^{Lt}(e^{Lt}-1)}{L} \|v-w\|_{L^\infty([0,t] \times \mathbb{R}^d; \mathbb{R}^d)}. \quad (6.11) $$

Now assume that there are two solutions $\mu$ and $\nu$ of (6.1), with a spatial support that is locally bounded in time, and having the same initial condition. Define $v(t,x) := \mathcal{X}[\mu(t)](x)$ and $w(t,x) := \mathcal{X}[\nu(t)](x)$. Also set

$$ t_0 := \inf\{t \ge 0: W_2(\mu(t), \nu(t)) \ne 0\}, $$

and assume that $t_0 \ne +\infty$. Fix $T > t_0$ and take $R > 0$ such that $\mu_t$ and $\nu_t$ are supported in $B(0, R)$ for any $t \in [0, T]$. Using (6.11) with $p = 2$, and setting $C_2(R) := 2\|Q^\top K\|_{\text{op}} \|V\|_{\text{op}} R^2$ in (6.4), we find

$$ W_2(\mu(t_0 + s), \nu(t_0 + s)) \le e^{2C_2(R)s} W_2(\mu(t_0), \nu(t_0)) \\ + e^{C_2(R)s} \frac{e^{C_2(R)s} - 1}{C_2(R)} \sup_{\tau \in [t_0, t_0+s]} \|v(\tau, \cdot) - w(\tau, \cdot)\|_{L^\infty(\mathbb{R}^d)}. $$

Choose $s > 0$ sufficiently small so that $e^{C_2(R)s} - 1 \le 2C_2(R)s$. Then, by virtue of (6.5) and the fact that $W_2(\mu(t_0), \nu(t_0)) = 0$, we deduce

$$ W_2(\mu(t_0 + s), \nu(t_0 + s)) \le 2se^{C_2(R)s} \sup_{\tau \in [t_0, t_0+s]} W_2(\mu(\tau), \nu(\tau)). \quad (6.12) $$

We choose $s' > 0$ satisfying both $e^{C_2(R)s'} - 1 \le 2C_2(R)s'$ and $2s'e^{C_2(R)s'} < 1$. Applying (6.12) to every $s \in [0, s']$ we obtain

$$ \sup_{s \in [0, s')} W_2(\mu(t_0 + s), \nu(t_0 + s)) \le 2s'e^{C_2(R)s'} \sup_{\tau \in [t_0, t_0 + s')} W_2(\mu(\tau), \nu(\tau)) \\ < \sup_{s \in [0, s')} W_2(\mu(t_0 + s), \nu(t_0 + s)), $$

which is a contradiction. Therefore $\mu(t) \equiv \nu(t)$ for any $t \ge 0$, which proves uniqueness, as desired.

**Part 3: Stability.** We do not detail the proof of estimate (6.9), which is very similar to the proof of (2.3) in Theorem 2.3 of [PRT15]: it follows from (6.11) with $p = 2$, and the argument after (A.7) in [PRT15], with $W_2$ instead of $W_1$. See also [PR13, Theorem 3]. □

We conclude this section with the proof of Proposition 6.2, which follows as a corollary of the above derivations.

*Proof of Proposition 6.2.* We first show existence. We apply Proposition 6.6 with $\mu_0 := \frac{1}{n} \sum_{j=1}^n \delta_{x_i^0}$, which in turn yields a solution $\mu(t)$ to (6.1). Following the proof of Proposition 6.6, we also know that this solution satisfies $\mu(t) = (\Phi_{\mathcal{X}[\mu(t)]}^t)_{\#}\mu_0$ for any $t \in \mathbb{R}$, and the vector field $\mathcal{X}[\mu(t)]$ satisfies the assumptions of the Cauchy-Lipschitz theorem. In particular, $\mu(t)$ is of the form $\mu(t) = \frac{1}{n} \sum_{j=1}^n \delta_{x_i(t)}$ for some Lipschitz curves $\mathbb{R} \ni t \mapsto x_i(t)$, for $i \in [n]$. Then $t \mapsto \mu(t) = \frac{1}{n} \sum_{j=1}^n \delta_{x_i(t)}$ is a solution to the Cauchy problem (6.1)-(6.2) in the sense of Definition 6.4.
**Part 2: Uniqueness.** Regarding uniqueness, we proceed as follows. We first recall the following estimate from [PR16, Proposition 4]. Let $p \ge 1$, let $t \ge 0$, let $v, w \in C^{0,1} \cap L^\infty([0,t] \times \mathbb{R}^d; \mathbb{R}^d)$ (both with Lipschitz constant $L > 0$, say), and let $\mu, \nu \in \mathcal{P}_c(\mathbb{R}^d)$. Then

$$W_p((\Phi_v^t)_{\#}\mu, (\Phi_w^t)_{\#}\nu) \le e^{\frac{p+1}{p}Lt} W_p(\mu, \nu) + \frac{e^{Lt}(e^{Lt}-1)}{L} \|v-w\|_{L^\infty([0,t] \times \mathbb{R}^d; \mathbb{R}^d)} \quad (6.11)$$

Now assume that there are two solutions $\mu$ and $\nu$ of (6.1), with a spatial support that is locally bounded in time, and having the same initial condition. Define $v(t, x) := \mathcal{X}[\mu(t)](x)$ and $w(t, x) := \mathcal{X}[\nu(t)](x)$. Also set

$$t_0 := \inf\{t \ge 0: W_2(\mu(t), \nu(t)) \ne 0\},$$

and assume that $t_0 \ne +\infty$. Fix $T > t_0$ and take $R > 0$ such that $\mu_t$ and $\nu_t$ are supported in $B(0, R)$ for any $t \in [0, T]$. Using (6.11) with $p = 2$, and setting $C_2(R) := 2\|Q^\top K\|_{\text{op}} \|V\|_{\text{op}} R^2$ in (6.4), we find

$$W_2(\mu(t_0 + s), \nu(t_0 + s)) \le e^{2C_2(R)s} W_2(\mu(t_0), \nu(t_0)) \\ + e^{C_2(R)s} \frac{e^{C_2(R)s} - 1}{C_2(R)} \sup_{\tau \in [t_0, t_0+s]} \|v(\tau, \cdot) - w(\tau, \cdot)\|_{L^\infty(\mathbb{R}^d)}.$$

Choose $s > 0$ sufficiently small so that $e^{C_2(R)s} - 1 \le 2C_2(R)s$. Then, by virtue of (6.5) and the fact that $W_2(\mu(t_0), \nu(t_0)) = 0$, we deduce

$$W_2(\mu(t_0 + s), \nu(t_0 + s)) \le 2s e^{C_2(R)s} \sup_{\tau \in [t_0, t_0+s]} W_2(\mu(\tau), \nu(\tau)). \quad (6.12)$$

We choose $s' > 0$ satisfying both $e^{C_2(R)s'} - 1 \le 2C_2(R)s'$ and $2s'e^{C_2(R)s'} < 1$. Applying (6.12) to every $s \in [0, s']$ we obtain

$$\sup_{s \in [0, s']} W_2(\mu(t_0 + s), \nu(t_0 + s)) \le 2s' e^{C_2(R)s'} \sup_{\tau \in [t_0, t_0 + s')} W_2(\mu(\tau), \nu(\tau)) \\ < \sup_{s \in [0, s']} W_2(\mu(t_0 + s), \nu(t_0 + s)),$$

which is a contradiction. Therefore $\mu(t) \equiv \nu(t)$ for any $t \ge 0$, which proves uniqueness, as desired.

**Part 3: Stability.** We do not detail the proof of estimate (6.9), which is very similar to the proof of (2.3) in Theorem 2.3 of [PRT15]: it follows from (6.11) with $p = 2$, and the argument after (A.7) in [PRT15], with $W_2$ instead of $W_1$. See also [PR13, Theorem 3]. □

We conclude this section with the proof of Proposition 6.2, which follows as a corollary of the above derivations.

*Proof of Proposition 6.2.* We first show existence. We apply Proposition 6.6 with $\mu_0 := \frac{1}{n} \sum_{j=1}^n \delta_{x_i^0}$, which in turn yields a solution $\mu(t)$ to (6.1). Following the proof of Proposition 6.6, we also know that this solution satisfies $\mu(t) = (\Phi_{\mathcal{X}[\mu(t)]}^t)_{\#}\mu_0$ for any $t \in \mathbb{R}$, and the vector field $\mathcal{X}[\mu(t)]$ satisfies the assumptions of the Cauchy-Lipschitz theorem. In particular, $\mu(t)$ is of the form $\mu(t) = \frac{1}{n} \sum_{j=1}^n \delta_{x_i(t)}$ for some Lipschitz curves $\mathbb{R} \ni t \mapsto x_i(t)$, for $i \in [n]$. Then $t \mapsto \mu(t) = \frac{1}{n} \sum_{j=1}^n \delta_{x_i(t)}$ is a solution to the Cauchy problem (6.1)-(6.2) in the sense of Definition 6.4.
Secondly, we show uniqueness. Suppose that $\mathbf{X}(t) = (x_1(t), \dots, x_n(t))$ and $\mathbf{X}^*(t)$ are two Lipschitz solutions to (1.1), with the same initial conditions. Then for a.e. $t \ge 0$, using the equation (1.1) and the fact that the attention matrix coefficients $P_{ij}(t)$ defined in (1.2) belong to $[0, 1]$, we obtain

$$ \frac{1}{2} \frac{d}{dt} \max_{i \in [n]} \|x_i(t)\|^2 \le \|V\|_{\text{op}} \max_{i \in [n]} \|x_i(t)\|^2 $$

(and analogously for $x_i^*(t)$). Using Grönwall's inequality, we deduce the existence of two constants $c_1, c_2 > 0$ such that for any $t > 0$ and for any $i \in [n]$, $\|x_i(t)\|$ and $\|x_i^*(t)\|$ are bounded from above by $c_1 e^{c_2 t}$. It then follows that the empirical measures $\mu(\cdot) = \frac{1}{n} \sum_{j=1}^n \delta_{x_i(\cdot)}$ and $\mu^*(\cdot) = \frac{1}{n} \sum_{j=1}^n \delta_{x_i^*(\cdot)}$ belong to $C_{\text{co}}^0(\mathbb{R}, \mathcal{P}_c(\mathbb{R}^d))$. Moreover, they satisfy $\mu(t) = (\Phi_{\chi[\mu(t)]}^t)^{\#\mu_0}$ and $\mu^*(t) = (\Phi_{\chi[\mu^*(t)]}^t)^{\#\mu_0}$ and are thus solutions to (6.1). Using the uniqueness result of Proposition 6.6, we obtain that $\mu = \mu^*$ which concludes the proof. $\square$

## 7. PROOF OF THEOREM 2.1

Throughout this section we focus on the following dynamics:

$$ \dot{x}_i(t) = \sum_{j=1}^{n} \left( \frac{e^{\langle x_i(t), x_j(t) \rangle}}{\sum_{k=1}^{n} e^{\langle x_i(t), x_k(t) \rangle}} \right) x_j(t). \quad (7.1) $$

Note that for $d = 1$, the dot products in (7.1) are just multiplications of scalars. We begin with the following observation, which holds for any $d \ge 1$.

**Lemma 7.1.** For any $x_1, \dots, x_n \in \mathbb{R}^d$, the function $f: \mathbb{R}^d \to \mathbb{R}$ defined by

$$ f: x \mapsto \log \left( \sum_{j=1}^{n} e^{\langle x, x_j \rangle} \right) \quad (7.2) $$

is convex.

*Proof.* Using the elementary inequality $(a+b) \ge 2(ab)^{\frac{1}{2}}$ for any $a, b \ge 0$, we have

$$ \begin{align} \exp(f(x) + f(y)) &= \left( \sum_{j=1}^{n} \exp(\langle x, x_j \rangle) \right) \left( \sum_{j=1}^{n} \exp(\langle y, x_j \rangle) \right) \nonumber \\ &= \frac{1}{2} \sum_{j=1}^{n} \sum_{k=1}^{n} \left[ \exp(\langle x, x_j \rangle + \langle y, x_k \rangle) + \exp(\langle x, x_k \rangle + \langle y, x_j \rangle) \right] \tag{7.3} \end{align} $$

$$ \ge \sum_{j=1}^{n} \sum_{k=1}^{n} \exp \left( \left\langle \frac{x+y}{2}, x_j + x_k \right\rangle \right) \quad (7.4) $$

$$ = \exp \left( 2f \left( \frac{x+y}{2} \right) \right). $$

Taking the log on both sides yields the statement. $\square$

The following lemma also holds for any $d \ge 1$.

**Lemma 7.2.** Let $\mathbb{R} \ni t \mapsto \{x_i(t)\}_{i \in [n]}$ be a solution to (7.1). Then for any $i, j \in [n]$, the map $\mathbb{R} \ni t \mapsto \|x_i(t) - x_j(t)\|$ is non-decreasing.
Secondly, we show uniqueness. Suppose that $\mathbf{X}(t) = (x_1(t), \dots, x_n(t))$ and $\mathbf{X}^*(t)$ are two Lipschitz solutions to (1.1), with the same initial conditions. Then for a.e. $t \ge 0$, using the equation (1.1) and the fact that the attention matrix coefficients $P_{ij}(t)$ defined in (1.2) belong to $[0, 1]$, we obtain

$$ \frac{1}{2} \frac{d}{dt} \max_{i \in [n]} \|x_i(t)\|^2 \le \|V\|_{\text{op}} \max_{i \in [n]} \|x_i(t)\|^2 $$

(and analogously for $x_i^*(t)$). Using Grönwall's inequality, we deduce the existence of two constants $c_1, c_2 > 0$ such that for any $t > 0$ and for any $i \in [n]$, $\|x_i(t)\|$ and $\|x_i^*(t)\|$ are bounded from above by $c_1 e^{c_2 t}$. It then follows that the empirical measures $\mu(\cdot) = \frac{1}{n} \sum_{j=1}^n \delta_{x_i(\cdot)}$ and $\mu^*(\cdot) = \frac{1}{n} \sum_{j=1}^n \delta_{x_i^*(\cdot)}$ belong to $C_{\text{co}}^0(\mathbb{R}, \mathcal{P}_c(\mathbb{R}^d))$. Moreover, they satisfy $\mu(t) = (\Phi_{\chi[\mu(t)]}^t)^{\#\mu_0}$ and $\mu^*(t) = (\Phi_{\chi[\mu^*(t)]}^t)^{\#\mu_0}$ and are thus solutions to (6.1). Using the uniqueness result of Proposition 6.6, we obtain that $\mu = \mu^*$ which concludes the proof. $\square$

## 7. PROOF OF THEOREM 2.1

Throughout this section we focus on the following dynamics:

$$ \dot{x}_i(t) = \sum_{j=1}^{n} \left( \frac{e^{\langle x_i(t), x_j(t) \rangle}}{\sum_{k=1}^{n} e^{\langle x_i(t), x_k(t) \rangle}} \right) x_j(t). \quad (7.1) $$

Note that for $d=1$, the dot products in (7.1) are just multiplications of scalars. We begin with the following observation, which holds for any $d \ge 1$.

**Lemma 7.1.** For any $x_1, \dots, x_n \in \mathbb{R}^d$, the function $f: \mathbb{R}^d \to \mathbb{R}$ defined by

$$ f: x \mapsto \log \left( \sum_{j=1}^{n} e^{\langle x, x_j \rangle} \right) \quad (7.2) $$

is convex.

*Proof.* Using the elementary inequality $(a+b) \ge 2(ab)^{\frac{1}{2}}$ for any $a, b \ge 0$, we have

$$ \begin{aligned} \exp(f(x) + f(y)) &= \left(\sum_{j=1}^{n} \exp(\langle x, x_j \rangle)\right) \left(\sum_{j=1}^{n} \exp(\langle y, x_j \rangle)\right) \\ &= \frac{1}{2} \sum_{j=1}^{n} \sum_{k=1}^{n} \left[ \exp(\langle x, x_j \rangle + \langle y, x_k \rangle) + \exp(\langle x, x_k \rangle + \langle y, x_j \rangle) \right] \end{aligned} \quad (7.3) $$

$$ \ge \sum_{j=1}^{n} \sum_{k=1}^{n} \exp \left( \left\langle \frac{x+y}{2}, x_j + x_k \right\rangle \right) \quad (7.4) $$

$$ = \exp \left( 2f \left( \frac{x+y}{2} \right) \right). $$

Taking the log on both sides yields the statement. $\square$

The following lemma also holds for any $d \ge 1$.

**Lemma 7.2.** Let $\mathbb{R} \ni t \mapsto \{x_i(t)\}_{i \in [n]}$ be a solution to (7.1). Then for any $i, j \in [n]$, the map $\mathbb{R} \ni t \mapsto \|x_i(t) - x_j(t)\|$ is non-decreasing.
*Proof.* The dynamics (7.1) can be equivalently written as

$$ \dot{x}_i(t) = \nabla f(x_i(t)) $$

where $f$ is as in (7.2). By convexity of $f$ (Lemma 7.1),

$$ \begin{aligned} \frac{1}{2} \frac{d}{dt} \|x_i(t) - x_j(t)\|^2 &= \langle \dot{x}_i(t) - \dot{x}_j(t), x_i(t) - x_j(t) \rangle \\ &= \langle \nabla f(x_i(t)) - \nabla f(x_j(t)), x_i(t) - x_j(t) \rangle \ge 0, \end{aligned} $$

as desired.

□

We now present the proof of Theorem 2.1, which assumes $d = 1$. We recall that in the statement, $V$ is a positive scalar, but by reparametrizing time we may assume that $V = 1$, so the 1d dynamics under consideration is really given by (7.1). Also, to ease notations we focus on $QK = 1$, but the proof adapts straightforwardly to the setting $QK > 0$ assumed in the statement of Theorem 2.1.

As seen in Section 7.1, it is not difficult to prove the convergence of the coefficients $P_{ij}(t)$ of the attention matrix for indices $i \in [n]$ for which $x_i(t)$ becomes unbounded as $t \to +\infty$. This is the case for at least $n-1$ of the particles $x_i(t)$ (Lemma 7.6). But should one particle $x_i(t)$ remain bounded, proving the convergence of $P_{ij}(t)$ for $j \in [n]$ is slightly tedious (Section 7.2). Since $d = 1$, up to relabeling, we can order the initial collection of particles (which, we recall, are assumed distinct):

$$ x_1(0) < \dots < x_n(0). $$

We set

$$ c := \min_{i \in [n-1]} |x_{i+1}(0) - x_i(0)|. \qquad (7.5) $$

According to Lemma 7.2, we have $|x_i(t) - x_j(t)| \ge c$ for any $i \ne j$ and any $t \ge 0$. In particular, particles never "collide".

**7.1. Results about unbounded particles.** In this section we gather several results concerning the indices $i$ corresponding to particles $x_i(t)$ which are not uniformly bounded in time. In particular, in Lemma 7.4 we show that for such indices $i$, $P_{ij}(t)$ converges toward 0 or 1 for any $j \in [n]$.

**Lemma 7.3.** Let $A > 0$ denote the unique positive real number satisfying $A^2 = n^2 \exp(-A^2)$. If $x_n(t_0) > A$ for some time $t_0 \ge 0$, then there exists $c_1 > 0$ such that $x_n(t) \ge c_1 e^t$ for any sufficiently large $t > 0$. Similarly, if $x_1(t_0) < -A$ for some $t_0 \ge 0$, then $x_1(t) \le -c_1 e^t$ for any sufficiently large $t > 0$.

*Proof.* The two cases are symmetric since the evolution (7.1) commutes with the involution of $(\mathbb{R}^d)^n$ given by $(x_1, \dots, x_n) \mapsto (-x_1, \dots, -x_n)$. We thus focus on the case $x_n(t_0) > A$.
*Proof.* The dynamics (7.1) can be equivalently written as

$$ \dot{x}_i(t) = \nabla f(x_i(t)) $$

where $f$ is as in (7.2). By convexity of $f$ (Lemma 7.1),

$$ \begin{aligned} \frac{1}{2} \frac{d}{dt} \|x_i(t) - x_j(t)\|^2 &= \langle \dot{x}_i(t) - \dot{x}_j(t), x_i(t) - x_j(t) \rangle \\ &= \langle \nabla f(x_i(t)) - \nabla f(x_j(t)), x_i(t) - x_j(t) \rangle \ge 0, \end{aligned} $$

as desired.

□

We now present the proof of Theorem 2.1, which assumes $d = 1$. We recall that in the statement, $V$ is a positive scalar, but by reparametrizing time we may assume that $V = 1$, so the 1d dynamics under consideration is really given by (7.1). Also, to ease notations we focus on $QK = 1$, but the proof adapts straightforwardly to the setting $QK > 0$ assumed in the statement of Theorem 2.1.

As seen in Section 7.1, it is not difficult to prove the convergence of the coefficients $P_{ij}(t)$ of the attention matrix for indices $i \in [n]$ for which $x_i(t)$ becomes unbounded as $t \to +\infty$. This is the case for at least $n-1$ of the particles $x_i(t)$ (Lemma 7.6). But should one particle $x_i(t)$ remain bounded, proving the convergence of $P_{ij}(t)$ for $j \in [n]$ is slightly tedious (Section 7.2). Since $d = 1$, up to relabeling, we can order the initial collection of particles (which, we recall, are assumed distinct):

$$ x_1(0) < \dots < x_n(0). $$

We set

$$ c := \min_{i \in [n-1]} |x_{i+1}(0) - x_i(0)|. \qquad (7.5) $$

According to Lemma 7.2, we have $|x_i(t) - x_j(t)| \ge c$ for any $i \ne j$ and any $t \ge 0$. In particular, particles never "collide".

**7.1. Results about unbounded particles.** In this section we gather several results concerning the indices $i$ corresponding to particles $x_i(t)$ which are not uniformly bounded in time. In particular, in Lemma 7.4 we show that for such indices $i$, $P_{ij}(t)$ converges toward 0 or 1 for any $j \in [n]$.

**Lemma 7.3.** Let $A > 0$ denote the unique positive real number satisfying $A^2 = n^2 \exp(-A^2)$. If $x_n(t_0) > A$ for some time $t_0 \ge 0$, then there exists $c_1 > 0$ such that $x_n(t) \ge c_1 e^t$ for any sufficiently large $t > 0$. Similarly, if $x_1(t_0) < -A$ for some $t_0 \ge 0$, then $x_1(t) \le -c_1 e^t$ for any sufficiently large $t > 0$.

*Proof.* The two cases are symmetric since the evolution (7.1) commutes with the involution of $(\mathbb{R}^d)^n$ given by $(x_1, \dots, x_n) \mapsto (-x_1, \dots, -x_n)$. We thus focus on the case $x_n(t_0) > A$.
If $x_n(t) \ge 0$ for some $t \ge 0$, then

$$ \dot{x}_n(t) = \sum_{j=1}^{n} \left( \frac{e^{x_n(t)(x_j(t)-x_n(t))}}{\sum_{k=1}^{n} e^{x_n(t)(x_k(t)-x_n(t))}} \right) x_j(t) \quad (7.6) $$

$$ \ge \frac{x_n(t)}{1 + (n-1)e^{-cx_n(t)}} + \sum_{\{j \in [n]: x_j(t) < 0\}} e^{x_n(t)(x_j(t)-x_n(t))} x_j(t) \quad (7.7) $$

$$ \ge \frac{x_n(t)}{1 + (n-1)e^{-cx_n(t)}} - n \frac{e^{-x_n(t)^2}}{x_n(t)} \quad (7.8) $$

$$ \ge \frac{x_n(t)}{n} - n \frac{e^{-x_n(t)^2}}{x_n(t)}. \quad (7.9) $$

We provide some detail on the above sequence of inequalities. First of all, to pass from (7.6) to (7.7), we use

$$ e^{x_n(t)(x_k(t)-x_n(t))} \le e^{-cx_n(t)} $$

for $j = n$ and for any $k \in [n]$ (which holds by virtue of (7.5)), combined with the fact that

$$ \sum_{k=1}^{n} e^{x_n(t)(x_k(t)-x_n(t))} \ge 1 $$

for all indices $j$ such that $x_j(t) < 0$. To pass from (7.7) to (7.8), we use $e^{x_n(t)z} \ge -\frac{1}{x_n(t)}$, which holds for any $z \le 0$.

For any $B > A$, we clearly have

$$ \frac{B}{n} - n \frac{e^{-B^2}}{B} > 0. $$

We then deduce from (7.8) and the fact that $x_n(t_0) > A$ that $x_n(t) \to +\infty$ as $t \to +\infty$. Moreover due to the fact that the expression in (7.9) is bounded from below by $\frac{x_n(t)}{2n}$ whenever $x_n(t)$ is sufficiently large, we deduce that

$$ x_n(t) \ge c_0 e^{\frac{t}{2n}} $$

for any sufficiently large $t > 0$.

Coming back to (7.8), we find that for sufficiently large $t > 0$,

$$ \dot{x}_n(t) \ge x_n(t) \left( \frac{1}{1 + (n-1)e^{-c c_0 e^{\frac{t}{2n}}}} - e^{-c_0^2 e^{\frac{t}{2n}}} \right). $$

This implies that

$$ \frac{d}{dt} \log(x_n(t)) \ge 1 - O\left(e^{-\frac{t}{3n}}\right), $$

whence

$$ \log(x_n(t)) \ge t + O(1) $$

for sufficiently large $t > 0$, as desired. $\square$

Here and in what follows, $\delta_{jk}$ denotes the Kronecker symbol.

**Lemma 7.4.** If $i \in [n]$ is such that $x_i(t)$ is not uniformly bounded with respect to $t > 0$, then $x_i(t)$ converges to either $-\infty$ or $+\infty$ as $t \to +\infty$. Moreover,

(1) if $x_i(t) \to +\infty$, then for any $j \in [n]$, $P_{ij}(t)$ converges to $\delta_{nj}$ as $t \to +\infty$, with doubly exponential rate.
If $x_n(t) \ge 0$ for some $t \ge 0$, then

$$ \dot{x}_n(t) = \sum_{j=1}^{n} \left( \frac{e^{x_n(t)(x_j(t)-x_n(t))}}{\sum_{k=1}^{n} e^{x_n(t)(x_k(t)-x_n(t))}} \right) x_j(t) \quad (7.6) $$

$$ \ge \frac{x_n(t)}{1+(n-1)e^{-cx_n(t)}} + \sum_{\{j \in [n]: x_j(t) < 0\}} e^{x_n(t)(x_j(t)-x_n(t))} x_j(t) \quad (7.7) $$

$$ \ge \frac{x_n(t)}{1+(n-1)e^{-cx_n(t)}} - n \frac{e^{-x_n(t)^2}}{x_n(t)} \quad (7.8) $$

$$ \ge \frac{x_n(t)}{n} - n \frac{e^{-x_n(t)^2}}{x_n(t)}. \quad (7.9) $$

We provide some detail on the above sequence of inequalities. First of all, to pass from (7.6) to (7.7), we use

$$ e^{x_n(t)(x_k(t)-x_n(t))} \le e^{-cx_n(t)} $$

for $j = n$ and for any $k \in [n]$ (which holds by virtue of (7.5)), combined with the fact that

$$ \sum_{k=1}^{n} e^{x_n(t)(x_k(t)-x_n(t))} \ge 1 $$

for all indices $j$ such that $x_j(t) < 0$. To pass from (7.7) to (7.8), we use $e^{x_n(t)z} \ge -\frac{1}{x_n(t)}$, which holds for any $z \le 0$.

For any $B > A$, we clearly have

$$ \frac{B}{n} - n \frac{e^{-B^2}}{B} > 0. $$

We then deduce from (7.8) and the fact that $x_n(t_0) > A$ that $x_n(t) \to +\infty$ as $t \to +\infty$. Moreover due to the fact that the expression in (7.9) is bounded from below by $\frac{x_n(t)}{2n}$ whenever $x_n(t)$ is sufficiently large, we deduce that

$$ x_n(t) \ge c_0 e^{\frac{t}{2n}} $$

for any sufficiently large $t > 0$.

Coming back to (7.8), we find that for sufficiently large $t > 0$,

$$ \dot{x}_n(t) \ge x_n(t) \left( \frac{1}{1 + (n-1)e^{-c c_0 e^{\frac{t}{2n}}}} - e^{-c_0^2 e^{\frac{t}{2n}}} \right). $$

This implies that

$$ \frac{d}{dt} \log(x_n(t)) \ge 1 - O\left(e^{-\frac{t}{3n}}\right), $$

whence

$$ \log(x_n(t)) \ge t + O(1) $$

for sufficiently large $t > 0$, as desired.

<figure><img src="image_16.png" /></figure>

Here and in what follows, $\delta_{jk}$ denotes the Kronecker symbol.

**Lemma 7.4.** If $i \in [n]$ is such that $x_i(t)$ is not uniformly bounded with respect to $t > 0$, then $x_i(t)$ converges to either $-\infty$ or $+\infty$ as $t \to +\infty$. Moreover,

(1) if $x_i(t) \to +\infty$, then for any $j \in [n]$, $P_{ij}(t)$ converges to $\delta_{nj}$ as $t \to +\infty$, with doubly exponential rate.
(2) if $x_i(t) \to -\infty$, then for any $j \in [n]$, $P_{ij}(t)$ converges to $\delta_{1j}$ as $t \to +\infty$, with doubly exponential rate.

*Proof.* We assume that $x_i(t)$ is not uniformly bounded with respect to $t > 0$. Without loss of generality, we assume that there exists a sequence of positive times $\{t_k\}_{k=1}^{+\infty}$ with $t_k \to +\infty$ such that $x_i(t_k) \to +\infty$. Necessarily, $x_n(t_k) \to +\infty$. We notice that if $x_i(t) > 0$ for some $t \ge 0$, then, arguing as in (7.6)-(7.7)-(7.8), we have

$$ \dot{x}_i(t) = \sum_{j=1}^{n} \left( \frac{e^{x_i(t)(x_j(t)-x_n(t))}}{\sum_{k=1}^{n} e^{x_i(t)(x_k(t)-x_n(t))}} \right) x_j(t) \ge \frac{x_n(t)}{n} - \frac{n}{x_i(t)} e^{-x_i(t)x_n(t)}. \quad (7.10) $$

For sufficiently large integers $k \ge 1$, from (7.10) we get $\dot{x}_i(t_k) > 0$ and $\dot{x}_n(t_k) > 0$. But as $x_i$ and $x_n$ increase, the lower bound in (7.10) becomes larger. It follows that

$$ \dot{x}_i(t) \ge \frac{x_n(t)}{2n} \ge \frac{x_i(t)}{2n} $$

for sufficiently large $t$, implying that $x_i(t) \to +\infty$ with exponential rate as $t \to +\infty$.

We now prove point 1. regarding $P(t)$. We assume that $x_i(t) \to +\infty$ as $t \to +\infty$. In this case, for $j \ne n$ (namely $j \in [n-1]$),

$$ P_{ij}(t) = \frac{e^{x_i(t)x_j(t)}}{\sum_{k=1}^{n} e^{x_i(t)x_k(t)}} \le e^{x_i(t)(x_j(t)-x_n(t))} \le e^{-cx_i(t)}, $$

thus $P_{ij}(t)$ converges to 0 as $t \to +\infty$ (with doubly exponential rate). Consequently, we also deduce that

$$ P_{in}(t) = 1 - \sum_{j=1}^{n-1} P_{ij}(t) $$

converges to 1, also with doubly exponential rate, as $t \to +\infty$.

The case where $x_i(t) \to -\infty$ is symmetric. This concludes the proof. $\square$

<figure><img src="image_16.png" /></figure>

Our last result is useful in the next section.

**Lemma 7.5.** For any $i \in [n]$ such that $x_i(t)$ is not uniformly bounded with respect to $t > 0$, there exists some $\gamma_i \in \mathbb{R}$, $\gamma_i \ne 0$ such that $x_i(t) = \gamma_i e^t + o(e^t)$ as $t \to +\infty$.

*Proof.* Without loss of generality we assume that $x_i(t) \to +\infty$ as $t \to +\infty$. For $j \ne n$, we find

$$ P_{ij}(t) = \frac{e^{x_i(t)x_j(t)}}{\sum_{k=1}^{n} e^{x_i(t)x_k(t)}} = \frac{e^{x_i(t)(x_j(t)-x_n(t))}}{\sum_{k=1}^{n} e^{x_i(t)(x_k(t)-x_n(t))}} \le e^{-cx_i(t)}. $$

Consequently,

$$ P_{in}(t) \ge 1 - ne^{-cx_i(t)}. $$

Therefore, using Lemma 7.3 and the fact that $x_i(t) \ge b_i e^{t/2n}$ for some $b_i > 0$ (thanks to (7.10)), we gather that

$$ \begin{aligned} \dot{x}_i(t) &\ge (1 - ne^{-cx_i(t)}) x_n(t) - ne^{-cx_i(t)} c_1 e^t \\ &\ge (1 - ne^{-cb_i e^{t/2n}}) x_n(t) - ne^{-cb_i e^{t/2n}} c_1 e^t \end{aligned} \quad (7.11) $$
(2) if $x_i(t) \to -\infty$, then for any $j \in [n]$, $P_{ij}(t)$ converges to $\delta_{1j}$ as $t \to +\infty$, with doubly exponential rate.

*Proof.* We assume that $x_i(t)$ is not uniformly bounded with respect to $t > 0$. Without loss of generality, we assume that there exists a sequence of positive times $\{t_k\}_{k=1}^{+\infty}$ with $t_k \to +\infty$ such that $x_i(t_k) \to +\infty$. Necessarily, $x_n(t_k) \to +\infty$. We notice that if $x_i(t) > 0$ for some $t \ge 0$, then, arguing as in (7.6)-(7.7)-(7.8), we have

$$ \dot{x}_i(t) = \sum_{j=1}^{n} \left( \frac{e^{x_i(t)(x_j(t)-x_n(t))}}{\sum_{k=1}^{n} e^{x_i(t)(x_k(t)-x_n(t))}} \right) x_j(t) \ge \frac{x_n(t)}{n} - \frac{n}{x_i(t)} e^{-x_i(t)x_n(t)}. \quad (7.10) $$

For sufficiently large integers $k \ge 1$, from (7.10) we get $\dot{x}_i(t_k) > 0$ and $\dot{x}_n(t_k) > 0$. But as $x_i$ and $x_n$ increase, the lower bound in (7.10) becomes larger. It follows that

$$ \dot{x}_i(t) \ge \frac{x_n(t)}{2n} \ge \frac{x_i(t)}{2n} $$

for sufficiently large $t$, implying that $x_i(t) \to +\infty$ with exponential rate as $t \to +\infty$.

We now prove point 1. regarding $P(t)$. We assume that $x_i(t) \to +\infty$ as $t \to +\infty$. In this case, for $j \ne n$ (namely $j \in [n-1]$),

$$ P_{ij}(t) = \frac{e^{x_i(t)x_j(t)}}{\sum_{k=1}^{n} e^{x_i(t)x_k(t)}} \le e^{x_i(t)(x_j(t)-x_n(t))} \le e^{-cx_i(t)}, $$

thus $P_{ij}(t)$ converges to 0 as $t \to +\infty$ (with doubly exponential rate). Consequently, we also deduce that

$$ P_{in}(t) = 1 - \sum_{j=1}^{n-1} P_{ij}(t) $$

converges to 1, also with doubly exponential rate, as $t \to +\infty$.

The case where $x_i(t) \to -\infty$ is symmetric. This concludes the proof. $\square$

<figure><img src="image_17.png" /></figure>

Our last result is useful in the next section.

**Lemma 7.5.** For any $i \in [n]$ such that $x_i(t)$ is not uniformly bounded with respect to $t > 0$, there exists some $\gamma_i \in \mathbb{R}$, $\gamma_i \ne 0$ such that $x_i(t) = \gamma_i e^t + o(e^t)$ as $t \to +\infty$.

*Proof.* Without loss of generality we assume that $x_i(t) \to +\infty$ as $t \to +\infty$. For $j \ne n$, we find

$$ P_{ij}(t) = \frac{e^{x_i(t)x_j(t)}}{\sum_{k=1}^{n} e^{x_i(t)x_k(t)}} = \frac{e^{x_i(t)(x_j(t)-x_n(t))}}{\sum_{k=1}^{n} e^{x_i(t)(x_k(t)-x_n(t))}} \le e^{-cx_i(t)}. $$

Consequently,

$$ P_{in}(t) \ge 1 - ne^{-cx_i(t)}. $$

Therefore, using Lemma 7.3 and the fact that $x_i(t) \ge b_i e^{t/2n}$ for some $b_i > 0$ (thanks to (7.10)), we gather that

$$ \begin{aligned} \dot{x}_i(t) &\ge (1 - ne^{-cx_i(t)}) x_n(t) - ne^{-cx_i(t)} c_1 e^t \\ &\ge (1 - ne^{-cb_i e^{t/2n}}) x_n(t) - ne^{-cb_i e^{t/2n}} c_1 e^t \end{aligned} \quad (7.11) $$
for some $c_1 > 0$ independent of $t$. We also notice that due to (7.1), $\dot{x}_i(t) \le x_n(t)$. Using (7.11), firstly for $i = n$, together with the trivial upper bound $x_n(t) \le Ce^t$ for some $C > 0$ independent of $t$ (immediately seen from (7.1)), we obtain

$$ \dot{x}_n(t) = x_n(t) \left( 1 + o \left( e^{-cb_i e^{\frac{t}{3n}}} \right) \right) $$

as $t \to +\infty$, which yields

$$ x_n(t) = \gamma_n e^t + o(e^t) $$

for some $\gamma_n > 0$. Now using (7.11) for the index $i$, we gather that

$$ \dot{x}_i(t) = x_n(t) + o \left( e^{-cb_i e^{\frac{t}{3n}}} \right), $$

and so we deduce that

$$ x_i(t) = \gamma_n e^t + o(e^t). $$

Similarly, if $x_i(t) \to -\infty$, then $x_i(t) = \gamma_1 e^t + o(e^t)$. This proves Lemma 7.5 (and shows that $\gamma_i \in \{\gamma_1, \gamma_n\}$). $\square$

**7.2. Results about bounded particles.** In this section we collect results concerning particles which remain uniformly bounded in time. The following lemma entails that there can be at most one particle with this property.

**Lemma 7.6.** Consider

$$ \mathcal{B} := \{ i \in [n] : x_i(\cdot) \in L^{\infty}([0, +\infty)) \}. $$

Then $\#\mathcal{B} \in \{0, 1\}$.

*Proof.* We first prove that either $x_1(t) \to -\infty$ or $x_n(t) \to +\infty$ as $t \to +\infty$. By contradiction, if this is not the case, then by Lemma 7.3, $(x_1(t), \dots, x_n(t)) \in [-A, A]^n$ for any $t \ge 0$. We denote by $\mathcal{I}$ the set of configurations $(x_1^*, \dots, x_n^*) \in [-A, A]^n$ such that $|x_i^* - x_j^*| \ge |x_i(0) - x_j(0)| > 0$ for any distinct $i, j \in [n]$. For any $\mathbf{X}^* = (x_1^*, \dots, x_n^*) \in \mathcal{I}$, the function $f$ defined in (7.2) (with anchor points given by $\mathbf{X}^*$) is strictly convex—the equality in the inequality between (7.3) and (7.4) is never achieved. Therefore, the proof of Lemma 7.2 shows that if $\mathbf{X}^*$ is seen as an initial datum for the dynamics (7.1), then

$$ v(\mathbf{X}^*) := \frac{d}{dt}\Big|_{t=0} |x_1^*(t) - x_n^*(t)| > 0. $$

Since $\mathcal{I}$ is compact, $v_0 := \inf_{\mathbf{X}^* \in \mathcal{I}} v(\mathbf{X}^*) > 0$. Hence, $t \mapsto |x_1(t) - x_n(t)|$ grows at least linearly, which is a contradiction.

We may therefore assume without loss of generality that $x_1(t) \to -\infty$ as $t \to +\infty$. We prove that $x_n(t)$ converges to either $-\infty$, or 0, or $+\infty$, as $t \to +\infty$. We assume in the sequel that $x_n(t)$ does not converge to $-\infty$ or 0. For any $i \in [n]$, if there exists $\varepsilon > 0$ and a sequence of positive times $\{s_k\}_{k=1}^{+\infty}$ tending to $+\infty$ such that $x_i(s_k) \le -\varepsilon$, then it follows from (7.10) that $x_i(t) \to -\infty$. Therefore, by our assumptions, we have $\liminf_{t \to +\infty} x_n(t) \ge 0$. Also, since $x_n(t) \to 0$, there exists $\varepsilon > 0$ and a sequence of positive times $\{t_k\}_{k=1}^{+\infty}$ tending to $+\infty$ such that $x_n(t_k) \ge \varepsilon$ for any integer $k \ge 1$. For any $t \ge 0$ such that $x_n(t) \ge \varepsilon$, we introduce the set of indices

$$ N(t) = \{i \in [n] : x_i(t) < 0\}, $$
for some $c_1 > 0$ independent of $t$. We also notice that due to (7.1), $\dot{x}_i(t) \le x_n(t)$. Using (7.11), firstly for $i = n$, together with the trivial upper bound $x_n(t) \le Ce^t$ for some $C > 0$ independent of $t$ (immediately seen from (7.1)), we obtain

$$ \dot{x}_n(t) = x_n(t) \left( 1 + o \left( e^{-cb_i e^{\frac{t}{3n}}} \right) \right) $$

as $t \to +\infty$, which yields

$$ x_n(t) = \gamma_n e^t + o(e^t) $$

for some $\gamma_n > 0$. Now using (7.11) for the index $i$, we gather that

$$ \dot{x}_i(t) = x_n(t) + o \left( e^{-cb_i e^{\frac{t}{3n}}} \right), $$

and so we deduce that

$$ x_i(t) = \gamma_n e^t + o(e^t). $$

Similarly, if $x_i(t) \to -\infty$, then $x_i(t) = \gamma_1 e^t + o(e^t)$. This proves Lemma 7.5 (and shows that $\gamma_i \in \{\gamma_1, \gamma_n\}$). $\square$

**7.2. Results about bounded particles.** In this section we collect results concerning particles which remain uniformly bounded in time. The following lemma entails that there can be at most one particle with this property.

**Lemma 7.6.** Consider

$$ \mathcal{B} := \{i \in [n] : x_i(\cdot) \in L^{\infty}([0, +\infty))\}. $$

Then $\#\mathcal{B} \in \{0, 1\}$.

*Proof.* We first prove that either $x_1(t) \to -\infty$ or $x_n(t) \to +\infty$ as $t \to +\infty$. By contradiction, if this is not the case, then by Lemma 7.3, $(x_1(t), \dots, x_n(t)) \in [-A, A]^n$ for any $t \ge 0$. We denote by $\mathcal{I}$ the set of configurations $(x_1^*, \dots, x_n^*) \in [-A, A]^n$ such that $|x_i^* - x_j^*| \ge |x_i(0) - x_j(0)| > 0$ for any distinct $i, j \in [n]$. For any $\mathbf{X}^* = (x_1^*, \dots, x_n^*) \in \mathcal{I}$, the function $f$ defined in (7.2) (with anchor points given by $\mathbf{X}^*$) is strictly convex—the equality in the inequality between (7.3) and (7.4) is never achieved. Therefore, the proof of Lemma 7.2 shows that if $\mathbf{X}^*$ is seen as an initial datum for the dynamics (7.1), then

$$ v(\mathbf{X}^*) := \frac{d}{dt}|_{t=0} |x_1^*(t) - x_n^*(t)| > 0. $$

Since $\mathcal{I}$ is compact, $v_0 := \inf_{\mathbf{X}^* \in \mathcal{I}} v(\mathbf{X}^*) > 0$. Hence, $t \mapsto |x_1(t) - x_n(t)|$ grows at least linearly, which is a contradiction.

We may therefore assume without loss of generality that $x_1(t) \to -\infty$ as $t \to +\infty$. We prove that $x_n(t)$ converges to either $-\infty$, or 0, or $+\infty$, as $t \to +\infty$. We assume in the sequel that $x_n(t)$ does not converge to $-\infty$ or 0. For any $i \in [n]$, if there exists $\varepsilon > 0$ and a sequence of positive times $\{s_k\}_{k=1}^{+\infty}$ tending to $+\infty$ such that $x_i(s_k) \le -\varepsilon$, then it follows from (7.10) that $x_i(t) \to -\infty$. Therefore, by our assumptions, we have $\liminf_{t \to +\infty} x_n(t) \ge 0$. Also, since $x_n(t) \to 0$, there exists $\varepsilon > 0$ and a sequence of positive times $\{t_k\}_{k=1}^{+\infty}$ tending to $+\infty$ such that $x_n(t_k) \ge \varepsilon$ for any integer $k \ge 1$. For any $t \ge 0$ such that $x_n(t) \ge \varepsilon$, we introduce the set of indices

$$ N(t) = \{i \in [n] : x_i(t) < 0\}, $$
and we write

$$ \dot{x}_n(t) \ge \frac{e^{x_n(t)^2} x_n(t)}{\sum_{k=1}^{n} e^{x_n(t)x_k(t)}} + \frac{\sum_{j \in N(t)} e^{x_j(t)x_n(t)} x_j(t)}{\sum_{k=1}^{n} e^{x_n(t)x_k(t)}} \ge \frac{\varepsilon}{n} + \frac{1}{e^{\varepsilon^2}} \sum_{j \in N(t)} e^{\varepsilon x_j(t)} x_j(t). \quad (7.12) $$

According to Lemma 7.4, any point $x_i(t)$ which takes negative values for arbitrarily large times and does not converge to $-\infty$ has to converge to 0. Therefore, the second term in the lowermost bound in (7.12) is lower bounded by $-\frac{\varepsilon}{2n}$ for sufficiently large $t$. All in all, we gather that $\dot{x}_n(t) \ge \frac{\varepsilon}{2n}$ and $x_n(t)$ converges to $+\infty$ as $t \to +\infty$. If it converges to 0, then necessarily $x_{n-1}(t) \to -\infty$ by combining Lemma 7.2 with Lemma 7.4. This proves Lemma 7.6 in this case.

From now on we assume that $x_n(t) \to +\infty$. Using (7.10) we see that if there exists $\varepsilon > 0$ such that $x_i(t) > \varepsilon$ for an unbounded sequence of times $t$, then $x_i(t) \to +\infty$. The same is true symmetrically when $x_i(t) < -\varepsilon$ for an unbounded sequence of times $t$. Thus if $i \in \mathcal{B}$, necessarily $x_i(t) \to 0$. By Lemma 7.2 this can be true for at most one index $i$, which concludes the proof of Lemma 7.6. $\square$

If $\mathcal{B} = \emptyset$, Theorem 2.1 follows from Lemma 7.4. From now on, we assume that $\#\mathcal{B} = 1$, and we denote by $i_0 \in [n]$ its unique element. We distinguish two cases: either $i_0 \in \{1, n\}$ (Lemma 7.7), or $i_0 \notin \{1, n\}$ (Lemma 7.8).

**Lemma 7.7.** If $x_n(t)$ is bounded as $t \to +\infty$, then $P_{nn}(t) \to 1$, and $P_{nj}(t) \to 0$ for any $j \in [n-1]$, as $t \to +\infty$. Similarly, if $x_1(t)$ is bounded as $t \to +\infty$, then $P_{11}(t) \to 1$, and $P_{1j}(t) \to 0$ for any $j \in [n-1]$, as $t \to +\infty$.

*Proof.* The two cases ($x_n(\cdot)$ bounded or $x_1(\cdot)$ bounded) are symmetric since the evolution (7.1) commutes with the involution of $(\mathbb{R}^d)^n$ given by $(x_1, \dots, x_n) \mapsto (-x_1, \dots, -x_n)$. Whence, we only address the first one: we assume that $x_n(t)$ is bounded as $t \to +\infty$. We first notice that all particles $x_j(t)$ for $j \in [n-1]$ tend to $-\infty$ as $t \to +\infty$ due to Lemma 7.6. We now prove the following properties:

(1) $x_n(t) > 0$ for any sufficiently large $t$;

(2) $x_n(t) \to 0$ as $t \to +\infty$;

(3) for any $j \in [n-1]$, $P_{nj}(t) \to 0$ as $t \to +\infty$.

To prove point (1), we notice that for sufficiently large $t$, $x_i(t) \le 0$ for any $i \in [n-1]$. If in addition $x_n(t) \le 0$, then due to (7.1), all $x_i(t)$ ($i \in [n]$) remain negative and due to (7.1), $x_n(t) \to -\infty$ as $t \to +\infty$, which is a contradiction.

For point (2), we fix $\varepsilon > 0$, and set

$$ T_{\varepsilon}^{+} := \{t \ge 0: x_n(t) \ge \varepsilon\}. $$

We prove that if $T_{\varepsilon}^{+}$ is unbounded, then $x_n(t) \to +\infty$ as $t \to +\infty$, which is a contradiction. As a consequence, $T_{\varepsilon}^{+}$ is bounded for any $\varepsilon > 0$, which implies (in conjunction with point 1.) that $x_n(t) \to 0$ as $t \to +\infty$. So let us assume that $T_{\varepsilon}^{+}$ is unbounded. We notice that for any $\delta > 0$, if $t \in T_{\varepsilon}^{+}$ is sufficiently large then

$$ |e^{x_n(t)x_j(t)} x_j(t)| \le \delta $$
and we write

$$ \dot{x}_n(t) \ge \frac{e^{x_n(t)^2} x_n(t)}{\sum_{k=1}^{n} e^{x_n(t)x_k(t)}} + \frac{\sum_{j \in N(t)} e^{x_j(t)x_n(t)} x_j(t)}{\sum_{k=1}^{n} e^{x_n(t)x_k(t)}} \ge \frac{\varepsilon}{n} + \frac{1}{e^{\varepsilon^2}} \sum_{j \in N(t)} e^{\varepsilon x_j(t)} x_j(t). \quad (7.12) $$

According to Lemma 7.4, any point $x_i(t)$ which takes negative values for arbitrarily large times and does not converge to $-\infty$ has to converge to 0. Therefore, the second term in the lowermost bound in (7.12) is lower bounded by $-\frac{\varepsilon}{2n}$ for sufficiently large $t$. All in all, we gather that $\dot{x}_n(t) \ge \frac{\varepsilon}{2n}$ and $x_n(t)$ converges to $+\infty$ as $t \to +\infty$. If it converges to 0, then necessarily $x_{n-1}(t) \to -\infty$ by combining Lemma 7.2 with Lemma 7.4. This proves Lemma 7.6 in this case.

From now on we assume that $x_n(t) \to +\infty$. Using (7.10) we see that if there exists $\varepsilon > 0$ such that $x_i(t) > \varepsilon$ for an unbounded sequence of times $t$, then $x_i(t) \to +\infty$. The same is true symmetrically when $x_i(t) < -\varepsilon$ for an unbounded sequence of times $t$. Thus if $i \in \mathcal{B}$, necessarily $x_i(t) \to 0$. By Lemma 7.2 this can be true for at most one index $i$, which concludes the proof of Lemma 7.6. $\square$

If $\mathcal{B} = \emptyset$, Theorem 2.1 follows from Lemma 7.4. From now on, we assume that $\#\mathcal{B} = 1$, and we denote by $i_0 \in [n]$ its unique element. We distinguish two cases: either $i_0 \in \{1, n\}$ (Lemma 7.7), or $i_0 \notin \{1, n\}$ (Lemma 7.8).

**Lemma 7.7.** If $x_n(t)$ is bounded as $t \to +\infty$, then $P_{nn}(t) \to 1$, and $P_{nj}(t) \to 0$ for any $j \in [n-1]$, as $t \to +\infty$. Similarly, if $x_1(t)$ is bounded as $t \to +\infty$, then $P_{11}(t) \to 1$, and $P_{1j}(t) \to 0$ for any $j \in [n-1]$, as $t \to +\infty$.

*Proof.* The two cases ($x_n(\cdot)$ bounded or $x_1(\cdot)$ bounded) are symmetric since the evolution (7.1) commutes with the involution of $(\mathbb{R}^d)^n$ given by $(x_1, \dots, x_n) \mapsto (-x_1, \dots, -x_n)$. Whence, we only address the first one: we assume that $x_n(t)$ is bounded as $t \to +\infty$. We first notice that all particles $x_j(t)$ for $j \in [n-1]$ tend to $-\infty$ as $t \to +\infty$ due to Lemma 7.6. We now prove the following properties:

(1) $x_n(t) > 0$ for any sufficiently large $t$;

(2) $x_n(t) \to 0$ as $t \to +\infty$;

(3) for any $j \in [n-1]$, $P_{nj}(t) \to 0$ as $t \to +\infty$.

To prove point (1), we notice that for sufficiently large $t$, $x_i(t) \le 0$ for any $i \in [n-1]$. If in addition $x_n(t) \le 0$, then due to (7.1), all $x_i(t)$ ($i \in [n]$) remain negative and due to (7.1), $x_n(t) \to -\infty$ as $t \to +\infty$, which is a contradiction.

For point (2), we fix $\varepsilon > 0$, and set

$$ T_{\varepsilon}^{+} := \{t \ge 0: x_n(t) \ge \varepsilon\}. $$

We prove that if $T_{\varepsilon}^{+}$ is unbounded, then $x_n(t) \to +\infty$ as $t \to +\infty$, which is a contradiction. As a consequence, $T_{\varepsilon}^{+}$ is bounded for any $\varepsilon > 0$, which implies (in conjunction with point 1.) that $x_n(t) \to 0$ as $t \to +\infty$. So let us assume that $T_{\varepsilon}^{+}$ is unbounded. We notice that for any $\delta > 0$, if $t \in T_{\varepsilon}^{+}$ is sufficiently large then

$$ |e^{x_n(t)x_j(t)} x_j(t)| \le \delta $$
for any $j \in [n-1]$ since $x_j(t) \to +\infty$ as $t \to +\infty$. Therefore,

$$ \sum_{j=1}^{n} e^{x_n(t)x_j(t)} x_j(t) \geq e^{\epsilon^2} \epsilon - (n-1)\delta \geq 0, $$

where we took $\delta > 0$ sufficiently small for the last inequality to hold. Consequently,

$$ \dot{x}_n(t) = \frac{\sum_{j=1}^{n} e^{x_n(t)x_j(t)} x_j(t)}{\sum_{j=1}^{n} e^{x_n(t)x_j(t)}} \geq \frac{e^{x_n(t)^2} x_n(t) - (n-1)\delta}{e^{x_n(t)^2} + n - 1}. $$

It is not difficult to see that this implies that $x_n(t) \to +\infty$ as $t \to +\infty$, which is a contradiction.

For point (3), we first notice that for any $j \neq n$, since $x_j(t) \to -\infty$,

$$ \dot{x}_j(t) = \sum_{k=1}^{n} \left( \frac{e^{x_j(t)(x_k(t)-x_n(t))}}{\sum_{\ell=1}^{n} e^{x_j(t)(x_\ell(t)-x_n(t))}} \right) x_k(t) \leq \frac{x_1(t)}{n} + \frac{n}{\epsilon} e^{-x_j(t)x_n(t)}. $$

Using Lemma 7.3, we deduce the existence of some $c_2 > 0$ such that

$$ x_j(t) \leq -c_2 e^t $$

for any sufficiently large $t > 0$. We now prove that for any $j \neq n$,

$$ x_j(t)x_n(t) - x_n(t)^2 \underset{t \to +\infty}{\longrightarrow} -\infty. \quad (7.13) $$

Due to the ordering of the particles, it is enough to prove (7.13) for $j = n-1$. Fix $j = n-1$ and $\kappa > 0$, and assume that

$$ x_n(t)x_j(t) \geq x_n(t)^2 - \kappa $$

for some $t \geq 0$. Then, using the fact that

$$ x_n(t)x_j(t) \geq x_n(t)x_k(t) $$

for any $k \in [n-2]$, we get

$$ P_{nj}(t) \geq \frac{e^{x_j(t)x_n(t)}}{e^{x_n(t)^2} + (n-1)e^{x_n(t)x_j(t)}} \geq \epsilon, $$

where $\epsilon = \frac{1}{n+e^\kappa}$. We obtain

$$ \dot{x}_n(t) \leq P_{nn}(t)x_n(t) + P_{nj}(t)x_j(t) \leq x_n(t) + \epsilon x_j(t), $$

hence

$$ \begin{aligned} \frac{d}{dt}(x_n(t)(x_n(t) - x_j(t))) &= \dot{x}_n(t)(2x_n(t) - x_j(t)) - x_n(t)\dot{x}_j(t) \\ &\leq (x_n(t) + \epsilon x_j(t))(2x_n(t) - x_j(t)) - x_n(t)\dot{x}_j(t) \\ &= -\epsilon x_j(t)^2 + x_n(t)(2\epsilon x_j(t) + 2x_n(t) - x_j(t) - \dot{x}_j(t)) \\ &\leq -\epsilon x_j(t)^2 + x_n(t)(2x_n(t) - 2x_1(t)), \end{aligned} \quad (7.14) $$

where in the last line we used the fact that $\dot{x}_j(t) \geq x_1(t)$, which is due to (7.1), and that $x_1(t) < x_j(t)$, which is due to the ordering of the particles. Since $x_j(t) \leq -c_2e^t$ and $x_1(t) \geq -c_1e^t$, the upper bound in (7.14) is negative if $t$ is large enough. We therefore conclude that for any fixed $\kappa$, if there exist unbounded times $t$ such that $x_n(t)x_j(t) \geq x_n(t)^2 - \kappa$, then $x_n(t)x_j(t) \geq x_n(t)^2 - \kappa$ for any $t$ large enough. But
for any $j \in [n-1]$ since $x_j(t) \to +\infty$ as $t \to +\infty$. Therefore,

$$ \sum_{j=1}^{n} e^{x_n(t)x_j(t)} x_j(t) \geq e^{\epsilon^2} \epsilon - (n-1)\delta \geq 0, $$

where we took $\delta > 0$ sufficiently small for the last inequality to hold. Consequently,

$$ \dot{x}_n(t) = \frac{\sum_{j=1}^{n} e^{x_n(t)x_j(t)} x_j(t)}{\sum_{j=1}^{n} e^{x_n(t)x_j(t)}} \geq \frac{e^{x_n(t)^2} x_n(t) - (n-1)\delta}{e^{x_n(t)^2} + n - 1}. $$

It is not difficult to see that this implies that $x_n(t) \to +\infty$ as $t \to +\infty$, which is a contradiction.

For point (3), we first notice that for any $j \neq n$, since $x_j(t) \to -\infty$,

$$ \dot{x}_j(t) = \sum_{k=1}^{n} \left( \frac{e^{x_j(t)(x_k(t)-x_n(t))}}{\sum_{\ell=1}^{n} e^{x_j(t)(x_\ell(t)-x_n(t))}} \right) x_k(t) \leq \frac{x_1(t)}{n} + \frac{n}{\epsilon} e^{-x_j(t)x_n(t)}. $$

Using Lemma 7.3, we deduce the existence of some $c_2 > 0$ such that

$$ x_j(t) \leq -c_2 e^t $$

for any sufficiently large $t > 0$. We now prove that for any $j \neq n$,

$$ x_j(t)x_n(t) - x_n(t)^2 \underset{t \to +\infty}{\longrightarrow} -\infty. \qquad (7.13) $$

Due to the ordering of the particles, it is enough to prove (7.13) for $j = n-1$. Fix $j = n-1$ and $\kappa > 0$, and assume that

$$ x_n(t)x_j(t) \geq x_n(t)^2 - \kappa $$

for some $t \geq 0$. Then, using the fact that

$$ x_n(t)x_j(t) \geq x_n(t)x_k(t) $$

for any $k \in [n-2]$, we get

$$ P_{nj}(t) \geq \frac{e^{x_j(t)x_n(t)}}{e^{x_n(t)^2} + (n-1)e^{x_n(t)x_j(t)}} \geq \epsilon, $$

where $\epsilon = \frac{1}{n+e^\kappa}$. We obtain

$$ \dot{x}_n(t) \leq P_{nn}(t)x_n(t) + P_{nj}(t)x_j(t) \leq x_n(t) + \epsilon x_j(t), $$

hence

$$ \begin{aligned} \frac{d}{dt}(x_n(t)(x_n(t) - x_j(t))) &= \dot{x}_n(t)(2x_n(t) - x_j(t)) - x_n(t)\dot{x}_j(t) \\ &\leq (x_n(t) + \epsilon x_j(t))(2x_n(t) - x_j(t)) - x_n(t)\dot{x}_j(t) \\ &= -\epsilon x_j(t)^2 + x_n(t)(2\epsilon x_j(t) + 2x_n(t) - x_j(t) - \dot{x}_j(t)) \\ &\leq -\epsilon x_j(t)^2 + x_n(t)(2x_n(t) - 2x_1(t)), \end{aligned} \qquad (7.14) $$

where in the last line we used the fact that $\dot{x}_j(t) \geq x_1(t)$, which is due to (7.1), and that $x_1(t) < x_j(t)$, which is due to the ordering of the particles. Since $x_j(t) \leq -c_2e^t$ and $x_1(t) \geq -c_1e^t$, the upper bound in (7.14) is negative if $t$ is large enough. We therefore conclude that for any fixed $\kappa$, if there exist unbounded times $t$ such that $x_n(t)x_j(t) \geq x_n(t)^2 - \kappa$, then $x_n(t)x_j(t) \geq x_n(t)^2 - \kappa$ for any $t$ large enough. But
this is excluded since $x_n(t) > 0$ and $x_j(t) \to -\infty$ as $t \to +\infty$. This concludes the proof of (7.13), and the lemma follows by plugging this information into the definition of $P_{nj}(t)$. $\square$

**Lemma 7.8.** If $i_0 \notin \{1, n\}$ and $x_{i_0}(t)$ remains uniformly bounded in $t$, then for any $j \in [n-1]$, there exists some $\alpha_j \in [0, 1]$ such that $P_{i_0j}(t) \to \alpha_j$ as $t \to +\infty$.

*Proof.* Assume that $i_0 \notin \{1, n\}$. Then $x_1(t) \to -\infty$ and $x_n(t) \to +\infty$ as $t \to +\infty$. Also, $x_{i_0}(t) \to 0$ due to (7.10). We write $x_{i_0}(t) = y_{i_0}(t)e^{-t}$. Since $\gamma_n > 0$ and $\gamma_1 < 0$, we notice that the function

$$g: \theta \mapsto \frac{\sum_{i \in [n] \setminus \{i_0\}} e^{\gamma_i \theta} \gamma_i}{1 + \sum_{i \in [n] \setminus \{i_0\}} e^{\gamma_i \theta}}$$

takes value $-\infty$ at $-\infty$, and $+\infty$ at $+\infty$, and has a positive derivative. Thus, it takes the value 0 exactly once, and we denote this point by $\theta_0$. We prove that $y_{i_0}(t) \to \theta_0$ as $t \to +\infty$. We observe that

$$e^{x_{i_0}(t)^2} = 1 + o(1).$$

Using Lemma 7.5 we have

$$\begin{aligned} \dot{y}_{i_0}(t) &= e^t \dot{x}_{i_0}(t) - y_{i_0}(t) \\ &= (P_{i_0 i_0}(t) - 1)y_{i_0}(t) \\ &\quad + e^{2t} \sum_{j \in [n] \setminus \{i_0\}} \left( \frac{e^{y_{i_0}(t)(\gamma_j + o(1))}}{1 + o(1) + \sum_{k \in [n] \setminus \{i_0\}} e^{y_{i_0}(t)(\gamma_k + o(1))}} \right) (\gamma_j + o(1)). \end{aligned}$$

We recognize that the sum in the above expression is roughly equal to $g(y_{i_0})$. If the latter is not close to 0 for large times, then $\dot{y}_{i_0}(t)$ necessarily have a huge magnitude due to the $e^{2t}$ factor, leading to a contradiction. Fix $\varepsilon > 0$. If $y_{i_0}(t) > \theta_0 + \varepsilon$ for some large time $t > 0$, then, noticing that

$$|y_{i_0}(t)| = e^t |x_{i_0}(t)| = o(e^t), \qquad (7.15)$$

we get

$$\dot{y}_{i_0}(t) = o(e^t) + e^{2t} \left( g(y_{i_0}(t) + o(y_{i_0}(t))) \right).$$

But $g(y_{i_0}(t)) \ge \delta = \delta(\varepsilon)$, and hence

$$\dot{y}_{i_0}(s) \ge \frac{\delta}{2} e^{2s}$$

for any larger time $s \ge t$, which contradicts (7.15). We get a similar contradiction if $y_{i_0}(t) < \theta_0 - \varepsilon$ for large enough $t$. This concludes the proof that $y_{i_0}(t) \to \theta_0$. As a consequence, $x_{i_0}(t)x_i(t) \to \theta_0 \gamma_i$ for any $i \ne i_0$, and we deduce Lemma 7.8. $\square$
this is excluded since $x_n(t) > 0$ and $x_j(t) \to -\infty$ as $t \to +\infty$. This concludes the proof of (7.13), and the lemma follows by plugging this information into the definition of $P_{nj}(t)$. □

**Lemma 7.8.** If $i_0 \notin \{1, n\}$ and $x_{i_0}(t)$ remains uniformly bounded in $t$, then for any $j \in [n-1]$, there exists some $\alpha_j \in [0, 1]$ such that $P_{i_0j}(t) \to \alpha_j$ as $t \to +\infty$.

*Proof.* Assume that $i_0 \notin \{1, n\}$. Then $x_1(t) \to -\infty$ and $x_n(t) \to +\infty$ as $t \to +\infty$. Also, $x_{i_0}(t) \to 0$ due to (7.10). We write $x_{i_0}(t) = y_{i_0}(t)e^{-t}$. Since $\gamma_n > 0$ and $\gamma_1 < 0$, we notice that the function

$$g: \theta \mapsto \frac{\sum_{i \in [n] \setminus \{i_0\}} e^{\gamma_i \theta} \gamma_i}{1 + \sum_{i \in [n] \setminus \{i_0\}} e^{\gamma_i \theta}}$$

takes value $-\infty$ at $-\infty$, and $+\infty$ at $+\infty$, and has a positive derivative. Thus, it takes the value 0 exactly once, and we denote this point by $\theta_0$. We prove that $y_{i_0}(t) \to \theta_0$ as $t \to +\infty$. We observe that

$$e^{x_{i_0}(t)^2} = 1 + o(1).$$

Using Lemma 7.5 we have

$$\begin{aligned} \dot{y}_{i_0}(t) &= e^t \dot{x}_{i_0}(t) - y_{i_0}(t) \\ &= (P_{i_0 i_0}(t) - 1)y_{i_0}(t) \\ &\quad + e^{2t} \sum_{j \in [n] \setminus \{i_0\}} \left( \frac{e^{y_{i_0}(t)(\gamma_j + o(1))}}{1 + o(1) + \sum_{k \in [n] \setminus \{i_0\}} e^{y_{i_0}(t)(\gamma_k + o(1))}} \right) (\gamma_j + o(1)). \end{aligned}$$

We recognize that the sum in the above expression is roughly equal to $g(y_{i_0})$. If the latter is not close to 0 for large times, then $\dot{y}_{i_0}(t)$ necessarily have a huge magnitude due to the $e^{2t}$ factor, leading to a contradiction. Fix $\varepsilon > 0$. If $y_{i_0}(t) > \theta_0 + \varepsilon$ for some large time $t > 0$, then, noticing that

$$|y_{i_0}(t)| = e^t |x_{i_0}(t)| = o(e^t), \quad (7.15)$$

we get

$$\dot{y}_{i_0}(t) = o(e^t) + e^{2t} \left( g(y_{i_0}(t) + o(y_{i_0}(t))) \right).$$

But $g(y_{i_0}(t)) \ge \delta = \delta(\varepsilon)$, and hence

$$\dot{y}_{i_0}(s) \ge \frac{\delta}{2} e^{2s}$$

for any larger time $s \ge t$, which contradicts (7.15). We get a similar contradiction if $y_{i_0}(t) < \theta_0 - \varepsilon$ for large enough $t$. This concludes the proof that $y_{i_0}(t) \to \theta_0$. As a consequence, $x_{i_0}(t)x_i(t) \to \theta_0 \gamma_i$ for any $i \ne i_0$, and we deduce Lemma 7.8. □
### 7.3. Concluding the proof of Theorem 2.1.

*Proof of Theorem 2.1.* By Lemma 7.6, there is at most one index $i_0 \in [n]$ for which the particle $x_{i_0}(t)$ remains bounded for any $t > 0$. In turn, for any $i \in [n]\setminus\{i_0\}$, we may invoke Lemma 7.4 which entails that $P_{ij}(t)$ converges to either $\delta_{1j}$ or $\delta_{nj}$ as $t \to +\infty$ (with doubly exponential rate). And by ordering of the particles, for indices $i_1 \le i_2$ different from $i_0$, and $P_{i_1j}(t) \to \delta_{nj}$ then necessarily $P_{i_2j}(t) \to \delta_{nj}$ as well. Consequently, all but at most one row of $P(t)$ converge to either $e_1 = (1, 0, \dots, 0)$ or $e_n = (0, \dots, 0, 1)$ as $t \to +\infty$. For the $i_0$-th row, we may invoke either Lemma 7.7 or Lemma 7.8. The former applies if $i_0 \in \{1, n\}$, and entails that the $i_0$-th row of $P(t)$ converges either to $e_1$ or $e_n$, while the latter applies if $i_0 \notin \{1, n\}$, and entails that the $i_0$-th row of $P(t)$ converges to some vector $\alpha \in \mathbb{R}^d$ with non-negative entries. Finally, since the $i_0$-th row of $P(t)$ has entries which sum up to 1, then so does $\alpha$. These conclusions lead us to a final limit matrix $P^*$ which has precisely the form indicated in Fig. 2 (namely, $P^* \in \mathcal{P}$), as desired. □

**Remark 7.9** (Higher dimensions). The extension of Theorem 2.1 to $d \ge 2$ is not straightforward due to rare pathological situations. For example, suppose $d = 2$, $n = 2$, and the initial configuration $x_1(0) = (1, \varepsilon)$ and $x_2(0) = (1, -\varepsilon)$. One can check that $x_i(t) \to (1, 0)$ as $t \to +\infty$, for $i = 1, 2$, which means that a single cluster appears. However, the self-attention matrix converges toward the identity (which has rank 2). Therefore, it is not true in full generality that the rank of the limiting self-attention matrix is equal to the number of clusters as $t \to +\infty$, although we believe that the result is true for almost all initial conditions.

## 8. PROOFS OF THEOREMS 3.1 AND 8.5

In this section, we focus on proving the result in the case

$$V = I_d.$$

We also provide a full picture of the behavior of the dynamics in the case $V = -I_d$ in Section 8.2.

**8.1. Clustering towards vertices of convex polytopes: Theorem 3.1.** In this section, we prove Theorem 8.1—namely, we show that particles $\{z_i(t)\}_{i \in [n]}$ following the rescaled dynamics

$$\dot{z}_i(t) = \sum_{j=1}^{n} \left( \frac{e^{2t} \langle Az_i(t), Az_j(t) \rangle}{\sum_{k=1}^{n} e^{2t} \langle Az_i(t), Az_k(t) \rangle} \right) (z_j(t) - z_i(t)) \quad (8.1)$$

converge, as $t \to \infty$, toward points lying on the boundary of a particular convex polytope. In (8.1) we made use of the shorthand notation

$$A := (Q^\top K)^{\frac{1}{2}}. \quad (8.2)$$

The precise statement is the following:

**Theorem 8.1.** Suppose $V = I_d$ and $Q^\top K > 0$. Then, for any initial datum $\{z_i(0)\}_{i \in [n]} \subset \mathbb{R}^d$, the solution to (8.1) is such that its convex hull $\text{conv}(\{z_i(t)\}_{i \in [n]})$ converges to some convex polytope $\mathcal{K} \subset \mathbb{R}^d$ as $t \to +\infty$. Furthermore, let $\mathcal{V} = \{v_1, \dots, v_m\}$ ($m \le n$) denote the set of vertices of $\mathcal{K}$, and consider

$$\mathcal{S} := \left\{ x \in \mathcal{K} : \|Ax\|^2 = \max_{j \in [m]} \langle Ax, Av_j \rangle \right\},$$
### 7.3. Concluding the proof of Theorem 2.1.

*Proof of Theorem 2.1.* By Lemma 7.6, there is at most one index $i_0 \in [n]$ for which the particle $x_{i_0}(t)$ remains bounded for any $t > 0$. In turn, for any $i \in [n]\setminus\{i_0\}$, we may invoke Lemma 7.4 which entails that $P_{ij}(t)$ converges to either $\delta_{1j}$ or $\delta_{nj}$ as $t \to +\infty$ (with doubly exponential rate). And by ordering of the particles, for indices $i_1 \le i_2$ different from $i_0$, and $P_{i_1j}(t) \to \delta_{nj}$ then necessarily $P_{i_2j}(t) \to \delta_{nj}$ as well. Consequently, all but at most one row of $P(t)$ converge to either $e_1 = (1, 0, \dots, 0)$ or $e_n = (0, \dots, 0, 1)$ as $t \to +\infty$. For the $i_0$-th row, we may invoke either Lemma 7.7 or Lemma 7.8. The former applies if $i_0 \in \{1, n\}$, and entails that the $i_0$-th row of $P(t)$ converges either to $e_1$ or $e_n$, while the latter applies if $i_0 \notin \{1, n\}$, and entails that the $i_0$-th row of $P(t)$ converges to some vector $\alpha \in \mathbb{R}^d$ with non-negative entries. Finally, since the $i_0$-th row of $P(t)$ has entries which sum up to 1, then so does $\alpha$. These conclusions lead us to a final limit matrix $P^*$ which has precisely the form indicated in Fig. 2 (namely, $P^* \in \mathcal{P}$), as desired. □

**Remark 7.9** (Higher dimensions). The extension of Theorem 2.1 to $d \ge 2$ is not straightforward due to rare pathological situations. For example, suppose $d = 2$, $n = 2$, and the initial configuration $x_1(0) = (1, \varepsilon)$ and $x_2(0) = (1, -\varepsilon)$. One can check that $x_i(t) \to (1, 0)$ as $t \to +\infty$, for $i = 1, 2$, which means that a single cluster appears. However, the self-attention matrix converges toward the identity (which has rank 2). Therefore, it is not true in full generality that the rank of the limiting self-attention matrix is equal to the number of clusters as $t \to +\infty$, although we believe that the result is true for almost all initial conditions.

## 8. PROOFS OF THEOREMS 3.1 AND 8.5

In this section, we focus on proving the result in the case

$$V = I_d.$$

We also provide a full picture of the behavior of the dynamics in the case $V = -I_d$ in Section 8.2.

### 8.1. Clustering towards vertices of convex polytopes: Theorem 3.1.
In this section, we prove Theorem 8.1—namely, we show that particles $\{z_i(t)\}_{i \in [n]}$ following the rescaled dynamics

$$\dot{z}_i(t) = \sum_{j=1}^{n} \left( \frac{e^{2t} \langle Az_i(t), Az_j(t) \rangle}{\sum_{k=1}^{n} e^{2t} \langle Az_i(t), Az_k(t) \rangle} \right) (z_j(t) - z_i(t)) \quad (8.1)$$

converge, as $t \to \infty$, toward points lying on the boundary of a particular convex polytope. In (8.1) we made use of the shorthand notation

$$A := (Q^\top K)^{\frac{1}{2}}. \quad (8.2)$$

The precise statement is the following:

**Theorem 8.1.** Suppose $V = I_d$ and $Q^\top K > 0$. Then, for any initial datum $\{z_i(0)\}_{i \in [n]} \subset \mathbb{R}^d$, the solution to (8.1) is such that its convex hull $\text{conv}(\{z_i(t)\}_{i \in [n]})$ converges to some convex polytope $\mathcal{K} \subset \mathbb{R}^d$ as $t \to +\infty$. Furthermore, let $\mathcal{V} = \{v_1, \dots, v_m\}$ ($m \le n$) denote the set of vertices of $\mathcal{K}$, and consider

$$\mathcal{S} := \left\{ x \in \mathcal{K} : \|Ax\|^2 = \max_{j \in [m]} \langle Ax, Av_j \rangle \right\},$$
with A defined in (8.2). Then $\mathcal{S}$ has finite cardinality, and $\mathcal{V} \subset \mathcal{S} \subset \partial\mathcal{K} \cup \{0\}$.
Finally, for any $i \in [n]$ there exists a point $\bar{z} \in \mathcal{S}$ such that $z_i(t) \to \bar{z}$ as $t \to +\infty$.
In particular, $z_i(t)$ converges either to some point on the boundary of $\mathcal{K}$, or to 0.

8.1.1. The convex hull is shrinking. To prove Theorem 8.1, we begin with the following illustrative result.

**Proposition 8.2.** Suppose $V = I_d$ and $Q^\top K > 0$. Then the solution $\{z_i(\cdot)\}_{i \in [n]}$ to (8.1) is such that $t \mapsto \text{conv}(\{z_i(t)\}_{i \in [n]})$ is non-increasing in the sense of set-inclusion.

*Proof of Proposition 8.2.* Fix $t > 0$ and let $H \subset \mathbb{R}^d$ be a closed half-space which does not contain any of the points $z_i(t)$. We define the map

$$ \alpha : s \mapsto \min_{i \in [n]} \text{dist}(z_i(s), H) $$

for $s \ge 0$. We claim that

$$ \alpha \text{ is non-decreasing on } [t, +\infty). \quad (8.3) $$

Before proving (8.3), let us show how to conclude the proof of Proposition 8.2 using this claim. It follows from (8.3) that if $\text{conv}(\{z_i(t)\}_{i \in [n]}) \cap H = \emptyset$, then $\text{conv}(\{z_i(t')\}_{i \in [n]}) \cap H = \emptyset$ for any $t' \ge t$. Writing the convex set $\text{conv}(\{z_i(t)\}_{i \in [n]})$ as

$$ \text{conv}(\{z_i(t)\}_{i \in [n]}) = \bigcap_{H' \text{ open half-space} \atop \text{conv}(\{z_i(t)\}_{i \in [n]}) \subset H'} H' = \bigcap_{H \text{ closed half-space} \atop \text{conv}(\{z_i(t)\}_{i \in [n]}) \cap H = \emptyset} \mathbb{R}^d \setminus H, $$

we get that $\text{conv}(\{z_i(t')\}_{i \in [n]}) \subset \text{conv}(\{z_i(t)\}_{i \in [n]})$ for any $t' \ge t$.

We now turn to the proof of the claim (8.3). Denoting by $\mathbf{n}$ the unit outer normal to $H$ and by $\text{proj}_H$ the orthogonal projection onto the closed set $H$, we have

$$ \text{dist}(x, H) = \langle x - \text{proj}_H(x), \mathbf{n} \rangle. $$

If $t \mapsto x(t)$ is a differentiable curve, writing $\dot{x}(t) = \langle \dot{x}(t), \mathbf{n} \rangle \mathbf{n} + v(t)$ where $v(t) \in H$ we have $\frac{d}{dt}(\text{proj}_H(x(t))) = v(t)$, whence

$$ \frac{d}{dt}\text{dist}(x(t), H) = \langle \dot{x}(t), \mathbf{n} \rangle. \quad (8.4) $$

Let $T > t$ denote the infimum of the times for which one of the points $z_i(t)$ lies in $H$. Now fix $s \in [t, T)$, and denote by $M(s)$ the set of indices $i \in [n]$ such that $\text{dist}(z_i(s), H)$ is minimal. For $h \to 0$, we have

$$ \begin{aligned} \alpha(s+h) &= \min_{i \in M(s)} \text{dist}(z_i(s+h), H) \\ &= \min_{i \in M(s)} \left( \text{dist}(z_i(s), H) + h \frac{d}{dt} \text{dist}(z_i(s), H) + o(h) \right) \\ &= \alpha(s) + h \left( \min_{i \in M(s)} \frac{d}{dt} \text{dist}(z_i(s), H) \right) + o(h). \end{aligned} $$

Consequently,

$$ \frac{d\alpha}{dt}(s) = \min_{i \in M(s)} \frac{d}{dt}\text{dist}(z_i(s), H). $$
with A defined in (8.2). Then $\mathcal{S}$ has finite cardinality, and $\mathcal{V} \subset \mathcal{S} \subset \partial\mathcal{K} \cup \{0\}$.
Finally, for any $i \in [n]$ there exists a point $\bar{z} \in \mathcal{S}$ such that $z_i(t) \to \bar{z}$ as $t \to +\infty$.
In particular, $z_i(t)$ converges either to some point on the boundary of $\mathcal{K}$, or to 0.

### 8.1.1. The convex hull is shrinking.
To prove Theorem 8.1, we begin with the following illustrative result.

**Proposition 8.2.** Suppose $V = I_d$ and $Q^\top K > 0$. Then the solution $\{z_i(\cdot)\}_{i \in [n]}$ to (8.1) is such that $t \mapsto \text{conv}(\{z_i(t)\}_{i \in [n]})$ is non-increasing in the sense of set-inclusion.

*Proof of Proposition 8.2.* Fix $t > 0$ and let $H \subset \mathbb{R}^d$ be a closed half-space which does not contain any of the points $z_i(t)$. We define the map

$$ \alpha : s \mapsto \min_{i \in [n]} \text{dist}(z_i(s), H) $$

for $s \ge 0$. We claim that

$$ \alpha \text{ is non-decreasing on } [t, +\infty). \quad (8.3) $$

Before proving (8.3), let us show how to conclude the proof of Proposition 8.2 using this claim. It follows from (8.3) that if $\text{conv}(\{z_i(t)\}_{i \in [n]}) \cap H = \emptyset$, then $\text{conv}(\{z_i(t')\}_{i \in [n]}) \cap H = \emptyset$ for any $t' \ge t$. Writing the convex set $\text{conv}(\{z_i(t)\}_{i \in [n]})$ as

$$ \text{conv}(\{z_i(t)\}_{i \in [n]}) = \bigcap_{H' \text{ open half-space} \atop \text{conv}(\{z_i(t)\}_{i \in [n]}) \subset H'} H' = \bigcap_{H \text{ closed half-space} \atop \text{conv}(\{z_i(t)\}_{i \in [n]}) \cap H = \emptyset} \mathbb{R}^d \setminus H, $$

we get that $\text{conv}(\{z_i(t')\}_{i \in [n]}) \subset \text{conv}(\{z_i(t)\}_{i \in [n]})$ for any $t' \ge t$.

We now turn to the proof of the claim (8.3). Denoting by $\mathbf{n}$ the unit outer normal to $H$ and by $\text{proj}_H$ the orthogonal projection onto the closed set $H$, we have

$$ \text{dist}(x, H) = \langle x - \text{proj}_H(x), \mathbf{n} \rangle. $$

If $t \mapsto x(t)$ is a differentiable curve, writing $\dot{x}(t) = \langle \dot{x}(t), \mathbf{n} \rangle \mathbf{n} + v(t)$ where $v(t) \in H$ we have $\frac{d}{dt}(\text{proj}_H(x(t))) = v(t)$, whence

$$ \frac{d}{dt}\text{dist}(x(t), H) = \langle \dot{x}(t), \mathbf{n} \rangle. \quad (8.4) $$

Let $T > t$ denote the infimum of the times for which one of the points $z_i(t)$ lies in $H$. Now fix $s \in [t, T)$, and denote by $M(s)$ the set of indices $i \in [n]$ such that $\text{dist}(z_i(s), H)$ is minimal. For $h \to 0$, we have

$$ \begin{aligned} \alpha(s+h) &= \min_{i \in M(s)} \text{dist}(z_i(s+h), H) \\ &= \min_{i \in M(s)} \left( \text{dist}(z_i(s), H) + h \frac{d}{dt} \text{dist}(z_i(s), H) + o(h) \right) \\ &= \alpha(s) + h \left( \min_{i \in M(s)} \frac{d}{dt} \text{dist}(z_i(s), H) \right) + o(h). \end{aligned} $$

Consequently,

$$ \frac{d\alpha}{dt}(s) = \min_{i \in M(s)} \frac{d}{dt}\text{dist}(z_i(s), H). $$
Moreover, for any $i \in M(s)$, one has

$$ \frac{d}{dt} \mathrm{dist}(z_i(s), H) \stackrel{(8.4)}{=} \langle \dot{z}_i(s), \mathbf{n} \rangle = \sum_{j=1}^{n} P_{ij}(s) \langle z_j(s) - z_i(s), \mathbf{n} \rangle \geq 0, $$

where the last inequality comes from the fact that each term in the sum is non-negative, since $i \in M(s)$. This proves (8.3) (and, as a byproduct, that $T = +\infty$). $\square$

The following fact immediately ensues.

**Corollary 8.3.** For any $i \in [n]$ and $t \ge 0$, $z_i(t) \in \mathrm{conv}(\{z_i(0)\}_{i \in [n]})$. In particular, $z_i(\cdot)$ is uniformly bounded in time.

### 8.1.2. Proof of Theorem 8.1.

*Proof of Theorem 8.1.* As a consequence of Proposition 8.2, the set $\mathrm{conv}(\{z_i(t)\}_{i \in [n]})$ converges as $t \to +\infty$ toward some convex polytope $\mathcal{K}$. In the remainder of the proof, we look to show that the particles $z_i(t)$ can in fact converge only to some well-distinguished points lying on the boundary of this polytope.

**Step 1. The candidate set of limit points.** We denote by $\mathcal{V} = \{v_1, \dots, v_m\}$ the set of vertices of $\mathcal{K}$. Writing any $x \in \mathcal{K}$ as a convex combination of these vertices: $x = \sum_{j=1}^m \alpha_j v_j$ for some weights $\alpha_j \ge 0$ with $\sum_{j=1}^m \alpha_j = 1$, we gather that

$$ \|Ax\|^2 = \left\langle Ax, \sum_{j=1}^{m} \alpha_j Av_j \right\rangle = \sum_{j=1}^{m} \alpha_j \langle Ax, Av_j \rangle \le \max_{j \in [m]} \langle Ax, Av_j \rangle. \quad (8.5) $$

Let $\mathcal{S} \subset \mathcal{K}$ denote the set of points $w \in \mathcal{K}$ such that

$$ \|Aw\|^2 = \max_{j \in [m]} \langle Aw, Av_j \rangle. \quad (8.6) $$

The following holds—we postpone the proof to after that of the theorem.

**Claim 1.** $\mathcal{V} \subset \mathcal{S}$. Moreover, if $0 \in \mathcal{K}$, then $0 \in \mathcal{S}$. Finally, $\mathcal{S} \subset \partial \mathcal{K} \cup \{0\}$, and $\mathcal{S}$ has finite cardinality.

Now, for $\delta > 0$, we define the set $\mathcal{S}_\delta$ of points in $\mathcal{K}$ at distance at most $\delta$ from $\mathcal{S}$:

$$ \mathcal{S}_\delta := \{x \in \mathcal{K}: \mathrm{dist}(x, \mathcal{S}) \le \delta\}. $$

Since $\mathcal{S}$ is finite, there exists a sufficiently small $\delta_0 > 0$ such that for any $\delta \le \delta_0$, the set $\mathcal{S}_\delta$ has $M := \#\mathcal{S}$ connected components, with any two of these connected components being separated by a distance of at least $\delta_0$. Our goal is to prove that for any $i \in [n]$, and for sufficiently large $t$, the particle $z_i(t)$ remains in one of these connected components. In the sequel, we fix $i \in [n]$.

**Step 2.** $z_i(t)$ must grow if it is not already in $\mathcal{S}_\delta$. We now prove that there exists some $\gamma = \gamma(\mathcal{K}) > 0$ (depending only on the geometry of $\mathcal{K}$) such that for any $\delta \in (0, \delta_0]$, there exists $T(\delta) > 0$ such that if $t \ge T(\delta)$ and $z_i(t) \notin \mathcal{S}_\delta$, then

$$ \frac{d}{dt} \|Az_i(t)\|^2 \ge \gamma\delta. \quad (8.7) $$
Moreover, for any $i \in M(s)$, one has

$$ \frac{d}{dt}\mathrm{dist}(z_i(s), H) \stackrel{(8.4)}{=} \langle \dot{z}_i(s), \mathbf{n} \rangle = \sum_{j=1}^{n} P_{ij}(s)\langle z_j(s) - z_i(s), \mathbf{n} \rangle \ge 0, $$

where the last inequality comes from the fact that each term in the sum is non-negative, since $i \in M(s)$. This proves (8.3) (and, as a byproduct, that $T = +\infty$). $\square$

The following fact immediately ensues.

**Corollary 8.3.** For any $i \in [n]$ and $t \ge 0$, $z_i(t) \in \mathrm{conv}(\{z_i(0)\}_{i \in [n]})$. In particular, $z_i(\cdot)$ is uniformly bounded in time.

### 8.1.2. *Proof of Theorem 8.1.*

*Proof of Theorem 8.1.* As a consequence of Proposition 8.2, the set $\mathrm{conv}(\{z_i(t)\}_{i \in [n]})$ converges as $t \to +\infty$ toward some convex polytope $\mathcal{K}$. In the remainder of the proof, we look to show that the particles $z_i(t)$ can in fact converge only to some well-distinguished points lying on the boundary of this polytope.

**Step 1. The candidate set of limit points.** We denote by $\mathcal{V} = \{v_1, \dots, v_m\}$ the set of vertices of $\mathcal{K}$. Writing any $x \in \mathcal{K}$ as a convex combination of these vertices: $x = \sum_{j=1}^m \alpha_j v_j$ for some weights $\alpha_j \ge 0$ with $\sum_{j=1}^m \alpha_j = 1$, we gather that

$$ \|Ax\|^2 = \left\langle Ax, \sum_{j=1}^{m} \alpha_j Av_j \right\rangle = \sum_{j=1}^{m} \alpha_j \langle Ax, Av_j \rangle \le \max_{j \in [m]} \langle Ax, Av_j \rangle. \quad (8.5) $$

Let $\mathcal{S} \subset \mathcal{K}$ denote the set of points $w \in \mathcal{K}$ such that

$$ \|Aw\|^2 = \max_{j \in [m]} \langle Aw, Av_j \rangle. \quad (8.6) $$

The following holds—we postpone the proof to after that of the theorem.

**Claim 1.** $\mathcal{V} \subset \mathcal{S}$. Moreover, if $0 \in \mathcal{K}$, then $0 \in \mathcal{S}$. Finally, $\mathcal{S} \subset \partial\mathcal{K} \cup \{0\}$, and $\mathcal{S}$ has finite cardinality.

Now, for $\delta > 0$, we define the set $\mathcal{S}_\delta$ of points in $\mathcal{K}$ at distance at most $\delta$ from $\mathcal{S}$:

$$ \mathcal{S}_\delta := \{x \in \mathcal{K}: \mathrm{dist}(x, \mathcal{S}) \le \delta\}. $$

Since $\mathcal{S}$ is finite, there exists a sufficiently small $\delta_0 > 0$ such that for any $\delta \le \delta_0$, the set $\mathcal{S}_\delta$ has $M := \#\mathcal{S}$ connected components, with any two of these connected components being separated by a distance of at least $\delta_0$. Our goal is to prove that for any $i \in [n]$, and for sufficiently large $t$, the particle $z_i(t)$ remains in one of these connected components. In the sequel, we fix $i \in [n]$.

**Step 2.** $z_i(t)$ must grow if it is not already in $\mathcal{S}_\delta$. We now prove that there exists some $\gamma = \gamma(\mathcal{K}) > 0$ (depending only on the geometry of $\mathcal{K}$) such that for any $\delta \in (0, \delta_0]$, there exists $T(\delta) > 0$ such that if $t \ge T(\delta)$ and $z_i(t) \notin \mathcal{S}_\delta$, then

$$ \frac{d}{dt} \|Az_i(t)\|^2 \ge \gamma\delta. \quad (8.7) $$
<figure><img src="image_17.png" /><figcaption>**Figure 9.** An example configuration of the sets $\mathcal{S}$ and $\mathcal{S}_\delta$ in $\mathbb{R}^2$. The set $\mathcal{S}$ consists of all green nodes along the boundary of $\partial\mathcal{K}$, while $\mathcal{S}_\delta$ is the union of all yellow "hemispheres". The latter are pairwise disjoint and are the connected components of $\mathcal{S}_\delta$, which we denote by $\mathcal{C}_k$, for $k \in [M]$.</figcaption></figure>

To this end, we observe that

$$ 
\begin{aligned}
\frac{1}{2} \frac{d}{dt} \|Az_i(t)\|^2 &= \langle A\dot{z}_i(t), Az_i(t) \rangle \\
&= \sum_{j=1}^{n} \left( \frac{e^{\langle Az_i(t), Az_j(t) \rangle e^{2t}}}{\sum_{k=1}^{n} e^{\langle Az_i(t), Az_k(t) \rangle e^{2t}}} \right) \langle A(z_j(t) - z_i(t)), Az_i(t) \rangle \\
&= \sum_{j=1}^{n} \underbrace{\left( \frac{e^{a_j(t)e^{2t}}}{\sum_{k=1}^{n} e^{a_k(t)e^{2t}}} \right)}_{:=b_j(t)} a_j(t) \qquad (8.8)
\end{aligned}
$$

where we have set

$$ a_j(t) := \langle A(z_j(t) - z_i(t)), Az_i(t) \rangle. $$

(To obtain the last equality in (8.8), divide both the numerator and the denominator by $e^{\|Az_i(t)\|^2 e^{2t}}$.) The following holds.

**Claim 2.** There exists some constant $\gamma' = \gamma'(\mathcal{K}) > 0$ depending only on the geometry of $\mathcal{K}$ such that the following holds. Fix $\delta \in (0, \delta_0]$. There exists $T'(\delta) > 0$ such that if $t \ge T'(\delta)$ and $z_i(t) \notin \mathcal{S}_\delta$, then there exists $j \in [n]$ such that $a_j(t) \ge \gamma'\delta$.

We postpone the proof of this claim to after that of the theorem. We seek to use this claim in obtaining a lower bound of $b_j(t)$ for any $j$, whenever $\delta$ is small enough and $t$ is large enough. Since by Corollary 8.3, for any $j \in [n]$, $t \mapsto z_j(t)$ is uniformly bounded on $[0, +\infty)$, we gather that $a_j(\cdot) \in L^\infty(0, +\infty)$. So, we may set

$$ \kappa := \max_{j \in [n]} \sup_{t \ge 0} |a_j(t)|. $$

Let $t \ge 0$ be fixed. We define

$$ B(t) := \{j \in [n] : a_j(t) \ge 0\}. $$
<figure><img src="image_18.png" /><figcaption>**Figure 9.** An example configuration of the sets $\mathcal{S}$ and $\mathcal{S}_\delta$ in $\mathbb{R}^2$. The set $\mathcal{S}$ consists of all green nodes along the boundary of $\partial\mathcal{K}$, while $\mathcal{S}_\delta$ is the union of all yellow "hemispheres". The latter are pairwise disjoint and are the connected components of $\mathcal{S}_\delta$, which we denote by $\mathcal{C}_k$, for $k \in [M]$.</figcaption></figure>

To this end, we observe that

$$ 
\begin{align} 
\frac{1}{2} \frac{d}{dt} \|Az_i(t)\|^2 &= \langle A\dot{z}_i(t), Az_i(t) \rangle \nonumber \\
&= \sum_{j=1}^{n} \left( \frac{e^{\langle Az_i(t), Az_j(t) \rangle e^{2t}}}{\sum_{k=1}^{n} e^{\langle Az_i(t), Az_k(t) \rangle e^{2t}}} \right) \langle A(z_j(t) - z_i(t)), Az_i(t) \rangle \nonumber \\
&= \sum_{j=1}^{n} \underbrace{\left( \frac{e^{a_j(t)e^{2t}}}{\sum_{k=1}^{n} e^{a_k(t)e^{2t}}} \right)}_{:=b_j(t)} a_j(t) \tag{8.8}
\end{align} 
$$

where we have set

$$ a_j(t) := \langle A(z_j(t) - z_i(t)), Az_i(t) \rangle. $$

(To obtain the last equality in (8.8), divide both the numerator and the denominator by $e^{\|Az_i(t)\|^2e^{2t}}$.) The following holds.

**Claim 2.** There exists some constant $\gamma' = \gamma'(\mathcal{K}) > 0$ depending only on the geometry of $\mathcal{K}$ such that the following holds. Fix $\delta \in (0, \delta_0]$. There exists $T'(\delta) > 0$ such that if $t \ge T'(\delta)$ and $z_i(t) \notin \mathcal{S}_\delta$, then there exists $j \in [n]$ such that $a_j(t) \ge \gamma'\delta$.

We postpone the proof of this claim to after that of the theorem. We seek to use this claim in obtaining a lower bound of $b_j(t)$ for any $j$, whenever $\delta$ is small enough and $t$ is large enough. Since by Corollary 8.3, for any $j \in [n]$, $t \mapsto z_j(t)$ is uniformly bounded on $[0, +\infty)$, we gather that $a_j(\cdot) \in L^\infty(0, +\infty)$. So, we may set

$$ \kappa := \max_{j \in [n]} \sup_{t \ge 0} |a_j(t)|. $$

Let $t \ge 0$ be fixed. We define

$$ B(t) := \{j \in [n] : a_j(t) \ge 0\}. $$
We pick an index $j_0(t)$ maximizing $a_j(t)$, namely

$$j_0(t) \in \operatorname{argmax}_{j \in [n]} a_j(t).$$

Observe that $j_0(t) \in B(t)$ since $a_{j_0(t)}(t) \ge a_i(t) = 0$. Clearly

$$b_j(t) \ge 0 \quad \text{for all } j \in B(t). \qquad (8.9)$$

In fact, we also have

$$b_{j_0(t)}(t) \ge \frac{a_{j_0(t)}(t)}{n}. \qquad (8.10)$$

Now suppose that $j \notin B(t)$; since $a_j(t) \ge -\kappa$, and

$$\frac{e^{a_j(t)} e^{2t}}{\sum_{k=1}^{n} e^{a_k(t)} e^{2t}} \le \frac{1}{\sum_{k=1}^{n} e^{a_k(t)} e^{2t}} \le e^{-a_{j_0(t)} e^{2t}},$$

we gather that

$$b_j(t) \ge -\kappa e^{-a_{j_0(t)}(t)} e^{2t} \quad \text{for all } j \in [n] \setminus B(t). \qquad (8.11)$$

Using (8.9), (8.10) and (8.11) in (8.8), we find

$$\frac{1}{2} \frac{d}{dt} \|Az_i(t)\|^2 \ge \frac{a_{j_0(t)}(t)}{n} - \kappa n e^{-a_{j_0(t)}(t)} e^{2t}.$$

The above inequality along with Claim 2 lead us to deduce that there exists $T(\delta) > 0$ (possibly larger than $T'(\delta)$) such that (8.7) holds whenever $t \ge T(\delta)$, with $\gamma = \frac{\gamma'}{2n}$, as desired.

**Step 3.** $z_i(t)$ cannot circulate indefinitely between the connected components of $S_\delta$. Since $z_i \in L^\infty([0, +\infty))$ by Corollary 8.3, from (8.1) we gather that $\dot{z}_i \in L^\infty([0, +\infty))$ as well. And since any two connected components of $S_{\delta_0}$ are separated by a distance at least $\delta_0$, we deduce that it takes a time at least

$$T_0 := \frac{\delta_0}{\|\dot{z}_i\|_{L^\infty([0, +\infty))}}$$

for $z_i$ to go from one connected component of $S_{\delta_0}$ to another one. Fix $\delta \in (0, \delta_0)$ such that

$$\delta < \frac{T_0 \gamma \delta_0}{8R \|A\|_{\text{op}}}, \qquad (8.12)$$

where $R := \max_{j \in [n]} \|z_j\|_{L^\infty([0, +\infty))}$. Denote by

$\mathscr{C}_1, \dots, \mathscr{C}_M$

the connected components of $S_\delta$, each of which being the intersection of $\mathcal{K}$ with a Euclidean ball of radius $\delta$ centered at some point of $S$ (see Fig. 9). For any $k \in [M]$,

$$\sup_{x \in \mathscr{C}_k} \|Ax\|^2 - \inf_{x \in \mathscr{C}_k} \|Ax\|^2 \le 4R \|A\|_{\text{op}} \delta. \qquad (8.13)$$

We introduce the following binary relation on $[M]$:

$$k > \ell \iff \inf_{x \in \mathscr{C}_k} \|Ax\|^2 > \sup_{x \in \mathscr{C}_\ell} \|Ax\|^2,$$

which is transitive. The underlying idea is the following: if $t$ is sufficiently large, and if $z_i$ starts from some connected component $\mathscr{C}_\ell$, then the only connected components $\mathscr{C}_k$ which $z_i$ is able to visit later on are those for which $k > \ell$. This travel of $z_i$
We pick an index $j_0(t)$ maximizing $a_j(t)$, namely

$$j_0(t) \in \operatorname{argmax}_{j \in [n]} a_j(t).$$

Observe that $j_0(t) \in B(t)$ since $a_{j_0(t)}(t) \ge a_i(t) = 0$. Clearly

$$b_j(t) \ge 0 \quad \text{for all } j \in B(t). \qquad (8.9)$$

In fact, we also have

$$b_{j_0(t)}(t) \ge \frac{a_{j_0(t)}(t)}{n}. \qquad (8.10)$$

Now suppose that $j \notin B(t)$; since $a_j(t) \ge -\kappa$, and

$$\frac{e^{a_j(t)} e^{2t}}{\sum_{k=1}^{n} e^{a_k(t)} e^{2t}} \le \frac{1}{\sum_{k=1}^{n} e^{a_k(t)} e^{2t}} \le e^{-a_{j_0(t)} e^{2t}},$$

we gather that

$$b_j(t) \ge -\kappa e^{-a_{j_0(t)}(t)} e^{2t} \quad \text{for all } j \in [n] \setminus B(t). \qquad (8.11)$$

Using (8.9), (8.10) and (8.11) in (8.8), we find

$$\frac{1}{2} \frac{d}{dt} \|Az_i(t)\|^2 \ge \frac{a_{j_0(t)}(t)}{n} - \kappa n e^{-a_{j_0(t)}(t)} e^{2t}.$$

The above inequality along with Claim 2 lead us to deduce that there exists $T(\delta) > 0$ (possibly larger than $T'(\delta)$) such that (8.7) holds whenever $t \ge T(\delta)$, with $\gamma = \frac{\gamma'}{2n}$, as desired.

**Step 3.** $z_i(t)$ cannot circulate indefinitely between the connected components of $S_\delta$. Since $z_i \in L^\infty([0, +\infty))$ by Corollary 8.3, from (8.1) we gather that $\dot{z}_i \in L^\infty([0, +\infty))$ as well. And since any two connected components of $S_{\delta_0}$ are separated by a distance at least $\delta_0$, we deduce that it takes a time at least

$$T_0 := \frac{\delta_0}{\|\dot{z}_i\|_{L^\infty([0, +\infty))}}$$

for $z_i$ to go from one connected component of $S_{\delta_0}$ to another one. Fix $\delta \in (0, \delta_0)$ such that

$$\delta < \frac{T_0 \gamma \delta_0}{8R \|A\|_{\text{op}}}, \qquad (8.12)$$

where $R := \max_{j \in [n]} \|z_j\|_{L^\infty([0, +\infty))}$. Denote by

$\mathcal{C}_1, \dots, \mathcal{C}_M$

the connected components of $S_\delta$, each of which being the intersection of $\mathcal{K}$ with a Euclidean ball of radius $\delta$ centered at some point of $S$ (see Fig. 9). For any $k \in [M]$,

$$\sup_{x \in \mathcal{C}_k} \|Ax\|^2 - \inf_{x \in \mathcal{C}_k} \|Ax\|^2 \le 4R \|A\|_{\text{op}} \delta. \qquad (8.13)$$

We introduce the following binary relation on $[M]$:

$$k > \ell \iff \inf_{x \in \mathcal{C}_k} \|Ax\|^2 > \sup_{x \in \mathcal{C}_\ell} \|Ax\|^2,$$

which is transitive. The underlying idea is the following: if $t$ is sufficiently large, and if $z_i$ starts from some connected component $\mathcal{C}_\ell$, then the only connected components $\mathcal{C}_k$ which $z_i$ is able to visit later on are those for which $k > \ell$. This travel of $z_i$
has to stop after some time since [M] is finite, > is transitive, and for any $\ell$, the relation $\ell > \ell$ does not hold.

Let $T = T(\delta)$ be as in Step 2. Suppose that $t_2 > t_1 \ge T$ and $k_1, k_2 \in [M]$ are distinct and such that $z_i(t_1) \in \mathscr{C}_{k_1}$, $z_i(t_2) \in \mathscr{C}_{k_2}$ and $z_i(t) \notin S_\delta$ for any $t \in (t_1, t_2)$. Per Step 2 (more specifically, (8.7)),

$$ \|Az_i(t_2)\|^2 \ge \|Az_i(t_1)\|^2 + T_0\gamma\delta_0. $$

Therefore using (8.13) twice and since $\delta$ is chosen as in (8.12), we gather that

$$ \begin{aligned} \inf_{x \in \mathscr{C}_{k_2}} \|Ax\|^2 &\ge \|Az_i(t_2)\|^2 - 4R\|A\|_{\text{op}}\delta \ge \|Az_i(t_1)\|^2 + T_0\gamma\delta_0 - 4R\|A\|_{\text{op}}\delta \\ &\ge \inf_{x \in \mathscr{C}_{k_1}} \|Ax\|^2 + T_0\gamma\delta_0 - 4R\|A\|_{\text{op}}\delta \\ &\ge \sup_{x \in \mathscr{C}_{k_1}} \|Ax\|^2 + T_0\gamma\delta_0 - 8R\|A\|_{\text{op}}\delta \quad (8.14) \\ &\ge \sup_{x \in \mathscr{C}_{k_1}} \|Ax\|^2. \end{aligned} $$

Whence $k_2 > k_1$. We therefore deduce that there exist some $T' \ge T$ and $k \in [M]$ such that $z_i(t) \notin S_\delta \setminus \mathscr{C}_k$ for any $t \ge T'$.

**Step 4. Conclusion.** To conclude, it remains to be shown that $z_i(t)$ stays in $\mathscr{C}_k$ for $t$ large enough. For this, in addition to (8.12), we impose

$$ \delta^{\frac{1}{4}} < \frac{\gamma T_0}{8R \|A\|_{\text{op}} \delta_0}. \quad (8.15) $$

For $r > 0$, we denote by $\mathscr{C}_k^r$ the intersection of $\mathcal{K}$ with the closed Euclidean ball of radius $\delta^r$ having the same center as $\mathscr{C}_k$. In particular, $\mathscr{C}_k^1 = \mathscr{C}_k$. If, after time $T'$, $z_i$ travels from $\mathscr{C}_k$ to the complement of $\mathscr{C}_k^{\frac{1}{4}}$, it spends a time at least

$$ \frac{(\delta^{\frac{1}{4}} - \delta^{\frac{1}{2}})}{\|\dot{z}_i\|_{L^\infty([0, +\infty))}} $$

in $\mathscr{C}_k^{\frac{1}{4}} \setminus \mathscr{C}_k^{\frac{1}{2}}$. Per Step 2 (used with $\delta^{\frac{1}{2}}$), $\|Az_i\|^2$ has to increase by at least

$$ \frac{\gamma \delta^{\frac{1}{2}} (\delta^{\frac{1}{4}} - \delta)}{\|\dot{z}_i\|_{L^\infty([0, +\infty))}} \ge \frac{\gamma \delta^{\frac{3}{4}}}{2 \|\dot{z}_i\|_{L^\infty([0, +\infty))}} > 4R \|A\|_{\text{op}} \delta \quad (8.16) $$

during this travel (the last inequality in (8.16) stems from (8.15)). This implies that $z_i$ cannot reenter $\mathscr{C}_k$ after having reached the boundary of $\mathscr{C}_k^{\frac{1}{4}}$, due to (8.13). Thus $z_i(t) \notin S_\delta$ for any sufficiently large $t$, which is impossible due to Step 2 and the uniform boundedness of $t \mapsto \|Az_i(t)\|$. Hence, for sufficiently large $t$, $z_i(t) \in \mathscr{C}_k^{\frac{1}{4}}$. Since $\delta$ may be chosen arbitrarily small, this concludes the proof of Theorem 8.1. $\square$

8.1.3. Proving Claims 1 and 2. We now address the proofs of the two claims which were instrumental in what precedes (along with a sketch of the proof of $\mathcal{V} \subset S$, as implied).

*Proof of Claim 1.* The fact that $0 \in S$ if $0 \in \mathcal{K}$ is immediate. We now show that $S$ is finite and $S \subset \partial\mathcal{K} \cup \{0\}$. Let $w \in S \setminus \{0\}$. As

$$ w = \sum_{j=1}^{m} \alpha_j v_j $$
has to stop after some time since [M] is finite, > is transitive, and for any $\ell$, the relation $\ell > \ell$ does not hold.

Let $T = T(\delta)$ be as in Step 2. Suppose that $t_2 > t_1 \ge T$ and $k_1, k_2 \in [M]$ are distinct and such that $z_i(t_1) \in \mathscr{C}_{k_1}$, $z_i(t_2) \in \mathscr{C}_{k_2}$ and $z_i(t) \notin S_\delta$ for any $t \in (t_1, t_2)$. Per Step 2 (more specifically, (8.7)),

$$ \|Az_i(t_2)\|^2 \ge \|Az_i(t_1)\|^2 + T_0\gamma\delta_0. $$

Therefore using (8.13) twice and since $\delta$ is chosen as in (8.12), we gather that

$$ \begin{aligned} \inf_{x \in \mathscr{C}_{k_2}} \|Ax\|^2 &\ge \|Az_i(t_2)\|^2 - 4R\|A\|_{\text{op}}\delta \ge \|Az_i(t_1)\|^2 + T_0\gamma\delta_0 - 4R\|A\|_{\text{op}}\delta \\ &\ge \inf_{x \in \mathscr{C}_{k_1}} \|Ax\|^2 + T_0\gamma\delta_0 - 4R\|A\|_{\text{op}}\delta \\ &\ge \sup_{x \in \mathscr{C}_{k_1}} \|Ax\|^2 + T_0\gamma\delta_0 - 8R\|A\|_{\text{op}}\delta \quad (8.14) \\ &\ge \sup_{x \in \mathscr{C}_{k_1}} \|Ax\|^2. \end{aligned} $$

Whence $k_2 > k_1$. We therefore deduce that there exist some $T' \ge T$ and $k \in [M]$ such that $z_i(t) \notin S_\delta \setminus \mathscr{C}_k$ for any $t \ge T'$.

**Step 4. Conclusion.** To conclude, it remains to be shown that $z_i(t)$ stays in $\mathscr{C}_k$ for $t$ large enough. For this, in addition to (8.12), we impose

$$ \delta^{\frac{1}{4}} < \frac{\gamma T_0}{8R \|A\|_{\text{op}} \delta_0}. \quad (8.15) $$

For $r > 0$, we denote by $\mathscr{C}_k^r$ the intersection of $\mathcal{K}$ with the closed Euclidean ball of radius $\delta^r$ having the same center as $\mathscr{C}_k$. In particular, $\mathscr{C}_k^1 = \mathscr{C}_k$. If, after time $T'$, $z_i$ travels from $\mathscr{C}_k$ to the complement of $\mathscr{C}_k^{\frac{1}{4}}$, it spends a time at least

$$ \frac{(\delta^{\frac{1}{4}} - \delta^{\frac{1}{2}})}{\|\dot{z}_i\|_{L^\infty([0, +\infty))}} $$

in $\mathscr{C}_k^{\frac{1}{4}} \setminus \mathscr{C}_k^{\frac{1}{2}}$. Per Step 2 (used with $\delta^{\frac{1}{2}}$), $\|Az_i\|^2$ has to increase by at least

$$ \frac{\gamma \delta^{\frac{1}{2}} (\delta^{\frac{1}{4}} - \delta)}{\|\dot{z}_i\|_{L^\infty([0, +\infty))}} \ge \frac{\gamma \delta^{\frac{3}{4}}}{2 \|\dot{z}_i\|_{L^\infty([0, +\infty))}} > 4R \|A\|_{\text{op}} \delta \quad (8.16) $$

during this travel (the last inequality in (8.16) stems from (8.15)). This implies that $z_i$ cannot reenter $\mathscr{C}_k$ after having reached the boundary of $\mathscr{C}_k^{\frac{1}{4}}$, due to (8.13). Thus $z_i(t) \notin S_\delta$ for any sufficiently large $t$, which is impossible due to Step 2 and the uniform boundedness of $t \mapsto \|Az_i(t)\|$. Hence, for sufficiently large $t$, $z_i(t) \in \mathscr{C}_k^{\frac{1}{4}}$. Since $\delta$ may be chosen arbitrarily small, this concludes the proof of Theorem 8.1. $\square$

8.1.3. *Proving Claims 1 and 2.* We now address the proofs of the two claims which were instrumental in what precedes (along with a sketch of the proof of $\mathcal{V} \subset \mathcal{S}$, as implied).

*Proof of Claim 1.* The fact that $0 \in \mathcal{S}$ if $0 \in \mathcal{K}$ is immediate. We now show that $\mathcal{S}$ is finite and $\mathcal{S} \subset \partial\mathcal{K} \cup \{0\}$. Let $w \in \mathcal{S} \setminus \{0\}$. As

$$ w = \sum_{j=1}^{m} \alpha_j v_j $$
for some $\alpha_j \ge 0$ with $\sum_{j=1}^m \alpha_j = 1$, and since (8.6) holds by definition, it follows
that $\alpha_j = 0$ for any $j$ not attaining the maximum in (8.6). Let $l \subset [m]$ denote the
set of all such indices. We have

$$
w = \sum_{j \in l} \alpha_j v_j
$$

with $\|Aw\|^2 = \langle Aw, Av_j \rangle$ for any $j \in l$. Whence $w$ is the orthogonal projection onto $\text{span}\{v_j\}_{j \in l}$ with respect to $\langle A^\cdot, A^\cdot \rangle$. This yields $S \subset \partial K$. Moreover, since for each subset $l \subset [m]$ there exists a unique such projection $w$, $S$ is finite. $\square$

Sketch of proof of $\mathcal{V} \subset \mathcal{S}$. We notice that for any $i \in [n]$ and for $t$ large enough, we have

$$
\dot{z}_i(t) = \sum_{j=1}^{n} \left( \frac{e^{2t} \langle Az_i(t), Az_j(t) \rangle}{\sum_{k=1}^{n} e^{2t} \langle Az_i(t), Az_k(t) \rangle} \right) (z_j(t) - z_i(t)) \quad (8.17)
$$

$$
\approx \sum_{j \in M_i(t)} \left( \frac{e^{2t} \langle Az_i(t), Az_j(t) \rangle}{\sum_{k=1}^{n} e^{2t} \langle Az_i(t), Az_k(t) \rangle} \right) (z_j(t) - z_i(t)), \quad (8.18)
$$

where $M_i(t)$ is the subset of $[n]$ containing all indices $j$ such that

$$
\max_{k \in [n]} \langle Az_i(t), Az_k(t) \rangle - \langle Az_i(t), Az_j(t) \rangle \le e^{-t}
$$

(all other terms in the sum (8.17) are negligible). Due to the convergence of
$\text{conv}(\{z_i(t)\}_{i \in [n]})$ toward $\mathcal{K}$, we also know that for $t$ large enough,

• all the points $z_i(t)$ are contained in a small neighborhood of $\mathcal{K}$,

• near any element of $\mathcal{V}$, there exists some particle $z_i(t)$.

Assume, for the sake of contradiction, that there exists a vertex $v_j \in \mathcal{V}$ such that $v_j \notin S$. Set $\mathcal{C} := \text{conv}(\{v_i\}_{i \in [m]\setminus\{j\}})$. In particular, $\text{dist}(v_j, \mathcal{C}) > 0$ since $v_j$ is a vertex of $\mathcal{K}$. If $l \subset [n]$ denotes the set of indices $i$ such that $z_i(t)$ lies near $v_j$, then $M_i(t) \cap l = \emptyset$ for any $i \in l$, since $v_j \notin S$. For $i \in l$, using (8.18), we find that $\text{dist}(z_i(t), \mathcal{C})$ decays as $t \to +\infty$ as long as $i \notin M_i(t)$—indeed, (8.18) implies that $z_i(t)$ is attracted by $\mathcal{C}$. This implies that $v_j \notin \text{conv}(\{z_k(t')\}_{k \in [n]})$ for $t'$ large enough. This is a contradiction since $\mathcal{K} \subset \text{conv}(\{z_k(t)\}_{k \in [n]})$ for any $t \ge 0$ according to Proposition 8.2. $\square$

Proof of Claim 2. To simplify the notation, we only prove Claim 2 when $A = I_d$.
Assume that $t \ge 0$ and that $z_i(t) \notin S_\delta$.

**First case.** Firstly, we prove the claim in the case where $z_i(t) \notin S_{\delta_0}$. For this, we notice that the function

$$
f : x \mapsto \max_{j \in [n]} \langle v_j, x \rangle - \|x\|^2
$$

is continuous, and by definition of $S$, $f$ is strictly positive on the compact set $K \setminus \text{Int}(S_{\delta_0})$ (the complement in $K$ of the interior of $S_{\delta_0}$). Hence $f(x) \ge c'$ in this set for some constant $c' > 0$. Setting

$$
\mathcal{K}_{\epsilon} := \{ x \in \mathbb{R}^d : \text{dist}(x, \mathcal{K}) \le \epsilon \},
$$

by continuity we find that $f(x) \ge c'/2$ for $x \in \mathcal{K}_\varepsilon \setminus \text{Int}(S_{\delta_0})$ and for sufficiently small $\varepsilon > 0$ (fixed in the sequel). For sufficiently large $t$, we have $z_i(t) \in \mathcal{K}_\varepsilon$ for any
for some $\alpha_j \ge 0$ with $\sum_{j=1}^m \alpha_j = 1$, and since (8.6) holds by definition, it follows
that $\alpha_j = 0$ for any $j$ not attaining the maximum in (8.6). Let $I \subset [m]$ denote the
set of all such indices. We have

$$
w = \sum_{j \in I} \alpha_j v_j
$$

with $\|Aw\|^2 = \langle Aw, Av_j \rangle$ for any $j \in I$. Whence $w$ is the orthogonal projection onto $\text{span}\{v_j\}_{j \in I}$ with respect to $\langle A^\cdot, A^\cdot \rangle$. This yields $S \subset \partial K$. Moreover, since for each subset $I \subset [m]$ there exists a unique such projection $w$, $S$ is finite. $\square$

Sketch of proof of $\mathcal{V} \subset \mathcal{S}$. We notice that for any $i \in [n]$ and for $t$ large enough, we have

$$
\dot{z}_i(t) = \sum_{j=1}^{n} \left( \frac{e^{2t} \langle Az_i(t), Az_j(t) \rangle}{\sum_{k=1}^{n} e^{2t} \langle Az_i(t), Az_k(t) \rangle} \right) (z_j(t) - z_i(t)) \quad (8.17)
$$

$$
\approx \sum_{j \in M_i(t)} \left( \frac{e^{2t} \langle Az_i(t), Az_j(t) \rangle}{\sum_{k=1}^{n} e^{2t} \langle Az_i(t), Az_k(t) \rangle} \right) (z_j(t) - z_i(t)), \quad (8.18)
$$

where $M_i(t)$ is the subset of $[n]$ containing all indices $j$ such that

$$
\max_{k \in [n]} \langle Az_i(t), Az_k(t) \rangle - \langle Az_i(t), Az_j(t) \rangle \le e^{-t}
$$

(all other terms in the sum (8.17) are negligible). Due to the convergence of
$\text{conv}(\{z_i(t)\}_{i \in [n]})$ toward $\mathcal{K}$, we also know that for $t$ large enough,

• all the points $z_i(t)$ are contained in a small neighborhood of $\mathcal{K}$,

• near any element of $\mathcal{V}$, there exists some particle $z_i(t)$.

Assume, for the sake of contradiction, that there exists a vertex $v_j \in \mathcal{V}$ such that $v_j \notin S$. Set $\mathcal{C} := \text{conv}(\{v_i\}_{i \in [m]\setminus\{j\}})$. In particular, $\text{dist}(v_j, \mathcal{C}) > 0$ since $v_j$ is a vertex of $\mathcal{K}$. If $I \subset [n]$ denotes the set of indices $i$ such that $z_i(t)$ lies near $v_j$, then $M_i(t) \cap I = \emptyset$ for any $i \in I$, since $v_j \notin S$. For $i \in I$, using (8.18), we find that $\text{dist}(z_i(t), \mathcal{C})$ decays as $t \to +\infty$ as long as $i \notin M_i(t)$—indeed, (8.18) implies that $z_i(t)$ is attracted by $\mathcal{C}$. This implies that $v_j \notin \text{conv}(\{z_k(t')\}_{k \in [n]})$ for $t'$ large enough. This is a contradiction since $\mathcal{K} \subset \text{conv}(\{z_k(t)\}_{k \in [n]})$ for any $t \ge 0$ according to Proposition 8.2. $\square$

Proof of Claim 2. To simplify the notation, we only prove Claim 2 when $A = I_d$.
Assume that $t \ge 0$ and that $z_i(t) \notin S_\delta$.

**First case.** Firstly, we prove the claim in the case where $z_i(t) \notin S_{\delta_0}$. For this, we notice that the function

$$
f : x \mapsto \max_{j \in [n]} \langle v_j, x \rangle - \|x\|^2
$$

is continuous, and by definition of $S$, $f$ is strictly positive on the compact set $K \setminus \text{Int}(S_{\delta_0})$ (the complement in $K$ of the interior of $S_{\delta_0}$). Hence $f(x) \ge c'$ in this set for some constant $c' > 0$. Setting

$$
\mathcal{K}_{\varepsilon} := \{ x \in \mathbb{R}^d : \text{dist}(x, \mathcal{K}) \le \varepsilon \},
$$

by continuity we find that $f(x) \ge c'/2$ for $x \in \mathcal{K}_{\varepsilon} \setminus \text{Int}(S_{\delta_0})$ and for sufficiently small $\varepsilon > 0$ (fixed in the sequel). For sufficiently large $t$, we have $z_i(t) \in \mathcal{K}_{\varepsilon}$ for any